mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-30 05:16:50 +01:00
fix(server): wake native status reassessment from committed changes (#17709)
## Thinking Path > - Paperclip manages agents, tasks and their durable native results. > - Changed evidence, task authority or review retirement can require a new status assessment. > - Periodic orphan recovery repeatedly reads history to find these changes. > - This change gives reassessment an owner driven by committed changes and saved deadlines. > - Existing result, assessment, decision and task rows remain durable truth. > - The status committer still verifies current sources and authority before writing. > - Settled or insufficient evidence becomes quiet until a supported producer changes it. ## Linked Issues or Issue Description Refs #17702 (merged), #17679 and #17680. This is slice 50, now based on master. It owns native status reassessment; workspace settlement, retained sessions and other orphan duties remain separate. **What happened?** Periodic recovery repeatedly inspects completed native results. Removing that scan without complete writer coverage or current authority checks could strand new evidence or commit an old result over newer work. **Expected behavior** Reassess from existing durable facts after relevant outer transactions commit. Retain real retry/lease deadlines and restart/drain recovery. Preserve current authority, quality-review and supersession checks. Make no recurring SQL when work is settled or dormant. **Steps to reproduce** Complete a native result awaiting evidence or review, leave it idle, then approve a real work product or retire the selected automatic review. Its saved result should advance once when authorized, or record the existing nonterminal business gate, without a periodic scan. ## What Changed - Extract reassessment and its pure status policy from finalization recovery into their owning modules. Add a scoped worker using the existing delivery coordinator and scheduler; no queue table or migration is added. - Retain the three durable discovery triggers: changed work-product history, authoritative task status/version, or retirement of the selected automatic completion review. Preserve selected-source, ownership, effects, retry and lease fences under the existing committer. - Notify from actual task/work-product, claim release, approval, interaction, OAuth, event evidence and workspace-settlement owners before their transaction writes. Keep provider calls outside short transactions and nested writes behind outer commit. - Scope ordinary milestone/finalization/coalesced-input wakes to an issue/run. Keep genuine company authority restoration explicit. Real Photon and coalescer regressions cover all three response/status queues. - Move the current parent's required quality policy into the owner and apply it to every reassessment that could complete a task. An unmet review records the existing in-progress decision and next action, avoiding an operational retry loop. Preserve conversation/session reset locks and the workspace omission-barrier signal. - Skip reassessment in periodic recovery while preserving explicit/startup recovery and the historical supersession guard before independent workspace work. Document the source contract, producer inventory, failures and qualifications in the linked audit. ## Verification - Broad source `9fddb3f0b1157df32b64fb1f22e9d6e9624470e8` passes **2731 distinct cases in 28 complete files**, with no skips: 2458 producer/owner/authority/workspace/quality cases in 394.43 seconds and all 273 governed-response cases in 53.74 seconds. Nineteen dedicated fresh PostgreSQL 17 databases migrate through 0413; eighteen embedded-only invocations preserve fresh-per-call isolation through the existing recovery helper. All 22 temporary imports restore exactly and all 28 hashes match. - Final runtime source `5ea3b0fd016bf7e63ba82a7044825a46a0cd2082` passes **1211 cases in five complete affected files** in 108.23 seconds after the narrowly reproduced evidence-trigger fix: all 43 worker, 145 corpus, 32 finalization-recovery, 968 native-session and 23 quality-review cases. Four fresh databases plus the quality helper's own fresh fixture use PostgreSQL 17. Its one adapted import restores exactly; all five hashes match. The other 27 broad-run test files are unchanged. These qualifications are source-scoped and not cumulatively double-counted. - Required/optional real regressions reproduce retry loops on retired-review and evidence-approval paths before repair; both become SQL quiet afterward with unchanged assertions/time limits. Controlled removal of only ordinary scope guards reproduces two coarse-hint SQL failures; exact guard restoration passes. Real producer, outer transaction, claim, authority, 103-item backlog, retry and drain cases pass. Photon transport is mocked; callbacks, locks and observed SQL are real. - Normal merges preserve 12661 parent/unrelated entries, 35 owned entries and 21 automatic shared results, with five original conflicts reviewed explicitly. The parent's actual squash reconciliation preserves the entire held child tree. The final quality fix changes only its owner and tests. Fourteen added comments are removed with identical nontrivia tokens; no directives are removed and the final comment check is empty. - Database build, server types/build, boundaries and exact-head migration order pass. The database tree is identical to merged #17702: no new SQL, queue table or snapshot is generated. Earlier benchmarks, host limitations, failed assertions and their repairs remain in `doc/plans/2026-10-26-native-status-reassessment-sleep-audit.md` and `doc/idle-native-status-reassessment.md`. - This qualifies targeted external PostgreSQL behavior, not embedded bootstrap on the semaphore-exhausted host, a full local repository suite, deployed sleep or measured fleet savings. Fresh exact-head CI and review remain required. ## Risks - Notifications retain the process-local supported-writer contract. Independent SQL writers and uncoordinated replicas are not covered by a new transport. - Reassessment competes with current authority changes. Existing row/source locks and retries remain required; arbitrary missing identities have no row to lock, so this is not a general predicate-lock guarantee. - Discovery pages 100 rows but retains outstanding candidates in memory. It remains proportional to backlog size, and cancellation occurs between queries. - Other orphan/workspace/session duties and Cloud sleep eligibility remain unchanged. Missing supported producer coverage could strand dormant work, so the producer audit and real callback/transaction tests remain part of acceptance. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell execution and PostgreSQL tests. The runtime does not expose a more specific serving snapshot or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted scope is stated above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
65 files changed
+3639
-511
No files matched your search
@@ -0,0 +1,100 @@
|
||||
# Native status reassessment without idle polling
|
||||
|
||||
Native status reassessment uses committed changes and the shared delivery
|
||||
scheduler. It no longer runs on every periodic orphan-recovery pass.
|
||||
|
||||
## What creates work
|
||||
|
||||
Existing native result, assessment, decision, task and review rows remain the
|
||||
source of truth. No new queue table or database migration is required.
|
||||
|
||||
| Change | Existing producer boundary |
|
||||
| --- | --- |
|
||||
| Work-product creation, update, import, removal, attachment or trusted promotion | Work-product/task services and the promotion transaction |
|
||||
| Task status/version, assignment, execution state or blocker configuration | Task mutation transaction |
|
||||
| Direct claim release, terminal/stale lock cleanup, queue release, recovery or tree restoration | Existing task-scoped claim mutation transaction |
|
||||
| Automatic review retirement | Automatic completion review owner |
|
||||
| Approval/link or interaction outcome, including OAuth card creation/reopening | Approval/review-state notification boundaries |
|
||||
| Accepted/rejected control-plane run evidence | Run-event append transaction |
|
||||
| Workspace barrier or physical ownership settlement | Workspace operation, finalizer and verified recovery owners |
|
||||
| Coordinator claim/retry, effect settlement or run projection | Native finalization and run-milestone boundaries |
|
||||
|
||||
These signals request inspection. They do not grant permission to complete a
|
||||
task. Discovery retains the three existing conditions: changed work-product
|
||||
history, changed authoritative task status/version, or a retired selected
|
||||
completion review. The owner still classifies the selected result and appends
|
||||
assessments and decisions through the existing status committer.
|
||||
|
||||
A settled worker makes no recurring queries. Insufficient evidence is dormant.
|
||||
Real contention or operational failure keeps a retry. A saved retry or lease
|
||||
expiry keeps its actual future deadline. Startup reconstructs candidates in
|
||||
100-row pages; the coordinator processes bounded batches and retains changes
|
||||
through drain/resume and concurrent dispatch. Shutdown uses its existing owner.
|
||||
|
||||
## Authority and separate responsibilities
|
||||
|
||||
Scheduled commit checks the selected source, current task/run, retry and lease,
|
||||
pending effects and native process ownership inside the status transaction.
|
||||
Retired-review completion also checks the latest contract, workspace barrier,
|
||||
physical workspace ownership, governance and dependencies. Evidence is
|
||||
reclassified under source locks. Contention defers instead of blocking an idle
|
||||
worker indefinitely or committing stale completion.
|
||||
|
||||
The legacy reconciler retains explicit recovery and a one-time startup pass.
|
||||
Its periodic call skips reassessment but preserves the superseded historical
|
||||
source guard before independent workspace recovery. Workspace/result settlement,
|
||||
retained session cleanup, native wake delivery and generic orphan recovery are
|
||||
separate duties; their recurring scans have not been removed by this slice.
|
||||
|
||||
Case-handoff delivery and lease changes do not improve evidence: every persisted
|
||||
non-cancelled state already counts as accepted. Cancellation is negative evidence
|
||||
and is fenced by the source lock at scheduled commit. Creation generates the
|
||||
handoff identity inside its transaction before returning it. No reassessment wake
|
||||
is added to every lease renewal. Target-agent status is also not a classification
|
||||
trigger: attention validation checks existence, not whether an agent is invokable.
|
||||
Creating an agent does not satisfy result criteria or verification. Missing
|
||||
arbitrary target identities have no row to lock; this change does not provide a
|
||||
general predicate-lock guarantee for independent identity insertion.
|
||||
|
||||
Pending child completion is a guard and a fresh parent-turn obligation. It is not
|
||||
silently consumed by completing the old result. New contracts supersede the old
|
||||
result's eligibility; creating one does not revive that old result. Direct SQL
|
||||
writers and independent replicas do not receive process-local notifications.
|
||||
The supported producer model remains the active server and integrated transaction
|
||||
owners; this change does not add a distributed notification transport.
|
||||
|
||||
## Evidence and limits
|
||||
|
||||
Current qualification is recorded in the [audit](plans/2026-10-26-native-status-reassessment-sleep-audit.md).
|
||||
The broad 9fddb source passes 2731 cases in 28 complete files. The final 5ea3
|
||||
runtime source, after the evidence-trigger quality fix, passes 1211 cases in five
|
||||
complete affected files. Their overlapping counts are not added together. Real
|
||||
producer/transaction coverage includes first writes, rollback/outer commit,
|
||||
authority races, saved deadlines, drain/resume, claim release and retained history.
|
||||
The final worker suite includes all 43 cases.
|
||||
|
||||
Required quality review remains mandatory on both retired-review and evidence
|
||||
reassessment. An unmet review produces its existing nonterminal decision and
|
||||
review recovery action, rather than repeatedly attempting completion as an
|
||||
operational retry. Real required/optional regressions and controlled coarse-hint
|
||||
negatives fail before repair and pass afterward without changing time limits.
|
||||
|
||||
Historical discovery measurements in the audit retain their original revision:
|
||||
10,000 selected-history tasks return no company candidates in about 35 ms;
|
||||
10,000 real candidates require 101 queries and about 1.56 seconds to discover.
|
||||
This is discovery cost, not complete backlog execution. The current worker still
|
||||
retains candidates in memory proportional to backlog size. Its permanent
|
||||
103-item lifecycle case verifies settlement across batches and SQL quiet.
|
||||
Cancellation is checked between pages, not during an already-running SQL call.
|
||||
|
||||
Database build, server typecheck/build, module boundaries and migration-order
|
||||
checks pass. The host's embedded PostgreSQL bootstrap remains limited by
|
||||
exhausted System V semaphores; acceptance uses fresh owned PostgreSQL 17 databases
|
||||
and preserves embedded-only fresh-per-call isolation through the existing recovery
|
||||
helper. Adapted test imports restore exactly; assertions are unchanged. No full
|
||||
local repository-suite or live external-provider qualification is claimed.
|
||||
|
||||
Remote exact-head CI and review still gate merging. The
|
||||
[stack ledger](plans/2026-10-24-recurring-server-sleep-stack.md) tracks other duties.
|
||||
Worker quiet does not establish database suspension, Cloud alarm coverage, host
|
||||
sleep or fleet cost savings.
|
||||
@@ -2876,3 +2876,179 @@ between pages on drain. Fresh migration and interrupted/concurrent replay pass.
|
||||
The audit preserves the initial query, migration and test-clock failures. Full
|
||||
local tests are running; remote acceptance, older stack findings, master
|
||||
reconciliation and every remaining original duty are still open. Nothing merged.
|
||||
|
||||
|
||||
## Slice 49 publication; slice 50 owner audit
|
||||
|
||||
Draft #17702 at `be6ca6151b` is open on #17696. Greptile reviewed that head
|
||||
at 5/5 with zero unresolved threads. CI remains in progress. The full local
|
||||
suite was stopped after repeated embedded PostgreSQL bootstrap failures and
|
||||
many skipped DB suites. One isolated native-session-executor failure reproduces
|
||||
in startEmbeddedPostgresTestDatabase before recovery behavior runs; logs are
|
||||
/tmp/sleep49-full-tests.log and /tmp/sleep49-full-failure-isolated.log. The
|
||||
255-case real-PostgreSQL owner/startup selection, workspace types/build and
|
||||
migration replay remain passing. This does not make full local acceptance green.
|
||||
|
||||
Slice 50 audits native status reassessment: changed evidence, authoritative task
|
||||
state and retired automatic reviews all feed the existing append-only assessment
|
||||
and decision owner. Its producer/guard inventory is in the dated reassessment
|
||||
audit. No implementation or periodic removal is claimed for slice 50 yet. Earlier
|
||||
stack gates, master reconciliation and the full remaining scope remain open.
|
||||
|
||||
|
||||
Slice 49 remote acceptance: #17702 at
|
||||
`be6ca6151bb04e25f691a830734ddb5856701d27` now has all 53 checks passing,
|
||||
Greptile 5/5 at that exact head and zero unresolved review threads. It remains
|
||||
draft and unmerged. /tmp/sleep49-final-gates.json, /tmp/sleep49-comments.json
|
||||
and /tmp/sleep49-review-threads.json preserve the observations. The PR body
|
||||
records the passing remote gates and keeps the local embedded-database failure
|
||||
visible. Slice 50 and all other remaining original obligations are incomplete.
|
||||
|
||||
|
||||
Slice 50 extraction checkpoint: reassessment and its pure policy now have native
|
||||
runtime owners, with the original reconciler delegating to them. All 94 status
|
||||
corpus cases pass before and after extraction; all 187 governed-response cases
|
||||
and server types pass afterward. No polling duty has been removed. The audit
|
||||
records the critical transaction boundary: scheduled guards belong inside the
|
||||
status committer, which publishes after its own transaction. Producer coverage,
|
||||
worker selection, lifecycle/cost proof and a new PR remain outstanding.
|
||||
|
||||
Slice 50 scheduled-commit checkpoint: nine competing source/lease/deadline/effect/
|
||||
ownership cases reproduce unsafe scheduled reassessment before the new guard.
|
||||
The final 103-case status corpus and server types pass. The audit preserves two
|
||||
initial invalid fixtures and the before-fix constraint rejection. The guard is
|
||||
opt-in at the existing committer transaction; no outer transaction or early
|
||||
publication is added. Worker discovery, producer/lifecycle/cost evidence and
|
||||
periodic removal remain outstanding. No new PR or merge is claimed; all original
|
||||
remaining duties and older stack gates remain open.
|
||||
|
||||
Slice 50 scoped-worker checkpoint: discovery/check/register now use the shared
|
||||
scheduler, with distinct dormant and retryable outcomes and exact saved deadlines.
|
||||
A real retirement missed wake and a near-term deadline delay were reproduced and
|
||||
fixed. Final local verification passes 116 status/worker cases plus 187 governed
|
||||
response consumer cases; server types pass. Driver SQL capture proves quiet after
|
||||
settlement and during drain for the tested path. Initial history cost measurements
|
||||
and their synthetic-fixture limits are recorded in the owner audit. The worker is
|
||||
not registered in startup and periodic reassessment remains. Producer coverage,
|
||||
concurrent authority audit, selected-current/backlog cost and full acceptance are
|
||||
still pending; no slice 50 PR or merge, and the full original scope stays open.
|
||||
|
||||
Slice 50 producer checkpoint: six real work-product/attachment transaction misses
|
||||
and three task mutation misses were reproduced and repaired. The real promotion
|
||||
HTTP route also wakes reassessment. All 13 owner lifecycle tests, 79 affected
|
||||
consumer cases across their documented runs, and server types pass. Initial
|
||||
embedded-PostgreSQL failures and the external-database test repairs are preserved
|
||||
in the owner audit. Approval/interaction/run evidence and workspace producers,
|
||||
concurrent authority, backlog cost, startup wiring and periodic removal remain.
|
||||
Master remains 1fbbef45a8; older stack gates and the full original scope stay open.
|
||||
|
||||
Slice 50 authority checkpoint: two real commit-boundary regressions reproduced
|
||||
stale completion after the source run failed or another run took task execution.
|
||||
The scheduled retired-review path now rechecks both facts under existing locks.
|
||||
All 115 status-corpus cases pass. A retained-history startup assertion failed in
|
||||
the combined run; after waiting for bounded startup batches while enforcing the
|
||||
saved retry deadline, all 28 lifecycle cases and server types pass. The owner
|
||||
audit preserves failures and handoff/contract producer findings. Remaining
|
||||
business-fact races, producer coverage, cost, wiring, periodic removal and PR
|
||||
acceptance are still open. Nothing merged; the full stack goal is incomplete.
|
||||
|
||||
Slice 50 contract checkpoint: a newer committed completion contract reproduced
|
||||
stale completion; a real uncommitted writer reproduced blocking on the task lock.
|
||||
The contract owner now exposes a nonblocking current-revision guard, acquired
|
||||
before status row locks and held through commit. All 145 corpus/lifecycle tests
|
||||
and server types pass. Failure logs and the lock-order rationale are recorded in
|
||||
the owner audit. Workspace/governance races, producer coverage, cost, startup
|
||||
integration, periodic removal, remote acceptance and the rest of the full stack
|
||||
remain open. Nothing merged.
|
||||
|
||||
Slice 50 workspace-barrier checkpoint: three stale-completion regressions now
|
||||
pass after checking and share-locking the current barrier inside status commit.
|
||||
The check uses existing row/FK serialization and NOWAIT contention handling.
|
||||
All 148 status/lifecycle cases and server types pass. The owner audit preserves
|
||||
before-fix failures and distinguishes barrier receipts from physical workspace
|
||||
ownership, which still needs audit. Governance/dependencies, remaining producers,
|
||||
cost, wiring, periodic removal and PR acceptance remain open; nothing merged.
|
||||
|
||||
Slice 50 physical-owner checkpoint: four unsafe-completion regressions and a
|
||||
subsequent release missed wake were reproduced and repaired. Scheduled retired-
|
||||
review reassessment respects the existing physical lock and durable owner receipt;
|
||||
normal/verified owner release wakes it through committed intent. All 217 status,
|
||||
lifecycle and workspace-owner consumer cases and server types pass. Existing
|
||||
orphan/stop-proof safeguards remain verified. Governance/dependencies, remaining
|
||||
producers, cost, integration and remote gates remain open, along with all other
|
||||
original duties and earlier stack findings. Nothing merged.
|
||||
|
||||
Slice 50 governance checkpoint: seven stale-completion races reproduced changed
|
||||
execution policy, approvals, interactions and question history. The governance
|
||||
owner now supplies nonblocking source locks; status commit verifies the gate
|
||||
used by the selected decision. All 159 status/lifecycle cases pass, the valid
|
||||
question-history fixture passes on recheck, and server types pass. Dependency and
|
||||
remaining authority/producer audits, task-history/query costs, wiring, periodic
|
||||
removal and remote gates remain outstanding. The full stack remains incomplete;
|
||||
nothing merged.
|
||||
|
||||
|
||||
Slice 50 dependency checkpoint: seven stale-completion races are reproduced and
|
||||
fixed through the existing dependency-readiness owner. Scheduled status commit
|
||||
locks/rechecks prerequisite and workspace-finalization sources with nonblocking
|
||||
contention handling; ordinary readers retain their behavior. All 181 distinct
|
||||
status/lifecycle/dependency consumer cases and server types pass. The audit keeps
|
||||
the initial unordered assertion and misconfigured embedded-test invocation
|
||||
failures and their repairs. Remaining authority/producer coverage, cost, startup
|
||||
integration, periodic removal, remote gates, earlier stack findings and the full
|
||||
original scope remain open. Nothing merged.
|
||||
|
||||
|
||||
Slice 50 evidence checkpoint: seven races reproduced completion using revoked,
|
||||
removed or concurrently edited evidence. The assessment owner now reclassifies
|
||||
and locks referenced evidence inside scheduled status commit; changed facts and
|
||||
contention defer through existing retry handling. All 177 status/lifecycle/
|
||||
classifier consumer cases and server types pass. The audit preserves before-fix
|
||||
failures and the remaining identity/producer and cost limits. Startup wiring,
|
||||
periodic removal, remote gates, older stack findings and all remaining recurring
|
||||
work are still incomplete. Nothing merged.
|
||||
|
||||
|
||||
Slice 50 OAuth producer checkpoint: authorization-card creation/reopening now
|
||||
signals both native review workers from the same short transaction as its card
|
||||
and state link. Three missing-settlement-signal regressions were reproduced;
|
||||
all 472 tool-access and 30 worker lifecycle tests plus server typecheck pass.
|
||||
The audit retains before/after evidence and the remaining handoff/authority
|
||||
coverage limits. Master is freshly verified at 1fbbef45a8. Discovery/cost proof,
|
||||
startup wiring, periodic removal, remote acceptance, earlier stack findings and
|
||||
the full original recurring-work scope remain incomplete. Nothing merged.
|
||||
|
||||
|
||||
Slice 50 scale checkpoint: 10,000 separate selected completed task/run histories
|
||||
produce zero company candidates (~35 ms); 10,000 actual candidates reconstruct
|
||||
in 101 bounded queries (~1.56 seconds/2.05 MB), with verified drain/abort behavior.
|
||||
A retired-review task with large review/comment/approval/dependency/workspace
|
||||
history settles in ~126 ms and disappears from discovery. The audit preserves
|
||||
two invalid benchmark-fixture attempts and the corrected fresh-task evidence.
|
||||
A permanent 103-item backlog test proves real settlement and SQL quiet; all 31
|
||||
lifecycle cases and server types pass. Remaining producer/authority audit,
|
||||
startup integration, periodic removal, remote gates and the full original stack
|
||||
scope remain open. Nothing merged.
|
||||
|
||||
|
||||
Slice 50 startup checkpoint: enabled maintenance now registers the reassessment
|
||||
worker and periodic recovery skips its classification/decision path. The separate
|
||||
historical supersession guard is preserved before workspace recovery. Two missing
|
||||
integration regressions fail before the change; all 246 corpus/startup/lifecycle
|
||||
cases, 32 existing finalization/cleanup consumer cases, and server types pass after.
|
||||
The owner audit records the intentional separation of reassessment from independent
|
||||
workspace settlement and the retained failure evidence. Remaining producer/authority
|
||||
audit, full acceptance, publication, remote gates and all other original duties
|
||||
remain incomplete. Nothing merged; the goal remains active.
|
||||
|
||||
|
||||
Slice 50 is draft [#17709](https://github.com/paperclipai/paperclip/pull/17709),
|
||||
based on #17702. Initial head 91a6c14170 passed all 52 CI checks; Greptile 4/5
|
||||
found direct claim-release writers missing reassessment notifications. Six real
|
||||
regressions reproduced the gap. Task-scoped signals now cover their existing
|
||||
transactions; the repaired selection passes 978 cases and server types, with
|
||||
15 embedded-DB runtime cases excluded for the diagnosed host semaphore exhaustion.
|
||||
The owner audit preserves the wrong initial ordinary-release completion assertion
|
||||
and its authority-preserving correction. Fresh-head gates remain pending; nothing
|
||||
merged. Earlier E2E failures on #17490/#17572 and voice findings on #17576/#17580
|
||||
were revalidated and remain unresolved. All original remaining duties stay in scope.
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,3 +1,4 @@
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture";
|
||||
@@ -15,6 +16,15 @@ import {
|
||||
} from "../services/heartbeat-run-events.js";
|
||||
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
|
||||
const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_EVENT_SEQUENCING_TEST_DATABASE_URL");
|
||||
async function clearExternalFixture(db: ReturnType<typeof createDb>, companyId: string) {
|
||||
if (!externalUrl) return;
|
||||
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.companyId, companyId));
|
||||
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId));
|
||||
await db.delete(agents).where(eq(agents.companyId, companyId));
|
||||
await db.delete(companies).where(eq(companies.id, companyId));
|
||||
}
|
||||
|
||||
describe("P6-11..13 / P6-17 canonical event allocator", () => {
|
||||
it("keeps native cursor responses bounded with long histories and a late gap fill", async () => {
|
||||
const temporary = await startEmbeddedPostgresTestDatabase("paperclip-native-cursor-budget-");
|
||||
@@ -73,9 +83,9 @@ describe("P6-11..13 / P6-17 canonical event allocator", () => {
|
||||
}, 60_000);
|
||||
|
||||
it.each([false, true])("deduplicates concurrent retry exhaustion (historical receipt: %s)", async (historical) => {
|
||||
const temporary = await startEmbeddedPostgresTestDatabase("paperclip-exhaustion-events-");
|
||||
const db = createDb(temporary.connectionString);
|
||||
const otherDb = createDb(temporary.connectionString);
|
||||
const temporary = externalUrl ? undefined : await startEmbeddedPostgresTestDatabase("paperclip-exhaustion-events-");
|
||||
const db = createDb(externalUrl ?? temporary!.connectionString);
|
||||
const otherDb = createDb(externalUrl ?? temporary!.connectionString);
|
||||
const companyId = "20000000-0000-4000-8000-000000000001";
|
||||
const agentId = "20000000-0000-4000-8000-000000000002";
|
||||
const runId = "20000000-0000-4000-8000-000000000003";
|
||||
@@ -129,13 +139,16 @@ describe("P6-11..13 / P6-17 canonical event allocator", () => {
|
||||
...event, companyId: "20000000-0000-4000-8000-000000000099", retryExhaustion: exhaustion,
|
||||
})).rejects.toThrow("heartbeat_run_event_binding_mismatch");
|
||||
} finally {
|
||||
await temporary.cleanup();
|
||||
await clearExternalFixture(db, companyId);
|
||||
await otherDb.$client.end({ timeout: 1 });
|
||||
await db.$client.end({ timeout: 1 });
|
||||
await temporary?.cleanup();
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
it("serializes concurrent writers and rejects conflicting replay without cursor drift", async () => {
|
||||
const temporary = await startEmbeddedPostgresTestDatabase("paperclip-native-events-");
|
||||
const db = createDb(temporary.connectionString);
|
||||
const temporary = externalUrl ? undefined : await startEmbeddedPostgresTestDatabase("paperclip-native-events-");
|
||||
const db = createDb(externalUrl ?? temporary!.connectionString);
|
||||
const companyId = "20000000-0000-4000-8000-000000000001";
|
||||
const agentId = "20000000-0000-4000-8000-000000000002";
|
||||
const runId = "20000000-0000-4000-8000-000000000003";
|
||||
@@ -219,7 +232,9 @@ describe("P6-11..13 / P6-17 canonical event allocator", () => {
|
||||
})).rejects.toBeInstanceOf(HeartbeatRunEventConflictError);
|
||||
expect(await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, runId))).toHaveLength(48);
|
||||
} finally {
|
||||
await temporary.cleanup();
|
||||
await clearExternalFixture(db, companyId);
|
||||
await db.$client.end({ timeout: 1 });
|
||||
await temporary?.cleanup();
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
|
||||
@@ -57,6 +57,8 @@ import {
|
||||
} from "../services/execution-workspace-policy.ts";
|
||||
import { buildAgentMentionHref, buildProjectMentionHref, MAX_ISSUE_REQUEST_DEPTH, type IssueWorkMode } from "@paperclipai/shared";
|
||||
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
@@ -4111,14 +4113,15 @@ describeEmbeddedPostgres("issueService.create workspace inheritance", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describeEmbeddedPostgres("issueService blockers and dependency wake readiness", () => {
|
||||
const dependencyTestDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_ISSUE_DEPENDENCIES_TEST_DATABASE_URL");
|
||||
(dependencyTestDatabaseUrl ? describe : describeEmbeddedPostgres)("issueService blockers and dependency wake readiness", () => {
|
||||
let db!: ReturnType<typeof createDb>;
|
||||
let svc!: ReturnType<typeof issueService>;
|
||||
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
|
||||
beforeAll(async () => {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-issues-blockers-");
|
||||
db = createDb(tempDb.connectionString);
|
||||
if (!dependencyTestDatabaseUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-issues-blockers-");
|
||||
db = createDb(dependencyTestDatabaseUrl ?? tempDb!.connectionString);
|
||||
svc = issueService(db);
|
||||
await ensureIssueRelationsTable(db);
|
||||
}, 20_000);
|
||||
@@ -4139,6 +4142,7 @@ describeEmbeddedPostgres("issueService blockers and dependency wake readiness",
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await db?.$client.end({ timeout: 5 });
|
||||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { createServer } from "node:http";
|
||||
import express from "express";
|
||||
@@ -55,12 +56,13 @@ import { issueService } from "../services/issues.js";
|
||||
import { heartbeatService } from "../services/heartbeat.js";
|
||||
import { LOW_TRUST_QUARANTINED_BODY } from "../services/source-trust.js";
|
||||
|
||||
const externalTestDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_LOW_TRUST_TEST_DATABASE_URL");
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported
|
||||
const describeEmbeddedPostgres = externalTestDatabaseUrl || embeddedPostgresSupport.supported
|
||||
? describe
|
||||
: describe.skip;
|
||||
|
||||
if (!embeddedPostgresSupport.supported) {
|
||||
if (!externalTestDatabaseUrl && !embeddedPostgresSupport.supported) {
|
||||
console.warn(
|
||||
`Skipping embedded Postgres low-trust route tests on this host: ${
|
||||
embeddedPostgresSupport.reason ?? "unsupported environment"
|
||||
@@ -885,10 +887,10 @@ describeEmbeddedPostgres(
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase(
|
||||
if (!externalTestDatabaseUrl) tempDb = await startEmbeddedPostgresTestDatabase(
|
||||
"paperclip-low-trust-red-team-routes-",
|
||||
);
|
||||
db = createDb(tempDb.connectionString);
|
||||
db = createDb(externalTestDatabaseUrl ?? tempDb!.connectionString);
|
||||
}, 20_000);
|
||||
|
||||
afterEach(async () => {
|
||||
@@ -927,6 +929,7 @@ describeEmbeddedPostgres(
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await db?.$client.end({ timeout: 1 });
|
||||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
|
||||
@@ -1,3 +1,8 @@
|
||||
import { pendingNativeGovernance } from "../services/native-runtime/native-governance.js";
|
||||
import { withNativeWorkspaceFinalizationOwnership } from "../services/native-runtime/native-workspace-finalization-ownership.js";
|
||||
import { ensureNativeCompletionContract } from "../services/native-runtime/completion-contracts.js";
|
||||
import { nativeStatusReassessmentWork } from "../services/native-runtime/native-status-reassessment-work.js";
|
||||
import { reassessNativeStatusFromSource } from "../services/native-runtime/native-status-reassessment.js";
|
||||
import { NativeRunCoordinatorStore } from "../services/native-runtime/native-run-coordinator-store.js";
|
||||
import { registerNativePendingEffectWork } from "../services/native-runtime/native-pending-effect-work.js";
|
||||
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
||||
@@ -9,12 +14,22 @@ import { randomUUID } from "node:crypto";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import {
|
||||
agents,
|
||||
assets,
|
||||
caseHandoffs,
|
||||
cases,
|
||||
heartbeatRunEvents,
|
||||
issueAttachments,
|
||||
approvals,
|
||||
issueApprovals,
|
||||
issueComments,
|
||||
agentWakeupRequests,
|
||||
companies,
|
||||
completionContracts,
|
||||
executionWorkspaces,
|
||||
issueRelations,
|
||||
createDb,
|
||||
heartbeatRuns,
|
||||
issueThreadInteractions,
|
||||
@@ -2451,6 +2466,514 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
|
||||
return seeded;
|
||||
}
|
||||
|
||||
it.each(["future", "claim", "unknown claim", "assessment", "result", "phase", "decision", "effects", "ownership", "run outcome", "task execution", "contract revision", "contract writer", "workspace status", "workspace operation", "workspace writer", "workspace owner", "workspace owner receipt"] as const)(
|
||||
"guards scheduled reassessment against a changed %s at the commit boundary", async changed => {
|
||||
const seeded = await seedAutomaticReview();
|
||||
await dismissAutomaticCompletionReviews(db, seeded.issueId);
|
||||
const [source] = await db.select({
|
||||
runId: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId,
|
||||
issueId: issues.id, issueStatus: issues.status, issueStatusVersion: issues.statusVersion,
|
||||
issueDecisionId: issues.lastStatusDecisionId, coordinatorPhase: nativeRunFinalizations.phase,
|
||||
assessmentId: nativeRunFinalizations.assessmentId, decisionId: nativeRunFinalizations.decisionId,
|
||||
contextSnapshot: heartbeatRuns.contextSnapshot,
|
||||
}).from(nativeRunFinalizations).innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId))
|
||||
.innerJoin(issues, eq(issues.id, nativeRunFinalizations.issueId))
|
||||
.where(eq(nativeRunFinalizations.runId, seeded.runId));
|
||||
const selected = { ...source, assessmentId: source.assessmentId! };
|
||||
const future = new Date(Date.now() + 60_000);
|
||||
const change = changed === "future" ? { nextAttemptAt: future }
|
||||
: changed === "claim" ? { leaseOwner: "other-finalizer", leaseExpiresAt: future }
|
||||
: changed === "unknown claim" ? { leaseOwner: "unknown-finalizer", leaseExpiresAt: null }
|
||||
: changed === "assessment" ? { assessmentId: null, decisionId: null }
|
||||
: changed === "result" ? { resultId: null, assessmentId: null, decisionId: null }
|
||||
: changed === "phase" ? { phase: "terminal_failure" }
|
||||
: changed === "decision" ? { decisionId: null } : null;
|
||||
const [originalRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId));
|
||||
const [originalIssue] = await db.select().from(issues).where(eq(issues.id, seeded.issueId));
|
||||
const [originalBarrier] = await db.select().from(workspaceOperations).where(and(
|
||||
eq(workspaceOperations.heartbeatRunId, seeded.runId), eq(workspaceOperations.phase, "workspace_finalize")));
|
||||
const newBarrierId = randomUUID();
|
||||
const replacementRunId = randomUUID();
|
||||
if (changed === "task execution") await db.insert(heartbeatRuns).values({
|
||||
id: replacementRunId, companyId, agentId, status: "running", runtimeMode: "native",
|
||||
nativeIssueId: seeded.issueId, contextSnapshot: { issueId: seeded.issueId },
|
||||
});
|
||||
const originalEffects = await db.select().from(statusDecisionEffects).where(eq(statusDecisionEffects.decisionId, selected.decisionId!)).orderBy(statusDecisionEffects.ordinal);
|
||||
expect(originalEffects.length).toBeGreaterThan(0);
|
||||
const transaction = db.transaction.bind(db);
|
||||
let newContractId: string | undefined;
|
||||
let releaseWriter: (() => void) | undefined;
|
||||
let writer: Promise<void> | undefined;
|
||||
const writeContract = async (owner: typeof db) => {
|
||||
const saved = await ensureNativeCompletionContract({ db: owner, companyId,
|
||||
issue: { id: seeded.issueId, title: "New direction", description: "Verify the revised objective" }, actorId: agentId });
|
||||
newContractId = saved.row.id;
|
||||
};
|
||||
const intercept = vi.spyOn(db, "transaction").mockImplementationOnce((async (callback, options) => {
|
||||
if (change) await db.update(nativeRunFinalizations).set(change).where(eq(nativeRunFinalizations.runId, seeded.runId));
|
||||
if (changed === "effects") await db.update(statusDecisionEffects).set({ deliveryState: "pending" })
|
||||
.where(eq(statusDecisionEffects.decisionId, selected.decisionId!));
|
||||
if (changed === "ownership") await db.update(heartbeatRuns).set({ status: "running",
|
||||
runtimeMode: "native", errorCode: "native_execution_ownership_unverified", nativePhase: "terminal_failure" })
|
||||
.where(eq(heartbeatRuns.id, seeded.runId));
|
||||
if (changed === "run outcome") await db.update(heartbeatRuns).set({ status: "failed" })
|
||||
.where(eq(heartbeatRuns.id, seeded.runId));
|
||||
if (changed === "task execution") await db.update(issues).set({ executionRunId: replacementRunId })
|
||||
.where(eq(issues.id, seeded.issueId));
|
||||
if (changed === "workspace status") await db.update(workspaceOperations).set({ status: "failed" })
|
||||
.where(eq(workspaceOperations.id, originalBarrier.id));
|
||||
if (changed === "workspace operation") await db.insert(workspaceOperations).values({
|
||||
id: newBarrierId, companyId, heartbeatRunId: seeded.runId, issueId: seeded.issueId,
|
||||
phase: "workspace_finalize", status: "running", cwd: process.cwd(),
|
||||
createdAt: new Date(Math.max(Date.now(), originalBarrier.createdAt.getTime() + 1)),
|
||||
});
|
||||
if (changed === "workspace owner receipt") await db.update(heartbeatRuns).set({
|
||||
runnerProfileJson: { ...originalRun.runnerProfileJson, nativeWorkspaceFinalizationOwner: {
|
||||
token: randomUUID(), hostname: "unverified-controller", pid: 12345, processStartedAt: new Date().toISOString(),
|
||||
} },
|
||||
}).where(eq(heartbeatRuns.id, seeded.runId));
|
||||
if (changed === "workspace owner") {
|
||||
let entered!: () => void;
|
||||
const ready = new Promise<void>(resolve => { entered = resolve; });
|
||||
const released = new Promise<void>(resolve => { releaseWriter = resolve; });
|
||||
writer = withNativeWorkspaceFinalizationOwnership({ db, companyId, runId: seeded.runId }, async () => {
|
||||
entered(); await released;
|
||||
}).then(() => undefined);
|
||||
await Promise.race([ready, writer]);
|
||||
}
|
||||
if (changed === "contract revision") await writeContract(db);
|
||||
if (changed === "contract writer" || changed === "workspace writer") {
|
||||
let writerReady!: () => void;
|
||||
const ready = new Promise<void>(resolve => { writerReady = resolve; });
|
||||
const released = new Promise<void>(resolve => { releaseWriter = resolve; });
|
||||
const rollback = new Error("rollback competing writer");
|
||||
writer = transaction(async tx => {
|
||||
if (changed === "contract writer") await writeContract(tx as unknown as typeof db);
|
||||
else await tx.update(workspaceOperations).set({ status: "failed" })
|
||||
.where(eq(workspaceOperations.id, originalBarrier.id));
|
||||
writerReady();
|
||||
await released;
|
||||
throw rollback;
|
||||
}).catch(error => { if (error !== rollback) throw error; });
|
||||
await Promise.race([ready, writer]);
|
||||
}
|
||||
return transaction(async tx => {
|
||||
if (changed === "contract writer" || changed === "workspace writer") await tx.execute(sql`set local statement_timeout = '2s'`);
|
||||
return callback(tx);
|
||||
}, options);
|
||||
}) as typeof db.transaction);
|
||||
try {
|
||||
expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toEqual({ kind: "deferred", retry: true });
|
||||
} finally { intercept.mockRestore(); releaseWriter?.(); await writer; }
|
||||
expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review");
|
||||
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(1);
|
||||
if (change) expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seeded.runId)))[0]).toMatchObject(change);
|
||||
if (changed === "effects") {
|
||||
expect(await db.select().from(statusDecisionEffects).where(eq(statusDecisionEffects.decisionId, selected.decisionId!)).orderBy(statusDecisionEffects.ordinal))
|
||||
.toEqual(originalEffects.map(effect => ({ ...effect, deliveryState: "pending" })));
|
||||
for (const effect of originalEffects) await db.update(statusDecisionEffects).set({ deliveryState: effect.deliveryState })
|
||||
.where(eq(statusDecisionEffects.id, effect.id));
|
||||
}
|
||||
if (changed === "ownership") {
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)))[0])
|
||||
.toMatchObject({ status: "running", errorCode: "native_execution_ownership_unverified" });
|
||||
await db.update(heartbeatRuns).set({ status: originalRun.status, runtimeMode: originalRun.runtimeMode,
|
||||
errorCode: originalRun.errorCode, nativePhase: originalRun.nativePhase }).where(eq(heartbeatRuns.id, seeded.runId));
|
||||
}
|
||||
if (changed === "run outcome") {
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)))[0].status).toBe("failed");
|
||||
await db.update(heartbeatRuns).set({ status: originalRun.status }).where(eq(heartbeatRuns.id, seeded.runId));
|
||||
}
|
||||
if (changed === "task execution") {
|
||||
expect((await issueService(db).getById(seeded.issueId))!.executionRunId).toBe(replacementRunId);
|
||||
await db.update(issues).set({ executionRunId: originalIssue.executionRunId }).where(eq(issues.id, seeded.issueId));
|
||||
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, replacementRunId));
|
||||
}
|
||||
if (changed === "contract revision") {
|
||||
expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toEqual({ kind: "deferred" });
|
||||
await db.delete(completionContracts).where(eq(completionContracts.id, newContractId!));
|
||||
}
|
||||
if (changed === "contract writer") expect(await db.select().from(completionContracts)
|
||||
.where(eq(completionContracts.id, newContractId!))).toHaveLength(0);
|
||||
if (changed === "workspace status" || changed === "workspace operation") {
|
||||
expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toEqual({ kind: "deferred" });
|
||||
if (changed === "workspace status") await db.update(workspaceOperations).set({ status: originalBarrier.status })
|
||||
.where(eq(workspaceOperations.id, originalBarrier.id));
|
||||
else await db.delete(workspaceOperations).where(eq(workspaceOperations.id, newBarrierId));
|
||||
}
|
||||
if (changed === "workspace writer") expect((await db.select().from(workspaceOperations)
|
||||
.where(eq(workspaceOperations.id, originalBarrier.id)))[0].status).toBe(originalBarrier.status);
|
||||
if (changed === "workspace owner receipt") {
|
||||
expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toEqual({ kind: "deferred" });
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: originalRun.runnerProfileJson })
|
||||
.where(eq(heartbeatRuns.id, seeded.runId));
|
||||
}
|
||||
await db.update(nativeRunFinalizations).set({ phase: selected.coordinatorPhase,
|
||||
resultId: seeded.resultId, assessmentId: selected.assessmentId, decisionId: selected.decisionId,
|
||||
nextAttemptAt: null, leaseOwner: null, leaseExpiresAt: null }).where(eq(nativeRunFinalizations.runId, seeded.runId));
|
||||
expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toMatchObject({ kind: "committed" });
|
||||
expect((await issueService(db).getById(seeded.issueId))!.status).toBe("done");
|
||||
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2);
|
||||
}, 30_000,
|
||||
);
|
||||
|
||||
it.each(["execution policy", "approval link", "approval reopening", "interaction", "approval writer", "interaction writer", "question history writer"] as const)(
|
||||
"fences reassessment against changed governance: %s", async changed => {
|
||||
const seeded = await seedAutomaticReview();
|
||||
await dismissAutomaticCompletionReviews(db, seeded.issueId);
|
||||
const [source] = await db.select({
|
||||
runId: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId,
|
||||
issueId: issues.id, issueStatus: issues.status, issueStatusVersion: issues.statusVersion,
|
||||
issueDecisionId: issues.lastStatusDecisionId, coordinatorPhase: nativeRunFinalizations.phase,
|
||||
assessmentId: nativeRunFinalizations.assessmentId, decisionId: nativeRunFinalizations.decisionId,
|
||||
contextSnapshot: heartbeatRuns.contextSnapshot,
|
||||
}).from(nativeRunFinalizations).innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId))
|
||||
.innerJoin(issues, eq(issues.id, nativeRunFinalizations.issueId))
|
||||
.where(eq(nativeRunFinalizations.runId, seeded.runId));
|
||||
const selected = { ...source, assessmentId: source.assessmentId! };
|
||||
const approvalId = randomUUID(), interactionId = randomUUID(), commentId = randomUUID();
|
||||
const approvalCase = changed.startsWith("approval");
|
||||
if (approvalCase) {
|
||||
await db.insert(approvals).values({ id: approvalId, companyId, type: "approve_ceo_strategy", payload: {},
|
||||
status: changed === "approval link" ? "pending" : "approved" });
|
||||
if (changed !== "approval link") await db.insert(issueApprovals).values({ companyId, issueId: seeded.issueId, approvalId });
|
||||
}
|
||||
if (changed === "interaction writer" || changed === "question history writer") {
|
||||
await db.insert(issueThreadInteractions).values({ id: interactionId, companyId, issueId: seeded.issueId,
|
||||
sourceRunId: seeded.runId, kind: changed === "question history writer" ? "ask_user_questions" : "request_confirmation",
|
||||
status: changed === "question history writer" ? "pending" : "accepted",
|
||||
payload: changed === "question history writer"
|
||||
? { version: 1, questions: [{ id: "review", prompt: "Continue?", selectionMode: "single",
|
||||
options: [{ id: "yes", label: "Yes" }] }] }
|
||||
: { version: 1, prompt: "Review" },
|
||||
createdAt: new Date(Date.now() - 2000) });
|
||||
if (changed === "question history writer") await db.insert(issueComments).values({ id: commentId,
|
||||
companyId, issueId: seeded.issueId, authorUserId: "operator", authorType: "user", body: "Continue with the new direction" });
|
||||
}
|
||||
const scope = { companyId, issueId: seeded.issueId, runId: seeded.runId, executionState: null };
|
||||
expect(await pendingNativeGovernance({ db, ...scope })).toBeNull();
|
||||
const transaction = db.transaction.bind(db);
|
||||
let release!: () => void;
|
||||
let writer: Promise<void> | undefined;
|
||||
const intercept = vi.spyOn(db, "transaction").mockImplementationOnce((async (callback, options) => {
|
||||
if (changed === "execution policy") await db.update(issues).set({ executionState: { status: "pending" } })
|
||||
.where(eq(issues.id, seeded.issueId));
|
||||
if (changed === "approval link") await db.insert(issueApprovals).values({ companyId, issueId: seeded.issueId, approvalId });
|
||||
if (changed === "approval reopening") await db.update(approvals).set({ status: "revision_requested" }).where(eq(approvals.id, approvalId));
|
||||
if (changed === "interaction") await db.insert(issueThreadInteractions).values({ id: interactionId,
|
||||
companyId, issueId: seeded.issueId, sourceRunId: seeded.runId, kind: "request_confirmation",
|
||||
status: "pending", payload: { version: 1, prompt: "Review" } });
|
||||
if (changed.endsWith("writer")) {
|
||||
let entered!: () => void;
|
||||
const ready = new Promise<void>(resolve => { entered = resolve; });
|
||||
const released = new Promise<void>(resolve => { release = resolve; });
|
||||
const rollback = new Error("rollback governance writer");
|
||||
writer = transaction(async tx => {
|
||||
if (changed === "approval writer") await tx.update(approvals).set({ status: "revision_requested" }).where(eq(approvals.id, approvalId));
|
||||
if (changed === "interaction writer") await tx.update(issueThreadInteractions).set({ status: "pending" }).where(eq(issueThreadInteractions.id, interactionId));
|
||||
if (changed === "question history writer") await tx.update(issueComments).set({ deletedAt: new Date() }).where(eq(issueComments.id, commentId));
|
||||
entered(); await released; throw rollback;
|
||||
}).catch(error => { if (error !== rollback) throw error; });
|
||||
await Promise.race([ready, writer]);
|
||||
}
|
||||
return transaction(async tx => {
|
||||
await tx.execute(sql`set local statement_timeout = '2s'`);
|
||||
return callback(tx);
|
||||
}, options);
|
||||
}) as typeof db.transaction);
|
||||
try {
|
||||
expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toEqual({ kind: "deferred", retry: true });
|
||||
} finally { intercept.mockRestore(); release?.(); await writer; }
|
||||
expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review");
|
||||
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(1);
|
||||
if (changed === "execution policy") await db.update(issues).set({ executionState: null }).where(eq(issues.id, seeded.issueId));
|
||||
if (changed === "approval link") await db.delete(issueApprovals).where(eq(issueApprovals.approvalId, approvalId));
|
||||
if (changed === "approval reopening") await db.update(approvals).set({ status: "approved" }).where(eq(approvals.id, approvalId));
|
||||
if (changed === "interaction") await db.delete(issueThreadInteractions).where(eq(issueThreadInteractions.id, interactionId));
|
||||
expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toMatchObject({ kind: "committed" });
|
||||
expect((await issueService(db).getById(seeded.issueId))!.status).toBe("done");
|
||||
}, 30_000,
|
||||
);
|
||||
|
||||
it.each(["new blocker", "reopened blocker", "blocker writer", "workspace operation", "workspace status", "workspace writer", "workspace insert writer"] as const)(
|
||||
"fences reassessment against changed dependency readiness: %s", async changed => {
|
||||
const seeded = await seedAutomaticReview();
|
||||
await dismissAutomaticCompletionReviews(db, seeded.issueId);
|
||||
const [source] = await db.select({
|
||||
runId: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId,
|
||||
issueId: issues.id, issueStatus: issues.status, issueStatusVersion: issues.statusVersion,
|
||||
issueDecisionId: issues.lastStatusDecisionId, coordinatorPhase: nativeRunFinalizations.phase,
|
||||
assessmentId: nativeRunFinalizations.assessmentId, decisionId: nativeRunFinalizations.decisionId,
|
||||
contextSnapshot: heartbeatRuns.contextSnapshot,
|
||||
}).from(nativeRunFinalizations).innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId))
|
||||
.innerJoin(issues, eq(issues.id, nativeRunFinalizations.issueId))
|
||||
.where(eq(nativeRunFinalizations.runId, seeded.runId));
|
||||
const selected = { ...source, assessmentId: source.assessmentId! };
|
||||
const blockerId = randomUUID(), workspaceId = randomUUID(), operationId = randomUUID();
|
||||
const workspaceCase = changed.startsWith("workspace");
|
||||
if (workspaceCase) await db.insert(executionWorkspaces).values({ id: workspaceId, companyId,
|
||||
mode: "isolated", strategyType: "git_worktree", name: "Dependency workspace" });
|
||||
await db.insert(issues).values({ id: blockerId, companyId, title: "Prerequisite",
|
||||
status: changed === "new blocker" ? "todo" : "done", executionWorkspaceId: workspaceCase ? workspaceId : null });
|
||||
if (changed !== "new blocker") await db.insert(issueRelations).values({ companyId, issueId: blockerId, relatedIssueId: seeded.issueId, type: "blocks" });
|
||||
if (changed === "workspace status" || changed === "workspace writer") await db.insert(workspaceOperations).values({
|
||||
id: operationId, companyId, issueId: blockerId, executionWorkspaceId: workspaceId, phase: "workspace_finalize", status: "succeeded" });
|
||||
expect((await issueService(db).getDependencyReadiness(seeded.issueId)).isDependencyReady).toBe(true);
|
||||
const transaction = db.transaction.bind(db);
|
||||
let release!: () => void;
|
||||
let writer: Promise<void> | undefined;
|
||||
const pendingOperation = { id: operationId, companyId, executionWorkspaceId: workspaceId,
|
||||
phase: "workspace_finalize", status: "running" };
|
||||
const intercept = vi.spyOn(db, "transaction").mockImplementationOnce((async (callback, options) => {
|
||||
if (changed === "new blocker") await db.insert(issueRelations).values({ companyId, issueId: blockerId, relatedIssueId: seeded.issueId, type: "blocks" });
|
||||
if (changed === "reopened blocker") await db.update(issues).set({ status: "todo" }).where(eq(issues.id, blockerId));
|
||||
if (changed === "workspace operation") await db.insert(workspaceOperations).values(pendingOperation);
|
||||
if (changed === "workspace status") await db.update(workspaceOperations).set({ status: "failed" }).where(eq(workspaceOperations.id, operationId));
|
||||
if (changed.endsWith("writer")) {
|
||||
let entered!: () => void;
|
||||
const ready = new Promise<void>(resolve => { entered = resolve; });
|
||||
const released = new Promise<void>(resolve => { release = resolve; });
|
||||
const rollback = new Error("rollback dependency writer");
|
||||
writer = transaction(async tx => {
|
||||
if (changed === "blocker writer") await tx.update(issues).set({ status: "todo" }).where(eq(issues.id, blockerId));
|
||||
if (changed === "workspace writer") await tx.update(workspaceOperations).set({ status: "failed" }).where(eq(workspaceOperations.id, operationId));
|
||||
if (changed === "workspace insert writer") await tx.insert(workspaceOperations).values(pendingOperation);
|
||||
entered(); await released; throw rollback;
|
||||
}).catch(error => { if (error !== rollback) throw error; });
|
||||
await Promise.race([ready, writer]);
|
||||
}
|
||||
return transaction(async tx => {
|
||||
await tx.execute(sql`set local statement_timeout = '2s'`);
|
||||
return callback(tx);
|
||||
}, options);
|
||||
}) as typeof db.transaction);
|
||||
try {
|
||||
expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toEqual({ kind: "deferred", retry: true });
|
||||
} finally { intercept.mockRestore(); release?.(); await writer; }
|
||||
expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review");
|
||||
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(1);
|
||||
if (changed === "new blocker" || changed === "reopened blocker") await db.update(issues).set({ status: "done" }).where(eq(issues.id, blockerId));
|
||||
if (changed === "workspace operation" || changed === "workspace status") await db.update(workspaceOperations).set({ status: "succeeded" }).where(eq(workspaceOperations.id, operationId));
|
||||
expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toMatchObject({ kind: "committed" });
|
||||
expect((await issueService(db).getById(seeded.issueId))!.status).toBe("done");
|
||||
}, 30_000,
|
||||
);
|
||||
|
||||
it.each(["work product", "work product writer", "approval", "interaction", "attachment", "event", "handoff"] as const)(
|
||||
"fences reassessment against changed evidence: %s", async kind => {
|
||||
const seeded = await seedAutomaticReview();
|
||||
await dismissAutomaticCompletionReviews(db, seeded.issueId);
|
||||
const evidenceId = randomUUID(), assetId = randomUUID(), caseId = randomUUID();
|
||||
let ref = `work_product:${seeded.workProductId}`;
|
||||
const attachment = { id: evidenceId, companyId, issueId: seeded.issueId, assetId };
|
||||
const event = { companyId, agentId, runId: seeded.runId, seq: 90001, eventType: "evidence.recorded",
|
||||
payload: { prpEvent: { sourceKind: "control_plane", payload: { evidenceOutcome: "accepted" } } } };
|
||||
if (kind === "approval") {
|
||||
ref = `approval:${evidenceId}`;
|
||||
await db.insert(approvals).values({ id: evidenceId, companyId, type: "hire_agent", status: "approved", payload: {} });
|
||||
await db.insert(issueApprovals).values({ companyId, issueId: seeded.issueId, approvalId: evidenceId });
|
||||
}
|
||||
if (kind === "interaction") {
|
||||
ref = `interaction:${evidenceId}`;
|
||||
await db.insert(issueThreadInteractions).values({ id: evidenceId, companyId, issueId: seeded.issueId,
|
||||
kind: "request_confirmation", status: "accepted", payload: { version: 1, prompt: "Evidence accepted" } });
|
||||
}
|
||||
if (kind === "attachment") {
|
||||
ref = `attachment:${evidenceId}`;
|
||||
await db.insert(assets).values({ id: assetId, companyId, provider: "local_disk", objectKey: assetId,
|
||||
contentType: "text/plain", byteSize: 1, sha256: assetId });
|
||||
await db.insert(issueAttachments).values(attachment);
|
||||
}
|
||||
if (kind === "event") { ref = "event:90001"; await db.insert(heartbeatRunEvents).values(event); }
|
||||
if (kind === "handoff") {
|
||||
ref = `case-handoff:${evidenceId}`;
|
||||
const existing = await db.select({ id: cases.id }).from(cases).where(eq(cases.companyId, companyId));
|
||||
await db.insert(cases).values({ id: caseId, companyId, caseNumber: existing.length + 1,
|
||||
identifier: caseId, caseType: "evidence", title: "Handoff evidence" });
|
||||
await db.insert(caseHandoffs).values({ id: evidenceId, companyId, caseId, issueId: seeded.issueId,
|
||||
version: 1, destinationKind: "done", phase: "complete", instructions: "Complete", actorType: "system",
|
||||
actorId: "test", sourceRunId: seeded.runId, wakeupRequestId: randomUUID(), idempotencyKey: evidenceId,
|
||||
inputFingerprint: evidenceId, state: "completed" });
|
||||
}
|
||||
const [storedResult] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, seeded.resultId!));
|
||||
const result = storedResult.resultJson.result as Record<string, unknown>;
|
||||
await db.update(nativeRunResults).set({ resultJson: { ...storedResult.resultJson, result: { ...result,
|
||||
completionClaim: { ...(result.completionClaim as object), criteria: [{ criterionId: "objective", status: "satisfied", evidenceRefs: [ref] }] },
|
||||
verification: [{ commandOrCheck: "evidence", status: "passed", artifactRef: ref }],
|
||||
} } }).where(eq(nativeRunResults.id, seeded.resultId!));
|
||||
const [source] = await db.select({
|
||||
runId: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId,
|
||||
issueId: issues.id, issueStatus: issues.status, issueStatusVersion: issues.statusVersion,
|
||||
issueDecisionId: issues.lastStatusDecisionId, coordinatorPhase: nativeRunFinalizations.phase,
|
||||
assessmentId: nativeRunFinalizations.assessmentId, decisionId: nativeRunFinalizations.decisionId,
|
||||
contextSnapshot: heartbeatRuns.contextSnapshot,
|
||||
}).from(nativeRunFinalizations).innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId))
|
||||
.innerJoin(issues, eq(issues.id, nativeRunFinalizations.issueId)).where(eq(nativeRunFinalizations.runId, seeded.runId));
|
||||
const selected = { ...source, assessmentId: source.assessmentId! };
|
||||
async function change(database: typeof db, revoke: boolean) {
|
||||
if (kind.startsWith("work product")) await database.update(issueWorkProducts)
|
||||
.set({ reviewState: revoke ? "rejected" : "approved" }).where(eq(issueWorkProducts.id, seeded.workProductId));
|
||||
if (kind === "approval") await database.update(approvals).set({ status: revoke ? "rejected" : "approved" }).where(eq(approvals.id, evidenceId));
|
||||
if (kind === "interaction") await database.update(issueThreadInteractions).set({ status: revoke ? "rejected" : "accepted" }).where(eq(issueThreadInteractions.id, evidenceId));
|
||||
if (kind === "handoff") await database.update(caseHandoffs).set({ state: revoke ? "cancelled" : "completed" }).where(eq(caseHandoffs.id, evidenceId));
|
||||
if (kind === "attachment") {
|
||||
if (revoke) await database.delete(issueAttachments).where(eq(issueAttachments.id, evidenceId));
|
||||
else await database.insert(issueAttachments).values(attachment);
|
||||
}
|
||||
if (kind === "event") {
|
||||
if (revoke) await database.delete(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, seeded.runId), eq(heartbeatRunEvents.seq, event.seq)));
|
||||
else await database.insert(heartbeatRunEvents).values(event);
|
||||
}
|
||||
}
|
||||
const transaction = db.transaction.bind(db);
|
||||
let release!: () => void;
|
||||
let writer: Promise<void> | undefined;
|
||||
const intercept = vi.spyOn(db, "transaction").mockImplementationOnce((async (callback, options) => {
|
||||
if (kind === "work product writer") {
|
||||
let entered!: () => void;
|
||||
const ready = new Promise<void>(resolve => { entered = resolve; });
|
||||
const released = new Promise<void>(resolve => { release = resolve; });
|
||||
const rollback = new Error("rollback evidence writer");
|
||||
writer = transaction(async tx => { await change(tx as unknown as typeof db, true); entered(); await released; throw rollback; })
|
||||
.catch(error => { if (error !== rollback) throw error; });
|
||||
await Promise.race([ready, writer]);
|
||||
} else await change(db, true);
|
||||
return transaction(async tx => {
|
||||
await tx.execute(sql`set local statement_timeout = '2s'`);
|
||||
return callback(tx);
|
||||
}, options);
|
||||
}) as typeof db.transaction);
|
||||
try {
|
||||
expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toEqual({ kind: "deferred", retry: true });
|
||||
} finally { intercept.mockRestore(); release?.(); await writer; }
|
||||
expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review");
|
||||
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(1);
|
||||
if (kind !== "work product writer") await change(db, false);
|
||||
expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toMatchObject({ kind: "committed" });
|
||||
expect((await issueService(db).getById(seeded.issueId))!.status).toBe("done");
|
||||
}, 30_000,
|
||||
);
|
||||
|
||||
it("leaves reassessment to its scheduled owner while periodic finalization continues", async () => {
|
||||
const seeded = await seedAutomaticReview();
|
||||
await dismissAutomaticCompletionReviews(db, seeded.issueId);
|
||||
const options = { skipNativeStatusReassessment: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true };
|
||||
const result = await reconcileNativeFinalizations(db, [seeded.runId], options);
|
||||
expect(result).toEqual([expect.objectContaining({ runId: seeded.runId, phase: "committed" })]);
|
||||
expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review");
|
||||
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(1);
|
||||
const work = nativeStatusReassessmentWork(db);
|
||||
const [pending] = await work.load({ companyId, issueId: seeded.issueId, runId: seeded.runId });
|
||||
expect(pending).toBeDefined();
|
||||
expect(await work.check(pending)).toBeNull();
|
||||
expect((await issueService(db).getById(seeded.issueId))!.status).toBe("done");
|
||||
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2);
|
||||
});
|
||||
|
||||
describe("native status reassessment work", () => {
|
||||
const selectedScope = (seeded: { issueId: string; runId: string }) => ({ companyId, issueId: seeded.issueId, runId: seeded.runId });
|
||||
|
||||
it.each(["retired review", "new evidence", "task status"] as const)("discovers and settles %s without revisiting settled history", async trigger => {
|
||||
const seeded = await seedAutomaticReview();
|
||||
const work = nativeStatusReassessmentWork(db), scope = selectedScope(seeded);
|
||||
expect(await work.load(scope)).toEqual([]);
|
||||
if (trigger === "retired review") await dismissAutomaticCompletionReviews(db, seeded.issueId);
|
||||
if (trigger === "new evidence") {
|
||||
const [assessment] = await db.select().from(workAssessments).where(eq(workAssessments.id, seeded.assessmentId));
|
||||
await db.update(workAssessments).set({ createdAt: new Date(Date.now() - 2000),
|
||||
assessmentJson: { ...assessment.assessmentJson, allCriteriaSatisfied: false } }).where(eq(workAssessments.id, seeded.assessmentId));
|
||||
await db.update(issueWorkProducts).set({ updatedAt: new Date() }).where(eq(issueWorkProducts.id, seeded.workProductId));
|
||||
}
|
||||
if (trigger === "task status") await db.update(issues).set({ status: "blocked", statusVersion: 2 }).where(eq(issues.id, seeded.issueId));
|
||||
const pending = await work.load(scope);
|
||||
expect(pending).toMatchObject([{ id: seeded.runId, dueAt: 0 }]);
|
||||
expect(await work.check(pending[0])).toBeNull();
|
||||
expect((await issueService(db).getById(seeded.issueId))!.status).toBe(trigger === "task status" ? "blocked" : "done");
|
||||
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2);
|
||||
expect(await work.load(scope)).toEqual([]);
|
||||
expect(await work.check(pending[0])).toBeNull();
|
||||
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("respects a saved retry and live lease, then reloads the released owner", async () => {
|
||||
const seeded = await seedAutomaticReview();
|
||||
await dismissAutomaticCompletionReviews(db, seeded.issueId);
|
||||
const work = nativeStatusReassessmentWork(db), scope = selectedScope(seeded);
|
||||
const future = new Date(Date.now() + 60_000), lease = new Date(Date.now() + 120_000);
|
||||
await db.update(nativeRunFinalizations).set({ nextAttemptAt: future, leaseOwner: "finalizer", leaseExpiresAt: lease })
|
||||
.where(eq(nativeRunFinalizations.runId, seeded.runId));
|
||||
const [pending] = await work.load(scope);
|
||||
expect(pending.dueAt).toBe(lease.getTime());
|
||||
expect(await work.check(pending)).toEqual(pending);
|
||||
expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review");
|
||||
await db.update(nativeRunFinalizations).set({ nextAttemptAt: null, leaseOwner: null, leaseExpiresAt: null })
|
||||
.where(eq(nativeRunFinalizations.runId, seeded.runId));
|
||||
expect(await work.check(pending)).toBeNull();
|
||||
expect((await issueService(db).getById(seeded.issueId))!.status).toBe("done");
|
||||
});
|
||||
|
||||
it("leaves insufficient evidence dormant until a later change satisfies the contract", async () => {
|
||||
const seeded = await seedAutomaticReview();
|
||||
const work = nativeStatusReassessmentWork(db), scope = selectedScope(seeded);
|
||||
const [assessment] = await db.select().from(workAssessments).where(eq(workAssessments.id, seeded.assessmentId));
|
||||
await db.update(workAssessments).set({ createdAt: new Date(Date.now() - 2000),
|
||||
assessmentJson: { ...assessment.assessmentJson, allCriteriaSatisfied: false } }).where(eq(workAssessments.id, seeded.assessmentId));
|
||||
await db.update(issueWorkProducts).set({ reviewState: "changes_requested", updatedAt: new Date() })
|
||||
.where(eq(issueWorkProducts.id, seeded.workProductId));
|
||||
const [pending] = await work.load(scope);
|
||||
expect(pending).toBeDefined();
|
||||
expect(await work.check(pending)).toBeNull();
|
||||
expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review");
|
||||
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(1);
|
||||
await db.update(issueWorkProducts).set({ reviewState: "approved", updatedAt: new Date() })
|
||||
.where(eq(issueWorkProducts.id, seeded.workProductId));
|
||||
const [changed] = await work.load(scope);
|
||||
expect(await work.check(changed)).toBeNull();
|
||||
expect((await issueService(db).getById(seeded.issueId))!.status).toBe("done");
|
||||
});
|
||||
|
||||
it.each(["unknown lease", "pending effect", "ownership"] as const)("waits for the %s owner to release its obligation", async held => {
|
||||
const seeded = await seedAutomaticReview();
|
||||
await dismissAutomaticCompletionReviews(db, seeded.issueId);
|
||||
const work = nativeStatusReassessmentWork(db), scope = selectedScope(seeded);
|
||||
const [pending] = await work.load(scope);
|
||||
if (held === "unknown lease") await db.update(nativeRunFinalizations).set({ leaseOwner: "unknown", leaseExpiresAt: null })
|
||||
.where(eq(nativeRunFinalizations.runId, seeded.runId));
|
||||
if (held === "pending effect") await db.update(statusDecisionEffects).set({ deliveryState: "pending" })
|
||||
.where(eq(statusDecisionEffects.decisionId, seeded.decision.id));
|
||||
if (held === "ownership") await db.update(heartbeatRuns).set({ status: "running", errorCode: "native_execution_ownership_unverified", nativePhase: "terminal_failure" })
|
||||
.where(eq(heartbeatRuns.id, seeded.runId));
|
||||
expect(await work.load(scope)).toEqual([]);
|
||||
expect(await work.check(pending)).toBeNull();
|
||||
expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review");
|
||||
});
|
||||
|
||||
it("keeps a newly saved near-term deadline after a competing commit", async () => {
|
||||
const seeded = await seedAutomaticReview();
|
||||
await dismissAutomaticCompletionReviews(db, seeded.issueId);
|
||||
const work = nativeStatusReassessmentWork(db), scope = selectedScope(seeded);
|
||||
const [pending] = await work.load(scope);
|
||||
const future = new Date(Date.now() + 5000);
|
||||
const transaction = db.transaction.bind(db);
|
||||
const intercept = vi.spyOn(db, "transaction").mockImplementationOnce((async (callback, options) => {
|
||||
await db.update(nativeRunFinalizations).set({ nextAttemptAt: future }).where(eq(nativeRunFinalizations.runId, seeded.runId));
|
||||
return transaction(callback, options);
|
||||
}) as typeof db.transaction);
|
||||
try { expect(await work.check(pending)).toEqual({ ...pending, dueAt: future.getTime() }); }
|
||||
finally { intercept.mockRestore(); }
|
||||
expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review");
|
||||
});
|
||||
|
||||
it("does not query after discovery was cancelled", async () => {
|
||||
const controller = new AbortController(); controller.abort();
|
||||
const select = vi.spyOn(db, "select");
|
||||
try {
|
||||
await expect(nativeStatusReassessmentWork(db).load(undefined, controller.signal)).rejects.toThrow("discovery stopped");
|
||||
expect(select).not.toHaveBeenCalled();
|
||||
} finally { select.mockRestore(); }
|
||||
});
|
||||
});
|
||||
|
||||
it("retires a proven automatic review and applies the current successful completion exactly once", async () => {
|
||||
const seeded = await seedAutomaticReview();
|
||||
// The persisted result is done, while its old assessment/decision required a review.
|
||||
@@ -3126,7 +3649,7 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
|
||||
]);
|
||||
});
|
||||
|
||||
it("ignores historical committed finalizations superseded by a newer authoritative decision", async () => {
|
||||
it.each([false, true])("ignores superseded historical finalizations with scheduled reassessment %s", async scheduled => {
|
||||
const fixture = corpus.fixtures.find((candidate) => candidate.mode === "native");
|
||||
if (!fixture) throw new Error("native corpus fixture missing");
|
||||
const seeded = await seedFixture(fixture);
|
||||
@@ -3188,7 +3711,7 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
|
||||
decisionId: historicalDecision!.id,
|
||||
}).where(eq(nativeRunFinalizations.runId, seeded.runId));
|
||||
|
||||
await expect(reconcileNativeFinalizations(db, [seeded.runId])).resolves.toEqual([]);
|
||||
await expect(reconcileNativeFinalizations(db, [seeded.runId], { skipNativeStatusReassessment: scheduled })).resolves.toEqual([]);
|
||||
await expect(db.select({
|
||||
decisionId: nativeRunFinalizations.decisionId,
|
||||
}).from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seeded.runId))).resolves.toEqual([
|
||||
|
||||
@@ -0,0 +1,640 @@
|
||||
import { notifyChatRunMilestones } from "../services/chat-run-milestone-notifications.js";
|
||||
import { notifyNativeFinalizationWork } from "../services/native-finalization-notifications.js";
|
||||
import { createPostgresWakeQueueAdapter } from "../modules/wake-queue/adapters/postgres.js";
|
||||
import { withNativeWorkspaceFinalizationOwnership } from "../services/native-runtime/native-workspace-finalization-ownership.js";
|
||||
import express from "express";
|
||||
import { appendHeartbeatRunEvent } from "../services/heartbeat-run-events.js";
|
||||
import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../services/delivery-work-notifications.js";
|
||||
import { mkdtemp, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { workspaceOperationService } from "../services/workspace-operations.js";
|
||||
import request from "supertest";
|
||||
import { LOW_TRUST_REVIEW_PRESET } from "@paperclipai/shared";
|
||||
import { issueRoutes } from "../routes/issues.js";
|
||||
import { errorHandler } from "../middleware/index.js";
|
||||
import { workProductService } from "../services/work-products.js";
|
||||
import { issueService } from "../services/issues.js";
|
||||
import { approvalService } from "../services/approvals.js";
|
||||
import { issueThreadInteractionService } from "../services/issue-thread-interactions.js";
|
||||
import { issueApprovalService } from "../services/issue-approvals.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { agents, companies, completionContracts, createDb, heartbeatRuns, issues, issueWorkProducts, issueThreadInteractions,
|
||||
nativeRunFinalizations, nativeRunResults, statusDecisions, workAssessments, workspaceOperations } from "@paperclipai/db";
|
||||
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
import { nativeStatusReassessmentWork } from "../services/native-runtime/native-status-reassessment-work.js";
|
||||
import { dismissAutomaticCompletionReviews } from "../services/native-runtime/automatic-completion-reviews.js";
|
||||
import { commitNativeStatusDecision } from "../services/native-runtime/status-decision-committer.js";
|
||||
import { NATIVE_STATUS_ARBITER_POLICY_VERSION } from "../services/native-runtime/status-arbiter.js";
|
||||
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
|
||||
describe("native reassessment worker lifecycle", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
|
||||
const companyId = randomUUID(), agentId = randomUUID();
|
||||
let logRoot: string;
|
||||
const priorLogRoot = process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
|
||||
let admitted = true;
|
||||
const errors: unknown[] = [];
|
||||
beforeAll(async () => {
|
||||
logRoot = await mkdtemp(join(tmpdir(), "reassessment-workspace-logs-"));
|
||||
process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = logRoot;
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_STATUS_REASSESSMENT_TEST_DATABASE_URL");
|
||||
if (!external) temporary = await startEmbeddedPostgresTestDatabase("paperclip-status-reassessment-");
|
||||
db = createDb(external ?? temporary!.connectionString);
|
||||
await db.insert(companies).values({ id: companyId, name: "Reassessment lifecycle", issuePrefix: `RS${randomUUID().slice(0, 8)}` });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Reassessment agent", adapterType: "codex_local", status: "idle" });
|
||||
}, 30_000);
|
||||
afterAll(async () => {
|
||||
await db?.$client.end({ timeout: 1 }); await temporary?.cleanup();
|
||||
await rm(logRoot, { recursive: true, force: true });
|
||||
if (priorLogRoot === undefined) delete process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
|
||||
else process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = priorLogRoot;
|
||||
});
|
||||
beforeEach(() => {
|
||||
admitted = true; errors.length = 0;
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
});
|
||||
afterEach(async () => {
|
||||
await coordinator?.stop(); coordinator = undefined;
|
||||
vi.useRealTimers(); vi.restoreAllMocks();
|
||||
expect(errors).toEqual([]);
|
||||
});
|
||||
async function start() {
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => admitted, canReconcile: () => admitted,
|
||||
onError: error => errors.push(error) });
|
||||
await nativeStatusReassessmentWork(db).register(coordinator, () => admitted).ready;
|
||||
}
|
||||
async function flush() {
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
expect(coordinator!.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 15_000 });
|
||||
}
|
||||
async function quiet() {
|
||||
const original = db.$client.options.debug;
|
||||
const queries: string[] = [];
|
||||
db.$client.options.debug = (_id, query) => { queries.push(query); };
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(120_000);
|
||||
await delay(10);
|
||||
expect(queries).toEqual([]);
|
||||
} finally { db.$client.options.debug = original; }
|
||||
}
|
||||
async function status(issueId: string) {
|
||||
return (await db.select({ status: issues.status }).from(issues).where(eq(issues.id, issueId)))[0].status;
|
||||
}
|
||||
async function seedReview(evidenceRef?: string) {
|
||||
const issueId = randomUUID(), runId = randomUUID(), resultId = randomUUID(), contractId = randomUUID();
|
||||
const assessmentId = randomUUID(), workProductId = randomUUID();
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Persisted native result", status: "in_progress", assigneeAgentId: agentId });
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "succeeded", runtimeMode: "native",
|
||||
nativeIssueId: issueId, completionContractId: contractId, contextSnapshot: { issueId }, finishedAt: new Date() });
|
||||
await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 1,
|
||||
schemaVersion: "paperclip.completion-contract.v1", policyVersion: "phase6-v1", risk: "low",
|
||||
completionAuthority: "agent_claim_policy", incompleteCriteriaPolicy: "preserve_non_terminal",
|
||||
contractJson: { revision: "test-v1", criteria: [{ id: "objective", requirement: "Verified artifact" }] },
|
||||
canonicalSha256: contractId, createdByActorType: "system", createdByActorId: "reassessment-test" });
|
||||
await db.insert(nativeRunResults).values({ id: resultId, companyId, issueId, runId, completionContractId: contractId,
|
||||
serverFingerprint: resultId, schemaStatus: "accepted", canonicalSha256: resultId,
|
||||
resultJson: { result: { reportedWorkDisposition: "done", summary: "Completed", attentionRequests: [],
|
||||
completionClaim: { contractRevision: "test-v1", objectiveSatisfied: true, remainingWork: [],
|
||||
criteria: [{ criterionId: "objective", status: "satisfied", evidenceRefs: [evidenceRef ?? `work_product:${workProductId}`] }] },
|
||||
verification: [{ commandOrCheck: "artifact", status: "passed", artifactRef: evidenceRef ?? `work_product:${workProductId}` }],
|
||||
}, terminal: { runTerminalState: "succeeded" } } });
|
||||
await db.insert(issueWorkProducts).values({ id: workProductId, companyId, issueId, type: "artifact", provider: "paperclip",
|
||||
title: "Verified artifact", status: "ready_for_review", reviewState: "approved",
|
||||
createdAt: new Date(Date.now() - 1000), updatedAt: new Date(Date.now() - 1000) });
|
||||
await db.insert(workAssessments).values({ id: assessmentId, companyId, issueId, runId, contractId, resultId,
|
||||
triggerKind: "native_result", triggerActorCompanyId: companyId, priorIssueStatus: "in_progress", priorStatusVersion: 0,
|
||||
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, inputDigest: assessmentId,
|
||||
assessmentJson: { reportedDisposition: "needs_review", attentionRequests: [], allCriteriaSatisfied: false } });
|
||||
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "assessing", resultId, assessmentId });
|
||||
const committed = await commitNativeStatusDecision({ db, companyId, issueId, runId, assessmentId,
|
||||
priorStatus: "in_progress", priorStatusVersion: 0, priorDecisionId: null,
|
||||
decision: { policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, statusAction: "in_review", toStatus: "in_review",
|
||||
reasonCode: "completion_claim_incomplete", unblockDescriptor: null,
|
||||
effects: [{ kind: "bind_reviewer", ownerUserId: null,
|
||||
prompt: "Review the persisted native-run evidence and confirm whether this issue may be completed." }] } });
|
||||
await db.insert(workspaceOperations).values({ companyId, heartbeatRunId: runId, issueId, phase: "workspace_finalize",
|
||||
status: "succeeded", exitCode: 0, cwd: process.cwd(), finishedAt: new Date() });
|
||||
return { issueId, runId, workProductId, decisionId: committed.decision.id };
|
||||
}
|
||||
|
||||
it.each(["milestone", "finalization"] as const)("ignores repeated unscoped routine %s hints without reconstructing dormant reassessment history", async source => {
|
||||
const seeded = await seedReview();
|
||||
const [other] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "failed",
|
||||
runtimeMode: "legacy", contextSnapshot: { issueId: seeded.issueId }, finishedAt: new Date() }).returning();
|
||||
await db.update(issues).set({ executionRunId: other.id }).where(eq(issues.id, seeded.issueId));
|
||||
await dismissAutomaticCompletionReviews(db, seeded.issueId);
|
||||
await start();
|
||||
await flush();
|
||||
await quiet();
|
||||
expect(await nativeStatusReassessmentWork(db).load({ companyId, runId: seeded.runId })).toHaveLength(1);
|
||||
const debug = db.$client.options.debug;
|
||||
const queries: string[] = [];
|
||||
db.$client.options.debug = (_connection, query) => {
|
||||
if (query.startsWith("select") && query.includes("native_run_finalizations")) queries.push(query);
|
||||
};
|
||||
try {
|
||||
for (let i = 0; i < 3; i++) {
|
||||
await db.transaction(tx => source === "milestone" ? notifyChatRunMilestones(tx, { companyId })
|
||||
: notifyNativeFinalizationWork(tx, { companyId }));
|
||||
await flush();
|
||||
}
|
||||
expect(queries).toEqual([]);
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
} finally { db.$client.options.debug = debug; }
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it.each([false, true])("preserves the current required quality-review gate during retired-review reassessment (required=%s)", async required => {
|
||||
const seeded = await seedReview();
|
||||
if (required) await db.update(issues).set({ executionPolicy: { requiredQualityReview: {
|
||||
version: 1, deliverables: [{ key: "output", label: "Verified output" }],
|
||||
} } }).where(eq(issues.id, seeded.issueId));
|
||||
await dismissAutomaticCompletionReviews(db, seeded.issueId);
|
||||
await start();
|
||||
await flush();
|
||||
expect(await status(seeded.issueId)).toBe(required ? "in_progress" : "done");
|
||||
const decisions = await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId));
|
||||
if (required) expect(decisions.find(row => row.id !== seeded.decisionId)?.reasonCode).toBe("required_quality_review_incomplete");
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it.each(["force release", "release", "terminal execution", "terminal checkout", "queue release"] as const)(
|
||||
"wakes dormant reassessment from real %s", async action => {
|
||||
const seeded = await seedReview();
|
||||
const [other] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "failed",
|
||||
runtimeMode: "legacy", contextSnapshot: { issueId: seeded.issueId }, finishedAt: new Date() }).returning();
|
||||
await db.update(issues).set({ executionRunId: other.id, checkoutRunId: other.id }).where(eq(issues.id, seeded.issueId));
|
||||
await dismissAutomaticCompletionReviews(db, seeded.issueId);
|
||||
await start(); await flush(); await quiet();
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
const service = issueService(db);
|
||||
if (action === "force release") await service.adminForceRelease(seeded.issueId);
|
||||
if (action === "release") await service.release(seeded.issueId);
|
||||
if (action === "terminal execution") await service.clearExecutionRunIfTerminal(seeded.issueId);
|
||||
if (action === "terminal checkout") await service.clearCheckoutRunIfTerminal(seeded.issueId);
|
||||
if (action === "queue release") {
|
||||
const adapter = createPostgresWakeQueueAdapter(db, {
|
||||
resolveResponsibleUserId: async () => "reassessment-user",
|
||||
getRoutineEnv: async () => ({ routineId: null, env: null, responsibleUserId: null }),
|
||||
resolveSessionBeforeForWakeup: async () => null,
|
||||
});
|
||||
await adapter.withIssueExecutionLock({ companyId, runId: other.id, now: new Date() },
|
||||
async () => ({ outcome: { kind: "released" }, postCommitEffects: [] }));
|
||||
}
|
||||
await flush();
|
||||
expect(await status(seeded.issueId)).toBe(action === "release" ? "in_review" : "done");
|
||||
const decisions = await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId));
|
||||
expect(decisions).toHaveLength(2);
|
||||
if (action === "release") {
|
||||
expect((await service.getById(seeded.issueId))!.assigneeAgentId).toBeNull();
|
||||
expect(decisions.find(decision => decision.id !== seeded.decisionId)?.reasonCode).toBe("prior_status_terminal_preserved");
|
||||
}
|
||||
await quiet();
|
||||
},
|
||||
);
|
||||
|
||||
it("retains dormant reassessment through outer force-release rollback and commit", async () => {
|
||||
const seeded = await seedReview();
|
||||
const [other] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "failed",
|
||||
runtimeMode: "legacy", contextSnapshot: { issueId: seeded.issueId }, finishedAt: new Date() }).returning();
|
||||
await db.update(issues).set({ executionRunId: other.id }).where(eq(issues.id, seeded.issueId));
|
||||
await dismissAutomaticCompletionReviews(db, seeded.issueId);
|
||||
await start(); await flush(); await quiet();
|
||||
const rollback = new Error("rollback release");
|
||||
await expect(db.transaction(async tx => {
|
||||
await issueService(tx as unknown as typeof db).adminForceRelease(seeded.issueId);
|
||||
await vi.advanceTimersByTimeAsync(2000);
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
throw rollback;
|
||||
})).rejects.toBe(rollback);
|
||||
await flush(); await quiet();
|
||||
expect((await issueService(db).getById(seeded.issueId))!.executionRunId).toBe(other.id);
|
||||
await db.transaction(async tx => {
|
||||
await issueService(tx as unknown as typeof db).adminForceRelease(seeded.issueId);
|
||||
await vi.advanceTimersByTimeAsync(2000);
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
});
|
||||
await flush();
|
||||
expect(await status(seeded.issueId)).toBe("done");
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("bounds discovery by pages and drains a real reassessment backlog to SQL quiet", async () => {
|
||||
const backlog = [];
|
||||
for (let index = 0; index < 103; index++) {
|
||||
const seeded = await seedReview();
|
||||
await dismissAutomaticCompletionReviews(db, seeded.issueId);
|
||||
backlog.push(seeded);
|
||||
}
|
||||
const work = nativeStatusReassessmentWork(db), scope = { companyId };
|
||||
const expected = new Set(backlog.map(row => row.runId));
|
||||
const discovered = await work.load(scope);
|
||||
expect(discovered.filter(row => expected.has(row.id))).toHaveLength(103);
|
||||
const debug = db.$client.options.debug;
|
||||
let queries = 0, admissionChecks = 0;
|
||||
db.$client.options.debug = (_id, query) => {
|
||||
if (query.startsWith("select") && query.includes("native_run_finalizations")) queries++;
|
||||
};
|
||||
try {
|
||||
await expect(work.load(scope, undefined, () => ++admissionChecks <= 1))
|
||||
.rejects.toThrow("Native status reassessment discovery stopped");
|
||||
expect(queries).toBe(1);
|
||||
queries = 0;
|
||||
await expect(work.load(scope, AbortSignal.abort())).rejects.toThrow("Native status reassessment discovery stopped");
|
||||
expect(queries).toBe(0);
|
||||
} finally { db.$client.options.debug = debug; }
|
||||
await start(); await flush();
|
||||
for (const seeded of backlog) {
|
||||
expect(await status(seeded.issueId)).toBe("done");
|
||||
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2);
|
||||
}
|
||||
expect((await work.load(scope)).filter(row => expected.has(row.id))).toEqual([]);
|
||||
await quiet();
|
||||
}, 60_000);
|
||||
|
||||
it("waits quietly for physical workspace ownership to release before reassessment", async () => {
|
||||
const seeded = await seedReview();
|
||||
await dismissAutomaticCompletionReviews(db, seeded.issueId);
|
||||
let enter!: () => void, release!: () => void;
|
||||
const entered = new Promise<void>(resolve => { enter = resolve; });
|
||||
const released = new Promise<void>(resolve => { release = resolve; });
|
||||
const owner = withNativeWorkspaceFinalizationOwnership({ db, companyId, runId: seeded.runId }, async () => {
|
||||
enter(); await released;
|
||||
});
|
||||
try {
|
||||
await Promise.race([entered, owner]);
|
||||
await start(); await flush();
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
await quiet();
|
||||
} finally { release(); await owner; }
|
||||
await flush();
|
||||
expect(await status(seeded.issueId)).toBe("done");
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("keeps an unverified workspace owner dormant despite a successful receipt", async () => {
|
||||
const seeded = await seedReview();
|
||||
await dismissAutomaticCompletionReviews(db, seeded.issueId);
|
||||
const owner = { token: randomUUID(), hostname: "unverified-controller", pid: 12345,
|
||||
processStartedAt: new Date().toISOString() };
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } })
|
||||
.where(eq(heartbeatRuns.id, seeded.runId));
|
||||
await start(); await flush();
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)))[0]
|
||||
.runnerProfileJson?.nativeWorkspaceFinalizationOwner).toEqual(owner);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it.each(["ordinary", "native source"] as const)("wakes from a committed control-plane evidence event through the %s append path", async path => {
|
||||
const seeded = await seedReview("event:1");
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await db.update(issueWorkProducts).set({ updatedAt: new Date() }).where(eq(issueWorkProducts.id, seeded.workProductId));
|
||||
await start(); await flush(); await quiet();
|
||||
const prpEvent = { sourceKind: "control_plane", payload: { evidenceOutcome: " accepted " } };
|
||||
const input = { companyId, agentId, runId: seeded.runId, eventType: "evidence.recorded", payload: { prpEvent },
|
||||
...(path === "native source" ? { nativeSource: { sourceInstanceId: randomUUID(), sourceEventId: randomUUID(),
|
||||
sourceSeq: 1, protocolSchemaVersion: 1, canonicalPayload: prpEvent } } : {}) };
|
||||
expect((await appendHeartbeatRunEvent(db, input)).disposition).toBe("committed");
|
||||
await flush();
|
||||
expect(await status(seeded.issueId)).toBe("done");
|
||||
if (path === "native source") expect((await appendHeartbeatRunEvent(db, input)).disposition).toBe("duplicate");
|
||||
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("preserves event evidence until outer commit and ignores rollback", async () => {
|
||||
const seeded = await seedReview("event:1");
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await db.update(issueWorkProducts).set({ updatedAt: new Date() }).where(eq(issueWorkProducts.id, seeded.workProductId));
|
||||
await start(); await flush(); await quiet();
|
||||
const write = async (tx: typeof db) => {
|
||||
await appendHeartbeatRunEvent(tx, { companyId, agentId, runId: seeded.runId, eventType: "evidence.recorded",
|
||||
payload: { prpEvent: { sourceKind: "control_plane", payload: { evidenceOutcome: "accepted" } } } });
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
};
|
||||
await expect(db.transaction(async tx => {
|
||||
await write(tx as unknown as typeof db); throw new Error("rollback event evidence");
|
||||
})).rejects.toThrow("rollback event evidence");
|
||||
await flush(); await quiet();
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
await db.transaction(tx => write(tx as unknown as typeof db));
|
||||
await flush();
|
||||
expect(await status(seeded.issueId)).toBe("done");
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("keeps ordinary and runner-authored logs from waking reassessment", async () => {
|
||||
const seeded = await seedReview("event:3");
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await db.update(issueWorkProducts).set({ updatedAt: new Date() }).where(eq(issueWorkProducts.id, seeded.workProductId));
|
||||
await start(); await flush(); await quiet();
|
||||
const wakes = vi.fn();
|
||||
const stop = subscribeDeliveryWork(db, DELIVERY_QUEUES.nativeStatusReassessment, wakes);
|
||||
try {
|
||||
await appendHeartbeatRunEvent(db, { companyId, agentId, runId: seeded.runId, eventType: "log", message: "Still working" });
|
||||
await appendHeartbeatRunEvent(db, { companyId, agentId, runId: seeded.runId, eventType: "evidence.recorded",
|
||||
payload: { prpEvent: { sourceKind: "runner", payload: { evidenceOutcome: "accepted" } } } });
|
||||
expect(wakes).not.toHaveBeenCalled();
|
||||
await quiet();
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
await appendHeartbeatRunEvent(db, { companyId, agentId, runId: seeded.runId, eventType: "evidence.recorded",
|
||||
payload: { prpEvent: { sourceKind: "control_plane", payload: { evidenceOutcome: "accepted" } } } });
|
||||
await flush();
|
||||
expect(await status(seeded.issueId)).toBe("done");
|
||||
await quiet();
|
||||
} finally { stop(); }
|
||||
});
|
||||
|
||||
it("wakes when a real workspace finalization settles after a quiet failure", async () => {
|
||||
const seeded = await seedReview();
|
||||
await db.update(workspaceOperations).set({ status: "running" }).where(eq(workspaceOperations.heartbeatRunId, seeded.runId));
|
||||
await dismissAutomaticCompletionReviews(db, seeded.issueId);
|
||||
await start(); await flush(); await quiet();
|
||||
const recorder = workspaceOperationService(db).createRecorder({ companyId, issueId: seeded.issueId, heartbeatRunId: seeded.runId });
|
||||
await expect(recorder.recordOperation({ phase: "workspace_finalize", run: async () => { throw new Error("export interrupted"); } }))
|
||||
.rejects.toThrow("export interrupted");
|
||||
await flush(); await quiet();
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
let release!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; });
|
||||
let started!: () => void;
|
||||
const running = new Promise<void>(resolve => { started = resolve; });
|
||||
const operation = recorder.recordOperation({ phase: "workspace_finalize", run: async () => {
|
||||
started(); await gate; return { status: "succeeded" };
|
||||
} });
|
||||
try {
|
||||
await running; await flush(); await quiet();
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
} finally { release(); await operation; }
|
||||
await flush();
|
||||
expect(await status(seeded.issueId)).toBe("done");
|
||||
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("defers workspace barrier writes until outer commit and ignores rollback", async () => {
|
||||
const seeded = await seedReview();
|
||||
await db.update(workspaceOperations).set({ status: "running" }).where(eq(workspaceOperations.heartbeatRunId, seeded.runId));
|
||||
await dismissAutomaticCompletionReviews(db, seeded.issueId);
|
||||
await start(); await flush(); await quiet();
|
||||
const write = async (tx: typeof db) => {
|
||||
await workspaceOperationService(tx).createRecorder({ companyId, issueId: seeded.issueId, heartbeatRunId: seeded.runId })
|
||||
.recordOperation({ phase: "workspace_finalize", run: async () => ({ status: "succeeded" }) });
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
};
|
||||
await expect(db.transaction(async tx => {
|
||||
await write(tx as unknown as typeof db); throw new Error("rollback workspace barrier");
|
||||
})).rejects.toThrow("rollback workspace barrier");
|
||||
await flush(); await quiet();
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
await db.transaction(tx => write(tx as unknown as typeof db));
|
||||
await flush();
|
||||
expect(await status(seeded.issueId)).toBe("done");
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it.each(["confirmation", "checkbox", "question", "verdict", "partial verdict"] as const)("wakes dormant evidence from real interaction %s", async mutation => {
|
||||
const interactionId = randomUUID();
|
||||
const seeded = await seedReview(`interaction:${interactionId}`);
|
||||
const kind = mutation === "confirmation" ? "request_confirmation" : mutation === "checkbox" ? "request_checkbox_confirmation"
|
||||
: mutation === "question" ? "ask_user_questions" : "request_item_verdicts";
|
||||
const payload = kind === "ask_user_questions"
|
||||
? { version: 1 as const, questions: [{ id: "verified", prompt: "Verified?", selectionMode: "single" as const,
|
||||
options: [{ id: "yes", label: "Yes" }] }] }
|
||||
: kind === "request_item_verdicts"
|
||||
? { version: 1 as const, prompt: "Review evidence", items: [{ id: "artifact", label: "Artifact" },
|
||||
...(mutation === "partial verdict" ? [{ id: "other", label: "Other" }] : [])] }
|
||||
: kind === "request_checkbox_confirmation"
|
||||
? { version: 1 as const, prompt: "Accept evidence", options: [{ id: "verified", label: "Verified" }] }
|
||||
: { version: 1 as const, prompt: "Accept evidence" };
|
||||
await db.insert(issueThreadInteractions).values({ id: interactionId, companyId, issueId: seeded.issueId, kind,
|
||||
sourceRunId: seeded.runId, continuationPolicy: "none", createdByUserId: "board-user", payload });
|
||||
await db.update(issues).set({ createdByUserId: "board-user" }).where(eq(issues.id, seeded.issueId));
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await db.update(issueWorkProducts).set({ updatedAt: new Date() }).where(eq(issueWorkProducts.id, seeded.workProductId));
|
||||
await start(); await flush(); await quiet();
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
const scope = { id: seeded.issueId, companyId, projectId: null, goalId: null };
|
||||
const interactions = issueThreadInteractionService(db);
|
||||
if (mutation === "confirmation" || mutation === "checkbox") {
|
||||
await interactions.acceptInteraction(scope, interactionId, mutation === "checkbox" ? { selectedOptionIds: ["verified"] } : {}, { userId: "board-user" });
|
||||
} else if (mutation === "question") {
|
||||
await interactions.answerQuestions(scope, interactionId, { answers: [{ questionId: "verified", optionIds: ["yes"] }] }, { userId: "board-user" });
|
||||
} else {
|
||||
await interactions.submitItemVerdicts(scope, interactionId, { verdicts: [{ id: "artifact", verdict: "approve" }] }, { userId: "board-user" });
|
||||
}
|
||||
await flush();
|
||||
expect(await status(seeded.issueId)).toBe("done");
|
||||
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it.each(["resolution", "link", "bulk link"] as const)("wakes dormant evidence from real approval %s", async mutation => {
|
||||
const approvalId = randomUUID();
|
||||
const seeded = await seedReview(`approval:${approvalId}`);
|
||||
await approvalService(db).create(companyId, { id: approvalId, type: "approve_ceo_strategy", payload: {},
|
||||
status: mutation === "resolution" ? "pending" : "approved" });
|
||||
if (mutation === "resolution") await issueApprovalService(db).link(seeded.issueId, approvalId);
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await db.update(issueWorkProducts).set({ updatedAt: new Date() }).where(eq(issueWorkProducts.id, seeded.workProductId));
|
||||
await start(); await flush(); await quiet();
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
if (mutation === "resolution") await approvalService(db).approve(approvalId, "board-user");
|
||||
if (mutation === "link") await issueApprovalService(db).link(seeded.issueId, approvalId);
|
||||
if (mutation === "bulk link") await issueApprovalService(db).linkManyForApproval(approvalId, [seeded.issueId]);
|
||||
await flush();
|
||||
expect(await status(seeded.issueId)).toBe("done");
|
||||
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("defers approval evidence until outer commit and ignores rolled-back approval", async () => {
|
||||
const approvalId = randomUUID();
|
||||
const seeded = await seedReview(`approval:${approvalId}`);
|
||||
await approvalService(db).create(companyId, { id: approvalId, type: "approve_ceo_strategy", payload: {} });
|
||||
await issueApprovalService(db).link(seeded.issueId, approvalId);
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await db.update(issueWorkProducts).set({ updatedAt: new Date() }).where(eq(issueWorkProducts.id, seeded.workProductId));
|
||||
await start(); await flush(); await quiet();
|
||||
await expect(db.transaction(async tx => {
|
||||
await approvalService(tx as unknown as typeof db).approve(approvalId, "board-user");
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
throw new Error("rollback approval evidence");
|
||||
})).rejects.toThrow("rollback approval evidence");
|
||||
await flush(); await quiet();
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
await db.transaction(async tx => {
|
||||
await approvalService(tx as unknown as typeof db).approve(approvalId, "board-user");
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
});
|
||||
await flush();
|
||||
expect(await status(seeded.issueId)).toBe("done");
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it.each([false, true])("preserves the required quality-review gate after real evidence approval (required=%s)", async required => {
|
||||
const seeded = await seedReview();
|
||||
await db.update(issueWorkProducts).set({ reviewState: "none" }).where(eq(issueWorkProducts.id, seeded.workProductId));
|
||||
if (required) await db.update(issues).set({ executionPolicy: { requiredQualityReview: {
|
||||
version: 1, deliverables: [{ key: "output", label: "Verified output" }],
|
||||
} } }).where(eq(issues.id, seeded.issueId));
|
||||
await start();
|
||||
await flush();
|
||||
await quiet();
|
||||
await workProductService(db).update(seeded.workProductId, { reviewState: "approved" });
|
||||
await flush();
|
||||
expect(await status(seeded.issueId)).toBe(required ? "in_progress" : "done");
|
||||
const decisions = await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId));
|
||||
if (required) expect(decisions.find(row => row.id !== seeded.decisionId)?.reasonCode).toBe("required_quality_review_incomplete");
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it.each(["update", "create", "primary replacement", "import", "attachment", "source promotion"] as const)("wakes from the real work-product %s writer", async writer => {
|
||||
const seeded = await seedReview();
|
||||
if (writer === "update") await db.update(issueWorkProducts).set({ reviewState: "none" }).where(eq(issueWorkProducts.id, seeded.workProductId));
|
||||
if (writer === "create") await db.delete(issueWorkProducts).where(eq(issueWorkProducts.id, seeded.workProductId));
|
||||
if (writer === "source promotion") await db.update(issueWorkProducts).set({ sourceTrust: {
|
||||
preset: LOW_TRUST_REVIEW_PRESET, disposition: "quarantined", sourceIssueId: seeded.issueId,
|
||||
sourceRunId: seeded.runId, sourceAgentId: agentId,
|
||||
} }).where(eq(issueWorkProducts.id, seeded.workProductId));
|
||||
await start(); await flush(); await quiet();
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
const products = workProductService(db);
|
||||
if (writer === "update") await products.update(seeded.workProductId, { reviewState: "approved" });
|
||||
if (writer === "create" || writer === "primary replacement") await products.createForIssue(seeded.issueId, companyId, {
|
||||
...(writer === "create" ? { id: seeded.workProductId } : {}),
|
||||
type: "artifact", provider: "paperclip", title: "Verified artifact", status: "approved", reviewState: "approved",
|
||||
isPrimary: writer === "primary replacement",
|
||||
});
|
||||
if (writer === "import") await products.createManyForImport([{ companyId, issueId: seeded.issueId, projectId: null,
|
||||
type: "artifact", provider: "paperclip", externalId: null, title: "Imported evidence", url: null, status: "approved",
|
||||
reviewState: "approved", isPrimary: false, healthStatus: "unknown", summary: null, metadata: null,
|
||||
executionWorkspaceId: null, runtimeServiceId: null, createdByRunId: null, sourceTrust: null }]);
|
||||
if (writer === "attachment") await issueService(db).createAttachment({ issueId: seeded.issueId, provider: "local_disk",
|
||||
objectKey: randomUUID(), contentType: "text/plain", byteSize: 4, sha256: "a".repeat(64), originalFilename: "result.txt",
|
||||
createdByAgentId: agentId, createdByRunId: seeded.runId });
|
||||
if (writer === "source promotion") {
|
||||
const app = express(); app.use(express.json());
|
||||
app.use((req, _res, next) => {
|
||||
req.actor = { type: "board", userId: "board-user", companyIds: [companyId],
|
||||
memberships: [{ companyId, membershipRole: "operator", status: "active" }],
|
||||
isInstanceAdmin: true, source: "local_implicit" }; next();
|
||||
});
|
||||
app.use("/api", issueRoutes(db, {} as any)); app.use(errorHandler);
|
||||
const promoted = await request(app).post(`/api/issues/${seeded.issueId}/low-trust/promotions`).send({
|
||||
sourceArtifactKind: "work_product", sourceArtifactId: seeded.workProductId,
|
||||
title: "Reviewed artifact", summary: "Reviewed completion evidence",
|
||||
});
|
||||
expect(promoted.status, JSON.stringify(promoted.body)).toBe(201);
|
||||
}
|
||||
await flush();
|
||||
expect(await status(seeded.issueId)).toBe("done");
|
||||
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("waits for an outer work-product transaction and ignores rollback", async () => {
|
||||
const seeded = await seedReview();
|
||||
await db.update(issueWorkProducts).set({ reviewState: "none" }).where(eq(issueWorkProducts.id, seeded.workProductId));
|
||||
await start(); await flush(); await quiet();
|
||||
await expect(db.transaction(async tx => {
|
||||
await workProductService(tx as unknown as typeof db).update(seeded.workProductId, { reviewState: "approved" });
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
throw new Error("rollback evidence approval");
|
||||
})).rejects.toThrow("rollback evidence approval");
|
||||
await flush(); await quiet();
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
await db.transaction(async tx => {
|
||||
await workProductService(tx as unknown as typeof db).update(seeded.workProductId, { reviewState: "approved" });
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
});
|
||||
await flush();
|
||||
expect(await status(seeded.issueId)).toBe("done");
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it.each(["status", "blocked reassertion", "assignment"] as const)("reassesses a real task %s mutation after quiet", async mutation => {
|
||||
const seeded = await seedReview();
|
||||
if (mutation === "blocked reassertion") await issueService(db).update(seeded.issueId, { status: "blocked" });
|
||||
await start(); await flush(); await quiet();
|
||||
const before = await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId));
|
||||
await issueService(db).update(seeded.issueId, mutation === "assignment" ? { assigneeAgentId: null } : { status: "blocked" });
|
||||
await flush();
|
||||
const after = await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId));
|
||||
expect(after).toHaveLength(before.length + 1);
|
||||
expect(await status(seeded.issueId)).toBe(mutation === "assignment" ? "in_review" : "blocked");
|
||||
const [current] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seeded.runId));
|
||||
const [assessment] = await db.select().from(workAssessments).where(eq(workAssessments.id, current.assessmentId!));
|
||||
const [issue] = await db.select().from(issues).where(eq(issues.id, seeded.issueId));
|
||||
expect(assessment.priorStatusVersion).toBe(issue.statusVersion);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("retains the first review-retirement wake through idle drain", async () => {
|
||||
const seeded = await seedReview();
|
||||
await start(); await flush(); await quiet();
|
||||
admitted = false;
|
||||
await dismissAutomaticCompletionReviews(db, seeded.issueId);
|
||||
await quiet();
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
admitted = true;
|
||||
await vi.advanceTimersByTimeAsync(30_000); await flush();
|
||||
expect(await status(seeded.issueId)).toBe("done");
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("reconstructs the saved retry after shutdown and never dispatches it early", async () => {
|
||||
const seeded = await seedReview();
|
||||
await dismissAutomaticCompletionReviews(db, seeded.issueId);
|
||||
const future = new Date(Date.now() + 60_000);
|
||||
await db.update(nativeRunFinalizations).set({ nextAttemptAt: future }).where(eq(nativeRunFinalizations.runId, seeded.runId));
|
||||
const waitForSavedDeadline = async () => {
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
expect(coordinator!.nextWakeAt()).toBe(future.getTime());
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
}, { timeout: 15_000 });
|
||||
};
|
||||
await start();
|
||||
await waitForSavedDeadline();
|
||||
await vi.advanceTimersByTimeAsync(20_000);
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
await coordinator!.stop(); coordinator = undefined;
|
||||
await start();
|
||||
await waitForSavedDeadline();
|
||||
await vi.advanceTimersByTimeAsync(Math.max(0, future.getTime() - Date.now())); await flush();
|
||||
expect(await status(seeded.issueId)).toBe("done");
|
||||
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("wakes from real review retirement after idle and becomes SQL quiet with completed history", async () => {
|
||||
const seeded = await seedReview();
|
||||
await start(); await flush(); await quiet();
|
||||
expect(await status(seeded.issueId)).toBe("in_review");
|
||||
await dismissAutomaticCompletionReviews(db, seeded.issueId);
|
||||
await flush();
|
||||
expect(await status(seeded.issueId)).toBe("done");
|
||||
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2);
|
||||
await quiet();
|
||||
});
|
||||
});
|
||||
@@ -1,3 +1,4 @@
|
||||
import { nativeStatusReassessmentWork } from "../../services/native-runtime/native-status-reassessment-work.js";
|
||||
import { nativeReviewResponseWork } from "../../services/native-runtime/native-review-response-work.js";
|
||||
import { nativeResponsePresentationWork } from "../../services/native-runtime/native-response-presentation-work.js";
|
||||
import { DELIVERY_QUEUES } from "../../services/delivery-work-notifications.js";
|
||||
@@ -377,7 +378,7 @@ describe("iMessage Photon channel control plane", () => {
|
||||
},
|
||||
};
|
||||
}
|
||||
it.each([DELIVERY_QUEUES.nativeResponsePresentation, DELIVERY_QUEUES.nativeReviewResponse])(
|
||||
it.each([DELIVERY_QUEUES.nativeResponsePresentation, DELIVERY_QUEUES.nativeReviewResponse, DELIVERY_QUEUES.nativeStatusReassessment])(
|
||||
"keeps repeated active Photon ownership assertions out of %s reconstruction and wakes once on recovery", async queue => {
|
||||
const t = await setup();
|
||||
await t.start();
|
||||
@@ -385,7 +386,8 @@ describe("iMessage Photon channel control plane", () => {
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
|
||||
coordinators.push(coordinator);
|
||||
const work = queue === DELIVERY_QUEUES.nativeResponsePresentation ? nativeResponsePresentationWork(db) : nativeReviewResponseWork(db);
|
||||
const work = queue === DELIVERY_QUEUES.nativeResponsePresentation ? nativeResponsePresentationWork(db)
|
||||
: queue === DELIVERY_QUEUES.nativeReviewResponse ? nativeReviewResponseWork(db) : nativeStatusReassessmentWork(db);
|
||||
const worker = work.register(coordinator, () => true);
|
||||
const debug = db.$client.options.debug;
|
||||
const hints: unknown[] = [];
|
||||
|
||||
@@ -116,6 +116,7 @@ const {
|
||||
reapOrphanedRuns: vi.fn(async () => ({ reaped: 0, runIds: [] })),
|
||||
registerScheduledRetryCancellationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerStoppedInstructionCollectionWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerNativeStatusReassessmentWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerNativeReviewResponseWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerNativeResponsePresentationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve(), stop: vi.fn() })),
|
||||
registerNativePendingEffectWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
@@ -571,6 +572,13 @@ describe("startServer feedback export wiring", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it.each([false, true])("registers native reassessment only when maintenance is enabled (%s)", async enabled => {
|
||||
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
|
||||
await startServer(); await settleStartupHousekeeping();
|
||||
expect(heartbeatServiceMock.registerNativeStatusReassessmentWork).toHaveBeenCalledTimes(enabled ? 1 : 0);
|
||||
if (enabled) expect(heartbeatServiceMock.registerNativeStatusReassessmentWork.mock.calls[0][1]()).toBe(true);
|
||||
});
|
||||
|
||||
it.each([false, true])("registers governed-response recovery only when maintenance is enabled (%s)", async enabled => {
|
||||
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
|
||||
await startServer(); await settleStartupHousekeeping();
|
||||
@@ -1025,7 +1033,7 @@ describe("startServer feedback export wiring", () => {
|
||||
intervalCallback?.();
|
||||
await new Promise<void>((resolve) => setImmediate(resolve));
|
||||
expect(heartbeatServiceMock.sweepStaleIssueLocks).toHaveBeenCalledTimes(2);
|
||||
expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true });
|
||||
expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true });
|
||||
expect(heartbeatServiceMock.reconcileCostAccounting).toHaveBeenCalledTimes(1);
|
||||
expect(retiredDetector).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../services/delivery-work-notifications.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import * as fs from "node:fs/promises";
|
||||
import { createHash, generateKeyPairSync, randomUUID } from "node:crypto";
|
||||
@@ -10088,6 +10089,45 @@ describeEmbeddedPostgres("tool access service", () => {
|
||||
|
||||
});
|
||||
|
||||
it.each(["creation", "reopening", "rollback"] as const)("signals native review recovery after OAuth card %s settles", async mode => {
|
||||
vi.stubEnv("PAPERCLIP_TOOL_OAUTH_SLACK_CLIENT_ID", "slack-client-id");
|
||||
vi.stubEnv("PAPERCLIP_TOOL_OAUTH_SLACK_CLIENT_SECRET", "slack-client-secret");
|
||||
const company = await createCompany(db);
|
||||
const agent = await createAgent(db, company.id);
|
||||
const { issue, run } = await createIssueAndRun(db, company.id, agent.id);
|
||||
const connected = await createTestToolAccessService(db).connectGalleryApp(company.id, { galleryKey: "slack", name: "Slack user auth" });
|
||||
if (mode !== "creation") await db.insert(issueThreadInteractions).values({ companyId: company.id,
|
||||
issueId: issue.id, kind: "request_confirmation", status: "accepted", payload: { version: 1, prompt: "Previous authorization" },
|
||||
idempotencyKey: `connection-authorization:${connected.connectionId}:user-for-run`, sourceRunId: run.id });
|
||||
const reassessment = vi.fn(), reviewResponse = vi.fn();
|
||||
const unsubscribe = [subscribeDeliveryWork(db, DELIVERY_QUEUES.nativeStatusReassessment, reassessment),
|
||||
subscribeDeliveryWork(db, DELIVERY_QUEUES.nativeReviewResponse, reviewResponse)];
|
||||
const rollback = new Error("rollback OAuth reopening");
|
||||
try {
|
||||
const operation = db.transaction(async tx => {
|
||||
const started = await createTestToolAccessService(tx as unknown as typeof db).startOAuth(company.id, connected.connectionId, {
|
||||
redirectUri: "https://paperclip.example/api/tools/oauth/callback", actor: { actorType: "agent", actorId: agent.id, sessionId: run.id },
|
||||
issueId: issue.id, subjectUserId: "user-for-run", scopes: ["channels:read"],
|
||||
});
|
||||
const [inside] = await tx.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.issueId, issue.id));
|
||||
expect(inside.status).toBe("pending");
|
||||
expect(inside.payload).toMatchObject({ target: { href: started.authorizationUrl } });
|
||||
expect(reassessment).not.toHaveBeenCalled(); expect(reviewResponse).not.toHaveBeenCalled();
|
||||
const [outside] = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.issueId, issue.id));
|
||||
expect(outside?.status).toBe(mode === "creation" ? undefined : "accepted");
|
||||
if (mode === "rollback") throw rollback;
|
||||
});
|
||||
if (mode === "rollback") await expect(operation).rejects.toBe(rollback); else await operation;
|
||||
|
||||
|
||||
expect(reassessment).toHaveBeenCalledTimes(1); expect(reviewResponse).toHaveBeenCalledTimes(1);
|
||||
const [persisted] = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.issueId, issue.id));
|
||||
expect(persisted.status).toBe(mode === "rollback" ? "accepted" : "pending");
|
||||
expect(await db.select().from(toolOauthStates).where(eq(toolOauthStates.issueId, issue.id)))
|
||||
.toHaveLength(mode === "rollback" ? 0 : 1);
|
||||
} finally { unsubscribe.forEach(stop => stop()); }
|
||||
});
|
||||
|
||||
it("creates and resolves an agent-initiated user authorization grant card", async () => {
|
||||
vi.stubEnv("PAPERCLIP_TOOL_OAUTH_SLACK_CLIENT_ID", "slack-client-id");
|
||||
vi.stubEnv(
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
@@ -65,7 +66,8 @@ describe("workspace runtime control serialization", () => {
|
||||
});
|
||||
});
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_WORKSPACE_OPERATIONS_TEST_DATABASE_URL");
|
||||
const embeddedPostgresSupport = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
if (!embeddedPostgresSupport.supported) {
|
||||
@@ -81,8 +83,8 @@ describeEmbeddedPostgres("workspace operation reconciliation", () => {
|
||||
let previousLogRoot: string | undefined;
|
||||
|
||||
beforeAll(async () => {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-workspace-operation-reconcile-");
|
||||
db = createDb(tempDb.connectionString);
|
||||
if (!externalUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-workspace-operation-reconcile-");
|
||||
db = createDb(externalUrl ?? tempDb!.connectionString);
|
||||
logRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-workspace-operation-logs-"));
|
||||
previousLogRoot = process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
|
||||
process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = logRoot;
|
||||
@@ -99,6 +101,7 @@ describeEmbeddedPostgres("workspace operation reconciliation", () => {
|
||||
if (previousLogRoot === undefined) delete process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
|
||||
else process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = previousLogRoot;
|
||||
await fs.rm(logRoot, { recursive: true, force: true });
|
||||
await db?.$client.end({ timeout: 1 });
|
||||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
|
||||
+3
-1
@@ -1825,6 +1825,8 @@ async function startServerWithDatabaseTeardown(
|
||||
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
|
||||
if (config.heartbeatSchedulerEnabled) await heartbeat.registerInstructionDirectoryCleanupWork(deliveryWork,
|
||||
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
|
||||
if (config.heartbeatSchedulerEnabled) await heartbeat.registerNativeStatusReassessmentWork(deliveryWork,
|
||||
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
|
||||
if (config.heartbeatSchedulerEnabled) await heartbeat.registerNativeReviewResponseWork(deliveryWork,
|
||||
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
|
||||
if (config.heartbeatSchedulerEnabled) {
|
||||
@@ -1974,7 +1976,7 @@ async function startServerWithDatabaseTeardown(
|
||||
// Periodically reap orphaned runs (5-min staleness threshold) and make sure
|
||||
// persisted queued work is still being driven forward.
|
||||
runPeriodicRecovery(() => heartbeat
|
||||
.reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true })
|
||||
.reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true })
|
||||
.then(() => heartbeat.promoteDueScheduledRetries())
|
||||
.then(async (promotion) => {
|
||||
await heartbeat.resumeQueuedRuns();
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeStatusReassessment } from "../../../services/native-status-reassessment-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "../../../services/unsafe-workspace-export-notifications.js";
|
||||
import { notifyCancelledWorkspacePreservation } from "../../../services/cancelled-workspace-preservation-notifications.js";
|
||||
import { notifyInstructionCollection } from "../../../services/instruction-collection-notifications.js";
|
||||
@@ -343,6 +344,7 @@ export function createPostgresWatchdogAdapter(db: Db): WatchdogRunReader & Watch
|
||||
await notifyInstructionCollection(tx, { companyId, runId: input.run.id });
|
||||
await notifyCancelledWorkspacePreservation(tx, { companyId, runId: input.run.id });
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId, issueId: input.sourceIssue.id });
|
||||
await notifyNativeStatusReassessment(tx, { companyId, issueId: input.sourceIssue.id });
|
||||
const [updatedRun] = await tx
|
||||
.update(heartbeatRuns)
|
||||
.set({
|
||||
|
||||
@@ -1224,7 +1224,7 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => {
|
||||
expect(wakeRow?.payload).not.toHaveProperty("foo");
|
||||
});
|
||||
|
||||
it.each([DELIVERY_QUEUES.nativeResponsePresentation, DELIVERY_QUEUES.nativeReviewResponse])(
|
||||
it.each([DELIVERY_QUEUES.nativeResponsePresentation, DELIVERY_QUEUES.nativeReviewResponse, DELIVERY_QUEUES.nativeStatusReassessment])(
|
||||
"scopes a real coalesced-input %s response wake to its active run after commit", async queue => {
|
||||
const companyId = await seedCompany();
|
||||
const agentId = await seedAgent({ companyId });
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeStatusReassessment } from "../../../services/native-status-reassessment-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "../../../services/unsafe-workspace-export-notifications.js";
|
||||
import { notifyChatInboundWakeNotices } from "../../../services/chat-inbound-wake-notice-notifications.js";
|
||||
import { notifyChatRunMilestones } from "../../../services/chat-run-milestone-notifications.js";
|
||||
@@ -1125,6 +1126,7 @@ export function createPostgresWakeQueueAdapter(db: Db, deps: WakeQueuePostgresAd
|
||||
for (const issue of candidateIssues) {
|
||||
if (issue.executionRunId === run.id || issue.checkoutRunId === run.id) {
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: issue.id });
|
||||
await notifyNativeStatusReassessment(tx, { companyId: input.companyId, issueId: issue.id });
|
||||
}
|
||||
}
|
||||
// Two separate updates: a retry can move `executionRunId` to a new run
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeStatusReassessment } from "../../../services/native-status-reassessment-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "../../../services/unsafe-workspace-export-notifications.js";
|
||||
import { notifyCancelledWorkspacePreservation } from "../../../services/cancelled-workspace-preservation-notifications.js";
|
||||
import { notifyInstructionCollection } from "../../../services/instruction-collection-notifications.js";
|
||||
@@ -120,6 +121,7 @@ function buildTransaction(tx: Db, companyId: string, issueId: string, deps: Queu
|
||||
},
|
||||
|
||||
async clearExecutionLockAndTouchIssue({ issueId, executionRunId, updatedAt }) {
|
||||
await notifyNativeStatusReassessment(tx, { companyId, issueId });
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId, issueId });
|
||||
await tx
|
||||
.update(issues)
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeStatusReassessment } from "../services/native-status-reassessment-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "../services/unsafe-workspace-export-notifications.js";
|
||||
import { notifyChatInboundWakeNotices } from "../services/chat-inbound-wake-notice-notifications.js";
|
||||
import { completeIssueCommentReceipt } from "../services/issue-comment-receipt.js";
|
||||
@@ -11816,6 +11817,7 @@ export function issueRoutes(
|
||||
promotedAt,
|
||||
});
|
||||
const product = await db.transaction(async (tx) => {
|
||||
await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
const markPromoted = {
|
||||
sourceTrust: promotionTrust,
|
||||
updatedAt: promotedAt,
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js";
|
||||
import { notifyChatCompanyAuthorityChange } from "./chat-run-milestone-notifications.js";
|
||||
import { notifyVoiceChange } from "./voice/voice-recovery-notifications.js";
|
||||
@@ -740,6 +741,7 @@ export function accessService(db: Db) {
|
||||
eq(issues.assigneeUserId, existing.principalId),
|
||||
sql`${issues.status} not in ('done', 'cancelled')`,
|
||||
);
|
||||
await notifyNativeStatusReassessment(tx, { companyId });
|
||||
const resetInProgress = await tx
|
||||
.update(issues)
|
||||
.set({
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { approvals, issueApprovals, type Db } from "@paperclipai/db";
|
||||
import { notifyNativeReviewResponse } from "./native-review-response-notifications.js";
|
||||
import { notifyNativeReviewState } from "./native-review-state-notifications.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
|
||||
export const notifyApprovalQueueWork = (transaction: object, companyId: string) => notifyDeliveryWork(transaction, DELIVERY_QUEUES.approvalQueueSeeding, companyId);
|
||||
@@ -21,7 +21,7 @@ export async function notifyApprovalWork(transaction: object, companyId: string,
|
||||
.where(and(eq(issueApprovals.companyId, companyId), eq(issueApprovals.approvalId, scope.approvalId)));
|
||||
for (const link of links) issueIds.add(link.issueId);
|
||||
}
|
||||
for (const issueId of issueIds) await notifyNativeReviewResponse(transaction, { companyId, issueId });
|
||||
for (const issueId of issueIds) await notifyNativeReviewState(transaction, { companyId, issueId });
|
||||
}
|
||||
|
||||
export function writeApprovalChange<T>(db: Db, companyId: string, write: (transaction: Db) => Promise<T>, scope?: ApprovalChangeScope): Promise<T> {
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { notifyNativeReviewResponse } from "./native-review-response-notifications.js";
|
||||
import { notifyNativeReviewState } from "./native-review-state-notifications.js";
|
||||
import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js";
|
||||
import { notifyChatResponseAvailability } from "./chat-response-availability.js";
|
||||
import { notifyChatConfirmationWork } from "./chat-confirmation-notifications.js";
|
||||
@@ -12,7 +12,7 @@ export type ChatRunMilestoneScope = { companyId: string; issueId?: string; runId
|
||||
|
||||
export async function notifyChatRunMilestones(tx: object, scope: ChatRunMilestoneScope) {
|
||||
if (scope.issueId || scope.runId) {
|
||||
await notifyNativeReviewResponse(tx, { companyId: scope.companyId, issueId: scope.issueId,
|
||||
await notifyNativeReviewState(tx, { companyId: scope.companyId, issueId: scope.issueId,
|
||||
runId: scope.issueId ? undefined : scope.runId });
|
||||
await notifyNativeResponsePresentation(tx, { companyId: scope.companyId, issueId: scope.issueId,
|
||||
runId: scope.issueId ? undefined : scope.runId });
|
||||
@@ -33,5 +33,5 @@ export async function notifyChatRunMilestones(tx: object, scope: ChatRunMileston
|
||||
export async function notifyChatCompanyAuthorityChange(tx: object, scope: { companyId: string }) {
|
||||
await notifyChatRunMilestones(tx, scope);
|
||||
await notifyNativeResponsePresentation(tx, scope);
|
||||
await notifyNativeReviewResponse(tx, scope);
|
||||
await notifyNativeReviewState(tx, scope);
|
||||
}
|
||||
@@ -2,6 +2,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db";
|
||||
|
||||
/** Topics for existing durable queues; register intent before their writes. */
|
||||
export const DELIVERY_QUEUES = {
|
||||
nativeStatusReassessment: "native-status-reassessment",
|
||||
nativeReviewResponse: "native-review-response",
|
||||
nativeResponsePresentation: "native-response-presentation",
|
||||
nativePendingEffects: "native-pending-effects",
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js";
|
||||
import { notifyNativeFinalizationWork } from "./native-finalization-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRunChange } from "./unsafe-workspace-export-notifications.js";
|
||||
import { notifyCancelledWorkspacePreservation } from "./cancelled-workspace-preservation-notifications.js";
|
||||
@@ -246,6 +247,7 @@ export async function reconcileAbandonedExecutionControl(
|
||||
if (updatedRun) await preserveNativeFailedWorkspace(tx as unknown as Db, updatedRun);
|
||||
return true;
|
||||
}
|
||||
await notifyNativeStatusReassessment(tx, { companyId: task.companyId, issueId: task.id });
|
||||
await tx
|
||||
.update(issues)
|
||||
.set({ executionRunId: null, checkoutRunId: null, updatedAt: now })
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js";
|
||||
import { notifyChatResponseAvailability } from "./chat-response-availability.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
@@ -591,6 +592,7 @@ export async function settleUnrecoverableExecutions(
|
||||
let nativeFailureBlock = action.evidence.nativeFailureBlock;
|
||||
let nativeBootstrapFailureBlock = action.evidence.nativeBootstrapFailureBlock;
|
||||
if (current) {
|
||||
await notifyNativeStatusReassessment(tx, { companyId: task.companyId, issueId: task.id });
|
||||
if (task.status !== "blocked") await notifyChatResponseAvailability(tx, { companyId: task.companyId, issueId: task.id });
|
||||
const [projected] = await tx
|
||||
.update(issues)
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import { readNativeEventEvidenceOutcome } from "./native-runtime/evidence-classifier.js";
|
||||
import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js";
|
||||
import { isNativeResponseEvidence, notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js";
|
||||
import { isSafeNativeChatProgressEventType } from "./safe-native-chat-progress.js";
|
||||
import { notifyChatNativeProgress } from "./chat-native-progress-notifications.js";
|
||||
@@ -72,9 +74,10 @@ export async function appendHeartbeatRunEvent(
|
||||
db: Db,
|
||||
input: AppendHeartbeatRunEventInput,
|
||||
): Promise<AppendHeartbeatRunEventResult> {
|
||||
const projectsReassessment = readNativeEventEvidenceOutcome(input.payload?.prpEvent) !== null;
|
||||
const projectsProgress = isSafeNativeChatProgressEventType(input.eventType);
|
||||
const projectsResponse = isNativeResponseEvidence(input.eventType, input.payload?.prpEvent);
|
||||
if (!input.nativeSource && !input.retryExhaustion && !projectsProgress && !projectsResponse) {
|
||||
if (!input.nativeSource && !input.retryExhaustion && !projectsProgress && !projectsResponse && !projectsReassessment) {
|
||||
const allocated = db.$with("allocated_run_event", {}).as(sql`
|
||||
update ${heartbeatRuns}
|
||||
set next_event_seq = ${heartbeatRuns.nextEventSeq} + 1,
|
||||
@@ -103,7 +106,7 @@ export async function appendHeartbeatRunEvent(
|
||||
.select({ companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId,
|
||||
// Only progress needs the task binding. Other native/recovery events
|
||||
// must not read a potentially large context snapshot to append a log.
|
||||
...(projectsProgress || projectsResponse ? { runtimeMode: heartbeatRuns.runtimeMode } : {}),
|
||||
...(projectsProgress || projectsResponse || projectsReassessment ? { runtimeMode: heartbeatRuns.runtimeMode } : {}),
|
||||
...(projectsProgress ? { issueId: boundRunIssueIdSql() } : {}) })
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, input.runId))
|
||||
@@ -175,6 +178,9 @@ export async function appendHeartbeatRunEvent(
|
||||
}
|
||||
}
|
||||
|
||||
if (projectsReassessment && run.runtimeMode === "native") {
|
||||
await notifyNativeStatusReassessment(tx, { companyId: run.companyId, runId: input.runId });
|
||||
}
|
||||
if (projectsProgress && run.runtimeMode === "native" && run.issueId) {
|
||||
await notifyChatNativeProgress(tx, { companyId: run.companyId, issueId: run.issueId });
|
||||
}
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { nativeStatusReassessmentWork } from "./native-runtime/native-status-reassessment-work.js";
|
||||
import { registerNativeReviewResponseWork } from "./native-runtime/native-review-response-work.js";
|
||||
import { registerNativeResponsePresentationWork } from "./native-runtime/native-response-presentation-work.js";
|
||||
import { registerNativePendingEffectWork } from "./native-runtime/native-pending-effect-work.js";
|
||||
@@ -4091,6 +4092,8 @@ export function heartbeatService(
|
||||
triggerIssueMonitor,
|
||||
|
||||
reportRunActivity: clearDetachedRunWarning,
|
||||
registerNativeStatusReassessmentWork: (coordinator: Parameters<ReturnType<typeof nativeStatusReassessmentWork>["register"]>[0], canRun: () => boolean) =>
|
||||
nativeStatusReassessmentWork(db).register(coordinator, () => canRun() && readTaskDrain(new Date()) === null),
|
||||
registerNativeReviewResponseWork: (coordinator: Parameters<typeof registerNativeReviewResponseWork>[1], canRun: () => boolean) =>
|
||||
registerNativeReviewResponseWork(db, coordinator, () => canRun() && readTaskDrain(new Date()) === null),
|
||||
registerNativeResponsePresentationWork: (coordinator: Parameters<typeof registerNativeResponsePresentationWork>[1], canRun: () => boolean) =>
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery, notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js";
|
||||
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
|
||||
import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
|
||||
@@ -1803,6 +1804,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
|
||||
|
||||
await db.transaction(async tx => {
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: run.companyId, issueId });
|
||||
await notifyNativeStatusReassessment(tx, { companyId: run.companyId, issueId });
|
||||
await tx
|
||||
.update(issues)
|
||||
.set({
|
||||
@@ -3694,6 +3696,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
|
||||
}
|
||||
|
||||
if (!activeExecutionRun && issue.executionRunId) {
|
||||
await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
await tx
|
||||
.update(issues)
|
||||
@@ -3893,6 +3896,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
|
||||
`- Next action: ${WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION}`,
|
||||
].join("\n");
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
if (issue.status !== "blocked") await notifyChatResponseAvailability(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
await tx
|
||||
.update(issues)
|
||||
|
||||
@@ -1947,7 +1947,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
|
||||
let settledProcessesDiscovered = false;
|
||||
|
||||
async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean; skipNativeResponsePresentation?: boolean; skipNativeReviewResponse?: boolean }) {
|
||||
async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean; skipNativeResponsePresentation?: boolean; skipNativeReviewResponse?: boolean; skipNativeStatusReassessment?: boolean }) {
|
||||
if (!opts?.skipCancelledWorkspacePreservation) await preserveCancelledNativeWorkspaces(db);
|
||||
const staleThresholdMs = opts?.staleThresholdMs ?? 0;
|
||||
const now = new Date();
|
||||
@@ -1971,6 +1971,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
skipNativePendingEffects: opts?.skipNativePendingEffects,
|
||||
skipNativeResponsePresentation: opts?.skipNativeResponsePresentation,
|
||||
skipNativeReviewResponse: opts?.skipNativeReviewResponse,
|
||||
skipNativeStatusReassessment: opts?.skipNativeStatusReassessment,
|
||||
environmentRuntime,
|
||||
pluginWorkerManager: dependencies.pluginWorkerManager,
|
||||
onWorkspaceSettled: settleRecoveredNativeWorkspace,
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { notifyNativeReviewResponse } from "./native-review-response-notifications.js";
|
||||
import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js";
|
||||
import { notifyNativeReviewState } from "./native-review-state-notifications.js";
|
||||
import { notifyChatConfirmationResolution } from "./chat-confirmation-notifications.js";
|
||||
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
|
||||
import { notifyToolReviewChange } from "./tool-review-work-notifications.js";
|
||||
@@ -1945,7 +1946,7 @@ async function expireStaleRequestConfirmationTarget(
|
||||
const expireInTransaction = async (tx: Db | any) => {
|
||||
await notifyChatConfirmationResolution(tx, args.row);
|
||||
if (args.row.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.row);
|
||||
else await notifyNativeReviewResponse(tx, args.row);
|
||||
else await notifyNativeReviewState(tx, args.row);
|
||||
const [updated] = await tx
|
||||
.update(issueThreadInteractions)
|
||||
.set({
|
||||
@@ -2225,7 +2226,7 @@ export function issueThreadInteractionService(
|
||||
await assertInteractionRunWriteAllowed(tx as unknown as Db, args.issue, args.actor);
|
||||
await notifyChatConfirmationResolution(tx, args.current);
|
||||
if (args.current.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.current);
|
||||
else await notifyNativeReviewResponse(tx, args.current);
|
||||
else await notifyNativeReviewState(tx, args.current);
|
||||
await args.mutationOptions?.beforeResolveInTransaction?.(tx);
|
||||
// Policy mutations and review transitions use the same issue-row lock,
|
||||
// so the authoritative review policy and requester are stable through
|
||||
@@ -2516,7 +2517,7 @@ export function issueThreadInteractionService(
|
||||
await assertInteractionRunWriteAllowed(tx as unknown as Db, args.issue, args.actor);
|
||||
await notifyChatConfirmationResolution(tx, args.current);
|
||||
if (args.current.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.current);
|
||||
else await notifyNativeReviewResponse(tx, args.current);
|
||||
else await notifyNativeReviewState(tx, args.current);
|
||||
await args.mutationOptions?.beforeResolveInTransaction?.(tx);
|
||||
const issueContext = await tx
|
||||
.select({
|
||||
@@ -2753,6 +2754,7 @@ export function issueThreadInteractionService(
|
||||
.where(and(eq(heartbeatRuns.id, input.sourceRunId), eq(heartbeatRuns.companyId, issue.companyId)));
|
||||
if (!sourceRun) throw unprocessable("Interaction source run is unavailable");
|
||||
const originCommentIds = await currentContinuationOrigins(tx as unknown as Db, issue.companyId, issue.id, sourceRun.context);
|
||||
await notifyNativeReviewState(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
const [row] = await tx
|
||||
.insert(issueThreadInteractions)
|
||||
.values({
|
||||
@@ -2909,7 +2911,7 @@ export function issueThreadInteractionService(
|
||||
const updated = await db.transaction(async (tx) => {
|
||||
await notifyChatConfirmationResolution(tx, current);
|
||||
await mutationOptions.beforeResolveInTransaction?.(tx);
|
||||
await notifyNativeReviewResponse(tx, current);
|
||||
await notifyNativeReviewState(tx, current);
|
||||
const [row] = await tx
|
||||
.update(issueThreadInteractions)
|
||||
.set({
|
||||
@@ -3345,7 +3347,7 @@ export function issueThreadInteractionService(
|
||||
const updated = await db.transaction(async (tx) => {
|
||||
await notifyChatConfirmationResolution(tx, row);
|
||||
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
|
||||
else await notifyNativeReviewResponse(tx, row);
|
||||
else await notifyNativeReviewState(tx, row);
|
||||
if (row.kind === "request_confirmation") await notifyToolProjectionForInteractions(tx, row.companyId, [row.id]);
|
||||
const [resolved] = await tx
|
||||
.update(issueThreadInteractions)
|
||||
@@ -3433,7 +3435,7 @@ export function issueThreadInteractionService(
|
||||
const updated = await db.transaction(async (tx) => {
|
||||
await notifyChatConfirmationResolution(tx, row);
|
||||
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
|
||||
else await notifyNativeReviewResponse(tx, row);
|
||||
else await notifyNativeReviewState(tx, row);
|
||||
if (row.kind === "request_confirmation") await notifyToolProjectionForInteractions(tx, row.companyId, [row.id]);
|
||||
const [updatedRow] = await tx
|
||||
.update(issueThreadInteractions)
|
||||
@@ -3863,6 +3865,8 @@ export function issueThreadInteractionService(
|
||||
}
|
||||
if (data.kind === "ask_user_questions" || data.kind === "request_confirmation") {
|
||||
await notifyChatRunMilestones(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
} else {
|
||||
await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
}
|
||||
const [row] = await tx
|
||||
.insert(issueThreadInteractions)
|
||||
@@ -4215,7 +4219,7 @@ export function issueThreadInteractionService(
|
||||
}
|
||||
}
|
||||
}
|
||||
await notifyNativeReviewResponse(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
await notifyNativeReviewState(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
const resolvedAt = new Date();
|
||||
const [claimed] = await tx
|
||||
.update(issueThreadInteractions)
|
||||
@@ -4479,7 +4483,8 @@ export function issueThreadInteractionService(
|
||||
complete,
|
||||
items,
|
||||
} satisfies RequestItemVerdictsResult;
|
||||
if (complete) await notifyNativeReviewResponse(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
if (complete) await notifyNativeReviewState(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
else await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
const [updated] = await tx
|
||||
.update(issueThreadInteractions)
|
||||
.set({
|
||||
@@ -4546,7 +4551,7 @@ export function issueThreadInteractionService(
|
||||
await notifyChatConfirmationResolution(tx, current);
|
||||
await mutationOptions.beforeResolveInTransaction?.(tx);
|
||||
await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor);
|
||||
await notifyNativeReviewResponse(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
await notifyNativeReviewState(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
const rows = await tx.update(issueThreadInteractions)
|
||||
.set({
|
||||
status: "rejected",
|
||||
@@ -4638,7 +4643,7 @@ export function issueThreadInteractionService(
|
||||
const updated = await db.transaction(async (tx) => {
|
||||
await notifyChatConfirmationResolution(tx, row);
|
||||
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
|
||||
else await notifyNativeReviewResponse(tx, row);
|
||||
else await notifyNativeReviewState(tx, row);
|
||||
if (row.kind === "connection_intent") {
|
||||
await tx
|
||||
.delete(toolOauthStates)
|
||||
@@ -4816,7 +4821,7 @@ export function issueThreadInteractionService(
|
||||
const updatedInteractions = await db.transaction(async (tx) => {
|
||||
await notifyChatConfirmationResolution(tx, sampleConfirmationRow);
|
||||
if (commentRows.some(row => row.kind === "request_confirmation")) await notifyChatRunMilestones(tx, sampleConfirmationRow);
|
||||
else await notifyNativeReviewResponse(tx, sampleConfirmationRow);
|
||||
else await notifyNativeReviewState(tx, sampleConfirmationRow);
|
||||
await notifyToolProjectionForInteractions(tx, issue.companyId, confirmationRowIds);
|
||||
const updatedRows = await tx
|
||||
.update(issueThreadInteractions)
|
||||
@@ -4852,7 +4857,7 @@ export function issueThreadInteractionService(
|
||||
|
||||
for (const row of itemVerdictRows) {
|
||||
const updated = await db.transaction(async tx => {
|
||||
await notifyNativeReviewResponse(tx, row);
|
||||
await notifyNativeReviewState(tx, row);
|
||||
const [retired] = await tx
|
||||
.update(issueThreadInteractions)
|
||||
.set({
|
||||
@@ -4878,7 +4883,7 @@ export function issueThreadInteractionService(
|
||||
|
||||
for (const row of connectionIntentRows) {
|
||||
const [updated] = await db.transaction(async (tx) => {
|
||||
await notifyNativeReviewResponse(tx, row);
|
||||
await notifyNativeReviewState(tx, row);
|
||||
await notifyChatConfirmationResolution(tx, row);
|
||||
await tx
|
||||
.delete(toolOauthStates)
|
||||
@@ -4967,7 +4972,7 @@ export function issueThreadInteractionService(
|
||||
const updated = await db.transaction(async (tx) => {
|
||||
await notifyChatConfirmationResolution(tx, row);
|
||||
if (row.kind === "request_confirmation") await notifyChatRunMilestones(tx, row);
|
||||
else await notifyNativeReviewResponse(tx, row);
|
||||
else await notifyNativeReviewState(tx, row);
|
||||
const [resolved] = await tx
|
||||
.update(issueThreadInteractions)
|
||||
.set({
|
||||
@@ -5031,7 +5036,7 @@ export function issueThreadInteractionService(
|
||||
? "the AI account"
|
||||
: `the ${payload?.serviceName?.trim() || "requested"} connection`;
|
||||
const updated = await db.transaction(async tx => {
|
||||
await notifyNativeReviewResponse(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
await notifyNativeReviewState(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
await notifyChatConfirmationResolution(tx, { companyId: issue.companyId, issueId: issue.id, kind: "connection_intent" });
|
||||
const [resolved] = await tx
|
||||
.update(issueThreadInteractions)
|
||||
@@ -5093,7 +5098,7 @@ export function issueThreadInteractionService(
|
||||
.transaction(async (tx) => {
|
||||
await notifyChatConfirmationResolution(tx, row);
|
||||
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
|
||||
else await notifyNativeReviewResponse(tx, row);
|
||||
else await notifyNativeReviewState(tx, row);
|
||||
if (row.kind === "connection_intent") {
|
||||
await tx
|
||||
.delete(toolOauthStates)
|
||||
@@ -5186,7 +5191,7 @@ export function issueThreadInteractionService(
|
||||
await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor);
|
||||
await notifyChatConfirmationResolution(tx, current);
|
||||
if (["ask_user_questions", "request_confirmation"].includes(current.kind)) await notifyChatRunMilestones(tx, current);
|
||||
else await notifyNativeReviewResponse(tx, current);
|
||||
else await notifyNativeReviewState(tx, current);
|
||||
await resolveLinkedToolActionRequests(tx, current, {
|
||||
status: "cancelled",
|
||||
fromStatuses: ["pending", "approved"],
|
||||
@@ -5449,7 +5454,7 @@ export function issueThreadInteractionService(
|
||||
await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor);
|
||||
await notifyChatConfirmationResolution(tx, current);
|
||||
if (["ask_user_questions", "request_confirmation"].includes(current.kind)) await notifyChatRunMilestones(tx, current);
|
||||
else await notifyNativeReviewResponse(tx, current);
|
||||
else await notifyNativeReviewState(tx, current);
|
||||
await resolveLinkedToolActionRequests(tx, current, {
|
||||
status: "cancelled",
|
||||
fromStatuses: ["pending", "approved"],
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js";
|
||||
import { notifyChatResponseAvailability } from "./chat-response-availability.js";
|
||||
import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js";
|
||||
@@ -1002,6 +1003,7 @@ export function issueTreeControlService(db: Db) {
|
||||
for (const [status, issueIdsForStatus] of issueIdsByStatus) {
|
||||
if (issueIdsForStatus.length === 0) continue;
|
||||
for (const issueId of issueIdsForStatus) {
|
||||
await notifyNativeStatusReassessment(tx, { companyId, issueId });
|
||||
await notifyTerminalWorkspaceWork(tx, `issue:${issueId}`);
|
||||
await notifyChatResponseAvailability(tx, { companyId, issueId });
|
||||
if (status === "blocked" || status === "in_review") await notifyUnsafeWorkspaceExportRecovery(tx, { companyId, issueId });
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js";
|
||||
import { notifyChatResponseAvailability } from "./chat-response-availability.js";
|
||||
import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js";
|
||||
@@ -2513,6 +2514,7 @@ async function listPendingFinalizeBlockerIssueIds(
|
||||
blockerIssueId: string;
|
||||
executionWorkspaceId: string;
|
||||
}>,
|
||||
lockSources = false,
|
||||
): Promise<Set<string>> {
|
||||
const pending = new Set<string>();
|
||||
const blockerIssueIds = [
|
||||
@@ -2529,7 +2531,14 @@ async function listPendingFinalizeBlockerIssueIds(
|
||||
),
|
||||
);
|
||||
|
||||
const rows = await dbOrTx
|
||||
if (lockSources) {
|
||||
|
||||
|
||||
await dbOrTx.select({ id: executionWorkspaces.id }).from(executionWorkspaces)
|
||||
.where(and(eq(executionWorkspaces.companyId, companyId), inArray(executionWorkspaces.id, executionWorkspaceIds)))
|
||||
.for("update", { noWait: true });
|
||||
}
|
||||
const query = dbOrTx
|
||||
.select({
|
||||
issueId: workspaceOperations.issueId,
|
||||
executionWorkspaceId: workspaceOperations.executionWorkspaceId,
|
||||
@@ -2548,6 +2557,8 @@ async function listPendingFinalizeBlockerIssueIds(
|
||||
),
|
||||
);
|
||||
|
||||
const rows = await (lockSources ? query.for("share", { noWait: true }) : query);
|
||||
|
||||
const latestAttributedByBlockerWorkspace = new Map<
|
||||
string,
|
||||
{ phase: string; status: string; startedAt: Date }
|
||||
@@ -2697,6 +2708,7 @@ async function listIssueDependencyReadinessMap(
|
||||
dbOrTx: Pick<Db, "select">,
|
||||
companyId: string,
|
||||
issueIds: string[],
|
||||
lockSources = false,
|
||||
) {
|
||||
const uniqueIssueIds = [...new Set(issueIds.filter(Boolean))];
|
||||
const readinessMap = new Map<string, IssueDependencyReadiness>();
|
||||
@@ -2705,7 +2717,7 @@ async function listIssueDependencyReadinessMap(
|
||||
}
|
||||
if (uniqueIssueIds.length === 0) return readinessMap;
|
||||
|
||||
const blockerRows = await dbOrTx
|
||||
const blockerQuery = dbOrTx
|
||||
.select({
|
||||
issueId: issueRelations.relatedIssueId,
|
||||
blockerIssueId: issueRelations.issueId,
|
||||
@@ -2722,6 +2734,10 @@ async function listIssueDependencyReadinessMap(
|
||||
),
|
||||
);
|
||||
|
||||
|
||||
|
||||
const blockerRows = await (lockSources ? blockerQuery.for("update", { noWait: true }) : blockerQuery);
|
||||
|
||||
// Collect issue/workspace pairs of "done" blockers — these are the only ones
|
||||
// subject to the workspace-finalize barrier. Blockers that aren't done already
|
||||
// mark the dependent as not-ready and don't need a finalize check.
|
||||
@@ -2742,6 +2758,7 @@ async function listIssueDependencyReadinessMap(
|
||||
dbOrTx,
|
||||
companyId,
|
||||
doneBlockerWorkspacePairs,
|
||||
lockSources,
|
||||
);
|
||||
|
||||
for (const row of blockerRows) {
|
||||
@@ -7823,6 +7840,7 @@ export function issueService(db: Db) {
|
||||
if (isExplicitContinuationRetryClaim(issue, run)) return false;
|
||||
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId });
|
||||
await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId });
|
||||
const updated = await tx
|
||||
.update(issues)
|
||||
.set({
|
||||
@@ -7897,6 +7915,7 @@ export function issueService(db: Db) {
|
||||
}
|
||||
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId });
|
||||
await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId });
|
||||
const updated = await tx
|
||||
.update(issues)
|
||||
.set({
|
||||
@@ -9311,7 +9330,13 @@ export function issueService(db: Db) {
|
||||
};
|
||||
},
|
||||
|
||||
getDependencyReadiness: async (issueId: string, dbOrTx: any = db) => {
|
||||
|
||||
|
||||
getDependencyReadiness: async (
|
||||
issueId: string,
|
||||
dbOrTx: any = db,
|
||||
options: { lockSources?: boolean } = {},
|
||||
) => {
|
||||
const issue = await dbOrTx
|
||||
.select({ id: issues.id, companyId: issues.companyId })
|
||||
.from(issues)
|
||||
@@ -9324,6 +9349,7 @@ export function issueService(db: Db) {
|
||||
dbOrTx,
|
||||
issue.companyId,
|
||||
[issueId],
|
||||
options.lockSources,
|
||||
);
|
||||
return readiness.get(issueId) ?? createIssueDependencyReadiness(issueId);
|
||||
},
|
||||
@@ -11473,6 +11499,11 @@ export function issueService(db: Db) {
|
||||
|| (patch.checkoutRunId !== undefined && patch.checkoutRunId !== receiptExisting.checkoutRunId)) {
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: receiptExisting.companyId, issueId: id });
|
||||
}
|
||||
if (patch.status !== undefined || patch.statusVersion !== undefined || patch.executionState !== undefined
|
||||
|| patch.executionRunId !== undefined || patch.checkoutRunId !== undefined
|
||||
|| patch.assigneeAgentId !== undefined || patch.assigneeUserId !== undefined || blockedByIssueIds !== undefined) {
|
||||
await notifyNativeStatusReassessment(tx, { companyId: receiptExisting.companyId, issueId: id });
|
||||
}
|
||||
const updated = await tx
|
||||
.update(issues)
|
||||
.set(patch)
|
||||
@@ -12383,6 +12414,7 @@ export function issueService(db: Db) {
|
||||
const releaseStatus =
|
||||
existing.status === "in_progress" ? "todo" : existing.status;
|
||||
await notifyTerminalWorkspaceWork(tx, `issue:${id}`);
|
||||
await notifyNativeStatusReassessment(tx, { companyId: existing.companyId, issueId: id });
|
||||
const updated = await tx
|
||||
.update(issues)
|
||||
.set({
|
||||
@@ -12434,6 +12466,9 @@ export function issueService(db: Db) {
|
||||
}
|
||||
|
||||
await notifyTerminalWorkspaceWork(tx, `issue:${id}`);
|
||||
if (existing.checkoutRunId || existing.executionRunId || options.clearAssignee) {
|
||||
await notifyNativeStatusReassessment(tx, { companyId: existing.companyId, issueId: id });
|
||||
}
|
||||
if (existing.checkoutRunId || existing.executionRunId) {
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: existing.companyId, issueId: id });
|
||||
}
|
||||
@@ -13497,6 +13532,7 @@ export function issueService(db: Db) {
|
||||
}
|
||||
}
|
||||
}
|
||||
await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
const [asset] = await tx
|
||||
.insert(assets)
|
||||
.values({
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js";
|
||||
import { notifyInstructionCollection } from "./instruction-collection-notifications.js";
|
||||
import { notifyExecutionRunWork } from "./execution-control-notifications.js";
|
||||
@@ -172,6 +173,9 @@ export async function terminalizeLegacyExecution(input: {
|
||||
? retainedLeaseIds.length > 0 || hasRequiredWorkspaceRecovery(updated.resultJson)
|
||||
: !input.reconcileIfNeeded || await legacyExecutionNeedsReconciliationWithEvidence(tx as unknown as Db, updated);
|
||||
if (!needsRecovery) return updated;
|
||||
if (!input.recordRestoreFailureOnly && task && (task.executionRunId === run.id || task.checkoutRunId === run.id)) {
|
||||
await notifyNativeStatusReassessment(tx, { companyId: task.companyId, issueId: task.id });
|
||||
}
|
||||
if (!input.recordRestoreFailureOnly && task?.executionRunId === run.id)
|
||||
await tx
|
||||
.update(issues)
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js";
|
||||
import { notifyNativeReviewResponse } from "./native-review-response-notifications.js";
|
||||
import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
@@ -6,6 +7,7 @@ export type NativeFinalizationWorkScope = { companyId: string; runId?: string; i
|
||||
|
||||
export async function notifyNativeFinalizationWork(transaction: object, scope: NativeFinalizationWorkScope) {
|
||||
if (scope.issueId || scope.runId) {
|
||||
await notifyNativeStatusReassessment(transaction, scope);
|
||||
await notifyNativeResponsePresentation(transaction, scope);
|
||||
await notifyNativeReviewResponse(transaction, scope);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
import { notifyNativeReviewResponse } from "./native-review-response-notifications.js";
|
||||
import { notifyNativeStatusReassessment, type NativeStatusReassessmentScope } from "./native-status-reassessment-notifications.js";
|
||||
|
||||
export async function notifyNativeReviewState(tx: object, scope: NativeStatusReassessmentScope) {
|
||||
await notifyNativeReviewResponse(tx, scope);
|
||||
await notifyNativeStatusReassessment(tx, scope);
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js";
|
||||
import type { AutomaticCompletionReviewScope } from "../automatic-completion-review-notifications.js";
|
||||
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
|
||||
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
|
||||
@@ -138,6 +139,7 @@ async function dismissCandidates(db: Db, candidates: Awaited<ReturnType<typeof f
|
||||
.for("update");
|
||||
if (!issue) return;
|
||||
await notifyChatRunMilestones(tx, interaction);
|
||||
await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id, runId: decision.runId });
|
||||
const now = new Date();
|
||||
const [retired] = await tx
|
||||
.update(issueThreadInteractions)
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
|
||||
import { createHash, randomBytes, randomUUID } from "node:crypto";
|
||||
import { mkdir, mkdtemp } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
@@ -70,8 +71,9 @@ describe("native same-conversation chat attachment reuse", () => {
|
||||
);
|
||||
|
||||
beforeAll(async () => {
|
||||
temporary = await startEmbeddedPostgresTestDatabase("native-chat-reuse-");
|
||||
db = createDb(temporary.connectionString);
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_CHAT_REUSE_TEST_DATABASE_URL");
|
||||
if (!external) temporary = await startEmbeddedPostgresTestDatabase("native-chat-reuse-");
|
||||
db = createDb(external ?? temporary!.connectionString);
|
||||
const storageRoot = await mkdtemp(
|
||||
path.join(tmpdir(), "paperclip-chat-reuse-"),
|
||||
);
|
||||
@@ -295,6 +297,7 @@ describe("native same-conversation chat attachment reuse", () => {
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await db?.$client.end({ timeout: 1 });
|
||||
await temporary?.cleanup();
|
||||
});
|
||||
|
||||
|
||||
@@ -133,6 +133,22 @@ export function buildNativeCompletionContract(
|
||||
};
|
||||
}
|
||||
|
||||
function nativeCompletionContractLockKey(companyId: string, issueId: string): string {
|
||||
return `paperclip:native-completion-contract:${companyId}:${issueId}`;
|
||||
}
|
||||
|
||||
export async function lockCurrentNativeCompletionContract(tx: Db, input: {
|
||||
companyId: string; issueId: string; contractId: string;
|
||||
}): Promise<boolean> {
|
||||
const [lock] = await tx.execute(sql`select pg_try_advisory_xact_lock(hashtextextended(
|
||||
${nativeCompletionContractLockKey(input.companyId, input.issueId)}, 0)) as acquired`) as unknown as Array<{ acquired: boolean }>;
|
||||
if (!lock.acquired) return false;
|
||||
const [latest] = await tx.select({ id: completionContracts.id }).from(completionContracts)
|
||||
.where(and(eq(completionContracts.companyId, input.companyId), eq(completionContracts.issueId, input.issueId)))
|
||||
.orderBy(desc(completionContracts.revision)).limit(1);
|
||||
return latest?.id === input.contractId;
|
||||
}
|
||||
|
||||
export async function ensureNativeCompletionContract(input: {
|
||||
db: Db;
|
||||
companyId: string;
|
||||
@@ -151,11 +167,8 @@ export async function ensureNativeCompletionContract(input: {
|
||||
humanResponseId?: string | null;
|
||||
}) {
|
||||
return input.db.transaction(async (tx) => {
|
||||
await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${[
|
||||
"paperclip:native-completion-contract",
|
||||
input.companyId,
|
||||
input.issue.id,
|
||||
].join(":")}, 0))`);
|
||||
await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(
|
||||
${nativeCompletionContractLockKey(input.companyId, input.issue.id)}, 0))`);
|
||||
const policy = resolveNativeCompletionPolicy(input.issue);
|
||||
const latest = await tx
|
||||
.select()
|
||||
|
||||
@@ -102,26 +102,35 @@ function uuidRef(ref: string, prefixes: string[]): string | null {
|
||||
return null;
|
||||
}
|
||||
|
||||
export function readNativeEventEvidenceOutcome(value: unknown): "accepted" | "rejected" | null {
|
||||
const event = record(value);
|
||||
if (event.sourceKind !== "control_plane") return null;
|
||||
const outcome = text(record(event.payload).evidenceOutcome);
|
||||
return outcome === "accepted" || outcome === "rejected" ? outcome : null;
|
||||
}
|
||||
|
||||
async function classifyEvidenceRef(input: {
|
||||
db: Db;
|
||||
companyId: string;
|
||||
issueId: string;
|
||||
runId: string;
|
||||
ref: string;
|
||||
lockSources?: boolean;
|
||||
}): Promise<NativeEvidenceRefAssessment> {
|
||||
const handoffId = uuidRef(input.ref, ["case-handoff"]);
|
||||
if (handoffId) {
|
||||
const [row] = await input.db.select({ id: caseHandoffs.id, state: caseHandoffs.state }).from(caseHandoffs).where(and(
|
||||
const query = input.db.select({ id: caseHandoffs.id, state: caseHandoffs.state }).from(caseHandoffs).where(and(
|
||||
eq(caseHandoffs.id, handoffId), eq(caseHandoffs.companyId, input.companyId), eq(caseHandoffs.issueId, input.issueId),
|
||||
eq(caseHandoffs.sourceRunId, input.runId),
|
||||
)).limit(1);
|
||||
const [row] = await (input.lockSources ? query.for("share", { noWait: true }) : query);
|
||||
return { ref: input.ref, kind: "case_handoff", outcome: !row ? "missing" : row.state === "cancelled" ? "rejected" : "accepted",
|
||||
reasonCode: !row ? "case_handoff_missing" : row.state === "cancelled" ? "case_handoff_cancelled" : "case_handoff_persisted", durableRecordId: row?.id ?? null };
|
||||
}
|
||||
const eventMatch = /^event:(\d+)$/.exec(input.ref);
|
||||
if (eventMatch) {
|
||||
const cursor = Number(eventMatch[1]);
|
||||
const event = await input.db.select({
|
||||
const query = input.db.select({
|
||||
id: heartbeatRunEvents.id,
|
||||
eventType: heartbeatRunEvents.eventType,
|
||||
payload: heartbeatRunEvents.payload,
|
||||
@@ -131,16 +140,16 @@ async function classifyEvidenceRef(input: {
|
||||
eq(heartbeatRunEvents.companyId, input.companyId),
|
||||
eq(heartbeatRunEvents.runId, input.runId),
|
||||
or(eq(heartbeatRunEvents.seq, cursor), eq(heartbeatRunEvents.sourceSeq, cursor)),
|
||||
)).limit(1).then((rows) => rows[0] ?? null);
|
||||
)).limit(1);
|
||||
const [event] = await (input.lockSources ? query.for("share", { noWait: true }) : query);
|
||||
if (!event) {
|
||||
return { ref: input.ref, kind: "event", outcome: "missing", reasonCode: "durable_run_event_missing", durableRecordId: null };
|
||||
}
|
||||
const prpEvent = record(record(event.payload).prpEvent);
|
||||
const evidenceOutcome = text(record(prpEvent.payload).evidenceOutcome);
|
||||
if (prpEvent.sourceKind === "control_plane" && evidenceOutcome === "accepted") {
|
||||
const evidenceOutcome = readNativeEventEvidenceOutcome(record(event.payload).prpEvent);
|
||||
if (evidenceOutcome === "accepted") {
|
||||
return { ref: input.ref, kind: "event", outcome: "accepted", reasonCode: "control_plane_evidence_accepted", durableRecordId: String(event.id) };
|
||||
}
|
||||
if (prpEvent.sourceKind === "control_plane" && evidenceOutcome === "rejected") {
|
||||
if (evidenceOutcome === "rejected") {
|
||||
return { ref: input.ref, kind: "event", outcome: "rejected", reasonCode: "control_plane_evidence_rejected", durableRecordId: String(event.id) };
|
||||
}
|
||||
return {
|
||||
@@ -156,7 +165,7 @@ async function classifyEvidenceRef(input: {
|
||||
|
||||
const workProductId = uuidRef(input.ref, ["work_product", "work-product", "artifact"]);
|
||||
if (workProductId) {
|
||||
const row = await input.db.select({
|
||||
const query = input.db.select({
|
||||
id: issueWorkProducts.id,
|
||||
status: issueWorkProducts.status,
|
||||
reviewState: issueWorkProducts.reviewState,
|
||||
@@ -164,7 +173,8 @@ async function classifyEvidenceRef(input: {
|
||||
eq(issueWorkProducts.id, workProductId),
|
||||
eq(issueWorkProducts.companyId, input.companyId),
|
||||
eq(issueWorkProducts.issueId, input.issueId),
|
||||
)).limit(1).then((rows) => rows[0] ?? null);
|
||||
)).limit(1);
|
||||
const [row] = await (input.lockSources ? query.for("share", { noWait: true }) : query);
|
||||
if (!row) return { ref: input.ref, kind: "work_product", outcome: "missing", reasonCode: "work_product_missing", durableRecordId: null };
|
||||
if (
|
||||
["failed", "error", "cancelled", "rejected", "changes_requested"].includes(row.status)
|
||||
@@ -183,7 +193,7 @@ async function classifyEvidenceRef(input: {
|
||||
|
||||
const approvalId = uuidRef(input.ref, ["approval"]);
|
||||
if (approvalId) {
|
||||
const row = await input.db.select({ id: approvals.id, status: approvals.status })
|
||||
const query = input.db.select({ id: approvals.id, status: approvals.status })
|
||||
.from(issueApprovals)
|
||||
.innerJoin(approvals, eq(issueApprovals.approvalId, approvals.id))
|
||||
.where(and(
|
||||
@@ -191,7 +201,8 @@ async function classifyEvidenceRef(input: {
|
||||
eq(issueApprovals.issueId, input.issueId),
|
||||
eq(approvals.companyId, input.companyId),
|
||||
eq(approvals.id, approvalId),
|
||||
)).limit(1).then((rows) => rows[0] ?? null);
|
||||
)).limit(1);
|
||||
const [row] = await (input.lockSources ? query.for("share", { noWait: true }) : query);
|
||||
if (!row) return { ref: input.ref, kind: "approval", outcome: "missing", reasonCode: "approval_missing", durableRecordId: null };
|
||||
if (row.status === "approved") return { ref: input.ref, kind: "approval", outcome: "accepted", reasonCode: "approval_approved", durableRecordId: row.id };
|
||||
if (["rejected", "cancelled"].includes(row.status)) return { ref: input.ref, kind: "approval", outcome: "rejected", reasonCode: "approval_rejected", durableRecordId: row.id };
|
||||
@@ -200,7 +211,7 @@ async function classifyEvidenceRef(input: {
|
||||
|
||||
const interactionId = uuidRef(input.ref, ["interaction"]);
|
||||
if (interactionId) {
|
||||
const row = await input.db.select({
|
||||
const query = input.db.select({
|
||||
id: issueThreadInteractions.id,
|
||||
status: issueThreadInteractions.status,
|
||||
result: issueThreadInteractions.result,
|
||||
@@ -208,7 +219,8 @@ async function classifyEvidenceRef(input: {
|
||||
eq(issueThreadInteractions.id, interactionId),
|
||||
eq(issueThreadInteractions.companyId, input.companyId),
|
||||
eq(issueThreadInteractions.issueId, input.issueId),
|
||||
)).limit(1).then((rows) => rows[0] ?? null);
|
||||
)).limit(1);
|
||||
const [row] = await (input.lockSources ? query.for("share", { noWait: true }) : query);
|
||||
if (!row) return { ref: input.ref, kind: "interaction", outcome: "missing", reasonCode: "interaction_missing", durableRecordId: null };
|
||||
const outcome = text(record(row.result).outcome);
|
||||
if (["accepted", "answered", "resolved"].includes(outcome ?? "") || ["accepted", "answered"].includes(row.status)) {
|
||||
@@ -222,11 +234,12 @@ async function classifyEvidenceRef(input: {
|
||||
|
||||
const attachmentId = uuidRef(input.ref, ["attachment"]);
|
||||
if (attachmentId) {
|
||||
const row = await input.db.select({ id: issueAttachments.id }).from(issueAttachments).where(and(
|
||||
const query = input.db.select({ id: issueAttachments.id }).from(issueAttachments).where(and(
|
||||
eq(issueAttachments.id, attachmentId),
|
||||
eq(issueAttachments.companyId, input.companyId),
|
||||
eq(issueAttachments.issueId, input.issueId),
|
||||
)).limit(1).then((rows) => rows[0] ?? null);
|
||||
)).limit(1);
|
||||
const [row] = await (input.lockSources ? query.for("share", { noWait: true }) : query);
|
||||
return row
|
||||
? { ref: input.ref, kind: "attachment", outcome: "accepted", reasonCode: "attachment_persisted", durableRecordId: row.id }
|
||||
: { ref: input.ref, kind: "attachment", outcome: "missing", reasonCode: "attachment_missing", durableRecordId: null };
|
||||
@@ -241,10 +254,10 @@ async function classifyEvidenceRef(input: {
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Classifies model claims against immutable server-owned records. A model's
|
||||
* `satisfied` or `passed` value is retained as a claim but never becomes proof.
|
||||
*/
|
||||
|
||||
|
||||
|
||||
|
||||
export async function classifyNativeEvidence(input: {
|
||||
db: Db;
|
||||
companyId: string;
|
||||
@@ -252,6 +265,8 @@ export async function classifyNativeEvidence(input: {
|
||||
runId: string;
|
||||
contract: Record<string, unknown>;
|
||||
result: Record<string, unknown>;
|
||||
|
||||
lockSources?: boolean;
|
||||
}): Promise<NativeEvidenceAssessment> {
|
||||
const claim = record(input.result.completionClaim);
|
||||
const claimedCriteria = Array.isArray(claim.criteria) ? claim.criteria.map(record) : [];
|
||||
@@ -355,11 +370,12 @@ export async function classifyNativeEvidence(input: {
|
||||
const ignoredAttentionRequests = [...normalizedSignals.ignoredAttentionRequests];
|
||||
for (const request of normalizedSignals.actionableAttentionRequests) {
|
||||
if (request.ownerClass === "agent" && request.targetAgentId) {
|
||||
const targetExists = await input.db.select({ id: agents.id }).from(agents).where(and(
|
||||
const targetQuery = input.db.select({ id: agents.id }).from(agents).where(and(
|
||||
eq(agents.id, request.targetAgentId),
|
||||
eq(agents.companyId, input.companyId),
|
||||
)).limit(1).then((rows) => rows.length > 0);
|
||||
if (!targetExists) {
|
||||
)).limit(1);
|
||||
const targets = await (input.lockSources ? targetQuery.for("share", { noWait: true }) : targetQuery);
|
||||
if (targets.length === 0) {
|
||||
ignoredAttentionRequests.push({
|
||||
sourceIndex: request.sourceIndex,
|
||||
sourceKind: request.sourceKind,
|
||||
|
||||
@@ -1,43 +1,37 @@
|
||||
import { nativeStatusReassessmentSourceIsSuperseded, reassessNativeStatusFromSource } from "./native-status-reassessment.js";
|
||||
import { resolveNativeReconciliationStatus } from "./native-reconciliation-status.js";
|
||||
export { resolveNativeReconciliationStatus, type NativeReconciliationFacts } from "./native-reconciliation-status.js";
|
||||
import { pendingNativeStatusEffectsCondition } from "./native-pending-effect-work.js";
|
||||
import { notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js";
|
||||
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
|
||||
import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
|
||||
import { readRequiredQualityReview } from "../required-quality-review.js";
|
||||
import type { WorkspaceWorkers } from "../plugin-workspaces.js";
|
||||
import { recoverLegacyUnsafeWorkspaceExports } from "./native-workspace-export-recovery.js";
|
||||
import { hasPendingNativeChildCompletion } from "./native-child-completion-delivery.js";
|
||||
import { dismissAutomaticCompletionReviews, decisionHasRetiredAutomaticReview } from "./automatic-completion-reviews.js";
|
||||
import { dismissAutomaticCompletionReviews } from "./automatic-completion-reviews.js";
|
||||
import { logger } from "../../middleware/logger.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js";
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { and, asc, desc, eq, gt, inArray, isNotNull, isNull, lt, lte, ne, not, notInArray, or, sql, type SQL } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import {
|
||||
completionContracts,
|
||||
agents,
|
||||
heartbeatRunEvents,
|
||||
heartbeatRuns,
|
||||
issueWorkProducts,
|
||||
issues,
|
||||
nativeRunFinalizations,
|
||||
nativeRunResults,
|
||||
statusDecisionEffects,
|
||||
statusDecisions,
|
||||
workAssessments,
|
||||
workspaceOperations,
|
||||
} from "@paperclipai/db";
|
||||
import {
|
||||
finalizeNativeRun,
|
||||
pendingNativeGovernance,
|
||||
resolveNativeFinalizerStatus,
|
||||
recordNativeFinalizationFailure,
|
||||
repairCommittedNativeReviewResponse,
|
||||
repairCommittedNativeChatResponse,
|
||||
} from "./native-run-finalizer.js";
|
||||
import {
|
||||
commitNativeStatusDecision,
|
||||
dispatchPendingNativeStatusEffects,
|
||||
NativeStatusRaceError,
|
||||
} from "./status-decision-committer.js";
|
||||
import { issueRecoveryActionService } from "../issue-recovery-actions.js";
|
||||
import { issueService } from "../issues.js";
|
||||
@@ -50,17 +44,13 @@ import {
|
||||
readNativeWorkspaceSyncReference,
|
||||
} from "./native-workspace-sync.js";
|
||||
import type { EnvironmentRuntimeService } from "../environment-runtime.js";
|
||||
import { classifyNativeEvidence } from "./evidence-classifier.js";
|
||||
import { createNativeReconciliationPassTiming } from "./native-reconciliation-timing.js";
|
||||
import { recordNativeWorkAssessment } from "./work-assessments.js";
|
||||
import {
|
||||
isNativeRunnerOwnershipHeld,
|
||||
nativeRunnerOwnershipNotHeldCondition,
|
||||
} from "./native-runner-ownership.js";
|
||||
import {
|
||||
NATIVE_STATUS_ARBITER_POLICY_VERSION,
|
||||
type NativeAuthoritativeIssueStatus,
|
||||
type NativeStatusDecision,
|
||||
} from "./status-arbiter.js";
|
||||
|
||||
import { getNativeReviewAssignment } from "./native-review-participant.js";
|
||||
@@ -70,125 +60,12 @@ import { readRunCancellation } from "../run-cancellation.js";
|
||||
import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
|
||||
import { retireSupersededNativeRetry } from "./native-superseded-retry.js";
|
||||
|
||||
export type NativeReconciliationFacts = {
|
||||
equivalentAttentionFamily?: boolean;
|
||||
canonicalReplay?: boolean;
|
||||
callerMaterialConflict?: boolean;
|
||||
workspaceOperationPending?: boolean;
|
||||
undeliveredEffectCount?: number;
|
||||
authoritativeStatusChanged?: boolean;
|
||||
newEvidenceSatisfiesContract?: boolean;
|
||||
hasPendingChildCompletion?: boolean;
|
||||
dependencyResolved?: boolean;
|
||||
authorizedResume?: boolean;
|
||||
statusVersionAdvanced?: boolean;
|
||||
};
|
||||
|
||||
function record(value: unknown): Record<string, unknown> {
|
||||
return value && typeof value === "object" && !Array.isArray(value)
|
||||
? value as Record<string, unknown>
|
||||
: {};
|
||||
}
|
||||
|
||||
/** Append-only recovery/reconciliation re-arbitration from durable facts. */
|
||||
export function resolveNativeReconciliationStatus(input: {
|
||||
facts: NativeReconciliationFacts;
|
||||
priorIssueStatus: NativeAuthoritativeIssueStatus;
|
||||
agentId: string;
|
||||
}): NativeStatusDecision {
|
||||
const preserve = (reasonCode: string, effects: NativeStatusDecision["effects"]): NativeStatusDecision => ({
|
||||
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
|
||||
statusAction: "preserve",
|
||||
toStatus: input.priorIssueStatus,
|
||||
reasonCode,
|
||||
unblockDescriptor: null,
|
||||
effects,
|
||||
});
|
||||
const continuation = (reasonCode: string, effects: NativeStatusDecision["effects"] = []): NativeStatusDecision => ({
|
||||
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
|
||||
statusAction: "in_progress",
|
||||
toStatus: "in_progress",
|
||||
reasonCode,
|
||||
unblockDescriptor: null,
|
||||
effects: [{
|
||||
kind: "enqueue_continuation",
|
||||
continuationKind: "same_agent",
|
||||
summary: `Continue native reconciliation after ${reasonCode}.`,
|
||||
idempotencyKey: `native-reconciliation:${reasonCode}`,
|
||||
agentId: input.agentId,
|
||||
}, ...effects],
|
||||
});
|
||||
|
||||
if (input.facts.equivalentAttentionFamily) {
|
||||
return preserve("attention_duplicate_suppressed", [{ kind: "link_canonical_request" }]);
|
||||
}
|
||||
|
||||
if (input.facts.callerMaterialConflict) {
|
||||
return preserve("structured_result_replay_conflict", [{
|
||||
kind: "record_finalization_error",
|
||||
cause: "structured_result_replay_conflict",
|
||||
nextAction: "Use a fresh canonical result identity.",
|
||||
agentId: input.agentId,
|
||||
}]);
|
||||
}
|
||||
if (input.facts.authoritativeStatusChanged) {
|
||||
return preserve("prior_status_terminal_preserved", [{ kind: "append_superseding_assessment" }]);
|
||||
}
|
||||
if (input.facts.statusVersionAdvanced) {
|
||||
return preserve("arbitration_conflict_reloaded", [
|
||||
{ kind: "increment_status_version" },
|
||||
{ kind: "schedule_reconciliation" },
|
||||
]);
|
||||
}
|
||||
if (input.facts.newEvidenceSatisfiesContract) {
|
||||
if (input.facts.hasPendingChildCompletion) {
|
||||
return {
|
||||
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
|
||||
statusAction: "in_progress",
|
||||
toStatus: "in_progress",
|
||||
reasonCode: "native_child_completion_pending",
|
||||
unblockDescriptor: null,
|
||||
// The existing child-result wake owns the continuation.
|
||||
effects: [{ kind: "release_checkout" }],
|
||||
};
|
||||
}
|
||||
return {
|
||||
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
|
||||
statusAction: "done",
|
||||
toStatus: "done",
|
||||
reasonCode: "decision_superseded_by_new_evidence",
|
||||
unblockDescriptor: null,
|
||||
effects: [{ kind: "release_checkout" }],
|
||||
};
|
||||
}
|
||||
if (input.facts.dependencyResolved) return continuation("dependency_resolved");
|
||||
if (input.facts.authorizedResume) return continuation("authorized_resume");
|
||||
if ((input.facts.undeliveredEffectCount ?? 0) > 0) {
|
||||
return continuation("live_continuation_registered", [{ kind: "dispatch_pending_effect" }]);
|
||||
}
|
||||
if (input.facts.workspaceOperationPending) {
|
||||
return {
|
||||
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
|
||||
statusAction: "done",
|
||||
toStatus: "done",
|
||||
reasonCode: "completion_contract_satisfied",
|
||||
unblockDescriptor: null,
|
||||
effects: [{ kind: "resume_workspace_operation" }, { kind: "release_checkout" }],
|
||||
};
|
||||
}
|
||||
if (input.facts.canonicalReplay) {
|
||||
return {
|
||||
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
|
||||
statusAction: "done",
|
||||
toStatus: "done",
|
||||
reasonCode: "completion_contract_satisfied",
|
||||
unblockDescriptor: null,
|
||||
effects: [{ kind: "release_checkout" }],
|
||||
};
|
||||
}
|
||||
throw new Error("native_reconciliation_facts_invalid");
|
||||
}
|
||||
|
||||
export type NativeSessionResumeClaim = { runId: string; leaseOwner: string };
|
||||
|
||||
type NativeCleanupOutcome = {
|
||||
@@ -646,6 +523,7 @@ type NativeReconciliationOptions = {
|
||||
skipNativePendingEffects?: boolean;
|
||||
skipNativeResponsePresentation?: boolean;
|
||||
skipNativeReviewResponse?: boolean;
|
||||
skipNativeStatusReassessment?: boolean;
|
||||
pluginWorkerManager?: WorkspaceWorkers;
|
||||
environmentRuntime?: EnvironmentRuntimeService;
|
||||
onWorkspaceSettled?: (input: {
|
||||
@@ -830,39 +708,12 @@ async function reconcileNativeFinalizationsPass(
|
||||
continue;
|
||||
}
|
||||
if (row.assessmentId) {
|
||||
const assessment = await db.select({
|
||||
id: workAssessments.id,
|
||||
contractId: workAssessments.contractId,
|
||||
resultId: workAssessments.resultId,
|
||||
policyVersion: workAssessments.policyVersion,
|
||||
priorIssueStatus: workAssessments.priorIssueStatus,
|
||||
priorStatusVersion: workAssessments.priorStatusVersion,
|
||||
priorDecisionId: workAssessments.priorDecisionId,
|
||||
assessmentJson: workAssessments.assessmentJson,
|
||||
createdAt: workAssessments.createdAt,
|
||||
}).from(workAssessments).where(and(
|
||||
eq(workAssessments.id, row.assessmentId),
|
||||
eq(workAssessments.companyId, row.companyId),
|
||||
eq(workAssessments.issueId, row.issueId),
|
||||
)).limit(1).then((entries) => entries[0] ?? null);
|
||||
const changedEvidence = assessment
|
||||
? await db.select({ id: issueWorkProducts.id }).from(issueWorkProducts).where(and(
|
||||
eq(issueWorkProducts.companyId, row.companyId),
|
||||
eq(issueWorkProducts.issueId, row.issueId),
|
||||
or(
|
||||
gt(issueWorkProducts.createdAt, assessment.createdAt),
|
||||
gt(issueWorkProducts.updatedAt, assessment.createdAt),
|
||||
),
|
||||
)).limit(1).then((entries) => entries[0] ?? null)
|
||||
: null;
|
||||
const currentDecision = row.decisionId
|
||||
const currentDecision = !options.skipNativeReviewResponse && row.coordinatorPhase === "committed" && row.decisionId
|
||||
? await db.select().from(statusDecisions).where(and(
|
||||
eq(statusDecisions.id, row.decisionId),
|
||||
eq(statusDecisions.companyId, row.companyId),
|
||||
eq(statusDecisions.id, row.decisionId), eq(statusDecisions.companyId, row.companyId),
|
||||
eq(statusDecisions.issueId, row.issueId),
|
||||
)).limit(1).then((entries) => entries[0] ?? null)
|
||||
)).limit(1).then(entries => entries[0] ?? null)
|
||||
: null;
|
||||
const currentDecisionJson = record(currentDecision?.decisionJson);
|
||||
if (row.coordinatorPhase === "committed" && !options.skipNativeResponsePresentation) {
|
||||
// A later decision can supersede this run's task status, but cannot
|
||||
// erase its accepted, still-authorized response. Repair presentation
|
||||
@@ -872,12 +723,12 @@ async function reconcileNativeFinalizationsPass(
|
||||
});
|
||||
}
|
||||
if (
|
||||
!options.skipNativeReviewResponse &&
|
||||
row.coordinatorPhase === "committed" &&
|
||||
row.decisionId &&
|
||||
row.resultId &&
|
||||
row.assessmentId &&
|
||||
currentDecisionJson.externalChatReviewPresentation &&
|
||||
!options.skipNativeReviewResponse
|
||||
record(currentDecision?.decisionJson).externalChatReviewPresentation
|
||||
) {
|
||||
// A later chat turn may retain the same pending review. Recover the
|
||||
// earlier response independently before status reconciliation skips
|
||||
@@ -891,168 +742,15 @@ async function reconcileNativeFinalizationsPass(
|
||||
assessmentId: row.assessmentId,
|
||||
});
|
||||
}
|
||||
if (row.coordinatorPhase === "committed" && row.decisionId && !currentDecision) {
|
||||
throw new Error("native_committed_decision_missing");
|
||||
}
|
||||
const supersededCommittedDecision = row.coordinatorPhase === "committed"
|
||||
&& row.decisionId !== null
|
||||
&& row.issueDecisionId !== null
|
||||
&& row.issueDecisionId !== row.decisionId
|
||||
&& !(
|
||||
currentDecisionJson.statusAction === "preserve"
|
||||
&& assessment?.priorDecisionId === row.issueDecisionId
|
||||
);
|
||||
if (supersededCommittedDecision) continue;
|
||||
const persistedProjectedVersion = Number(currentDecisionJson.projectedStatusVersion);
|
||||
const expectedProjectedVersion = currentDecision
|
||||
? Number.isFinite(persistedProjectedVersion)
|
||||
? persistedProjectedVersion
|
||||
: Number(currentDecision.decisionVersion) - (currentDecisionJson.statusAction === "preserve" ? 1 : 0)
|
||||
: null;
|
||||
const issueMatchesCurrentDecision = !!assessment
|
||||
&& !!currentDecision
|
||||
&& row.issueDecisionId === row.decisionId
|
||||
&& currentDecision.assessmentId === assessment.id
|
||||
&& currentDecision.toStatus === row.issueStatus
|
||||
&& expectedProjectedVersion === Number(row.issueStatusVersion);
|
||||
const authoritativeStatusChanged = !!assessment && !issueMatchesCurrentDecision && (
|
||||
assessment.priorIssueStatus !== row.issueStatus
|
||||
|| Number(assessment.priorStatusVersion) !== Number(row.issueStatusVersion)
|
||||
);
|
||||
const retiredAutomaticReview = issueMatchesCurrentDecision && currentDecision
|
||||
? await decisionHasRetiredAutomaticReview(db, currentDecision) : false;
|
||||
let reassessment = null;
|
||||
let resultRow = null;
|
||||
let contractRow = null;
|
||||
if (assessment && (authoritativeStatusChanged || changedEvidence || retiredAutomaticReview)) {
|
||||
[resultRow, contractRow] = await Promise.all([
|
||||
db.select().from(nativeRunResults).where(and(
|
||||
eq(nativeRunResults.id, assessment.resultId),
|
||||
eq(nativeRunResults.companyId, row.companyId),
|
||||
eq(nativeRunResults.issueId, row.issueId),
|
||||
eq(nativeRunResults.runId, row.runId),
|
||||
)).limit(1).then((entries) => entries[0] ?? null),
|
||||
db.select().from(completionContracts).where(and(
|
||||
eq(completionContracts.id, assessment.contractId),
|
||||
eq(completionContracts.companyId, row.companyId),
|
||||
eq(completionContracts.issueId, row.issueId),
|
||||
)).limit(1).then((entries) => entries[0] ?? null),
|
||||
]);
|
||||
if (!resultRow || !contractRow) throw new Error("native_reconciliation_evidence_binding_missing");
|
||||
reassessment = await classifyNativeEvidence({
|
||||
db,
|
||||
companyId: row.companyId,
|
||||
issueId: row.issueId,
|
||||
runId: row.runId,
|
||||
contract: record(contractRow.contractJson),
|
||||
result: record(record(resultRow.resultJson).result),
|
||||
});
|
||||
}
|
||||
const previousAssessment = record(assessment?.assessmentJson);
|
||||
const newEvidenceSatisfiesContract = !!changedEvidence
|
||||
&& previousAssessment.allCriteriaSatisfied !== true
|
||||
&& reassessment?.allCriteriaSatisfied === true
|
||||
&& reassessment.verificationPassed === true;
|
||||
const facts: NativeReconciliationFacts = authoritativeStatusChanged
|
||||
? { authoritativeStatusChanged: true }
|
||||
: newEvidenceSatisfiesContract
|
||||
? { newEvidenceSatisfiesContract: true }
|
||||
: {};
|
||||
if (Object.keys(facts).length > 0 || retiredAutomaticReview) {
|
||||
if (!assessment || !reassessment || !resultRow || !contractRow) {
|
||||
throw new Error("native_reconciliation_reassessment_missing");
|
||||
const source = { ...row, assessmentId: row.assessmentId };
|
||||
if (options.skipNativeStatusReassessment) {
|
||||
if (await nativeStatusReassessmentSourceIsSuperseded(db, source)) continue;
|
||||
} else {
|
||||
const reassessed = await reassessNativeStatusFromSource(db, source);
|
||||
if (reassessed.kind !== "unchanged") {
|
||||
if (reassessed.kind === "committed") results.push(reassessed.result);
|
||||
continue;
|
||||
}
|
||||
const currentIssue = retiredAutomaticReview
|
||||
? await db.select().from(issues).where(and(eq(issues.id, row.issueId), eq(issues.companyId, row.companyId))).then((entries) => entries[0])
|
||||
: null;
|
||||
const readiness = retiredAutomaticReview ? await issueService(db).getDependencyReadiness(row.issueId, db) : null;
|
||||
const currentRun = retiredAutomaticReview ? await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, row.runId)).then((entries) => entries[0]) : null;
|
||||
// Never replay an old result over a new run or a later task contract.
|
||||
const latestContract = retiredAutomaticReview ? await db.select({ id: completionContracts.id }).from(completionContracts)
|
||||
.where(and(eq(completionContracts.companyId, row.companyId), eq(completionContracts.issueId, row.issueId)))
|
||||
.orderBy(desc(completionContracts.revision)).limit(1).then((entries) => entries[0]) : null;
|
||||
if (retiredAutomaticReview && (!currentIssue || currentRun?.status !== "succeeded"
|
||||
|| latestContract?.id !== contractRow.id
|
||||
|| (currentIssue.executionRunId && currentIssue.executionRunId !== row.runId))) continue;
|
||||
// A committed decision proves the original barrier passed. If a later
|
||||
// workspace operation exists, do not ignore a pending or failed retry.
|
||||
const reviewBarrier = retiredAutomaticReview ? await db.select({ status: workspaceOperations.status })
|
||||
.from(workspaceOperations).where(and(eq(workspaceOperations.companyId, row.companyId),
|
||||
eq(workspaceOperations.heartbeatRunId, row.runId), eq(workspaceOperations.phase, "workspace_finalize")))
|
||||
.orderBy(desc(workspaceOperations.createdAt)).limit(1).then((entries) => entries[0]) : null;
|
||||
if (reviewBarrier && reviewBarrier.status !== "succeeded") continue;
|
||||
const childCompletionRecipient = {
|
||||
companyId: row.companyId, issueId: row.issueId, agentId: row.agentId, runId: row.runId,
|
||||
sourceIntentId: typeof row.nativeStatusWakeIntentId === "string" ? row.nativeStatusWakeIntentId : null,
|
||||
};
|
||||
const hasPendingChildCompletion = await hasPendingNativeChildCompletion(db, childCompletionRecipient);
|
||||
const reassessmentRow = await recordNativeWorkAssessment({
|
||||
db,
|
||||
companyId: row.companyId,
|
||||
issueId: row.issueId,
|
||||
runId: row.runId,
|
||||
turnId: resultRow.turnId,
|
||||
contractId: contractRow.id,
|
||||
contractCanonicalSha256: contractRow.canonicalSha256,
|
||||
resultId: resultRow.id,
|
||||
resultCanonicalSha256: resultRow.canonicalSha256,
|
||||
priorIssueStatus: row.issueStatus,
|
||||
priorStatusVersion: Number(row.issueStatusVersion),
|
||||
priorDecisionId: row.issueDecisionId,
|
||||
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
|
||||
assessment: reassessment,
|
||||
supersedesAssessmentId: assessment.id,
|
||||
});
|
||||
const qualityReview = currentIssue ? await readRequiredQualityReview(db, currentIssue) : null;
|
||||
let decision = retiredAutomaticReview && currentIssue
|
||||
? resolveNativeFinalizerStatus({
|
||||
hasPendingChildCompletion,
|
||||
requiredQualityReviewSatisfied: qualityReview?.satisfied,
|
||||
assessment: reassessment, terminalState: "succeeded", workspaceFinalizeStatus: "succeeded",
|
||||
governanceGate: await pendingNativeGovernance({ db, companyId: row.companyId, issueId: row.issueId,
|
||||
runId: row.runId, executionState: record(currentIssue.executionState) }),
|
||||
completionClaimPolicyAccepted: contractRow.risk === "low" && contractRow.completionAuthority === "agent_claim_policy",
|
||||
hasUnresolvedIssueBlockers: (readiness?.unresolvedBlockerCount ?? 0) > 0,
|
||||
reviewOwnerUserId: currentIssue.responsibleUserId ?? currentIssue.createdByUserId,
|
||||
priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus, agentId: row.agentId,
|
||||
})
|
||||
: resolveNativeReconciliationStatus({
|
||||
facts: { ...facts, hasPendingChildCompletion },
|
||||
priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus, agentId: row.agentId,
|
||||
});
|
||||
if (decision.statusAction === "done" && qualityReview?.satisfied === false) {
|
||||
decision = resolveNativeFinalizerStatus({ assessment: reassessment, terminalState: "succeeded", workspaceFinalizeStatus: "succeeded",
|
||||
requiredQualityReviewSatisfied: false, priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus, agentId: row.agentId });
|
||||
}
|
||||
let committed: Awaited<ReturnType<typeof commitNativeStatusDecision>>;
|
||||
try {
|
||||
committed = await commitNativeStatusDecision({
|
||||
db,
|
||||
companyId: row.companyId,
|
||||
issueId: row.issueId,
|
||||
runId: row.runId,
|
||||
assessmentId: reassessmentRow.id,
|
||||
priorStatus: row.issueStatus,
|
||||
priorStatusVersion: Number(row.issueStatusVersion),
|
||||
priorDecisionId: row.issueDecisionId,
|
||||
decision,
|
||||
requireNoPendingChildCompletion: decision.statusAction === "done" ? childCompletionRecipient : undefined,
|
||||
supersedesCommittedDecisionId: row.coordinatorPhase === "committed"
|
||||
? row.decisionId ?? undefined
|
||||
: undefined,
|
||||
});
|
||||
} catch (error) {
|
||||
if (error instanceof NativeStatusRaceError) continue;
|
||||
throw error;
|
||||
}
|
||||
results.push({
|
||||
runId: row.runId,
|
||||
phase: "committed",
|
||||
reconciliationAction: decision.reasonCode,
|
||||
decisionId: committed.decision.id,
|
||||
reconciliationDecision: decision,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
}
|
||||
const barrier = await db.select({ status: workspaceOperations.status })
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { and, eq, inArray, notInArray } from "drizzle-orm";
|
||||
import { approvals, issueApprovals, issueThreadInteractions, issues, type Db } from "@paperclipai/db";
|
||||
import { approvals, issueApprovals, issueComments, issueThreadInteractions, issues, type Db } from "@paperclipai/db";
|
||||
import { activeIssueInteractionCondition } from "../issue-question-context.js";
|
||||
import { isConversation } from "../agent-conversations.js";
|
||||
import type { NativeGovernanceGate } from "./status-arbiter.js";
|
||||
@@ -8,6 +8,22 @@ function record(value: unknown): Record<string, unknown> {
|
||||
return typeof value === "object" && value !== null ? value as Record<string, unknown> : {};
|
||||
}
|
||||
|
||||
|
||||
export async function lockNativeGovernanceSources(input: { db: Db; companyId: string; issueId: string }): Promise<void> {
|
||||
const interactions = await input.db.select({ kind: issueThreadInteractions.kind }).from(issueThreadInteractions)
|
||||
.where(and(eq(issueThreadInteractions.companyId, input.companyId), eq(issueThreadInteractions.issueId, input.issueId)))
|
||||
.for("share", { noWait: true });
|
||||
if (interactions.some(interaction => interaction.kind === "ask_user_questions")) {
|
||||
await input.db.select({ id: issueComments.id }).from(issueComments)
|
||||
.where(and(eq(issueComments.companyId, input.companyId), eq(issueComments.issueId, input.issueId)))
|
||||
.for("share", { noWait: true });
|
||||
}
|
||||
await input.db.select({ id: approvals.id }).from(issueApprovals)
|
||||
.innerJoin(approvals, eq(issueApprovals.approvalId, approvals.id))
|
||||
.where(and(eq(issueApprovals.companyId, input.companyId), eq(issueApprovals.issueId, input.issueId),
|
||||
eq(approvals.companyId, input.companyId))).for("share", { noWait: true });
|
||||
}
|
||||
|
||||
export async function pendingNativeGovernance(input: {
|
||||
db: Db;
|
||||
companyId: string;
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
import { NATIVE_STATUS_ARBITER_POLICY_VERSION, type NativeAuthoritativeIssueStatus, type NativeStatusDecision } from "./status-arbiter.js";
|
||||
|
||||
export type NativeReconciliationFacts = {
|
||||
equivalentAttentionFamily?: boolean;
|
||||
canonicalReplay?: boolean;
|
||||
callerMaterialConflict?: boolean;
|
||||
workspaceOperationPending?: boolean;
|
||||
undeliveredEffectCount?: number;
|
||||
authoritativeStatusChanged?: boolean;
|
||||
newEvidenceSatisfiesContract?: boolean;
|
||||
hasPendingChildCompletion?: boolean;
|
||||
dependencyResolved?: boolean;
|
||||
authorizedResume?: boolean;
|
||||
statusVersionAdvanced?: boolean;
|
||||
};
|
||||
|
||||
|
||||
export function resolveNativeReconciliationStatus(input: {
|
||||
facts: NativeReconciliationFacts;
|
||||
priorIssueStatus: NativeAuthoritativeIssueStatus;
|
||||
agentId: string;
|
||||
}): NativeStatusDecision {
|
||||
const preserve = (reasonCode: string, effects: NativeStatusDecision["effects"]): NativeStatusDecision => ({
|
||||
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
|
||||
statusAction: "preserve",
|
||||
toStatus: input.priorIssueStatus,
|
||||
reasonCode,
|
||||
unblockDescriptor: null,
|
||||
effects,
|
||||
});
|
||||
const continuation = (reasonCode: string, effects: NativeStatusDecision["effects"] = []): NativeStatusDecision => ({
|
||||
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
|
||||
statusAction: "in_progress",
|
||||
toStatus: "in_progress",
|
||||
reasonCode,
|
||||
unblockDescriptor: null,
|
||||
effects: [{
|
||||
kind: "enqueue_continuation",
|
||||
continuationKind: "same_agent",
|
||||
summary: `Continue native reconciliation after ${reasonCode}.`,
|
||||
idempotencyKey: `native-reconciliation:${reasonCode}`,
|
||||
agentId: input.agentId,
|
||||
}, ...effects],
|
||||
});
|
||||
|
||||
if (input.facts.equivalentAttentionFamily) {
|
||||
return preserve("attention_duplicate_suppressed", [{ kind: "link_canonical_request" }]);
|
||||
}
|
||||
|
||||
if (input.facts.callerMaterialConflict) {
|
||||
return preserve("structured_result_replay_conflict", [{
|
||||
kind: "record_finalization_error",
|
||||
cause: "structured_result_replay_conflict",
|
||||
nextAction: "Use a fresh canonical result identity.",
|
||||
agentId: input.agentId,
|
||||
}]);
|
||||
}
|
||||
if (input.facts.authoritativeStatusChanged) {
|
||||
return preserve("prior_status_terminal_preserved", [{ kind: "append_superseding_assessment" }]);
|
||||
}
|
||||
if (input.facts.statusVersionAdvanced) {
|
||||
return preserve("arbitration_conflict_reloaded", [
|
||||
{ kind: "increment_status_version" },
|
||||
{ kind: "schedule_reconciliation" },
|
||||
]);
|
||||
}
|
||||
if (input.facts.newEvidenceSatisfiesContract) {
|
||||
if (input.facts.hasPendingChildCompletion) {
|
||||
return {
|
||||
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
|
||||
statusAction: "in_progress",
|
||||
toStatus: "in_progress",
|
||||
reasonCode: "native_child_completion_pending",
|
||||
unblockDescriptor: null,
|
||||
effects: [{ kind: "release_checkout" }],
|
||||
};
|
||||
}
|
||||
return {
|
||||
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
|
||||
statusAction: "done",
|
||||
toStatus: "done",
|
||||
reasonCode: "decision_superseded_by_new_evidence",
|
||||
unblockDescriptor: null,
|
||||
effects: [{ kind: "release_checkout" }],
|
||||
};
|
||||
}
|
||||
if (input.facts.dependencyResolved) return continuation("dependency_resolved");
|
||||
if (input.facts.authorizedResume) return continuation("authorized_resume");
|
||||
if ((input.facts.undeliveredEffectCount ?? 0) > 0) {
|
||||
return continuation("live_continuation_registered", [{ kind: "dispatch_pending_effect" }]);
|
||||
}
|
||||
if (input.facts.workspaceOperationPending) {
|
||||
return {
|
||||
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
|
||||
statusAction: "done",
|
||||
toStatus: "done",
|
||||
reasonCode: "completion_contract_satisfied",
|
||||
unblockDescriptor: null,
|
||||
effects: [{ kind: "resume_workspace_operation" }, { kind: "release_checkout" }],
|
||||
};
|
||||
}
|
||||
if (input.facts.canonicalReplay) {
|
||||
return {
|
||||
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
|
||||
statusAction: "done",
|
||||
toStatus: "done",
|
||||
reasonCode: "completion_contract_satisfied",
|
||||
unblockDescriptor: null,
|
||||
effects: [{ kind: "release_checkout" }],
|
||||
};
|
||||
}
|
||||
throw new Error("native_reconciliation_facts_invalid");
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js";
|
||||
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
|
||||
import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
|
||||
@@ -942,6 +943,7 @@ export async function claimNativeRestartRecoveries(input: {
|
||||
if (updatedRun && isNewTerminalRunFailure(row.run, updatedRun)) {
|
||||
terminalRunToReport = updatedRun;
|
||||
}
|
||||
await notifyNativeStatusReassessment(tx, { companyId: row.run.companyId, issueId: row.coordinator.issueId });
|
||||
await tx
|
||||
.update(issues)
|
||||
.set({ executionRunId: null, updatedAt: now })
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js";
|
||||
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
|
||||
import { logger } from "../../middleware/logger.js";
|
||||
@@ -487,6 +488,7 @@ export async function reconcileSafeNativeReplacements(
|
||||
.update(heartbeatRuns)
|
||||
.set({ executionStatusDeliveryId: randomUUID() })
|
||||
.where(eq(heartbeatRuns.id, run.id));
|
||||
await notifyNativeStatusReassessment(tx, { companyId: run.companyId, issueId: task.id });
|
||||
// The successor gets task authority only through the existing admission gate.
|
||||
await tx
|
||||
.update(issues)
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js";
|
||||
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js";
|
||||
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
|
||||
@@ -9882,6 +9883,7 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
stillOwnsTask &&
|
||||
phase === "terminal_failure"
|
||||
) {
|
||||
await notifyNativeStatusReassessment(tx, { companyId: input.execution.binding.companyId, issueId: input.execution.binding.issueId });
|
||||
await tx
|
||||
.update(issues)
|
||||
.set({ executionRunId: null, checkoutRunId: null, updatedAt: now })
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
import { and, asc, eq, exists, gt, inArray, isNotNull, isNull, ne, not, notExists, or, sql } from "drizzle-orm";
|
||||
import { heartbeatRuns, issues, issueThreadInteractions, issueWorkProducts, nativeRunFinalizations,
|
||||
statusDecisionEffects, statusDecisions, workAssessments, type Db } from "@paperclipai/db";
|
||||
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
|
||||
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
|
||||
import type { NativeStatusReassessmentScope } from "../native-status-reassessment-notifications.js";
|
||||
import { registerScopedDeadlineWork } from "../scoped-deadline-work.js";
|
||||
import { nativeRunnerOwnershipNotHeldCondition } from "./native-runner-ownership.js";
|
||||
import { reassessNativeStatusFromSource } from "./native-status-reassessment.js";
|
||||
|
||||
type Entry = { id: string; companyId: string; issueId: string; agentId: string; dueAt: number };
|
||||
|
||||
export function nativeStatusReassessmentWork(db: Db) {
|
||||
function candidates(scope?: NativeStatusReassessmentScope, after?: string, runId?: string) {
|
||||
const projectedVersion = sql`case
|
||||
when jsonb_typeof(${statusDecisions.decisionJson}->'projectedStatusVersion') = 'number'
|
||||
then (${statusDecisions.decisionJson}->>'projectedStatusVersion')::numeric
|
||||
else ${statusDecisions.decisionVersion} - case when ${statusDecisions.decisionJson}->>'statusAction' = 'preserve' then 1 else 0 end
|
||||
end`;
|
||||
const matchesDecision = sql`coalesce(${issues.lastStatusDecisionId} = ${nativeRunFinalizations.decisionId}
|
||||
and ${statusDecisions.assessmentId} = ${workAssessments.id} and ${statusDecisions.toStatus} = ${issues.status}
|
||||
and ${projectedVersion} = ${issues.statusVersion}, false)`;
|
||||
const statusChanged = and(not(matchesDecision), or(ne(workAssessments.priorIssueStatus, issues.status),
|
||||
ne(workAssessments.priorStatusVersion, issues.statusVersion)));
|
||||
const evidenceChanged = exists(db.select({ id: issueWorkProducts.id }).from(issueWorkProducts).where(and(
|
||||
eq(issueWorkProducts.companyId, nativeRunFinalizations.companyId), eq(issueWorkProducts.issueId, nativeRunFinalizations.issueId),
|
||||
or(gt(issueWorkProducts.createdAt, workAssessments.createdAt), gt(issueWorkProducts.updatedAt, workAssessments.createdAt)),
|
||||
)));
|
||||
const retiredReview = exists(db.select({ id: issueThreadInteractions.id }).from(issueThreadInteractions).where(and(
|
||||
eq(issueThreadInteractions.companyId, nativeRunFinalizations.companyId), eq(issueThreadInteractions.issueId, nativeRunFinalizations.issueId),
|
||||
eq(issueThreadInteractions.status, "cancelled"), sql`${issueThreadInteractions.result}->>'reason' = 'automatic_completion_review_removed'`,
|
||||
sql`(${issueThreadInteractions.payload}->'target'->>'revisionId' = ${statusDecisions.id}::text
|
||||
or ${statusDecisions.decisionJson}->'effects' @> jsonb_build_array(jsonb_build_object('gate',
|
||||
jsonb_build_object('kind', 'interaction', 'id', ${issueThreadInteractions.id}::text))))`,
|
||||
)));
|
||||
return db.select({ runId: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId,
|
||||
issueId: issues.id, issueStatus: issues.status, issueStatusVersion: issues.statusVersion,
|
||||
issueDecisionId: issues.lastStatusDecisionId, coordinatorPhase: nativeRunFinalizations.phase,
|
||||
assessmentId: workAssessments.id, decisionId: nativeRunFinalizations.decisionId, nativeStatusWakeIntentId: sql<unknown>`${heartbeatRuns.contextSnapshot}->'nativeStatusWakeIntentId'`,
|
||||
nextAttemptAt: nativeRunFinalizations.nextAttemptAt, leaseOwner: nativeRunFinalizations.leaseOwner,
|
||||
leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt,
|
||||
}).from(nativeRunFinalizations)
|
||||
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, nativeRunFinalizations.runId),
|
||||
eq(heartbeatRuns.companyId, nativeRunFinalizations.companyId), eq(heartbeatRuns.nativeIssueId, nativeRunFinalizations.issueId)))
|
||||
.innerJoin(issues, and(eq(issues.id, nativeRunFinalizations.issueId), eq(issues.companyId, nativeRunFinalizations.companyId)))
|
||||
.innerJoin(workAssessments, and(eq(workAssessments.id, nativeRunFinalizations.assessmentId),
|
||||
eq(workAssessments.companyId, nativeRunFinalizations.companyId), eq(workAssessments.issueId, nativeRunFinalizations.issueId),
|
||||
eq(workAssessments.runId, nativeRunFinalizations.runId)))
|
||||
.leftJoin(statusDecisions, and(eq(statusDecisions.id, nativeRunFinalizations.decisionId),
|
||||
eq(statusDecisions.companyId, nativeRunFinalizations.companyId), eq(statusDecisions.issueId, nativeRunFinalizations.issueId)))
|
||||
.where(and(eq(heartbeatRuns.runtimeMode, "native"), nativeRunnerOwnershipNotHeldCondition(),
|
||||
ne(nativeRunFinalizations.phase, "terminal_failure"), isNotNull(nativeRunFinalizations.resultId),
|
||||
or(isNull(nativeRunFinalizations.leaseOwner), isNotNull(nativeRunFinalizations.leaseExpiresAt)),
|
||||
sql`not coalesce(${nativeRunFinalizations.phase} = 'committed'
|
||||
and ${nativeRunFinalizations.decisionId} is not null and ${issues.lastStatusDecisionId} is not null
|
||||
and ${issues.lastStatusDecisionId} <> ${nativeRunFinalizations.decisionId}
|
||||
and not coalesce(${statusDecisions.decisionJson}->>'statusAction' = 'preserve'
|
||||
and ${workAssessments.priorDecisionId} = ${issues.lastStatusDecisionId}, false), false)`,
|
||||
or(statusChanged, evidenceChanged, and(matchesDecision, retiredReview)),
|
||||
notExists(db.select({ id: statusDecisionEffects.id }).from(statusDecisionEffects).where(and(
|
||||
eq(statusDecisionEffects.companyId, nativeRunFinalizations.companyId), eq(statusDecisionEffects.issueId, nativeRunFinalizations.issueId),
|
||||
eq(statusDecisionEffects.decisionId, nativeRunFinalizations.decisionId), inArray(statusDecisionEffects.deliveryState, ["pending", "failed"]),
|
||||
))),
|
||||
scope ? eq(nativeRunFinalizations.companyId, scope.companyId) : undefined,
|
||||
scope?.issueId ? eq(nativeRunFinalizations.issueId, scope.issueId) : undefined,
|
||||
scope?.runId ? eq(nativeRunFinalizations.runId, scope.runId) : undefined,
|
||||
scope?.agentId ? eq(heartbeatRuns.agentId, scope.agentId) : undefined,
|
||||
after ? gt(nativeRunFinalizations.runId, after) : undefined,
|
||||
runId ? eq(nativeRunFinalizations.runId, runId) : undefined,
|
||||
)).orderBy(asc(nativeRunFinalizations.runId)).limit(100);
|
||||
}
|
||||
function entry(row: Awaited<ReturnType<typeof candidates>>[number]): Entry {
|
||||
return { id: row.runId, companyId: row.companyId, issueId: row.issueId, agentId: row.agentId,
|
||||
dueAt: Math.max(row.nextAttemptAt?.getTime() ?? 0, row.leaseOwner !== null ? row.leaseExpiresAt!.getTime() : 0) };
|
||||
}
|
||||
async function load(scope?: NativeStatusReassessmentScope, signal?: AbortSignal, canRun = () => true): Promise<Entry[]> {
|
||||
const entries: Entry[] = [];
|
||||
let after: string | undefined;
|
||||
while (canRun() && !signal?.aborted) {
|
||||
const rows = await candidates(scope, after);
|
||||
entries.push(...rows.map(entry));
|
||||
if (rows.length < 100) return entries;
|
||||
after = rows[rows.length - 1].runId;
|
||||
}
|
||||
throw new Error("Native status reassessment discovery stopped");
|
||||
}
|
||||
async function check(current: Entry, signal?: AbortSignal): Promise<Entry | null> {
|
||||
if (signal?.aborted) throw new Error("Native status reassessment stopped");
|
||||
const scope = { companyId: current.companyId, issueId: current.issueId, runId: current.id };
|
||||
const [source] = await candidates(scope);
|
||||
if (!source) return null;
|
||||
const due = entry(source);
|
||||
if (due.dueAt > Date.now()) return due;
|
||||
if (signal?.aborted) throw new Error("Native status reassessment stopped");
|
||||
const outcome = await reassessNativeStatusFromSource(db, source, { onlyWhenDue: true });
|
||||
if (outcome.kind !== "deferred" || !outcome.retry) return null;
|
||||
const [remaining] = await candidates(scope);
|
||||
if (!remaining) return null;
|
||||
const retry = entry(remaining);
|
||||
return retry.dueAt > Date.now() ? retry : { ...retry, dueAt: Date.now() + 30_000 };
|
||||
}
|
||||
function register(coordinator: ReturnType<typeof createDeliveryWorkCoordinator>, canRun: () => boolean) {
|
||||
return registerScopedDeadlineWork(coordinator, {
|
||||
queue: DELIVERY_QUEUES.nativeStatusReassessment,
|
||||
decodeScope: (hint: string): NativeStatusReassessmentScope | undefined => hint === "*" ? undefined : JSON.parse(hint),
|
||||
matches: (item: Entry, scope?: NativeStatusReassessmentScope) => !scope || item.companyId === scope.companyId
|
||||
&& (!scope.issueId || item.issueId === scope.issueId) && (!scope.runId || item.id === scope.runId)
|
||||
&& (!scope.agentId || item.agentId === scope.agentId),
|
||||
load: (scope, signal) => load(scope, signal, canRun),
|
||||
check: (item, signal) => {
|
||||
if (!canRun()) throw new Error("Native status reassessment paused");
|
||||
return check(item, signal);
|
||||
},
|
||||
}, canRun);
|
||||
}
|
||||
return { load, check, register };
|
||||
}
|
||||
@@ -0,0 +1,266 @@
|
||||
import { readRequiredQualityReview } from "../required-quality-review.js";
|
||||
import { hasNativeWorkspaceFinalizationOwner } from "./native-workspace-finalization-recovery-state.js";
|
||||
import { and, desc, eq, gt, or } from "drizzle-orm";
|
||||
import { completionContracts, heartbeatRuns, issueWorkProducts, issues, nativeRunResults,
|
||||
statusDecisions, workAssessments, workspaceOperations, type Db } from "@paperclipai/db";
|
||||
import { issueService } from "../issues.js";
|
||||
import { decisionHasRetiredAutomaticReview } from "./automatic-completion-reviews.js";
|
||||
import { classifyNativeEvidence } from "./evidence-classifier.js";
|
||||
import { hasPendingNativeChildCompletion } from "./native-child-completion-delivery.js";
|
||||
import { resolveNativeReconciliationStatus, type NativeReconciliationFacts } from "./native-reconciliation-status.js";
|
||||
import { pendingNativeGovernance, resolveNativeFinalizerStatus } from "./native-run-finalizer.js";
|
||||
import { NATIVE_STATUS_ARBITER_POLICY_VERSION, type NativeAuthoritativeIssueStatus, type NativeStatusDecision } from "./status-arbiter.js";
|
||||
import { commitNativeStatusDecision, NativeStatusRaceError } from "./status-decision-committer.js";
|
||||
import { recordNativeWorkAssessment } from "./work-assessments.js";
|
||||
|
||||
export type NativeStatusReassessmentSource = {
|
||||
runId: string; companyId: string; agentId: string; issueId: string;
|
||||
issueStatus: string; issueStatusVersion: number; issueDecisionId: string | null;
|
||||
coordinatorPhase: string; assessmentId: string; decisionId: string | null;
|
||||
nativeStatusWakeIntentId: unknown;
|
||||
};
|
||||
|
||||
type Outcome = { kind: "unchanged" | "superseded" } | { kind: "deferred"; retry?: boolean } | {
|
||||
kind: "committed";
|
||||
result: { runId: string; phase: "committed"; reconciliationAction: string | null;
|
||||
decisionId: string; reconciliationDecision: NativeStatusDecision };
|
||||
};
|
||||
|
||||
function record(value: unknown): Record<string, unknown> {
|
||||
return value && typeof value === "object" && !Array.isArray(value)
|
||||
? value as Record<string, unknown>
|
||||
: {};
|
||||
}
|
||||
|
||||
function isSupersededSource(
|
||||
row: NativeStatusReassessmentSource,
|
||||
decision: Record<string, unknown>,
|
||||
priorDecisionId: string | null | undefined,
|
||||
): boolean {
|
||||
return row.coordinatorPhase === "committed"
|
||||
&& row.decisionId !== null
|
||||
&& row.issueDecisionId !== null
|
||||
&& row.issueDecisionId !== row.decisionId
|
||||
&& !(
|
||||
decision.statusAction === "preserve"
|
||||
&& priorDecisionId === row.issueDecisionId
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
|
||||
export async function nativeStatusReassessmentSourceIsSuperseded(
|
||||
db: Db,
|
||||
row: NativeStatusReassessmentSource,
|
||||
): Promise<boolean> {
|
||||
if (row.coordinatorPhase !== "committed" || row.decisionId === null
|
||||
|| row.issueDecisionId === null || row.issueDecisionId === row.decisionId) return false;
|
||||
const [decision] = await db.select({ decisionJson: statusDecisions.decisionJson })
|
||||
.from(statusDecisions).where(and(eq(statusDecisions.id, row.decisionId),
|
||||
eq(statusDecisions.companyId, row.companyId), eq(statusDecisions.issueId, row.issueId))).limit(1);
|
||||
if (!decision) throw new Error("native_committed_decision_missing");
|
||||
const [assessment] = await db.select({ priorDecisionId: workAssessments.priorDecisionId })
|
||||
.from(workAssessments).where(and(eq(workAssessments.id, row.assessmentId),
|
||||
eq(workAssessments.companyId, row.companyId), eq(workAssessments.issueId, row.issueId))).limit(1);
|
||||
return isSupersededSource(row, record(decision.decisionJson), assessment?.priorDecisionId);
|
||||
}
|
||||
|
||||
export async function reassessNativeStatusFromSource(
|
||||
db: Db,
|
||||
row: NativeStatusReassessmentSource,
|
||||
options: { onlyWhenDue?: boolean } = {},
|
||||
): Promise<Outcome> {
|
||||
const assessment = await db.select({
|
||||
id: workAssessments.id,
|
||||
contractId: workAssessments.contractId,
|
||||
resultId: workAssessments.resultId,
|
||||
policyVersion: workAssessments.policyVersion,
|
||||
priorIssueStatus: workAssessments.priorIssueStatus,
|
||||
priorStatusVersion: workAssessments.priorStatusVersion,
|
||||
priorDecisionId: workAssessments.priorDecisionId,
|
||||
assessmentJson: workAssessments.assessmentJson,
|
||||
createdAt: workAssessments.createdAt,
|
||||
}).from(workAssessments).where(and(
|
||||
eq(workAssessments.id, row.assessmentId),
|
||||
eq(workAssessments.companyId, row.companyId),
|
||||
eq(workAssessments.issueId, row.issueId),
|
||||
)).limit(1).then((entries) => entries[0] ?? null);
|
||||
const changedEvidence = assessment
|
||||
? await db.select({ id: issueWorkProducts.id }).from(issueWorkProducts).where(and(
|
||||
eq(issueWorkProducts.companyId, row.companyId),
|
||||
eq(issueWorkProducts.issueId, row.issueId),
|
||||
or(
|
||||
gt(issueWorkProducts.createdAt, assessment.createdAt),
|
||||
gt(issueWorkProducts.updatedAt, assessment.createdAt),
|
||||
),
|
||||
)).limit(1).then((entries) => entries[0] ?? null)
|
||||
: null;
|
||||
const currentDecision = row.decisionId
|
||||
? await db.select().from(statusDecisions).where(and(
|
||||
eq(statusDecisions.id, row.decisionId),
|
||||
eq(statusDecisions.companyId, row.companyId),
|
||||
eq(statusDecisions.issueId, row.issueId),
|
||||
)).limit(1).then((entries) => entries[0] ?? null)
|
||||
: null;
|
||||
const currentDecisionJson = record(currentDecision?.decisionJson);
|
||||
if (row.coordinatorPhase === "committed" && row.decisionId && !currentDecision) {
|
||||
throw new Error("native_committed_decision_missing");
|
||||
}
|
||||
if (isSupersededSource(row, currentDecisionJson, assessment?.priorDecisionId)) return { kind: "superseded" };
|
||||
const persistedProjectedVersion = Number(currentDecisionJson.projectedStatusVersion);
|
||||
const expectedProjectedVersion = currentDecision
|
||||
? Number.isFinite(persistedProjectedVersion)
|
||||
? persistedProjectedVersion
|
||||
: Number(currentDecision.decisionVersion) - (currentDecisionJson.statusAction === "preserve" ? 1 : 0)
|
||||
: null;
|
||||
const issueMatchesCurrentDecision = !!assessment
|
||||
&& !!currentDecision
|
||||
&& row.issueDecisionId === row.decisionId
|
||||
&& currentDecision.assessmentId === assessment.id
|
||||
&& currentDecision.toStatus === row.issueStatus
|
||||
&& expectedProjectedVersion === Number(row.issueStatusVersion);
|
||||
const authoritativeStatusChanged = !!assessment && !issueMatchesCurrentDecision && (
|
||||
assessment.priorIssueStatus !== row.issueStatus
|
||||
|| Number(assessment.priorStatusVersion) !== Number(row.issueStatusVersion)
|
||||
);
|
||||
const retiredAutomaticReview = issueMatchesCurrentDecision && currentDecision
|
||||
? await decisionHasRetiredAutomaticReview(db, currentDecision) : false;
|
||||
let reassessment = null;
|
||||
let resultRow = null;
|
||||
let contractRow = null;
|
||||
if (assessment && (authoritativeStatusChanged || changedEvidence || retiredAutomaticReview)) {
|
||||
[resultRow, contractRow] = await Promise.all([
|
||||
db.select().from(nativeRunResults).where(and(
|
||||
eq(nativeRunResults.id, assessment.resultId),
|
||||
eq(nativeRunResults.companyId, row.companyId),
|
||||
eq(nativeRunResults.issueId, row.issueId),
|
||||
eq(nativeRunResults.runId, row.runId),
|
||||
)).limit(1).then((entries) => entries[0] ?? null),
|
||||
db.select().from(completionContracts).where(and(
|
||||
eq(completionContracts.id, assessment.contractId),
|
||||
eq(completionContracts.companyId, row.companyId),
|
||||
eq(completionContracts.issueId, row.issueId),
|
||||
)).limit(1).then((entries) => entries[0] ?? null),
|
||||
]);
|
||||
if (!resultRow || !contractRow) throw new Error("native_reconciliation_evidence_binding_missing");
|
||||
reassessment = await classifyNativeEvidence({
|
||||
db,
|
||||
companyId: row.companyId,
|
||||
issueId: row.issueId,
|
||||
runId: row.runId,
|
||||
contract: record(contractRow.contractJson),
|
||||
result: record(record(resultRow.resultJson).result),
|
||||
});
|
||||
}
|
||||
const previousAssessment = record(assessment?.assessmentJson);
|
||||
const newEvidenceSatisfiesContract = !!changedEvidence
|
||||
&& previousAssessment.allCriteriaSatisfied !== true
|
||||
&& reassessment?.allCriteriaSatisfied === true
|
||||
&& reassessment.verificationPassed === true;
|
||||
const facts: NativeReconciliationFacts = authoritativeStatusChanged
|
||||
? { authoritativeStatusChanged: true }
|
||||
: newEvidenceSatisfiesContract
|
||||
? { newEvidenceSatisfiesContract: true }
|
||||
: {};
|
||||
if (Object.keys(facts).length > 0 || retiredAutomaticReview) {
|
||||
if (!assessment || !reassessment || !resultRow || !contractRow) {
|
||||
throw new Error("native_reconciliation_reassessment_missing");
|
||||
}
|
||||
const currentIssue = await db.select().from(issues)
|
||||
.where(and(eq(issues.id, row.issueId), eq(issues.companyId, row.companyId))).then((entries) => entries[0]);
|
||||
const readiness = retiredAutomaticReview ? await issueService(db).getDependencyReadiness(row.issueId, db) : null;
|
||||
const currentRun = retiredAutomaticReview ? await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, row.runId)).then((entries) => entries[0]) : null;
|
||||
const latestContract = retiredAutomaticReview ? await db.select({ id: completionContracts.id }).from(completionContracts)
|
||||
.where(and(eq(completionContracts.companyId, row.companyId), eq(completionContracts.issueId, row.issueId)))
|
||||
.orderBy(desc(completionContracts.revision)).limit(1).then((entries) => entries[0]) : null;
|
||||
if (retiredAutomaticReview && (!currentIssue || currentRun?.status !== "succeeded"
|
||||
|| (options.onlyWhenDue && hasNativeWorkspaceFinalizationOwner(currentRun.runnerProfileJson))
|
||||
|| latestContract?.id !== contractRow.id
|
||||
|| (currentIssue.executionRunId && currentIssue.executionRunId !== row.runId))) return { kind: "deferred" };
|
||||
const reviewBarrier = retiredAutomaticReview ? await db.select({ status: workspaceOperations.status })
|
||||
.from(workspaceOperations).where(and(eq(workspaceOperations.companyId, row.companyId),
|
||||
eq(workspaceOperations.heartbeatRunId, row.runId), eq(workspaceOperations.phase, "workspace_finalize")))
|
||||
.orderBy(desc(workspaceOperations.createdAt)).limit(1).then((entries) => entries[0]) : null;
|
||||
if (reviewBarrier && reviewBarrier.status !== "succeeded") return { kind: "deferred" };
|
||||
const childCompletionRecipient = {
|
||||
companyId: row.companyId, issueId: row.issueId, agentId: row.agentId, runId: row.runId,
|
||||
sourceIntentId: typeof row.nativeStatusWakeIntentId === "string" ? row.nativeStatusWakeIntentId : null,
|
||||
};
|
||||
const hasPendingChildCompletion = await hasPendingNativeChildCompletion(db, childCompletionRecipient);
|
||||
const reassessmentRow = await recordNativeWorkAssessment({
|
||||
db,
|
||||
companyId: row.companyId,
|
||||
issueId: row.issueId,
|
||||
runId: row.runId,
|
||||
turnId: resultRow.turnId,
|
||||
contractId: contractRow.id,
|
||||
contractCanonicalSha256: contractRow.canonicalSha256,
|
||||
resultId: resultRow.id,
|
||||
resultCanonicalSha256: resultRow.canonicalSha256,
|
||||
priorIssueStatus: row.issueStatus,
|
||||
priorStatusVersion: Number(row.issueStatusVersion),
|
||||
priorDecisionId: row.issueDecisionId,
|
||||
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
|
||||
assessment: reassessment,
|
||||
supersedesAssessmentId: assessment.id,
|
||||
});
|
||||
const hasUnresolvedIssueBlockers = (readiness?.unresolvedBlockerCount ?? 0) > 0;
|
||||
const governanceGate = retiredAutomaticReview && currentIssue
|
||||
? await pendingNativeGovernance({ db, companyId: row.companyId, issueId: row.issueId,
|
||||
runId: row.runId, executionState: record(currentIssue.executionState) }) : null;
|
||||
const qualityReview = currentIssue ? await readRequiredQualityReview(db, currentIssue) : null;
|
||||
let decision = retiredAutomaticReview && currentIssue
|
||||
? resolveNativeFinalizerStatus({
|
||||
hasPendingChildCompletion,
|
||||
requiredQualityReviewSatisfied: qualityReview?.satisfied,
|
||||
assessment: reassessment, terminalState: "succeeded", workspaceFinalizeStatus: "succeeded",
|
||||
governanceGate,
|
||||
completionClaimPolicyAccepted: contractRow.risk === "low" && contractRow.completionAuthority === "agent_claim_policy",
|
||||
hasUnresolvedIssueBlockers,
|
||||
reviewOwnerUserId: currentIssue.responsibleUserId ?? currentIssue.createdByUserId,
|
||||
priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus, agentId: row.agentId,
|
||||
})
|
||||
: resolveNativeReconciliationStatus({
|
||||
facts: { ...facts, hasPendingChildCompletion },
|
||||
priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus, agentId: row.agentId,
|
||||
});
|
||||
if (decision.statusAction === "done" && qualityReview?.satisfied === false) {
|
||||
decision = resolveNativeFinalizerStatus({ assessment: reassessment, terminalState: "succeeded", workspaceFinalizeStatus: "succeeded",
|
||||
requiredQualityReviewSatisfied: false, priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus, agentId: row.agentId });
|
||||
}
|
||||
let committed: Awaited<ReturnType<typeof commitNativeStatusDecision>>;
|
||||
try {
|
||||
committed = await commitNativeStatusDecision({
|
||||
db,
|
||||
companyId: row.companyId,
|
||||
issueId: row.issueId,
|
||||
runId: row.runId,
|
||||
assessmentId: reassessmentRow.id,
|
||||
priorStatus: row.issueStatus,
|
||||
priorStatusVersion: Number(row.issueStatusVersion),
|
||||
priorDecisionId: row.issueDecisionId,
|
||||
decision,
|
||||
requireReassessmentSource: options.onlyWhenDue ? {
|
||||
phase: row.coordinatorPhase, resultId: resultRow.id, assessmentId: assessment.id,
|
||||
decisionId: row.decisionId, agentId: row.agentId, retiredAutomaticReview: retiredAutomaticReview ? { contractId: contractRow.id, governanceGate, hasUnresolvedIssueBlockers } : null,
|
||||
} : undefined,
|
||||
requireNoPendingChildCompletion: decision.statusAction === "done" ? childCompletionRecipient : undefined,
|
||||
supersedesCommittedDecisionId: row.coordinatorPhase === "committed"
|
||||
? row.decisionId ?? undefined
|
||||
: undefined,
|
||||
});
|
||||
} catch (error) {
|
||||
if (error instanceof NativeStatusRaceError) return options.onlyWhenDue
|
||||
? { kind: "deferred", retry: true } : { kind: "deferred" };
|
||||
throw error;
|
||||
}
|
||||
return { kind: "committed", result: {
|
||||
runId: row.runId,
|
||||
phase: "committed",
|
||||
reconciliationAction: decision.reasonCode,
|
||||
decisionId: committed.decision.id,
|
||||
reconciliationDecision: decision,
|
||||
} };
|
||||
}
|
||||
return { kind: "unchanged" };
|
||||
}
|
||||
@@ -95,13 +95,11 @@ async function recoverCandidate(db: Db, candidate: { runId: string; companyId: s
|
||||
let updatedRun: typeof heartbeatRuns.$inferSelect | undefined;
|
||||
if (eligible) {
|
||||
await ownership.assertHeld();
|
||||
|
||||
|
||||
await notifyNativeFinalizationWork(tx, { companyId: run.companyId, runId: run.id });
|
||||
await tx.insert(workspaceOperations).values({ companyId: run.companyId, heartbeatRunId: run.id, issueId: issue.id,
|
||||
phase: "workspace_finalize", status: "succeeded", exitCode: 0, finishedAt: now,
|
||||
metadata: { owningService: "native_workspace_finalizer", workspaceSync: { omitted: true, reason: "restore_unsafe_archive", legacy: true } },
|
||||
});
|
||||
await notifyNativeFinalizationWork(tx, { companyId: run.companyId, runId: run.id });
|
||||
await tx.update(nativeRunFinalizations).set({ phase: "result_accepted", failureCode: null, nextAttemptAt: null,
|
||||
failureDetail: { workspaceFinalizeAttempt: 0, legacyUnsafeExportOmitted: { resultId: result.id, recoveredAt: now.toISOString() } }, updatedAt: now,
|
||||
}).where(eq(nativeRunFinalizations.runId, run.id));
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js";
|
||||
import { beginRunPhase } from "../execution-preparation-timing.js";
|
||||
import { logActivity } from "../activity-log.js";
|
||||
import { issueRecoveryActionService } from "../issue-recovery-actions.js";
|
||||
@@ -21,6 +22,12 @@ const OWNER_KEY = "nativeWorkspaceFinalizationOwner";
|
||||
export function nativeWorkspaceFinalizationLockKey(companyId: string, runId: string): string {
|
||||
return `native-workspace-finalization:${companyId}:${runId}`;
|
||||
}
|
||||
export async function tryLockNativeWorkspaceFinalization(tx: Db, input: { companyId: string; runId: string }): Promise<boolean> {
|
||||
const [lock] = await tx.execute(sql`select pg_try_advisory_xact_lock(hashtextextended(
|
||||
${nativeWorkspaceFinalizationLockKey(input.companyId, input.runId)}, 0)) as acquired`) as unknown as Array<{ acquired: boolean }>;
|
||||
return lock.acquired;
|
||||
}
|
||||
|
||||
const activeOwnerTokens = new Set<string>();
|
||||
// Positive same-boot evidence, retained only when durable cleanup could not be
|
||||
// confirmed. Absence from the active set alone never proves another token joined.
|
||||
@@ -258,9 +265,12 @@ export async function withNativeWorkspaceFinalizationOwnership<T>(
|
||||
activeOwnerTokens.delete(owner.token);
|
||||
joinedOwnerTokens.add(owner.token);
|
||||
try {
|
||||
await input.db.update(heartbeatRuns).set({
|
||||
runnerProfileJson: sql`coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb) - ${OWNER_KEY}`,
|
||||
}).where(owned);
|
||||
await input.db.transaction(async tx => {
|
||||
await notifyNativeStatusReassessment(tx, { companyId: input.companyId, runId: input.runId });
|
||||
await tx.update(heartbeatRuns).set({
|
||||
runnerProfileJson: sql`coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb) - ${OWNER_KEY}`,
|
||||
}).where(owned);
|
||||
});
|
||||
joinedOwnerTokens.delete(owner.token);
|
||||
} catch {
|
||||
// A reconnect in this exact controller boot can use the positive join
|
||||
|
||||
@@ -5,6 +5,10 @@ import { nativeSha256 } from "./canonical.js";
|
||||
|
||||
export const NATIVE_WORKSPACE_OWNER_UNVERIFIED = "native_workspace_finalization_owner_unverified";
|
||||
|
||||
export function hasNativeWorkspaceFinalizationOwner(profile: Record<string, unknown> | null | undefined): boolean {
|
||||
return profile?.nativeWorkspaceFinalizationOwner != null;
|
||||
}
|
||||
|
||||
/** A terminal task does not prove that an older controller's copyback stopped. */
|
||||
export async function hasUnverifiedNativeWorkspaceFinalizationOwner(
|
||||
db: Db,
|
||||
@@ -20,5 +24,5 @@ export async function hasUnverifiedNativeWorkspaceFinalizationOwner(
|
||||
eq(heartbeatRuns.runtimeMode, "native"),
|
||||
)).limit(1);
|
||||
const owner = run?.profile?.nativeWorkspaceFinalizationOwner;
|
||||
return owner != null && action.fingerprint === nativeSha256({ runId, owner });
|
||||
return hasNativeWorkspaceFinalizationOwner(run?.profile) && action.fingerprint === nativeSha256({ runId, owner });
|
||||
}
|
||||
@@ -1,3 +1,5 @@
|
||||
import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../delivery-work-notifications.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
|
||||
import * as pluginWorkspaces from "../plugin-workspaces.js";
|
||||
import * as platformProof from "./native-workspace-platform-proof.js";
|
||||
import { currentNativeControllerIdentity } from "./native-restart-recovery.js";
|
||||
@@ -38,7 +40,8 @@ import { resumeNativeWorkspaceFinalization } from "./native-workspace-finalizer.
|
||||
import { reconcileNativeFinalizations } from "./native-finalization-reconciler.js";
|
||||
|
||||
describe("native workspace finalization recovery", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let connectionString: string;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let workspaceRoot: string;
|
||||
let priorLogRoot: string | undefined;
|
||||
@@ -125,10 +128,10 @@ describe("native workspace finalization recovery", () => {
|
||||
workspaceRoot,
|
||||
"operation-logs",
|
||||
);
|
||||
temporary = await startEmbeddedPostgresTestDatabase(
|
||||
"paperclip-native-workspace-finalizer-",
|
||||
);
|
||||
db = createDb(temporary.connectionString);
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_NATIVE_WORKSPACE_FINALIZER_TEST_DATABASE_URL");
|
||||
if (!external) temporary = await startEmbeddedPostgresTestDatabase("paperclip-native-workspace-finalizer-");
|
||||
connectionString = external ?? temporary!.connectionString;
|
||||
db = createDb(connectionString);
|
||||
|
||||
await db.insert(companies).values([
|
||||
{ id: companyId, name: "Native workspace owner", issuePrefix: "NWO" },
|
||||
@@ -162,7 +165,8 @@ describe("native workspace finalization recovery", () => {
|
||||
}, 30_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await temporary.cleanup();
|
||||
await db?.$client.end({ timeout: 1 });
|
||||
await temporary?.cleanup();
|
||||
await fs.rm(workspaceRoot, { recursive: true, force: true });
|
||||
if (priorLogRoot === undefined) {
|
||||
delete process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
|
||||
@@ -210,7 +214,17 @@ describe("native workspace finalization recovery", () => {
|
||||
const seed = await seedAbandonedCopyback();
|
||||
const [before] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId));
|
||||
expect(await withNativeWorkspaceFinalizationOwnership(seed, async () => "unexpected")).toEqual({ acquired: false });
|
||||
const receipt = await resumeNativeWorkspaceAfterOwnerStop(seed);
|
||||
const observed: Array<Promise<unknown>> = [];
|
||||
const stop = subscribeDeliveryWork(db, DELIVERY_QUEUES.nativeStatusReassessment, () => {
|
||||
observed.push(db.select({ profile: heartbeatRuns.runnerProfileJson }).from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, seed.runId)).then(rows => rows[0].profile));
|
||||
});
|
||||
let receipt: Awaited<ReturnType<typeof resumeNativeWorkspaceAfterOwnerStop>>;
|
||||
try {
|
||||
receipt = await resumeNativeWorkspaceAfterOwnerStop(seed);
|
||||
await vi.waitFor(() => expect(observed).toHaveLength(1));
|
||||
expect(await Promise.all(observed)).toEqual([{ preserved: "workspace descriptor" }]);
|
||||
} finally { stop(); }
|
||||
expect(receipt).toEqual({ runId: seed.runId, resultId: before.resultId, status: "queued" });
|
||||
const [after] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId));
|
||||
expect(after).toEqual(before);
|
||||
@@ -425,7 +439,7 @@ describe("native workspace finalization recovery", () => {
|
||||
|
||||
it("keeps ordinary progress writable even when the application pool has one connection", async () => {
|
||||
const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Small application pool" });
|
||||
const smallPool = createDb(temporary.connectionString, { maxConnections: 1 });
|
||||
const smallPool = createDb(connectionString, { maxConnections: 1 });
|
||||
const result = await withNativeWorkspaceFinalizationOwnership({ db: smallPool, companyId, runId: seeded.runId }, async (ownership) => {
|
||||
await ownership.assertHeld();
|
||||
await smallPool.update(heartbeatRuns).set({ updatedAt: new Date() }).where(eq(heartbeatRuns.id, seeded.runId));
|
||||
@@ -438,7 +452,7 @@ describe("native workspace finalization recovery", () => {
|
||||
const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Live copyback" });
|
||||
const live = await withNativeWorkspaceFinalizationOwnership({ db, companyId, runId: seeded.runId }, async (ownership) => {
|
||||
await ownership.assertHeld();
|
||||
expect(await resumeNativeWorkspaceFinalization({ db: createDb(temporary.connectionString), runId: seeded.runId })).toBeNull();
|
||||
expect(await resumeNativeWorkspaceFinalization({ db: createDb(connectionString), runId: seeded.runId })).toBeNull();
|
||||
expect(await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, seeded.runId))).toEqual([]);
|
||||
return "copied";
|
||||
});
|
||||
@@ -462,7 +476,7 @@ describe("native workspace finalization recovery", () => {
|
||||
try {
|
||||
await entered;
|
||||
await db.execute(sql`select pg_terminate_backend(pid) from pg_locks where locktype = 'advisory' and database = (select oid from pg_database where datname = current_database()) and pid <> pg_backend_pid()`);
|
||||
const second = await withNativeWorkspaceFinalizationOwnership({ db: createDb(temporary.connectionString), companyId, runId: seeded.runId }, async () => "must not run");
|
||||
const second = await withNativeWorkspaceFinalizationOwnership({ db: createDb(connectionString), companyId, runId: seeded.runId }, async () => "must not run");
|
||||
expect(second).toEqual({ acquired: false });
|
||||
const [held] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId));
|
||||
expect(held.runnerProfileJson?.nativeWorkspaceFinalizationOwner).toBeTruthy();
|
||||
@@ -475,7 +489,7 @@ describe("native workspace finalization recovery", () => {
|
||||
|
||||
it("recovers its own joined copyback receipt after the application pool disconnects during cleanup", async () => {
|
||||
const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Reconnect joined copyback" });
|
||||
const interruptedDb = createDb(temporary.connectionString);
|
||||
const interruptedDb = createDb(connectionString);
|
||||
let joined = false;
|
||||
await expect(withNativeWorkspaceFinalizationOwnership({ db: interruptedDb, companyId, runId: seeded.runId }, async () => {
|
||||
await fs.writeFile(path.join(workspaceRoot, "joined-copyback.txt"), "durable work");
|
||||
@@ -493,7 +507,7 @@ describe("native workspace finalization recovery", () => {
|
||||
.toEqual({ acquired: false });
|
||||
}
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: beforeReconnect.runnerProfileJson }).where(eq(heartbeatRuns.id, seeded.runId));
|
||||
expect(await withNativeWorkspaceFinalizationOwnership({ db: createDb(temporary.connectionString), companyId, runId: seeded.runId }, async () => "resumed"))
|
||||
expect(await withNativeWorkspaceFinalizationOwnership({ db: createDb(connectionString), companyId, runId: seeded.runId }, async () => "resumed"))
|
||||
.toEqual({ acquired: true, value: "resumed" });
|
||||
});
|
||||
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import { tryLockNativeWorkspaceFinalization } from "./native-workspace-finalization-ownership.js";
|
||||
import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js";
|
||||
import os from "node:os";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, type Db } from "@paperclipai/db";
|
||||
@@ -24,8 +26,7 @@ export async function resumeNativeWorkspaceAfterOwnerStop(input: {
|
||||
const admitted = await input.db.transaction(async tx => {
|
||||
// Share the physical copyback lock. Do not wait behind an active exporter
|
||||
// and then apply a confirmation that described an earlier owner.
|
||||
const locks = await tx.execute(sql`select pg_try_advisory_xact_lock(hashtextextended(${`native-workspace-finalization:${input.companyId}:${input.runId}`}, 0)) as acquired`);
|
||||
if (locks[0]?.acquired !== true) throw conflict("Workspace copyback is still active. Wait for it to stop before confirming recovery.");
|
||||
if (!await tryLockNativeWorkspaceFinalization(tx as unknown as Db, input)) throw conflict("Workspace copyback is still active. Wait for it to stop before confirming recovery.");
|
||||
// Match the status arbiter's coordinator -> issue -> run lock order.
|
||||
const [coordinator] = await tx.select().from(nativeRunFinalizations).where(and(eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId), eq(nativeRunFinalizations.runId, input.runId))).for("update");
|
||||
const [issue] = await tx.select().from(issues).where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("update");
|
||||
@@ -65,6 +66,7 @@ export async function resumeNativeWorkspaceAfterOwnerStop(input: {
|
||||
const now = new Date();
|
||||
const proof = { owner: prior, resultId: result.id, actorId: input.actorId,
|
||||
stopEvidence: confirmation.stopEvidence, confirmedAt: now.toISOString() };
|
||||
await notifyNativeStatusReassessment(tx, { companyId: input.companyId, issueId: input.issueId, runId: input.runId });
|
||||
await tx.update(heartbeatRuns).set({
|
||||
runnerProfileJson: sql`${heartbeatRuns.runnerProfileJson} - ${OWNER_KEY}`, updatedAt: now,
|
||||
}).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, run.id),
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { Readable } from "node:stream";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
@@ -11,10 +12,14 @@ import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.
|
||||
|
||||
const sha = (body: Buffer) => createHash("sha256").update(body).digest("hex");
|
||||
describe("Dot assigned-task attachment reading", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
beforeAll(async () => { temporary = await startEmbeddedPostgresTestDatabase("dot-task-files-"); db = createDb(temporary.connectionString); });
|
||||
afterAll(async () => { await temporary?.cleanup(); });
|
||||
beforeAll(async () => {
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_RUNNER_ATTACHMENTS_TEST_DATABASE_URL");
|
||||
if (!external) temporary = await startEmbeddedPostgresTestDatabase("dot-task-files-");
|
||||
db = createDb(external ?? temporary!.connectionString);
|
||||
});
|
||||
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
|
||||
afterEach(() => { vi.unstubAllEnvs(); vi.unstubAllGlobals(); });
|
||||
async function fixture(body = Buffer.from("only in the file 🌍"), enabled = true) {
|
||||
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID();
|
||||
|
||||
@@ -1,3 +1,8 @@
|
||||
import { lockCurrentNativeWorkAssessmentEvidence } from "./work-assessments.js";
|
||||
import { hasNativeWorkspaceFinalizationOwner } from "./native-workspace-finalization-recovery-state.js";
|
||||
import { tryLockNativeWorkspaceFinalization } from "./native-workspace-finalization-ownership.js";
|
||||
import { lockCurrentNativeCompletionContract } from "./completion-contracts.js";
|
||||
import { isNativeRunnerOwnershipHeld } from "./native-runner-ownership.js";
|
||||
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js";
|
||||
import { notifyAutomaticCompletionReviews } from "../automatic-completion-review-notifications.js";
|
||||
@@ -8,7 +13,7 @@ import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js"
|
||||
import { scheduledIssueMonitorWait } from "../issue-monitors.js";
|
||||
import { onboardingFirstTaskCompletionAllowed } from "../onboarding-first-task-admission.js";
|
||||
import { hasPublishedGitHubReview, readInheritedGitHubReviewWait } from "../chat-channels/github/review-wait.js";
|
||||
import { pendingNativeGovernance } from "./native-governance.js";
|
||||
import { lockNativeGovernanceSources, pendingNativeGovernance } from "./native-governance.js";
|
||||
import { findSupersededExternalReviewRequests, supersedeExternalReviewRequestsInTransaction } from "./superseded-external-review-requests.js";
|
||||
import { normalizePrpResultSignals, type PrpStructuredRunResult } from "../../vendor/paperclip-runner/index.js";
|
||||
import { readNativeMonitorDelivery, readNativePreservedMonitorWait, type NativeMonitorDelivery, type NativePreservedMonitorWait } from "./native-monitor-delivery.js";
|
||||
@@ -38,6 +43,7 @@ import {
|
||||
} from "@paperclipai/db";
|
||||
import {
|
||||
NATIVE_STATUS_ARBITER_POLICY_VERSION,
|
||||
type NativeGovernanceGate,
|
||||
type NativeStatusDecision,
|
||||
type NativeStatusEffect,
|
||||
} from "./status-arbiter.js";
|
||||
@@ -1603,6 +1609,10 @@ export async function commitNativeStatusDecision(input: {
|
||||
requireMonitorDelivery?: NativeMonitorDelivery;
|
||||
requirePreservedMonitorWait?: NativePreservedMonitorWait;
|
||||
requireNoPendingChildCompletion?: NativeChildCompletionRecipient;
|
||||
requireReassessmentSource?: {
|
||||
phase: string; resultId: string; assessmentId: string; decisionId: string | null; agentId: string;
|
||||
retiredAutomaticReview: { contractId: string; governanceGate: NativeGovernanceGate | null; hasUnresolvedIssueBlockers: boolean } | null;
|
||||
};
|
||||
requireModelRejectionOwner?: { agentId: string; reviewContext: NativeReviewAssignmentContext | null };
|
||||
requireProviderFailureOwner?: { agentId: string; reviewContext: NativeReviewAssignmentContext | null };
|
||||
requireBoardResponseWaitSource?: NativeBoardResponseWaitSource;
|
||||
@@ -1626,6 +1636,11 @@ export async function commitNativeStatusDecision(input: {
|
||||
const publications: ActivityPublication[] = [];
|
||||
const terminalRunsToEmit: (typeof heartbeatRuns.$inferSelect)[] = [];
|
||||
const committed = await input.db.transaction(async (tx) => {
|
||||
const retiredReview = input.requireReassessmentSource?.retiredAutomaticReview;
|
||||
if (retiredReview && !await lockCurrentNativeCompletionContract(tx as unknown as Db, {
|
||||
companyId: input.companyId, issueId: input.issueId, contractId: retiredReview.contractId,
|
||||
})) throw new NativeStatusRaceError();
|
||||
if (retiredReview && !await tryLockNativeWorkspaceFinalization(tx as unknown as Db, input)) throw new NativeStatusRaceError();
|
||||
const coordinator = await tx
|
||||
.select()
|
||||
.from(nativeRunFinalizations)
|
||||
@@ -1654,6 +1669,67 @@ export async function commitNativeStatusDecision(input: {
|
||||
.limit(1)
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (!issue) throw new NativeStatusRaceError();
|
||||
if (input.requireReassessmentSource) {
|
||||
const expected = input.requireReassessmentSource;
|
||||
const now = Date.now();
|
||||
if (coordinator.phase === "terminal_failure" || coordinator.phase !== expected.phase ||
|
||||
coordinator.resultId !== expected.resultId || coordinator.assessmentId !== expected.assessmentId ||
|
||||
coordinator.decisionId !== expected.decisionId ||
|
||||
(coordinator.nextAttemptAt !== null && coordinator.nextAttemptAt.getTime() > now) ||
|
||||
(coordinator.leaseOwner !== null &&
|
||||
(coordinator.leaseExpiresAt === null || coordinator.leaseExpiresAt.getTime() > now))) {
|
||||
throw new NativeStatusRaceError();
|
||||
}
|
||||
const run = await tx.select().from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId),
|
||||
eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.agentId, expected.agentId),
|
||||
)).for("update").limit(1).then(rows => rows[0]);
|
||||
if (!run || run.runtimeMode !== "native" || isNativeRunnerOwnershipHeld(run)) throw new NativeStatusRaceError();
|
||||
if (expected.retiredAutomaticReview && (run.status !== "succeeded" ||
|
||||
hasNativeWorkspaceFinalizationOwner(run.runnerProfileJson) ||
|
||||
(issue.executionRunId !== null && issue.executionRunId !== input.runId))) {
|
||||
throw new NativeStatusRaceError();
|
||||
}
|
||||
if (expected.retiredAutomaticReview) {
|
||||
try {
|
||||
const [barrier] = await tx.select({ status: workspaceOperations.status }).from(workspaceOperations)
|
||||
.where(and(eq(workspaceOperations.companyId, input.companyId),
|
||||
eq(workspaceOperations.heartbeatRunId, input.runId), eq(workspaceOperations.phase, "workspace_finalize")))
|
||||
.orderBy(desc(workspaceOperations.createdAt)).limit(1).for("share", { noWait: true });
|
||||
if (barrier && barrier.status !== "succeeded") throw new NativeStatusRaceError();
|
||||
const executionState = record(issue.executionState);
|
||||
if (executionState.status !== "pending") await lockNativeGovernanceSources({ db: tx as unknown as Db,
|
||||
companyId: input.companyId, issueId: input.issueId });
|
||||
const gate = await pendingNativeGovernance({ db: tx as unknown as Db, companyId: input.companyId,
|
||||
issueId: input.issueId, runId: input.runId, executionState });
|
||||
const expectedGate = expected.retiredAutomaticReview.governanceGate;
|
||||
if (gate?.kind !== expectedGate?.kind || gate?.id !== expectedGate?.id) throw new NativeStatusRaceError();
|
||||
const readiness = await issueService(tx as unknown as Db).getDependencyReadiness(input.issueId, tx, { lockSources: true });
|
||||
if ((readiness.unresolvedBlockerCount > 0) !== expected.retiredAutomaticReview.hasUnresolvedIssueBlockers) {
|
||||
throw new NativeStatusRaceError();
|
||||
}
|
||||
} catch (error) {
|
||||
if (record(error).code === "55P03" || record(record(error).cause).code === "55P03") throw new NativeStatusRaceError();
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
const pendingEffect = expected.decisionId ? await tx.select({ id: statusDecisionEffects.id })
|
||||
.from(statusDecisionEffects).where(and(
|
||||
eq(statusDecisionEffects.companyId, input.companyId), eq(statusDecisionEffects.issueId, input.issueId),
|
||||
eq(statusDecisionEffects.decisionId, expected.decisionId),
|
||||
inArray(statusDecisionEffects.deliveryState, ["pending", "failed"]),
|
||||
)).limit(1).then(rows => rows[0]) : null;
|
||||
if (pendingEffect) throw new NativeStatusRaceError();
|
||||
try {
|
||||
if (!await lockCurrentNativeWorkAssessmentEvidence(tx as unknown as Db, {
|
||||
companyId: input.companyId, issueId: input.issueId, runId: input.runId,
|
||||
resultId: expected.resultId, assessmentId: input.assessmentId,
|
||||
})) throw new NativeStatusRaceError();
|
||||
} catch (error) {
|
||||
if (record(error).code === "55P03" || record(record(error).cause).code === "55P03") throw new NativeStatusRaceError();
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
// A completed model turn cannot close a persistent conversation. Preserve
|
||||
// the task here; the response finalizer records its durable waiting state.
|
||||
if (issue.conversationAgentId && input.decision.statusAction === "done") {
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { workAssessments } from "@paperclipai/db";
|
||||
import type { NativeEvidenceAssessment } from "./evidence-classifier.js";
|
||||
import { completionContracts, nativeRunResults, workAssessments } from "@paperclipai/db";
|
||||
import { classifyNativeEvidence, type NativeEvidenceAssessment } from "./evidence-classifier.js";
|
||||
import { nativeSha256 } from "./canonical.js";
|
||||
import type { NativeMonitorDelivery, NativePreservedMonitorWait } from "./native-monitor-delivery.js";
|
||||
|
||||
@@ -144,3 +144,33 @@ export async function recordNativeAttentionAssessment(input: {
|
||||
if (!row) throw new Error("native_attention_assessment_not_persisted");
|
||||
return row;
|
||||
}
|
||||
|
||||
|
||||
|
||||
export async function lockCurrentNativeWorkAssessmentEvidence(db: Db, input: {
|
||||
companyId: string;
|
||||
issueId: string;
|
||||
runId: string;
|
||||
resultId: string;
|
||||
assessmentId: string;
|
||||
}): Promise<boolean> {
|
||||
const [row] = await db.select({
|
||||
assessment: workAssessments.assessmentJson,
|
||||
contract: completionContracts.contractJson,
|
||||
envelope: nativeRunResults.resultJson,
|
||||
}).from(workAssessments)
|
||||
.innerJoin(completionContracts, and(eq(completionContracts.id, workAssessments.contractId),
|
||||
eq(completionContracts.companyId, input.companyId), eq(completionContracts.issueId, input.issueId)))
|
||||
.innerJoin(nativeRunResults, and(eq(nativeRunResults.id, workAssessments.resultId),
|
||||
eq(nativeRunResults.companyId, input.companyId), eq(nativeRunResults.issueId, input.issueId),
|
||||
eq(nativeRunResults.runId, input.runId)))
|
||||
.where(and(eq(workAssessments.id, input.assessmentId), eq(workAssessments.companyId, input.companyId),
|
||||
eq(workAssessments.issueId, input.issueId), eq(workAssessments.runId, input.runId),
|
||||
eq(workAssessments.resultId, input.resultId)))
|
||||
.limit(1).for("share", { noWait: true });
|
||||
if (!row) return false;
|
||||
const current = await classifyNativeEvidence({ db, companyId: input.companyId,
|
||||
issueId: input.issueId, runId: input.runId, contract: row.contract,
|
||||
result: row.envelope.result as Record<string, unknown>, lockSources: true });
|
||||
return nativeSha256(current) === nativeSha256(row.assessment);
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
|
||||
export type NativeStatusReassessmentScope = { companyId: string; issueId?: string; runId?: string; agentId?: string };
|
||||
|
||||
export function notifyNativeStatusReassessment(tx: object, scope: NativeStatusReassessmentScope) {
|
||||
return notifyDeliveryWork(tx, DELIVERY_QUEUES.nativeStatusReassessment,
|
||||
JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId, runId: scope.runId, agentId: scope.agentId }));
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js";
|
||||
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
|
||||
import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
|
||||
@@ -6453,6 +6454,7 @@ export function recoveryService(
|
||||
|
||||
const updated = await db.transaction(async tx => {
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
return tx
|
||||
.update(issues)
|
||||
.set({
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { notifyNativeReviewResponse } from "./native-review-response-notifications.js";
|
||||
import { notifyNativeReviewState } from "./native-review-state-notifications.js";
|
||||
import { notifyChatConfirmationResolution } from "./chat-confirmation-notifications.js";
|
||||
import { enqueueTerminalIssueInteractionChatPublications } from "./chat-interaction-publications.js";
|
||||
import { connectionManagementService } from "./connection-management.js";
|
||||
@@ -156,7 +156,7 @@ export function satisfiedConnectionIntentService(db: Db) {
|
||||
}
|
||||
const connection = await findSatisfiedToolConnection(txDb, task, intent);
|
||||
if (!connection) return false;
|
||||
await notifyNativeReviewResponse(tx, candidate);
|
||||
await notifyNativeReviewState(tx, candidate);
|
||||
const now = new Date();
|
||||
// Retire the obsolete request without inventing a human approval.
|
||||
await notifyChatConfirmationResolution(tx, intent);
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeReviewState } from "../../native-review-state-notifications.js";
|
||||
import { vercelConnectHttpError, vercelCredentialFor, vercelConnectResourcesFor } from "../vercel-credentials.js";
|
||||
import { credentialScope, managedConnectorProfile } from "./client-settings.js";
|
||||
import { createOAuthClientRegistration } from "./client-registration.js";
|
||||
@@ -563,65 +564,69 @@ export function createOAuthAuthorizationStart(db: Db, dependencies: Dependencies
|
||||
href: authorizationUrl.toString(),
|
||||
},
|
||||
};
|
||||
const [existingInteraction] = await db
|
||||
.select()
|
||||
.from(issueThreadInteractions)
|
||||
.where(
|
||||
and(
|
||||
eq(issueThreadInteractions.companyId, companyId),
|
||||
eq(issueThreadInteractions.issueId, input.issueId),
|
||||
eq(issueThreadInteractions.idempotencyKey, idempotencyKey),
|
||||
),
|
||||
)
|
||||
.limit(1);
|
||||
const [interaction] = existingInteraction
|
||||
? await db
|
||||
.update(issueThreadInteractions)
|
||||
.set({
|
||||
status: "pending",
|
||||
requestedResolverPolicy: "human_only",
|
||||
effectiveResolverPolicy: "human_only",
|
||||
resolverPolicyProvenance: "explicit",
|
||||
effectiveResolverPolicySource: "requested",
|
||||
addresseeUserId: authorizationSubjectUserId,
|
||||
payload,
|
||||
result: null,
|
||||
resolvedAt: null,
|
||||
updatedAt: new Date(),
|
||||
})
|
||||
.where(eq(issueThreadInteractions.id, existingInteraction.id))
|
||||
.returning()
|
||||
: await db
|
||||
.insert(issueThreadInteractions)
|
||||
.values({
|
||||
companyId,
|
||||
issueId: input.issueId,
|
||||
kind: "request_confirmation",
|
||||
status: "pending",
|
||||
continuationPolicy: "none",
|
||||
requestedResolverPolicy: "human_only",
|
||||
effectiveResolverPolicy: "human_only",
|
||||
resolverPolicyProvenance: "explicit",
|
||||
effectiveResolverPolicySource: "requested",
|
||||
addresseeUserId: authorizationSubjectUserId,
|
||||
idempotencyKey,
|
||||
sourceRunId:
|
||||
binding.actorType === "agent"
|
||||
? (input.actor.sessionId ?? null)
|
||||
: null,
|
||||
title: `Connect your ${providerName} to continue`,
|
||||
summary: `${requestingAgent?.name ?? "An agent"} needs your ${providerName} identity for work running as you.`,
|
||||
createdByAgentId:
|
||||
binding.actorType === "agent" ? binding.actorId : null,
|
||||
payload,
|
||||
})
|
||||
.returning();
|
||||
if (interaction) {
|
||||
await db
|
||||
.update(toolOauthStates)
|
||||
.set({ interactionId: interaction.id })
|
||||
.where(eq(toolOauthStates.state, state));
|
||||
}
|
||||
const issueId = input.issueId;
|
||||
await db.transaction(async tx => {
|
||||
await notifyNativeReviewState(tx, { companyId, issueId });
|
||||
const [existingInteraction] = await tx
|
||||
.select()
|
||||
.from(issueThreadInteractions)
|
||||
.where(
|
||||
and(
|
||||
eq(issueThreadInteractions.companyId, companyId),
|
||||
eq(issueThreadInteractions.issueId, issueId),
|
||||
eq(issueThreadInteractions.idempotencyKey, idempotencyKey),
|
||||
),
|
||||
)
|
||||
.limit(1);
|
||||
const [interaction] = existingInteraction
|
||||
? await tx
|
||||
.update(issueThreadInteractions)
|
||||
.set({
|
||||
status: "pending",
|
||||
requestedResolverPolicy: "human_only",
|
||||
effectiveResolverPolicy: "human_only",
|
||||
resolverPolicyProvenance: "explicit",
|
||||
effectiveResolverPolicySource: "requested",
|
||||
addresseeUserId: authorizationSubjectUserId,
|
||||
payload,
|
||||
result: null,
|
||||
resolvedAt: null,
|
||||
updatedAt: new Date(),
|
||||
})
|
||||
.where(eq(issueThreadInteractions.id, existingInteraction.id))
|
||||
.returning()
|
||||
: await tx
|
||||
.insert(issueThreadInteractions)
|
||||
.values({
|
||||
companyId,
|
||||
issueId: issueId,
|
||||
kind: "request_confirmation",
|
||||
status: "pending",
|
||||
continuationPolicy: "none",
|
||||
requestedResolverPolicy: "human_only",
|
||||
effectiveResolverPolicy: "human_only",
|
||||
resolverPolicyProvenance: "explicit",
|
||||
effectiveResolverPolicySource: "requested",
|
||||
addresseeUserId: authorizationSubjectUserId,
|
||||
idempotencyKey,
|
||||
sourceRunId:
|
||||
binding.actorType === "agent"
|
||||
? (input.actor.sessionId ?? null)
|
||||
: null,
|
||||
title: `Connect your ${providerName} to continue`,
|
||||
summary: `${requestingAgent?.name ?? "An agent"} needs your ${providerName} identity for work running as you.`,
|
||||
createdByAgentId:
|
||||
binding.actorType === "agent" ? binding.actorId : null,
|
||||
payload,
|
||||
})
|
||||
.returning();
|
||||
if (interaction) {
|
||||
await tx
|
||||
.update(toolOauthStates)
|
||||
.set({ interactionId: interaction.id })
|
||||
.where(eq(toolOauthStates.state, state));
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
const nextConfig = {
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { notifyNativeReviewResponse } from "../../native-review-response-notifications.js";
|
||||
import { notifyNativeReviewState } from "../../native-review-state-notifications.js";
|
||||
import { notifyToolConnectionHealth } from "../../tool-connection-health-notifications.js";
|
||||
import { managedConnectorProfile } from "./client-settings.js";
|
||||
import type { ActorInfo } from "../types.js";
|
||||
@@ -421,7 +421,7 @@ export function createCloudOAuthCallback(db: Db, dependencies: Dependencies) {
|
||||
stateRow.interactionId &&
|
||||
linkedInteraction?.kind === "request_confirmation"
|
||||
) {
|
||||
await notifyNativeReviewResponse(tx, { companyId: connection.companyId, issueId: linkedInteraction.issueId });
|
||||
await notifyNativeReviewState(tx, { companyId: connection.companyId, issueId: linkedInteraction.issueId });
|
||||
await tx
|
||||
.update(issueThreadInteractions)
|
||||
.set({
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { notifyNativeReviewResponse } from "../../native-review-response-notifications.js";
|
||||
import { notifyNativeReviewState } from "../../native-review-state-notifications.js";
|
||||
import { notifyToolConnectionHealth } from "../../tool-connection-health-notifications.js";
|
||||
import { credentialScope } from "./client-settings.js";
|
||||
import { withoutOAuthRefreshLease } from "./refresh-leases.js";
|
||||
@@ -387,7 +387,7 @@ export function createDirectOAuthCallback(db: Db, dependencies: Dependencies) {
|
||||
stateRow.interactionId &&
|
||||
linkedInteraction?.kind === "request_confirmation"
|
||||
) {
|
||||
await notifyNativeReviewResponse(tx, { companyId: connection.companyId, issueId: linkedInteraction.issueId });
|
||||
await notifyNativeReviewState(tx, { companyId: connection.companyId, issueId: linkedInteraction.issueId });
|
||||
await tx
|
||||
.update(issueThreadInteractions)
|
||||
.set({
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { notifyNativeReviewResponse } from "../../native-review-response-notifications.js";
|
||||
import { notifyNativeReviewState } from "../../native-review-state-notifications.js";
|
||||
import type { ActorInfo } from "../types.js";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
@@ -164,7 +164,7 @@ async function rejectPendingOAuthInteraction(
|
||||
}
|
||||
if (!linked) return;
|
||||
await db.transaction(async tx => {
|
||||
await notifyNativeReviewResponse(tx, { companyId: stateRow.companyId, issueId: linked.issueId });
|
||||
await notifyNativeReviewState(tx, { companyId: stateRow.companyId, issueId: linked.issueId });
|
||||
await tx
|
||||
.update(issueThreadInteractions)
|
||||
.set({
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js";
|
||||
import { notifyApprovalQueueWork } from "./approval-work-notifications.js";
|
||||
import { and, desc, eq, inArray } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
@@ -274,6 +275,7 @@ export function workProductService(
|
||||
|
||||
createForIssue: async (issueId: string, companyId: string, data: Omit<typeof issueWorkProducts.$inferInsert, "issueId" | "companyId">) => {
|
||||
const row = await db.transaction(async (tx) => {
|
||||
await notifyNativeStatusReassessment(tx, { companyId, issueId });
|
||||
if (data.type === "pull_request") {
|
||||
await notifyTerminalWorkspaceWork(tx, `issue:${issueId}`);
|
||||
await notifyApprovalQueueWork(tx, companyId);
|
||||
@@ -312,6 +314,10 @@ export function workProductService(
|
||||
.where(eq(issueWorkProducts.id, id))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (!existing) return null;
|
||||
await notifyNativeStatusReassessment(tx, { companyId: existing.companyId, issueId: existing.issueId });
|
||||
if ((patch.companyId && patch.companyId !== existing.companyId) || (patch.issueId && patch.issueId !== existing.issueId)) {
|
||||
await notifyNativeStatusReassessment(tx, { companyId: patch.companyId ?? existing.companyId, issueId: patch.issueId ?? existing.issueId });
|
||||
}
|
||||
|
||||
if (existing.type === "pull_request" || patch.type === "pull_request") {
|
||||
await notifyTerminalWorkspaceWork(tx, `issue:${existing.issueId}`);
|
||||
@@ -385,6 +391,9 @@ export function workProductService(
|
||||
// prefix behind (which a retry would then duplicate). Mirrors the
|
||||
// per-writer transaction the batched issue/document writers use.
|
||||
await db.transaction(async (tx) => {
|
||||
for (const scope of new Map(values.map(value => [value.issueId, { companyId: value.companyId, issueId: value.issueId }])).values()) {
|
||||
await notifyNativeStatusReassessment(tx, scope);
|
||||
}
|
||||
for (const companyId of new Set(values.filter(value => value.type === "pull_request").map(value => value.companyId))) {
|
||||
await notifyApprovalQueueWork(tx, companyId);
|
||||
}
|
||||
@@ -399,6 +408,7 @@ export function workProductService(
|
||||
const row = await db.transaction(async tx => {
|
||||
const [existing] = await tx.select({ type: issueWorkProducts.type, issueId: issueWorkProducts.issueId, companyId: issueWorkProducts.companyId })
|
||||
.from(issueWorkProducts).where(eq(issueWorkProducts.id, id));
|
||||
if (existing) await notifyNativeStatusReassessment(tx, { companyId: existing.companyId, issueId: existing.issueId });
|
||||
if (existing?.type === "pull_request") {
|
||||
await notifyTerminalWorkspaceWork(tx, `issue:${existing.issueId}`);
|
||||
await notifyApprovalQueueWork(tx, existing.companyId);
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { workspaceOperations } from "@paperclipai/db";
|
||||
@@ -476,6 +477,14 @@ export function workspaceOperationService(db: Db) {
|
||||
},
|
||||
|
||||
async recordOperation(recordInput) {
|
||||
async function writeState<T>(write: (database: Db) => Promise<T>): Promise<T> {
|
||||
if (recordInput.phase !== "workspace_finalize" || !input.heartbeatRunId) return write(db);
|
||||
return db.transaction(async tx => {
|
||||
await notifyNativeStatusReassessment(tx, { companyId: input.companyId,
|
||||
runId: input.heartbeatRunId!, issueId: input.issueId ?? undefined });
|
||||
return write(tx as unknown as Db);
|
||||
});
|
||||
}
|
||||
const currentUserRedactionOptions = {
|
||||
enabled: (await instanceSettings.getGeneral()).censorUsernameInLogs,
|
||||
};
|
||||
@@ -521,7 +530,7 @@ export function workspaceOperationService(db: Db) {
|
||||
if (runtimeControlAction) liveRuntimeControlOperationIds.add(id);
|
||||
|
||||
try {
|
||||
await db.insert(workspaceOperations).values({
|
||||
await writeState(database => database.insert(workspaceOperations).values({
|
||||
id,
|
||||
companyId: input.companyId,
|
||||
executionWorkspaceId,
|
||||
@@ -539,7 +548,7 @@ export function workspaceOperationService(db: Db) {
|
||||
) as Record<string, unknown> | null,
|
||||
startedAt,
|
||||
updatedAt: startedAt,
|
||||
});
|
||||
}));
|
||||
} catch (insertError) {
|
||||
liveRuntimeControlOperationIds.delete(id);
|
||||
throw insertError;
|
||||
@@ -630,7 +639,7 @@ export function workspaceOperationService(db: Db) {
|
||||
await append("stderr", result.stderr ?? null);
|
||||
const finalized = await logStore.finalize(handle);
|
||||
const finishedAt = new Date();
|
||||
const row = await db
|
||||
const row = await writeState(database => database
|
||||
.update(workspaceOperations)
|
||||
.set({
|
||||
executionWorkspaceId,
|
||||
@@ -650,7 +659,7 @@ export function workspaceOperationService(db: Db) {
|
||||
})
|
||||
.where(eq(workspaceOperations.id, id))
|
||||
.returning()
|
||||
.then((rows) => rows[0] ?? null);
|
||||
.then((rows) => rows[0] ?? null));
|
||||
if (!row) throw notFound("Workspace operation not found");
|
||||
return toWorkspaceOperation(row);
|
||||
} catch (error) {
|
||||
@@ -660,7 +669,7 @@ export function workspaceOperationService(db: Db) {
|
||||
await append("stderr", error instanceof Error ? error.message : String(error)).catch(() => undefined);
|
||||
const finalized = await logStore.finalize(handle).catch(() => null);
|
||||
const finishedAt = new Date();
|
||||
await db
|
||||
await writeState(database => database
|
||||
.update(workspaceOperations)
|
||||
.set({
|
||||
executionWorkspaceId,
|
||||
@@ -687,7 +696,7 @@ export function workspaceOperationService(db: Db) {
|
||||
finishedAt,
|
||||
updatedAt: finishedAt,
|
||||
})
|
||||
.where(eq(workspaceOperations.id, id));
|
||||
.where(eq(workspaceOperations.id, id)));
|
||||
throw error;
|
||||
} finally {
|
||||
// Releasing the in-process claim before the request unwinds is what lets the very
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
|
||||
import { DELIVERY_QUEUES, subscribeDeliveryWork } from "./delivery-work-notifications.js";
|
||||
import { prepareHeartbeatWorkspace } from "./heartbeat/workspace-preparation.js";
|
||||
import { createHeartbeatWorkspaceResolver, resolveExecutionWorkspaceReuseRequestForIssue } from "./heartbeat/workspaces.js";
|
||||
import { environmentService } from "./environments.js";
|
||||
@@ -29,7 +31,6 @@ import { resumePreservedResultFinalization } from "./preserved-result-finalizati
|
||||
import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
|
||||
import type { EnvironmentLease } from "@paperclipai/shared";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
|
||||
import { prepareNativeWorkspaceSync } from "./native-runtime/native-workspace-sync.js";
|
||||
import { nativeWorkspaceFinalizationLockKey } from "./native-runtime/native-workspace-finalization-ownership.js";
|
||||
import { remoteTerminationReceipt } from "./remote-execution-termination.js";
|
||||
@@ -54,7 +55,8 @@ vi.mock("./environment-execution-target.js", async importOriginal => {
|
||||
});
|
||||
|
||||
describe("automatic retained workspace recovery", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let connectionString: string;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let root: string;
|
||||
const priorHome = process.env.PAPERCLIP_HOME, priorInstance = process.env.PAPERCLIP_INSTANCE_ID;
|
||||
@@ -66,7 +68,8 @@ describe("automatic retained workspace recovery", () => {
|
||||
await applyPendingMigrations(external);
|
||||
temporary = { connectionString: external, cleanup: async () => {} };
|
||||
} else temporary = await startEmbeddedPostgresTestDatabase("workspace-recovery-");
|
||||
db = createDb(temporary.connectionString);
|
||||
connectionString = temporary.connectionString;
|
||||
db = createDb(connectionString);
|
||||
if (external) await db.execute(sql`TRUNCATE TABLE companies CASCADE`);
|
||||
}, 30_000);
|
||||
afterAll(async () => {
|
||||
@@ -478,6 +481,21 @@ describe("automatic retained workspace recovery", () => {
|
||||
systemRecovery: { ...state, nextAttemptAt: 0 } } }).where(eq(issueRecoveryActions.id, actionId));
|
||||
}
|
||||
|
||||
it("publishes the committed finalization barrier after real retained workspace recovery", async () => {
|
||||
const f = await seed();
|
||||
const observed: Array<Promise<Array<{ status: string }>>> = [];
|
||||
const stop = subscribeDeliveryWork(db, DELIVERY_QUEUES.nativeStatusReassessment, () => {
|
||||
observed.push(db.select({ status: workspaceOperations.status }).from(workspaceOperations).where(and(
|
||||
eq(workspaceOperations.companyId, f.companyId), eq(workspaceOperations.heartbeatRunId, f.runId),
|
||||
eq(workspaceOperations.phase, "workspace_finalize"),
|
||||
)).then(rows => rows));
|
||||
});
|
||||
try { await recoverRetainedWorkspace(f.input); } finally { stop(); }
|
||||
expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete");
|
||||
expect(observed.length).toBeGreaterThan(0);
|
||||
expect((await Promise.all(observed)).some(rows => rows.some(row => row.status === "succeeded"))).toBe(true);
|
||||
});
|
||||
|
||||
it("re-enters only the existing finalizer for an accepted result, once per intent, preserving failure history", async () => {
|
||||
const f = await seed();
|
||||
const contractId = randomUUID(), resultId = randomUUID();
|
||||
@@ -582,7 +600,7 @@ describe("automatic retained workspace recovery", () => {
|
||||
expect(pending.status).toBe("running");
|
||||
expect(uploads).toBe(2);
|
||||
await makeRetryDue(f.actionId);
|
||||
const restarted = createDb(temporary.connectionString);
|
||||
const restarted = createDb(connectionString);
|
||||
try { await recoverRetainedWorkspace({ ...f.input, db: restarted }); }
|
||||
finally { await restarted.$client.end(); }
|
||||
const operations = await db.select().from(workspaceOperations).where(and(
|
||||
@@ -791,7 +809,7 @@ describe("automatic retained workspace recovery", () => {
|
||||
const pending = await action(f.actionId);
|
||||
await db.update(issueRecoveryActions).set({ evidence: { ...pending.evidence,
|
||||
systemRecovery: { ...readSystemRecovery(pending.evidence), nextAttemptAt: 0 } } }).where(eq(issueRecoveryActions.id, f.actionId));
|
||||
const restarted = createDb(temporary.connectionString);
|
||||
const restarted = createDb(connectionString);
|
||||
try {
|
||||
await recoverRetainedWorkspace({ ...f.input, db: restarted });
|
||||
expect(await hasVerifiedWorkspaceRepair(restarted, await run(f.runId))).toBe(true);
|
||||
@@ -1143,7 +1161,7 @@ describe("automatic retained workspace recovery", () => {
|
||||
}
|
||||
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0);
|
||||
expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0].status).toBe("released");
|
||||
const restarted = createDb(temporary.connectionString), onRecovered = vi.fn(async (_source: { id: string }) => {});
|
||||
const restarted = createDb(connectionString), onRecovered = vi.fn(async (_source: { id: string }) => {});
|
||||
try {
|
||||
await expect.poll(async () => {
|
||||
await reconcileRetainedWorkspaces(restarted, f.input.environmentRuntime, onRecovered);
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
import { beginIdleTrackedWork } from "./task-admission.js";
|
||||
import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
@@ -246,6 +247,7 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ
|
||||
const next = { ...current!, revision: current!.revision + 1, phase: "reconcile_effects" as const,
|
||||
state: "complete" as const, nextAttemptAt: null };
|
||||
const activity = await db.transaction(async tx => {
|
||||
await notifyNativeStatusReassessment(tx, input);
|
||||
await tx.insert(workspaceOperations).values({ companyId: input.companyId, issueId: input.issueId,
|
||||
heartbeatRunId: input.runId, phase: "workspace_finalize", status: "succeeded", finishedAt: new Date(),
|
||||
metadata: { owningService: "workspace_recovery", preservationOperationIds: next.preservationOperationIds } });
|
||||
|
||||
Reference in new issue
Block a user