mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-30 05:16:50 +01:00
fix(server): resolve native finalization notices from committed wakes (#17758)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native finalization can commit before its recovery notice is resolved. > - The periodic orphan pass revisits committed runs to resolve those notices. > - Retained completed history should not require repeated discovery. > - This PR moves that one duty to transaction wakes and concrete failure retries. > - Existing notice rows reconstruct work after restart. The action service still owns resolution. > - Other completion and resource duties keep their current recovery paths. ## Linked Issues or Issue Description Refs #17752 (stack parent: committed run projection), #17748 (accepted-result commit), and #17734 (workspace finalization). These are distinct recovery obligations. This slice incorporates parent #17752 at `a49a2dac13b0e1c19da5d3d6b435c781703b1c0b`, now merged as `50fe32684d2d4e1db129e3c1bb455571344a0c09`. Related #17724 retires obsolete exact-session notices for terminal runs with no result and preserves their replay hold. This slice resolves notices after a result committed; it does not change that no-replay policy. **Current behavior** Committed native finalization recovery repeatedly inspects active watchdog notices. A notice can also be updated in place to describe a different run, so resolving a previously selected ID alone is unsafe. **Proposed behavior** Wake a scoped worker from relevant committed writes. Reconstruct active finalization notices at startup. Resolve only the current notice after the native owner rechecks its binding under lifecycle locks. Stop scheduled work when no matching obligation remains. **Reason and benefit** Notice recovery becomes quiet after settlement and reacts to real changes without a polling delay. Reused notices remain attached to their current incident. This does not claim that every remaining server duty or the host can sleep yet. ## What Changed - Track the initial recovery-notice sweep through shutdown without blocking HTTP readiness on task locks. - Require issue/run scope for ordinary finalization fanout. Routine company-only hints leave dormant notices quiet; dedicated owner and explicit settings signals retain their scopes. - Extract the native finalization notice predicate and resolution owner. Keep the existing recovery-action service responsible for writing the resolution. - Bind candidates by company, issue and run. Require a committed result and an active or escalated finalization notice. Exclude held ownership, pending status effects and unfinished run projection. - Recheck coordinator, issue, run and matching actions under locks. A newer incident cannot be resolved from a stale candidate snapshot. - Use the shared scoped worker, transaction signals, retry backoff and lifecycle handling. Add no table, per-feature timer or provider branch. - Signal action insertion, update, supersession and resolution; finalization and effect changes; run projection and ownership release; issue changes; settings activation; and the recovery route that reopens an action. - Preserve no-op resolution quiet. Serialize only the three supported notification IDs, even when a caller supplies a larger internal object. - Delegate only notice resolution from the periodic path. Keep manual/default finalization and other completion/resource work. - Add real PostgreSQL producer, concurrency and lifecycle coverage and a detailed audit. ## Verification - Current local source `730e00a7b007a36d188ca0a328ef050d3a25d1ee` incorporates the parent with a normal merge. All 1,209 parent-only, eight owned-only and 12 automatic shared paths are preserved. The two manual paths retain both chat import sets and scope the new notice fanout. The entire package tree matches parent profile 80 byte-for-byte. - Controlled real-database regression: original notice-only unscoped fanout produces three candidate SELECTs for routine company hints. The same zero-query assertion passes after restoring scoped fanout. The real notice writer then creates and resolves an action for a committed run and returns to quiet. Both phases restore production bytes exactly; the 22 cases outside this focused pattern are not counted as passing there. - The complete main command passes **1,873 cases in 21 files**, without skips, in 309.69 seconds. It includes all 23 notice-worker cases, full presentation/ownership read coverage, 88 startup cases, and the projection/result/physical-work consumers. Process recovery passes 410 in 163.54 seconds and external-chat response passes 273 in 53.48 seconds. Total complete-file acceptance: **2,556 cases in 23 complete files**, with no skips. The additional focused chat selection passes 18 cases in 13.44 seconds; 1,796 unselected chat cases do not count. Total selected passes: **2,574**. Normal merge `a0f2b3f4348b078219a53d8cc135c84202ba505e` adopts actual parent ancestry with this complete tested child tree unchanged. - The chat selection covers the real reply/notice row-lock regression, Slack and Telegram response recovery, current-authority denial cases, exact canonical response binding and internal GitHub attachment navigation. Other cases in the giant chat file remain unselected; this is not full chat-suite acceptance. - Production server typecheck, build, module boundaries, migration order and source identity checks pass. Twenty dedicated PostgreSQL 17 databases use the unchanged migration 413 schema. Embedded-only invocations use the existing fresh-per-call recovery helper. All temporary imports restore exactly and all 24 selected test hashes match after completion. No assertion or timeout is relaxed. - Parent package acceptance remains applicable: 130 passing installation/compatibility/transport cases, 22 installed ACPX/Grok contract cases and 26 native-provider Rust cases. One existing opt-in closed Pi snapshot case is skipped because its distribution root is unset. This does not qualify a live provider. - Failure history and scope are retained in `doc/plans/2026-10-26-native-recovery-notice-sleep-audit.md`. Current-head acceptance: all 57 checks are clean, including protected CI and security. Greptile gives 5/5 on `2a8777bbed8b1e10810e3d6d99aae3f25b4854ef`, with no actionable findings. Prior scores do not substitute. No Ponytail verdict is present under the authorized missing-review policy. No full local repository suite, deployed sleep, expanded Cloud wake coverage or measured savings is claimed. ## Risks - Resolution takes coordinator, issue, run and action locks in the native recovery order. Contention must preserve the current incident; the real concurrent-upsert and competing-resolver cases exercise this boundary. - Process-local signals cover instrumented application writers. They do not add support for independent SQL writers or replicas without supported ingress. Startup reconstructs the existing durable rows. - Completed finalization is not proof that external conversation settlement, Muse reconciliation or physical cleanup finished. Those obligations retain their owners and recovery paths. - Discovery uses narrow keyset pages. No fleet benchmark, host suspension or cost reduction is claimed. Cloud wake coverage remains unchanged. - Parent #17752 has merged. This PR targets master at `50fe32684d2d4e1db129e3c1bb455571344a0c09`. Fresh current-head CI and review must pass before the authorized protected merge. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The exact serving model ID and context-window size are not exposed in this session. No subagent was used for this slice. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
22 files changed
+917
-52
No files matched your search
@@ -3270,3 +3270,20 @@ obligations. Workspace typechecking passes; the full local suite remains limited
|
||||
by embedded PostgreSQL startup capacity. Final local
|
||||
and remote acceptance is pending. Master refreshed to `ad861a7f94`; stack integration,
|
||||
prior failed gates, suppression audits and all remaining duties stay in scope.
|
||||
|
||||
## Slice 54 accepted; slice 55 recovery-notice resolution
|
||||
|
||||
Slice 54 [#17752](https://github.com/paperclipai/paperclip/pull/17752) is verified at
|
||||
`f98e6b5b57558c78ba111170ab3d8d19ff667827`: all 53 checks/status entries passed,
|
||||
Greptile 5/5, zero unresolved threads, 400 passing local cases, workspace types
|
||||
and build. The PR remains draft/open and unmerged. Intermediate failures and the
|
||||
full-local-suite limitation remain recorded in its audit and PR body.
|
||||
|
||||
Slice 55, `codex/sleep-55-native-recovery-notices`, moves only committed native
|
||||
finalization recovery-notice resolution to scoped transaction wakes. The existing
|
||||
action rows remain truth; the owner rechecks reused action identities under
|
||||
lifecycle locks. [Its audit](2026-10-26-native-recovery-notice-sleep-audit.md)
|
||||
records producer coverage, 102 passing worker/startup cases and 340 passing
|
||||
affected cases (442 total), plus workspace typechecking. Build and remote
|
||||
acceptance remain pending. All other inventory, prior gate failures,
|
||||
suppression audits and master propagation remain active obligations.
|
||||
@@ -0,0 +1,275 @@
|
||||
# Committed native recovery notices: sleep audit
|
||||
|
||||
Date: 2026-10-26. Slice 55 starts at
|
||||
`f98e6b5b57558c78ba111170ab3d8d19ff667827`,
|
||||
[parent #17752](https://github.com/paperclipai/paperclip/pull/17752).
|
||||
Branch: `codex/sleep-55-native-recovery-notices`. The full scope remains in the
|
||||
[recurring-server migration ledger](2026-10-24-recurring-server-sleep-stack.md).
|
||||
|
||||
## One obligation and its owner
|
||||
|
||||
A native finalization may commit while its recovery notice remains active. The
|
||||
old periodic finalizer replay called `resolveCommittedFinalizationRecovery` to
|
||||
resolve that notice. This slice extracts that duty into `native-recovery-notices`
|
||||
and schedules it from committed changes. The recovery-action service still owns
|
||||
the actual resolution write and its existing notifications. Other watchdogs,
|
||||
external conversation settlement, Muse reconciliation and physical resource
|
||||
settlement are separate obligations.
|
||||
|
||||
The existing `issue_recovery_actions` rows are durable truth. No new table,
|
||||
per-feature timer or provider branch is introduced. Direct/manual finalizer calls
|
||||
retain notice resolution by default. The periodic path passes a narrow
|
||||
`skipNativeRecoveryNotices` flag; all its other completion/resource behavior
|
||||
remains enabled. A real reconciler test verifies the resource-settlement callback
|
||||
still runs when notice resolution is delegated.
|
||||
|
||||
## Pending predicate and quiet states
|
||||
|
||||
A candidate binds company, issue and run across the native coordinator, heartbeat
|
||||
run and issue. The coordinator must be committed and have a result. Discovery
|
||||
excludes held runner ownership, pending status effects and unfinished run
|
||||
projection. A correlated notice lookup requires:
|
||||
|
||||
- The same company, source issue and `evidence.runId`.
|
||||
- Kind `active_run_watchdog` and status `active` or `escalated`.
|
||||
- `wakePolicy.kind = resume_native_run`, or the existing special cause
|
||||
`native_workspace_finalization_owner_unverified`.
|
||||
|
||||
An exhausted owner-verification notice can still need resolution even after its
|
||||
retry policy is cleared. Unrelated notices, uncommitted runs, resolved history,
|
||||
and otherwise idle configuration do not arm a recurring scan. Discovery reads
|
||||
IDs in keyset pages of 100 and the shared worker executes company-fair batches of
|
||||
25. Real resolution failures retain shared per-entry backoff, starting at 30
|
||||
seconds and bounded at five minutes. A successful attempt with remaining work
|
||||
retains a scoped 30-second retry. Startup reconstructs durable unfinished notices.
|
||||
|
||||
The owner first checks for a matching notice to avoid taking lifecycle locks on
|
||||
the normal already-settled path. That read is an optimization, not authority.
|
||||
Actual resolution locks coordinator, issue, run, then matching actions, following
|
||||
native recovery's existing order. It rechecks the current committed coordinator,
|
||||
run binding and physical ownership. The background caller also checks effect and
|
||||
projection settlement under those locks. The existing action service performs
|
||||
the resolution inside that transaction.
|
||||
|
||||
## Why the action must be read again
|
||||
|
||||
`upsertSourceScoped` can reuse an active action ID and change its run, cause,
|
||||
fingerprint or policy. Resolving a captured ID alone can erase a newer incident.
|
||||
The owner selects matching actions only after acquiring lifecycle locks and locks
|
||||
them before calling the resolution service. The regression holds the run row lock,
|
||||
starts resolution, changes the notice through the real upsert service, and then
|
||||
releases the lock. The new notice stays active. A separate four-writer regression
|
||||
observes exactly one recovery-action UPDATE.
|
||||
|
||||
## Producer and boundary audit
|
||||
|
||||
| Producer | Wake contract |
|
||||
| --- | --- |
|
||||
| Recovery-action insert, supersession, in-place update and budget exhaustion | Scoped notice hint before the existing transaction write. This includes changes to kind or wake policy alone, not just run/cause changes. |
|
||||
| Recovery-action resolution | Signal only after a matching pending row is found and before the write. An empty resolution call must not wake this worker. |
|
||||
| Native decision/finalization and pending-effect settlement | Existing finalization notification fanout now includes notices. The committed transaction remains authoritative. |
|
||||
| Run projection and heartbeat lifecycle status writers | Scoped signal before the write; finishing projection or releasing an ownership hold can make a notice actionable. |
|
||||
| Relevant issue changes | Existing issue transaction sends an issue-scoped hint. |
|
||||
| Worktree execution activation | Global reconstruction hint through the existing settings transaction. |
|
||||
| Recovery route reopening a resolved action | Explicit issue-scoped hint in the reopening transaction. |
|
||||
|
||||
`native-workspace-export-retry` already uses the finalization fanout when it changes
|
||||
the action to `resume_native_run`; its coordinator becomes `result_accepted`, so
|
||||
notice resolution waits for the later commit. Native workspace owner recovery
|
||||
likewise queues result finalization; the subsequent commit supplies the activation.
|
||||
|
||||
Other direct action writers were reviewed. Legacy execution admission and
|
||||
historical backfill create different causes without a native-resume policy.
|
||||
Backfill, system recovery, workspace owner progress and queue continuation writes
|
||||
preserve the top-level run/cause/kind/status eligibility while adding evidence.
|
||||
The wake-queue adapter adds failure-block evidence. Explicit continuation creates
|
||||
resolved authorization receipts or resolves previous actions. Safe replacement,
|
||||
workspace reconstruction/export recovery and execution-recovery resolution remove
|
||||
active obligations. Their existing queued candidate is rechecked and disarmed;
|
||||
they cannot create new eligible notice work. Recovery-service monitor/deadline and
|
||||
board-disposition writes do not introduce the matching native policy. New eligible
|
||||
notice creation remains with the instrumented action service.
|
||||
|
||||
All producer hints use the existing database intent and outer transaction protocol.
|
||||
Nested savepoint release is not commit. Unknown outcomes stay with that shared
|
||||
protocol. Process-local signals cover instrumented writers in this process;
|
||||
independent SQL writers and replicas without supported ingress are not newly
|
||||
supported. No database LISTEN connection is added.
|
||||
|
||||
The notification boundary explicitly copies only company, issue and run IDs. A
|
||||
structurally compatible caller can carry additional internal fields; serializing
|
||||
its entire object is not safe. A real finalization regression exposed this during
|
||||
implementation when a caller's database object caused circular JSON serialization.
|
||||
The corrected hint uses the same narrow payload contract as adjacent workers.
|
||||
|
||||
## Lifecycle and verification
|
||||
|
||||
Public heartbeat registration preserves task drain, restore and worktree execution
|
||||
suppression. Settings activation reconstructs work. The coordinator retains a
|
||||
first write during drain and a failed retry through drain/restart, stops future
|
||||
subscription work on shutdown and joins active resolution. A new startup discovers
|
||||
notices written while stopped.
|
||||
|
||||
At this checkpoint, **102 tests pass**: 22 new worker/owner cases and 80 startup
|
||||
cases. Coverage includes startup, zero idle SQL, resolved history, real first and
|
||||
late producers, outer commit/rollback, finalizer commit, policy/kind-only changes,
|
||||
supersession, escalated owner notices, concurrent resolution, action reuse while
|
||||
waiting for a real row lock, no-op resolution, pending effects, projection,
|
||||
ownership release, suppression/activation, drain/retry/restart, shutdown and
|
||||
retained periodic resource settlement.
|
||||
|
||||
Preserved failures:
|
||||
|
||||
- Before producer wiring, six tests produced three failures: late notice, outer
|
||||
commit and later finalizer commit did not wake the worker. The other three passed.
|
||||
- The first wiring run failed all six notice tests while all 80 startup cases
|
||||
passed. A stronger fixture assertion exposed `side_effect_planning_failed` with
|
||||
circular JSON serialization. The hint now copies its three supported IDs. All
|
||||
initial and expanded cases pass with the real finalizer.
|
||||
|
||||
Local logs: `/tmp/sleep55-initial-red.log`, `/tmp/sleep55-producers-green.log`
|
||||
(the failed intermediate run), `/tmp/sleep55-producer-diagnostic.log`,
|
||||
`/tmp/sleep55-lifecycle.log` (96 passes), and `/tmp/sleep55-final-lifecycle.log`
|
||||
(102 passes). Six complete affected suites also pass: 340 cases across native finalization
|
||||
recovery, finalizer telemetry, recovery actions, the status arbiter corpus,
|
||||
result-commit work and run projection. Total: **442 passing local cases**.
|
||||
Workspace typechecking passes. Evidence is in `/tmp/sleep55-affected.log` and
|
||||
`/tmp/sleep55-types.log`. Build, publication and exact-head CI/review remain
|
||||
pending at this checkpoint. The full local suite is not claimed: this host's
|
||||
embedded PostgreSQL startup previously exhausted semaphore capacity. No further
|
||||
shared OS cleanup was performed; fresh-host CI is required for full acceptance.
|
||||
|
||||
## Remaining work
|
||||
|
||||
Master was refreshed; `origin/master` was subsequently observed at `125ddcabe5`.
|
||||
Full-stack integration of later master changes and fresh gates remains required.
|
||||
The older failed gates and eight-worker restore/worktree suppression audit remain
|
||||
open. All other duties in the ledger remain in scope. This establishes worker
|
||||
behavior, not deployed database suspension, host sleep or fleet savings. The stack
|
||||
is unmerged, and this request does not authorize merging it.
|
||||
|
||||
## Reply and notice contention repair
|
||||
|
||||
The committed-response owner previously locked issue, run, then coordinator.
|
||||
Notice resolution, status arbitration, and physical workspace recovery lock
|
||||
coordinator, issue, then run. Concurrent reply and notice repair could therefore
|
||||
deadlock even though both were authorized to settle the same committed result.
|
||||
The reply owner now acquires and validates its exact committed, unleased
|
||||
coordinator first, then retains its existing issue/run and presentation checks.
|
||||
No publication authority, completion policy, retry cadence, or transaction scope
|
||||
is broadened. The change belongs with this notice worker because introducing
|
||||
that concurrent owner exposed the conflicting order.
|
||||
|
||||
The regression uses a real committed Telegram reply, real notice creation, and
|
||||
the real resolution/publication owners. It holds the notice transaction at its
|
||||
first coordinator lock, observes the reply's actual PostgreSQL blocking state,
|
||||
then continues notice resolution with a bounded local lock timeout. Before the
|
||||
fix PostgreSQL rejected reply recovery with `40P01` (deadlock detected). After
|
||||
the fix both operations commit, the notice resolves, and exactly one accepted
|
||||
comment/publication exists; replay publishes nothing.
|
||||
|
||||
The focused regression passed, followed by 73 selected committed/accepted-response
|
||||
chat cases (1,570 unrelated chat cases filtered out) and all 141 cases in the
|
||||
notice, finalizer telemetry, startup, and response-authority suites. Logs:
|
||||
`/tmp/sleep55-lock-order-{red,green,chat,consumers}.log`. This is 214 distinct
|
||||
passing cases across five files, not a full chat-file or full repository test
|
||||
run. Integrated typecheck, build, and fresh remote checks remain required.
|
||||
|
||||
## Projection fixture follow-through after the lock-order repair
|
||||
|
||||
The coordinator-first response-recovery guard exposed stale projection fixtures.
|
||||
Thirteen cases inserted a native run without a committed finalization record.
|
||||
Recovery correctly stopped before loading that run, so the expected measured
|
||||
projection was absent. The failure reproduced locally against a new disposable
|
||||
PostgreSQL database: 13 failures and four passes, matching the CI failures on
|
||||
later stack branches.
|
||||
|
||||
The fixture now uses the existing accepted-result and finalization setup. It
|
||||
retains a future retry while comparing narrow and full-row reads, so both paths
|
||||
observe the same state without publishing between comparisons. The selected
|
||||
answer guard, typed context flags, non-object result evidence, payload byte
|
||||
limits, unchanged-row assertions, company/issue scope checks, and separate real
|
||||
publication comparison remain. All 17 cases now pass; server typechecking also
|
||||
passes. The existing external-test-database convention permits a real database
|
||||
replay without requiring another embedded PostgreSQL instance. No production
|
||||
lock, recovery eligibility, retry policy, or projection was weakened.
|
||||
|
||||
## Startup readiness follow-up (2026-10-27)
|
||||
|
||||
A held recovery-notice sweep reproduced HTTP startup waiting for task locks.
|
||||
Registration now tracks its readiness promise through the existing heartbeat
|
||||
shutdown owner instead of awaiting it on the readiness path. The coordinator
|
||||
continues to own worker cancellation. The regression proves HTTP readiness and
|
||||
subsequent registration proceed while shutdown waits for the held work. All 86
|
||||
startup cases pass on the master-integrated branch; no timer or persistence is
|
||||
added. Full integrated typechecking and head-specific CI remain separate gates.
|
||||
|
||||
## Current parent integration and routine-hint regression (2026-10-29)
|
||||
|
||||
This slice incorporates parent #17752 at
|
||||
`a49a2dac13b0e1c19da5d3d6b435c781703b1c0b` through a normal merge. A
|
||||
batched tree comparison preserves 1,209 parent-only entries, eight owned-only
|
||||
entries, 11,524 identical entries and 12 automatic shared merges. Two manual
|
||||
paths remain explicit: ordinary notice fanout is inside the existing issue/run
|
||||
guard; the chat test's leading import conflict retains all four independent
|
||||
imports and removes only the three marker lines. Every other automatically
|
||||
merged byte in that test remains intact before comment removal. The complete
|
||||
package tree matches parent profile 80, including all historical identities.
|
||||
|
||||
Thirteen added comment objects in five files were removed with identical nontrivia
|
||||
TypeScript token streams. No directive was removed. Notice settlement retains
|
||||
coordinator, issue, run and action lock ordering; exact current incident matching;
|
||||
existing action-resolution side effects; and the separate status/effect/resource
|
||||
owners. A discovered action ID alone is not authority to resolve a later incident.
|
||||
|
||||
The new real-database regression submits three routine company-only finalization
|
||||
hints after startup becomes quiet. A controlled baseline moves only notice fanout
|
||||
outside the guard and keeps every other parent guard. It issues three candidate
|
||||
SELECTs and fails the zero-query assertion. Restoring production bytes passes the
|
||||
same assertion. The real notice writer then creates and resolves an action for a
|
||||
committed run, returns outcome `restored`, and settles back to quiet. Red fails one
|
||||
selected case in 7.46 seconds; green passes that same case. Twenty-two cases outside
|
||||
the focused pattern are not counted as passing there. Both phases use fresh
|
||||
dedicated PostgreSQL 17 databases and restore the production helper exactly.
|
||||
|
||||
The complete main command passes 1,873 cases in 21 files in 309.69 seconds, with
|
||||
no skips. Process recovery and external-chat response run in separate database
|
||||
phases. The giant chat integration file is intentionally a focused selection,
|
||||
including the real reply/notice row-lock regression and related provider and
|
||||
current-authority cases; unselected chat cases do not constitute acceptance.
|
||||
All test assertions and timeouts are unchanged.
|
||||
|
||||
### Complete current-source acceptance
|
||||
|
||||
Source `730e00a7b007a36d188ca0a328ef050d3a25d1ee` passes **2,556 cases
|
||||
in 23 complete files**, with no skips in those files. The main command passes
|
||||
1,873 cases in 21 files in 309.69 seconds. Process recovery passes 410 in 163.54
|
||||
seconds; external-chat response passes 273 in 53.48 seconds. A separate focused
|
||||
chat selection passes 18 cases in 13.44 seconds, including the actual reply/notice
|
||||
row-lock race, Slack/Telegram recovery, current-authority refusals, canonical
|
||||
response binding and internal GitHub navigation. The other 1,796 cases in that
|
||||
chat file remain unselected. Total selected passing cases: 2,574. This is not full
|
||||
chat-suite or repository acceptance.
|
||||
|
||||
The complete files include all 23 notice-worker cases, 17 presentation/ownership
|
||||
read cases and 88 startup cases. Production server typechecking, build, module
|
||||
boundaries, migration order and the source identity check pass. Twenty dedicated
|
||||
PostgreSQL 17 databases use the unchanged migration 413 schema. Embedded-only
|
||||
calls use existing fresh-per-call recovery fixtures. Every temporary import is
|
||||
restored byte-for-byte and all 24 selected file hashes match after completion.
|
||||
No assertion or timeout is relaxed. The focused scope green takes 6.69 seconds;
|
||||
the intended failing unscoped baseline is not counted as acceptance.
|
||||
|
||||
Parent #17752 merged as `50fe32684d2d4e1db129e3c1bb455571344a0c09`, after
|
||||
57 clean current-head checks and Greptile 5/5. Its complete reviewed tree equals
|
||||
the actual squash. Normal merge `a0f2b3f4348b078219a53d8cc135c84202ba505e`
|
||||
adopts that ancestry with this child's entire tested tree unchanged. This slice
|
||||
can now target master. Fresh exact-head CI and review remain separate gates.
|
||||
|
||||
The unchanged package tree retains parent profile 80 acceptance: 130 passing
|
||||
installation/compatibility/transport cases, 22 installed ACPX/Grok contract cases
|
||||
and 26 native-provider Rust cases. One existing opt-in closed Pi snapshot case
|
||||
was skipped because its distribution root was unset. This does not qualify a live
|
||||
Pi installation. Earlier failures in this audit retain their revision and
|
||||
subsequent disposition. No deployed sleep, expanded Cloud alarm coverage or
|
||||
measured savings is claimed.
|
||||
@@ -1,3 +1,6 @@
|
||||
import { resolveCommittedNativeRecoveryNotices, pendingNativeRecoveryNotices } from "../services/native-runtime/native-recovery-notices.js";
|
||||
import { issueRecoveryActionService } from "../services/issue-recovery-actions.js";
|
||||
import { projectCommittedRun } from "../services/native-runtime/native-run-projection.js";
|
||||
import { notifyNativeFinalizationWork } from "../services/native-finalization-notifications.js";
|
||||
import { nativeResponsePresentationWork } from "../services/native-runtime/native-response-presentation-work.js";
|
||||
import { notifyChatRunMilestones } from "../services/chat-run-milestone-notifications.js";
|
||||
@@ -74469,6 +74472,54 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
};
|
||||
}
|
||||
|
||||
it("repairs a committed reply and its recovery notice without reversing row locks", async () => {
|
||||
const context = await committedChatResponseRecoveryFixture("telegram");
|
||||
const scope = { companyId: context.fixture.companyId, issueId: context.issue.id, runId: context.runId };
|
||||
let response: Promise<boolean> | undefined;
|
||||
try {
|
||||
const [run] = await readHeartbeatRunsById(context.runId);
|
||||
await projectCommittedRun({ db, run, coordinator: context.coordinator });
|
||||
const notice = await issueRecoveryActionService(db).upsertSourceScoped({
|
||||
companyId: scope.companyId, sourceIssueId: scope.issueId, kind: "active_run_watchdog",
|
||||
ownerType: "system", cause: "native_workspace_sync_out_failed", fingerprint: scope.runId,
|
||||
evidence: { runId: scope.runId }, nextAction: "Retry finalization",
|
||||
wakePolicy: { kind: "resume_native_run", runId: scope.runId },
|
||||
});
|
||||
expect(await pendingNativeRecoveryNotices(db, scope)).toHaveLength(1);
|
||||
await db.transaction(async tx => {
|
||||
|
||||
|
||||
await tx.select().from(nativeRunFinalizations)
|
||||
.where(eq(nativeRunFinalizations.runId, scope.runId)).for("update");
|
||||
const [backend] = await tx.execute(sql`select pg_backend_pid() as pid`) as unknown as Array<{ pid: number }>;
|
||||
response = recoverCommittedNativeChatResponse(db, { ...scope, onlyWhenDue: true });
|
||||
|
||||
void response.catch(() => {});
|
||||
await vi.waitFor(async () => {
|
||||
const [state] = await db.execute(sql`select exists (
|
||||
select 1 from pg_stat_activity where datname = current_database()
|
||||
and ${backend.pid} = any(pg_blocking_pids(pid))
|
||||
) as blocked`) as unknown as Array<{ blocked: boolean }>;
|
||||
expect(state.blocked).toBe(true);
|
||||
});
|
||||
await tx.execute(sql`set local lock_timeout = '1s'`);
|
||||
await resolveCommittedNativeRecoveryNotices(tx as unknown as Db, scope, true);
|
||||
});
|
||||
await expect(response).resolves.toBe(true);
|
||||
expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, notice.id)))[0])
|
||||
.toMatchObject({ status: "resolved", outcome: "restored" });
|
||||
const comments = await db.select().from(issueComments).where(eq(issueComments.createdByRunId, scope.runId));
|
||||
expect(comments).toHaveLength(1);
|
||||
expect(comments[0].body).toBe(context.result.summary);
|
||||
expect(await readChatPublicationsByCommentId(comments[0].id)).toHaveLength(1);
|
||||
await expect(recoverCommittedNativeChatResponse(db, scope)).resolves.toBe(false);
|
||||
expect(await pendingNativeRecoveryNotices(db, scope)).toEqual([]);
|
||||
} finally {
|
||||
await response?.catch(() => {});
|
||||
await context.service.shutdown();
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["slack", "telegram"] as const)(
|
||||
"recovers a committed %s answer once without rerunning or changing review/quarantine evidence",
|
||||
async (provider) => {
|
||||
|
||||
@@ -0,0 +1,345 @@
|
||||
import { projectCommittedRun } from "../services/native-runtime/native-run-projection.js";
|
||||
import { heartbeatService } from "../services/heartbeat.js";
|
||||
import { instanceSettingsService } from "../services/instance-settings.js";
|
||||
import { createHeartbeatLifecycle, type HeartbeatLifecycleDependencies } from "../services/heartbeat/run-lifecycle.js";
|
||||
import { reconcileNativeFinalizations } from "../services/native-runtime/native-finalization-reconciler.js";
|
||||
import { statusDecisionEffects } from "@paperclipai/db";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { mkdtemp, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { agents, companies, completionContracts, createDb, heartbeatRuns, issues, nativeRunFinalizations, type Db } from "@paperclipai/db";
|
||||
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
||||
import { nativeRecoveryNoticeWork } from "../services/native-runtime/native-recovery-notice-work.js";
|
||||
import { resolveCommittedNativeRecoveryNotices } from "../services/native-runtime/native-recovery-notices.js";
|
||||
import { notifyNativeFinalizationWork } from "../services/native-finalization-notifications.js";
|
||||
import { issueRecoveryActionService } from "../services/issue-recovery-actions.js";
|
||||
import { issueRecoveryActions } from "@paperclipai/db";
|
||||
import { finalizeNativeRun } from "../services/native-runtime/native-run-finalizer.js";
|
||||
import * as statusCommitter from "../services/native-runtime/status-decision-committer.js";
|
||||
import { PaperclipControlPlanePort } from "../services/native-runtime/paperclip-control-plane-port.js";
|
||||
import { NativeRunCoordinatorStore } from "../services/native-runtime/native-run-coordinator-store.js";
|
||||
import { workspaceOperationService } from "../services/workspace-operations.js";
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
import { CONTROL_PLANE_CONFORMANCE_RESULT, CONTROL_PLANE_CONFORMANCE_TERMINAL } from "../vendor/paperclip-runner/testing.js";
|
||||
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { emitAgentTaskRun } from "../services/agent-task-run-telemetry.js";
|
||||
vi.mock("../services/agent-task-run-telemetry.js", () => ({ emitAgentTaskRun: vi.fn(async () => undefined) }));
|
||||
vi.mock("../services/run-failure-report.js", async importOriginal => ({
|
||||
...await importOriginal<typeof import("../services/run-failure-report.js")>(), reportRunFailure: vi.fn(async () => undefined),
|
||||
}));
|
||||
|
||||
describe("committed native recovery notice lifecycle", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let db: Db, root: string;
|
||||
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
|
||||
let worker: ReturnType<ReturnType<typeof nativeRecoveryNoticeWork>["register"]> | undefined;
|
||||
const companyId = randomUUID(), agentId = randomUUID();
|
||||
const priorLogs = process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
|
||||
let admitted = true;
|
||||
const errors: unknown[] = [];
|
||||
beforeAll(async () => {
|
||||
root = await mkdtemp(join(tmpdir(), "native-recovery-notice-"));
|
||||
process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = root;
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_NATIVE_RECOVERY_NOTICE_TEST_DATABASE_URL");
|
||||
if (!external) temporary = await startEmbeddedPostgresTestDatabase("native-recovery-notice-");
|
||||
db = createDb(external ?? temporary!.connectionString);
|
||||
await db.insert(companies).values({ id: companyId, name: "Result commits", issuePrefix: `RC${randomUUID().slice(0, 6)}` });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Result worker", adapterType: "paperclip_runner" });
|
||||
}, 30_000);
|
||||
beforeEach(() => {
|
||||
admitted = true; errors.length = 0; vi.mocked(emitAgentTaskRun).mockClear();
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
});
|
||||
afterEach(async () => {
|
||||
await coordinator?.stop(); coordinator = undefined; worker = undefined;
|
||||
vi.useRealTimers(); vi.restoreAllMocks();
|
||||
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, companyId));
|
||||
expect(errors).toEqual([]);
|
||||
});
|
||||
afterAll(async () => {
|
||||
await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); await rm(root, { recursive: true, force: true });
|
||||
if (priorLogs === undefined) delete process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
|
||||
else process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = priorLogs;
|
||||
});
|
||||
async function start(project = resolveCommittedNativeRecoveryNotices) {
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => admitted, canReconcile: () => admitted,
|
||||
onError: error => errors.push(error) });
|
||||
worker = nativeRecoveryNoticeWork(db, project).register(coordinator, () => admitted);
|
||||
await worker.ready;
|
||||
}
|
||||
async function flush() {
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
expect(coordinator!.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 5000 });
|
||||
}
|
||||
async function quiet() {
|
||||
const previous = db.$client.options.debug;
|
||||
const queries: string[] = [];
|
||||
db.$client.options.debug = (_id, query) => { queries.push(query); };
|
||||
try { await vi.advanceTimersByTimeAsync(120_000); await delay(10); expect(queries).toEqual([]); }
|
||||
finally { db.$client.options.debug = previous; }
|
||||
}
|
||||
async function seed() {
|
||||
const runId = randomUUID(), issueId = randomUUID(), contractId = randomUUID(), sourceInstanceId = randomUUID();
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Commit accepted work", status: "in_progress", assigneeAgentId: agentId });
|
||||
await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 1,
|
||||
schemaVersion: "paperclip.completion-contract.v1", policyVersion: "result-worker-v1", risk: "standard",
|
||||
completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal",
|
||||
contractJson: { revision: "result-worker-v1", objective: "Finish", criteria: [{ id: "objective", requirement: "Complete the task" }] },
|
||||
canonicalSha256: contractId, createdByActorType: "system", createdByActorId: "test" });
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, runtimeMode: "native", status: "running",
|
||||
nativeIssueId: issueId, nativeSessionId: runId, runnerInstanceId: sourceInstanceId,
|
||||
completionContractId: contractId, completionContractSha256: contractId, contextSnapshot: { issueId } });
|
||||
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
|
||||
return { companyId, issueId, runId, agentId, sessionId: runId, completionContractId: contractId,
|
||||
completionContractSha256: contractId, sourceInstanceId, controlPlaneSourceInstanceId: "result-worker-test" };
|
||||
}
|
||||
async function complete(binding: Awaited<ReturnType<typeof seed>>, producer: "control_plane" | "coordinator", writer: Db = db, terminalStatus: "succeeded" | "failed" | "cancelled" = "succeeded") {
|
||||
const port = producer === "control_plane" ? new PaperclipControlPlanePort(writer, binding)
|
||||
: new NativeRunCoordinatorStore(writer, { ...binding, normalizedSessionId: binding.sessionId,
|
||||
runnerSourceInstanceId: binding.sourceInstanceId,
|
||||
completionContractRevision: CONTROL_PLANE_CONFORMANCE_RESULT.completionClaim.contractRevision,
|
||||
completionContractCriterionIds: CONTROL_PLANE_CONFORMANCE_RESULT.completionClaim.criteria.map(item => item.criterionId) });
|
||||
await port.completeRun({ result: CONTROL_PLANE_CONFORMANCE_RESULT, terminal: { ...CONTROL_PLANE_CONFORMANCE_TERMINAL, runTerminalState: terminalStatus },
|
||||
callerResultId: `${binding.runId}:result` });
|
||||
}
|
||||
async function barrier(binding: Awaited<ReturnType<typeof seed>>) {
|
||||
await workspaceOperationService(db).createRecorder({ companyId, issueId: binding.issueId, heartbeatRunId: binding.runId })
|
||||
.recordOperation({ phase: "workspace_finalize", cwd: root, run: async () => ({ status: "succeeded", exitCode: 0 }) });
|
||||
}
|
||||
async function state(runId: string) {
|
||||
return db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)).then(rows => rows[0]);
|
||||
}
|
||||
async function read(runId: string) {
|
||||
return db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).then(rows => rows[0]);
|
||||
}
|
||||
async function accepted(status: "succeeded" | "failed" | "cancelled" = "succeeded") {
|
||||
const binding = await seed(); await complete(binding, "control_plane", db, status); await barrier(binding); return binding;
|
||||
}
|
||||
async function committed() {
|
||||
const binding = await accepted();
|
||||
const result = await finalizeNativeRun({ db, runId: binding.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
|
||||
expect(result.phase, JSON.stringify(await state(binding.runId))).toBe("committed");
|
||||
return binding;
|
||||
}
|
||||
async function notice(binding: Awaited<ReturnType<typeof seed>>, writer: Db = db,
|
||||
patch: Partial<Parameters<ReturnType<typeof issueRecoveryActionService>["upsertSourceScoped"]>[0]> = {}) {
|
||||
return issueRecoveryActionService(writer).upsertSourceScoped({ companyId, sourceIssueId: binding.issueId,
|
||||
kind: "active_run_watchdog", ownerType: "system", cause: "native_workspace_sync_out_failed",
|
||||
fingerprint: binding.runId, evidence: { runId: binding.runId }, nextAction: "Retry finalization",
|
||||
wakePolicy: { kind: "resume_native_run", runId: binding.runId }, ...patch });
|
||||
}
|
||||
async function action(id: string) {
|
||||
return db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, id)).then(rows => rows[0]);
|
||||
}
|
||||
it("reconstructs pending notices and leaves resolved history quiet across restart", async () => {
|
||||
const binding = await committed(); const pending = await notice(binding);
|
||||
await start(); await flush(); expect(await action(pending.id)).toMatchObject({ status: "resolved", outcome: "restored" });
|
||||
await quiet(); const settled = await action(pending.id);
|
||||
await coordinator!.stop(); await start(); await flush(); await quiet(); expect(await action(pending.id)).toEqual(settled);
|
||||
});
|
||||
it("ignores routine company-only finalization hints and still resolves a real notice", async () => {
|
||||
await start(); await flush(); await quiet();
|
||||
const previous = db.$client.options.debug;
|
||||
const queries: string[] = [];
|
||||
db.$client.options.debug = (_id, query) => {
|
||||
if (/from\s+"native_run_finalizations"/i.test(query)) queries.push(query);
|
||||
};
|
||||
try {
|
||||
for (let index = 0; index < 3; index += 1) {
|
||||
await db.transaction(async tx => { await notifyNativeFinalizationWork(tx, { companyId }); });
|
||||
await flush();
|
||||
}
|
||||
expect(queries).toEqual([]);
|
||||
} finally { db.$client.options.debug = previous; }
|
||||
const binding = await committed();
|
||||
const pending = await notice(binding); await flush();
|
||||
expect(await action(pending.id)).toMatchObject({ status: "resolved", outcome: "restored" });
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("wakes from a real notice created after startup is quiet", async () => {
|
||||
const binding = await committed(); await start(); await flush(); await quiet();
|
||||
const pending = await notice(binding); await flush(); expect((await action(pending.id)).status).toBe("resolved"); await quiet();
|
||||
});
|
||||
it.each(["commit", "rollback"])("honors the notice writer's outer %s", async outcome => {
|
||||
const binding = await committed(); await start(); await flush();
|
||||
let actionId = "";
|
||||
const transaction = db.transaction(async tx => {
|
||||
actionId = (await notice(binding, tx as unknown as Db)).id;
|
||||
await vi.advanceTimersByTimeAsync(5000);
|
||||
expect(await action(actionId)).toBeUndefined();
|
||||
if (outcome === "rollback") throw new Error("rollback notice");
|
||||
});
|
||||
if (outcome === "rollback") await expect(transaction).rejects.toThrow("rollback notice"); else await transaction;
|
||||
await flush();
|
||||
if (outcome === "rollback") expect(await action(actionId)).toBeUndefined();
|
||||
else expect((await action(actionId)).status).toBe("resolved");
|
||||
await quiet();
|
||||
});
|
||||
it("wakes when the real finalizer commits after a notice was already present", async () => {
|
||||
const binding = await accepted(); const pending = await notice(binding);
|
||||
await start(); await flush(); await quiet();
|
||||
await finalizeNativeRun({ db, runId: binding.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true,
|
||||
skipNativeRecoveryNotices: true });
|
||||
await flush(); expect((await action(pending.id)).status).toBe("resolved"); await quiet();
|
||||
});
|
||||
it("preserves a notice rebound to a new run after discovery", async () => {
|
||||
const binding = await committed(); const pending = await notice(binding);
|
||||
const differentRun = randomUUID();
|
||||
const resolve: typeof resolveCommittedNativeRecoveryNotices = async (database, scope, settled) => {
|
||||
const updated = await notice({ ...binding, runId: differentRun }); expect(updated.id).toBe(pending.id);
|
||||
await resolveCommittedNativeRecoveryNotices(database, scope, settled);
|
||||
};
|
||||
await start(resolve); await flush();
|
||||
expect(await action(pending.id)).toMatchObject({ status: "active", evidence: { runId: differentRun } }); await quiet();
|
||||
});
|
||||
it("does not wake for an already-resolved notice", async () => {
|
||||
const binding = await committed(); const pending = await notice(binding);
|
||||
await start(); await flush();
|
||||
const queries: string[] = []; const previous = db.$client.options.debug;
|
||||
db.$client.options.debug = (_id, query) => { queries.push(query); };
|
||||
try {
|
||||
await issueRecoveryActionService(db).resolveActiveForIssue({ companyId, sourceIssueId: binding.issueId,
|
||||
actionId: pending.id, status: "resolved", outcome: "restored" });
|
||||
await flush();
|
||||
expect(queries.filter(query => query.includes('from "native_run_finalizations"'))).toEqual([]);
|
||||
} finally { db.$client.options.debug = previous; }
|
||||
await quiet();
|
||||
});
|
||||
it.each(["restore", "worktree"])("preserves %s suppression and settings activation", async mode => {
|
||||
const binding = await committed(); const pending = await notice(binding);
|
||||
const runtimeEnv = mode === "restore" ? { PAPERCLIP_DATABASE_RESTORE_IN_PROGRESS: "true" }
|
||||
: { PAPERCLIP_IN_WORKTREE: "true", PAPERCLIP_INSTANCE_ID: "notice-suppression-test" };
|
||||
const settings = instanceSettingsService(db, { runtimeEnv });
|
||||
if (mode === "worktree") await settings.updateExperimental({ enableWorktreeRunExecution: false });
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => admitted, canReconcile: () => admitted,
|
||||
onError: error => errors.push(error) });
|
||||
worker = heartbeatService(db, { runtimeEnv }).registerNativeRecoveryNoticeWork(coordinator, () => admitted);
|
||||
await worker.ready; await flush(); await quiet(); expect((await action(pending.id)).status).toBe("active");
|
||||
if (mode === "worktree") {
|
||||
try { await settings.updateExperimental({ enableWorktreeRunExecution: true }); await flush();
|
||||
expect((await action(pending.id)).status).toBe("resolved"); await quiet();
|
||||
} finally { await settings.updateExperimental({ enableWorktreeRunExecution: false }); }
|
||||
}
|
||||
});
|
||||
it("retains the first notice committed during drain", async () => {
|
||||
const binding = await committed(); await start(); await flush(); admitted = false;
|
||||
const pending = await notice(binding); await vi.advanceTimersByTimeAsync(120_000); await quiet();
|
||||
expect((await action(pending.id)).status).toBe("active"); admitted = true;
|
||||
await vi.advanceTimersByTimeAsync(30_000); await flush(); expect((await action(pending.id)).status).toBe("resolved"); await quiet();
|
||||
});
|
||||
it("retains a real failure retry through drain and restart", async () => {
|
||||
const binding = await committed(); const pending = await notice(binding);
|
||||
const resolve: typeof resolveCommittedNativeRecoveryNotices = async () => { throw new Error("notice unavailable"); };
|
||||
await start(resolve); expect(errors).toHaveLength(1); errors.length = 0;
|
||||
expect(coordinator!.nextWakeAt()).not.toBeNull(); admitted = false;
|
||||
await vi.advanceTimersByTimeAsync(120_000); await quiet(); expect((await action(pending.id)).status).toBe("active");
|
||||
await coordinator!.stop(); admitted = true; await start(); await flush();
|
||||
expect((await action(pending.id)).status).toBe("resolved"); await quiet();
|
||||
});
|
||||
it("joins in-flight resolution on shutdown and reconstructs later notices", async () => {
|
||||
let release!: () => void; const gate = new Promise<void>(resolve => { release = resolve; }); const entered = vi.fn();
|
||||
const resolve: typeof resolveCommittedNativeRecoveryNotices = async (...args) => {
|
||||
entered(); await gate; return resolveCommittedNativeRecoveryNotices(...args);
|
||||
};
|
||||
const first = await committed(); const second = await committed(); await start(resolve); await flush();
|
||||
const pending = await notice(first);
|
||||
await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(1000); expect(entered).toHaveBeenCalledTimes(1); });
|
||||
const later = await notice(second); let stopped = false;
|
||||
const stopping = coordinator!.stop().then(() => { stopped = true; });
|
||||
await delay(20); expect(stopped).toBe(false); release(); await stopping;
|
||||
expect((await action(pending.id)).status).toBe("resolved"); expect((await action(later.id)).status).toBe("active");
|
||||
await start(); await flush(); expect((await action(later.id)).status).toBe("resolved"); await quiet();
|
||||
});
|
||||
it("waits for real pending-effect settlement", async () => {
|
||||
const binding = await committed(); const pending = await notice(binding); const current = await state(binding.runId);
|
||||
const [effect] = await db.select().from(statusDecisionEffects).where(eq(statusDecisionEffects.decisionId, current.decisionId!));
|
||||
expect(effect).toBeDefined();
|
||||
await db.update(statusDecisionEffects).set({ deliveryState: "pending", deliveredAt: null }).where(eq(statusDecisionEffects.id, effect.id));
|
||||
await start(); await flush(); await quiet(); expect((await action(pending.id)).status).toBe("active");
|
||||
await statusCommitter.dispatchPendingNativeStatusEffects({ db, ...binding, decisionId: current.decisionId!, onlyWhenDue: true });
|
||||
await flush(); expect((await action(pending.id)).status).toBe("resolved"); await quiet();
|
||||
});
|
||||
it("waits for projection and wakes from the real projection owner", async () => {
|
||||
const binding = await committed(); const pending = await notice(binding);
|
||||
await db.update(heartbeatRuns).set({ status: "running", finishedAt: null }).where(eq(heartbeatRuns.id, binding.runId));
|
||||
await start(); await flush(); await quiet(); expect((await action(pending.id)).status).toBe("active");
|
||||
await projectCommittedRun({ db, run: await read(binding.runId), coordinator: await state(binding.runId) });
|
||||
await flush(); expect((await action(pending.id)).status).toBe("resolved"); await quiet();
|
||||
});
|
||||
it("wakes after the real lifecycle writer releases an ownership hold", async () => {
|
||||
const binding = await committed(); const pending = await notice(binding);
|
||||
await db.update(heartbeatRuns).set({ status: "running", nativePhase: "terminal_failure", errorCode: "native_execution_ownership_unverified" })
|
||||
.where(eq(heartbeatRuns.id, binding.runId));
|
||||
await start(); await flush(); await quiet(); expect((await action(pending.id)).status).toBe("active");
|
||||
const lifecycle = createHeartbeatLifecycle(db, { emitTerminalAgentTaskRun: vi.fn() } as unknown as HeartbeatLifecycleDependencies);
|
||||
await lifecycle.setRunStatus(binding.runId, "succeeded", { nativePhase: "committed", errorCode: null });
|
||||
await flush(); expect((await action(pending.id)).status).toBe("resolved"); await quiet();
|
||||
});
|
||||
it("delegates only notice repair and preserves periodic resource settlement", async () => {
|
||||
const binding = await committed(); const pending = await notice(binding); const settled = vi.fn(async () => undefined);
|
||||
await reconcileNativeFinalizations(db, [binding.runId], { skipNativeRecoveryNotices: true,
|
||||
skipNativeResponsePresentation: true, skipNativeReviewResponse: true, onWorkspaceSettled: settled });
|
||||
expect((await action(pending.id)).status).toBe("active"); expect(settled).toHaveBeenCalledTimes(1);
|
||||
await start(); await flush(); expect((await action(pending.id)).status).toBe("resolved"); await quiet();
|
||||
});
|
||||
it.each(["wakePolicy", "kind"])("activates an existing notice after only its %s changes", async field => {
|
||||
const binding = await committed();
|
||||
const pending = await notice(binding, db, field === "kind" ? { kind: "missing_disposition" } : { wakePolicy: null });
|
||||
await start(); await flush(); await quiet(); expect((await action(pending.id)).status).toBe("active");
|
||||
const updated = await notice(binding); expect(updated.id).toBe(pending.id);
|
||||
await flush(); expect((await action(pending.id)).status).toBe("resolved"); await quiet();
|
||||
});
|
||||
it("resolves the escalated owner-verification notice without a retry policy", async () => {
|
||||
const binding = await committed();
|
||||
const pending = await notice(binding, db, { cause: "native_workspace_finalization_owner_unverified", maxAttempts: 1 });
|
||||
await notice(binding, db, { cause: "native_workspace_finalization_owner_unverified", maxAttempts: 1 });
|
||||
expect(await action(pending.id)).toMatchObject({ status: "escalated", wakePolicy: null });
|
||||
await start(); await flush(); expect((await action(pending.id)).status).toBe("resolved"); await quiet();
|
||||
});
|
||||
it("wakes for a new identity while preserving the superseded notice", async () => {
|
||||
const binding = await committed(); const old = await notice({ ...binding, runId: randomUUID() });
|
||||
await start(); await flush(); await quiet();
|
||||
const pending = await notice(binding, db, { supersedeOnIdentityChange: true });
|
||||
expect(pending.id).not.toBe(old.id); await flush();
|
||||
expect((await action(old.id)).status).toBe("cancelled"); expect((await action(pending.id)).status).toBe("resolved"); await quiet();
|
||||
});
|
||||
it("writes a resolution once across competing live/recovery owners", async () => {
|
||||
const binding = await committed(); const pending = await notice(binding);
|
||||
const queries: string[] = []; const previous = db.$client.options.debug;
|
||||
db.$client.options.debug = (_id, query) => { queries.push(query); };
|
||||
try {
|
||||
await Promise.all(Array.from({ length: 4 }, () => resolveCommittedNativeRecoveryNotices(db, binding, true)));
|
||||
expect(queries.filter(query => query.startsWith('update "issue_recovery_actions"'))).toHaveLength(1);
|
||||
} finally { db.$client.options.debug = previous; }
|
||||
expect((await action(pending.id)).status).toBe("resolved");
|
||||
});
|
||||
it("rechecks the action after waiting for the run lock", async () => {
|
||||
const binding = await committed(); const pending = await notice(binding);
|
||||
let release!: () => void; let locked!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; });
|
||||
const ready = new Promise<void>(resolve => { locked = resolve; });
|
||||
const holding = db.transaction(async tx => {
|
||||
await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(eq(heartbeatRuns.id, binding.runId)).for("update");
|
||||
locked(); await gate;
|
||||
});
|
||||
await ready;
|
||||
let waiting = false; const previous = db.$client.options.debug;
|
||||
db.$client.options.debug = (_id, query) => { if (query.includes('from "heartbeat_runs"') && query.includes('for update')) waiting = true; };
|
||||
const resolving = resolveCommittedNativeRecoveryNotices(db, binding, true);
|
||||
const replacementRun = randomUUID();
|
||||
try {
|
||||
await vi.waitFor(() => { expect(waiting).toBe(true); });
|
||||
const updated = await notice({ ...binding, runId: replacementRun }); expect(updated.id).toBe(pending.id);
|
||||
} finally { release(); db.$client.options.debug = previous; await holding; await resolving; }
|
||||
expect(await action(pending.id)).toMatchObject({ status: "active", evidence: { runId: replacementRun } });
|
||||
});
|
||||
});
|
||||
@@ -120,6 +120,7 @@ const {
|
||||
registerRetainedNativeSessionCleanupWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve(), stop: vi.fn() })),
|
||||
registerNativeWorkspaceFinalizationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve(), stop: vi.fn() })),
|
||||
registerNativeResultCommitWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerNativeRecoveryNoticeWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerNativeRunProjectionWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerNativeStatusReassessmentWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerNativeReviewResponseWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
@@ -599,6 +600,13 @@ describe("startServer feedback export wiring", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it.each([false, true])("registers native recovery notices only when maintenance is enabled (%s)", async enabled => {
|
||||
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
|
||||
await startServer(); await settleStartupHousekeeping();
|
||||
expect(heartbeatServiceMock.registerNativeRecoveryNoticeWork).toHaveBeenCalledTimes(enabled ? 1 : 0);
|
||||
if (enabled) expect(heartbeatServiceMock.registerNativeRecoveryNoticeWork.mock.calls[0][1]()).toBe(true);
|
||||
});
|
||||
|
||||
it.each([false, true])("registers native run projection only when maintenance is enabled (%s)", async enabled => {
|
||||
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
|
||||
await startServer(); await settleStartupHousekeeping();
|
||||
@@ -1074,7 +1082,7 @@ describe("startServer feedback export wiring", () => {
|
||||
intervalCallback?.();
|
||||
await new Promise<void>((resolve) => setImmediate(resolve));
|
||||
expect(heartbeatServiceMock.sweepStaleIssueLocks).toHaveBeenCalledTimes(2);
|
||||
expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true, skipRetainedNativeSessionCleanup: true, skipNativeWorkspaceFinalization: true, skipNativeResultCommit: true, skipNativeRunProjection: true });
|
||||
expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true, skipRetainedNativeSessionCleanup: true, skipNativeWorkspaceFinalization: true, skipNativeResultCommit: true, skipNativeRunProjection: true, skipNativeRecoveryNotices: true });
|
||||
expect(heartbeatServiceMock.reconcileCostAccounting).toHaveBeenCalledTimes(1);
|
||||
expect(retiredDetector).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
@@ -1230,7 +1238,7 @@ describe("startServer feedback export wiring", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["retained cleanup", "workspace finalization", "native run projection", "native result commit"] as const)("becomes ready while %s is held but joins it at shutdown", async owner => {
|
||||
it.each(["retained cleanup", "workspace finalization", "native run projection", "native result commit", "native recovery notices"] as const)("becomes ready while %s is held but joins it at shutdown", async owner => {
|
||||
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: true }));
|
||||
let releaseCleanup!: () => void;
|
||||
const ready = new Promise<void>(resolve => { releaseCleanup = resolve; });
|
||||
@@ -1238,9 +1246,11 @@ describe("startServer feedback export wiring", () => {
|
||||
const register = owner === "retained cleanup" ? heartbeatServiceMock.registerRetainedNativeSessionCleanupWork
|
||||
: owner === "workspace finalization" ? heartbeatServiceMock.registerNativeWorkspaceFinalizationWork
|
||||
: owner === "native run projection" ? heartbeatServiceMock.registerNativeRunProjectionWork
|
||||
: heartbeatServiceMock.registerNativeResultCommitWork;
|
||||
: owner === "native result commit" ? heartbeatServiceMock.registerNativeResultCommitWork
|
||||
: heartbeatServiceMock.registerNativeRecoveryNoticeWork;
|
||||
if (owner === "native run projection") heartbeatServiceMock.registerNativeRunProjectionWork.mockReturnValueOnce({ ready });
|
||||
else if (owner === "native result commit") heartbeatServiceMock.registerNativeResultCommitWork.mockReturnValueOnce({ ready });
|
||||
else if (owner === "native recovery notices") heartbeatServiceMock.registerNativeRecoveryNoticeWork.mockReturnValueOnce({ ready });
|
||||
else (owner === "retained cleanup" ? heartbeatServiceMock.registerRetainedNativeSessionCleanupWork
|
||||
: heartbeatServiceMock.registerNativeWorkspaceFinalizationWork).mockReturnValueOnce({ ready, stop });
|
||||
const timer = vi.spyOn(globalThis, "setInterval").mockImplementation(() => 1 as unknown as ReturnType<typeof setInterval>);
|
||||
|
||||
+7
-1
@@ -1842,6 +1842,12 @@ async function startServerWithDatabaseTeardown(
|
||||
server.on("close", worker.stop);
|
||||
|
||||
|
||||
trackHeartbeatSchedulerWork(worker.ready);
|
||||
}
|
||||
if (config.heartbeatSchedulerEnabled) {
|
||||
const worker = heartbeat.registerNativeRecoveryNoticeWork(deliveryWork,
|
||||
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby());
|
||||
|
||||
trackHeartbeatSchedulerWork(worker.ready);
|
||||
}
|
||||
if (config.heartbeatSchedulerEnabled) {
|
||||
@@ -2008,7 +2014,7 @@ async function startServerWithDatabaseTeardown(
|
||||
// Periodically reap orphaned runs (5-min staleness threshold) and make sure
|
||||
// persisted queued work is still being driven forward.
|
||||
runPeriodicRecovery(() => heartbeat
|
||||
.reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true, skipRetainedNativeSessionCleanup: true, skipNativeWorkspaceFinalization: true, skipNativeResultCommit: true, skipNativeRunProjection: true })
|
||||
.reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true, skipRetainedNativeSessionCleanup: true, skipNativeWorkspaceFinalization: true, skipNativeResultCommit: true, skipNativeRunProjection: true, skipNativeRecoveryNotices: true })
|
||||
.then(() => heartbeat.promoteDueScheduledRetries())
|
||||
.then(async (promotion) => {
|
||||
await heartbeat.resumeQueuedRuns();
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeRecoveryNotices } from "../services/native-recovery-notice-notifications.js";
|
||||
import { notifyNativeStatusReassessment } from "../services/native-status-reassessment-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "../services/unsafe-workspace-export-notifications.js";
|
||||
import { notifyChatInboundWakeNotices } from "../services/chat-inbound-wake-notice-notifications.js";
|
||||
@@ -9933,6 +9934,7 @@ export function issueRoutes(
|
||||
);
|
||||
}
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: lockedIssue.companyId, issueId: lockedIssue.id });
|
||||
await notifyNativeRecoveryNotices(tx, { companyId: lockedIssue.companyId, issueId: lockedIssue.id });
|
||||
const [reopened] = await tx
|
||||
.update(issueRecoveryActions)
|
||||
.set({ status: "active", outcome: null, resolvedAt: null })
|
||||
|
||||
@@ -2,6 +2,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db";
|
||||
|
||||
/** Topics for existing durable queues; register intent before their writes. */
|
||||
export const DELIVERY_QUEUES = {
|
||||
nativeRecoveryNotices: "native-recovery-notices",
|
||||
nativeRunProjection: "native-run-projection",
|
||||
nativeResultCommit: "native-result-commit",
|
||||
nativeWorkspaceFinalization: "native-workspace-finalization",
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { nativeRecoveryNoticeWork } from "./native-runtime/native-recovery-notice-work.js";
|
||||
import { nativeRunProjectionWork } from "./native-runtime/native-run-projection-work.js";
|
||||
import { nativeResultCommitWork } from "./native-runtime/native-result-commit-work.js";
|
||||
import { nativeStatusReassessmentWork } from "./native-runtime/native-status-reassessment-work.js";
|
||||
@@ -4096,6 +4097,14 @@ export function heartbeatService(
|
||||
triggerIssueMonitor,
|
||||
|
||||
reportRunActivity: clearDetachedRunWarning,
|
||||
registerNativeRecoveryNoticeWork: (
|
||||
coordinator: Parameters<ReturnType<typeof nativeRecoveryNoticeWork>["register"]>[0], canRun: () => boolean,
|
||||
) => nativeRecoveryNoticeWork(db).register(coordinator,
|
||||
() => canRun() && readTaskDrain(new Date()) === null,
|
||||
async () => {
|
||||
const state = await getSchedulingSuppression(true);
|
||||
return !state.suppressed || state.reason === "task_drain";
|
||||
}),
|
||||
registerNativeRunProjectionWork: (
|
||||
coordinator: Parameters<ReturnType<typeof nativeRunProjectionWork>["register"]>[0], canRun: () => boolean,
|
||||
) => nativeRunProjectionWork(db).register(coordinator,
|
||||
|
||||
@@ -1967,7 +1967,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
|
||||
let settledProcessesDiscovered = false;
|
||||
|
||||
async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean; skipNativeResponsePresentation?: boolean; skipNativeReviewResponse?: boolean; skipNativeStatusReassessment?: boolean; skipRetainedNativeSessionCleanup?: boolean; skipNativeWorkspaceFinalization?: boolean; skipNativeResultCommit?: boolean; skipNativeRunProjection?: boolean }) {
|
||||
async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean; skipNativeResponsePresentation?: boolean; skipNativeReviewResponse?: boolean; skipNativeStatusReassessment?: boolean; skipRetainedNativeSessionCleanup?: boolean; skipNativeWorkspaceFinalization?: boolean; skipNativeResultCommit?: boolean; skipNativeRunProjection?: boolean; skipNativeRecoveryNotices?: boolean }) {
|
||||
if (!opts?.skipCancelledWorkspacePreservation) await preserveCancelledNativeWorkspaces(db);
|
||||
const staleThresholdMs = opts?.staleThresholdMs ?? 0;
|
||||
const now = new Date();
|
||||
@@ -1995,6 +1995,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
skipNativeWorkspaceFinalization: opts?.skipNativeWorkspaceFinalization,
|
||||
skipNativeResultCommit: opts?.skipNativeResultCommit,
|
||||
skipNativeRunProjection: opts?.skipNativeRunProjection,
|
||||
skipNativeRecoveryNotices: opts?.skipNativeRecoveryNotices,
|
||||
environmentRuntime,
|
||||
pluginWorkerManager: dependencies.pluginWorkerManager,
|
||||
onWorkspaceSettled: settleRecoveredNativeWorkspace,
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeRecoveryNotices } from "../native-recovery-notice-notifications.js";
|
||||
import { notifyNativeRunProjection } from "../native-run-projection-notifications.js";
|
||||
import { notifyRetainedNativeSessionCleanup } from "../retained-native-session-cleanup-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js";
|
||||
@@ -1026,6 +1027,7 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl
|
||||
// commit cannot strand newly-actionable accounting after the live event.
|
||||
return db.transaction(async transaction => {
|
||||
const tx = transaction as unknown as Db;
|
||||
if (previous?.runtimeMode === "native") await notifyNativeRecoveryNotices(tx, { companyId: previous.companyId, runId: previous.id });
|
||||
if (previous?.runtimeMode === "native") await notifyNativeRunProjection(tx, { companyId: previous.companyId, runId: previous.id });
|
||||
if (previous && previous.status !== status) await notifyUnsafeWorkspaceExportRunChange(tx, previous);
|
||||
if (previous && (previous.status === "scheduled_retry" || status === "scheduled_retry"))
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeRecoveryNotices } from "./native-recovery-notice-notifications.js";
|
||||
import { notifyNativeRunProjection } from "./native-run-projection-notifications.js";
|
||||
import { notifyNativeResultCommit } from "./native-result-commit-notifications.js";
|
||||
import { notifyNativeWorkspaceFinalization } from "./native-workspace-finalization-notifications.js";
|
||||
@@ -629,6 +630,7 @@ export function instanceSettingsService(db: Db, options: InstanceSettingsService
|
||||
await notifyDirectoryCleanup(tx);
|
||||
await notifyInstructionCollection(tx);
|
||||
await notifyCancelledWorkspacePreservation(tx);
|
||||
await notifyNativeRecoveryNotices(tx);
|
||||
await notifyNativeRunProjection(tx);
|
||||
await notifyNativeResultCommit(tx);
|
||||
await notifyNativeWorkspaceFinalization(tx);
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeRecoveryNotices } from "./native-recovery-notice-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js";
|
||||
import { writeExecutionControl } from "./execution-control-notifications.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
@@ -321,6 +322,7 @@ export function issueRecoveryActionService(db: DbOrTransaction) {
|
||||
): Promise<IssueRecoveryAction> {
|
||||
try {
|
||||
const created = await db.transaction(async (tx) => {
|
||||
await notifyNativeRecoveryNotices(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionDisposition);
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionContinuation);
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
@@ -411,6 +413,7 @@ export function issueRecoveryActionService(db: DbOrTransaction) {
|
||||
Math.min(nextAttemptCount, effectiveMaxAttempts),
|
||||
);
|
||||
const [exhausted] = await writeExecutionControl(db, [DELIVERY_QUEUES.executionDisposition, DELIVERY_QUEUES.executionContinuation], async tx => {
|
||||
await notifyNativeRecoveryNotices(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
if ((!input.preserveExistingOwner && existing.cause !== input.cause)
|
||||
|| (input.evidence?.runId !== undefined && input.evidence.runId !== existing.evidence?.runId)) {
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
@@ -471,6 +474,7 @@ export function issueRecoveryActionService(db: DbOrTransaction) {
|
||||
return toReadModel(exhausted);
|
||||
}
|
||||
const [updated] = await writeExecutionControl(db, [DELIVERY_QUEUES.executionDisposition, DELIVERY_QUEUES.executionContinuation], async tx => {
|
||||
await notifyNativeRecoveryNotices(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
if ((!input.preserveExistingOwner && existing.cause !== input.cause)
|
||||
|| (input.evidence?.runId !== undefined && input.evidence.runId !== existing.evidence?.runId)) {
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
@@ -544,6 +548,7 @@ export function issueRecoveryActionService(db: DbOrTransaction) {
|
||||
|
||||
try {
|
||||
const [created] = await writeExecutionControl(db, [DELIVERY_QUEUES.executionDisposition, DELIVERY_QUEUES.executionContinuation], async tx => {
|
||||
await notifyNativeRecoveryNotices(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
return tx
|
||||
.insert(issueRecoveryActions)
|
||||
@@ -590,6 +595,7 @@ export function issueRecoveryActionService(db: DbOrTransaction) {
|
||||
const pending = await tx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions)
|
||||
.where(and(...predicates)).limit(1);
|
||||
if (!pending.length) return [];
|
||||
await notifyNativeRecoveryNotices(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: input.sourceIssueId });
|
||||
return tx
|
||||
.update(issueRecoveryActions)
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeRecoveryNotices } from "./native-recovery-notice-notifications.js";
|
||||
import { notifyNativeRunProjection } from "./native-run-projection-notifications.js";
|
||||
import { notifyNativeResultCommit } from "./native-result-commit-notifications.js";
|
||||
import { notifyNativeWorkspaceFinalization } from "./native-workspace-finalization-notifications.js";
|
||||
@@ -11505,6 +11506,7 @@ export function issueService(db: Db) {
|
||||
if (patch.status !== undefined || patch.statusVersion !== undefined || patch.executionState !== undefined
|
||||
|| patch.executionRunId !== undefined || patch.checkoutRunId !== undefined
|
||||
|| patch.assigneeAgentId !== undefined || patch.assigneeUserId !== undefined || blockedByIssueIds !== undefined) {
|
||||
await notifyNativeRecoveryNotices(tx, { companyId: receiptExisting.companyId, issueId: id });
|
||||
await notifyNativeRunProjection(tx, { companyId: receiptExisting.companyId, issueId: id });
|
||||
await notifyNativeResultCommit(tx, { companyId: receiptExisting.companyId, issueId: id });
|
||||
await notifyNativeWorkspaceFinalization(tx, { companyId: receiptExisting.companyId, issueId: id });
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeRecoveryNotices } from "./native-recovery-notice-notifications.js";
|
||||
import { notifyNativeRunProjection } from "./native-run-projection-notifications.js";
|
||||
import { notifyNativeResultCommit } from "./native-result-commit-notifications.js";
|
||||
import { notifyNativeWorkspaceFinalization } from "./native-workspace-finalization-notifications.js";
|
||||
@@ -11,6 +12,7 @@ export type NativeFinalizationWorkScope = { companyId: string; runId?: string; i
|
||||
|
||||
export async function notifyNativeFinalizationWork(transaction: object, scope: NativeFinalizationWorkScope) {
|
||||
if (scope.issueId || scope.runId) {
|
||||
await notifyNativeRecoveryNotices(transaction, scope);
|
||||
await notifyNativeRunProjection(transaction, scope);
|
||||
await notifyNativeResultCommit(transaction, scope);
|
||||
await notifyNativeWorkspaceFinalization(transaction, scope);
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
|
||||
export type NativeRecoveryNoticeScope = { companyId: string; issueId?: string; runId?: string };
|
||||
|
||||
export function notifyNativeRecoveryNotices(tx: object, scope?: NativeRecoveryNoticeScope) {
|
||||
return notifyDeliveryWork(tx, DELIVERY_QUEUES.nativeRecoveryNotices,
|
||||
scope ? JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId, runId: scope.runId }) : "*");
|
||||
}
|
||||
@@ -391,6 +391,7 @@ type NativeReconciliationOptions = {
|
||||
skipNativeWorkspaceFinalization?: boolean;
|
||||
skipNativeResultCommit?: boolean;
|
||||
skipNativeRunProjection?: boolean;
|
||||
skipNativeRecoveryNotices?: boolean;
|
||||
pluginWorkerManager?: WorkspaceWorkers;
|
||||
environmentRuntime?: EnvironmentRuntimeService;
|
||||
onWorkspaceSettled?: (input: {
|
||||
@@ -688,6 +689,7 @@ async function reconcileNativeFinalizationsPass(
|
||||
skipNativeResponsePresentation: options.skipNativeResponsePresentation,
|
||||
skipNativeReviewResponse: options.skipNativeReviewResponse,
|
||||
skipCommittedRunProjection: options.skipNativeRunProjection,
|
||||
skipNativeRecoveryNotices: options.skipNativeRecoveryNotices,
|
||||
preserveProviderAttempt: Boolean(
|
||||
readNativeWorkspaceSyncReference(
|
||||
row.nativeWorkspaceSync,
|
||||
@@ -750,6 +752,7 @@ async function reconcileNativeFinalizationsPass(
|
||||
skipNativeResponsePresentation: options.skipNativeResponsePresentation,
|
||||
skipNativeReviewResponse: options.skipNativeReviewResponse,
|
||||
skipCommittedRunProjection: options.skipNativeRunProjection,
|
||||
skipNativeRecoveryNotices: options.skipNativeRecoveryNotices,
|
||||
preserveProviderAttempt: Boolean(
|
||||
readNativeWorkspaceSyncReference(
|
||||
row.nativeWorkspaceSync,
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
|
||||
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
|
||||
import type { NativeRecoveryNoticeScope } from "../native-recovery-notice-notifications.js";
|
||||
import { registerScopedDeadlineWork } from "../scoped-deadline-work.js";
|
||||
import { pendingNativeRecoveryNotices, resolveCommittedNativeRecoveryNotices } from "./native-recovery-notices.js";
|
||||
|
||||
type Entry = { id: string; companyId: string; issueId: string; dueAt: number };
|
||||
|
||||
export function nativeRecoveryNoticeWork(db: Db, resolve = resolveCommittedNativeRecoveryNotices) {
|
||||
async function load(scope?: NativeRecoveryNoticeScope, signal?: AbortSignal, canRun = () => true): Promise<Entry[]> {
|
||||
const entries: Entry[] = [];
|
||||
let after: string | undefined;
|
||||
while (canRun() && !signal?.aborted) {
|
||||
const rows = await pendingNativeRecoveryNotices(db, scope, after);
|
||||
entries.push(...rows.map(row => ({ ...row, dueAt: 0 })));
|
||||
if (rows.length < 100) return entries;
|
||||
after = rows[rows.length - 1].id;
|
||||
}
|
||||
throw new Error("Native recovery notice discovery stopped");
|
||||
}
|
||||
async function check(entry: Entry, signal?: AbortSignal): Promise<Entry | null> {
|
||||
if (signal?.aborted) throw new Error("Native recovery notice resolution stopped");
|
||||
const scope = { companyId: entry.companyId, issueId: entry.issueId, runId: entry.id };
|
||||
await resolve(db, scope, true);
|
||||
return (await pendingNativeRecoveryNotices(db, scope)).length ? { ...entry, dueAt: Date.now() + 30_000 } : null;
|
||||
}
|
||||
function register(coordinator: ReturnType<typeof createDeliveryWorkCoordinator>, canRun: () => boolean,
|
||||
canRecover: () => Promise<boolean> = async () => true) {
|
||||
return registerScopedDeadlineWork(coordinator, {
|
||||
queue: DELIVERY_QUEUES.nativeRecoveryNotices, batchSize: 25,
|
||||
decodeScope: (hint: string): NativeRecoveryNoticeScope | undefined => hint === "*" ? undefined : JSON.parse(hint),
|
||||
matches: (entry: Entry, scope?: NativeRecoveryNoticeScope) => !scope || entry.companyId === scope.companyId
|
||||
&& (!scope.issueId || entry.issueId === scope.issueId) && (!scope.runId || entry.id === scope.runId),
|
||||
load: async (scope, signal) => await canRecover() ? load(scope, signal, canRun) : [],
|
||||
check: async (entry, signal) => {
|
||||
if (!canRun()) throw new Error("Native recovery notice resolution paused");
|
||||
return await canRecover() ? check(entry, signal) : null;
|
||||
},
|
||||
}, canRun);
|
||||
}
|
||||
return { load, check, register };
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
import { and, asc, eq, gt, inArray, isNotNull, not, or, sql } from "drizzle-orm";
|
||||
import { heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, type Db } from "@paperclipai/db";
|
||||
import { issueRecoveryActionService } from "../issue-recovery-actions.js";
|
||||
import type { NativeRecoveryNoticeScope } from "../native-recovery-notice-notifications.js";
|
||||
import { nativeRunnerOwnershipNotHeldCondition } from "./native-runner-ownership.js";
|
||||
import { pendingNativeStatusEffectsCondition } from "./native-pending-effect-work.js";
|
||||
import { pendingNativeRunProjectionCondition } from "./native-run-projection.js";
|
||||
|
||||
function matchingNotice(runId: string | typeof nativeRunFinalizations.runId) {
|
||||
return and(
|
||||
eq(issueRecoveryActions.kind, "active_run_watchdog"),
|
||||
inArray(issueRecoveryActions.status, ["active", "escalated"]),
|
||||
sql`${issueRecoveryActions.evidence}->>'runId' = ${runId}::text`,
|
||||
or(sql`${issueRecoveryActions.wakePolicy}->>'kind' = 'resume_native_run'`,
|
||||
eq(issueRecoveryActions.cause, "native_workspace_finalization_owner_unverified")),
|
||||
);
|
||||
}
|
||||
|
||||
export function pendingNativeRecoveryNotices(db: Db, scope?: NativeRecoveryNoticeScope, after?: string) {
|
||||
return db.select({ id: nativeRunFinalizations.runId, companyId: nativeRunFinalizations.companyId,
|
||||
issueId: nativeRunFinalizations.issueId })
|
||||
.from(nativeRunFinalizations)
|
||||
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, nativeRunFinalizations.runId),
|
||||
eq(heartbeatRuns.companyId, nativeRunFinalizations.companyId), eq(heartbeatRuns.nativeIssueId, nativeRunFinalizations.issueId)))
|
||||
.innerJoin(issues, and(eq(issues.id, nativeRunFinalizations.issueId), eq(issues.companyId, nativeRunFinalizations.companyId)))
|
||||
.where(and(eq(nativeRunFinalizations.phase, "committed"), isNotNull(nativeRunFinalizations.resultId),
|
||||
eq(heartbeatRuns.runtimeMode, "native"), nativeRunnerOwnershipNotHeldCondition(),
|
||||
not(pendingNativeStatusEffectsCondition(db)), not(pendingNativeRunProjectionCondition(db)),
|
||||
sql`exists (${db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and(
|
||||
eq(issueRecoveryActions.companyId, nativeRunFinalizations.companyId),
|
||||
eq(issueRecoveryActions.sourceIssueId, nativeRunFinalizations.issueId), matchingNotice(nativeRunFinalizations.runId)))})`,
|
||||
scope ? eq(nativeRunFinalizations.companyId, scope.companyId) : undefined,
|
||||
scope?.issueId ? eq(nativeRunFinalizations.issueId, scope.issueId) : undefined,
|
||||
scope?.runId ? eq(nativeRunFinalizations.runId, scope.runId) : undefined,
|
||||
after ? gt(nativeRunFinalizations.runId, after) : undefined,
|
||||
)).orderBy(asc(nativeRunFinalizations.runId)).limit(100);
|
||||
}
|
||||
|
||||
|
||||
|
||||
export async function resolveCommittedNativeRecoveryNotices(db: Db,
|
||||
scope: { companyId: string; issueId: string; runId: string }, onlyWhenSettled = false) {
|
||||
const [pending] = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and(
|
||||
eq(issueRecoveryActions.companyId, scope.companyId), eq(issueRecoveryActions.sourceIssueId, scope.issueId),
|
||||
matchingNotice(scope.runId))).limit(1);
|
||||
if (!pending) return;
|
||||
return db.transaction(async transaction => {
|
||||
const tx = transaction as unknown as Db;
|
||||
|
||||
|
||||
const [coordinator] = await tx.select({ phase: nativeRunFinalizations.phase, resultId: nativeRunFinalizations.resultId })
|
||||
.from(nativeRunFinalizations).where(and(eq(nativeRunFinalizations.companyId, scope.companyId),
|
||||
eq(nativeRunFinalizations.issueId, scope.issueId), eq(nativeRunFinalizations.runId, scope.runId))).for("update");
|
||||
if (!coordinator || coordinator.phase !== "committed" || !coordinator.resultId) return;
|
||||
const [issue] = await tx.select({ id: issues.id }).from(issues).where(and(
|
||||
eq(issues.companyId, scope.companyId), eq(issues.id, scope.issueId))).for("update");
|
||||
if (!issue) return;
|
||||
const [run] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.id, scope.runId), eq(heartbeatRuns.companyId, scope.companyId),
|
||||
eq(heartbeatRuns.nativeIssueId, scope.issueId), eq(heartbeatRuns.runtimeMode, "native"),
|
||||
nativeRunnerOwnershipNotHeldCondition())).for("update");
|
||||
if (!run) return;
|
||||
if (onlyWhenSettled && !(await pendingNativeRecoveryNotices(tx, scope)).length) return;
|
||||
const actions = await tx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and(
|
||||
eq(issueRecoveryActions.companyId, scope.companyId), eq(issueRecoveryActions.sourceIssueId, scope.issueId),
|
||||
matchingNotice(scope.runId))).for("update");
|
||||
for (const action of actions) await issueRecoveryActionService(tx).resolveActiveForIssue({
|
||||
companyId: scope.companyId, sourceIssueId: scope.issueId, actionId: action.id,
|
||||
status: "resolved", outcome: "restored",
|
||||
resolutionNote: "The accepted native result and workspace finalization committed successfully; this run needs no further finalization retry.",
|
||||
});
|
||||
});
|
||||
}
|
||||
@@ -1,3 +1,5 @@
|
||||
import { notifyNativeRecoveryNotices } from "../native-recovery-notice-notifications.js";
|
||||
import { resolveCommittedNativeRecoveryNotices } from "./native-recovery-notices.js";
|
||||
import { notifyNativeRunProjection } from "../native-run-projection-notifications.js";
|
||||
import { projectCommittedRun, recoveredExecutionFailureMetadata } from "./native-run-projection.js";
|
||||
import { notifyNativeResultCommit } from "../native-result-commit-notifications.js";
|
||||
@@ -37,7 +39,6 @@ import {
|
||||
heartbeatRuns,
|
||||
heartbeatRunEvents,
|
||||
issueComments,
|
||||
issueRecoveryActions,
|
||||
issueThreadInteractions,
|
||||
issues,
|
||||
nativeRunFinalizations,
|
||||
@@ -594,21 +595,6 @@ export async function recordNativeFinalizationFailure(input: {
|
||||
});
|
||||
}
|
||||
|
||||
async function resolveCommittedFinalizationRecovery(db: Db, run: Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId">, issueId: string) {
|
||||
const actions = await db.select().from(issueRecoveryActions).where(and(
|
||||
eq(issueRecoveryActions.companyId, run.companyId), eq(issueRecoveryActions.sourceIssueId, issueId),
|
||||
eq(issueRecoveryActions.kind, "active_run_watchdog"), inArray(issueRecoveryActions.status, ["active", "escalated"]),
|
||||
sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
|
||||
or(
|
||||
sql`${issueRecoveryActions.wakePolicy}->>'kind' = 'resume_native_run'`,
|
||||
eq(issueRecoveryActions.cause, "native_workspace_finalization_owner_unverified"),
|
||||
),
|
||||
));
|
||||
for (const action of actions) await issueRecoveryActionService(db).resolveActiveForIssue({
|
||||
companyId: run.companyId, sourceIssueId: issueId, actionId: action.id,
|
||||
status: "resolved", outcome: "restored", resolutionNote: "The accepted native result and workspace finalization committed successfully; this run needs no further finalization retry.",
|
||||
});
|
||||
}
|
||||
|
||||
async function materializeNativeReviewResponse(db: Db, runId: string, dbOrTx: Db = db) {
|
||||
const [run] = await dbOrTx
|
||||
@@ -754,6 +740,27 @@ export async function recoverCommittedNativeChatResponse(
|
||||
try {
|
||||
materialized = await db.transaction(async (transaction) => {
|
||||
const tx = transaction as unknown as Db;
|
||||
|
||||
|
||||
const [coordinator] = await tx
|
||||
.select()
|
||||
.from(nativeRunFinalizations)
|
||||
.where(
|
||||
and(
|
||||
eq(nativeRunFinalizations.runId, input.runId),
|
||||
eq(nativeRunFinalizations.companyId, input.companyId),
|
||||
eq(nativeRunFinalizations.issueId, input.issueId),
|
||||
eq(nativeRunFinalizations.phase, "committed"),
|
||||
isNull(nativeRunFinalizations.leaseOwner),
|
||||
),
|
||||
)
|
||||
.for("update");
|
||||
if (
|
||||
!coordinator?.resultId ||
|
||||
!coordinator.decisionId ||
|
||||
!coordinator.assessmentId
|
||||
)
|
||||
return false;
|
||||
const [issue] = await tx
|
||||
.select()
|
||||
.from(issues)
|
||||
@@ -816,25 +823,6 @@ export async function recoverCommittedNativeChatResponse(
|
||||
record(run.contextSnapshot).skipIssueComment === true
|
||||
)
|
||||
return false;
|
||||
const [coordinator] = await tx
|
||||
.select()
|
||||
.from(nativeRunFinalizations)
|
||||
.where(
|
||||
and(
|
||||
eq(nativeRunFinalizations.runId, run.id),
|
||||
eq(nativeRunFinalizations.companyId, input.companyId),
|
||||
eq(nativeRunFinalizations.issueId, input.issueId),
|
||||
eq(nativeRunFinalizations.phase, "committed"),
|
||||
isNull(nativeRunFinalizations.leaseOwner),
|
||||
),
|
||||
)
|
||||
.for("update");
|
||||
if (
|
||||
!coordinator?.resultId ||
|
||||
!coordinator.decisionId ||
|
||||
!coordinator.assessmentId
|
||||
)
|
||||
return false;
|
||||
if (input.onlyWhenDue) {
|
||||
if (coordinator.nextAttemptAt && coordinator.nextAttemptAt.getTime() > Date.now()) return false;
|
||||
const [pendingEffect] = await tx.select({ id: statusDecisionEffects.id }).from(statusDecisionEffects).where(and(
|
||||
@@ -1100,6 +1088,7 @@ export async function finalizeNativeRun(input: {
|
||||
skipNativeResponsePresentation?: boolean;
|
||||
skipNativeReviewResponse?: boolean;
|
||||
skipCommittedRunProjection?: boolean;
|
||||
skipNativeRecoveryNotices?: boolean;
|
||||
/** Workspace-only replay must not consume the provider recovery budget. */
|
||||
preserveProviderAttempt?: boolean;
|
||||
failpoint?: NativeStatusCommitFailpoint;
|
||||
@@ -1165,7 +1154,7 @@ export async function finalizeNativeRun(input: {
|
||||
logger.warn({ err, runId: admittedRun.id }, "Slack conversation settlement deferred to reconciliation");
|
||||
});
|
||||
}
|
||||
await resolveCommittedFinalizationRecovery(input.db, admittedRun, coordinator.issueId);
|
||||
if (!input.skipNativeRecoveryNotices) await resolveCommittedNativeRecoveryNotices(input.db, { companyId: admittedRun.companyId, runId: admittedRun.id, issueId: coordinator.issueId });
|
||||
if (input.projectRunStatus) {
|
||||
const [projectedRun] = await input.db.select({
|
||||
id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId,
|
||||
@@ -1645,6 +1634,7 @@ export async function finalizeNativeRun(input: {
|
||||
.from(heartbeatRuns).where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId)))
|
||||
.for("update").limit(1);
|
||||
if (!before) return null;
|
||||
await notifyNativeRecoveryNotices(tx, { companyId: run.companyId, runId: run.id });
|
||||
await notifyNativeRunProjection(tx, { companyId: run.companyId, runId: run.id });
|
||||
await notifyRetainedNativeSessionCleanup(tx, { companyId: run.companyId, runId: run.id });
|
||||
if (input.projectRunStatus) await notifyInstructionCollection(tx, { companyId: run.companyId, runId: run.id });
|
||||
@@ -1681,7 +1671,7 @@ export async function finalizeNativeRun(input: {
|
||||
logger.warn({ err, runId: run.id }, "Slack conversation settlement deferred to reconciliation");
|
||||
});
|
||||
}
|
||||
if (finalizationPhase === "committed") await resolveCommittedFinalizationRecovery(input.db, run, coordinator.issueId);
|
||||
if (finalizationPhase === "committed" && !input.skipNativeRecoveryNotices) await resolveCommittedNativeRecoveryNotices(input.db, { companyId: run.companyId, runId: run.id, issueId: coordinator.issueId });
|
||||
if (input.projectRunStatus && updatedRun) await reconcileMuseNativeTerminalRun(input.db, updatedRun);
|
||||
return {
|
||||
...coordinator,
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeRecoveryNotices } from "../native-recovery-notice-notifications.js";
|
||||
import { pendingNativeStatusEffectsCondition } from "./native-pending-effect-work.js";
|
||||
import { notifyNativeRunProjection } from "../native-run-projection-notifications.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
@@ -126,6 +127,7 @@ export async function projectCommittedRun(input: {
|
||||
nativePhase: heartbeatRuns.nativePhase, finishedAt: heartbeatRuns.finishedAt })
|
||||
.from(heartbeatRuns).where(needsRepair).for("update").limit(1);
|
||||
if (!before) return null;
|
||||
await notifyNativeRecoveryNotices(tx, { companyId: input.run.companyId, runId: input.run.id });
|
||||
await notifyNativeRunProjection(tx, { companyId: input.run.companyId, runId: input.run.id });
|
||||
await notifyRetainedNativeSessionCleanup(tx, { companyId: input.run.companyId, runId: input.run.id });
|
||||
await notifyInstructionCollection(tx, { companyId: input.run.companyId, runId: input.run.id });
|
||||
|
||||
@@ -6,15 +6,18 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } fro
|
||||
import { NativeRunCoordinatorStore } from "./native-runtime/native-run-coordinator-store.js";
|
||||
import { getConversationOwnershipBlocker } from "./conversation-continuation.js";
|
||||
import { isExternalChatPresentationContext } from "./heartbeat-run-summary.js";
|
||||
import { finalizeNativeRun, repairCommittedNativeChatResponse } from "./native-runtime/native-run-finalizer.js";
|
||||
import { finalizeNativeRun, recoverCommittedNativeChatResponse, repairCommittedNativeChatResponse } from "./native-runtime/native-run-finalizer.js";
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
|
||||
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_PRESENTATION_READ_TEST_DATABASE_URL");
|
||||
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
describe.skipIf(!support.supported)("presentation and ownership PostgreSQL read projections", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
beforeAll(async () => {
|
||||
database = await startEmbeddedPostgresTestDatabase("presentation-ownership-read-");
|
||||
db = createDb(database.connectionString);
|
||||
if (!external) database = await startEmbeddedPostgresTestDatabase("presentation-ownership-read-");
|
||||
db = createDb(external ?? database!.connectionString);
|
||||
}, 60_000);
|
||||
afterAll(async () => { await db?.$client.end(); await database?.cleanup(); });
|
||||
|
||||
@@ -113,8 +116,8 @@ describe.skipIf(!support.supported)("presentation and ownership PostgreSQL read
|
||||
async function compareRepair(f: Awaited<ReturnType<typeof fixture>>) {
|
||||
const baseline = captureReads(true), projected = captureReads();
|
||||
const input = { companyId: f.companyId, issueId: f.issueId, runId: f.run.id };
|
||||
expect(await repairCommittedNativeChatResponse(projected.recorded, input))
|
||||
.toBe(await repairCommittedNativeChatResponse(baseline.recorded, input));
|
||||
expect(await recoverCommittedNativeChatResponse(projected.recorded, { ...input, onlyWhenDue: true }))
|
||||
.toBe(await recoverCommittedNativeChatResponse(baseline.recorded, { ...input, onlyWhenDue: true }));
|
||||
expect(projected.rows.flat()).toHaveLength(1);
|
||||
expect(Buffer.byteLength(JSON.stringify(projected.rows))).toBeLessThan(4096);
|
||||
expect(Buffer.byteLength(JSON.stringify(baseline.rows))).toBeGreaterThan(128 * 1024);
|
||||
@@ -123,7 +126,7 @@ describe.skipIf(!support.supported)("presentation and ownership PostgreSQL read
|
||||
}
|
||||
|
||||
it("preserves selected answers without downloading prompts, profiles, or excerpts", async () => {
|
||||
const f = await fixture({ runtimeMode: "native", status: "succeeded", resultJson: {
|
||||
const f = await repairFixture({ resultJson: {
|
||||
presentationDecision: { commentId: randomUUID() }, unknownFutureReceipt: { preserved: true } } });
|
||||
const row = await compareRepair(f);
|
||||
expect(row.resultJson).toEqual(f.run.resultJson);
|
||||
@@ -167,6 +170,18 @@ describe.skipIf(!support.supported)("presentation and ownership PostgreSQL read
|
||||
return { ...f, before, reply };
|
||||
}
|
||||
|
||||
async function repairFixture(patch: Partial<typeof heartbeatRuns.$inferInsert>) {
|
||||
|
||||
|
||||
const f = await unpublishedAcceptedReply();
|
||||
const [run] = await db.update(heartbeatRuns).set(patch).where(eq(heartbeatRuns.id, f.run.id)).returning();
|
||||
|
||||
|
||||
await db.update(nativeRunFinalizations).set({ nextAttemptAt: new Date(Date.now() + 60_000) })
|
||||
.where(eq(nativeRunFinalizations.runId, run.id));
|
||||
return { ...f, run };
|
||||
}
|
||||
|
||||
it("publishes the same accepted reply and decision from equivalent full-row and projected starting states", async () => {
|
||||
const outcomes: unknown[] = [];
|
||||
for (const fullRows of [true, false]) {
|
||||
@@ -200,7 +215,7 @@ describe.skipIf(!support.supported)("presentation and ownership PostgreSQL read
|
||||
{ source: "chat:fixture", skipIssueComment: true, paperclipWake: { externalInteractionContinuation: "true" } },
|
||||
{ source: "internal", paperclipWake: { externalInteractionContinuation: true } },
|
||||
].map(context => ({ context })))("preserves typed publication flags for $context", async ({ context }) => {
|
||||
const f = await fixture({ runtimeMode: "native", status: "succeeded", contextSnapshot: context as Record<string, unknown> | null });
|
||||
const f = await repairFixture({ contextSnapshot: context as Record<string, unknown> | null });
|
||||
const row = await compareRepair(f);
|
||||
expect(isExternalChatPresentationContext(row.contextSnapshot)).toBe(isExternalChatPresentationContext(context));
|
||||
const original = context && typeof context === "object" && !Array.isArray(context) ? context : {};
|
||||
@@ -208,7 +223,7 @@ describe.skipIf(!support.supported)("presentation and ownership PostgreSQL read
|
||||
});
|
||||
|
||||
it.each([null, ["stdout", { receipt: true }], "scalar", 42].map(result => ({ result })))("preserves non-object result evidence $result", async ({ result }) => {
|
||||
const f = await fixture({ runtimeMode: "native", status: "succeeded", resultJson: result as Record<string, unknown> | null });
|
||||
const f = await repairFixture({ resultJson: result as Record<string, unknown> | null });
|
||||
expect((await compareRepair(f)).resultJson).toEqual(result);
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user