fix(server): schedule accepted native response recovery (#17696)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs store accepted results and their finalization decisions.
> - A completed run can still owe its public answer after an
interruption.
> - Periodic recovery scans for that answer even when no new work
exists.
> - This pull request wakes recovery from committed changes and saved
retry deadlines.
> - The existing owner still verifies the result, source, and current
publication authority.
> - Settled or dormant response work no longer needs its own periodic
scan.

## Linked Issues or Issue Description

Refs #17690 (merged). This is slice 48 of the sleep migration stack, now
based on master. Related: #17661 handles GitHub terminal publication;
this change handles native accepted-response presentation.
Governed-review presentation remains a separate duty.

**What happened?**

Native finalization recovery repeatedly checks already accepted results
for a missing answer. Removing that check without complete notifications
would lose responses when evidence arrives late, an agent resumes, or
admission settles after execution.

**Expected behavior**

Recover the answer from its saved result when a supported change makes
it eligible. Retain real retry deadlines. Stop querying when the answer
is selected or cannot currently be published. Reconstruct work after
restart.

**Steps to reproduce**

1. Commit a native accepted result with no selected response comment.
2. Run finalization recovery while its source is unavailable or locked.
3. Restore that source or release the lock. The answer must appear once
without repeating provider execution.

## What Changed

- Honor recorded non-create decisions and skip-comment policy in both
discovery and recovery. Apply durable conversation/result/continuation
guards before recovery locks. Preserve genuine unanswered and legacy
recovery.
- Scope routine response wakes to the affected issue/run. Coalesced
input names its active run; pending-effect settlement names its issue.
Sixteen existing global authority/configuration hooks use one explicit
company invalidation boundary. No new transaction wrappers or tables are
added.
- Add an accepted-response worker using the existing delivery
coordinator and scoped deadline helper. Preserve saved retry deadlines
and pending-effect fences under the owner's lock.
- Share the finalization notification boundary. Wake on finalization
settlement, exact accepted-result/final-message evidence, agent
availability, retry admission, and existing chat authority changes.
- Keep lifecycle policy in services. A narrow pre-write status hook lets
lifecycle retain ownership of its transaction and status transitions.
- Register the worker during enabled maintenance startup. Remove its
authority subscription on shutdown. Periodic replay skips this duty on
both committed and fresh-commit paths; ordinary finalization and
explicit recovery retain their behavior.
- Add a partial index and expression statistics for unpresented native
runs. Migration 0412 supports concurrent replay and adds no queue table.
- Add real PostgreSQL and lifecycle coverage. Preserve the
implementation and producer audit in the plan document.

## Verification

- Additional real Photon callback acceptance on the published production
code disproves the security warning: ten active assertions cause zero
response-recovery hints/SELECTs, while three concurrent assertions
during an actual attention-to-active recovery cause exactly one
wake/SELECT. The existing locked status guard already enforces this
boundary. No production change was needed; the resolved finding now has
a successful current-head security check. The regression will be
retained with the next slice.
- Eligibility repair source `892f36e5f66ba1af123034936a7160c64c3ab883`
passes **1489 distinct server cases**: 1139 in six complete
recovery/finalizer/monitor/summary/workspace suites (81.02 seconds),
plus 350 selected chat cases (297.68s). Another 1463 chat cases are
outside the pattern; the complete 1813-case file is not claimed. All
eleven new regressions are included in that count. Five dedicated fresh
PostgreSQL 17 databases migrate through unchanged 0412, with no
fixture-source adaptations. Database build, server types/build,
boundaries and full-SHA migration ordering pass. All seven test source
hashes match.
- The exact published baseline first fails eight behavioral assertions
while the three genuine-answer cases pass. The same corrected test
source passes all eleven after repair: recorded none/reuse remain
suppressed; five ineligible history families take zero recovery row
locks across repeated issue hints; missing/legacy/create-null answers
still publish once; fresh periodic finalization defers presentation to
its worker. Earlier fixture setup and variable/compiler failures remain
explained in the audit, rather than erased by later success.
- Repair source `2fc8b0614aaf9ac7337045de4ced7325f95edc45` passes **1821
server cases**: 1496 in 13 complete producer/lifecycle/authority suites
(110.63 seconds), plus 325 selected chat cases (278.99 seconds).
Coverage includes all native-session cases, queue admission, status
decisions, membership/board access, the complete GitHub workflow,
accepted-response lifecycle and real authority restoration. The other
1477 chat cases are outside this selection; the complete 1802-case file
is not claimed for this repair run.
- Actual PostgreSQL regressions first reproduce a coalesced-input hint
missing its run scope and two routine hint paths producing 12
response-history SELECTs each after three notifications. The same
assertions pass with run-scoped coalescing and zero response-history
SELECTs for unscoped routine hints. Actual membership restoration still
publishes the waiting response once. Assertion time limits are
unchanged; this measures real driver SQL and persisted producer work.
- Eleven dedicated fresh PostgreSQL 17 databases migrate through 0412.
One embedded-only wake-queue fixture import temporarily uses its own
external database and restores byte for byte. All 14 selected source
files match pre-run hashes. This qualifies external-database behavior
rather than reproducing embedded CI locally.
- Master advanced independently with #18126's Pi archive retry fix.
Normal merge `2a535bc61259ef99387369e8b32498f774e5ba2c` preserves both
complete binary patches. All accepted server/database/UI/test/CLI trees
stay identical; the three independent runner installer/manifest files
are retained. All 13 installer tests pass, including stalled header/body
cancellation. Combined server types/build, boundaries and full-SHA
migration order pass. A subsequent normal merge of #18128 at
`e8b78cce5bb557b086bee9ca930eeaab1d09b244` retains its three test-only
file changes with both complete patches preserved. The changed HTTPS
backfill case passes on a fresh PostgreSQL fixture (1 selected case; 178
outside the pattern), and combined types/build, boundaries and migration
order pass again. All 14 repair source hashes remain unchanged. The
independent browser fixture changes are preserved; no local browser run
is claimed.
- The earlier source `9feec170ef8174011fc3c5d3d6d76f0a0f39fea0` passed
the 4171-case full baseline before this scope repair. That scope is
preserved in the audit rather than represented as a current full rerun.
All 0412 database bytes remain unchanged: original concurrent
index/statistics/ANALYZE, prior executable history, journal entries and
retained snapshots. Five snapshots 0408 through 0412 remain. The
populated 0411-to0412 upgrade preserved 410 prior receipts and 38
companies, appended one receipt and none on replay; index/statistic
checks passed.
- The earlier P2 security failure, Greptile's former conditional
dependency on #17702, and the subsequent actual Ponytail eligibility
findings are recorded in
`doc/plans/2026-10-25-native-response-presentation-sleep-audit.md`. This
PR now includes the issue-scoped pending-effect notification fix itself,
so it is independently correct before #17702. Fresh published-head CI,
Greptile 5/5 and a clean security scan remain required. No
deployed-sleep, external-wake or fleet-savings claim is made.


## Risks

- The writer contract is process-local. Independent SQL writers and
other replicas are not covered by these notifications.
- Unpresented backlog reconstruction still has memory and time cost
proportional to candidate count. Pages contain at most 100 rows;
execution uses the shared fair batch. Selected history is excluded.
Health-only signals do not trigger reconstruction.
- Publication policy remains with its current owner. Digest, company/run
binding, source authorization, selected-answer suppression, and
ownership fences remain required.
- The rest of native recovery and governed-review presentation still run
separately. This PR does not relax Cloud sleep eligibility or add
external wake coverage.
- The stack parent has merged and its actual ancestry is reconciled.
This publication still needs fresh current-head CI and review.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell
execution, and PostgreSQL tests. The runtime does not expose a more
specific model identifier or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
authored and GitHub committed 2026-10-29 02:31:20 -05:00
1 parent e8b78cce5b
commit ec3b558277
53 files changed
+1960 -108

No files matched your search

@@ -2715,3 +2715,114 @@ and all final-head PR gates remain open. No PR has been merged by this stack wor
Final slice 47 source passes workspace typecheck/build and 158 worker/startup
cases, including exact near-term concurrent deadlines. Remote publication and
acceptance are the next gates; local success does not satisfy the full stack.
## Slice 47 published; slice 48 response-owner audit
Slice 47 is draft [#17690](https://github.com/paperclipai/paperclip/pull/17690),
head `56c3601ea605b333141e86e0a810ed790fb6ce48`, based on #17687. Its final local
checks passed; remote CI and Greptile are running. No merge or remote acceptance
is claimed yet.
Slice 48 starts accepted-response presentation recovery, excluding governed-review
presentation. Its current helper conflates dormant/settled work and transient
failures as false; resolve that owner contract before removing its backstop.
The exact null-lease, digest, selected-answer, and publication-authority fences
must remain intact. The existing external-chat baseline passed 33 selected cases;
1,551 others were not selected. See the
[response-presentation audit](2026-10-25-native-response-presentation-sleep-audit.md).
The old action-required check on #17576 is Superagent Security Scan. Its existing
voice-review findings and the older E2E failures remain outstanding. Work on the
rest of the original stack continues; none of these gates are waived.
Slice 48's first owner-contract change now passes the same 33 real-PostgreSQL
response cases plus server typecheck. The strict recovery entry point propagates
operational errors; existing callers keep their logging/boolean contract. No
response worker has been wired and no response polling is removed yet.
The Superagent action-required gate on #17576 was traced to review 5643793631,
inline finding 4371283330: repeated connection/health writes reload the full live
voice-session inventory. This is the already-recorded bounded-discovery issue,
not a missing credential or permission. It still needs a code/test repair and
fresh review; the gate must not be bypassed.
## Slice 47 accepted
[#17690](https://github.com/paperclipai/paperclip/pull/17690) at exact head
`56c3601ea605b333141e86e0a810ed790fb6ce48` has all 53 checks passing, Greptile 5/5
with no actionable findings or review threads, and a mergeable parent. Its PR body
records those gates. It remains draft and unmerged. Evidence is retained in
`/tmp/sleep47-checks-accepted.json`, `/tmp/sleep47-final-gates.json`,
`/tmp/sleep47-review-accepted.json`, and `/tmp/sleep47-review-threads.json`.
Slice 48 remains local and incomplete: owner error separation and the 33-case
baseline pass; candidate discovery, producer coverage, lifecycle wiring, and
periodic removal remain. Older unresolved stack gates and every other original
recurring-server duty still require completion. The active goal is not complete.
## Slice 48 local worker progress
Accepted-response recovery now has a local scoped worker, locked retry/effect
fences, milestone-topic notifications, and live authority registration invalidation.
The final selected real-PostgreSQL/lifecycle run passes 45 cases, and server types
pass. An elapsed-deadline race was reproduced before repair. A generated concurrent
partial index plus expression statistics avoids selected-history scans in the
10,000-row fixture; fresh migration and replay pass. See the
[response audit](2026-10-25-native-response-presentation-sleep-audit.md) for exact
scope and preserved failures.
The worker is not wired into startup and periodic response recovery remains.
Producer coverage, remaining variants/failure tests, large dormant-backlog cost,
full local/remote acceptance, and PR publication are still required. Master was
refreshed to `7703220d38`; the complete stack has not yet been reconciled with it.
No merge, complete migration, whole-instance sleep, or full-goal completion is
claimed.
## Slice 48 producer and lifecycle checkpoint
Response recovery now shares the finalization notification boundary, receives
run-scoped late evidence, and wakes when a paused agent resumes or retry admission
settles. The resume gap was reproduced before fixing it at lifecycle's transaction
boundary. The existing lifecycle suite, both response variants' supporting suites,
and the combined acceptance selection cover 175 distinct passing cases across
runs; server types pass. A real source-row lock verifies retry across drain.
Earlier harness/test failures and the 10,000-unpresented-row discovery measurements
are preserved in the response audit. No startup wiring, periodic removal, PR
publication, remote acceptance, or whole-instance sleep is claimed yet.
## Slice 48 startup integration, 2026-10-26
The response worker now registers under enabled maintenance and releases its
subscription during shutdown. Periodic recovery skips both its direct repair and
already-committed finalizer replay; fresh finalization and explicit recovery stay
active. The startup suite passes 66 cases, and two database regressions confirm
periodic calls defer to the new worker. Workspace types pass. Full local tests
remain in progress with embedded-database initialization failures; build and
remote checks remain pending. The stack goal is still incomplete.
## Slice 48 review repair, 2026-10-26
Draft #17696 is open and attached. Review identified company-wide response reloads
from routine run-status notifications; three real database regressions reproduced
it. Status/claim/release wakes now retain task or exact-run scope while task
lineage remains task-wide. All 34 heartbeat lifecycle and 63 response/authority/
admission tests pass, as do server types and the workspace build. The full local
suite was stopped after repeated embedded PostgreSQL startup failures and remains
unproven. Remote failures at `79bf36cbf8` include an early startup-settlement test
assertion (repaired), a workspace staging timeout, and an E2E runner shutdown.
See the response audit for logs. Fresh-head CI/review, earlier stack findings,
master reconciliation, and all remaining original duties stay open. Nothing merged.
## Slice 48 full-history acceptance repair
Fresh review of `c4beddad1e` is 5/5 with its scope finding resolved. Remote CI
exposed a second early startup assertion. Full local chat verification finished
1,595 passed / seven failed; all seven were new worker tests observing a transient
null deadline or unfinished startup batch. Their settlement predicate now includes
zero active work. All 18 affected cases pass against the full suite's retained
history. Fresh remote acceptance remains pending; these are recorded test repairs,
not a waiver of failed gates or a change to production batch behavior.
@@ -160,6 +160,12 @@ deadline repair (`/tmp/sleep47-deadlines-typecheck.log`,
`/tmp/sleep47-deadlines-build.log`). The final worker/startup run now passes 158
cases, giving 683 distinct passing cases with the earlier unaffected consumers.
## Remote acceptance
Published as [#17690](https://github.com/paperclipai/paperclip/pull/17690).
Exact head `56c3601ea605b333141e86e0a810ed790fb6ce48`: all 53 checks pass;
Greptile 5/5 with no actionable findings or review threads; mergeable parent.
The PR remains draft and unmerged. This accepts slice 47 only, not the full stack.
## Current migration411 integration and complete acceptance
@@ -0,0 +1,665 @@
# Native accepted-response presentation recovery
Status: worker, producers, and startup/periodic integration implemented; full local and remote acceptance pending.
Branch: `codex/sleep-48-native-response-presentation`.
Parent: [#17690](https://github.com/paperclipai/paperclip/pull/17690),
`56c3601ea605b333141e86e0a810ed790fb6ce48`.
## Reason for work
A native run can have an accepted, committed result but no saved public reply.
The finalization reconciler calls `repairCommittedNativeChatResponse` before
skipping superseded status decisions. A later task decision must not erase an
already accepted, still-authorized response. This work neither re-executes the
provider nor changes issue disposition. It is separate from governed-review
presentation, evidence reassessment, and result/workspace settlement.
The current helper covers three variants of the same accepted-response duty:
a saved same-agent/monitor continuation summary, an internal task's exact final
assistant message, and an authorized external response-wake summary. Governed
review responses are explicitly excluded and retain their own stricter owner.
Do not merge the review response migration into this slice merely because both
helpers live in the finalizer.
## Existing authority and mutation boundary
`repairCommittedNativeChatResponse` locks issue, run, and coordinator in that
order. It requires native runtime; succeeded, failed, or timed_out with finishedAt;
a committed coordinator with result/assessment/decision and a null leaseOwner.
It refuses to replace a selected comment, including a deleted selected answer.
It checks the accepted result's digest using either the coordinator format or
the older authenticated control-plane format, retaining its bounded journal
source lookup. It excludes externalChatReviewPresentation decisions.
A lease that expired but still has an owner is not permission to publish: the
helper requires actual release. Do not reuse the pending-effects lease rule.
The outer periodic candidate query additionally respects saved nextAttemptAt.
A future worker must preserve that fence where applicable and wake on release.
Internal completion needs an exact final assistant event for the accepted turn,
current internal-write authorization, and permission to create a new completion
comment. Continuation summaries check provider-owned interactions. External
response-wake summaries require the live registered chat authority, exact durable
source provenance, and current presentation authorization. The chat owner validates
source edits/deletion, membership, endpoint state/generation, consumed failure
lanes, and ambiguous delivery. None of this policy belongs in a generic scheduler.
The comment and presentationDecision are written in the same transaction, through
the existing comment/publication owner. Notification follows durable commit.
Recovery must not revive private provider narration, duplicate a target, publish
under another source, or reinterpret the saved result using new task settings.
## Contract gap to resolve before retiring polling
The helper currently returns false for several distinct outcomes: already settled,
structurally ineligible, authority unavailable/revoked, and transient SQL/lock or
materialization failure. It logs most errors then returns false. A worker cannot
use that boolean as a pending-work predicate: dropping false loses retryable
responses, while retrying every false rescans ineligible history indefinitely.
Keep the current boolean caller contract where needed, but expose the smallest
owner-level result/error contract that distinguishes dormant history from concrete
retry debt. Do not translate all authorization failures into retry forever. The
registered authority lives in durable-chat-wakeup.ts and is installed by
chat-channels/failed-run-retry.ts. Missing registration and later registration need
an explicit lifecycle wake path, separate from committed database mutations.
Audit and test the candidate predicate before introducing a worker. Startup scans
must filter selected answers, irrelevant result kinds, review-owned responses,
and other passive history. Full authorization stays in the locked owner; SQL
prefiltering is not a new publication authorization mechanism. Measure actual query
plans with substantial completed history before claiming bounded discovery.
## Producer audit still required
- Native coordinator decision commit, persisted-attention replay, and ownership
release; result/assessment/decision binding changes.
- Run terminalization and accepted-turn event persistence, including late journal
arrival and direct recovery writers. Avoid waking on every unrelated progress
event merely because it shares the journal.
- Presentation selection/materialization and existing-comment suppression. A
deleted selected answer remains settled rather than becoming repair debt.
- Issue conversation/context changes and skipIssueComment behavior.
- Exact chat source, action/admission, publication, endpoint generation/status,
principal/membership, continuation provenance, and ambiguous-delivery resolution.
Separate mutations that only revoke work from supported reactivation paths.
- Live chat-authority registration/replacement/shutdown; subscription ownership.
- Startup, first write during drain, retry during drain, resume, and shutdown.
Use existing durable records, scoped transaction signals, and the shared scheduler.
No new response queue table or permissive publication fallback is justified.
Independent SQL/multi-process writes remain outside the existing local signal
contract unless explicitly integrated; preserve the documented limitation.
## Baseline evidence
On the unchanged parent source, a fresh external PostgreSQL selection passed
33 accepted-response cases in chat-channels.integration.test.ts. The remaining
1,551 cases in that file were not selected, not claimed passed. The selection
covers duplicate/concurrent repair, source/authority changes, digest changes,
private disposition, cross-company/run binding, live heartbeat fences, the older
port-bound digest, GitHub internal prose/attachments, failure-lane replacement,
provider-dispatch reauthorization, selected/deleted answers, and contention.
Log: `/tmp/sleep48-chat-baseline.log`. Database:
`paperclip_sleep48_chat_baseline_20261025_test`.
The monitor-continuation and control-plane-port suites currently require embedded
PostgreSQL directly. Qualify a validated external-test path if needed before using
their results on this host; do not count skipped or initialization-failed suites.
No sleep or migrated-worker behavior is established by this baseline.
## Initial owner-contract change
`recoverCommittedNativeChatResponse` now exposes the existing transaction and
returns false for settled/ineligible/authorization-denied state, while propagating
operational failures. The existing `repairCommittedNativeChatResponse` wrapper
retains logging and false-on-error behavior for current callers. Authority checks,
lock order, target materialization, and commit signaling remain in the same owner.
The commit signal itself contains subscriber errors, so the wrapper does not hide
a previously propagated publication error.
The same 33-case real-PostgreSQL selection passes after this change, with added
assertions that the new entry point rejects under actual source-action lock
contention and returns false for the existing authorization/binding fences. The
server typecheck passes (`/tmp/sleep48-contract-typecheck.log`). Worker discovery,
producer notifications, startup wiring, and periodic removal are not implemented.
## Existing notification boundary to reuse
The current `notifyChatRunMilestones` boundary already covers 102 production call
sites and fans out lineage/authority changes to native progress, inbound notices,
confirmations, and GitHub work. Its scope is company plus optional issue because
source changes can authorize descendants. Audit its coverage for this duty before
adding caller wrappers. It already reaches status-decision settlement and run
projection, but raw coordinator-store/session release and accepted-turn journal
writers need individual inspection. Reuse is not proof that all producers exist.
## Local worker and locked scheduling fences
The local worker now discovers exact run/coordinator/result/decision bindings,
filters selected answers and structurally irrelevant results, and dispatches
through `recoverCommittedNativeChatResponse`. It does not own publication policy.
`onlyWhenDue` rechecks the saved retry and outstanding status effects while the
existing coordinator lock is held. Existing explicit callers retain their old
behavior. A future retry is retained even if it becomes due between the locked
check and the follow-up read; its next attempt is immediate rather than discarded.
Operational errors remain retryable through the shared coordinator. A structurally
eligible but unauthorized row becomes dormant until an eligibility mutation or
live authority registration invalidates its scope. Selected comments, including
deleted selections, remain excluded. No provider rerun or new work table is added.
The existing milestone notification boundary now includes the response topic.
Hints explicitly select company and issue IDs: structural TypeScript callers may
pass larger objects, so serializing the whole argument is not safe. The first
integration run exposed that mistake as circular-object serialization during
status materialization; it was corrected before the passing runs below.
Two narrow lifecycle capabilities support the new consumer:
- `registerScopedDeadlineWork.invalidate` marks reconstruction dirty before waking.
A plain wake is insufficient after a prior dormant result removed its entry.
Existing consumers keep the same registration and scheduling behavior.
- The committed-response authority owner exposes registration notifications.
Registering/replacing the actual chat service reactivates dormant responses.
The worker's `stop` removes this subscription; the delivery coordinator still
owns worker cancellation and awaiting in-flight work. Repeated unsubscription
cannot delete a later listener set. The authority remains scoped to the exact Db.
No startup registration or periodic skip is wired yet. The old recovery path is
still present. Full producer coverage, internal/continuation variants, recovery
failure/drain tests, and repeated large-scope mutation cost remain acceptance work.
Current discovery pages by run ID but reconstructs all matching pages in one
load; its resource bound under a large unresolved/dormant backlog needs proof
before activation. Do not describe the worker as a completed migration yet.
## Query-plan and migration evidence
The actual discovery query was captured from the driver. On a disposable database
with 10,000 additional selected runs, results, assessments, decisions, and
coordinators, it initially scanned completed history (22.656 ms). A partial
unpresented-run index alone still chose the wrong join order (16.532 ms).
Expression statistics for the selected-comment predicate allowed PostgreSQL to
start from the one unpresented run (0.091 ms).
Generated migration `0404_futuristic_earthquake.sql` creates the partial index
concurrently, adds idempotent expression statistics, and analyzes the table.
The existing top-level migration runner owns concurrent-index replay/repair.
Snapshot retention contains exactly 0396–0400. Fresh migration, journal replay,
direct SQL replay, ready/valid index flags, and statistics presence passed.
Rechecking the actual migration's objects (after removing experimental objects)
used the driver's real global, company, and issue-scoped queries. Global discovery
found one candidate in 0.078 ms; history-only company/issue scopes returned none
in 0.024/0.019 ms without sequential scans. These are query-plan fixtures, not
latency or fleet-sleep claims. Large unselected ineligible histories and repeated
configuration signals still need separate qualification.
Evidence: `/tmp/sleep48-query.json`, `/tmp/sleep48-query-plan.sql`,
`/tmp/sleep48-query-plan.log`, `/tmp/sleep48-query-statistics.log`,
`/tmp/sleep48-production-plan.log`, `/tmp/sleep48-migration-verify.log`.
## Current verification and preserved failures
The expanded final selection passed 45 cases in two files: 33 existing accepted-
response cases, ten new worker/owner cases, and two authority lifecycle cases.
The 1,551 unselected chat cases are not claimed passed. Server typecheck passed.
The worker cases establish startup recovery, selected-history quiet over former
poll periods, real chat-service authority restoration, outer commit/rollback,
first notification during drain, SQL-free drain waiting, resume, saved future
retry, pending/failed status-effect ordering, and an elapsed-deadline race.
The elapsed-deadline test failed before its fix: the worker returned null even
though the locked owner had deferred the response. It now retains the due entry
and the next attempt publishes exactly once. Log:
`/tmp/sleep48-elapsed-deadline-before.log`.
Earlier worker runs exposed the hint serialization defect (nine fixture failures),
then two test-observation mistakes. A timer created before fake-time installation
was not advanced by the fake clock; the test now installs time before registration.
A null next deadline can occur while a sweep is in flight; commit assertions now
wait for the persisted publication, zero tracked activity, and no next deadline
jointly. The original logs remain `/tmp/sleep48-worker-tests.log`,
`/tmp/sleep48-worker-tests-v2.log`, `/tmp/sleep48-drain-isolated.log`, and
`/tmp/sleep48-worker-tests-v3.log`. The corrected initial selection passed eleven
cases in `/tmp/sleep48-worker-tests-v4.log`; the final expanded result is
`/tmp/sleep48-acceptance-v2.log` with server types in
`/tmp/sleep48-acceptance-typecheck-v2.log`.
No slice 48 PR, full workspace acceptance, remote CI/review, or provider sleep
qualification is claimed by these local results. The earlier stack gates and
remaining recurring-server duties are unchanged.
## Producer integration and lifecycle evidence
The existing pending-effect notification boundary is now named
`notifyNativeFinalizationWork` and wakes both consumers. Its existing coordinator
claim/retry, effect settlement, persisted-attention replay, retained-session release,
and recovery callers retain transaction ownership. No new wrappers are threaded
through those callers. Response discovery now accepts exact run and agent scopes.
Both journal writers notify for accepted-result evidence and normalized completed
final assistant messages, after source/deduplication checks and before persistence.
Deltas, tool completions, and analysis messages do not activate response recovery.
The port's late accepted-result journal test and the internal final-message test
exercise these existing writers. Monitor continuation recovery is also exercised
through the worker, without another accepted result or provider execution.
Agent availability was a real missing producer: the recovery owner rejects a
paused agent, but resuming that agent did not invalidate dormant responses. A
real resume regression failed before the fix. Lifecycle now exposes a narrow
`beforeStatusChange` integration callback, invoked inside its existing transaction
before the compatibility status write. The service supplies response policy only
for a transition from non-invokable to invokable, scoped to that agent. Lifecycle
still owns status transitions and the transaction; its existing resource-event
callback retains its semantics. With no response worker registered, this adds only
the existing notification protocol, not an independent worker or provider call.
The complete lifecycle suite exercises existing consumers of the new contract.
Failed-chat retry admission also notifies response recovery when its source action
becomes processed. This can establish the ancestry proof after a very fast child
run has already finished. Five real admission scenarios verify that the response
consumer observes the committed action and the exact issue scope: commit, retry,
startup, stale claim, and rollback. An assignment-reactivation experiment instead
hit the existing prohibition on changing a channel-bound task's agent; that
unsupported fixture was removed without relaxing the guard.
Validation across these runs covers 175 distinct cases: 59 response/authority and
retry-admission cases, 58 lifecycle cases, 21 monitor-continuation cases, and 37
control-plane-port cases. Server typechecking passes. These are selected suites,
not a claim that the entire repository suite or remote PR checks passed.
- `/tmp/sleep48-producers-rerun.log`: port 37/37, continuation 20/20, server types.
- `/tmp/sleep48-producer-acceptance.log`: lifecycle 58/58, expanded continuation
21/21, response/authority 53 selected, server types.
- `/tmp/sleep48-acceptance-v5.log`: combined response/authority/admission 59 passed;
1,546 unrelated cases not selected. Types in
`/tmp/sleep48-acceptance-typecheck-v5.log`.
- Real action-row contention now verifies retry retention through drain, zero
passive SQL while paused, and eventual single publication after resume.
Failures remain recorded. The first external-port run had 36 passes and one
observer connection using the absent embedded database; the observer now uses the
validated external URL. The resume regression and invalid reassignment experiment
are in `/tmp/sleep48-eligibility-before.log`. A test-only observer initially omitted
its required retry interval and produced finite-deadline warnings; corrected in
`/tmp/sleep48-retry-producer-v2.log`. The combined v4 run exposed a global fault
injection targeting another retained fixture (13 transactions, not one). The v5
run uses an actual lock on the tested response's source instead of assuming global
candidate order; all 59 cases pass with that stronger fault boundary.
## Unpresented-backlog measurement
A disposable clone of the earlier plan fixture has 10,008 structurally eligible,
unpresented candidates. The actual worker loader took 2,980 ms and 101 queries for
global reconstruction; 10,001 candidates in one company took 2,828 ms; the exact
run took 1.73 ms and one query. Agent reconstruction for that company's 10,001
candidates took 2,826 ms. Serialized entries occupy about 2.05 MB. The drain gate
stopped discovery after two pages. Evidence:
`/tmp/sleep48-unpresented-backlog.mts` and `.log`.
This measures discovery, not authorization or publication throughput: the cloned
rows do not establish valid independent publication authority. Each SQL page is
bounded to 100 rows, but total reconstruction memory/time still scales with the
unpresented backlog. Ordinary finalization and journal notifications are run-scoped;
agent resume is agent-scoped; actual authority/configuration changes may reconstruct
a larger scope. Health-only configuration signals do not invalidate this worker.
Do not describe total reconstruction as constant-cost or claim fleet qualification.
Startup/shutdown integration, removal of this duty's periodic call, final producer
coverage review, full PR verification, and publication remain outstanding. The old
backstop is still present. Master was refreshed to `6bdcb70244`; the full stack
still needs reconciliation and its earlier failing/review gates remain open.
## Startup integration, 2026-10-26
The heartbeat facade now registers this worker with the existing task-drain gate.
Enabled maintenance awaits reconstruction during startup. Shutdown and listener
close release its live-authority subscription; coordinator shutdown retains
ownership of cancellation and in-flight work.
The periodic orphan-recovery caller now passes `skipNativeResponsePresentation`.
The reconciler skips its direct repair call and passes the same option into
canonical finalizer replay. The finalizer skips only its already-committed response
repair in that mode. Fresh finalization still delivers immediately. Explicit
recovery and startup recovery keep their default behavior. Governed-review
presentation and other native finalization duties remain separate.
All 66 startup tests pass, including disabled maintenance and close subscription
registration. Two real PostgreSQL regressions verify that both periodic entry
points leave the response unmaterialized, and the worker then creates exactly one
publication from the accepted result. Log: `/tmp/sleep48-wiring-tests.log`.
Workspace typecheck passes (`/tmp/sleep48-full-typecheck.log`). The full local suite
is still running and has reported embedded PostgreSQL initialization failures;
its final result and build status remain pending. No full-suite or remote check
acceptance is claimed at this checkpoint.
## Review repair and current verification, 2026-10-26
Published draft #17696 at `79bf36cbf8`. Greptile found a real precision gap:
`writeRunStatus` still sent a company-only milestone hint. Every unrelated run
status could reconstruct dormant response history. Three real-PostgreSQL cases
(native binding, legacy context binding, and taskless run) failed before repair
(`/tmp/sleep48-scope-before.log`).
The milestone contract now accepts an optional run identity. Status producers
supply the task when bound and the exact run otherwise. The response fanout
prefers task scope over run scope, because lineage changes can authorize other
runs on that task. Known run claim/release and completion producers also preserve
an exact-run fallback. Existing company configuration invalidations remain broad.
No new durable rows or caller recovery wrappers are introduced.
All 34 heartbeat lifecycle tests pass after repair. Two real worker regressions
retain a dormant eligible response, send unrelated task/run wakes, and observe
one actual candidate SQL query at the postgres.js debug boundary, without
rechecking the dormant row. The outer-commit/rollback cases additionally send a
different run identity within the same task and retain task-level recovery.
An initial test mistakenly counted Drizzle subquery builder construction as a
second SQL statement; driver-level observation corrected the measurement.
The response/authority/admission selection now passes all 63 cases
(`/tmp/sleep48-scope-acceptance.log`). Server typechecking and the workspace build
pass (`/tmp/sleep48-scope-typecheck-v3.log`, `/tmp/sleep48-scope-build.log`).
The previous full workspace typecheck passed at `79bf36cbf8`. The full local
Vitest run was explicitly stopped after repeated embedded PostgreSQL bootstrap
failures; `/tmp/sleep48-full-tests.log` preserves those failures. It is not a
full-suite pass. Focused database acceptance uses the validated external
PostgreSQL path and isolated disposable test databases.
Remote CI at `79bf36cbf8` also exposed a late-journal test that assumed worker
startup drains all candidates in one batch. It now waits for actual settlement,
as the shared worker deliberately processes fair batches. The workspaces-b job
hit a five-second sandbox staging test timeout; E2E shard five received a runner
shutdown signal. These failures remain recorded in `/tmp/sleep48-ci-chat3.log`,
`/tmp/sleep48-ci-workspaces-b.log`, and `/tmp/sleep48-ci-e2e5.log`. Fresh-head CI and
review are required after the repair; no failing gate is waived.
## Full-history test repair, 2026-10-26
At `c4beddad1e`, fresh Greptile review is 5/5 and the scope thread is resolved.
Remote CI passed 51 contexts but failed the chat shard and its aggregate gate:
a second startup assertion assumed `worker.ready` drained every fair batch.
The full local chat file finished with 1,595 passing and seven failing tests
(`/tmp/sleep48-chat-full.log`); all seven were in the newly added worker tests.
The larger run exposed two related observation errors: startup can have another
batch scheduled, and `nextWakeAt() === null` can be transient while a batch runs.
The latter caused query instrumentation to count unfinished startup checks and
an outcome assertion to run before its response had been materialized. An early
future-deadline test failure then left its fixture's real deadline for later tests.
Worker startup/settlement assertions now require both zero active work and the
expected next deadline. This changes test observation, not scheduler behavior or
batch limits. All 18 affected cases pass against the retained history database
from the full 1,602-case run (`/tmp/sleep48-settlement-history.log`), including
single-query unrelated wakes, future deadlines, and coordinator retry release.
The full-run failure remains recorded; fresh remote CI is still required.
## Current migration 0412 integration and complete acceptance
The original response-presentation patch is integrated with the validated current
native-effect parent. The complete owned runtime/schema changes are preserved,
including fourteen files with combined upstream edits and 10,401 unchanged
unrelated upstream source files. The control-plane fixture conflict retains its
dedicated external database path and the current recovery-helper fallback; all
upstream conformance cases and owned assertions remain. Both audit histories
are retained.
Only this slice's unlanded migration artifacts were deferred before integration.
`pnpm db:generate` generated `0412_tearful_squadron_supreme.sql`. The generated
index definition was checked against the original index before restoring all
three original statements byte for byte: concurrent idempotent index creation,
selection statistics, and ANALYZE. No original tuning statement is dropped. All
prior executable SQL, journal entries and retained snapshot bytes are unchanged.
Five snapshots 0408 through 0412 remain; the concurrent allowlist includes 0412.
Source `9feec170ef8174011fc3c5d3d6d76f0a0f39fea0` passes **4171 distinct cases
in 29 complete files**, without skips: 2313 owner/producer/consumer cases in
221.96 seconds, 59 safe-replacement cases in 29.10 seconds, and all 1799
assembled chat integration cases in 769.87 seconds. This includes the complete
GitHub workflow, accepted-response worker lifecycle, authenticated late evidence,
outer commit/rollback, contention, authority registration, agent resume, startup,
retry, drain/resume, and unrelated retained-history isolation. Per-file counts:
| Complete file | Passing cases |
| --- | ---: |
| src/__tests__/chat-channels.integration.test.ts | 1799 |
| src/__tests__/committed-chat-response-authority.test.ts | 7 |
| src/__tests__/issue-recovery-actions.test.ts | 89 |
| src/__tests__/native-finalization-recovery.test.ts | 32 |
| src/__tests__/native-interaction-bridge.test.ts | 7 |
| src/__tests__/native-status-arbiter-corpus.test.ts | 94 |
| src/__tests__/native-workspace-sync.test.ts | 10 |
| src/__tests__/recovery-stale-issue-lock-sweep.test.ts | 40 |
| src/__tests__/server-startup-feedback-export.test.ts | 69 |
| src/__tests__/slack-conversation-lifecycle.test.ts | 41 |
| src/modules/run-dispatch/adapters/postgres.test.ts | 69 |
| src/modules/wake-queue/adapters/postgres.test.ts | 93 |
| src/modules/wake-queue/adapters/queued-comment-postgres.test.ts | 5 |
| src/services/chat-monitor-continuation.test.ts | 21 |
| src/services/execution-control-reconciliation.test.ts | 10 |
| src/services/explicit-native-continuation.test.ts | 335 |
| src/services/heartbeat/queue.test.ts | 32 |
| src/services/heartbeat/recovery.test.ts | 45 |
| src/services/heartbeat/run-control.test.ts | 51 |
| src/services/heartbeat/run-lifecycle.test.ts | 34 |
| src/services/heartbeat/upgrade-readiness.test.ts | 7 |
| src/services/legacy-execution-recovery.test.ts | 15 |
| src/services/native-runtime/native-restart-recovery.test.ts | 25 |
| src/services/native-runtime/native-run-finalizer-telemetry.test.ts | 29 |
| src/services/native-runtime/native-safe-replacement.test.ts | 59 |
| src/services/native-runtime/native-session-executor.test.ts | 968 |
| src/services/native-runtime/native-workspace-export-retry.test.ts | 80 |
| src/services/native-runtime/native-workspace-reconstruction.test.ts | 50 |
| src/services/native-runtime/paperclip-control-plane-port.test.ts | 55 |
The complete 94-case status-authority corpus also passes again on the same
database in 16.68 seconds. It is not double-counted. Three retained PSC/OUT and
sentinel company rows remain unchanged, and each invocation uses distinct
UUID-derived fixture prefixes. Twenty-four separate fresh PostgreSQL 17
databases migrate through 0412 before setup. Four embedded-only fixture imports
temporarily use their own external database on this IPC-exhausted host, then
restore byte for byte. All assertions and timing limits remain unchanged. This
is qualified external-database acceptance, not an unmodified embedded CI run.
Actual database build, server types/build, module boundaries and exact-head
migration order pass. A populated 0411-to0412 upgrade preserves all 410 prior
migration receipts and all 38 companies, appends exactly one receipt, and adds
none on replay. The partial index is valid and ready on both fresh and upgraded
databases; the existing selection statistic is present. Earlier failed runs
above retain their historical scopes and explanations. The current complete
chat run passes, rather than substituting a focused retry for the full suite.
Normal merge `1c019b48c98cd8273aa3b7e3cf95433a369b9531` adopts published native-effect
parent `5bb5ddacd3d1e6817de4825b283c6e92f85bd124` and actual unsafe-export ancestry.
All complete server, package, UI and test trees remain identical to the accepted
source. Publication remains held until #17690 lands and its actual squash
ancestry is reconciled. Fresh published-head CI and exact-head Greptile 5/5
remain required. These checks do not establish deployed suspension, external
Cloud alarm delivery or fleet savings.
## Actual native-effect prerequisite ancestry
Native-effect prerequisite #17690 merged as
`a8a038e659c50c1c5e3ce532d13072a29a5cf951`, reviewed at
`5bb5ddacd3d1e6817de4825b283c6e92f85bd124`. Its entire squash tree equals
the reviewed parent already adopted by this child. The normal ancestry merge
automatically restored the removed pending-effect notification module. The
whole-tree identity check rejected that composition; its deletion was restored
before publication. All accepted source, including the complete 0412 database
tree and all 29 test files, remains byte-identical. The following edit records
the ancestry proof. The 4171-case acceptance keeps its original source scope;
fresh published-head CI and exact-head Greptile 5/5 are required.
## 2026-10-29 review repair: routine hints cannot invalidate a company
Superagent review of `a5648c92f4601fb72cec26b9ff2e99e40d502042` identified a
real P2: coalesced inbound messages used a company-only milestone hint, and
the new response fanout reconstructed every unpresented response in that
company. Pending-effect settlement also used a company-only finalization hint.
The current-head scan required action, so the branch was not merged. Greptile
rated that head 5/5 but conditioned readiness on the later #17702 scoped
pending-effect fix. That fix is included here; this slice no longer requires
its later dependent PR to be independently correct.
The repair keeps routine milestones and finalization hints from entering the
response queue without an issue or run. Real coalescing names its already-known
active run. Pending-effect settlement names the affected issue, covering older
decisions on that issue. Retry promotion/cancellation retain their known run
when older context omits an issue. Existing pending-effect and chat consumers
keep their previous serialized hints; only response reconstruction is narrowed.
Genuine company-wide authority changes remain supported through one explicit
`notifyChatCompanyAuthorityChange` boundary. Sixteen existing mutation hooks in
nine owning files use it: membership and board-claim restoration, Cloud actor
membership repair, GitHub/Slack identity and sponsor changes, endpoint policy,
resource enablement and connection configuration. They keep their original
change predicates and transaction owners. Two issue-scoped resource/publication
hooks remain issue-scoped. No transaction wrappers, queue tables, polling or
new persisted bookkeeping are introduced. Dynamic milestone producers carry
issue-bound interaction rows or typed recovery inputs.
The ordinary coalescing regression first fails on the actual PostgreSQL writer:
its settled hint has only the company instead of the active run. Two receiver
regressions each issue three routine company hints against dormant response
history and observe **12 response-history SELECTs**. Both must observe zero.
A positive test restores suspended membership through the actual access owner
and delivers the waiting response once; it passes before and after the repair.
The unchanged four focused assertions pass after the fix. Assertion time limits
are unchanged. These tests measure real driver SQL and persisted producer work.
Repair source `2fc8b0614aaf9ac7337045de4ced7325f95edc45` passes **1821 distinct
server cases**: 1496 in 13 complete suites in 110.63 seconds, and 325 selected
chat cases in 278.99 seconds. The chat selection includes the complete GitHub
workflow, accepted-response lifecycle, the new regressions, and real post-close
authority restoration. Another 1477 chat cases are outside the name pattern;
the complete 1802-case file is not claimed for this repair run. Counts:
| File | Passing cases |
| --- | ---: |
| src/__tests__/access-service.test.ts | 18 |
| src/__tests__/auth-session-route.test.ts | 12 |
| src/__tests__/board-claim.test.ts | 3 |
| src/__tests__/chat-channels.integration.test.ts (focused) | 325 |
| src/__tests__/committed-chat-response-authority.test.ts | 7 |
| src/__tests__/native-finalization-recovery.test.ts | 32 |
| src/__tests__/native-status-arbiter-corpus.test.ts | 94 |
| src/__tests__/server-startup-feedback-export.test.ts | 69 |
| src/modules/run-dispatch/adapters/postgres.test.ts | 69 |
| src/modules/wake-queue/adapters/postgres.test.ts | 94 |
| src/modules/wake-queue/application/use-cases.test.ts | 67 |
| src/services/heartbeat/run-lifecycle.test.ts | 34 |
| src/services/native-runtime/native-run-finalizer-telemetry.test.ts | 29 |
| src/services/native-runtime/native-session-executor.test.ts | 968 |
Eleven dedicated fresh PostgreSQL 17 databases migrate through unchanged 0412.
The wake-queue fixture's embedded-only import temporarily uses its own external
database and restores byte for byte. All 14 selected source files match their
pre-run hashes. Actual database build, server types/build, module boundaries
and full-SHA migration order pass. The earlier 4171-case full baseline keeps
its original pre-repair source scope; it is not represented as this repair run.
Master independently advanced to `522c653d562a41ecad64561d35457cb99ecf6cd6`
with #18126's Pi archive-download retry fix. Normal merge
`2a535bc61259ef99387369e8b32498f774e5ba2c` preserves both complete binary patches.
All accepted server, database, UI, test and CLI trees remain identical; only
the three independently changed runner installer/manifest files are added.
All 13 installer tests pass in 200.18 milliseconds, including stalled-header
and stalled-body cancellation. Combined server types/build, boundaries and
migration order pass. No unrelated upstream source is discarded.
Explicit global authority invalidation and startup still reconstruct their
company's owed candidates with existing 100-row discovery pages and fair
execution batches. Discovery retains linear time and memory; it is not a
constant-cost global operation. The prior 10,001-candidate profile remains
relevant to those genuine global obligations. This repair removes ordinary
inbound traffic's ability to request that global reconstruction and prevents
unscoped routine traffic from rebuilding dormant response history. It does not waive an actual review
failure or establish deployed sleep, Cloud wake delivery or fleet savings.
Fresh published-head CI, Greptile 5/5 and a clean security scan remain required.
A second publication guard detected master #18128 at
`e8b78cce5bb557b086bee9ca930eeaab1d09b244`. Normal merge
`06d55ca9f872f7e7794680335811b496942be034` retains its three test-only file changes.
Both complete owned and upstream binary patches are preserved. An isolated Git
index projects only the owned audit onto the original repair source to prove
this equality; it does not rewrite branch history or include master changes
as owned edits. All 14 repair test hashes and the entire package tree remain
unchanged. The changed HTTPS backfill case passes on a fresh 0412 PostgreSQL
fixture in 7.62 seconds; 178 other workspace cases are outside that focused
selection. Its embedded-only fixture import restores byte for byte. Combined
server types/build, boundaries and migration order pass again. The independent
browser fixture changes are preserved exactly; no local browser run is claimed.
### Recorded presentation and structural eligibility repair
The exact published head `ae75e21bd5aa5841b66f0d3f421181cdcfa9a3be`
received Greptile 5/5, a clean security scan, and passing CI. Its completed
Ponytail 4/5 verdict nevertheless required two correctness fixes. That actual
failing verdict was honored; no review timeout or numeric score waived it.
The review also found a missing fresh-commit skip guard, which is repaired here.
A recorded non-create presentation decision (`none`, `reuse`, or another explicit
non-create value) now excludes both discovery and locked recovery. A string
selected comment ID still fences recovery, including deleted comments. The
`skipIssueComment` policy applies to all recovery branches. Missing legacy
presentation metadata and explicit `create` with a null comment ID remain owed
work and can recover once. No per-worker table or transaction wrapper is added.
Discovery joins the owning company/issue and applies the cheap durable guards
before taking recovery locks: done results belong to succeeded non-conversation
runs; response waits have empty attention requests and usable summary/key;
monitor and same-agent continuations require succeeded runs and their accepted
reason codes. Digest integrity, current publication authority, provider-owned
interactions, final-message selection and other owner checks remain under the
existing transaction. These predicates narrow structural candidates; they do not
claim to establish authorization or replace late-evidence and authority-change
notifications. Fresh finalization now honors `skipNativeResponsePresentation`
as the already-committed path does.
The final corrected regression source is unchanged between the red and green
runs. The published baseline fails eight behavioral assertions and passes the
three positive genuine-answer cases. Repair source
`892f36e5f66ba1af123034936a7160c64c3ab883` passes all eleven. Recorded none/reuse
produce no new comments or publications and keep their decision. Five families
of structurally ineligible retained rows produce zero issue/run/finalization
`FOR UPDATE` queries across three issue-scoped notifications each. Those selector
fixtures deliberately vary durable payload fields; integrity and authorization
remain separate recovery requirements. Genuine missing, legacy and create-null
presentations still publish once. A genuinely uncommitted accepted run proves
that periodic finalization leaves presentation to the worker.
Earlier material failures remain in the verification record: an incomplete
conversation identity fixture, a duplicate variable declaration, a remaining
stale variable reference and its downstream retry contamination, a test that
accidentally used the matching continuation reason, and a missing fixture import.
The fixture was corrected to exercise the actual incompatible reason; assertions
and time limits were not weakened. The final red comparison restores the exact
published production files temporarily and verifies byte-exact restoration of
the repair afterward. These were explained local setup/implementation failures,
not inferred infrastructure faults.
Repair source passes **1489 distinct cases**: 1139 across six complete consumer
suites in 81.02 seconds, plus 350 selected chat integration cases in 297.68s.
The remaining 1463 chat cases are outside the name pattern; the full 1813-case
file is not claimed for this repair. The eleven new regressions are included
in the selected count, not added again. The six complete files cover 968 native
session cases, 79 response-summary cases, 21 monitor-continuation cases, 29 native
finalizer cases, 32 finalization-recovery cases and 10 workspace-sync cases.
No executable migration, schema, prior history, index/statistics statement or
snapshot changes in this repair. Five fresh PostgreSQL 17 fixtures migrate
through 0412; no embedded fixture adaptation is needed for these suites. All
selected test hashes match the captured source. Database build, server
types/build, module boundaries and full-SHA migration ordering pass. Actual host
sleep, external wake and fleet cost savings remain unqualified. A newly published
head still requires fresh exact-head Greptile 5/5, passing CI and no actionable
review findings before merge.
+1
View File
@@ -561,6 +561,7 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([
"0409_uneven_zuras.sql",
"0410_freezing_kinsey_walden.sql",
"0411_real_scrambler.sql",
"0412_tearful_squadron_supreme.sql",
"0384_funny_adam_warlock.sql",
@@ -0,0 +1,3 @@
CREATE INDEX CONCURRENTLY IF NOT EXISTS "heartbeat_runs_native_unpresented_response_idx" ON "heartbeat_runs" USING btree ("company_id","native_issue_id","id") WHERE "heartbeat_runs"."runtime_mode" = 'native' AND "heartbeat_runs"."status" in ('succeeded', 'failed', 'timed_out') AND "heartbeat_runs"."finished_at" is not null AND jsonb_typeof("heartbeat_runs"."result_json" #> '{presentationDecision,commentId}') is distinct from 'string';--> statement-breakpoint
CREATE STATISTICS IF NOT EXISTS "heartbeat_runs_native_response_selection_stats" ON (jsonb_typeof("result_json" #> '{presentationDecision,commentId}')) FROM "heartbeat_runs";--> statement-breakpoint
ANALYZE "heartbeat_runs";
@@ -1,6 +1,6 @@
{
"id": "856a147c-73dc-4908-a8d9-4f19cab7dbf6",
"prevId": "c34fcd91-0d57-4241-b8b4-399bc9f6dcb0",
"id": "3b8bb508-028d-4154-abdd-af870f1f3ca6",
"prevId": "ff56ee93-8cfc-465a-b26a-1c12d10773ff",
"version": "7",
"dialect": "postgresql",
"tables": {
@@ -2665,6 +2665,28 @@
}
},
"indexes": {
"agent_instruction_copies_collection_idx": {
"name": "agent_instruction_copies_collection_idx",
"columns": [
{
"expression": "run_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "NOT (coalesce(\"agent_instruction_working_copies\".\"receipt\", '{}'::jsonb) ? 'retainedByRunId') AND (\n (\"agent_instruction_working_copies\".\"attempts\" < 3 AND (\"agent_instruction_working_copies\".\"state\" IN ('prepared', 'pending_collection', 'unchanged_turn', 'warm_saved')\n OR (\"agent_instruction_working_copies\".\"state\" = 'preparing' AND \"agent_instruction_working_copies\".\"receipt\"->>'schema' = 'paperclip.agent-files.v1')))\n OR (\"agent_instruction_working_copies\".\"state\" = 'unavailable' AND \"agent_instruction_working_copies\".\"process_stopped_at\" IS NULL\n AND \"agent_instruction_working_copies\".\"location\" LIKE 'remote:%' AND \"agent_instruction_working_copies\".\"receipt\"->>'schema' = 'paperclip.agent-files.v1'))",
"concurrently": false,
"method": "btree",
"with": {}
},
"agent_instruction_copies_cleanup_idx": {
"name": "agent_instruction_copies_cleanup_idx",
"columns": [
@@ -33749,6 +33771,56 @@
}
},
"indexes": {
"heartbeat_runs_native_unpresented_response_idx": {
"name": "heartbeat_runs_native_unpresented_response_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "native_issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' AND \"heartbeat_runs\".\"status\" in ('succeeded', 'failed', 'timed_out') AND \"heartbeat_runs\".\"finished_at\" is not null AND jsonb_typeof(\"heartbeat_runs\".\"result_json\" #> '{presentationDecision,commentId}') is distinct from 'string'",
"concurrently": false,
"method": "btree",
"with": {}
},
"heartbeat_runs_cancelled_workspace_preservation_idx": {
"name": "heartbeat_runs_cancelled_workspace_preservation_idx",
"columns": [
{
"expression": "id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' AND \"heartbeat_runs\".\"status\" = 'cancelled'\n AND \"heartbeat_runs\".\"runner_profile_json\"->'nativeWorkspaceSync'->>'state' = 'prepared'\n AND \"heartbeat_runs\".\"result_json\"->'nativeCancellation'->>'dispatchState' = 'acknowledged'\n AND (\"heartbeat_runs\".\"result_json\"->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'\n AND \"heartbeat_runs\".\"result_json\"->>'workspaceRestoreFailure' IN ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_failed')) IS NOT TRUE",
"concurrently": false,
"method": "btree",
"with": {}
},
"heartbeat_runs_retained_recovery_adoption_idx": {
"name": "heartbeat_runs_retained_recovery_adoption_idx",
"columns": [
@@ -43064,6 +43136,62 @@
}
},
"indexes": {
"native_run_finalizations_pending_effect_owner_idx": {
"name": "native_run_finalizations_pending_effect_owner_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "decision_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"native_run_finalizations\".\"decision_id\" is not null AND \"native_run_finalizations\".\"result_id\" is not null AND \"native_run_finalizations\".\"phase\" <> 'terminal_failure'",
"concurrently": false,
"method": "btree",
"with": {}
},
"native_run_finalizations_unsafe_export_recovery_idx": {
"name": "native_run_finalizations_unsafe_export_recovery_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "run_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"native_run_finalizations\".\"phase\" = 'terminal_failure' AND \"native_run_finalizations\".\"failure_code\" = 'native_workspace_sync_out_unsafe_archive'",
"concurrently": false,
"method": "btree",
"with": {}
},
"native_run_finalizations_control_deadline_idx": {
"name": "native_run_finalizations_control_deadline_idx",
"columns": [
@@ -52368,6 +52496,34 @@
}
},
"indexes": {
"status_decision_effects_pending_delivery_idx": {
"name": "status_decision_effects_pending_delivery_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "decision_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"status_decision_effects\".\"delivery_state\" in ('pending', 'failed')",
"concurrently": false,
"method": "btree",
"with": {}
},
"status_decision_effects_decision_ordinal_uq": {
"name": "status_decision_effects_decision_ordinal_uq",
"columns": [
+7
View File
@@ -2871,6 +2871,13 @@
"when": 1793246172969,
"tag": "0411_real_scrambler",
"breakpoints": true
},
{
"idx": 412,
"version": "7",
"when": 1793249495056,
"tag": "0412_tearful_squadron_supreme",
"breakpoints": true
}
]
}
+3
View File
@@ -111,6 +111,9 @@ export const heartbeatRuns = pgTable(
updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(),
},
(table) => ({
nativeUnpresentedResponseIdx: index("heartbeat_runs_native_unpresented_response_idx")
.on(table.companyId, table.nativeIssueId, table.id)
.where(sql`${table.runtimeMode} = 'native' AND ${table.status} in ('succeeded', 'failed', 'timed_out') AND ${table.finishedAt} is not null AND jsonb_typeof(${table.resultJson} #> '{presentationDecision,commentId}') is distinct from 'string'`),
cancelledWorkspacePreservationIdx: index("heartbeat_runs_cancelled_workspace_preservation_idx").on(table.id, table.companyId)
.where(sql`${table.runtimeMode} = 'native' AND ${table.status} = 'cancelled'
AND ${table.runnerProfileJson}->'nativeWorkspaceSync'->>'state' = 'prepared'
@@ -1,6 +1,8 @@
import { notifyNativeFinalizationWork } from "../services/native-finalization-notifications.js";
import { nativeResponsePresentationWork } from "../services/native-runtime/native-response-presentation-work.js";
import { notifyChatRunMilestones } from "../services/chat-run-milestone-notifications.js";
import { registerGitHubReviewProjectionWork } from "../services/chat-channels/github/review-projection-work.js";
import { retainGitHubReviewReceipt } from "../services/chat-channels/github/task-preparation.js";
import { notifyChatRunMilestones } from "../services/chat-run-milestone-notifications.js";
import { settleUnrecoverableExecutions } from "../services/execution-recovery-resolution.js";
import { recordRunPresentationDecision } from "../services/heartbeat/run-completion.js";
import { environmentService } from "../services/environments.js";
@@ -16,13 +18,13 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db";
import { claimTaskRetryInput, releaseUnadmittedTaskRetryInput } from "../services/chat-channels/task-retry-input.js";
import { restoreNativeChatReviewPresentationInTransaction } from "../services/native-runtime/native-chat-review-presentation.js";
import { notifyChatEndpointConfiguration, notifyChatCompanyConfiguration } from "../services/chat-channels/configuration-work-notifications.js";
import { configureAgentLifecycle, createAgentLifecycle, processAgentLifecycle } from "../services/agent-lifecycle.js";
import { deleteCompany } from "../services/company-deletion.js";
import { adapterExecutionControls, createAdapterExecutionControl } from "../services/adapter-execution-control.js";
import { chatRunSourceMessageId } from "../services/chat-channels/run-origin.js";
import * as durableChatWakeup from "../services/durable-chat-wakeup.js";
import { conflict } from "../errors.js";
import * as githubTaskPreparation from "../services/chat-channels/github/task-preparation.js";
import { configureAgentLifecycle, createAgentLifecycle } from "../services/agent-lifecycle.js";
import { budgetPolicies } from "@paperclipai/db";
import { budgetService } from "../services/budgets.js";
import { idleWorkSnapshot } from "../services/task-admission.js";
@@ -150,6 +152,8 @@ import {
issues,
nativeRunResults,
nativeRunFinalizations,
statusDecisions,
statusDecisionEffects,
environmentLeases,
environments,
principalPermissionGrants,
@@ -267,6 +271,7 @@ import type {
import {
finalizeNativeRun,
repairCommittedNativeChatResponse,
recoverCommittedNativeChatResponse,
} from "../services/native-runtime/native-run-finalizer.js";
import { NativeRunCoordinatorStore } from "../services/native-runtime/native-run-coordinator-store.js";
import { reconcileNativeFinalizations } from "../services/native-runtime/native-finalization-reconciler.js";
@@ -66562,6 +66567,19 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
const input = { companyId: context.fixture.companyId, issueId: context.issue.id,
agentId: context.fixture.assignedAgentId, failedRunId: context.runId, initiatedByUserId: "owner-user" };
const stage = () => db.transaction(tx => context.service.prepareFailedChatRunRetry(tx, input));
const observedResponseAdmissions: string[] = [];
const responseHints: string[] = [];
await coordinator.register(DELIVERY_QUEUES.nativeResponsePresentation, {
retryMs: 30_000,
onSettled: hints => { responseHints.push(...hints ?? []); },
run: async () => {
const admitted = await db.select({ id: chatActions.id }).from(chatActions).where(and(
eq(chatActions.companyId, context.fixture.companyId), eq(chatActions.kind, "failed_run_retry"), eq(chatActions.status, "processed"),
));
observedResponseAdmissions.push(...admitted.map(row => row.id));
},
hasPending: async () => false,
}).ready;
context.wakeup.mockClear();
try {
let actionId: string;
@@ -66607,6 +66625,8 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
expect(action?.result.attemptCount).toBe(mode === "retry" ? 2 : 1);
expect(coordinator.nextWakeAt()).toBeNull();
}, { timeout: 10_000 });
await vi.waitFor(() => expect(observedResponseAdmissions).toContain(actionId), { timeout: 5000 });
expect(responseHints.map(hint => JSON.parse(hint))).toContainEqual({ companyId: context.fixture.companyId, issueId: context.issue.id });
expect(context.wakeup).toHaveBeenCalledTimes(mode === "retry" ? 2 : 1);
const receipts = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, actionId));
expect(receipts).toHaveLength(1);
@@ -74255,6 +74275,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
context: Awaited<ReturnType<typeof failedChatRetryFixture>>,
) => Promise<void>,
summary?: (issueId: string) => string,
finalize = true,
) {
const context = await failedChatRetryFixture(
provider,
@@ -74409,7 +74430,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
.from(nativeRunResults)
.where(eq(nativeRunResults.runId, context.runId));
const accepted = { resultId: acceptedRow.id };
await finalizeNativeRun({
if (finalize) await finalizeNativeRun({
db,
runId: context.runId,
workspaceFinalizeStatus: "succeeded",
@@ -74418,7 +74439,8 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
.select()
.from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, context.runId));
expect(coordinator.phase).toBe("committed");
if (finalize) expect(coordinator.phase).toBe("committed");
else expect(coordinator.phase).not.toBe("committed");
await db
.update(heartbeatRuns)
.set({
@@ -74616,6 +74638,11 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
})
: context.repair(),
).resolves.toBe(false);
await expect(recoverCommittedNativeChatResponse(db, {
companyId: mutation === "cross_company" ? randomUUID() : context.fixture.companyId,
issueId: context.issue.id,
runId: mutation === "cross_run" ? randomUUID() : context.runId,
})).resolves.toBe(false);
await expect(
db
.select()
@@ -75257,6 +75284,519 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
},
);
it.each(["none", "reuse"] as const)("preserves a recorded %s presentation instead of recovering an unsolicited answer", async commentAction => {
const context = await committedChatResponseRecoveryFixture("telegram");
const presentationDecision = { commentAction, commentId: null, reasonCodes: ["interaction_owns_response"] };
try {
await db.update(heartbeatRuns).set({ resultJson: { presentationDecision } }).where(eq(heartbeatRuns.id, context.runId));
await expect(recoverCommittedNativeChatResponse(db, { companyId: context.fixture.companyId,
issueId: context.issue.id, runId: context.runId })).resolves.toBe(false);
expect(await nativeResponsePresentationWork(db).load({ companyId: context.fixture.companyId, runId: context.runId })).toEqual([]);
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toEqual([]);
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual([]);
expect((await readHeartbeatRunsById(context.runId))[0].resultJson).toEqual({ presentationDecision });
} finally { await context.service.shutdown(); }
});
it.each(["create", "legacy", "missing"] as const)("recovers a genuine unanswered %s presentation", async action => {
const context = await committedChatResponseRecoveryFixture("telegram");
try {
await db.update(heartbeatRuns).set({ resultJson: action === "missing" ? {} : {
presentationDecision: { commentId: null, ...(action === "create" ? { commentAction: "create" } : {}) },
} }).where(eq(heartbeatRuns.id, context.runId));
const work = nativeResponsePresentationWork(db);
const entries = await work.load({ companyId: context.fixture.companyId, runId: context.runId });
expect(entries).toHaveLength(1);
await expect(work.check(entries[0])).resolves.toBeNull();
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1);
await expect(context.repair()).resolves.toBe(false);
} finally { await context.service.shutdown(); }
});
it.each(["conversation_done", "attention_request", "same_agent_reason", "monitor_reason", "skip_comment"] as const)(
"does not lock durably ineligible %s history on repeated issue milestones", async kind => {
const context = await committedChatResponseRecoveryFixture("telegram");
await context.service.shutdown();
await db.update(heartbeatRuns).set({ status: "succeeded",
...(kind === "skip_comment" ? { contextSnapshot: { skipIssueComment: true } } : {}) }).where(eq(heartbeatRuns.id, context.runId));
const [accepted] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, context.accepted.resultId));
const envelope = accepted.resultJson as { result: PrpStructuredRunResult; terminal: PrpTerminalState };
if (kind === "conversation_done") {
envelope.result = { ...envelope.result, reportedWorkDisposition: "done" };
envelope.terminal = { ...envelope.terminal, reportedWorkDisposition: "done" };
await db.update(issues).set({ conversationAgentId: context.fixture.assignedAgentId, conversationUserId: "owner-user", conversationState: "active", assigneeUserId: null })
.where(eq(issues.id, context.issue.id));
} else if (kind === "attention_request") {
envelope.result = { ...envelope.result, attentionRequests: [{ kind: "review", summary: "Needs a decision", ownerClass: "human" }] };
} else if (kind === "same_agent_reason" || kind === "monitor_reason") {
envelope.result = { ...envelope.result, continuation: { ...envelope.result.continuation,
kind: kind === "same_agent_reason" ? "same_agent" : "monitor" } as PrpStructuredRunResult["continuation"] };
await db.update(statusDecisions).set({ reasonCode: "external_chat_response_waiting" })
.where(eq(statusDecisions.id, context.coordinator.decisionId!));
}
await db.update(nativeRunResults).set({ resultJson: envelope }).where(eq(nativeRunResults.id, accepted.id));
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const work = nativeResponsePresentationWork(db);
const worker = work.register(coordinator, () => true);
const debug = db.$client.options.debug;
const locks: string[] = [];
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 10_000 })
.toEqual({ active: 0, next: null });
db.$client.options.debug = (_connection, query) => {
if (query.startsWith("select") && query.includes("for update") && /"(?:issues|heartbeat_runs|native_run_finalizations)"/.test(query)) locks.push(query);
};
for (let i = 0; i < 3; i++) {
await db.transaction(tx => notifyChatRunMilestones(tx, { companyId: context.fixture.companyId, issueId: context.issue.id }));
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 5000 })
.toEqual({ active: 0, next: null });
}
expect(locks).toEqual([]);
expect(await work.load({ companyId: context.fixture.companyId, issueId: context.issue.id })).toEqual([]);
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual([]);
expect(errors).toEqual([]);
} finally { db.$client.options.debug = debug; worker.stop(); await coordinator.stop(); }
});
it("leaves fresh-commit accepted-response presentation to the worker when periodic replay skips it", async () => {
const context = await committedChatResponseRecoveryFixture("telegram", "coordinator", false, undefined, undefined, false);
try {
await finalizeNativeRun({ db, runId: context.runId, workspaceFinalizeStatus: "succeeded",
projectRunStatus: true, skipNativeResponsePresentation: true });
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual([]);
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toEqual([]);
const work = nativeResponsePresentationWork(db);
const entries = await work.load({ companyId: context.fixture.companyId, runId: context.runId });
expect(entries).toHaveLength(1);
await expect(work.check(entries[0])).resolves.toBeNull();
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1);
} finally { await context.service.shutdown(); }
});
it.each(["reconciler", "finalizer"])("leaves accepted-response presentation to its worker during periodic %s replay", async caller => {
const context = await committedChatResponseRecoveryFixture("telegram");
try {
if (caller === "reconciler") {
await reconcileNativeFinalizations(db, [context.runId], { skipNativeResponsePresentation: true });
} else {
await finalizeNativeRun({ db, runId: context.runId, workspaceFinalizeStatus: "succeeded",
projectRunStatus: true, skipNativeResponsePresentation: true });
}
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toHaveLength(0);
const work = nativeResponsePresentationWork(db);
const entries = await work.load({ companyId: context.fixture.companyId, runId: context.runId });
expect(entries).toHaveLength(1);
await expect(work.check(entries[0])).resolves.toBeNull();
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1);
await expect(context.repair()).resolves.toBe(false);
} finally { await context.service.shutdown(); }
});
it("recovers accepted-response work on startup and leaves selected history quiet", async () => {
const context = await committedChatResponseRecoveryFixture("telegram");
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const work = nativeResponsePresentationWork(db);
const worker = work.register(coordinator, () => true);
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
const comments = await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId));
expect(comments).toHaveLength(1);
expect(comments[0].body).toBe(context.result.summary);
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1);
expect(await work.load({ companyId: context.fixture.companyId })).toEqual([]);
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
try {
await vi.advanceTimersByTimeAsync(120_000);
expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled();
} finally { reads.mockRestore(); writes.mockRestore(); }
expect(errors).toEqual([]);
} finally { worker.stop(); await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); }
});
it("reactivates dormant accepted-response work when the live chat authority registers", async () => {
const context = await committedChatResponseRecoveryFixture("telegram");
await context.service.shutdown();
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const worker = nativeResponsePresentationWork(db).register(coordinator, () => true);
let replacement: ReturnType<typeof createService> | undefined;
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
replacement = createService(new FakeChatSdkRuntime(), fakeTelegramFetch() as typeof fetch);
await expect.poll(async () => (await readChatPublicationsByEndpointId(context.endpoint.id)).length, { timeout: 5000 }).toBe(1);
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
expect(errors).toEqual([]);
} finally { worker.stop(); await coordinator.stop(); await replacement?.service.shutdown(); }
});
it.each(["issue", "run"])("does not reload dormant accepted-response history for an unrelated %s wake", async scope => {
const context = await committedChatResponseRecoveryFixture("telegram");
await context.service.shutdown();
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const work = nativeResponsePresentationWork(db);
const worker = work.register(coordinator, () => true);
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
expect(await work.load({ companyId: context.fixture.companyId, runId: context.runId })).toHaveLength(1);
const previousDebug = db.$client.options.debug;
const queries: string[] = [];
db.$client.options.debug = (_connection, query) => {
if (query.startsWith("select") && query.includes("native_run_finalizations")) queries.push(query);
};
try {
await db.transaction(tx => notifyChatRunMilestones(tx, { companyId: context.fixture.companyId,
...(scope === "issue" ? { issueId: randomUUID() } : { runId: randomUUID() }) }));
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 5000 }).toEqual({ active: 0, next: null });
expect(queries).toHaveLength(1);
} finally { db.$client.options.debug = previousDebug; }
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
expect(errors).toEqual([]);
} finally { worker.stop(); await coordinator.stop(); }
});
it.each(["milestone", "finalization"] as const)("ignores repeated unscoped routine %s hints without rebuilding dormant accepted-response history", async source => {
const context = await committedChatResponseRecoveryFixture("telegram");
await context.service.shutdown();
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const work = nativeResponsePresentationWork(db);
const worker = work.register(coordinator, () => true);
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 10_000 })
.toEqual({ active: 0, next: null });
expect(await work.load({ companyId: context.fixture.companyId, runId: context.runId })).toHaveLength(1);
const debug = db.$client.options.debug;
const queries: string[] = [];
db.$client.options.debug = (_connection, query) => {
if (query.startsWith("select") && query.includes("native_run_finalizations")) queries.push(query);
};
try {
for (let i = 0; i < 3; i++) {
await db.transaction(tx => source === "milestone"
? notifyChatRunMilestones(tx, { companyId: context.fixture.companyId })
: notifyNativeFinalizationWork(tx, { companyId: context.fixture.companyId }));
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 5000 })
.toEqual({ active: 0, next: null });
}
expect(queries).toEqual([]);
} finally { db.$client.options.debug = debug; }
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
expect(errors).toEqual([]);
} finally { worker.stop(); await coordinator.stop(); }
});
it("wakes dormant accepted-response history from the real company membership restoration owner", async () => {
const context = await committedChatResponseRecoveryFixture("telegram");
const [membership] = await db.select().from(companyMemberships).where(and(
eq(companyMemberships.companyId, context.fixture.companyId), eq(companyMemberships.principalId, "owner-user")));
await db.update(companyMemberships).set({ status: "suspended" }).where(eq(companyMemberships.id, membership.id));
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const worker = nativeResponsePresentationWork(db).register(coordinator, () => true);
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 10_000 })
.toEqual({ active: 0, next: null });
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
await accessService(db).updateMember(context.fixture.companyId, membership.id, { status: "active" });
await expect.poll(async () => ({ count: (await readChatPublicationsByEndpointId(context.endpoint.id)).length,
active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 5000 })
.toEqual({ count: 1, active: 0, next: null });
expect(errors).toEqual([]);
} finally { worker.stop(); await coordinator.stop(); await context.service.shutdown(); }
});
it.each(["commit", "rollback"])("keeps accepted-response work behind the outer %s boundary", async outcome => {
const context = await committedChatResponseRecoveryFixture("telegram");
await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, context.runId));
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const worker = nativeResponsePresentationWork(db).register(coordinator, () => true);
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
const transaction = db.transaction(async tx => {
await tx.transaction(async nested => {
await notifyChatRunMilestones(nested, { companyId: context.fixture.companyId, issueId: context.issue.id, runId: randomUUID() });
await nested.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, context.runId));
});
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
expect(idleWorkSnapshot().active).toBeGreaterThan(0);
if (outcome === "rollback") throw new Error("response fixture rollback");
});
if (outcome === "rollback") await expect(transaction).rejects.toThrow("response fixture rollback");
else await transaction;
await expect.poll(async () => ({
publications: (await readChatPublicationsByEndpointId(context.endpoint.id)).length,
active: idleWorkSnapshot().active,
next: coordinator.nextWakeAt(),
}), { timeout: 5000 }).toEqual({ publications: outcome === "commit" ? 1 : 0, active: 0, next: null });
expect(errors).toEqual([]);
} finally { worker.stop(); await coordinator.stop(); await context.service.shutdown(); }
});
it("retains the first accepted-response wake through idle drain without passive SQL", async () => {
const context = await committedChatResponseRecoveryFixture("telegram");
await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, context.runId));
let canRun = true;
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => canRun, canReconcile: () => true, onError: error => errors.push(error) });
const worker = nativeResponsePresentationWork(db).register(coordinator, () => canRun);
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
canRun = false;
await db.transaction(async tx => {
await notifyChatRunMilestones(tx, { companyId: context.fixture.companyId, issueId: context.issue.id });
await tx.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, context.runId));
});
await expect.poll(() => idleWorkSnapshot().active, { timeout: 5000 }).toBe(0);
expect(coordinator.nextWakeAt()).not.toBeNull();
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
try {
await vi.advanceTimersByTimeAsync(120_000);
expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled();
} finally { reads.mockRestore(); writes.mockRestore(); }
canRun = true; worker.wake();
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(1000);
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1);
expect(coordinator.nextWakeAt()).toBeNull();
}, { timeout: 5000 });
expect(errors).toEqual([]);
} finally { worker.stop(); await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); }
});
it("retains a failed accepted-response attempt across drain and retries after resume", async () => {
const context = await committedChatResponseRecoveryFixture("telegram");
let release!: () => void, observed!: () => void;
const hold = new Promise<void>(resolve => { release = resolve; });
const locked = new Promise<void>(resolve => { observed = resolve; });
const holder = db.transaction(async tx => {
await tx.select().from(chatActions).where(eq(chatActions.id, context.action.id)).for("update");
observed();
await hold;
});
await locked;
let canRun = true;
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => canRun, canReconcile: () => true, onError: error => errors.push(error) });
const worker = nativeResponsePresentationWork(db).register(coordinator, () => canRun);
try {
await worker.ready;
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(1000);
expect(errors).toHaveLength(1);
expect(idleWorkSnapshot().active).toBe(0);
}, { timeout: 5000 });
expect(coordinator.nextWakeAt()).toBeGreaterThan(Date.now());
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
canRun = false;
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
try {
await vi.advanceTimersByTimeAsync(120_000);
expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled();
} finally { reads.mockRestore(); writes.mockRestore(); }
release(); await holder;
canRun = true; worker.wake();
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(1000);
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1);
expect(idleWorkSnapshot().active).toBe(0);
expect(coordinator.nextWakeAt()).toBeNull();
}, { timeout: 5000 });
expect(errors).toHaveLength(1);
} finally { release(); await holder; worker.stop(); await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); }
});
it("reconstructs a future accepted-response deadline without early publication", async () => {
const context = await committedChatResponseRecoveryFixture("telegram");
const at = new Date(Date.now() + 60_000);
await db.update(nativeRunFinalizations).set({ nextAttemptAt: at }).where(eq(nativeRunFinalizations.runId, context.runId));
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const worker = nativeResponsePresentationWork(db).register(coordinator, () => true);
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: at.getTime() });
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
await vi.advanceTimersByTimeAsync(Math.max(0, at.getTime() - Date.now()));
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(1000);
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1);
expect(coordinator.nextWakeAt()).toBeNull();
}, { timeout: 5000 });
expect(errors).toEqual([]);
} finally { worker.stop(); await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); }
});
it("retains an accepted-response deadline that elapses during the locked recovery check", async () => {
const context = await committedChatResponseRecoveryFixture("telegram");
const work = nativeResponsePresentationWork(db);
const [entry] = await work.load({ companyId: context.fixture.companyId });
expect(entry?.id).toBe(context.runId);
vi.useFakeTimers({ toFake: ["Date"] });
const due = new Date(Date.now() + 1000);
const transaction = db.transaction.bind(db);
const intercept = vi.spyOn(db, "transaction").mockImplementationOnce((async (callback, options) => {
await transaction(async tx => {
await tx.update(nativeRunFinalizations).set({ nextAttemptAt: due }).where(eq(nativeRunFinalizations.runId, context.runId));
});
const result = await transaction(callback, options);
vi.setSystemTime(due.getTime() + 1);
return result;
}) as Db["transaction"]);
try {
const deferred = await work.check(entry);
intercept.mockRestore();
expect(deferred).toEqual({ ...entry, dueAt: due.getTime() });
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
await expect(work.check(deferred!)).resolves.toBeNull();
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1);
} finally { intercept.mockRestore(); vi.useRealTimers(); await context.service.shutdown(); }
});
it("wakes dormant accepted-response recovery after agent resume", async () => {
const context = await committedChatResponseRecoveryFixture("telegram");
await db.update(agents).set({ status: "paused", lifecycleState: "paused", lifecycleHolds: ["manual"], lifecycleOperation: null })
.where(eq(agents.id, context.binding.agentId));
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const work = nativeResponsePresentationWork(db);
const worker = work.register(coordinator, () => true);
const lifecycle = configureAgentLifecycle(db, {
requiredPluginIds: async () => [], runHost: async () => "complete", runPlugin: async () => "complete",
});
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
expect(await work.load({ companyId: context.fixture.companyId, agentId: randomUUID() })).toEqual([]);
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
await createAgentLifecycle(db).resumeAgent(context.binding.agentId);
await processAgentLifecycle(db, context.binding.agentId);
await expect.poll(async () => (await db.select().from(agents).where(eq(agents.id, context.binding.agentId)))[0].status).toBe("idle");
await expect.poll(async () => (await readChatPublicationsByEndpointId(context.endpoint.id)).length, { timeout: 5000 }).toBe(1);
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
expect(errors).toEqual([]);
} finally { await lifecycle.stop(); worker.stop(); await coordinator.stop(); await context.service.shutdown(); }
});
it("wakes accepted-response recovery from the real coordinator retry release", async () => {
const context = await committedChatResponseRecoveryFixture("telegram");
const store = new NativeRunCoordinatorStore(db, context.binding);
await store.claimFinalization({ leaseOwner: "response-recovery-fixture", leaseTtlMs: 1000 });
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const worker = nativeResponsePresentationWork(db).register(coordinator, () => true);
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
await vi.advanceTimersByTimeAsync(2000);
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
expect(coordinator.nextWakeAt()).toBeNull();
await store.markFinalizationRetry({ leaseOwner: "response-recovery-fixture", failureCode: "response_fixture_retry", retryAfterMs: 1000 });
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(1000);
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1);
expect(idleWorkSnapshot().active).toBe(0);
expect(coordinator.nextWakeAt()).toBeNull();
}, { timeout: 5000 });
expect(errors).toEqual([]);
} finally { worker.stop(); await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); }
});
it("wakes accepted-response recovery when its authenticated accepted-result journal arrives late", async () => {
const context = await committedChatResponseRecoveryFixture("telegram", "port");
await db.delete(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, context.runId), eq(heartbeatRunEvents.eventType, "run.result.accepted")));
const errors: unknown[] = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
const worker = nativeResponsePresentationWork(db).register(coordinator, () => true);
try {
await worker.ready;
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
const port = new PaperclipControlPlanePort(db, {
companyId: context.binding.companyId, issueId: context.binding.issueId,
runId: context.runId, agentId: context.binding.agentId, sessionId: context.binding.normalizedSessionId,
completionContractId: context.binding.completionContractId,
completionContractSha256: context.binding.completionContractSha256,
sourceInstanceId: context.binding.runnerSourceInstanceId, controlPlaneSourceInstanceId: `control-${context.runId}`,
});
await port.appendEvent({
schema: "paperclip.prp.event.v1", sourceEventId: `accepted-${context.runId}`, sourceSeq: 1,
sourceInstanceId: `control-${context.runId}`, sourceKind: "control_plane",
runId: context.runId, normalizedSessionId: context.binding.normalizedSessionId,
turnId: `turn-${context.runId}`, eventType: "run.result.accepted", schemaVersion: 1,
priority: 0, emittedAt: new Date().toISOString(), payload: { result: context.result },
});
await expect.poll(async () => ({
publications: (await readChatPublicationsByEndpointId(context.endpoint.id)).length,
active: idleWorkSnapshot().active, next: coordinator.nextWakeAt(),
}), { timeout: 5000 }).toEqual({ publications: 1, active: 0, next: null });
expect(errors).toEqual([]);
} finally { worker.stop(); await coordinator.stop(); await context.service.shutdown(); }
});
it("defers scheduled accepted-response recovery until the persisted retry is due", async () => {
const context = await committedChatResponseRecoveryFixture("telegram");
const input = { companyId: context.fixture.companyId, issueId: context.issue.id, runId: context.runId, onlyWhenDue: true };
try {
await db.update(nativeRunFinalizations).set({ nextAttemptAt: new Date(Date.now() + 60_000) })
.where(eq(nativeRunFinalizations.runId, context.runId));
await expect(recoverCommittedNativeChatResponse(db, input)).resolves.toBe(false);
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toHaveLength(0);
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
await db.update(nativeRunFinalizations).set({ nextAttemptAt: new Date(0) })
.where(eq(nativeRunFinalizations.runId, context.runId));
await expect(recoverCommittedNativeChatResponse(db, input)).resolves.toBe(true);
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toHaveLength(1);
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1);
} finally { await context.service.shutdown(); }
});
it.each(["pending", "failed"])("defers scheduled accepted-response recovery behind %s status effects", async deliveryState => {
const context = await committedChatResponseRecoveryFixture("telegram");
const input = { companyId: context.fixture.companyId, issueId: context.issue.id, runId: context.runId, onlyWhenDue: true };
try {
const [effect] = await db.select().from(statusDecisionEffects).where(eq(statusDecisionEffects.decisionId, context.coordinator.decisionId!));
expect(effect).toBeDefined();
await db.update(statusDecisionEffects).set({ deliveryState }).where(eq(statusDecisionEffects.id, effect.id));
await expect(recoverCommittedNativeChatResponse(db, input)).resolves.toBe(false);
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toHaveLength(0);
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
await db.update(statusDecisionEffects).set({ deliveryState: "delivered" }).where(eq(statusDecisionEffects.id, effect.id));
await expect(recoverCommittedNativeChatResponse(db, input)).resolves.toBe(true);
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1);
} finally { await context.service.shutdown(); }
});
it("leaves a contended accepted response retryable without reclassifying or partially publishing it", async () => {
const context = await committedChatResponseRecoveryFixture("telegram");
let release!: () => void;
@@ -75279,6 +75819,9 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
try {
await locked;
await expect(context.repair()).resolves.toBe(false);
await expect(recoverCommittedNativeChatResponse(db, {
companyId: context.fixture.companyId, issueId: context.issue.id, runId: context.runId,
})).rejects.toThrow();
await expect(
db
.select()
@@ -0,0 +1,55 @@
import { isNativeResponseEvidence } from "../services/native-response-presentation-notifications.js";
import { describe, expect, it, vi } from "vitest";
import type { Db } from "@paperclipai/db";
import { authorizeCommittedChatResponse, onCommittedChatResponseAuthorityRegistered, registerCommittedChatResponseAuthority } from "../services/durable-chat-wakeup.js";
describe("committed response authority lifecycle", () => {
it("reactivates only its database and unsubscribes without removing a replacement listener", () => {
const db = {} as Db, otherDb = {} as Db;
const first = vi.fn(), second = vi.fn(), other = vi.fn();
const disposeFirst = onCommittedChatResponseAuthorityRegistered(db, first);
const disposeOther = onCommittedChatResponseAuthorityRegistered(otherDb, other);
const removeAuthority = registerCommittedChatResponseAuthority(db, async () => {});
expect(first).toHaveBeenCalledTimes(1);
expect(other).not.toHaveBeenCalled();
disposeFirst();
const disposeSecond = onCommittedChatResponseAuthorityRegistered(db, second);
disposeFirst();
const removeReplacement = registerCommittedChatResponseAuthority(db, async () => {});
expect(first).toHaveBeenCalledTimes(1);
expect(second).toHaveBeenCalledTimes(1);
removeAuthority();
expect(second).toHaveBeenCalledTimes(1);
disposeSecond(); disposeOther(); removeReplacement();
const removeLast = registerCommittedChatResponseAuthority(db, async () => {});
expect(second).toHaveBeenCalledTimes(1);
removeLast();
});
it("retains the replacement authority when the old service shuts down", async () => {
const db = {} as Db;
const first = vi.fn(async () => {}), second = vi.fn(async () => {});
const removeFirst = registerCommittedChatResponseAuthority(db, first);
const removeSecond = registerCommittedChatResponseAuthority(db, second);
try {
removeFirst();
const input = { companyId: "company", issueId: "issue", agentId: "agent", runId: "run", resultId: "result" };
await authorizeCommittedChatResponse(db, db, input);
expect(first).not.toHaveBeenCalled();
expect(second).toHaveBeenCalledExactlyOnceWith(db, input);
} finally { removeSecond(); }
});
});
describe("accepted response evidence selection", () => {
it.each([
{ type: "item.delta", kind: "agentMessage", channel: "final", expected: false },
{ type: "item.completed", kind: "agentMessage", channel: "analysis", expected: false },
{ type: "item.completed", kind: "commandExecution", channel: "final", expected: false },
{ type: "item.completed", kind: "agentMessage", channel: "final", expected: true },
{ type: "run.result.accepted", kind: "agentMessage", channel: "final", expected: true },
])("signals only response evidence: $type/$kind/$channel", ({ type, kind, channel, expected }) => {
expect(isNativeResponseEvidence(type, { payload: { kind, channel, text: "Saved evidence" } })).toBe(expected);
});
});
@@ -116,6 +116,7 @@ const {
reapOrphanedRuns: vi.fn(async () => ({ reaped: 0, runIds: [] })),
registerScheduledRetryCancellationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
registerStoppedInstructionCollectionWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
registerNativeResponsePresentationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve(), stop: vi.fn() })),
registerNativePendingEffectWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
registerUnsafeWorkspaceExportWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
registerAutomaticCompletionReviewWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
@@ -558,6 +559,17 @@ describe("startServer feedback export wiring", () => {
process.env.BETTER_AUTH_SECRET = "test-secret";
});
it.each([false, true])("registers response recovery and its shutdown subscription only when maintenance is enabled (%s)", async enabled => {
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
await startServer(); await settleStartupHousekeeping();
expect(heartbeatServiceMock.registerNativeResponsePresentationWork).toHaveBeenCalledTimes(enabled ? 1 : 0);
if (enabled) {
expect(heartbeatServiceMock.registerNativeResponsePresentationWork.mock.calls[0][1]()).toBe(true);
const worker = heartbeatServiceMock.registerNativeResponsePresentationWork.mock.results[0].value;
expect(fakeServer.on).toHaveBeenCalledWith("close", worker.stop);
}
});
it.each([false, true])("registers native pending effects only when maintenance is enabled (%s)", async enabled => {
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
await startServer(); await settleStartupHousekeeping();
@@ -1005,7 +1017,7 @@ describe("startServer feedback export wiring", () => {
intervalCallback?.();
await new Promise<void>((resolve) => setImmediate(resolve));
expect(heartbeatServiceMock.sweepStaleIssueLocks).toHaveBeenCalledTimes(2);
expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true });
expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true });
expect(heartbeatServiceMock.reconcileCostAccounting).toHaveBeenCalledTimes(1);
expect(retiredDetector).not.toHaveBeenCalled();
} finally {
+3 -3
View File
@@ -4,7 +4,7 @@ import type { Db } from "@paperclipai/db";
import { companies, companyMemberships, instanceUserRoles } from "@paperclipai/db";
import type { DeploymentMode } from "@paperclipai/shared";
import { ensureHumanRoleDefaultGrants } from "./services/principal-access-compatibility.js";
import { notifyChatRunMilestones } from "./services/chat-run-milestone-notifications.js";
import { notifyChatCompanyAuthorityChange } from "./services/chat-run-milestone-notifications.js";
const LOCAL_BOARD_USER_ID = "local-board";
const CLAIM_TTL_MS = 1000 * 60 * 60 * 24;
@@ -125,7 +125,7 @@ export async function claimBoardOwnership(
.then((rows) => rows[0] ?? null);
if (!existing) {
await notifyChatRunMilestones(tx, { companyId: company.id });
await notifyChatCompanyAuthorityChange(tx, { companyId: company.id });
await tx.insert(companyMemberships).values({
companyId: company.id,
principalType: "user",
@@ -137,7 +137,7 @@ export async function claimBoardOwnership(
}
if (existing.status !== "active") {
await notifyChatRunMilestones(tx, { companyId: company.id });
await notifyChatCompanyAuthorityChange(tx, { companyId: company.id });
await tx
.update(companyMemberships)
.set({ status: "active", membershipRole: "owner", updatedAt: new Date() })
+10 -1
View File
@@ -1403,6 +1403,7 @@ async function startServerWithDatabaseTeardown(
hasPending: () => caseHandoffDeliveries.hasPending(),
});
let unregisterTerminalWorkspaceProbe: (() => void) | undefined;
let unsubscribeNativeResponsePresentation: (() => void) | undefined;
let unsubscribeNativeWorkspaceSyncRetention: (() => void) | undefined;
const fastResponses = app.locals.fastResponses as ReturnType<typeof import("./services/fast-responses.js").fastResponseService>;
const fastResponseWorker = deliveryWork.register(DELIVERY_QUEUES.fastResponse, {
@@ -1824,6 +1825,13 @@ async function startServerWithDatabaseTeardown(
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) await heartbeat.registerInstructionDirectoryCleanupWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) {
const worker = heartbeat.registerNativeResponsePresentationWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby());
unsubscribeNativeResponsePresentation = worker.stop;
server.on("close", worker.stop);
await worker.ready;
}
if (config.heartbeatSchedulerEnabled) await heartbeat.registerNativePendingEffectWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) await heartbeat.registerUnsafeWorkspaceExportWork(deliveryWork,
@@ -1964,7 +1972,7 @@ async function startServerWithDatabaseTeardown(
// Periodically reap orphaned runs (5-min staleness threshold) and make sure
// persisted queued work is still being driven forward.
runPeriodicRecovery(() => heartbeat
.reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true })
.reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true })
.then(() => heartbeat.promoteDueScheduledRetries())
.then(async (promotion) => {
await heartbeat.resumeQueuedRuns();
@@ -2160,6 +2168,7 @@ async function startServerWithDatabaseTeardown(
await skillVersionRetention.dispose();
await skillSourceSync.stop();
unsubscribeChatCompletions();
unsubscribeNativeResponsePresentation?.();
await deliveryWork.stop();
await workspaceRecoveryWork.stop();
unregisterTerminalWorkspaceProbe?.();
+2 -2
View File
@@ -30,7 +30,7 @@ import { captureRunIdentity } from "../services/run-identity.js";
import { boardAuthService } from "../services/board-auth.js";
import { retryIdempotentDatabaseOperation } from "../database-retry.js";
import { beginIdleTrackedWork } from "../services/task-admission.js";
import { notifyChatRunMilestones } from "../services/chat-run-milestone-notifications.js";
import { notifyChatCompanyAuthorityChange } from "../services/chat-run-milestone-notifications.js";
export {
isTransientDbConnectionError,
@@ -665,7 +665,7 @@ async function resolveCloudTenantActorOnce(
// Serialize authority restoration and its notice while retaining the
// guarded upsert that leaves unchanged memberships untouched.
if (!existing || existing.status !== "active" || existing.membershipRole === "viewer") {
await notifyChatRunMilestones(tx, { companyId });
await notifyChatCompanyAuthorityChange(tx, { companyId });
}
return tx
.insert(companyMemberships)
@@ -673,7 +673,7 @@ export function createPostgresRunDispatchAdapter(
await notifyInstructionCollection(tx, { companyId: input.companyId, runId: input.runId });
await notifyCancelledWorkspacePreservation(tx, { companyId: input.companyId, runId: input.runId });
await notifyScheduledRetryCancellation(tx, { companyId: input.companyId, runId: input.runId });
await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: issueId ?? undefined });
await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: issueId ?? undefined, runId: input.runId });
const [row] = await tx
.update(heartbeatRuns)
.set({
@@ -719,7 +719,7 @@ export function createPostgresRunDispatchAdapter(
await notifyCancelledWorkspacePreservation(tx, { companyId: input.companyId, runId: input.runId });
await notifyScheduledRetryCancellation(tx, { companyId: input.companyId, runId: input.runId });
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: input.issueId ?? undefined });
await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: input.issueId ?? undefined });
await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: input.issueId ?? undefined, runId: input.runId });
const [row] = await tx
.update(heartbeatRuns)
.set({
@@ -1,3 +1,4 @@
import { DELIVERY_QUEUES } from "../../../services/delivery-work-notifications.js";
import { instanceSettingsService } from "../../../services/instance-settings.js";
import { randomBytes, randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
@@ -16,6 +17,7 @@ import {
issueRecoveryActions,
issues,
nativeRunFinalizations,
subscribeDatabaseWork,
} from "@paperclipai/db";
import {
getEmbeddedPostgresTestSupport,
@@ -1222,6 +1224,30 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => {
expect(wakeRow?.payload).not.toHaveProperty("foo");
});
it("scopes a real coalesced-input response wake to its active run after commit", async () => {
const companyId = await seedCompany();
const agentId = await seedAgent({ companyId });
const runId = await seedRun({ companyId, agentId, status: "running" });
const hints: Array<Record<string, unknown>> = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.nativeResponsePresentation, (event, values) => {
if (event === "settled") for (const value of values ?? []) hints.push(JSON.parse(value));
});
try {
await db.transaction(async tx => {
await createWakeAdmissionWriter().coalesceIntoActiveExecutionRun(createAdmissionTransactionScope(companyId, tx as unknown as Db), {
companyId, activeExecutionRunId: runId, mergedContextSnapshot: { commentId: "fresh-input" },
agentId, source: "on_demand", triggerDetail: null, payload: null,
requestedByActorType: "user", requestedByActorId: null, idempotencyKey: null,
});
expect(hints).toEqual([]);
});
expect(hints).toEqual([{ companyId, runId }]);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0].contextSnapshot)
.toMatchObject({ commentId: "fresh-input" });
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.runId, runId))).toHaveLength(1);
} finally { unsubscribe(); }
});
// Review test (b), the coalesce write: a foreign company affects no row
// on the active execution run either.
it("refuses to coalesce into a run for a company that does not own it, and leaves the run and the wake table untouched", async () => {
@@ -956,7 +956,7 @@ export function createWakeAdmissionWriter(): WakeAdmissionWriter {
async coalesceIntoActiveExecutionRun(scope, input) {
const tx = requireAdmissionTx(scope, input.companyId);
const now = new Date();
await notifyChatRunMilestones(tx, { companyId: input.companyId });
await notifyChatRunMilestones(tx, { companyId: input.companyId, runId: input.activeExecutionRunId });
const mergedRun = await tx
.update(heartbeatRuns)
.set({ contextSnapshot: input.mergedContextSnapshot, updatedAt: now })
+7 -7
View File
@@ -1,5 +1,5 @@
import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js";
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { notifyChatCompanyAuthorityChange } from "./chat-run-milestone-notifications.js";
import { notifyVoiceChange } from "./voice/voice-recovery-notifications.js";
import { notifyChatEndpointWork } from "./chat-work-notifications.js";
import { notifyToolConnectionHealth } from "./tool-connection-health-notifications.js";
@@ -558,7 +558,7 @@ export function accessService(db: Db) {
if (existing.principalType === "user" && nextStatus === "active" && nextMembershipRole !== "viewer"
&& (existing.status !== "active" || existing.membershipRole === "viewer")) {
await notifyChatRunMilestones(tx, { companyId });
await notifyChatCompanyAuthorityChange(tx, { companyId });
}
if (
@@ -894,7 +894,7 @@ export function accessService(db: Db) {
const existingMembership = existingByCompany.get(companyId);
if (existingMembership) {
if (existingMembership.status !== "active") {
if (existingMembership.membershipRole !== "viewer") await notifyChatRunMilestones(tx, { companyId });
if (existingMembership.membershipRole !== "viewer") await notifyChatCompanyAuthorityChange(tx, { companyId });
await tx
.update(companyMemberships)
.set({
@@ -906,7 +906,7 @@ export function accessService(db: Db) {
}
continue;
}
await notifyChatRunMilestones(tx, { companyId });
await notifyChatCompanyAuthorityChange(tx, { companyId });
await tx.insert(companyMemberships).values({
companyId,
principalType: "user",
@@ -936,7 +936,7 @@ export function accessService(db: Db) {
if (existing.status !== status || existing.membershipRole !== membershipRole) {
if (principalType === "user" && status === "active" && membershipRole !== "viewer"
&& (existing.status !== "active" || existing.membershipRole === "viewer")) {
await notifyChatRunMilestones(tx, { companyId });
await notifyChatCompanyAuthorityChange(tx, { companyId });
}
const updated = await tx
.update(companyMemberships)
@@ -950,7 +950,7 @@ export function accessService(db: Db) {
}
if (principalType === "user" && status === "active" && membershipRole !== "viewer") {
await notifyChatRunMilestones(tx, { companyId });
await notifyChatCompanyAuthorityChange(tx, { companyId });
}
return tx
.insert(companyMemberships)
@@ -1146,7 +1146,7 @@ export function accessService(db: Db) {
if (existing.principalType === "user" && nextStatus === "active" && nextMembershipRole !== "viewer"
&& (existing.status !== "active" || existing.membershipRole === "viewer")) {
await notifyChatRunMilestones(tx, { companyId });
await notifyChatCompanyAuthorityChange(tx, { companyId });
}
if (
+3 -1
View File
@@ -1,4 +1,5 @@
import { isAgentStatusInvokable } from "@paperclipai/shared";
import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js";
import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js";
import { logger } from "../middleware/logger.js";
import { companies, type Db } from "@paperclipai/db";
@@ -35,8 +36,9 @@ export function createAgentLifecycleEffects(_hooks: BudgetServiceHooks = {}): li
return true;
},
notifyWork: tx => notifyDeliveryWork(tx, DELIVERY_QUEUES.agentLifecycle),
async beforeStatusChange(tx, companyId, _agentId, before, after) {
async beforeStatusChange(tx, companyId, agentId, before, after) {
if (!isAgentStatusInvokable(before) && isAgentStatusInvokable(after)) {
await notifyNativeResponsePresentation(tx, { companyId, agentId });
await notifyChatInboundWakeNotices(tx, { companyId });
}
},
@@ -1,4 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { notifyChatCompanyAuthorityChange } from "../chat-run-milestone-notifications.js";
import { chatConnectionTransport } from "./provider-catalog.js";
import type { ConnectionManagementContext } from "../connection-management.js";
import { slackSetupState } from "./slack/setup.js";
@@ -458,7 +458,7 @@ function providerSetupState(
if ((input.allowDirectMessages === true && !existing.endpoint.allowDirectMessages)
|| (input.allowGroupChats === true && !existing.endpoint.allowGroupChats)
|| (input.allowUnlinkedPeople === true && !existing.endpoint.allowUnlinkedPeople)) {
await notifyChatRunMilestones(tx, { companyId: existing.endpoint.companyId });
await notifyChatCompanyAuthorityChange(tx, { companyId: existing.endpoint.companyId });
}
if (input.executionDefaults !== undefined) {
await assertChatExecutionDefaultsAccess(db, {
@@ -1,3 +1,4 @@
import { notifyNativeResponsePresentation } from "../native-response-presentation-notifications.js";
import { notifyChatInboundWakeNotices } from "../chat-inbound-wake-notice-notifications.js";
import { createFailedRetryNotices } from "./failed-retry-notices.js";
import { safeChatWakeAuthorityReason, type ChatWakeAuthorityReason } from "./inbound/wakeup-authorization-error.js";
@@ -1784,6 +1785,9 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
const settle = async (status: string, result: Record<string, unknown>) => {
const [current] = await db.transaction(async (tx) => {
await notifyChatActionWork(tx, "failed_run_retry");
if (status === "processed") await notifyNativeResponsePresentation(tx, {
companyId: claimed.companyId, issueId: String(claimed.payload.issueId),
});
return tx
.update(chatActions)
.set({ status, result, updatedAt: new Date() })
@@ -1,4 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { notifyChatCompanyAuthorityChange } from "../chat-run-milestone-notifications.js";
import { githubIdentityFields } from "./github/identity-links.js";
import { createSlackIdentityLinks, type SlackIdentityEffectStager } from "./slack/identity-links.js";
import { createHash, randomBytes } from "node:crypto";
@@ -320,7 +320,7 @@ export function createChatIdentityLinks(db: Db, options: ChatIdentityLinkOptions
},
);
}
await notifyChatRunMilestones(tx, { companyId: currentLink.companyId });
await notifyChatCompanyAuthorityChange(tx, { companyId: currentLink.companyId });
const confirmedLink = await tx
.update(chatIdentityLinks)
.set({
@@ -1,4 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { notifyChatRunMilestones, notifyChatCompanyAuthorityChange } from "../chat-run-milestone-notifications.js";
import { notifyChatEndpointConfiguration } from "./configuration-work-notifications.js";
import { notifyChatConnectionHealth } from "./connection-health-notifications.js";
import type { ConnectionManagementContext } from "../connection-management.js";
@@ -287,7 +287,7 @@ export function createChatResourceReconciliation(db: Db, options: ResourceReconc
after: { enabled: finalEnabled.get(row.id)!, ...(JSON.stringify(row.executionDefaults) !== JSON.stringify(finalDefaults.get(row.id)) ? { executionDefaults: finalDefaults.get(row.id) } : {}) },
}));
if (changes.some(change => !change.before.enabled && change.after.enabled)) {
await notifyChatRunMilestones(tx, { companyId: endpoint.companyId });
await notifyChatCompanyAuthorityChange(tx, { companyId: endpoint.companyId });
}
if (bulk) {
// One atomic database update, independent of pagination/search and
@@ -1,4 +1,4 @@
import { notifyChatRunMilestones } from "../../chat-run-milestone-notifications.js";
import { notifyChatCompanyAuthorityChange } from "../../chat-run-milestone-notifications.js";
import { chatIdentityLinks, chatExternalPrincipals, companyMemberships, toolConnections, type Db } from "@paperclipai/db";
import { and, eq, ne, sql } from "drizzle-orm";
import { conflict, forbidden, notFound } from "../../../errors.js";
@@ -47,7 +47,7 @@ export function createSlackSetupServices(db: Db, options: SetupServicesOptions)
const links = await tx.select().from(chatIdentityLinks).where(and(eq(chatIdentityLinks.companyId, record.endpoint.companyId), eq(chatIdentityLinks.principalId, principal.id))).for("update");
if (links.some(link => link.status === "linked" && link.paperclipUserId !== actor.userId) || links.some(link => link.endpointId === endpointId && link.status === "revoked")) throw conflict("This Slack account has an existing link", { code: "chat_identity_link_conflict" });
if (links.some(link => link.endpointId === endpointId && link.status === "linked")) return;
await notifyChatRunMilestones(tx, { companyId: record.endpoint.companyId });
await notifyChatCompanyAuthorityChange(tx, { companyId: record.endpoint.companyId });
await tx.insert(chatIdentityLinks).values({ companyId: record.endpoint.companyId, endpointId, principalId: principal.id,
paperclipUserId: actor.userId, status: "linked", confirmedAt: new Date() })
.onConflictDoUpdate({ target: [chatIdentityLinks.endpointId, chatIdentityLinks.principalId], set: {
@@ -24,7 +24,7 @@ import {
import { HttpError, badRequest, conflict, forbidden, notFound, unprocessable } from "../errors.js";
import { toolAccessService } from "./tool-access.js";
import { logActivity } from "./activity-log.js";
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { notifyChatCompanyAuthorityChange } from "./chat-run-milestone-notifications.js";
import { githubBotRequest } from "./chat-github-client.js";
import { environmentService } from "./environments.js";
import { instanceSettingsService } from "./instance-settings.js";
@@ -165,7 +165,7 @@ export function githubChatManagementService(db: Db, fetchImpl = fetch) {
const allowUnlinkedPeople = config.people.some((p) => p.kind === "guest");
if ((!currentEndpoint.allowUnlinkedPeople && allowUnlinkedPeople) ||
(allowUnlinkedPeople && currentEndpoint.sponsorUserId !== config.responsibleUserId)) {
await notifyChatRunMilestones(tx, { companyId: bot.companyId });
await notifyChatCompanyAuthorityChange(tx, { companyId: bot.companyId });
}
const values = {
companyId: bot.companyId,
@@ -335,7 +335,7 @@ export function githubChatManagementService(db: Db, fetchImpl = fetch) {
updatedAt: new Date(),
};
if (prior?.status !== "linked") {
await notifyChatRunMilestones(tx, { companyId: bot.companyId });
await notifyChatCompanyAuthorityChange(tx, { companyId: bot.companyId });
}
await tx
.insert(chatIdentityLinks)
@@ -1,5 +1,7 @@
import { nativeResponsePresentationWork } from "./native-runtime/native-response-presentation-work.js";
import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
import { randomBytes, randomUUID } from "node:crypto";
import { and, eq } from "drizzle-orm";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import {
agents, agentWakeupRequests, authUsers, chatConversations, chatEndpoints, chatMessageLinks,
@@ -16,14 +18,18 @@ import { createHeartbeatQueueTestDependencies } from "../__tests__/helpers/heart
import { createHeartbeatQueue } from "./heartbeat/queue.js";
import { instanceSettingsService } from "./instance-settings.js";
const support = await getEmbeddedPostgresTestSupport();
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_MONITOR_CONTINUATION_TEST_DATABASE_URL");
const support = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
if (!support.supported) console.warn(`Monitor publication database unavailable: ${support.reason}`);
(support.supported ? describe : describe.skip)("monitor conversation continuation", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: ReturnType<typeof createDb>;
beforeAll(async () => {
temporary = await startEmbeddedPostgresTestDatabase("paperclip-monitor-publications-");
db = createDb(temporary.connectionString);
if (external) db = createDb(external);
else {
temporary = await startEmbeddedPostgresTestDatabase("paperclip-monitor-publications-");
db = createDb(temporary.connectionString);
}
await db.insert(authUsers).values({ id: "requester", name: "Requester", email: "requester@example.test", createdAt: new Date(), updatedAt: new Date() });
});
afterAll(async () => { await temporary?.cleanup(); });
@@ -235,7 +241,7 @@ if (!support.supported) console.warn(`Monitor publication database unavailable:
expect(await f.bindings()).toEqual([]);
});
it("recovers the accepted continuation explanation after interrupted finalization without another provider result", async () => {
it.each([false, true])("recovers the accepted continuation explanation without another provider result (missing presentation: %s)", async missingPresentation => {
const f = await fixture();
const executionPayload = randomBytes(128 * 1024).toString("base64");
await db.update(heartbeatRuns).set({
@@ -281,6 +287,20 @@ if (!support.supported) console.warn(`Monitor publication database unavailable:
expect(settledRun.stdoutExcerpt).toBe(executionPayload);
const priorComments = await db.select({ body: issueComments.body }).from(issueComments).where(eq(issueComments.createdByRunId, f.source.id));
expect(priorComments).toEqual([{ body: summary }]);
expect(await nativeResponsePresentationWork(db).load({ companyId: f.company.id, runId: f.source.id })).toEqual([]);
if (missingPresentation) {
await db.transaction(async tx => {
await tx.delete(chatPublications).where(eq(chatPublications.issueId, f.issue.id));
await tx.delete(issueComments).where(eq(issueComments.createdByRunId, f.source.id));
await tx.update(heartbeatRuns).set({ resultJson: sql`${heartbeatRuns.resultJson} - 'presentationDecision'` })
.where(eq(heartbeatRuns.id, f.source.id));
});
const work = nativeResponsePresentationWork(db);
const entries = await work.load({ companyId: f.company.id, runId: f.source.id });
expect(entries).toHaveLength(1);
await expect(work.check(entries[0])).resolves.toBeNull();
expect(await work.load({ companyId: f.company.id, runId: f.source.id })).toEqual([]);
}
expect(await repair()).toBe(false);
expect(await repair()).toBe(false);
const comments = await db.select().from(issueComments).where(eq(issueComments.createdByRunId, f.source.id));
@@ -1,3 +1,4 @@
import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js";
import { notifyChatResponseAvailability } from "./chat-response-availability.js";
import { notifyChatConfirmationWork } from "./chat-confirmation-notifications.js";
import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js";
@@ -6,9 +7,13 @@ import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notificatio
/** Lineage changes can authorize descendants, so invalidate the issue rather
* than just the run whose row changed. Configuration may invalidate a company. */
export type ChatRunMilestoneScope = { companyId: string; issueId?: string };
export type ChatRunMilestoneScope = { companyId: string; issueId?: string; runId?: string };
export async function notifyChatRunMilestones(tx: object, scope: ChatRunMilestoneScope) {
if (scope.issueId || scope.runId) {
await notifyNativeResponsePresentation(tx, { companyId: scope.companyId, issueId: scope.issueId,
runId: scope.issueId ? undefined : scope.runId });
}
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatGitHubReviewChecks, JSON.stringify({ companyId: scope.companyId }));
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatGitHubTerminalResponses, JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId }));
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatRunMilestones, JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId }));
@@ -21,3 +26,8 @@ export async function notifyChatRunMilestones(tx: object, scope: ChatRunMileston
// Native presentation and run ownership can unblock a parked acknowledgement.
await notifyChatResponseAvailability(tx, scope);
}
export async function notifyChatCompanyAuthorityChange(tx: object, scope: { companyId: string }) {
await notifyChatRunMilestones(tx, scope);
await notifyNativeResponsePresentation(tx, scope);
}
@@ -1,5 +1,5 @@
import { notifyChatResponseAvailability } from "./chat-response-availability.js";
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { notifyChatRunMilestones, notifyChatCompanyAuthorityChange } from "./chat-run-milestone-notifications.js";
import { notifyChatConnectionConfiguration } from "./chat-channels/configuration-work-notifications.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
@@ -47,7 +47,7 @@ export async function notifyChatVerificationWork(tx: object): Promise<void> {
export async function notifyChatEndpointWork(tx: object, connectionId: string, companyId: string): Promise<void> {
// Configuration/status transitions can restore post-control publication
// authority. Ordinary health checks use the narrower configuration signals.
await notifyChatRunMilestones(tx, { companyId });
await notifyChatCompanyAuthorityChange(tx, { companyId });
await notifyChatDeliveryWork(tx);
await notifyChatPublicationWork(tx);
for (const queue of Object.values(CHAT_ACTION_QUEUES)) await notifyDeliveryWork(tx, queue);
@@ -2,6 +2,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db";
/** Topics for existing durable queues; register intent before their writes. */
export const DELIVERY_QUEUES = {
nativeResponsePresentation: "native-response-presentation",
nativePendingEffects: "native-pending-effects",
unsafeWorkspaceExportRecovery: "unsafe-workspace-export-recovery",
automaticCompletionReviews: "automatic-completion-reviews",
@@ -21,6 +21,21 @@ const committedChatResponseAuthorities = new WeakMap<
CommittedChatResponseAuthority
>();
const committedChatResponseAuthorityListeners = new WeakMap<Db, Set<() => void>>();
export function onCommittedChatResponseAuthorityRegistered(db: Db, listener: () => void): () => void {
let listeners = committedChatResponseAuthorityListeners.get(db);
if (!listeners) {
listeners = new Set();
committedChatResponseAuthorityListeners.set(db, listeners);
}
listeners.add(listener);
return () => {
listeners.delete(listener);
if (listeners.size === 0 && committedChatResponseAuthorityListeners.get(db) === listeners) committedChatResponseAuthorityListeners.delete(db);
};
}
export class CommittedChatResponseAuthorizationError extends HttpError {
constructor() {
super(
@@ -41,6 +56,7 @@ export function registerCommittedChatResponseAuthority(
authority: CommittedChatResponseAuthority,
): () => void {
committedChatResponseAuthorities.set(db, authority);
for (const listener of committedChatResponseAuthorityListeners.get(db) ?? []) listener();
return () => {
if (committedChatResponseAuthorities.get(db) === authority) {
committedChatResponseAuthorities.delete(db);
@@ -1,4 +1,4 @@
import { notifyNativePendingEffects } from "./native-pending-effect-notifications.js";
import { notifyNativeFinalizationWork } from "./native-finalization-notifications.js";
import { notifyUnsafeWorkspaceExportRunChange } from "./unsafe-workspace-export-notifications.js";
import { notifyCancelledWorkspacePreservation } from "./cancelled-workspace-preservation-notifications.js";
import { notifyInstructionCollection } from "./instruction-collection-notifications.js";
@@ -147,7 +147,7 @@ export async function reconcileAbandonedExecutionControl(
}
// A persisted result belongs to the existing finalizer, not a replacement provider.
if (coordinator?.resultId) {
await notifyNativePendingEffects(tx, { companyId: run.companyId, runId: run.id });
await notifyNativeFinalizationWork(tx, { companyId: run.companyId, runId: run.id });
await tx
.update(nativeRunFinalizations)
.set({
+8 -2
View File
@@ -1,3 +1,4 @@
import { isNativeResponseEvidence, notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js";
import { isSafeNativeChatProgressEventType } from "./safe-native-chat-progress.js";
import { notifyChatNativeProgress } from "./chat-native-progress-notifications.js";
import { boundRunIssueIdSql } from "./run-task-binding.js";
@@ -72,7 +73,8 @@ export async function appendHeartbeatRunEvent(
input: AppendHeartbeatRunEventInput,
): Promise<AppendHeartbeatRunEventResult> {
const projectsProgress = isSafeNativeChatProgressEventType(input.eventType);
if (!input.nativeSource && !input.retryExhaustion && !projectsProgress) {
const projectsResponse = isNativeResponseEvidence(input.eventType, input.payload?.prpEvent);
if (!input.nativeSource && !input.retryExhaustion && !projectsProgress && !projectsResponse) {
const allocated = db.$with("allocated_run_event", {}).as(sql`
update ${heartbeatRuns}
set next_event_seq = ${heartbeatRuns.nextEventSeq} + 1,
@@ -101,7 +103,8 @@ export async function appendHeartbeatRunEvent(
.select({ companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId,
// Only progress needs the task binding. Other native/recovery events
// must not read a potentially large context snapshot to append a log.
...(projectsProgress ? { runtimeMode: heartbeatRuns.runtimeMode, issueId: boundRunIssueIdSql() } : {}) })
...(projectsProgress || projectsResponse ? { runtimeMode: heartbeatRuns.runtimeMode } : {}),
...(projectsProgress ? { issueId: boundRunIssueIdSql() } : {}) })
.from(heartbeatRuns)
.where(eq(heartbeatRuns.id, input.runId))
.for("update")
@@ -175,6 +178,9 @@ export async function appendHeartbeatRunEvent(
if (projectsProgress && run.runtimeMode === "native" && run.issueId) {
await notifyChatNativeProgress(tx, { companyId: run.companyId, issueId: run.issueId });
}
if (projectsResponse && run.runtimeMode === "native") {
await notifyNativeResponsePresentation(tx, { companyId: run.companyId, runId: input.runId });
}
const seq = await allocateHeartbeatRunEventSeq(
tx as unknown as Db,
input.runId,
+3
View File
@@ -1,3 +1,4 @@
import { registerNativeResponsePresentationWork } from "./native-runtime/native-response-presentation-work.js";
import { registerNativePendingEffectWork } from "./native-runtime/native-pending-effect-work.js";
import { registerUnsafeWorkspaceExportWork } from "./native-runtime/native-workspace-export-recovery.js";
import { registerAutomaticCompletionReviewWork } from "./native-runtime/automatic-completion-reviews.js";
@@ -4089,6 +4090,8 @@ export function heartbeatService(
triggerIssueMonitor,
reportRunActivity: clearDetachedRunWarning,
registerNativeResponsePresentationWork: (coordinator: Parameters<typeof registerNativeResponsePresentationWork>[1], canRun: () => boolean) =>
registerNativeResponsePresentationWork(db, coordinator, () => canRun() && readTaskDrain(new Date()) === null),
registerNativePendingEffectWork: (coordinator: Parameters<typeof registerNativePendingEffectWork>[1], canRun: () => boolean) =>
registerNativePendingEffectWork(db, coordinator, () => canRun() && readTaskDrain(new Date()) === null),
registerUnsafeWorkspaceExportWork: (coordinator: Parameters<typeof registerUnsafeWorkspaceExportWork>[1], canRun: () => boolean) =>
+4 -3
View File
@@ -976,7 +976,8 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.status, "queued")))
.for("update");
if (!currentRun) return null;
await notifyChatRunMilestones(tx, { companyId: run.companyId });
await notifyChatRunMilestones(tx, { companyId: run.companyId, runId: run.id,
issueId: run.nativeIssueId ?? run.issueId ?? readNonEmptyString(parseObject(run.contextSnapshot).issueId) ?? undefined });
await notifyInstructionCollection(tx, { companyId: run.companyId, runId: run.id });
await notifyCancelledWorkspacePreservation(tx, { companyId: run.companyId, runId: run.id });
await notifyUnsafeWorkspaceExportRunChange(tx, run);
@@ -1625,7 +1626,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
)).for("update");
if (!lockedRun || lockedRun.status !== "queued") return null;
await withExternalAdmissionGuard(claimTx, run.companyId, run.agentId, () => assertNoExternalOverlap(claimTx, run.companyId, run.agentId));
await notifyChatRunMilestones(claimTx, { companyId: run.companyId, issueId: issueId ?? undefined });
await notifyChatRunMilestones(claimTx, { companyId: run.companyId, runId: run.id, issueId: issueId ?? undefined });
const claimedRun = await claimTx.update(heartbeatRuns).set(claimValues).where(and(
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.status, "queued"),
)).returning().then((rows) => rows[0] ?? null);
@@ -1726,7 +1727,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
const [candidate] = await tx.select({ companyId: heartbeatRuns.companyId, contextSnapshot: heartbeatRuns.contextSnapshot })
.from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.status, "running"))).for("update");
if (!candidate) return;
await notifyChatRunMilestones(tx, { companyId: candidate.companyId,
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, runId,
issueId: readNonEmptyString(parseObject(candidate.contextSnapshot).issueId) ?? undefined });
const released = await tx
.update(heartbeatRuns)
+2 -1
View File
@@ -1947,7 +1947,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
let settledProcessesDiscovered = false;
async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean }) {
async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean; skipNativeResponsePresentation?: boolean }) {
if (!opts?.skipCancelledWorkspacePreservation) await preserveCancelledNativeWorkspaces(db);
const staleThresholdMs = opts?.staleThresholdMs ?? 0;
const now = new Date();
@@ -1969,6 +1969,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
skipAutomaticCompletionReviews: opts?.skipAutomaticCompletionReviews,
skipUnsafeWorkspaceExports: opts?.skipUnsafeWorkspaceExports,
skipNativePendingEffects: opts?.skipNativePendingEffects,
skipNativeResponsePresentation: opts?.skipNativeResponsePresentation,
environmentRuntime,
pluginWorkerManager: dependencies.pluginWorkerManager,
onWorkspaceSettled: settleRecoveredNativeWorkspace,
@@ -351,7 +351,7 @@ export async function recordRunPresentationDecision(db: Db, input: {
companyId: string; runId: string; issueId?: string; decision: RunPresentationDecision;
}) {
await db.transaction(async tx => {
await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: input.issueId });
await notifyChatRunMilestones(tx, { companyId: input.companyId, runId: input.runId, issueId: input.issueId });
await tx.update(heartbeatRuns).set({
resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({ presentationDecision: input.decision })}::jsonb`,
updatedAt: new Date(),
@@ -1,7 +1,7 @@
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
import { randomBytes } from "node:crypto";
import type { Db } from "@paperclipai/db";
import { agents, agentWakeupRequests, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRunEvents, heartbeatRuns } from "@paperclipai/db";
import { agents, agentWakeupRequests, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRunEvents, heartbeatRuns, issues, subscribeDatabaseWork } from "@paperclipai/db";
import { eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
@@ -241,6 +241,23 @@ describe.skipIf(!support.supported)("heartbeat lifecycle database wiring", () =>
} finally { unsubscribe(); }
});
it.each(["native", "legacy", "unbound"] as const)("keeps %s run-status response recovery scoped to its task or run", async binding => {
const { run } = await fixture();
const [issue] = await db.insert(issues).values({ companyId: run.companyId, title: "Scope owner" }).returning();
await db.update(heartbeatRuns).set(binding === "native" ? { nativeIssueId: issue.id }
: binding === "legacy" ? { contextSnapshot: { issueId: issue.id } } : { contextSnapshot: {} }).where(eq(heartbeatRuns.id, run.id));
const hints: unknown[] = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.nativeResponsePresentation, (event, scopes) => {
if (event === "settled") for (const hint of scopes ?? []) hints.push(JSON.parse(hint));
});
try {
await createHeartbeatLifecycle(db, callbacks(db)).setRunStatusIfRunning(run.id, "succeeded", { finishedAt: new Date() });
expect(hints).toEqual([binding === "unbound" ? { companyId: run.companyId, runId: run.id }
: { companyId: run.companyId, issueId: issue.id }]);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].status).toBe("succeeded");
} finally { unsubscribe(); }
});
it("does not request accounting for an unchanged running status", async () => {
const { run } = await fixture();
const settled = vi.fn();
@@ -1017,7 +1017,7 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl
return recordHeartbeatRunRuntimeProgress(currentRun, update, issueId);
}
async function writeRunStatus<T>(status: string, previous: Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "status" | "nativeIssueId" | "contextSnapshot"> | null,
async function writeRunStatus<T>(status: string, previous: Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "status" | "nativeIssueId" | "issueId" | "contextSnapshot"> | null,
write: (tx: Db) => Promise<T>) {
const companyId = previous?.companyId;
// Register on the outer transaction before the terminal write. An unknown
@@ -1031,7 +1031,8 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl
if (previous) await notifyCancelledWorkspacePreservation(tx, { companyId: previous.companyId, runId: previous.id });
if (isHeartbeatRunTerminalStatus(status)) await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, companyId);
await notifyExecutionRunWork(tx);
if (companyId) await notifyChatRunMilestones(tx, { companyId });
if (previous) await notifyChatRunMilestones(tx, { companyId: previous.companyId, runId: previous.id,
issueId: previous.nativeIssueId ?? previous.issueId ?? readNonEmptyString(parseObject(previous.contextSnapshot).issueId) ?? undefined });
return write(tx);
});
}
@@ -0,0 +1,10 @@
import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
export type NativeFinalizationWorkScope = { companyId: string; runId?: string; issueId?: string };
export async function notifyNativeFinalizationWork(transaction: object, scope: NativeFinalizationWorkScope) {
if (scope.issueId || scope.runId) await notifyNativeResponsePresentation(transaction, scope);
await notifyDeliveryWork(transaction, DELIVERY_QUEUES.nativePendingEffects,
JSON.stringify({ companyId: scope.companyId, runId: scope.runId }));
}
@@ -1,8 +0,0 @@
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
export type NativePendingEffectScope = { companyId: string; runId?: string };
export async function notifyNativePendingEffects(transaction: object, scope: NativePendingEffectScope) {
await notifyDeliveryWork(transaction, DELIVERY_QUEUES.nativePendingEffects,
JSON.stringify({ companyId: scope.companyId, runId: scope.runId }));
}
@@ -0,0 +1,14 @@
import { readCompletedAssistantMessageCandidate } from "./heartbeat-run-summary.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
export type NativeResponsePresentationScope = { companyId: string; issueId?: string; runId?: string; agentId?: string };
export async function notifyNativeResponsePresentation(tx: object, scope: NativeResponsePresentationScope) {
await notifyDeliveryWork(tx, DELIVERY_QUEUES.nativeResponsePresentation,
JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId, runId: scope.runId, agentId: scope.agentId }));
}
export function isNativeResponseEvidence(eventType: string, prpEvent: unknown): boolean {
return eventType === "run.result.accepted" || eventType === "item.completed" &&
readCompletedAssistantMessageCandidate({ seq: 0, prpEvent })?.channel === "final";
}
@@ -644,6 +644,7 @@ type NativeReconciliationOptions = {
skipAutomaticCompletionReviews?: boolean;
skipUnsafeWorkspaceExports?: boolean;
skipNativePendingEffects?: boolean;
skipNativeResponsePresentation?: boolean;
pluginWorkerManager?: WorkspaceWorkers;
environmentRuntime?: EnvironmentRuntimeService;
onWorkspaceSettled?: (input: {
@@ -861,7 +862,7 @@ async function reconcileNativeFinalizationsPass(
)).limit(1).then((entries) => entries[0] ?? null)
: null;
const currentDecisionJson = record(currentDecision?.decisionJson);
if (row.coordinatorPhase === "committed") {
if (row.coordinatorPhase === "committed" && !options.skipNativeResponsePresentation) {
// A later decision can supersede this run's task status, but cannot
// erase its accepted, still-authorized response. Repair presentation
// before the status-only early return below, without rerunning work.
@@ -1123,6 +1124,7 @@ async function reconcileNativeFinalizationsPass(
runId: row.runId,
workspaceFinalizeStatus,
projectRunStatus: true,
skipNativeResponsePresentation: options.skipNativeResponsePresentation,
preserveProviderAttempt: Boolean(
readNativeWorkspaceSyncReference(
row.nativeWorkspaceSync,
@@ -1181,6 +1183,7 @@ async function reconcileNativeFinalizationsPass(
runId: row.runId,
workspaceFinalizeStatus: barrier.status as "succeeded" | "failed",
projectRunStatus: true,
skipNativeResponsePresentation: options.skipNativeResponsePresentation,
preserveProviderAttempt: Boolean(
readNativeWorkspaceSyncReference(
row.nativeWorkspaceSync,
@@ -1,4 +1,4 @@
import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js";
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js";
import type { Db } from "@paperclipai/db";
import type { NativeInteractionResponseEnvelope } from "../../vendor/paperclip-runner/index.js";
@@ -633,7 +633,7 @@ export async function routePersistedNativeResultAttention(input: {
const delegation = effects.find((effect) => effect.kind === "create_delegated_issue");
const finalizationError = effects.find((effect) => effect.kind === "record_finalization_error");
await input.db.transaction(async tx => {
await notifyNativePendingEffects(tx, { companyId: binding.companyId, runId: input.runId });
await notifyNativeFinalizationWork(tx, { companyId: binding.companyId, runId: input.runId });
await tx.update(nativeRunFinalizations).set({
phase: finalizationError ? "retryable_failure" : "committed",
assessmentId: existingAssessment.id,
@@ -2,7 +2,7 @@ import { and, asc, eq, exists, gt, inArray, isNotNull, ne } from "drizzle-orm";
import { heartbeatRuns, issues, nativeRunFinalizations, statusDecisionEffects, type Db } from "@paperclipai/db";
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
import type { NativePendingEffectScope } from "../native-pending-effect-notifications.js";
import type { NativeFinalizationWorkScope } from "../native-finalization-notifications.js";
import { registerScopedDeadlineWork } from "../scoped-deadline-work.js";
import { dispatchPendingNativeStatusEffects, nativePendingEffectDueAt } from "./status-decision-committer.js";
@@ -15,7 +15,7 @@ export function pendingNativeStatusEffectsCondition(db: Db) {
)));
}
function findPendingEffects(db: Db, scope?: NativePendingEffectScope, after?: string) {
function findPendingEffects(db: Db, scope?: NativeFinalizationWorkScope, after?: string) {
return db.select({
id: nativeRunFinalizations.runId, companyId: nativeRunFinalizations.companyId,
issueId: nativeRunFinalizations.issueId, decisionId: nativeRunFinalizations.decisionId,
@@ -36,7 +36,7 @@ function findPendingEffects(db: Db, scope?: NativePendingEffectScope, after?: st
export function registerNativePendingEffectWork(db: Db, coordinator: ReturnType<typeof createDeliveryWorkCoordinator>, canRun: () => boolean) {
type Entry = { id: string; companyId: string; dueAt: number };
async function load(scope?: NativePendingEffectScope, signal?: AbortSignal): Promise<Entry[]> {
async function load(scope?: NativeFinalizationWorkScope, signal?: AbortSignal): Promise<Entry[]> {
const entries: Entry[] = [];
let after: string | undefined;
while (canRun() && !signal?.aborted) {
@@ -49,8 +49,8 @@ export function registerNativePendingEffectWork(db: Db, coordinator: ReturnType<
}
return registerScopedDeadlineWork(coordinator, {
queue: DELIVERY_QUEUES.nativePendingEffects, batchSize: 25,
decodeScope: (hint: string): NativePendingEffectScope | undefined => hint === "*" ? undefined : JSON.parse(hint),
matches: (entry: Entry, scope?: NativePendingEffectScope) => !scope || entry.companyId === scope.companyId && (!scope.runId || entry.id === scope.runId),
decodeScope: (hint: string): NativeFinalizationWorkScope | undefined => hint === "*" ? undefined : JSON.parse(hint),
matches: (entry: Entry, scope?: NativeFinalizationWorkScope) => !scope || entry.companyId === scope.companyId && (!scope.runId || entry.id === scope.runId),
load,
check: async (entry: Entry, signal?: AbortSignal) => {
if (!canRun() || signal?.aborted) throw new Error("Native pending effect recovery stopped");
@@ -0,0 +1,117 @@
import { and, asc, eq, gt, inArray, isNotNull, isNull, notExists, or, sql } from "drizzle-orm";
import { heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, statusDecisions, statusDecisionEffects, type Db } from "@paperclipai/db";
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
import type { NativeResponsePresentationScope } from "../native-response-presentation-notifications.js";
import { onCommittedChatResponseAuthorityRegistered } from "../durable-chat-wakeup.js";
import { registerScopedDeadlineWork } from "../scoped-deadline-work.js";
import { recoverCommittedNativeChatResponse } from "./native-run-finalizer.js";
export function registerNativeResponsePresentationWork(db: Db,
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>, canRun: () => boolean) {
return nativeResponsePresentationWork(db).register(coordinator, canRun);
}
type Entry = { id: string; companyId: string; issueId: string; agentId: string; dueAt: number };
export function nativeResponsePresentationWork(db: Db) {
function candidates(scope?: NativeResponsePresentationScope, after?: string, runId?: string) {
return db.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId,
issueId: nativeRunFinalizations.issueId, agentId: heartbeatRuns.agentId, nextAttemptAt: nativeRunFinalizations.nextAttemptAt,
}).from(nativeRunFinalizations)
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, nativeRunFinalizations.runId),
eq(heartbeatRuns.companyId, nativeRunFinalizations.companyId), eq(heartbeatRuns.nativeIssueId, nativeRunFinalizations.issueId)))
.innerJoin(issues, and(eq(issues.id, nativeRunFinalizations.issueId), eq(issues.companyId, nativeRunFinalizations.companyId)))
.innerJoin(nativeRunResults, and(eq(nativeRunResults.id, nativeRunFinalizations.resultId),
eq(nativeRunResults.companyId, nativeRunFinalizations.companyId), eq(nativeRunResults.issueId, nativeRunFinalizations.issueId),
eq(nativeRunResults.runId, heartbeatRuns.id), eq(nativeRunResults.completionContractId, heartbeatRuns.completionContractId)))
.innerJoin(statusDecisions, and(eq(statusDecisions.id, nativeRunFinalizations.decisionId),
eq(statusDecisions.companyId, nativeRunFinalizations.companyId), eq(statusDecisions.issueId, nativeRunFinalizations.issueId),
eq(statusDecisions.runId, heartbeatRuns.id), eq(statusDecisions.assessmentId, nativeRunFinalizations.assessmentId)))
.where(and(
eq(heartbeatRuns.runtimeMode, "native"), inArray(heartbeatRuns.status, ["succeeded", "failed", "timed_out"]),
isNotNull(heartbeatRuns.finishedAt), eq(nativeRunFinalizations.phase, "committed"), isNull(nativeRunFinalizations.leaseOwner),
eq(nativeRunResults.schemaStatus, "accepted"),
sql`jsonb_typeof(${heartbeatRuns.resultJson} #> '{presentationDecision,commentId}') is distinct from 'string'`,
sql`coalesce(${heartbeatRuns.resultJson} #> '{presentationDecision,commentAction}', '"create"'::jsonb) = '"create"'::jsonb`,
sql`${heartbeatRuns.contextSnapshot}->'skipIssueComment' is distinct from 'true'::jsonb`,
sql`coalesce(${statusDecisions.decisionJson}->'externalChatReviewPresentation', 'null'::jsonb) in ('null'::jsonb, 'false'::jsonb, '0'::jsonb, '""'::jsonb)`,
sql`${nativeRunResults.resultJson} #>> '{result,schema}' = 'paperclip.run_result.v1'`,
sql`${nativeRunResults.resultJson} #>> '{terminal,runTerminalState}' = 'succeeded'`,
sql`${nativeRunResults.resultJson} #>> '{terminal,turnTerminalState}' = 'completed'`,
or(
and(eq(heartbeatRuns.status, "succeeded"),
sql`(coalesce(${issues.conversationAgentId}::text, '') = '' or coalesce(${issues.conversationUserId}, '') = '')`,
sql`${nativeRunResults.resultJson} #>> '{result,reportedWorkDisposition}' = 'done'`,
sql`${nativeRunResults.resultJson} #>> '{terminal,reportedWorkDisposition}' = 'done'`),
and(sql`${nativeRunResults.resultJson} #>> '{result,reportedWorkDisposition}' = 'yielded'`,
sql`${nativeRunResults.resultJson} #>> '{terminal,reportedWorkDisposition}' = 'yielded'`,
sql`jsonb_typeof(${nativeRunResults.resultJson} #> '{result,summary}') = 'string'
and ${nativeRunResults.resultJson} #>> '{result,summary}' ~ '[^[:space:]]'`,
sql`jsonb_typeof(${nativeRunResults.resultJson} #> '{result,continuation,idempotencyKey}') = 'string'
and ${nativeRunResults.resultJson} #>> '{result,continuation,idempotencyKey}' ~ '[^[:space:]]'`,
or(
and(sql`${nativeRunResults.resultJson} #>> '{result,continuation,kind}' = 'response_wake'`,
sql`${nativeRunResults.resultJson} #> '{result,attentionRequests}' = '[]'::jsonb`,
sql`jsonb_typeof(${nativeRunResults.resultJson} #> '{result,continuation,summary}') = 'string'
and ${nativeRunResults.resultJson} #>> '{result,continuation,summary}' ~ '[^[:space:]]'`),
and(eq(heartbeatRuns.status, "succeeded"), or(
and(sql`${nativeRunResults.resultJson} #>> '{result,continuation,kind}' = 'same_agent'`,
eq(statusDecisions.reasonCode, "live_continuation_registered")),
and(sql`${nativeRunResults.resultJson} #>> '{result,continuation,kind}' = 'monitor'`,
inArray(statusDecisions.reasonCode, ["scheduled_monitor_waiting", "scheduled_monitor_dispatched", "scheduled_monitor_preserved"])),
)),
)),
),
notExists(db.select({ id: statusDecisionEffects.id }).from(statusDecisionEffects).where(and(
eq(statusDecisionEffects.companyId, nativeRunFinalizations.companyId),
eq(statusDecisionEffects.issueId, nativeRunFinalizations.issueId),
eq(statusDecisionEffects.decisionId, nativeRunFinalizations.decisionId),
inArray(statusDecisionEffects.deliveryState, ["pending", "failed"]),
))),
scope ? and(eq(heartbeatRuns.companyId, scope.companyId), scope.issueId ? eq(nativeRunFinalizations.issueId, scope.issueId) : undefined) : undefined,
scope?.runId ? eq(heartbeatRuns.id, scope.runId) : undefined,
scope?.agentId ? eq(heartbeatRuns.agentId, scope.agentId) : undefined,
after ? gt(heartbeatRuns.id, after) : undefined, runId ? eq(heartbeatRuns.id, runId) : undefined,
)).orderBy(asc(heartbeatRuns.id)).limit(100);
}
async function load(scope?: NativeResponsePresentationScope, signal?: AbortSignal, canRun = () => true): Promise<Entry[]> {
const entries: Entry[] = [];
let after: string | undefined;
while (canRun() && !signal?.aborted) {
const rows = await candidates(scope, after);
for (const row of rows) entries.push({ id: row.id, companyId: row.companyId, issueId: row.issueId, agentId: row.agentId, dueAt: row.nextAttemptAt?.getTime() ?? 0 });
if (rows.length < 100) return entries;
after = rows[rows.length - 1].id;
}
throw new Error("Native response presentation discovery stopped");
}
async function check(entry: Entry, signal?: AbortSignal): Promise<Entry | null> {
if (signal?.aborted) throw new Error("Native response presentation recovery stopped");
const scope = { companyId: entry.companyId, issueId: entry.issueId };
const [current] = await candidates(scope, undefined, entry.id);
if (!current) return null;
if (current.nextAttemptAt && current.nextAttemptAt.getTime() > Date.now()) return { ...entry, dueAt: current.nextAttemptAt.getTime() };
if (signal?.aborted) throw new Error("Native response presentation recovery stopped");
const recoveryStartedAt = Date.now();
await recoverCommittedNativeChatResponse(db, { ...scope, runId: entry.id, onlyWhenDue: true });
const [remaining] = await candidates(scope, undefined, entry.id);
return remaining?.nextAttemptAt && remaining.nextAttemptAt.getTime() > recoveryStartedAt
? { ...entry, dueAt: remaining.nextAttemptAt.getTime() } : null;
}
function register(coordinator: ReturnType<typeof createDeliveryWorkCoordinator>, canRun: () => boolean) {
const worker = registerScopedDeadlineWork(coordinator, {
queue: DELIVERY_QUEUES.nativeResponsePresentation,
decodeScope: (hint: string): NativeResponsePresentationScope | undefined => hint === "*" ? undefined : JSON.parse(hint),
matches: (entry: Entry, scope?: NativeResponsePresentationScope) => !scope || entry.companyId === scope.companyId && (!scope.issueId || entry.issueId === scope.issueId) && (!scope.runId || entry.id === scope.runId) && (!scope.agentId || entry.agentId === scope.agentId),
load: (scope, signal) => load(scope, signal, canRun),
check: (entry, signal) => {
if (!canRun()) throw new Error("Native response presentation recovery paused");
return check(entry, signal);
},
}, canRun);
const unsubscribe = onCommittedChatResponseAuthorityRegistered(db, () => worker.invalidate());
return { ...worker, stop: unsubscribe };
}
return { load, check, register };
}
@@ -1,4 +1,5 @@
import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js";
import { isNativeResponseEvidence, notifyNativeResponsePresentation } from "../native-response-presentation-notifications.js";
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js";
import { isSafeNativeChatProgressEventType } from "../safe-native-chat-progress.js";
import { notifyChatNativeProgress } from "../chat-native-progress-notifications.js";
@@ -297,6 +298,9 @@ export class NativeRunCoordinatorStore {
if (isSafeNativeChatProgressEventType(event.eventType)) {
await notifyChatNativeProgress(tx, { companyId: this.#binding.companyId, issueId: this.#binding.issueId });
}
if (isNativeResponseEvidence(event.eventType, canonicalPayload)) {
await notifyNativeResponsePresentation(tx, this.#binding);
}
const cursor = run.nextEventSeq;
const [inserted] = await tx
.insert(heartbeatRunEvents)
@@ -523,7 +527,7 @@ export class NativeRunCoordinatorStore {
) {
throw new Error("native_finalization_lease_conflict");
}
if (row.decisionId) await notifyNativePendingEffects(tx, this.#binding);
if (row.decisionId) await notifyNativeFinalizationWork(tx, this.#binding);
if (
row.leaseOwner === input.leaseOwner &&
row.leaseExpiresAt &&
@@ -566,7 +570,7 @@ export class NativeRunCoordinatorStore {
throw new Error("native_finalization_retry_invalid");
}
const [updated] = await this.#db.transaction(async tx => {
await notifyNativePendingEffects(tx, this.#binding);
await notifyNativeFinalizationWork(tx, this.#binding);
if (input.failureCode === "native_workspace_sync_out_unsafe_archive") {
await notifyUnsafeWorkspaceExportRecovery(tx, this.#binding);
}
@@ -1,4 +1,4 @@
import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js";
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
import { notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js";
import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js";
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
@@ -39,6 +39,7 @@ import {
nativeRunFinalizations,
nativeRunResults,
statusDecisions,
statusDecisionEffects,
workAssessments,
workspaceOperations,
} from "@paperclipai/db";
@@ -215,7 +216,7 @@ async function claimCoordinator(input: {
coordinator.leaseOwner !== leaseOwner
)
throw new Error("native_finalization_lease_busy");
if (coordinator.decisionId) await notifyNativePendingEffects(tx, coordinator);
if (coordinator.decisionId) await notifyNativeFinalizationWork(tx, coordinator);
const [updated] = await tx
.update(nativeRunFinalizations)
.set({
@@ -383,7 +384,7 @@ async function recordRetryableFailure(input: {
? "native_workspace_sync_out_retry_exhausted"
: "native_finalization_retry_exhausted"
: input.failureCode;
await notifyNativePendingEffects(tx, { companyId: input.run.companyId, runId: input.run.id });
await notifyNativeFinalizationWork(tx, { companyId: input.run.companyId, runId: input.run.id });
if (phase === "terminal_failure") await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, input.run.companyId);
if (phase === "terminal_failure" && failureCode === "native_workspace_sync_out_unsafe_archive") {
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.run.companyId, issueId: input.coordinator.issueId });
@@ -848,6 +849,21 @@ async function acceptedResponseDigestMatches(
export async function repairCommittedNativeChatResponse(
db: Db,
input: { companyId: string; issueId: string; runId: string },
): Promise<boolean> {
try {
return await recoverCommittedNativeChatResponse(db, input);
} catch (error) {
logger.warn(
{ runId: input.runId, error },
"Committed chat response is awaiting presentation retry",
);
return false;
}
}
export async function recoverCommittedNativeChatResponse(
db: Db,
input: { companyId: string; issueId: string; runId: string; onlyWhenDue?: boolean },
): Promise<boolean> {
let agentId: string | null = null;
let materialized = false;
@@ -909,9 +925,11 @@ export async function repairCommittedNativeChatResponse(
return false;
// An existing selected answer, including a subsequently deleted one, must
// never be replaced or resurrected by a background presentation repair.
const recordedPresentation = record(record(run.resultJson).presentationDecision);
if (
typeof record(record(run.resultJson).presentationDecision).commentId ===
"string"
typeof recordedPresentation.commentId === "string" ||
(recordedPresentation.commentAction !== undefined && recordedPresentation.commentAction !== "create") ||
record(run.contextSnapshot).skipIssueComment === true
)
return false;
const [coordinator] = await tx
@@ -933,6 +951,16 @@ export async function repairCommittedNativeChatResponse(
!coordinator.assessmentId
)
return false;
if (input.onlyWhenDue) {
if (coordinator.nextAttemptAt && coordinator.nextAttemptAt.getTime() > Date.now()) return false;
const [pendingEffect] = await tx.select({ id: statusDecisionEffects.id }).from(statusDecisionEffects).where(and(
eq(statusDecisionEffects.companyId, input.companyId),
eq(statusDecisionEffects.issueId, input.issueId),
eq(statusDecisionEffects.decisionId, coordinator.decisionId),
inArray(statusDecisionEffects.deliveryState, ["pending", "failed"]),
)).limit(1);
if (pendingEffect) return false;
}
const [accepted] = await tx
.select()
.from(nativeRunResults)
@@ -1123,13 +1151,7 @@ export async function repairCommittedNativeChatResponse(
});
} catch (error) {
if (error instanceof CommittedChatResponseAuthorizationError) return false;
// Keep transient failures visible and retryable without reclassifying the
// committed run or acknowledging a publication that never committed.
logger.warn(
{ runId: input.runId, error },
"Committed chat response is awaiting presentation retry",
);
return false;
throw error;
}
if (materialized && agentId)
publishChatPublicationCommitSignal({
@@ -1167,6 +1189,7 @@ export async function finalizeNativeRun(input: {
workspaceFinalizeStatus: "succeeded" | "failed";
/** Reconciliation owns terminal run projection; the live heartbeat does it afterward. */
projectRunStatus?: boolean;
skipNativeResponsePresentation?: boolean;
/** Workspace-only replay must not consume the provider recovery budget. */
preserveProviderAttempt?: boolean;
failpoint?: NativeStatusCommitFailpoint;
@@ -1220,7 +1243,7 @@ export async function finalizeNativeRun(input: {
await projectCommittedRun({ db: input.db, run: admittedRun, coordinator });
if (input.projectRunStatus && !presentationAlreadyMaterialized)
await materializeCommittedReviewResponse(input.db, input.runId);
if (input.projectRunStatus)
if (input.projectRunStatus && !input.skipNativeResponsePresentation)
await repairCommittedNativeChatResponse(input.db, {
companyId: admittedRun.companyId,
issueId: coordinator.issueId,
@@ -1721,7 +1744,7 @@ export async function finalizeNativeRun(input: {
}
if (input.projectRunStatus)
await materializeCommittedReviewResponse(input.db, input.runId);
if (input.projectRunStatus && finalizationPhase === "committed")
if (input.projectRunStatus && finalizationPhase === "committed" && !input.skipNativeResponsePresentation)
await repairCommittedNativeChatResponse(input.db, {
companyId: run.companyId,
issueId: coordinator.issueId,
@@ -1,4 +1,4 @@
import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js";
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js";
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
@@ -3898,7 +3898,7 @@ export async function reconcileRetainedNativeSessionCleanup(
renameSync(owned.root, archive);
}
renameSync(copy, owned.root);
await notifyNativePendingEffects(tx, { companyId: owned.run.companyId, runId: owned.run.id });
await notifyNativeFinalizationWork(tx, { companyId: owned.run.companyId, runId: owned.run.id });
await tx
.update(nativeRunFinalizations)
.set({
@@ -3956,7 +3956,7 @@ export async function reconcileRetainedNativeSessionCleanup(
.limit(1)
.then((rows) => rows[0]);
if (!current || current.leaseOwner !== leaseOwner) return;
await notifyNativePendingEffects(tx, { companyId: owned.run.companyId, runId: owned.run.id });
await notifyNativeFinalizationWork(tx, { companyId: owned.run.companyId, runId: owned.run.id });
await tx
.update(nativeRunFinalizations)
.set({
@@ -9700,7 +9700,7 @@ async function executePaperclipNativeSessionWithinScope(
)
.for("update");
await notifyExecutionRunWork(tx);
await notifyNativePendingEffects(tx, input.execution.binding);
await notifyNativeFinalizationWork(tx, input.execution.binding);
const updated = await tx
.update(nativeRunFinalizations)
.set({
@@ -10036,7 +10036,7 @@ async function executePaperclipNativeSessionWithinScope(
}
const releaseNow = new Date();
const released = await timeController("coordinator_release", () => input.db.transaction(async tx => {
await notifyNativePendingEffects(tx, input.execution.binding);
await notifyNativeFinalizationWork(tx, input.execution.binding);
return tx.update(nativeRunFinalizations)
.set({
leaseOwner: null,
@@ -1,4 +1,4 @@
import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js";
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
import { alias } from "drizzle-orm/pg-core";
import { notifyUnsafeWorkspaceExportRecovery, type UnsafeWorkspaceExportScope } from "../unsafe-workspace-export-notifications.js";
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
@@ -101,7 +101,7 @@ async function recoverCandidate(db: Db, candidate: { runId: string; companyId: s
phase: "workspace_finalize", status: "succeeded", exitCode: 0, finishedAt: now,
metadata: { owningService: "native_workspace_finalizer", workspaceSync: { omitted: true, reason: "restore_unsafe_archive", legacy: true } },
});
await notifyNativePendingEffects(tx, { companyId: run.companyId, runId: run.id });
await notifyNativeFinalizationWork(tx, { companyId: run.companyId, runId: run.id });
await tx.update(nativeRunFinalizations).set({ phase: "result_accepted", failureCode: null, nextAttemptAt: null,
failureDetail: { workspaceFinalizeAttempt: 0, legacyUnsafeExportOmitted: { resultId: result.id, recoveredAt: now.toISOString() } }, updatedAt: now,
}).where(eq(nativeRunFinalizations.runId, run.id));
@@ -1,4 +1,4 @@
import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js";
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
import { notifyEnvironmentCleanup } from "../environment-cleanup-notifications.js";
import { randomUUID } from "node:crypto";
import type { Environment, EnvironmentLease } from "@paperclipai/shared";
@@ -135,7 +135,7 @@ export async function retryNativeWorkspaceExport(input: {
const now = new Date();
const retry = { ...receipt, requestId: randomUUID(), actorId: input.actorId, repairNote: input.repairNote, requestedAt: now.toISOString(),
stoppedProvider: initial.lease.metadata?.remoteExecutionTermination };
await notifyNativePendingEffects(tx, { companyId: current.run.companyId, runId: current.run.id });
await notifyNativeFinalizationWork(tx, { companyId: current.run.companyId, runId: current.run.id });
await tx.update(nativeRunFinalizations).set({ phase: "result_accepted", failureCode: null, nextAttemptAt: null,
failureDetail: { ...current.coordinator.failureDetail, workspaceFinalizeAttempt: 0, workspaceExportRetry: retry }, updatedAt: now }).where(eq(nativeRunFinalizations.runId, current.run.id));
await tx.update(heartbeatRuns).set({ status: "running", finishedAt: null, error: null, errorCode: null,
@@ -1,3 +1,5 @@
import { nativeResponsePresentationWork } from "./native-response-presentation-work.js";
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { asc, eq, sql } from "drizzle-orm";
import { generateKeyPairSync, randomUUID } from "node:crypto";
@@ -65,6 +67,8 @@ import {
subscribeCompanyLiveEvents,
} from "../live-events.js";
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_CONTROL_PLANE_TEST_DATABASE_URL");
describe("PaperclipControlPlanePort conformance", () => {
let temporary: Awaited<ReturnType<typeof startRecoveryPostgres>> | null = null;
let db: ReturnType<typeof createDb>;
@@ -87,8 +91,11 @@ describe("PaperclipControlPlanePort conformance", () => {
const governanceRunnerId = "00000000-0000-4000-8000-000000000021";
beforeAll(async () => {
temporary = await startRecoveryPostgres("paperclip-native-port-");
db = createDb(temporary.connectionString);
if (external) db = createDb(external);
else {
temporary = await startRecoveryPostgres("paperclip-native-port-");
db = createDb(temporary.connectionString);
}
const identity = CONTROL_PLANE_CONFORMANCE_OPEN.identity;
await db.insert(companies).values({ id: identity.companyId, name: "Phase 6", issuePrefix: "P6C" });
await db.insert(agents).values({
@@ -645,7 +652,7 @@ describe("PaperclipControlPlanePort conformance", () => {
completionContractSha256: localContractSha,
contextSnapshot: { issueId },
});
const observerDb = createDb(temporary!.connectionString);
const observerDb = createDb(external ?? temporary!.connectionString);
const observed: Array<Record<string, unknown>> = [];
const committedCallbacks: string[] = [];
let visibilityCheck: Promise<Array<{ eventType: string }>> | null = null;
@@ -1096,7 +1103,10 @@ describe("PaperclipControlPlanePort conformance", () => {
.where(eq(heartbeatRuns.id, taskRunId));
expect(await repair()).toBe(false);
await db.update(heartbeatRuns).set({ contextSnapshot: {} }).where(eq(heartbeatRuns.id, taskRunId));
expect(await repair()).toBe(true);
const work = nativeResponsePresentationWork(db);
const [entry] = await work.load({ companyId: identity.companyId, runId: taskRunId });
expect(entry?.id).toBe(taskRunId);
expect(await work.check(entry)).toBeNull();
expect(await repair()).toBe(false);
expect(await db.select({ body: issueComments.body }).from(issueComments)
.where(eq(issueComments.createdByRunId, taskRunId)).orderBy(asc(issueComments.createdAt)))
@@ -1,4 +1,4 @@
import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js";
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js";
import { notifyAutomaticCompletionReviews } from "../automatic-completion-review-notifications.js";
import { notifyObsoletePolicyReviews } from "../obsolete-policy-review-notifications.js";
@@ -154,7 +154,7 @@ export async function dispatchPendingNativeStatusEffects(input: {
targetId: effect.targetId,
});
}
await notifyNativePendingEffects(tx, input);
await notifyNativeFinalizationWork(tx, input);
await notifyChatRunMilestones(tx, input);
await tx
.update(statusDecisionEffects)
@@ -1241,7 +1241,7 @@ async function materializeDecisionEffect(input: {
)
.for("update");
if (pending.length === 0) throw new Error("native_pending_effect_missing");
await notifyNativePendingEffects(input.tx, { companyId: input.companyId });
await notifyNativeFinalizationWork(input.tx, { companyId: input.companyId, issueId: input.issue.id });
await input.tx
.update(statusDecisionEffects)
.set({
@@ -1931,7 +1931,7 @@ export async function commitNativeStatusDecision(input: {
effect => effect.kind === "record_finalization_error",
);
await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, input.companyId);
await notifyNativePendingEffects(tx, input);
await notifyNativeFinalizationWork(tx, input);
await tx
.update(nativeRunFinalizations)
.set({
@@ -1,4 +1,4 @@
import { notifyNativePendingEffects } from "./native-pending-effect-notifications.js";
import { notifyNativeFinalizationWork } from "./native-finalization-notifications.js";
import { and, desc, eq, gte, ne, or, sql } from "drizzle-orm";
import { agentWakeupRequests, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db";
import { readSystemRecovery } from "./system-recovery-state.js";
@@ -46,7 +46,7 @@ export async function resumePreservedResultFinalization(db: Db, companyId: strin
const generationId = intent?.id ?? recovery.generationId;
if (coordinator.recoveryHistory.some(entry => entry.kind === "preserved_result_finalization" &&
entry.generationId === generationId)) return false;
await notifyNativePendingEffects(tx, { companyId, runId });
await notifyNativeFinalizationWork(tx, { companyId, runId });
await tx.update(nativeRunFinalizations).set({
phase: coordinator.assessmentId ? "arbitrating" : "workspace_finalizing",
nextAttemptAt: new Date(), failureCode: null, failureDetail: null, updatedAt: new Date(),