fix(server): schedule native pending effect recovery (#17690)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native status decisions store effect targets and a delivery ledger.
> - Historical pending or failed effects still need recovery.
> - The periodic finalizer queries that ledger even when work is
complete.
> - This pull request gives pending effects their own commit-driven
worker and saved deadlines.
> - Completed history becomes quiet while incomplete effects remain
recoverable.

## Linked Issues or Issue Description

Refs #17687 (merged). This is slice 47 of the sleep migration stack, now
based on master.

**What problem does this solve?**

Periodic native finalization repeatedly checks effect delivery. A tenant
with completed history should not perform recurring SQL for this duty.
Historical unresolved effects must still recover without duplicating
their targets.

**Proposed solution**

Reconstruct current pending effects at startup. Wake from committed
coordinator changes. Use the saved retry and ownership deadlines. Keep
the existing dispatcher as the only recovery mutation owner.

**Alternatives considered**

A slower unconditional scan still creates idle traffic. A new queue
table duplicates existing durable state. Replaying materialization can
duplicate effects. This change uses the existing ledger, transaction
signals, and shared scheduler.

## What Changed

- Add a scoped worker with 100-row discovery pages and fair 25-entry
dispatch batches. Delivered, superseded, and terminal-failure history
has no recurring scan.
- Preserve exact company/issue/run/decision binding and validate
materialized targets before settling effects. Recheck phase, saved
result, retry, and lease under the coordinator lock. Missing targets
retain recovery.
- Notify from decision replay, accepted-result recovery, coordinator
claim/retry/release, attention replay, and execution-control recovery.
Only two DB-only producer writes need a new transaction wrapper. Nested
transactions use the existing outer-commit boundary.
- Exclude pending-effect candidates from periodic finalization so it
cannot skip ahead of their owner. Keep explicit and startup recovery
behavior compatible. Respect maintenance disablement, drain, standby,
and shutdown.
- Add concurrent, idempotent migration 0411 with partial indexes for
pending effects and their current coordinator binding. Retain the five
newest schema snapshots.
- Document producers, query evidence, failure history, and scope in
`doc/plans/2026-10-25-native-pending-effects-sleep-audit.md`.

## Verification

- Source `ba38ee465df02b6026286b5d5e15ae49fe07a367` passes all 2284
distinct cases across 25 complete files, without skips. The main
selection passes 2225 in 203.94 seconds; safe-replacement passes 59 in
26.03 seconds. Coverage includes all 968 native-session cases, the
94-case status-authority/pending-effect corpus, actual decision/lease
producers, Stop, queue, interaction, reconstruction and Slack consumers.
The audit gives per-file counts.
- The complete 94-case corpus passes again on the same database in 16.03
seconds. Retained PSC/OUT rows and an unrelated sentinel remain
unchanged. The repeat is not counted again as distinct coverage.
- Twenty-one separate fresh PostgreSQL 17 databases migrate through 0411
before setup. Four embedded-only fixture imports temporarily use their
own external database on this IPC-exhausted host and are restored byte
for byte. Assertions and timing limits stay unchanged. This qualifies
external-database behavior; it does not claim an unmodified embedded CI
run.
- Database build, server typecheck/build, module boundaries and
exact-head migration order pass. Original runtime/schema edits and
unrelated upstream blobs are preserved. Both original concurrent index
statements, all prior executable history, journal entries and retained
snapshot bytes are unchanged; five snapshots 0407 through 0411 remain.
- Populated 0410-to0411 upgrade preserves all 409 prior migration
receipts and 38 companies, appends one receipt and adds none on replay.
Both indexes are valid and ready on fresh and upgraded databases.
- Adopting the latest prepared prerequisite changes ancestry and parent
audits only. All accepted core trees remain identical. The audit retains
earlier failures and qualifications. Actual landed-parent ancestry is
now reconciled; fresh published-head CI and exact-head Greptile 5/5
remain required. No deployed sleep, Cloud-wake or fleet-cost claim is
made.

- Current parent context: integration
`5f7cd7bc589ad0169dfb5f33df611ead56510ca3` changes only the parent's
GitHub observation bound and independent cleanup assertion. All
production code, packages and 25 accepted test files remain unchanged.
Both focused GitHub idle cases pass in 15.26 seconds and the complete
six-case cleanup suite passes in 8.47 seconds on two fresh 0411
PostgreSQL fixtures. This does not claim the full parent workflow was
rerun here.

## Risks

- A missing producer can delay work until restart. The audit covers
current supported writers. Independent SQL and other processes remain
outside the local notification contract.
- Recovery validates existing targets; it cannot recreate a missing
target or silently mark it delivered. Such work retains an error and
retry.
- The indexes add write/storage cost and use the existing restart-safe
concurrent migration runner.
- Other finalization duties still poll until their separate migrations.
This PR does not prove host/database suspension, extend Cloud alarms, or
claim fleet savings.
- The actual prerequisite squash is adopted through a normal merge. Its
complete tree equals the reviewed parent, and the complete accepted
child tree is preserved. Fresh published-head CI and review remain
required.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning and code-execution tools.
The exact serving model ID and context-window size are not exposed in
this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
authored and GitHub committed 2026-10-29 00:28:00 -05:00
1 parent 4e715eb7ea
commit a8a038e659
26 files changed
+877 -52

No files matched your search

@@ -2662,3 +2662,56 @@ producer coverage, fixture failures, and the supported writer boundary. Workspac
typechecks and build passed; remote acceptance remains outstanding. Origin/master was fetched at
`6336c34041`; the full stack still needs final reconciliation with master. No PR
in this stack has been merged, and the original remaining scope is unchanged.
## Slice 46 published; slice 47 pending-effects audit
Unsafe-export recovery is draft [#17687](https://github.com/paperclipai/paperclip/pull/17687)
at `b9fe9e2e40ab48104cf18822b5302a740d1957a7`, stacked on #17680. Workspace types
and build passed. Greptile reviewed that exact head at 5/5 with no actionable
findings; full remote CI remains active. This is not acceptance or merge proof.
The attempted full local `pnpm test:run` hit embedded PostgreSQL initialization
failures (15 native-session failures plus database-suite skips) and was stopped
with its log preserved at `/tmp/sleep46-full-tests.log`. It did not pass. The
626 focused external-database cases remain the local acceptance evidence.
Slice 47 begins with native pending status effects. The current writer already
materializes targets and records delivered effects atomically; historical pending
or failed rows still require recovery. The existing owner validates targets and
settles the ledger without recreating effects. The audit and open producer/
ordering questions are in [the pending-effects audit](2026-10-25-native-pending-effects-sleep-audit.md).
Its 72-case status-arbiter baseline passed. No source migration is implemented yet.
Older gates were revalidated: #17490 and #17572 still have E2E shard 3 and aggregate
failures at their recorded heads. #17576 additionally reports an action-required
check. #17580 has successful/neutral CI, but its recorded review findings still
require resolution. These remain part of the full requested stack finish line.
## Slice 46 accepted; slice 47 locally verified
#17687 at `b9fe9e2e40ab48104cf18822b5302a740d1957a7` has all 53 checks passing,
Greptile 5/5, no unresolved review threads, and a mergeable base. It remains
unmerged. Its full-local-suite PostgreSQL failure above remains part of the record.
Slice 47 implements native pending-effect recovery with exact coordinator binding,
commit-aware producers, saved lease/retry deadlines, lock-time eligibility checks,
and pending-only indexed discovery. Periodic finalization skips this duty and
preserves ordering; explicit/startup recovery remains compatible. No provider turn,
new queue table, or host-sleep exemption is introduced. See the
[pending-effects audit](2026-10-25-native-pending-effects-sleep-audit.md).
The expanded consumer run passed 680 tests across 15 files. The final producer
audit found a lease-shortening wake gap; its new real-store regression failed
before the fix and passed afterward. Final owner/startup tests pass 158 cases;
the finalizer rerun passes 29 on a fresh database (683 distinct local cases across
these runs). The initial finalizer rerun reused a fixed company prefix and failed
setup; that failure is preserved. Index discovery was measured with 10,000 delivered
effects and 10,000 completed coordinators. Migration replay and valid/ready checks
pass. Final workspace and remote gates must be recorded before acceptance.
The remaining original scope, older CI/review issues, full master reconciliation,
and all final-head PR gates remain open. No PR has been merged by this stack work.
Final slice 47 source passes workspace typecheck/build and 158 worker/startup
cases, including exact near-term concurrent deadlines. Remote publication and
acceptance are the next gates; local success does not satisfy the full stack.
@@ -0,0 +1,276 @@
# Native pending status-effect recovery
Branch: `codex/sleep-47-native-pending-effects`.
Current published parent: #17687, `b28399babac72ba9db1c4a4ef882c5e87aded38c`.
Original draft parent: `b9fe9e2e40ab48104cf18822b5302a740d1957a7`.
Status: implemented and locally verified; remote PR acceptance remains required.
## Duty and existing ownership
The native finalization reconciler reads pending/failed `status_decision_effects`
for each candidate's current decision before other finalization work. When any
exist, it calls `dispatchPendingNativeStatusEffects` and skips the remaining
candidate processing for that pass. This slice moves only that duty. Response
presentation, evidence reassessment, and accepted-result/workspace settlement
remain separate migrations.
The existing dispatcher owns the recovery transaction. It locks the coordinator,
checks the exact company/issue/run/decision binding, locks pending effects, and
checks that their materialized targets exist. It then marks the ledger delivered,
increments attempts, clears retry/error state, updates delivered-effect IDs, and
signals chat milestones. It does not recreate targets or arbitrate status again.
A missing target remains an error with an owned retry, never successful cleanup.
## Source of truth and scheduling
The only current production insert creates effect targets and delivered ledger
entries atomically in `commitNativeStatusDecision`. Both production effect-update
paths also settle to delivered. No current pending/failed producer was found.
Historical rows and the schema default still require recovery. Ordinary successful
commits therefore discover no pending-effects timer.
The worker reconstructs existing pending/failed effects belonging to the exact
current coordinator decision. It requires native runtime, an existing issue,
a saved result, and a phase other than terminal_failure. Superseded decisions and
terminal-failure history remain dormant. Its deadline is the later of the saved
coordinator retry and a held lease's expiry. An owner without an expiry preserves
the old reconciler's immediately eligible behavior. The ledger's nextAttemptAt
was not an eligibility fence in the old dispatcher and this slice does not invent
one. Actual errors use the shared bounded retry backoff.
Discovery uses keyset pages of 100. The existing scoped deadline helper coalesces
commit hints, handles dirty generations/drain/failures, and fairly dispatches 25
entries per batch. Empty, delivered, and dormant history leaves no recurring SQL.
The dispatcher rechecks eligibility under its existing coordinator lock so a
concurrent lease or retry extension cannot be bypassed.
Startup registration respects the existing maintenance toggle and task-drain,
shutdown, and warm-standby gates. Periodic finalization excludes pending-effect
candidates and retains a late pending-effect guard to preserve ordering. Explicit
and startup recovery retain the old dispatcher default. The remaining finalizer
can process a candidate after its effects settle; this slice does not remove its
other polling duties.
## Producer audit and shared boundaries
Notifications carry only company and optional run IDs. They register intent before
the write and use the existing outer-transaction/uncertain-outcome protocol.
| Mutation owner | Why it matters |
| --- | --- |
| Status decision commit/replay | Can restore the current decision and phase; new effects already delivered remain quiet. |
| Pending-effect dispatcher/materializer | Retires an existing obligation and refreshes remembered deadlines. |
| Preserved-result finalization | Reactivates accepted-result history. |
| Workspace export retry and historical export recovery | Restore accepted-result eligibility. |
| Persisted-attention replay | Rebinds an existing decision and clears its lease/retry. Its DB-only write now has an explicit transaction. |
| Finalizer claim and retryable failure | Claim can clear a later retry; failure releases ownership and changes the next retry. |
| Coordinator-store claim/renewal and retry | An existing owner can shorten its lease; release can move eligibility earlier. |
| Retained-session cleanup success/failure | Releases the coordinator lease. |
| Native execution failure/release | Can leave a saved result eligible and release its lease. The final DB-only release now has an explicit transaction. |
| Abandoned execution-control recovery | Releases ownership of an already accepted result. |
The audit enumerated 34 coordinator writes that change phase, result, decision,
lease, or retry fields, then inspected their guards. Provider execution claims,
restart recovery, session resumption, and stopped-execution checks reject saved
results or explicitly require resultId null. Terminal-failure writes disable
eligibility; safe replacement keeps that terminal phase. Retained cleanup claims
and normal session renewals only postpone eligibility, which is rechecked at the
remembered deadline. Claim/renewal paths that can shorten it are signaled.
The accepted-result insertion cannot expose an existing pending decision without
a result: schema checks require decision -> assessment -> result, and replay of
an existing accepted result returns before replacing the coordinator. Runtime-mode
writes occur in initial native preparation or the signaled status-decision owner.
Deleting records removes work; a scheduled lookup drops a missing candidate.
This remains the supported single-owner client notification model. Independent
SQL writes and other processes do not produce these local signals; startup can
reconstruct their durable records but this change does not add live cross-process
delivery. No new queue table, provider protocol, or host-sleep exemption is added.
## Query and migration evidence
Captured the actual worker SQL using the PostgreSQL client's debug hook. A fixture
added 10,000 delivered effects and 10,000 completed coordinators, with one eligible
pending effect. The committed fixture matters: an earlier rolled-back experiment
could not consistently use the newly built partial index under its old snapshot.
| Discovery | Existing indexes | Pending-effect index only | Both new indexes |
| --- | ---: | ---: | ---: |
| Global startup | 2.247 ms | 1.080 ms | 0.036 ms |
| Company scope | 1.807 ms | 1.048 ms | 0.015 ms |
Without the indexes, discovery filtered 10,809 delivered effects and scanned 10,353
coordinators. The partial effect index selects pending/failed rows; the partial
coordinator index resolves their current company/issue/decision owner. Both plans
used index access for those relations after the change. This fixture establishes
history cost for this query, not universal tenant-size or fleet performance.
Current migration 0411 uses concurrent IF NOT EXISTS index builds through the existing
nontransactional migration owner. Generated snapshots retain the newest five
(0407–0411). Fresh schema application, migration replay, direct SQL replay, and
valid/ready index checks pass. The first replay command failed before SQL because
the tsx evaluator rejected top-level await; the corrected promise-chain command
passed. Logs: `/tmp/sleep47-query-plan-committed.log`,
`/tmp/sleep47-migration-replay-v2.log`.
## Verification and remaining gates
The pre-change corpus passed 72 cases. Initial worker verification passed 86.
Two initial test failures compared epoch scheduler deadlines to Date objects;
the assertions were corrected and passed without weakening production behavior.
Worker/startup wiring passed 154 cases. The expanded owner and consumer selection
passed 680 cases across 15 files on fresh external PostgreSQL. Workspace types
and build passed. The later producer audit identified a shortened-lease signal
gap; its regression and final verification are recorded below before acceptance.
Tests exercise delivered/empty/terminal history quiet, pending/failed recovery,
no duplicate decisions or targets, missing-target failure, future retry/lease
quiet, concurrent deadline extension, real replay, outer commit/rollback, real
lease release, company-scope isolation, drain/resume, restart reconstruction,
103-candidate paging, and periodic/manual owner ordering. Existing consumers cover
finalization, export recovery, execution control, lifecycle, and chat delivery.
The full local suite is not claimed green. The parent attempt encountered embedded
PostgreSQL initialization failures and was intentionally stopped; that failure is
preserved in `/tmp/sleep46-full-tests.log`. This slice uses fresh external databases
for affected suites. Remote CI/review must pass at the final PR head. Actual host
or database suspension and the remaining recurring-server duties are not covered
by this worker's quiet proof.
## Final local acceptance
The real-store shortened-lease regression failed before the missing claim signal:
recovery waited beyond the old deadline. After notifying at the two claim owners,
the regression passes. A concurrently saved ten-second deadline now remains ten
seconds rather than being rounded up to the fallback retry. The saved lease-owner
check preserves SQL null semantics, including legacy empty-string owners.
The final worker/startup selection passes 158 tests (94 corpus, 64 startup),
including all 22 cases added to the corpus by this slice. The finalizer selection
passes 29 tests on a fresh database. A first rerun reused the TLM company-prefix
fixture and failed setup with companies_issue_prefix_idx; it did not pass, and
its log remains `/tmp/sleep47-owner-final.log`. The fresh rerun is
`/tmp/sleep47-finalizer-final.log`. Combined with unaffected consumer cases from
the 680-case run, 683 distinct cases have passed. The final worker log is
`/tmp/sleep47-deadlines-final.log`.
Final source verification: `pnpm typecheck` and `pnpm build` pass after the near-term
deadline repair (`/tmp/sleep47-deadlines-typecheck.log`,
`/tmp/sleep47-deadlines-build.log`). The final worker/startup run now passes 158
cases, giving 683 distinct passing cases with the earlier unaffected consumers.
## Current migration411 integration and complete acceptance
Only this slice's unlanded 0403 migration artifacts were deferred before a
normal merge of the verified current unsafe-export parent. No conflicts occur.
The complete original runtime/schema patch and every unrelated upstream blob
are preserved. The migration directory and client matched the exact parent
before generation. `pnpm db:generate` produced `0411_real_scrambler.sql`. Both
concurrent, idempotent index statements are byte-identical to the original.
All earlier SQL, journal entries and retained snapshot bytes remain unchanged,
with five snapshots 0407 through 0411. No new code comments are added.
Source `ba38ee465df02b6026286b5d5e15ae49fe07a367` passes **2284 distinct
cases in 25 complete files**, without skips. The main selection passes 2225
cases in 203.94 seconds and safe-replacement passes 59 in 26.03 seconds.
The complete 94-case status-authority corpus then passes again on the same
database in 16.03 seconds. The repeat is not double-counted. Three preexisting
PSC/OUT/sentinel company rows remain unchanged in their inspected ID/name/prefix
fields after both runs; each invocation appends two distinct UUID-derived
fixture prefixes. Current per-file counts:
| Complete file | Passing cases |
| --- | ---: |
| src/__tests__/issue-recovery-actions.test.ts | 89 |
| src/__tests__/native-finalization-recovery.test.ts | 32 |
| src/__tests__/native-interaction-bridge.test.ts | 7 |
| src/__tests__/native-status-arbiter-corpus.test.ts | 94 |
| src/__tests__/native-workspace-sync.test.ts | 10 |
| src/__tests__/recovery-stale-issue-lock-sweep.test.ts | 40 |
| src/__tests__/server-startup-feedback-export.test.ts | 67 |
| src/__tests__/slack-conversation-lifecycle.test.ts | 41 |
| src/modules/run-dispatch/adapters/postgres.test.ts | 69 |
| src/modules/wake-queue/adapters/postgres.test.ts | 93 |
| src/modules/wake-queue/adapters/queued-comment-postgres.test.ts | 5 |
| src/services/execution-control-reconciliation.test.ts | 10 |
| src/services/explicit-native-continuation.test.ts | 335 |
| src/services/heartbeat/queue.test.ts | 32 |
| src/services/heartbeat/recovery.test.ts | 45 |
| src/services/heartbeat/run-control.test.ts | 51 |
| src/services/heartbeat/run-lifecycle.test.ts | 31 |
| src/services/heartbeat/upgrade-readiness.test.ts | 7 |
| src/services/legacy-execution-recovery.test.ts | 15 |
| src/services/native-runtime/native-restart-recovery.test.ts | 25 |
| src/services/native-runtime/native-run-finalizer-telemetry.test.ts | 29 |
| src/services/native-runtime/native-safe-replacement.test.ts | 59 |
| src/services/native-runtime/native-session-executor.test.ts | 968 |
| src/services/native-runtime/native-workspace-export-retry.test.ts | 80 |
| src/services/native-runtime/native-workspace-reconstruction.test.ts | 50 |
Twenty-one separate fresh PostgreSQL 17 databases migrate through 0411 before
setup. Four embedded-only fixture imports temporarily use their own external
database on this IPC-exhausted host and are restored byte for byte: interaction
bridge, workspace reconstruction, Slack lifecycle and wake-queue PostgreSQL.
All assertions and timing limits remain unchanged. Other suites use their
existing supported external-database paths. This is qualified external-database
acceptance, not an unmodified embedded CI reproduction. Actual database build,
server types/build, module boundaries and exact-head migration order pass.
A populated 0410-to0411 upgrade preserves all 409 prior migration receipts and
all 38 companies, appends exactly one receipt and appends none on replay. Both
partial indexes are valid and ready on fresh and upgraded databases. Earlier
failures above retain their original scope and causes. These checks verify
current worker and producer/consumer behavior, not deployed host/database
suspension, external Cloud wakes or fleet savings.
Normal merge `6cc9e0f4a05ac7aee225b8ac7dce119c31799257` adopts prepared parent
`d64d307484633485716124b24ad8bbdbea61f399`, carrying the published automatic-review
ancestry and parent documentation. Complete server, package, UI and test trees
remain identical to the accepted source, including the whole 0411 database
tree. Publication stays held for the actual unsafe-export prerequisite merge
and ancestry reconciliation. Fresh published-head CI and reviews remain required.
## Inherited CI correction checked in native-effect context
Normal integration `5f7cd7bc589ad0169dfb5f33df611ead56510ca3` carries the
upstream GitHub observation-bound correction and independent cleanup assertion.
Only those two previously unselected test files and parent audits change.
All production source, all packages including migration 0411, and all 25 files
in the accepted 2284-case selection remain byte-identical. That acceptance and
its 94-case same-database repeat retain their original source scope.
Both focused native-presentation GitHub idle variants pass in 15.26 seconds.
All six cases in the complete cleanup-removal suite pass in 8.47 seconds. Two
separate fresh PostgreSQL 17 databases migrate through 0411; supported external
URLs require no fixture source adaptation. The other 1779 cases in the large
chat file are outside this focused selection. The full parent 274-case workflow
is not claimed as executed here. The parent audit preserves the original CI
failure and controlled comparison, including its limit on exact CI causality.
Actual prerequisite ancestry and fresh published-head CI/reviews remain required.
## Published unsafe-export parent
Normal merge `01ff4da4457bf2afc3d2e5e65fa33c16d56a4e6f` adopts published prerequisite
`b28399babac72ba9db1c4a4ef882c5e87aded38c`, including the actual automatic-review
squash ancestry. Complete server, package, UI and test trees remain identical
to the accepted child. The whole 0411 database tree and all earlier acceptance
scopes remain unchanged. Publication remains held until the unsafe-export
parent lands and its actual ancestry is reconciled. Fresh current-head CI
and review are still required.
## Actual unsafe-export squash ancestry
Unsafe-export prerequisite #17687 merged as
`4e715eb7ea56d49a099530c3dce59e61e698077c`, reviewed at
`b28399babac72ba9db1c4a4ef882c5e87aded38c`. Its entire squash tree equals
the reviewed parent, which was already an ancestor of this prepared child.
Normal merge `45a1f20c8830efb0d60794f01fd7a5013ff41198` resolves duplicate
squash ancestry while preserving the complete held child tree byte for byte.
All accepted source, all migrations through 0411, and all earlier validation
scopes remain unchanged. The only following edit records this proof. Fresh
published-head CI and exact-head Greptile 5/5 are still required.
+1
View File
@@ -560,6 +560,7 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([
"0408_short_red_skull.sql",
"0409_uneven_zuras.sql",
"0410_freezing_kinsey_walden.sql",
"0411_real_scrambler.sql",
"0384_funny_adam_warlock.sql",
@@ -0,0 +1,2 @@
CREATE INDEX CONCURRENTLY IF NOT EXISTS "native_run_finalizations_pending_effect_owner_idx" ON "native_run_finalizations" USING btree ("company_id","issue_id","decision_id") WHERE "native_run_finalizations"."decision_id" is not null AND "native_run_finalizations"."result_id" is not null AND "native_run_finalizations"."phase" <> 'terminal_failure';--> statement-breakpoint
CREATE INDEX CONCURRENTLY IF NOT EXISTS "status_decision_effects_pending_delivery_idx" ON "status_decision_effects" USING btree ("company_id","issue_id","decision_id") WHERE "status_decision_effects"."delivery_state" in ('pending', 'failed');
@@ -1,6 +1,6 @@
{
"id": "c34fcd91-0d57-4241-b8b4-399bc9f6dcb0",
"prevId": "c87ad360-f306-459b-b814-fde37327c6da",
"id": "ff56ee93-8cfc-465a-b26a-1c12d10773ff",
"prevId": "c02426e4-de1a-4ce3-a541-99bb970c272a",
"version": "7",
"dialect": "postgresql",
"tables": {
@@ -2665,6 +2665,50 @@
}
},
"indexes": {
"agent_instruction_copies_collection_idx": {
"name": "agent_instruction_copies_collection_idx",
"columns": [
{
"expression": "run_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "NOT (coalesce(\"agent_instruction_working_copies\".\"receipt\", '{}'::jsonb) ? 'retainedByRunId') AND (\n (\"agent_instruction_working_copies\".\"attempts\" < 3 AND (\"agent_instruction_working_copies\".\"state\" IN ('prepared', 'pending_collection', 'unchanged_turn', 'warm_saved')\n OR (\"agent_instruction_working_copies\".\"state\" = 'preparing' AND \"agent_instruction_working_copies\".\"receipt\"->>'schema' = 'paperclip.agent-files.v1')))\n OR (\"agent_instruction_working_copies\".\"state\" = 'unavailable' AND \"agent_instruction_working_copies\".\"process_stopped_at\" IS NULL\n AND \"agent_instruction_working_copies\".\"location\" LIKE 'remote:%' AND \"agent_instruction_working_copies\".\"receipt\"->>'schema' = 'paperclip.agent-files.v1'))",
"concurrently": false,
"method": "btree",
"with": {}
},
"agent_instruction_copies_cleanup_idx": {
"name": "agent_instruction_copies_cleanup_idx",
"columns": [
{
"expression": "run_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"agent_instruction_working_copies\".\"state\" IN ('saved', 'unchanged', 'resolved', 'unavailable')\n AND \"agent_instruction_working_copies\".\"process_stopped_at\" IS NOT NULL\n AND \"agent_instruction_working_copies\".\"receipt\"->>'schema' = 'paperclip.agent-files.v1'\n AND coalesce(\"agent_instruction_working_copies\".\"receipt\"->>'retainedByRunId', '') = ''\n AND \"agent_instruction_working_copies\".\"receipt\"->>'cleanupPending' IS DISTINCT FROM 'false'\n AND (\"agent_instruction_working_copies\".\"receipt\" ? 'baseline' OR \"agent_instruction_working_copies\".\"receipt\"->>'cleanupPending' = 'true')",
"concurrently": false,
"method": "btree",
"with": {}
},
"agent_instruction_copies_pending_idx": {
"name": "agent_instruction_copies_pending_idx",
"columns": [
@@ -33727,6 +33771,28 @@
}
},
"indexes": {
"heartbeat_runs_cancelled_workspace_preservation_idx": {
"name": "heartbeat_runs_cancelled_workspace_preservation_idx",
"columns": [
{
"expression": "id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' AND \"heartbeat_runs\".\"status\" = 'cancelled'\n AND \"heartbeat_runs\".\"runner_profile_json\"->'nativeWorkspaceSync'->>'state' = 'prepared'\n AND \"heartbeat_runs\".\"result_json\"->'nativeCancellation'->>'dispatchState' = 'acknowledged'\n AND (\"heartbeat_runs\".\"result_json\"->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'\n AND \"heartbeat_runs\".\"result_json\"->>'workspaceRestoreFailure' IN ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_failed')) IS NOT TRUE",
"concurrently": false,
"method": "btree",
"with": {}
},
"heartbeat_runs_retained_recovery_adoption_idx": {
"name": "heartbeat_runs_retained_recovery_adoption_idx",
"columns": [
@@ -43042,6 +43108,62 @@
}
},
"indexes": {
"native_run_finalizations_pending_effect_owner_idx": {
"name": "native_run_finalizations_pending_effect_owner_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "decision_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"native_run_finalizations\".\"decision_id\" is not null AND \"native_run_finalizations\".\"result_id\" is not null AND \"native_run_finalizations\".\"phase\" <> 'terminal_failure'",
"concurrently": false,
"method": "btree",
"with": {}
},
"native_run_finalizations_unsafe_export_recovery_idx": {
"name": "native_run_finalizations_unsafe_export_recovery_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "run_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"native_run_finalizations\".\"phase\" = 'terminal_failure' AND \"native_run_finalizations\".\"failure_code\" = 'native_workspace_sync_out_unsafe_archive'",
"concurrently": false,
"method": "btree",
"with": {}
},
"native_run_finalizations_control_deadline_idx": {
"name": "native_run_finalizations_control_deadline_idx",
"columns": [
@@ -52346,6 +52468,34 @@
}
},
"indexes": {
"status_decision_effects_pending_delivery_idx": {
"name": "status_decision_effects_pending_delivery_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "decision_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"status_decision_effects\".\"delivery_state\" in ('pending', 'failed')",
"concurrently": false,
"method": "btree",
"with": {}
},
"status_decision_effects_decision_ordinal_uq": {
"name": "status_decision_effects_decision_ordinal_uq",
"columns": [
+7
View File
@@ -2864,6 +2864,13 @@
"when": 1793244017955,
"tag": "0410_freezing_kinsey_walden",
"breakpoints": true
},
{
"idx": 411,
"version": "7",
"when": 1793246172969,
"tag": "0411_real_scrambler",
"breakpoints": true
}
]
}
@@ -50,6 +50,9 @@ export const nativeRunFinalizations = pgTable(
updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(),
},
(table) => ({
pendingEffectOwnerIdx: index("native_run_finalizations_pending_effect_owner_idx")
.on(table.companyId, table.issueId, table.decisionId)
.where(sql`${table.decisionId} is not null AND ${table.resultId} is not null AND ${table.phase} <> 'terminal_failure'`),
unsafeExportRecoveryIdx: index("native_run_finalizations_unsafe_export_recovery_idx")
.on(table.companyId, table.issueId, table.runId)
.where(sql`${table.phase} = 'terminal_failure' AND ${table.failureCode} = 'native_workspace_sync_out_unsafe_archive'`),
@@ -1,5 +1,7 @@
import { sql } from "drizzle-orm";
import {
foreignKey,
index,
pgTable,
uuid,
text,
@@ -34,6 +36,9 @@ export const statusDecisionEffects = pgTable(
updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(),
},
(table) => ({
pendingDeliveryIdx: index("status_decision_effects_pending_delivery_idx")
.on(table.companyId, table.issueId, table.decisionId)
.where(sql`${table.deliveryState} in ('pending', 'failed')`),
issueCompanyFk: foreignKey({
columns: [table.companyId, table.issueId],
foreignColumns: [issues.companyId, issues.id],
@@ -1,3 +1,5 @@
import { NativeRunCoordinatorStore } from "../services/native-runtime/native-run-coordinator-store.js";
import { registerNativePendingEffectWork } from "../services/native-runtime/native-pending-effect-work.js";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { idleWorkSnapshot } from "../services/task-admission.js";
import { dismissAutomaticCompletionReviews, registerAutomaticCompletionReviewWork } from "../services/native-runtime/automatic-completion-reviews.js";
@@ -417,7 +419,8 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
async function seedFixture(fixture: Fixture) {
async function seedFixture(fixture: Fixture, tenant = { companyId, agentId }) {
const { companyId, agentId } = tenant;
const issueId = randomUUID();
const runId = randomUUID();
const contractId = randomUUID();
@@ -2050,6 +2053,186 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
}
describe("native pending-effect scheduling", () => {
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
let admitted = true;
const errors: unknown[] = [];
beforeEach(async () => {
await db.update(statusDecisionEffects).set({ deliveryState: "delivered" }).where(eq(statusDecisionEffects.companyId, companyId));
admitted = true; errors.length = 0;
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
});
afterEach(async () => {
await coordinator?.stop(); coordinator = undefined;
vi.useRealTimers(); vi.restoreAllMocks(); expect(errors).toEqual([]);
});
async function seedEffects(state = "pending", tenant = { companyId, agentId }) {
const { companyId } = tenant;
const template = corpus.fixtures.find(candidate => candidate.mode === "native")!;
const f = await seedFixture({ ...template, id: `pending-effects-${randomUUID()}`,
given: { ...template.given, priorIssueStatus: "in_progress", completionState: "pending_effect_recovery" } }, tenant);
const commitInput: Parameters<typeof commitNativeStatusDecision>[0] = { db, companyId, issueId: f.issueId, runId: f.runId,
assessmentId: f.assessmentId, priorStatus: "in_progress", priorStatusVersion: 0, priorDecisionId: null,
decision: { policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, statusAction: "preserve", toStatus: "in_progress",
reasonCode: "live_continuation_registered", unblockDescriptor: null, effects: [{ kind: "release_checkout" }] } };
const committed = await commitNativeStatusDecision(commitInput);
const [effect] = await db.update(statusDecisionEffects).set({ deliveryState: state })
.where(eq(statusDecisionEffects.decisionId, committed.decision.id)).returning();
return { ...f, decisionId: committed.decision.id, effect, commitInput };
}
async function start() {
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => admitted, canReconcile: () => admitted,
onError: error => errors.push(error) });
await registerNativePendingEffectWork(db, coordinator, () => admitted).ready;
}
async function flush() {
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(1000);
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBeNull();
}, { timeout: 15_000 });
}
async function quiet(milliseconds = 120_000) {
const read = vi.spyOn(db, "select"), write = vi.spyOn(db, "transaction");
try { await vi.advanceTimersByTimeAsync(milliseconds); expect(read).not.toHaveBeenCalled(); expect(write).not.toHaveBeenCalled(); }
finally { read.mockRestore(); write.mockRestore(); }
}
const effect = async (id: string) => (await db.select().from(statusDecisionEffects).where(eq(statusDecisionEffects.id, id)))[0];
it.each(["empty", "delivered"])("stays quiet with %s history", async state => {
if (state === "delivered") await seedEffects(state);
await start(); await flush(); await quiet();
});
it.each(["pending", "failed"])("recovers %s effects without materializing another target or decision", async state => {
const f = await seedEffects(state);
await start(); await flush();
expect(await effect(f.effect.id)).toMatchObject({ deliveryState: "delivered", attemptCount: f.effect.attemptCount + 1 });
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, f.issueId))).toHaveLength(1);
expect(await db.select().from(statusDecisionEffects).where(eq(statusDecisionEffects.issueId, f.issueId))).toHaveLength(1);
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0].status).toBe("in_progress");
await quiet();
});
it("leaves terminal-failure history dormant", async () => {
const f = await seedEffects();
await db.update(nativeRunFinalizations).set({ phase: "terminal_failure" }).where(eq(nativeRunFinalizations.runId, f.runId));
await start(); await flush(); expect((await effect(f.effect.id)).deliveryState).toBe("pending"); await quiet();
});
it.each(["retry", "lease", "empty lease owner"])("honors the saved %s deadline without polling before it", async kind => {
const f = await seedEffects(); const due = new Date(Date.now() + 180_000);
await db.update(nativeRunFinalizations).set(kind === "retry" ? { nextAttemptAt: due } : { leaseOwner: kind === "empty lease owner" ? "" : "current-owner", leaseExpiresAt: due })
.where(eq(nativeRunFinalizations.runId, f.runId));
await start(); expect(coordinator!.nextWakeAt()).toBe(due.getTime());
await quiet(120_000); expect((await effect(f.effect.id)).deliveryState).toBe("pending");
await vi.advanceTimersByTimeAsync(60_000); await flush();
expect((await effect(f.effect.id)).deliveryState).toBe("delivered"); await quiet();
});
it.each([["retry", 180_000], ["lease", 180_000], ["retry", 10_000]] as const)("rechecks a concurrent %s extension of %i ms under the coordinator lock", async (kind, delay) => {
const f = await seedEffects();
const due = new Date(Date.now() + delay);
const original = db.transaction.bind(db);
const transaction = vi.spyOn(db, "transaction").mockImplementationOnce(async (callback, config) => {
await db.update(nativeRunFinalizations).set(kind === "retry"
? { nextAttemptAt: due } : { leaseOwner: "new-owner", leaseExpiresAt: due })
.where(eq(nativeRunFinalizations.runId, f.runId));
return original(callback, config);
});
await start();
await vi.waitFor(() => expect(coordinator!.nextWakeAt()).toBe(due.getTime()));
transaction.mockRestore();
expect((await effect(f.effect.id)).deliveryState).toBe("pending");
await quiet(delay / 2);
await vi.advanceTimersByTimeAsync(delay / 2); await flush();
expect((await effect(f.effect.id)).deliveryState).toBe("delivered"); await quiet();
});
it("leaves pending effects to their owner during periodic finalization and preserves explicit recovery", async () => {
const f = await seedEffects();
expect(await reconcileNativeFinalizations(db, [f.runId], { skipNativePendingEffects: true })).toEqual([]);
expect((await effect(f.effect.id)).deliveryState).toBe("pending");
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, f.issueId))).toHaveLength(1);
await reconcileNativeFinalizations(db, [f.runId]);
expect((await effect(f.effect.id)).deliveryState).toBe("delivered");
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, f.issueId))).toHaveLength(1);
});
it("limits a committed run wake to its owner and recovers other durable work on restart", async () => {
const local = await seedEffects();
const other = await seedEffects("pending", { companyId: outsideCompanyId, agentId: outsideAgentId });
for (const f of [local, other]) await db.update(nativeRunFinalizations).set({ phase: "terminal_failure" })
.where(eq(nativeRunFinalizations.runId, f.runId));
await start(); await flush();
await db.update(nativeRunFinalizations).set({ phase: "committed" }).where(eq(nativeRunFinalizations.runId, other.runId));
await commitNativeStatusDecision(local.commitInput); await flush();
expect((await effect(local.effect.id)).deliveryState).toBe("delivered");
expect((await effect(other.effect.id)).deliveryState).toBe("pending"); await quiet();
await coordinator!.stop(); coordinator = undefined;
await start(); await flush();
expect((await effect(other.effect.id)).deliveryState).toBe("delivered"); await quiet();
});
it("drains more than one discovery page and stops querying delivered history", async () => {
const fixtures = [];
for (let index = 0; index < 103; index++) fixtures.push(await seedEffects());
await start(); await flush();
for (const f of fixtures) expect((await effect(f.effect.id)).deliveryState).toBe("delivered");
await quiet();
}, 60_000);
it("preserves failed target verification and retries only that obligation", async () => {
const f = await seedEffects();
await db.update(statusDecisionEffects).set({ targetId: randomUUID() }).where(eq(statusDecisionEffects.id, f.effect.id));
await start(); await vi.waitFor(() => expect(errors.length).toBeGreaterThan(0));
expect((await effect(f.effect.id)).deliveryState).toBe("pending"); errors.length = 0;
await db.update(statusDecisionEffects).set({ targetId: f.effect.targetId }).where(eq(statusDecisionEffects.id, f.effect.id));
await vi.advanceTimersByTimeAsync(30_000); await flush();
expect((await effect(f.effect.id)).deliveryState).toBe("delivered"); await quiet();
});
it("reactivates dormant history when the real decision replay restores its binding", async () => {
const f = await seedEffects();
await db.update(nativeRunFinalizations).set({ phase: "terminal_failure" }).where(eq(nativeRunFinalizations.runId, f.runId));
await start(); await flush(); await quiet();
expect((await commitNativeStatusDecision(f.commitInput)).replayed).toBe(true);
await flush(); expect((await effect(f.effect.id)).deliveryState).toBe("delivered"); await quiet();
});
it.each(["commit", "rollback"] as const)("defers decision reactivation until the outer transaction %s", async outcome => {
const f = await seedEffects();
await db.update(nativeRunFinalizations).set({ phase: "terminal_failure" }).where(eq(nativeRunFinalizations.runId, f.runId));
await start(); await flush();
const writing = db.transaction(async tx => {
await commitNativeStatusDecision({ ...f.commitInput, db: tx as unknown as typeof db });
await vi.advanceTimersByTimeAsync(1000);
expect((await effect(f.effect.id)).deliveryState).toBe("pending");
if (outcome === "rollback") throw new Error("roll back decision replay");
});
if (outcome === "rollback") await expect(writing).rejects.toThrow("roll back decision replay");
else await writing;
await flush(); expect((await effect(f.effect.id)).deliveryState).toBe(outcome === "rollback" ? "pending" : "delivered"); await quiet();
});
it.each(["releases", "shortens"])("moves a future ownership deadline earlier when the real store %s its lease", async change => {
const f = await seedEffects();
const due = new Date(Date.now() + 300_000);
await db.update(nativeRunFinalizations).set({ leaseOwner: "release-owner", leaseExpiresAt: due }).where(eq(nativeRunFinalizations.runId, f.runId));
await start(); expect(coordinator!.nextWakeAt()).toBe(due.getTime()); await quiet();
const store = new NativeRunCoordinatorStore(db, { companyId, issueId: f.issueId, runId: f.runId, agentId,
normalizedSessionId: "pending-effects", runnerSourceInstanceId: "pending-effects", completionContractId: f.contractId!,
completionContractSha256: "test", completionContractRevision: "test", completionContractCriterionIds: [] });
if (change === "shortens") await store.claimFinalization({ leaseOwner: "release-owner", leaseTtlMs: 1000 });
else await store.markFinalizationRetry({ leaseOwner: "release-owner", failureCode: "temporary_failure", retryAfterMs: 1000 });
await flush(); expect(Date.now()).toBeLessThan(due.getTime());
expect((await effect(f.effect.id)).deliveryState).toBe("delivered"); await quiet();
});
it("retains a first reactivation arriving during drain without passive reads", async () => {
const f = await seedEffects();
await db.update(nativeRunFinalizations).set({ phase: "terminal_failure" }).where(eq(nativeRunFinalizations.runId, f.runId));
await start(); await flush(); admitted = false;
await commitNativeStatusDecision(f.commitInput); await quiet();
expect((await effect(f.effect.id)).deliveryState).toBe("pending");
admitted = true; await vi.advanceTimersByTimeAsync(30_000); await flush();
expect((await effect(f.effect.id)).deliveryState).toBe("delivered"); await quiet();
});
it("retains startup work through drain and reconstructs an owed retry after shutdown", async () => {
const f = await seedEffects(); admitted = false;
await start(); await quiet(); expect((await effect(f.effect.id)).deliveryState).toBe("pending");
await coordinator!.stop(); coordinator = undefined;
admitted = true; await start(); await flush();
expect((await effect(f.effect.id)).deliveryState).toBe("delivered"); await quiet();
});
});
describe.each(["obsolete policy", "automatic completion"] as const)("%s review scheduling", kind => {
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
let admitted = true;
@@ -116,6 +116,7 @@ const {
reapOrphanedRuns: vi.fn(async () => ({ reaped: 0, runIds: [] })),
registerScheduledRetryCancellationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
registerStoppedInstructionCollectionWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
registerNativePendingEffectWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
registerUnsafeWorkspaceExportWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
registerAutomaticCompletionReviewWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
registerObsoletePolicyReviewWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
@@ -557,6 +558,13 @@ describe("startServer feedback export wiring", () => {
process.env.BETTER_AUTH_SECRET = "test-secret";
});
it.each([false, true])("registers native pending effects only when maintenance is enabled (%s)", async enabled => {
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
await startServer(); await settleStartupHousekeeping();
expect(heartbeatServiceMock.registerNativePendingEffectWork).toHaveBeenCalledTimes(enabled ? 1 : 0);
if (enabled) expect(heartbeatServiceMock.registerNativePendingEffectWork.mock.calls[0][1]()).toBe(true);
});
it.each([false, true])("registers unsafe export recovery only when maintenance is enabled (%s)", async enabled => {
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
await startServer(); await settleStartupHousekeeping();
@@ -997,7 +1005,7 @@ describe("startServer feedback export wiring", () => {
intervalCallback?.();
await new Promise<void>((resolve) => setImmediate(resolve));
expect(heartbeatServiceMock.sweepStaleIssueLocks).toHaveBeenCalledTimes(2);
expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true });
expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true });
expect(heartbeatServiceMock.reconcileCostAccounting).toHaveBeenCalledTimes(1);
expect(retiredDetector).not.toHaveBeenCalled();
} finally {
+3 -1
View File
@@ -1824,6 +1824,8 @@ async function startServerWithDatabaseTeardown(
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) await heartbeat.registerInstructionDirectoryCleanupWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) await heartbeat.registerNativePendingEffectWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) await heartbeat.registerUnsafeWorkspaceExportWork(deliveryWork,
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
if (config.heartbeatSchedulerEnabled) await heartbeat.registerAutomaticCompletionReviewWork(deliveryWork,
@@ -1962,7 +1964,7 @@ async function startServerWithDatabaseTeardown(
// Periodically reap orphaned runs (5-min staleness threshold) and make sure
// persisted queued work is still being driven forward.
runPeriodicRecovery(() => heartbeat
.reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true })
.reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true })
.then(() => heartbeat.promoteDueScheduledRetries())
.then(async (promotion) => {
await heartbeat.resumeQueuedRuns();
@@ -2,6 +2,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db";
/** Topics for existing durable queues; register intent before their writes. */
export const DELIVERY_QUEUES = {
nativePendingEffects: "native-pending-effects",
unsafeWorkspaceExportRecovery: "unsafe-workspace-export-recovery",
automaticCompletionReviews: "automatic-completion-reviews",
obsoletePolicyReviews: "obsolete-policy-reviews",
@@ -1,3 +1,4 @@
import { notifyNativePendingEffects } from "./native-pending-effect-notifications.js";
import { notifyUnsafeWorkspaceExportRunChange } from "./unsafe-workspace-export-notifications.js";
import { notifyCancelledWorkspacePreservation } from "./cancelled-workspace-preservation-notifications.js";
import { notifyInstructionCollection } from "./instruction-collection-notifications.js";
@@ -146,6 +147,7 @@ export async function reconcileAbandonedExecutionControl(
}
// A persisted result belongs to the existing finalizer, not a replacement provider.
if (coordinator?.resultId) {
await notifyNativePendingEffects(tx, { companyId: run.companyId, runId: run.id });
await tx
.update(nativeRunFinalizations)
.set({
+3
View File
@@ -1,3 +1,4 @@
import { registerNativePendingEffectWork } from "./native-runtime/native-pending-effect-work.js";
import { registerUnsafeWorkspaceExportWork } from "./native-runtime/native-workspace-export-recovery.js";
import { registerAutomaticCompletionReviewWork } from "./native-runtime/automatic-completion-reviews.js";
import { registerObsoletePolicyReviewWork } from "./native-runtime/obsolete-policy-reviews.js";
@@ -4088,6 +4089,8 @@ export function heartbeatService(
triggerIssueMonitor,
reportRunActivity: clearDetachedRunWarning,
registerNativePendingEffectWork: (coordinator: Parameters<typeof registerNativePendingEffectWork>[1], canRun: () => boolean) =>
registerNativePendingEffectWork(db, coordinator, () => canRun() && readTaskDrain(new Date()) === null),
registerUnsafeWorkspaceExportWork: (coordinator: Parameters<typeof registerUnsafeWorkspaceExportWork>[1], canRun: () => boolean) =>
registerUnsafeWorkspaceExportWork(db, coordinator, () => canRun() && readTaskDrain(new Date()) === null),
registerAutomaticCompletionReviewWork: (coordinator: Parameters<typeof registerAutomaticCompletionReviewWork>[1], canRun: () => boolean) =>
+2 -1
View File
@@ -1947,7 +1947,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
let settledProcessesDiscovered = false;
async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean }) {
async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean }) {
if (!opts?.skipCancelledWorkspacePreservation) await preserveCancelledNativeWorkspaces(db);
const staleThresholdMs = opts?.staleThresholdMs ?? 0;
const now = new Date();
@@ -1968,6 +1968,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
skipObsoletePolicyReviews: opts?.skipObsoletePolicyReviews,
skipAutomaticCompletionReviews: opts?.skipAutomaticCompletionReviews,
skipUnsafeWorkspaceExports: opts?.skipUnsafeWorkspaceExports,
skipNativePendingEffects: opts?.skipNativePendingEffects,
environmentRuntime,
pluginWorkerManager: dependencies.pluginWorkerManager,
onWorkspaceSettled: settleRecoveredNativeWorkspace,
@@ -0,0 +1,8 @@
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
export type NativePendingEffectScope = { companyId: string; runId?: string };
export async function notifyNativePendingEffects(transaction: object, scope: NativePendingEffectScope) {
await notifyDeliveryWork(transaction, DELIVERY_QUEUES.nativePendingEffects,
JSON.stringify({ companyId: scope.companyId, runId: scope.runId }));
}
@@ -1,3 +1,4 @@
import { pendingNativeStatusEffectsCondition } from "./native-pending-effect-work.js";
import { notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js";
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
@@ -642,6 +643,7 @@ type NativeReconciliationOptions = {
skipObsoletePolicyReviews?: boolean;
skipAutomaticCompletionReviews?: boolean;
skipUnsafeWorkspaceExports?: boolean;
skipNativePendingEffects?: boolean;
pluginWorkerManager?: WorkspaceWorkers;
environmentRuntime?: EnvironmentRuntimeService;
onWorkspaceSettled?: (input: {
@@ -776,6 +778,7 @@ async function reconcileNativeFinalizationsPass(
eq(heartbeatRuns.runtimeMode, "native"),
isNotNull(nativeRunFinalizations.resultId),
notInArray(nativeRunFinalizations.phase, ["terminal_failure"]),
options.skipNativePendingEffects ? not(pendingNativeStatusEffectsCondition(db)) : undefined,
or(
isNull(nativeRunFinalizations.nextAttemptAt),
lte(nativeRunFinalizations.nextAttemptAt, new Date()),
@@ -805,6 +808,7 @@ async function reconcileNativeFinalizationsPass(
))
: [];
if (row.decisionId && pendingEffects.length > 0) {
if (options.skipNativePendingEffects) continue;
const decision = resolveNativeReconciliationStatus({
facts: { undeliveredEffectCount: pendingEffects.length },
priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus,
@@ -1,3 +1,4 @@
import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js";
import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js";
import type { Db } from "@paperclipai/db";
import type { NativeInteractionResponseEnvelope } from "../../vendor/paperclip-runner/index.js";
@@ -631,26 +632,29 @@ export async function routePersistedNativeResultAttention(input: {
const effects = Array.isArray(decisionJson.effects) ? decisionJson.effects.map(record) : [];
const delegation = effects.find((effect) => effect.kind === "create_delegated_issue");
const finalizationError = effects.find((effect) => effect.kind === "record_finalization_error");
await input.db.update(nativeRunFinalizations).set({
phase: finalizationError ? "retryable_failure" : "committed",
assessmentId: existingAssessment.id,
decisionId: existingDecision.id,
leaseOwner: null,
leaseExpiresAt: null,
recoveryState: null,
recoveryRequestId: null,
failureCode: nonEmptyText(finalizationError?.cause),
failureDetail: finalizationError ? {
originalFailureCode: finalizationError.cause,
nextAction: finalizationError.nextAction,
} : null,
nextAttemptAt: finalizationError ? new Date(Date.now() + 30_000) : null,
updatedAt: new Date(),
}).where(and(
eq(nativeRunFinalizations.runId, input.runId),
eq(nativeRunFinalizations.companyId, binding.companyId),
eq(nativeRunFinalizations.issueId, binding.issueId),
));
await input.db.transaction(async tx => {
await notifyNativePendingEffects(tx, { companyId: binding.companyId, runId: input.runId });
await tx.update(nativeRunFinalizations).set({
phase: finalizationError ? "retryable_failure" : "committed",
assessmentId: existingAssessment.id,
decisionId: existingDecision.id,
leaseOwner: null,
leaseExpiresAt: null,
recoveryState: null,
recoveryRequestId: null,
failureCode: nonEmptyText(finalizationError?.cause),
failureDetail: finalizationError ? {
originalFailureCode: finalizationError.cause,
nextAction: finalizationError.nextAction,
} : null,
nextAttemptAt: finalizationError ? new Date(Date.now() + 30_000) : null,
updatedAt: new Date(),
}).where(and(
eq(nativeRunFinalizations.runId, input.runId),
eq(nativeRunFinalizations.companyId, binding.companyId),
eq(nativeRunFinalizations.issueId, binding.issueId),
));
});
receipts.push({
requestId,
route: candidate.route,
@@ -0,0 +1,71 @@
import { and, asc, eq, exists, gt, inArray, isNotNull, ne } from "drizzle-orm";
import { heartbeatRuns, issues, nativeRunFinalizations, statusDecisionEffects, type Db } from "@paperclipai/db";
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
import type { NativePendingEffectScope } from "../native-pending-effect-notifications.js";
import { registerScopedDeadlineWork } from "../scoped-deadline-work.js";
import { dispatchPendingNativeStatusEffects, nativePendingEffectDueAt } from "./status-decision-committer.js";
export function pendingNativeStatusEffectsCondition(db: Db) {
return exists(db.select({ id: statusDecisionEffects.id }).from(statusDecisionEffects).where(and(
eq(statusDecisionEffects.companyId, nativeRunFinalizations.companyId),
eq(statusDecisionEffects.issueId, nativeRunFinalizations.issueId),
eq(statusDecisionEffects.decisionId, nativeRunFinalizations.decisionId),
inArray(statusDecisionEffects.deliveryState, ["pending", "failed"]),
)));
}
function findPendingEffects(db: Db, scope?: NativePendingEffectScope, after?: string) {
return db.select({
id: nativeRunFinalizations.runId, companyId: nativeRunFinalizations.companyId,
issueId: nativeRunFinalizations.issueId, decisionId: nativeRunFinalizations.decisionId,
phase: nativeRunFinalizations.phase, resultId: nativeRunFinalizations.resultId,
nextAttemptAt: nativeRunFinalizations.nextAttemptAt,
leaseOwner: nativeRunFinalizations.leaseOwner, leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt,
}).from(nativeRunFinalizations)
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, nativeRunFinalizations.runId),
eq(heartbeatRuns.companyId, nativeRunFinalizations.companyId)))
.innerJoin(issues, and(eq(issues.id, nativeRunFinalizations.issueId), eq(issues.companyId, nativeRunFinalizations.companyId)))
.where(and(eq(heartbeatRuns.runtimeMode, "native"), isNotNull(nativeRunFinalizations.resultId),
isNotNull(nativeRunFinalizations.decisionId), ne(nativeRunFinalizations.phase, "terminal_failure"),
scope ? and(eq(nativeRunFinalizations.companyId, scope.companyId), scope.runId ? eq(nativeRunFinalizations.runId, scope.runId) : undefined) : undefined,
after ? gt(nativeRunFinalizations.runId, after) : undefined,
pendingNativeStatusEffectsCondition(db),
)).orderBy(asc(nativeRunFinalizations.runId)).limit(100);
}
export function registerNativePendingEffectWork(db: Db, coordinator: ReturnType<typeof createDeliveryWorkCoordinator>, canRun: () => boolean) {
type Entry = { id: string; companyId: string; dueAt: number };
async function load(scope?: NativePendingEffectScope, signal?: AbortSignal): Promise<Entry[]> {
const entries: Entry[] = [];
let after: string | undefined;
while (canRun() && !signal?.aborted) {
const rows = await findPendingEffects(db, scope, after);
for (const row of rows) entries.push({ id: row.id, companyId: row.companyId, dueAt: nativePendingEffectDueAt(row)! });
if (rows.length < 100) return entries;
after = rows[rows.length - 1].id;
}
throw new Error("Native pending effect discovery stopped");
}
return registerScopedDeadlineWork(coordinator, {
queue: DELIVERY_QUEUES.nativePendingEffects, batchSize: 25,
decodeScope: (hint: string): NativePendingEffectScope | undefined => hint === "*" ? undefined : JSON.parse(hint),
matches: (entry: Entry, scope?: NativePendingEffectScope) => !scope || entry.companyId === scope.companyId && (!scope.runId || entry.id === scope.runId),
load,
check: async (entry: Entry, signal?: AbortSignal) => {
if (!canRun() || signal?.aborted) throw new Error("Native pending effect recovery stopped");
const scope = { companyId: entry.companyId, runId: entry.id };
const [current] = await findPendingEffects(db, scope);
if (!current) return null;
const dueAt = nativePendingEffectDueAt(current)!;
if (dueAt > Date.now()) return { ...entry, dueAt };
if (!canRun() || signal?.aborted) throw new Error("Native pending effect recovery stopped");
await dispatchPendingNativeStatusEffects({ db, companyId: current.companyId, issueId: current.issueId,
runId: current.id, decisionId: current.decisionId!, onlyWhenDue: true });
const [pending] = await findPendingEffects(db, scope);
if (!pending) return null;
const nextDueAt = nativePendingEffectDueAt(pending)!;
return { ...entry, dueAt: nextDueAt > Date.now() ? nextDueAt : Date.now() + 30_000 };
},
}, canRun);
}
@@ -1,3 +1,4 @@
import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js";
import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js";
import { isSafeNativeChatProgressEventType } from "../safe-native-chat-progress.js";
import { notifyChatNativeProgress } from "../chat-native-progress-notifications.js";
@@ -522,6 +523,7 @@ export class NativeRunCoordinatorStore {
) {
throw new Error("native_finalization_lease_conflict");
}
if (row.decisionId) await notifyNativePendingEffects(tx, this.#binding);
if (
row.leaseOwner === input.leaseOwner &&
row.leaseExpiresAt &&
@@ -564,6 +566,7 @@ export class NativeRunCoordinatorStore {
throw new Error("native_finalization_retry_invalid");
}
const [updated] = await this.#db.transaction(async tx => {
await notifyNativePendingEffects(tx, this.#binding);
if (input.failureCode === "native_workspace_sync_out_unsafe_archive") {
await notifyUnsafeWorkspaceExportRecovery(tx, this.#binding);
}
@@ -1,3 +1,4 @@
import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js";
import { notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js";
import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js";
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
@@ -214,6 +215,7 @@ async function claimCoordinator(input: {
coordinator.leaseOwner !== leaseOwner
)
throw new Error("native_finalization_lease_busy");
if (coordinator.decisionId) await notifyNativePendingEffects(tx, coordinator);
const [updated] = await tx
.update(nativeRunFinalizations)
.set({
@@ -381,6 +383,7 @@ async function recordRetryableFailure(input: {
? "native_workspace_sync_out_retry_exhausted"
: "native_finalization_retry_exhausted"
: input.failureCode;
await notifyNativePendingEffects(tx, { companyId: input.run.companyId, runId: input.run.id });
if (phase === "terminal_failure") await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, input.run.companyId);
if (phase === "terminal_failure" && failureCode === "native_workspace_sync_out_unsafe_archive") {
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.run.companyId, issueId: input.coordinator.issueId });
@@ -1,3 +1,4 @@
import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js";
import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js";
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
@@ -3897,6 +3898,7 @@ export async function reconcileRetainedNativeSessionCleanup(
renameSync(owned.root, archive);
}
renameSync(copy, owned.root);
await notifyNativePendingEffects(tx, { companyId: owned.run.companyId, runId: owned.run.id });
await tx
.update(nativeRunFinalizations)
.set({
@@ -3954,6 +3956,7 @@ export async function reconcileRetainedNativeSessionCleanup(
.limit(1)
.then((rows) => rows[0]);
if (!current || current.leaseOwner !== leaseOwner) return;
await notifyNativePendingEffects(tx, { companyId: owned.run.companyId, runId: owned.run.id });
await tx
.update(nativeRunFinalizations)
.set({
@@ -9697,6 +9700,7 @@ async function executePaperclipNativeSessionWithinScope(
)
.for("update");
await notifyExecutionRunWork(tx);
await notifyNativePendingEffects(tx, input.execution.binding);
const updated = await tx
.update(nativeRunFinalizations)
.set({
@@ -10031,33 +10035,35 @@ async function executePaperclipNativeSessionWithinScope(
}
}
const releaseNow = new Date();
const released = await timeController("coordinator_release", () => input.db
.update(nativeRunFinalizations)
.set({
leaseOwner: null,
leaseExpiresAt: null,
recoveryState: null,
recoveryRequestId: null,
updatedAt: releaseNow,
})
.where(
and(
eq(nativeRunFinalizations.runId, input.execution.binding.runId),
eq(nativeRunFinalizations.companyId, input.execution.binding.companyId),
eq(nativeRunFinalizations.issueId, input.execution.binding.issueId),
eq(nativeRunFinalizations.leaseOwner, leaseOwner),
eq(nativeRunFinalizations.attempt, attempt),
eq(nativeRunFinalizations.controllerBootId, controller.bootId),
eq(nativeRunFinalizations.controllerPid, controller.pid),
eq(
nativeRunFinalizations.controllerProcessStartedAt,
controller.processStartedAt,
const released = await timeController("coordinator_release", () => input.db.transaction(async tx => {
await notifyNativePendingEffects(tx, input.execution.binding);
return tx.update(nativeRunFinalizations)
.set({
leaseOwner: null,
leaseExpiresAt: null,
recoveryState: null,
recoveryRequestId: null,
updatedAt: releaseNow,
})
.where(
and(
eq(nativeRunFinalizations.runId, input.execution.binding.runId),
eq(nativeRunFinalizations.companyId, input.execution.binding.companyId),
eq(nativeRunFinalizations.issueId, input.execution.binding.issueId),
eq(nativeRunFinalizations.leaseOwner, leaseOwner),
eq(nativeRunFinalizations.attempt, attempt),
eq(nativeRunFinalizations.controllerBootId, controller.bootId),
eq(nativeRunFinalizations.controllerPid, controller.pid),
eq(
nativeRunFinalizations.controllerProcessStartedAt,
controller.processStartedAt,
),
gt(nativeRunFinalizations.leaseExpiresAt, sql`now()`),
),
gt(nativeRunFinalizations.leaseExpiresAt, sql`now()`),
),
)
.returning({ runId: nativeRunFinalizations.runId })
.then((rows) => rows[0] ?? null));
)
.returning({ runId: nativeRunFinalizations.runId })
.then((rows) => rows[0] ?? null);
}));
if (!released) throw new Error("native_session_lease_lost");
const finalization: NativeFinalizationResult = {
schema: "paperclip.native-finalization.v1",
@@ -1,3 +1,4 @@
import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js";
import { alias } from "drizzle-orm/pg-core";
import { notifyUnsafeWorkspaceExportRecovery, type UnsafeWorkspaceExportScope } from "../unsafe-workspace-export-notifications.js";
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
@@ -100,6 +101,7 @@ async function recoverCandidate(db: Db, candidate: { runId: string; companyId: s
phase: "workspace_finalize", status: "succeeded", exitCode: 0, finishedAt: now,
metadata: { owningService: "native_workspace_finalizer", workspaceSync: { omitted: true, reason: "restore_unsafe_archive", legacy: true } },
});
await notifyNativePendingEffects(tx, { companyId: run.companyId, runId: run.id });
await tx.update(nativeRunFinalizations).set({ phase: "result_accepted", failureCode: null, nextAttemptAt: null,
failureDetail: { workspaceFinalizeAttempt: 0, legacyUnsafeExportOmitted: { resultId: result.id, recoveredAt: now.toISOString() } }, updatedAt: now,
}).where(eq(nativeRunFinalizations.runId, run.id));
@@ -1,3 +1,4 @@
import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js";
import { notifyEnvironmentCleanup } from "../environment-cleanup-notifications.js";
import { randomUUID } from "node:crypto";
import type { Environment, EnvironmentLease } from "@paperclipai/shared";
@@ -134,6 +135,7 @@ export async function retryNativeWorkspaceExport(input: {
const now = new Date();
const retry = { ...receipt, requestId: randomUUID(), actorId: input.actorId, repairNote: input.repairNote, requestedAt: now.toISOString(),
stoppedProvider: initial.lease.metadata?.remoteExecutionTermination };
await notifyNativePendingEffects(tx, { companyId: current.run.companyId, runId: current.run.id });
await tx.update(nativeRunFinalizations).set({ phase: "result_accepted", failureCode: null, nextAttemptAt: null,
failureDetail: { ...current.coordinator.failureDetail, workspaceFinalizeAttempt: 0, workspaceExportRetry: retry }, updatedAt: now }).where(eq(nativeRunFinalizations.runId, current.run.id));
await tx.update(heartbeatRuns).set({ status: "running", finishedAt: null, error: null, errorCode: null,
@@ -1,3 +1,4 @@
import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js";
import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js";
import { notifyAutomaticCompletionReviews } from "../automatic-completion-review-notifications.js";
import { notifyObsoletePolicyReviews } from "../obsolete-policy-review-notifications.js";
@@ -80,6 +81,15 @@ export class NativeStatusRaceError extends Error {
}
}
export function nativePendingEffectDueAt(coordinator: {
phase: string; resultId: string | null; nextAttemptAt: Date | null;
leaseOwner: string | null; leaseExpiresAt: Date | null;
}): number | null {
if (coordinator.phase === "terminal_failure" || coordinator.resultId === null) return null;
return Math.max(coordinator.nextAttemptAt?.getTime() ?? 0,
coordinator.leaseOwner !== null ? coordinator.leaseExpiresAt?.getTime() ?? 0 : 0);
}
/** Reconciliation delivery resumes ledger work only; it never re-arbitrates. */
export async function dispatchPendingNativeStatusEffects(input: {
db: Db;
@@ -87,11 +97,17 @@ export async function dispatchPendingNativeStatusEffects(input: {
issueId: string;
decisionId: string;
runId: string;
onlyWhenDue?: boolean;
}) {
return input.db.transaction(async (tx) => {
const coordinator = await tx
.select({
decisionId: nativeRunFinalizations.decisionId,
phase: nativeRunFinalizations.phase,
resultId: nativeRunFinalizations.resultId,
nextAttemptAt: nativeRunFinalizations.nextAttemptAt,
leaseOwner: nativeRunFinalizations.leaseOwner,
leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt,
})
.from(nativeRunFinalizations)
.where(
@@ -107,6 +123,10 @@ export async function dispatchPendingNativeStatusEffects(input: {
if (!coordinator || coordinator.decisionId !== input.decisionId) {
throw new Error("native_pending_effect_decision_binding_mismatch");
}
if (input.onlyWhenDue) {
const dueAt = nativePendingEffectDueAt(coordinator);
if (dueAt === null || dueAt > Date.now()) return [];
}
const pending = await tx
.select({
id: statusDecisionEffects.id,
@@ -134,6 +154,7 @@ export async function dispatchPendingNativeStatusEffects(input: {
targetId: effect.targetId,
});
}
await notifyNativePendingEffects(tx, input);
await notifyChatRunMilestones(tx, input);
await tx
.update(statusDecisionEffects)
@@ -1220,6 +1241,7 @@ async function materializeDecisionEffect(input: {
)
.for("update");
if (pending.length === 0) throw new Error("native_pending_effect_missing");
await notifyNativePendingEffects(input.tx, { companyId: input.companyId });
await input.tx
.update(statusDecisionEffects)
.set({
@@ -1909,6 +1931,7 @@ export async function commitNativeStatusDecision(input: {
effect => effect.kind === "record_finalization_error",
);
await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, input.companyId);
await notifyNativePendingEffects(tx, input);
await tx
.update(nativeRunFinalizations)
.set({
@@ -1,3 +1,4 @@
import { notifyNativePendingEffects } from "./native-pending-effect-notifications.js";
import { and, desc, eq, gte, ne, or, sql } from "drizzle-orm";
import { agentWakeupRequests, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db";
import { readSystemRecovery } from "./system-recovery-state.js";
@@ -45,6 +46,7 @@ export async function resumePreservedResultFinalization(db: Db, companyId: strin
const generationId = intent?.id ?? recovery.generationId;
if (coordinator.recoveryHistory.some(entry => entry.kind === "preserved_result_finalization" &&
entry.generationId === generationId)) return false;
await notifyNativePendingEffects(tx, { companyId, runId });
await tx.update(nativeRunFinalizations).set({
phase: coordinator.assessmentId ? "arbitrating" : "workspace_finalizing",
nextAttemptAt: new Date(), failureCode: null, failureDetail: null, updatedAt: new Date(),