PaperClipAI/doc/idle-decision-effect-recovery.md

Idle decision-effect recovery

Interrupted decision effects now have a dedicated deadline worker. The main maintenance tick no longer scans running decisions. Target invalidation remains in that tick; TTL expiry and continuation delivery have their own workers.

Obligation and activation

The existing decision row is durable truth. A row with status = decided and executionStatus = running owes effect recovery. Its next deadline is updatedAt + PAPERCLIP_DECISIONS_RECOVERY_GRACE_MS, default 60 seconds. Zero is supported; invalid or negative values retain the existing default. Reads do not move the timestamp. Open decisions and completed history create no recovery debt.

The existing writeDecisionChange transaction boundary signals effect recovery for running claims, effect/archive settlement, cancellation, dismissal and expiry. The generic notifyDecisionChange helper signals continuation and expiry only; open-decision creation does not owe effect recovery. Notifications are awaited before writes. Outer commit activates work; rollback does not publish a claim. The additional topic does not wrap provider work or agent wakes in SQL transactions and does not introduce a new queue table.

Committed notification bursts coalesce for up to one second after a successful pass. This bounds sequential tenant writes to at most one notification-driven global pass per window, rather than only coalescing overlapping scans. A saved recovery deadline can preempt that window. New obligations discovered by a signal can wait up to one second; foreground effects still execute immediately. The window alone never schedules work when there is no notification or saved debt.

The supported writer model is the application DB signal owner and its child pools. Independent SQL clients and multiple active writers need a separate notification contract. Startup reconstructs committed claims; it does not observe future writes from unsupported writers.

Ownership and recovery

decision-effect-recovery.ts owns selection, grace, bounded paging, and earliest deadline lookup. decisions.ts owns execution, current membership, effect claims, audit, archive policy, and continuation delivery. The worker adapter only binds that domain contract to the shared coordinator. Startup registers it only when heartbeat scheduling is enabled. The public combined sweep retains its default recovery behavior for existing callers.

Each pass honors the existing configured batch size (default 100, finite values clamped to at least one integer). Its timestamp/ID cursor advances past failed candidates within a frozen final timestamp/ID. Newer due claims wait for the next rotation, so a continuous stream cannot postpone an earlier failure forever. Both cursor positions retain PostgreSQL precision. Genuine execution or settlement failures retain their retry obligation.

A missing saved decider or chosen option is a deterministic malformed outcome. Recovery rechecks it under a row lock, records an audited failed outcome (partial if a completed effect receipt exists), and preserves all effect receipts. It creates no authority and executes no effect for the malformed claim. The failure reason and continuation obligation are saved in the same transaction as the audit. An audit or settlement failure rolls back and leaves the claim pending. A claim repaired before the lock is acquired remains eligible for normal recovery.

The decision service re-reads current status before recovery. Foreground and recovery execution share the existing per-decision flight scoped to the DB owner. Individual effects keep their durable records and transaction advisory locks. Completed effects are not repeated after a failed final decision status write. Recovery derives authority from the saved decider's current active membership. It preserves the current authorization intersection and current target checks.

The scheduler has a 30-second minimum retry spacing for pending or failed work. A first claim or failed retry during drain retains its wake without passive SQL. Shutdown stops subscriptions and checks cancellation between candidates. An already admitted effect/continuation is awaited; later candidates remain durable for restart. No periodic pass remains after all claims settle.

Existing guarantees and limits

Regular effects preserve their existing per-effect transaction boundaries. Continuation delivery retains its separate at-least-once wake/acknowledgement contract. This migration does not establish exactly-once external delivery.

Archive proposals have a separate versioned manifest contract. Recovery still verifies the saved signature/manifest and current authority. A crash after an archive commits but before the decision settles can encounter the changed manifest version on restart. The existing result is a failed decision with an archive error, while the archive remains committed once. The new worker preserves and tests that outcome; it does not claim to repair this preexisting atomicity gap.

Cloud sleep guards remain conservative. The internal next deadline is not a persisted external alarm and does not authorize host sleep with running decisions. This change establishes worker quiet, not actual instance/database suspension.

Verification

The new real-PostgreSQL suite covers 17 cases: empty/open/completed-history quiet, real foreground claim and failed settlement, exact grace and zero override, restart from a claimed effect, active foreground/recovery overlap, outer commit and rollback, first claim and failed retry through drain, revoked authority, continuation failure, aggregate exclusion, fair paging past missing-decider and malformed claims, archive recovery before and after an archive commit, a claim created during delivery, and shutdown between candidates. Quiet assertions observe SQL methods through one hour of fake time after settling.

The combined selection passes 212 tests: the 17 new cases, 50 existing decision service cases, 14 TTL cases, 39 startup cases, and 92 agent-history route cases. The first 15-case recovery run also passed. Workspace typecheck/build and the final server typecheck pass. No failure was suppressed or timeout increased.

The current regenerated migration 0390_salty_killraven.sql adds one partial index concurrently, with IF NOT EXISTS, through the migration owner's top-level statement path. Generation retains the newest five schema snapshots. Fresh application, replay, safety, and numbering checks pass. With 40,000 completed decisions and one running claim, both due selection and earliest-deadline lookup use the partial index and read two buffers (0.012 ms and 0.003 ms in the local probe). The probe transaction is rolled back.

Full PR CI and review must pass separately before this slice is ready.

A follow-up regression seeds a legacy invalid claim with a PostgreSQL-generated microsecond timestamp ahead of a valid claim. With a one-row page, the original Date cursor selected the invalid claim again and starved the valid effect. That regression failed before repair. Recovery now retains the timestamp as database text for cursor comparisons, leaving business grace deadlines unchanged. The unresolved claim remains durable debt and is revisited after paging wraps; the valid claim progresses without repeating its completed effect. The regression now holds a candidate unresolved through the dispatch callback, because known malformed claims receive a durable failed outcome as described above. All 18 effect-recovery lifecycle cases and server typechecking pass after repair.

Foreground join test timing

CI exposed a timing assumption in the foreground/recovery overlap test: it expected exactly two lifetime recovery sweeps. A controlled PostgreSQL replay waited for the committed claim's deadline discovery before advancing time. It recorded two completed, zero-resumption scans before the deadline (startup and claim notification), followed by the legitimate deadline scan. The old assertion then failed deterministically with three calls. No recurring idle poll or duplicate effect was found in that sequence.

The test now waits for discovery, measures the deadline recovery relative to that point, and proves that its promise stays pending while foreground execution holds the effect lock. Advancing through several retry periods cannot start another sweep or effect. After release, recovery completes, one comment and one continuation wake exist, and the worker performs no SQL through the existing hour-long quiet check. Counting actual lock attempts also distinguishes joining from a second execution blocked on the same PostgreSQL advisory lock. A temporary mutation that removed flight joining made this stronger assertion fail; production code was restored before final verification. No runtime code, retry policy, timeout, or business deadline changed.

All 123 tests across the four complete effect-recovery, decision-service, expiry, and startup suites passed, followed by server typechecking. Evidence: /tmp/sleep-effect-join-red.log, /tmp/sleep-effect-join-green.log, /tmp/sleep-effect-join-mutation.log, /tmp/sleep-effect-join-owning-tests.log, and /tmp/sleep-effect-join-root-types.log with their status files. The initial failure and mutation failure are expected evidence, not passing runs. Remaining review findings and fresh stack CI remain separate acceptance requirements.

Recovery review repairs (October 26)

Two regressions failed before repair: an old failure was not revisited while newer due claims kept each page full, and creating an open decision caused an unnecessary effect-recovery scan. Frozen rotation boundaries now preserve both retry fairness and forward progress. Creation still signals its continuation/expiry consumers; the effect queue is signaled by the existing claim/settlement write boundary.

Four malformed-outcome cases also failed before repair. Recovery now records effectRecoveryError and a decision.recovery_failed activity atomically. Live activity publication follows commit. Existing completed effect receipts remain unchanged, and the normal continuation owner delivers the result. A real audit insert failure leaves both status and audit uncommitted, then later recovery settles successfully. A real PostgreSQL row-lock test repairs a claim while settlement waits; the locked recheck leaves that claim running, and its valid effect executes once on the next pass.

The shared execution flight now loads the current claim and derives recovery authority together. Recovery no longer performs a redundant full-row lookup before entering the flight. Foreground overlap, current membership checks, per-effect locks, continuation behavior, and the resumption count are preserved. Only the two known malformed states are terminalized; other failures retain their existing retry policy. No schema, queue, or external-wake contract is added.

All 24 recovery lifecycle cases and server typechecking pass. Controlled failures and final evidence are retained in /tmp/sleep-effect-fairness-red.log, /tmp/sleep-effect-invalid-red.log, /tmp/sleep-effect-owner-tests.log, and /tmp/sleep-effect-owner-types.log. Broader stack and remote acceptance remain separate requirements; these results do not prove host suspension.

The owning branch also passes all 129 cases across the four complete recovery, decision-service, expiry, and startup suites. Database-backed suites use separate migrated disposable databases. See /tmp/sleep-effect-review-owner-acceptance.log.

Bounded notification review repair

Ten sequential real cancellation transactions reproduced ten extra recovery scans. After enabling the existing shared notification-coalescing capability, the same transactions cause one follow-up scan and then return to SQL quiet. A separate real-database case preserves an already loaded deadline 500 milliseconds away. The initial deadline fixture observed the scheduler while its SQL pass was still active; waiting for that read removed the fixture race before the baseline was recorded. The cancellation amplification still failed before the production fix.

All 131 tests in the four complete recovery, decision-service, expiry and startup suites pass at the owning branch, as does server typechecking. The helper's optional coalescing capability is promoted from the later plugin scheduling slice into this PR; consumers that omit it keep their previous behavior. The integrated stack is verified separately before publication. The original full-suite and external-wake limitations remain.

Current parent integration

The expiry and continuation review repairs are integrated normally. The running claim index is regenerated as 0390 after parent 0388/0389, with concurrent creation registered in the existing executor. All 388 parent journal entries are unchanged; the five newest generated snapshots remain. Schema drift, numbering/safety and fresh migration apply/replay pass. No generated snapshot was hand-edited.

All 141 cases in the five complete effect-recovery, decision/continuation, expiry, startup and scoped-deadline suites pass. This includes all 26 recovery cases and the parent's deterministic cross-pool archive contention test. Server types and build pass. The first integrated run passed 140 cases and failed an old scheduler assertion: the coordinator now also holds an earlier continuation-coalescing deadline. The overlap test services that earlier registered wake, then still requires the original effect grace deadline, one active effect flight, two total advisory-lock attempts (test gate plus foreground), one committed effect and one continuation. No production timing or retry was changed to pass the test.

Existing one-second effect-notification coalescing and thirty-second failure retry remain. Continuation notification coalescing is owned separately by the parent. Extra upstream embedded-only tests attempted during parent integration could not initialize locally; fresh current-head CI must cover that environment. Full CI and both automated reviews remain merge gates.

After normal integration of master-target expiry parent f7c61e9912, the effect-recovery owner, worker, decision execution and notification implementations are unchanged. All 69 current recovery/startup/deadline cases and server types/ build pass with the updated Runner vendor. The earlier full 141-case selection remains evidence at its stated revision. Parent migration history is an exact prefix and five generated snapshots remain. Fresh head CI and both automated reviews are required after this parent integration.

Landed expiry integration and CI interruption

Expiry-only creation notification from aa03d2c4c2 is integrated. All 143 cases in the complete recovery, decision, expiry, startup and scoped-deadline suites pass, including single/bundle creation without continuation signals. The normal merge of landed expiry 3421366903 also preserves all 21 independently merged master files. The same 143 cases pass after that reconciliation.

The previous candidate f54a707d74 had both reviews at 5/5, but CI run 40597921712 failed because server shard 6 was interrupted. Its test step was cancelled and orphan cleanup ran; no failing test assertion or suite summary was reported. The workflow eventually completed with that shard and its dependent verify gate failed. It was not retried or presented as passing. The necessary expiry parent repair creates a new candidate with fresh CI and reviews required.