PaperClipAI/doc/idle-decision-expiry.md

Decision TTL expiry while idle

Open decisions already contain an expiresAt deadline. The expiry worker now uses that timestamp instead of scanning TTL candidates on each maintenance tick. This slice owns time-based expiry only. Target invalidation, interrupted effect recovery, and retention remain separate duties.

Producers and transaction ownership

Creation and bundle creation call notifyDecisionCreated before the decision insert in their existing transaction. It signals expiry only: an open row cannot owe continuation delivery yet. Existing decision updates share notifyDecisionChange, which now signals expiry as well as continuation delivery. Cancellation, dismissal, and execution claims therefore remove obsolete expiry deadlines. All production inserts and updates are in the decision owner; acknowledgement only changes terminal delivery metadata and cannot change an open expiry. No supported API currently edits an existing decision's expiry timestamp. A future such writer must use this same notification boundary.

The helper adds no recovery wrappers to routes. Outer transactions and nested savepoints retain the shared commit/rollback and uncertain-outcome protocol. Creation without a committed row cannot publish a business deadline. Startup reconstructs saved deadlines, including already-overdue work. Independent SQL writers and another active server process need their own wake contract.

decision-expiry.ts owns due selection, the next deadline, and expiry settlement. decisions.ts delegates expiry-on-use and the existing target-driven expiration transition to that owner. Target selection remains outside the TTL worker. Startup registers TTL work before the combined decision sweep. The periodic combined sweep opts out of TTL selection and cannot expire a row merely because its TTL elapsed. Its public default still includes TTL and preserves the configured combined-sweep batch limit.

Deadline, settlement, and recovery

The next deadline is the earliest open expiresAt. A partial index on (expires_at, id) covers open decisions. The query selects one indexed row; retained terminal history does not enlarge the deadline lookup. Expiry is due when expiresAt <= now, including equality. The old combined path skipped the exact equality despite selecting that row; this migration fixes that boundary.

Each worker pass honors PAPERCLIP_DECISIONS_SWEEP_BATCH_SIZE (default 100), ordered by expiry and ID. Its in-memory cursor advances across failures and wraps after the last page. Remaining overdue work retains a one-second continuation through the saved earliest deadline after a successful pass. Failures retain the existing 30-second retry throttle. New earlier creation or resolution of the earliest decision causes deadline reconciliation. When only a future expiry remains, the worker does not query again until that deadline or a committed change. No open decisions means no recurring expiry work.

Settlement requires the row to remain open. TTL settlement also rechecks that the saved deadline is due. Metadata is merged into the current row. The existing expiry activity is persisted in the same transaction as the terminal transition. Publication and origin continuation follow commit. If the audit insert fails, the expiry and continuation obligation both roll back. A deadline-driven expiry records ttl; target-driven expiration retains its own reason. When both causes are possible, the first successful terminal transition determines the reason. Continuation failures belong to the continuation worker after the decision is expired. Neither owner replays decision effects.

Drain and standby defer dispatch while retaining the request. The worker checks the coordinator's AbortSignal between candidates. Shutdown finishes the admitted expiry and leaves untouched candidates for restart. It does not process the whole backlog before stopping. There is no new queue table or permanent DB listener, and no provider call is held inside a new transaction.

Limits

The disabled-heartbeat branch keeps its existing behavior and does not register this worker. Expiry-on-use remains enforced independently of background work. The in-process deadline is not an external Cloud alarm. Existing conservative sleep blockers remain; this migration does not permit an instance to sleep with unsupported future obligations. Worker quiet is not proof of host or database suspension.

Verification

The original combined selection passed 103 tests: 14 initial real-PostgreSQL expiry worker cases, 50 decision service/continuation cases, and 39 startup cases. It covered empty and terminal-history quiet, actual create and bundle producers, exact expiry, moving the next deadline earlier, cancellation, restart, outer rollback and commit, first creation during drain, failed retry through drain, failed wake delivery, separate aggregate/TTL ownership, 101 candidates with a failed first row, creation during delivery, and shutdown/restart with outstanding candidates. The parent archive race and bounded continuation shutdown regressions also pass.

The first selection had 97 passes and one fixture failure. The fault test cleared its error collector before an already-admitted failed query had finished. It now checks all collected causes after recovery settles. The next selection passed 101 cases; after the parent review repairs and bounded expiry/shutdown coverage, all 103 pass. No production hold, retry, or timeout was weakened.

The original index applied and replayed successfully; after normal parent integration it is regenerated as 0389_brave_hulk.sql. Fresh apply also passes. With 40,000 synthetic decisions and one open row, the next-deadline query uses the partial index, reads two shared buffers, and executes in 0.022 ms locally. The probe transaction rolls back. This is query evidence, not a production latency or savings claim. Workspace typecheck and build pass on the combined branch. Full PR CI and fresh review remain required before readiness.

A follow-up paging regression uses 101 overdue rows sharing a PostgreSQL microsecond timestamp, with the first 100 writes failing. The original Date-based cursor rounded the timestamp to milliseconds and repeatedly selected that failed page, starving the last valid row. The cursor now carries PostgreSQL timestamp text into its tuple comparison; business deadline calculations still use Date. The regression failed before the fix and verifies progress past the failed page, then recovery of all failed rows after the fault is removed. The repaired expiry suite passes all 15 cases, and server typechecking passes.

Current review repairs and verification

Three review findings failed real PostgreSQL regressions before repair: the standalone worker ignored the configured batch limit, a successful overdue backlog waited thirty seconds between batches, and a failed audit write left the row expired without durable activity. The worker now honors its existing batch configuration and continues successful overdue work after one second. The shared deadline adapter accepts an optional continuationMs; omitted values preserve every existing consumer's retry interval. Errors still retain the existing thirty-second retry. Expiry and durable activity commit atomically, then publish activity and deliver the origin continuation. Provider calls stay outside that transaction.

All 129 selected cases pass: 18 expiry, 53 decision/continuation, 15 archive, 42 startup and one scoped-deadline case. The archive selection exercises an existing shared-deadline consumer with the new option absent. Server typecheck/build and generated snapshot/schema drift pass. Parent journal entries remain unchanged and five generated snapshots remain. No snapshot was hand-edited.

The first complete selection passed 113 cases and failed the future-startup assertion because it inspected the scheduler before an additional committed notification reconciliation finished. Coordinator ready awaits the first admitted pass, while a notification arriving during that pass can schedule another immediate pass. The assertion now waits for reconciliation to expose the exact saved future deadline, then still proves the decision does not expire early. The three production regressions and this fixture failure remain recorded; no error retry or business deadline was relaxed to pass the tests. Fresh current-head CI and reviews remain merge gates.

After normal integration of continuation recovery's final coalescing and deterministic contention tests and pinned master 8161f2a813, all 130 selected cases pass: 76 expiry/archive/startup/deadline cases plus all 54 decision cases. Server types/build pass. Existing notification coalescing remains thirty seconds for continuation recovery; expiry's successful backlog continuation is one second, and failure retry remains thirty seconds. Migration replay reports no pending migrations and the parent journal remains an exact prefix. Extra upstream embedded-only tests attempted during parent integration could not initialize locally; fresh CI must cover that environment, as recorded in the archive parent PR.

Normal integration with master-target continuation parent 18f682863c preserves expiry, deadline and producer implementations exactly. All 61 expiry/startup/ scoped-deadline cases and server types/build pass with the rebuilt Runner vendor. The previous full 130-case selection remains evidence at its stated revision. No migration SQL changed, and five generated snapshots remain. Fresh head CI and reviews are required after this parent integration.

Creation notification review repair

Single and bundle creation previously notified continuation recovery even though the inserted decision was open. Two real producer regressions failed before repair because continuation received intent and settlement. Creation now uses the existing central notification file's expiry-only notifyDecisionCreated helper. Update and terminal transitions keep notifyDecisionChange, including the normal continuation signal. No optional flags, per-route bookkeeping or new table is needed. Both regressions verify the exact future expiry, no early wake, and a subsequent cancellation that still signals and delivers continuation normally.

All 117 current expiry/decision/startup/deadline cases pass, including 20 expiry cases. Server types/build pass. The preceding f7 candidate's identical-tree merge proof remains history; this subsequent creation-signal repair changes runtime behavior and has its own acceptance above. Fresh current-head CI and both reviews are required.