fix(server): wake failed retry notices from receipt changes

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed 2026-10-25 14:10:02 -05:00
1 parent 7c0246893f
commit cbfaab918a
13 files changed
+462 -20

No files matched your search

@@ -1774,3 +1774,16 @@ voice inventory amplification findings 4371283330/4371307607, and #17580 still
has unresolved company-offboarding/cleanup findings 4370459800/4370543984. These
are required repairs, not exemptions. The full original migration scope,
master reconciliation, and all latest-head gates remain required.
## Slice 31 worker and producer checkpoint (2026-10-25)
Failed-retry notices now register a scoped worker and leave periodic publication
maintenance. Real missed wakes were reproduced for admission, agent/tree
cancellation, resumed-admission denial, early admission refusal, and requester
restoration. Their owning transactions now notify through the existing shared
boundary. Fourteen selected lifecycle cases and all 1,503 chat/startup acceptance
cases pass. Workspace typecheck and build pass. See the retry audit for retained
failures, exact commands/logs, and native source-cleanup qualification still open.
This is a draft checkpoint, not a merge-ready claim. Older parent gates and every
remaining original recurring-server duty stay in scope. No merge occurred.
@@ -3,9 +3,10 @@
Date: 2026-10-25. Branch: `codex/sleep-31-failed-retry-notices`.
Parent: [#17647](https://github.com/paperclipai/paperclip/pull/17647),
`fee07f3fa8cbc489cb9bbf657a6065c6e00748c3`.
This is slice 31 of the recurring-server-work stack. No production worker is
registered yet, and the periodic retry-notice pass remains until producer and
lifecycle acceptance is complete. No PR has been opened for this slice yet.
This is slice 31 of the recurring-server-work stack. The branch now registers
a scoped worker and removes retry notices from periodic publication maintenance.
Producer audit and full acceptance remain in progress; this is not merge-ready.
No PR has been opened for this slice yet.
## Obligation and boundary
@@ -139,3 +140,94 @@ Additional producer audit findings to resolve before wiring:
the separate task-intent receipt. The notice candidate uses the chat-action
receipt; examine generation/source authorization before adding blanket wakes
to all intent timestamp writes.
## Admission and cancellation migration checkpoint
The worker now uses the shared scoped-deadline coordinator. It reconstructs
existing receipt obligations at startup, resets an issue cursor on a committed
invalidation, and retains bounded pages, real contention, and failed selection.
Settled and denied history has no recurring scan. No new table is added.
`notifyChatInboundWakeNotices` now also notifies the retry-notice topic. This
keeps task-tree, agent, budget, company, deferred-queue, source/privacy, and
existing milestone authority signals at one boundary. The hint serializes only
company and optional issue IDs. The new worker is absent from older parents;
there it creates no new timer or database listener.
New producer changes are narrow:
| Owner | Obligation and notification |
| --- | --- |
| Retry admission callback | After current authorization succeeds, register intent inside the heartbeat transaction before the receipt write. |
| Retry authorization denial | Retire the exact unadmitted receipt in a transaction that registers notice intent before its conditional update. No provider I/O enters the transaction. |
| Heartbeat early refusal | A durable chat receipt can be skipped before its authorization callback. Signal its existing execution-wait transaction, or create one transaction for the previously standalone skipped-row insert. Non-chat callers retain their existing path. |
| Deferred-queue requester merge | A manual requester replacement can restore or revoke the original receipt identity. Register intent before that update. Payload-only merges do not add a notification. |
Causal evidence retained:
- `/tmp/sleep31-cancellation-before.log`: agent cancellation missed the notice;
the tree case initially used an invalid direct function reference.
- `/tmp/sleep31-tree-cancellation-before.log`: corrected real task-tree service
also reproduces the missing notice. The shared notification repair passes
39 related cases in `/tmp/sleep31-cancellation-fixed.log`.
- `/tmp/sleep31-real-admission-lifecycle.log`: real heartbeat admission missed
the notice in normal, drain, and failed-selection modes. Restart alone found
it. The callback notification fixes that causal gap.
- `/tmp/sleep31-real-admission-fixed.log`: one normal-path assertion observed
an unexpected worker error without printing it. This error was not classified
as a flake. The assertion now shows full errors. The diagnostic run instead
exposed the expected failure's AggregateError wrapper; that expectation now
checks its nested cause. Retain the first observation for broad acceptance.
- `/tmp/sleep31-denial-before.log`: the actual recovery service restages a
deferred retry; a controlled admission rejection saves `failed`, but no
not-started notice appears. The conditional transaction notification fixes it.
- `/tmp/sleep31-early-refusal-before.log`: real heartbeat admission with
wake-on-demand disabled saves a skipped receipt but misses its notice. The
early-refusal writer notification fixes it.
- `/tmp/sleep31-requester-before.log`: the real deferred-queue Postgres writer
restores the original requester but misses the notice. This exercises the
producer transaction, not a synthetic notification. It does not establish
every end-to-end ordinary/retry coalescing path.
- `/tmp/sleep31-producer-lifecycle.log`: all 14 selected cases pass, including
real admission, first-write drain/resume, failed selection followed by drain,
stop/restart, agent/tree rollback and commit, admission denial, early refusal,
privacy restoration after idle and during a bounded page, and requester merge.
The quiet assertions observe no root SQL/transaction calls for 120 seconds.
- `/tmp/sleep31-workspace-typecheck.log` and
`/tmp/sleep31-workspace-build.log`: workspace typecheck and build pass.
The full chat/startup suite passes all 1,503 cases on fresh migrated
`paperclip_sleep31_wired_chat_20261025_test`, with evidence in
`/tmp/sleep31-wired-full-chat.log` (456.47 seconds, exit 0). This includes
the existing provider authorization and publication acceptance suite.
The stopped-worker case also checks that publication maintenance no longer
projects retry notices before startup reconstruction resumes the worker.
### Remaining producer qualifications
`readTaskChatRetryIntent` uses saved identity, generation, source, actor and
binding. It does not use the separate task-intent status or updatedAt. Ordinary
status reconciliation therefore does not itself restore notice eligibility.
Restaging the chat action from processed to issued also stays within the set of
statuses already allowed for publication. Changes that bind a receipt to a run
remove it from this notice obligation; the projection rechecks before writing.
`assertFailedRetryReceipt` validates stable receipt identity, not normalized
queued-comment payload. Payload-only comment normalization is not an authority
restoration for retry notices. The requester replacement is different and is
now covered at its actual writer. Existing ordinary-source notice behavior is
still included through the shared signal fanout.
Native exact-retry source authorization needs further qualification before
readiness: it reads native finalization ownership, cleanup history, environment
lease settlement, and retained checkpoint proof. `native-session-executor.ts`
commits a settled maintenance history entry and then removes its activation
file; `environments.ts::releaseLease` uses environment-cleanup notifications.
Determine which of these can restore an already-existing retry notice and prove
the required wake after the full authority boundary. A signal before the final
filesystem handoff may be too early. Do not claim all native cleanup producers
are covered by the general milestone fanout merely because run finalizers use it.
All older stack failures and original recurring-server duties remain required.
No merge, expanded Cloud wake coverage, or deployed host sleep is claimed.