fix(server): wake chat milestones after native dispatch receipts commit

Co-Authored-By: Paperclip <noreply@paperclip.ing>

doc/plans/2026-10-24-recurring-server-sleep-stack.md

@@ -1442,3 +1442,14 @@ server production types pass. The slice audit preserves the initial invalid
 staleness fixture and corrected causal evidence. Periodic projection remains
 until all supported producers are covered. Remaining original inventory, open
 PR gates/findings and final master reconciliation are unchanged.
+
+
+Slice 28 native wake receipt checkpoint: interrupted recovery, same-scope
+coalescence and fresh admission receipt writes now notify from narrow database
+transactions. Both missed-recovery and admission-before-receipt races were
+reproduced. All 56 milestone/queue cases, 13 existing dependency cases and server
+production types pass; four native cases were rerun after fixing external-test
+fixture cleanup. The audit preserves setup failures and provider-acceptance
+limits. No periodic projector has been removed yet. Fresh remote inspection
+still finds #17590 Paperclip Review queued, #17592 neutral, and #17490 E2E shard
+3 plus aggregate failing. Those gates and all remaining original scope stay open.

doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md

@@ -435,3 +435,51 @@ Queued-comment changes, raw queue cancellation and wake-receipt writers, native
 review admission acceptance, and remaining dependency producers are still open.
 The periodic combined projector remains in production; this checkpoint does
 not claim slice publication, full-suite/PR acceptance, or instance sleep.
+
+
+## Native wake dispatch receipt checkpoint
+
+`dispatchPendingNativeStatusWakeups` now registers milestone intent when it
+repairs an interrupted dispatch receipt, coalesces another same-scope intent,
+or records the result of fresh admission. Each receipt UPDATE has a narrow
+transaction; enqueue/provider execution remains outside it. Existing status,
+claim, provenance, and update guards are unchanged. The parsed issue identity
+already owned by this dispatcher scopes the notification (company fallback
+when absent). Claims, retry timestamps, and rejected targets do not acquire
+milestone intent just because they were inspected.
+
+The existing continuation reader requires both the delivered decision effect
+and the coalesced original wake plus matching dispatch receipt. Consequently,
+the run-admission notification alone is insufficient: it can be consumed while
+the original intent is still claimed. The new normal-dispatch regression forces
+that ordering using the dispatch suppression callback, then verifies the final
+receipt reactivates projection without starting provider execution.
+
+Evidence and limitations:
+
+- `/tmp/sleep28-native-wake-receipt-before.log`: interrupted dispatch recovery
+  linked the run successfully but its queued notice remained absent.
+- `/tmp/sleep28-native-wake-fresh-without-signal.log`: removing only the final
+  receipt notification reproduced a missing notice after the early admission
+  sweep. The production signal was restored in a finally block.
+- `/tmp/sleep28-native-wake-restored-tests.log`: all 56 milestone/queue cases
+  pass (31 milestone cases, 25 existing queue cases), including normal and
+  recovered receipts, repeat recovery quiet, and drain-held dispatch.
+- `/tmp/sleep28-native-wake-dependency-tests.log`: all 13 existing dependency
+  scheduling cases pass on a new isolated PostgreSQL database. The suite logs
+  responsible-user setup denial for some existing fixture paths; this is not
+  evidence of successful provider execution or full product acceptance.
+- The dependency suite now accepts guarded `PAPERCLIP_DEPENDENCY_TEST_DATABASE_URL`
+  and removes its fixed fixture user/closes its client during teardown. Its
+  existing broad cleanup must run only on a dedicated database. Initial reuse
+  retained the old fixture user; a manual cleanup command used the wrong physical
+  table name, and the following run failed in setup with duplicate `user_pkey`:
+  `/tmp/sleep28-native-wake-dependency-final.log`. Corrected teardown removed the
+  fixture; a fresh read confirmed zero rows before the successful four native
+  cases in `/tmp/sleep28-native-wake-dependency-clean.log` (nine unselected).
+- Server production typecheck passes: `/tmp/sleep28-native-wake-typecheck.log`.
+  Test files remain outside that typecheck. No whole-slice PR acceptance claimed.
+
+Raw queue cancellation, queued-comment/lineage changes, native reviewer claim
+acceptance, and the remaining dependency producer audit still prevent retiring
+the production periodic projector. All original stack scope remains required.

server/src/__tests__/chat-run-milestone-work.test.ts

@@ -373,6 +373,55 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu
     await expect(dispatchPendingNativeStatusEffects({ db, ...input, decisionId: decision.id })).resolves.toEqual([]);
     await quiet();
   });
+  it("wakes an admitted continuation when native wake dispatch repairs its original receipt", async () => {
+    const input = await continuationDecisionFixture();
+    await commitNativeStatusDecision({ db, ...input });
+    const [intent] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId));
+    const target = await run(f, { status: "queued", contextSnapshot: {
+      issueId: f.issueId, source: "native_status_decision", nativeStatusWakeIntentId: intent.id,
+    } });
+    await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",
+      reason: "issue_status_changed", requestedByActorType: "system", requestedByActorId: `native-status-wake-dispatch:${intent.id}`,
+      runId: target.id, status: "queued", idempotencyKey: intent.idempotencyKey, payload: { issueId: f.issueId },
+    });
+    await start(); await flush();
+    expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${target.id}:`))).toBe(false);
+    await quiet();
+    const { service, dependencies } = await queue();
+    await expect(service.dispatchPendingNativeStatusWakeups({ companyId: f.companyId })).resolves.toMatchObject({ recovered: 1, dispatched: 0 });
+    await flush();
+    expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${target.id}:queued:${f.endpointId}`);
+    expect(dependencies.executeRun).not.toHaveBeenCalled();
+    await quiet();
+    await expect(service.dispatchPendingNativeStatusWakeups({ companyId: f.companyId })).resolves.toMatchObject({ scanned: 0 });
+    await quiet();
+  });
+  it("rechecks a fresh native continuation after admission finishes its wake receipt", async () => {
+    const input = await continuationDecisionFixture();
+    await commitNativeStatusDecision({ db, ...input });
+    await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, input.runId));
+    await start(); await flush(); await quiet();
+    const { service, dependencies } = await queue();
+    let admittedId: string | undefined;
+    dependencies.getSchedulingSuppression.mockImplementation(async () => {
+      const rows = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId));
+      const admitted = rows.find(row => row.id !== input.runId);
+      if (admitted && !admittedId) {
+        admittedId = admitted.id;
+        // Admission already committed and signalled, but the original wake
+        // receipt is still claimed. Let that first projection finish now.
+        await flush();
+        expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${admittedId}:`))).toBe(false);
+      }
+      return { suppressed: true, reason: "task_drain" };
+    });
+    await expect(service.dispatchPendingNativeStatusWakeups({ companyId: f.companyId })).resolves.toMatchObject({ dispatched: 1 });
+    expect(admittedId).toBeDefined();
+    await flush();
+    expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${admittedId}:queued:${f.endpointId}`);
+    expect(dependencies.executeRun).not.toHaveBeenCalled();
+    await quiet();
+  });
   it("projects a continuation only after the real question delivery receipt commits", async () => {
     const source = await run(f, { status: "succeeded" });
     const interactions = issueThreadInteractionService(db);

server/src/__tests__/heartbeat-dependency-scheduling.test.ts

@@ -37,6 +37,7 @@ import {
   getEmbeddedPostgresTestSupport,
   startEmbeddedPostgresTestDatabase,
 } from "./helpers/embedded-postgres.js";
+import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
 import { heartbeatService } from "../services/heartbeat.ts";
 import { runningProcesses } from "../adapters/index.ts";
 
@@ -63,7 +64,8 @@ vi.mock("../adapters/index.ts", async () => {
   };
 });
 
-const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
+const external = resolveExternalTestDatabaseUrl("PAPERCLIP_DEPENDENCY_TEST_DATABASE_URL");
+const embeddedPostgresSupport = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
 const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
 
 if (!embeddedPostgresSupport.supported) {
@@ -103,8 +105,8 @@ describeEmbeddedPostgres("heartbeat dependency-aware queued run selection", () =
   let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
 
   beforeAll(async () => {
-    tempDb = await startEmbeddedPostgresTestDatabase("paperclip-heartbeat-dependency-scheduling-");
-    db = createDb(tempDb.connectionString);
+    if (!external) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-heartbeat-dependency-scheduling-");
+    db = createDb(external ?? tempDb!.connectionString);
     await db.insert(authUsers).values({ id: "responsible-user", name: "Fixture owner", email: "owner@example.test", createdAt: new Date(), updatedAt: new Date() });
     heartbeat = heartbeatService(db);
     await ensureIssueRelationsTable(db);
@@ -185,6 +187,8 @@ describeEmbeddedPostgres("heartbeat dependency-aware queued run selection", () =
   });
 
   afterAll(async () => {
+    await db?.delete(authUsers).where(eq(authUsers.id, "responsible-user"));
+    await db?.$client.end();
     await tempDb?.cleanup();
   });
 

server/src/services/heartbeat/queue.ts

@@ -4659,6 +4659,13 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
     >();
 
     for (const candidate of candidates) {
+      const payload = parseObject(candidate.payload);
+      const wakeContext = parseObject(payload._paperclipWakeContext);
+      const issueId =
+        readNonEmptyString(payload.issueId) ??
+        readNonEmptyString(payload.taskId) ??
+        readNonEmptyString(wakeContext.issueId) ??
+        null;
       const dispatchActorId = `native-status-wake-dispatch:${candidate.id}`;
       const existingDispatch = await db
         .select()
@@ -4680,24 +4687,27 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
           : existingDispatch.status === "deferred_issue_execution"
             ? "coalesced"
             : existingDispatch.status;
-        await db
-          .update(agentWakeupRequests)
-          .set({
-            status: recoveredStatus,
-            runId: existingDispatch.runId,
-            finishedAt:
-              (existingDispatch.runId ?? existingDispatch.finishedAt)
-                ? (existingDispatch.finishedAt ?? now)
-                : null,
-            error: existingDispatch.error,
-            updatedAt: now,
-          })
-          .where(
-            and(
-              eq(agentWakeupRequests.id, candidate.id),
-              isNull(agentWakeupRequests.runId),
-            ),
-          );
+        await db.transaction(async tx => {
+          await notifyChatRunMilestones(tx, { companyId: candidate.companyId, issueId: issueId ?? undefined });
+          await tx
+            .update(agentWakeupRequests)
+            .set({
+              status: recoveredStatus,
+              runId: existingDispatch.runId,
+              finishedAt:
+                (existingDispatch.runId ?? existingDispatch.finishedAt)
+                  ? (existingDispatch.finishedAt ?? now)
+                  : null,
+              error: existingDispatch.error,
+              updatedAt: now,
+            })
+            .where(
+              and(
+                eq(agentWakeupRequests.id, candidate.id),
+                isNull(agentWakeupRequests.runId),
+              ),
+            );
+        });
         recovered += 1;
         continue;
       }
@@ -4731,13 +4741,6 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
         .returning({ id: agentWakeupRequests.id });
       if (claimed.length === 0) continue;
 
-      const payload = parseObject(candidate.payload);
-      const wakeContext = parseObject(payload._paperclipWakeContext);
-      const issueId =
-        readNonEmptyString(payload.issueId) ??
-        readNonEmptyString(payload.taskId) ??
-        readNonEmptyString(wakeContext.issueId) ??
-        null;
       const scopeKey = issueId
         ? `${candidate.companyId}:${candidate.agentId}:${issueId}:${readNativeReviewAssignmentContext(wakeContext)?.nativeReviewInteractionId ?? ""}`
         : null;
@@ -4745,16 +4748,19 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
         ? deliveredByIssueScope.get(scopeKey)
         : null;
       if (priorDelivery) {
-        await db
-          .update(agentWakeupRequests)
-          .set({
-            status: "coalesced",
-            runId: priorDelivery.runId,
-            finishedAt: new Date(),
-            error: null,
-            updatedAt: new Date(),
-          })
-          .where(eq(agentWakeupRequests.id, candidate.id));
+        await db.transaction(async tx => {
+          await notifyChatRunMilestones(tx, { companyId: candidate.companyId, issueId: issueId ?? undefined });
+          await tx
+            .update(agentWakeupRequests)
+            .set({
+              status: "coalesced",
+              runId: priorDelivery.runId,
+              finishedAt: new Date(),
+              error: null,
+              updatedAt: new Date(),
+            })
+            .where(eq(agentWakeupRequests.id, candidate.id));
+        });
         recovered += 1;
         continue;
       }
@@ -4853,21 +4859,24 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
         // promote both rows (the original has no nativeStatusWakeIntentId
         // provenance and would otherwise run once before the dispatch receipt).
         const deliveredStatus = delivered?.status ?? "queued";
-        await db
-          .update(agentWakeupRequests)
-          .set({
-            status: wakeRun || deliveredStatus === "deferred_issue_execution"
-              ? "coalesced"
-              : deliveredStatus,
-            runId: wakeRun?.id ?? delivered?.runId ?? null,
-            finishedAt:
-              wakeRun || delivered?.finishedAt
-                ? (delivered?.finishedAt ?? new Date())
-                : null,
-            error: delivered?.error ?? null,
-            updatedAt: new Date(),
-          })
-          .where(eq(agentWakeupRequests.id, candidate.id));
+        await db.transaction(async tx => {
+          await notifyChatRunMilestones(tx, { companyId: candidate.companyId, issueId: issueId ?? undefined });
+          await tx
+            .update(agentWakeupRequests)
+            .set({
+              status: wakeRun || deliveredStatus === "deferred_issue_execution"
+                ? "coalesced"
+                : deliveredStatus,
+              runId: wakeRun?.id ?? delivered?.runId ?? null,
+              finishedAt:
+                wakeRun || delivered?.finishedAt
+                  ? (delivered?.finishedAt ?? new Date())
+                  : null,
+              error: delivered?.error ?? null,
+              updatedAt: new Date(),
+            })
+            .where(eq(agentWakeupRequests.id, candidate.id));
+        });
 
         if (scopeKey) {
           deliveredByIssueScope.set(scopeKey, {