From 706beb01f2a8363ef9bf107fbd7e27bdc68547e8 Mon Sep 17 00:00:00 2001 From: Dotta Date: Sat, 24 Oct 2026 14:04:03 -0700 Subject: [PATCH] test(server): verify Grok dispatch lifecycle and quiet history Co-Authored-By: Paperclip --- doc/idle-grok-dispatch.md | 98 +++++++++++++++++++ ...2026-10-24-recurring-server-sleep-stack.md | 20 ++++ .../src/__tests__/grok-dispatch-work.test.ts | 32 ++++++ 3 files changed, 150 insertions(+) create mode 100644 doc/idle-grok-dispatch.md diff --git a/doc/idle-grok-dispatch.md b/doc/idle-grok-dispatch.md new file mode 100644 index 00000000000..a7d64e1b9f2 --- /dev/null +++ b/doc/idle-grok-dispatch.md @@ -0,0 +1,98 @@ +# Idle Grok routine dispatch + +The Grok dispatcher previously started a one-second timer whenever public MCP +was configured. Each tick read experimental settings, even when Grok was +otherwise disabled. Enabled ticks scanned prepared dispatches regardless of +whether any work existed. The disabled baseline regression observed three +settings queries in 3.2 seconds. + +The dispatcher now registers with the existing delivery coordinator and shared +scheduler. It performs one startup reconciliation. The worker becomes quiet +when Grok is disabled or when no prepared dispatch remains. Quiet means no +recurring worker SQL or provider calls, including with retained terminal history. +This does not establish whole-instance or database suspension. + +## Ownership and producers + +`grokRoutineService.reserve` is the only supported creator of prepared dispatches. +It signals intent before its insert in the existing transaction. Route verification, +manual retry of a confirmed rejection, and native-broker assignment admission all +use this service. The native broker passes its transaction, so the wake follows +the outer commit. Savepoint release is not publication; rollback does not send. + +`instanceSettingsService.updateExperimental` signals the same queue before its +existing transactional update. This covers enablePublicMcp, enableNativeRunner, +and enableGrokBot changes, including caller-owned outer transactions. It permits +prepared work retained while disabled to resume when enabled. Managed settings +and environment overlays are process configuration and are reconstructed on restart. +Other settings writers update general/default-environment fields, not these flags. + +All other dispatch writers move rows away from prepared: send claims, delivery +settlement, acknowledgement, explicit recovery, assignment completion, and +revocation. None reopens a row as prepared. They cannot introduce hidden new work. +A previously scheduled backlog pass may observe cancellation and disarm. + +Signals cover the current process's shared database owner and child pools. +Independent SQL writers or another process need explicit ingress integration; +this change does not claim cross-process notifications. Startup reconstructs +committed prepared rows from the existing dispatch table. No queue table is added. + +## Delivery and recovery + +The Grok service continues to own current authority checks, binding locks, +prepared-to-sending compare-and-set, secret access, provider delivery, and outcome +settlement. The worker does not interpret provider payloads or relocate policy +into app wiring. A foreground send and background recovery share the durable +claim; only the winner contacts the provider. + +Each pass processes at most ten prepared rows. The next lookup uses the same +predicate and schedules the existing one-second cadence only while prepared work +remains, or after a reconciliation failure. New commits wake the shared worker; +its deadline adapter preserves requested generations through concurrent changes +and drain. The partial created_at/id index keeps both selection and pending-work +lookup independent of retained completed history. The migration builds it +concurrently, is replay-safe, and uses the normal snapshot-retention workflow. + +The existing send-error policy is preserved: if sending throws and the row is +still prepared, the dispatcher cancels it. A row already claimed as sending is +never reset or replayed. Accepted, sending, unknown, acknowledged, rejected, and +cancelled rows do not become automatic retry work. Explicit operator retry still +creates a new intent only for confirmed rejection. This migration does not claim +exactly-once delivery or resolve uncertain provider outcomes automatically. + +## Lifecycle and sleep boundary + +The app registers only when public MCP is configured, after the shared coordinator +exists. Registration does not delay readiness for a provider send. The coordinator +tracks startup work and joins admitted sends on shutdown. Abort prevents the next +row from starting; existing HTTP timeouts bound a current transport operation. +Later prepared rows remain durable for restart. + +Drain and warm standby retain pending requests without passive worker queries. +Resume uses the same coordinator retry path; a first write or failed sweep cannot +be forgotten merely because dispatch was paused. A disabled feature may retain +prepared rows without scheduling a scan; a committed settings change reactivates +it. Disabled feature state is not permission to discard durable work or relax +Cloud sleep guards. Existing binding/assignment/hold blockers remain unchanged. + +## Verification + +- The baseline failed the disabled-feature quiet assertion with three SQL reads. +- 21 real-PostgreSQL lifecycle cases cover disabled/configured idle, first real + reservation, prepared startup recovery, all six non-prepared states, re-enable, + outer commit/rollback, drain/startup, failed scan through drain, concurrent + enqueue, two-page draining, foreground claim overlap, expired unsent work, + settings transaction rollback/commit, and shutdown joining a send. +- The existing 45 Grok qualification cases and 39 startup cases pass together + with the new suite: 105 cases total. +- The four affected settings suites pass 154 cases (259 selected cases total). +- Workspace typecheck, production build, migration checks and replay pass. A + generic-plan probe with 40,000 completed dispatches and one prepared row uses + the partial index for both queries (six shared buffers including the routine + join; 0.032ms batch and 0.025ms next-work execution in this local probe). +- Remote PR review and CI must pass before the slice is ready. + +The tests use the existing real OAuth/agent/routine fixture and a controlled +provider response. A validated disposable external PostgreSQL option permits +local execution where embedded PostgreSQL is unavailable; CI retains embedded +PostgreSQL by default. Provider-host suspension is outside this local proof. diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index 41013294a78..e7c74b72a77 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -28,6 +28,7 @@ Do not mark an item done from a timer removal or a green check alone. | 14 | Skill-version retention and native-harness backup expiration | Separate retention owners and existing retention guarantees | Pending | | 15 | Automatic database backups | Shared scheduling, actual backup deadlines, existing checkpoint contract | Pending | | 16 | Aggregate maintenance tick | Retire enabled and disabled branches after all constituent owners migrate | Pending | +| 17 | Grok routine dispatch | Committed reservations and feature changes; prepared-only recovery | Slice 13 implemented, local verification in progress | ## Stack and verification contract @@ -621,3 +622,22 @@ when public MCP is configured, even when Grok is disabled. Include this in the full stack as slice 13, separately from decision retention. The disabled baseline issued three settings queries in 3.2 seconds. Its durable prepared dispatches, atomic claim, and existing uncertain-outcome policy must remain authoritative. + + +## Slice 13: Grok routine dispatch + +The one-second idle scan is replaced with a shared-coordinator worker. The existing +reservation and experimental-settings transactions notify it; the Grok service +retains all authority, claims, delivery, and uncertainty handling. Prepared-only +selection uses a concurrent partial index. The 21 lifecycle cases, 45 existing +Grok cases, and 39 startup cases pass together (105). Workspace typecheck passes. +See [Grok dispatch](../idle-grok-dispatch.md). Build, populated query proof, PR +publication, and fresh review/CI remain required. Decision retention and every +other pending inventory owner remain in scope. + + +Slice 13 final local verification also passes all 154 affected settings cases +(259 selected tests total), workspace typecheck, production build, migration +checks/replay, and a generic-plan query probe with 40,000 completed dispatches. +Both queries use the prepared-only partial index with six shared buffers, +including the routine join. Remote CI/review remain required. diff --git a/server/src/__tests__/grok-dispatch-work.test.ts b/server/src/__tests__/grok-dispatch-work.test.ts index 5423d191432..6d1c243dc5a 100644 --- a/server/src/__tests__/grok-dispatch-work.test.ts +++ b/server/src/__tests__/grok-dispatch-work.test.ts @@ -127,6 +127,38 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes const row = await reserve(first); await delivered(row.id); await vi.waitFor(() => expect(next).toBeDefined()); await delivered(next.id); await quiet(); expect(delivery).toHaveBeenCalledTimes(2); }); + it("drains more than one page then stops with retained history", async () => { + const rows = []; + for (let i = 0; i < 11; i++) rows.push(await reserve(await grokRoutineFixture(db))); + await start(); await delivered(rows.at(-1)!.id); await quiet(); + expect(delivery).toHaveBeenCalledTimes(11); + expect((await db.select().from(grokBotDispatches)).every(row => row.status === "accepted")).toBe(true); + }); + it("shares the durable claim with a simultaneous foreground send", async () => { + const f = await grokRoutineFixture(db); await start(); await quiet(); + const row = await reserve(f); + await Promise.all([grokRoutineService(db, delivery).send(f.company.id, f.agent.id, row.id), worker.tick()]); + await delivered(row.id); await quiet(); expect(delivery).toHaveBeenCalledTimes(1); + }); + it("cancels an expired unsent dispatch without contacting the provider", async () => { + const f = await grokRoutineFixture(db); const row = await reserve(f); + await db.update(grokBotDispatches).set({ expiresAt: new Date(Date.now() - 1000) }).where(eq(grokBotDispatches.id, row.id)); + await start(); await quiet(); expect(await status(row.id)).toBe("cancelled"); expect(delivery).not.toHaveBeenCalled(); + }); + it("does not enable pending dispatches before the settings transaction commits", async () => { + const f = await grokRoutineFixture(db); const row = await reserve(f); await enable(false); + await start(); await quiet(); + await expect(db.transaction(async tx => { + await instanceSettingsService(tx as unknown as typeof db).updateExperimental({ enableGrokBot: true }); + await vi.advanceTimersByTimeAsync(2000); expect(delivery).not.toHaveBeenCalled(); throw new Error("rollback"); + })).rejects.toThrow("rollback"); + await quiet(); expect(await status(row.id)).toBe("prepared"); + await db.transaction(async tx => { + await instanceSettingsService(tx as unknown as typeof db).updateExperimental({ enableGrokBot: true }); + await vi.advanceTimersByTimeAsync(2000); expect(delivery).not.toHaveBeenCalled(); + }); + await delivered(row.id); await quiet(); expect(delivery).toHaveBeenCalledTimes(1); + }); it("joins an admitted send on shutdown and leaves later rows for restart", async () => { const first = await grokRoutineFixture(db), second = await grokRoutineFixture(db); let release!: () => void; const held = new Promise(resolve => { release = resolve; });