Files
PaperClipAI/server/src/services/grok-bot/routine.ts
T
DottaandPaperclip 5a9e6b6315 fix(grok): avoid false file recovery after Stop (#16427)
## Thinking Path

> - Paperclip lets people manage AI agents and their work.
> - Grok Bot uses a managed Runner and a fixed external work session.
> - Its admitted contract gives it no access to Paperclip workspace
files.
> - The Runner still has a sync descriptor to support durable recovery.
> - Cancellation treated that descriptor as proof that files needed
repair.
> - This pull request uses the admitted access contract while preserving
ownership settlement.
> - Users get the correct recovery action after Stop, with native file
safeguards intact.

## Linked Issues or Issue Description

Refs #16219 and #16422. The duplicate search found no open Grok
workspace correction.

**What happened?**

A real Grok Stop test cancelled its run, fenced its assignment, denied a
stale progress update and cleaned up the sandbox. Paperclip also created
a file-repair hold. The recovery panel told the user to recover
workspace files, although the admitted external contract has workspace
access set to none.

**Expected behavior**

Keep the stopped session fenced and require inspection before new work.
Keep failed-run ownership settlement. Do not invent a file export
obligation for a session that cannot access the workspace. Native
sessions with writable or unknown workspace access must keep their file
safeguards.

**Steps to reproduce**

1. Connect a Grok Bot and assign a task.
2. Let it accept the task and save a Paperclip checkpoint document.
3. Use Stop while it is working.
4. Inspect the task recovery panel. It asks for file repair after
sandbox cleanup succeeds.

**Paperclip version or commit**

Observed on Core 11d69d04c7 through the
actual Grok client. Reproduced against master with database-backed
regressions.

## What Changed

- Match the admitted native input to its run, company, agent, task and
external driver before treating workspace access as disabled.
- Keep coordinator ownership checks and failure settlement. Skip file
retention and the false file-repair obligation for that exact contract.
- Add inspection-only settlement for old Grok file holds through Runtime
recovery. Verify exact run ownership, fenced assignments, terminal
coordinators and managed lease stop receipts. Archive the false file
evidence and preserve no-replay.
- Keep historical Dot recovery controls. Do not claim global
external-provider termination.
- Fix the later dispatch-failure owner so it cannot retain a
no-workspace external sandbox for a false file obligation.
- Cover both Grok and Dot, mismatched access evidence, native file
recovery and the assembled heartbeat cleanup path.
- Update the Grok guide and the existing release evidence plan.

## Verification

- Four database regressions fail before the correction.
- 68 file-recovery and cancellation checks pass. Grok, Dot and the
shared recovery notice pass 98 checks. The assembled heartbeat selection
passes both cases; 404 unrelated cases were not selected in that local
command.
- The lifecycle regression caught an unsafe early-return prototype. The
final code keeps coordinator settlement for both workspace cases. A
separate assertion was corrected for the run-state projection's explicit
nulls. The failed attempts remain in the private evidence record.
- Repository typecheck and build, token gates and Storybook build pass.
The branch was rebased from ea4aa185b0
onto 7420af26fc, whose additional change
affects CI profile exports. The core recovery checks are repeated after
rebase.
- On the earlier deployed source, a real 824-second task survived an
actual same-image controller restart with its original run, assignment,
process and lease. It completed canonically and cleaned up without an
extra wake. The active Stop test denied a real stale update and
completed connection inspection and authenticated re-verification. These
results establish the observed baseline and defect; they do not qualify
this patch's deployed behavior.
- Review found two valid gaps: the old file obligation still blocked
fresh input, and dispatch failure could request retention earlier. Both
are corrected. The follow-up passes 128 focused tests and server
typecheck. The recovery test proves fresh user continuation after
inspection, authenticated re-verification, and managed Runner cleanup
while the old assignment stays fenced.
- A second review found a partial-transaction gap. Routine readiness now
commits as invalid before task holds retire. All 43 Grok routine tests
pass, including a held task lock and an unrelated active assignment.
- Repository typecheck and build pass after the first review
corrections; final server typecheck also passes. The full local suite is
still running and is not claimed as passed. The first follow-up CI timed
out at 15 seconds in one unchanged run-history case; all 91 cases pass
locally. The cause is unproven.
- The final head, 33e4361911, passes 54
checks plus one successful status context. Two optional Storybook jobs
are skipped. Greptile reviewed that exact head at 5/5 with no unresolved
threads. All source CI suites pass on this final head without retry. The
earlier runner-shutdown and timeout attempts remain recorded.
- An actual deployed historical recovery and fresh Stop repeat are still
required.

## Risks

An incomplete or mismatched contract must retain file safeguards. The
change uses stored admission evidence rather than current agent
configuration. Historical records stay auditable and do not gain
automatic replay or a generic Retry action. Operator inspection and a
fresh authenticated verification remain required before new Grok work.
Global Grok termination remains unconfirmed. No schema migration is
needed.

## Model Used

OpenAI GPT-6 in Codex, with reasoning, repository editing, shell tools
and computer use. The exact serving model ID, snapshot and context
window are not exposed by this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-15 08:52:50 -05:00

330 lines
28 KiB
TypeScript

import { createHash, randomBytes, randomUUID, timingSafeEqual } from "node:crypto";
import { and, asc, desc, eq, inArray, isNull, isNotNull, lte, or, sql } from "drizzle-orm";
import { z } from "zod";
import { companies, dotAgentBindings as bindings, dotRunnerAssignments as assignments,
grokBotRoutines as routines, grokBotDispatches as dispatches, mcpOauthGrants, type Db } from "@paperclipai/db";
import { authorizeExternalBotBinding, authorizeExternalBotOperator, externalBotEnabled } from "../external-bot-authority.js";
import { McpOAuthError, type McpPrincipal } from "../public-mcp/oauth.js";
import { secretService } from "../secrets.js";
import { scheduleAgentLifecycle } from "../agent-lifecycle.js";
import { logActivity } from "../activity-log.js";
import { deliverGrokRoutineWake, validateGrokRoutineCredential, type GrokRoutineDelivery, type GrokRoutineWake } from "./webhook.js";
import { grokWorkspaceInspectionCandidates, inspectGrokWorkspaceObligations } from "./workspace-inspection.js";
const digest = (value: string) => createHash("sha256").update(value).digest("hex");
const fail = (message: string) => new McpOAuthError("access_denied", message, 409);
export const grokWakeAcknowledgementSchema = z.object({ correlationId: z.uuid(), bindingId: z.uuid(),
bindingGeneration: z.number().int().positive(), challenge: z.string().min(20).max(100).optional() }).strict();
type Ack = z.infer<typeof grokWakeAcknowledgementSchema>;
type Delivery = typeof deliverGrokRoutineWake;
/** Serialize work acceptance with the bounded HTTP handoff, without holding
* binding/assignment row locks that would block Stop or authenticated replies. */
export async function lockGrokRoutineAcceptance(database: Db, bindingId: string) {
await database.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`grok-routine-acceptance:${bindingId}`}, 0))`);
}
/** Existing shared agent grants and assignments own authority; routines only wake the Bot. */
export function grokRoutineService(db: Db, deliver: Delivery = deliverGrokRoutineWake) {
async function current(database: Db, companyId: string, agentId: string, bindingId: string, lock = false, allowPaused = false) {
const query = database.select().from(bindings).where(and(eq(bindings.id, bindingId), eq(bindings.companyId, companyId),
eq(bindings.agentId, agentId), eq(bindings.providerKind, "grok_bot"), isNull(bindings.revokedAt)));
const [binding] = lock ? await query.for("update") : await query;
if (!binding?.grantId || !await externalBotEnabled(database, "grok_bot")) throw fail("Connect the Grok agent before configuring assignments.");
const [grant] = await database.select().from(mcpOauthGrants).where(and(eq(mcpOauthGrants.id, binding.grantId), isNull(mcpOauthGrants.revokedAt)));
const [company] = await database.select().from(companies).where(eq(companies.id, companyId));
if (!grant || company?.status !== "active") throw fail("Agent connection authority is unavailable.");
await authorizeExternalBotBinding(database, { grant, actor: { type: "agent", companyId, agentId },
company: { id: company.id, name: company.name, issuePrefix: company.issuePrefix, status: company.status } },
{ provider: "grok_bot", requireReady: false, allowPaused });
return { binding, grant };
}
async function recoveryNeeded(companyId: string, agentId: string) {
const [held] = await db.select({ id: routines.id }).from(routines).where(and(eq(routines.companyId, companyId), eq(routines.agentId, agentId), isNotNull(routines.recoveryRequiredAt))).limit(1);
const [uncertain] = await db.select({ id: dispatches.id }).from(dispatches).innerJoin(routines, eq(routines.id, dispatches.routineId))
.where(and(eq(dispatches.companyId, companyId), eq(routines.agentId, agentId),
or(eq(dispatches.status, "unknown"), and(eq(dispatches.status, "sending"), lte(dispatches.updatedAt, new Date(Date.now() - 30_000))), and(eq(dispatches.status, "accepted"), lte(dispatches.expiresAt, new Date()))))).limit(1);
return !!held || !!uncertain || (await grokWorkspaceInspectionCandidates(db, companyId, agentId)).length > 0;
}
async function recover(companyId: string, agentId: string, operatorId: string) {
await authorizeExternalBotOperator(db, companyId, operatorId);
await db.transaction(async tx => {
// All routine writers serialize on the binding before dispatch/routine rows.
// Include retired bindings so recovery cannot bypass an old activity hold.
await tx.select({ id: bindings.id }).from(bindings).where(and(eq(bindings.companyId, companyId),
eq(bindings.agentId, agentId), eq(bindings.providerKind, "grok_bot"))).orderBy(asc(bindings.id)).for("update");
const rows = await tx.select().from(routines).where(and(eq(routines.companyId, companyId), eq(routines.agentId, agentId))).for("update");
const [active] = await tx.select({ id: assignments.id }).from(assignments).where(and(eq(assignments.companyId, companyId), eq(assignments.agentId, agentId), inArray(assignments.status, ["offered", "accepted"])));
if (active) throw fail("Stop or finish the current Paperclip work session before clearing recovery.");
for (const routine of rows) {
await tx.update(dispatches).set({ status: "cancelled", updatedAt: new Date() }).where(and(eq(dispatches.routineId, routine.id), inArray(dispatches.status, ["prepared", "sending", "accepted", "unknown"])));
await tx.update(routines).set({ recoveryRequiredAt: null, recoveryReason: null, verifiedAt: null, verifiedGrantId: null, updatedAt: new Date() }).where(eq(routines.id, routine.id));
await tx.update(bindings).set({ status: "connected", readyAt: null, updatedAt: new Date() }).where(and(eq(bindings.id, routine.bindingId), isNull(bindings.revokedAt)));
}
await logActivity(tx as unknown as Db, { companyId, actorType: "user", actorId: operatorId, action: "grok.recovery_inspected",
entityType: "agent", entityId: agentId, details: { inspectedRemoteWork: true, remoteTerminationConfirmed: false, requiresVerification: true } });
});
// Commit loss of readiness before retiring task holds. If inspection fails
// or waits for a task lock, the old verification cannot admit new work.
// The remaining task hold also prevents verification during this gap.
await inspectGrokWorkspaceObligations(db, companyId, agentId, operatorId);
return { recovered: true, requiresVerification: true, remoteTerminationConfirmed: false };
}
async function state(companyId: string, agentId: string, bindingId: string) {
const [routine] = await db.select().from(routines).where(and(eq(routines.companyId, companyId),
eq(routines.agentId, agentId), eq(routines.bindingId, bindingId)));
if (!routine) return null;
const [last] = await db.select({ id: dispatches.id, kind: dispatches.kind, status: dispatches.status,
httpStatus: dispatches.httpStatus, sentAt: dispatches.sentAt, acknowledgedAt: dispatches.acknowledgedAt, expiresAt: dispatches.expiresAt })
.from(dispatches).where(and(eq(dispatches.companyId, companyId), eq(dispatches.routineId, routine.id),
eq(dispatches.routineRevision, routine.revision))).orderBy(desc(dispatches.createdAt)).limit(1);
const [binding] = await db.select().from(bindings).where(and(eq(bindings.id, bindingId), eq(bindings.companyId, companyId),
eq(bindings.agentId, agentId), eq(bindings.generation, routine.bindingGeneration), isNull(bindings.revokedAt)));
const [grant] = binding?.grantId ? await db.select().from(mcpOauthGrants).where(and(eq(mcpOauthGrants.id, binding.grantId), isNull(mcpOauthGrants.revokedAt))) : [];
return { bindingId, botName: routine.botName, revision: routine.revision, configured: true,
verified: !!routine.verifiedAt && !!binding && !!grant && binding.grantId === routine.verifiedGrantId,
verifiedAt: routine.verifiedAt, lastContactAt: routine.lastContactAt,
recoveryNeeded: await recoveryNeeded(companyId, agentId), lastExchange: last ?? null };
}
async function configure(input: { companyId: string; agentId: string; bindingId: string; operatorId: string;
expectedRevision: number; botName: string; webhookUrl: string; webhookKey: string }) {
await authorizeExternalBotOperator(db, input.companyId, input.operatorId);
if (await recoveryNeeded(input.companyId, input.agentId)) throw fail("Inspect and recover remote work before replacing routine credentials.");
const secret = validateGrokRoutineCredential(input);
const botName = z.string().trim().min(1).max(100).parse(input.botName);
await db.transaction(async tx => {
const database = tx as unknown as Db;
const { binding } = await current(database, input.companyId, input.agentId, input.bindingId, true);
const [active] = await tx.select({ id: assignments.id }).from(assignments).where(and(eq(assignments.bindingId, binding.id), inArray(assignments.status, ["offered", "accepted"])));
const [old] = await tx.select().from(routines).where(eq(routines.bindingId, binding.id)).for("update");
const [pending] = old ? await tx.select({ id: dispatches.id }).from(dispatches).where(and(eq(dispatches.routineId, old.id), eq(dispatches.routineRevision, old.revision), inArray(dispatches.status, ["prepared", "sending", "accepted", "unknown"]))) : [];
if (active || pending || old?.recoveryRequiredAt) throw fail("Inspect and recover existing Paperclip work before replacing routine credentials.");
if ((old?.revision ?? 0) !== input.expectedRevision) throw fail("Routine configuration changed. Reload before replacing it.");
const revision = (old?.revision ?? 0) + 1;
// One fresh immutable secret version prevents partial URL/key replacement.
const stored = await secretService(database).create(input.companyId, {
name: `Grok routine ${binding.id} revision ${revision}`, key: `GROK_ROUTINE_${binding.id.replaceAll("-", "_").toUpperCase()}_${revision}`,
provider: "local_encrypted", value: JSON.stringify(secret), description: "Grok routine sender credential; never provider-visible",
}, { userId: input.operatorId });
const values = { companyId: input.companyId, agentId: input.agentId, bindingId: binding.id, bindingGeneration: binding.generation,
botName, secretId: stored.id, secretVersion: stored.latestVersion, revision,
verifiedAt: null, verifiedGrantId: null, updatedAt: new Date() };
if (old) await tx.update(routines).set(values).where(eq(routines.id, old.id));
else await tx.insert(routines).values(values);
await tx.update(bindings).set({ status: "connected", readyAt: null, updatedAt: new Date() }).where(eq(bindings.id, binding.id));
await logActivity(database, { companyId: input.companyId, actorType: "user", actorId: input.operatorId,
action: "grok.routine_configured", entityType: "agent", entityId: input.agentId,
details: { bindingId: binding.id, revision } });
});
return state(input.companyId, input.agentId, input.bindingId);
}
async function reserve(input: { companyId: string; agentId: string; bindingId: string; sourceEventId: string;
kind: GrokRoutineWake["kind"]; assignmentId?: string; after?: number;
expectedAuthority?: { grantId: string; generation: number; routineRevision: number } }) {
if (await recoveryNeeded(input.companyId, input.agentId)) throw fail("Inspect and recover remote work before another delivery.");
return db.transaction(async tx => {
const database = tx as unknown as Db;
const { binding, grant } = await current(database, input.companyId, input.agentId, input.bindingId, true);
const [routine] = await tx.select().from(routines).where(and(eq(routines.bindingId, binding.id),
eq(routines.companyId, input.companyId), eq(routines.agentId, input.agentId))).for("update");
if (!routine || routine.bindingGeneration !== binding.generation || routine.recoveryRequiredAt) throw fail("Configure or recover the Grok routine first.");
if (input.expectedAuthority && (input.expectedAuthority.grantId !== grant.id
|| input.expectedAuthority.generation !== binding.generation || input.expectedAuthority.routineRevision !== routine.revision)) {
throw fail("Rejected delivery authority changed.");
}
const [old] = await tx.select().from(dispatches).where(and(eq(dispatches.bindingId, binding.id),
eq(dispatches.bindingGeneration, binding.generation), eq(dispatches.routineRevision, routine.revision), eq(dispatches.sourceEventId, input.sourceEventId)));
if (old) return old;
const [active] = input.kind === "verify" ? await tx.select({ id: assignments.id }).from(assignments).where(and(eq(assignments.bindingId, binding.id), inArray(assignments.status, ["offered", "accepted"]))) : [];
if (active) throw fail("Finish or stop current Paperclip work before verifying the routine again.");
const [pending] = input.kind === "verify" ? await tx.select({ id: dispatches.id }).from(dispatches).where(and(eq(dispatches.routineId, routine.id), eq(dispatches.routineRevision, routine.revision), eq(dispatches.kind, "verify"), inArray(dispatches.status, ["prepared", "sending", "accepted", "unknown"]))) : [];
if (pending) throw fail("Verification already has an outstanding delivery. Inspect it before sending another.");
const correlationId = randomUUID();
const common = { schema: "paperclip.external_bot_wake.v1" as const, bindingId: binding.id, bindingGeneration: binding.generation, correlationId };
const challenge = input.kind === "verify" ? randomBytes(24).toString("base64url") : null;
const wake: GrokRoutineWake = input.kind === "verify" ? { ...common, kind: "verify", challenge: challenge! }
: input.kind === "authority_revoked" ? { ...common, kind: "authority_revoked", assignmentId: z.uuid().parse(input.assignmentId) }
: { ...common, kind: "mailbox_updated", after: z.number().int().nonnegative().max(Number.MAX_SAFE_INTEGER).parse(input.after ?? 0) };
if (input.kind === "verify") {
await tx.update(routines).set({ verifiedAt: null, verifiedGrantId: null, updatedAt: new Date() }).where(eq(routines.id, routine.id));
await tx.update(bindings).set({ status: "connected", readyAt: null, updatedAt: new Date() }).where(eq(bindings.id, binding.id));
}
const [row] = await tx.insert(dispatches).values({ id: correlationId, companyId: input.companyId, routineId: routine.id,
bindingId: binding.id, bindingGeneration: binding.generation, grantId: grant.id, routineRevision: routine.revision,
sourceEventId: z.string().min(1).max(240).parse(input.sourceEventId), kind: input.kind,
assignmentId: input.assignmentId, payload: { ...wake }, challengeHash: challenge ? digest(challenge) : null,
expiresAt: new Date(Date.now() + 10 * 60_000) }).returning();
return row!;
});
}
async function send(companyId: string, agentId: string, correlationId: string) {
const intent = await db.transaction(async tx => {
const database = tx as unknown as Db;
const [row] = await tx.select().from(dispatches).where(and(eq(dispatches.id, correlationId), eq(dispatches.companyId, companyId)));
if (!row) throw fail("Dispatch is unavailable.");
const { binding } = await current(database, companyId, agentId, row.bindingId, true);
const [routine] = await tx.select().from(routines).where(eq(routines.id, row.routineId)).for("update");
if (!routine || routine.agentId !== agentId || row.grantId !== binding.grantId || row.bindingGeneration !== binding.generation
|| row.routineRevision !== routine.revision || routine.recoveryRequiredAt) throw fail("Dispatch authority changed.");
if (row.status !== "prepared") return null;
if (row.kind === "mailbox_updated") {
if (!row.assignmentId) throw fail("Work wake requires an admitted assignment.");
const { dotRunnerBroker } = await import("../dot-runner-broker.js");
await dotRunnerBroker(db).assertWakeAuthority(companyId, agentId, row.bindingId, row.assignmentId, { requireOffered: true });
}
// Accepted, acknowledged and uncertain work is never sent again after restart.
if (row.expiresAt <= new Date()) throw fail("Unsent dispatch expired. Inspect before creating a new verification.");
const [claimed] = await tx.update(dispatches).set({ status: "sending", sentAt: new Date(), updatedAt: new Date() })
.where(and(eq(dispatches.id, row.id), eq(dispatches.status, "prepared"))).returning();
return claimed ? { row: claimed, routine } : null;
});
if (!intent) {
const [row] = await db.select().from(dispatches).where(and(eq(dispatches.id, correlationId), eq(dispatches.companyId, companyId)));
return row ? state(companyId, agentId, row.bindingId) : null;
}
let networkAttempted = false;
let outcome: { delivery: GrokRoutineDelivery["delivery"] | "cancelled"; httpStatus: number | null } = { delivery: "unknown", httpStatus: null };
try {
const secret = await secretService(db).resolveSecretValue(companyId, intent.routine.secretId, intent.routine.secretVersion,
{ accessContext: { consumerType: "agent", consumerId: agentId, configPath: "grok.routine", actorType: "system", actorId: "grok-dispatch" } });
const handoff = async (database: Db) => {
// Acceptance may have committed while credentials were resolving. An
// obsolete, unsent wake is cancellation, not a provider rejection.
if (intent.row.kind === "mailbox_updated") {
const [assignment] = await database.select({ status: assignments.status }).from(assignments)
.where(and(eq(assignments.id, intent.row.assignmentId!), eq(assignments.bindingId, intent.row.bindingId)));
if (assignment && assignment.status !== "offered") return { delivery: "cancelled" as const, httpStatus: null };
}
// Recheck after acquiring the acceptance lock and resolving credentials.
await current(database, companyId, agentId, intent.row.bindingId);
if (intent.row.kind === "mailbox_updated") {
const { dotRunnerBroker } = await import("../dot-runner-broker.js");
await dotRunnerBroker(database).assertWakeAuthority(companyId, agentId, intent.row.bindingId, intent.row.assignmentId!, { requireOffered: true });
}
networkAttempted = true;
return deliver(JSON.parse(secret), intent.row.payload as GrokRoutineWake);
};
outcome = intent.row.kind === "mailbox_updated" ? await db.transaction(async tx => {
const database = tx as unknown as Db;
await lockGrokRoutineAcceptance(database, intent.row.bindingId);
return handoff(database);
}) : await handoff(db);
} catch { outcome = { delivery: networkAttempted ? "unknown" : "rejected", httpStatus: null }; /* Do not persist exception or response text. */ }
await db.transaction(async tx => {
await tx.select({ id: bindings.id }).from(bindings).where(eq(bindings.id, intent.row.bindingId)).for("update");
const [changed] = await tx.update(dispatches).set({ status: outcome.delivery, httpStatus: outcome.httpStatus, updatedAt: new Date() })
.where(and(eq(dispatches.id, correlationId), eq(dispatches.status, "sending"))).returning();
// A matching authenticated reply can win this race and reconcile delivery.
if (changed?.status === "unknown") await tx.update(routines).set({ recoveryRequiredAt: new Date(), recoveryReason: "dispatch", updatedAt: new Date() })
.where(and(eq(routines.id, intent.routine.id), eq(routines.revision, intent.routine.revision),
or(isNull(routines.recoveryReason), eq(routines.recoveryReason, "dispatch"))));
});
return state(companyId, agentId, intent.row.bindingId);
}
async function retryRejected(companyId: string, agentId: string, operatorId: string, correlationId: string) {
await authorizeExternalBotOperator(db, companyId, operatorId);
const [row] = await db.select({ dispatch: dispatches, routine: routines }).from(dispatches)
.innerJoin(routines, eq(routines.id, dispatches.routineId)).where(and(eq(dispatches.id, correlationId),
eq(dispatches.companyId, companyId), eq(routines.companyId, companyId), eq(routines.agentId, agentId)));
if (!row || row.dispatch.status !== "rejected") throw fail("Only a confirmed rejected delivery can be retried.");
const { binding } = await current(db, companyId, agentId, row.dispatch.bindingId);
if (row.dispatch.grantId !== binding.grantId || row.dispatch.bindingGeneration !== binding.generation
|| row.dispatch.routineRevision !== row.routine.revision) throw fail("Rejected delivery authority changed.");
if (row.dispatch.kind === "mailbox_updated") {
if (!row.dispatch.assignmentId) throw fail("Work wake requires an admitted assignment.");
const { dotRunnerBroker } = await import("../dot-runner-broker.js");
await dotRunnerBroker(db).assertWakeAuthority(companyId, agentId, binding.id, row.dispatch.assignmentId, { requireOffered: true });
}
// Retain the rejected receipt. Concurrent clicks reserve the same next intent;
// accepted/unknown work can never enter this path, and is never replayed.
const intent = await reserve({ companyId, agentId, bindingId: binding.id,
sourceEventId: `retry:${row.dispatch.id}`, kind: row.dispatch.kind,
expectedAuthority: { grantId: row.dispatch.grantId, generation: row.dispatch.bindingGeneration, routineRevision: row.dispatch.routineRevision },
...(row.dispatch.assignmentId ? { assignmentId: row.dispatch.assignmentId } : {}),
after: (row.dispatch.payload as GrokRoutineWake & { after?: number }).after });
await logActivity(db, { companyId, actorType: "user", actorId: operatorId, action: "grok.delivery_retry_requested",
entityType: "agent", entityId: agentId, details: { rejectedCorrelationId: row.dispatch.id, correlationId: intent.id } });
return send(companyId, agentId, intent.id);
}
/** Called in the authoritative successful-finish transaction, with the binding lock held. */
async function settleWorkNotifications(companyId: string, bindingId: string, assignmentId: string) {
const [assignment] = await db.select({ id: assignments.id }).from(assignments).where(and(eq(assignments.id, assignmentId),
eq(assignments.companyId, companyId), eq(assignments.bindingId, bindingId), eq(assignments.status, "settled")));
if (!assignment) throw fail("Only settled work can close its notification obligation.");
await db.update(dispatches).set({ status: "cancelled", updatedAt: new Date() }).where(and(eq(dispatches.companyId, companyId),
eq(dispatches.bindingId, bindingId), eq(dispatches.assignmentId, assignmentId), eq(dispatches.kind, "mailbox_updated"),
inArray(dispatches.status, ["prepared", "sending", "accepted", "unknown"])));
const [uncertain] = await db.select({ id: dispatches.id }).from(dispatches).where(and(eq(dispatches.companyId, companyId),
eq(dispatches.bindingId, bindingId), or(eq(dispatches.status, "unknown"),
and(eq(dispatches.status, "sending"), lte(dispatches.updatedAt, new Date(Date.now() - 30_000))),
and(eq(dispatches.status, "accepted"), lte(dispatches.expiresAt, new Date()))))).limit(1);
// A canonical finish closes this session; it never proves a webhook ACK or
// restores execution authority. Other uncertain deliveries keep their hold.
if (!uncertain) await db.update(routines).set({ recoveryRequiredAt: null, recoveryReason: null, updatedAt: new Date() })
.where(and(eq(routines.companyId, companyId), eq(routines.bindingId, bindingId), eq(routines.recoveryReason, "dispatch")));
}
async function acknowledge(principal: McpPrincipal, raw: Ack) {
const input = grokWakeAcknowledgementSchema.parse(raw);
const binding = await authorizeExternalBotBinding(db, principal, { provider: "grok_bot", requireReady: false, allowPaused: true });
if (input.bindingId !== binding.id || input.bindingGeneration !== binding.generation) throw fail("Wake is outside this connection.");
const result = await db.transaction(async tx => {
const database = tx as unknown as Db;
const { binding: currentBinding } = await current(database, binding.companyId, binding.agentId, binding.id, true, true);
const [row] = await tx.select().from(dispatches).where(and(eq(dispatches.id, input.correlationId),
eq(dispatches.companyId, binding.companyId), eq(dispatches.bindingId, binding.id), eq(dispatches.bindingGeneration, binding.generation),
eq(dispatches.grantId, principal.grant.id))).for("update");
const [routine] = row ? await tx.select().from(routines).where(eq(routines.id, row.routineId)).for("update") : [];
if (!row || !routine || routine.bindingGeneration !== currentBinding.generation || row.routineRevision !== routine.revision
|| row.expiresAt <= new Date() || !["sending", "accepted", "unknown", "acknowledged", "cancelled"].includes(row.status)) throw fail("Wake expired, was cancelled, or its authority changed.");
if (row.kind === "verify") {
if (row.status === "cancelled") throw fail("Verification was cancelled.");
await authorizeExternalBotBinding(database, principal, { provider: "grok_bot", requireReady: false });
if (!input.challenge || !row.challengeHash || !timingSafeEqual(Buffer.from(digest(input.challenge)), Buffer.from(row.challengeHash))) throw fail("Verification challenge does not match.");
} else {
if (input.challenge) throw fail("This wake is not a verification challenge.");
const [assignment] = row.assignmentId ? await tx.select().from(assignments).where(and(eq(assignments.id, row.assignmentId),
eq(assignments.companyId, binding.companyId), eq(assignments.bindingId, binding.id), eq(assignments.bindingGeneration, binding.generation),
eq(assignments.agentId, binding.agentId))) : [];
const settledDelivery = row.kind === "mailbox_updated" && assignment?.status === "settled" && !!row.sentAt;
if (row.status === "cancelled" && !settledDelivery) throw fail("Wake was cancelled.");
if (row.kind === "mailbox_updated" && row.assignmentId && !settledDelivery) {
const { dotRunnerBroker } = await import("../dot-runner-broker.js");
await dotRunnerBroker(db).assertWakeAuthority(binding.companyId, binding.agentId, binding.id, row.assignmentId);
}
if (!assignment || (row.kind === "authority_revoked" ? assignment.status !== "fenced"
: !settledDelivery && (!["offered", "accepted"].includes(assignment.status) || assignment.expiresAt <= new Date()))) throw fail("Work wake authority is no longer current.");
}
if (row.status === "acknowledged") return { status: "acknowledged", correlationId: row.id, workSessionClosed: false, remoteTerminationConfirmed: false };
await tx.update(dispatches).set({ status: "acknowledged", acknowledgedAt: new Date(), updatedAt: new Date() }).where(eq(dispatches.id, row.id));
await tx.update(routines).set({ lastContactAt: new Date(), updatedAt: new Date(),
...(row.kind === "verify" ? { verifiedAt: new Date(), verifiedGrantId: principal.grant.id } : {}) }).where(eq(routines.id, routine.id));
if (row.kind !== "authority_revoked") {
const [other] = await tx.select({ id: dispatches.id }).from(dispatches).where(and(eq(dispatches.routineId, routine.id),
or(eq(dispatches.status, "unknown"), and(eq(dispatches.status, "sending"), lte(dispatches.updatedAt, new Date(Date.now() - 30_000))),
and(eq(dispatches.status, "accepted"), lte(dispatches.expiresAt, new Date()))))).limit(1);
if (!other) await tx.update(routines).set({ recoveryRequiredAt: null, recoveryReason: null })
.where(and(eq(routines.id, routine.id), eq(routines.recoveryReason, "dispatch")));
}
if (row.kind === "verify") await tx.update(bindings).set({ status: "ready", readyAt: new Date(), updatedAt: new Date() }).where(eq(bindings.id, binding.id));
await logActivity(database, { companyId: binding.companyId, actorType: "agent", actorId: binding.agentId,
action: row.kind === "verify" ? "grok.wake_verified" : "grok.wake_acknowledged", entityType: "agent", entityId: binding.agentId,
details: { bindingId: binding.id, generation: binding.generation, correlationId: row.id, kind: row.kind,
authorizingPersonId: binding.operatorId, grantId: principal.grant.id, remoteTerminationConfirmed: false } });
return { status: "acknowledged", correlationId: row.id, workSessionClosed: false, remoteTerminationConfirmed: false };
});
if (input.challenge) scheduleAgentLifecycle(db, binding.agentId);
return result;
}
return { state, configure, reserve, send, acknowledge, settleWorkNotifications, recoveryNeeded, recover, retryRejected };
}