Files
PaperClipAI/server/src/services/durable-chat-wakeup.ts
T
DottaandClaude Opus 5.5 9c230ace94 fix(heartbeat): settle deferred chat wakes that can never resume instead of retrying every pass (#21678)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat deferred-receipt worker resumes wakes that waited
behind a running task execution
> - A chat message that arrives while its task runs is saved as a
deferred chat wake
> - When the task finishes (done or cancelled) first, the saved wake can
never resume, but the sweep retried it forever and touched `updated_at`
on every pass
> - An idle instance did one write and one failed resume about 4 times a
second, and the log line did not say why
> - This pull request gives each such wake one recorded outcome, and
makes every other failure wait on a stored, spaced, bounded next attempt
> - The benefit is an idle database stays idle, and a person sees the
existing "not started" result instead of silence

## Linked Issues or Issue Description

**What happened?**
The stranded deferred-queue sweep in
`server/src/services/heartbeat/queue.ts` selects
`deferred_issue_execution` chat-inbound wakes. On every pass it runs
`update agent_wakeup_requests set updated_at = now()` for each wake.
Then `resumeDeferredChatWake` fails because the task is done or
cancelled, so the chat authority denies the work start
(`task_assignment`). Nothing records the failure, so the next pass
repeats it.

**Steps to reproduce**
1. Save a chat message on a task while a run is active, so the message
becomes a deferred chat wake.
2. Let the run finish and mark the task done.
3. Leave the instance idle. The log shows "saved chat source could not
resume after recovery" again and again for the same request ids.

**Expected behavior**
A wake that can never resume gets one terminal outcome with a stored
reason. A wake that can resume later waits on a stored time. An idle
instance runs no statements for these wakes.

**Paperclip version or commit**
master at 36101ebcb9 and later.

**Relevant logs or output**
On the hosted instance with no active runs: 1,405 failed resumes in 6
minutes (3.9/s) across 363 request ids, plus one write per attempt. With
the test harness, 12 stale wakes over 30 idle minutes caused 1,344
statements, 84 wake writes and 84 failure log lines in one process.

## What Changed

- `heartbeat/queue.ts`: the stranded-queue sweep no longer touches
`updated_at` on chat-inbound wakes. Non-chat saved wakes keep the touch.
It moves `updated_at`, which spaces the worker's 30-second deadline for
wakes with an execution wait, and a board chat continuation
(`ai-personal-connection-recovery` e2e) depends on that spacing. The
chat branch now records the outcome of each resume. The log line now has
`reason`, `outcome` and `attempts`. Unscoped sweeps skip a wake whose
stored next attempt time is in the future.
- `durable-chat-wakeup.ts`: add `recordDeferredChatWakeDecline`,
`clearDeferredChatWakeRetry` and `deferredChatWakeDeclineForError`.
- Settle: set the wake to `skipped` with the reason in `error`, set
`finishedAt`, mark the chat delivery with the existing "Input saved. Its
work request was skipped…" text, and notify the inbound wake notice
worker. That worker shows the existing "This follow-up was not started"
notice where a "queued" notice is visible. Wakes older than 24 hours
settle silently (see Risks).
- Retry: store `payload.chatResume = { attempts, reason, nextAttemptAt
}` and the reason in `error`. Delays are 1 min, 10 min, 1 h, 6 h and 24
h. After five attempts the wake settles as `retry_exhausted:<reason>`.
- Permanent authority reasons (`source_shape`, `delivery_binding`,
`conversation_binding`, `destination_binding`, `source_comment`) settle.
Other reasons, such as principal access, runtime fence, privacy or
infrastructure errors, retry.
- `chat-channels/inbound/wakeups.ts` (the deferred chat wake owner): for
the recovery sweep only, return a decline instead of a silent return or
an authority error:
- The task is done or cancelled and the wake is older than
`CLOSED_TASK_CHAT_WAKE_REOPEN_WINDOW_MS` (1 hour): settle with
`task_closed` and start no run.
- The task is done or cancelled and the wake is recent: reopen the task
to `todo` in the same transaction as the authorization check, then
resume through the normal chat admission (`queueIssueAssignmentWakeup`).
This is what ingress does today for a message on a finished task:
`chat-channels.ts` reopens the bound task and admits the message. An
authorization failure rolls back the reopen.
- Several recent saved messages from the same conversation on the
finished task start one follow-up run. The wake that reached the owner
is authorized first, and any decline is recorded on that wake. Each
other recent message is authorized in a savepoint, and a message that
fails keeps its own receipt. The newest authorized message is admitted,
and the run's `wakeCommentIds` holds the whole batch. Inside the
admission transaction, the durable request's `authorize` callback locks
the other receipts. If another run took one of them, it refuses the
whole batch (`chat_inbound_wakeup_batch_changed`, stored retry).
Otherwise `onAdmitted` marks them `coalesced` with the run id in the
same transaction. A restart or a refused admission leaves them deferred.
- The sweep acts only on `inbound_wakeup` chat sources. Task-retry chat
receipts that share the `chat-inbound:` key keep their own owner.
- A refused start ends in the state that admission already gives a
refused chat message. A paused agent saves `executionWait` and resumes
when the pause lifts. A budget hard stop or a non-invokable agent
produces a skipped receipt or a saved wait. Nothing retries on a timer.
- The task is missing or reassigned: settle. The task is in backlog, or
the chat source is still in ingress: retry.
  - Other callers (task retry, cleanup) keep their old behavior.
- `issue-assignment-wakeup.ts`: add the optional `wakeCommentIds` input.
It goes into the run context so that the follow-up run carries every
folded message.
- `heartbeat/deferred-receipt-work.ts`: the deferred worker schedules a
wake at its stored `nextAttemptAt`. A notification that names the exact
wake (for example, the run it waited on finished) still resumes it at
once.
- Tests: add `deferred-chat-wake-sweep.test.ts`. Update three scenarios
in `chat-channels.integration.test.ts` whose source is permanently gone
and which now settle.

## Verification

- New `server/src/__tests__/deferred-chat-wake-sweep.test.ts` (embedded
PostgreSQL, real deferred-receipt worker and heartbeat queue, fake
clock, statement counter):
- 12 stale wakes on done or cancelled tasks, 30 idle minutes. Before
(master): 1,344 statements, 84 wake updates, 84 failure log lines in the
idle window. After: 0 statements, 0 updates, 0 log lines. Each wake is
`skipped` with `error = task_closed` after one attempt, and no run
starts.
- A recent wake (5 minutes old) on a done task reopens the task and
starts one run.
- Two recent wakes on one done task start exactly one run. That run's
`wakeCommentIds` holds both comments, and both wakes record its run id.
- A recent wake whose start is refused (paused agent) keeps the saved
pause wait, makes 0 statements in 30 idle minutes, and starts one run
when the pause lifts.
- A chat wake whose task row no longer exists settles once as
`task_missing`.
- When another run takes a batched message before admission, the batch
is refused with no run. The next attempt answers the remaining message
alone.
- A 3-day-old and a 3-hour-old wake on finished tasks both settle. Only
the 3-hour-old one gets a delivery text and one "not started" notice. A
second notice pass and a fresh notice worker (a restart) add nothing,
and the old one is never a notice candidate.
- When a newer saved message on the same finished task lost its source
comment, the older valid message still gets its run, and the newer one
is never folded into it.
- A retryable failure (paused endpoint) makes 3 attempts in 30 idle
minutes (at 0, 1 and 11 minutes), with the reason in `error`. Then the
endpoint is unpaused and a notification names the wake, and it starts a
run at once.
- A wake deferred behind a running run makes 0 statements in 10 idle
minutes. It resumes at once when that run finishes.
- Red/green: on master the first three cases fail, and on this branch
all four pass.
- Focused suites pass locally: `queue.test.ts`,
`deferred-execution-receipt-work.test.ts`,
`heartbeat-held-comment-wake.test.ts`, `human-directed-work.test.ts`,
`owner-chat-instruction-authority.test.ts`,
`explicit-native-continuation.test.ts`, and the 20 "reconsiders the same
GitHub reply after stopped native workspace cleanup" cases in
`chat-channels.integration.test.ts`.
- `pnpm verify --explain` selected 2,728 files, so I ran only focused
suites on this shared machine. CI supplies the broad coverage.

## Risks

- On first deploy, the existing stuck wakes drain on their own. On the
hosted instance, the 359 chat-inbound wakes on done or cancelled tasks
are all older than the 1-hour window, so they settle once as
`skipped`/`task_closed` and start 0 runs. All but 2 of the 364 are older
than a day, and none is younger than an hour, so the deploy starts 0
follow-up runs. The 5 on blocked or todo tasks get one resume attempt
each. They then resume, wait on a persisted execution wait, or follow
the stored retry schedule and settle within about 31 hours.
- Product decision: the owner chose "close silently" for old saved
messages. A wake older than `SILENT_CHAT_WAKE_SETTLEMENT_AFTER_MS`
settles with its reason in `error` and `finishedAt`, and has no visible
chat effect. Its delivery text is not written, no notice is signaled,
and the receipt gets `payload.silentSettlement = true`. Both the notice
candidate query (`wakeup-notices.ts`) and `resolveInboundWakeReceipt`
skip such wakes, so a later pass or a restart never edits an old
"queued" notice. The value of 24 hours from the wake's `created_at` was
picked by the reviewer, not the owner, so it is easy to change.
- A wake between the two bounds (older than 1 hour, newer than 24 hours)
keeps the existing visible result. Its "Your follow-up is queued."
notice changes once to "This follow-up was not started."
- Visible edits on deploy: on the hosted instance, only wakes younger
than 24 hours can get one. There were 2 at measurement time, and likely
0 by deploy. The other wakes close silently.
- Product decision: the owner chose "yes, if recent" for a message that
arrived just as its task finished. The freshness bound is
`CLOSED_TASK_CHAT_WAKE_REOPEN_WINDOW_MS` in `durable-chat-wakeup.ts`.
Its value of 1 hour from the wake's `created_at` was picked by the
reviewer, not the owner, so it is easy to change. Set it to 0 to settle
every closed-task wake without a run.
- Non-chat saved wakes keep the per-pass touch. The other 75 deferred
wakes on the hosted instance are not chat-inbound, so this PR does not
change their timing.
- No change to authorization, budget, pause or approval checks. The
reopen runs in the same transaction as the full chat authorization, and
the authorization still decides. No wait is added before a run's first
turn.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:31:40 -06:00

497 lines
21 KiB
TypeScript

import { slackDeliveryMeetsMentionRequirement } from "./chat-channels/slack/wake-authority.js";
import { createHash } from "node:crypto";
import { agentWakeupRequests, chatActions, chatConversations, chatDeliveries, chatEndpoints, type Db } from "@paperclipai/db";
import { and, eq, exists, inArray, isNull, or, sql, type SQLWrapper } from "drizzle-orm";
import { HttpError } from "../errors.js";
import { safeChatWakeAuthorityReason } from "./chat-channels/inbound/wakeup-authorization-error.js";
import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js";
export interface CommittedChatResponseAuthorizationInput {
companyId: string;
issueId: string;
agentId: string;
runId: string;
resultId: string;
}
type CommittedChatResponseAuthority = (
tx: Db,
input: CommittedChatResponseAuthorizationInput,
) => Promise<void>;
const committedChatResponseAuthorities = new WeakMap<
Db,
CommittedChatResponseAuthority
>();
const committedChatResponseAuthorityListeners = new WeakMap<Db, Set<() => void>>();
export function onCommittedChatResponseAuthorityRegistered(db: Db, listener: () => void): () => void {
let listeners = committedChatResponseAuthorityListeners.get(db);
if (!listeners) {
listeners = new Set();
committedChatResponseAuthorityListeners.set(db, listeners);
}
listeners.add(listener);
return () => {
listeners.delete(listener);
if (listeners.size === 0 && committedChatResponseAuthorityListeners.get(db) === listeners) committedChatResponseAuthorityListeners.delete(db);
};
}
export class CommittedChatResponseAuthorizationError extends HttpError {
constructor() {
super(
409,
"The accepted chat response is no longer authorized for this conversation.",
{
code: "chat_committed_response_not_authorized",
},
);
this.name = "CommittedChatResponseAuthorizationError";
}
}
/** Separate, server-only authority for presenting an already accepted result.
* This grants no permission to execute or repair the provider session. */
export function registerCommittedChatResponseAuthority(
db: Db,
authority: CommittedChatResponseAuthority,
): () => void {
committedChatResponseAuthorities.set(db, authority);
for (const listener of committedChatResponseAuthorityListeners.get(db) ?? []) listener();
return () => {
if (committedChatResponseAuthorities.get(db) === authority) {
committedChatResponseAuthorities.delete(db);
}
};
}
export async function authorizeCommittedChatResponse(
db: Db,
tx: Db,
input: CommittedChatResponseAuthorizationInput,
): Promise<void> {
const authority = committedChatResponseAuthorities.get(db);
if (!authority) throw new CommittedChatResponseAuthorizationError();
await authority(tx, input);
}
export interface FailedChatRunRetryAuthorizationInput {
phase: "admission" | "promotion" | "execution";
wakeupRequestId: string | null;
companyId: string;
agentId: string;
issueId: string | null;
runId?: string;
contextSnapshot: Record<string, unknown>;
}
type FailedChatRunRetryAuthority = (
tx: Db,
input: FailedChatRunRetryAuthorizationInput & {
wakeupRequestId: string;
issueId: string;
},
) => Promise<void>;
const failedChatRunRetryAuthorities = new WeakMap<
Db,
FailedChatRunRetryAuthority
>();
export class FailedChatRunRetryAuthorizationError extends HttpError {
constructor() {
super(
409,
"The exact failed chat request is no longer authorized. Send a new request in the current connected conversation.",
{
code: "chat_failed_run_retry_not_authorized",
},
);
this.name = "FailedChatRunRetryAuthorizationError";
}
}
/** The live chat service supplies the authority; serialized wake hints never do. */
export function registerFailedChatRunRetryAuthority(
db: Db,
authority: FailedChatRunRetryAuthority,
): () => void {
failedChatRunRetryAuthorities.set(db, authority);
return () => {
if (failedChatRunRetryAuthorities.get(db) === authority) {
failedChatRunRetryAuthorities.delete(db);
}
};
}
/** Re-discover retry provenance from the durable action, including after restart
* or when an untrusted caller strips the serialized retry selector. */
export async function authorizeFailedChatRunRetryWake(
db: Db,
tx: Db,
input: FailedChatRunRetryAuthorizationInput,
): Promise<boolean> {
const action = input.wakeupRequestId
? await tx
.select({ companyId: chatActions.companyId, kind: chatActions.kind, payload: chatActions.payload })
.from(chatActions)
.where(eq(chatActions.id, input.wakeupRequestId))
.limit(1)
.then((rows) => rows[0] ?? null)
: null;
if (action?.kind !== "failed_run_retry") {
if (Object.hasOwn(input.contextSnapshot, "chatFailedRunRetry")) {
throw new FailedChatRunRetryAuthorizationError();
}
return false;
}
const authority = failedChatRunRetryAuthorities.get(db);
const selector = input.contextSnapshot.chatFailedRunRetry;
if (
!authority ||
action.companyId !== input.companyId ||
!input.issueId ||
!input.wakeupRequestId ||
!selector ||
typeof selector !== "object" ||
Array.isArray(selector) ||
(selector as Record<string, unknown>).version !== 1 ||
(selector as Record<string, unknown>).actionId !== input.wakeupRequestId ||
typeof (selector as Record<string, unknown>).failedRunId !== "string" ||
!(selector as Record<string, unknown>).failedRunId ||
(selector as Record<string, unknown>).failedRunId !==
input.contextSnapshot.retryOfRunId ||
(input.contextSnapshot.forceFreshSession === true &&
(typeof action.payload.taskRetryIntentId !== "string" ||
input.contextSnapshot.taskRetryRequestId !== action.payload.taskRetryIntentId)) ||
["resumeFromRunId", "resumeSessionParams", "resumeSessionDisplayId"].some(
(key) => Object.hasOwn(input.contextSnapshot, key),
)
) {
throw new FailedChatRunRetryAuthorizationError();
}
await authority(tx, {
...input,
issueId: input.issueId,
wakeupRequestId: input.wakeupRequestId,
});
return true;
}
/** Prevent automatic recovery from treating committed-but-unadmitted input as
* an ordinary stranded task. A receipt is historical admission evidence even
* if its run subsequently fails or is cancelled; skipped was never admitted. */
export function unadmittedChatWakeupCondition(
issueId: SQLWrapper,
companyId: SQLWrapper,
) {
return sql`exists (
select 1 from chat_actions chat_unadmitted
where chat_unadmitted.company_id = ${companyId}
and chat_unadmitted.payload->>'issueId' = ${issueId}::text
and chat_unadmitted.kind = 'inbound_wakeup'
and chat_unadmitted.status in ('preparing', 'issued', 'processing', 'failed')
and not exists (
select 1 from agent_wakeup_requests chat_admission
where chat_admission.id = chat_unadmitted.id
and chat_admission.company_id = chat_unadmitted.company_id
and chat_admission.agent_id::text = chat_unadmitted.payload->>'agentId'
and chat_admission.status <> 'skipped'
)
and not exists (
select 1 from chat_actions chat_retry
join agent_wakeup_requests chat_retry_admission on chat_retry_admission.id = chat_retry.id
where chat_retry.company_id = chat_unadmitted.company_id
and chat_retry.endpoint_id = chat_unadmitted.endpoint_id
and chat_retry.conversation_id = chat_unadmitted.conversation_id
and chat_retry.kind = 'failed_run_retry'
and chat_retry.payload->'taskRetryInput'->'skippedActionIds' ? chat_unadmitted.id::text
and chat_retry.payload->>'issueId' = chat_unadmitted.payload->>'issueId'
and chat_retry_admission.company_id = chat_retry.company_id
and chat_retry_admission.agent_id::text = chat_retry.payload->>'agentId'
and chat_retry_admission.status <> 'skipped'
)
and (chat_unadmitted.status <> 'failed' or not exists (
select 1 from chat_actions chat_later
join agent_wakeup_requests chat_later_admission on chat_later_admission.id = chat_later.id
where chat_later.company_id = chat_unadmitted.company_id
and chat_later.endpoint_id = chat_unadmitted.endpoint_id
and chat_later.conversation_id = chat_unadmitted.conversation_id
and chat_later.payload->>'sessionGeneration' = chat_unadmitted.payload->>'sessionGeneration'
and chat_later.payload->>'issueId' = chat_unadmitted.payload->>'issueId'
and chat_later.kind = 'inbound_wakeup'
and (chat_later.created_at, chat_later.id) > (chat_unadmitted.created_at, chat_unadmitted.id)
and chat_later_admission.company_id = chat_later.company_id
and chat_later_admission.agent_id::text = chat_later.payload->>'agentId'
and chat_later_admission.status <> 'skipped'
))
)`;
}
export interface DeferredChatWakeScope {
companyId: string;
agentId: string;
issueId: string;
requestId: string;
recoverySweep?: boolean;
}
export type DeferredChatWakeDecline = { settle: string } | { retry: string };
export const CLOSED_TASK_CHAT_WAKE_REOPEN_WINDOW_MS = 60 * 60_000;
export const SILENT_CHAT_WAKE_SETTLEMENT_AFTER_MS = 24 * 60 * 60_000;
export const DEFERRED_CHAT_WAKE_RETRY_KEY = "chatResume";
export const SILENT_CHAT_WAKE_SETTLEMENT_KEY = "silentSettlement";
export const DEFERRED_CHAT_WAKE_RETRY_DELAYS_MS = [60_000, 10 * 60_000, 60 * 60_000, 6 * 60 * 60_000, 24 * 60 * 60_000];
const SETTLED_AUTHORITY_REASONS = new Set<string>(["source_shape", "delivery_binding", "conversation_binding",
"destination_binding", "source_comment"]);
const DECLINED_CHAT_INPUT_NOTICE =
"Input saved. Its work request was skipped, cancelled, or failed; no automatic replay was performed.";
export function deferredChatWakeNextAttemptAt(payload: Record<string, unknown> | null | undefined): number | null {
const retry = payload?.[DEFERRED_CHAT_WAKE_RETRY_KEY];
if (!retry || typeof retry !== "object" || Array.isArray(retry)) return null;
const at = Date.parse(String((retry as Record<string, unknown>).nextAttemptAt));
return Number.isFinite(at) ? at : null;
}
export function deferredChatWakeDeclineForError(error: unknown): DeferredChatWakeDecline {
const authority = safeChatWakeAuthorityReason(error);
if (authority) return SETTLED_AUTHORITY_REASONS.has(authority)
? { settle: `authority_${authority}` } : { retry: `authority_${authority}` };
if (error instanceof HttpError) {
const details = error.details && typeof error.details === "object" ? error.details as Record<string, unknown> : {};
return { retry: typeof details.code === "string" ? details.code : `http_${error.status}` };
}
return { retry: error instanceof Error && /^chat_[a-z_]+$/.test(error.message) ? error.message : "resume_failed" };
}
export async function recordDeferredChatWakeDecline(db: Db,
scope: { companyId: string; issueId: string; requestId: string }, decline: DeferredChatWakeDecline) {
return db.transaction(async tx => {
const [receipt] = await tx.select({ payload: agentWakeupRequests.payload, createdAt: agentWakeupRequests.createdAt })
.from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, scope.companyId), eq(agentWakeupRequests.id, scope.requestId),
eq(agentWakeupRequests.status, "deferred_issue_execution"), isNull(agentWakeupRequests.runId))).for("update");
if (!receipt) return null;
const previous = receipt.payload?.[DEFERRED_CHAT_WAKE_RETRY_KEY];
const attempts = (previous && typeof previous === "object" && !Array.isArray(previous)
? Number((previous as Record<string, unknown>).attempts) || 0 : 0) + 1;
const now = new Date();
if ("retry" in decline && attempts <= DEFERRED_CHAT_WAKE_RETRY_DELAYS_MS.length) {
const nextAttemptAt = now.getTime() + DEFERRED_CHAT_WAKE_RETRY_DELAYS_MS[attempts - 1]!;
await tx.update(agentWakeupRequests).set({ error: decline.retry, updatedAt: now,
payload: sql`coalesce(${agentWakeupRequests.payload}, '{}'::jsonb) || ${JSON.stringify({ [DEFERRED_CHAT_WAKE_RETRY_KEY]: {
attempts, reason: decline.retry, nextAttemptAt: new Date(nextAttemptAt).toISOString() } })}::jsonb`,
}).where(eq(agentWakeupRequests.id, scope.requestId));
return { outcome: "retry" as const, reason: decline.retry, attempts, nextAttemptAt };
}
const reason = "settle" in decline ? decline.settle : `retry_exhausted:${decline.retry}`;
const silent = receipt.createdAt.getTime() < now.getTime() - SILENT_CHAT_WAKE_SETTLEMENT_AFTER_MS;
await tx.update(agentWakeupRequests).set({ status: "skipped", error: reason, finishedAt: now, updatedAt: now,
payload: silent
? sql`(${agentWakeupRequests.payload} - ${DEFERRED_CHAT_WAKE_RETRY_KEY}::text) || ${JSON.stringify({ [SILENT_CHAT_WAKE_SETTLEMENT_KEY]: true })}::jsonb`
: sql`${agentWakeupRequests.payload} - ${DEFERRED_CHAT_WAKE_RETRY_KEY}::text`,
}).where(eq(agentWakeupRequests.id, scope.requestId));
if (silent) return { outcome: "settled" as const, reason, attempts, silent };
await notifyChatInboundWakeNotices(tx, { companyId: scope.companyId, issueId: scope.issueId });
await tx.update(chatDeliveries).set({ redactedError: DECLINED_CHAT_INPUT_NOTICE, updatedAt: now }).where(and(
eq(chatDeliveries.companyId, scope.companyId), eq(chatDeliveries.state, "processed"),
sql`${chatDeliveries.id} = (select ${chatActions.deliveryId} from ${chatActions}
where ${chatActions.companyId} = ${scope.companyId} and ${chatActions.id} = ${scope.requestId}
and ${chatActions.kind} = 'inbound_wakeup')`));
return { outcome: "settled" as const, reason, attempts, silent };
});
}
export async function clearDeferredChatWakeRetry(db: Db, scope: { companyId: string; requestId: string }) {
await db.update(agentWakeupRequests).set({ error: null,
payload: sql`${agentWakeupRequests.payload} - ${DEFERRED_CHAT_WAKE_RETRY_KEY}::text`,
}).where(and(eq(agentWakeupRequests.companyId, scope.companyId), eq(agentWakeupRequests.id, scope.requestId),
sql`${agentWakeupRequests.payload} ? ${DEFERRED_CHAT_WAKE_RETRY_KEY}::text`));
}
/** Identify saved input by its canonical receipt, never by a caller's source
* label. This is selection only; the inbound owner rechecks live authority. */
export function deferredChatInputPredicate(db: Db) {
return exists(db.select({ id: chatActions.id }).from(chatActions)
.innerJoin(chatConversations, and(eq(chatConversations.companyId, chatActions.companyId),
eq(chatConversations.id, chatActions.conversationId)))
.innerJoin(chatEndpoints, and(eq(chatEndpoints.companyId, chatActions.companyId),
eq(chatEndpoints.id, chatActions.endpointId), eq(chatEndpoints.id, chatConversations.endpointId)))
.where(and(eq(chatActions.companyId, agentWakeupRequests.companyId), eq(chatActions.id, agentWakeupRequests.id),
eq(chatActions.kind, "inbound_wakeup"), eq(chatActions.status, "processed"),
eq(chatEndpoints.assignedAgentId, agentWakeupRequests.agentId), eq(chatEndpoints.externalExecutionPolicy, "restricted"),
sql`${chatConversations.issueId}::text = ${agentWakeupRequests.payload}->>'issueId'`,
sql`${chatActions.payload}->>'issueId' = ${agentWakeupRequests.payload}->>'issueId'`,
sql`${chatActions.payload}->>'agentId' = ${agentWakeupRequests.agentId}::text`)));
}
type DeferredChatWakeOwner = (input: DeferredChatWakeScope) => Promise<DeferredChatWakeDecline | void>;
const deferredChatWakeOwners = new WeakMap<Db, DeferredChatWakeOwner>();
/** The live inbound owner reauthorizes the original source. Cleanup owns no chat authority. */
export function registerDeferredChatWakeOwner(db: Db, owner: DeferredChatWakeOwner) {
deferredChatWakeOwners.set(db, owner);
return () => { if (deferredChatWakeOwners.get(db) === owner) deferredChatWakeOwners.delete(db); };
}
export async function resumeDeferredChatWake(db: Db, input: DeferredChatWakeScope): Promise<DeferredChatWakeDecline | undefined> {
const owner = deferredChatWakeOwners.get(db);
if (!owner) return undefined;
return (await owner(input)) ?? undefined;
}
/** Internal authority: this object cannot be supplied through a JSON API. */
export interface DurableChatWakeupRequest {
readonly id: string;
readonly companyId: string;
readonly agentId: string;
readonly issueId: string;
readonly commentId: string;
readonly requestedByActorType: "user" | "system";
readonly requestedByActorId: string;
readonly requestedAt: Date;
readonly idempotencyKey: string;
readonly failedRunRetry?: { readonly failedRunId: string };
/** Only the current inbound owner may reconsider its already saved, unconsumed receipt. */
readonly resumeDeferred?: true;
readonly authorize: (tx: Db) => Promise<void>;
readonly onAdmitted?: (tx: Db, receipt: { runId: string | null }) => Promise<void>;
}
const trustedRequests = new WeakSet<object>();
export function createDurableChatWakeupRequest(
input: Omit<DurableChatWakeupRequest, "idempotencyKey">,
): DurableChatWakeupRequest {
const idempotencyKey = `chat-inbound:${createHash("sha256")
.update(
JSON.stringify([
input.id,
input.companyId,
input.agentId,
input.issueId,
input.commentId,
input.requestedByActorType,
input.requestedByActorId,
]),
)
.digest("hex")}`;
const request = Object.freeze({
...input,
...(input.failedRunRetry
? { failedRunRetry: Object.freeze({ ...input.failedRunRetry }) }
: {}),
idempotencyKey,
});
trustedRequests.add(request);
return request;
}
export function assertDurableChatWakeupRequest(
request: DurableChatWakeupRequest,
scope: {
agentId: string;
companyId: string;
issueId: string | null;
commentId: string | null;
requestedByActorType?: string;
requestedByActorId?: string | null;
},
) {
if (
!trustedRequests.has(request) ||
!scope.issueId ||
request.agentId !== scope.agentId ||
request.companyId !== scope.companyId ||
request.issueId !== scope.issueId ||
request.commentId !== scope.commentId ||
request.requestedByActorType !== scope.requestedByActorType ||
request.requestedByActorId !== scope.requestedByActorId
) {
throw new Error("chat_inbound_wakeup_binding_denied");
}
}
export function assertDurableChatWakeupReceipt(
request: DurableChatWakeupRequest,
receipt: {
id: string;
companyId: string;
agentId: string;
idempotencyKey: string | null;
requestedByActorType: string | null;
requestedByActorId: string | null;
},
) {
if (
receipt.id !== request.id ||
receipt.companyId !== request.companyId ||
receipt.agentId !== request.agentId ||
receipt.idempotencyKey !== request.idempotencyKey ||
receipt.requestedByActorType !== request.requestedByActorType ||
receipt.requestedByActorId !== request.requestedByActorId
) {
throw new Error("chat_inbound_wakeup_receipt_conflict");
}
}
export function withDurableChatAdmission<T extends { kind: string; admitted?: boolean }>(request: DurableChatWakeupRequest | undefined,
admit: (tx: Parameters<Parameters<Db["transaction"]>[0]>[0]) => Promise<T>) {
return async (tx: Parameters<Parameters<Db["transaction"]>[0]>[0]) => {
const outcome = await admit(tx);
// An existing deferred receipt is not proof this transaction passed its
// gates. Only dispatch or an explicitly persisted wait admits the input.
if (request?.onAdmitted && (outcome.admitted === true || ["queued", "durable", "coalesced"].includes(outcome.kind))) {
const [receipt] = await tx.select().from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, request.companyId), eq(agentWakeupRequests.id, request.id),
inArray(agentWakeupRequests.status, ["queued", "claimed", "deferred_issue_execution", "coalesced"]),
));
if (receipt) {
assertDurableChatWakeupReceipt(request, receipt);
await request.onAdmitted(tx as unknown as Db, { runId: receipt.runId });
}
}
return outcome;
};
}
/** Work-start policy only: ongoing tool and publication authorization must not
* revoke an already-started run when the operator enables mention gating. */
export async function slackMentionAllowsRunStart(
db: Db,
run: { id: string; companyId: string; agentId: string; wakeupRequestId: string | null },
): Promise<boolean> {
if (!run.wakeupRequestId) return true;
const inputs = await db.select({ event: chatDeliveries.normalizedEvent })
.from(agentWakeupRequests)
.innerJoin(chatActions, and(
eq(chatActions.id, agentWakeupRequests.id),
eq(chatActions.companyId, run.companyId),
eq(chatActions.kind, "inbound_wakeup"),
))
.innerJoin(chatEndpoints, and(
eq(chatEndpoints.id, chatActions.endpointId),
eq(chatEndpoints.companyId, run.companyId),
eq(chatEndpoints.provider, "slack"),
eq(chatEndpoints.requireAtMention, true),
))
.innerJoin(chatDeliveries, and(
eq(chatDeliveries.id, chatActions.deliveryId),
eq(chatDeliveries.endpointId, chatEndpoints.id),
eq(chatDeliveries.companyId, run.companyId),
))
.where(and(
eq(agentWakeupRequests.companyId, run.companyId),
eq(agentWakeupRequests.agentId, run.agentId),
or(
eq(agentWakeupRequests.runId, run.id),
eq(agentWakeupRequests.id, run.wakeupRequestId),
sql`${agentWakeupRequests.payload}->>'coalescedIntoWakeupRequestId' = ${run.wakeupRequestId}`,
),
));
return inputs.every(({ event }) =>
slackDeliveryMeetsMentionRequirement({ requireAtMention: true }, event));
}