Files
PaperClipAI/server/src/services/durable-chat-wakeup.ts
T
DottaandPaperclip 0710c56842 fix: retry saved chat input after a successful run (#18621)
## Thinking Path

> - Paperclip manages AI agents and the work they receive.
> - Connected conversations save incoming requests before they can run.
> - A request can remain queued when the previous turn finishes
successfully.
> - Task Retry sent that case through the failed-run retry guard, which
rejected the successful run.
> - This change resumes the saved request through its existing admission
owner.
> - The operator can continue saved work while preserving the prior
result and current permissions.

## Linked Issues or Issue Description

**What happened?**

Task Retry returned a conflict for a restricted connected conversation
when its latest run succeeded and newer input remained saved. A GitHub
review can reach this state when a new PR head arrives during the
previous review. The prior review cannot publish against the new head,
but its completed turn is not a failed run that can be replayed.

**Expected behavior**

Retry should resume the saved request with its original source identity
and current authorization. It must preserve the previous successful
result and must not replay that turn.

**Steps to reproduce**

1. Start a review through a restricted GitHub endpoint.
2. Send a PR synchronize event while that review runs and retain its
deferred input receipt.
3. Let the original run finish successfully with the task still open.
4. Click Retry. Before this change, the failed-run guard rejects the
request. With this change, the saved input creates one successor through
normal admission.

Related work: #17725, #16965, #15033.

## What Changed

- Select the newest saved input by its canonical chat action and wake
receipt.
- Store its receipt ID in the task retry intent and resume it through
the existing chat owner.
- Preserve caller permissions, source authorization, task state, agent
state, cleanup waits, and duplicate-click handling.
- Exclude saved-input intents from generic board-message replay. Keep
board messages in separate queued receipts and allow independently saved
input after successful board turns.
- Add signed webhook, HTTP endpoint, accepted-result preservation, and
refusal tests. Update execution semantics and capability maps. Stabilize
the existing Teams lifecycle test with explicit database result
ordering.

## Verification

- Current-head [CI
passes](https://github.com/paperclipai/paperclip/actions/runs/41482727909):
54 successful checks, one neutral review, three expected skips, and a
successful Snyk status. Greptile gives commit `53f02e9105` 5/5 with no
unresolved review threads.
- Reproduced the original HTTP 409 before the fix. Reproduced both
review findings with three failing cases before fixing them.
- Passed 25 focused and related GitHub retry cases, 52 ordinary queue
tests, and three native startup cases against disposable PostgreSQL.
Earlier related verification also passed 83 GitHub retry cases and 109
recovery/receipt cases.
- Passed `pnpm -r typecheck`, `pnpm build`, `node
scripts/check-maps.mjs`, and the module boundary check. The final server
typecheck passed after the review fixes.
- The full local `pnpm test:run` attempt stopped on an existing Slack
queue deadline assertion with a 1 ms database/host clock difference.
That case passed in isolation. A second attempt was stopped when review
changes superseded its source. Full local completion is not claimed; all
chat, server, browser, build, and runner CI jobs passed on the final
commit.
- The runner CI job needed one rerun after a two-second startup timing
assertion failed. That case and its two related cases passed locally;
the rerun passed without code changes.
- Live GitHub review publication with this patch is not verified. The
signed provider fixture uses mocked GitHub responses. This branch is not
deployed.

## Risks

- This changes how Retry dispatches deferred external input after a
successful run. It relies on the existing inbound owner to recheck
source authority before execution.
- Tests cover paused connections and agents, changed source bindings,
completed tasks, pending cleanup, duplicate clicks, and preservation of
the accepted result.
- No schema migration or public API shape change is required.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, code execution, and browser
tools. The exact deployed model ID and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-02 11:46:46 -06:00

417 lines
16 KiB
TypeScript

import { createHash } from "node:crypto";
import { agentWakeupRequests, chatActions, chatConversations, chatDeliveries, chatEndpoints, type Db } from "@paperclipai/db";
import { and, eq, exists, inArray, or, sql, type SQLWrapper } from "drizzle-orm";
import { HttpError } from "../errors.js";
export interface CommittedChatResponseAuthorizationInput {
companyId: string;
issueId: string;
agentId: string;
runId: string;
resultId: string;
}
type CommittedChatResponseAuthority = (
tx: Db,
input: CommittedChatResponseAuthorizationInput,
) => Promise<void>;
const committedChatResponseAuthorities = new WeakMap<
Db,
CommittedChatResponseAuthority
>();
const committedChatResponseAuthorityListeners = new WeakMap<Db, Set<() => void>>();
export function onCommittedChatResponseAuthorityRegistered(db: Db, listener: () => void): () => void {
let listeners = committedChatResponseAuthorityListeners.get(db);
if (!listeners) {
listeners = new Set();
committedChatResponseAuthorityListeners.set(db, listeners);
}
listeners.add(listener);
return () => {
listeners.delete(listener);
if (listeners.size === 0 && committedChatResponseAuthorityListeners.get(db) === listeners) committedChatResponseAuthorityListeners.delete(db);
};
}
export class CommittedChatResponseAuthorizationError extends HttpError {
constructor() {
super(
409,
"The accepted chat response is no longer authorized for this conversation.",
{
code: "chat_committed_response_not_authorized",
},
);
this.name = "CommittedChatResponseAuthorizationError";
}
}
/** Separate, server-only authority for presenting an already accepted result.
* This grants no permission to execute or repair the provider session. */
export function registerCommittedChatResponseAuthority(
db: Db,
authority: CommittedChatResponseAuthority,
): () => void {
committedChatResponseAuthorities.set(db, authority);
for (const listener of committedChatResponseAuthorityListeners.get(db) ?? []) listener();
return () => {
if (committedChatResponseAuthorities.get(db) === authority) {
committedChatResponseAuthorities.delete(db);
}
};
}
export async function authorizeCommittedChatResponse(
db: Db,
tx: Db,
input: CommittedChatResponseAuthorizationInput,
): Promise<void> {
const authority = committedChatResponseAuthorities.get(db);
if (!authority) throw new CommittedChatResponseAuthorizationError();
await authority(tx, input);
}
export interface FailedChatRunRetryAuthorizationInput {
phase: "admission" | "promotion" | "execution";
wakeupRequestId: string | null;
companyId: string;
agentId: string;
issueId: string | null;
runId?: string;
contextSnapshot: Record<string, unknown>;
}
type FailedChatRunRetryAuthority = (
tx: Db,
input: FailedChatRunRetryAuthorizationInput & {
wakeupRequestId: string;
issueId: string;
},
) => Promise<void>;
const failedChatRunRetryAuthorities = new WeakMap<
Db,
FailedChatRunRetryAuthority
>();
export class FailedChatRunRetryAuthorizationError extends HttpError {
constructor() {
super(
409,
"The exact failed chat request is no longer authorized. Send a new request in the current connected conversation.",
{
code: "chat_failed_run_retry_not_authorized",
},
);
this.name = "FailedChatRunRetryAuthorizationError";
}
}
/** The live chat service supplies the authority; serialized wake hints never do. */
export function registerFailedChatRunRetryAuthority(
db: Db,
authority: FailedChatRunRetryAuthority,
): () => void {
failedChatRunRetryAuthorities.set(db, authority);
return () => {
if (failedChatRunRetryAuthorities.get(db) === authority) {
failedChatRunRetryAuthorities.delete(db);
}
};
}
/** Re-discover retry provenance from the durable action, including after restart
* or when an untrusted caller strips the serialized retry selector. */
export async function authorizeFailedChatRunRetryWake(
db: Db,
tx: Db,
input: FailedChatRunRetryAuthorizationInput,
): Promise<boolean> {
const action = input.wakeupRequestId
? await tx
.select({ companyId: chatActions.companyId, kind: chatActions.kind, payload: chatActions.payload })
.from(chatActions)
.where(eq(chatActions.id, input.wakeupRequestId))
.limit(1)
.then((rows) => rows[0] ?? null)
: null;
if (action?.kind !== "failed_run_retry") {
if (Object.hasOwn(input.contextSnapshot, "chatFailedRunRetry")) {
throw new FailedChatRunRetryAuthorizationError();
}
return false;
}
const authority = failedChatRunRetryAuthorities.get(db);
const selector = input.contextSnapshot.chatFailedRunRetry;
if (
!authority ||
action.companyId !== input.companyId ||
!input.issueId ||
!input.wakeupRequestId ||
!selector ||
typeof selector !== "object" ||
Array.isArray(selector) ||
(selector as Record<string, unknown>).version !== 1 ||
(selector as Record<string, unknown>).actionId !== input.wakeupRequestId ||
typeof (selector as Record<string, unknown>).failedRunId !== "string" ||
!(selector as Record<string, unknown>).failedRunId ||
(selector as Record<string, unknown>).failedRunId !==
input.contextSnapshot.retryOfRunId ||
(input.contextSnapshot.forceFreshSession === true &&
(typeof action.payload.taskRetryIntentId !== "string" ||
input.contextSnapshot.taskRetryRequestId !== action.payload.taskRetryIntentId)) ||
["resumeFromRunId", "resumeSessionParams", "resumeSessionDisplayId"].some(
(key) => Object.hasOwn(input.contextSnapshot, key),
)
) {
throw new FailedChatRunRetryAuthorizationError();
}
await authority(tx, {
...input,
issueId: input.issueId,
wakeupRequestId: input.wakeupRequestId,
});
return true;
}
/** Prevent automatic recovery from treating committed-but-unadmitted input as
* an ordinary stranded task. A receipt is historical admission evidence even
* if its run subsequently fails or is cancelled; skipped was never admitted. */
export function unadmittedChatWakeupCondition(
issueId: SQLWrapper,
companyId: SQLWrapper,
) {
return sql`exists (
select 1 from chat_actions chat_unadmitted
where chat_unadmitted.company_id = ${companyId}
and chat_unadmitted.payload->>'issueId' = ${issueId}::text
and chat_unadmitted.kind = 'inbound_wakeup'
and chat_unadmitted.status in ('preparing', 'issued', 'processing', 'failed')
and not exists (
select 1 from agent_wakeup_requests chat_admission
where chat_admission.id = chat_unadmitted.id
and chat_admission.company_id = chat_unadmitted.company_id
and chat_admission.agent_id::text = chat_unadmitted.payload->>'agentId'
and chat_admission.status <> 'skipped'
)
and not exists (
select 1 from chat_actions chat_retry
join agent_wakeup_requests chat_retry_admission on chat_retry_admission.id = chat_retry.id
where chat_retry.company_id = chat_unadmitted.company_id
and chat_retry.endpoint_id = chat_unadmitted.endpoint_id
and chat_retry.conversation_id = chat_unadmitted.conversation_id
and chat_retry.kind = 'failed_run_retry'
and chat_retry.payload->'taskRetryInput'->'skippedActionIds' ? chat_unadmitted.id::text
and chat_retry.payload->>'issueId' = chat_unadmitted.payload->>'issueId'
and chat_retry_admission.company_id = chat_retry.company_id
and chat_retry_admission.agent_id::text = chat_retry.payload->>'agentId'
and chat_retry_admission.status <> 'skipped'
)
and (chat_unadmitted.status <> 'failed' or not exists (
select 1 from chat_actions chat_later
join agent_wakeup_requests chat_later_admission on chat_later_admission.id = chat_later.id
where chat_later.company_id = chat_unadmitted.company_id
and chat_later.endpoint_id = chat_unadmitted.endpoint_id
and chat_later.conversation_id = chat_unadmitted.conversation_id
and chat_later.payload->>'sessionGeneration' = chat_unadmitted.payload->>'sessionGeneration'
and chat_later.payload->>'issueId' = chat_unadmitted.payload->>'issueId'
and chat_later.kind = 'inbound_wakeup'
and (chat_later.created_at, chat_later.id) > (chat_unadmitted.created_at, chat_unadmitted.id)
and chat_later_admission.company_id = chat_later.company_id
and chat_later_admission.agent_id::text = chat_later.payload->>'agentId'
and chat_later_admission.status <> 'skipped'
))
)`;
}
export interface DeferredChatWakeScope {
companyId: string;
agentId: string;
issueId: string;
requestId: string;
}
/** Identify saved input by its canonical receipt, never by a caller's source
* label. This is selection only; the inbound owner rechecks live authority. */
export function deferredChatInputPredicate(db: Db) {
return exists(db.select({ id: chatActions.id }).from(chatActions)
.innerJoin(chatConversations, and(eq(chatConversations.companyId, chatActions.companyId),
eq(chatConversations.id, chatActions.conversationId)))
.innerJoin(chatEndpoints, and(eq(chatEndpoints.companyId, chatActions.companyId),
eq(chatEndpoints.id, chatActions.endpointId), eq(chatEndpoints.id, chatConversations.endpointId)))
.where(and(eq(chatActions.companyId, agentWakeupRequests.companyId), eq(chatActions.id, agentWakeupRequests.id),
eq(chatActions.kind, "inbound_wakeup"), eq(chatActions.status, "processed"),
eq(chatEndpoints.assignedAgentId, agentWakeupRequests.agentId), eq(chatEndpoints.externalExecutionPolicy, "restricted"),
sql`${chatConversations.issueId}::text = ${agentWakeupRequests.payload}->>'issueId'`,
sql`${chatActions.payload}->>'issueId' = ${agentWakeupRequests.payload}->>'issueId'`,
sql`${chatActions.payload}->>'agentId' = ${agentWakeupRequests.agentId}::text`)));
}
const deferredChatWakeOwners = new WeakMap<Db, (input: DeferredChatWakeScope) => Promise<void>>();
/** The live inbound owner reauthorizes the original source. Cleanup owns no chat authority. */
export function registerDeferredChatWakeOwner(db: Db, owner: (input: DeferredChatWakeScope) => Promise<void>) {
deferredChatWakeOwners.set(db, owner);
return () => { if (deferredChatWakeOwners.get(db) === owner) deferredChatWakeOwners.delete(db); };
}
export async function resumeDeferredChatWake(db: Db, input: DeferredChatWakeScope) {
await deferredChatWakeOwners.get(db)?.(input);
}
/** Internal authority: this object cannot be supplied through a JSON API. */
export interface DurableChatWakeupRequest {
readonly id: string;
readonly companyId: string;
readonly agentId: string;
readonly issueId: string;
readonly commentId: string;
readonly requestedByActorType: "user" | "system";
readonly requestedByActorId: string;
readonly requestedAt: Date;
readonly idempotencyKey: string;
readonly failedRunRetry?: { readonly failedRunId: string };
/** Only the current inbound owner may reconsider its already saved, unconsumed receipt. */
readonly resumeDeferred?: true;
readonly authorize: (tx: Db) => Promise<void>;
readonly onAdmitted?: (tx: Db, receipt: { runId: string | null }) => Promise<void>;
}
const trustedRequests = new WeakSet<object>();
export function createDurableChatWakeupRequest(
input: Omit<DurableChatWakeupRequest, "idempotencyKey">,
): DurableChatWakeupRequest {
const idempotencyKey = `chat-inbound:${createHash("sha256")
.update(
JSON.stringify([
input.id,
input.companyId,
input.agentId,
input.issueId,
input.commentId,
input.requestedByActorType,
input.requestedByActorId,
]),
)
.digest("hex")}`;
const request = Object.freeze({
...input,
...(input.failedRunRetry
? { failedRunRetry: Object.freeze({ ...input.failedRunRetry }) }
: {}),
idempotencyKey,
});
trustedRequests.add(request);
return request;
}
export function assertDurableChatWakeupRequest(
request: DurableChatWakeupRequest,
scope: {
agentId: string;
companyId: string;
issueId: string | null;
commentId: string | null;
requestedByActorType?: string;
requestedByActorId?: string | null;
},
) {
if (
!trustedRequests.has(request) ||
!scope.issueId ||
request.agentId !== scope.agentId ||
request.companyId !== scope.companyId ||
request.issueId !== scope.issueId ||
request.commentId !== scope.commentId ||
request.requestedByActorType !== scope.requestedByActorType ||
request.requestedByActorId !== scope.requestedByActorId
) {
throw new Error("chat_inbound_wakeup_binding_denied");
}
}
export function assertDurableChatWakeupReceipt(
request: DurableChatWakeupRequest,
receipt: {
id: string;
companyId: string;
agentId: string;
idempotencyKey: string | null;
requestedByActorType: string | null;
requestedByActorId: string | null;
},
) {
if (
receipt.id !== request.id ||
receipt.companyId !== request.companyId ||
receipt.agentId !== request.agentId ||
receipt.idempotencyKey !== request.idempotencyKey ||
receipt.requestedByActorType !== request.requestedByActorType ||
receipt.requestedByActorId !== request.requestedByActorId
) {
throw new Error("chat_inbound_wakeup_receipt_conflict");
}
}
export function withDurableChatAdmission<T extends { kind: string; admitted?: boolean }>(request: DurableChatWakeupRequest | undefined,
admit: (tx: Parameters<Parameters<Db["transaction"]>[0]>[0]) => Promise<T>) {
return async (tx: Parameters<Parameters<Db["transaction"]>[0]>[0]) => {
const outcome = await admit(tx);
// An existing deferred receipt is not proof this transaction passed its
// gates. Only dispatch or an explicitly persisted wait admits the input.
if (request?.onAdmitted && (outcome.admitted === true || ["queued", "durable", "coalesced"].includes(outcome.kind))) {
const [receipt] = await tx.select().from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, request.companyId), eq(agentWakeupRequests.id, request.id),
inArray(agentWakeupRequests.status, ["queued", "claimed", "deferred_issue_execution", "coalesced"]),
));
if (receipt) {
assertDurableChatWakeupReceipt(request, receipt);
await request.onAdmitted(tx as unknown as Db, { runId: receipt.runId });
}
}
return outcome;
};
}
/** Work-start policy only: ongoing tool and publication authorization must not
* revoke an already-started run when the operator enables mention gating. */
export async function slackMentionAllowsRunStart(
db: Db,
run: { id: string; companyId: string; agentId: string; wakeupRequestId: string | null },
): Promise<boolean> {
if (!run.wakeupRequestId) return true;
const inputs = await db.select({ event: chatDeliveries.normalizedEvent })
.from(agentWakeupRequests)
.innerJoin(chatActions, and(
eq(chatActions.id, agentWakeupRequests.id),
eq(chatActions.companyId, run.companyId),
eq(chatActions.kind, "inbound_wakeup"),
))
.innerJoin(chatEndpoints, and(
eq(chatEndpoints.id, chatActions.endpointId),
eq(chatEndpoints.companyId, run.companyId),
eq(chatEndpoints.provider, "slack"),
eq(chatEndpoints.requireAtMention, true),
))
.innerJoin(chatDeliveries, and(
eq(chatDeliveries.id, chatActions.deliveryId),
eq(chatDeliveries.endpointId, chatEndpoints.id),
eq(chatDeliveries.companyId, run.companyId),
))
.where(and(
eq(agentWakeupRequests.companyId, run.companyId),
eq(agentWakeupRequests.agentId, run.agentId),
or(
eq(agentWakeupRequests.runId, run.id),
eq(agentWakeupRequests.id, run.wakeupRequestId),
sql`${agentWakeupRequests.payload}->>'coalescedIntoWakeupRequestId' = ${run.wakeupRequestId}`,
),
));
return inputs.every(({ event }) =>
(event.conversation as { isDirectMessage?: boolean } | undefined)?.isDirectMessage === true ||
event.trigger === "mention" ||
(event.message as { mentionedBot?: boolean } | undefined)?.mentionedBot === true);
}