Files
PaperClipAI/server/src/services/durable-chat-wakeup.ts
T
ec3b558277 fix(server): schedule accepted native response recovery (#17696)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs store accepted results and their finalization decisions.
> - A completed run can still owe its public answer after an
interruption.
> - Periodic recovery scans for that answer even when no new work
exists.
> - This pull request wakes recovery from committed changes and saved
retry deadlines.
> - The existing owner still verifies the result, source, and current
publication authority.
> - Settled or dormant response work no longer needs its own periodic
scan.

## Linked Issues or Issue Description

Refs #17690 (merged). This is slice 48 of the sleep migration stack, now
based on master. Related: #17661 handles GitHub terminal publication;
this change handles native accepted-response presentation.
Governed-review presentation remains a separate duty.

**What happened?**

Native finalization recovery repeatedly checks already accepted results
for a missing answer. Removing that check without complete notifications
would lose responses when evidence arrives late, an agent resumes, or
admission settles after execution.

**Expected behavior**

Recover the answer from its saved result when a supported change makes
it eligible. Retain real retry deadlines. Stop querying when the answer
is selected or cannot currently be published. Reconstruct work after
restart.

**Steps to reproduce**

1. Commit a native accepted result with no selected response comment.
2. Run finalization recovery while its source is unavailable or locked.
3. Restore that source or release the lock. The answer must appear once
without repeating provider execution.

## What Changed

- Honor recorded non-create decisions and skip-comment policy in both
discovery and recovery. Apply durable conversation/result/continuation
guards before recovery locks. Preserve genuine unanswered and legacy
recovery.
- Scope routine response wakes to the affected issue/run. Coalesced
input names its active run; pending-effect settlement names its issue.
Sixteen existing global authority/configuration hooks use one explicit
company invalidation boundary. No new transaction wrappers or tables are
added.
- Add an accepted-response worker using the existing delivery
coordinator and scoped deadline helper. Preserve saved retry deadlines
and pending-effect fences under the owner's lock.
- Share the finalization notification boundary. Wake on finalization
settlement, exact accepted-result/final-message evidence, agent
availability, retry admission, and existing chat authority changes.
- Keep lifecycle policy in services. A narrow pre-write status hook lets
lifecycle retain ownership of its transaction and status transitions.
- Register the worker during enabled maintenance startup. Remove its
authority subscription on shutdown. Periodic replay skips this duty on
both committed and fresh-commit paths; ordinary finalization and
explicit recovery retain their behavior.
- Add a partial index and expression statistics for unpresented native
runs. Migration 0412 supports concurrent replay and adds no queue table.
- Add real PostgreSQL and lifecycle coverage. Preserve the
implementation and producer audit in the plan document.

## Verification

- Additional real Photon callback acceptance on the published production
code disproves the security warning: ten active assertions cause zero
response-recovery hints/SELECTs, while three concurrent assertions
during an actual attention-to-active recovery cause exactly one
wake/SELECT. The existing locked status guard already enforces this
boundary. No production change was needed; the resolved finding now has
a successful current-head security check. The regression will be
retained with the next slice.
- Eligibility repair source `892f36e5f66ba1af123034936a7160c64c3ab883`
passes **1489 distinct server cases**: 1139 in six complete
recovery/finalizer/monitor/summary/workspace suites (81.02 seconds),
plus 350 selected chat cases (297.68s). Another 1463 chat cases are
outside the pattern; the complete 1813-case file is not claimed. All
eleven new regressions are included in that count. Five dedicated fresh
PostgreSQL 17 databases migrate through unchanged 0412, with no
fixture-source adaptations. Database build, server types/build,
boundaries and full-SHA migration ordering pass. All seven test source
hashes match.
- The exact published baseline first fails eight behavioral assertions
while the three genuine-answer cases pass. The same corrected test
source passes all eleven after repair: recorded none/reuse remain
suppressed; five ineligible history families take zero recovery row
locks across repeated issue hints; missing/legacy/create-null answers
still publish once; fresh periodic finalization defers presentation to
its worker. Earlier fixture setup and variable/compiler failures remain
explained in the audit, rather than erased by later success.
- Repair source `2fc8b0614aaf9ac7337045de4ced7325f95edc45` passes **1821
server cases**: 1496 in 13 complete producer/lifecycle/authority suites
(110.63 seconds), plus 325 selected chat cases (278.99 seconds).
Coverage includes all native-session cases, queue admission, status
decisions, membership/board access, the complete GitHub workflow,
accepted-response lifecycle and real authority restoration. The other
1477 chat cases are outside this selection; the complete 1802-case file
is not claimed for this repair run.
- Actual PostgreSQL regressions first reproduce a coalesced-input hint
missing its run scope and two routine hint paths producing 12
response-history SELECTs each after three notifications. The same
assertions pass with run-scoped coalescing and zero response-history
SELECTs for unscoped routine hints. Actual membership restoration still
publishes the waiting response once. Assertion time limits are
unchanged; this measures real driver SQL and persisted producer work.
- Eleven dedicated fresh PostgreSQL 17 databases migrate through 0412.
One embedded-only wake-queue fixture import temporarily uses its own
external database and restores byte for byte. All 14 selected source
files match pre-run hashes. This qualifies external-database behavior
rather than reproducing embedded CI locally.
- Master advanced independently with #18126's Pi archive retry fix.
Normal merge `2a535bc61259ef99387369e8b32498f774e5ba2c` preserves both
complete binary patches. All accepted server/database/UI/test/CLI trees
stay identical; the three independent runner installer/manifest files
are retained. All 13 installer tests pass, including stalled header/body
cancellation. Combined server types/build, boundaries and full-SHA
migration order pass. A subsequent normal merge of #18128 at
`e8b78cce5bb557b086bee9ca930eeaab1d09b244` retains its three test-only
file changes with both complete patches preserved. The changed HTTPS
backfill case passes on a fresh PostgreSQL fixture (1 selected case; 178
outside the pattern), and combined types/build, boundaries and migration
order pass again. All 14 repair source hashes remain unchanged. The
independent browser fixture changes are preserved; no local browser run
is claimed.
- The earlier source `9feec170ef8174011fc3c5d3d6d76f0a0f39fea0` passed
the 4171-case full baseline before this scope repair. That scope is
preserved in the audit rather than represented as a current full rerun.
All 0412 database bytes remain unchanged: original concurrent
index/statistics/ANALYZE, prior executable history, journal entries and
retained snapshots. Five snapshots 0408 through 0412 remain. The
populated 0411-to0412 upgrade preserved 410 prior receipts and 38
companies, appended one receipt and none on replay; index/statistic
checks passed.
- The earlier P2 security failure, Greptile's former conditional
dependency on #17702, and the subsequent actual Ponytail eligibility
findings are recorded in
`doc/plans/2026-10-25-native-response-presentation-sleep-audit.md`. This
PR now includes the issue-scoped pending-effect notification fix itself,
so it is independently correct before #17702. Fresh published-head CI,
Greptile 5/5 and a clean security scan remain required. No
deployed-sleep, external-wake or fleet-savings claim is made.


## Risks

- The writer contract is process-local. Independent SQL writers and
other replicas are not covered by these notifications.
- Unpresented backlog reconstruction still has memory and time cost
proportional to candidate count. Pages contain at most 100 rows;
execution uses the shared fair batch. Selected history is excluded.
Health-only signals do not trigger reconstruction.
- Publication policy remains with its current owner. Digest, company/run
binding, source authorization, selected-answer suppression, and
ownership fences remain required.
- The rest of native recovery and governed-review presentation still run
separately. This PR does not relax Cloud sleep eligibility or add
external wake coverage.
- The stack parent has merged and its actual ancestry is reconciled.
This publication still needs fresh current-head CI and review.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell
execution, and PostgreSQL tests. The runtime does not expose a more
specific model identifier or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 02:31:20 -05:00

401 lines
15 KiB
TypeScript

import { createHash } from "node:crypto";
import { agentWakeupRequests, chatActions, chatDeliveries, chatEndpoints, type Db } from "@paperclipai/db";
import { and, eq, inArray, or, sql, type SQLWrapper } from "drizzle-orm";
import { HttpError } from "../errors.js";
export interface CommittedChatResponseAuthorizationInput {
companyId: string;
issueId: string;
agentId: string;
runId: string;
resultId: string;
}
type CommittedChatResponseAuthority = (
tx: Db,
input: CommittedChatResponseAuthorizationInput,
) => Promise<void>;
const committedChatResponseAuthorities = new WeakMap<
Db,
CommittedChatResponseAuthority
>();
const committedChatResponseAuthorityListeners = new WeakMap<Db, Set<() => void>>();
export function onCommittedChatResponseAuthorityRegistered(db: Db, listener: () => void): () => void {
let listeners = committedChatResponseAuthorityListeners.get(db);
if (!listeners) {
listeners = new Set();
committedChatResponseAuthorityListeners.set(db, listeners);
}
listeners.add(listener);
return () => {
listeners.delete(listener);
if (listeners.size === 0 && committedChatResponseAuthorityListeners.get(db) === listeners) committedChatResponseAuthorityListeners.delete(db);
};
}
export class CommittedChatResponseAuthorizationError extends HttpError {
constructor() {
super(
409,
"The accepted chat response is no longer authorized for this conversation.",
{
code: "chat_committed_response_not_authorized",
},
);
this.name = "CommittedChatResponseAuthorizationError";
}
}
/** Separate, server-only authority for presenting an already accepted result.
* This grants no permission to execute or repair the provider session. */
export function registerCommittedChatResponseAuthority(
db: Db,
authority: CommittedChatResponseAuthority,
): () => void {
committedChatResponseAuthorities.set(db, authority);
for (const listener of committedChatResponseAuthorityListeners.get(db) ?? []) listener();
return () => {
if (committedChatResponseAuthorities.get(db) === authority) {
committedChatResponseAuthorities.delete(db);
}
};
}
export async function authorizeCommittedChatResponse(
db: Db,
tx: Db,
input: CommittedChatResponseAuthorizationInput,
): Promise<void> {
const authority = committedChatResponseAuthorities.get(db);
if (!authority) throw new CommittedChatResponseAuthorizationError();
await authority(tx, input);
}
export interface FailedChatRunRetryAuthorizationInput {
phase: "admission" | "promotion" | "execution";
wakeupRequestId: string | null;
companyId: string;
agentId: string;
issueId: string | null;
runId?: string;
contextSnapshot: Record<string, unknown>;
}
type FailedChatRunRetryAuthority = (
tx: Db,
input: FailedChatRunRetryAuthorizationInput & {
wakeupRequestId: string;
issueId: string;
},
) => Promise<void>;
const failedChatRunRetryAuthorities = new WeakMap<
Db,
FailedChatRunRetryAuthority
>();
export class FailedChatRunRetryAuthorizationError extends HttpError {
constructor() {
super(
409,
"The exact failed chat request is no longer authorized. Send a new request in the current connected conversation.",
{
code: "chat_failed_run_retry_not_authorized",
},
);
this.name = "FailedChatRunRetryAuthorizationError";
}
}
/** The live chat service supplies the authority; serialized wake hints never do. */
export function registerFailedChatRunRetryAuthority(
db: Db,
authority: FailedChatRunRetryAuthority,
): () => void {
failedChatRunRetryAuthorities.set(db, authority);
return () => {
if (failedChatRunRetryAuthorities.get(db) === authority) {
failedChatRunRetryAuthorities.delete(db);
}
};
}
/** Re-discover retry provenance from the durable action, including after restart
* or when an untrusted caller strips the serialized retry selector. */
export async function authorizeFailedChatRunRetryWake(
db: Db,
tx: Db,
input: FailedChatRunRetryAuthorizationInput,
): Promise<boolean> {
const action = input.wakeupRequestId
? await tx
.select({ companyId: chatActions.companyId, kind: chatActions.kind, payload: chatActions.payload })
.from(chatActions)
.where(eq(chatActions.id, input.wakeupRequestId))
.limit(1)
.then((rows) => rows[0] ?? null)
: null;
if (action?.kind !== "failed_run_retry") {
if (Object.hasOwn(input.contextSnapshot, "chatFailedRunRetry")) {
throw new FailedChatRunRetryAuthorizationError();
}
return false;
}
const authority = failedChatRunRetryAuthorities.get(db);
const selector = input.contextSnapshot.chatFailedRunRetry;
if (
!authority ||
action.companyId !== input.companyId ||
!input.issueId ||
!input.wakeupRequestId ||
!selector ||
typeof selector !== "object" ||
Array.isArray(selector) ||
(selector as Record<string, unknown>).version !== 1 ||
(selector as Record<string, unknown>).actionId !== input.wakeupRequestId ||
typeof (selector as Record<string, unknown>).failedRunId !== "string" ||
!(selector as Record<string, unknown>).failedRunId ||
(selector as Record<string, unknown>).failedRunId !==
input.contextSnapshot.retryOfRunId ||
(input.contextSnapshot.forceFreshSession === true &&
(typeof action.payload.taskRetryIntentId !== "string" ||
input.contextSnapshot.taskRetryRequestId !== action.payload.taskRetryIntentId)) ||
["resumeFromRunId", "resumeSessionParams", "resumeSessionDisplayId"].some(
(key) => Object.hasOwn(input.contextSnapshot, key),
)
) {
throw new FailedChatRunRetryAuthorizationError();
}
await authority(tx, {
...input,
issueId: input.issueId,
wakeupRequestId: input.wakeupRequestId,
});
return true;
}
/** Prevent automatic recovery from treating committed-but-unadmitted input as
* an ordinary stranded task. A receipt is historical admission evidence even
* if its run subsequently fails or is cancelled; skipped was never admitted. */
export function unadmittedChatWakeupCondition(
issueId: SQLWrapper,
companyId: SQLWrapper,
) {
return sql`exists (
select 1 from chat_actions chat_unadmitted
where chat_unadmitted.company_id = ${companyId}
and chat_unadmitted.payload->>'issueId' = ${issueId}::text
and chat_unadmitted.kind = 'inbound_wakeup'
and chat_unadmitted.status in ('preparing', 'issued', 'processing', 'failed')
and not exists (
select 1 from agent_wakeup_requests chat_admission
where chat_admission.id = chat_unadmitted.id
and chat_admission.company_id = chat_unadmitted.company_id
and chat_admission.agent_id::text = chat_unadmitted.payload->>'agentId'
and chat_admission.status <> 'skipped'
)
and not exists (
select 1 from chat_actions chat_retry
join agent_wakeup_requests chat_retry_admission on chat_retry_admission.id = chat_retry.id
where chat_retry.company_id = chat_unadmitted.company_id
and chat_retry.endpoint_id = chat_unadmitted.endpoint_id
and chat_retry.conversation_id = chat_unadmitted.conversation_id
and chat_retry.kind = 'failed_run_retry'
and chat_retry.payload->'taskRetryInput'->'skippedActionIds' ? chat_unadmitted.id::text
and chat_retry.payload->>'issueId' = chat_unadmitted.payload->>'issueId'
and chat_retry_admission.company_id = chat_retry.company_id
and chat_retry_admission.agent_id::text = chat_retry.payload->>'agentId'
and chat_retry_admission.status <> 'skipped'
)
and (chat_unadmitted.status <> 'failed' or not exists (
select 1 from chat_actions chat_later
join agent_wakeup_requests chat_later_admission on chat_later_admission.id = chat_later.id
where chat_later.company_id = chat_unadmitted.company_id
and chat_later.endpoint_id = chat_unadmitted.endpoint_id
and chat_later.conversation_id = chat_unadmitted.conversation_id
and chat_later.payload->>'sessionGeneration' = chat_unadmitted.payload->>'sessionGeneration'
and chat_later.payload->>'issueId' = chat_unadmitted.payload->>'issueId'
and chat_later.kind = 'inbound_wakeup'
and (chat_later.created_at, chat_later.id) > (chat_unadmitted.created_at, chat_unadmitted.id)
and chat_later_admission.company_id = chat_later.company_id
and chat_later_admission.agent_id::text = chat_later.payload->>'agentId'
and chat_later_admission.status <> 'skipped'
))
)`;
}
export interface DeferredChatWakeScope {
companyId: string;
agentId: string;
issueId: string;
requestId: string;
}
const deferredChatWakeOwners = new WeakMap<Db, (input: DeferredChatWakeScope) => Promise<void>>();
/** The live inbound owner reauthorizes the original source. Cleanup owns no chat authority. */
export function registerDeferredChatWakeOwner(db: Db, owner: (input: DeferredChatWakeScope) => Promise<void>) {
deferredChatWakeOwners.set(db, owner);
return () => { if (deferredChatWakeOwners.get(db) === owner) deferredChatWakeOwners.delete(db); };
}
export async function resumeDeferredChatWake(db: Db, input: DeferredChatWakeScope) {
await deferredChatWakeOwners.get(db)?.(input);
}
/** Internal authority: this object cannot be supplied through a JSON API. */
export interface DurableChatWakeupRequest {
readonly id: string;
readonly companyId: string;
readonly agentId: string;
readonly issueId: string;
readonly commentId: string;
readonly requestedByActorType: "user" | "system";
readonly requestedByActorId: string;
readonly requestedAt: Date;
readonly idempotencyKey: string;
readonly failedRunRetry?: { readonly failedRunId: string };
/** Only the current inbound owner may reconsider its already saved, unconsumed receipt. */
readonly resumeDeferred?: true;
readonly authorize: (tx: Db) => Promise<void>;
readonly onAdmitted?: (tx: Db, receipt: { runId: string | null }) => Promise<void>;
}
const trustedRequests = new WeakSet<object>();
export function createDurableChatWakeupRequest(
input: Omit<DurableChatWakeupRequest, "idempotencyKey">,
): DurableChatWakeupRequest {
const idempotencyKey = `chat-inbound:${createHash("sha256")
.update(
JSON.stringify([
input.id,
input.companyId,
input.agentId,
input.issueId,
input.commentId,
input.requestedByActorType,
input.requestedByActorId,
]),
)
.digest("hex")}`;
const request = Object.freeze({
...input,
...(input.failedRunRetry
? { failedRunRetry: Object.freeze({ ...input.failedRunRetry }) }
: {}),
idempotencyKey,
});
trustedRequests.add(request);
return request;
}
export function assertDurableChatWakeupRequest(
request: DurableChatWakeupRequest,
scope: {
agentId: string;
companyId: string;
issueId: string | null;
commentId: string | null;
requestedByActorType?: string;
requestedByActorId?: string | null;
},
) {
if (
!trustedRequests.has(request) ||
!scope.issueId ||
request.agentId !== scope.agentId ||
request.companyId !== scope.companyId ||
request.issueId !== scope.issueId ||
request.commentId !== scope.commentId ||
request.requestedByActorType !== scope.requestedByActorType ||
request.requestedByActorId !== scope.requestedByActorId
) {
throw new Error("chat_inbound_wakeup_binding_denied");
}
}
export function assertDurableChatWakeupReceipt(
request: DurableChatWakeupRequest,
receipt: {
id: string;
companyId: string;
agentId: string;
idempotencyKey: string | null;
requestedByActorType: string | null;
requestedByActorId: string | null;
},
) {
if (
receipt.id !== request.id ||
receipt.companyId !== request.companyId ||
receipt.agentId !== request.agentId ||
receipt.idempotencyKey !== request.idempotencyKey ||
receipt.requestedByActorType !== request.requestedByActorType ||
receipt.requestedByActorId !== request.requestedByActorId
) {
throw new Error("chat_inbound_wakeup_receipt_conflict");
}
}
export function withDurableChatAdmission<T extends { kind: string; admitted?: boolean }>(request: DurableChatWakeupRequest | undefined,
admit: (tx: Parameters<Parameters<Db["transaction"]>[0]>[0]) => Promise<T>) {
return async (tx: Parameters<Parameters<Db["transaction"]>[0]>[0]) => {
const outcome = await admit(tx);
// An existing deferred receipt is not proof this transaction passed its
// gates. Only dispatch or an explicitly persisted wait admits the input.
if (request?.onAdmitted && (outcome.admitted === true || ["queued", "durable", "coalesced"].includes(outcome.kind))) {
const [receipt] = await tx.select().from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.companyId, request.companyId), eq(agentWakeupRequests.id, request.id),
inArray(agentWakeupRequests.status, ["queued", "claimed", "deferred_issue_execution", "coalesced"]),
));
if (receipt) {
assertDurableChatWakeupReceipt(request, receipt);
await request.onAdmitted(tx as unknown as Db, { runId: receipt.runId });
}
}
return outcome;
};
}
/** Work-start policy only: ongoing tool and publication authorization must not
* revoke an already-started run when the operator enables mention gating. */
export async function slackMentionAllowsRunStart(
db: Db,
run: { id: string; companyId: string; agentId: string; wakeupRequestId: string | null },
): Promise<boolean> {
if (!run.wakeupRequestId) return true;
const inputs = await db.select({ event: chatDeliveries.normalizedEvent })
.from(agentWakeupRequests)
.innerJoin(chatActions, and(
eq(chatActions.id, agentWakeupRequests.id),
eq(chatActions.companyId, run.companyId),
eq(chatActions.kind, "inbound_wakeup"),
))
.innerJoin(chatEndpoints, and(
eq(chatEndpoints.id, chatActions.endpointId),
eq(chatEndpoints.companyId, run.companyId),
eq(chatEndpoints.provider, "slack"),
eq(chatEndpoints.requireAtMention, true),
))
.innerJoin(chatDeliveries, and(
eq(chatDeliveries.id, chatActions.deliveryId),
eq(chatDeliveries.endpointId, chatEndpoints.id),
eq(chatDeliveries.companyId, run.companyId),
))
.where(and(
eq(agentWakeupRequests.companyId, run.companyId),
eq(agentWakeupRequests.agentId, run.agentId),
or(
eq(agentWakeupRequests.runId, run.id),
eq(agentWakeupRequests.id, run.wakeupRequestId),
sql`${agentWakeupRequests.payload}->>'coalescedIntoWakeupRequestId' = ${run.wakeupRequestId}`,
),
));
return inputs.every(({ event }) =>
(event.conversation as { isDirectMessage?: boolean } | undefined)?.isDirectMessage === true ||
event.trigger === "mention" ||
(event.message as { mentionedBot?: boolean } | undefined)?.mentionedBot === true);
}