Files
PaperClipAI/server/src/services/chat-channels/publications/resolution.ts
T
DottaandPaperclip 1ed0af4186 fix(server): wake native chat progress from committed changes (#17639)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Chat users receive coarse progress updates while native runs
execute.
> - A recurring chat pass scans saved run events even when no progress
changes.
> - This traffic prevents an idle database from staying idle.
> - Events, lineage changes, and publication state already identify when
projection can change.
> - This PR gives native progress its own commit-driven worker and
schedules only unfinished work.
> - Idle progress stops scanning while restart and recovery preserve
delivery.

## Linked Issues or Issue Description

Refs #17606. This is slice 29 of the recurring-server-work stack, based
on `codex/sleep-28-chat-run-milestones`. It owns native chat progress
only. Related historical optimization: #16845. The parent and this PR
must land in stack order.

**What happened?**

Native chat progress is still called by the one-second chat
reconciliation loop. It repeatedly selects run events with no new work.
Its cursor also truncates PostgreSQL timestamp precision, which can
repeat a rejected full page forever.

**Expected behavior**

Committed events and eligibility changes should wake the projection
owner. Existing rows should reconstruct work after restart. A settled
worker should have no recurring database activity. Full batches, lock
contention, and errors must retain recovery.

**Steps to reproduce**

Start an otherwise idle instance with chat history. Observe the native
progress lane in the recurring chat coordinator. For the cursor defect,
create a full page of suppressed candidates sharing a microsecond
timestamp and one eligible later candidate. The old cursor fails to
advance.

## What Changed

- Extract native progress from lifecycle milestones into its owning
service. Keep the public combined API and budget compatible.
- Register a scoped worker with the existing delivery coordinator and
scheduler. Remove native progress from the recurring chat coordinator.
- Notify before allowlisted event writes in both persistence owners.
Preserve outer commit, rollback, duplicate detection, and the ordinary
non-progress log fast path.
- Extend the existing lineage/authority notification boundary once.
Failed-publication replay and operator retry carry their issue scope.
First working publications signal progress so commit ordering cannot
strand an already-inspected event.
- Limit notification payloads to company/task IDs so wider caller inputs
never serialize database handles or private fields. Load task-binding
fields only for progress event kinds; other native/recovery events
retain their narrow lock query.
- Preserve current safe phrases, cadence, phase deduplication, provider
policy, authorization, and final-output precedence.
- Retry concrete issue/run lock contention. Disarm deleted or terminal
candidates. Preserve scoped dirty work across batches, failures, drain
and shutdown.
- Preserve PostgreSQL timestamp precision in pagination. Add a bounded
regression for the former infinite rejected-page scan.
- Document the producer audit, lifecycle contract, failures, and
remaining limits in
`doc/plans/2026-10-25-native-chat-progress-sleep-audit.md`.

## Verification

Published head: `b7ee820e49fd06c780b36b79d9b1c0883557fb73`, targeting
master after #17606 merged at
`5f0da562dbfdf3229f3cd4bb3dc742ab541986b4`. A normal ancestry merge
preserves the complete 18-path native-progress patch byte-for-byte
relative to its verified parent, with all six unrelated current master
paths intact. No new production change was introduced during final
reconciliation.

- **163 distinct selected cases pass on the verified unchanged patch:**
137 across eight complete suites (native progress, atomic event append,
all five milestone worker suites, and app admission wiring), plus 26
real chat integration cases selected by `native progress`. The other
1,595 chat integration cases were skipped, not passed.
- The full worker suites cover restart, committed events, trace
reprojection, duplicate/no-progress events, first-working publication
ordering, concrete lock contention, issue scope, backlog, drain/resume,
failures and shutdown. Settled history leaves no recurring SQL. The real
chat cases verify operator retry and replay, microsecond pagination,
authority, cadence, phase deduplication, final/question precedence and
redaction.
- The sole earlier parent conflict preserves both this slice's
issue-scoped retry wake and the exact voice conversation scope. The
current parent's milestone producers, onboarding completion guard,
identity query/read-scope coverage, migration 0397 and voice
cleanup/retention repairs remain present.
- Server production typechecking, build and module-boundary checks pass.
The selected server tests live under the excluded `src/__tests__` path
and were executed separately. Each database owner uses a new isolated
app-migrated PostgreSQL fixture.
- [Current-parent acceptance and failure
ledger](https://github.com/paperclipai/paperclip/blob/b7ee820e49fd06c780b36b79d9b1c0883557fb73/doc/plans/2026-10-25-native-chat-progress-sleep-audit.md#current-milestone-parent-acceptance-october-28)
records current counts and scope. Older combined checks and failures
retain their original revision scope and are not added to 163. No new
table, dependency, workflow, migration, or scheduler contract is
introduced.
- **Exact-head Greptile is 5/5 and all 55 current checks are clean.** No
Ponytail review is present, which is an explicitly authorized
nonblocker; no failing verdict is waived. Older head/parent reviews are
not current passes. Actual failing Ponytail findings must be fixed.
Missing/stopped/paused turns without a verdict are authorized
nonblockers; active unfinished reviews retain the original per-head
timer with a 25-minute monitor cutoff respecting the 30-minute maximum.
- No complete local repository matrix, live provider tenant, external
wake, physical host/database suspension or savings claim is made.
Following stack duties remain separate.

## Risks

- Signals are process-local under the existing single active writer
contract. Independent SQL writers or another active process need an
explicit wake integration.
- Allowlisted direct log writes now use a transaction. Native bound runs
register transaction intent; unrelated direct event types retain their
atomic-statement path.
- The shared eligibility boundary wakes both lifecycle and native
projection. It does not decide provider policy or copy event payloads.
- The existing 24-hour selection window, event-time cadence and
authority checks remain. A suppressed event is not retried just because
wall-clock time passes.
- This establishes worker quiet, not whole-instance suspension. Cloud
alarm coverage and conservative sleep gates do not change. Other chat
maintenance remains on its existing cadence.
- This actual-master reconciliation still requires fresh exact-head
review and CI before protected merge. Subsequent slices retain their own
gates.

## Model Used

OpenAI Codex, GPT-6 family, with repository inspection, code execution
and browser-test tools. The exact serving variant and context-window
size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 11:22:29 -05:00

632 lines
24 KiB
TypeScript

import { notifyVoicePublication } from "../../voice/voice-reply-notifications.js";
import { providerEffectPayload } from "../provider-effects.js";
import { notifyChatDeliveryWork, notifyChatPublicationWork, notifyChatActionWork } from "../../chat-work-notifications.js";
import { and, eq, inArray } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import { chatActions, chatConversations, chatDeliveries, chatEndpoints, chatPublications, chatTeamsFileTransfers, issueThreadInteractions } from "@paperclipai/db";
import { conflict, notFound } from "../../../errors.js";
import { logActivity } from "../../activity-log.js";
import type { EndpointRecord } from "../runtime-types.js";
import type { chatCredentialMutationLease } from "../../chat-credential-mutation-lease.js";
import type { createTeamsFileDelivery } from "../microsoft-teams/file-delivery.js";
import type { createChatPublicationTaskControl } from "./task-control.js";
import type { createChatPublicationQueue } from "./queue.js";
import type { createProviderEffects } from "../provider-effects.js";
import { resolveTeamsFilePublication } from "../microsoft-teams/file-resolution.js";
import { recordPhotonPublicationRetry } from "../imessage-photon/publication-retry.js";
import { resolveSlackFileReceipt } from "../slack/file-receipt-resolution.js";
type ResolutionOptions = {
endpointRecord: (endpointId: string) => Promise<EndpointRecord | null>;
withCredentialMutationLease: ReturnType<typeof chatCredentialMutationLease>;
teamsTransferProtocol: ReturnType<typeof createTeamsFileDelivery>["teamsTransferProtocol"];
commitTaskControlCompletion: ReturnType<typeof createChatPublicationTaskControl>["commitTaskControlCompletion"];
processPendingPublications: ReturnType<typeof createChatPublicationQueue>["processPendingPublications"];
processProviderEffect: ReturnType<typeof createProviderEffects>["processProviderEffect"];
};
export function createChatPublicationResolution(db: Db, options: ResolutionOptions) {
const { endpointRecord, withCredentialMutationLease, teamsTransferProtocol, commitTaskControlCompletion, processPendingPublications, processProviderEffect } = options;
async function replayPublication(endpointId: string, publicationId: string) {
const publication = await db
.select()
.from(chatPublications)
.where(
and(
eq(chatPublications.endpointId, endpointId),
eq(chatPublications.id, publicationId),
),
)
.then((rows) => rows[0] ?? null);
if (!publication) throw notFound("Publication not found");
if (publication.state !== "failed") {
throw conflict("Only a failed publication can be replayed", {
code:
publication.state === "delivery_unknown"
? "chat_publication_resolution_required"
: "chat_publication_not_replayable",
});
}
const interactionId = publication.payload.interactionId;
const isInteractionPrompt =
Boolean(interactionId) &&
publication.idempotencyKey ===
`interaction:${interactionId}:${publication.endpointId}`;
if (isInteractionPrompt) {
if (!publication.issueId || !interactionId) {
throw conflict(
"This interaction prompt is missing its authoritative task binding and cannot be replayed safely",
{ code: "chat_terminal_interaction_publication_not_replayable" },
);
}
const interaction = await db
.select({ status: issueThreadInteractions.status })
.from(issueThreadInteractions)
.where(
and(
eq(issueThreadInteractions.id, interactionId),
eq(issueThreadInteractions.companyId, publication.companyId),
eq(issueThreadInteractions.issueId, publication.issueId),
),
)
.then((rows) => rows[0] ?? null);
if (!interaction || interaction.status !== "pending") {
throw conflict(
"A resolved interaction prompt cannot be replayed into the provider",
{ code: "chat_terminal_interaction_publication_not_replayable" },
);
}
}
const claimed = await db.transaction(async (tx) => {
const current = await tx
.select()
.from(chatPublications)
.where(
and(
eq(chatPublications.companyId, publication.companyId),
eq(chatPublications.endpointId, endpointId),
eq(chatPublications.id, publication.id),
),
)
.for("update")
.then((rows) => rows[0]);
if (
!current ||
current.state !== publication.state ||
current.attempts !== publication.attempts
)
return [];
// Lock the publication before checking the transfer lane: issue() takes
// the same lock, so a concurrent consent/upload cannot slip into replay.
const transfer = await tx
.select({ id: chatTeamsFileTransfers.id })
.from(chatTeamsFileTransfers)
.where(
and(
eq(chatTeamsFileTransfers.companyId, current.companyId),
eq(chatTeamsFileTransfers.publicationId, current.id),
),
)
.then((rows) => rows[0]);
if (transfer)
throw conflict(
"File delivery requires a stage-specific operator resolution",
{
code: "chat_file_transfer_resolution_required",
},
);
await notifyChatPublicationWork(tx, current);
return tx
.update(chatPublications)
.set({
state: "retry",
nextAttemptAt: new Date(),
redactedError: null,
updatedAt: new Date(),
})
.where(
and(
eq(chatPublications.id, publication.id),
eq(chatPublications.state, publication.state),
eq(chatPublications.attempts, publication.attempts),
),
)
.returning({ id: chatPublications.id });
});
if (!claimed.length) {
throw conflict("This publication is already being replayed", {
code: "chat_publication_replay_conflict",
});
}
await processPendingPublications(1);
}
async function resolvePublication(
endpointId: string,
publicationId: string,
action: "mark_delivered" | "retry_anyway" | "cancel",
userId: string,
fileTransfer?: {
phase: import("@paperclipai/shared").ChatFileTransferPhase;
version: number;
},
) {
const initialRecord = await endpointRecord(endpointId);
if (!initialRecord) throw notFound("Chat endpoint not found");
await withCredentialMutationLease(
initialRecord.endpoint,
(credentialLease) =>
db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
const publication = await tx
.select()
.from(chatPublications)
.where(
and(
eq(chatPublications.endpointId, endpointId),
eq(chatPublications.id, publicationId),
),
)
.for("update")
.then((rows) => rows[0] ?? null);
if (!publication) throw notFound("Publication not found");
if (await resolveTeamsFilePublication(tx, { endpointId, publication, action, userId, credentialLease, fileTransfer }, teamsTransferProtocol)) return;
if (publication.state !== "delivery_unknown") {
throw conflict(
"Only an unconfirmed publication needs an operator resolution",
{
code: "chat_publication_resolution_not_required",
},
);
}
const interactionId = publication.payload.interactionId;
const isInteractionPrompt =
Boolean(interactionId) &&
publication.idempotencyKey ===
`interaction:${interactionId}:${publication.endpointId}`;
if (action === "retry_anyway" && isInteractionPrompt) {
if (!publication.issueId || !interactionId) {
throw conflict(
"This interaction prompt is missing its authoritative task binding and cannot be retried safely",
{
code: "chat_terminal_interaction_publication_not_replayable",
},
);
}
const interaction = await tx
.select({ status: issueThreadInteractions.status })
.from(issueThreadInteractions)
.where(
and(
eq(issueThreadInteractions.id, interactionId),
eq(issueThreadInteractions.companyId, publication.companyId),
eq(issueThreadInteractions.issueId, publication.issueId),
),
)
.then((rows) => rows[0] ?? null);
if (!interaction || interaction.status !== "pending") {
throw conflict(
"A resolved interaction prompt cannot be retried with stale provider actions; mark it delivered or cancel it instead",
{
code: "chat_terminal_interaction_publication_not_replayable",
},
);
}
}
if (
initialRecord.endpoint.provider === "imessage-photon" &&
action === "retry_anyway"
) { await recordPhotonPublicationRetry(tx, endpointId, publication, userId); }
const now = new Date();
await notifyChatPublicationWork(tx, publication);
await notifyVoicePublication(tx, initialRecord.endpoint, publication.conversationId);
await tx
.update(chatPublications)
.set(
action === "mark_delivered"
? {
state: "published",
publishedAt: now,
nextAttemptAt: null,
redactedError: null,
updatedAt: now,
}
: action === "retry_anyway"
? {
state: "retry",
nextAttemptAt: now,
redactedError: null,
updatedAt: now,
}
: {
state: "cancelled",
nextAttemptAt: null,
redactedError:
"Cancelled by an operator after an unconfirmed provider delivery",
updatedAt: now,
},
)
.where(
and(
eq(chatPublications.id, publication.id),
eq(chatPublications.state, "delivery_unknown"),
),
);
await resolveSlackFileReceipt(tx, publication, action, now);
if (publication.idempotencyKey.startsWith("control:")) {
await tx
.update(chatActions)
.set({
status:
action === "mark_delivered"
? "processed"
: action === "retry_anyway"
? "issued"
: "cancelled",
result: {
code:
action === "mark_delivered"
? "task_control_marked_delivered_by_operator"
: action === "retry_anyway"
? "task_control_retry_requested"
: "task_control_cancelled_by_operator",
},
updatedAt: now,
})
.where(
and(
eq(chatActions.endpointId, publication.endpointId),
eq(
chatActions.providerActionId,
`task-control-authorization:${publication.id}`,
),
inArray(chatActions.status, [
"issued",
"processing",
"delivery_unknown",
]),
),
);
}
if (action === "mark_delivered") {
await tx
.update(chatEndpoints)
.set({ lastPublicationAt: now, updatedAt: now })
.where(eq(chatEndpoints.id, endpointId));
await commitTaskControlCompletion(
tx as unknown as Db,
publication,
now,
);
}
await logActivity(tx as unknown as Db, {
companyId: publication.companyId,
actorType: "user",
actorId: userId,
action: `chat.publication_${action}`,
entityType: "chat_publication",
entityId: publication.id,
issueId: publication.issueId,
details: {
endpointId,
conversationId: publication.conversationId,
previousState: "delivery_unknown",
nextState:
action === "mark_delivered"
? "published"
: action === "retry_anyway"
? "retry"
: "cancelled",
duplicateRiskAccepted: action === "retry_anyway",
},
});
await credentialLease.assertOwned(tx);
}),
);
await processPendingPublications();
}
async function resolveProviderEffect(
endpointId: string,
actionId: string,
resolution: "mark_delivered" | "retry_anyway" | "cancel",
userId: string,
) {
const initialRecord = await endpointRecord(endpointId);
if (!initialRecord) throw notFound("Chat endpoint not found");
if (resolution === "retry_anyway") {
await withCredentialMutationLease(
initialRecord.endpoint,
async (credentialLease) => {
const currentRecord = await endpointRecord(endpointId);
if (
!currentRecord ||
!["verifying", "active"].includes(currentRecord.endpoint.status)
) {
throw conflict(
"The chat connection must be active before retrying this provider reply",
{
code: "chat_action_endpoint_unavailable",
},
);
}
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
const action = await tx
.select()
.from(chatActions)
.where(
and(
eq(chatActions.endpointId, endpointId),
eq(chatActions.id, actionId),
eq(chatActions.kind, "provider_effect"),
),
)
.for("update")
.then((rows) => rows[0] ?? null);
if (!action) throw notFound("Provider action not found");
if (action.status !== "delivery_unknown") {
throw conflict(
"Only an unconfirmed provider action needs operator resolution",
{
code: "chat_action_resolution_not_required",
},
);
}
const payload = providerEffectPayload(action.payload);
if (!payload) {
throw conflict("This provider action cannot be retried safely", {
code: "chat_action_retry_context_missing",
});
}
const delivery =
payload.settleDelivery && action.deliveryId
? await tx
.select({ state: chatDeliveries.state })
.from(chatDeliveries)
.where(
and(
eq(chatDeliveries.endpointId, endpointId),
eq(chatDeliveries.id, action.deliveryId),
),
)
.for("update")
.then((rows) => rows[0] ?? null)
: null;
if (payload.settleDelivery && action.deliveryId) {
if (!delivery || delivery.state !== "failed") {
throw conflict(
"The provider action's delivery is no longer awaiting resolution",
{
code: "chat_action_delivery_resolution_conflict",
},
);
}
await notifyChatDeliveryWork(tx);
await tx
.update(chatDeliveries)
.set({
state: "processing",
nextAttemptAt: null,
redactedError: null,
updatedAt: new Date(),
})
.where(
and(
eq(chatDeliveries.id, action.deliveryId),
eq(chatDeliveries.state, "failed"),
),
);
}
const now = new Date();
await notifyChatActionWork(tx, "provider_effect");
const [claimed] = await tx
.update(chatActions)
.set({
status: "received",
result: {
...(action.result ?? {}),
code: "provider_effect_retry_requested",
retryable: false,
retryAt: null,
},
updatedAt: now,
})
.where(
and(
eq(chatActions.id, action.id),
eq(chatActions.status, "delivery_unknown"),
),
)
.returning({ id: chatActions.id });
if (!claimed) {
throw conflict("This provider action is already being resolved", {
code: "chat_action_resolution_conflict",
});
}
await logActivity(tx as unknown as Db, {
companyId: action.companyId,
actorType: "user",
actorId: userId,
action: "chat.provider_effect_retry_anyway",
entityType: "chat_action",
entityId: action.id,
details: {
endpointId,
conversationId: action.conversationId,
deliveryId: action.deliveryId,
previousState: "delivery_unknown",
nextState: "received",
duplicateRiskAccepted: true,
},
});
});
},
);
// The audited retry request is durable before the new transport claim.
// Do not keep the endpoint mutation lease across provider I/O.
await processProviderEffect(actionId);
return;
}
await withCredentialMutationLease(
initialRecord.endpoint,
(credentialLease) =>
db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
const action = await tx
.select()
.from(chatActions)
.where(
and(
eq(chatActions.endpointId, endpointId),
eq(chatActions.id, actionId),
eq(chatActions.kind, "provider_effect"),
),
)
.for("update")
.then((rows) => rows[0] ?? null);
if (!action) throw notFound("Provider action not found");
if (action.status !== "delivery_unknown") {
throw conflict(
"Only an unconfirmed provider action needs operator resolution",
{
code: "chat_action_resolution_not_required",
},
);
}
const payload = providerEffectPayload(action.payload);
if (!payload) {
throw conflict("This provider action cannot be resolved safely", {
code: "chat_action_resolution_context_missing",
});
}
const delivery =
payload.settleDelivery && action.deliveryId
? await tx
.select({ state: chatDeliveries.state })
.from(chatDeliveries)
.where(
and(
eq(chatDeliveries.endpointId, endpointId),
eq(chatDeliveries.id, action.deliveryId),
),
)
.for("update")
.then((rows) => rows[0] ?? null)
: null;
if (
payload.settleDelivery &&
action.deliveryId &&
(!delivery || delivery.state !== "failed")
) {
throw conflict(
"The provider action's delivery is no longer awaiting resolution",
{
code: "chat_action_delivery_resolution_conflict",
},
);
}
const now = new Date();
const nextStatus =
resolution === "mark_delivered" ? "processed" : "cancelled";
await notifyChatActionWork(tx, "provider_effect");
const [resolved] = await tx
.update(chatActions)
.set({
status: nextStatus,
result: {
...(action.result ?? {}),
code:
resolution === "mark_delivered"
? "provider_effect_marked_delivered_by_operator"
: "provider_effect_cancelled_by_operator",
retryable: false,
retryAt: null,
},
updatedAt: now,
})
.where(
and(
eq(chatActions.id, action.id),
eq(chatActions.status, "delivery_unknown"),
),
)
.returning({ id: chatActions.id });
if (!resolved) {
throw conflict("This provider action is already being resolved", {
code: "chat_action_resolution_conflict",
});
}
if (payload.settleDelivery && action.deliveryId) {
await notifyChatDeliveryWork(tx);
await tx
.update(chatDeliveries)
.set(
resolution === "mark_delivered"
? {
state: "processed",
processedAt: now,
nextAttemptAt: null,
redactedError: null,
updatedAt: now,
}
: {
state: "filtered",
processedAt: now,
nextAttemptAt: null,
redactedError:
"Cancelled by an operator after an unconfirmed provider reply",
updatedAt: now,
},
)
.where(
and(
eq(chatDeliveries.id, action.deliveryId),
eq(chatDeliveries.state, "failed"),
),
);
}
if (resolution === "mark_delivered") {
if (payload.completeConversationId) {
await tx
.update(chatConversations)
.set({ state: "completed", updatedAt: now })
.where(
and(
eq(chatConversations.id, payload.completeConversationId),
eq(chatConversations.endpointId, endpointId),
),
);
}
await tx
.update(chatEndpoints)
.set({ lastEventAt: now, updatedAt: now })
.where(eq(chatEndpoints.id, endpointId));
}
await logActivity(tx as unknown as Db, {
companyId: action.companyId,
actorType: "user",
actorId: userId,
action: `chat.provider_effect_${resolution}`,
entityType: "chat_action",
entityId: action.id,
details: {
endpointId,
conversationId: action.conversationId,
deliveryId: action.deliveryId,
completeConversationId: payload.completeConversationId ?? null,
previousState: "delivery_unknown",
nextState: nextStatus,
duplicateRiskAccepted: false,
},
});
await credentialLease.assertOwned(tx);
}),
);
}
return { replayPublication, resolvePublication, resolveProviderEffect };
}