fix(server): wake chat lifecycle milestones from committed changes (#17606)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connected chats show safe notices when a run starts, waits,
finishes, or fails.
> - A recurring scan projected these notices even when no eligible work
existed.
> - Eligibility depends on committed run state, conversation lineage,
and current authority.
> - Existing database rows can reconstruct that work after restart.
> - This PR wakes a scoped worker from those writes and removes
lifecycle polling.
> - Native progress events keep their separate existing owner until the
next migration.

## Linked Issues or Issue Description

**What happened?**

The chat maintenance timer repeatedly scanned run lifecycle state. Empty
and settled instances still queried the database. A run-status event
alone cannot replace the scan: prompt receipts, continuation receipts,
identity changes, and restored authority can make a run eligible later.

**What did you expect?**

Committed changes wake projection. Startup reconstructs eligible work. A
completed pass stops until another relevant change arrives. Failed work
retains recovery without scanning settled history forever.

**Steps to reproduce**

Configure an automatic chat endpoint. Finish its work, or leave it
unused. Observe recurring lifecycle projection queries from the chat
maintenance callback.

Stacked on #17592. Related foundation: #15625. This PR owns lifecycle
milestones only.

## What Changed

- Preserve upstream coalesced Cloud membership writes together with
locked membership restoration and its committed chat notification.

- Signal milestone eligibility from the real Slack account-link
transaction.

- Register the lifecycle worker with the existing delivery coordinator.
Remove lifecycle projection from the periodic chat callback. Keep native
progress in a separately named lane and preserve the explicit combined
API.
- Reconstruct company/issue scopes from the existing selector. Drain
bounded batches with company fairness. Retry failures, retain writes
through drain/resume, and disarm settled work.
- Signal from supported run, presentation, interaction, receipt,
lineage, conversation, recovery, and authority writers. Preserve their
transaction ownership, including nested savepoints and uncertain
settlement. No new table is added.
- Preserve provider authorization, authored-final suppression,
responsible-user continuity, deduplication, the 24-hour selection
window, and raw-output restrictions. Keep provider interpretation in its
owner.
- Preserve PostgreSQL timestamp precision in lifecycle pagination.
Rejected submillisecond candidates must not repeat the same page and
starve later notices.
- Document producer coverage and alternate-writer dispositions in
`doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md`.

## Verification

Published head: `1d056f68c5b1005933b9e12de8745bf1c4cbeeca`, targeting
master after #17592 merged at
`7d75ae0569e41786640cf09647ae8849411d2f22`. A normal merge integrates
the actual landed parent. The entire 77-path milestone patch is
binary-identical to the locally verified preparation across that
ancestry reconciliation; both unrelated master paths are preserved. The
four earlier producer/test integration conflicts retain scoped milestone
and voice notices, current onboarding completion admission, and expanded
identity query/read-scope coverage.

- **330 distinct selected cases pass on the verified unchanged patch:**
305 across 14 complete suites (all five milestone worker suites, run
identity, board claim, Cloud provisioning, auth session routes, Cloud
actor authentication, stale-lock recovery, GitHub credentials, project
Git authorization, and app admission wiring), plus 25 selected real chat
producer cases. The remaining 1,590 chat integration cases were skipped,
not passed.
- The real producer cases cover saved milestone discovery, GitHub and
Slack identity changes, account-link rollback/reconfirmation, ordinary
ingress reactivation, post-close action/delivery receipt settlement, all
ten authority-restoration paths including Cloud synchronization, and
changed native review metadata.
- Worker coverage verifies outer commit, rollback, uncertain-write
recovery, restart, bounded batching/fairness, timestamp precision,
retries, drain/resume, and settled SQL quiet. Authorization and existing
publication deduplication remain with their owning services.
- Server production typechecking, build, and module-boundary checks
pass. The tests above were executed separately. Production typechecking
excludes `src/__tests__` and does not replace those runs. Each database
owner uses its own newly created and app-migrated PostgreSQL fixture.
- [Current-parent acceptance and preserved failure
ledger](https://github.com/paperclipai/paperclip/blob/1d056f68c5b1005933b9e12de8745bf1c4cbeeca/doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md#current-parent-integration-and-acceptance-october-28)
records commands, counts, producer scope, conflict dispositions, and
limits. Earlier combined-stack counts and failures keep their original
revision scope; local acceptance does not erase old CI failures.
- No new migration, queue table, dependency, workflow, or shared
scheduler contract is introduced. Parent migration 0397, bounded consent
recovery, voice cleanup/retention repairs, and runner checkpoint
restoration remain present.
- **Exact-head Greptile is 5/5 and all 55 current checks are clean.** No
Ponytail review is present, which is an explicitly authorized
nonblocker; no failing review is waived. Older reviews and parent checks
are not current passes. Actual failing Ponytail findings must be fixed;
an absent/stopped/paused turn without a verdict is an authorized
nonblocker. Active unfinished Ponytail uses its original request/start
timer, with a 25-minute monitor cutoff respecting the user's 30-minute
maximum.
- No complete local repository matrix, live provider tenant, external
alarm, physical host/database suspension, or fleet savings claim is
made. Native progress remains a separate following slice.

## Risks

- A future eligibility writer must notify the existing projection
boundary. Independent SQL writers or another active server process are
outside this process-local signal contract.
- Authority and lineage remain sensitive. The worker uses the existing
selectors and publication policy; it does not grant execution or
disclosure authority.
- Native progress and other publication maintenance still have recurring
work. This PR removes one reason, not the entire chat maintenance timer.
- Worker quiet does not prove database suspension, host sleep, or
external alarm coverage. Cloud sleep blockers remain unchanged.
- Merge eligibility requires passing current-head CI and Greptile 5/5
after this actual-master reconciliation. Subsequent stack slices retain
their own gates.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and
context-window size are not exposed. Used repository inspection,
reasoning, shell execution, PostgreSQL tests, browser E2Es and GitHub
tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details available in this session)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run the documented focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-28 11:07:54 -05:00
1 parent 143d20e466
commit 5f0da562db
77 files changed
+5002 -501

No files matched your search

+49 -24
View File
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { notifyVoiceChange } from "./voice/voice-recovery-notifications.js";
import { notifyChatEndpointWork } from "./chat-work-notifications.js";
import { notifyToolConnectionHealth } from "./tool-connection-health-notifications.js";
@@ -328,7 +329,7 @@ export function accessService(db: Db) {
&& grant.credentialSecretRefs.length > 0
&& grant.credentialSecretRefs.every((ref) => !connectionSecretIdsToRemove.has(ref.secretId)),
);
if (connection.transport === "chat_sdk") await notifyChatEndpointWork(tx, connection.id);
if (connection.transport === "chat_sdk") await notifyChatEndpointWork(tx, connection.id, companyId);
if (connection.transport === "voice") await notifyVoiceChange(tx, { kind: "connection", id: connection.id });
if (!hasUnaffectedActiveGrant) await notifyToolConnectionHealth(tx, { companyId, id: connection.id });
await tx.update(toolConnections).set({
@@ -554,6 +555,11 @@ export function accessService(db: Db) {
data.membershipRole !== undefined ? data.membershipRole : existing.membershipRole;
const nextStatus = data.status ?? existing.status;
if (existing.principalType === "user" && nextStatus === "active" && nextMembershipRole !== "viewer"
&& (existing.status !== "active" || existing.membershipRole === "viewer")) {
await notifyChatRunMilestones(tx, { companyId });
}
if (
existing.principalType === "user" &&
existing.status === "active" &&
@@ -886,6 +892,7 @@ export function accessService(db: Db) {
const existingMembership = existingByCompany.get(companyId);
if (existingMembership) {
if (existingMembership.status !== "active") {
if (existingMembership.membershipRole !== "viewer") await notifyChatRunMilestones(tx, { companyId });
await tx
.update(companyMemberships)
.set({
@@ -897,6 +904,7 @@ export function accessService(db: Db) {
}
continue;
}
await notifyChatRunMilestones(tx, { companyId });
await tx.insert(companyMemberships).values({
companyId,
principalType: "user",
@@ -917,31 +925,43 @@ export function accessService(db: Db) {
membershipRole: string | null = "member",
status: "pending" | "active" | "suspended" = "active",
) {
const existing = await getMembership(companyId, principalType, principalId);
if (existing) {
if (existing.status !== status || existing.membershipRole !== membershipRole) {
const updated = await db
.update(companyMemberships)
.set({ status, membershipRole, updatedAt: new Date() })
.where(eq(companyMemberships.id, existing.id))
.returning()
.then((rows) => rows[0] ?? null);
return updated ?? existing;
return db.transaction(async tx => {
const existing = await tx.select().from(companyMemberships).where(and(
eq(companyMemberships.companyId, companyId), eq(companyMemberships.principalType, principalType),
eq(companyMemberships.principalId, principalId),
)).for("update").then(rows => rows[0] ?? null);
if (existing) {
if (existing.status !== status || existing.membershipRole !== membershipRole) {
if (principalType === "user" && status === "active" && membershipRole !== "viewer"
&& (existing.status !== "active" || existing.membershipRole === "viewer")) {
await notifyChatRunMilestones(tx, { companyId });
}
const updated = await tx
.update(companyMemberships)
.set({ status, membershipRole, updatedAt: new Date() })
.where(eq(companyMemberships.id, existing.id))
.returning()
.then((rows) => rows[0] ?? null);
return updated ?? existing;
}
return existing;
}
return existing;
}
return db
.insert(companyMemberships)
.values({
companyId,
principalType,
principalId,
status,
membershipRole,
})
.returning()
.then((rows) => rows[0]);
if (principalType === "user" && status === "active" && membershipRole !== "viewer") {
await notifyChatRunMilestones(tx, { companyId });
}
return tx
.insert(companyMemberships)
.values({
companyId,
principalType,
principalId,
status,
membershipRole,
})
.returning()
.then((rows) => rows[0]);
});
}
async function setPrincipalGrants(
@@ -1122,6 +1142,11 @@ export function accessService(db: Db) {
data.membershipRole !== undefined ? data.membershipRole : existing.membershipRole;
const nextStatus = data.status ?? existing.status;
if (existing.principalType === "user" && nextStatus === "active" && nextMembershipRole !== "viewer"
&& (existing.status !== "active" || existing.membershipRole === "viewer")) {
await notifyChatRunMilestones(tx, { companyId });
}
if (
existing.principalType === "user" &&
existing.status === "active" &&
@@ -17,6 +17,7 @@ import {
type Db,
} from "@paperclipai/db";
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { sanitizeQuarantinedCommentForHigherTrust } from "./source-trust.js";
export type ConversationIdentity = {
@@ -166,6 +167,7 @@ export async function prepareConversationTurn(
}
// The boundary lives on the command comment. A crash/retry reuses it instead of resetting twice.
if (reset && comment && comment.conversationSessionGeneration == null) {
await notifyChatRunMilestones(tx, { companyId: issue.companyId, issueId: issue.id });
generation += 1;
await tx
.update(issues)
+5
View File
@@ -1,6 +1,7 @@
import { leasedChatProvider } from "./chat-channels/runtime/receiver-policy.js";
import { chatReceiverSchedule } from "./chat-channels/runtime/receiver-schedule.js";
import { registerChatReceiverWork } from "./chat-channels/runtime/receiver-work.js";
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { notifyChatEndpointConfiguration } from "./chat-channels/configuration-work-notifications.js";
import { registerDiscordCommandWork } from "./chat-channels/discord/command-work.js";
import { discordCommandSchedule } from "./chat-channels/discord/command-schedule.js";
@@ -2121,6 +2122,9 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
taskUserId: string | null,
) => {
let conversation = existingConversation;
if (conversation) {
await notifyChatRunMilestones(taskTx, { companyId: taskEndpoint.companyId, issueId: conversation.issueId });
}
if (!conversation) {
const [taskResource] = await taskTx.select({ executionDefaults: chatEndpointResources.executionDefaults })
.from(chatEndpointResources).where(and(eq(chatEndpointResources.id, resource.id), eq(chatEndpointResources.endpointId, taskEndpoint.id)));
@@ -2162,6 +2166,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
},
taskTx,
);
await notifyChatRunMilestones(taskTx, { companyId: taskEndpoint.companyId, issueId: issue.id });
await taskTx
.insert(chatConversations)
.values({
@@ -196,7 +196,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
code: "chat_endpoint_not_active",
});
}
await notifyChatEndpointWork(tx, endpoint.connectionId);
await notifyChatEndpointWork(tx, endpoint.connectionId, endpoint.companyId);
await tx
.update(chatEndpoints)
.set({
@@ -334,7 +334,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
inArray(chatDeliveries.state, ["received", "retry"]),
),
);
await notifyChatEndpointWork(tx, endpoint.connectionId);
await notifyChatEndpointWork(tx, endpoint.connectionId, endpoint.companyId);
[activatedEndpoint] = await tx
.update(chatEndpoints)
.set({
@@ -381,9 +381,9 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
const failure = redactError(error);
const failedAt = new Date();
await managementContext?.reauthorize();
await db.transaction(async (tx) => {
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
await notifyChatEndpointWork(tx, endpoint.connectionId);
await notifyChatEndpointWork(tx, endpoint.connectionId, endpoint.companyId);
await tx
.update(chatEndpoints)
.set({
@@ -487,7 +487,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
code: "chat_endpoint_already_removed",
});
}
await notifyChatEndpointWork(tx, endpoint.connectionId);
await notifyChatEndpointWork(tx, endpoint.connectionId, endpoint.companyId);
await tx
.update(chatEndpoints)
.set({
@@ -721,7 +721,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
await managementContext?.reauthorize();
const claimed = await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
await notifyChatEndpointWork(tx, endpoint.connectionId);
await notifyChatEndpointWork(tx, endpoint.connectionId, endpoint.companyId);
const rows = await tx
.update(chatEndpoints)
.set({
@@ -800,7 +800,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
await installSpekoVoiceTools(tx, endpoint, actorUserId);
}
const preserveSlackVerification = preservesProvisionedSlackVerification(current.setup, credentials.signingSecret ?? "", `${getWebhookPublicBaseUrl()}/api/chat-webhooks/${endpoint.publicId}/slack`);
await notifyChatEndpointWork(tx, endpoint.connectionId);
await notifyChatEndpointWork(tx, endpoint.connectionId, endpoint.companyId);
await tx
.update(chatEndpoints)
.set({
@@ -887,7 +887,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
await managementContext?.reauthorize();
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
await notifyChatEndpointWork(tx, endpoint.connectionId);
await notifyChatEndpointWork(tx, endpoint.connectionId, endpoint.companyId);
await tx
.update(chatEndpoints)
.set({
@@ -383,7 +383,7 @@ export function createChatCredentials(db: Db, options: ChatCredentialsOptions) {
.set({ credentialSecretRefs: refs, updatedAt: new Date() })
.where(eq(toolConnections.id, endpoint.connectionId));
if (changedFields.length) {
await notifyChatEndpointWork(tx, endpoint.connectionId);
await notifyChatEndpointWork(tx, endpoint.connectionId, endpoint.companyId);
const [currentEndpoint] = await tx.select({ setup: chatEndpoints.setup }).from(chatEndpoints)
.where(and(eq(chatEndpoints.id, endpoint.id), eq(chatEndpoints.companyId, endpoint.companyId))).for("update");
if (!currentEndpoint) throw notFound("Chat endpoint not found");
@@ -165,7 +165,7 @@ export function createDiscordGatewayEvents(db: Db, options: DiscordGatewayEvents
current.setup.step === "complete" ? "active" : "verifying";
await notifyChatReceiver(tx, current);
await notifyDiscordCommandRegistration(tx, { kind: "endpoint", id: current.id });
if (status !== current.status) await notifyChatEndpointWork(tx, current.connectionId);
if (status !== current.status) await notifyChatEndpointWork(tx, current.connectionId, current.companyId);
await tx
.update(chatEndpoints)
.set({
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { chatConnectionTransport } from "./provider-catalog.js";
import type { ConnectionManagementContext } from "../connection-management.js";
import { slackSetupState } from "./slack/setup.js";
@@ -454,6 +455,11 @@ function providerSetupState(
await managementContext?.reauthorize();
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
if ((input.allowDirectMessages === true && !existing.endpoint.allowDirectMessages)
|| (input.allowGroupChats === true && !existing.endpoint.allowGroupChats)
|| (input.allowUnlinkedPeople === true && !existing.endpoint.allowUnlinkedPeople)) {
await notifyChatRunMilestones(tx, { companyId: existing.endpoint.companyId });
}
if (input.executionDefaults !== undefined) {
await assertChatExecutionDefaultsAccess(db, {
companyId: existing.endpoint.companyId,
@@ -201,11 +201,12 @@ export async function reconcileGitHubRepositoryCoordinates(tx: DbTransaction, en
type GitHubResourceOptions = {
endpointRecord: (endpointId: string) => Promise<{ endpoint: EndpointRow } | null>;
upsertProviderResourceRow: (tx: DbTransaction, endpoint: EndpointRow, item: ChatProviderResourceInventoryItem) => Promise<{ id: string } | null>;
restoreResourceConversations: (tx: DbTransaction, endpoint: EndpointRow, resourceId: string) => Promise<void>;
updateResourceSelection: (endpointId: string, updates: Array<{ id: string; enabled: boolean }>, actorUserId?: string | null, options?: { allGitHubRepositoriesEnabled?: boolean }) => Promise<void>;
};
export function createGitHubResources(db: Db, options: GitHubResourceOptions) {
const { endpointRecord, upsertProviderResourceRow, updateResourceSelection } = options;
const { endpointRecord, upsertProviderResourceRow, restoreResourceConversations, updateResourceSelection } = options;
async function reconcileGitHubWebhookRepository(
endpoint: EndpointRow,
payload: unknown,
@@ -220,17 +221,7 @@ export function createGitHubResources(db: Db, options: GitHubResourceOptions) {
// A correctly signed repository callback is current provider proof. Keep
// Paperclip's enabled choice, and reopen only conversations previously
// quarantined for provider availability loss.
await tx
.update(chatConversations)
.set({ state: "active", updatedAt: new Date() })
.where(
and(
eq(chatConversations.companyId, endpoint.companyId),
eq(chatConversations.endpointId, endpoint.id),
eq(chatConversations.resourceId, resource.id),
eq(chatConversations.state, "unavailable"),
),
);
await restoreResourceConversations(tx, endpoint, resource.id);
await credentialLease.assertOwned(tx);
});
}
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { githubIdentityFields } from "./github/identity-links.js";
import { createSlackIdentityLinks, type SlackIdentityEffectStager } from "./slack/identity-links.js";
import { createHash, randomBytes } from "node:crypto";
@@ -319,6 +320,7 @@ export function createChatIdentityLinks(db: Db, options: ChatIdentityLinkOptions
},
);
}
await notifyChatRunMilestones(tx, { companyId: currentLink.companyId });
const confirmedLink = await tx
.update(chatIdentityLinks)
.set({
@@ -32,7 +32,7 @@ export function createPhotonRuntimeCallbacks(db: Db, options: PhotonRuntimeOptio
if (!current) return;
const connection = await tx.select().from(toolConnections).where(eq(toolConnections.id, current.connectionId)).then((rows) => rows[0]);
if (!connection?.enabled || connection.healthStatus === "failed") throw new Error("Photon endpoint requires operator recovery");
await notifyChatEndpointWork(tx, connectionId);
await notifyChatEndpointWork(tx, connectionId, companyId);
await tx.update(chatEndpoints).set({ status: current.setup.step === "complete" ? "active" : "verifying", healthMessage: "Photon receiver connected", lastError: null, updatedAt: new Date() }).where(eq(chatEndpoints.id, endpointId));
});
},
@@ -1,5 +1,6 @@
import { DELIVERY_PROCESSING_STALE_MS } from "./delivery-ledger.js";
import type { EndpointRow, DbOrTransaction } from "../runtime-types.js";
import { notifyChatRunMilestones } from "../../chat-run-milestone-notifications.js";
import { notifyChatDeliveryWork } from "../../chat-work-notifications.js";
import { enqueueFastResponse } from "../../fast-responses.js";
import { and, eq, sql } from "drizzle-orm";
@@ -110,6 +111,9 @@ export function createInboundWakeups(db: Db, options: WakeupOptions) {
);
}
await notifyChatDeliveryWork(tx);
if (action.status === "processed") {
await notifyChatRunMilestones(tx, { companyId: action.companyId, issueId: String(action.payload.issueId) });
}
const acceptedAt = new Date();
const accepted = await tx
.update(chatDeliveries)
@@ -249,6 +253,9 @@ export function createInboundWakeups(db: Db, options: WakeupOptions) {
const settle = async (status: string, result: Record<string, unknown>) => {
const changed = await db.transaction(async (tx) => {
await notifyChatDeliveryWork(tx);
if (status === "processed") {
await notifyChatRunMilestones(tx, { companyId: claimed.companyId, issueId: request!.issueId });
}
const changed = await tx
.update(chatActions)
.set({ status, result, updatedAt: new Date() })
@@ -787,7 +787,7 @@ export function createChatInteractionCallbacks(db: Db, options: InteractionCallb
.then((rows) => rows[0] ?? null);
if (occupied && occupied.publicationId !== current.id) return null;
const reconciledAt = new Date();
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, current.state === "delivery_unknown" ? current : undefined);
const [reconciled] = await tx
.update(chatPublications)
.set({
@@ -120,7 +120,7 @@ export function createChatPublicationFailures(db: Db, options: PublicationFailur
: disposition.kind === "resource_unavailable"
? "cancelled"
: "failed";
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, state === "failed" || state === "cancelled" ? publication : undefined);
const [ownedPublication] = await tx
.update(chatPublications)
.set({
@@ -218,7 +218,7 @@ export function createChatPublicationResolution(db: Db, options: ResolutionOptio
action === "retry_anyway"
) { await recordPhotonPublicationRetry(tx, endpointId, publication, userId); }
const now = new Date();
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, action === "retry_anyway" ? undefined : publication);
await notifyVoicePublication(tx, initialRecord.endpoint, publication.conversationId);
await tx
.update(chatPublications)
@@ -154,7 +154,7 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
? classified.retryAfterMs
: Math.min(60_000, 2 ** Math.max(0, attempts) * 1_000);
const result = await db.transaction(async (tx) => {
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, terminal ? publication : undefined);
const [settled] = await tx
.update(chatPublications)
.set({
@@ -380,7 +380,7 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
}
if (explicitSlackDelivery === "delivered") {
await db.transaction(async (tx) => {
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, publication);
return tx.update(chatPublications).set({ state: "cancelled", redactedError: "Identical response already delivered by Slack tool", updatedAt: new Date() }).where(and(eq(chatPublications.id, publication.id), inArray(chatPublications.state, ["pending", "retry"])))
;
});
@@ -507,7 +507,7 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
) {
await publicationLease.assertOwned();
await db.transaction(async (tx) => {
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, publication);
return tx
.update(chatPublications)
.set({
@@ -531,7 +531,7 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
if (!endpoint.allowDirectMessages) {
await publicationLease.assertOwned();
await db.transaction(async (tx) => {
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, publication);
return tx
.update(chatPublications)
.set({
@@ -566,7 +566,7 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
if (!nonDirectDestinationAllowed(endpoint, resource)) {
await publicationLease.assertOwned();
await db.transaction(async (tx) => {
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, publication);
return tx
.update(chatPublications)
.set({
@@ -605,7 +605,7 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
if (!currentInteraction || currentInteraction.status !== "pending") {
await publicationLease.assertOwned();
await db.transaction(async (tx) => {
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, publication);
return tx
.update(chatPublications)
.set({
@@ -638,7 +638,7 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
if (progressSupersessionReason) {
await publicationLease.assertOwned();
await db.transaction(async (tx) => {
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, publication);
return tx
.update(chatPublications)
.set({
@@ -710,7 +710,7 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
if (!authorizationClaim) {
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, publication);
await tx
.update(chatPublications)
.set({
@@ -858,7 +858,7 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
const committedAt = new Date();
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, publication);
await notifyVoicePublication(tx, authorizationClaim.endpoint, publication.conversationId);
const [completedPublication] = await tx
.update(chatPublications)
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { notifyChatEndpointConfiguration } from "./configuration-work-notifications.js";
import { notifyChatConnectionHealth } from "./connection-health-notifications.js";
import type { ConnectionManagementContext } from "../connection-management.js";
@@ -80,6 +81,32 @@ export function createChatResourceReconciliation(db: Db, options: ResourceReconc
return resource ?? null;
}
async function setResourceConversationAvailability(
tx: DbTransaction,
endpoint: Pick<EndpointRow, "id" | "companyId">,
resourceId: string,
available: boolean,
now: Date,
) {
const affected = and(
eq(chatConversations.companyId, endpoint.companyId),
eq(chatConversations.endpointId, endpoint.id),
eq(chatConversations.resourceId, resourceId),
available ? eq(chatConversations.state, "unavailable") : inArray(chatConversations.state, ["active", "waiting"]),
);
if (available) {
// Only a reopened binding creates projection work. Repeated inventory
// refreshes and completed history must not wake the milestone scanner.
const reopened = await tx.select({ issueId: chatConversations.issueId })
.from(chatConversations).where(affected).for("update");
for (const issueId of new Set(reopened.map(row => row.issueId))) {
await notifyChatRunMilestones(tx, { companyId: endpoint.companyId, issueId });
}
}
await tx.update(chatConversations)
.set({ state: available ? "active" : "unavailable", updatedAt: now }).where(affected);
}
async function reconcileProviderResourceRows(
endpoint: EndpointRow,
inventory: ChatProviderInventoryResult,
@@ -99,17 +126,7 @@ export function createChatResourceReconciliation(db: Db, options: ResourceReconc
if (resource) {
// Resource recovery reopens only bindings that Paperclip marked
// unavailable. Completed historical tasks stay completed.
await tx
.update(chatConversations)
.set({ state: "active", updatedAt: new Date() })
.where(
and(
eq(chatConversations.companyId, endpoint.companyId),
eq(chatConversations.endpointId, endpoint.id),
eq(chatConversations.resourceId, resource.id),
eq(chatConversations.state, "unavailable"),
),
);
await setResourceConversationAvailability(tx, endpoint, resource.id, true, new Date());
}
}
@@ -132,17 +149,7 @@ export function createChatResourceReconciliation(db: Db, options: ResourceReconc
.update(chatEndpointResources)
.set({ availability: absentAvailability, updatedAt: new Date() })
.where(eq(chatEndpointResources.id, resource.id));
await tx
.update(chatConversations)
.set({ state: "unavailable", updatedAt: new Date() })
.where(
and(
eq(chatConversations.companyId, endpoint.companyId),
eq(chatConversations.endpointId, endpoint.id),
eq(chatConversations.resourceId, resource.id),
inArray(chatConversations.state, ["active", "waiting"]),
),
);
await setResourceConversationAvailability(tx, endpoint, resource.id, false, new Date());
}
await credentialLease.assertOwned(tx);
});
@@ -279,6 +286,9 @@ export function createChatResourceReconciliation(db: Db, options: ResourceReconc
before: { enabled: row.enabled, ...(JSON.stringify(row.executionDefaults) !== JSON.stringify(finalDefaults.get(row.id)) ? { executionDefaults: row.executionDefaults } : {}) },
after: { enabled: finalEnabled.get(row.id)!, ...(JSON.stringify(row.executionDefaults) !== JSON.stringify(finalDefaults.get(row.id)) ? { executionDefaults: finalDefaults.get(row.id) } : {}) },
}));
if (changes.some(change => !change.before.enabled && change.after.enabled)) {
await notifyChatRunMilestones(tx, { companyId: endpoint.companyId });
}
if (bulk) {
// One atomic database update, independent of pagination/search and
// serialized with refresh, reconnect, and individual row changes.
@@ -471,25 +481,7 @@ export function createChatResourceReconciliation(db: Db, options: ResourceReconc
await migrateTelegramResourceConversations(tx, currentEndpoint, previousResource, resource, previousProviderResourceId, providerResourceId, migratedLabel, now);
}
if (resource) {
await tx
.update(chatConversations)
.set({
state:
effect.availability === "available"
? "active"
: "unavailable",
updatedAt: now,
})
.where(
and(
eq(chatConversations.companyId, currentEndpoint.companyId),
eq(chatConversations.endpointId, currentEndpoint.id),
eq(chatConversations.resourceId, resource.id),
effect.availability === "available"
? eq(chatConversations.state, "unavailable")
: inArray(chatConversations.state, ["active", "waiting"]),
),
);
await setResourceConversationAvailability(tx, currentEndpoint, resource.id, effect.availability === "available", now);
}
await tx
.update(chatEndpoints)
@@ -512,7 +504,7 @@ export function createChatResourceReconciliation(db: Db, options: ResourceReconc
? "verifying"
: currentEndpoint.status;
await notifyChatEndpointConfiguration(tx, currentEndpoint);
if (status !== currentEndpoint.status) await notifyChatEndpointWork(tx, currentEndpoint.connectionId);
if (status !== currentEndpoint.status) await notifyChatEndpointWork(tx, currentEndpoint.connectionId, currentEndpoint.companyId);
await tx
.update(chatEndpoints)
.set({
@@ -616,7 +608,10 @@ export function createChatResourceReconciliation(db: Db, options: ResourceReconc
});
}
const { reconcileGitHubWebhookRepository, listGitHubRepositories, toggleAllGitHubRepositories } = createGitHubResources(db, { endpointRecord, upsertProviderResourceRow, updateResourceSelection });
const { reconcileGitHubWebhookRepository, listGitHubRepositories, toggleAllGitHubRepositories } = createGitHubResources(db, {
endpointRecord, upsertProviderResourceRow, updateResourceSelection,
restoreResourceConversations: (tx, endpoint, resourceId) => setResourceConversationAvailability(tx, endpoint, resourceId, true, new Date()),
});
return { reconcileGitHubWebhookRepository, reconcileProviderResourceRows, listResources, listGitHubRepositories, replaceResources, toggleAllGitHubRepositories, hasNewerProcessedProviderLifecycleEffect, applyProviderAvailability };
}
@@ -300,7 +300,7 @@ export function createSetupProbes(db: Db, options: SetupProbesOptions) {
)).limit(1).then((rows) => rows[0]);
if (!linked || !(await lockCurrentPrincipalAuthorization(tx, endpoint, linked.principalId)).allowed) throw forbidden("Your connected account is no longer authorized");
}
await notifyChatEndpointWork(tx, endpoint.connectionId);
await notifyChatEndpointWork(tx, endpoint.connectionId, endpoint.companyId);
const [activated] = await tx
.update(chatEndpoints)
.set({
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../../chat-run-milestone-notifications.js";
import { chatIdentityLinks, chatExternalPrincipals, companyMemberships, toolConnections, type Db } from "@paperclipai/db";
import { and, eq, ne, sql } from "drizzle-orm";
import { conflict, forbidden, notFound } from "../../../errors.js";
@@ -46,6 +47,7 @@ export function createSlackSetupServices(db: Db, options: SetupServicesOptions)
const links = await tx.select().from(chatIdentityLinks).where(and(eq(chatIdentityLinks.companyId, record.endpoint.companyId), eq(chatIdentityLinks.principalId, principal.id))).for("update");
if (links.some(link => link.status === "linked" && link.paperclipUserId !== actor.userId) || links.some(link => link.endpointId === endpointId && link.status === "revoked")) throw conflict("This Slack account has an existing link", { code: "chat_identity_link_conflict" });
if (links.some(link => link.endpointId === endpointId && link.status === "linked")) return;
await notifyChatRunMilestones(tx, { companyId: record.endpoint.companyId });
await tx.insert(chatIdentityLinks).values({ companyId: record.endpoint.companyId, endpointId, principalId: principal.id,
paperclipUserId: actor.userId, status: "linked", confirmedAt: new Date() })
.onConflictDoUpdate({ target: [chatIdentityLinks.endpointId, chatIdentityLinks.principalId], set: {
@@ -1,5 +1,6 @@
import { and, eq, sql } from "drizzle-orm";
import { chatConversations, chatEndpointResources, chatEndpoints, type Db } from "@paperclipai/db";
import { notifyChatRunMilestones } from "../../chat-run-milestone-notifications.js";
import type { ChatProviderLifecycleEffect } from "../../chat-provider-lifecycle.js";
type DbTransaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
type EndpointRow = typeof chatEndpoints.$inferSelect;
@@ -70,6 +71,18 @@ export async function migrateTelegramResourceConversations(
resource: { id: string }, previousProviderResourceId: string | undefined,
providerResourceId: string, migratedLabel: string, now: Date,
) {
const affected = and(
eq(chatConversations.companyId, currentEndpoint.companyId),
eq(chatConversations.endpointId, currentEndpoint.id),
eq(chatConversations.resourceId, previousResource.id),
);
const migrated = await tx.select({ issueId: chatConversations.issueId })
.from(chatConversations).where(affected).for("update");
// Migration restores the binding before generic resource availability runs.
// Register the wake here, within the same owning transaction as that change.
for (const issueId of new Set(migrated.map(row => row.issueId))) {
await notifyChatRunMilestones(tx, { companyId: currentEndpoint.companyId, issueId });
}
const previousThreadPrefix = `telegram:${previousProviderResourceId}`;
const migratedThreadPrefix = `telegram:${providerResourceId}`;
await tx
@@ -94,11 +107,5 @@ export async function migrateTelegramResourceConversations(
state: "active",
updatedAt: now,
})
.where(
and(
eq(chatConversations.companyId, currentEndpoint.companyId),
eq(chatConversations.endpointId, currentEndpoint.id),
eq(chatConversations.resourceId, previousResource.id),
),
);
.where(affected);
}
+13 -3
View File
@@ -24,6 +24,7 @@ import {
import { HttpError, badRequest, conflict, forbidden, notFound, unprocessable } from "../errors.js";
import { toolAccessService } from "./tool-access.js";
import { logActivity } from "./activity-log.js";
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { githubBotRequest } from "./chat-github-client.js";
import { environmentService } from "./environments.js";
import { instanceSettingsService } from "./instance-settings.js";
@@ -145,11 +146,12 @@ export function githubChatManagementService(db: Db, fetchImpl = fetch) {
"Repository overrides must refer to this App installation",
);
await db.transaction(async (tx) => {
await tx
.select({ id: chatEndpoints.id })
const [currentEndpoint] = await tx
.select({ allowUnlinkedPeople: chatEndpoints.allowUnlinkedPeople, sponsorUserId: chatEndpoints.sponsorUserId })
.from(chatEndpoints)
.where(eq(chatEndpoints.id, endpointId))
.for("update");
if (!currentEndpoint) throw notFound("GitHub bot not found");
await tx.execute(
sql`select pg_advisory_xact_lock(hashtextextended(${`github-config:${endpointId}`}, 0))`,
);
@@ -160,6 +162,11 @@ export function githubChatManagementService(db: Db, fetchImpl = fetch) {
.for("update");
if ((current?.revision ?? 0) !== parsed.expectedRevision)
throw conflict("This configuration changed. Refresh before saving.");
const allowUnlinkedPeople = config.people.some((p) => p.kind === "guest");
if ((!currentEndpoint.allowUnlinkedPeople && allowUnlinkedPeople) ||
(allowUnlinkedPeople && currentEndpoint.sponsorUserId !== config.responsibleUserId)) {
await notifyChatRunMilestones(tx, { companyId: bot.companyId });
}
const values = {
companyId: bot.companyId,
endpointId,
@@ -180,7 +187,7 @@ export function githubChatManagementService(db: Db, fetchImpl = fetch) {
await tx
.update(chatEndpoints)
.set({
allowUnlinkedPeople: config.people.some((p) => p.kind === "guest"),
allowUnlinkedPeople,
sponsorUserId: config.responsibleUserId,
updatedAt: new Date(),
})
@@ -327,6 +334,9 @@ export function githubChatManagementService(db: Db, fetchImpl = fetch) {
expiresAt: null,
updatedAt: new Date(),
};
if (prior?.status !== "linked") {
await notifyChatRunMilestones(tx, { companyId: bot.companyId });
}
await tx
.insert(chatIdentityLinks)
.values(values)
@@ -15,19 +15,20 @@ import {
toolConnections,
} from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
import { hasChatRunOwnedProviderInteraction } from "./chat-interaction-arbitration.js";
const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_ARBITRATION_TEST_DATABASE_URL");
describe("provider-owned interaction arbitration", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: ReturnType<typeof createDb>;
let companyCount = 0;
beforeAll(async () => {
temporary = await startEmbeddedPostgresTestDatabase(
"chat-interaction-owner-",
);
db = createDb(temporary.connectionString);
if (!externalDatabaseUrl) temporary = await startEmbeddedPostgresTestDatabase("chat-interaction-owner-");
db = createDb(externalDatabaseUrl ?? temporary!.connectionString);
}, 30_000);
afterAll(async () => {
await db?.$client.end();
await temporary?.cleanup();
});
@@ -43,7 +44,7 @@ describe("provider-owned interaction arbitration", () => {
.values({
id: companyId,
name: "Arbitration",
issuePrefix: `CI${String.fromCharCode(65 + companyCount++)}`,
issuePrefix: `C${companyId.slice(0, 7)}`,
});
await db.insert(agents).values(
[agentId, otherAgentId].map((id) => ({
@@ -0,0 +1,9 @@
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
/** Lineage changes can authorize descendants, so invalidate the issue rather
* than just the run whose row changed. Configuration may invalidate a company. */
export type ChatRunMilestoneScope = { companyId: string; issueId?: string };
export function notifyChatRunMilestones(tx: object, scope: ChatRunMilestoneScope) {
return notifyDeliveryWork(tx, DELIVERY_QUEUES.chatRunMilestones, JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId }));
}
@@ -0,0 +1,38 @@
import type { Db } from "@paperclipai/db";
import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
import { registerScopedDeadlineWork } from "./scoped-deadline-work.js";
import type { ChatRunMilestoneScope } from "./chat-run-milestone-notifications.js";
import { enqueueChatRunLifecycleMilestones, loadChatRunLifecycleMilestoneScopes } from "./chat-run-publications.js";
type Entry = { id: string; companyId: string; issueId: string; dueAt: number };
const batchSize = 100;
/** No periodic discovery: reconstruct on startup/committed changes, drain
* current notices, and retain a retry only for failure or an unfinished batch. */
export function chatRunMilestoneWork(db: Db, publicBaseUrl?: string | null) {
async function load(scope?: ChatRunMilestoneScope, signal?: AbortSignal): Promise<Entry[]> {
return (await loadChatRunLifecycleMilestoneScopes(db, scope, signal)).map(row => ({
...row, id: `${row.companyId}:${row.issueId}`, dueAt: 0,
}));
}
async function check(entry: Entry, signal?: AbortSignal): Promise<Entry | null> {
const inserted = await enqueueChatRunLifecycleMilestones(db, {
scope: { companyId: entry.companyId, issueId: entry.issueId },
publicBaseUrl, limit: batchSize, signal,
});
// A short batch exhausted the selector, including currently suppressed
// candidates. Their dependencies must signal a later eligibility change.
return inserted === batchSize ? { ...entry, dueAt: Date.now() } : null;
}
function register(coordinator: ReturnType<typeof createDeliveryWorkCoordinator>, canRun: () => boolean) {
return registerScopedDeadlineWork(coordinator, {
queue: DELIVERY_QUEUES.chatRunMilestones,
decodeScope: (hint: string): ChatRunMilestoneScope | undefined => hint === "*" ? undefined : JSON.parse(hint),
matches: (entry: Entry, scope?: ChatRunMilestoneScope) => !scope ||
(entry.companyId === scope.companyId && (!scope.issueId || entry.issueId === scope.issueId)),
load, check,
}, canRun);
}
return { load, check, register };
}
+99 -61
View File
@@ -1,3 +1,4 @@
import type { ChatRunMilestoneScope } from "./chat-run-milestone-notifications.js";
import { notifyChatPublicationWork } from "./chat-work-notifications.js";
import {
and,
@@ -218,7 +219,7 @@ type ChatRunMilestoneCandidate = {
runId: string;
runStatus: string;
runErrorCode: string | null;
runUpdatedAt: Date;
runUpdatedAt: string;
issueId: string;
companyId: string;
endpointId: string;
@@ -297,10 +298,11 @@ export function safeMilestoneText(input: {
* lane. The selector intentionally reads only event identity, type, sequence,
* and time; native messages and payloads stay inside Paperclip.
*/
async function enqueueSafeNativeChatProgress(
export async function enqueueSafeNativeChatProgress(
db: Db,
input: { since: Date; limit: number },
options: { since?: Date; limit?: number } = {},
): Promise<number> {
const input = chatRunProjectionWindow(options);
const basePublication = alias(
chatPublications,
"safe_native_progress_base_publication",
@@ -670,26 +672,19 @@ async function enqueueSafeNativeChatProgress(
return inserted;
}
/**
* Project only coarse run lifecycle into bound external conversations. Raw
* output, errors, tool events, and reasoning stay in Paperclip. Idempotency is
* keyed by run, milestone, and endpoint so polling and restarts are harmless.
*/
export async function enqueueChatRunMilestones(
db: Db,
input: {
publicBaseUrl?: string | null;
since?: Date;
limit?: number;
} = {},
): Promise<number> {
const since = input.since ?? new Date(Date.now() - 24 * 60 * 60_000);
const limit = Math.max(1, Math.min(input.limit ?? 200, 1_000));
const issueIdFromContext = boundRunIssueIdSql();
const explicitlyAuthoredCommentReason = sql<boolean>`(
type ChatRunMilestoneCursor = {
updatedAt: string;
runId: string;
conversationId: string;
};
const explicitlyAuthoredCommentReason = sql<boolean>`(
${issueComments.metadata} ->> 'authorizationReason' = 'paperclip_runner_protocol'
or left(coalesce(${issueComments.metadata} ->> 'authorizationReason', ''), 6) = 'allow_'
)`;
function lifecycleMilestonePages(db: Db, since: Date, scope?: ChatRunMilestoneScope) {
const issueIdFromContext = boundRunIssueIdSql();
const milestoneFromStatus = sql<string>`case
when ${heartbeatRuns.status} = 'queued' then 'queued'
when ${heartbeatRuns.status} = 'running'
@@ -801,26 +796,15 @@ export async function enqueueChatRunMilestones(
and continuation_wake.payload->>'issueId' = ${issueIdFromContext}
)
)`;
let inserted = 0;
let cursor: {
updatedAt: Date;
runId: string;
conversationId: string;
} | null = null;
const bindingsCache = new Map<
string,
Awaited<ReturnType<typeof resolveChatOriginPublicationBindings>>
>();
const providerInteractionCache = new Map<string, boolean>();
while (inserted < limit) {
const pageSize = Math.max(25, Math.min(200, limit - inserted));
const pageCursor: typeof cursor = cursor;
const rows: ChatRunMilestoneCandidate[] = await db
return (pageSize: number, pageCursor: ChatRunMilestoneCursor | null): Promise<ChatRunMilestoneCandidate[]> => {
return db
.select({
runId: heartbeatRuns.id,
runStatus: heartbeatRuns.status,
runErrorCode: heartbeatRuns.errorCode,
runUpdatedAt: heartbeatRuns.updatedAt,
// Keep PostgreSQL precision: a Date cursor can repeatedly select a
// rejected page whose updated_at has submillisecond digits.
runUpdatedAt: sql<string>`${heartbeatRuns.updatedAt}::text`,
issueId: chatConversations.issueId,
companyId: chatConversations.companyId,
endpointId: chatConversations.endpointId,
@@ -979,22 +963,13 @@ export async function enqueueChatRunMilestones(
and target_delivery.status in ('delivered', 'fallback_queued')
)`,
),
scope ? eq(heartbeatRuns.companyId, scope.companyId) : undefined,
scope?.issueId ? eq(chatConversations.issueId, scope.issueId) : undefined,
gte(heartbeatRuns.updatedAt, since),
isNull(chatPublications.id),
pageCursor
? or(
gt(heartbeatRuns.updatedAt, pageCursor.updatedAt),
and(
eq(heartbeatRuns.updatedAt, pageCursor.updatedAt),
or(
gt(heartbeatRuns.id, pageCursor.runId),
and(
eq(heartbeatRuns.id, pageCursor.runId),
gt(chatConversations.id, pageCursor.conversationId),
),
),
),
)
? sql`(${heartbeatRuns.updatedAt}, ${heartbeatRuns.id}, ${chatConversations.id}) >
(${pageCursor.updatedAt}::timestamptz, ${pageCursor.runId}::uuid, ${pageCursor.conversationId}::uuid)`
: undefined,
),
)
@@ -1004,15 +979,59 @@ export async function enqueueChatRunMilestones(
asc(chatConversations.id),
)
.limit(pageSize);
};
}
function milestoneCursor(row: ChatRunMilestoneCandidate): ChatRunMilestoneCursor {
return { updatedAt: row.runUpdatedAt, runId: row.runId, conversationId: row.conversationId };
}
/** Reconstruct issue-scoped projection work. Candidate rows are only hints;
* projection still applies current lineage and provider authority. */
export async function loadChatRunLifecycleMilestoneScopes(
db: Db,
scope?: ChatRunMilestoneScope,
signal?: AbortSignal,
): Promise<Array<{ companyId: string; issueId: string }>> {
const pages = lifecycleMilestonePages(db, chatRunProjectionWindow({}).since, scope);
const scopes = new Map<string, { companyId: string; issueId: string }>();
let cursor: ChatRunMilestoneCursor | null = null;
for (;;) {
signal?.throwIfAborted();
const rows = await pages(100, cursor);
for (const row of rows) scopes.set(`${row.companyId}:${row.issueId}`, { companyId: row.companyId, issueId: row.issueId });
if (rows.length < 100) return [...scopes.values()];
cursor = milestoneCursor(rows[rows.length - 1]);
}
}
/**
* Project only coarse run lifecycle into bound external conversations. Raw
* output, errors, tool events, and reasoning stay in Paperclip. Idempotency is
* keyed by run, milestone, and endpoint so polling and restarts are harmless.
*/
export async function enqueueChatRunLifecycleMilestones(
db: Db,
input: ChatRunProjectionInput & { scope?: ChatRunMilestoneScope; signal?: AbortSignal } = {},
): Promise<number> {
const { since, limit } = chatRunProjectionWindow(input);
let inserted = 0;
const pages = lifecycleMilestonePages(db, since, input.scope);
let cursor: ChatRunMilestoneCursor | null = null;
const bindingsCache = new Map<
string,
Awaited<ReturnType<typeof resolveChatOriginPublicationBindings>>
>();
const providerInteractionCache = new Map<string, boolean>();
while (inserted < limit) {
const pageSize = Math.max(25, Math.min(200, limit - inserted));
input.signal?.throwIfAborted();
const rows = await pages(pageSize, cursor);
if (rows.length === 0) break;
const lastRow = rows.at(-1)!;
cursor = {
updatedAt: lastRow.runUpdatedAt,
runId: lastRow.runId,
conversationId: lastRow.conversationId,
};
cursor = milestoneCursor(rows[rows.length - 1]);
for (const row of rows) {
input.signal?.throwIfAborted();
if (inserted >= limit) break;
const milestone = milestoneForStatus(row.runStatus, row.runErrorCode);
if (!milestone) continue;
@@ -1107,11 +1126,30 @@ export async function enqueueChatRunMilestones(
}
if (rows.length < pageSize) break;
}
if (inserted < limit) {
inserted += await enqueueSafeNativeChatProgress(db, {
since,
limit: limit - inserted,
});
}
return inserted;
}
type ChatRunProjectionInput = {
publicBaseUrl?: string | null;
since?: Date;
limit?: number;
};
function chatRunProjectionWindow(input: ChatRunProjectionInput) {
return {
since: input.since ?? new Date(Date.now() - 24 * 60 * 60_000),
limit: Math.max(1, Math.min(input.limit ?? 200, 1_000)),
};
}
/** Explicit combined reconciliation retains lifecycle priority and one shared
* publication budget. Background owners can reconcile the two duties separately. */
export async function enqueueChatRunMilestones(
db: Db,
input: ChatRunProjectionInput = {},
): Promise<number> {
const { since, limit } = chatRunProjectionWindow(input);
const inserted = await enqueueChatRunLifecycleMilestones(db, { ...input, since, limit });
if (inserted >= limit) return inserted;
return inserted + await enqueueSafeNativeChatProgress(db, { since, limit: limit - inserted });
}
+15 -2
View File
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { notifyChatConnectionConfiguration } from "./chat-channels/configuration-work-notifications.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
@@ -5,10 +6,19 @@ export async function notifyChatDeliveryWork(tx: object): Promise<void> {
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatInbound);
}
export async function notifyChatPublicationWork(tx: object): Promise<void> {
export async function notifyChatPublicationWork(
tx: object,
milestoneScope?: { companyId: string; issueId: string | null },
): Promise<void> {
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatPublications);
// Receipt recovery waits for the publication owner to release its claim.
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatReceipts);
// Published prompts and terminal delivery outcomes can change run lineage or
// release its provider response slot. Callers supply the affected issue only
// for those changes; transport claim/lease bookkeeping has no projection debt.
if (milestoneScope?.issueId) {
await notifyChatRunMilestones(tx, { companyId: milestoneScope.companyId, issueId: milestoneScope.issueId });
}
}
export const CHAT_ACTION_QUEUES = {
@@ -32,7 +42,10 @@ export async function notifyChatVerificationWork(tx: object): Promise<void> {
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatVerification);
}
export async function notifyChatEndpointWork(tx: object, connectionId: string): Promise<void> {
export async function notifyChatEndpointWork(tx: object, connectionId: string, companyId: string): Promise<void> {
// Configuration/status transitions can restore post-control publication
// authority. Ordinary health checks use the narrower configuration signals.
await notifyChatRunMilestones(tx, { companyId });
await notifyChatDeliveryWork(tx);
await notifyChatPublicationWork(tx);
for (const queue of Object.values(CHAT_ACTION_QUEUES)) await notifyDeliveryWork(tx, queue);
@@ -57,6 +57,7 @@ export const DELIVERY_QUEUES = {
chatVerification: "chat-verification",
chatInbound: "chat-inbound",
chatPublications: "chat-publications",
chatRunMilestones: "chat-run-milestones",
chatReceipts: "chat-slack-receipts",
browser: "browser-use",
email: "email-channels",
+6 -4
View File
@@ -3,6 +3,7 @@ import type { Db } from "@paperclipai/db";
import { agents, agentWakeupRequests, caseHandoffs, cases, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns, issueComments, issues, nativeRunFinalizations } from "@paperclipai/db";
import { eq, inArray, sql } from "drizzle-orm";
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { createPostgresWakeQueueAdapter } from "../../modules/wake-queue/adapters/postgres.js";
import { createReleaseIssueExecution } from "../../modules/wake-queue/application/use-cases.js";
@@ -74,14 +75,15 @@ describe("heartbeat queue module boundary", () => {
});
});
const support = await getEmbeddedPostgresTestSupport();
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_QUEUE_TEST_DATABASE_URL");
const support = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
if (!support.supported) console.warn(`Skipping queue module database tests: ${support.reason}`);
describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: Db;
beforeAll(async () => {
database = await startEmbeddedPostgresTestDatabase("paperclip-heartbeat-queue-module");
db = createDb(database.connectionString);
if (!external) database = await startEmbeddedPostgresTestDatabase("paperclip-heartbeat-queue-module");
db = createDb(external ?? database!.connectionString);
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
afterAll(async () => {
await db?.$client.end();
+85 -53
View File
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { assertTaskRetryConversationAllowed, readTaskRetryIntent, readTaskRetryBoardMessageOrigin, retireSupersededTaskRetry, taskRetryMutationAllowed } from "../task-retry.js";
import { insertIssueComment } from "../issue-comment-writes.js";
import { withExternalAdmissionGuard, assertNoExternalOverlap } from "../../modules/external-agents/index.js";
@@ -845,6 +846,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
const error = stopped
? "Automatic continuation stopped by the committed chat conversation close. Send a new request in chat or on the Board to start fresh work."
: "Automatic continuation source could not be verified before provider admission. Review the task and send a fresh request; this attempt will not automatically retry.";
await notifyChatRunMilestones(tx, { companyId: current.companyId, issueId });
[terminal] = await tx
.update(heartbeatRuns)
.set({
@@ -960,9 +962,17 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
if (!["pending_dispatch", "delivered"].includes(caseHandoffReceipt.state) || caseHandoffReceipt.toAgentId !== run.agentId ||
!current || current.version !== caseHandoffReceipt.version || current.issueId !== caseHandoffReceipt.issueId ||
current.agentId !== run.agentId || (!run.startedAt && current.statusVersion !== caseHandoffReceipt.appliedIssueStatusVersion)) {
const [cancelled] = await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date(),
errorCode: "case_handoff_superseded", error: "This case handoff was cancelled or superseded.", updatedAt: new Date() })
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.status, "queued"))).returning();
const cancelled = await db.transaction(async tx => {
const [currentRun] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.status, "queued")))
.for("update");
if (!currentRun) return null;
await notifyChatRunMilestones(tx, { companyId: run.companyId });
const [row] = await tx.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date(),
errorCode: "case_handoff_superseded", error: "This case handoff was cancelled or superseded.", updatedAt: new Date() })
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.status, "queued"))).returning();
return row;
});
if (cancelled) await setWakeupStatus(run.wakeupRequestId, "cancelled", { finishedAt: new Date(), error: cancelled.error });
return null;
}
@@ -1317,6 +1327,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
if (proof.kind === "stopped") {
const error =
"Automatic continuation stopped by the committed chat conversation close. Send a fresh request in chat or on the Board.";
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
const [cancelled] = await tx
.update(heartbeatRuns)
.set({
@@ -1395,6 +1406,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
// explicitly bound queued-message envelope is subject to the
// live-comment discard gate below.
await withExternalAdmissionGuard(tx, run.companyId, run.agentId, () => assertNoExternalOverlap(tx, run.companyId, run.agentId));
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
const [claimedRun] = await tx
.update(heartbeatRuns)
.set({
@@ -1436,6 +1448,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
});
if (liveIds.length === 0) {
const reason = "Queued messages were discarded before dispatch";
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
const [cancelled] = await tx
.update(heartbeatRuns)
.set({
@@ -1483,6 +1496,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
}
await withExternalAdmissionGuard(tx, run.companyId, run.agentId, () => assertNoExternalOverlap(tx, run.companyId, run.agentId));
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
await tx
.update(agentWakeupRequests)
.set({
@@ -1593,6 +1607,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
)).for("update");
if (!lockedRun || lockedRun.status !== "queued") return null;
await withExternalAdmissionGuard(claimTx, run.companyId, run.agentId, () => assertNoExternalOverlap(claimTx, run.companyId, run.agentId));
await notifyChatRunMilestones(claimTx, { companyId: run.companyId, issueId: issueId ?? undefined });
const claimedRun = await claimTx.update(heartbeatRuns).set(claimValues).where(and(
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.status, "queued"),
)).returning().then((rows) => rows[0] ?? null);
@@ -1690,6 +1705,11 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
async function releaseRunClaimedJustBeforeSuppression(runId: string) {
const now = new Date();
await db.transaction(async (tx) => {
const [candidate] = await tx.select({ companyId: heartbeatRuns.companyId, contextSnapshot: heartbeatRuns.contextSnapshot })
.from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.status, "running"))).for("update");
if (!candidate) return;
await notifyChatRunMilestones(tx, { companyId: candidate.companyId,
issueId: readNonEmptyString(parseObject(candidate.contextSnapshot).issueId) ?? undefined });
const released = await tx
.update(heartbeatRuns)
.set({
@@ -3454,6 +3474,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
const reason = issueCancelled
? "Cancelled because the issue was cancelled before the scheduled retry became due"
: "Cancelled because the issue was reassigned before the scheduled retry became due";
await notifyChatRunMilestones(tx, { companyId: scheduledRun.companyId, issueId: issue.id });
const cancelled = await tx
.update(heartbeatRuns)
.set({
@@ -3590,6 +3611,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
issue.assigneeAgentId &&
activeExecutionRun.agentId !== issue.assigneeAgentId
) {
await notifyChatRunMilestones(tx, { companyId: activeExecutionRun.companyId, issueId: issue.id });
const cancelled = await tx
.update(heartbeatRuns)
.set({
@@ -4309,6 +4331,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
adoptedCommentIds = await undeliveredLegacyUserCommentIds(tx as unknown as Db,
agent.companyId, issueId, agentId, adoptedCommentIds);
}
await notifyChatRunMilestones(tx, { companyId: agent.companyId, issueId: issueId ?? undefined });
const newRun = await tx
.insert(heartbeatRuns)
.values({
@@ -4701,6 +4724,13 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
>();
for (const candidate of candidates) {
const payload = parseObject(candidate.payload);
const wakeContext = parseObject(payload._paperclipWakeContext);
const issueId =
readNonEmptyString(payload.issueId) ??
readNonEmptyString(payload.taskId) ??
readNonEmptyString(wakeContext.issueId) ??
null;
const dispatchActorId = `native-status-wake-dispatch:${candidate.id}`;
const existingDispatch = await db
.select()
@@ -4722,24 +4752,27 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
: existingDispatch.status === "deferred_issue_execution"
? "coalesced"
: existingDispatch.status;
await db
.update(agentWakeupRequests)
.set({
status: recoveredStatus,
runId: existingDispatch.runId,
finishedAt:
(existingDispatch.runId ?? existingDispatch.finishedAt)
? (existingDispatch.finishedAt ?? now)
: null,
error: existingDispatch.error,
updatedAt: now,
})
.where(
and(
eq(agentWakeupRequests.id, candidate.id),
isNull(agentWakeupRequests.runId),
),
);
await db.transaction(async tx => {
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, issueId: issueId ?? undefined });
await tx
.update(agentWakeupRequests)
.set({
status: recoveredStatus,
runId: existingDispatch.runId,
finishedAt:
(existingDispatch.runId ?? existingDispatch.finishedAt)
? (existingDispatch.finishedAt ?? now)
: null,
error: existingDispatch.error,
updatedAt: now,
})
.where(
and(
eq(agentWakeupRequests.id, candidate.id),
isNull(agentWakeupRequests.runId),
),
);
});
recovered += 1;
continue;
}
@@ -4773,13 +4806,6 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
.returning({ id: agentWakeupRequests.id });
if (claimed.length === 0) continue;
const payload = parseObject(candidate.payload);
const wakeContext = parseObject(payload._paperclipWakeContext);
const issueId =
readNonEmptyString(payload.issueId) ??
readNonEmptyString(payload.taskId) ??
readNonEmptyString(wakeContext.issueId) ??
null;
const scopeKey = issueId
? `${candidate.companyId}:${candidate.agentId}:${issueId}:${readNativeReviewAssignmentContext(wakeContext)?.nativeReviewInteractionId ?? ""}`
: null;
@@ -4787,16 +4813,19 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
? deliveredByIssueScope.get(scopeKey)
: null;
if (priorDelivery) {
await db
.update(agentWakeupRequests)
.set({
status: "coalesced",
runId: priorDelivery.runId,
finishedAt: new Date(),
error: null,
updatedAt: new Date(),
})
.where(eq(agentWakeupRequests.id, candidate.id));
await db.transaction(async tx => {
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, issueId: issueId ?? undefined });
await tx
.update(agentWakeupRequests)
.set({
status: "coalesced",
runId: priorDelivery.runId,
finishedAt: new Date(),
error: null,
updatedAt: new Date(),
})
.where(eq(agentWakeupRequests.id, candidate.id));
});
recovered += 1;
continue;
}
@@ -4895,21 +4924,24 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
// promote both rows (the original has no nativeStatusWakeIntentId
// provenance and would otherwise run once before the dispatch receipt).
const deliveredStatus = delivered?.status ?? "queued";
await db
.update(agentWakeupRequests)
.set({
status: wakeRun || deliveredStatus === "deferred_issue_execution"
? "coalesced"
: deliveredStatus,
runId: wakeRun?.id ?? delivered?.runId ?? null,
finishedAt:
wakeRun || delivered?.finishedAt
? (delivered?.finishedAt ?? new Date())
: null,
error: delivered?.error ?? null,
updatedAt: new Date(),
})
.where(eq(agentWakeupRequests.id, candidate.id));
await db.transaction(async tx => {
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, issueId: issueId ?? undefined });
await tx
.update(agentWakeupRequests)
.set({
status: wakeRun || deliveredStatus === "deferred_issue_execution"
? "coalesced"
: deliveredStatus,
runId: wakeRun?.id ?? delivered?.runId ?? null,
finishedAt:
wakeRun || delivered?.finishedAt
? (delivered?.finishedAt ?? new Date())
: null,
error: delivered?.error ?? null,
updatedAt: new Date(),
})
.where(eq(agentWakeupRequests.id, candidate.id));
});
if (scopeKey) {
deliveredByIssueScope.set(scopeKey, {
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { measureRunPhase } from "../execution-preparation-timing.js";
import { PROJECT_ACCESS_DENIED, ProjectAccessDeniedError } from "../project-access-denial.js";
import { pendingCaseHandoffForRun } from "./case-handoff-lifecycle.js";
@@ -891,13 +892,13 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
};
}
await db
.update(heartbeatRuns)
.set({
await db.transaction(async tx => {
await notifyChatRunMilestones(tx, { companyId: livenessRun.companyId, issueId: issueId ?? undefined });
await tx.update(heartbeatRuns).set({
resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({ presentationDecision })}::jsonb`,
updatedAt: new Date(),
})
.where(eq(heartbeatRuns.id, livenessRun.id));
}).where(eq(heartbeatRuns.id, livenessRun.id));
});
await appendRunEvent(livenessRun, {
eventType: "run.presentation.resolved",
stream: "system",
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
import os from "node:os";
import {
@@ -1019,6 +1020,7 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl
const tx = transaction as unknown as Db;
if (isHeartbeatRunTerminalStatus(status)) await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, companyId);
await notifyExecutionRunWork(tx);
if (companyId) await notifyChatRunMilestones(tx, { companyId });
return write(tx);
});
}
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { notifyToolReviewChange } from "./tool-review-work-notifications.js";
import { normalizeEscapedLineBreaks } from "@paperclipai/shared/validators/text";
import { notifyDeliveryWork, DELIVERY_QUEUES } from "./delivery-work-notifications.js";
@@ -1912,6 +1913,7 @@ async function expireStaleRequestConfirmationTarget(
snapshot,
});
const expireInTransaction = async (tx: Db | any) => {
if (args.row.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.row);
const [updated] = await tx
.update(issueThreadInteractions)
.set({
@@ -2189,6 +2191,7 @@ export function issueThreadInteractionService(
const postCommitActivityPublications: ActivityPublication[] = [];
const result = await db.transaction(async (tx) => {
await assertInteractionRunWriteAllowed(tx as unknown as Db, args.issue, args.actor);
if (args.current.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.current);
await args.mutationOptions?.beforeResolveInTransaction?.(tx);
// Policy mutations and review transitions use the same issue-row lock,
// so the authoritative review policy and requester are stable through
@@ -2472,6 +2475,7 @@ export function issueThreadInteractionService(
const now = new Date();
const updated = await db.transaction(async (tx) => {
await assertInteractionRunWriteAllowed(tx as unknown as Db, args.issue, args.actor);
if (args.current.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.current);
await args.mutationOptions?.beforeResolveInTransaction?.(tx);
const issueContext = await tx
.select({
@@ -3290,6 +3294,7 @@ export function issueThreadInteractionService(
const cancelled: IssueThreadInteraction[] = [];
for (const row of rows) {
const updated = await db.transaction(async (tx) => {
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
if (row.kind === "request_confirmation") await notifyToolProjectionForInteractions(tx, row.companyId, [row.id]);
const [resolved] = await tx
.update(issueThreadInteractions)
@@ -3375,6 +3380,7 @@ export function issueThreadInteractionService(
const expired: IssueThreadInteraction[] = [];
for (const { row, replacementInteractionId } of supersededRows) {
const updated = await db.transaction(async (tx) => {
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
if (row.kind === "request_confirmation") await notifyToolProjectionForInteractions(tx, row.companyId, [row.id]);
const [updatedRow] = await tx
.update(issueThreadInteractions)
@@ -3736,6 +3742,9 @@ export function issueThreadInteractionService(
eq(chatVoiceSessions.callerAuthority, "guest_intake"))).limit(1);
if (guest) throw unprocessable("Ask this unverified caller a clarification in a task comment; submit_request will deliver their spoken follow-up. Protected human-input questions require authenticated access.", {code: "voice_guest_use_task_comment"});
}
if (data.kind === "ask_user_questions" || data.kind === "request_confirmation") {
await notifyChatRunMilestones(tx, { companyId: issue.companyId, issueId: issue.id });
}
const [row] = await tx
.insert(issueThreadInteractions)
.values({
@@ -4459,6 +4468,7 @@ export function issueThreadInteractionService(
const expired: IssueThreadInteraction[] = [];
for (const row of superseded) {
const updated = await db.transaction(async (tx) => {
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
if (row.kind === "connection_intent") {
await tx
.delete(toolOauthStates)
@@ -4595,6 +4605,7 @@ export function issueThreadInteractionService(
);
if (!sampleQuestionRow) continue;
const updatedInteractions = await db.transaction(async (tx) => {
await notifyChatRunMilestones(tx, sampleQuestionRow);
const updatedRows = await tx
.update(issueThreadInteractions)
.set({
@@ -4633,6 +4644,7 @@ export function issueThreadInteractionService(
);
if (!sampleConfirmationRow) continue;
const updatedInteractions = await db.transaction(async (tx) => {
if (commentRows.some(row => row.kind === "request_confirmation")) await notifyChatRunMilestones(tx, sampleConfirmationRow);
await notifyToolProjectionForInteractions(tx, issue.companyId, confirmationRowIds);
const updatedRows = await tx
.update(issueThreadInteractions)
@@ -4778,6 +4790,7 @@ export function issueThreadInteractionService(
document,
});
const updated = await db.transaction(async (tx) => {
if (row.kind === "request_confirmation") await notifyChatRunMilestones(tx, row);
const [resolved] = await tx
.update(issueThreadInteractions)
.set({
@@ -4898,6 +4911,7 @@ export function issueThreadInteractionService(
// lifecycle reflection.
const updated = await db
.transaction(async (tx) => {
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
if (row.kind === "connection_intent") {
await tx
.delete(toolOauthStates)
@@ -4988,6 +5002,7 @@ export function issueThreadInteractionService(
// request must not outlive a withdrawn card.
const updated = await db.transaction(async (tx) => {
await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor);
if (["ask_user_questions", "request_confirmation"].includes(current.kind)) await notifyChatRunMilestones(tx, current);
await resolveLinkedToolActionRequests(tx, current, {
status: "cancelled",
fromStatuses: ["pending", "approved"],
@@ -5123,6 +5138,7 @@ export function issueThreadInteractionService(
if (!historicalAnswer) throw interactionIssueClosedError();
}
await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor);
await notifyChatRunMilestones(tx, current);
await mutationOptions.beforeResolveInTransaction?.(tx);
const resolvedAt = new Date();
const [row] = await tx
@@ -5247,6 +5263,7 @@ export function issueThreadInteractionService(
const now = new Date();
const updated = await db.transaction(async (tx) => {
await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor);
if (["ask_user_questions", "request_confirmation"].includes(current.kind)) await notifyChatRunMilestones(tx, current);
await resolveLinkedToolActionRequests(tx, current, {
status: "cancelled",
fromStatuses: ["pending", "approved"],
@@ -5344,6 +5361,7 @@ export function issueThreadInteractionService(
const reason = data.reason?.trim() || null;
const updated = await db.transaction(async (tx) => {
await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor);
await notifyChatRunMilestones(tx, current);
const resolvedAt = new Date();
const [row] = await tx
.update(issueThreadInteractions)
+4
View File
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { notifyApprovalWork } from "./approval-work-notifications.js";
import { notifyDecisionTargetWork, notifyDecisionTargetIssueRemoval } from "./decision-targets.js";
import { assertOnboardingFirstTaskCreationAllowed } from "./onboarding-first-task-admission.js";
@@ -11413,6 +11414,9 @@ export function issueService(db: Db) {
if (patch.status === "done" || patch.status === "cancelled") {
await notifyDecisionTargetWork(tx, `issue:${id}`);
}
if (patch.hiddenAt === null && receiptExisting.hiddenAt !== null) {
await notifyChatRunMilestones(tx, { companyId: receiptExisting.companyId, issueId: id });
}
if (patch.hiddenAt !== undefined || patch.harnessKind !== undefined || patch.conversationAgentId !== undefined) {
await notifyApprovalWork(tx, receiptExisting.companyId);
}
@@ -15,6 +15,7 @@ import {
type ActivityPublication,
} from "../activity-log.js";
import { enqueueTerminalIssueInteractionChatPublications } from "../chat-interaction-publications.js";
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { issueThreadInteractionService } from "../issue-thread-interactions.js";
import { logger } from "../../middleware/logger.js";
@@ -126,6 +127,7 @@ export async function dismissAutomaticCompletionReviews(
)
.for("update");
if (!issue) return;
await notifyChatRunMilestones(tx, interaction);
const now = new Date();
const [retired] = await tx
.update(issueThreadInteractions)
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { and, eq, inArray, ne, sql } from "drizzle-orm";
import {
agents,
@@ -573,20 +574,28 @@ export async function restoreNativeChatReviewPresentationInTransaction(
assessmentId: input.assessmentId,
externalChatReviewPresentation: { ...proof, decisionId: input.decisionId },
};
const runOwner = and(
eq(heartbeatRuns.id, input.runId),
eq(heartbeatRuns.companyId, input.companyId),
eq(heartbeatRuns.nativeIssueId, input.issueId),
eq(heartbeatRuns.runtimeMode, "native"),
);
const projectedResult = sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify(projection)}::jsonb`;
const [run] = await db.select({
needsUpdate: sql<boolean>`${projectedResult} is distinct from ${heartbeatRuns.resultJson}`,
}).from(heartbeatRuns).where(runOwner).for("update");
if (!run) return false;
// A valid existing projection still permits the caller to materialize its
// response. Replaying recovery must not rewrite it or signal another scan.
if (!run.needsUpdate) return true;
await notifyChatRunMilestones(db, input);
const [updated] = await db
.update(heartbeatRuns)
.set({
resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify(projection)}::jsonb`,
resultJson: projectedResult,
updatedAt: new Date(),
})
.where(
and(
eq(heartbeatRuns.id, input.runId),
eq(heartbeatRuns.companyId, input.companyId),
eq(heartbeatRuns.nativeIssueId, input.issueId),
eq(heartbeatRuns.runtimeMode, "native"),
),
)
.where(runOwner)
.returning({ id: heartbeatRuns.id });
return Boolean(updated);
}
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { withExternalAdmissionGuard, assertNoExternalOverlap } from "../../modules/external-agents/index.js";
import { agentWakeupRequests, heartbeatRuns, issues, type Db } from "@paperclipai/db";
import { and, eq } from "drizzle-orm";
@@ -40,6 +41,7 @@ export async function claimQueuedNativeReviewRun(db: Db, input: {
contextSnapshot: context, agentNameKey: input.agentNameKey, claimedAt: input.claimedAt,
});
if (!locked) return null;
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
const [claimed] = await tx.update(heartbeatRuns).set({
...input.claimValues, status: "running", startedAt: run.startedAt ?? input.claimedAt,
updatedAt: input.claimedAt,
@@ -1,4 +1,5 @@
import { pendingNativeGovernance } from "./native-governance.js";
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { onboardingFirstTaskCompletionAllowed } from "../onboarding-first-task-admission.js";
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
import { reconcileMuseNativeTerminalRun } from "./muse-terminal-reconciliation.js";
@@ -377,6 +378,7 @@ async function recordRetryableFailure(input: {
: input.failureCode;
if (phase === "terminal_failure") await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, input.run.companyId);
await notifyExecutionRunWork(tx);
await notifyChatRunMilestones(tx, { companyId: input.run.companyId, issueId: input.coordinator.issueId });
await tx
.update(nativeRunFinalizations)
.set({
@@ -666,6 +668,7 @@ async function projectCommittedRun(input: {
const tx = transaction as unknown as Db;
await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, input.run.companyId);
await notifyExecutionRunWork(tx);
await notifyChatRunMilestones(tx, { companyId: input.run.companyId, issueId: input.coordinator.issueId });
return tx
.update(heartbeatRuns)
.set({
@@ -965,6 +968,7 @@ export async function repairCommittedNativeChatResponse(
const authorizationReason = await resolveChatRunPresentationAuthorizationReason(tx, input);
const resolved = resolveHeartbeatRunResponse({ resultJson: monitorResult });
if (!resolved.text) return false;
await notifyChatRunMilestones(tx, input);
const comment = await issueService(db).addComment(input.issueId, resolved.text,
{ agentId: run.agentId, runId: run.id }, { authorizationReason, completionReply: true }, tx);
await tx.update(heartbeatRuns).set({
@@ -1019,6 +1023,7 @@ export async function repairCommittedNativeChatResponse(
finalAgentMessage,
});
if (!resolved.text || resolved.decision.commentAction !== "create") return false;
await notifyChatRunMilestones(tx, input);
const comment = await issueService(db).addComment(input.issueId, resolved.text,
{ agentId: run.agentId, runId: run.id },
{ authorizationReason: "internal_agent_write", completionReply: true }, tx);
@@ -1062,6 +1067,7 @@ export async function repairCommittedNativeChatResponse(
});
if (!resolved.text || resolved.decision.commentAction !== "create")
return false;
await notifyChatRunMilestones(tx, input);
const comment = await issueService(db).addComment(
input.issueId,
resolved.text,
@@ -1678,6 +1684,7 @@ export async function finalizeNativeRun(input: {
const tx = transaction as unknown as Db;
if (input.projectRunStatus) await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, run.companyId);
await notifyExecutionRunWork(tx);
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId: coordinator.issueId });
return projectRun(tx);
});
if (
@@ -8,6 +8,7 @@ import { logger } from "../../middleware/logger.js";
import { issueService } from "../issues.js";
import { issueThreadInteractionService } from "../issue-thread-interactions.js";
import { enqueueTerminalIssueInteractionChatPublications } from "../chat-interaction-publications.js";
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { persistActivity, publishActivity, type ActivityPublication } from "../activity-log.js";
const obsoletePrompt = "Review the superseding native policy assessment.";
@@ -68,6 +69,7 @@ export async function dismissObsoleteNativePolicyReviews(db: Db, runIds?: string
eq(issues.id, decision.issueId), eq(issues.companyId, decision.companyId),
)).for("update").then((rows) => rows[0]);
if (!issue) return;
await notifyChatRunMilestones(tx, interaction);
const now = new Date();
const [cancelled] = await tx.update(issueThreadInteractions).set({
status: "cancelled",
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { scheduledIssueMonitorWait } from "../issue-monitors.js";
import { onboardingFirstTaskCompletionAllowed } from "../onboarding-first-task-admission.js";
import { hasPublishedGitHubReview, readInheritedGitHubReviewWait } from "../chat-channels/github/review-wait.js";
@@ -127,6 +128,7 @@ export async function dispatchPendingNativeStatusEffects(input: {
targetId: effect.targetId,
});
}
await notifyChatRunMilestones(tx, input);
await tx
.update(statusDecisionEffects)
.set({
@@ -1973,6 +1975,7 @@ export async function commitNativeStatusDecision(input: {
await recordCoordinatorDecision(decisionRow.id);
return { decision: decisionRow, issue, replayed: true };
}
await notifyChatRunMilestones(tx, input);
if (!decisionRow) {
const latestDecisionVersion = await tx
.select({
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { and, asc, desc, eq, inArray, isNull, lte, or, sql } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
@@ -454,6 +455,9 @@ export function questionResponseDeliveryService(
current.lastAttemptAt.getTime() > claimAt.getTime() - claimStaleMs
)
return null;
if (current.status !== "delivering") {
await notifyChatRunMilestones(tx, { companyId: current.companyId, issueId: current.issueId });
}
return tx
.update(issueQuestionResponseDeliveries)
.set({
@@ -504,6 +508,7 @@ export function questionResponseDeliveryService(
}): Promise<QuestionResponseDeliveryOutcome> {
const at = now();
const updated = await db.transaction(async (tx) => {
await notifyChatRunMilestones(tx, { companyId: input.delivery.companyId, issueId: input.delivery.issueId });
const row = await tx
.update(issueQuestionResponseDeliveries)
.set({
@@ -591,25 +596,28 @@ export function questionResponseDeliveryService(
const nextErrorCount = delivery.errorCount + (options.bounded ? 1 : 0);
const exhausted =
options.bounded && nextErrorCount >= MAX_DELIVERY_ATTEMPTS;
await db
.update(issueQuestionResponseDeliveries)
.set({
// Keep an exhausted claim owned until recordTerminal commits its outcome.
status: exhausted ? "delivering" : "pending",
...(options.bounded ? { errorCount: nextErrorCount } : {}),
lastErrorCode: errorCode,
updatedAt: at,
})
.where(
and(
eq(issueQuestionResponseDeliveries.id, delivery.id),
eq(issueQuestionResponseDeliveries.status, "delivering"),
eq(
issueQuestionResponseDeliveries.attemptCount,
delivery.attemptCount,
),
),
await db.transaction(async (tx) => {
const ownedClaim = and(
eq(issueQuestionResponseDeliveries.id, delivery.id),
eq(issueQuestionResponseDeliveries.status, "delivering"),
eq(issueQuestionResponseDeliveries.attemptCount, delivery.attemptCount),
);
const [current] = await tx.select({ id: issueQuestionResponseDeliveries.id })
.from(issueQuestionResponseDeliveries).where(ownedClaim).for("update");
if (!current) return;
if (!exhausted) {
await notifyChatRunMilestones(tx, { companyId: delivery.companyId, issueId: delivery.issueId });
}
await tx.update(issueQuestionResponseDeliveries)
.set({
// Keep an exhausted claim owned until recordTerminal commits its outcome.
status: exhausted ? "delivering" : "pending",
...(options.bounded ? { errorCount: nextErrorCount } : {}),
lastErrorCode: errorCode,
updatedAt: at,
})
.where(ownedClaim);
});
return exhausted;
}
@@ -1301,21 +1309,32 @@ export function questionResponseDeliveryService(
async function sweepPending(limit = 50) {
const sweepAt = now();
const staleAt = new Date(sweepAt.getTime() - claimStaleMs);
await db
.update(issueQuestionResponseDeliveries)
.set({
status: "pending",
updatedAt: sweepAt,
})
.where(
and(
eq(issueQuestionResponseDeliveries.status, "delivering"),
or(
isNull(issueQuestionResponseDeliveries.lastAttemptAt),
lte(issueQuestionResponseDeliveries.lastAttemptAt, staleAt),
),
await db.transaction(async (tx) => {
const staleClaims = await tx.select({
id: issueQuestionResponseDeliveries.id,
companyId: issueQuestionResponseDeliveries.companyId,
issueId: issueQuestionResponseDeliveries.issueId,
}).from(issueQuestionResponseDeliveries).where(and(
eq(issueQuestionResponseDeliveries.status, "delivering"),
or(
isNull(issueQuestionResponseDeliveries.lastAttemptAt),
lte(issueQuestionResponseDeliveries.lastAttemptAt, staleAt),
),
);
)).orderBy(asc(issueQuestionResponseDeliveries.id)).for("update");
if (staleClaims.length === 0) return;
const notified = new Set<string>();
for (const claim of staleClaims) {
const scope = `${claim.companyId}:${claim.issueId}`;
if (notified.has(scope)) continue;
await notifyChatRunMilestones(tx, claim);
notified.add(scope);
}
for (let offset = 0; offset < staleClaims.length; offset += 500) {
await tx.update(issueQuestionResponseDeliveries)
.set({ status: "pending", updatedAt: sweepAt })
.where(inArray(issueQuestionResponseDeliveries.id, staleClaims.slice(offset, offset + 500).map(claim => claim.id)));
}
});
const ids = await db
.select({ interactionId: issueQuestionResponseDeliveries.interactionId })
.from(issueQuestionResponseDeliveries)
+92 -72
View File
@@ -1,3 +1,5 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { boundRunIssueId } from "../run-task-binding.js";
import { recoveryRunContextColumn, recoveryRunVersionColumn } from "./run-context-read.js";
import { recoveryRunResultColumn } from "./run-result-read.js";
import { readLegacyDispositionSource } from "./legacy-disposition-read.js";
@@ -4340,6 +4342,7 @@ export function recoveryService(
}
async function persistAdapterFailureRecoveryClassification(
issue: Pick<typeof issues.$inferSelect, "id" | "companyId">,
latestRun: NonNullable<LatestIssueRun>,
classification: NonNullable<AdapterFailureRecoveryClassification>,
): Promise<NonNullable<LatestIssueRun>> {
@@ -4351,17 +4354,20 @@ export function recoveryService(
const classificationMetadata = withAdapterFailureRecoveryClassification(
{ ...latestRun, resultJson: {} }, classification,
).resultJson;
const [updated] = await db
.update(heartbeatRuns)
.set({
errorCode: classifiedRun.errorCode,
// Classification owns retry metadata, not workspace repair receipts
// that may have committed after this run was selected.
resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify(classificationMetadata)}::jsonb`,
updatedAt: new Date(),
})
.where(eq(heartbeatRuns.id, latestRun.id))
.returning({ resultJson: heartbeatRuns.resultJson, errorCode: heartbeatRuns.errorCode });
const [updated] = await db.transaction(async tx => {
await notifyChatRunMilestones(tx, { companyId: issue.companyId, issueId: issue.id });
return tx
.update(heartbeatRuns)
.set({
errorCode: classifiedRun.errorCode,
// Classification owns retry metadata, not workspace repair receipts
// that may have committed after this run was selected.
resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify(classificationMetadata)}::jsonb`,
updatedAt: new Date(),
})
.where(and(eq(heartbeatRuns.id, latestRun.id), eq(heartbeatRuns.companyId, issue.companyId)))
.returning({ resultJson: heartbeatRuns.resultJson, errorCode: heartbeatRuns.errorCode });
});
return updated ? { ...classifiedRun, ...updated } : classifiedRun;
}
@@ -4960,6 +4966,7 @@ export function recoveryService(
});
if (monitored) {
latestRun = await persistAdapterFailureRecoveryClassification(
issue,
latestRun,
adapterFailureClassification,
);
@@ -4981,6 +4988,7 @@ export function recoveryService(
});
if (updated) {
latestRun = await persistAdapterFailureRecoveryClassification(
issue,
latestRun,
adapterFailureClassification,
);
@@ -5184,6 +5192,7 @@ export function recoveryService(
});
if (monitored) {
latestRun = await persistAdapterFailureRecoveryClassification(
issue,
participantLatestRun,
participantAdapterFailureClassification,
);
@@ -5210,6 +5219,7 @@ export function recoveryService(
});
if (updated) {
latestRun = await persistAdapterFailureRecoveryClassification(
issue,
participantLatestRun,
participantAdapterFailureClassification,
);
@@ -6179,70 +6189,80 @@ export function recoveryService(
await deps.beforeOrphanedRunTerminalWrite?.(run.id);
const now = new Date();
const updated = await db
.update(heartbeatRuns)
.set({
status: terminalStatus,
finishedAt: run.finishedAt ?? now,
error: run.error ?? (terminalStatus === "interrupted" ? message : null),
errorCode:
run.errorCode ??
(terminalStatus === "interrupted" ? errorCode : null),
updatedAt: now,
})
.where(
and(
eq(heartbeatRuns.id, run.id),
eq(heartbeatRuns.status, "running"),
eq(heartbeatRuns.runtimeMode, run.runtimeMode),
nativeRunnerOwnershipNotHeldCondition(),
// Provider exit is expected while the native coordinator resumes a
// session or copies its completed workspace back. The coordinator
// owns those retries, including expired leases and future attempts.
// Check at the write so a newly recorded result cannot be orphaned
// using the earlier liveness snapshot. Terminal issue status remains
// the stronger authority.
!issueTerminalStatus && run.runtimeMode === "native"
? notExists(db.select({ runId: nativeRunFinalizations.runId })
.from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.runId, heartbeatRuns.id),
eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId),
or(
and(
isNotNull(nativeRunFinalizations.resultId),
inArray(nativeRunFinalizations.phase, [
"observed", "workspace_finalizing", "ready_for_assessment",
"arbitrating", "retryable_failure",
]),
),
and(
isNull(nativeRunFinalizations.resultId),
or(
eq(nativeRunFinalizations.phase, "retryable_failure"),
and(eq(nativeRunFinalizations.phase, "observed"), gt(nativeRunFinalizations.attempt, 0)),
),
const updated = await db.transaction(async tx => {
const eligible = and(
eq(heartbeatRuns.id, run.id),
eq(heartbeatRuns.status, "running"),
eq(heartbeatRuns.runtimeMode, run.runtimeMode),
nativeRunnerOwnershipNotHeldCondition(),
// Provider exit is expected while the native coordinator resumes a
// session or copies its completed workspace back. The coordinator
// owns those retries, including expired leases and future attempts.
// Check at the write so a newly recorded result cannot be orphaned
// using the earlier liveness snapshot. Terminal issue status remains
// the stronger authority.
!issueTerminalStatus && run.runtimeMode === "native"
? notExists(tx.select({ runId: nativeRunFinalizations.runId })
.from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.runId, heartbeatRuns.id),
eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId),
or(
and(
isNotNull(nativeRunFinalizations.resultId),
inArray(nativeRunFinalizations.phase, [
"observed", "workspace_finalizing", "ready_for_assessment",
"arbitrating", "retryable_failure",
]),
),
and(
isNull(nativeRunFinalizations.resultId),
or(
eq(nativeRunFinalizations.phase, "retryable_failure"),
and(eq(nativeRunFinalizations.phase, "observed"), gt(nativeRunFinalizations.attempt, 0)),
),
),
)))
: undefined,
// Recheck ownership in the write: a controller can renew or claim
// the run after the liveness read. An old snapshot cannot end a new
// controller's run, even if that controller's lease later expires.
run.runtimeMode === "legacy"
? and(
run.controllerBootId
? eq(heartbeatRuns.controllerBootId, run.controllerBootId)
: isNull(heartbeatRuns.controllerBootId),
or(
isNull(heartbeatRuns.controllerBootId),
sql`${heartbeatRuns.controllerLeaseExpiresAt} <= clock_timestamp()`,
),
)
: undefined,
),
)
.returning()
.then((rows) => rows[0] ?? null);
)))
: undefined,
// Recheck ownership in the write: a controller can renew or claim
// the run after the liveness read. An old snapshot cannot end a new
// controller's run, even if that controller's lease later expires.
run.runtimeMode === "legacy"
? and(
run.controllerBootId
? eq(heartbeatRuns.controllerBootId, run.controllerBootId)
: isNull(heartbeatRuns.controllerBootId),
or(
isNull(heartbeatRuns.controllerBootId),
sql`${heartbeatRuns.controllerLeaseExpiresAt} <= clock_timestamp()`,
),
)
: undefined,
);
// Do not wake projection for a retained native finalization or a run
// whose execution owner changed. Keep the same predicate on the write
// to recheck finalization records committed after this lock was taken.
const [candidate] = await tx.select().from(heartbeatRuns).where(eligible).for("update");
if (!candidate) return null;
const boundIssueId = boundRunIssueId(candidate);
if (boundIssueId) {
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId: boundIssueId });
}
return tx
.update(heartbeatRuns)
.set({
status: terminalStatus,
finishedAt: run.finishedAt ?? now,
error: run.error ?? (terminalStatus === "interrupted" ? message : null),
errorCode:
run.errorCode ??
(terminalStatus === "interrupted" ? errorCode : null),
updatedAt: now,
})
.where(eligible)
.returning()
.then((rows) => rows[0] ?? null);
});
if (!updated) {
// Another path finalized the run between the read and this write. Keep
// that terminal outcome authoritative.
+10 -1
View File
@@ -11,7 +11,8 @@ import {
type Db,
} from "@paperclipai/db";
import { conflict, forbidden } from "../errors.js";
import { boundRunIssueId, runTaskContextColumn } from "./run-task-binding.js";
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { boundRunIssueId, boundRunIssueIdSql, runTaskContextColumn } from "./run-task-binding.js";
import { isUuidLike } from "@paperclipai/shared";
import { queuedCommentIdsFromRunContext, queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js";
@@ -138,6 +139,13 @@ async function append(
}
async function activate(executor: Executor, context: RunIdentityContext) {
const [binding] = await executor.select({ issueId: boundRunIssueIdSql() })
.from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, context.runId), eq(heartbeatRuns.companyId, context.companyId),
));
if (binding?.issueId) {
await notifyChatRunMilestones(executor, { companyId: context.companyId, issueId: binding.issueId });
}
const [run] = await executor
.update(heartbeatRuns)
.set({
@@ -447,6 +455,7 @@ export async function rejectSteeredIdentity(
);
}
/** Accept within the caller's task/run transaction; standalone recovery uses reconcileSteeredIdentity. */
export async function acceptSteeredIdentity(
executor: Executor,
context: RunIdentityContext,
+2
View File
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
import { updateAgentConfigurationInTransaction } from "./agent-configuration-transaction.js";
import { withAgentAppearance } from "@paperclipai/shared";
@@ -290,6 +291,7 @@ export function createSecretProposalsService(db: Db) {
: outcome === "rejected"
? "rejected"
: "withdrawn";
if (current.status !== status) await notifyChatRunMilestones(txDb, current);
await txDb
.update(issueThreadInteractions)
.set({
+1 -1
View File
@@ -4665,7 +4665,7 @@ export function toolAccessService(
.returning();
const [row] = existing.transport === "chat_sdk"
? await db.transaction(async tx => {
await notifyChatEndpointWork(tx, existing.id);
await notifyChatEndpointWork(tx, existing.id, existing.companyId);
const rows = await write(tx as unknown as Db);
// Credential resolution must never observe the binding rebuild's
// delete/insert gap after a configuration wake.
@@ -541,7 +541,7 @@ async function removeConnection(
.returning({ id: connectionTokenIssuances.id });
const archived = await db.transaction(async (tx) => {
if (connection.transport === "chat_sdk") await notifyChatEndpointWork(tx, connection.id);
if (connection.transport === "chat_sdk") await notifyChatEndpointWork(tx, connection.id, connection.companyId);
if (connection.transport === "voice") await notifyVoiceChange(tx, { kind: "connection", id: connection.id });
await notifyToolConnectionHealth(tx, connection);
await tx.select({ id: toolApplications.id }).from(toolApplications)
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { notifyToolReviewChange } from "./tool-review-work-notifications.js";
import { notifyDeliveryWork, DELIVERY_QUEUES } from "./delivery-work-notifications.js";
import { and, eq } from "drizzle-orm";
@@ -141,6 +142,7 @@ export async function commitToolActionReview(
throw conflict("This review has already been resolved");
}
const now = new Date();
if (interaction && issue) await notifyChatRunMilestones(tx, interaction);
await notifyToolReviewChange(tx, current.companyId);
const [updated] = await tx
.update(toolActionRequests)
+4
View File
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { writeToolConnectionHealthChange } from "./tool-connection-health-notifications.js";
import { writeApprovalChange } from "./approval-work-notifications.js";
import { toolReviewProjectionService } from "./tool-review-projection.js";
@@ -2408,6 +2409,9 @@ export function createToolGatewayService(
&& projection.resultSummary === (input.resultSummary ?? null)) return false;
const now = new Date();
if (linked.issueId && interaction.status === "pending" && ["expired", "cancelled"].includes(input.status)) {
await notifyChatRunMilestones(tx, { companyId: linked.companyId, issueId: linked.issueId });
}
await tx
.update(issueThreadInteractions)
.set({