fix(server): wake chat lifecycle milestones from committed changes (#17606)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connected chats show safe notices when a run starts, waits,
finishes, or fails.
> - A recurring scan projected these notices even when no eligible work
existed.
> - Eligibility depends on committed run state, conversation lineage,
and current authority.
> - Existing database rows can reconstruct that work after restart.
> - This PR wakes a scoped worker from those writes and removes
lifecycle polling.
> - Native progress events keep their separate existing owner until the
next migration.

## Linked Issues or Issue Description

**What happened?**

The chat maintenance timer repeatedly scanned run lifecycle state. Empty
and settled instances still queried the database. A run-status event
alone cannot replace the scan: prompt receipts, continuation receipts,
identity changes, and restored authority can make a run eligible later.

**What did you expect?**

Committed changes wake projection. Startup reconstructs eligible work. A
completed pass stops until another relevant change arrives. Failed work
retains recovery without scanning settled history forever.

**Steps to reproduce**

Configure an automatic chat endpoint. Finish its work, or leave it
unused. Observe recurring lifecycle projection queries from the chat
maintenance callback.

Stacked on #17592. Related foundation: #15625. This PR owns lifecycle
milestones only.

## What Changed

- Preserve upstream coalesced Cloud membership writes together with
locked membership restoration and its committed chat notification.

- Signal milestone eligibility from the real Slack account-link
transaction.

- Register the lifecycle worker with the existing delivery coordinator.
Remove lifecycle projection from the periodic chat callback. Keep native
progress in a separately named lane and preserve the explicit combined
API.
- Reconstruct company/issue scopes from the existing selector. Drain
bounded batches with company fairness. Retry failures, retain writes
through drain/resume, and disarm settled work.
- Signal from supported run, presentation, interaction, receipt,
lineage, conversation, recovery, and authority writers. Preserve their
transaction ownership, including nested savepoints and uncertain
settlement. No new table is added.
- Preserve provider authorization, authored-final suppression,
responsible-user continuity, deduplication, the 24-hour selection
window, and raw-output restrictions. Keep provider interpretation in its
owner.
- Preserve PostgreSQL timestamp precision in lifecycle pagination.
Rejected submillisecond candidates must not repeat the same page and
starve later notices.
- Document producer coverage and alternate-writer dispositions in
`doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md`.

## Verification

Published head: `1d056f68c5b1005933b9e12de8745bf1c4cbeeca`, targeting
master after #17592 merged at
`7d75ae0569e41786640cf09647ae8849411d2f22`. A normal merge integrates
the actual landed parent. The entire 77-path milestone patch is
binary-identical to the locally verified preparation across that
ancestry reconciliation; both unrelated master paths are preserved. The
four earlier producer/test integration conflicts retain scoped milestone
and voice notices, current onboarding completion admission, and expanded
identity query/read-scope coverage.

- **330 distinct selected cases pass on the verified unchanged patch:**
305 across 14 complete suites (all five milestone worker suites, run
identity, board claim, Cloud provisioning, auth session routes, Cloud
actor authentication, stale-lock recovery, GitHub credentials, project
Git authorization, and app admission wiring), plus 25 selected real chat
producer cases. The remaining 1,590 chat integration cases were skipped,
not passed.
- The real producer cases cover saved milestone discovery, GitHub and
Slack identity changes, account-link rollback/reconfirmation, ordinary
ingress reactivation, post-close action/delivery receipt settlement, all
ten authority-restoration paths including Cloud synchronization, and
changed native review metadata.
- Worker coverage verifies outer commit, rollback, uncertain-write
recovery, restart, bounded batching/fairness, timestamp precision,
retries, drain/resume, and settled SQL quiet. Authorization and existing
publication deduplication remain with their owning services.
- Server production typechecking, build, and module-boundary checks
pass. The tests above were executed separately. Production typechecking
excludes `src/__tests__` and does not replace those runs. Each database
owner uses its own newly created and app-migrated PostgreSQL fixture.
- [Current-parent acceptance and preserved failure
ledger](https://github.com/paperclipai/paperclip/blob/1d056f68c5b1005933b9e12de8745bf1c4cbeeca/doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md#current-parent-integration-and-acceptance-october-28)
records commands, counts, producer scope, conflict dispositions, and
limits. Earlier combined-stack counts and failures keep their original
revision scope; local acceptance does not erase old CI failures.
- No new migration, queue table, dependency, workflow, or shared
scheduler contract is introduced. Parent migration 0397, bounded consent
recovery, voice cleanup/retention repairs, and runner checkpoint
restoration remain present.
- **Exact-head Greptile is 5/5 and all 55 current checks are clean.** No
Ponytail review is present, which is an explicitly authorized
nonblocker; no failing review is waived. Older reviews and parent checks
are not current passes. Actual failing Ponytail findings must be fixed;
an absent/stopped/paused turn without a verdict is an authorized
nonblocker. Active unfinished Ponytail uses its original request/start
timer, with a 25-minute monitor cutoff respecting the user's 30-minute
maximum.
- No complete local repository matrix, live provider tenant, external
alarm, physical host/database suspension, or fleet savings claim is
made. Native progress remains a separate following slice.

## Risks

- A future eligibility writer must notify the existing projection
boundary. Independent SQL writers or another active server process are
outside this process-local signal contract.
- Authority and lineage remain sensitive. The worker uses the existing
selectors and publication policy; it does not grant execution or
disclosure authority.
- Native progress and other publication maintenance still have recurring
work. This PR removes one reason, not the entire chat maintenance timer.
- Worker quiet does not prove database suspension, host sleep, or
external alarm coverage. Cloud sleep blockers remain unchanged.
- Merge eligibility requires passing current-head CI and Greptile 5/5
after this actual-master reconciliation. Subsequent stack slices retain
their own gates.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and
context-window size are not exposed. Used repository inspection,
reasoning, shell execution, PostgreSQL tests, browser E2Es and GitHub
tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details available in this session)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run the documented focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-28 11:07:54 -05:00
1 parent 143d20e466
commit 5f0da562db
77 files changed
+5002 -501

No files matched your search

@@ -25,7 +25,7 @@ describe("createChatReconciliationCoordinator", () => {
processFailedGitHubWebhookDeliveries,
processPendingDeliveries,
flushPublications,
projectRunMilestones: async () => 0,
projectNativeProgress: async () => 0,
processPendingSlackFileUploadReceipts: async () => undefined,
processPendingSlackSessionSyncs: async () => undefined,
onError,
@@ -142,14 +142,14 @@ describe("createChatReconciliationCoordinator", () => {
});
it.each([false, true])(
"isolates a blocked milestone projector and joins its completion (rejects: %s)",
"isolates a blocked native progress projector and joins its completion (rejects: %s)",
async (rejects) => {
let release!: () => void;
const held = new Promise<void>((resolve) => {
release = resolve;
});
const failure = new Error("milestone projection failed");
const projectRunMilestones = vi
const failure = new Error("native progress projection failed");
const projectNativeProgress = vi
.fn(async () => 0)
.mockImplementationOnce(async () => {
await held;
@@ -159,7 +159,7 @@ describe("createChatReconciliationCoordinator", () => {
const flushPublications = vi.fn(async () => undefined);
const onError = vi.fn();
const coordinator = createChatReconciliationCoordinator({
projectRunMilestones,
projectNativeProgress,
processPendingDeliveries: async () => undefined,
flushPublications,
processPendingSlackFileUploadReceipts: async () => undefined,
@@ -169,7 +169,7 @@ describe("createChatReconciliationCoordinator", () => {
try {
coordinator.reconcile();
await vi.waitFor(() => {
expect(projectRunMilestones).toHaveBeenCalledTimes(1);
expect(projectNativeProgress).toHaveBeenCalledTimes(1);
expect(flushPublications).toHaveBeenCalledTimes(1);
});
// A newly committed final/question must dispatch even while a different
@@ -179,7 +179,7 @@ describe("createChatReconciliationCoordinator", () => {
await vi.waitFor(() =>
expect(flushPublications).toHaveBeenCalledTimes(2),
);
expect(projectRunMilestones).toHaveBeenCalledTimes(1);
expect(projectNativeProgress).toHaveBeenCalledTimes(1);
let drained = false;
const draining = coordinator.drain().then(() => {
drained = true;
@@ -188,17 +188,17 @@ describe("createChatReconciliationCoordinator", () => {
expect(drained).toBe(false);
release();
await draining;
expect(projectRunMilestones).toHaveBeenCalledTimes(2);
expect(projectNativeProgress).toHaveBeenCalledTimes(2);
expect(onError).toHaveBeenCalledTimes(rejects ? 1 : 0);
if (rejects)
expect(onError).toHaveBeenCalledWith("run milestones", failure);
expect(onError).toHaveBeenCalledWith("native progress", failure);
const completedCounts = [
projectRunMilestones.mock.calls.length,
projectNativeProgress.mock.calls.length,
flushPublications.mock.calls.length,
];
await new Promise<void>((resolve) => setImmediate(resolve));
expect([
projectRunMilestones.mock.calls.length,
projectNativeProgress.mock.calls.length,
flushPublications.mock.calls.length,
]).toEqual(completedCounts);
} finally {
@@ -209,18 +209,18 @@ describe("createChatReconciliationCoordinator", () => {
},
);
it("joins a blocked milestone projector at shutdown without starting dirty follow-ups", async () => {
it("joins a blocked native progress projector at shutdown without starting dirty follow-ups", async () => {
let release!: () => void;
const held = new Promise<void>((resolve) => {
release = resolve;
});
const projectRunMilestones = vi.fn(async () => {
const projectNativeProgress = vi.fn(async () => {
await held;
return 1;
});
const flushPublications = vi.fn(async () => undefined);
const coordinator = createChatReconciliationCoordinator({
projectRunMilestones,
projectNativeProgress,
flushPublications,
processPendingDeliveries: async () => undefined,
processPendingSlackFileUploadReceipts: async () => undefined,
@@ -230,7 +230,7 @@ describe("createChatReconciliationCoordinator", () => {
try {
coordinator.reconcile();
await vi.waitFor(() => {
expect(projectRunMilestones).toHaveBeenCalledOnce();
expect(projectNativeProgress).toHaveBeenCalledOnce();
expect(flushPublications).toHaveBeenCalledOnce();
});
coordinator.notifyPublications();
@@ -243,7 +243,7 @@ describe("createChatReconciliationCoordinator", () => {
expect(drained).toBe(false);
release();
await draining;
expect(projectRunMilestones).toHaveBeenCalledOnce();
expect(projectNativeProgress).toHaveBeenCalledOnce();
expect(flushPublications).toHaveBeenCalledOnce();
} finally {
release();
@@ -269,7 +269,7 @@ describe("createChatReconciliationCoordinator", () => {
);
const onError = vi.fn();
const coordinator = createChatReconciliationCoordinator({
projectRunMilestones: async () => 0,
projectNativeProgress: async () => 0,
processPendingDeliveries,
flushPublications,
processPendingSlackFileUploadReceipts,
@@ -317,7 +317,7 @@ describe("createChatReconciliationCoordinator", () => {
const processPendingSlackSessionSyncs = vi.fn(async () => undefined);
const onError = vi.fn();
const coordinator = createChatReconciliationCoordinator({
projectRunMilestones: async () => 0,
projectNativeProgress: async () => 0,
processPendingDeliveries,
flushPublications,
processPendingSlackFileUploadReceipts,
@@ -10,12 +10,16 @@ import {
import { errorHandler } from "../middleware/error-handler.js";
import { assertCompanyAccess } from "../routes/authz.js";
function queryResult<T>(rows: T) {
return Object.assign(Promise.resolve(rows), { for: () => Promise.resolve(rows) });
}
function createSelectChain(rows: unknown[]) {
return {
from() {
return {
where() {
return Promise.resolve(rows);
return queryResult(rows);
},
};
},
@@ -102,6 +106,7 @@ describe("actorMiddleware authenticated session profile", () => {
delete: vi.fn(() => ({ where: () => Promise.resolve(undefined) })),
select: vi.fn(() => createSelectChain([])),
} as any;
db.transaction = vi.fn(async (run: (tx: typeof db) => Promise<unknown>) => run(db));
const app = express();
app.use(
actorMiddleware(db, {
@@ -176,7 +181,7 @@ describe("actorMiddleware authenticated session profile", () => {
select: vi.fn(() => ({
from: (table: unknown) => ({
where: () =>
Promise.resolve(
queryResult(
table === companyMemberships
? [{ companyId: importedCompanyId, membershipRole: "member", status: "active" }]
: [],
@@ -186,6 +191,7 @@ describe("actorMiddleware authenticated session profile", () => {
insert: vi.fn(() => insertChain),
delete: vi.fn(() => ({ where: () => Promise.resolve(undefined) })),
} as any;
db.transaction = vi.fn(async (run: (tx: typeof db) => Promise<unknown>) => run(db));
const app = express();
app.use(
actorMiddleware(db, {
@@ -252,7 +258,7 @@ describe("actorMiddleware authenticated session profile", () => {
select: vi.fn(() => ({
from: (table: unknown) => ({
where: () =>
Promise.resolve(
queryResult(
table === companies
? [{ name: "paperclip-stack-purple-rain" }]
: [],
@@ -280,7 +286,7 @@ describe("actorMiddleware authenticated session profile", () => {
})),
delete: vi.fn(() => ({ where: () => Promise.resolve(undefined) })),
} as any;
db.transaction = vi.fn(async (run: (tx: typeof db) => Promise<void>) => run(db));
db.transaction = vi.fn(async (run: (tx: typeof db) => Promise<unknown>) => run(db));
const app = express();
app.use(
actorMiddleware(db, {
@@ -349,7 +355,7 @@ describe("actorMiddleware authenticated session profile", () => {
select: vi.fn(() => ({
from: (table: unknown) => ({
where: () =>
Promise.resolve(
queryResult(
table === instanceUserRoles && state.staleInstanceAdminRow ? [{ id: "stale-role-row" }] : [],
),
}),
@@ -362,6 +368,7 @@ describe("actorMiddleware authenticated session profile", () => {
},
})),
} as any;
db.transaction = vi.fn(async (run: (tx: typeof db) => Promise<unknown>) => run(db));
const app = express();
app.use(
actorMiddleware(db, {
+26 -10
View File
@@ -1,3 +1,5 @@
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
import { randomUUID } from "node:crypto";
import { and, eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
@@ -8,6 +10,7 @@ import {
createDb,
instanceUserRoles,
principalPermissionGrants,
subscribeDatabaseWork,
} from "@paperclipai/db";
import {
claimBoardOwnership,
@@ -20,7 +23,8 @@ import {
startEmbeddedPostgresTestDatabase,
} from "./helpers/embedded-postgres.js";
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_BOARD_CLAIM_TEST_DATABASE_URL");
const embeddedPostgresSupport = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
describeEmbeddedPostgres("board claim", () => {
@@ -28,8 +32,8 @@ describeEmbeddedPostgres("board claim", () => {
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
beforeAll(async () => {
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-board-claim-");
db = createDb(tempDb.connectionString);
if (!external) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-board-claim-");
db = createDb(external ?? tempDb!.connectionString);
}, 20_000);
afterEach(async () => {
@@ -42,10 +46,11 @@ describeEmbeddedPostgres("board claim", () => {
});
afterAll(async () => {
await db?.$client.end({ timeout: 1 });
await tempDb?.cleanup();
});
it("lets a signed-in user claim a local-board-only authenticated instance", async () => {
it.each(["new", "suspended", "active"] as const)("claims a local-board-only instance with %s membership and wakes only restored authority", async prior => {
const now = new Date();
const userId = `claim-user-${randomUUID()}`;
const company = await db
@@ -70,6 +75,10 @@ describeEmbeddedPostgres("board claim", () => {
role: "instance_admin",
});
if (prior !== "new") await db.insert(companyMemberships).values({
companyId: company.id, principalType: "user", principalId: userId,
status: prior, membershipRole: "owner",
});
await initializeBoardClaimChallenge(db, { deploymentMode: "authenticated" });
const warningUrl = getBoardClaimWarningUrl("127.0.0.1", 3197);
expect(warningUrl).toBeTruthy();
@@ -84,12 +93,19 @@ describeEmbeddedPostgres("board claim", () => {
claimedByUserId: null,
});
await expect(
claimBoardOwnership(db, { token, code, userId }),
).resolves.toEqual({
status: "claimed",
claimedByUserId: userId,
});
const events: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, event => events.push(event));
try {
await db.transaction(async tx => {
await expect(claimBoardOwnership(tx as unknown as typeof db, { token, code, userId }))
.resolves.toEqual({ status: "claimed", claimedByUserId: userId });
expect(events).toEqual(prior === "active" ? [] : ["intent"]);
// A nested claim must not publish a commit before its caller settles.
const outside = await db.select().from(companyMemberships).where(eq(companyMemberships.companyId, company.id));
expect(outside.map(row => row.status)).toEqual(prior === "new" ? [] : [prior]);
});
expect(events).toEqual(prior === "active" ? [] : ["intent", "settled"]);
} finally { unsubscribe(); }
await expect(
db
@@ -47,7 +47,7 @@ describe("chat action scheduling", () => {
const work = s.start();
await vi.advanceTimersByTimeAsync(1);
expect(s.service.processTelegramSubscriptionDiscovery).toHaveBeenCalledTimes(1);
await s.owner.transaction(tx => notifyChatEndpointWork(tx, "connection-fixture"));
await s.owner.transaction(tx => notifyChatEndpointWork(tx, "connection-fixture", "company-fixture"));
expect(s.service.scheduleTelegramSubscriptionDiscovery).toHaveBeenCalledWith("connection-fixture");
s.service.processTelegramSubscriptionDiscovery.mockRejectedValueOnce(new Error("discovery unavailable"));
release();
@@ -87,7 +87,7 @@ describe("chat action scheduling", () => {
for (const run of s.runs) expect(run).toHaveBeenCalledTimes(2);
// Queue claims/settlements do not rediscover every configured endpoint.
expect(s.service.processTelegramSubscriptionDiscovery).toHaveBeenCalledTimes(1);
await s.owner.transaction(tx => notifyChatEndpointWork(tx, "connection-fixture"));
await s.owner.transaction(tx => notifyChatEndpointWork(tx, "connection-fixture", "company-fixture"));
await vi.advanceTimersByTimeAsync(1);
for (const run of s.runs) expect(run).toHaveBeenCalledTimes(3);
expect(s.service.processTelegramSubscriptionDiscovery).toHaveBeenCalledTimes(2);
@@ -1,3 +1,8 @@
import { cloudActorHeaderSourceFromHeaders, resolveCloudTenantActor } from "../middleware/auth.js";
import { cloudTenantPrimaryCompanyId } from "../services/cloud-instance.js";
import { accessService } from "../services/access.js";
import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db";
import { restoreNativeChatReviewPresentationInTransaction } from "../services/native-runtime/native-chat-review-presentation.js";
import { notifyChatEndpointConfiguration, notifyChatCompanyConfiguration } from "../services/chat-channels/configuration-work-notifications.js";
import { deleteCompany } from "../services/company-deletion.js";
import { adapterExecutionControls, createAdapterExecutionControl } from "../services/adapter-execution-control.js";
@@ -23,6 +28,7 @@ import { chatSlackManagerGrants, chatSlackRegistrations, toolOauthStates, enviro
import type { PaperclipCloudConnector, SealedConnectorCredentials } from "../services/paperclip-cloud-connector.js";
import { buildSlackAppManifest, SLACK_MANAGER_SCOPES } from "@paperclipai/shared";
import { registerChatActionWork, dueChatAction } from "../services/chat-action-work.js";
import { chatRunMilestoneWork } from "../services/chat-run-milestone-work.js";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { registerChatDeliveryWork } from "../services/chat-delivery-work.js";
import { notifyChatPublicationWork, notifyChatActionWork, CHAT_ACTION_QUEUES } from "../services/chat-work-notifications.js";
@@ -202,6 +208,8 @@ import {
} from "../services/chat-interaction-publications.js";
import {
enqueueChatRunMilestones,
enqueueChatRunLifecycleMilestones,
enqueueSafeNativeChatProgress,
githubRunReplyState,
resolveChatRunPresentationAuthorizationReason,
CHAT_RUN_PRESENTATION_AUTHORIZATION_REASON,
@@ -1144,8 +1152,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
const companyPrefixAttempts = 8;
const freshCompanyPrefix = () => `C${randomUUID().replaceAll("-", "").slice(0, 7).toUpperCase()}`;
async function seedCompany(nextIssuePrefix = freshCompanyPrefix) {
const companyId = randomUUID();
async function seedCompany(nextIssuePrefix = freshCompanyPrefix, companyId = randomUUID()) {
const assignedAgentId = randomUUID();
const replacementAgentId = randomUUID();
let inserted = false;
@@ -6029,6 +6036,68 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
.where(eq(chatEndpointLeases.endpointId, f.endpoint.id)),
).toHaveLength(0);
});
it("wakes GitHub milestone identity restoration only after outer commit and stays quiet on reconfirmation", async () => {
const f = await reviewBotFixture();
await f.service.revokeLink(f.endpoint.id, f.principal.id);
const verified = { githubUserId: "42", login: "octocat" };
const events: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, (event, scopes) => {
if ([...scopes ?? []].some(scope => JSON.parse(scope).companyId === f.companyId)) events.push(event);
});
const link = () => db.select().from(chatIdentityLinks).where(and(
eq(chatIdentityLinks.endpointId, f.endpoint.id), eq(chatIdentityLinks.principalId, f.principal.id),
)).then(rows => rows[0]);
try {
await expect(db.transaction(async tx => {
await githubChatManagementService(tx as unknown as typeof db, f.providerFetch)
.linkObservedIdentity(f.endpoint.id, "owner-user", verified);
expect(events).toEqual(["intent"]);
expect((await link()).status).toBe("revoked");
throw new Error("rollback identity restoration");
})).rejects.toThrow("rollback identity restoration");
expect((await link()).status).toBe("revoked");
events.length = 0;
await db.transaction(async tx => {
await githubChatManagementService(tx as unknown as typeof db, f.providerFetch)
.linkObservedIdentity(f.endpoint.id, "owner-user", verified);
expect(events).toEqual(["intent"]);
expect((await link()).status).toBe("revoked");
});
expect(events).toEqual(["intent", "settled"]);
expect((await link()).status).toBe("linked");
events.length = 0;
await f.management.linkObservedIdentity(f.endpoint.id, "owner-user", verified);
expect(events).toEqual([]);
} finally { unsubscribe(); }
});
it("wakes milestone authority only when GitHub guest settings change, after outer commit", async () => {
const f = await reviewBotFixture();
const initial = await f.management.configuration(f.endpoint.id, "owner-user");
const configuration = { ...initial.configuration, people: [{
kind: "guest" as const, githubUserId: "84", login: "contributor",
sponsorUserId: "owner-user", permissionProfile: "restricted" as const, automaticReviews: false,
}] };
const events: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, (event, scopes) => {
if ([...scopes ?? []].some(scope => JSON.parse(scope).companyId === f.companyId)) events.push(event);
});
try {
await db.transaction(async tx => {
await githubChatManagementService(tx as unknown as typeof db, f.providerFetch)
.saveConfiguration(f.endpoint.id, { expectedRevision: initial.revision, configuration }, "owner-user");
expect(events).toEqual(["intent"]);
expect((await f.service.get(f.endpoint.id)).allowUnlinkedPeople).toBe(false);
});
expect(events).toEqual(["intent", "settled"]);
expect((await f.service.get(f.endpoint.id)).allowUnlinkedPeople).toBe(true);
events.length = 0;
const current = await f.management.configuration(f.endpoint.id, "owner-user");
await f.management.saveConfiguration(f.endpoint.id, { expectedRevision: current.revision, configuration }, "owner-user");
expect(events).toEqual([]);
} finally { unsubscribe(); }
});
it("scopes guest tasks and rechecks their sponsor before every bot tool", async () => {
const f = await reviewBotFixture();
const initial = await f.management.configuration(
@@ -8295,11 +8364,62 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
expect(posts(f)).toHaveLength(1);
expect(new URLSearchParams(String(posts(f)[0]![1]!.body)).get("channel")).toBe(f.installerId);
});
it("links the actor's own manager identity without requiring a connect command", async () => {
it("signals milestone eligibility around the real Slack account link and stays quiet on reconfirmation", async () => {
const f = await managedFixture(); await f.provision();
await db.delete(chatIdentityLinks).where(eq(chatIdentityLinks.endpointId, f.endpoint.id));
expect(await f.service.sendSlackTestMessage(f.endpoint.id, actor, { requestId: randomUUID() })).toMatchObject({ status: "sent" });
expect(await db.select().from(chatIdentityLinks).where(eq(chatIdentityLinks.endpointId, f.endpoint.id))).toEqual([expect.objectContaining({ paperclipUserId: actor.userId, status: "linked" })]);
const events: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, (event, scopes) => {
if ([...scopes ?? []].some(scope => JSON.parse(scope).companyId === f.companyId)) events.push(event);
});
let release!: () => void, acquired!: () => void;
const gate = new Promise<void>(resolve => { release = resolve; });
const locked = new Promise<void>(resolve => { acquired = resolve; });
const lock = db.transaction(async tx => {
await tx.execute(sql`lock table chat_identity_links in share mode`);
acquired(); await gate;
});
await locked;
const linking = f.service.sendSlackTestMessage(f.endpoint.id, actor, { requestId: randomUUID() });
void linking.catch(() => undefined);
try {
try {
await vi.waitFor(async () => {
const [waiting] = await db.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity
where datname = current_database() and state = 'active' and wait_event_type = 'Lock'
and query like 'insert into "chat_identity_links"%'`);
expect(waiting.count).toBeGreaterThan(0);
});
expect(events).toEqual(["intent"]);
expect(await db.select().from(chatIdentityLinks).where(eq(chatIdentityLinks.endpointId, f.endpoint.id))).toEqual([]);
} finally { release(); await lock; await linking; }
expect(events).toEqual(["intent", "settled"]);
expect(await db.select().from(chatIdentityLinks).where(eq(chatIdentityLinks.endpointId, f.endpoint.id)))
.toEqual([expect.objectContaining({ paperclipUserId: actor.userId, status: "linked" })]);
events.length = 0;
expect(await f.service.sendSlackTestMessage(f.endpoint.id, actor, { requestId: randomUUID() })).toMatchObject({ status: "sent" });
expect(events).toEqual([]);
} finally { unsubscribe(); }
});
it("settles a failed Slack account-link intent without committing identity or sending a test message", async () => {
const f = await managedFixture(); await f.provision();
await db.delete(chatIdentityLinks).where(eq(chatIdentityLinks.endpointId, f.endpoint.id));
f.provider.mockClear();
const events: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, (event, scopes) => {
if ([...scopes ?? []].some(scope => JSON.parse(scope).companyId === f.companyId)) events.push(event);
});
await db.execute(sql`create function sleep_slack_link_failure() returns trigger language plpgsql as $$ begin raise exception 'link write unavailable'; end $$`);
await db.execute(sql`create trigger sleep_slack_link_failure before insert on chat_identity_links for each row execute function sleep_slack_link_failure()`);
try {
await expect(f.service.sendSlackTestMessage(f.endpoint.id, actor, { requestId: randomUUID() })).rejects.toThrow();
expect(events).toEqual(["intent", "settled"]);
expect(await db.select().from(chatIdentityLinks).where(eq(chatIdentityLinks.endpointId, f.endpoint.id))).toEqual([]);
expect(posts(f)).toHaveLength(0);
} finally {
unsubscribe();
await db.execute(sql`drop trigger sleep_slack_link_failure on chat_identity_links`);
await db.execute(sql`drop function sleep_slack_link_failure()`);
}
});
it("requires a private Slack challenge for an unknown account before linking and sending the test", async () => {
const f = await managedFixture(); await f.provision();
@@ -34933,7 +35053,44 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
]);
});
it("drains bounded Slack chat-origin milestones without admitting an internal issue run", async () => {
it("wakes an existing chat milestone when ordinary ingress reactivates its conversation", async () => {
const fixture = await seedCompany();
const { callbacks, endpoint, wakeup } = await configuredSlackEndpoint(fixture);
const thread = makeThread({ channelId: "C-REOPEN-MILESTONE", id: "slack:C-REOPEN-MILESTONE:4100.1", name: "reopen-milestone" });
const send = (id: string) => deliverMessage({ callbacks, endpointId: endpoint.id, thread: thread.thread,
message: makeMessage({ id, text: "@maya continue this conversation", mentioned: true }), trigger: "mention" });
await send("4100.1");
const [conversation] = await readChatConversationsByEndpointId(endpoint.id);
const [link] = await db.select().from(chatMessageLinks).where(and(
eq(chatMessageLinks.conversationId, conversation.id), eq(chatMessageLinks.providerMessageId, "4100.1")));
const runId = randomUUID();
await db.insert(heartbeatRuns).values({ id: runId, companyId: fixture.companyId, agentId: fixture.assignedAgentId,
status: "failed", errorCode: "low_trust_isolation_unavailable", updatedAt: new Date(),
contextSnapshot: { issueId: conversation.issueId, source: "chat:slack", endpointId: endpoint.id, wakeCommentId: link.commentId },
});
await db.update(chatConversations).set({ state: "unavailable" }).where(eq(chatConversations.id, conversation.id));
const onError = vi.fn();
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError });
const notice = () => db.select().from(chatPublications).where(and(eq(chatPublications.issueId, conversation.issueId),
eq(chatPublications.idempotencyKey, `run:${runId}:failed:${endpoint.id}`)));
try {
await chatRunMilestoneWork(db).register(coordinator, () => true).ready;
await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBeNull());
expect(await notice()).toEqual([]);
const wakesBefore = wakeup.mock.calls.length;
await send("4100.2");
expect(wakeup.mock.calls.length).toBe(wakesBefore + 1);
// The harness accepts the new wake without creating a run. Existing
// projection debt must wake from ingress, not a later run transition.
await expect.poll(notice, { timeout: 5000 }).toEqual([
expect.objectContaining({ idempotencyKey: `run:${runId}:failed:${endpoint.id}` }),
]);
await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBeNull());
expect(onError).not.toHaveBeenCalled();
} finally { await coordinator.stop(); }
});
it.each(["milliseconds", "microseconds"])("drains bounded Slack chat-origin milestones without admitting an internal issue run (%s)", async (precision) => {
const fixture = await seedCompany();
const { callbacks, endpoint } = await configuredSlackEndpoint(fixture);
const thread = makeThread({
@@ -35028,15 +35185,32 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
})),
]);
const inserted: number[] = [];
for (let index = 0; index < 4; index += 1) {
inserted.push(
await enqueueChatRunMilestones(db, {
since: new Date("2026-09-05T14:00:00.000Z"),
limit: 1,
}),
);
if (precision === "microseconds") {
await db.update(heartbeatRuns).set({
updatedAt: sql`'2026-09-05T14:59:00.000123Z'::timestamptz`,
}).where(inArray(heartbeatRuns.id, resolverRejectedRuns.map(run => run.id)));
}
// Fail a non-advancing pagination loop explicitly instead of leaving a
// background query chain running after the test timeout.
const select = db.select.bind(db);
let reads = 0;
const readGuard = vi.spyOn(db, "select").mockImplementation((...args) => {
if (args[0] && "runUpdatedAt" in args[0] && ++reads > 100) {
throw new Error("Milestone projection did not advance past rejected candidates");
}
return select(...args);
});
const inserted: number[] = [];
try {
for (let index = 0; index < 4; index += 1) {
inserted.push(
await enqueueChatRunLifecycleMilestones(db, {
since: new Date("2026-09-05T14:00:00.000Z"),
limit: 1,
}),
);
}
} finally { readGuard.mockRestore(); }
expect(inserted).toEqual([1, 1, 1, 0]);
const publications = await db
.select()
@@ -41727,13 +41901,18 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
}
});
it("publishes a closed-choice question, settles its Slack card, and delivers its exact continuation response", async () => {
it("publishes a closed-choice question, settles its Slack card, and delivers its exact continuation response", async ({ onTestFinished }) => {
let receiptWakeObserved = false;
const fixture = await seedCompany();
const continuationRunId = randomUUID();
const resolveNativeQuestion = vi.fn(async () => "not_native" as const);
const { callbacks, endpoint, runtime, service } =
await configuredSlackEndpoint(fixture, {
resolveNativeQuestion,
questionResolutionPersistBarrier: async () => {
// Receipt commit must wake projection before the separate answer write.
expect(receiptWakeObserved).toBe(true);
},
wakeup: async (agentId, options) => {
if (options.contextSnapshot?.source !== "issue.interaction.respond") {
return { accepted: true };
@@ -42056,6 +42235,12 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
.set({ state: "delivery_unknown", providerMessageId: null })
.where(eq(chatPublications.id, publication.id));
const unsubscribeReceipt = subscribeDatabaseWork(db, "chat-run-milestones", (event, hints) => {
if (event === "settled" && hints?.has(JSON.stringify({ companyId: fixture.companyId, issueId: conversation.issueId }))) {
receiptWakeObserved = true;
}
});
onTestFinished(unsubscribeReceipt);
const providerAcknowledgement = callbacks.onAction(actionEvent());
await expect(
Promise.race([
@@ -51802,8 +51987,8 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
const registeredCommandId = "888888888888888819";
let interactionSequence = 0n;
async function commandFixture() {
const fixture = await seedCompany();
async function commandFixture(companyId?: string) {
const fixture = await seedCompany(undefined, companyId);
const applicationId = uniqueDiscordApplicationId();
let holdWork = false;
const queued: Array<() => void> = [];
@@ -52897,8 +53082,8 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
: lastFreshDiscordMessageId + 1n;
return lastFreshDiscordMessageId.toString();
};
async function reopenedFixture(includeDuringConfirmationSource = false) {
const f = await commandFixture();
async function reopenedFixture(includeDuringConfirmationSource = false, companyId?: string) {
const f = await commandFixture(companyId);
const threadId = `discord:${guildId}:${channelId}:${rootMessageId}`;
const admitMessage = async (message: ReturnType<typeof makeMessage>) => {
await deliverMessage({
@@ -53046,6 +53231,142 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
};
}
it.each(["action", "delivery"] as const)("wakes a post-close milestone when interrupted inbound acceptance settles its %s receipt", async receipt => {
const f = await reopenedFixture();
const onError = vi.fn();
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError });
const notice = () => db.select().from(chatPublications).where(and(
eq(chatPublications.issueId, f.conversation.issueId),
eq(chatPublications.idempotencyKey, `run:${f.fresh.runId}:working:${f.endpoint.id}`),
));
try {
// Crash boundary: the run/wake is durable, but its inbound action has
// not yet recorded settlement. Recovery must not dispatch it again.
if (receipt === "action") {
await db.update(chatActions).set({ status: "issued", result: {}, updatedAt: new Date() })
.where(eq(chatActions.id, f.fresh.action.id));
} else {
// Operator repair of a failed ledger keeps the already-admitted
// action/run. Only delivery acceptance needs to commit again.
await db.update(chatDeliveries).set({ state: "retry", processedAt: null, nextAttemptAt: null })
.where(eq(chatDeliveries.id, f.delivery.id));
}
await chatRunMilestoneWork(db).register(coordinator, () => true).ready;
await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBeNull());
expect(await notice()).toEqual([]);
const wakesBefore = f.wakeup.mock.calls.length;
await expect.poll(async () => {
await f.service.processPendingDeliveries(25, f.delivery.id);
const [action] = await db.select().from(chatActions).where(eq(chatActions.id, f.fresh.action.id));
const [delivery] = await db.select().from(chatDeliveries).where(eq(chatDeliveries.id, f.delivery.id));
return [action.status, delivery.state];
}, { timeout: 10_000 }).toEqual(["processed", "processed"]);
expect(f.wakeup.mock.calls.length).toBe(wakesBefore);
await expect.poll(notice, { timeout: 5000 }).toEqual([
expect.objectContaining({ idempotencyKey: `run:${f.fresh.runId}:working:${f.endpoint.id}` }),
]);
await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBeNull());
expect(onError).not.toHaveBeenCalled();
} finally { await coordinator.stop(); await f.close(); }
});
it.each(["connection", "resource", "unlinked", "identity", "member", "member_permissions", "ensure_membership", "company_access", "visibility", "cloud_sync"] as const)(
"wakes a post-close milestone when %s authority is restored", async owner => {
const stackId = randomUUID();
const f = await reopenedFixture(false, owner === "cloud_sync" ? cloudTenantPrimaryCompanyId(stackId) : undefined);
const previousToken = process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN;
const [previousUser] = owner === "cloud_sync"
? await db.select().from(authUsers).where(eq(authUsers.id, "owner-user"))
: [];
const onError = vi.fn();
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError });
const notice = () => db.select().from(chatPublications).where(and(
eq(chatPublications.issueId, f.conversation.issueId),
eq(chatPublications.idempotencyKey, `run:${f.fresh.runId}:working:${f.endpoint.id}`),
));
const access = accessService(db);
const membership = (await db.select().from(companyMemberships).where(and(
eq(companyMemberships.companyId, f.fixture.companyId), eq(companyMemberships.principalId, "owner-user"),
)))[0];
try {
let restore: () => Promise<unknown>;
if (owner === "connection") {
await toolAccessService(db).updateConnection(f.endpoint.connectionId, { enabled: false });
restore = () => toolAccessService(db).updateConnection(f.endpoint.connectionId, { enabled: true });
} else if (owner === "resource") {
const [conversation] = await db.select().from(chatConversations).where(eq(chatConversations.id, f.conversation.id));
expect(conversation.resourceId).not.toBeNull();
await f.service.replaceResources(f.endpoint.id, [{ id: conversation.resourceId!, enabled: false }], "owner-user");
restore = () => f.service.replaceResources(f.endpoint.id, [{ id: conversation.resourceId!, enabled: true }], "owner-user");
} else if (owner === "identity" || owner === "unlinked") {
await f.service.update(f.endpoint.id, { allowUnlinkedPeople: false }, "owner-user");
await f.service.revokeLink(f.endpoint.id, f.fresh.action.principalId!);
if (owner === "unlinked") {
restore = () => f.service.update(f.endpoint.id, { allowUnlinkedPeople: true }, "owner-user");
} else {
const intent = await f.service.createLinkIntent(f.endpoint.id, f.fresh.action.principalId!, 1800);
restore = () => f.service.confirmIdentityLink(new URL(intent.confirmationUrl).searchParams.get("token")!, "owner-user");
}
} else if (owner === "visibility") {
await issueService(db).update(f.conversation.issueId, { hiddenAt: new Date() });
restore = () => issueService(db).update(f.conversation.issueId, { hiddenAt: null });
} else {
// Model existing denied authority without exercising revocation's
// unrelated credential cleanup; restoration uses its actual owner.
await db.update(companyMemberships).set({ status: "suspended" }).where(eq(companyMemberships.id, membership.id));
if (owner === "member") restore = () => access.updateMember(f.fixture.companyId, membership.id, { status: "active" });
else if (owner === "member_permissions") restore = () => access.updateMemberAndPermissions(f.fixture.companyId, membership.id, { status: "active", grants: [] }, "owner-user");
else if (owner === "cloud_sync") {
process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN = "milestone-cloud-sync-test";
let request = 0;
restore = () => resolveCloudTenantActor(db, cloudActorHeaderSourceFromHeaders({
"x-paperclip-cloud-tenant-token": "milestone-cloud-sync-test",
"x-paperclip-cloud-user-id": "owner-user",
"x-paperclip-cloud-user-email": "owner@example.test",
// Change the profile to force an actual sync on replay. The
// debounce must not hide repeated unchanged membership wakes.
"x-paperclip-cloud-user-name": `Owner ${request++}`,
"x-paperclip-cloud-stack-id": stackId,
"x-paperclip-cloud-stack-role": "owner",
}));
}
else if (owner === "ensure_membership") restore = () => access.ensureMembership(f.fixture.companyId, "user", "owner-user", membership.membershipRole, "active");
else {
const existing = await db.select().from(companyMemberships).where(and(
eq(companyMemberships.principalType, "user"), eq(companyMemberships.principalId, "owner-user"),
));
restore = () => access.setUserCompanyAccess("owner-user", existing.map(row => row.companyId));
}
}
await chatRunMilestoneWork(db).register(coordinator, () => true).ready;
await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBeNull());
expect(await notice()).toEqual([]);
await restore();
await expect.poll(notice, { timeout: 5000 }).toEqual([
expect.objectContaining({ idempotencyKey: `run:${f.fresh.runId}:working:${f.endpoint.id}` }),
]);
await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBeNull());
if (owner !== "identity" && owner !== "connection") {
const notices: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, event => notices.push(event));
try { await restore(); expect(notices).toEqual([]); }
finally { unsubscribe(); }
expect(await notice()).toHaveLength(1);
}
expect(onError).not.toHaveBeenCalled();
} finally {
if (previousToken === undefined) delete process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN;
else process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN = previousToken;
await coordinator.stop();
if (previousUser) await db.update(authUsers).set({
name: previousUser.name, email: previousUser.email,
emailVerified: previousUser.emailVerified, updatedAt: previousUser.updatedAt,
}).where(eq(authUsers.id, previousUser.id));
await f.close();
}
},
);
it("publishes working and the exact fresh final after pinned close without resurrecting the old source", async () => {
const f = await reopenedFixture();
try {
@@ -65325,6 +65646,36 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
};
}
it.each(["separate owners", "combined budget"] as const)(
"keeps lifecycle and native progress projection independent with %s",
async (mode) => {
await enqueueChatRunMilestones(db, { since: new Date(0) });
const context = await safeNativeProgressFixture("telegram", mode === "separate owners" ? "111" : "112");
try {
const progressRun = await context.createRun("ongoing work");
await context.addEvent(progressRun);
const failedRun = await context.createRun("failed work");
await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, failedRun.runId));
const input = { since: new Date(0), limit: 1 };
await expect((mode === "separate owners" ? enqueueChatRunLifecycleMilestones : enqueueChatRunMilestones)(db, input)).resolves.toBe(1);
const rows = () => db.select({ key: chatPublications.idempotencyKey }).from(chatPublications)
.where(eq(chatPublications.conversationId, context.conversation.id));
expect((await rows()).map(row => row.key)).toContain(`run:${failedRun.runId}:failed:${context.endpoint.id}`);
expect((await rows()).some(row => row.key.includes(":native:"))).toBe(false);
if (mode === "separate owners") {
await expect(enqueueChatRunLifecycleMilestones(db, input)).resolves.toBe(0);
}
await expect((mode === "separate owners" ? enqueueSafeNativeChatProgress : enqueueChatRunMilestones)(db, input)).resolves.toBe(1);
expect((await rows()).filter(row => row.key.includes(":native:"))).toEqual([
{ key: `run:${progressRun.runId}:working:${context.endpoint.id}:native:making_progress:1` },
]);
await expect(enqueueChatRunMilestones(db, input)).resolves.toBe(0);
} finally {
await retirePublicationFixture(context.service, context.endpoint.id);
}
},
);
it.each(["issue", "run"] as const)(
"skips a locked native progress %s while unrelated final and question publications deliver",
async (lockTarget) => {
@@ -72379,6 +72730,48 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
};
}
it("restores changed native review metadata once and signals only its committed change", async () => {
const context = await committedNativeReviewPublicationFixture("milestone-repair");
// Earlier failure cases can leave intent awaiting XID reconciliation on the
// suite's shared owner. Subscribing to that owner immediately replays intent.
await expect(db.transaction(async tx => {
await signalDatabaseWork(tx, "chat-run-milestones");
throw new Error("unrelated rolled-back producer");
})).rejects.toThrow("unrelated rolled-back producer");
const repairDb = createDb(externalTestDatabaseUrl ?? tempDb!.connectionString);
const events: string[] = [];
const unsubscribe = subscribeDatabaseWork(repairDb, "chat-run-milestones", event => events.push(event));
try {
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, context.runId));
const input = { companyId: context.fixture.companyId, issueId: coordinator.issueId, runId: context.runId,
decisionId: coordinator.decisionId!, resultId: coordinator.resultId!, assessmentId: coordinator.assessmentId! };
const savedAt = new Date(Date.now() - 60_000);
await db.update(heartbeatRuns).set({ updatedAt: savedAt }).where(eq(heartbeatRuns.id, context.runId));
// Already-restored metadata still authorizes materialization by the caller,
// but must neither rewrite the run nor wake projection again.
await expect(repairDb.transaction(tx => restoreNativeChatReviewPresentationInTransaction(tx as unknown as typeof db, input))).resolves.toBe(true);
expect((await readHeartbeatRunsById(context.runId))[0].updatedAt).toEqual(savedAt);
expect(events).toEqual([]);
await db.update(heartbeatRuns).set({ resultJson: sql`${heartbeatRuns.resultJson} - 'externalChatReviewPresentation'` })
.where(eq(heartbeatRuns.id, context.runId));
await repairDb.transaction(async tx => {
expect(await restoreNativeChatReviewPresentationInTransaction(tx as unknown as typeof db, input)).toBe(true);
expect(events).toEqual(["intent"]);
});
expect(events).toEqual(["intent", "settled"]);
expect((await readHeartbeatRunsById(context.runId))[0].resultJson).toMatchObject({
externalChatReviewPresentation: { decisionId: coordinator.decisionId, runId: context.runId },
});
events.length = 0;
await repairDb.transaction(tx => restoreNativeChatReviewPresentationInTransaction(tx as unknown as typeof db, input));
expect(events).toEqual([]);
} finally {
unsubscribe();
await repairDb.$client.end({ timeout: 1 });
await context.service.shutdown();
}
});
it.each(["principal_revoked", "gate_changed"] as const)(
"blocks a committed native review response and selected files at transport after %s",
async (kind) => {
@@ -0,0 +1,230 @@
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import {
completionContracts, nativeRunResults, nativeRunFinalizations, workAssessments, statusDecisions, statusDecisionEffects,
agents, toolActionDeliveries, toolActionRequests, toolInvocations, toolGatewaySessions, toolCallEvents, toolAccessAuditEvents, toolPolicies,
companySecretBindings, companySecretProposals, companySecrets, createDb,
chatPublications, heartbeatRuns, issueComments, issueThreadInteractions, issues,
} from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { createChatMilestoneFixtures } from "./helpers/chat-milestone-fixtures.js";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { chatRunMilestoneWork } from "../services/chat-run-milestone-work.js";
import { createSecretProposalsService } from "../services/secret-proposals.js";
import { commitToolActionReview } from "../services/tool-action-review.js";
import { createToolGatewayService } from "../services/tool-gateway.js";
import { issueThreadInteractionService } from "../services/issue-thread-interactions.js";
import { prepareConversationTurn } from "../services/agent-conversations.js";
import { dismissAutomaticCompletionReviews } from "../services/native-runtime/automatic-completion-reviews.js";
import { dismissObsoleteNativePolicyReviews } from "../services/native-runtime/obsolete-policy-reviews.js";
import { idleWorkSnapshot } from "../services/task-admission.js";
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_MILESTONE_GOVERNED_TEST_DATABASE_URL");
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
(support.supported ? describe : describe.skip)("chat milestones after governed interaction writes", () => {
let db: ReturnType<typeof createDb>;
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let fixtures: ReturnType<typeof createChatMilestoneFixtures>;
let f: Awaited<ReturnType<typeof fixtures.create>>;
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
const errors: unknown[] = [];
beforeAll(async () => {
if (!external) temporary = await startEmbeddedPostgresTestDatabase("chat-governed-milestones-");
db = createDb(external ?? temporary!.connectionString, { idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
fixtures = createChatMilestoneFixtures(db);
}, 60_000);
beforeEach(async () => {
errors.length = 0; f = await fixtures.create();
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
});
afterEach(async () => {
await coordinator?.stop(); coordinator = undefined; vi.useRealTimers(); vi.restoreAllMocks();
await db.delete(companySecretProposals).where(eq(companySecretProposals.companyId, f.companyId));
await db.delete(companySecretBindings).where(eq(companySecretBindings.companyId, f.companyId));
await db.delete(companySecrets).where(eq(companySecrets.companyId, f.companyId));
await db.delete(toolActionDeliveries).where(eq(toolActionDeliveries.companyId, f.companyId));
await db.delete(toolGatewaySessions).where(eq(toolGatewaySessions.companyId, f.companyId));
await db.delete(toolCallEvents).where(eq(toolCallEvents.companyId, f.companyId));
await db.delete(toolAccessAuditEvents).where(eq(toolAccessAuditEvents.companyId, f.companyId));
await db.delete(toolActionRequests).where(eq(toolActionRequests.companyId, f.companyId));
await db.delete(toolInvocations).where(eq(toolInvocations.companyId, f.companyId));
await db.delete(toolPolicies).where(eq(toolPolicies.companyId, f.companyId));
await fixtures.cleanup();
expect(errors).toEqual([]);
});
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
async function start() {
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: error => errors.push(error) });
await chatRunMilestoneWork(db).register(coordinator, () => true).ready;
await flush();
}
async function flush() {
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(1000);
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBeNull();
}, { timeout: 10_000 });
}
async function quiet() {
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
try { await vi.advanceTimersByTimeAsync(120_000); expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); }
finally { reads.mockRestore(); writes.mockRestore(); }
}
const publications = () => db.select().from(chatPublications).where(eq(chatPublications.issueId, f.issueId));
async function run(status: "failed" | "running" = "failed") {
const runId = randomUUID();
await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: f.agentId,
responsibleUserId: f.userId, status, errorCode: "test_run_failed", updatedAt: new Date(),
contextSnapshot: { issueId: f.issueId, source: "chat:slack", endpointId: f.endpointId, wakeCommentId: f.commentId },
});
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, f.issueId));
return runId;
}
it.each(["approved", "rejected", "withdrawn", "expired", "rolled back", "parent rejected"] as const)("reactivates after a binding proposal is %s", async status => {
const runId = await run();
// Binding review needs secret metadata, not the provider value. No secret is resolved.
const [secret] = await db.insert(companySecrets).values({ companyId: f.companyId, name: "test/binding", key: "BINDING_TEST" }).returning();
const parent = status === "parent rejected" ? (await db.insert(companySecretProposals).values({
companyId: f.companyId, kind: "secret", proposedName: "test/parent", proposedKey: "PARENT_TEST",
justification: "Pending parent secret", proposedByAgentId: f.agentId, originRunId: runId,
originIssueId: f.issueId, expiresAt: new Date(Date.now() + 86_400_000),
}).returning())[0] : null;
const proposals = createSecretProposalsService(db);
const proposal = await proposals.createBinding({ companyId: f.companyId, heartbeatRunId: runId }, {
...(parent ? { secretProposalId: parent.id } : { secretId: secret.id }), configPath: "env.BINDING_TEST", justification: "Use the existing company credential", bindingTargetPolicy: "self_and_reports",
});
expect(proposal.interactionId).toBeTruthy();
await start(); expect(await publications()).toEqual([]); await quiet();
if (status === "rolled back") {
await expect(db.transaction(async tx => {
await createSecretProposalsService(tx as unknown as typeof db).transition(f.companyId, proposal.id, "rejected");
throw new Error("rollback proposal");
})).rejects.toThrow("rollback proposal");
expect((await proposals.getById(f.companyId, proposal.id))?.status).toBe("pending");
await flush(); expect(await publications()).toEqual([]); await quiet();
await proposals.transition(f.companyId, proposal.id, "rejected");
} else if (parent) {
await proposals.transition(f.companyId, parent.id, "rejected");
expect((await proposals.getById(f.companyId, proposal.id))?.status).toBe("rejected");
} else if (status === "approved") {
await proposals.approve(f.companyId, proposal.id, { resolvedByUserId: f.userId });
} else if (status === "expired") {
expect(await proposals.sweepExpired(new Date(proposal.expiresAt.getTime() + 1))).toBe(1);
} else if (status === "rejected" || status === "withdrawn") {
await proposals.transition(f.companyId, proposal.id, status, { resolvedByUserId: f.userId, proposerAgentId: f.agentId });
}
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, proposal.interactionId!)))[0].status)
.toBe(status === "approved" ? "accepted" : status === "withdrawn" ? "cancelled" : status === "rolled back" || status === "parent rejected" ? "rejected" : status);
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${runId}:failed:${f.endpointId}`);
await quiet();
});
it.each(["approved", "rejected", "expired"] as const)("reactivates after a tool review is %s", async status => {
await db.update(agents).set({ adapterType: "process" }).where(eq(agents.id, f.agentId));
const runId = await run("running");
await db.insert(toolPolicies).values({ companyId: f.companyId, name: "Ask first", policyType: "require_approval",
selectors: { toolName: "mcp-remote-fixture:update_note" } });
const remote = vi.fn(async () => { throw new Error("Review must not invoke the provider"); });
const gateway = createToolGatewayService(db, { toolActionSigningSecret: "milestone-review-fixture", remoteHttpRequest: remote });
const session = await gateway.createSession({ companyId: f.companyId, agentId: f.agentId, runId });
await expect(gateway.executeTool({ sessionToken: session.token, tool: "mcp-remote-fixture:update_note", parameters: { noteId: "test" } }))
.rejects.toMatchObject({ reasonCode: "approval_required" });
const [request] = await db.select().from(toolActionRequests).where(eq(toolActionRequests.companyId, f.companyId));
expect(request.interactionId).toBeTruthy();
// The prompt could not be delivered, and the run failed while awaiting review.
// The pending review still owns its response until the board resolves it.
await db.update(chatPublications).set({ state: "failed" }).where(eq(chatPublications.issueId, f.issueId));
await db.update(heartbeatRuns).set({ status: "failed", updatedAt: new Date() }).where(eq(heartbeatRuns.id, runId));
await start();
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${runId}:`))).toBe(false);
await quiet();
if (status === "expired") {
await db.update(toolActionRequests).set({ expiresAt: new Date(Date.now() - 1) }).where(eq(toolActionRequests.id, request.id));
await gateway.sweepExpiredActionReviews();
} else {
await commitToolActionReview(db, { companyId: f.companyId, actionRequestId: request.id, decision: status, actor: { userId: f.userId } });
}
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, request.interactionId!)))[0].status)
.toBe(status === "approved" ? "accepted" : status);
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${runId}:failed:${f.endpointId}`);
await quiet(); expect(remote).not.toHaveBeenCalled();
});
it.each([
["automatic", "prompt"], ["automatic", "no prompt"], ["automatic", "explicit request"],
["obsolete", "prompt"], ["obsolete", "no prompt"], ["obsolete", "explicit request"],
] as const)("handles %s review retirement with %s", async (kind, history) => {
const runId = await run(), contractId = randomUUID(), resultId = randomUUID(), assessmentId = randomUUID(), decisionId = randomUUID();
const prompt = kind === "automatic"
? "Review the persisted native-run evidence and confirm whether this issue may be completed."
: "Review the superseding native policy assessment.";
await db.insert(completionContracts).values({ id: contractId, companyId: f.companyId, issueId: f.issueId, revision: 1,
schemaVersion: "paperclip.completion-contract.v1", policyVersion: "phase6-v1", risk: kind === "automatic" ? "low" : "standard",
completionAuthority: kind === "automatic" ? "agent_claim_policy" : "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal",
contractJson: { objective: "Respond to chat" }, canonicalSha256: contractId, createdByActorType: "system", createdByActorId: "test" });
await db.update(heartbeatRuns).set({ status: "succeeded", runtimeMode: "native", nativeIssueId: f.issueId, completionContractId: contractId,
resultJson: { presentationDecision: { source: "none" } } }).where(eq(heartbeatRuns.id, runId));
await db.insert(nativeRunResults).values({ id: resultId, companyId: f.companyId, issueId: f.issueId, runId,
completionContractId: contractId, serverFingerprint: resultId, schemaStatus: "accepted", canonicalSha256: resultId,
resultJson: { terminal: { runTerminalState: "succeeded" } } });
await db.insert(nativeRunFinalizations).values({ runId, companyId: f.companyId, issueId: f.issueId, phase: "committed", resultId });
await db.insert(workAssessments).values({ id: assessmentId, companyId: f.companyId, issueId: f.issueId, runId,
contractId, resultId, triggerKind: "run_result", triggerActorCompanyId: f.companyId, priorIssueStatus: "in_progress",
priorStatusVersion: 0, policyVersion: "phase6-v1", assessmentJson: { attentionRequests: [] }, inputDigest: assessmentId });
await db.insert(statusDecisions).values({ id: decisionId, companyId: f.companyId, issueId: f.issueId, runId, assessmentId,
decisionVersion: 1, policyVersion: "phase6-v1", fromStatus: "in_progress", toStatus: "in_review", applicationState: "applied",
reasonCode: kind === "automatic" ? "completion_claim_incomplete" : "completion_review_required", decisionDigest: decisionId,
decisionJson: { projectedStatusVersion: 1, effects: [{ kind: "bind_reviewer", prompt, ownerUserId: null },
...(kind === "obsolete" ? [{ kind: "append_superseding_assessment" }] : [])] } });
await db.update(issues).set({ status: "in_review", statusVersion: 1, lastStatusDecisionId: decisionId }).where(eq(issues.id, f.issueId));
const [interaction] = await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId: f.issueId,
kind: "request_confirmation", sourceRunId: runId, idempotencyKey: `native-review:${decisionId}`, status: "pending",
payload: { version: 1, prompt: history === "explicit request" ? "Review the release before publishing." : prompt,
target: { type: "custom", key: "native_completion_review", revisionId: decisionId } } }).returning();
await db.insert(statusDecisionEffects).values({ companyId: f.companyId, issueId: f.issueId, decisionId,
ordinal: 0, effectKind: "bind_reviewer", targetType: "issue_thread_interaction", targetId: interaction.id,
idempotencyKey: decisionId, payload: {}, deliveryState: "delivered" });
// System cards are no longer externalized, but stored prompt rows still own
// the response slot. Exercise their supported retirement rather than insert
// a synthetic new provider send through today's author-validation boundary.
if (history !== "no prompt") await db.insert(chatPublications).values({ companyId: f.companyId, issueId: f.issueId,
endpointId: f.endpointId, conversationId: f.conversationId, state: "pending", payload: { text: prompt, interactionId: interaction.id },
idempotencyKey: `interaction:${interaction.id}:${f.endpointId}` });
const notices = async () => (await publications()).filter(row => row.idempotencyKey === `run:${runId}:completed:${f.endpointId}`);
const retire = () => kind === "automatic" ? dismissAutomaticCompletionReviews(db, f.issueId) : dismissObsoleteNativePolicyReviews(db, [runId]);
await start(); expect(await notices()).toHaveLength(history === "no prompt" ? 1 : 0); await quiet();
await retire();
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interaction.id)))[0].status)
.toBe(history === "explicit request" ? "pending" : "cancelled");
await flush(); expect(await notices()).toHaveLength(history === "explicit request" ? 0 : 1); await quiet();
await retire(); await flush(); expect(await notices()).toHaveLength(history === "explicit request" ? 0 : 1); await quiet();
});
it("rechecks a stored chat binding when a conversation reset expires its question", async () => {
const runId = await run();
await db.update(issues).set({ conversationAgentId: f.agentId, conversationUserId: f.userId, conversationState: "active" }).where(eq(issues.id, f.issueId));
const interaction = await issueThreadInteractionService(db).create({ id: f.issueId, companyId: f.companyId }, {
kind: "ask_user_questions", sourceRunId: runId, continuationPolicy: "wake_assignee",
payload: { version: 1, title: "Choose", questions: [{ id: "choice", prompt: "Continue?", selectionMode: "single",
options: [{ id: "yes", label: "Yes" }], required: true }] },
}, { agentId: f.agentId, runId });
// Failed delivery leaves the pending question itself as the response owner.
await db.update(chatPublications).set({ state: "failed" }).where(eq(chatPublications.issueId, f.issueId));
const [comment] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId,
authorType: "user", authorUserId: f.userId, body: "/new" }).returning();
const [resetRun] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId,
status: "running", contextSnapshot: { issueId: f.issueId, source: "issue.comment", wakeCommentId: comment.id } }).returning();
await start();
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${runId}:`))).toBe(false); await quiet();
expect(await prepareConversationTurn(db, resetRun)).toMatchObject({ reset: true });
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interaction.id)))[0].status).toBe("expired");
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${runId}:failed:${f.endpointId}`);
await quiet();
});
});
@@ -0,0 +1,204 @@
import { randomUUID } from "node:crypto";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { agentWakeupRequests, chatActions, chatExternalPrincipals, chatIdentityLinks, chatMessageLinks, chatPublications, companyMemberships, createDb, heartbeatRuns, issueComments, issues, subscribeDatabaseWork } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { createChatMilestoneFixtures } from "./helpers/chat-milestone-fixtures.js";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
import { chatRunMilestoneWork } from "../services/chat-run-milestone-work.js";
import { initializeRunIdentity, reserveSteeredIdentity, acceptSteeredIdentity } from "../services/run-identity.js";
import { idleWorkSnapshot } from "../services/task-admission.js";
import { accessService } from "../services/access.js";
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_MILESTONE_IDENTITY_TEST_DATABASE_URL");
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
(support.supported ? describe : describe.skip)("chat milestones after identity activation", () => {
let db: ReturnType<typeof createDb>;
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let fixtures: ReturnType<typeof createChatMilestoneFixtures>;
let f: Awaited<ReturnType<typeof fixtures.create>>;
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
const errors: unknown[] = [];
beforeAll(async () => {
if (!external) temporary = await startEmbeddedPostgresTestDatabase("chat-identity-milestones-");
db = createDb(external ?? temporary!.connectionString, { idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
fixtures = createChatMilestoneFixtures(db);
}, 60_000);
beforeEach(async () => {
errors.length = 0; f = await fixtures.create();
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
});
afterEach(async () => {
await coordinator?.stop(); coordinator = undefined; vi.useRealTimers(); vi.restoreAllMocks();
await fixtures.cleanup(); expect(errors).toEqual([]);
});
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
async function flush() {
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(1000);
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBeNull();
}, { timeout: 10_000 });
}
async function quiet() {
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
try { await vi.advanceTimersByTimeAsync(120_000); expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); }
finally { reads.mockRestore(); writes.mockRestore(); }
}
const publications = () => db.select().from(chatPublications).where(eq(chatPublications.issueId, f.issueId));
it.each([false, true])("rechecks concurrent membership suspension before restoring notice access (rollback=%s)", async rollback => {
const membershipWhere = and(eq(companyMemberships.companyId, f.companyId), eq(companyMemberships.principalId, f.userId));
// A Board-origin reply uses the linked author and current membership.
// Remove the fixture's unrelated provider-ingress provenance.
await db.delete(chatMessageLinks).where(eq(chatMessageLinks.commentId, f.commentId));
await db.update(issueComments).set({ authorUserId: f.userId }).where(eq(issueComments.id, f.commentId));
const [principal] = await db.insert(chatExternalPrincipals).values({ companyId: f.companyId,
provider: "slack", providerAccountId: "", externalId: f.userId,
}).returning();
await db.insert(chatIdentityLinks).values({ companyId: f.companyId, endpointId: f.endpointId,
principalId: principal.id, paperclipUserId: f.userId, status: "linked",
});
await db.insert(chatActions).values({ companyId: f.companyId, endpointId: f.endpointId,
conversationId: f.conversationId, kind: "slack_board_message", status: "processed",
providerActionId: `board-comment:${f.commentId}`, payload: { issueId: f.issueId, commentId: f.commentId,
userId: f.userId, principalId: principal.id, agentId: f.agentId },
});
const runId = randomUUID(), sourceId = randomUUID();
const nextCheckAt = new Date(Date.now() - 1000).toISOString();
await db.insert(heartbeatRuns).values({ id: sourceId, companyId: f.companyId, agentId: f.agentId,
nativeIssueId: f.issueId, runtimeMode: "native", status: "succeeded", responsibleUserId: f.userId,
contextSnapshot: { issueId: f.issueId, source: "chat:slack", endpointId: f.endpointId, wakeCommentId: f.commentId },
resultJson: { semanticToolReceipts: { schedule: { operationId: "set_task_monitor",
result: { taskId: f.issueId, monitor: { status: "scheduled", nextCheckAt } } } } },
});
await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: f.agentId,
nativeIssueId: f.issueId, runtimeMode: "native", status: "running", responsibleUserId: f.userId,
contextSnapshot: { issueId: f.issueId, source: "issue.monitor" },
});
await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, runId,
source: "automation", reason: "issue_monitor_due", status: "claimed",
idempotencyKey: `issue-monitor:${f.issueId}:${nextCheckAt}`, payload: { issueId: f.issueId, nextCheckAt },
});
await initializeRunIdentity(db, { companyId: f.companyId, runId, issueId: f.issueId,
responsibleUserId: f.userId, cause: "dispatch" });
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, f.issueId));
let releaseSuspension!: () => void;
const suspensionHeld = new Promise<void>(resolve => { releaseSuspension = resolve; });
let markSuspended!: () => void;
const suspended = new Promise<void>(resolve => { markSuspended = resolve; });
let releaseRestoration!: () => void;
const restorationHeld = new Promise<void>(resolve => { releaseRestoration = resolve; });
let restored = false;
let restorationPid = 0;
const suspending = db.transaction(async tx => {
await tx.update(companyMemberships).set({ status: "suspended" }).where(membershipWhere);
markSuspended();
await suspensionHeld;
});
await suspended;
const restoring = db.transaction(async tx => {
const [backend] = await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`);
restorationPid = backend.pid;
await accessService(tx as unknown as typeof db).ensureMembership(f.companyId, "user", f.userId, "operator");
restored = true;
await restorationHeld;
if (rollback) throw new Error("outer membership rollback");
});
void restoring.catch(() => undefined);
try {
// The old read sees active membership, then its UPDATE waits. The fixed
// read waits first and observes the suspension after that writer commits.
await vi.waitFor(async () => {
expect(restorationPid).toBeGreaterThan(0);
const [state] = await db.execute<{ blocked: boolean }>(sql`select cardinality(pg_blocking_pids(${restorationPid})) > 0 as blocked`);
expect(state.blocked).toBe(true);
});
releaseSuspension();
await suspending;
await vi.waitFor(() => expect(restored).toBe(true));
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: error => errors.push(error) });
await chatRunMilestoneWork(db).register(coordinator, () => true).ready;
await vi.advanceTimersByTimeAsync(1000);
expect(await publications()).toEqual([]);
expect((await db.select().from(companyMemberships).where(membershipWhere))[0].status).toBe("suspended");
releaseRestoration();
if (rollback) await expect(restoring).rejects.toThrow("outer membership rollback");
else await restoring;
await flush();
expect((await db.select().from(companyMemberships).where(membershipWhere))[0].status)
.toBe(rollback ? "suspended" : "active");
expect(await publications()).toEqual(rollback ? [] : [expect.objectContaining({ idempotencyKey: `run:${runId}:working:${f.endpointId}` })]);
await quiet();
if (!rollback) {
const notices: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, event => notices.push(event));
try {
await accessService(db).ensureMembership(f.companyId, "user", f.userId, "operator");
expect(notices).toEqual([]);
} finally { unsubscribe(); }
await quiet();
}
} finally {
releaseSuspension();
releaseRestoration();
await Promise.allSettled([suspending, restoring]);
}
});
it.each(["initialization", "steering"] as const)("wakes matching monitor lineage after %s commits, preserving rollback and replay", async kind => {
const nextCheckAt = new Date(Date.now() - 1000).toISOString();
const sourceId = randomUUID(), successorId = randomUUID();
await db.insert(heartbeatRuns).values({ id: sourceId, companyId: f.companyId, agentId: f.agentId,
nativeIssueId: f.issueId, runtimeMode: "native", status: "succeeded", responsibleUserId: f.userId,
contextSnapshot: { issueId: f.issueId, source: "chat:slack", endpointId: f.endpointId, wakeCommentId: f.commentId },
resultJson: { semanticToolReceipts: { schedule: { operationId: "set_task_monitor",
result: { taskId: f.issueId, monitor: { status: "scheduled", nextCheckAt } } } } },
});
await db.insert(heartbeatRuns).values({ id: successorId, companyId: f.companyId, agentId: f.agentId,
nativeIssueId: f.issueId, runtimeMode: "native", status: "running", responsibleUserId: null,
contextSnapshot: { issueId: f.issueId, source: "issue.monitor" },
});
await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, runId: successorId,
source: "automation", reason: "issue_monitor_due", status: "claimed",
idempotencyKey: `issue-monitor:${f.issueId}:${nextCheckAt}`, payload: { issueId: f.issueId, nextCheckAt },
});
const identityInput = { companyId: f.companyId, runId: successorId, issueId: f.issueId, responsibleUserId: f.userId, cause: "dispatch" };
if (kind === "steering") {
await initializeRunIdentity(db, { ...identityInput, responsibleUserId: null });
await db.update(issueComments).set({ authorUserId: f.userId }).where(eq(issueComments.id, f.commentId));
}
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: error => errors.push(error) });
await chatRunMilestoneWork(db).register(coordinator, () => true).ready;
await flush(); expect(await publications()).toEqual([]); await quiet();
const pending = kind === "steering" ? await reserveSteeredIdentity(db, {
companyId: f.companyId, runId: successorId, issueId: f.issueId, messageId: f.commentId,
}) : null;
// Reserving an unacknowledged message must not activate its identity.
if (kind === "steering") expect(pending?.status).toBe("pending");
await flush(); expect(await publications()).toEqual([]); await quiet();
const activate = (rollback: boolean) => db.transaction(async tx => {
if (kind === "steering") await acceptSteeredIdentity(tx, pending!);
else await initializeRunIdentity(tx as unknown as typeof db, identityInput);
await vi.advanceTimersByTimeAsync(1000);
expect(await publications()).toEqual([]);
if (rollback) throw new Error("outer rollback");
});
await expect(activate(true)).rejects.toThrow("outer rollback");
await flush(); expect(await publications()).toEqual([]); await quiet();
await activate(false); await flush();
expect(await publications()).toEqual([expect.objectContaining({ idempotencyKey: `run:${successorId}:working:${f.endpointId}` })]);
await quiet();
const notices: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, event => notices.push(event));
try {
await db.transaction(async tx => {
if (kind === "steering") await acceptSteeredIdentity(tx, pending!);
else await initializeRunIdentity(tx as unknown as typeof db, identityInput);
});
expect(notices).toEqual([]);
} finally { unsubscribe(); }
await flush(); await quiet();
});
});
@@ -0,0 +1,103 @@
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { chatPublications, createDb, heartbeatRuns, issues } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { createChatMilestoneFixtures } from "./helpers/chat-milestone-fixtures.js";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
import { chatRunMilestoneWork } from "../services/chat-run-milestone-work.js";
import { recoveryService } from "../services/recovery/service.js";
import { idleWorkSnapshot } from "../services/task-admission.js";
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_MILESTONE_RECOVERY_TEST_DATABASE_URL");
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
(support.supported ? describe : describe.skip)("chat milestones after run recovery", () => {
let db: ReturnType<typeof createDb>;
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let fixtures: ReturnType<typeof createChatMilestoneFixtures>;
let f: Awaited<ReturnType<typeof fixtures.create>>;
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
const errors: unknown[] = [];
beforeAll(async () => {
if (!external) temporary = await startEmbeddedPostgresTestDatabase("chat-recovery-milestones-");
db = createDb(external ?? temporary!.connectionString, { idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
fixtures = createChatMilestoneFixtures(db);
}, 60_000);
beforeEach(async () => {
errors.length = 0; f = await fixtures.create();
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
});
afterEach(async () => {
await coordinator?.stop(); coordinator = undefined; vi.useRealTimers(); vi.restoreAllMocks();
await fixtures.cleanup(); expect(errors).toEqual([]);
});
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
async function flush() {
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(1000);
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBeNull();
}, { timeout: 10_000 });
}
async function quiet() {
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
try { await vi.advanceTimersByTimeAsync(120_000); expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); }
finally { reads.mockRestore(); writes.mockRestore(); }
}
const publications = () => db.select().from(chatPublications).where(eq(chatPublications.issueId, f.issueId));
it.each(["process_gone", "issue_cancelled"] as const)("publishes the recovered terminal state after %s and becomes quiet", async reason => {
const runId = randomUUID();
await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: f.agentId,
status: "running", runtimeMode: "legacy", responsibleUserId: f.userId,
processPid: reason === "process_gone" ? 2_000_000_000 : process.pid,
contextSnapshot: { issueId: f.issueId, source: "chat:slack", endpointId: f.endpointId, wakeCommentId: f.commentId },
});
await db.update(issues).set({ executionRunId: runId,
status: reason === "issue_cancelled" ? "cancelled" : "in_progress",
}).where(eq(issues.id, f.issueId));
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: error => errors.push(error) });
await chatRunMilestoneWork(db).register(coordinator, () => true).ready;
await flush();
expect(await publications()).toEqual([expect.objectContaining({ idempotencyKey: `run:${runId}:working:${f.endpointId}` })]);
await quiet();
const recovery = recoveryService(db, { enqueueWakeup: vi.fn() });
expect(await recovery.sweepStaleIssueLocks()).toEqual({ cleared: 1, issueIds: [f.issueId], terminalizedRunIds: [runId] });
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0].status)
.toBe(reason === "process_gone" ? "interrupted" : "cancelled");
await flush();
expect((await publications()).map(row => row.idempotencyKey).sort()).toEqual([
`run:${runId}:failed:${f.endpointId}`, `run:${runId}:working:${f.endpointId}`,
].sort());
await quiet();
expect(await recovery.sweepStaleIssueLocks()).toEqual({ cleared: 0, issueIds: [], terminalizedRunIds: [] });
await flush(); await quiet(); expect(await publications()).toHaveLength(2);
});
it("wakes an older failed run when recovery refreshes its quota classification", async () => {
const runId = randomUUID(), old = new Date(Date.now() - 48 * 60 * 60_000);
await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: f.agentId,
status: "failed", runtimeMode: "legacy", responsibleUserId: f.userId,
createdAt: old, startedAt: old, finishedAt: old, updatedAt: old,
error: "You've hit your usage limit for GPT-5. Try again at 12:00 AM (UTC).", errorCode: "adapter_failed",
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
contextSnapshot: { issueId: f.issueId, source: "chat:slack", endpointId: f.endpointId, wakeCommentId: f.commentId },
});
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: error => errors.push(error) });
await chatRunMilestoneWork(db).register(coordinator, () => true).ready;
await flush(); expect(await publications()).toEqual([]); await quiet();
const enqueueWakeup = vi.fn(async () => null);
const recovery = recoveryService(db, { enqueueWakeup });
expect(await recovery.reconcileStrandedAssignedIssues()).toMatchObject({ providerQuotaMonitored: 1 });
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0].errorCode).toBe("provider_quota");
expect(enqueueWakeup).not.toHaveBeenCalled();
await flush();
expect(await publications()).toEqual([expect.objectContaining({ idempotencyKey: `run:${runId}:failed:${f.endpointId}` })]);
await quiet();
expect(await recovery.reconcileStrandedAssignedIssues()).toMatchObject({ providerQuotaMonitored: 0, skipped: 1 });
await flush(); await quiet(); expect(await publications()).toHaveLength(1);
});
});
@@ -0,0 +1,125 @@
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { chatConversations, chatDeliveries, chatEndpointResources, chatEndpoints, chatPublications, createDb, heartbeatRuns, issues, subscribeDatabaseWork } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { createChatMilestoneFixtures } from "./helpers/chat-milestone-fixtures.js";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
import { chatRunMilestoneWork } from "../services/chat-run-milestone-work.js";
import { chatCredentialMutationLease } from "../services/chat-credential-mutation-lease.js";
import { createChatResourceReconciliation } from "../services/chat-channels/resource-reconciliation.js";
import { idleWorkSnapshot } from "../services/task-admission.js";
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_MILESTONE_RESOURCE_TEST_DATABASE_URL");
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
(support.supported ? describe : describe.skip)("chat milestones after resource recovery", () => {
let db: ReturnType<typeof createDb>;
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let fixtures: ReturnType<typeof createChatMilestoneFixtures>;
let f: Awaited<ReturnType<typeof fixtures.create>>;
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
const errors: unknown[] = [];
beforeAll(async () => {
if (!external) temporary = await startEmbeddedPostgresTestDatabase("chat-resource-milestones-");
db = createDb(external ?? temporary!.connectionString, { idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
fixtures = createChatMilestoneFixtures(db);
}, 60_000);
beforeEach(async () => {
errors.length = 0; f = await fixtures.create();
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
});
afterEach(async () => {
await coordinator?.stop(); coordinator = undefined; vi.useRealTimers(); vi.restoreAllMocks();
await db.delete(chatDeliveries).where(eq(chatDeliveries.companyId, f.companyId));
await fixtures.cleanup(); expect(errors).toEqual([]);
});
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
async function flush() {
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(1000);
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBeNull();
}, { timeout: 10_000 });
}
async function quiet() {
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
try { await vi.advanceTimersByTimeAsync(120_000); expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); }
finally { reads.mockRestore(); writes.mockRestore(); }
}
const publications = () => db.select().from(chatPublications).where(eq(chatPublications.issueId, f.issueId));
it.each(["inventory", "lifecycle", "github repository", "telegram migration"] as const)("wakes only reopened conversations after %s commits, preserving rollback and completed history", async path => {
const provider = path === "github repository" ? "github" : path === "telegram migration" ? "telegram" : "slack";
const resourceType = provider === "github" ? "repository" : provider === "telegram" ? "chat" : "channel";
const providerResourceId = provider === "github" ? "owner/repo" : provider === "telegram" ? "-100222" : "recovered-channel";
await db.update(chatEndpoints).set({ provider }).where(eq(chatEndpoints.id, f.endpointId));
const runId = randomUUID();
await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: f.agentId,
responsibleUserId: f.userId, status: "failed", errorCode: "test_run_failed", updatedAt: new Date(),
contextSnapshot: { issueId: f.issueId, source: `chat:${provider}`, endpointId: f.endpointId, wakeCommentId: f.commentId },
});
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, f.issueId));
const [endpoint] = await db.select().from(chatEndpoints).where(eq(chatEndpoints.id, f.endpointId));
const [resource] = await db.insert(chatEndpointResources).values({ companyId: f.companyId, endpointId: f.endpointId,
type: resourceType, providerResourceId: provider === "telegram" ? "-111" : providerResourceId, label: "Recovered", availability: "unavailable", enabled: true,
}).returning();
await db.update(chatConversations).set({ resourceId: resource.id, state: "unavailable", ...(provider === "telegram" ? { externalConversationId: "telegram:-111", externalThreadId: "telegram:-111:42" } : {}) }).where(eq(chatConversations.id, f.conversationId));
const completedId = randomUUID();
await db.insert(chatConversations).values({ id: completedId, companyId: f.companyId, endpointId: f.endpointId,
// Telegram migration intentionally rebinds all conversations on the old resource.
// Unrelated completed history must remain passive.
resourceId: provider === "telegram" ? null : resource.id, issueId: f.issueId, state: "completed", externalConversationId: completedId, externalThreadId: completedId, externalLabel: "History" });
const [delivery] = await db.insert(chatDeliveries).values({ companyId: f.companyId, endpointId: f.endpointId,
providerEventId: randomUUID(), deduplicationKey: randomUUID(), eventKind: "installation", normalizedEvent: {}, state: "processing", attempts: 1,
}).returning();
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: error => errors.push(error) });
await chatRunMilestoneWork(db).register(coordinator, () => true).ready;
await flush(); expect(await publications()).toEqual([]); await quiet();
const withLease = chatCredentialMutationLease(db);
const recover = (rollback: boolean, alreadyPublished = false) => withLease(endpoint, lease => db.transaction(async tx => {
const resources = createChatResourceReconciliation(tx as unknown as typeof db, {
endpointRecord: async () => ({ endpoint }), withCredentialMutationLease: withLease, runtimeGeneration: () => 0, maxErrorText: 1000,
});
if (path === "inventory") {
await resources.reconcileProviderResourceRows(endpoint, { provider, resources: [{ type: resourceType, providerResourceId, label: "Recovered" }] }, lease);
} else if (path === "github repository") {
await resources.reconcileGitHubWebhookRepository(endpoint, { repository: { id: 123, full_name: providerResourceId, name: "repo", owner: { id: 321, login: "owner" } } }, lease);
} else {
await resources.applyProviderAvailability({ currentEndpoint: endpoint, credentialLease: lease, staleResourceEffect: false,
claimed: delivery, candidate: { attempts: 0 }, effect: { kind: "resource", provider, providerEventId: delivery.providerEventId,
providerResourceId, resourceType, ...(provider === "telegram" ? { previousProviderResourceId: "-111" } : {}), label: "Recovered", availability: "available" },
});
}
// The owner used a nested transaction. Its savepoint release cannot publish.
await vi.advanceTimersByTimeAsync(1000);
expect(await publications()).toHaveLength(alreadyPublished ? 1 : 0);
if (rollback) throw new Error("outer rollback");
}));
await expect(recover(true)).rejects.toThrow("outer rollback");
await flush(); expect(await publications()).toEqual([]); await quiet();
await recover(false); await flush();
expect(await publications()).toEqual([expect.objectContaining({ idempotencyKey: `run:${runId}:failed:${f.endpointId}` })]);
expect((await db.select().from(chatConversations).where(eq(chatConversations.id, completedId)))[0].state).toBe("completed");
if (provider === "telegram") {
expect((await db.select().from(chatConversations).where(eq(chatConversations.id, f.conversationId)))[0])
.toMatchObject({ externalConversationId: "telegram:-100222", externalThreadId: "telegram:-100222:42", state: "active" });
}
await quiet();
// Unchanged inventory must not turn a provider refresh into milestone polling.
const notices: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, event => notices.push(event));
try {
await withLease(endpoint, lease => createChatResourceReconciliation(db, {
endpointRecord: async () => ({ endpoint }), withCredentialMutationLease: withLease, runtimeGeneration: () => 0, maxErrorText: 1000,
}).reconcileProviderResourceRows(endpoint, { provider, resources: [{ type: resourceType, providerResourceId, label: "Recovered" }] }, lease));
expect(notices).toEqual([]);
if (provider === "telegram") {
// A duplicate migration has no old bindings left to reactivate.
await recover(false, true);
expect(notices).toEqual([]);
}
} finally { unsubscribe(); }
await flush(); await quiet();
});
});
@@ -0,0 +1,987 @@
import { documentService } from "../services/documents.js";
import { createChatMilestoneFixtures } from "./helpers/chat-milestone-fixtures.js";
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { agents, agentWakeupRequests, caseHandoffs, cases, chatConversations, chatEndpoints, chatPublications, completionContracts, createDb, nativeRunFinalizations, nativeRunResults, statusDecisions, statusDecisionEffects, workAssessments, subscribeDatabaseWork, heartbeatRuns, issueComments, issueQuestionResponseDeliveries, issueThreadInteractions, issues } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { chatRunMilestoneWork } from "../services/chat-run-milestone-work.js";
import { notifyChatRunMilestones } from "../services/chat-run-milestone-notifications.js";
import { issueThreadInteractionService } from "../services/issue-thread-interactions.js";
import { questionResponseDeliveryService } from "../services/question-response-delivery.js";
import { createQueuedCommentIssueLockWriter } from "../modules/wake-queue/adapters/queued-comment-postgres.js";
import { createDiscardQueuedComment, createReorderQueuedComments } from "../modules/wake-queue/application/queued-comment-use-cases.js";
import { createRunDispatch } from "../modules/run-dispatch/index.js";
import { createAdmissionTransactionScope, createPostgresWakeQueueAdapter, createWakeAdmissionWriter } from "../modules/wake-queue/adapters/postgres.js";
import { instanceSettingsService } from "../services/instance-settings.js";
import { createHeartbeatQueue } from "../services/heartbeat/queue.js";
import { createHeartbeatQueueTestDependencies } from "./helpers/heartbeat-queue.js";
import { CHAT_CONTROL_RECOVERY_ADMISSION_KEY, CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE } from "../services/chat-control-recovery-stop.js";
import { heartbeatService } from "../services/heartbeat.js";
import { commitNativeStatusDecision, dispatchPendingNativeStatusEffects } from "../services/native-runtime/status-decision-committer.js";
import { claimQueuedNativeReviewRun } from "../services/native-runtime/native-review-dispatch.js";
import { createChatPublicationResolution } from "../services/chat-channels/publications/resolution.js";
import { createChatPublicationWorker } from "../services/chat-channels/publications/worker.js";
import { CAPABILITIES } from "../services/chat-channels/provider-catalog.js";
import { createChatPublicationFailures } from "../services/chat-channels/publications/failures.js";
import { PaperclipRunnerToolAuthority } from "../services/native-runtime/paperclip-runner-tool-authority.js";
import { finalizeNativeRun } from "../services/native-runtime/native-run-finalizer.js";
import { idleWorkSnapshot } from "../services/task-admission.js";
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_MILESTONE_TEST_DATABASE_URL");
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
(support.supported ? describe : describe.skip)("chat lifecycle milestone worker", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: ReturnType<typeof createDb>;
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
let enabled = true;
const errors: unknown[] = [];
let fixtures: ReturnType<typeof createChatMilestoneFixtures>;
type Fixture = Awaited<ReturnType<typeof fixture>>;
let f: Fixture;
beforeAll(async () => {
if (!external) temporary = await startEmbeddedPostgresTestDatabase("chat-milestones-");
db = createDb(external ?? temporary!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
fixtures = createChatMilestoneFixtures(db);
}, 60_000);
const fixture = (companyId?: string) => fixtures.create(companyId);
beforeEach(async () => {
enabled = true; errors.length = 0; f = await fixture();
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
});
afterEach(async () => {
await coordinator?.stop(); coordinator = undefined; vi.useRealTimers(); vi.restoreAllMocks();
await fixtures.cleanup();
expect(errors).toEqual([]);
});
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
function runValues(scope = f, patch: Partial<typeof heartbeatRuns.$inferInsert> = {}) {
return { id: randomUUID(), companyId: scope.companyId, agentId: scope.agentId, responsibleUserId: scope.userId, status: "running", updatedAt: new Date(),
contextSnapshot: { issueId: scope.issueId, source: "chat:slack", endpointId: scope.endpointId, wakeCommentId: scope.commentId }, ...patch };
}
async function run(scope = f, patch: Partial<typeof heartbeatRuns.$inferInsert> = {}) {
const values = runValues(scope, patch);
await db.transaction(async tx => { await notifyChatRunMilestones(tx, scope); await tx.insert(heartbeatRuns).values(values); });
return values;
}
async function start() {
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => enabled, canReconcile: () => true, onError: error => errors.push(error) });
await chatRunMilestoneWork(db).register(coordinator, () => enabled).ready;
}
async function flush() {
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(1000);
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBeNull();
}, { timeout: 10_000 });
}
async function quiet(ms = 120_000) {
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
try { await vi.advanceTimersByTimeAsync(ms); expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); }
finally { reads.mockRestore(); writes.mockRestore(); }
}
const publications = (scope = f) => db.select().from(chatPublications).where(eq(chatPublications.issueId, scope.issueId));
it("leaves configured idle, internal runs and old history quiet", async () => {
await run(f, { contextSnapshot: { issueId: f.issueId, source: "issue.comment" } });
await run(f, { updatedAt: new Date(Date.now() - 2 * 86_400_000) });
await start(); await flush(); expect(await publications()).toEqual([]); await quiet();
});
it("reconstructs missing notices at startup and leaves published history quiet across restart", async () => {
const row = await run(); await start(); await flush();
expect(await publications()).toEqual([expect.objectContaining({ idempotencyKey: `run:${row.id}:working:${f.endpointId}` })]);
await quiet(); await coordinator!.stop(); await start(); await flush(); await quiet();
expect(await publications()).toHaveLength(1);
});
it("wakes on the first committed row after becoming quiet", async () => {
await start(); await flush(); await quiet(); await run(); await flush();
expect(await publications()).toHaveLength(1); await quiet();
});
it("wakes from the real run cancellation transaction without a manual milestone signal", async () => {
const row = await run(f, { status: "queued", runtimeMode: "legacy" });
await start(); await flush(); await quiet();
// This fixture exercises the Stop producer, not successor execution.
await heartbeatService(db).cancelRun(row.id, "Operator stopped queued work", { suppressImmediateRecovery: true });
await flush();
expect((await publications()).map(publication => publication.idempotencyKey).sort()).toEqual([
`run:${row.id}:failed:${f.endpointId}`, `run:${row.id}:queued:${f.endpointId}`,
].sort());
await quiet();
});
it.each(["promote", "suppress"] as const)("wakes when real retry dispatch chooses to %s", async action => {
const row = await run(f, { status: "scheduled_retry", runtimeMode: "legacy", scheduledRetryAt: new Date(Date.now() - 1000), scheduledRetryReason: "transient_error" });
if (action === "suppress") await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, f.issueId));
await start(); await flush(); expect(await publications()).toEqual([]); await quiet();
const dispatch = createRunDispatch(db);
const result = await dispatch.promoteScheduledRetry({ companyId: f.companyId, runId: row.id });
expect(result.outcome).toBe(action === "promote" ? "promoted" : "gate_suppressed");
await flush();
expect((await publications()).map(publication => publication.idempotencyKey)).toContain(
`run:${row.id}:${action === "promote" ? "queued" : "failed"}:${f.endpointId}`,
);
await quiet();
expect((await dispatch.promoteScheduledRetry({ companyId: f.companyId, runId: row.id })).outcome).toBe("not_promoted");
await quiet();
});
it("wakes when the real staleness gate cancels queued work", async () => {
const row = await run(f, { status: "queued", runtimeMode: "legacy", scheduledRetryReason: "max_turns_continuation" });
await start(); await flush(); await quiet();
await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, f.issueId));
const dispatch = createRunDispatch(db);
await dispatch.cancelStaleQueuedRun({ companyId: f.companyId, runId: row.id, expectedStatus: "queued" });
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, row.id)))[0].status).toBe("cancelled");
await flush();
expect((await publications()).map(publication => publication.idempotencyKey)).toContain(`run:${row.id}:failed:${f.endpointId}`);
await quiet();
await dispatch.cancelStaleQueuedRun({ companyId: f.companyId, runId: row.id, expectedStatus: "queued" });
await quiet();
});
it.each(["deferred", "immediate"] as const)("wakes when real %s recovery admits chat work", async kind => {
const finishing = await run(f, { status: "succeeded", runtimeMode: "legacy", finishedAt: new Date(),
contextSnapshot: { issueId: f.issueId, source: "issue_assigned" },
});
const wakeId = randomUUID();
if (kind === "deferred") await db.insert(agentWakeupRequests).values({ id: wakeId, companyId: f.companyId, agentId: f.agentId,
source: "on_demand", status: "deferred_issue_execution", payload: { issueId: f.issueId },
});
const adapter = createPostgresWakeQueueAdapter(db, {
resolveResponsibleUserId: async () => f.userId,
getRoutineEnv: async () => ({ routineId: null, env: null, responsibleUserId: null }),
resolveSessionBeforeForWakeup: async () => null,
});
await start(); await flush(); expect(await publications()).toEqual([]); await quiet();
let admittedId: string | undefined;
await adapter.withIssueExecutionLock({ companyId: f.companyId, runId: finishing.id, now: new Date() }, async (locked, ports) => {
const input = { companyId: f.companyId, issue: locked.primaryIssue, finishingRun: locked.run,
contextSnapshot: runValues().contextSnapshot, responsibleUserId: f.userId, sessionBefore: null, now: new Date(),
};
const agent = { id: f.agentId, companyId: f.companyId, name: "Maya", invokable: true };
if (kind === "deferred") {
expect(await ports.transaction.claimDeferredWakeForPromotion({ companyId: f.companyId, wakeId, now: new Date() })).toBe(true);
admittedId = (await ports.transaction.finalizePromotedWake({ ...input, wakeId, deferredAgent: agent,
reason: "issue_execution_promoted", source: "on_demand", triggerDetail: null, payload: {},
})).id;
} else {
admittedId = (await ports.transaction.queueImmediateRecoveryRun({ ...input, recoveryAgent: agent, reason: "issue_execution_recovery" })).id;
}
expect(await publications()).toEqual([]);
return { outcome: { kind: "released" }, postCommitEffects: [] };
});
expect(admittedId).toBeDefined();
await flush();
expect((await publications()).map(publication => publication.idempotencyKey)).toContain(`run:${admittedId}:queued:${f.endpointId}`);
await quiet();
});
async function queue() {
const dependencies = createHeartbeatQueueTestDependencies(db);
const [agent] = await db.select().from(agents).where(eq(agents.id, f.agentId));
dependencies.getAgent.mockResolvedValue(agent);
dependencies.instanceSettings.getExperimental.mockImplementation(() => instanceSettingsService(db).getExperimental());
return { service: createHeartbeatQueue(db, dependencies), dependencies };
}
it("wakes on real queue admission while execution is held by drain", async () => {
const { service, dependencies } = await queue();
dependencies.getSchedulingSuppression.mockResolvedValue({ suppressed: true, reason: "task_drain" });
await start(); await flush(); await quiet();
const admitted = await service.enqueueWakeup(f.agentId, { source: "on_demand", reason: "issue_commented",
contextSnapshot: runValues().contextSnapshot, requestedByActorType: "user", requestedByActorId: f.userId,
});
expect(admitted).toMatchObject({ status: "queued", companyId: f.companyId });
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${admitted!.id}:queued:${f.endpointId}`);
expect(dependencies.executeRun).not.toHaveBeenCalled();
await quiet();
});
it.each(["ordinary", "legacy_comments", "queued_comments"] as const)("wakes on a real %s queue claim", async kind => {
const wakeId = randomUUID();
const context = runValues().contextSnapshot;
await db.insert(agentWakeupRequests).values({ id: wakeId, companyId: f.companyId, agentId: f.agentId,
source: "on_demand", status: "queued",
payload: kind === "queued_comments" ? { _paperclipWakeContext: { wakeCommentIds: [f.commentId] } } : {},
});
const seeded = await run(f, { status: "queued", runtimeMode: "legacy", wakeupRequestId: wakeId,
contextSnapshot: kind === "ordinary" ? context : { ...context, wakeCommentIds: [f.commentId] },
});
await db.update(agentWakeupRequests).set({ runId: seeded.id }).where(eq(agentWakeupRequests.id, wakeId));
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.id));
const { service, dependencies } = await queue();
await start(); await flush(); await quiet();
expect(await service.claimQueuedRun(before)).toMatchObject({ id: seeded.id, status: "running" });
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${seeded.id}:working:${f.endpointId}`);
expect(dependencies.executeRun).not.toHaveBeenCalled();
await quiet();
expect(await service.claimQueuedRun(before)).toBeNull();
await quiet();
});
it("wakes when claim rejects a superseded case handoff and leaves repeated claims quiet", async () => {
const wakeId = randomUUID();
await db.insert(agentWakeupRequests).values({ id: wakeId, companyId: f.companyId, agentId: f.agentId,
source: "assignment", status: "queued", payload: { issueId: f.issueId },
});
const seeded = await run(f, { status: "queued", runtimeMode: "legacy", wakeupRequestId: wakeId });
const [record] = await db.insert(cases).values({ companyId: f.companyId, caseNumber: 1, identifier: "M-CASE-1",
caseType: "work", title: "Chat task", workIssueId: f.issueId, handoffVersion: 2 }).returning();
await db.insert(caseHandoffs).values({ companyId: f.companyId, caseId: record.id, issueId: f.issueId,
version: 1, toAgentId: f.agentId, phase: "implementation", instructions: "Build the change", actorType: "system",
actorId: "test", idempotencyKey: "superseded", inputFingerprint: "fixture", wakeupRequestId: wakeId,
expectedIssueStatusVersion: 0, appliedIssueStatusVersion: 0, state: "delivered" });
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.id));
const { service, dependencies } = await queue();
await start(); await flush(); await quiet();
expect(await service.claimQueuedRun(before)).toBeNull();
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.id)))[0])
.toMatchObject({ status: "cancelled", errorCode: "case_handoff_superseded" });
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${seeded.id}:failed:${f.endpointId}`);
expect(dependencies.executeRun).not.toHaveBeenCalled();
await quiet();
expect(await service.claimQueuedRun(before)).toBeNull();
await quiet();
});
it("wakes when dispatch fails an unverifiable chat recovery and leaves repeated gates quiet", async () => {
const seeded = await run(f, { status: "running", runtimeMode: "legacy", invocationSource: "automation",
runnerProfileJson: { [CHAT_CONTROL_RECOVERY_ADMISSION_KEY]: {} },
});
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.id));
const { service, dependencies } = await queue();
const dispatch = vi.fn(async () => null);
await start(); await flush(); await quiet();
await service.withChatControlRecoveryGate(before, "dispatch", dispatch);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.id)))[0])
.toMatchObject({ status: "failed", errorCode: CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE });
expect(dispatch).not.toHaveBeenCalled(); expect(dependencies.executeRun).not.toHaveBeenCalled();
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${seeded.id}:failed:${f.endpointId}`);
await quiet();
await service.withChatControlRecoveryGate(before, "dispatch", dispatch);
await quiet();
});
it.each(["queued", "scheduled_retry"] as const)("publishes cancellation when admission replaces a reassigned %s owner", async status => {
const old = await run(f, { status, runtimeMode: "legacy", scheduledRetryAt: status === "scheduled_retry" ? new Date(Date.now() + 60_000) : null });
const [replacement] = await db.insert(agents).values({ companyId: f.companyId, name: "Replacement", adapterType: "paperclip_runner" }).returning();
await db.update(issues).set({ assigneeAgentId: replacement.id, executionRunId: old.id }).where(eq(issues.id, f.issueId));
const { service, dependencies } = await queue();
dependencies.getAgent.mockResolvedValue(replacement);
dependencies.getSchedulingSuppression.mockResolvedValue({ suppressed: true, reason: "task_drain" });
await start(); await flush(); await quiet();
const admitted = await service.enqueueWakeup(replacement.id, { source: "on_demand", reason: "issue_commented",
contextSnapshot: runValues().contextSnapshot, requestedByActorType: "user", requestedByActorId: f.userId,
});
expect(admitted).toMatchObject({ agentId: replacement.id, status: "queued" });
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, old.id)))[0])
.toMatchObject({ status: "cancelled", errorCode: status === "queued" ? "lock_released_on_reassignment" : "issue_reassigned" });
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${old.id}:failed:${f.endpointId}`);
expect(dependencies.executeRun).not.toHaveBeenCalled(); await quiet();
});
async function editableQueue(staleContext = false) {
const wakeId = randomUUID();
await db.update(issueComments).set({ authorUserId: f.userId }).where(eq(issueComments.id, f.commentId));
await db.insert(agentWakeupRequests).values({ id: wakeId, companyId: f.companyId, agentId: f.agentId,
source: "on_demand", status: "queued", payload: { issueId: f.issueId, _paperclipWakeContext: { wakeCommentIds: [f.commentId] } },
});
const seeded = await run(f, { status: "queued", runtimeMode: "legacy", wakeupRequestId: wakeId,
contextSnapshot: { ...runValues().contextSnapshot, wakeCommentId: staleContext ? randomUUID() : f.commentId },
});
await db.update(agentWakeupRequests).set({ runId: seeded.id }).where(eq(agentWakeupRequests.id, wakeId));
const issue = { id: f.issueId, companyId: f.companyId, assigneeAgentId: f.agentId, executionRunId: seeded.id };
const actor = { actorType: "user" as const, actorId: f.userId, agentId: null, runId: null, agentApiKeyId: null };
const issueLock = createQueuedCommentIssueLockWriter(db, {
syncCommentReferences: async () => {}, deleteCommentReferenceSource: async () => {}, syncCommentExternalObjectsSafely: async () => {},
});
const snapshot = await issueLock.withLockedQueue({ issue, actor, queueId: wakeId }, async locked => locked.queue);
return { seeded, issueLock, input: { issue, actor, queueId: wakeId, revision: snapshot.revision, now: new Date() } };
}
it("wakes when queued-comment mutation reconstructs run lineage from its authoritative queue", async () => {
const { seeded, issueLock, input } = await editableQueue(true);
await start(); await flush(); expect(await publications()).toEqual([]); await quiet();
await createReorderQueuedComments({ issueLock })({ ...input, orderedCommentIds: [f.commentId] });
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${seeded.id}:queued:${f.endpointId}`);
await quiet();
});
it("settles queued-comment discard intent only after the owning transaction commits", async () => {
const { seeded, input } = await editableQueue();
const events: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, "chat-run-milestones", event => events.push(event));
try {
await db.transaction(async tx => {
const writer = createQueuedCommentIssueLockWriter(tx as unknown as typeof db, {
syncCommentReferences: async () => {}, deleteCommentReferenceSource: async () => {}, syncCommentExternalObjectsSafely: async () => {},
});
await createDiscardQueuedComment({ issueLock: writer })({ ...input, commentId: f.commentId });
expect(events).toContain("intent"); expect(events).not.toContain("settled");
});
expect(events).toContain("settled");
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.id)))[0])
.toMatchObject({ status: "cancelled", errorCode: "queued_comment_discarded" });
} finally { unsubscribe(); }
});
it("wakes when the real coalescing writer gives an existing run chat lineage", async () => {
const seeded = await run(f, { contextSnapshot: { issueId: f.issueId, source: "issue_assigned" } });
await start(); await flush(); expect(await publications()).toEqual([]); await quiet();
await db.transaction(async tx => {
await createWakeAdmissionWriter().coalesceIntoActiveExecutionRun(
createAdmissionTransactionScope(f.companyId, tx as unknown as typeof db), {
companyId: f.companyId, activeExecutionRunId: seeded.id, mergedContextSnapshot: runValues().contextSnapshot,
agentId: f.agentId, source: "on_demand", triggerDetail: null, payload: { issueId: f.issueId },
requestedByActorType: "user", requestedByActorId: f.userId, idempotencyKey: null,
});
expect(await publications()).toEqual([]);
});
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${seeded.id}:working:${f.endpointId}`);
await quiet();
});
it("wakes on a claim released for drain and leaves repeated release quiet", async () => {
const seeded = await run();
const { service } = await queue();
await start(); await flush(); await quiet();
await service.releaseRunClaimedJustBeforeSuppression(seeded.id);
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${seeded.id}:queued:${f.endpointId}`);
await quiet();
await service.releaseRunClaimedJustBeforeSuppression(seeded.id);
await quiet();
});
it("wakes an admitted monitor through its previously committed source receipt", async () => {
const source = await run(f, { runtimeMode: "native", nativeIssueId: f.issueId });
await db.update(issues).set({ executionRunId: source.id }).where(eq(issues.id, f.issueId));
await start(); await flush(); await quiet();
const nextCheckAt = new Date(Date.now() + 60_000).toISOString();
const authority = new PaperclipRunnerToolAuthority(db, { ...f, runId: source.id });
const request = { tool: "set_task_monitor", callId: randomUUID(), arguments: {
idempotencyKey: "milestone-monitor", monitor: { nextCheckAt, notes: "Check the saved task" },
} };
await expect(authority.execute(request)).resolves.toMatchObject({ taskId: f.issueId, replayed: false });
// The monitor and source receipt commit together, before due admission. This
// fixture finishes the source without executing a harness; admission below
// uses the real queue, with execution held by task drain.
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, source.id));
await db.update(issues).set({ executionRunId: null }).where(eq(issues.id, f.issueId));
await quiet();
const { service, dependencies } = await queue();
dependencies.getSchedulingSuppression.mockResolvedValue({ suppressed: true, reason: "task_drain" });
const admitted = await service.enqueueWakeup(f.agentId, { source: "automation", reason: "issue_monitor_due",
idempotencyKey: `issue-monitor:${f.issueId}:${nextCheckAt}`,
payload: { issueId: f.issueId, nextCheckAt },
contextSnapshot: { issueId: f.issueId, source: "issue.monitor" },
requestedByActorType: "user", requestedByActorId: f.userId,
});
expect(admitted).toMatchObject({ status: "queued" });
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${admitted!.id}:queued:${f.endpointId}`);
expect(dependencies.executeRun).not.toHaveBeenCalled();
await quiet();
});
async function nativeFixture(phase = "committed") {
const contractId = randomUUID(), resultId = randomUUID();
await db.insert(completionContracts).values({
id: contractId, companyId: f.companyId, issueId: f.issueId, revision: 1,
schemaVersion: "paperclip.completion-contract.v1", policyVersion: "phase6-v1", risk: "standard",
completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal",
contractJson: { objective: "Respond to chat" }, canonicalSha256: "milestone-contract",
createdByActorType: "system", createdByActorId: "test",
});
// Persisted audit-only routing can commit without a status decision. Recovery
// still owes the terminal run projection after the controller restarts.
const row = await run(f, { runtimeMode: "native", nativeIssueId: f.issueId, completionContractId: contractId,
resultJson: { nativeAttentionRouting: [{ decisionId: null, reasonCode: "attention_duplicate_suppressed",
materializedTargets: [{ kind: "existing_attention" }] }] },
});
await db.insert(nativeRunResults).values({ id: resultId, companyId: f.companyId, issueId: f.issueId,
runId: row.id, completionContractId: contractId, serverFingerprint: resultId, schemaStatus: "accepted",
resultJson: { terminal: { runTerminalState: "cancelled" } }, canonicalSha256: "milestone-result",
});
await db.insert(nativeRunFinalizations).values({ runId: row.id, companyId: f.companyId, issueId: f.issueId,
phase, resultId,
});
return { row, contractId, resultId };
}
async function continuationDecisionFixture() {
const native = await nativeFixture("assessing");
const assessmentId = randomUUID();
await db.insert(workAssessments).values({ id: assessmentId, companyId: f.companyId, issueId: f.issueId,
runId: native.row.id, contractId: native.contractId, resultId: native.resultId, triggerKind: "run_result",
triggerActorCompanyId: f.companyId, priorIssueStatus: "in_progress", priorStatusVersion: 0,
policyVersion: "phase6-v1", assessmentJson: {}, inputDigest: assessmentId,
});
return { companyId: f.companyId, issueId: f.issueId, runId: native.row.id, assessmentId,
priorStatus: "in_progress", priorStatusVersion: 0, priorDecisionId: null,
decision: { policyVersion: "phase6-v1", statusAction: "preserve" as const, toStatus: "in_progress" as const,
reasonCode: "live_continuation_registered", unblockDescriptor: null,
effects: [{ kind: "enqueue_continuation" as const, agentId: f.agentId, continuationKind: "same_agent" as const,
summary: "Continue responding", idempotencyKey: `continuation:${native.row.id}` }],
},
};
}
it("wakes from a governed native reviewer claim and leaves repeated claims quiet", async () => {
const input = await continuationDecisionFixture();
const { decision } = await commitNativeStatusDecision({ db, ...input });
const [reviewer] = await db.insert(agents).values({ companyId: f.companyId, name: "Reviewer", adapterType: "paperclip_runner" }).returning();
await db.update(statusDecisions).set({ decisionJson: { ...decision.decisionJson, projectedStatusVersion: 1 } }).where(eq(statusDecisions.id, decision.id));
await db.update(issues).set({ status: "in_review", statusVersion: 1, lastStatusDecisionId: decision.id, executionRunId: null }).where(eq(issues.id, f.issueId));
await db.update(chatEndpoints).set({ assignedAgentId: reviewer.id }).where(eq(chatEndpoints.id, f.endpointId));
const [interaction] = await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId: f.issueId,
kind: "request_confirmation", status: "pending", sourceRunId: input.runId, addresseeAgentId: reviewer.id,
effectiveResolverPolicy: "anyone", resolverPolicyProvenance: "inherited",
payload: { version: 1, prompt: "Review the work", target: { type: "custom", key: "native_completion_review", revisionId: decision.id } },
}).returning();
const wakeId = randomUUID();
await db.insert(agentWakeupRequests).values({ id: wakeId, companyId: f.companyId, agentId: reviewer.id, source: "automation", status: "queued" });
const seeded = await run(f, { agentId: reviewer.id, status: "queued", runtimeMode: "native", nativeIssueId: f.issueId, wakeupRequestId: wakeId,
contextSnapshot: { ...runValues().contextSnapshot, nativeReviewInteractionId: interaction.id, nativeReviewDecisionId: decision.id },
});
await db.update(agentWakeupRequests).set({ runId: seeded.id }).where(eq(agentWakeupRequests.id, wakeId));
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.id));
const claim = { run: before, agentNameKey: "reviewer", claimedAt: new Date(), claimValues: {} };
await start(); await flush(); await quiet();
expect(await claimQueuedNativeReviewRun(db, claim)).toMatchObject({ id: seeded.id, status: "running" });
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${seeded.id}:working:${f.endpointId}`);
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0].executionRunId).toBe(seeded.id);
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wakeId)))[0].status).toBe("claimed");
await quiet();
expect(await claimQueuedNativeReviewRun(db, claim)).toBeNull(); await quiet();
});
it("wakes from native committed-result projection and leaves repeated finalization quiet", async () => {
const { row } = await nativeFixture();
await start(); await flush(); await quiet();
await finalizeNativeRun({ db, runId: row.id, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
await flush();
expect((await publications()).map(publication => publication.idempotencyKey).sort()).toEqual([
`run:${row.id}:failed:${f.endpointId}`, `run:${row.id}:working:${f.endpointId}`,
].sort());
await quiet();
await finalizeNativeRun({ db, runId: row.id, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
// A no-op repair must not enqueue another scan. This also catches deferred
// coordinator callbacks that start after the finalizer returns.
await quiet();
expect(await publications()).toHaveLength(2);
});
it("signals a new native decision at outer commit but not an already committed replay", async () => {
const input = await continuationDecisionFixture();
const events: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, "chat-run-milestones", event => events.push(event));
try {
await db.transaction(async tx => {
await commitNativeStatusDecision({ db: tx as unknown as typeof db, ...input });
expect(events).toEqual(["intent"]);
});
expect(events).toEqual(["intent", "settled"]);
events.length = 0;
await expect(commitNativeStatusDecision({ db, ...input })).resolves.toMatchObject({ replayed: true });
expect(events).toEqual([]);
} finally { unsubscribe(); }
});
it("wakes a suppressed continuation when its native effect delivery receipt commits", async () => {
const input = await continuationDecisionFixture();
const { decision } = await commitNativeStatusDecision({ db, ...input });
const [intent] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId));
// Reconstruct an interrupted delivery with its already admitted successor.
// The target has no direct chat identity: only the delivered ledger can
// authorize inheriting the source run's conversation.
await db.update(statusDecisionEffects).set({ deliveryState: "pending", deliveredAt: null })
.where(eq(statusDecisionEffects.decisionId, decision.id));
const target = await run(f, { status: "queued", contextSnapshot: {
issueId: f.issueId, source: "native_status_decision", nativeStatusWakeIntentId: intent.id,
} });
await db.update(agentWakeupRequests).set({ status: "coalesced", runId: target.id }).where(eq(agentWakeupRequests.id, intent.id));
await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",
reason: "issue_status_changed", requestedByActorType: "system", requestedByActorId: `native-status-wake-dispatch:${intent.id}`,
runId: target.id, status: "queued", idempotencyKey: intent.idempotencyKey, payload: { issueId: f.issueId },
});
await start(); await flush();
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${target.id}:`))).toBe(false);
await quiet();
await dispatchPendingNativeStatusEffects({ db, ...input, decisionId: decision.id });
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${target.id}:queued:${f.endpointId}`);
await quiet();
await expect(dispatchPendingNativeStatusEffects({ db, ...input, decisionId: decision.id })).resolves.toEqual([]);
await quiet();
});
it("wakes an admitted continuation when native wake dispatch repairs its original receipt", async () => {
const input = await continuationDecisionFixture();
await commitNativeStatusDecision({ db, ...input });
const [intent] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId));
const target = await run(f, { status: "queued", contextSnapshot: {
issueId: f.issueId, source: "native_status_decision", nativeStatusWakeIntentId: intent.id,
} });
await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",
reason: "issue_status_changed", requestedByActorType: "system", requestedByActorId: `native-status-wake-dispatch:${intent.id}`,
runId: target.id, status: "queued", idempotencyKey: intent.idempotencyKey, payload: { issueId: f.issueId },
});
await start(); await flush();
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${target.id}:`))).toBe(false);
await quiet();
const { service, dependencies } = await queue();
await expect(service.dispatchPendingNativeStatusWakeups({ companyId: f.companyId })).resolves.toMatchObject({ recovered: 1, dispatched: 0 });
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${target.id}:queued:${f.endpointId}`);
expect(dependencies.executeRun).not.toHaveBeenCalled();
await quiet();
await expect(service.dispatchPendingNativeStatusWakeups({ companyId: f.companyId })).resolves.toMatchObject({ scanned: 0 });
await quiet();
});
it("rechecks a fresh native continuation after admission finishes its wake receipt", async () => {
const input = await continuationDecisionFixture();
await commitNativeStatusDecision({ db, ...input });
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, input.runId));
await start(); await flush(); await quiet();
const { service, dependencies } = await queue();
let admittedId: string | undefined;
dependencies.getSchedulingSuppression.mockImplementation(async () => {
const rows = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId));
const admitted = rows.find(row => row.id !== input.runId);
if (admitted && !admittedId) {
admittedId = admitted.id;
// Admission already committed and signalled, but the original wake
// receipt is still claimed. Let that first projection finish now.
await flush();
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${admittedId}:`))).toBe(false);
}
return { suppressed: true, reason: "task_drain" };
});
await expect(service.dispatchPendingNativeStatusWakeups({ companyId: f.companyId })).resolves.toMatchObject({ dispatched: 1 });
expect(admittedId).toBeDefined();
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${admittedId}:queued:${f.endpointId}`);
expect(dependencies.executeRun).not.toHaveBeenCalled();
await quiet();
});
it("projects a continuation only after the real question delivery receipt commits", async () => {
const source = await run(f, { status: "succeeded" });
const interactions = issueThreadInteractionService(db);
const interaction = await interactions.create({ id: f.issueId, companyId: f.companyId }, {
kind: "ask_user_questions", continuationPolicy: "wake_assignee", sourceRunId: source.id,
title: "Choose", payload: { version: 1, title: "Choose", questions: [{
id: "choice", prompt: "Continue?", selectionMode: "single", required: true,
options: [{ id: "yes", label: "Yes" }],
}] },
}, { agentId: f.agentId, runId: source.id });
await interactions.answerQuestions({ id: f.issueId, companyId: f.companyId, status: "in_progress" }, interaction.id,
{ answers: [{ questionId: "choice", optionIds: ["yes"] }] }, { userId: f.userId });
// A provider acknowledgement is durable input to this projection test.
await db.insert(chatPublications).values({ companyId: f.companyId, endpointId: f.endpointId, conversationId: f.conversationId,
issueId: f.issueId, idempotencyKey: `interaction:${interaction.id}:${f.endpointId}`,
payload: { text: "Continue?", interactionId: interaction.id }, state: "published", providerMessageId: "question-1",
}).onConflictDoNothing();
await db.update(chatPublications).set({ state: "published", providerMessageId: "question-1" }).where(eq(chatPublications.issueId, f.issueId));
await start(); await flush(); await quiet();
let targetId!: string;
const delivery = questionResponseDeliveryService(db, { heartbeat: {
cancelRun: async () => { throw new Error("The source is already terminal"); },
wakeup: async (_agentId, options) => {
const values = runValues(f, { status: "queued", contextSnapshot: options.contextSnapshot });
const [target] = await db.insert(heartbeatRuns).values(values).returning();
targetId = target.id;
await flush();
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${target.id}:`))).toBe(false);
return target;
},
} });
await expect(delivery.deliver(interaction.id)).resolves.toMatchObject({ status: "fallback_queued" });
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${targetId}:queued:${f.endpointId}`);
await quiet();
});
it.each([
["question", "answer"], ["question", "skip"], ["question", "cancel"],
["question", "withdraw"], ["question", "expire"], ["question", "historical answer"],
["confirmation", "accept"], ["confirmation", "reject"], ["confirmation", "skip"],
["confirmation", "withdraw"], ["confirmation", "expire"],
["question", "comment"], ["question", "historical comment"],
["confirmation", "comment"], ["confirmation", "historical comment"],
["question", "replacement"], ["confirmation", "replacement"], ["confirmation", "startup supersession"],
["question", "addressee deletion"], ["confirmation", "addressee deletion"],
] as const)("rechecks an unsent %s after %s", async (kind, action) => {
const source = await run(f, { status: "failed", errorCode: "test_run_failed" });
const interactions = issueThreadInteractionService(db);
const issue = { id: f.issueId, companyId: f.companyId, status: "in_progress" };
const addressee = action === "addressee deletion"
? (await db.insert(agents).values({ companyId: f.companyId, name: "Reviewer", adapterType: "paperclip_runner" }).returning())[0] : null;
const createInteraction = (sourceRunId: string, supersedePendingSiblingInteractions = true) => interactions.create(issue, kind === "confirmation" ? {
kind: "request_confirmation", continuationPolicy: "wake_assignee", sourceRunId, addresseeAgentId: addressee?.id,
title: "Confirm", payload: { version: 1, prompt: "Continue?", supersedeOnUserComment: true },
} : {
kind: "ask_user_questions", continuationPolicy: "wake_assignee", sourceRunId, addresseeAgentId: addressee?.id,
title: "Choose", payload: { version: 1, title: "Choose", supersedeOnUserComment: true, questions: [{
id: "choice", prompt: "Continue?", selectionMode: "single", required: true,
options: [{ id: "yes", label: "Yes" }],
}] },
}, { agentId: f.agentId, runId: sourceRunId }, { supersedePendingSiblingInteractions });
const interaction = await createInteraction(source.id);
const replacement = action === "replacement" || action === "startup supersession" ? await run() : null;
if (action === "startup supersession") {
await createInteraction(replacement!.id, false);
await db.update(issueThreadInteractions).set({ createdAt: new Date(Date.now() - 1000) }).where(eq(issueThreadInteractions.id, interaction.id));
}
if (action === "historical answer") {
await interactions.dismissInteraction(issue, interaction.id, { userId: f.userId });
await db.update(issues).set({ status: "done" }).where(eq(issues.id, f.issueId));
}
await start(); await flush();
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${source.id}:`))).toBe(false);
await quiet();
if (action === "addressee deletion") {
expect((await interactions.cancelPendingForDeletedAddressee(f.companyId, addressee!.id)).map(row => row.id)).toContain(interaction.id);
} else if (action === "startup supersession") {
expect(await interactions.sweepSupersededPendingRequestConfirmations()).toMatchObject({ expired: 1 });
} else if (action === "replacement") {
await createInteraction(replacement!.id);
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interaction.id)))[0].status).toBe("expired");
} else if (action === "comment" || action === "historical comment") {
const [comment] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId,
authorType: "user", authorUserId: f.userId, origin: "comment", body: "Ignore the earlier request" }).returning();
const expired = action === "comment"
? await interactions.expireRequestConfirmationsSupersededByComment(issue, comment, { userId: f.userId })
: await interactions.expireRequestConfirmationsSupersededByHistoricalComments(issue);
expect(expired.map(row => row.id)).toContain(interaction.id);
} else if (action === "accept") {
await interactions.acceptInteraction({ ...issue, projectId: null, goalId: null }, interaction.id, {}, { userId: f.userId });
} else if (action === "reject") {
await interactions.rejectInteraction(issue, interaction.id, { reason: "No longer needed" }, { userId: f.userId });
} else if (action === "skip") {
await interactions.skipInteraction(issue, interaction.id, { reason: "No longer needed" }, { userId: f.userId });
} else if (action === "cancel") {
await interactions.cancelQuestions(issue, interaction.id, { reason: "No longer needed" }, { userId: f.userId });
} else if (action === "withdraw") {
await interactions.withdrawInteraction(issue, interaction.id, { reason: "No longer needed" }, { userId: f.userId });
} else if (action === "expire") {
await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, f.issueId));
await interactions.expirePendingInteractionsForTerminalIssue({ ...issue, status: "cancelled" }, { userId: f.userId });
} else {
await interactions.answerQuestions(issue, interaction.id,
{ answers: [{ questionId: "choice", optionIds: ["yes"] }] }, { userId: f.userId });
}
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${source.id}:failed:${f.endpointId}`);
if (action === "historical answer") {
expect(await db.select().from(issueQuestionResponseDeliveries).where(eq(issueQuestionResponseDeliveries.interactionId, interaction.id))).toEqual([]);
}
await quiet();
});
it.each(["change", "delete", "accept", "reject"] as const)("rechecks a document confirmation after %s detects a stale target", async mode => {
const source = await run(f, { status: "failed", errorCode: "test_run_failed" });
const interactions = issueThreadInteractionService(db), docs = documentService(db);
const issue = { id: f.issueId, companyId: f.companyId, status: "in_progress", projectId: null, goalId: null };
const input = { issueId: f.issueId, key: "plan", format: "markdown", body: "First plan", createdByUserId: f.userId };
const { document } = await docs.upsertIssueDocument(input);
const interaction = await interactions.create(issue, {
kind: "request_confirmation", continuationPolicy: "wake_assignee", sourceRunId: source.id,
payload: { version: 1, prompt: "Apply this plan?", target: { type: "issue_document", issueId: f.issueId,
documentId: document.id, key: document.key, revisionId: document.latestRevisionId!, revisionNumber: document.latestRevisionNumber } },
}, { agentId: f.agentId, runId: source.id });
await start(); await flush();
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${source.id}:`))).toBe(false);
await quiet();
const changed = mode === "delete"
? { ...(await docs.deleteIssueDocument(f.issueId, document.key))!, latestRevisionId: null, latestRevisionNumber: null }
: (await docs.upsertIssueDocument({ ...input, body: "Revised plan", baseRevisionId: document.latestRevisionId })).document;
await flush(); await quiet();
if (mode === "accept" || mode === "reject") {
await expect(mode === "accept"
? interactions.acceptInteraction(issue, interaction.id, {}, { userId: f.userId })
: interactions.rejectInteraction(issue, interaction.id, {}, { userId: f.userId }))
.rejects.toMatchObject({ status: 409, details: { code: "interaction_stale_target" } });
} else {
expect((await interactions.expireStaleRequestConfirmationsForIssueDocument(issue, changed, { userId: f.userId })).map(row => row.id)).toContain(interaction.id);
}
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interaction.id)))[0].status).toBe("expired");
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${source.id}:failed:${f.endpointId}`);
await quiet();
});
it("wakes a question continuation after its in-flight prompt receipt commits", async () => {
await db.update(chatEndpoints).set({ provider: "discord" }).where(eq(chatEndpoints.id, f.endpointId));
const source = await run(f, { status: "failed", contextSnapshot: { ...runValues().contextSnapshot, source: "chat:discord" } });
const interactions = issueThreadInteractionService(db);
const interaction = await interactions.create({ id: f.issueId, companyId: f.companyId }, {
kind: "ask_user_questions", continuationPolicy: "wake_assignee", sourceRunId: source.id,
title: "Choose", payload: { version: 1, title: "Choose", questions: [{
id: "choice", prompt: "Continue?", selectionMode: "single", required: true,
options: [{ id: "yes", label: "Yes" }],
}] },
}, { agentId: f.agentId, runId: source.id });
const prompt = (await publications()).find(row => row.idempotencyKey === `interaction:${interaction.id}:${f.endpointId}`)!;
const [endpoint] = await db.select().from(chatEndpoints).where(eq(chatEndpoints.id, f.endpointId));
const [conversation] = await db.select().from(chatConversations).where(eq(chatConversations.id, f.conversationId));
const targetId = randomUUID();
let beforeReceiptKeys: string[] | undefined;
await start(); await flush(); await quiet();
const deliver = questionResponseDeliveryService(db, { heartbeat: {
wakeup: async () => {
const [target] = await db.insert(heartbeatRuns).values(runValues(f, {
id: targetId, status: "queued", contextSnapshot: { issueId: f.issueId },
})).returning();
return target;
},
} });
const post = vi.fn(async () => {
await interactions.answerQuestions({ id: f.issueId, companyId: f.companyId, status: "in_progress" }, interaction.id,
{ answers: [{ questionId: "choice", optionIds: ["yes"] }] }, { userId: f.userId });
await deliver.deliver(interaction.id);
await flush();
beforeReceiptKeys = (await publications()).map(row => row.idempotencyKey);
return { id: "published-question" };
});
// Controlled transport/authorization ports; actual attempt lease, receipt
// transaction, question delivery receipt and milestone worker use PostgreSQL.
const worker = createChatPublicationWorker(db, {
CAPABILITIES, settleGitHubAutomaticPublication: async () => false,
ensureDurablePublicationTransport: async publication => publication,
tryProcessTeamsFilePublication: async () => false,
endpointRecord: async () => ({ endpoint, assignedAgentName: "Maya", connectionName: "Chat", applicationId: "test", credentialSecretRefs: [] }),
runtimeContextForRecord: () => ({ generation: 0, localEpoch: 0, credentialFingerprint: "test", version: "test" }),
hasCommittedTaskControlCompletion: async () => false, nonDirectDestinationAllowed: () => true,
runOwnershipMilestoneSupersessionReason: async () => null,
progressSupersededByFastResponse: async () => false, runProgressSupersededByPublishedInteraction: async () => false,
currentTaskControlPayload: async publication => publication.payload,
withCredentialMutationLease: async (_endpoint, operation) => operation({ assertOwned: async () => {} }),
closeProgressToReplace: async () => null, interactionResolutionPublicationToReplace: async () => null,
interactionPromptPublicationToReplace: async () => null, runPublicationToReplace: async () => null,
inboundWakePublicationToReplace: async () => null, taskStatusPublicationToReplace: async () => null,
claimPublicationTransportAuthorization: async () => ({ endpoint, conversation, authorizationActionId: null, preparedText: null }),
postSafePublication: post, commitTaskControlCompletion: async () => {}, receiptReactionCompletionRunId: async () => null,
finalizePublicationFailure: async (_publication, error) => { errors.push(error); return false; },
redactError: String,
} as Parameters<typeof createChatPublicationWorker>[1]);
await worker.processSelectedPublication(prompt);
expect(post).toHaveBeenCalledTimes(1);
expect((await db.select().from(chatPublications).where(eq(chatPublications.id, prompt.id)))[0]).toMatchObject({ state: "published", providerMessageId: "published-question" });
expect(beforeReceiptKeys).toBeDefined();
expect(beforeReceiptKeys).not.toContain(`run:${targetId}:queued:${f.endpointId}`);
await flush();
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${targetId}:queued:${f.endpointId}`);
await quiet();
});
it.each([
["retry", null], ["failed", null], ["delivery_unknown", "cancel"], ["delivery_unknown", "mark_delivered"],
] as const)("rechecks resolved prompt ownership after delivery %s (%s)", async (state, resolution) => {
const source = await run(f, { status: "failed", errorCode: "test_run_failed" });
const interactions = issueThreadInteractionService(db);
const interaction = await interactions.create({ id: f.issueId, companyId: f.companyId }, {
kind: "ask_user_questions", continuationPolicy: "wake_assignee", sourceRunId: source.id,
title: "Choose", payload: { version: 1, title: "Choose", questions: [{
id: "choice", prompt: "Continue?", selectionMode: "single", required: true,
options: [{ id: "yes", label: "Yes" }],
}] },
}, { agentId: f.agentId, runId: source.id });
// Model a send already claimed when the user answers through the UI.
const [prompt] = await db.update(chatPublications).set({ state: "streaming", attempts: 1 })
.where(eq(chatPublications.idempotencyKey, `interaction:${interaction.id}:${f.endpointId}`)).returning();
await interactions.answerQuestions({ id: f.issueId, companyId: f.companyId, status: "in_progress" }, interaction.id,
{ answers: [{ questionId: "choice", optionIds: ["yes"] }] }, { userId: f.userId });
expect(prompt).toBeDefined();
await start(); await flush();
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${source.id}:`))).toBe(false);
await quiet();
const failures = createChatPublicationFailures(db, {
FailedChatRetryPublicationReadError: class extends Error {},
MAX_ERROR_TEXT: 1000, getTaskBaseUrl: () => null,
invalidateRuntime: async () => false, runtimeCallbackEndpoint: async () => null,
});
const failure = Object.assign(new Error("provider rejected the send"), { status: state === "retry" ? 429 : 400 });
// The finalizer receives the selection from before the attempt increment.
await expect(failures.finalizePublicationFailure({ ...prompt, attempts: 0 }, failure, state === "delivery_unknown")).resolves.toBe(true);
expect((await db.select().from(chatPublications).where(eq(chatPublications.id, prompt.id)))[0].state).toBe(state);
await flush();
expect((await publications()).some(row => row.idempotencyKey === `run:${source.id}:failed:${f.endpointId}`)).toBe(state === "failed");
await quiet();
if (resolution) {
const targetId = randomUUID();
await questionResponseDeliveryService(db, { heartbeat: { wakeup: async () => {
const [target] = await db.insert(heartbeatRuns).values(runValues(f, {
id: targetId, status: "queued", contextSnapshot: { issueId: f.issueId },
})).returning();
return target;
} } }).deliver(interaction.id);
await flush();
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${targetId}:`))).toBe(false);
await quiet();
const [endpoint] = await db.select().from(chatEndpoints).where(eq(chatEndpoints.id, f.endpointId));
const resolver = createChatPublicationResolution(db, {
endpointRecord: async () => ({ endpoint, assignedAgentName: "Maya", connectionName: "Chat", applicationId: "test", credentialSecretRefs: [] }),
withCredentialMutationLease: async (_endpoint, operation) => operation({ assertOwned: async () => {} }),
commitTaskControlCompletion: async () => {}, processPendingPublications: async () => {},
} as Parameters<typeof createChatPublicationResolution>[1]);
await resolver.resolvePublication(f.endpointId, prompt.id, resolution, f.userId);
await flush();
const keys = (await publications()).map(row => row.idempotencyKey);
expect(keys.includes(`run:${source.id}:failed:${f.endpointId}`)).toBe(resolution === "cancel");
expect(keys.includes(`run:${targetId}:queued:${f.endpointId}`)).toBe(resolution === "mark_delivered");
await quiet();
}
});
it.each(["retry", "stale claim", "live claim", "rolled-back reset"] as const)("rechecks question prompt binding after %s", async (mode) => {
const source = await run();
const interactions = issueThreadInteractionService(db);
const interaction = await interactions.create({ id: f.issueId, companyId: f.companyId }, {
kind: "ask_user_questions", continuationPolicy: "wake_assignee", sourceRunId: source.id,
title: "Choose", payload: { version: 1, title: "Choose", questions: [{
id: "choice", prompt: "Continue?", selectionMode: "single", required: true,
options: [{ id: "yes", label: "Yes" }],
}] },
}, { agentId: f.agentId, runId: source.id });
await interactions.answerQuestions({ id: f.issueId, companyId: f.companyId, status: "in_progress" }, interaction.id,
{ answers: [{ questionId: "choice", optionIds: ["yes"] }] }, { userId: f.userId });
const other = await fixture(f.companyId);
await db.update(chatEndpoints).set({ assignedAgentId: f.agentId }).where(eq(chatEndpoints.id, other.endpointId));
await db.update(chatConversations).set({ issueId: f.issueId }).where(eq(chatConversations.id, other.conversationId));
await db.insert(chatPublications).values({ companyId: f.companyId, endpointId: other.endpointId, conversationId: other.conversationId,
issueId: f.issueId, idempotencyKey: `interaction:${interaction.id}:${other.endpointId}`,
payload: { text: "Continue?", interactionId: interaction.id }, state: "published", providerMessageId: "other-question",
});
const events: string[] = [];
let claimEvents: string[] = [];
let duringClaimNotices: string[] | undefined;
const unsubscribe = subscribeDatabaseWork(db, "chat-run-milestones", event => events.push(event));
const delivery = questionResponseDeliveryService(db, { heartbeat: {
cancelRun: async () => {
claimEvents = [...events];
await start(); await flush();
duringClaimNotices = (await publications()).map(row => row.idempotencyKey);
throw new Error("temporary cancellation failure");
},
wakeup: async () => { throw new Error("Must retain the still-active source"); },
} });
try {
if (mode === "retry") {
await expect(delivery.deliver(interaction.id)).resolves.toBeNull();
} else {
await db.update(issueQuestionResponseDeliveries).set({ status: "delivering", attemptCount: 1,
lastAttemptAt: new Date(Date.now() - (mode === "live claim" ? 0 : 60_000)),
}).where(eq(issueQuestionResponseDeliveries.interactionId, interaction.id));
await start(); await flush();
duringClaimNotices = (await publications()).map(row => row.idempotencyKey);
if (mode === "rolled-back reset") {
await expect(db.transaction(async tx => {
await questionResponseDeliveryService(tx as unknown as typeof db, { heartbeat: {} as never }).sweepPending(0);
expect(events).toEqual(["intent"]);
throw new Error("rollback reset");
})).rejects.toThrow("rollback reset");
} else {
await delivery.sweepPending(0);
}
}
expect(duringClaimNotices).toBeDefined();
expect(duringClaimNotices!.some(key => key.startsWith(`run:${source.id}:`))).toBe(false);
expect((await db.select().from(issueQuestionResponseDeliveries).where(eq(issueQuestionResponseDeliveries.interactionId, interaction.id)))[0].status).toBe(mode === "live claim" || mode === "rolled-back reset" ? "delivering" : "pending");
await flush();
const noticeExists = (await publications()).some(row => row.idempotencyKey === `run:${source.id}:working:${f.endpointId}`);
expect(noticeExists).toBe(mode === "retry" || mode === "stale claim");
if (mode === "retry") expect(claimEvents).toEqual(["intent", "settled"]);
else if (mode === "live claim") expect(events).toEqual([]);
else {
expect(events[0]).toBe("intent");
expect(events.slice(1)).toContain("settled");
expect(events.filter(event => event === "intent")).toHaveLength(1);
}
await quiet();
} finally { unsubscribe(); }
});
it("waits for outer commit and ignores rolled-back nested writes", async () => {
await start(); await flush();
await expect(db.transaction(async tx => {
await tx.transaction(async nested => { await notifyChatRunMilestones(nested, f); await nested.insert(heartbeatRuns).values(runValues()); });
throw new Error("rollback fixture");
})).rejects.toThrow("rollback fixture");
await flush(); expect(await publications()).toEqual([]); await quiet();
let release!: () => void, staged!: () => void;
const gate = new Promise<void>(resolve => { release = resolve; });
const stagedPromise = new Promise<void>(resolve => { staged = resolve; });
const writing = db.transaction(async tx => {
await tx.transaction(async nested => { await notifyChatRunMilestones(nested, f); await nested.insert(heartbeatRuns).values(runValues()); });
staged(); await gate;
});
try { await stagedPromise; await vi.advanceTimersByTimeAsync(1000); expect(await publications()).toEqual([]); }
finally { release(); await writing; }
await flush(); expect(await publications()).toHaveLength(1); await quiet();
});
it("retains the first committed wake during drain without periodic SQL", async () => {
await start(); await flush(); enabled = false;
await run(); await vi.advanceTimersByTimeAsync(1000);
await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0));
const drained = idleWorkSnapshot();
expect(await publications()).toEqual([]); await quiet(); expect(idleWorkSnapshot()).toEqual(drained);
enabled = true; await vi.advanceTimersByTimeAsync(30_000); await flush();
expect(await publications()).toHaveLength(1); await quiet();
});
it("scopes mutation wakes to their issue and company", async () => {
const sibling = await fixture(f.companyId), foreign = await fixture();
await start(); await flush();
// Unsupported raw writers deliberately do not wake: they provide a sentinel
// proving a supported issue mutation does not perform a global scan.
await db.insert(heartbeatRuns).values([runValues(sibling), runValues(foreign)]);
await run(); await flush();
expect(await publications()).toHaveLength(1); expect(await publications(sibling)).toEqual([]); expect(await publications(foreign)).toEqual([]);
await db.transaction(tx => notifyChatRunMilestones(tx, { companyId: f.companyId }));
await flush(); expect(await publications(sibling)).toHaveLength(1); expect(await publications(foreign)).toEqual([]);
await quiet();
});
it("disarms suppressed lineage and reactivates after its binding changes", async () => {
const row = await run(f, { contextSnapshot: { issueId: f.issueId, source: "chat:slack", endpointId: randomUUID(), wakeCommentId: f.commentId } });
await start(); await flush(); expect(await publications()).toEqual([]); await quiet();
await db.transaction(async tx => {
await notifyChatRunMilestones(tx, f);
await tx.update(heartbeatRuns).set({ contextSnapshot: runValues().contextSnapshot }).where(eq(heartbeatRuns.id, row.id));
});
await flush(); expect(await publications()).toHaveLength(1); await quiet();
});
it("waits quietly for a late presentation decision", async () => {
const row = await run(f, { status: "succeeded" }); await start(); await flush();
expect(await publications()).toEqual([]); await quiet();
await db.transaction(async tx => {
await notifyChatRunMilestones(tx, f);
await tx.update(heartbeatRuns).set({ resultJson: { presentationDecision: { source: "none" } } }).where(eq(heartbeatRuns.id, row.id));
});
await flush(); expect(await publications()).toEqual([expect.objectContaining({ idempotencyKey: `run:${row.id}:completed:${f.endpointId}` })]); await quiet();
});
it("drains more than one batch without duplicate notices or starving another company", async () => {
const other = await fixture();
await db.insert(heartbeatRuns).values([...Array.from({ length: 205 }, () => runValues()), runValues(other)]);
await start(); await flush(); expect(await publications()).toHaveLength(205); expect(await publications(other)).toHaveLength(1); await quiet();
});
it("retains a failed selection for retry through drain", async () => {
await run();
const select = vi.spyOn(db, "select").mockImplementationOnce(() => { throw new Error("temporary read failure"); });
await start(); select.mockRestore(); expect(errors.splice(0)).toHaveLength(1);
enabled = false; await vi.advanceTimersByTimeAsync(30_000);
await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0));
const drained = idleWorkSnapshot();
await quiet(); expect(idleWorkSnapshot()).toEqual(drained);
enabled = true; await vi.advanceTimersByTimeAsync(30_000); await flush();
expect(await publications()).toHaveLength(1); await quiet();
});
it("retries a failed publication write without idle scans before its deadline", async () => {
await run();
const transaction = vi.spyOn(db, "transaction").mockRejectedValueOnce(new Error("temporary publication failure"));
await start(); transaction.mockRestore(); expect(errors.splice(0)).toHaveLength(1);
expect(await publications()).toEqual([]); await quiet(10_000);
await vi.advanceTimersByTimeAsync(30_000); await flush(); expect(await publications()).toHaveLength(1); await quiet();
});
it("preserves a committed change arriving while a projection batch is running", async () => {
await run();
const transaction = db.transaction.bind(db);
let release!: () => void, entered!: () => void;
const gate = new Promise<void>(resolve => { release = resolve; });
const staged = new Promise<void>(resolve => { entered = resolve; });
const held = vi.spyOn(db, "transaction").mockImplementationOnce(async (callback, config) => {
entered(); await gate; return transaction(callback, config);
});
const starting = start();
try {
await staged;
await transaction(async tx => { await notifyChatRunMilestones(tx, f); await tx.insert(heartbeatRuns).values(runValues()); });
} finally { release(); await starting; held.mockRestore(); }
await flush(); expect(await publications()).toHaveLength(2); await quiet();
});
it("stops future wakes and reconstructs writes made after shutdown", async () => {
await start(); await flush(); await coordinator!.stop(); await run(); await quiet();
expect(await publications()).toEqual([]); await start(); await flush(); expect(await publications()).toHaveLength(1);
});
});
@@ -1,3 +1,4 @@
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { createHash } from "node:crypto";
import { afterAll, afterEach, beforeEach, beforeAll, describe, expect, it } from "vitest";
import { and, eq } from "drizzle-orm";
@@ -18,7 +19,8 @@ import {
} from "./helpers/embedded-postgres.js";
import { cloudActorHeaderSourceFromHeaders, resolveCloudTenantActor } from "../middleware/auth.js";
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_CLOUD_PROVISIONING_TEST_DATABASE_URL");
const embeddedPostgresSupport = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
if (!embeddedPostgresSupport.supported) {
@@ -59,8 +61,8 @@ describeEmbeddedPostgres("cloud tenant company provisioning", () => {
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
beforeAll(async () => {
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-tenant-provisioning-");
db = createDb(tempDb.connectionString);
if (!external) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-tenant-provisioning-");
db = createDb(external ?? tempDb!.connectionString);
}, 20_000);
beforeEach(() => {
@@ -80,6 +82,7 @@ describeEmbeddedPostgres("cloud tenant company provisioning", () => {
});
afterAll(async () => {
await db?.$client.end({ timeout: 1 });
await tempDb?.cleanup();
});
@@ -1,3 +1,4 @@
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { githubCredentialsFixture } from "./helpers/github-credentials-fixture.js";
import { createNativeGitHubAccess } from "../services/native-runtime/native-github-access.js";
import express from "express";
@@ -34,8 +35,6 @@ import {
import {
initializeRunIdentity,
captureRunIdentity,
reserveSteeredIdentity,
acceptSteeredIdentity,
} from "../services/run-identity.js";
import { resolveGitHubOperationCredentials } from "../services/github-operation-credentials.js";
import {
@@ -52,23 +51,25 @@ const vault = vi.hoisted(() => ({
resolveSecretValue: vi.fn(async () => "test-dedicated-token"),
}));
vi.mock("../services/secrets.js", () => ({ secretService: () => vault }));
const support = await getEmbeddedPostgresTestSupport();
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_GITHUB_CREDENTIAL_TEST_DATABASE_URL");
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
(support.supported ? describe : describe.skip)(
"operation-time GitHub credential resolution",
() => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>,
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined,
db: ReturnType<typeof createDb>;
beforeAll(async () => {
vi.stubEnv(
"PAPERCLIP_AGENT_JWT_SECRET",
"test-github-broker-signing-secret",
);
database = await startEmbeddedPostgresTestDatabase(
if (!external) database = await startEmbeddedPostgresTestDatabase(
"paperclip-github-operation-",
);
db = createDb(database.connectionString);
db = createDb(external ?? database!.connectionString);
}, 30_000);
afterAll(async () => {
await db?.$client.end({ timeout: 1 });
await database?.cleanup();
vi.unstubAllEnvs();
}, 60_000);
@@ -85,7 +86,7 @@ const support = await getEmbeddedPostgresTestSupport();
transport: "mcp_remote" as const, config: { sourceTemplateKey, payload }, transportConfig: { sourceTemplateKey } };
});
await db.insert(toolConnections).values(irrelevant);
const observed = createDb(database.connectionString), events: Array<Record<string, unknown>> = [], now = Date.now();
const observed = createDb(external ?? database!.connectionString), events: Array<Record<string, unknown>> = [], now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(observed, { toolConnections }, capture);
try {
@@ -117,7 +118,7 @@ const support = await getEmbeddedPostgresTestSupport();
resultJson: { stdout: payload }, stdoutExcerpt: payload, stderrExcerpt: payload,
}).where(eq(heartbeatRuns.id, input.runId));
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId));
const observed = createDb(database.connectionString), events: Array<Record<string, unknown>> = [], now = Date.now();
const observed = createDb(external ?? database!.connectionString), events: Array<Record<string, unknown>> = [], now = Date.now();
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(observed, { heartbeatRuns }, capture);
try {
@@ -37,6 +37,7 @@ import {
getEmbeddedPostgresTestSupport,
startEmbeddedPostgresTestDatabase,
} from "./helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { heartbeatService } from "../services/heartbeat.ts";
import { runningProcesses } from "../adapters/index.ts";
@@ -63,7 +64,8 @@ vi.mock("../adapters/index.ts", async () => {
};
});
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_DEPENDENCY_TEST_DATABASE_URL");
const embeddedPostgresSupport = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
if (!embeddedPostgresSupport.supported) {
@@ -103,8 +105,8 @@ describeEmbeddedPostgres("heartbeat dependency-aware queued run selection", () =
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
beforeAll(async () => {
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-heartbeat-dependency-scheduling-");
db = createDb(tempDb.connectionString);
if (!external) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-heartbeat-dependency-scheduling-");
db = createDb(external ?? tempDb!.connectionString);
await db.insert(authUsers).values({ id: "responsible-user", name: "Fixture owner", email: "owner@example.test", createdAt: new Date(), updatedAt: new Date() });
heartbeat = heartbeatService(db);
await ensureIssueRelationsTable(db);
@@ -185,6 +187,8 @@ describeEmbeddedPostgres("heartbeat dependency-aware queued run selection", () =
});
afterAll(async () => {
await db?.delete(authUsers).where(eq(authUsers.id, "responsible-user"));
await db?.$client.end();
await tempDb?.cleanup();
});
@@ -0,0 +1,72 @@
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import {
activityLog, agents, agentWakeupRequests, authUsers,
caseHandoffs, cases, companyMemberships, chatActions,
chatConversations, chatEndpoints, chatMessageLinks, chatPublications,
companies, completionContracts, documentRevisions, documents,
issueDocuments, type Db, nativeRunFinalizations, nativeRunResults,
statusDecisions, statusDecisionEffects, workAssessments, heartbeatRunEvents,
heartbeatRuns, issueComments, issueQuestionResponseDeliveries, issueThreadInteractions,
issues, toolApplications, toolConnections,
} from "@paperclipai/db";
// This owns only fixture data. Each test retains its worker, clock and quiet
// assertions, and must stop the worker before cleanup.
export function createChatMilestoneFixtures(db: Db) {
const fixtureCompanies: string[] = [];
const fixtureUsers: string[] = [];
async function create(companyId: string = randomUUID()) {
const agentId = randomUUID(), issueId = randomUUID(), endpointId = randomUUID(), conversationId = randomUUID(), commentId = randomUUID(), userId = randomUUID();
if (!fixtureCompanies.includes(companyId)) {
fixtureCompanies.push(companyId);
await db.insert(companies).values({ id: companyId, name: "Milestones", issuePrefix: `M${companyId.slice(0, 7)}` });
}
fixtureUsers.push(userId);
await db.insert(authUsers).values({ id: userId, name: "Owner", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() });
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, status: "active", membershipRole: "owner" });
await db.insert(agents).values({ id: agentId, companyId, name: "Maya", adapterType: "paperclip_runner" });
await db.insert(issues).values({ id: issueId, companyId, title: "Chat task", status: "in_progress", assigneeAgentId: agentId, responsibleUserId: userId });
const applicationId = randomUUID(), connectionId = randomUUID();
await db.insert(toolApplications).values({ id: applicationId, companyId, name: `Chat ${applicationId}`, type: "chat" });
await db.insert(toolConnections).values({ id: connectionId, companyId, applicationId, name: "Chat", uid: connectionId, transport: "chat_sdk", connectionPurpose: "channel", enabled: true, status: "active" });
await db.insert(chatEndpoints).values({ id: endpointId, companyId, connectionId, assignedAgentId: agentId, publicId: endpointId, provider: "slack", status: "active" });
await db.insert(chatConversations).values({ id: conversationId, companyId, endpointId, issueId, externalConversationId: conversationId, externalThreadId: conversationId, externalLabel: "Chat" });
await db.insert(issueComments).values({ id: commentId, companyId, issueId, authorType: "user", body: "Start work" });
await db.insert(chatMessageLinks).values({ companyId, endpointId, conversationId, commentId, providerMessageId: commentId, direction: "inbound" });
return { companyId, agentId, issueId, endpointId, conversationId, commentId, userId };
}
async function cleanup() {
for (const companyId of fixtureCompanies.splice(0)) {
await db.delete(activityLog).where(eq(activityLog.companyId, companyId));
await db.delete(caseHandoffs).where(eq(caseHandoffs.companyId, companyId));
await db.delete(cases).where(eq(cases.companyId, companyId));
await db.delete(chatActions).where(eq(chatActions.companyId, companyId));
await db.delete(chatMessageLinks).where(eq(chatMessageLinks.companyId, companyId));
await db.delete(chatPublications).where(eq(chatPublications.companyId, companyId));
await db.delete(chatConversations).where(eq(chatConversations.companyId, companyId));
await db.delete(chatEndpoints).where(eq(chatEndpoints.companyId, companyId));
await db.delete(issueQuestionResponseDeliveries).where(eq(issueQuestionResponseDeliveries.companyId, companyId));
await db.delete(issueThreadInteractions).where(eq(issueThreadInteractions.companyId, companyId));
await db.delete(issueComments).where(eq(issueComments.companyId, companyId));
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.companyId, companyId));
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, companyId));
await db.delete(statusDecisionEffects).where(eq(statusDecisionEffects.companyId, companyId));
await db.delete(statusDecisions).where(eq(statusDecisions.companyId, companyId));
await db.delete(workAssessments).where(eq(workAssessments.companyId, companyId));
await db.delete(nativeRunResults).where(eq(nativeRunResults.companyId, companyId));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId));
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId));
await db.delete(completionContracts).where(eq(completionContracts.companyId, companyId));
await db.delete(issueDocuments).where(eq(issueDocuments.companyId, companyId));
await db.delete(documentRevisions).where(eq(documentRevisions.companyId, companyId));
await db.delete(documents).where(eq(documents.companyId, companyId));
await db.delete(issues).where(eq(issues.companyId, companyId));
await db.delete(agents).where(eq(agents.companyId, companyId));
await db.delete(companyMemberships).where(eq(companyMemberships.companyId, companyId));
await db.delete(companies).where(eq(companies.id, companyId));
}
for (const userId of fixtureUsers.splice(0)) await db.delete(authUsers).where(eq(authUsers.id, userId));
}
return { create, cleanup };
}
@@ -1,6 +1,6 @@
import { randomUUID } from "node:crypto";
import { agents, companies, companyMemberships, companySecrets, connectionGrants, heartbeatRuns, issueComments, issues, toolApplications, toolConnectionInstalls, toolConnections, userSecretDefinitions, type Db } from "@paperclipai/db";
import { initializeRunIdentity, reserveSteeredIdentity, acceptSteeredIdentity } from "../../services/run-identity.js";
import { initializeRunIdentity, reserveSteeredIdentity, reconcileSteeredIdentity } from "../../services/run-identity.js";
export function githubCredentialsFixture(getDb: () => Db) {
async function seed() {
@@ -145,7 +145,7 @@ export function githubCredentialsFixture(getDb: () => Db) {
...input,
messageId: id,
});
await acceptSteeredIdentity(db, context!);
await reconcileSteeredIdentity(db, context!);
}
return { seed, grant, switchTo };
}
@@ -1,3 +1,4 @@
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { randomUUID } from "node:crypto";
import fs from "node:fs/promises";
import os from "node:os";
@@ -48,7 +49,8 @@ import { instanceSettingsService } from "../services/instance-settings.js";
import { withNativeWorkspaceFinalizationOwnership } from "../services/native-runtime/native-workspace-finalization-ownership.js";
import { noticeMetadataReferencesRecoveryAction } from "../services/recovery/successful-run-handoff.js";
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_RECOVERY_ACTION_TEST_DATABASE_URL");
const embeddedPostgresSupport = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
function makeRecoveryActionRow(overrides: Record<string, unknown> = {}) {
@@ -154,8 +156,8 @@ describeEmbeddedPostgres("issue recovery actions", () => {
beforeAll(async () => {
operationLogRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-recovery-operation-logs-"));
process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = operationLogRoot;
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-issue-recovery-actions-");
db = createDb(tempDb.connectionString);
if (!external) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-issue-recovery-actions-");
db = createDb(external ?? tempDb!.connectionString);
}, 30_000);
afterEach(async () => {
@@ -182,6 +184,7 @@ describeEmbeddedPostgres("issue recovery actions", () => {
});
afterAll(async () => {
await db?.$client.end({ timeout: 1 });
await tempDb?.cleanup();
if (priorOperationLogRoot === undefined) delete process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
else process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = priorOperationLogRoot;
@@ -616,8 +619,8 @@ describeEmbeddedPostgres("issue recovery actions", () => {
await db.update(agents).set({ status: "paused" }).where(eq(agents.id, coderId));
const enqueueWakeup = vi.fn(async () => null);
const recovery = recoveryService(db, { enqueueWakeup });
const operatorDb = createDb(tempDb!.connectionString, { maxConnections: 1 });
const probeDb = createDb(tempDb!.connectionString, { maxConnections: 1 });
const operatorDb = createDb(external ?? tempDb!.connectionString, { maxConnections: 1 });
const probeDb = createDb(external ?? tempDb!.connectionString, { maxConnections: 1 });
let release!: () => void;
let ready!: () => void;
let holderPid = 0;
@@ -766,7 +769,7 @@ describeEmbeddedPostgres("issue recovery actions", () => {
.where(eq(issues.id, sourceIssue.id));
}
const [observed] = await db.select().from(issues).where(eq(issues.id, sourceIssue.id));
const singleConnectionDb = createDb(tempDb!.connectionString, { maxConnections: 1 });
const singleConnectionDb = createDb(external ?? tempDb!.connectionString, { maxConnections: 1 });
let timer: ReturnType<typeof setTimeout> | undefined;
try {
const work = recoveryService(singleConnectionDb, { enqueueWakeup: async () => null })
@@ -1,3 +1,4 @@
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import express from "express";
import request from "supertest";
import { eq } from "drizzle-orm";
@@ -20,18 +21,19 @@ vi.mock("../services/github-fetch.js", async importOriginal => ({
...await importOriginal<typeof import("../services/github-fetch.js")>(),
ghFetch: vi.fn(async () => new Response(null, { status: 404 })),
}));
const support = await getEmbeddedPostgresTestSupport();
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_PROJECT_GIT_TEST_DATABASE_URL");
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
(support.supported ? describe : describe.skip)("explicit project Git authorization", () => {
let db: ReturnType<typeof createDb>, database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
const { seed, grant, switchTo } = githubCredentialsFixture(() => db);
beforeAll(async () => {
database = await startEmbeddedPostgresTestDatabase("paperclip-project-git-authorization-");
db = createDb(database.connectionString);
if (!external) database = await startEmbeddedPostgresTestDatabase("paperclip-project-git-authorization-");
db = createDb(external ?? database!.connectionString);
await db.insert(authUsers).values(["A", "B"].map(id => ({
id, name: id, email: `${id}@example.invalid`, createdAt: new Date(), updatedAt: new Date(),
})));
}, 30_000);
afterAll(async () => { await database?.cleanup(); }, 60_000);
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await database?.cleanup(); }, 60_000);
async function projectAuthorizationFixture() {
const input = await seed();
const selected = await grant(input, "A");
@@ -26,6 +26,7 @@ import {
getEmbeddedPostgresTestSupport,
startEmbeddedPostgresTestDatabase,
} from "./helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { issueThreadInteractionService } from "../services/issue-thread-interactions.js";
import {
buildQuestionResponseDeliveryEnvelope,
@@ -33,7 +34,8 @@ import {
questionResponseDeliveryService,
} from "../services/question-response-delivery.js";
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_QUESTION_DELIVERY_TEST_DATABASE_URL");
const embeddedPostgresSupport = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported
? describe
: describe.skip;
@@ -64,10 +66,10 @@ describeEmbeddedPostgres("question response delivery", () => {
> | null = null;
beforeAll(async () => {
tempDb = await startEmbeddedPostgresTestDatabase(
"paperclip-question-delivery-",
);
db = createDb(tempDb.connectionString);
if (!externalDatabaseUrl) {
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-question-delivery-");
}
db = createDb(externalDatabaseUrl ?? tempDb!.connectionString);
}, 20_000);
afterEach(async () => {
@@ -84,6 +86,7 @@ describeEmbeddedPostgres("question response delivery", () => {
});
afterAll(async () => {
await db?.$client.end();
await tempDb?.cleanup();
});
@@ -1,7 +1,9 @@
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { randomUUID } from "node:crypto";
import { eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
subscribeDatabaseWork,
costEvents,
activityLog,
agents,
@@ -25,9 +27,11 @@ const mockTelemetryClient = vi.hoisted(() => ({ track: vi.fn() }));
vi.mock("../telemetry.ts", () => ({ getTelemetryClient: () => mockTelemetryClient }));
import { heartbeatService } from "../services/heartbeat.ts";
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
import { recoveryService } from "../services/recovery/service.ts";
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_STALE_LOCK_TEST_DATABASE_URL");
const embeddedPostgresSupport = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
if (!embeddedPostgresSupport.supported) {
@@ -43,8 +47,8 @@ describeEmbeddedPostgres("recovery sweepStaleIssueLocks", () => {
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
beforeAll(async () => {
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-stale-lock-sweep-");
db = createDb(tempDb.connectionString);
if (!external) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-stale-lock-sweep-");
db = createDb(external ?? tempDb!.connectionString);
}, 20_000);
afterEach(async () => {
@@ -64,6 +68,7 @@ describeEmbeddedPostgres("recovery sweepStaleIssueLocks", () => {
});
afterAll(async () => {
await db?.$client.end({ timeout: 1 });
await tempDb?.cleanup();
});
@@ -585,8 +590,15 @@ describeEmbeddedPostgres("recovery sweepStaleIssueLocks", () => {
it.each(["observed", "workspace_finalizing", "ready_for_assessment", "arbitrating", "retryable_failure"])(
"preserves a completed native result in %s after its provider exits", async phase => {
const { runningRunId, issueId } = await seedPendingNativeResult(phase);
const result = await recoveryService(db, { enqueueWakeup: vi.fn() }).sweepStaleIssueLocks();
expect(result).toEqual({ cleared: 0, issueIds: [], terminalizedRunIds: [] });
const events: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, event => events.push(event));
try {
const recovery = recoveryService(db, { enqueueWakeup: vi.fn() });
for (let pass = 0; pass < 3; pass++) {
expect(await recovery.sweepStaleIssueLocks()).toEqual({ cleared: 0, issueIds: [], terminalizedRunIds: [] });
}
expect(events).toEqual([]);
} finally { unsubscribe(); }
expect(await db.select({ status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode }).from(heartbeatRuns)
.where(eq(heartbeatRuns.id, runningRunId))).toEqual([{ status: "running", errorCode: null }]);
expect(await db.select({ executionRunId: issues.executionRunId }).from(issues)
+4 -4
View File
@@ -3,7 +3,7 @@ import { eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { agentWakeupRequests, agents, companies, applyPendingMigrations, createDb, closeRegisteredClients, heartbeatRuns, heartbeatRunEvents, issueComments, issueThreadInteractions, issues, secretAccessEvents } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { acceptSteeredIdentity, captureRunIdentity, initializeRunIdentity, listRunIdentityContexts, rejectSteeredIdentity, reserveSteeredIdentity } from "../services/run-identity.js";
import { acceptSteeredIdentity, reconcileSteeredIdentity, captureRunIdentity, initializeRunIdentity, listRunIdentityContexts, rejectSteeredIdentity, reserveSteeredIdentity } from "../services/run-identity.js";
const externalDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL;
const support = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
@@ -143,8 +143,8 @@ const support = externalDatabaseUrl ? { supported: true } : await getEmbeddedPos
expect((await captureRunIdentity(db, input, options)).context?.responsibleUserId).toBe("B");
expect(startedUnderA.context?.responsibleUserId).toBe("A");
const a = await reserveSteeredIdentity(db, { ...input, messageId: input.messageIds[2] });
await acceptSteeredIdentity(db, a!);
await acceptSteeredIdentity(db, b!);
await reconcileSteeredIdentity(db, a!);
await reconcileSteeredIdentity(db, b!);
expect((await captureRunIdentity(db, input, options)).context?.responsibleUserId).toBe("A");
});
it("keeps rejected steering unchanged and preserves the originating identity on retry", async () => {
@@ -203,7 +203,7 @@ const support = externalDatabaseUrl ? { supported: true } : await getEmbeddedPos
sourceIdentityContextId: origin.id, payload: { prompt: "Continue?" } as never,
});
const b = await reserveSteeredIdentity(db, { ...input, messageId: input.messageIds[1] });
await acceptSteeredIdentity(db, b!);
await reconcileSteeredIdentity(db, b!);
for (const approval of [true, false]) {
const runId = randomUUID();
await db.insert(heartbeatRuns).values({ id: runId, companyId: input.companyId, agentId: input.agentId, status: "running" });
@@ -270,7 +270,7 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu
await db.update(chatVoiceInboundCalls).set({ expiresAt: new Date(0), updatedAt: new Date(0) }).where(eq(chatVoiceInboundCalls.id, f.call.id));
const onError = vi.fn();
const maintenance = createChatReconciliationCoordinator({
processPendingDeliveries: async () => undefined, projectRunMilestones: async () => 0,
processPendingDeliveries: async () => undefined, projectNativeProgress: async () => 0,
flushPublications: async () => undefined, onError,
});
try {
+18 -16
View File
@@ -195,7 +195,8 @@ import { createToolGatewayService } from "./services/tool-gateway.js";
import { toolAccessService } from "./services/tool-access.js";
import { chatChannelService } from "./services/chat-channels.js";
import { deliverNativeQuestionResponse } from "./services/native-runtime/native-question-bridge.js";
import { enqueueChatRunMilestones } from "./services/chat-run-publications.js";
import { enqueueSafeNativeChatProgress } from "./services/chat-run-publications.js";
import { chatRunMilestoneWork } from "./services/chat-run-milestone-work.js";
import {
createCoalescedAsyncTrigger,
isChatPublicationCommitSignal,
@@ -323,7 +324,7 @@ export function shouldEnablePrivateHostnameGuard(opts: {
type ChatReconciliationLane =
| "deliveries"
| "GitHub webhook recovery"
| "run milestones"
| "native progress"
| "publications"
| "Slack file receipts"
| "Slack session status";
@@ -336,7 +337,7 @@ type ChatReconciliationLane =
export function createChatReconciliationCoordinator(input: {
processPendingDeliveries: () => Promise<unknown>;
processFailedGitHubWebhookDeliveries?: () => Promise<unknown>;
projectRunMilestones: () => Promise<number>;
projectNativeProgress: () => Promise<number>;
flushPublications: () => Promise<unknown>;
processPendingSlackFileUploadReceipts?: () => Promise<unknown>;
processPendingSlackSessionSyncs?: () => Promise<unknown>;
@@ -348,15 +349,15 @@ export function createChatReconciliationCoordinator(input: {
run: input.flushPublications,
onError: (error) => input.onError("publications", error),
});
const milestoneReconciliation = createCoalescedAsyncTrigger({
const progressReconciliation = createCoalescedAsyncTrigger({
run: async () => {
const inserted = await input.projectRunMilestones();
const inserted = await input.projectNativeProgress();
// Existing final/question publications never wait on this optional
// projection. Newly committed milestones get a bounded dispatch wake;
// projection. Newly committed progress gets a bounded dispatch wake;
// an empty/contended pass does not create a self-sustaining loop.
if (inserted > 0) publicationReconciliation.notify();
},
onError: (error) => input.onError("run milestones", error),
onError: (error) => input.onError("native progress", error),
});
const start = (
lane: ChatReconciliationLane,
@@ -382,24 +383,24 @@ export function createChatReconciliationCoordinator(input: {
input.processFailedGitHubWebhookDeliveries,
);
}
milestoneReconciliation.poll();
progressReconciliation.poll();
publicationReconciliation.poll();
if (input.processPendingSlackFileUploadReceipts) start("Slack file receipts", input.processPendingSlackFileUploadReceipts);
if (input.processPendingSlackSessionSyncs) start("Slack session status", input.processPendingSlackSessionSyncs);
},
notifyPublications() {
milestoneReconciliation.notify();
progressReconciliation.notify();
publicationReconciliation.notify();
},
stop() {
stopped = true;
milestoneReconciliation.stop();
progressReconciliation.stop();
publicationReconciliation.stop();
},
async drain() {
await Promise.allSettled([
...inFlight.values(),
milestoneReconciliation.drain(),
progressReconciliation.drain(),
]);
// Projecting the final batch can notify dispatch after an earlier drain
// would have returned. Join dispatch only after its producer has drained.
@@ -1304,6 +1305,9 @@ export async function createApp(
chatChannels.registerTeamsConsentExpiryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerTeamsAttemptRecoveryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatChannels.registerCommandWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
chatRunMilestoneWork(db, opts.authPublicBaseUrl).register(
deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive(),
);
emailChannels.start();
const reconcileChatPublicationMaintenance = async () => {
await chatChannels.processPublicationMaintenance();
@@ -1314,10 +1318,7 @@ export async function createApp(
processPendingDeliveries: async () => undefined,
processFailedGitHubWebhookDeliveries: () =>
chatChannels.processFailedGitHubWebhookDeliveries(),
projectRunMilestones: () =>
enqueueChatRunMilestones(db, {
publicBaseUrl: opts.authPublicBaseUrl,
}),
projectNativeProgress: () => enqueueSafeNativeChatProgress(db),
flushPublications: () => reconcileChatPublicationMaintenance(),
onError: (lane, err) => {
logger.error({ err, lane }, `Failed to reconcile chat ${lane}`);
@@ -1329,7 +1330,8 @@ export async function createApp(
chatReconciliation.notifyPublications();
},
);
// Provider/action maintenance and milestone projection still use this cadence.
// Native progress and remaining publication maintenance still use this cadence.
// Lifecycle milestones use committed changes and their own recovery worker.
// Inbox, outbox, and file receipt queues have independent commit/deadline wakes.
let chatPublicationTimer: ReturnType<typeof setInterval> | null = setInterval(
() => {
+3
View File
@@ -4,6 +4,7 @@ import type { Db } from "@paperclipai/db";
import { companies, companyMemberships, instanceUserRoles } from "@paperclipai/db";
import type { DeploymentMode } from "@paperclipai/shared";
import { ensureHumanRoleDefaultGrants } from "./services/principal-access-compatibility.js";
import { notifyChatRunMilestones } from "./services/chat-run-milestone-notifications.js";
const LOCAL_BOARD_USER_ID = "local-board";
const CLAIM_TTL_MS = 1000 * 60 * 60 * 24;
@@ -124,6 +125,7 @@ export async function claimBoardOwnership(
.then((rows) => rows[0] ?? null);
if (!existing) {
await notifyChatRunMilestones(tx, { companyId: company.id });
await tx.insert(companyMemberships).values({
companyId: company.id,
principalType: "user",
@@ -135,6 +137,7 @@ export async function claimBoardOwnership(
}
if (existing.status !== "active") {
await notifyChatRunMilestones(tx, { companyId: company.id });
await tx
.update(companyMemberships)
.set({ status: "active", membershipRole: "owner", updatedAt: new Date() })
+29 -13
View File
@@ -30,6 +30,7 @@ import { captureRunIdentity } from "../services/run-identity.js";
import { boardAuthService } from "../services/board-auth.js";
import { retryIdempotentDatabaseOperation } from "../database-retry.js";
import { beginIdleTrackedWork } from "../services/task-admission.js";
import { notifyChatRunMilestones } from "../services/chat-run-milestone-notifications.js";
export {
isTransientDbConnectionError,
@@ -652,19 +653,34 @@ async function resolveCloudTenantActorOnce(
// exact legacy-marker guards and the operator's own edits.
await repairCloudTenantCompanyProvisionDefaults(db, { companyId, stackId, now });
effectiveMembership = await db
.insert(companyMemberships)
.values({ companyId, principalType: "user", principalId: userId, status: "active", membershipRole, updatedAt: now })
.onConflictDoUpdate({
target: [companyMemberships.companyId, companyMemberships.principalType, companyMemberships.principalId],
set: { status: "active", membershipRole, updatedAt: now },
setWhere: sql`${companyMemberships.status} is distinct from 'active'
or ${companyMemberships.membershipRole} is distinct from ${membershipRole}`,
})
.returning({ companyId: companyMemberships.companyId, membershipRole: companyMemberships.membershipRole, status: companyMemberships.status })
// PostgreSQL returns no row for an unchanged ON CONFLICT ... WHERE.
// That condition itself proves the existing role/status match the input.
.then((rows) => rows[0] ?? { companyId, membershipRole, status: "active" });
effectiveMembership = await db.transaction(async tx => {
const [existing] = await tx.select({
status: companyMemberships.status,
membershipRole: companyMemberships.membershipRole,
}).from(companyMemberships).where(and(
eq(companyMemberships.companyId, companyId),
eq(companyMemberships.principalType, "user"),
eq(companyMemberships.principalId, userId),
)).for("update");
// Serialize authority restoration and its notice while retaining the
// guarded upsert that leaves unchanged memberships untouched.
if (!existing || existing.status !== "active" || existing.membershipRole === "viewer") {
await notifyChatRunMilestones(tx, { companyId });
}
return tx
.insert(companyMemberships)
.values({ companyId, principalType: "user", principalId: userId, status: "active", membershipRole, updatedAt: now })
.onConflictDoUpdate({
target: [companyMemberships.companyId, companyMemberships.principalType, companyMemberships.principalId],
set: { status: "active", membershipRole, updatedAt: now },
setWhere: sql`${companyMemberships.status} is distinct from 'active'
or ${companyMemberships.membershipRole} is distinct from ${membershipRole}`,
})
.returning({ companyId: companyMemberships.companyId, membershipRole: companyMemberships.membershipRole, status: companyMemberships.status })
// PostgreSQL returns no row for an unchanged ON CONFLICT ... WHERE.
// That condition itself proves the existing role/status match the input.
.then((rows) => rows[0] ?? { companyId, membershipRole, status: "active" });
});
// An unchanged membership can still lack default grants after a partial
// sync or an older deployment; keep this repair inside every actual sync.
@@ -46,7 +46,7 @@ function createFakeDb(options: {
: options.settingsRow;
const insertedTables: unknown[] = [];
const deletedTables: unknown[] = [];
const selectWheres: Array<{ table: unknown; condition: unknown }> = [];
const selectWheres: Array<{ table: unknown; condition: unknown; locked?: boolean }> = [];
function writeChain(table: unknown) {
let values: unknown;
const chain: Record<string, unknown> = {};
@@ -63,6 +63,7 @@ function createFakeDb(options: {
return chain;
}
const db = {
transaction: async (work: (tx: Db) => Promise<unknown>) => work(db),
insert: (table: unknown) => {
insertedTables.push(table);
return writeChain(table);
@@ -75,21 +76,32 @@ function createFakeDb(options: {
return {
from: (table: unknown) => ({
where: (condition: unknown) => {
// Role-default grant seeding reads existing grants first; keep that
// read working so selectThrows exercises only the degradable reads.
if (options.selectThrows && table !== principalPermissionGrants) {
// Only settings lookup is degradable. Membership locking and
// permission seeding must still work in the fail-closed flag test.
if (options.selectThrows && table === instanceSettings) {
throw new Error("select unavailable");
}
selectWheres.push({ table, condition });
const selected: (typeof selectWheres)[number] = { table, condition };
selectWheres.push(selected);
const rows =
table === instanceSettings && settingsRow
? [settingsRow]
: table === companyMemberships
? (options.membershipQueryRows ?? [])
: [];
return {
then: (resolve: (v: unknown) => unknown) => Promise.resolve(rows).then(resolve),
let locked = false;
const result = {
for: () => { locked = true; selected.locked = true; return result; },
then: (resolve: (v: unknown) => unknown, reject?: (error: unknown) => unknown) => {
// The optional cross-company read can fail independently of
// the primary membership transaction, which must succeed.
const query = options.selectThrows && table === companyMemberships && !locked
? Promise.reject(new Error("select unavailable"))
: Promise.resolve(rows);
return query.then(resolve, reject);
},
};
return result;
},
}),
};
@@ -258,7 +270,8 @@ describe("resolveCloudTenantActor (shared-pool hardening)", () => {
expect(insertedTables.filter(table => table === companyMemberships)).toHaveLength(1);
expect(deletedTables.filter(table => table === instanceUserRoles)).toHaveLength(20);
expect(selectWheres.filter(entry => entry.table === instanceSettings)).toHaveLength(20);
expect(selectWheres.filter(entry => entry.table === companyMemberships)).toHaveLength(20);
expect(selectWheres.filter(entry => entry.table === companyMemberships && !entry.locked)).toHaveLength(20);
expect(selectWheres.filter(entry => entry.table === companyMemberships && entry.locked)).toHaveLength(1);
expect(actors.every(actor => actor?.memberships?.[0]?.membershipRole === "owner")).toBe(true);
});
@@ -461,11 +474,13 @@ describe("resolveCloudTenantActor (shared-pool hardening)", () => {
const { db, selectWheres } = createFakeDb();
await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
const membershipSelects = selectWheres.filter((entry) => entry.table === companyMemberships);
expect(membershipSelects).toHaveLength(1);
expect(membershipSelects).toHaveLength(2);
// The first read locks the primary membership before its upsert; the
// second loads the user's active memberships for request authorization.
// Rows for other users and rows in any non-active status are excluded
// by the query itself: the filter binds exactly this user id and the
// "active" status — the same condition the session path applies.
expect(membershipSelects[0]!.condition).toEqual(
expect(membershipSelects[1]!.condition).toEqual(
and(
eq(companyMemberships.principalType, "user"),
eq(companyMemberships.principalId, "user-123"),
@@ -26,6 +26,7 @@ import { budgetService, budgetServiceInTransaction, recheckAdmittedRunBudgetRead
import { lockRunBudgetRecoveryOwnerInTransaction, recheckRunBudgetReservationInTransaction } from "../../../services/budget-reservations.js";
import { isCompletedOnboardingHandoffWake } from "../../../services/chat-completion-delivery.js";
import { isHeartbeatWakeOnDemandEnabled } from "../../../services/heartbeat-policy.js";
import { notifyChatRunMilestones } from "../../../services/chat-run-milestone-notifications.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../../../services/delivery-work-notifications.js";
import { connectionContinuationPendingResponse } from "../../../services/satisfied-connection-intents.js";
import { collectDispositionRepairSourceState } from "../../../services/recovery/disposition-repair.js";
@@ -660,10 +661,12 @@ export function createPostgresRunDispatchAdapter(
async function promoteDueRetryInTx(
tx: Db,
input: PromoteOrCancelDueRetryInput,
issueId: string | null,
): Promise<
| { applied: true; run: HeartbeatRun; postCommitEffects: PostCommitEffect[] }
| { applied: false }
> {
await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: issueId ?? undefined });
const [row] = await tx
.update(heartbeatRuns)
.set({
@@ -705,6 +708,7 @@ export function createPostgresRunDispatchAdapter(
tx: Db,
input: CancelSuppressedRetryInput,
): Promise<CancelSuppressedRetryResult> {
await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: input.issueId ?? undefined });
const [row] = await tx
.update(heartbeatRuns)
.set({
@@ -873,7 +877,7 @@ export function createPostgresRunDispatchAdapter(
runId: input.runId,
companyId: input.companyId,
now,
});
}, issueId);
return promoted.applied
? {
outcome: {
@@ -911,6 +915,7 @@ export function createPostgresRunDispatchAdapter(
providerDispatchNotStarted = false,
): Promise<CancelStaleQueuedRunOutcome> {
await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, run.companyId);
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
const [row] = await tx
.update(heartbeatRuns)
.set({
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../../../services/chat-run-milestone-notifications.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../../../services/delivery-work-notifications.js";
import { isAcknowledgedNativeReassignmentStop, isAcknowledgedNativeStop } from "../../../services/acknowledged-native-stop.js";
import { readRunCancellation } from "../../../services/run-cancellation.js";
@@ -472,6 +473,7 @@ function buildTransaction(tx: Db, deps: WakeQueuePostgresAdapterDeps, db: Db, ru
},
async finalizePromotedWake(input) {
await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: input.issue.id });
const newRun = await tx
.insert(heartbeatRuns)
.values({
@@ -688,6 +690,7 @@ function buildTransaction(tx: Db, deps: WakeQueuePostgresAdapterDeps, db: Db, ru
sessionBefore,
now,
}) {
await notifyChatRunMilestones(tx, { companyId, issueId: issue.id });
const wakeupRequest = await tx
.insert(agentWakeupRequests)
.values({
@@ -949,6 +952,7 @@ export function createWakeAdmissionWriter(): WakeAdmissionWriter {
async coalesceIntoActiveExecutionRun(scope, input) {
const tx = requireAdmissionTx(scope, input.companyId);
const now = new Date();
await notifyChatRunMilestones(tx, { companyId: input.companyId });
const mergedRun = await tx
.update(heartbeatRuns)
.set({ contextSnapshot: input.mergedContextSnapshot, updatedAt: now })
@@ -7,6 +7,7 @@ import {
getEmbeddedPostgresTestSupport,
startEmbeddedPostgresTestDatabase,
} from "../../../__tests__/helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "../../../__tests__/helpers/external-test-database.js";
import { createQueuedCommentIssueLockWriter } from "./queued-comment-postgres.js";
import type { QueuedCommentQueuePostgresAdapterDeps } from "./queued-comment-postgres.js";
import { QueuedCommentMutationError } from "../application/queued-comment-use-cases.js";
@@ -19,7 +20,8 @@ import { QueuedCommentMutationError } from "../application/queued-comment-use-ca
// leaving a partial write. The decision branching itself is proven against
// plain facts in `domain/policy.test.ts`; the use-case orchestration is
// proven against a mocked port in `application/queued-comment-use-cases.test.ts`.
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_QUEUED_COMMENT_TEST_DATABASE_URL");
const embeddedPostgresSupport = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
if (!embeddedPostgresSupport.supported) {
@@ -39,8 +41,8 @@ describeEmbeddedPostgres("queued-comment postgres adapter", () => {
};
beforeAll(async () => {
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-queued-comment-postgres-adapter-");
db = createDb(tempDb.connectionString);
if (!external) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-queued-comment-postgres-adapter-");
db = createDb(external ?? tempDb!.connectionString);
}, 20_000);
afterEach(async () => {
@@ -56,6 +58,7 @@ describeEmbeddedPostgres("queued-comment postgres adapter", () => {
});
afterAll(async () => {
await db?.$client.end();
await tempDb?.cleanup();
});
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../../../services/chat-run-milestone-notifications.js";
import { and, eq, inArray } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import { agentWakeupRequests, agents, heartbeatRuns, issueComments, issues } from "@paperclipai/db";
@@ -45,7 +46,7 @@ export type QueuedCommentQueuePostgresAdapterDeps = {
syncCommentExternalObjectsSafely(commentId: string, tx: Db): Promise<void>;
};
function buildTransaction(tx: Db, companyId: string, deps: QueuedCommentQueuePostgresAdapterDeps): QueuedCommentQueueTransaction {
function buildTransaction(tx: Db, companyId: string, issueId: string, deps: QueuedCommentQueuePostgresAdapterDeps): QueuedCommentQueueTransaction {
return {
async updateCommentBody({ issueId, commentId, body, updatedAt }) {
const updated = await tx
@@ -72,6 +73,7 @@ function buildTransaction(tx: Db, companyId: string, deps: QueuedCommentQueuePos
},
async updateQueueRunCommentIds({ queueRunId, contextSnapshot, ids, updatedAt }) {
await notifyChatRunMilestones(tx, { companyId, issueId });
const row = await tx
.update(heartbeatRuns)
.set({ contextSnapshot: withQueuedCommentIdsInRunContext(contextSnapshot, ids), updatedAt })
@@ -82,6 +84,7 @@ function buildTransaction(tx: Db, companyId: string, deps: QueuedCommentQueuePos
},
async deleteComment({ issueId, commentId }) {
await notifyChatRunMilestones(tx, { companyId, issueId });
const row = await tx
.delete(issueComments)
.where(and(eq(issueComments.id, commentId), eq(issueComments.issueId, issueId), eq(issueComments.companyId, companyId)))
@@ -98,6 +101,7 @@ function buildTransaction(tx: Db, companyId: string, deps: QueuedCommentQueuePos
},
async cancelQueueRun({ queueRunId, reason, now }) {
await notifyChatRunMilestones(tx, { companyId, issueId });
const row = await tx
.update(heartbeatRuns)
.set({ status: "cancelled", finishedAt: now, error: reason, errorCode: "queued_comment_discarded", updatedAt: now })
@@ -291,7 +295,7 @@ export function createQueuedCommentIssueLockWriter(db: Db, deps: QueuedCommentQu
.then((rows) => rows[0] ?? null)
: null;
const transaction = buildTransaction(tx, companyId, deps);
const transaction = buildTransaction(tx, companyId, input.issue.id, deps);
const wake = toWakeRow(wakeRow);
const queueRun = queueRunRow ? toRunRow(queueRunRow) : null;
const activeRun = activeRunRow ? toRunRow(activeRunRow) : null;
+49 -24
View File
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { notifyVoiceChange } from "./voice/voice-recovery-notifications.js";
import { notifyChatEndpointWork } from "./chat-work-notifications.js";
import { notifyToolConnectionHealth } from "./tool-connection-health-notifications.js";
@@ -328,7 +329,7 @@ export function accessService(db: Db) {
&& grant.credentialSecretRefs.length > 0
&& grant.credentialSecretRefs.every((ref) => !connectionSecretIdsToRemove.has(ref.secretId)),
);
if (connection.transport === "chat_sdk") await notifyChatEndpointWork(tx, connection.id);
if (connection.transport === "chat_sdk") await notifyChatEndpointWork(tx, connection.id, companyId);
if (connection.transport === "voice") await notifyVoiceChange(tx, { kind: "connection", id: connection.id });
if (!hasUnaffectedActiveGrant) await notifyToolConnectionHealth(tx, { companyId, id: connection.id });
await tx.update(toolConnections).set({
@@ -554,6 +555,11 @@ export function accessService(db: Db) {
data.membershipRole !== undefined ? data.membershipRole : existing.membershipRole;
const nextStatus = data.status ?? existing.status;
if (existing.principalType === "user" && nextStatus === "active" && nextMembershipRole !== "viewer"
&& (existing.status !== "active" || existing.membershipRole === "viewer")) {
await notifyChatRunMilestones(tx, { companyId });
}
if (
existing.principalType === "user" &&
existing.status === "active" &&
@@ -886,6 +892,7 @@ export function accessService(db: Db) {
const existingMembership = existingByCompany.get(companyId);
if (existingMembership) {
if (existingMembership.status !== "active") {
if (existingMembership.membershipRole !== "viewer") await notifyChatRunMilestones(tx, { companyId });
await tx
.update(companyMemberships)
.set({
@@ -897,6 +904,7 @@ export function accessService(db: Db) {
}
continue;
}
await notifyChatRunMilestones(tx, { companyId });
await tx.insert(companyMemberships).values({
companyId,
principalType: "user",
@@ -917,31 +925,43 @@ export function accessService(db: Db) {
membershipRole: string | null = "member",
status: "pending" | "active" | "suspended" = "active",
) {
const existing = await getMembership(companyId, principalType, principalId);
if (existing) {
if (existing.status !== status || existing.membershipRole !== membershipRole) {
const updated = await db
.update(companyMemberships)
.set({ status, membershipRole, updatedAt: new Date() })
.where(eq(companyMemberships.id, existing.id))
.returning()
.then((rows) => rows[0] ?? null);
return updated ?? existing;
return db.transaction(async tx => {
const existing = await tx.select().from(companyMemberships).where(and(
eq(companyMemberships.companyId, companyId), eq(companyMemberships.principalType, principalType),
eq(companyMemberships.principalId, principalId),
)).for("update").then(rows => rows[0] ?? null);
if (existing) {
if (existing.status !== status || existing.membershipRole !== membershipRole) {
if (principalType === "user" && status === "active" && membershipRole !== "viewer"
&& (existing.status !== "active" || existing.membershipRole === "viewer")) {
await notifyChatRunMilestones(tx, { companyId });
}
const updated = await tx
.update(companyMemberships)
.set({ status, membershipRole, updatedAt: new Date() })
.where(eq(companyMemberships.id, existing.id))
.returning()
.then((rows) => rows[0] ?? null);
return updated ?? existing;
}
return existing;
}
return existing;
}
return db
.insert(companyMemberships)
.values({
companyId,
principalType,
principalId,
status,
membershipRole,
})
.returning()
.then((rows) => rows[0]);
if (principalType === "user" && status === "active" && membershipRole !== "viewer") {
await notifyChatRunMilestones(tx, { companyId });
}
return tx
.insert(companyMemberships)
.values({
companyId,
principalType,
principalId,
status,
membershipRole,
})
.returning()
.then((rows) => rows[0]);
});
}
async function setPrincipalGrants(
@@ -1122,6 +1142,11 @@ export function accessService(db: Db) {
data.membershipRole !== undefined ? data.membershipRole : existing.membershipRole;
const nextStatus = data.status ?? existing.status;
if (existing.principalType === "user" && nextStatus === "active" && nextMembershipRole !== "viewer"
&& (existing.status !== "active" || existing.membershipRole === "viewer")) {
await notifyChatRunMilestones(tx, { companyId });
}
if (
existing.principalType === "user" &&
existing.status === "active" &&
@@ -17,6 +17,7 @@ import {
type Db,
} from "@paperclipai/db";
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { sanitizeQuarantinedCommentForHigherTrust } from "./source-trust.js";
export type ConversationIdentity = {
@@ -166,6 +167,7 @@ export async function prepareConversationTurn(
}
// The boundary lives on the command comment. A crash/retry reuses it instead of resetting twice.
if (reset && comment && comment.conversationSessionGeneration == null) {
await notifyChatRunMilestones(tx, { companyId: issue.companyId, issueId: issue.id });
generation += 1;
await tx
.update(issues)
+5
View File
@@ -1,6 +1,7 @@
import { leasedChatProvider } from "./chat-channels/runtime/receiver-policy.js";
import { chatReceiverSchedule } from "./chat-channels/runtime/receiver-schedule.js";
import { registerChatReceiverWork } from "./chat-channels/runtime/receiver-work.js";
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { notifyChatEndpointConfiguration } from "./chat-channels/configuration-work-notifications.js";
import { registerDiscordCommandWork } from "./chat-channels/discord/command-work.js";
import { discordCommandSchedule } from "./chat-channels/discord/command-schedule.js";
@@ -2121,6 +2122,9 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
taskUserId: string | null,
) => {
let conversation = existingConversation;
if (conversation) {
await notifyChatRunMilestones(taskTx, { companyId: taskEndpoint.companyId, issueId: conversation.issueId });
}
if (!conversation) {
const [taskResource] = await taskTx.select({ executionDefaults: chatEndpointResources.executionDefaults })
.from(chatEndpointResources).where(and(eq(chatEndpointResources.id, resource.id), eq(chatEndpointResources.endpointId, taskEndpoint.id)));
@@ -2162,6 +2166,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
},
taskTx,
);
await notifyChatRunMilestones(taskTx, { companyId: taskEndpoint.companyId, issueId: issue.id });
await taskTx
.insert(chatConversations)
.values({
@@ -196,7 +196,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
code: "chat_endpoint_not_active",
});
}
await notifyChatEndpointWork(tx, endpoint.connectionId);
await notifyChatEndpointWork(tx, endpoint.connectionId, endpoint.companyId);
await tx
.update(chatEndpoints)
.set({
@@ -334,7 +334,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
inArray(chatDeliveries.state, ["received", "retry"]),
),
);
await notifyChatEndpointWork(tx, endpoint.connectionId);
await notifyChatEndpointWork(tx, endpoint.connectionId, endpoint.companyId);
[activatedEndpoint] = await tx
.update(chatEndpoints)
.set({
@@ -381,9 +381,9 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
const failure = redactError(error);
const failedAt = new Date();
await managementContext?.reauthorize();
await db.transaction(async (tx) => {
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
await notifyChatEndpointWork(tx, endpoint.connectionId);
await notifyChatEndpointWork(tx, endpoint.connectionId, endpoint.companyId);
await tx
.update(chatEndpoints)
.set({
@@ -487,7 +487,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
code: "chat_endpoint_already_removed",
});
}
await notifyChatEndpointWork(tx, endpoint.connectionId);
await notifyChatEndpointWork(tx, endpoint.connectionId, endpoint.companyId);
await tx
.update(chatEndpoints)
.set({
@@ -721,7 +721,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
await managementContext?.reauthorize();
const claimed = await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
await notifyChatEndpointWork(tx, endpoint.connectionId);
await notifyChatEndpointWork(tx, endpoint.connectionId, endpoint.companyId);
const rows = await tx
.update(chatEndpoints)
.set({
@@ -800,7 +800,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
await installSpekoVoiceTools(tx, endpoint, actorUserId);
}
const preserveSlackVerification = preservesProvisionedSlackVerification(current.setup, credentials.signingSecret ?? "", `${getWebhookPublicBaseUrl()}/api/chat-webhooks/${endpoint.publicId}/slack`);
await notifyChatEndpointWork(tx, endpoint.connectionId);
await notifyChatEndpointWork(tx, endpoint.connectionId, endpoint.companyId);
await tx
.update(chatEndpoints)
.set({
@@ -887,7 +887,7 @@ export function createChatConfiguration(db: Db, options: ChatConfigurationOption
await managementContext?.reauthorize();
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
await notifyChatEndpointWork(tx, endpoint.connectionId);
await notifyChatEndpointWork(tx, endpoint.connectionId, endpoint.companyId);
await tx
.update(chatEndpoints)
.set({
@@ -383,7 +383,7 @@ export function createChatCredentials(db: Db, options: ChatCredentialsOptions) {
.set({ credentialSecretRefs: refs, updatedAt: new Date() })
.where(eq(toolConnections.id, endpoint.connectionId));
if (changedFields.length) {
await notifyChatEndpointWork(tx, endpoint.connectionId);
await notifyChatEndpointWork(tx, endpoint.connectionId, endpoint.companyId);
const [currentEndpoint] = await tx.select({ setup: chatEndpoints.setup }).from(chatEndpoints)
.where(and(eq(chatEndpoints.id, endpoint.id), eq(chatEndpoints.companyId, endpoint.companyId))).for("update");
if (!currentEndpoint) throw notFound("Chat endpoint not found");
@@ -165,7 +165,7 @@ export function createDiscordGatewayEvents(db: Db, options: DiscordGatewayEvents
current.setup.step === "complete" ? "active" : "verifying";
await notifyChatReceiver(tx, current);
await notifyDiscordCommandRegistration(tx, { kind: "endpoint", id: current.id });
if (status !== current.status) await notifyChatEndpointWork(tx, current.connectionId);
if (status !== current.status) await notifyChatEndpointWork(tx, current.connectionId, current.companyId);
await tx
.update(chatEndpoints)
.set({
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { chatConnectionTransport } from "./provider-catalog.js";
import type { ConnectionManagementContext } from "../connection-management.js";
import { slackSetupState } from "./slack/setup.js";
@@ -454,6 +455,11 @@ function providerSetupState(
await managementContext?.reauthorize();
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
if ((input.allowDirectMessages === true && !existing.endpoint.allowDirectMessages)
|| (input.allowGroupChats === true && !existing.endpoint.allowGroupChats)
|| (input.allowUnlinkedPeople === true && !existing.endpoint.allowUnlinkedPeople)) {
await notifyChatRunMilestones(tx, { companyId: existing.endpoint.companyId });
}
if (input.executionDefaults !== undefined) {
await assertChatExecutionDefaultsAccess(db, {
companyId: existing.endpoint.companyId,
@@ -201,11 +201,12 @@ export async function reconcileGitHubRepositoryCoordinates(tx: DbTransaction, en
type GitHubResourceOptions = {
endpointRecord: (endpointId: string) => Promise<{ endpoint: EndpointRow } | null>;
upsertProviderResourceRow: (tx: DbTransaction, endpoint: EndpointRow, item: ChatProviderResourceInventoryItem) => Promise<{ id: string } | null>;
restoreResourceConversations: (tx: DbTransaction, endpoint: EndpointRow, resourceId: string) => Promise<void>;
updateResourceSelection: (endpointId: string, updates: Array<{ id: string; enabled: boolean }>, actorUserId?: string | null, options?: { allGitHubRepositoriesEnabled?: boolean }) => Promise<void>;
};
export function createGitHubResources(db: Db, options: GitHubResourceOptions) {
const { endpointRecord, upsertProviderResourceRow, updateResourceSelection } = options;
const { endpointRecord, upsertProviderResourceRow, restoreResourceConversations, updateResourceSelection } = options;
async function reconcileGitHubWebhookRepository(
endpoint: EndpointRow,
payload: unknown,
@@ -220,17 +221,7 @@ export function createGitHubResources(db: Db, options: GitHubResourceOptions) {
// A correctly signed repository callback is current provider proof. Keep
// Paperclip's enabled choice, and reopen only conversations previously
// quarantined for provider availability loss.
await tx
.update(chatConversations)
.set({ state: "active", updatedAt: new Date() })
.where(
and(
eq(chatConversations.companyId, endpoint.companyId),
eq(chatConversations.endpointId, endpoint.id),
eq(chatConversations.resourceId, resource.id),
eq(chatConversations.state, "unavailable"),
),
);
await restoreResourceConversations(tx, endpoint, resource.id);
await credentialLease.assertOwned(tx);
});
}
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { githubIdentityFields } from "./github/identity-links.js";
import { createSlackIdentityLinks, type SlackIdentityEffectStager } from "./slack/identity-links.js";
import { createHash, randomBytes } from "node:crypto";
@@ -319,6 +320,7 @@ export function createChatIdentityLinks(db: Db, options: ChatIdentityLinkOptions
},
);
}
await notifyChatRunMilestones(tx, { companyId: currentLink.companyId });
const confirmedLink = await tx
.update(chatIdentityLinks)
.set({
@@ -32,7 +32,7 @@ export function createPhotonRuntimeCallbacks(db: Db, options: PhotonRuntimeOptio
if (!current) return;
const connection = await tx.select().from(toolConnections).where(eq(toolConnections.id, current.connectionId)).then((rows) => rows[0]);
if (!connection?.enabled || connection.healthStatus === "failed") throw new Error("Photon endpoint requires operator recovery");
await notifyChatEndpointWork(tx, connectionId);
await notifyChatEndpointWork(tx, connectionId, companyId);
await tx.update(chatEndpoints).set({ status: current.setup.step === "complete" ? "active" : "verifying", healthMessage: "Photon receiver connected", lastError: null, updatedAt: new Date() }).where(eq(chatEndpoints.id, endpointId));
});
},
@@ -1,5 +1,6 @@
import { DELIVERY_PROCESSING_STALE_MS } from "./delivery-ledger.js";
import type { EndpointRow, DbOrTransaction } from "../runtime-types.js";
import { notifyChatRunMilestones } from "../../chat-run-milestone-notifications.js";
import { notifyChatDeliveryWork } from "../../chat-work-notifications.js";
import { enqueueFastResponse } from "../../fast-responses.js";
import { and, eq, sql } from "drizzle-orm";
@@ -110,6 +111,9 @@ export function createInboundWakeups(db: Db, options: WakeupOptions) {
);
}
await notifyChatDeliveryWork(tx);
if (action.status === "processed") {
await notifyChatRunMilestones(tx, { companyId: action.companyId, issueId: String(action.payload.issueId) });
}
const acceptedAt = new Date();
const accepted = await tx
.update(chatDeliveries)
@@ -249,6 +253,9 @@ export function createInboundWakeups(db: Db, options: WakeupOptions) {
const settle = async (status: string, result: Record<string, unknown>) => {
const changed = await db.transaction(async (tx) => {
await notifyChatDeliveryWork(tx);
if (status === "processed") {
await notifyChatRunMilestones(tx, { companyId: claimed.companyId, issueId: request!.issueId });
}
const changed = await tx
.update(chatActions)
.set({ status, result, updatedAt: new Date() })
@@ -787,7 +787,7 @@ export function createChatInteractionCallbacks(db: Db, options: InteractionCallb
.then((rows) => rows[0] ?? null);
if (occupied && occupied.publicationId !== current.id) return null;
const reconciledAt = new Date();
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, current.state === "delivery_unknown" ? current : undefined);
const [reconciled] = await tx
.update(chatPublications)
.set({
@@ -120,7 +120,7 @@ export function createChatPublicationFailures(db: Db, options: PublicationFailur
: disposition.kind === "resource_unavailable"
? "cancelled"
: "failed";
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, state === "failed" || state === "cancelled" ? publication : undefined);
const [ownedPublication] = await tx
.update(chatPublications)
.set({
@@ -218,7 +218,7 @@ export function createChatPublicationResolution(db: Db, options: ResolutionOptio
action === "retry_anyway"
) { await recordPhotonPublicationRetry(tx, endpointId, publication, userId); }
const now = new Date();
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, action === "retry_anyway" ? undefined : publication);
await notifyVoicePublication(tx, initialRecord.endpoint, publication.conversationId);
await tx
.update(chatPublications)
@@ -154,7 +154,7 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
? classified.retryAfterMs
: Math.min(60_000, 2 ** Math.max(0, attempts) * 1_000);
const result = await db.transaction(async (tx) => {
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, terminal ? publication : undefined);
const [settled] = await tx
.update(chatPublications)
.set({
@@ -380,7 +380,7 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
}
if (explicitSlackDelivery === "delivered") {
await db.transaction(async (tx) => {
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, publication);
return tx.update(chatPublications).set({ state: "cancelled", redactedError: "Identical response already delivered by Slack tool", updatedAt: new Date() }).where(and(eq(chatPublications.id, publication.id), inArray(chatPublications.state, ["pending", "retry"])))
;
});
@@ -507,7 +507,7 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
) {
await publicationLease.assertOwned();
await db.transaction(async (tx) => {
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, publication);
return tx
.update(chatPublications)
.set({
@@ -531,7 +531,7 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
if (!endpoint.allowDirectMessages) {
await publicationLease.assertOwned();
await db.transaction(async (tx) => {
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, publication);
return tx
.update(chatPublications)
.set({
@@ -566,7 +566,7 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
if (!nonDirectDestinationAllowed(endpoint, resource)) {
await publicationLease.assertOwned();
await db.transaction(async (tx) => {
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, publication);
return tx
.update(chatPublications)
.set({
@@ -605,7 +605,7 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
if (!currentInteraction || currentInteraction.status !== "pending") {
await publicationLease.assertOwned();
await db.transaction(async (tx) => {
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, publication);
return tx
.update(chatPublications)
.set({
@@ -638,7 +638,7 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
if (progressSupersessionReason) {
await publicationLease.assertOwned();
await db.transaction(async (tx) => {
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, publication);
return tx
.update(chatPublications)
.set({
@@ -710,7 +710,7 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
if (!authorizationClaim) {
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, publication);
await tx
.update(chatPublications)
.set({
@@ -858,7 +858,7 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
await db.transaction(async (tx) => {
await credentialLease.assertOwned(tx);
const committedAt = new Date();
await notifyChatPublicationWork(tx);
await notifyChatPublicationWork(tx, publication);
await notifyVoicePublication(tx, authorizationClaim.endpoint, publication.conversationId);
const [completedPublication] = await tx
.update(chatPublications)
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { notifyChatEndpointConfiguration } from "./configuration-work-notifications.js";
import { notifyChatConnectionHealth } from "./connection-health-notifications.js";
import type { ConnectionManagementContext } from "../connection-management.js";
@@ -80,6 +81,32 @@ export function createChatResourceReconciliation(db: Db, options: ResourceReconc
return resource ?? null;
}
async function setResourceConversationAvailability(
tx: DbTransaction,
endpoint: Pick<EndpointRow, "id" | "companyId">,
resourceId: string,
available: boolean,
now: Date,
) {
const affected = and(
eq(chatConversations.companyId, endpoint.companyId),
eq(chatConversations.endpointId, endpoint.id),
eq(chatConversations.resourceId, resourceId),
available ? eq(chatConversations.state, "unavailable") : inArray(chatConversations.state, ["active", "waiting"]),
);
if (available) {
// Only a reopened binding creates projection work. Repeated inventory
// refreshes and completed history must not wake the milestone scanner.
const reopened = await tx.select({ issueId: chatConversations.issueId })
.from(chatConversations).where(affected).for("update");
for (const issueId of new Set(reopened.map(row => row.issueId))) {
await notifyChatRunMilestones(tx, { companyId: endpoint.companyId, issueId });
}
}
await tx.update(chatConversations)
.set({ state: available ? "active" : "unavailable", updatedAt: now }).where(affected);
}
async function reconcileProviderResourceRows(
endpoint: EndpointRow,
inventory: ChatProviderInventoryResult,
@@ -99,17 +126,7 @@ export function createChatResourceReconciliation(db: Db, options: ResourceReconc
if (resource) {
// Resource recovery reopens only bindings that Paperclip marked
// unavailable. Completed historical tasks stay completed.
await tx
.update(chatConversations)
.set({ state: "active", updatedAt: new Date() })
.where(
and(
eq(chatConversations.companyId, endpoint.companyId),
eq(chatConversations.endpointId, endpoint.id),
eq(chatConversations.resourceId, resource.id),
eq(chatConversations.state, "unavailable"),
),
);
await setResourceConversationAvailability(tx, endpoint, resource.id, true, new Date());
}
}
@@ -132,17 +149,7 @@ export function createChatResourceReconciliation(db: Db, options: ResourceReconc
.update(chatEndpointResources)
.set({ availability: absentAvailability, updatedAt: new Date() })
.where(eq(chatEndpointResources.id, resource.id));
await tx
.update(chatConversations)
.set({ state: "unavailable", updatedAt: new Date() })
.where(
and(
eq(chatConversations.companyId, endpoint.companyId),
eq(chatConversations.endpointId, endpoint.id),
eq(chatConversations.resourceId, resource.id),
inArray(chatConversations.state, ["active", "waiting"]),
),
);
await setResourceConversationAvailability(tx, endpoint, resource.id, false, new Date());
}
await credentialLease.assertOwned(tx);
});
@@ -279,6 +286,9 @@ export function createChatResourceReconciliation(db: Db, options: ResourceReconc
before: { enabled: row.enabled, ...(JSON.stringify(row.executionDefaults) !== JSON.stringify(finalDefaults.get(row.id)) ? { executionDefaults: row.executionDefaults } : {}) },
after: { enabled: finalEnabled.get(row.id)!, ...(JSON.stringify(row.executionDefaults) !== JSON.stringify(finalDefaults.get(row.id)) ? { executionDefaults: finalDefaults.get(row.id) } : {}) },
}));
if (changes.some(change => !change.before.enabled && change.after.enabled)) {
await notifyChatRunMilestones(tx, { companyId: endpoint.companyId });
}
if (bulk) {
// One atomic database update, independent of pagination/search and
// serialized with refresh, reconnect, and individual row changes.
@@ -471,25 +481,7 @@ export function createChatResourceReconciliation(db: Db, options: ResourceReconc
await migrateTelegramResourceConversations(tx, currentEndpoint, previousResource, resource, previousProviderResourceId, providerResourceId, migratedLabel, now);
}
if (resource) {
await tx
.update(chatConversations)
.set({
state:
effect.availability === "available"
? "active"
: "unavailable",
updatedAt: now,
})
.where(
and(
eq(chatConversations.companyId, currentEndpoint.companyId),
eq(chatConversations.endpointId, currentEndpoint.id),
eq(chatConversations.resourceId, resource.id),
effect.availability === "available"
? eq(chatConversations.state, "unavailable")
: inArray(chatConversations.state, ["active", "waiting"]),
),
);
await setResourceConversationAvailability(tx, currentEndpoint, resource.id, effect.availability === "available", now);
}
await tx
.update(chatEndpoints)
@@ -512,7 +504,7 @@ export function createChatResourceReconciliation(db: Db, options: ResourceReconc
? "verifying"
: currentEndpoint.status;
await notifyChatEndpointConfiguration(tx, currentEndpoint);
if (status !== currentEndpoint.status) await notifyChatEndpointWork(tx, currentEndpoint.connectionId);
if (status !== currentEndpoint.status) await notifyChatEndpointWork(tx, currentEndpoint.connectionId, currentEndpoint.companyId);
await tx
.update(chatEndpoints)
.set({
@@ -616,7 +608,10 @@ export function createChatResourceReconciliation(db: Db, options: ResourceReconc
});
}
const { reconcileGitHubWebhookRepository, listGitHubRepositories, toggleAllGitHubRepositories } = createGitHubResources(db, { endpointRecord, upsertProviderResourceRow, updateResourceSelection });
const { reconcileGitHubWebhookRepository, listGitHubRepositories, toggleAllGitHubRepositories } = createGitHubResources(db, {
endpointRecord, upsertProviderResourceRow, updateResourceSelection,
restoreResourceConversations: (tx, endpoint, resourceId) => setResourceConversationAvailability(tx, endpoint, resourceId, true, new Date()),
});
return { reconcileGitHubWebhookRepository, reconcileProviderResourceRows, listResources, listGitHubRepositories, replaceResources, toggleAllGitHubRepositories, hasNewerProcessedProviderLifecycleEffect, applyProviderAvailability };
}
@@ -300,7 +300,7 @@ export function createSetupProbes(db: Db, options: SetupProbesOptions) {
)).limit(1).then((rows) => rows[0]);
if (!linked || !(await lockCurrentPrincipalAuthorization(tx, endpoint, linked.principalId)).allowed) throw forbidden("Your connected account is no longer authorized");
}
await notifyChatEndpointWork(tx, endpoint.connectionId);
await notifyChatEndpointWork(tx, endpoint.connectionId, endpoint.companyId);
const [activated] = await tx
.update(chatEndpoints)
.set({
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../../chat-run-milestone-notifications.js";
import { chatIdentityLinks, chatExternalPrincipals, companyMemberships, toolConnections, type Db } from "@paperclipai/db";
import { and, eq, ne, sql } from "drizzle-orm";
import { conflict, forbidden, notFound } from "../../../errors.js";
@@ -46,6 +47,7 @@ export function createSlackSetupServices(db: Db, options: SetupServicesOptions)
const links = await tx.select().from(chatIdentityLinks).where(and(eq(chatIdentityLinks.companyId, record.endpoint.companyId), eq(chatIdentityLinks.principalId, principal.id))).for("update");
if (links.some(link => link.status === "linked" && link.paperclipUserId !== actor.userId) || links.some(link => link.endpointId === endpointId && link.status === "revoked")) throw conflict("This Slack account has an existing link", { code: "chat_identity_link_conflict" });
if (links.some(link => link.endpointId === endpointId && link.status === "linked")) return;
await notifyChatRunMilestones(tx, { companyId: record.endpoint.companyId });
await tx.insert(chatIdentityLinks).values({ companyId: record.endpoint.companyId, endpointId, principalId: principal.id,
paperclipUserId: actor.userId, status: "linked", confirmedAt: new Date() })
.onConflictDoUpdate({ target: [chatIdentityLinks.endpointId, chatIdentityLinks.principalId], set: {
@@ -1,5 +1,6 @@
import { and, eq, sql } from "drizzle-orm";
import { chatConversations, chatEndpointResources, chatEndpoints, type Db } from "@paperclipai/db";
import { notifyChatRunMilestones } from "../../chat-run-milestone-notifications.js";
import type { ChatProviderLifecycleEffect } from "../../chat-provider-lifecycle.js";
type DbTransaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
type EndpointRow = typeof chatEndpoints.$inferSelect;
@@ -70,6 +71,18 @@ export async function migrateTelegramResourceConversations(
resource: { id: string }, previousProviderResourceId: string | undefined,
providerResourceId: string, migratedLabel: string, now: Date,
) {
const affected = and(
eq(chatConversations.companyId, currentEndpoint.companyId),
eq(chatConversations.endpointId, currentEndpoint.id),
eq(chatConversations.resourceId, previousResource.id),
);
const migrated = await tx.select({ issueId: chatConversations.issueId })
.from(chatConversations).where(affected).for("update");
// Migration restores the binding before generic resource availability runs.
// Register the wake here, within the same owning transaction as that change.
for (const issueId of new Set(migrated.map(row => row.issueId))) {
await notifyChatRunMilestones(tx, { companyId: currentEndpoint.companyId, issueId });
}
const previousThreadPrefix = `telegram:${previousProviderResourceId}`;
const migratedThreadPrefix = `telegram:${providerResourceId}`;
await tx
@@ -94,11 +107,5 @@ export async function migrateTelegramResourceConversations(
state: "active",
updatedAt: now,
})
.where(
and(
eq(chatConversations.companyId, currentEndpoint.companyId),
eq(chatConversations.endpointId, currentEndpoint.id),
eq(chatConversations.resourceId, previousResource.id),
),
);
.where(affected);
}
+13 -3
View File
@@ -24,6 +24,7 @@ import {
import { HttpError, badRequest, conflict, forbidden, notFound, unprocessable } from "../errors.js";
import { toolAccessService } from "./tool-access.js";
import { logActivity } from "./activity-log.js";
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { githubBotRequest } from "./chat-github-client.js";
import { environmentService } from "./environments.js";
import { instanceSettingsService } from "./instance-settings.js";
@@ -145,11 +146,12 @@ export function githubChatManagementService(db: Db, fetchImpl = fetch) {
"Repository overrides must refer to this App installation",
);
await db.transaction(async (tx) => {
await tx
.select({ id: chatEndpoints.id })
const [currentEndpoint] = await tx
.select({ allowUnlinkedPeople: chatEndpoints.allowUnlinkedPeople, sponsorUserId: chatEndpoints.sponsorUserId })
.from(chatEndpoints)
.where(eq(chatEndpoints.id, endpointId))
.for("update");
if (!currentEndpoint) throw notFound("GitHub bot not found");
await tx.execute(
sql`select pg_advisory_xact_lock(hashtextextended(${`github-config:${endpointId}`}, 0))`,
);
@@ -160,6 +162,11 @@ export function githubChatManagementService(db: Db, fetchImpl = fetch) {
.for("update");
if ((current?.revision ?? 0) !== parsed.expectedRevision)
throw conflict("This configuration changed. Refresh before saving.");
const allowUnlinkedPeople = config.people.some((p) => p.kind === "guest");
if ((!currentEndpoint.allowUnlinkedPeople && allowUnlinkedPeople) ||
(allowUnlinkedPeople && currentEndpoint.sponsorUserId !== config.responsibleUserId)) {
await notifyChatRunMilestones(tx, { companyId: bot.companyId });
}
const values = {
companyId: bot.companyId,
endpointId,
@@ -180,7 +187,7 @@ export function githubChatManagementService(db: Db, fetchImpl = fetch) {
await tx
.update(chatEndpoints)
.set({
allowUnlinkedPeople: config.people.some((p) => p.kind === "guest"),
allowUnlinkedPeople,
sponsorUserId: config.responsibleUserId,
updatedAt: new Date(),
})
@@ -327,6 +334,9 @@ export function githubChatManagementService(db: Db, fetchImpl = fetch) {
expiresAt: null,
updatedAt: new Date(),
};
if (prior?.status !== "linked") {
await notifyChatRunMilestones(tx, { companyId: bot.companyId });
}
await tx
.insert(chatIdentityLinks)
.values(values)
@@ -15,19 +15,20 @@ import {
toolConnections,
} from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
import { hasChatRunOwnedProviderInteraction } from "./chat-interaction-arbitration.js";
const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_ARBITRATION_TEST_DATABASE_URL");
describe("provider-owned interaction arbitration", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: ReturnType<typeof createDb>;
let companyCount = 0;
beforeAll(async () => {
temporary = await startEmbeddedPostgresTestDatabase(
"chat-interaction-owner-",
);
db = createDb(temporary.connectionString);
if (!externalDatabaseUrl) temporary = await startEmbeddedPostgresTestDatabase("chat-interaction-owner-");
db = createDb(externalDatabaseUrl ?? temporary!.connectionString);
}, 30_000);
afterAll(async () => {
await db?.$client.end();
await temporary?.cleanup();
});
@@ -43,7 +44,7 @@ describe("provider-owned interaction arbitration", () => {
.values({
id: companyId,
name: "Arbitration",
issuePrefix: `CI${String.fromCharCode(65 + companyCount++)}`,
issuePrefix: `C${companyId.slice(0, 7)}`,
});
await db.insert(agents).values(
[agentId, otherAgentId].map((id) => ({
@@ -0,0 +1,9 @@
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
/** Lineage changes can authorize descendants, so invalidate the issue rather
* than just the run whose row changed. Configuration may invalidate a company. */
export type ChatRunMilestoneScope = { companyId: string; issueId?: string };
export function notifyChatRunMilestones(tx: object, scope: ChatRunMilestoneScope) {
return notifyDeliveryWork(tx, DELIVERY_QUEUES.chatRunMilestones, JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId }));
}
@@ -0,0 +1,38 @@
import type { Db } from "@paperclipai/db";
import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
import { registerScopedDeadlineWork } from "./scoped-deadline-work.js";
import type { ChatRunMilestoneScope } from "./chat-run-milestone-notifications.js";
import { enqueueChatRunLifecycleMilestones, loadChatRunLifecycleMilestoneScopes } from "./chat-run-publications.js";
type Entry = { id: string; companyId: string; issueId: string; dueAt: number };
const batchSize = 100;
/** No periodic discovery: reconstruct on startup/committed changes, drain
* current notices, and retain a retry only for failure or an unfinished batch. */
export function chatRunMilestoneWork(db: Db, publicBaseUrl?: string | null) {
async function load(scope?: ChatRunMilestoneScope, signal?: AbortSignal): Promise<Entry[]> {
return (await loadChatRunLifecycleMilestoneScopes(db, scope, signal)).map(row => ({
...row, id: `${row.companyId}:${row.issueId}`, dueAt: 0,
}));
}
async function check(entry: Entry, signal?: AbortSignal): Promise<Entry | null> {
const inserted = await enqueueChatRunLifecycleMilestones(db, {
scope: { companyId: entry.companyId, issueId: entry.issueId },
publicBaseUrl, limit: batchSize, signal,
});
// A short batch exhausted the selector, including currently suppressed
// candidates. Their dependencies must signal a later eligibility change.
return inserted === batchSize ? { ...entry, dueAt: Date.now() } : null;
}
function register(coordinator: ReturnType<typeof createDeliveryWorkCoordinator>, canRun: () => boolean) {
return registerScopedDeadlineWork(coordinator, {
queue: DELIVERY_QUEUES.chatRunMilestones,
decodeScope: (hint: string): ChatRunMilestoneScope | undefined => hint === "*" ? undefined : JSON.parse(hint),
matches: (entry: Entry, scope?: ChatRunMilestoneScope) => !scope ||
(entry.companyId === scope.companyId && (!scope.issueId || entry.issueId === scope.issueId)),
load, check,
}, canRun);
}
return { load, check, register };
}
+99 -61
View File
@@ -1,3 +1,4 @@
import type { ChatRunMilestoneScope } from "./chat-run-milestone-notifications.js";
import { notifyChatPublicationWork } from "./chat-work-notifications.js";
import {
and,
@@ -218,7 +219,7 @@ type ChatRunMilestoneCandidate = {
runId: string;
runStatus: string;
runErrorCode: string | null;
runUpdatedAt: Date;
runUpdatedAt: string;
issueId: string;
companyId: string;
endpointId: string;
@@ -297,10 +298,11 @@ export function safeMilestoneText(input: {
* lane. The selector intentionally reads only event identity, type, sequence,
* and time; native messages and payloads stay inside Paperclip.
*/
async function enqueueSafeNativeChatProgress(
export async function enqueueSafeNativeChatProgress(
db: Db,
input: { since: Date; limit: number },
options: { since?: Date; limit?: number } = {},
): Promise<number> {
const input = chatRunProjectionWindow(options);
const basePublication = alias(
chatPublications,
"safe_native_progress_base_publication",
@@ -670,26 +672,19 @@ async function enqueueSafeNativeChatProgress(
return inserted;
}
/**
* Project only coarse run lifecycle into bound external conversations. Raw
* output, errors, tool events, and reasoning stay in Paperclip. Idempotency is
* keyed by run, milestone, and endpoint so polling and restarts are harmless.
*/
export async function enqueueChatRunMilestones(
db: Db,
input: {
publicBaseUrl?: string | null;
since?: Date;
limit?: number;
} = {},
): Promise<number> {
const since = input.since ?? new Date(Date.now() - 24 * 60 * 60_000);
const limit = Math.max(1, Math.min(input.limit ?? 200, 1_000));
const issueIdFromContext = boundRunIssueIdSql();
const explicitlyAuthoredCommentReason = sql<boolean>`(
type ChatRunMilestoneCursor = {
updatedAt: string;
runId: string;
conversationId: string;
};
const explicitlyAuthoredCommentReason = sql<boolean>`(
${issueComments.metadata} ->> 'authorizationReason' = 'paperclip_runner_protocol'
or left(coalesce(${issueComments.metadata} ->> 'authorizationReason', ''), 6) = 'allow_'
)`;
function lifecycleMilestonePages(db: Db, since: Date, scope?: ChatRunMilestoneScope) {
const issueIdFromContext = boundRunIssueIdSql();
const milestoneFromStatus = sql<string>`case
when ${heartbeatRuns.status} = 'queued' then 'queued'
when ${heartbeatRuns.status} = 'running'
@@ -801,26 +796,15 @@ export async function enqueueChatRunMilestones(
and continuation_wake.payload->>'issueId' = ${issueIdFromContext}
)
)`;
let inserted = 0;
let cursor: {
updatedAt: Date;
runId: string;
conversationId: string;
} | null = null;
const bindingsCache = new Map<
string,
Awaited<ReturnType<typeof resolveChatOriginPublicationBindings>>
>();
const providerInteractionCache = new Map<string, boolean>();
while (inserted < limit) {
const pageSize = Math.max(25, Math.min(200, limit - inserted));
const pageCursor: typeof cursor = cursor;
const rows: ChatRunMilestoneCandidate[] = await db
return (pageSize: number, pageCursor: ChatRunMilestoneCursor | null): Promise<ChatRunMilestoneCandidate[]> => {
return db
.select({
runId: heartbeatRuns.id,
runStatus: heartbeatRuns.status,
runErrorCode: heartbeatRuns.errorCode,
runUpdatedAt: heartbeatRuns.updatedAt,
// Keep PostgreSQL precision: a Date cursor can repeatedly select a
// rejected page whose updated_at has submillisecond digits.
runUpdatedAt: sql<string>`${heartbeatRuns.updatedAt}::text`,
issueId: chatConversations.issueId,
companyId: chatConversations.companyId,
endpointId: chatConversations.endpointId,
@@ -979,22 +963,13 @@ export async function enqueueChatRunMilestones(
and target_delivery.status in ('delivered', 'fallback_queued')
)`,
),
scope ? eq(heartbeatRuns.companyId, scope.companyId) : undefined,
scope?.issueId ? eq(chatConversations.issueId, scope.issueId) : undefined,
gte(heartbeatRuns.updatedAt, since),
isNull(chatPublications.id),
pageCursor
? or(
gt(heartbeatRuns.updatedAt, pageCursor.updatedAt),
and(
eq(heartbeatRuns.updatedAt, pageCursor.updatedAt),
or(
gt(heartbeatRuns.id, pageCursor.runId),
and(
eq(heartbeatRuns.id, pageCursor.runId),
gt(chatConversations.id, pageCursor.conversationId),
),
),
),
)
? sql`(${heartbeatRuns.updatedAt}, ${heartbeatRuns.id}, ${chatConversations.id}) >
(${pageCursor.updatedAt}::timestamptz, ${pageCursor.runId}::uuid, ${pageCursor.conversationId}::uuid)`
: undefined,
),
)
@@ -1004,15 +979,59 @@ export async function enqueueChatRunMilestones(
asc(chatConversations.id),
)
.limit(pageSize);
};
}
function milestoneCursor(row: ChatRunMilestoneCandidate): ChatRunMilestoneCursor {
return { updatedAt: row.runUpdatedAt, runId: row.runId, conversationId: row.conversationId };
}
/** Reconstruct issue-scoped projection work. Candidate rows are only hints;
* projection still applies current lineage and provider authority. */
export async function loadChatRunLifecycleMilestoneScopes(
db: Db,
scope?: ChatRunMilestoneScope,
signal?: AbortSignal,
): Promise<Array<{ companyId: string; issueId: string }>> {
const pages = lifecycleMilestonePages(db, chatRunProjectionWindow({}).since, scope);
const scopes = new Map<string, { companyId: string; issueId: string }>();
let cursor: ChatRunMilestoneCursor | null = null;
for (;;) {
signal?.throwIfAborted();
const rows = await pages(100, cursor);
for (const row of rows) scopes.set(`${row.companyId}:${row.issueId}`, { companyId: row.companyId, issueId: row.issueId });
if (rows.length < 100) return [...scopes.values()];
cursor = milestoneCursor(rows[rows.length - 1]);
}
}
/**
* Project only coarse run lifecycle into bound external conversations. Raw
* output, errors, tool events, and reasoning stay in Paperclip. Idempotency is
* keyed by run, milestone, and endpoint so polling and restarts are harmless.
*/
export async function enqueueChatRunLifecycleMilestones(
db: Db,
input: ChatRunProjectionInput & { scope?: ChatRunMilestoneScope; signal?: AbortSignal } = {},
): Promise<number> {
const { since, limit } = chatRunProjectionWindow(input);
let inserted = 0;
const pages = lifecycleMilestonePages(db, since, input.scope);
let cursor: ChatRunMilestoneCursor | null = null;
const bindingsCache = new Map<
string,
Awaited<ReturnType<typeof resolveChatOriginPublicationBindings>>
>();
const providerInteractionCache = new Map<string, boolean>();
while (inserted < limit) {
const pageSize = Math.max(25, Math.min(200, limit - inserted));
input.signal?.throwIfAborted();
const rows = await pages(pageSize, cursor);
if (rows.length === 0) break;
const lastRow = rows.at(-1)!;
cursor = {
updatedAt: lastRow.runUpdatedAt,
runId: lastRow.runId,
conversationId: lastRow.conversationId,
};
cursor = milestoneCursor(rows[rows.length - 1]);
for (const row of rows) {
input.signal?.throwIfAborted();
if (inserted >= limit) break;
const milestone = milestoneForStatus(row.runStatus, row.runErrorCode);
if (!milestone) continue;
@@ -1107,11 +1126,30 @@ export async function enqueueChatRunMilestones(
}
if (rows.length < pageSize) break;
}
if (inserted < limit) {
inserted += await enqueueSafeNativeChatProgress(db, {
since,
limit: limit - inserted,
});
}
return inserted;
}
type ChatRunProjectionInput = {
publicBaseUrl?: string | null;
since?: Date;
limit?: number;
};
function chatRunProjectionWindow(input: ChatRunProjectionInput) {
return {
since: input.since ?? new Date(Date.now() - 24 * 60 * 60_000),
limit: Math.max(1, Math.min(input.limit ?? 200, 1_000)),
};
}
/** Explicit combined reconciliation retains lifecycle priority and one shared
* publication budget. Background owners can reconcile the two duties separately. */
export async function enqueueChatRunMilestones(
db: Db,
input: ChatRunProjectionInput = {},
): Promise<number> {
const { since, limit } = chatRunProjectionWindow(input);
const inserted = await enqueueChatRunLifecycleMilestones(db, { ...input, since, limit });
if (inserted >= limit) return inserted;
return inserted + await enqueueSafeNativeChatProgress(db, { since, limit: limit - inserted });
}
+15 -2
View File
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { notifyChatConnectionConfiguration } from "./chat-channels/configuration-work-notifications.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
@@ -5,10 +6,19 @@ export async function notifyChatDeliveryWork(tx: object): Promise<void> {
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatInbound);
}
export async function notifyChatPublicationWork(tx: object): Promise<void> {
export async function notifyChatPublicationWork(
tx: object,
milestoneScope?: { companyId: string; issueId: string | null },
): Promise<void> {
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatPublications);
// Receipt recovery waits for the publication owner to release its claim.
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatReceipts);
// Published prompts and terminal delivery outcomes can change run lineage or
// release its provider response slot. Callers supply the affected issue only
// for those changes; transport claim/lease bookkeeping has no projection debt.
if (milestoneScope?.issueId) {
await notifyChatRunMilestones(tx, { companyId: milestoneScope.companyId, issueId: milestoneScope.issueId });
}
}
export const CHAT_ACTION_QUEUES = {
@@ -32,7 +42,10 @@ export async function notifyChatVerificationWork(tx: object): Promise<void> {
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatVerification);
}
export async function notifyChatEndpointWork(tx: object, connectionId: string): Promise<void> {
export async function notifyChatEndpointWork(tx: object, connectionId: string, companyId: string): Promise<void> {
// Configuration/status transitions can restore post-control publication
// authority. Ordinary health checks use the narrower configuration signals.
await notifyChatRunMilestones(tx, { companyId });
await notifyChatDeliveryWork(tx);
await notifyChatPublicationWork(tx);
for (const queue of Object.values(CHAT_ACTION_QUEUES)) await notifyDeliveryWork(tx, queue);
@@ -57,6 +57,7 @@ export const DELIVERY_QUEUES = {
chatVerification: "chat-verification",
chatInbound: "chat-inbound",
chatPublications: "chat-publications",
chatRunMilestones: "chat-run-milestones",
chatReceipts: "chat-slack-receipts",
browser: "browser-use",
email: "email-channels",
+6 -4
View File
@@ -3,6 +3,7 @@ import type { Db } from "@paperclipai/db";
import { agents, agentWakeupRequests, caseHandoffs, cases, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns, issueComments, issues, nativeRunFinalizations } from "@paperclipai/db";
import { eq, inArray, sql } from "drizzle-orm";
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { createPostgresWakeQueueAdapter } from "../../modules/wake-queue/adapters/postgres.js";
import { createReleaseIssueExecution } from "../../modules/wake-queue/application/use-cases.js";
@@ -74,14 +75,15 @@ describe("heartbeat queue module boundary", () => {
});
});
const support = await getEmbeddedPostgresTestSupport();
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_QUEUE_TEST_DATABASE_URL");
const support = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
if (!support.supported) console.warn(`Skipping queue module database tests: ${support.reason}`);
describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: Db;
beforeAll(async () => {
database = await startEmbeddedPostgresTestDatabase("paperclip-heartbeat-queue-module");
db = createDb(database.connectionString);
if (!external) database = await startEmbeddedPostgresTestDatabase("paperclip-heartbeat-queue-module");
db = createDb(external ?? database!.connectionString);
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
afterAll(async () => {
await db?.$client.end();
+85 -53
View File
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { assertTaskRetryConversationAllowed, readTaskRetryIntent, readTaskRetryBoardMessageOrigin, retireSupersededTaskRetry, taskRetryMutationAllowed } from "../task-retry.js";
import { insertIssueComment } from "../issue-comment-writes.js";
import { withExternalAdmissionGuard, assertNoExternalOverlap } from "../../modules/external-agents/index.js";
@@ -845,6 +846,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
const error = stopped
? "Automatic continuation stopped by the committed chat conversation close. Send a new request in chat or on the Board to start fresh work."
: "Automatic continuation source could not be verified before provider admission. Review the task and send a fresh request; this attempt will not automatically retry.";
await notifyChatRunMilestones(tx, { companyId: current.companyId, issueId });
[terminal] = await tx
.update(heartbeatRuns)
.set({
@@ -960,9 +962,17 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
if (!["pending_dispatch", "delivered"].includes(caseHandoffReceipt.state) || caseHandoffReceipt.toAgentId !== run.agentId ||
!current || current.version !== caseHandoffReceipt.version || current.issueId !== caseHandoffReceipt.issueId ||
current.agentId !== run.agentId || (!run.startedAt && current.statusVersion !== caseHandoffReceipt.appliedIssueStatusVersion)) {
const [cancelled] = await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date(),
errorCode: "case_handoff_superseded", error: "This case handoff was cancelled or superseded.", updatedAt: new Date() })
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.status, "queued"))).returning();
const cancelled = await db.transaction(async tx => {
const [currentRun] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.status, "queued")))
.for("update");
if (!currentRun) return null;
await notifyChatRunMilestones(tx, { companyId: run.companyId });
const [row] = await tx.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date(),
errorCode: "case_handoff_superseded", error: "This case handoff was cancelled or superseded.", updatedAt: new Date() })
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.status, "queued"))).returning();
return row;
});
if (cancelled) await setWakeupStatus(run.wakeupRequestId, "cancelled", { finishedAt: new Date(), error: cancelled.error });
return null;
}
@@ -1317,6 +1327,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
if (proof.kind === "stopped") {
const error =
"Automatic continuation stopped by the committed chat conversation close. Send a fresh request in chat or on the Board.";
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
const [cancelled] = await tx
.update(heartbeatRuns)
.set({
@@ -1395,6 +1406,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
// explicitly bound queued-message envelope is subject to the
// live-comment discard gate below.
await withExternalAdmissionGuard(tx, run.companyId, run.agentId, () => assertNoExternalOverlap(tx, run.companyId, run.agentId));
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
const [claimedRun] = await tx
.update(heartbeatRuns)
.set({
@@ -1436,6 +1448,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
});
if (liveIds.length === 0) {
const reason = "Queued messages were discarded before dispatch";
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
const [cancelled] = await tx
.update(heartbeatRuns)
.set({
@@ -1483,6 +1496,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
}
await withExternalAdmissionGuard(tx, run.companyId, run.agentId, () => assertNoExternalOverlap(tx, run.companyId, run.agentId));
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
await tx
.update(agentWakeupRequests)
.set({
@@ -1593,6 +1607,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
)).for("update");
if (!lockedRun || lockedRun.status !== "queued") return null;
await withExternalAdmissionGuard(claimTx, run.companyId, run.agentId, () => assertNoExternalOverlap(claimTx, run.companyId, run.agentId));
await notifyChatRunMilestones(claimTx, { companyId: run.companyId, issueId: issueId ?? undefined });
const claimedRun = await claimTx.update(heartbeatRuns).set(claimValues).where(and(
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.status, "queued"),
)).returning().then((rows) => rows[0] ?? null);
@@ -1690,6 +1705,11 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
async function releaseRunClaimedJustBeforeSuppression(runId: string) {
const now = new Date();
await db.transaction(async (tx) => {
const [candidate] = await tx.select({ companyId: heartbeatRuns.companyId, contextSnapshot: heartbeatRuns.contextSnapshot })
.from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.status, "running"))).for("update");
if (!candidate) return;
await notifyChatRunMilestones(tx, { companyId: candidate.companyId,
issueId: readNonEmptyString(parseObject(candidate.contextSnapshot).issueId) ?? undefined });
const released = await tx
.update(heartbeatRuns)
.set({
@@ -3454,6 +3474,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
const reason = issueCancelled
? "Cancelled because the issue was cancelled before the scheduled retry became due"
: "Cancelled because the issue was reassigned before the scheduled retry became due";
await notifyChatRunMilestones(tx, { companyId: scheduledRun.companyId, issueId: issue.id });
const cancelled = await tx
.update(heartbeatRuns)
.set({
@@ -3590,6 +3611,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
issue.assigneeAgentId &&
activeExecutionRun.agentId !== issue.assigneeAgentId
) {
await notifyChatRunMilestones(tx, { companyId: activeExecutionRun.companyId, issueId: issue.id });
const cancelled = await tx
.update(heartbeatRuns)
.set({
@@ -4309,6 +4331,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
adoptedCommentIds = await undeliveredLegacyUserCommentIds(tx as unknown as Db,
agent.companyId, issueId, agentId, adoptedCommentIds);
}
await notifyChatRunMilestones(tx, { companyId: agent.companyId, issueId: issueId ?? undefined });
const newRun = await tx
.insert(heartbeatRuns)
.values({
@@ -4701,6 +4724,13 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
>();
for (const candidate of candidates) {
const payload = parseObject(candidate.payload);
const wakeContext = parseObject(payload._paperclipWakeContext);
const issueId =
readNonEmptyString(payload.issueId) ??
readNonEmptyString(payload.taskId) ??
readNonEmptyString(wakeContext.issueId) ??
null;
const dispatchActorId = `native-status-wake-dispatch:${candidate.id}`;
const existingDispatch = await db
.select()
@@ -4722,24 +4752,27 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
: existingDispatch.status === "deferred_issue_execution"
? "coalesced"
: existingDispatch.status;
await db
.update(agentWakeupRequests)
.set({
status: recoveredStatus,
runId: existingDispatch.runId,
finishedAt:
(existingDispatch.runId ?? existingDispatch.finishedAt)
? (existingDispatch.finishedAt ?? now)
: null,
error: existingDispatch.error,
updatedAt: now,
})
.where(
and(
eq(agentWakeupRequests.id, candidate.id),
isNull(agentWakeupRequests.runId),
),
);
await db.transaction(async tx => {
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, issueId: issueId ?? undefined });
await tx
.update(agentWakeupRequests)
.set({
status: recoveredStatus,
runId: existingDispatch.runId,
finishedAt:
(existingDispatch.runId ?? existingDispatch.finishedAt)
? (existingDispatch.finishedAt ?? now)
: null,
error: existingDispatch.error,
updatedAt: now,
})
.where(
and(
eq(agentWakeupRequests.id, candidate.id),
isNull(agentWakeupRequests.runId),
),
);
});
recovered += 1;
continue;
}
@@ -4773,13 +4806,6 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
.returning({ id: agentWakeupRequests.id });
if (claimed.length === 0) continue;
const payload = parseObject(candidate.payload);
const wakeContext = parseObject(payload._paperclipWakeContext);
const issueId =
readNonEmptyString(payload.issueId) ??
readNonEmptyString(payload.taskId) ??
readNonEmptyString(wakeContext.issueId) ??
null;
const scopeKey = issueId
? `${candidate.companyId}:${candidate.agentId}:${issueId}:${readNativeReviewAssignmentContext(wakeContext)?.nativeReviewInteractionId ?? ""}`
: null;
@@ -4787,16 +4813,19 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
? deliveredByIssueScope.get(scopeKey)
: null;
if (priorDelivery) {
await db
.update(agentWakeupRequests)
.set({
status: "coalesced",
runId: priorDelivery.runId,
finishedAt: new Date(),
error: null,
updatedAt: new Date(),
})
.where(eq(agentWakeupRequests.id, candidate.id));
await db.transaction(async tx => {
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, issueId: issueId ?? undefined });
await tx
.update(agentWakeupRequests)
.set({
status: "coalesced",
runId: priorDelivery.runId,
finishedAt: new Date(),
error: null,
updatedAt: new Date(),
})
.where(eq(agentWakeupRequests.id, candidate.id));
});
recovered += 1;
continue;
}
@@ -4895,21 +4924,24 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
// promote both rows (the original has no nativeStatusWakeIntentId
// provenance and would otherwise run once before the dispatch receipt).
const deliveredStatus = delivered?.status ?? "queued";
await db
.update(agentWakeupRequests)
.set({
status: wakeRun || deliveredStatus === "deferred_issue_execution"
? "coalesced"
: deliveredStatus,
runId: wakeRun?.id ?? delivered?.runId ?? null,
finishedAt:
wakeRun || delivered?.finishedAt
? (delivered?.finishedAt ?? new Date())
: null,
error: delivered?.error ?? null,
updatedAt: new Date(),
})
.where(eq(agentWakeupRequests.id, candidate.id));
await db.transaction(async tx => {
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, issueId: issueId ?? undefined });
await tx
.update(agentWakeupRequests)
.set({
status: wakeRun || deliveredStatus === "deferred_issue_execution"
? "coalesced"
: deliveredStatus,
runId: wakeRun?.id ?? delivered?.runId ?? null,
finishedAt:
wakeRun || delivered?.finishedAt
? (delivered?.finishedAt ?? new Date())
: null,
error: delivered?.error ?? null,
updatedAt: new Date(),
})
.where(eq(agentWakeupRequests.id, candidate.id));
});
if (scopeKey) {
deliveredByIssueScope.set(scopeKey, {
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { measureRunPhase } from "../execution-preparation-timing.js";
import { PROJECT_ACCESS_DENIED, ProjectAccessDeniedError } from "../project-access-denial.js";
import { pendingCaseHandoffForRun } from "./case-handoff-lifecycle.js";
@@ -891,13 +892,13 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC
};
}
await db
.update(heartbeatRuns)
.set({
await db.transaction(async tx => {
await notifyChatRunMilestones(tx, { companyId: livenessRun.companyId, issueId: issueId ?? undefined });
await tx.update(heartbeatRuns).set({
resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({ presentationDecision })}::jsonb`,
updatedAt: new Date(),
})
.where(eq(heartbeatRuns.id, livenessRun.id));
}).where(eq(heartbeatRuns.id, livenessRun.id));
});
await appendRunEvent(livenessRun, {
eventType: "run.presentation.resolved",
stream: "system",
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
import os from "node:os";
import {
@@ -1019,6 +1020,7 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl
const tx = transaction as unknown as Db;
if (isHeartbeatRunTerminalStatus(status)) await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, companyId);
await notifyExecutionRunWork(tx);
if (companyId) await notifyChatRunMilestones(tx, { companyId });
return write(tx);
});
}
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { notifyToolReviewChange } from "./tool-review-work-notifications.js";
import { normalizeEscapedLineBreaks } from "@paperclipai/shared/validators/text";
import { notifyDeliveryWork, DELIVERY_QUEUES } from "./delivery-work-notifications.js";
@@ -1912,6 +1913,7 @@ async function expireStaleRequestConfirmationTarget(
snapshot,
});
const expireInTransaction = async (tx: Db | any) => {
if (args.row.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.row);
const [updated] = await tx
.update(issueThreadInteractions)
.set({
@@ -2189,6 +2191,7 @@ export function issueThreadInteractionService(
const postCommitActivityPublications: ActivityPublication[] = [];
const result = await db.transaction(async (tx) => {
await assertInteractionRunWriteAllowed(tx as unknown as Db, args.issue, args.actor);
if (args.current.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.current);
await args.mutationOptions?.beforeResolveInTransaction?.(tx);
// Policy mutations and review transitions use the same issue-row lock,
// so the authoritative review policy and requester are stable through
@@ -2472,6 +2475,7 @@ export function issueThreadInteractionService(
const now = new Date();
const updated = await db.transaction(async (tx) => {
await assertInteractionRunWriteAllowed(tx as unknown as Db, args.issue, args.actor);
if (args.current.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.current);
await args.mutationOptions?.beforeResolveInTransaction?.(tx);
const issueContext = await tx
.select({
@@ -3290,6 +3294,7 @@ export function issueThreadInteractionService(
const cancelled: IssueThreadInteraction[] = [];
for (const row of rows) {
const updated = await db.transaction(async (tx) => {
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
if (row.kind === "request_confirmation") await notifyToolProjectionForInteractions(tx, row.companyId, [row.id]);
const [resolved] = await tx
.update(issueThreadInteractions)
@@ -3375,6 +3380,7 @@ export function issueThreadInteractionService(
const expired: IssueThreadInteraction[] = [];
for (const { row, replacementInteractionId } of supersededRows) {
const updated = await db.transaction(async (tx) => {
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
if (row.kind === "request_confirmation") await notifyToolProjectionForInteractions(tx, row.companyId, [row.id]);
const [updatedRow] = await tx
.update(issueThreadInteractions)
@@ -3736,6 +3742,9 @@ export function issueThreadInteractionService(
eq(chatVoiceSessions.callerAuthority, "guest_intake"))).limit(1);
if (guest) throw unprocessable("Ask this unverified caller a clarification in a task comment; submit_request will deliver their spoken follow-up. Protected human-input questions require authenticated access.", {code: "voice_guest_use_task_comment"});
}
if (data.kind === "ask_user_questions" || data.kind === "request_confirmation") {
await notifyChatRunMilestones(tx, { companyId: issue.companyId, issueId: issue.id });
}
const [row] = await tx
.insert(issueThreadInteractions)
.values({
@@ -4459,6 +4468,7 @@ export function issueThreadInteractionService(
const expired: IssueThreadInteraction[] = [];
for (const row of superseded) {
const updated = await db.transaction(async (tx) => {
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
if (row.kind === "connection_intent") {
await tx
.delete(toolOauthStates)
@@ -4595,6 +4605,7 @@ export function issueThreadInteractionService(
);
if (!sampleQuestionRow) continue;
const updatedInteractions = await db.transaction(async (tx) => {
await notifyChatRunMilestones(tx, sampleQuestionRow);
const updatedRows = await tx
.update(issueThreadInteractions)
.set({
@@ -4633,6 +4644,7 @@ export function issueThreadInteractionService(
);
if (!sampleConfirmationRow) continue;
const updatedInteractions = await db.transaction(async (tx) => {
if (commentRows.some(row => row.kind === "request_confirmation")) await notifyChatRunMilestones(tx, sampleConfirmationRow);
await notifyToolProjectionForInteractions(tx, issue.companyId, confirmationRowIds);
const updatedRows = await tx
.update(issueThreadInteractions)
@@ -4778,6 +4790,7 @@ export function issueThreadInteractionService(
document,
});
const updated = await db.transaction(async (tx) => {
if (row.kind === "request_confirmation") await notifyChatRunMilestones(tx, row);
const [resolved] = await tx
.update(issueThreadInteractions)
.set({
@@ -4898,6 +4911,7 @@ export function issueThreadInteractionService(
// lifecycle reflection.
const updated = await db
.transaction(async (tx) => {
if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row);
if (row.kind === "connection_intent") {
await tx
.delete(toolOauthStates)
@@ -4988,6 +5002,7 @@ export function issueThreadInteractionService(
// request must not outlive a withdrawn card.
const updated = await db.transaction(async (tx) => {
await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor);
if (["ask_user_questions", "request_confirmation"].includes(current.kind)) await notifyChatRunMilestones(tx, current);
await resolveLinkedToolActionRequests(tx, current, {
status: "cancelled",
fromStatuses: ["pending", "approved"],
@@ -5123,6 +5138,7 @@ export function issueThreadInteractionService(
if (!historicalAnswer) throw interactionIssueClosedError();
}
await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor);
await notifyChatRunMilestones(tx, current);
await mutationOptions.beforeResolveInTransaction?.(tx);
const resolvedAt = new Date();
const [row] = await tx
@@ -5247,6 +5263,7 @@ export function issueThreadInteractionService(
const now = new Date();
const updated = await db.transaction(async (tx) => {
await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor);
if (["ask_user_questions", "request_confirmation"].includes(current.kind)) await notifyChatRunMilestones(tx, current);
await resolveLinkedToolActionRequests(tx, current, {
status: "cancelled",
fromStatuses: ["pending", "approved"],
@@ -5344,6 +5361,7 @@ export function issueThreadInteractionService(
const reason = data.reason?.trim() || null;
const updated = await db.transaction(async (tx) => {
await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor);
await notifyChatRunMilestones(tx, current);
const resolvedAt = new Date();
const [row] = await tx
.update(issueThreadInteractions)
+4
View File
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { notifyApprovalWork } from "./approval-work-notifications.js";
import { notifyDecisionTargetWork, notifyDecisionTargetIssueRemoval } from "./decision-targets.js";
import { assertOnboardingFirstTaskCreationAllowed } from "./onboarding-first-task-admission.js";
@@ -11413,6 +11414,9 @@ export function issueService(db: Db) {
if (patch.status === "done" || patch.status === "cancelled") {
await notifyDecisionTargetWork(tx, `issue:${id}`);
}
if (patch.hiddenAt === null && receiptExisting.hiddenAt !== null) {
await notifyChatRunMilestones(tx, { companyId: receiptExisting.companyId, issueId: id });
}
if (patch.hiddenAt !== undefined || patch.harnessKind !== undefined || patch.conversationAgentId !== undefined) {
await notifyApprovalWork(tx, receiptExisting.companyId);
}
@@ -15,6 +15,7 @@ import {
type ActivityPublication,
} from "../activity-log.js";
import { enqueueTerminalIssueInteractionChatPublications } from "../chat-interaction-publications.js";
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { issueThreadInteractionService } from "../issue-thread-interactions.js";
import { logger } from "../../middleware/logger.js";
@@ -126,6 +127,7 @@ export async function dismissAutomaticCompletionReviews(
)
.for("update");
if (!issue) return;
await notifyChatRunMilestones(tx, interaction);
const now = new Date();
const [retired] = await tx
.update(issueThreadInteractions)
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { and, eq, inArray, ne, sql } from "drizzle-orm";
import {
agents,
@@ -573,20 +574,28 @@ export async function restoreNativeChatReviewPresentationInTransaction(
assessmentId: input.assessmentId,
externalChatReviewPresentation: { ...proof, decisionId: input.decisionId },
};
const runOwner = and(
eq(heartbeatRuns.id, input.runId),
eq(heartbeatRuns.companyId, input.companyId),
eq(heartbeatRuns.nativeIssueId, input.issueId),
eq(heartbeatRuns.runtimeMode, "native"),
);
const projectedResult = sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify(projection)}::jsonb`;
const [run] = await db.select({
needsUpdate: sql<boolean>`${projectedResult} is distinct from ${heartbeatRuns.resultJson}`,
}).from(heartbeatRuns).where(runOwner).for("update");
if (!run) return false;
// A valid existing projection still permits the caller to materialize its
// response. Replaying recovery must not rewrite it or signal another scan.
if (!run.needsUpdate) return true;
await notifyChatRunMilestones(db, input);
const [updated] = await db
.update(heartbeatRuns)
.set({
resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify(projection)}::jsonb`,
resultJson: projectedResult,
updatedAt: new Date(),
})
.where(
and(
eq(heartbeatRuns.id, input.runId),
eq(heartbeatRuns.companyId, input.companyId),
eq(heartbeatRuns.nativeIssueId, input.issueId),
eq(heartbeatRuns.runtimeMode, "native"),
),
)
.where(runOwner)
.returning({ id: heartbeatRuns.id });
return Boolean(updated);
}
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { withExternalAdmissionGuard, assertNoExternalOverlap } from "../../modules/external-agents/index.js";
import { agentWakeupRequests, heartbeatRuns, issues, type Db } from "@paperclipai/db";
import { and, eq } from "drizzle-orm";
@@ -40,6 +41,7 @@ export async function claimQueuedNativeReviewRun(db: Db, input: {
contextSnapshot: context, agentNameKey: input.agentNameKey, claimedAt: input.claimedAt,
});
if (!locked) return null;
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId });
const [claimed] = await tx.update(heartbeatRuns).set({
...input.claimValues, status: "running", startedAt: run.startedAt ?? input.claimedAt,
updatedAt: input.claimedAt,
@@ -1,4 +1,5 @@
import { pendingNativeGovernance } from "./native-governance.js";
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { onboardingFirstTaskCompletionAllowed } from "../onboarding-first-task-admission.js";
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
import { reconcileMuseNativeTerminalRun } from "./muse-terminal-reconciliation.js";
@@ -377,6 +378,7 @@ async function recordRetryableFailure(input: {
: input.failureCode;
if (phase === "terminal_failure") await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, input.run.companyId);
await notifyExecutionRunWork(tx);
await notifyChatRunMilestones(tx, { companyId: input.run.companyId, issueId: input.coordinator.issueId });
await tx
.update(nativeRunFinalizations)
.set({
@@ -666,6 +668,7 @@ async function projectCommittedRun(input: {
const tx = transaction as unknown as Db;
await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, input.run.companyId);
await notifyExecutionRunWork(tx);
await notifyChatRunMilestones(tx, { companyId: input.run.companyId, issueId: input.coordinator.issueId });
return tx
.update(heartbeatRuns)
.set({
@@ -965,6 +968,7 @@ export async function repairCommittedNativeChatResponse(
const authorizationReason = await resolveChatRunPresentationAuthorizationReason(tx, input);
const resolved = resolveHeartbeatRunResponse({ resultJson: monitorResult });
if (!resolved.text) return false;
await notifyChatRunMilestones(tx, input);
const comment = await issueService(db).addComment(input.issueId, resolved.text,
{ agentId: run.agentId, runId: run.id }, { authorizationReason, completionReply: true }, tx);
await tx.update(heartbeatRuns).set({
@@ -1019,6 +1023,7 @@ export async function repairCommittedNativeChatResponse(
finalAgentMessage,
});
if (!resolved.text || resolved.decision.commentAction !== "create") return false;
await notifyChatRunMilestones(tx, input);
const comment = await issueService(db).addComment(input.issueId, resolved.text,
{ agentId: run.agentId, runId: run.id },
{ authorizationReason: "internal_agent_write", completionReply: true }, tx);
@@ -1062,6 +1067,7 @@ export async function repairCommittedNativeChatResponse(
});
if (!resolved.text || resolved.decision.commentAction !== "create")
return false;
await notifyChatRunMilestones(tx, input);
const comment = await issueService(db).addComment(
input.issueId,
resolved.text,
@@ -1678,6 +1684,7 @@ export async function finalizeNativeRun(input: {
const tx = transaction as unknown as Db;
if (input.projectRunStatus) await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, run.companyId);
await notifyExecutionRunWork(tx);
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId: coordinator.issueId });
return projectRun(tx);
});
if (
@@ -8,6 +8,7 @@ import { logger } from "../../middleware/logger.js";
import { issueService } from "../issues.js";
import { issueThreadInteractionService } from "../issue-thread-interactions.js";
import { enqueueTerminalIssueInteractionChatPublications } from "../chat-interaction-publications.js";
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { persistActivity, publishActivity, type ActivityPublication } from "../activity-log.js";
const obsoletePrompt = "Review the superseding native policy assessment.";
@@ -68,6 +69,7 @@ export async function dismissObsoleteNativePolicyReviews(db: Db, runIds?: string
eq(issues.id, decision.issueId), eq(issues.companyId, decision.companyId),
)).for("update").then((rows) => rows[0]);
if (!issue) return;
await notifyChatRunMilestones(tx, interaction);
const now = new Date();
const [cancelled] = await tx.update(issueThreadInteractions).set({
status: "cancelled",
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { scheduledIssueMonitorWait } from "../issue-monitors.js";
import { onboardingFirstTaskCompletionAllowed } from "../onboarding-first-task-admission.js";
import { hasPublishedGitHubReview, readInheritedGitHubReviewWait } from "../chat-channels/github/review-wait.js";
@@ -127,6 +128,7 @@ export async function dispatchPendingNativeStatusEffects(input: {
targetId: effect.targetId,
});
}
await notifyChatRunMilestones(tx, input);
await tx
.update(statusDecisionEffects)
.set({
@@ -1973,6 +1975,7 @@ export async function commitNativeStatusDecision(input: {
await recordCoordinatorDecision(decisionRow.id);
return { decision: decisionRow, issue, replayed: true };
}
await notifyChatRunMilestones(tx, input);
if (!decisionRow) {
const latestDecisionVersion = await tx
.select({
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { and, asc, desc, eq, inArray, isNull, lte, or, sql } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
@@ -454,6 +455,9 @@ export function questionResponseDeliveryService(
current.lastAttemptAt.getTime() > claimAt.getTime() - claimStaleMs
)
return null;
if (current.status !== "delivering") {
await notifyChatRunMilestones(tx, { companyId: current.companyId, issueId: current.issueId });
}
return tx
.update(issueQuestionResponseDeliveries)
.set({
@@ -504,6 +508,7 @@ export function questionResponseDeliveryService(
}): Promise<QuestionResponseDeliveryOutcome> {
const at = now();
const updated = await db.transaction(async (tx) => {
await notifyChatRunMilestones(tx, { companyId: input.delivery.companyId, issueId: input.delivery.issueId });
const row = await tx
.update(issueQuestionResponseDeliveries)
.set({
@@ -591,25 +596,28 @@ export function questionResponseDeliveryService(
const nextErrorCount = delivery.errorCount + (options.bounded ? 1 : 0);
const exhausted =
options.bounded && nextErrorCount >= MAX_DELIVERY_ATTEMPTS;
await db
.update(issueQuestionResponseDeliveries)
.set({
// Keep an exhausted claim owned until recordTerminal commits its outcome.
status: exhausted ? "delivering" : "pending",
...(options.bounded ? { errorCount: nextErrorCount } : {}),
lastErrorCode: errorCode,
updatedAt: at,
})
.where(
and(
eq(issueQuestionResponseDeliveries.id, delivery.id),
eq(issueQuestionResponseDeliveries.status, "delivering"),
eq(
issueQuestionResponseDeliveries.attemptCount,
delivery.attemptCount,
),
),
await db.transaction(async (tx) => {
const ownedClaim = and(
eq(issueQuestionResponseDeliveries.id, delivery.id),
eq(issueQuestionResponseDeliveries.status, "delivering"),
eq(issueQuestionResponseDeliveries.attemptCount, delivery.attemptCount),
);
const [current] = await tx.select({ id: issueQuestionResponseDeliveries.id })
.from(issueQuestionResponseDeliveries).where(ownedClaim).for("update");
if (!current) return;
if (!exhausted) {
await notifyChatRunMilestones(tx, { companyId: delivery.companyId, issueId: delivery.issueId });
}
await tx.update(issueQuestionResponseDeliveries)
.set({
// Keep an exhausted claim owned until recordTerminal commits its outcome.
status: exhausted ? "delivering" : "pending",
...(options.bounded ? { errorCount: nextErrorCount } : {}),
lastErrorCode: errorCode,
updatedAt: at,
})
.where(ownedClaim);
});
return exhausted;
}
@@ -1301,21 +1309,32 @@ export function questionResponseDeliveryService(
async function sweepPending(limit = 50) {
const sweepAt = now();
const staleAt = new Date(sweepAt.getTime() - claimStaleMs);
await db
.update(issueQuestionResponseDeliveries)
.set({
status: "pending",
updatedAt: sweepAt,
})
.where(
and(
eq(issueQuestionResponseDeliveries.status, "delivering"),
or(
isNull(issueQuestionResponseDeliveries.lastAttemptAt),
lte(issueQuestionResponseDeliveries.lastAttemptAt, staleAt),
),
await db.transaction(async (tx) => {
const staleClaims = await tx.select({
id: issueQuestionResponseDeliveries.id,
companyId: issueQuestionResponseDeliveries.companyId,
issueId: issueQuestionResponseDeliveries.issueId,
}).from(issueQuestionResponseDeliveries).where(and(
eq(issueQuestionResponseDeliveries.status, "delivering"),
or(
isNull(issueQuestionResponseDeliveries.lastAttemptAt),
lte(issueQuestionResponseDeliveries.lastAttemptAt, staleAt),
),
);
)).orderBy(asc(issueQuestionResponseDeliveries.id)).for("update");
if (staleClaims.length === 0) return;
const notified = new Set<string>();
for (const claim of staleClaims) {
const scope = `${claim.companyId}:${claim.issueId}`;
if (notified.has(scope)) continue;
await notifyChatRunMilestones(tx, claim);
notified.add(scope);
}
for (let offset = 0; offset < staleClaims.length; offset += 500) {
await tx.update(issueQuestionResponseDeliveries)
.set({ status: "pending", updatedAt: sweepAt })
.where(inArray(issueQuestionResponseDeliveries.id, staleClaims.slice(offset, offset + 500).map(claim => claim.id)));
}
});
const ids = await db
.select({ interactionId: issueQuestionResponseDeliveries.interactionId })
.from(issueQuestionResponseDeliveries)
+92 -72
View File
@@ -1,3 +1,5 @@
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
import { boundRunIssueId } from "../run-task-binding.js";
import { recoveryRunContextColumn, recoveryRunVersionColumn } from "./run-context-read.js";
import { recoveryRunResultColumn } from "./run-result-read.js";
import { readLegacyDispositionSource } from "./legacy-disposition-read.js";
@@ -4340,6 +4342,7 @@ export function recoveryService(
}
async function persistAdapterFailureRecoveryClassification(
issue: Pick<typeof issues.$inferSelect, "id" | "companyId">,
latestRun: NonNullable<LatestIssueRun>,
classification: NonNullable<AdapterFailureRecoveryClassification>,
): Promise<NonNullable<LatestIssueRun>> {
@@ -4351,17 +4354,20 @@ export function recoveryService(
const classificationMetadata = withAdapterFailureRecoveryClassification(
{ ...latestRun, resultJson: {} }, classification,
).resultJson;
const [updated] = await db
.update(heartbeatRuns)
.set({
errorCode: classifiedRun.errorCode,
// Classification owns retry metadata, not workspace repair receipts
// that may have committed after this run was selected.
resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify(classificationMetadata)}::jsonb`,
updatedAt: new Date(),
})
.where(eq(heartbeatRuns.id, latestRun.id))
.returning({ resultJson: heartbeatRuns.resultJson, errorCode: heartbeatRuns.errorCode });
const [updated] = await db.transaction(async tx => {
await notifyChatRunMilestones(tx, { companyId: issue.companyId, issueId: issue.id });
return tx
.update(heartbeatRuns)
.set({
errorCode: classifiedRun.errorCode,
// Classification owns retry metadata, not workspace repair receipts
// that may have committed after this run was selected.
resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify(classificationMetadata)}::jsonb`,
updatedAt: new Date(),
})
.where(and(eq(heartbeatRuns.id, latestRun.id), eq(heartbeatRuns.companyId, issue.companyId)))
.returning({ resultJson: heartbeatRuns.resultJson, errorCode: heartbeatRuns.errorCode });
});
return updated ? { ...classifiedRun, ...updated } : classifiedRun;
}
@@ -4960,6 +4966,7 @@ export function recoveryService(
});
if (monitored) {
latestRun = await persistAdapterFailureRecoveryClassification(
issue,
latestRun,
adapterFailureClassification,
);
@@ -4981,6 +4988,7 @@ export function recoveryService(
});
if (updated) {
latestRun = await persistAdapterFailureRecoveryClassification(
issue,
latestRun,
adapterFailureClassification,
);
@@ -5184,6 +5192,7 @@ export function recoveryService(
});
if (monitored) {
latestRun = await persistAdapterFailureRecoveryClassification(
issue,
participantLatestRun,
participantAdapterFailureClassification,
);
@@ -5210,6 +5219,7 @@ export function recoveryService(
});
if (updated) {
latestRun = await persistAdapterFailureRecoveryClassification(
issue,
participantLatestRun,
participantAdapterFailureClassification,
);
@@ -6179,70 +6189,80 @@ export function recoveryService(
await deps.beforeOrphanedRunTerminalWrite?.(run.id);
const now = new Date();
const updated = await db
.update(heartbeatRuns)
.set({
status: terminalStatus,
finishedAt: run.finishedAt ?? now,
error: run.error ?? (terminalStatus === "interrupted" ? message : null),
errorCode:
run.errorCode ??
(terminalStatus === "interrupted" ? errorCode : null),
updatedAt: now,
})
.where(
and(
eq(heartbeatRuns.id, run.id),
eq(heartbeatRuns.status, "running"),
eq(heartbeatRuns.runtimeMode, run.runtimeMode),
nativeRunnerOwnershipNotHeldCondition(),
// Provider exit is expected while the native coordinator resumes a
// session or copies its completed workspace back. The coordinator
// owns those retries, including expired leases and future attempts.
// Check at the write so a newly recorded result cannot be orphaned
// using the earlier liveness snapshot. Terminal issue status remains
// the stronger authority.
!issueTerminalStatus && run.runtimeMode === "native"
? notExists(db.select({ runId: nativeRunFinalizations.runId })
.from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.runId, heartbeatRuns.id),
eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId),
or(
and(
isNotNull(nativeRunFinalizations.resultId),
inArray(nativeRunFinalizations.phase, [
"observed", "workspace_finalizing", "ready_for_assessment",
"arbitrating", "retryable_failure",
]),
),
and(
isNull(nativeRunFinalizations.resultId),
or(
eq(nativeRunFinalizations.phase, "retryable_failure"),
and(eq(nativeRunFinalizations.phase, "observed"), gt(nativeRunFinalizations.attempt, 0)),
),
const updated = await db.transaction(async tx => {
const eligible = and(
eq(heartbeatRuns.id, run.id),
eq(heartbeatRuns.status, "running"),
eq(heartbeatRuns.runtimeMode, run.runtimeMode),
nativeRunnerOwnershipNotHeldCondition(),
// Provider exit is expected while the native coordinator resumes a
// session or copies its completed workspace back. The coordinator
// owns those retries, including expired leases and future attempts.
// Check at the write so a newly recorded result cannot be orphaned
// using the earlier liveness snapshot. Terminal issue status remains
// the stronger authority.
!issueTerminalStatus && run.runtimeMode === "native"
? notExists(tx.select({ runId: nativeRunFinalizations.runId })
.from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.runId, heartbeatRuns.id),
eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId),
or(
and(
isNotNull(nativeRunFinalizations.resultId),
inArray(nativeRunFinalizations.phase, [
"observed", "workspace_finalizing", "ready_for_assessment",
"arbitrating", "retryable_failure",
]),
),
and(
isNull(nativeRunFinalizations.resultId),
or(
eq(nativeRunFinalizations.phase, "retryable_failure"),
and(eq(nativeRunFinalizations.phase, "observed"), gt(nativeRunFinalizations.attempt, 0)),
),
),
)))
: undefined,
// Recheck ownership in the write: a controller can renew or claim
// the run after the liveness read. An old snapshot cannot end a new
// controller's run, even if that controller's lease later expires.
run.runtimeMode === "legacy"
? and(
run.controllerBootId
? eq(heartbeatRuns.controllerBootId, run.controllerBootId)
: isNull(heartbeatRuns.controllerBootId),
or(
isNull(heartbeatRuns.controllerBootId),
sql`${heartbeatRuns.controllerLeaseExpiresAt} <= clock_timestamp()`,
),
)
: undefined,
),
)
.returning()
.then((rows) => rows[0] ?? null);
)))
: undefined,
// Recheck ownership in the write: a controller can renew or claim
// the run after the liveness read. An old snapshot cannot end a new
// controller's run, even if that controller's lease later expires.
run.runtimeMode === "legacy"
? and(
run.controllerBootId
? eq(heartbeatRuns.controllerBootId, run.controllerBootId)
: isNull(heartbeatRuns.controllerBootId),
or(
isNull(heartbeatRuns.controllerBootId),
sql`${heartbeatRuns.controllerLeaseExpiresAt} <= clock_timestamp()`,
),
)
: undefined,
);
// Do not wake projection for a retained native finalization or a run
// whose execution owner changed. Keep the same predicate on the write
// to recheck finalization records committed after this lock was taken.
const [candidate] = await tx.select().from(heartbeatRuns).where(eligible).for("update");
if (!candidate) return null;
const boundIssueId = boundRunIssueId(candidate);
if (boundIssueId) {
await notifyChatRunMilestones(tx, { companyId: run.companyId, issueId: boundIssueId });
}
return tx
.update(heartbeatRuns)
.set({
status: terminalStatus,
finishedAt: run.finishedAt ?? now,
error: run.error ?? (terminalStatus === "interrupted" ? message : null),
errorCode:
run.errorCode ??
(terminalStatus === "interrupted" ? errorCode : null),
updatedAt: now,
})
.where(eligible)
.returning()
.then((rows) => rows[0] ?? null);
});
if (!updated) {
// Another path finalized the run between the read and this write. Keep
// that terminal outcome authoritative.
+10 -1
View File
@@ -11,7 +11,8 @@ import {
type Db,
} from "@paperclipai/db";
import { conflict, forbidden } from "../errors.js";
import { boundRunIssueId, runTaskContextColumn } from "./run-task-binding.js";
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { boundRunIssueId, boundRunIssueIdSql, runTaskContextColumn } from "./run-task-binding.js";
import { isUuidLike } from "@paperclipai/shared";
import { queuedCommentIdsFromRunContext, queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js";
@@ -138,6 +139,13 @@ async function append(
}
async function activate(executor: Executor, context: RunIdentityContext) {
const [binding] = await executor.select({ issueId: boundRunIssueIdSql() })
.from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, context.runId), eq(heartbeatRuns.companyId, context.companyId),
));
if (binding?.issueId) {
await notifyChatRunMilestones(executor, { companyId: context.companyId, issueId: binding.issueId });
}
const [run] = await executor
.update(heartbeatRuns)
.set({
@@ -447,6 +455,7 @@ export async function rejectSteeredIdentity(
);
}
/** Accept within the caller's task/run transaction; standalone recovery uses reconcileSteeredIdentity. */
export async function acceptSteeredIdentity(
executor: Executor,
context: RunIdentityContext,
+2
View File
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
import { updateAgentConfigurationInTransaction } from "./agent-configuration-transaction.js";
import { withAgentAppearance } from "@paperclipai/shared";
@@ -290,6 +291,7 @@ export function createSecretProposalsService(db: Db) {
: outcome === "rejected"
? "rejected"
: "withdrawn";
if (current.status !== status) await notifyChatRunMilestones(txDb, current);
await txDb
.update(issueThreadInteractions)
.set({
+1 -1
View File
@@ -4665,7 +4665,7 @@ export function toolAccessService(
.returning();
const [row] = existing.transport === "chat_sdk"
? await db.transaction(async tx => {
await notifyChatEndpointWork(tx, existing.id);
await notifyChatEndpointWork(tx, existing.id, existing.companyId);
const rows = await write(tx as unknown as Db);
// Credential resolution must never observe the binding rebuild's
// delete/insert gap after a configuration wake.
@@ -541,7 +541,7 @@ async function removeConnection(
.returning({ id: connectionTokenIssuances.id });
const archived = await db.transaction(async (tx) => {
if (connection.transport === "chat_sdk") await notifyChatEndpointWork(tx, connection.id);
if (connection.transport === "chat_sdk") await notifyChatEndpointWork(tx, connection.id, connection.companyId);
if (connection.transport === "voice") await notifyVoiceChange(tx, { kind: "connection", id: connection.id });
await notifyToolConnectionHealth(tx, connection);
await tx.select({ id: toolApplications.id }).from(toolApplications)
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { notifyToolReviewChange } from "./tool-review-work-notifications.js";
import { notifyDeliveryWork, DELIVERY_QUEUES } from "./delivery-work-notifications.js";
import { and, eq } from "drizzle-orm";
@@ -141,6 +142,7 @@ export async function commitToolActionReview(
throw conflict("This review has already been resolved");
}
const now = new Date();
if (interaction && issue) await notifyChatRunMilestones(tx, interaction);
await notifyToolReviewChange(tx, current.companyId);
const [updated] = await tx
.update(toolActionRequests)
+4
View File
@@ -1,3 +1,4 @@
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
import { writeToolConnectionHealthChange } from "./tool-connection-health-notifications.js";
import { writeApprovalChange } from "./approval-work-notifications.js";
import { toolReviewProjectionService } from "./tool-review-projection.js";
@@ -2408,6 +2409,9 @@ export function createToolGatewayService(
&& projection.resultSummary === (input.resultSummary ?? null)) return false;
const now = new Date();
if (linked.issueId && interaction.status === "pending" && ["expired", "cancelled"].includes(input.status)) {
await notifyChatRunMilestones(tx, { companyId: linked.companyId, issueId: linked.issueId });
}
await tx
.update(issueThreadInteractions)
.set({