mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-29 21:07:40 +01:00
fix(server): wake chat lifecycle milestones from committed changes (#17606)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Connected chats show safe notices when a run starts, waits, finishes, or fails. > - A recurring scan projected these notices even when no eligible work existed. > - Eligibility depends on committed run state, conversation lineage, and current authority. > - Existing database rows can reconstruct that work after restart. > - This PR wakes a scoped worker from those writes and removes lifecycle polling. > - Native progress events keep their separate existing owner until the next migration. ## Linked Issues or Issue Description **What happened?** The chat maintenance timer repeatedly scanned run lifecycle state. Empty and settled instances still queried the database. A run-status event alone cannot replace the scan: prompt receipts, continuation receipts, identity changes, and restored authority can make a run eligible later. **What did you expect?** Committed changes wake projection. Startup reconstructs eligible work. A completed pass stops until another relevant change arrives. Failed work retains recovery without scanning settled history forever. **Steps to reproduce** Configure an automatic chat endpoint. Finish its work, or leave it unused. Observe recurring lifecycle projection queries from the chat maintenance callback. Stacked on #17592. Related foundation: #15625. This PR owns lifecycle milestones only. ## What Changed - Preserve upstream coalesced Cloud membership writes together with locked membership restoration and its committed chat notification. - Signal milestone eligibility from the real Slack account-link transaction. - Register the lifecycle worker with the existing delivery coordinator. Remove lifecycle projection from the periodic chat callback. Keep native progress in a separately named lane and preserve the explicit combined API. - Reconstruct company/issue scopes from the existing selector. Drain bounded batches with company fairness. Retry failures, retain writes through drain/resume, and disarm settled work. - Signal from supported run, presentation, interaction, receipt, lineage, conversation, recovery, and authority writers. Preserve their transaction ownership, including nested savepoints and uncertain settlement. No new table is added. - Preserve provider authorization, authored-final suppression, responsible-user continuity, deduplication, the 24-hour selection window, and raw-output restrictions. Keep provider interpretation in its owner. - Preserve PostgreSQL timestamp precision in lifecycle pagination. Rejected submillisecond candidates must not repeat the same page and starve later notices. - Document producer coverage and alternate-writer dispositions in `doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md`. ## Verification Published head: `1d056f68c5b1005933b9e12de8745bf1c4cbeeca`, targeting master after #17592 merged at `7d75ae0569e41786640cf09647ae8849411d2f22`. A normal merge integrates the actual landed parent. The entire 77-path milestone patch is binary-identical to the locally verified preparation across that ancestry reconciliation; both unrelated master paths are preserved. The four earlier producer/test integration conflicts retain scoped milestone and voice notices, current onboarding completion admission, and expanded identity query/read-scope coverage. - **330 distinct selected cases pass on the verified unchanged patch:** 305 across 14 complete suites (all five milestone worker suites, run identity, board claim, Cloud provisioning, auth session routes, Cloud actor authentication, stale-lock recovery, GitHub credentials, project Git authorization, and app admission wiring), plus 25 selected real chat producer cases. The remaining 1,590 chat integration cases were skipped, not passed. - The real producer cases cover saved milestone discovery, GitHub and Slack identity changes, account-link rollback/reconfirmation, ordinary ingress reactivation, post-close action/delivery receipt settlement, all ten authority-restoration paths including Cloud synchronization, and changed native review metadata. - Worker coverage verifies outer commit, rollback, uncertain-write recovery, restart, bounded batching/fairness, timestamp precision, retries, drain/resume, and settled SQL quiet. Authorization and existing publication deduplication remain with their owning services. - Server production typechecking, build, and module-boundary checks pass. The tests above were executed separately. Production typechecking excludes `src/__tests__` and does not replace those runs. Each database owner uses its own newly created and app-migrated PostgreSQL fixture. - [Current-parent acceptance and preserved failure ledger](https://github.com/paperclipai/paperclip/blob/1d056f68c5b1005933b9e12de8745bf1c4cbeeca/doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md#current-parent-integration-and-acceptance-october-28) records commands, counts, producer scope, conflict dispositions, and limits. Earlier combined-stack counts and failures keep their original revision scope; local acceptance does not erase old CI failures. - No new migration, queue table, dependency, workflow, or shared scheduler contract is introduced. Parent migration 0397, bounded consent recovery, voice cleanup/retention repairs, and runner checkpoint restoration remain present. - **Exact-head Greptile is 5/5 and all 55 current checks are clean.** No Ponytail review is present, which is an explicitly authorized nonblocker; no failing review is waived. Older reviews and parent checks are not current passes. Actual failing Ponytail findings must be fixed; an absent/stopped/paused turn without a verdict is an authorized nonblocker. Active unfinished Ponytail uses its original request/start timer, with a 25-minute monitor cutoff respecting the user's 30-minute maximum. - No complete local repository matrix, live provider tenant, external alarm, physical host/database suspension, or fleet savings claim is made. Native progress remains a separate following slice. ## Risks - A future eligibility writer must notify the existing projection boundary. Independent SQL writers or another active server process are outside this process-local signal contract. - Authority and lineage remain sensitive. The worker uses the existing selectors and publication policy; it does not grant execution or disclosure authority. - Native progress and other publication maintenance still have recurring work. This PR removes one reason, not the entire chat maintenance timer. - Worker quiet does not prove database suspension, host sleep, or external alarm coverage. Cloud sleep blockers remain unchanged. - Merge eligibility requires passing current-head CI and Greptile 5/5 after this actual-master reconciliation. Subsequent stack slices retain their own gates. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window size are not exposed. Used repository inspection, reasoning, shell execution, PostgreSQL tests, browser E2Es and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details available in this session) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run the documented focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
143d20e466
commit
5f0da562db
77 files changed
+5002
-501
No files matched your search
@@ -25,7 +25,7 @@ describe("createChatReconciliationCoordinator", () => {
|
||||
processFailedGitHubWebhookDeliveries,
|
||||
processPendingDeliveries,
|
||||
flushPublications,
|
||||
projectRunMilestones: async () => 0,
|
||||
projectNativeProgress: async () => 0,
|
||||
processPendingSlackFileUploadReceipts: async () => undefined,
|
||||
processPendingSlackSessionSyncs: async () => undefined,
|
||||
onError,
|
||||
@@ -142,14 +142,14 @@ describe("createChatReconciliationCoordinator", () => {
|
||||
});
|
||||
|
||||
it.each([false, true])(
|
||||
"isolates a blocked milestone projector and joins its completion (rejects: %s)",
|
||||
"isolates a blocked native progress projector and joins its completion (rejects: %s)",
|
||||
async (rejects) => {
|
||||
let release!: () => void;
|
||||
const held = new Promise<void>((resolve) => {
|
||||
release = resolve;
|
||||
});
|
||||
const failure = new Error("milestone projection failed");
|
||||
const projectRunMilestones = vi
|
||||
const failure = new Error("native progress projection failed");
|
||||
const projectNativeProgress = vi
|
||||
.fn(async () => 0)
|
||||
.mockImplementationOnce(async () => {
|
||||
await held;
|
||||
@@ -159,7 +159,7 @@ describe("createChatReconciliationCoordinator", () => {
|
||||
const flushPublications = vi.fn(async () => undefined);
|
||||
const onError = vi.fn();
|
||||
const coordinator = createChatReconciliationCoordinator({
|
||||
projectRunMilestones,
|
||||
projectNativeProgress,
|
||||
processPendingDeliveries: async () => undefined,
|
||||
flushPublications,
|
||||
processPendingSlackFileUploadReceipts: async () => undefined,
|
||||
@@ -169,7 +169,7 @@ describe("createChatReconciliationCoordinator", () => {
|
||||
try {
|
||||
coordinator.reconcile();
|
||||
await vi.waitFor(() => {
|
||||
expect(projectRunMilestones).toHaveBeenCalledTimes(1);
|
||||
expect(projectNativeProgress).toHaveBeenCalledTimes(1);
|
||||
expect(flushPublications).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
// A newly committed final/question must dispatch even while a different
|
||||
@@ -179,7 +179,7 @@ describe("createChatReconciliationCoordinator", () => {
|
||||
await vi.waitFor(() =>
|
||||
expect(flushPublications).toHaveBeenCalledTimes(2),
|
||||
);
|
||||
expect(projectRunMilestones).toHaveBeenCalledTimes(1);
|
||||
expect(projectNativeProgress).toHaveBeenCalledTimes(1);
|
||||
let drained = false;
|
||||
const draining = coordinator.drain().then(() => {
|
||||
drained = true;
|
||||
@@ -188,17 +188,17 @@ describe("createChatReconciliationCoordinator", () => {
|
||||
expect(drained).toBe(false);
|
||||
release();
|
||||
await draining;
|
||||
expect(projectRunMilestones).toHaveBeenCalledTimes(2);
|
||||
expect(projectNativeProgress).toHaveBeenCalledTimes(2);
|
||||
expect(onError).toHaveBeenCalledTimes(rejects ? 1 : 0);
|
||||
if (rejects)
|
||||
expect(onError).toHaveBeenCalledWith("run milestones", failure);
|
||||
expect(onError).toHaveBeenCalledWith("native progress", failure);
|
||||
const completedCounts = [
|
||||
projectRunMilestones.mock.calls.length,
|
||||
projectNativeProgress.mock.calls.length,
|
||||
flushPublications.mock.calls.length,
|
||||
];
|
||||
await new Promise<void>((resolve) => setImmediate(resolve));
|
||||
expect([
|
||||
projectRunMilestones.mock.calls.length,
|
||||
projectNativeProgress.mock.calls.length,
|
||||
flushPublications.mock.calls.length,
|
||||
]).toEqual(completedCounts);
|
||||
} finally {
|
||||
@@ -209,18 +209,18 @@ describe("createChatReconciliationCoordinator", () => {
|
||||
},
|
||||
);
|
||||
|
||||
it("joins a blocked milestone projector at shutdown without starting dirty follow-ups", async () => {
|
||||
it("joins a blocked native progress projector at shutdown without starting dirty follow-ups", async () => {
|
||||
let release!: () => void;
|
||||
const held = new Promise<void>((resolve) => {
|
||||
release = resolve;
|
||||
});
|
||||
const projectRunMilestones = vi.fn(async () => {
|
||||
const projectNativeProgress = vi.fn(async () => {
|
||||
await held;
|
||||
return 1;
|
||||
});
|
||||
const flushPublications = vi.fn(async () => undefined);
|
||||
const coordinator = createChatReconciliationCoordinator({
|
||||
projectRunMilestones,
|
||||
projectNativeProgress,
|
||||
flushPublications,
|
||||
processPendingDeliveries: async () => undefined,
|
||||
processPendingSlackFileUploadReceipts: async () => undefined,
|
||||
@@ -230,7 +230,7 @@ describe("createChatReconciliationCoordinator", () => {
|
||||
try {
|
||||
coordinator.reconcile();
|
||||
await vi.waitFor(() => {
|
||||
expect(projectRunMilestones).toHaveBeenCalledOnce();
|
||||
expect(projectNativeProgress).toHaveBeenCalledOnce();
|
||||
expect(flushPublications).toHaveBeenCalledOnce();
|
||||
});
|
||||
coordinator.notifyPublications();
|
||||
@@ -243,7 +243,7 @@ describe("createChatReconciliationCoordinator", () => {
|
||||
expect(drained).toBe(false);
|
||||
release();
|
||||
await draining;
|
||||
expect(projectRunMilestones).toHaveBeenCalledOnce();
|
||||
expect(projectNativeProgress).toHaveBeenCalledOnce();
|
||||
expect(flushPublications).toHaveBeenCalledOnce();
|
||||
} finally {
|
||||
release();
|
||||
@@ -269,7 +269,7 @@ describe("createChatReconciliationCoordinator", () => {
|
||||
);
|
||||
const onError = vi.fn();
|
||||
const coordinator = createChatReconciliationCoordinator({
|
||||
projectRunMilestones: async () => 0,
|
||||
projectNativeProgress: async () => 0,
|
||||
processPendingDeliveries,
|
||||
flushPublications,
|
||||
processPendingSlackFileUploadReceipts,
|
||||
@@ -317,7 +317,7 @@ describe("createChatReconciliationCoordinator", () => {
|
||||
const processPendingSlackSessionSyncs = vi.fn(async () => undefined);
|
||||
const onError = vi.fn();
|
||||
const coordinator = createChatReconciliationCoordinator({
|
||||
projectRunMilestones: async () => 0,
|
||||
projectNativeProgress: async () => 0,
|
||||
processPendingDeliveries,
|
||||
flushPublications,
|
||||
processPendingSlackFileUploadReceipts,
|
||||
|
||||
@@ -10,12 +10,16 @@ import {
|
||||
import { errorHandler } from "../middleware/error-handler.js";
|
||||
import { assertCompanyAccess } from "../routes/authz.js";
|
||||
|
||||
function queryResult<T>(rows: T) {
|
||||
return Object.assign(Promise.resolve(rows), { for: () => Promise.resolve(rows) });
|
||||
}
|
||||
|
||||
function createSelectChain(rows: unknown[]) {
|
||||
return {
|
||||
from() {
|
||||
return {
|
||||
where() {
|
||||
return Promise.resolve(rows);
|
||||
return queryResult(rows);
|
||||
},
|
||||
};
|
||||
},
|
||||
@@ -102,6 +106,7 @@ describe("actorMiddleware authenticated session profile", () => {
|
||||
delete: vi.fn(() => ({ where: () => Promise.resolve(undefined) })),
|
||||
select: vi.fn(() => createSelectChain([])),
|
||||
} as any;
|
||||
db.transaction = vi.fn(async (run: (tx: typeof db) => Promise<unknown>) => run(db));
|
||||
const app = express();
|
||||
app.use(
|
||||
actorMiddleware(db, {
|
||||
@@ -176,7 +181,7 @@ describe("actorMiddleware authenticated session profile", () => {
|
||||
select: vi.fn(() => ({
|
||||
from: (table: unknown) => ({
|
||||
where: () =>
|
||||
Promise.resolve(
|
||||
queryResult(
|
||||
table === companyMemberships
|
||||
? [{ companyId: importedCompanyId, membershipRole: "member", status: "active" }]
|
||||
: [],
|
||||
@@ -186,6 +191,7 @@ describe("actorMiddleware authenticated session profile", () => {
|
||||
insert: vi.fn(() => insertChain),
|
||||
delete: vi.fn(() => ({ where: () => Promise.resolve(undefined) })),
|
||||
} as any;
|
||||
db.transaction = vi.fn(async (run: (tx: typeof db) => Promise<unknown>) => run(db));
|
||||
const app = express();
|
||||
app.use(
|
||||
actorMiddleware(db, {
|
||||
@@ -252,7 +258,7 @@ describe("actorMiddleware authenticated session profile", () => {
|
||||
select: vi.fn(() => ({
|
||||
from: (table: unknown) => ({
|
||||
where: () =>
|
||||
Promise.resolve(
|
||||
queryResult(
|
||||
table === companies
|
||||
? [{ name: "paperclip-stack-purple-rain" }]
|
||||
: [],
|
||||
@@ -280,7 +286,7 @@ describe("actorMiddleware authenticated session profile", () => {
|
||||
})),
|
||||
delete: vi.fn(() => ({ where: () => Promise.resolve(undefined) })),
|
||||
} as any;
|
||||
db.transaction = vi.fn(async (run: (tx: typeof db) => Promise<void>) => run(db));
|
||||
db.transaction = vi.fn(async (run: (tx: typeof db) => Promise<unknown>) => run(db));
|
||||
const app = express();
|
||||
app.use(
|
||||
actorMiddleware(db, {
|
||||
@@ -349,7 +355,7 @@ describe("actorMiddleware authenticated session profile", () => {
|
||||
select: vi.fn(() => ({
|
||||
from: (table: unknown) => ({
|
||||
where: () =>
|
||||
Promise.resolve(
|
||||
queryResult(
|
||||
table === instanceUserRoles && state.staleInstanceAdminRow ? [{ id: "stale-role-row" }] : [],
|
||||
),
|
||||
}),
|
||||
@@ -362,6 +368,7 @@ describe("actorMiddleware authenticated session profile", () => {
|
||||
},
|
||||
})),
|
||||
} as any;
|
||||
db.transaction = vi.fn(async (run: (tx: typeof db) => Promise<unknown>) => run(db));
|
||||
const app = express();
|
||||
app.use(
|
||||
actorMiddleware(db, {
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
|
||||
@@ -8,6 +10,7 @@ import {
|
||||
createDb,
|
||||
instanceUserRoles,
|
||||
principalPermissionGrants,
|
||||
subscribeDatabaseWork,
|
||||
} from "@paperclipai/db";
|
||||
import {
|
||||
claimBoardOwnership,
|
||||
@@ -20,7 +23,8 @@ import {
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_BOARD_CLAIM_TEST_DATABASE_URL");
|
||||
const embeddedPostgresSupport = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
describeEmbeddedPostgres("board claim", () => {
|
||||
@@ -28,8 +32,8 @@ describeEmbeddedPostgres("board claim", () => {
|
||||
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
|
||||
beforeAll(async () => {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-board-claim-");
|
||||
db = createDb(tempDb.connectionString);
|
||||
if (!external) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-board-claim-");
|
||||
db = createDb(external ?? tempDb!.connectionString);
|
||||
}, 20_000);
|
||||
|
||||
afterEach(async () => {
|
||||
@@ -42,10 +46,11 @@ describeEmbeddedPostgres("board claim", () => {
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await db?.$client.end({ timeout: 1 });
|
||||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
it("lets a signed-in user claim a local-board-only authenticated instance", async () => {
|
||||
it.each(["new", "suspended", "active"] as const)("claims a local-board-only instance with %s membership and wakes only restored authority", async prior => {
|
||||
const now = new Date();
|
||||
const userId = `claim-user-${randomUUID()}`;
|
||||
const company = await db
|
||||
@@ -70,6 +75,10 @@ describeEmbeddedPostgres("board claim", () => {
|
||||
role: "instance_admin",
|
||||
});
|
||||
|
||||
if (prior !== "new") await db.insert(companyMemberships).values({
|
||||
companyId: company.id, principalType: "user", principalId: userId,
|
||||
status: prior, membershipRole: "owner",
|
||||
});
|
||||
await initializeBoardClaimChallenge(db, { deploymentMode: "authenticated" });
|
||||
const warningUrl = getBoardClaimWarningUrl("127.0.0.1", 3197);
|
||||
expect(warningUrl).toBeTruthy();
|
||||
@@ -84,12 +93,19 @@ describeEmbeddedPostgres("board claim", () => {
|
||||
claimedByUserId: null,
|
||||
});
|
||||
|
||||
await expect(
|
||||
claimBoardOwnership(db, { token, code, userId }),
|
||||
).resolves.toEqual({
|
||||
status: "claimed",
|
||||
claimedByUserId: userId,
|
||||
});
|
||||
const events: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, event => events.push(event));
|
||||
try {
|
||||
await db.transaction(async tx => {
|
||||
await expect(claimBoardOwnership(tx as unknown as typeof db, { token, code, userId }))
|
||||
.resolves.toEqual({ status: "claimed", claimedByUserId: userId });
|
||||
expect(events).toEqual(prior === "active" ? [] : ["intent"]);
|
||||
// A nested claim must not publish a commit before its caller settles.
|
||||
const outside = await db.select().from(companyMemberships).where(eq(companyMemberships.companyId, company.id));
|
||||
expect(outside.map(row => row.status)).toEqual(prior === "new" ? [] : [prior]);
|
||||
});
|
||||
expect(events).toEqual(prior === "active" ? [] : ["intent", "settled"]);
|
||||
} finally { unsubscribe(); }
|
||||
|
||||
await expect(
|
||||
db
|
||||
|
||||
@@ -47,7 +47,7 @@ describe("chat action scheduling", () => {
|
||||
const work = s.start();
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
expect(s.service.processTelegramSubscriptionDiscovery).toHaveBeenCalledTimes(1);
|
||||
await s.owner.transaction(tx => notifyChatEndpointWork(tx, "connection-fixture"));
|
||||
await s.owner.transaction(tx => notifyChatEndpointWork(tx, "connection-fixture", "company-fixture"));
|
||||
expect(s.service.scheduleTelegramSubscriptionDiscovery).toHaveBeenCalledWith("connection-fixture");
|
||||
s.service.processTelegramSubscriptionDiscovery.mockRejectedValueOnce(new Error("discovery unavailable"));
|
||||
release();
|
||||
@@ -87,7 +87,7 @@ describe("chat action scheduling", () => {
|
||||
for (const run of s.runs) expect(run).toHaveBeenCalledTimes(2);
|
||||
// Queue claims/settlements do not rediscover every configured endpoint.
|
||||
expect(s.service.processTelegramSubscriptionDiscovery).toHaveBeenCalledTimes(1);
|
||||
await s.owner.transaction(tx => notifyChatEndpointWork(tx, "connection-fixture"));
|
||||
await s.owner.transaction(tx => notifyChatEndpointWork(tx, "connection-fixture", "company-fixture"));
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
for (const run of s.runs) expect(run).toHaveBeenCalledTimes(3);
|
||||
expect(s.service.processTelegramSubscriptionDiscovery).toHaveBeenCalledTimes(2);
|
||||
|
||||
@@ -1,3 +1,8 @@
|
||||
import { cloudActorHeaderSourceFromHeaders, resolveCloudTenantActor } from "../middleware/auth.js";
|
||||
import { cloudTenantPrimaryCompanyId } from "../services/cloud-instance.js";
|
||||
import { accessService } from "../services/access.js";
|
||||
import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db";
|
||||
import { restoreNativeChatReviewPresentationInTransaction } from "../services/native-runtime/native-chat-review-presentation.js";
|
||||
import { notifyChatEndpointConfiguration, notifyChatCompanyConfiguration } from "../services/chat-channels/configuration-work-notifications.js";
|
||||
import { deleteCompany } from "../services/company-deletion.js";
|
||||
import { adapterExecutionControls, createAdapterExecutionControl } from "../services/adapter-execution-control.js";
|
||||
@@ -23,6 +28,7 @@ import { chatSlackManagerGrants, chatSlackRegistrations, toolOauthStates, enviro
|
||||
import type { PaperclipCloudConnector, SealedConnectorCredentials } from "../services/paperclip-cloud-connector.js";
|
||||
import { buildSlackAppManifest, SLACK_MANAGER_SCOPES } from "@paperclipai/shared";
|
||||
import { registerChatActionWork, dueChatAction } from "../services/chat-action-work.js";
|
||||
import { chatRunMilestoneWork } from "../services/chat-run-milestone-work.js";
|
||||
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
||||
import { registerChatDeliveryWork } from "../services/chat-delivery-work.js";
|
||||
import { notifyChatPublicationWork, notifyChatActionWork, CHAT_ACTION_QUEUES } from "../services/chat-work-notifications.js";
|
||||
@@ -202,6 +208,8 @@ import {
|
||||
} from "../services/chat-interaction-publications.js";
|
||||
import {
|
||||
enqueueChatRunMilestones,
|
||||
enqueueChatRunLifecycleMilestones,
|
||||
enqueueSafeNativeChatProgress,
|
||||
githubRunReplyState,
|
||||
resolveChatRunPresentationAuthorizationReason,
|
||||
CHAT_RUN_PRESENTATION_AUTHORIZATION_REASON,
|
||||
@@ -1144,8 +1152,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
const companyPrefixAttempts = 8;
|
||||
const freshCompanyPrefix = () => `C${randomUUID().replaceAll("-", "").slice(0, 7).toUpperCase()}`;
|
||||
|
||||
async function seedCompany(nextIssuePrefix = freshCompanyPrefix) {
|
||||
const companyId = randomUUID();
|
||||
async function seedCompany(nextIssuePrefix = freshCompanyPrefix, companyId = randomUUID()) {
|
||||
const assignedAgentId = randomUUID();
|
||||
const replacementAgentId = randomUUID();
|
||||
let inserted = false;
|
||||
@@ -6029,6 +6036,68 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
.where(eq(chatEndpointLeases.endpointId, f.endpoint.id)),
|
||||
).toHaveLength(0);
|
||||
});
|
||||
it("wakes GitHub milestone identity restoration only after outer commit and stays quiet on reconfirmation", async () => {
|
||||
const f = await reviewBotFixture();
|
||||
await f.service.revokeLink(f.endpoint.id, f.principal.id);
|
||||
const verified = { githubUserId: "42", login: "octocat" };
|
||||
const events: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, (event, scopes) => {
|
||||
if ([...scopes ?? []].some(scope => JSON.parse(scope).companyId === f.companyId)) events.push(event);
|
||||
});
|
||||
const link = () => db.select().from(chatIdentityLinks).where(and(
|
||||
eq(chatIdentityLinks.endpointId, f.endpoint.id), eq(chatIdentityLinks.principalId, f.principal.id),
|
||||
)).then(rows => rows[0]);
|
||||
try {
|
||||
await expect(db.transaction(async tx => {
|
||||
await githubChatManagementService(tx as unknown as typeof db, f.providerFetch)
|
||||
.linkObservedIdentity(f.endpoint.id, "owner-user", verified);
|
||||
expect(events).toEqual(["intent"]);
|
||||
expect((await link()).status).toBe("revoked");
|
||||
throw new Error("rollback identity restoration");
|
||||
})).rejects.toThrow("rollback identity restoration");
|
||||
expect((await link()).status).toBe("revoked");
|
||||
events.length = 0;
|
||||
await db.transaction(async tx => {
|
||||
await githubChatManagementService(tx as unknown as typeof db, f.providerFetch)
|
||||
.linkObservedIdentity(f.endpoint.id, "owner-user", verified);
|
||||
expect(events).toEqual(["intent"]);
|
||||
expect((await link()).status).toBe("revoked");
|
||||
});
|
||||
expect(events).toEqual(["intent", "settled"]);
|
||||
expect((await link()).status).toBe("linked");
|
||||
events.length = 0;
|
||||
await f.management.linkObservedIdentity(f.endpoint.id, "owner-user", verified);
|
||||
expect(events).toEqual([]);
|
||||
} finally { unsubscribe(); }
|
||||
});
|
||||
|
||||
it("wakes milestone authority only when GitHub guest settings change, after outer commit", async () => {
|
||||
const f = await reviewBotFixture();
|
||||
const initial = await f.management.configuration(f.endpoint.id, "owner-user");
|
||||
const configuration = { ...initial.configuration, people: [{
|
||||
kind: "guest" as const, githubUserId: "84", login: "contributor",
|
||||
sponsorUserId: "owner-user", permissionProfile: "restricted" as const, automaticReviews: false,
|
||||
}] };
|
||||
const events: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, (event, scopes) => {
|
||||
if ([...scopes ?? []].some(scope => JSON.parse(scope).companyId === f.companyId)) events.push(event);
|
||||
});
|
||||
try {
|
||||
await db.transaction(async tx => {
|
||||
await githubChatManagementService(tx as unknown as typeof db, f.providerFetch)
|
||||
.saveConfiguration(f.endpoint.id, { expectedRevision: initial.revision, configuration }, "owner-user");
|
||||
expect(events).toEqual(["intent"]);
|
||||
expect((await f.service.get(f.endpoint.id)).allowUnlinkedPeople).toBe(false);
|
||||
});
|
||||
expect(events).toEqual(["intent", "settled"]);
|
||||
expect((await f.service.get(f.endpoint.id)).allowUnlinkedPeople).toBe(true);
|
||||
events.length = 0;
|
||||
const current = await f.management.configuration(f.endpoint.id, "owner-user");
|
||||
await f.management.saveConfiguration(f.endpoint.id, { expectedRevision: current.revision, configuration }, "owner-user");
|
||||
expect(events).toEqual([]);
|
||||
} finally { unsubscribe(); }
|
||||
});
|
||||
|
||||
it("scopes guest tasks and rechecks their sponsor before every bot tool", async () => {
|
||||
const f = await reviewBotFixture();
|
||||
const initial = await f.management.configuration(
|
||||
@@ -8295,11 +8364,62 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
expect(posts(f)).toHaveLength(1);
|
||||
expect(new URLSearchParams(String(posts(f)[0]![1]!.body)).get("channel")).toBe(f.installerId);
|
||||
});
|
||||
it("links the actor's own manager identity without requiring a connect command", async () => {
|
||||
it("signals milestone eligibility around the real Slack account link and stays quiet on reconfirmation", async () => {
|
||||
const f = await managedFixture(); await f.provision();
|
||||
await db.delete(chatIdentityLinks).where(eq(chatIdentityLinks.endpointId, f.endpoint.id));
|
||||
expect(await f.service.sendSlackTestMessage(f.endpoint.id, actor, { requestId: randomUUID() })).toMatchObject({ status: "sent" });
|
||||
expect(await db.select().from(chatIdentityLinks).where(eq(chatIdentityLinks.endpointId, f.endpoint.id))).toEqual([expect.objectContaining({ paperclipUserId: actor.userId, status: "linked" })]);
|
||||
const events: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, (event, scopes) => {
|
||||
if ([...scopes ?? []].some(scope => JSON.parse(scope).companyId === f.companyId)) events.push(event);
|
||||
});
|
||||
let release!: () => void, acquired!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; });
|
||||
const locked = new Promise<void>(resolve => { acquired = resolve; });
|
||||
const lock = db.transaction(async tx => {
|
||||
await tx.execute(sql`lock table chat_identity_links in share mode`);
|
||||
acquired(); await gate;
|
||||
});
|
||||
await locked;
|
||||
const linking = f.service.sendSlackTestMessage(f.endpoint.id, actor, { requestId: randomUUID() });
|
||||
void linking.catch(() => undefined);
|
||||
try {
|
||||
try {
|
||||
await vi.waitFor(async () => {
|
||||
const [waiting] = await db.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity
|
||||
where datname = current_database() and state = 'active' and wait_event_type = 'Lock'
|
||||
and query like 'insert into "chat_identity_links"%'`);
|
||||
expect(waiting.count).toBeGreaterThan(0);
|
||||
});
|
||||
expect(events).toEqual(["intent"]);
|
||||
expect(await db.select().from(chatIdentityLinks).where(eq(chatIdentityLinks.endpointId, f.endpoint.id))).toEqual([]);
|
||||
} finally { release(); await lock; await linking; }
|
||||
expect(events).toEqual(["intent", "settled"]);
|
||||
expect(await db.select().from(chatIdentityLinks).where(eq(chatIdentityLinks.endpointId, f.endpoint.id)))
|
||||
.toEqual([expect.objectContaining({ paperclipUserId: actor.userId, status: "linked" })]);
|
||||
events.length = 0;
|
||||
expect(await f.service.sendSlackTestMessage(f.endpoint.id, actor, { requestId: randomUUID() })).toMatchObject({ status: "sent" });
|
||||
expect(events).toEqual([]);
|
||||
} finally { unsubscribe(); }
|
||||
});
|
||||
it("settles a failed Slack account-link intent without committing identity or sending a test message", async () => {
|
||||
const f = await managedFixture(); await f.provision();
|
||||
await db.delete(chatIdentityLinks).where(eq(chatIdentityLinks.endpointId, f.endpoint.id));
|
||||
f.provider.mockClear();
|
||||
const events: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, (event, scopes) => {
|
||||
if ([...scopes ?? []].some(scope => JSON.parse(scope).companyId === f.companyId)) events.push(event);
|
||||
});
|
||||
await db.execute(sql`create function sleep_slack_link_failure() returns trigger language plpgsql as $$ begin raise exception 'link write unavailable'; end $$`);
|
||||
await db.execute(sql`create trigger sleep_slack_link_failure before insert on chat_identity_links for each row execute function sleep_slack_link_failure()`);
|
||||
try {
|
||||
await expect(f.service.sendSlackTestMessage(f.endpoint.id, actor, { requestId: randomUUID() })).rejects.toThrow();
|
||||
expect(events).toEqual(["intent", "settled"]);
|
||||
expect(await db.select().from(chatIdentityLinks).where(eq(chatIdentityLinks.endpointId, f.endpoint.id))).toEqual([]);
|
||||
expect(posts(f)).toHaveLength(0);
|
||||
} finally {
|
||||
unsubscribe();
|
||||
await db.execute(sql`drop trigger sleep_slack_link_failure on chat_identity_links`);
|
||||
await db.execute(sql`drop function sleep_slack_link_failure()`);
|
||||
}
|
||||
});
|
||||
it("requires a private Slack challenge for an unknown account before linking and sending the test", async () => {
|
||||
const f = await managedFixture(); await f.provision();
|
||||
@@ -34933,7 +35053,44 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
]);
|
||||
});
|
||||
|
||||
it("drains bounded Slack chat-origin milestones without admitting an internal issue run", async () => {
|
||||
it("wakes an existing chat milestone when ordinary ingress reactivates its conversation", async () => {
|
||||
const fixture = await seedCompany();
|
||||
const { callbacks, endpoint, wakeup } = await configuredSlackEndpoint(fixture);
|
||||
const thread = makeThread({ channelId: "C-REOPEN-MILESTONE", id: "slack:C-REOPEN-MILESTONE:4100.1", name: "reopen-milestone" });
|
||||
const send = (id: string) => deliverMessage({ callbacks, endpointId: endpoint.id, thread: thread.thread,
|
||||
message: makeMessage({ id, text: "@maya continue this conversation", mentioned: true }), trigger: "mention" });
|
||||
await send("4100.1");
|
||||
const [conversation] = await readChatConversationsByEndpointId(endpoint.id);
|
||||
const [link] = await db.select().from(chatMessageLinks).where(and(
|
||||
eq(chatMessageLinks.conversationId, conversation.id), eq(chatMessageLinks.providerMessageId, "4100.1")));
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId: fixture.companyId, agentId: fixture.assignedAgentId,
|
||||
status: "failed", errorCode: "low_trust_isolation_unavailable", updatedAt: new Date(),
|
||||
contextSnapshot: { issueId: conversation.issueId, source: "chat:slack", endpointId: endpoint.id, wakeCommentId: link.commentId },
|
||||
});
|
||||
await db.update(chatConversations).set({ state: "unavailable" }).where(eq(chatConversations.id, conversation.id));
|
||||
const onError = vi.fn();
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError });
|
||||
const notice = () => db.select().from(chatPublications).where(and(eq(chatPublications.issueId, conversation.issueId),
|
||||
eq(chatPublications.idempotencyKey, `run:${runId}:failed:${endpoint.id}`)));
|
||||
try {
|
||||
await chatRunMilestoneWork(db).register(coordinator, () => true).ready;
|
||||
await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBeNull());
|
||||
expect(await notice()).toEqual([]);
|
||||
const wakesBefore = wakeup.mock.calls.length;
|
||||
await send("4100.2");
|
||||
expect(wakeup.mock.calls.length).toBe(wakesBefore + 1);
|
||||
// The harness accepts the new wake without creating a run. Existing
|
||||
// projection debt must wake from ingress, not a later run transition.
|
||||
await expect.poll(notice, { timeout: 5000 }).toEqual([
|
||||
expect.objectContaining({ idempotencyKey: `run:${runId}:failed:${endpoint.id}` }),
|
||||
]);
|
||||
await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBeNull());
|
||||
expect(onError).not.toHaveBeenCalled();
|
||||
} finally { await coordinator.stop(); }
|
||||
});
|
||||
|
||||
it.each(["milliseconds", "microseconds"])("drains bounded Slack chat-origin milestones without admitting an internal issue run (%s)", async (precision) => {
|
||||
const fixture = await seedCompany();
|
||||
const { callbacks, endpoint } = await configuredSlackEndpoint(fixture);
|
||||
const thread = makeThread({
|
||||
@@ -35028,15 +35185,32 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
})),
|
||||
]);
|
||||
|
||||
const inserted: number[] = [];
|
||||
for (let index = 0; index < 4; index += 1) {
|
||||
inserted.push(
|
||||
await enqueueChatRunMilestones(db, {
|
||||
since: new Date("2026-09-05T14:00:00.000Z"),
|
||||
limit: 1,
|
||||
}),
|
||||
);
|
||||
if (precision === "microseconds") {
|
||||
await db.update(heartbeatRuns).set({
|
||||
updatedAt: sql`'2026-09-05T14:59:00.000123Z'::timestamptz`,
|
||||
}).where(inArray(heartbeatRuns.id, resolverRejectedRuns.map(run => run.id)));
|
||||
}
|
||||
// Fail a non-advancing pagination loop explicitly instead of leaving a
|
||||
// background query chain running after the test timeout.
|
||||
const select = db.select.bind(db);
|
||||
let reads = 0;
|
||||
const readGuard = vi.spyOn(db, "select").mockImplementation((...args) => {
|
||||
if (args[0] && "runUpdatedAt" in args[0] && ++reads > 100) {
|
||||
throw new Error("Milestone projection did not advance past rejected candidates");
|
||||
}
|
||||
return select(...args);
|
||||
});
|
||||
const inserted: number[] = [];
|
||||
try {
|
||||
for (let index = 0; index < 4; index += 1) {
|
||||
inserted.push(
|
||||
await enqueueChatRunLifecycleMilestones(db, {
|
||||
since: new Date("2026-09-05T14:00:00.000Z"),
|
||||
limit: 1,
|
||||
}),
|
||||
);
|
||||
}
|
||||
} finally { readGuard.mockRestore(); }
|
||||
expect(inserted).toEqual([1, 1, 1, 0]);
|
||||
const publications = await db
|
||||
.select()
|
||||
@@ -41727,13 +41901,18 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("publishes a closed-choice question, settles its Slack card, and delivers its exact continuation response", async () => {
|
||||
it("publishes a closed-choice question, settles its Slack card, and delivers its exact continuation response", async ({ onTestFinished }) => {
|
||||
let receiptWakeObserved = false;
|
||||
const fixture = await seedCompany();
|
||||
const continuationRunId = randomUUID();
|
||||
const resolveNativeQuestion = vi.fn(async () => "not_native" as const);
|
||||
const { callbacks, endpoint, runtime, service } =
|
||||
await configuredSlackEndpoint(fixture, {
|
||||
resolveNativeQuestion,
|
||||
questionResolutionPersistBarrier: async () => {
|
||||
// Receipt commit must wake projection before the separate answer write.
|
||||
expect(receiptWakeObserved).toBe(true);
|
||||
},
|
||||
wakeup: async (agentId, options) => {
|
||||
if (options.contextSnapshot?.source !== "issue.interaction.respond") {
|
||||
return { accepted: true };
|
||||
@@ -42056,6 +42235,12 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
.set({ state: "delivery_unknown", providerMessageId: null })
|
||||
.where(eq(chatPublications.id, publication.id));
|
||||
|
||||
const unsubscribeReceipt = subscribeDatabaseWork(db, "chat-run-milestones", (event, hints) => {
|
||||
if (event === "settled" && hints?.has(JSON.stringify({ companyId: fixture.companyId, issueId: conversation.issueId }))) {
|
||||
receiptWakeObserved = true;
|
||||
}
|
||||
});
|
||||
onTestFinished(unsubscribeReceipt);
|
||||
const providerAcknowledgement = callbacks.onAction(actionEvent());
|
||||
await expect(
|
||||
Promise.race([
|
||||
@@ -51802,8 +51987,8 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
const registeredCommandId = "888888888888888819";
|
||||
let interactionSequence = 0n;
|
||||
|
||||
async function commandFixture() {
|
||||
const fixture = await seedCompany();
|
||||
async function commandFixture(companyId?: string) {
|
||||
const fixture = await seedCompany(undefined, companyId);
|
||||
const applicationId = uniqueDiscordApplicationId();
|
||||
let holdWork = false;
|
||||
const queued: Array<() => void> = [];
|
||||
@@ -52897,8 +53082,8 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
: lastFreshDiscordMessageId + 1n;
|
||||
return lastFreshDiscordMessageId.toString();
|
||||
};
|
||||
async function reopenedFixture(includeDuringConfirmationSource = false) {
|
||||
const f = await commandFixture();
|
||||
async function reopenedFixture(includeDuringConfirmationSource = false, companyId?: string) {
|
||||
const f = await commandFixture(companyId);
|
||||
const threadId = `discord:${guildId}:${channelId}:${rootMessageId}`;
|
||||
const admitMessage = async (message: ReturnType<typeof makeMessage>) => {
|
||||
await deliverMessage({
|
||||
@@ -53046,6 +53231,142 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
};
|
||||
}
|
||||
|
||||
it.each(["action", "delivery"] as const)("wakes a post-close milestone when interrupted inbound acceptance settles its %s receipt", async receipt => {
|
||||
const f = await reopenedFixture();
|
||||
const onError = vi.fn();
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError });
|
||||
const notice = () => db.select().from(chatPublications).where(and(
|
||||
eq(chatPublications.issueId, f.conversation.issueId),
|
||||
eq(chatPublications.idempotencyKey, `run:${f.fresh.runId}:working:${f.endpoint.id}`),
|
||||
));
|
||||
try {
|
||||
// Crash boundary: the run/wake is durable, but its inbound action has
|
||||
// not yet recorded settlement. Recovery must not dispatch it again.
|
||||
if (receipt === "action") {
|
||||
await db.update(chatActions).set({ status: "issued", result: {}, updatedAt: new Date() })
|
||||
.where(eq(chatActions.id, f.fresh.action.id));
|
||||
} else {
|
||||
// Operator repair of a failed ledger keeps the already-admitted
|
||||
// action/run. Only delivery acceptance needs to commit again.
|
||||
await db.update(chatDeliveries).set({ state: "retry", processedAt: null, nextAttemptAt: null })
|
||||
.where(eq(chatDeliveries.id, f.delivery.id));
|
||||
}
|
||||
await chatRunMilestoneWork(db).register(coordinator, () => true).ready;
|
||||
await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBeNull());
|
||||
expect(await notice()).toEqual([]);
|
||||
const wakesBefore = f.wakeup.mock.calls.length;
|
||||
await expect.poll(async () => {
|
||||
await f.service.processPendingDeliveries(25, f.delivery.id);
|
||||
const [action] = await db.select().from(chatActions).where(eq(chatActions.id, f.fresh.action.id));
|
||||
const [delivery] = await db.select().from(chatDeliveries).where(eq(chatDeliveries.id, f.delivery.id));
|
||||
return [action.status, delivery.state];
|
||||
}, { timeout: 10_000 }).toEqual(["processed", "processed"]);
|
||||
expect(f.wakeup.mock.calls.length).toBe(wakesBefore);
|
||||
await expect.poll(notice, { timeout: 5000 }).toEqual([
|
||||
expect.objectContaining({ idempotencyKey: `run:${f.fresh.runId}:working:${f.endpoint.id}` }),
|
||||
]);
|
||||
await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBeNull());
|
||||
expect(onError).not.toHaveBeenCalled();
|
||||
} finally { await coordinator.stop(); await f.close(); }
|
||||
});
|
||||
|
||||
it.each(["connection", "resource", "unlinked", "identity", "member", "member_permissions", "ensure_membership", "company_access", "visibility", "cloud_sync"] as const)(
|
||||
"wakes a post-close milestone when %s authority is restored", async owner => {
|
||||
const stackId = randomUUID();
|
||||
const f = await reopenedFixture(false, owner === "cloud_sync" ? cloudTenantPrimaryCompanyId(stackId) : undefined);
|
||||
const previousToken = process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN;
|
||||
const [previousUser] = owner === "cloud_sync"
|
||||
? await db.select().from(authUsers).where(eq(authUsers.id, "owner-user"))
|
||||
: [];
|
||||
const onError = vi.fn();
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError });
|
||||
const notice = () => db.select().from(chatPublications).where(and(
|
||||
eq(chatPublications.issueId, f.conversation.issueId),
|
||||
eq(chatPublications.idempotencyKey, `run:${f.fresh.runId}:working:${f.endpoint.id}`),
|
||||
));
|
||||
const access = accessService(db);
|
||||
const membership = (await db.select().from(companyMemberships).where(and(
|
||||
eq(companyMemberships.companyId, f.fixture.companyId), eq(companyMemberships.principalId, "owner-user"),
|
||||
)))[0];
|
||||
try {
|
||||
let restore: () => Promise<unknown>;
|
||||
if (owner === "connection") {
|
||||
await toolAccessService(db).updateConnection(f.endpoint.connectionId, { enabled: false });
|
||||
restore = () => toolAccessService(db).updateConnection(f.endpoint.connectionId, { enabled: true });
|
||||
} else if (owner === "resource") {
|
||||
const [conversation] = await db.select().from(chatConversations).where(eq(chatConversations.id, f.conversation.id));
|
||||
expect(conversation.resourceId).not.toBeNull();
|
||||
await f.service.replaceResources(f.endpoint.id, [{ id: conversation.resourceId!, enabled: false }], "owner-user");
|
||||
restore = () => f.service.replaceResources(f.endpoint.id, [{ id: conversation.resourceId!, enabled: true }], "owner-user");
|
||||
} else if (owner === "identity" || owner === "unlinked") {
|
||||
await f.service.update(f.endpoint.id, { allowUnlinkedPeople: false }, "owner-user");
|
||||
await f.service.revokeLink(f.endpoint.id, f.fresh.action.principalId!);
|
||||
if (owner === "unlinked") {
|
||||
restore = () => f.service.update(f.endpoint.id, { allowUnlinkedPeople: true }, "owner-user");
|
||||
} else {
|
||||
const intent = await f.service.createLinkIntent(f.endpoint.id, f.fresh.action.principalId!, 1800);
|
||||
restore = () => f.service.confirmIdentityLink(new URL(intent.confirmationUrl).searchParams.get("token")!, "owner-user");
|
||||
}
|
||||
} else if (owner === "visibility") {
|
||||
await issueService(db).update(f.conversation.issueId, { hiddenAt: new Date() });
|
||||
restore = () => issueService(db).update(f.conversation.issueId, { hiddenAt: null });
|
||||
} else {
|
||||
// Model existing denied authority without exercising revocation's
|
||||
// unrelated credential cleanup; restoration uses its actual owner.
|
||||
await db.update(companyMemberships).set({ status: "suspended" }).where(eq(companyMemberships.id, membership.id));
|
||||
if (owner === "member") restore = () => access.updateMember(f.fixture.companyId, membership.id, { status: "active" });
|
||||
else if (owner === "member_permissions") restore = () => access.updateMemberAndPermissions(f.fixture.companyId, membership.id, { status: "active", grants: [] }, "owner-user");
|
||||
else if (owner === "cloud_sync") {
|
||||
process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN = "milestone-cloud-sync-test";
|
||||
let request = 0;
|
||||
restore = () => resolveCloudTenantActor(db, cloudActorHeaderSourceFromHeaders({
|
||||
"x-paperclip-cloud-tenant-token": "milestone-cloud-sync-test",
|
||||
"x-paperclip-cloud-user-id": "owner-user",
|
||||
"x-paperclip-cloud-user-email": "owner@example.test",
|
||||
// Change the profile to force an actual sync on replay. The
|
||||
// debounce must not hide repeated unchanged membership wakes.
|
||||
"x-paperclip-cloud-user-name": `Owner ${request++}`,
|
||||
"x-paperclip-cloud-stack-id": stackId,
|
||||
"x-paperclip-cloud-stack-role": "owner",
|
||||
}));
|
||||
}
|
||||
else if (owner === "ensure_membership") restore = () => access.ensureMembership(f.fixture.companyId, "user", "owner-user", membership.membershipRole, "active");
|
||||
else {
|
||||
const existing = await db.select().from(companyMemberships).where(and(
|
||||
eq(companyMemberships.principalType, "user"), eq(companyMemberships.principalId, "owner-user"),
|
||||
));
|
||||
restore = () => access.setUserCompanyAccess("owner-user", existing.map(row => row.companyId));
|
||||
}
|
||||
}
|
||||
await chatRunMilestoneWork(db).register(coordinator, () => true).ready;
|
||||
await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBeNull());
|
||||
expect(await notice()).toEqual([]);
|
||||
await restore();
|
||||
await expect.poll(notice, { timeout: 5000 }).toEqual([
|
||||
expect.objectContaining({ idempotencyKey: `run:${f.fresh.runId}:working:${f.endpoint.id}` }),
|
||||
]);
|
||||
await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBeNull());
|
||||
if (owner !== "identity" && owner !== "connection") {
|
||||
const notices: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, event => notices.push(event));
|
||||
try { await restore(); expect(notices).toEqual([]); }
|
||||
finally { unsubscribe(); }
|
||||
expect(await notice()).toHaveLength(1);
|
||||
}
|
||||
expect(onError).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
if (previousToken === undefined) delete process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN;
|
||||
else process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN = previousToken;
|
||||
await coordinator.stop();
|
||||
if (previousUser) await db.update(authUsers).set({
|
||||
name: previousUser.name, email: previousUser.email,
|
||||
emailVerified: previousUser.emailVerified, updatedAt: previousUser.updatedAt,
|
||||
}).where(eq(authUsers.id, previousUser.id));
|
||||
await f.close();
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it("publishes working and the exact fresh final after pinned close without resurrecting the old source", async () => {
|
||||
const f = await reopenedFixture();
|
||||
try {
|
||||
@@ -65325,6 +65646,36 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
};
|
||||
}
|
||||
|
||||
it.each(["separate owners", "combined budget"] as const)(
|
||||
"keeps lifecycle and native progress projection independent with %s",
|
||||
async (mode) => {
|
||||
await enqueueChatRunMilestones(db, { since: new Date(0) });
|
||||
const context = await safeNativeProgressFixture("telegram", mode === "separate owners" ? "111" : "112");
|
||||
try {
|
||||
const progressRun = await context.createRun("ongoing work");
|
||||
await context.addEvent(progressRun);
|
||||
const failedRun = await context.createRun("failed work");
|
||||
await db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, failedRun.runId));
|
||||
const input = { since: new Date(0), limit: 1 };
|
||||
await expect((mode === "separate owners" ? enqueueChatRunLifecycleMilestones : enqueueChatRunMilestones)(db, input)).resolves.toBe(1);
|
||||
const rows = () => db.select({ key: chatPublications.idempotencyKey }).from(chatPublications)
|
||||
.where(eq(chatPublications.conversationId, context.conversation.id));
|
||||
expect((await rows()).map(row => row.key)).toContain(`run:${failedRun.runId}:failed:${context.endpoint.id}`);
|
||||
expect((await rows()).some(row => row.key.includes(":native:"))).toBe(false);
|
||||
if (mode === "separate owners") {
|
||||
await expect(enqueueChatRunLifecycleMilestones(db, input)).resolves.toBe(0);
|
||||
}
|
||||
await expect((mode === "separate owners" ? enqueueSafeNativeChatProgress : enqueueChatRunMilestones)(db, input)).resolves.toBe(1);
|
||||
expect((await rows()).filter(row => row.key.includes(":native:"))).toEqual([
|
||||
{ key: `run:${progressRun.runId}:working:${context.endpoint.id}:native:making_progress:1` },
|
||||
]);
|
||||
await expect(enqueueChatRunMilestones(db, input)).resolves.toBe(0);
|
||||
} finally {
|
||||
await retirePublicationFixture(context.service, context.endpoint.id);
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["issue", "run"] as const)(
|
||||
"skips a locked native progress %s while unrelated final and question publications deliver",
|
||||
async (lockTarget) => {
|
||||
@@ -72379,6 +72730,48 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
};
|
||||
}
|
||||
|
||||
it("restores changed native review metadata once and signals only its committed change", async () => {
|
||||
const context = await committedNativeReviewPublicationFixture("milestone-repair");
|
||||
// Earlier failure cases can leave intent awaiting XID reconciliation on the
|
||||
// suite's shared owner. Subscribing to that owner immediately replays intent.
|
||||
await expect(db.transaction(async tx => {
|
||||
await signalDatabaseWork(tx, "chat-run-milestones");
|
||||
throw new Error("unrelated rolled-back producer");
|
||||
})).rejects.toThrow("unrelated rolled-back producer");
|
||||
const repairDb = createDb(externalTestDatabaseUrl ?? tempDb!.connectionString);
|
||||
const events: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(repairDb, "chat-run-milestones", event => events.push(event));
|
||||
try {
|
||||
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, context.runId));
|
||||
const input = { companyId: context.fixture.companyId, issueId: coordinator.issueId, runId: context.runId,
|
||||
decisionId: coordinator.decisionId!, resultId: coordinator.resultId!, assessmentId: coordinator.assessmentId! };
|
||||
const savedAt = new Date(Date.now() - 60_000);
|
||||
await db.update(heartbeatRuns).set({ updatedAt: savedAt }).where(eq(heartbeatRuns.id, context.runId));
|
||||
// Already-restored metadata still authorizes materialization by the caller,
|
||||
// but must neither rewrite the run nor wake projection again.
|
||||
await expect(repairDb.transaction(tx => restoreNativeChatReviewPresentationInTransaction(tx as unknown as typeof db, input))).resolves.toBe(true);
|
||||
expect((await readHeartbeatRunsById(context.runId))[0].updatedAt).toEqual(savedAt);
|
||||
expect(events).toEqual([]);
|
||||
await db.update(heartbeatRuns).set({ resultJson: sql`${heartbeatRuns.resultJson} - 'externalChatReviewPresentation'` })
|
||||
.where(eq(heartbeatRuns.id, context.runId));
|
||||
await repairDb.transaction(async tx => {
|
||||
expect(await restoreNativeChatReviewPresentationInTransaction(tx as unknown as typeof db, input)).toBe(true);
|
||||
expect(events).toEqual(["intent"]);
|
||||
});
|
||||
expect(events).toEqual(["intent", "settled"]);
|
||||
expect((await readHeartbeatRunsById(context.runId))[0].resultJson).toMatchObject({
|
||||
externalChatReviewPresentation: { decisionId: coordinator.decisionId, runId: context.runId },
|
||||
});
|
||||
events.length = 0;
|
||||
await repairDb.transaction(tx => restoreNativeChatReviewPresentationInTransaction(tx as unknown as typeof db, input));
|
||||
expect(events).toEqual([]);
|
||||
} finally {
|
||||
unsubscribe();
|
||||
await repairDb.$client.end({ timeout: 1 });
|
||||
await context.service.shutdown();
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["principal_revoked", "gate_changed"] as const)(
|
||||
"blocks a committed native review response and selected files at transport after %s",
|
||||
async (kind) => {
|
||||
|
||||
@@ -0,0 +1,230 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
completionContracts, nativeRunResults, nativeRunFinalizations, workAssessments, statusDecisions, statusDecisionEffects,
|
||||
agents, toolActionDeliveries, toolActionRequests, toolInvocations, toolGatewaySessions, toolCallEvents, toolAccessAuditEvents, toolPolicies,
|
||||
companySecretBindings, companySecretProposals, companySecrets, createDb,
|
||||
chatPublications, heartbeatRuns, issueComments, issueThreadInteractions, issues,
|
||||
} from "@paperclipai/db";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { createChatMilestoneFixtures } from "./helpers/chat-milestone-fixtures.js";
|
||||
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
||||
import { chatRunMilestoneWork } from "../services/chat-run-milestone-work.js";
|
||||
import { createSecretProposalsService } from "../services/secret-proposals.js";
|
||||
import { commitToolActionReview } from "../services/tool-action-review.js";
|
||||
import { createToolGatewayService } from "../services/tool-gateway.js";
|
||||
import { issueThreadInteractionService } from "../services/issue-thread-interactions.js";
|
||||
import { prepareConversationTurn } from "../services/agent-conversations.js";
|
||||
import { dismissAutomaticCompletionReviews } from "../services/native-runtime/automatic-completion-reviews.js";
|
||||
import { dismissObsoleteNativePolicyReviews } from "../services/native-runtime/obsolete-policy-reviews.js";
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_MILESTONE_GOVERNED_TEST_DATABASE_URL");
|
||||
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
(support.supported ? describe : describe.skip)("chat milestones after governed interaction writes", () => {
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let fixtures: ReturnType<typeof createChatMilestoneFixtures>;
|
||||
let f: Awaited<ReturnType<typeof fixtures.create>>;
|
||||
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
|
||||
const errors: unknown[] = [];
|
||||
beforeAll(async () => {
|
||||
if (!external) temporary = await startEmbeddedPostgresTestDatabase("chat-governed-milestones-");
|
||||
db = createDb(external ?? temporary!.connectionString, { idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
|
||||
fixtures = createChatMilestoneFixtures(db);
|
||||
}, 60_000);
|
||||
beforeEach(async () => {
|
||||
errors.length = 0; f = await fixtures.create();
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
});
|
||||
afterEach(async () => {
|
||||
await coordinator?.stop(); coordinator = undefined; vi.useRealTimers(); vi.restoreAllMocks();
|
||||
await db.delete(companySecretProposals).where(eq(companySecretProposals.companyId, f.companyId));
|
||||
await db.delete(companySecretBindings).where(eq(companySecretBindings.companyId, f.companyId));
|
||||
await db.delete(companySecrets).where(eq(companySecrets.companyId, f.companyId));
|
||||
await db.delete(toolActionDeliveries).where(eq(toolActionDeliveries.companyId, f.companyId));
|
||||
await db.delete(toolGatewaySessions).where(eq(toolGatewaySessions.companyId, f.companyId));
|
||||
await db.delete(toolCallEvents).where(eq(toolCallEvents.companyId, f.companyId));
|
||||
await db.delete(toolAccessAuditEvents).where(eq(toolAccessAuditEvents.companyId, f.companyId));
|
||||
await db.delete(toolActionRequests).where(eq(toolActionRequests.companyId, f.companyId));
|
||||
await db.delete(toolInvocations).where(eq(toolInvocations.companyId, f.companyId));
|
||||
await db.delete(toolPolicies).where(eq(toolPolicies.companyId, f.companyId));
|
||||
await fixtures.cleanup();
|
||||
expect(errors).toEqual([]);
|
||||
});
|
||||
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
|
||||
async function start() {
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: error => errors.push(error) });
|
||||
await chatRunMilestoneWork(db).register(coordinator, () => true).ready;
|
||||
await flush();
|
||||
}
|
||||
async function flush() {
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 10_000 });
|
||||
}
|
||||
async function quiet() {
|
||||
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
|
||||
try { await vi.advanceTimersByTimeAsync(120_000); expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); }
|
||||
finally { reads.mockRestore(); writes.mockRestore(); }
|
||||
}
|
||||
const publications = () => db.select().from(chatPublications).where(eq(chatPublications.issueId, f.issueId));
|
||||
|
||||
async function run(status: "failed" | "running" = "failed") {
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: f.agentId,
|
||||
responsibleUserId: f.userId, status, errorCode: "test_run_failed", updatedAt: new Date(),
|
||||
contextSnapshot: { issueId: f.issueId, source: "chat:slack", endpointId: f.endpointId, wakeCommentId: f.commentId },
|
||||
});
|
||||
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, f.issueId));
|
||||
return runId;
|
||||
}
|
||||
|
||||
it.each(["approved", "rejected", "withdrawn", "expired", "rolled back", "parent rejected"] as const)("reactivates after a binding proposal is %s", async status => {
|
||||
const runId = await run();
|
||||
// Binding review needs secret metadata, not the provider value. No secret is resolved.
|
||||
const [secret] = await db.insert(companySecrets).values({ companyId: f.companyId, name: "test/binding", key: "BINDING_TEST" }).returning();
|
||||
const parent = status === "parent rejected" ? (await db.insert(companySecretProposals).values({
|
||||
companyId: f.companyId, kind: "secret", proposedName: "test/parent", proposedKey: "PARENT_TEST",
|
||||
justification: "Pending parent secret", proposedByAgentId: f.agentId, originRunId: runId,
|
||||
originIssueId: f.issueId, expiresAt: new Date(Date.now() + 86_400_000),
|
||||
}).returning())[0] : null;
|
||||
const proposals = createSecretProposalsService(db);
|
||||
const proposal = await proposals.createBinding({ companyId: f.companyId, heartbeatRunId: runId }, {
|
||||
...(parent ? { secretProposalId: parent.id } : { secretId: secret.id }), configPath: "env.BINDING_TEST", justification: "Use the existing company credential", bindingTargetPolicy: "self_and_reports",
|
||||
});
|
||||
expect(proposal.interactionId).toBeTruthy();
|
||||
await start(); expect(await publications()).toEqual([]); await quiet();
|
||||
if (status === "rolled back") {
|
||||
await expect(db.transaction(async tx => {
|
||||
await createSecretProposalsService(tx as unknown as typeof db).transition(f.companyId, proposal.id, "rejected");
|
||||
throw new Error("rollback proposal");
|
||||
})).rejects.toThrow("rollback proposal");
|
||||
expect((await proposals.getById(f.companyId, proposal.id))?.status).toBe("pending");
|
||||
await flush(); expect(await publications()).toEqual([]); await quiet();
|
||||
await proposals.transition(f.companyId, proposal.id, "rejected");
|
||||
} else if (parent) {
|
||||
await proposals.transition(f.companyId, parent.id, "rejected");
|
||||
expect((await proposals.getById(f.companyId, proposal.id))?.status).toBe("rejected");
|
||||
} else if (status === "approved") {
|
||||
await proposals.approve(f.companyId, proposal.id, { resolvedByUserId: f.userId });
|
||||
} else if (status === "expired") {
|
||||
expect(await proposals.sweepExpired(new Date(proposal.expiresAt.getTime() + 1))).toBe(1);
|
||||
} else if (status === "rejected" || status === "withdrawn") {
|
||||
await proposals.transition(f.companyId, proposal.id, status, { resolvedByUserId: f.userId, proposerAgentId: f.agentId });
|
||||
}
|
||||
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, proposal.interactionId!)))[0].status)
|
||||
.toBe(status === "approved" ? "accepted" : status === "withdrawn" ? "cancelled" : status === "rolled back" || status === "parent rejected" ? "rejected" : status);
|
||||
await flush();
|
||||
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${runId}:failed:${f.endpointId}`);
|
||||
await quiet();
|
||||
});
|
||||
it.each(["approved", "rejected", "expired"] as const)("reactivates after a tool review is %s", async status => {
|
||||
await db.update(agents).set({ adapterType: "process" }).where(eq(agents.id, f.agentId));
|
||||
const runId = await run("running");
|
||||
await db.insert(toolPolicies).values({ companyId: f.companyId, name: "Ask first", policyType: "require_approval",
|
||||
selectors: { toolName: "mcp-remote-fixture:update_note" } });
|
||||
const remote = vi.fn(async () => { throw new Error("Review must not invoke the provider"); });
|
||||
const gateway = createToolGatewayService(db, { toolActionSigningSecret: "milestone-review-fixture", remoteHttpRequest: remote });
|
||||
const session = await gateway.createSession({ companyId: f.companyId, agentId: f.agentId, runId });
|
||||
await expect(gateway.executeTool({ sessionToken: session.token, tool: "mcp-remote-fixture:update_note", parameters: { noteId: "test" } }))
|
||||
.rejects.toMatchObject({ reasonCode: "approval_required" });
|
||||
const [request] = await db.select().from(toolActionRequests).where(eq(toolActionRequests.companyId, f.companyId));
|
||||
expect(request.interactionId).toBeTruthy();
|
||||
// The prompt could not be delivered, and the run failed while awaiting review.
|
||||
// The pending review still owns its response until the board resolves it.
|
||||
await db.update(chatPublications).set({ state: "failed" }).where(eq(chatPublications.issueId, f.issueId));
|
||||
await db.update(heartbeatRuns).set({ status: "failed", updatedAt: new Date() }).where(eq(heartbeatRuns.id, runId));
|
||||
await start();
|
||||
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${runId}:`))).toBe(false);
|
||||
await quiet();
|
||||
if (status === "expired") {
|
||||
await db.update(toolActionRequests).set({ expiresAt: new Date(Date.now() - 1) }).where(eq(toolActionRequests.id, request.id));
|
||||
await gateway.sweepExpiredActionReviews();
|
||||
} else {
|
||||
await commitToolActionReview(db, { companyId: f.companyId, actionRequestId: request.id, decision: status, actor: { userId: f.userId } });
|
||||
}
|
||||
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, request.interactionId!)))[0].status)
|
||||
.toBe(status === "approved" ? "accepted" : status);
|
||||
await flush();
|
||||
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${runId}:failed:${f.endpointId}`);
|
||||
await quiet(); expect(remote).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([
|
||||
["automatic", "prompt"], ["automatic", "no prompt"], ["automatic", "explicit request"],
|
||||
["obsolete", "prompt"], ["obsolete", "no prompt"], ["obsolete", "explicit request"],
|
||||
] as const)("handles %s review retirement with %s", async (kind, history) => {
|
||||
const runId = await run(), contractId = randomUUID(), resultId = randomUUID(), assessmentId = randomUUID(), decisionId = randomUUID();
|
||||
const prompt = kind === "automatic"
|
||||
? "Review the persisted native-run evidence and confirm whether this issue may be completed."
|
||||
: "Review the superseding native policy assessment.";
|
||||
await db.insert(completionContracts).values({ id: contractId, companyId: f.companyId, issueId: f.issueId, revision: 1,
|
||||
schemaVersion: "paperclip.completion-contract.v1", policyVersion: "phase6-v1", risk: kind === "automatic" ? "low" : "standard",
|
||||
completionAuthority: kind === "automatic" ? "agent_claim_policy" : "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal",
|
||||
contractJson: { objective: "Respond to chat" }, canonicalSha256: contractId, createdByActorType: "system", createdByActorId: "test" });
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded", runtimeMode: "native", nativeIssueId: f.issueId, completionContractId: contractId,
|
||||
resultJson: { presentationDecision: { source: "none" } } }).where(eq(heartbeatRuns.id, runId));
|
||||
await db.insert(nativeRunResults).values({ id: resultId, companyId: f.companyId, issueId: f.issueId, runId,
|
||||
completionContractId: contractId, serverFingerprint: resultId, schemaStatus: "accepted", canonicalSha256: resultId,
|
||||
resultJson: { terminal: { runTerminalState: "succeeded" } } });
|
||||
await db.insert(nativeRunFinalizations).values({ runId, companyId: f.companyId, issueId: f.issueId, phase: "committed", resultId });
|
||||
await db.insert(workAssessments).values({ id: assessmentId, companyId: f.companyId, issueId: f.issueId, runId,
|
||||
contractId, resultId, triggerKind: "run_result", triggerActorCompanyId: f.companyId, priorIssueStatus: "in_progress",
|
||||
priorStatusVersion: 0, policyVersion: "phase6-v1", assessmentJson: { attentionRequests: [] }, inputDigest: assessmentId });
|
||||
await db.insert(statusDecisions).values({ id: decisionId, companyId: f.companyId, issueId: f.issueId, runId, assessmentId,
|
||||
decisionVersion: 1, policyVersion: "phase6-v1", fromStatus: "in_progress", toStatus: "in_review", applicationState: "applied",
|
||||
reasonCode: kind === "automatic" ? "completion_claim_incomplete" : "completion_review_required", decisionDigest: decisionId,
|
||||
decisionJson: { projectedStatusVersion: 1, effects: [{ kind: "bind_reviewer", prompt, ownerUserId: null },
|
||||
...(kind === "obsolete" ? [{ kind: "append_superseding_assessment" }] : [])] } });
|
||||
await db.update(issues).set({ status: "in_review", statusVersion: 1, lastStatusDecisionId: decisionId }).where(eq(issues.id, f.issueId));
|
||||
const [interaction] = await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId: f.issueId,
|
||||
kind: "request_confirmation", sourceRunId: runId, idempotencyKey: `native-review:${decisionId}`, status: "pending",
|
||||
payload: { version: 1, prompt: history === "explicit request" ? "Review the release before publishing." : prompt,
|
||||
target: { type: "custom", key: "native_completion_review", revisionId: decisionId } } }).returning();
|
||||
await db.insert(statusDecisionEffects).values({ companyId: f.companyId, issueId: f.issueId, decisionId,
|
||||
ordinal: 0, effectKind: "bind_reviewer", targetType: "issue_thread_interaction", targetId: interaction.id,
|
||||
idempotencyKey: decisionId, payload: {}, deliveryState: "delivered" });
|
||||
// System cards are no longer externalized, but stored prompt rows still own
|
||||
// the response slot. Exercise their supported retirement rather than insert
|
||||
// a synthetic new provider send through today's author-validation boundary.
|
||||
if (history !== "no prompt") await db.insert(chatPublications).values({ companyId: f.companyId, issueId: f.issueId,
|
||||
endpointId: f.endpointId, conversationId: f.conversationId, state: "pending", payload: { text: prompt, interactionId: interaction.id },
|
||||
idempotencyKey: `interaction:${interaction.id}:${f.endpointId}` });
|
||||
const notices = async () => (await publications()).filter(row => row.idempotencyKey === `run:${runId}:completed:${f.endpointId}`);
|
||||
const retire = () => kind === "automatic" ? dismissAutomaticCompletionReviews(db, f.issueId) : dismissObsoleteNativePolicyReviews(db, [runId]);
|
||||
await start(); expect(await notices()).toHaveLength(history === "no prompt" ? 1 : 0); await quiet();
|
||||
await retire();
|
||||
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interaction.id)))[0].status)
|
||||
.toBe(history === "explicit request" ? "pending" : "cancelled");
|
||||
await flush(); expect(await notices()).toHaveLength(history === "explicit request" ? 0 : 1); await quiet();
|
||||
await retire(); await flush(); expect(await notices()).toHaveLength(history === "explicit request" ? 0 : 1); await quiet();
|
||||
});
|
||||
|
||||
it("rechecks a stored chat binding when a conversation reset expires its question", async () => {
|
||||
const runId = await run();
|
||||
await db.update(issues).set({ conversationAgentId: f.agentId, conversationUserId: f.userId, conversationState: "active" }).where(eq(issues.id, f.issueId));
|
||||
const interaction = await issueThreadInteractionService(db).create({ id: f.issueId, companyId: f.companyId }, {
|
||||
kind: "ask_user_questions", sourceRunId: runId, continuationPolicy: "wake_assignee",
|
||||
payload: { version: 1, title: "Choose", questions: [{ id: "choice", prompt: "Continue?", selectionMode: "single",
|
||||
options: [{ id: "yes", label: "Yes" }], required: true }] },
|
||||
}, { agentId: f.agentId, runId });
|
||||
// Failed delivery leaves the pending question itself as the response owner.
|
||||
await db.update(chatPublications).set({ state: "failed" }).where(eq(chatPublications.issueId, f.issueId));
|
||||
const [comment] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId,
|
||||
authorType: "user", authorUserId: f.userId, body: "/new" }).returning();
|
||||
const [resetRun] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId,
|
||||
status: "running", contextSnapshot: { issueId: f.issueId, source: "issue.comment", wakeCommentId: comment.id } }).returning();
|
||||
await start();
|
||||
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${runId}:`))).toBe(false); await quiet();
|
||||
expect(await prepareConversationTurn(db, resetRun)).toMatchObject({ reset: true });
|
||||
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interaction.id)))[0].status).toBe("expired");
|
||||
await flush();
|
||||
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${runId}:failed:${f.endpointId}`);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
});
|
||||
@@ -0,0 +1,204 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { agentWakeupRequests, chatActions, chatExternalPrincipals, chatIdentityLinks, chatMessageLinks, chatPublications, companyMemberships, createDb, heartbeatRuns, issueComments, issues, subscribeDatabaseWork } from "@paperclipai/db";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { createChatMilestoneFixtures } from "./helpers/chat-milestone-fixtures.js";
|
||||
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
||||
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
|
||||
import { chatRunMilestoneWork } from "../services/chat-run-milestone-work.js";
|
||||
import { initializeRunIdentity, reserveSteeredIdentity, acceptSteeredIdentity } from "../services/run-identity.js";
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
import { accessService } from "../services/access.js";
|
||||
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_MILESTONE_IDENTITY_TEST_DATABASE_URL");
|
||||
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
(support.supported ? describe : describe.skip)("chat milestones after identity activation", () => {
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let fixtures: ReturnType<typeof createChatMilestoneFixtures>;
|
||||
let f: Awaited<ReturnType<typeof fixtures.create>>;
|
||||
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
|
||||
const errors: unknown[] = [];
|
||||
beforeAll(async () => {
|
||||
if (!external) temporary = await startEmbeddedPostgresTestDatabase("chat-identity-milestones-");
|
||||
db = createDb(external ?? temporary!.connectionString, { idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
|
||||
fixtures = createChatMilestoneFixtures(db);
|
||||
}, 60_000);
|
||||
beforeEach(async () => {
|
||||
errors.length = 0; f = await fixtures.create();
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
});
|
||||
afterEach(async () => {
|
||||
await coordinator?.stop(); coordinator = undefined; vi.useRealTimers(); vi.restoreAllMocks();
|
||||
await fixtures.cleanup(); expect(errors).toEqual([]);
|
||||
});
|
||||
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
|
||||
async function flush() {
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 10_000 });
|
||||
}
|
||||
async function quiet() {
|
||||
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
|
||||
try { await vi.advanceTimersByTimeAsync(120_000); expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); }
|
||||
finally { reads.mockRestore(); writes.mockRestore(); }
|
||||
}
|
||||
const publications = () => db.select().from(chatPublications).where(eq(chatPublications.issueId, f.issueId));
|
||||
|
||||
it.each([false, true])("rechecks concurrent membership suspension before restoring notice access (rollback=%s)", async rollback => {
|
||||
const membershipWhere = and(eq(companyMemberships.companyId, f.companyId), eq(companyMemberships.principalId, f.userId));
|
||||
// A Board-origin reply uses the linked author and current membership.
|
||||
// Remove the fixture's unrelated provider-ingress provenance.
|
||||
await db.delete(chatMessageLinks).where(eq(chatMessageLinks.commentId, f.commentId));
|
||||
await db.update(issueComments).set({ authorUserId: f.userId }).where(eq(issueComments.id, f.commentId));
|
||||
const [principal] = await db.insert(chatExternalPrincipals).values({ companyId: f.companyId,
|
||||
provider: "slack", providerAccountId: "", externalId: f.userId,
|
||||
}).returning();
|
||||
await db.insert(chatIdentityLinks).values({ companyId: f.companyId, endpointId: f.endpointId,
|
||||
principalId: principal.id, paperclipUserId: f.userId, status: "linked",
|
||||
});
|
||||
await db.insert(chatActions).values({ companyId: f.companyId, endpointId: f.endpointId,
|
||||
conversationId: f.conversationId, kind: "slack_board_message", status: "processed",
|
||||
providerActionId: `board-comment:${f.commentId}`, payload: { issueId: f.issueId, commentId: f.commentId,
|
||||
userId: f.userId, principalId: principal.id, agentId: f.agentId },
|
||||
});
|
||||
const runId = randomUUID(), sourceId = randomUUID();
|
||||
const nextCheckAt = new Date(Date.now() - 1000).toISOString();
|
||||
await db.insert(heartbeatRuns).values({ id: sourceId, companyId: f.companyId, agentId: f.agentId,
|
||||
nativeIssueId: f.issueId, runtimeMode: "native", status: "succeeded", responsibleUserId: f.userId,
|
||||
contextSnapshot: { issueId: f.issueId, source: "chat:slack", endpointId: f.endpointId, wakeCommentId: f.commentId },
|
||||
resultJson: { semanticToolReceipts: { schedule: { operationId: "set_task_monitor",
|
||||
result: { taskId: f.issueId, monitor: { status: "scheduled", nextCheckAt } } } } },
|
||||
});
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: f.agentId,
|
||||
nativeIssueId: f.issueId, runtimeMode: "native", status: "running", responsibleUserId: f.userId,
|
||||
contextSnapshot: { issueId: f.issueId, source: "issue.monitor" },
|
||||
});
|
||||
await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, runId,
|
||||
source: "automation", reason: "issue_monitor_due", status: "claimed",
|
||||
idempotencyKey: `issue-monitor:${f.issueId}:${nextCheckAt}`, payload: { issueId: f.issueId, nextCheckAt },
|
||||
});
|
||||
await initializeRunIdentity(db, { companyId: f.companyId, runId, issueId: f.issueId,
|
||||
responsibleUserId: f.userId, cause: "dispatch" });
|
||||
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, f.issueId));
|
||||
let releaseSuspension!: () => void;
|
||||
const suspensionHeld = new Promise<void>(resolve => { releaseSuspension = resolve; });
|
||||
let markSuspended!: () => void;
|
||||
const suspended = new Promise<void>(resolve => { markSuspended = resolve; });
|
||||
let releaseRestoration!: () => void;
|
||||
const restorationHeld = new Promise<void>(resolve => { releaseRestoration = resolve; });
|
||||
let restored = false;
|
||||
let restorationPid = 0;
|
||||
const suspending = db.transaction(async tx => {
|
||||
await tx.update(companyMemberships).set({ status: "suspended" }).where(membershipWhere);
|
||||
markSuspended();
|
||||
await suspensionHeld;
|
||||
});
|
||||
await suspended;
|
||||
const restoring = db.transaction(async tx => {
|
||||
const [backend] = await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`);
|
||||
restorationPid = backend.pid;
|
||||
await accessService(tx as unknown as typeof db).ensureMembership(f.companyId, "user", f.userId, "operator");
|
||||
restored = true;
|
||||
await restorationHeld;
|
||||
if (rollback) throw new Error("outer membership rollback");
|
||||
});
|
||||
void restoring.catch(() => undefined);
|
||||
try {
|
||||
// The old read sees active membership, then its UPDATE waits. The fixed
|
||||
// read waits first and observes the suspension after that writer commits.
|
||||
await vi.waitFor(async () => {
|
||||
expect(restorationPid).toBeGreaterThan(0);
|
||||
const [state] = await db.execute<{ blocked: boolean }>(sql`select cardinality(pg_blocking_pids(${restorationPid})) > 0 as blocked`);
|
||||
expect(state.blocked).toBe(true);
|
||||
});
|
||||
releaseSuspension();
|
||||
await suspending;
|
||||
await vi.waitFor(() => expect(restored).toBe(true));
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: error => errors.push(error) });
|
||||
await chatRunMilestoneWork(db).register(coordinator, () => true).ready;
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(await publications()).toEqual([]);
|
||||
expect((await db.select().from(companyMemberships).where(membershipWhere))[0].status).toBe("suspended");
|
||||
releaseRestoration();
|
||||
if (rollback) await expect(restoring).rejects.toThrow("outer membership rollback");
|
||||
else await restoring;
|
||||
await flush();
|
||||
expect((await db.select().from(companyMemberships).where(membershipWhere))[0].status)
|
||||
.toBe(rollback ? "suspended" : "active");
|
||||
expect(await publications()).toEqual(rollback ? [] : [expect.objectContaining({ idempotencyKey: `run:${runId}:working:${f.endpointId}` })]);
|
||||
await quiet();
|
||||
if (!rollback) {
|
||||
const notices: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, event => notices.push(event));
|
||||
try {
|
||||
await accessService(db).ensureMembership(f.companyId, "user", f.userId, "operator");
|
||||
expect(notices).toEqual([]);
|
||||
} finally { unsubscribe(); }
|
||||
await quiet();
|
||||
}
|
||||
} finally {
|
||||
releaseSuspension();
|
||||
releaseRestoration();
|
||||
await Promise.allSettled([suspending, restoring]);
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["initialization", "steering"] as const)("wakes matching monitor lineage after %s commits, preserving rollback and replay", async kind => {
|
||||
const nextCheckAt = new Date(Date.now() - 1000).toISOString();
|
||||
const sourceId = randomUUID(), successorId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: sourceId, companyId: f.companyId, agentId: f.agentId,
|
||||
nativeIssueId: f.issueId, runtimeMode: "native", status: "succeeded", responsibleUserId: f.userId,
|
||||
contextSnapshot: { issueId: f.issueId, source: "chat:slack", endpointId: f.endpointId, wakeCommentId: f.commentId },
|
||||
resultJson: { semanticToolReceipts: { schedule: { operationId: "set_task_monitor",
|
||||
result: { taskId: f.issueId, monitor: { status: "scheduled", nextCheckAt } } } } },
|
||||
});
|
||||
await db.insert(heartbeatRuns).values({ id: successorId, companyId: f.companyId, agentId: f.agentId,
|
||||
nativeIssueId: f.issueId, runtimeMode: "native", status: "running", responsibleUserId: null,
|
||||
contextSnapshot: { issueId: f.issueId, source: "issue.monitor" },
|
||||
});
|
||||
await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, runId: successorId,
|
||||
source: "automation", reason: "issue_monitor_due", status: "claimed",
|
||||
idempotencyKey: `issue-monitor:${f.issueId}:${nextCheckAt}`, payload: { issueId: f.issueId, nextCheckAt },
|
||||
});
|
||||
const identityInput = { companyId: f.companyId, runId: successorId, issueId: f.issueId, responsibleUserId: f.userId, cause: "dispatch" };
|
||||
if (kind === "steering") {
|
||||
await initializeRunIdentity(db, { ...identityInput, responsibleUserId: null });
|
||||
await db.update(issueComments).set({ authorUserId: f.userId }).where(eq(issueComments.id, f.commentId));
|
||||
}
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: error => errors.push(error) });
|
||||
await chatRunMilestoneWork(db).register(coordinator, () => true).ready;
|
||||
await flush(); expect(await publications()).toEqual([]); await quiet();
|
||||
const pending = kind === "steering" ? await reserveSteeredIdentity(db, {
|
||||
companyId: f.companyId, runId: successorId, issueId: f.issueId, messageId: f.commentId,
|
||||
}) : null;
|
||||
// Reserving an unacknowledged message must not activate its identity.
|
||||
if (kind === "steering") expect(pending?.status).toBe("pending");
|
||||
await flush(); expect(await publications()).toEqual([]); await quiet();
|
||||
const activate = (rollback: boolean) => db.transaction(async tx => {
|
||||
if (kind === "steering") await acceptSteeredIdentity(tx, pending!);
|
||||
else await initializeRunIdentity(tx as unknown as typeof db, identityInput);
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(await publications()).toEqual([]);
|
||||
if (rollback) throw new Error("outer rollback");
|
||||
});
|
||||
await expect(activate(true)).rejects.toThrow("outer rollback");
|
||||
await flush(); expect(await publications()).toEqual([]); await quiet();
|
||||
await activate(false); await flush();
|
||||
expect(await publications()).toEqual([expect.objectContaining({ idempotencyKey: `run:${successorId}:working:${f.endpointId}` })]);
|
||||
await quiet();
|
||||
const notices: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, event => notices.push(event));
|
||||
try {
|
||||
await db.transaction(async tx => {
|
||||
if (kind === "steering") await acceptSteeredIdentity(tx, pending!);
|
||||
else await initializeRunIdentity(tx as unknown as typeof db, identityInput);
|
||||
});
|
||||
expect(notices).toEqual([]);
|
||||
} finally { unsubscribe(); }
|
||||
await flush(); await quiet();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,103 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { chatPublications, createDb, heartbeatRuns, issues } from "@paperclipai/db";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { createChatMilestoneFixtures } from "./helpers/chat-milestone-fixtures.js";
|
||||
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
||||
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
|
||||
import { chatRunMilestoneWork } from "../services/chat-run-milestone-work.js";
|
||||
import { recoveryService } from "../services/recovery/service.js";
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_MILESTONE_RECOVERY_TEST_DATABASE_URL");
|
||||
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
(support.supported ? describe : describe.skip)("chat milestones after run recovery", () => {
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let fixtures: ReturnType<typeof createChatMilestoneFixtures>;
|
||||
let f: Awaited<ReturnType<typeof fixtures.create>>;
|
||||
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
|
||||
const errors: unknown[] = [];
|
||||
beforeAll(async () => {
|
||||
if (!external) temporary = await startEmbeddedPostgresTestDatabase("chat-recovery-milestones-");
|
||||
db = createDb(external ?? temporary!.connectionString, { idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
|
||||
fixtures = createChatMilestoneFixtures(db);
|
||||
}, 60_000);
|
||||
beforeEach(async () => {
|
||||
errors.length = 0; f = await fixtures.create();
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
});
|
||||
afterEach(async () => {
|
||||
await coordinator?.stop(); coordinator = undefined; vi.useRealTimers(); vi.restoreAllMocks();
|
||||
await fixtures.cleanup(); expect(errors).toEqual([]);
|
||||
});
|
||||
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
|
||||
async function flush() {
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 10_000 });
|
||||
}
|
||||
async function quiet() {
|
||||
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
|
||||
try { await vi.advanceTimersByTimeAsync(120_000); expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); }
|
||||
finally { reads.mockRestore(); writes.mockRestore(); }
|
||||
}
|
||||
const publications = () => db.select().from(chatPublications).where(eq(chatPublications.issueId, f.issueId));
|
||||
|
||||
it.each(["process_gone", "issue_cancelled"] as const)("publishes the recovered terminal state after %s and becomes quiet", async reason => {
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: f.agentId,
|
||||
status: "running", runtimeMode: "legacy", responsibleUserId: f.userId,
|
||||
processPid: reason === "process_gone" ? 2_000_000_000 : process.pid,
|
||||
contextSnapshot: { issueId: f.issueId, source: "chat:slack", endpointId: f.endpointId, wakeCommentId: f.commentId },
|
||||
});
|
||||
await db.update(issues).set({ executionRunId: runId,
|
||||
status: reason === "issue_cancelled" ? "cancelled" : "in_progress",
|
||||
}).where(eq(issues.id, f.issueId));
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: error => errors.push(error) });
|
||||
await chatRunMilestoneWork(db).register(coordinator, () => true).ready;
|
||||
await flush();
|
||||
expect(await publications()).toEqual([expect.objectContaining({ idempotencyKey: `run:${runId}:working:${f.endpointId}` })]);
|
||||
await quiet();
|
||||
|
||||
const recovery = recoveryService(db, { enqueueWakeup: vi.fn() });
|
||||
expect(await recovery.sweepStaleIssueLocks()).toEqual({ cleared: 1, issueIds: [f.issueId], terminalizedRunIds: [runId] });
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0].status)
|
||||
.toBe(reason === "process_gone" ? "interrupted" : "cancelled");
|
||||
await flush();
|
||||
expect((await publications()).map(row => row.idempotencyKey).sort()).toEqual([
|
||||
`run:${runId}:failed:${f.endpointId}`, `run:${runId}:working:${f.endpointId}`,
|
||||
].sort());
|
||||
await quiet();
|
||||
expect(await recovery.sweepStaleIssueLocks()).toEqual({ cleared: 0, issueIds: [], terminalizedRunIds: [] });
|
||||
await flush(); await quiet(); expect(await publications()).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("wakes an older failed run when recovery refreshes its quota classification", async () => {
|
||||
const runId = randomUUID(), old = new Date(Date.now() - 48 * 60 * 60_000);
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: f.agentId,
|
||||
status: "failed", runtimeMode: "legacy", responsibleUserId: f.userId,
|
||||
createdAt: old, startedAt: old, finishedAt: old, updatedAt: old,
|
||||
error: "You've hit your usage limit for GPT-5. Try again at 12:00 AM (UTC).", errorCode: "adapter_failed",
|
||||
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
||||
contextSnapshot: { issueId: f.issueId, source: "chat:slack", endpointId: f.endpointId, wakeCommentId: f.commentId },
|
||||
});
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: error => errors.push(error) });
|
||||
await chatRunMilestoneWork(db).register(coordinator, () => true).ready;
|
||||
await flush(); expect(await publications()).toEqual([]); await quiet();
|
||||
const enqueueWakeup = vi.fn(async () => null);
|
||||
const recovery = recoveryService(db, { enqueueWakeup });
|
||||
expect(await recovery.reconcileStrandedAssignedIssues()).toMatchObject({ providerQuotaMonitored: 1 });
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0].errorCode).toBe("provider_quota");
|
||||
expect(enqueueWakeup).not.toHaveBeenCalled();
|
||||
await flush();
|
||||
expect(await publications()).toEqual([expect.objectContaining({ idempotencyKey: `run:${runId}:failed:${f.endpointId}` })]);
|
||||
await quiet();
|
||||
expect(await recovery.reconcileStrandedAssignedIssues()).toMatchObject({ providerQuotaMonitored: 0, skipped: 1 });
|
||||
await flush(); await quiet(); expect(await publications()).toHaveLength(1);
|
||||
});
|
||||
|
||||
});
|
||||
@@ -0,0 +1,125 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { chatConversations, chatDeliveries, chatEndpointResources, chatEndpoints, chatPublications, createDb, heartbeatRuns, issues, subscribeDatabaseWork } from "@paperclipai/db";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { createChatMilestoneFixtures } from "./helpers/chat-milestone-fixtures.js";
|
||||
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
||||
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
|
||||
import { chatRunMilestoneWork } from "../services/chat-run-milestone-work.js";
|
||||
import { chatCredentialMutationLease } from "../services/chat-credential-mutation-lease.js";
|
||||
import { createChatResourceReconciliation } from "../services/chat-channels/resource-reconciliation.js";
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_MILESTONE_RESOURCE_TEST_DATABASE_URL");
|
||||
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
(support.supported ? describe : describe.skip)("chat milestones after resource recovery", () => {
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let fixtures: ReturnType<typeof createChatMilestoneFixtures>;
|
||||
let f: Awaited<ReturnType<typeof fixtures.create>>;
|
||||
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
|
||||
const errors: unknown[] = [];
|
||||
beforeAll(async () => {
|
||||
if (!external) temporary = await startEmbeddedPostgresTestDatabase("chat-resource-milestones-");
|
||||
db = createDb(external ?? temporary!.connectionString, { idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
|
||||
fixtures = createChatMilestoneFixtures(db);
|
||||
}, 60_000);
|
||||
beforeEach(async () => {
|
||||
errors.length = 0; f = await fixtures.create();
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
});
|
||||
afterEach(async () => {
|
||||
await coordinator?.stop(); coordinator = undefined; vi.useRealTimers(); vi.restoreAllMocks();
|
||||
await db.delete(chatDeliveries).where(eq(chatDeliveries.companyId, f.companyId));
|
||||
await fixtures.cleanup(); expect(errors).toEqual([]);
|
||||
});
|
||||
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
|
||||
async function flush() {
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 10_000 });
|
||||
}
|
||||
async function quiet() {
|
||||
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
|
||||
try { await vi.advanceTimersByTimeAsync(120_000); expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); }
|
||||
finally { reads.mockRestore(); writes.mockRestore(); }
|
||||
}
|
||||
const publications = () => db.select().from(chatPublications).where(eq(chatPublications.issueId, f.issueId));
|
||||
|
||||
it.each(["inventory", "lifecycle", "github repository", "telegram migration"] as const)("wakes only reopened conversations after %s commits, preserving rollback and completed history", async path => {
|
||||
const provider = path === "github repository" ? "github" : path === "telegram migration" ? "telegram" : "slack";
|
||||
const resourceType = provider === "github" ? "repository" : provider === "telegram" ? "chat" : "channel";
|
||||
const providerResourceId = provider === "github" ? "owner/repo" : provider === "telegram" ? "-100222" : "recovered-channel";
|
||||
await db.update(chatEndpoints).set({ provider }).where(eq(chatEndpoints.id, f.endpointId));
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId: f.companyId, agentId: f.agentId,
|
||||
responsibleUserId: f.userId, status: "failed", errorCode: "test_run_failed", updatedAt: new Date(),
|
||||
contextSnapshot: { issueId: f.issueId, source: `chat:${provider}`, endpointId: f.endpointId, wakeCommentId: f.commentId },
|
||||
});
|
||||
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, f.issueId));
|
||||
const [endpoint] = await db.select().from(chatEndpoints).where(eq(chatEndpoints.id, f.endpointId));
|
||||
const [resource] = await db.insert(chatEndpointResources).values({ companyId: f.companyId, endpointId: f.endpointId,
|
||||
type: resourceType, providerResourceId: provider === "telegram" ? "-111" : providerResourceId, label: "Recovered", availability: "unavailable", enabled: true,
|
||||
}).returning();
|
||||
await db.update(chatConversations).set({ resourceId: resource.id, state: "unavailable", ...(provider === "telegram" ? { externalConversationId: "telegram:-111", externalThreadId: "telegram:-111:42" } : {}) }).where(eq(chatConversations.id, f.conversationId));
|
||||
const completedId = randomUUID();
|
||||
await db.insert(chatConversations).values({ id: completedId, companyId: f.companyId, endpointId: f.endpointId,
|
||||
// Telegram migration intentionally rebinds all conversations on the old resource.
|
||||
// Unrelated completed history must remain passive.
|
||||
resourceId: provider === "telegram" ? null : resource.id, issueId: f.issueId, state: "completed", externalConversationId: completedId, externalThreadId: completedId, externalLabel: "History" });
|
||||
const [delivery] = await db.insert(chatDeliveries).values({ companyId: f.companyId, endpointId: f.endpointId,
|
||||
providerEventId: randomUUID(), deduplicationKey: randomUUID(), eventKind: "installation", normalizedEvent: {}, state: "processing", attempts: 1,
|
||||
}).returning();
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, canReconcile: () => true, onError: error => errors.push(error) });
|
||||
await chatRunMilestoneWork(db).register(coordinator, () => true).ready;
|
||||
await flush(); expect(await publications()).toEqual([]); await quiet();
|
||||
const withLease = chatCredentialMutationLease(db);
|
||||
const recover = (rollback: boolean, alreadyPublished = false) => withLease(endpoint, lease => db.transaction(async tx => {
|
||||
const resources = createChatResourceReconciliation(tx as unknown as typeof db, {
|
||||
endpointRecord: async () => ({ endpoint }), withCredentialMutationLease: withLease, runtimeGeneration: () => 0, maxErrorText: 1000,
|
||||
});
|
||||
if (path === "inventory") {
|
||||
await resources.reconcileProviderResourceRows(endpoint, { provider, resources: [{ type: resourceType, providerResourceId, label: "Recovered" }] }, lease);
|
||||
} else if (path === "github repository") {
|
||||
await resources.reconcileGitHubWebhookRepository(endpoint, { repository: { id: 123, full_name: providerResourceId, name: "repo", owner: { id: 321, login: "owner" } } }, lease);
|
||||
} else {
|
||||
await resources.applyProviderAvailability({ currentEndpoint: endpoint, credentialLease: lease, staleResourceEffect: false,
|
||||
claimed: delivery, candidate: { attempts: 0 }, effect: { kind: "resource", provider, providerEventId: delivery.providerEventId,
|
||||
providerResourceId, resourceType, ...(provider === "telegram" ? { previousProviderResourceId: "-111" } : {}), label: "Recovered", availability: "available" },
|
||||
});
|
||||
}
|
||||
// The owner used a nested transaction. Its savepoint release cannot publish.
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(await publications()).toHaveLength(alreadyPublished ? 1 : 0);
|
||||
if (rollback) throw new Error("outer rollback");
|
||||
}));
|
||||
await expect(recover(true)).rejects.toThrow("outer rollback");
|
||||
await flush(); expect(await publications()).toEqual([]); await quiet();
|
||||
await recover(false); await flush();
|
||||
expect(await publications()).toEqual([expect.objectContaining({ idempotencyKey: `run:${runId}:failed:${f.endpointId}` })]);
|
||||
expect((await db.select().from(chatConversations).where(eq(chatConversations.id, completedId)))[0].state).toBe("completed");
|
||||
if (provider === "telegram") {
|
||||
expect((await db.select().from(chatConversations).where(eq(chatConversations.id, f.conversationId)))[0])
|
||||
.toMatchObject({ externalConversationId: "telegram:-100222", externalThreadId: "telegram:-100222:42", state: "active" });
|
||||
}
|
||||
await quiet();
|
||||
// Unchanged inventory must not turn a provider refresh into milestone polling.
|
||||
const notices: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, event => notices.push(event));
|
||||
try {
|
||||
await withLease(endpoint, lease => createChatResourceReconciliation(db, {
|
||||
endpointRecord: async () => ({ endpoint }), withCredentialMutationLease: withLease, runtimeGeneration: () => 0, maxErrorText: 1000,
|
||||
}).reconcileProviderResourceRows(endpoint, { provider, resources: [{ type: resourceType, providerResourceId, label: "Recovered" }] }, lease));
|
||||
expect(notices).toEqual([]);
|
||||
if (provider === "telegram") {
|
||||
// A duplicate migration has no old bindings left to reactivate.
|
||||
await recover(false, true);
|
||||
expect(notices).toEqual([]);
|
||||
}
|
||||
} finally { unsubscribe(); }
|
||||
await flush(); await quiet();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,987 @@
|
||||
import { documentService } from "../services/documents.js";
|
||||
import { createChatMilestoneFixtures } from "./helpers/chat-milestone-fixtures.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { agents, agentWakeupRequests, caseHandoffs, cases, chatConversations, chatEndpoints, chatPublications, completionContracts, createDb, nativeRunFinalizations, nativeRunResults, statusDecisions, statusDecisionEffects, workAssessments, subscribeDatabaseWork, heartbeatRuns, issueComments, issueQuestionResponseDeliveries, issueThreadInteractions, issues } from "@paperclipai/db";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
||||
import { chatRunMilestoneWork } from "../services/chat-run-milestone-work.js";
|
||||
import { notifyChatRunMilestones } from "../services/chat-run-milestone-notifications.js";
|
||||
import { issueThreadInteractionService } from "../services/issue-thread-interactions.js";
|
||||
import { questionResponseDeliveryService } from "../services/question-response-delivery.js";
|
||||
import { createQueuedCommentIssueLockWriter } from "../modules/wake-queue/adapters/queued-comment-postgres.js";
|
||||
import { createDiscardQueuedComment, createReorderQueuedComments } from "../modules/wake-queue/application/queued-comment-use-cases.js";
|
||||
import { createRunDispatch } from "../modules/run-dispatch/index.js";
|
||||
import { createAdmissionTransactionScope, createPostgresWakeQueueAdapter, createWakeAdmissionWriter } from "../modules/wake-queue/adapters/postgres.js";
|
||||
import { instanceSettingsService } from "../services/instance-settings.js";
|
||||
import { createHeartbeatQueue } from "../services/heartbeat/queue.js";
|
||||
import { createHeartbeatQueueTestDependencies } from "./helpers/heartbeat-queue.js";
|
||||
import { CHAT_CONTROL_RECOVERY_ADMISSION_KEY, CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE } from "../services/chat-control-recovery-stop.js";
|
||||
import { heartbeatService } from "../services/heartbeat.js";
|
||||
import { commitNativeStatusDecision, dispatchPendingNativeStatusEffects } from "../services/native-runtime/status-decision-committer.js";
|
||||
import { claimQueuedNativeReviewRun } from "../services/native-runtime/native-review-dispatch.js";
|
||||
import { createChatPublicationResolution } from "../services/chat-channels/publications/resolution.js";
|
||||
import { createChatPublicationWorker } from "../services/chat-channels/publications/worker.js";
|
||||
import { CAPABILITIES } from "../services/chat-channels/provider-catalog.js";
|
||||
import { createChatPublicationFailures } from "../services/chat-channels/publications/failures.js";
|
||||
import { PaperclipRunnerToolAuthority } from "../services/native-runtime/paperclip-runner-tool-authority.js";
|
||||
import { finalizeNativeRun } from "../services/native-runtime/native-run-finalizer.js";
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_MILESTONE_TEST_DATABASE_URL");
|
||||
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
(support.supported ? describe : describe.skip)("chat lifecycle milestone worker", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
|
||||
let enabled = true;
|
||||
const errors: unknown[] = [];
|
||||
let fixtures: ReturnType<typeof createChatMilestoneFixtures>;
|
||||
type Fixture = Awaited<ReturnType<typeof fixture>>;
|
||||
let f: Fixture;
|
||||
|
||||
beforeAll(async () => {
|
||||
if (!external) temporary = await startEmbeddedPostgresTestDatabase("chat-milestones-");
|
||||
db = createDb(external ?? temporary!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
|
||||
fixtures = createChatMilestoneFixtures(db);
|
||||
}, 60_000);
|
||||
const fixture = (companyId?: string) => fixtures.create(companyId);
|
||||
beforeEach(async () => {
|
||||
enabled = true; errors.length = 0; f = await fixture();
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
});
|
||||
afterEach(async () => {
|
||||
await coordinator?.stop(); coordinator = undefined; vi.useRealTimers(); vi.restoreAllMocks();
|
||||
await fixtures.cleanup();
|
||||
expect(errors).toEqual([]);
|
||||
});
|
||||
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
|
||||
function runValues(scope = f, patch: Partial<typeof heartbeatRuns.$inferInsert> = {}) {
|
||||
return { id: randomUUID(), companyId: scope.companyId, agentId: scope.agentId, responsibleUserId: scope.userId, status: "running", updatedAt: new Date(),
|
||||
contextSnapshot: { issueId: scope.issueId, source: "chat:slack", endpointId: scope.endpointId, wakeCommentId: scope.commentId }, ...patch };
|
||||
}
|
||||
async function run(scope = f, patch: Partial<typeof heartbeatRuns.$inferInsert> = {}) {
|
||||
const values = runValues(scope, patch);
|
||||
await db.transaction(async tx => { await notifyChatRunMilestones(tx, scope); await tx.insert(heartbeatRuns).values(values); });
|
||||
return values;
|
||||
}
|
||||
async function start() {
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => enabled, canReconcile: () => true, onError: error => errors.push(error) });
|
||||
await chatRunMilestoneWork(db).register(coordinator, () => enabled).ready;
|
||||
}
|
||||
async function flush() {
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 10_000 });
|
||||
}
|
||||
async function quiet(ms = 120_000) {
|
||||
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
|
||||
try { await vi.advanceTimersByTimeAsync(ms); expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); }
|
||||
finally { reads.mockRestore(); writes.mockRestore(); }
|
||||
}
|
||||
const publications = (scope = f) => db.select().from(chatPublications).where(eq(chatPublications.issueId, scope.issueId));
|
||||
|
||||
it("leaves configured idle, internal runs and old history quiet", async () => {
|
||||
await run(f, { contextSnapshot: { issueId: f.issueId, source: "issue.comment" } });
|
||||
await run(f, { updatedAt: new Date(Date.now() - 2 * 86_400_000) });
|
||||
await start(); await flush(); expect(await publications()).toEqual([]); await quiet();
|
||||
});
|
||||
it("reconstructs missing notices at startup and leaves published history quiet across restart", async () => {
|
||||
const row = await run(); await start(); await flush();
|
||||
expect(await publications()).toEqual([expect.objectContaining({ idempotencyKey: `run:${row.id}:working:${f.endpointId}` })]);
|
||||
await quiet(); await coordinator!.stop(); await start(); await flush(); await quiet();
|
||||
expect(await publications()).toHaveLength(1);
|
||||
});
|
||||
it("wakes on the first committed row after becoming quiet", async () => {
|
||||
await start(); await flush(); await quiet(); await run(); await flush();
|
||||
expect(await publications()).toHaveLength(1); await quiet();
|
||||
});
|
||||
it("wakes from the real run cancellation transaction without a manual milestone signal", async () => {
|
||||
const row = await run(f, { status: "queued", runtimeMode: "legacy" });
|
||||
await start(); await flush(); await quiet();
|
||||
// This fixture exercises the Stop producer, not successor execution.
|
||||
await heartbeatService(db).cancelRun(row.id, "Operator stopped queued work", { suppressImmediateRecovery: true });
|
||||
await flush();
|
||||
expect((await publications()).map(publication => publication.idempotencyKey).sort()).toEqual([
|
||||
`run:${row.id}:failed:${f.endpointId}`, `run:${row.id}:queued:${f.endpointId}`,
|
||||
].sort());
|
||||
await quiet();
|
||||
});
|
||||
it.each(["promote", "suppress"] as const)("wakes when real retry dispatch chooses to %s", async action => {
|
||||
const row = await run(f, { status: "scheduled_retry", runtimeMode: "legacy", scheduledRetryAt: new Date(Date.now() - 1000), scheduledRetryReason: "transient_error" });
|
||||
if (action === "suppress") await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, f.issueId));
|
||||
await start(); await flush(); expect(await publications()).toEqual([]); await quiet();
|
||||
const dispatch = createRunDispatch(db);
|
||||
const result = await dispatch.promoteScheduledRetry({ companyId: f.companyId, runId: row.id });
|
||||
expect(result.outcome).toBe(action === "promote" ? "promoted" : "gate_suppressed");
|
||||
await flush();
|
||||
expect((await publications()).map(publication => publication.idempotencyKey)).toContain(
|
||||
`run:${row.id}:${action === "promote" ? "queued" : "failed"}:${f.endpointId}`,
|
||||
);
|
||||
await quiet();
|
||||
expect((await dispatch.promoteScheduledRetry({ companyId: f.companyId, runId: row.id })).outcome).toBe("not_promoted");
|
||||
await quiet();
|
||||
});
|
||||
it("wakes when the real staleness gate cancels queued work", async () => {
|
||||
const row = await run(f, { status: "queued", runtimeMode: "legacy", scheduledRetryReason: "max_turns_continuation" });
|
||||
await start(); await flush(); await quiet();
|
||||
await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, f.issueId));
|
||||
const dispatch = createRunDispatch(db);
|
||||
await dispatch.cancelStaleQueuedRun({ companyId: f.companyId, runId: row.id, expectedStatus: "queued" });
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, row.id)))[0].status).toBe("cancelled");
|
||||
await flush();
|
||||
expect((await publications()).map(publication => publication.idempotencyKey)).toContain(`run:${row.id}:failed:${f.endpointId}`);
|
||||
await quiet();
|
||||
await dispatch.cancelStaleQueuedRun({ companyId: f.companyId, runId: row.id, expectedStatus: "queued" });
|
||||
await quiet();
|
||||
});
|
||||
it.each(["deferred", "immediate"] as const)("wakes when real %s recovery admits chat work", async kind => {
|
||||
const finishing = await run(f, { status: "succeeded", runtimeMode: "legacy", finishedAt: new Date(),
|
||||
contextSnapshot: { issueId: f.issueId, source: "issue_assigned" },
|
||||
});
|
||||
const wakeId = randomUUID();
|
||||
if (kind === "deferred") await db.insert(agentWakeupRequests).values({ id: wakeId, companyId: f.companyId, agentId: f.agentId,
|
||||
source: "on_demand", status: "deferred_issue_execution", payload: { issueId: f.issueId },
|
||||
});
|
||||
const adapter = createPostgresWakeQueueAdapter(db, {
|
||||
resolveResponsibleUserId: async () => f.userId,
|
||||
getRoutineEnv: async () => ({ routineId: null, env: null, responsibleUserId: null }),
|
||||
resolveSessionBeforeForWakeup: async () => null,
|
||||
});
|
||||
await start(); await flush(); expect(await publications()).toEqual([]); await quiet();
|
||||
let admittedId: string | undefined;
|
||||
await adapter.withIssueExecutionLock({ companyId: f.companyId, runId: finishing.id, now: new Date() }, async (locked, ports) => {
|
||||
const input = { companyId: f.companyId, issue: locked.primaryIssue, finishingRun: locked.run,
|
||||
contextSnapshot: runValues().contextSnapshot, responsibleUserId: f.userId, sessionBefore: null, now: new Date(),
|
||||
};
|
||||
const agent = { id: f.agentId, companyId: f.companyId, name: "Maya", invokable: true };
|
||||
if (kind === "deferred") {
|
||||
expect(await ports.transaction.claimDeferredWakeForPromotion({ companyId: f.companyId, wakeId, now: new Date() })).toBe(true);
|
||||
admittedId = (await ports.transaction.finalizePromotedWake({ ...input, wakeId, deferredAgent: agent,
|
||||
reason: "issue_execution_promoted", source: "on_demand", triggerDetail: null, payload: {},
|
||||
})).id;
|
||||
} else {
|
||||
admittedId = (await ports.transaction.queueImmediateRecoveryRun({ ...input, recoveryAgent: agent, reason: "issue_execution_recovery" })).id;
|
||||
}
|
||||
expect(await publications()).toEqual([]);
|
||||
return { outcome: { kind: "released" }, postCommitEffects: [] };
|
||||
});
|
||||
expect(admittedId).toBeDefined();
|
||||
await flush();
|
||||
expect((await publications()).map(publication => publication.idempotencyKey)).toContain(`run:${admittedId}:queued:${f.endpointId}`);
|
||||
await quiet();
|
||||
});
|
||||
async function queue() {
|
||||
const dependencies = createHeartbeatQueueTestDependencies(db);
|
||||
const [agent] = await db.select().from(agents).where(eq(agents.id, f.agentId));
|
||||
dependencies.getAgent.mockResolvedValue(agent);
|
||||
dependencies.instanceSettings.getExperimental.mockImplementation(() => instanceSettingsService(db).getExperimental());
|
||||
return { service: createHeartbeatQueue(db, dependencies), dependencies };
|
||||
}
|
||||
it("wakes on real queue admission while execution is held by drain", async () => {
|
||||
const { service, dependencies } = await queue();
|
||||
dependencies.getSchedulingSuppression.mockResolvedValue({ suppressed: true, reason: "task_drain" });
|
||||
await start(); await flush(); await quiet();
|
||||
const admitted = await service.enqueueWakeup(f.agentId, { source: "on_demand", reason: "issue_commented",
|
||||
contextSnapshot: runValues().contextSnapshot, requestedByActorType: "user", requestedByActorId: f.userId,
|
||||
});
|
||||
expect(admitted).toMatchObject({ status: "queued", companyId: f.companyId });
|
||||
await flush();
|
||||
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${admitted!.id}:queued:${f.endpointId}`);
|
||||
expect(dependencies.executeRun).not.toHaveBeenCalled();
|
||||
await quiet();
|
||||
});
|
||||
it.each(["ordinary", "legacy_comments", "queued_comments"] as const)("wakes on a real %s queue claim", async kind => {
|
||||
const wakeId = randomUUID();
|
||||
const context = runValues().contextSnapshot;
|
||||
await db.insert(agentWakeupRequests).values({ id: wakeId, companyId: f.companyId, agentId: f.agentId,
|
||||
source: "on_demand", status: "queued",
|
||||
payload: kind === "queued_comments" ? { _paperclipWakeContext: { wakeCommentIds: [f.commentId] } } : {},
|
||||
});
|
||||
const seeded = await run(f, { status: "queued", runtimeMode: "legacy", wakeupRequestId: wakeId,
|
||||
contextSnapshot: kind === "ordinary" ? context : { ...context, wakeCommentIds: [f.commentId] },
|
||||
});
|
||||
await db.update(agentWakeupRequests).set({ runId: seeded.id }).where(eq(agentWakeupRequests.id, wakeId));
|
||||
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.id));
|
||||
const { service, dependencies } = await queue();
|
||||
await start(); await flush(); await quiet();
|
||||
expect(await service.claimQueuedRun(before)).toMatchObject({ id: seeded.id, status: "running" });
|
||||
await flush();
|
||||
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${seeded.id}:working:${f.endpointId}`);
|
||||
expect(dependencies.executeRun).not.toHaveBeenCalled();
|
||||
await quiet();
|
||||
expect(await service.claimQueuedRun(before)).toBeNull();
|
||||
await quiet();
|
||||
});
|
||||
it("wakes when claim rejects a superseded case handoff and leaves repeated claims quiet", async () => {
|
||||
const wakeId = randomUUID();
|
||||
await db.insert(agentWakeupRequests).values({ id: wakeId, companyId: f.companyId, agentId: f.agentId,
|
||||
source: "assignment", status: "queued", payload: { issueId: f.issueId },
|
||||
});
|
||||
const seeded = await run(f, { status: "queued", runtimeMode: "legacy", wakeupRequestId: wakeId });
|
||||
const [record] = await db.insert(cases).values({ companyId: f.companyId, caseNumber: 1, identifier: "M-CASE-1",
|
||||
caseType: "work", title: "Chat task", workIssueId: f.issueId, handoffVersion: 2 }).returning();
|
||||
await db.insert(caseHandoffs).values({ companyId: f.companyId, caseId: record.id, issueId: f.issueId,
|
||||
version: 1, toAgentId: f.agentId, phase: "implementation", instructions: "Build the change", actorType: "system",
|
||||
actorId: "test", idempotencyKey: "superseded", inputFingerprint: "fixture", wakeupRequestId: wakeId,
|
||||
expectedIssueStatusVersion: 0, appliedIssueStatusVersion: 0, state: "delivered" });
|
||||
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.id));
|
||||
const { service, dependencies } = await queue();
|
||||
await start(); await flush(); await quiet();
|
||||
expect(await service.claimQueuedRun(before)).toBeNull();
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.id)))[0])
|
||||
.toMatchObject({ status: "cancelled", errorCode: "case_handoff_superseded" });
|
||||
await flush();
|
||||
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${seeded.id}:failed:${f.endpointId}`);
|
||||
expect(dependencies.executeRun).not.toHaveBeenCalled();
|
||||
await quiet();
|
||||
expect(await service.claimQueuedRun(before)).toBeNull();
|
||||
await quiet();
|
||||
});
|
||||
it("wakes when dispatch fails an unverifiable chat recovery and leaves repeated gates quiet", async () => {
|
||||
const seeded = await run(f, { status: "running", runtimeMode: "legacy", invocationSource: "automation",
|
||||
runnerProfileJson: { [CHAT_CONTROL_RECOVERY_ADMISSION_KEY]: {} },
|
||||
});
|
||||
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.id));
|
||||
const { service, dependencies } = await queue();
|
||||
const dispatch = vi.fn(async () => null);
|
||||
await start(); await flush(); await quiet();
|
||||
await service.withChatControlRecoveryGate(before, "dispatch", dispatch);
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.id)))[0])
|
||||
.toMatchObject({ status: "failed", errorCode: CHAT_CONTROL_RECOVERY_UNRESOLVED_CODE });
|
||||
expect(dispatch).not.toHaveBeenCalled(); expect(dependencies.executeRun).not.toHaveBeenCalled();
|
||||
await flush();
|
||||
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${seeded.id}:failed:${f.endpointId}`);
|
||||
await quiet();
|
||||
await service.withChatControlRecoveryGate(before, "dispatch", dispatch);
|
||||
await quiet();
|
||||
});
|
||||
it.each(["queued", "scheduled_retry"] as const)("publishes cancellation when admission replaces a reassigned %s owner", async status => {
|
||||
const old = await run(f, { status, runtimeMode: "legacy", scheduledRetryAt: status === "scheduled_retry" ? new Date(Date.now() + 60_000) : null });
|
||||
const [replacement] = await db.insert(agents).values({ companyId: f.companyId, name: "Replacement", adapterType: "paperclip_runner" }).returning();
|
||||
await db.update(issues).set({ assigneeAgentId: replacement.id, executionRunId: old.id }).where(eq(issues.id, f.issueId));
|
||||
const { service, dependencies } = await queue();
|
||||
dependencies.getAgent.mockResolvedValue(replacement);
|
||||
dependencies.getSchedulingSuppression.mockResolvedValue({ suppressed: true, reason: "task_drain" });
|
||||
await start(); await flush(); await quiet();
|
||||
const admitted = await service.enqueueWakeup(replacement.id, { source: "on_demand", reason: "issue_commented",
|
||||
contextSnapshot: runValues().contextSnapshot, requestedByActorType: "user", requestedByActorId: f.userId,
|
||||
});
|
||||
expect(admitted).toMatchObject({ agentId: replacement.id, status: "queued" });
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, old.id)))[0])
|
||||
.toMatchObject({ status: "cancelled", errorCode: status === "queued" ? "lock_released_on_reassignment" : "issue_reassigned" });
|
||||
await flush();
|
||||
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${old.id}:failed:${f.endpointId}`);
|
||||
expect(dependencies.executeRun).not.toHaveBeenCalled(); await quiet();
|
||||
});
|
||||
async function editableQueue(staleContext = false) {
|
||||
const wakeId = randomUUID();
|
||||
await db.update(issueComments).set({ authorUserId: f.userId }).where(eq(issueComments.id, f.commentId));
|
||||
await db.insert(agentWakeupRequests).values({ id: wakeId, companyId: f.companyId, agentId: f.agentId,
|
||||
source: "on_demand", status: "queued", payload: { issueId: f.issueId, _paperclipWakeContext: { wakeCommentIds: [f.commentId] } },
|
||||
});
|
||||
const seeded = await run(f, { status: "queued", runtimeMode: "legacy", wakeupRequestId: wakeId,
|
||||
contextSnapshot: { ...runValues().contextSnapshot, wakeCommentId: staleContext ? randomUUID() : f.commentId },
|
||||
});
|
||||
await db.update(agentWakeupRequests).set({ runId: seeded.id }).where(eq(agentWakeupRequests.id, wakeId));
|
||||
const issue = { id: f.issueId, companyId: f.companyId, assigneeAgentId: f.agentId, executionRunId: seeded.id };
|
||||
const actor = { actorType: "user" as const, actorId: f.userId, agentId: null, runId: null, agentApiKeyId: null };
|
||||
const issueLock = createQueuedCommentIssueLockWriter(db, {
|
||||
syncCommentReferences: async () => {}, deleteCommentReferenceSource: async () => {}, syncCommentExternalObjectsSafely: async () => {},
|
||||
});
|
||||
const snapshot = await issueLock.withLockedQueue({ issue, actor, queueId: wakeId }, async locked => locked.queue);
|
||||
return { seeded, issueLock, input: { issue, actor, queueId: wakeId, revision: snapshot.revision, now: new Date() } };
|
||||
}
|
||||
it("wakes when queued-comment mutation reconstructs run lineage from its authoritative queue", async () => {
|
||||
const { seeded, issueLock, input } = await editableQueue(true);
|
||||
await start(); await flush(); expect(await publications()).toEqual([]); await quiet();
|
||||
await createReorderQueuedComments({ issueLock })({ ...input, orderedCommentIds: [f.commentId] });
|
||||
await flush();
|
||||
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${seeded.id}:queued:${f.endpointId}`);
|
||||
await quiet();
|
||||
});
|
||||
it("settles queued-comment discard intent only after the owning transaction commits", async () => {
|
||||
const { seeded, input } = await editableQueue();
|
||||
const events: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, "chat-run-milestones", event => events.push(event));
|
||||
try {
|
||||
await db.transaction(async tx => {
|
||||
const writer = createQueuedCommentIssueLockWriter(tx as unknown as typeof db, {
|
||||
syncCommentReferences: async () => {}, deleteCommentReferenceSource: async () => {}, syncCommentExternalObjectsSafely: async () => {},
|
||||
});
|
||||
await createDiscardQueuedComment({ issueLock: writer })({ ...input, commentId: f.commentId });
|
||||
expect(events).toContain("intent"); expect(events).not.toContain("settled");
|
||||
});
|
||||
expect(events).toContain("settled");
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.id)))[0])
|
||||
.toMatchObject({ status: "cancelled", errorCode: "queued_comment_discarded" });
|
||||
} finally { unsubscribe(); }
|
||||
});
|
||||
it("wakes when the real coalescing writer gives an existing run chat lineage", async () => {
|
||||
const seeded = await run(f, { contextSnapshot: { issueId: f.issueId, source: "issue_assigned" } });
|
||||
await start(); await flush(); expect(await publications()).toEqual([]); await quiet();
|
||||
await db.transaction(async tx => {
|
||||
await createWakeAdmissionWriter().coalesceIntoActiveExecutionRun(
|
||||
createAdmissionTransactionScope(f.companyId, tx as unknown as typeof db), {
|
||||
companyId: f.companyId, activeExecutionRunId: seeded.id, mergedContextSnapshot: runValues().contextSnapshot,
|
||||
agentId: f.agentId, source: "on_demand", triggerDetail: null, payload: { issueId: f.issueId },
|
||||
requestedByActorType: "user", requestedByActorId: f.userId, idempotencyKey: null,
|
||||
});
|
||||
expect(await publications()).toEqual([]);
|
||||
});
|
||||
await flush();
|
||||
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${seeded.id}:working:${f.endpointId}`);
|
||||
await quiet();
|
||||
});
|
||||
it("wakes on a claim released for drain and leaves repeated release quiet", async () => {
|
||||
const seeded = await run();
|
||||
const { service } = await queue();
|
||||
await start(); await flush(); await quiet();
|
||||
await service.releaseRunClaimedJustBeforeSuppression(seeded.id);
|
||||
await flush();
|
||||
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${seeded.id}:queued:${f.endpointId}`);
|
||||
await quiet();
|
||||
await service.releaseRunClaimedJustBeforeSuppression(seeded.id);
|
||||
await quiet();
|
||||
});
|
||||
it("wakes an admitted monitor through its previously committed source receipt", async () => {
|
||||
const source = await run(f, { runtimeMode: "native", nativeIssueId: f.issueId });
|
||||
await db.update(issues).set({ executionRunId: source.id }).where(eq(issues.id, f.issueId));
|
||||
await start(); await flush(); await quiet();
|
||||
const nextCheckAt = new Date(Date.now() + 60_000).toISOString();
|
||||
const authority = new PaperclipRunnerToolAuthority(db, { ...f, runId: source.id });
|
||||
const request = { tool: "set_task_monitor", callId: randomUUID(), arguments: {
|
||||
idempotencyKey: "milestone-monitor", monitor: { nextCheckAt, notes: "Check the saved task" },
|
||||
} };
|
||||
await expect(authority.execute(request)).resolves.toMatchObject({ taskId: f.issueId, replayed: false });
|
||||
// The monitor and source receipt commit together, before due admission. This
|
||||
// fixture finishes the source without executing a harness; admission below
|
||||
// uses the real queue, with execution held by task drain.
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, source.id));
|
||||
await db.update(issues).set({ executionRunId: null }).where(eq(issues.id, f.issueId));
|
||||
await quiet();
|
||||
const { service, dependencies } = await queue();
|
||||
dependencies.getSchedulingSuppression.mockResolvedValue({ suppressed: true, reason: "task_drain" });
|
||||
const admitted = await service.enqueueWakeup(f.agentId, { source: "automation", reason: "issue_monitor_due",
|
||||
idempotencyKey: `issue-monitor:${f.issueId}:${nextCheckAt}`,
|
||||
payload: { issueId: f.issueId, nextCheckAt },
|
||||
contextSnapshot: { issueId: f.issueId, source: "issue.monitor" },
|
||||
requestedByActorType: "user", requestedByActorId: f.userId,
|
||||
});
|
||||
expect(admitted).toMatchObject({ status: "queued" });
|
||||
await flush();
|
||||
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${admitted!.id}:queued:${f.endpointId}`);
|
||||
expect(dependencies.executeRun).not.toHaveBeenCalled();
|
||||
await quiet();
|
||||
});
|
||||
async function nativeFixture(phase = "committed") {
|
||||
const contractId = randomUUID(), resultId = randomUUID();
|
||||
await db.insert(completionContracts).values({
|
||||
id: contractId, companyId: f.companyId, issueId: f.issueId, revision: 1,
|
||||
schemaVersion: "paperclip.completion-contract.v1", policyVersion: "phase6-v1", risk: "standard",
|
||||
completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal",
|
||||
contractJson: { objective: "Respond to chat" }, canonicalSha256: "milestone-contract",
|
||||
createdByActorType: "system", createdByActorId: "test",
|
||||
});
|
||||
// Persisted audit-only routing can commit without a status decision. Recovery
|
||||
// still owes the terminal run projection after the controller restarts.
|
||||
const row = await run(f, { runtimeMode: "native", nativeIssueId: f.issueId, completionContractId: contractId,
|
||||
resultJson: { nativeAttentionRouting: [{ decisionId: null, reasonCode: "attention_duplicate_suppressed",
|
||||
materializedTargets: [{ kind: "existing_attention" }] }] },
|
||||
});
|
||||
await db.insert(nativeRunResults).values({ id: resultId, companyId: f.companyId, issueId: f.issueId,
|
||||
runId: row.id, completionContractId: contractId, serverFingerprint: resultId, schemaStatus: "accepted",
|
||||
resultJson: { terminal: { runTerminalState: "cancelled" } }, canonicalSha256: "milestone-result",
|
||||
});
|
||||
await db.insert(nativeRunFinalizations).values({ runId: row.id, companyId: f.companyId, issueId: f.issueId,
|
||||
phase, resultId,
|
||||
});
|
||||
return { row, contractId, resultId };
|
||||
}
|
||||
async function continuationDecisionFixture() {
|
||||
const native = await nativeFixture("assessing");
|
||||
const assessmentId = randomUUID();
|
||||
await db.insert(workAssessments).values({ id: assessmentId, companyId: f.companyId, issueId: f.issueId,
|
||||
runId: native.row.id, contractId: native.contractId, resultId: native.resultId, triggerKind: "run_result",
|
||||
triggerActorCompanyId: f.companyId, priorIssueStatus: "in_progress", priorStatusVersion: 0,
|
||||
policyVersion: "phase6-v1", assessmentJson: {}, inputDigest: assessmentId,
|
||||
});
|
||||
return { companyId: f.companyId, issueId: f.issueId, runId: native.row.id, assessmentId,
|
||||
priorStatus: "in_progress", priorStatusVersion: 0, priorDecisionId: null,
|
||||
decision: { policyVersion: "phase6-v1", statusAction: "preserve" as const, toStatus: "in_progress" as const,
|
||||
reasonCode: "live_continuation_registered", unblockDescriptor: null,
|
||||
effects: [{ kind: "enqueue_continuation" as const, agentId: f.agentId, continuationKind: "same_agent" as const,
|
||||
summary: "Continue responding", idempotencyKey: `continuation:${native.row.id}` }],
|
||||
},
|
||||
};
|
||||
}
|
||||
it("wakes from a governed native reviewer claim and leaves repeated claims quiet", async () => {
|
||||
const input = await continuationDecisionFixture();
|
||||
const { decision } = await commitNativeStatusDecision({ db, ...input });
|
||||
const [reviewer] = await db.insert(agents).values({ companyId: f.companyId, name: "Reviewer", adapterType: "paperclip_runner" }).returning();
|
||||
await db.update(statusDecisions).set({ decisionJson: { ...decision.decisionJson, projectedStatusVersion: 1 } }).where(eq(statusDecisions.id, decision.id));
|
||||
await db.update(issues).set({ status: "in_review", statusVersion: 1, lastStatusDecisionId: decision.id, executionRunId: null }).where(eq(issues.id, f.issueId));
|
||||
await db.update(chatEndpoints).set({ assignedAgentId: reviewer.id }).where(eq(chatEndpoints.id, f.endpointId));
|
||||
const [interaction] = await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId: f.issueId,
|
||||
kind: "request_confirmation", status: "pending", sourceRunId: input.runId, addresseeAgentId: reviewer.id,
|
||||
effectiveResolverPolicy: "anyone", resolverPolicyProvenance: "inherited",
|
||||
payload: { version: 1, prompt: "Review the work", target: { type: "custom", key: "native_completion_review", revisionId: decision.id } },
|
||||
}).returning();
|
||||
const wakeId = randomUUID();
|
||||
await db.insert(agentWakeupRequests).values({ id: wakeId, companyId: f.companyId, agentId: reviewer.id, source: "automation", status: "queued" });
|
||||
const seeded = await run(f, { agentId: reviewer.id, status: "queued", runtimeMode: "native", nativeIssueId: f.issueId, wakeupRequestId: wakeId,
|
||||
contextSnapshot: { ...runValues().contextSnapshot, nativeReviewInteractionId: interaction.id, nativeReviewDecisionId: decision.id },
|
||||
});
|
||||
await db.update(agentWakeupRequests).set({ runId: seeded.id }).where(eq(agentWakeupRequests.id, wakeId));
|
||||
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.id));
|
||||
const claim = { run: before, agentNameKey: "reviewer", claimedAt: new Date(), claimValues: {} };
|
||||
await start(); await flush(); await quiet();
|
||||
expect(await claimQueuedNativeReviewRun(db, claim)).toMatchObject({ id: seeded.id, status: "running" });
|
||||
await flush();
|
||||
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${seeded.id}:working:${f.endpointId}`);
|
||||
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0].executionRunId).toBe(seeded.id);
|
||||
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wakeId)))[0].status).toBe("claimed");
|
||||
await quiet();
|
||||
expect(await claimQueuedNativeReviewRun(db, claim)).toBeNull(); await quiet();
|
||||
});
|
||||
it("wakes from native committed-result projection and leaves repeated finalization quiet", async () => {
|
||||
const { row } = await nativeFixture();
|
||||
await start(); await flush(); await quiet();
|
||||
await finalizeNativeRun({ db, runId: row.id, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
|
||||
await flush();
|
||||
expect((await publications()).map(publication => publication.idempotencyKey).sort()).toEqual([
|
||||
`run:${row.id}:failed:${f.endpointId}`, `run:${row.id}:working:${f.endpointId}`,
|
||||
].sort());
|
||||
await quiet();
|
||||
await finalizeNativeRun({ db, runId: row.id, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
|
||||
// A no-op repair must not enqueue another scan. This also catches deferred
|
||||
// coordinator callbacks that start after the finalizer returns.
|
||||
await quiet();
|
||||
expect(await publications()).toHaveLength(2);
|
||||
});
|
||||
it("signals a new native decision at outer commit but not an already committed replay", async () => {
|
||||
const input = await continuationDecisionFixture();
|
||||
const events: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, "chat-run-milestones", event => events.push(event));
|
||||
try {
|
||||
await db.transaction(async tx => {
|
||||
await commitNativeStatusDecision({ db: tx as unknown as typeof db, ...input });
|
||||
expect(events).toEqual(["intent"]);
|
||||
});
|
||||
expect(events).toEqual(["intent", "settled"]);
|
||||
events.length = 0;
|
||||
await expect(commitNativeStatusDecision({ db, ...input })).resolves.toMatchObject({ replayed: true });
|
||||
expect(events).toEqual([]);
|
||||
} finally { unsubscribe(); }
|
||||
});
|
||||
it("wakes a suppressed continuation when its native effect delivery receipt commits", async () => {
|
||||
const input = await continuationDecisionFixture();
|
||||
const { decision } = await commitNativeStatusDecision({ db, ...input });
|
||||
const [intent] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId));
|
||||
// Reconstruct an interrupted delivery with its already admitted successor.
|
||||
// The target has no direct chat identity: only the delivered ledger can
|
||||
// authorize inheriting the source run's conversation.
|
||||
await db.update(statusDecisionEffects).set({ deliveryState: "pending", deliveredAt: null })
|
||||
.where(eq(statusDecisionEffects.decisionId, decision.id));
|
||||
const target = await run(f, { status: "queued", contextSnapshot: {
|
||||
issueId: f.issueId, source: "native_status_decision", nativeStatusWakeIntentId: intent.id,
|
||||
} });
|
||||
await db.update(agentWakeupRequests).set({ status: "coalesced", runId: target.id }).where(eq(agentWakeupRequests.id, intent.id));
|
||||
await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",
|
||||
reason: "issue_status_changed", requestedByActorType: "system", requestedByActorId: `native-status-wake-dispatch:${intent.id}`,
|
||||
runId: target.id, status: "queued", idempotencyKey: intent.idempotencyKey, payload: { issueId: f.issueId },
|
||||
});
|
||||
await start(); await flush();
|
||||
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${target.id}:`))).toBe(false);
|
||||
await quiet();
|
||||
await dispatchPendingNativeStatusEffects({ db, ...input, decisionId: decision.id });
|
||||
await flush();
|
||||
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${target.id}:queued:${f.endpointId}`);
|
||||
await quiet();
|
||||
await expect(dispatchPendingNativeStatusEffects({ db, ...input, decisionId: decision.id })).resolves.toEqual([]);
|
||||
await quiet();
|
||||
});
|
||||
it("wakes an admitted continuation when native wake dispatch repairs its original receipt", async () => {
|
||||
const input = await continuationDecisionFixture();
|
||||
await commitNativeStatusDecision({ db, ...input });
|
||||
const [intent] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId));
|
||||
const target = await run(f, { status: "queued", contextSnapshot: {
|
||||
issueId: f.issueId, source: "native_status_decision", nativeStatusWakeIntentId: intent.id,
|
||||
} });
|
||||
await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",
|
||||
reason: "issue_status_changed", requestedByActorType: "system", requestedByActorId: `native-status-wake-dispatch:${intent.id}`,
|
||||
runId: target.id, status: "queued", idempotencyKey: intent.idempotencyKey, payload: { issueId: f.issueId },
|
||||
});
|
||||
await start(); await flush();
|
||||
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${target.id}:`))).toBe(false);
|
||||
await quiet();
|
||||
const { service, dependencies } = await queue();
|
||||
await expect(service.dispatchPendingNativeStatusWakeups({ companyId: f.companyId })).resolves.toMatchObject({ recovered: 1, dispatched: 0 });
|
||||
await flush();
|
||||
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${target.id}:queued:${f.endpointId}`);
|
||||
expect(dependencies.executeRun).not.toHaveBeenCalled();
|
||||
await quiet();
|
||||
await expect(service.dispatchPendingNativeStatusWakeups({ companyId: f.companyId })).resolves.toMatchObject({ scanned: 0 });
|
||||
await quiet();
|
||||
});
|
||||
it("rechecks a fresh native continuation after admission finishes its wake receipt", async () => {
|
||||
const input = await continuationDecisionFixture();
|
||||
await commitNativeStatusDecision({ db, ...input });
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, input.runId));
|
||||
await start(); await flush(); await quiet();
|
||||
const { service, dependencies } = await queue();
|
||||
let admittedId: string | undefined;
|
||||
dependencies.getSchedulingSuppression.mockImplementation(async () => {
|
||||
const rows = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId));
|
||||
const admitted = rows.find(row => row.id !== input.runId);
|
||||
if (admitted && !admittedId) {
|
||||
admittedId = admitted.id;
|
||||
// Admission already committed and signalled, but the original wake
|
||||
// receipt is still claimed. Let that first projection finish now.
|
||||
await flush();
|
||||
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${admittedId}:`))).toBe(false);
|
||||
}
|
||||
return { suppressed: true, reason: "task_drain" };
|
||||
});
|
||||
await expect(service.dispatchPendingNativeStatusWakeups({ companyId: f.companyId })).resolves.toMatchObject({ dispatched: 1 });
|
||||
expect(admittedId).toBeDefined();
|
||||
await flush();
|
||||
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${admittedId}:queued:${f.endpointId}`);
|
||||
expect(dependencies.executeRun).not.toHaveBeenCalled();
|
||||
await quiet();
|
||||
});
|
||||
it("projects a continuation only after the real question delivery receipt commits", async () => {
|
||||
const source = await run(f, { status: "succeeded" });
|
||||
const interactions = issueThreadInteractionService(db);
|
||||
const interaction = await interactions.create({ id: f.issueId, companyId: f.companyId }, {
|
||||
kind: "ask_user_questions", continuationPolicy: "wake_assignee", sourceRunId: source.id,
|
||||
title: "Choose", payload: { version: 1, title: "Choose", questions: [{
|
||||
id: "choice", prompt: "Continue?", selectionMode: "single", required: true,
|
||||
options: [{ id: "yes", label: "Yes" }],
|
||||
}] },
|
||||
}, { agentId: f.agentId, runId: source.id });
|
||||
await interactions.answerQuestions({ id: f.issueId, companyId: f.companyId, status: "in_progress" }, interaction.id,
|
||||
{ answers: [{ questionId: "choice", optionIds: ["yes"] }] }, { userId: f.userId });
|
||||
// A provider acknowledgement is durable input to this projection test.
|
||||
await db.insert(chatPublications).values({ companyId: f.companyId, endpointId: f.endpointId, conversationId: f.conversationId,
|
||||
issueId: f.issueId, idempotencyKey: `interaction:${interaction.id}:${f.endpointId}`,
|
||||
payload: { text: "Continue?", interactionId: interaction.id }, state: "published", providerMessageId: "question-1",
|
||||
}).onConflictDoNothing();
|
||||
await db.update(chatPublications).set({ state: "published", providerMessageId: "question-1" }).where(eq(chatPublications.issueId, f.issueId));
|
||||
await start(); await flush(); await quiet();
|
||||
let targetId!: string;
|
||||
const delivery = questionResponseDeliveryService(db, { heartbeat: {
|
||||
cancelRun: async () => { throw new Error("The source is already terminal"); },
|
||||
wakeup: async (_agentId, options) => {
|
||||
const values = runValues(f, { status: "queued", contextSnapshot: options.contextSnapshot });
|
||||
const [target] = await db.insert(heartbeatRuns).values(values).returning();
|
||||
targetId = target.id;
|
||||
await flush();
|
||||
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${target.id}:`))).toBe(false);
|
||||
return target;
|
||||
},
|
||||
} });
|
||||
await expect(delivery.deliver(interaction.id)).resolves.toMatchObject({ status: "fallback_queued" });
|
||||
await flush();
|
||||
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${targetId}:queued:${f.endpointId}`);
|
||||
await quiet();
|
||||
});
|
||||
it.each([
|
||||
["question", "answer"], ["question", "skip"], ["question", "cancel"],
|
||||
["question", "withdraw"], ["question", "expire"], ["question", "historical answer"],
|
||||
["confirmation", "accept"], ["confirmation", "reject"], ["confirmation", "skip"],
|
||||
["confirmation", "withdraw"], ["confirmation", "expire"],
|
||||
["question", "comment"], ["question", "historical comment"],
|
||||
["confirmation", "comment"], ["confirmation", "historical comment"],
|
||||
["question", "replacement"], ["confirmation", "replacement"], ["confirmation", "startup supersession"],
|
||||
["question", "addressee deletion"], ["confirmation", "addressee deletion"],
|
||||
] as const)("rechecks an unsent %s after %s", async (kind, action) => {
|
||||
const source = await run(f, { status: "failed", errorCode: "test_run_failed" });
|
||||
const interactions = issueThreadInteractionService(db);
|
||||
const issue = { id: f.issueId, companyId: f.companyId, status: "in_progress" };
|
||||
const addressee = action === "addressee deletion"
|
||||
? (await db.insert(agents).values({ companyId: f.companyId, name: "Reviewer", adapterType: "paperclip_runner" }).returning())[0] : null;
|
||||
const createInteraction = (sourceRunId: string, supersedePendingSiblingInteractions = true) => interactions.create(issue, kind === "confirmation" ? {
|
||||
kind: "request_confirmation", continuationPolicy: "wake_assignee", sourceRunId, addresseeAgentId: addressee?.id,
|
||||
title: "Confirm", payload: { version: 1, prompt: "Continue?", supersedeOnUserComment: true },
|
||||
} : {
|
||||
kind: "ask_user_questions", continuationPolicy: "wake_assignee", sourceRunId, addresseeAgentId: addressee?.id,
|
||||
title: "Choose", payload: { version: 1, title: "Choose", supersedeOnUserComment: true, questions: [{
|
||||
id: "choice", prompt: "Continue?", selectionMode: "single", required: true,
|
||||
options: [{ id: "yes", label: "Yes" }],
|
||||
}] },
|
||||
}, { agentId: f.agentId, runId: sourceRunId }, { supersedePendingSiblingInteractions });
|
||||
const interaction = await createInteraction(source.id);
|
||||
const replacement = action === "replacement" || action === "startup supersession" ? await run() : null;
|
||||
if (action === "startup supersession") {
|
||||
await createInteraction(replacement!.id, false);
|
||||
await db.update(issueThreadInteractions).set({ createdAt: new Date(Date.now() - 1000) }).where(eq(issueThreadInteractions.id, interaction.id));
|
||||
}
|
||||
if (action === "historical answer") {
|
||||
await interactions.dismissInteraction(issue, interaction.id, { userId: f.userId });
|
||||
await db.update(issues).set({ status: "done" }).where(eq(issues.id, f.issueId));
|
||||
}
|
||||
await start(); await flush();
|
||||
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${source.id}:`))).toBe(false);
|
||||
await quiet();
|
||||
if (action === "addressee deletion") {
|
||||
expect((await interactions.cancelPendingForDeletedAddressee(f.companyId, addressee!.id)).map(row => row.id)).toContain(interaction.id);
|
||||
} else if (action === "startup supersession") {
|
||||
expect(await interactions.sweepSupersededPendingRequestConfirmations()).toMatchObject({ expired: 1 });
|
||||
} else if (action === "replacement") {
|
||||
await createInteraction(replacement!.id);
|
||||
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interaction.id)))[0].status).toBe("expired");
|
||||
} else if (action === "comment" || action === "historical comment") {
|
||||
const [comment] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId,
|
||||
authorType: "user", authorUserId: f.userId, origin: "comment", body: "Ignore the earlier request" }).returning();
|
||||
const expired = action === "comment"
|
||||
? await interactions.expireRequestConfirmationsSupersededByComment(issue, comment, { userId: f.userId })
|
||||
: await interactions.expireRequestConfirmationsSupersededByHistoricalComments(issue);
|
||||
expect(expired.map(row => row.id)).toContain(interaction.id);
|
||||
} else if (action === "accept") {
|
||||
await interactions.acceptInteraction({ ...issue, projectId: null, goalId: null }, interaction.id, {}, { userId: f.userId });
|
||||
} else if (action === "reject") {
|
||||
await interactions.rejectInteraction(issue, interaction.id, { reason: "No longer needed" }, { userId: f.userId });
|
||||
} else if (action === "skip") {
|
||||
await interactions.skipInteraction(issue, interaction.id, { reason: "No longer needed" }, { userId: f.userId });
|
||||
} else if (action === "cancel") {
|
||||
await interactions.cancelQuestions(issue, interaction.id, { reason: "No longer needed" }, { userId: f.userId });
|
||||
} else if (action === "withdraw") {
|
||||
await interactions.withdrawInteraction(issue, interaction.id, { reason: "No longer needed" }, { userId: f.userId });
|
||||
} else if (action === "expire") {
|
||||
await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, f.issueId));
|
||||
await interactions.expirePendingInteractionsForTerminalIssue({ ...issue, status: "cancelled" }, { userId: f.userId });
|
||||
} else {
|
||||
await interactions.answerQuestions(issue, interaction.id,
|
||||
{ answers: [{ questionId: "choice", optionIds: ["yes"] }] }, { userId: f.userId });
|
||||
}
|
||||
await flush();
|
||||
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${source.id}:failed:${f.endpointId}`);
|
||||
if (action === "historical answer") {
|
||||
expect(await db.select().from(issueQuestionResponseDeliveries).where(eq(issueQuestionResponseDeliveries.interactionId, interaction.id))).toEqual([]);
|
||||
}
|
||||
await quiet();
|
||||
});
|
||||
it.each(["change", "delete", "accept", "reject"] as const)("rechecks a document confirmation after %s detects a stale target", async mode => {
|
||||
const source = await run(f, { status: "failed", errorCode: "test_run_failed" });
|
||||
const interactions = issueThreadInteractionService(db), docs = documentService(db);
|
||||
const issue = { id: f.issueId, companyId: f.companyId, status: "in_progress", projectId: null, goalId: null };
|
||||
const input = { issueId: f.issueId, key: "plan", format: "markdown", body: "First plan", createdByUserId: f.userId };
|
||||
const { document } = await docs.upsertIssueDocument(input);
|
||||
const interaction = await interactions.create(issue, {
|
||||
kind: "request_confirmation", continuationPolicy: "wake_assignee", sourceRunId: source.id,
|
||||
payload: { version: 1, prompt: "Apply this plan?", target: { type: "issue_document", issueId: f.issueId,
|
||||
documentId: document.id, key: document.key, revisionId: document.latestRevisionId!, revisionNumber: document.latestRevisionNumber } },
|
||||
}, { agentId: f.agentId, runId: source.id });
|
||||
await start(); await flush();
|
||||
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${source.id}:`))).toBe(false);
|
||||
await quiet();
|
||||
const changed = mode === "delete"
|
||||
? { ...(await docs.deleteIssueDocument(f.issueId, document.key))!, latestRevisionId: null, latestRevisionNumber: null }
|
||||
: (await docs.upsertIssueDocument({ ...input, body: "Revised plan", baseRevisionId: document.latestRevisionId })).document;
|
||||
await flush(); await quiet();
|
||||
if (mode === "accept" || mode === "reject") {
|
||||
await expect(mode === "accept"
|
||||
? interactions.acceptInteraction(issue, interaction.id, {}, { userId: f.userId })
|
||||
: interactions.rejectInteraction(issue, interaction.id, {}, { userId: f.userId }))
|
||||
.rejects.toMatchObject({ status: 409, details: { code: "interaction_stale_target" } });
|
||||
} else {
|
||||
expect((await interactions.expireStaleRequestConfirmationsForIssueDocument(issue, changed, { userId: f.userId })).map(row => row.id)).toContain(interaction.id);
|
||||
}
|
||||
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interaction.id)))[0].status).toBe("expired");
|
||||
await flush();
|
||||
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${source.id}:failed:${f.endpointId}`);
|
||||
await quiet();
|
||||
});
|
||||
it("wakes a question continuation after its in-flight prompt receipt commits", async () => {
|
||||
await db.update(chatEndpoints).set({ provider: "discord" }).where(eq(chatEndpoints.id, f.endpointId));
|
||||
const source = await run(f, { status: "failed", contextSnapshot: { ...runValues().contextSnapshot, source: "chat:discord" } });
|
||||
const interactions = issueThreadInteractionService(db);
|
||||
const interaction = await interactions.create({ id: f.issueId, companyId: f.companyId }, {
|
||||
kind: "ask_user_questions", continuationPolicy: "wake_assignee", sourceRunId: source.id,
|
||||
title: "Choose", payload: { version: 1, title: "Choose", questions: [{
|
||||
id: "choice", prompt: "Continue?", selectionMode: "single", required: true,
|
||||
options: [{ id: "yes", label: "Yes" }],
|
||||
}] },
|
||||
}, { agentId: f.agentId, runId: source.id });
|
||||
const prompt = (await publications()).find(row => row.idempotencyKey === `interaction:${interaction.id}:${f.endpointId}`)!;
|
||||
const [endpoint] = await db.select().from(chatEndpoints).where(eq(chatEndpoints.id, f.endpointId));
|
||||
const [conversation] = await db.select().from(chatConversations).where(eq(chatConversations.id, f.conversationId));
|
||||
const targetId = randomUUID();
|
||||
let beforeReceiptKeys: string[] | undefined;
|
||||
await start(); await flush(); await quiet();
|
||||
const deliver = questionResponseDeliveryService(db, { heartbeat: {
|
||||
wakeup: async () => {
|
||||
const [target] = await db.insert(heartbeatRuns).values(runValues(f, {
|
||||
id: targetId, status: "queued", contextSnapshot: { issueId: f.issueId },
|
||||
})).returning();
|
||||
return target;
|
||||
},
|
||||
} });
|
||||
const post = vi.fn(async () => {
|
||||
await interactions.answerQuestions({ id: f.issueId, companyId: f.companyId, status: "in_progress" }, interaction.id,
|
||||
{ answers: [{ questionId: "choice", optionIds: ["yes"] }] }, { userId: f.userId });
|
||||
await deliver.deliver(interaction.id);
|
||||
await flush();
|
||||
beforeReceiptKeys = (await publications()).map(row => row.idempotencyKey);
|
||||
return { id: "published-question" };
|
||||
});
|
||||
// Controlled transport/authorization ports; actual attempt lease, receipt
|
||||
// transaction, question delivery receipt and milestone worker use PostgreSQL.
|
||||
const worker = createChatPublicationWorker(db, {
|
||||
CAPABILITIES, settleGitHubAutomaticPublication: async () => false,
|
||||
ensureDurablePublicationTransport: async publication => publication,
|
||||
tryProcessTeamsFilePublication: async () => false,
|
||||
endpointRecord: async () => ({ endpoint, assignedAgentName: "Maya", connectionName: "Chat", applicationId: "test", credentialSecretRefs: [] }),
|
||||
runtimeContextForRecord: () => ({ generation: 0, localEpoch: 0, credentialFingerprint: "test", version: "test" }),
|
||||
hasCommittedTaskControlCompletion: async () => false, nonDirectDestinationAllowed: () => true,
|
||||
runOwnershipMilestoneSupersessionReason: async () => null,
|
||||
progressSupersededByFastResponse: async () => false, runProgressSupersededByPublishedInteraction: async () => false,
|
||||
currentTaskControlPayload: async publication => publication.payload,
|
||||
withCredentialMutationLease: async (_endpoint, operation) => operation({ assertOwned: async () => {} }),
|
||||
closeProgressToReplace: async () => null, interactionResolutionPublicationToReplace: async () => null,
|
||||
interactionPromptPublicationToReplace: async () => null, runPublicationToReplace: async () => null,
|
||||
inboundWakePublicationToReplace: async () => null, taskStatusPublicationToReplace: async () => null,
|
||||
claimPublicationTransportAuthorization: async () => ({ endpoint, conversation, authorizationActionId: null, preparedText: null }),
|
||||
postSafePublication: post, commitTaskControlCompletion: async () => {}, receiptReactionCompletionRunId: async () => null,
|
||||
finalizePublicationFailure: async (_publication, error) => { errors.push(error); return false; },
|
||||
redactError: String,
|
||||
} as Parameters<typeof createChatPublicationWorker>[1]);
|
||||
await worker.processSelectedPublication(prompt);
|
||||
expect(post).toHaveBeenCalledTimes(1);
|
||||
expect((await db.select().from(chatPublications).where(eq(chatPublications.id, prompt.id)))[0]).toMatchObject({ state: "published", providerMessageId: "published-question" });
|
||||
expect(beforeReceiptKeys).toBeDefined();
|
||||
expect(beforeReceiptKeys).not.toContain(`run:${targetId}:queued:${f.endpointId}`);
|
||||
await flush();
|
||||
expect((await publications()).map(row => row.idempotencyKey)).toContain(`run:${targetId}:queued:${f.endpointId}`);
|
||||
await quiet();
|
||||
});
|
||||
it.each([
|
||||
["retry", null], ["failed", null], ["delivery_unknown", "cancel"], ["delivery_unknown", "mark_delivered"],
|
||||
] as const)("rechecks resolved prompt ownership after delivery %s (%s)", async (state, resolution) => {
|
||||
const source = await run(f, { status: "failed", errorCode: "test_run_failed" });
|
||||
const interactions = issueThreadInteractionService(db);
|
||||
const interaction = await interactions.create({ id: f.issueId, companyId: f.companyId }, {
|
||||
kind: "ask_user_questions", continuationPolicy: "wake_assignee", sourceRunId: source.id,
|
||||
title: "Choose", payload: { version: 1, title: "Choose", questions: [{
|
||||
id: "choice", prompt: "Continue?", selectionMode: "single", required: true,
|
||||
options: [{ id: "yes", label: "Yes" }],
|
||||
}] },
|
||||
}, { agentId: f.agentId, runId: source.id });
|
||||
// Model a send already claimed when the user answers through the UI.
|
||||
const [prompt] = await db.update(chatPublications).set({ state: "streaming", attempts: 1 })
|
||||
.where(eq(chatPublications.idempotencyKey, `interaction:${interaction.id}:${f.endpointId}`)).returning();
|
||||
await interactions.answerQuestions({ id: f.issueId, companyId: f.companyId, status: "in_progress" }, interaction.id,
|
||||
{ answers: [{ questionId: "choice", optionIds: ["yes"] }] }, { userId: f.userId });
|
||||
expect(prompt).toBeDefined();
|
||||
await start(); await flush();
|
||||
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${source.id}:`))).toBe(false);
|
||||
await quiet();
|
||||
const failures = createChatPublicationFailures(db, {
|
||||
FailedChatRetryPublicationReadError: class extends Error {},
|
||||
MAX_ERROR_TEXT: 1000, getTaskBaseUrl: () => null,
|
||||
invalidateRuntime: async () => false, runtimeCallbackEndpoint: async () => null,
|
||||
});
|
||||
const failure = Object.assign(new Error("provider rejected the send"), { status: state === "retry" ? 429 : 400 });
|
||||
// The finalizer receives the selection from before the attempt increment.
|
||||
await expect(failures.finalizePublicationFailure({ ...prompt, attempts: 0 }, failure, state === "delivery_unknown")).resolves.toBe(true);
|
||||
expect((await db.select().from(chatPublications).where(eq(chatPublications.id, prompt.id)))[0].state).toBe(state);
|
||||
await flush();
|
||||
expect((await publications()).some(row => row.idempotencyKey === `run:${source.id}:failed:${f.endpointId}`)).toBe(state === "failed");
|
||||
await quiet();
|
||||
if (resolution) {
|
||||
const targetId = randomUUID();
|
||||
await questionResponseDeliveryService(db, { heartbeat: { wakeup: async () => {
|
||||
const [target] = await db.insert(heartbeatRuns).values(runValues(f, {
|
||||
id: targetId, status: "queued", contextSnapshot: { issueId: f.issueId },
|
||||
})).returning();
|
||||
return target;
|
||||
} } }).deliver(interaction.id);
|
||||
await flush();
|
||||
expect((await publications()).some(row => row.idempotencyKey.startsWith(`run:${targetId}:`))).toBe(false);
|
||||
await quiet();
|
||||
const [endpoint] = await db.select().from(chatEndpoints).where(eq(chatEndpoints.id, f.endpointId));
|
||||
const resolver = createChatPublicationResolution(db, {
|
||||
endpointRecord: async () => ({ endpoint, assignedAgentName: "Maya", connectionName: "Chat", applicationId: "test", credentialSecretRefs: [] }),
|
||||
withCredentialMutationLease: async (_endpoint, operation) => operation({ assertOwned: async () => {} }),
|
||||
commitTaskControlCompletion: async () => {}, processPendingPublications: async () => {},
|
||||
} as Parameters<typeof createChatPublicationResolution>[1]);
|
||||
await resolver.resolvePublication(f.endpointId, prompt.id, resolution, f.userId);
|
||||
await flush();
|
||||
const keys = (await publications()).map(row => row.idempotencyKey);
|
||||
expect(keys.includes(`run:${source.id}:failed:${f.endpointId}`)).toBe(resolution === "cancel");
|
||||
expect(keys.includes(`run:${targetId}:queued:${f.endpointId}`)).toBe(resolution === "mark_delivered");
|
||||
await quiet();
|
||||
}
|
||||
});
|
||||
it.each(["retry", "stale claim", "live claim", "rolled-back reset"] as const)("rechecks question prompt binding after %s", async (mode) => {
|
||||
const source = await run();
|
||||
const interactions = issueThreadInteractionService(db);
|
||||
const interaction = await interactions.create({ id: f.issueId, companyId: f.companyId }, {
|
||||
kind: "ask_user_questions", continuationPolicy: "wake_assignee", sourceRunId: source.id,
|
||||
title: "Choose", payload: { version: 1, title: "Choose", questions: [{
|
||||
id: "choice", prompt: "Continue?", selectionMode: "single", required: true,
|
||||
options: [{ id: "yes", label: "Yes" }],
|
||||
}] },
|
||||
}, { agentId: f.agentId, runId: source.id });
|
||||
await interactions.answerQuestions({ id: f.issueId, companyId: f.companyId, status: "in_progress" }, interaction.id,
|
||||
{ answers: [{ questionId: "choice", optionIds: ["yes"] }] }, { userId: f.userId });
|
||||
const other = await fixture(f.companyId);
|
||||
await db.update(chatEndpoints).set({ assignedAgentId: f.agentId }).where(eq(chatEndpoints.id, other.endpointId));
|
||||
await db.update(chatConversations).set({ issueId: f.issueId }).where(eq(chatConversations.id, other.conversationId));
|
||||
await db.insert(chatPublications).values({ companyId: f.companyId, endpointId: other.endpointId, conversationId: other.conversationId,
|
||||
issueId: f.issueId, idempotencyKey: `interaction:${interaction.id}:${other.endpointId}`,
|
||||
payload: { text: "Continue?", interactionId: interaction.id }, state: "published", providerMessageId: "other-question",
|
||||
});
|
||||
const events: string[] = [];
|
||||
let claimEvents: string[] = [];
|
||||
let duringClaimNotices: string[] | undefined;
|
||||
const unsubscribe = subscribeDatabaseWork(db, "chat-run-milestones", event => events.push(event));
|
||||
const delivery = questionResponseDeliveryService(db, { heartbeat: {
|
||||
cancelRun: async () => {
|
||||
claimEvents = [...events];
|
||||
await start(); await flush();
|
||||
duringClaimNotices = (await publications()).map(row => row.idempotencyKey);
|
||||
throw new Error("temporary cancellation failure");
|
||||
},
|
||||
wakeup: async () => { throw new Error("Must retain the still-active source"); },
|
||||
} });
|
||||
try {
|
||||
if (mode === "retry") {
|
||||
await expect(delivery.deliver(interaction.id)).resolves.toBeNull();
|
||||
} else {
|
||||
await db.update(issueQuestionResponseDeliveries).set({ status: "delivering", attemptCount: 1,
|
||||
lastAttemptAt: new Date(Date.now() - (mode === "live claim" ? 0 : 60_000)),
|
||||
}).where(eq(issueQuestionResponseDeliveries.interactionId, interaction.id));
|
||||
await start(); await flush();
|
||||
duringClaimNotices = (await publications()).map(row => row.idempotencyKey);
|
||||
if (mode === "rolled-back reset") {
|
||||
await expect(db.transaction(async tx => {
|
||||
await questionResponseDeliveryService(tx as unknown as typeof db, { heartbeat: {} as never }).sweepPending(0);
|
||||
expect(events).toEqual(["intent"]);
|
||||
throw new Error("rollback reset");
|
||||
})).rejects.toThrow("rollback reset");
|
||||
} else {
|
||||
await delivery.sweepPending(0);
|
||||
}
|
||||
}
|
||||
expect(duringClaimNotices).toBeDefined();
|
||||
expect(duringClaimNotices!.some(key => key.startsWith(`run:${source.id}:`))).toBe(false);
|
||||
expect((await db.select().from(issueQuestionResponseDeliveries).where(eq(issueQuestionResponseDeliveries.interactionId, interaction.id)))[0].status).toBe(mode === "live claim" || mode === "rolled-back reset" ? "delivering" : "pending");
|
||||
await flush();
|
||||
const noticeExists = (await publications()).some(row => row.idempotencyKey === `run:${source.id}:working:${f.endpointId}`);
|
||||
expect(noticeExists).toBe(mode === "retry" || mode === "stale claim");
|
||||
if (mode === "retry") expect(claimEvents).toEqual(["intent", "settled"]);
|
||||
else if (mode === "live claim") expect(events).toEqual([]);
|
||||
else {
|
||||
expect(events[0]).toBe("intent");
|
||||
expect(events.slice(1)).toContain("settled");
|
||||
expect(events.filter(event => event === "intent")).toHaveLength(1);
|
||||
}
|
||||
await quiet();
|
||||
} finally { unsubscribe(); }
|
||||
});
|
||||
it("waits for outer commit and ignores rolled-back nested writes", async () => {
|
||||
await start(); await flush();
|
||||
await expect(db.transaction(async tx => {
|
||||
await tx.transaction(async nested => { await notifyChatRunMilestones(nested, f); await nested.insert(heartbeatRuns).values(runValues()); });
|
||||
throw new Error("rollback fixture");
|
||||
})).rejects.toThrow("rollback fixture");
|
||||
await flush(); expect(await publications()).toEqual([]); await quiet();
|
||||
let release!: () => void, staged!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; });
|
||||
const stagedPromise = new Promise<void>(resolve => { staged = resolve; });
|
||||
const writing = db.transaction(async tx => {
|
||||
await tx.transaction(async nested => { await notifyChatRunMilestones(nested, f); await nested.insert(heartbeatRuns).values(runValues()); });
|
||||
staged(); await gate;
|
||||
});
|
||||
try { await stagedPromise; await vi.advanceTimersByTimeAsync(1000); expect(await publications()).toEqual([]); }
|
||||
finally { release(); await writing; }
|
||||
await flush(); expect(await publications()).toHaveLength(1); await quiet();
|
||||
});
|
||||
it("retains the first committed wake during drain without periodic SQL", async () => {
|
||||
await start(); await flush(); enabled = false;
|
||||
await run(); await vi.advanceTimersByTimeAsync(1000);
|
||||
await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0));
|
||||
const drained = idleWorkSnapshot();
|
||||
expect(await publications()).toEqual([]); await quiet(); expect(idleWorkSnapshot()).toEqual(drained);
|
||||
enabled = true; await vi.advanceTimersByTimeAsync(30_000); await flush();
|
||||
expect(await publications()).toHaveLength(1); await quiet();
|
||||
});
|
||||
it("scopes mutation wakes to their issue and company", async () => {
|
||||
const sibling = await fixture(f.companyId), foreign = await fixture();
|
||||
await start(); await flush();
|
||||
// Unsupported raw writers deliberately do not wake: they provide a sentinel
|
||||
// proving a supported issue mutation does not perform a global scan.
|
||||
await db.insert(heartbeatRuns).values([runValues(sibling), runValues(foreign)]);
|
||||
await run(); await flush();
|
||||
expect(await publications()).toHaveLength(1); expect(await publications(sibling)).toEqual([]); expect(await publications(foreign)).toEqual([]);
|
||||
await db.transaction(tx => notifyChatRunMilestones(tx, { companyId: f.companyId }));
|
||||
await flush(); expect(await publications(sibling)).toHaveLength(1); expect(await publications(foreign)).toEqual([]);
|
||||
await quiet();
|
||||
});
|
||||
it("disarms suppressed lineage and reactivates after its binding changes", async () => {
|
||||
const row = await run(f, { contextSnapshot: { issueId: f.issueId, source: "chat:slack", endpointId: randomUUID(), wakeCommentId: f.commentId } });
|
||||
await start(); await flush(); expect(await publications()).toEqual([]); await quiet();
|
||||
await db.transaction(async tx => {
|
||||
await notifyChatRunMilestones(tx, f);
|
||||
await tx.update(heartbeatRuns).set({ contextSnapshot: runValues().contextSnapshot }).where(eq(heartbeatRuns.id, row.id));
|
||||
});
|
||||
await flush(); expect(await publications()).toHaveLength(1); await quiet();
|
||||
});
|
||||
it("waits quietly for a late presentation decision", async () => {
|
||||
const row = await run(f, { status: "succeeded" }); await start(); await flush();
|
||||
expect(await publications()).toEqual([]); await quiet();
|
||||
await db.transaction(async tx => {
|
||||
await notifyChatRunMilestones(tx, f);
|
||||
await tx.update(heartbeatRuns).set({ resultJson: { presentationDecision: { source: "none" } } }).where(eq(heartbeatRuns.id, row.id));
|
||||
});
|
||||
await flush(); expect(await publications()).toEqual([expect.objectContaining({ idempotencyKey: `run:${row.id}:completed:${f.endpointId}` })]); await quiet();
|
||||
});
|
||||
it("drains more than one batch without duplicate notices or starving another company", async () => {
|
||||
const other = await fixture();
|
||||
await db.insert(heartbeatRuns).values([...Array.from({ length: 205 }, () => runValues()), runValues(other)]);
|
||||
await start(); await flush(); expect(await publications()).toHaveLength(205); expect(await publications(other)).toHaveLength(1); await quiet();
|
||||
});
|
||||
it("retains a failed selection for retry through drain", async () => {
|
||||
await run();
|
||||
const select = vi.spyOn(db, "select").mockImplementationOnce(() => { throw new Error("temporary read failure"); });
|
||||
await start(); select.mockRestore(); expect(errors.splice(0)).toHaveLength(1);
|
||||
enabled = false; await vi.advanceTimersByTimeAsync(30_000);
|
||||
await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0));
|
||||
const drained = idleWorkSnapshot();
|
||||
await quiet(); expect(idleWorkSnapshot()).toEqual(drained);
|
||||
enabled = true; await vi.advanceTimersByTimeAsync(30_000); await flush();
|
||||
expect(await publications()).toHaveLength(1); await quiet();
|
||||
});
|
||||
it("retries a failed publication write without idle scans before its deadline", async () => {
|
||||
await run();
|
||||
const transaction = vi.spyOn(db, "transaction").mockRejectedValueOnce(new Error("temporary publication failure"));
|
||||
await start(); transaction.mockRestore(); expect(errors.splice(0)).toHaveLength(1);
|
||||
expect(await publications()).toEqual([]); await quiet(10_000);
|
||||
await vi.advanceTimersByTimeAsync(30_000); await flush(); expect(await publications()).toHaveLength(1); await quiet();
|
||||
});
|
||||
it("preserves a committed change arriving while a projection batch is running", async () => {
|
||||
await run();
|
||||
const transaction = db.transaction.bind(db);
|
||||
let release!: () => void, entered!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; });
|
||||
const staged = new Promise<void>(resolve => { entered = resolve; });
|
||||
const held = vi.spyOn(db, "transaction").mockImplementationOnce(async (callback, config) => {
|
||||
entered(); await gate; return transaction(callback, config);
|
||||
});
|
||||
const starting = start();
|
||||
try {
|
||||
await staged;
|
||||
await transaction(async tx => { await notifyChatRunMilestones(tx, f); await tx.insert(heartbeatRuns).values(runValues()); });
|
||||
} finally { release(); await starting; held.mockRestore(); }
|
||||
await flush(); expect(await publications()).toHaveLength(2); await quiet();
|
||||
});
|
||||
it("stops future wakes and reconstructs writes made after shutdown", async () => {
|
||||
await start(); await flush(); await coordinator!.stop(); await run(); await quiet();
|
||||
expect(await publications()).toEqual([]); await start(); await flush(); expect(await publications()).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
@@ -1,3 +1,4 @@
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { createHash } from "node:crypto";
|
||||
import { afterAll, afterEach, beforeEach, beforeAll, describe, expect, it } from "vitest";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
@@ -18,7 +19,8 @@ import {
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
import { cloudActorHeaderSourceFromHeaders, resolveCloudTenantActor } from "../middleware/auth.js";
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_CLOUD_PROVISIONING_TEST_DATABASE_URL");
|
||||
const embeddedPostgresSupport = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
if (!embeddedPostgresSupport.supported) {
|
||||
@@ -59,8 +61,8 @@ describeEmbeddedPostgres("cloud tenant company provisioning", () => {
|
||||
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
|
||||
beforeAll(async () => {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-tenant-provisioning-");
|
||||
db = createDb(tempDb.connectionString);
|
||||
if (!external) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-tenant-provisioning-");
|
||||
db = createDb(external ?? tempDb!.connectionString);
|
||||
}, 20_000);
|
||||
|
||||
beforeEach(() => {
|
||||
@@ -80,6 +82,7 @@ describeEmbeddedPostgres("cloud tenant company provisioning", () => {
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await db?.$client.end({ timeout: 1 });
|
||||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { githubCredentialsFixture } from "./helpers/github-credentials-fixture.js";
|
||||
import { createNativeGitHubAccess } from "../services/native-runtime/native-github-access.js";
|
||||
import express from "express";
|
||||
@@ -34,8 +35,6 @@ import {
|
||||
import {
|
||||
initializeRunIdentity,
|
||||
captureRunIdentity,
|
||||
reserveSteeredIdentity,
|
||||
acceptSteeredIdentity,
|
||||
} from "../services/run-identity.js";
|
||||
import { resolveGitHubOperationCredentials } from "../services/github-operation-credentials.js";
|
||||
import {
|
||||
@@ -52,23 +51,25 @@ const vault = vi.hoisted(() => ({
|
||||
resolveSecretValue: vi.fn(async () => "test-dedicated-token"),
|
||||
}));
|
||||
vi.mock("../services/secrets.js", () => ({ secretService: () => vault }));
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_GITHUB_CREDENTIAL_TEST_DATABASE_URL");
|
||||
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
(support.supported ? describe : describe.skip)(
|
||||
"operation-time GitHub credential resolution",
|
||||
() => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>,
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined,
|
||||
db: ReturnType<typeof createDb>;
|
||||
beforeAll(async () => {
|
||||
vi.stubEnv(
|
||||
"PAPERCLIP_AGENT_JWT_SECRET",
|
||||
"test-github-broker-signing-secret",
|
||||
);
|
||||
database = await startEmbeddedPostgresTestDatabase(
|
||||
if (!external) database = await startEmbeddedPostgresTestDatabase(
|
||||
"paperclip-github-operation-",
|
||||
);
|
||||
db = createDb(database.connectionString);
|
||||
db = createDb(external ?? database!.connectionString);
|
||||
}, 30_000);
|
||||
afterAll(async () => {
|
||||
await db?.$client.end({ timeout: 1 });
|
||||
await database?.cleanup();
|
||||
vi.unstubAllEnvs();
|
||||
}, 60_000);
|
||||
@@ -85,7 +86,7 @@ const support = await getEmbeddedPostgresTestSupport();
|
||||
transport: "mcp_remote" as const, config: { sourceTemplateKey, payload }, transportConfig: { sourceTemplateKey } };
|
||||
});
|
||||
await db.insert(toolConnections).values(irrelevant);
|
||||
const observed = createDb(database.connectionString), events: Array<Record<string, unknown>> = [], now = Date.now();
|
||||
const observed = createDb(external ?? database!.connectionString), events: Array<Record<string, unknown>> = [], now = Date.now();
|
||||
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
|
||||
installDatabaseResponseCapture(observed, { toolConnections }, capture);
|
||||
try {
|
||||
@@ -117,7 +118,7 @@ const support = await getEmbeddedPostgresTestSupport();
|
||||
resultJson: { stdout: payload }, stdoutExcerpt: payload, stderrExcerpt: payload,
|
||||
}).where(eq(heartbeatRuns.id, input.runId));
|
||||
const [before] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId));
|
||||
const observed = createDb(database.connectionString), events: Array<Record<string, unknown>> = [], now = Date.now();
|
||||
const observed = createDb(external ?? database!.connectionString), events: Array<Record<string, unknown>> = [], now = Date.now();
|
||||
const capture = createDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
|
||||
installDatabaseResponseCapture(observed, { heartbeatRuns }, capture);
|
||||
try {
|
||||
|
||||
@@ -37,6 +37,7 @@ import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { heartbeatService } from "../services/heartbeat.ts";
|
||||
import { runningProcesses } from "../adapters/index.ts";
|
||||
|
||||
@@ -63,7 +64,8 @@ vi.mock("../adapters/index.ts", async () => {
|
||||
};
|
||||
});
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_DEPENDENCY_TEST_DATABASE_URL");
|
||||
const embeddedPostgresSupport = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
if (!embeddedPostgresSupport.supported) {
|
||||
@@ -103,8 +105,8 @@ describeEmbeddedPostgres("heartbeat dependency-aware queued run selection", () =
|
||||
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
|
||||
beforeAll(async () => {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-heartbeat-dependency-scheduling-");
|
||||
db = createDb(tempDb.connectionString);
|
||||
if (!external) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-heartbeat-dependency-scheduling-");
|
||||
db = createDb(external ?? tempDb!.connectionString);
|
||||
await db.insert(authUsers).values({ id: "responsible-user", name: "Fixture owner", email: "owner@example.test", createdAt: new Date(), updatedAt: new Date() });
|
||||
heartbeat = heartbeatService(db);
|
||||
await ensureIssueRelationsTable(db);
|
||||
@@ -185,6 +187,8 @@ describeEmbeddedPostgres("heartbeat dependency-aware queued run selection", () =
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await db?.delete(authUsers).where(eq(authUsers.id, "responsible-user"));
|
||||
await db?.$client.end();
|
||||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import {
|
||||
activityLog, agents, agentWakeupRequests, authUsers,
|
||||
caseHandoffs, cases, companyMemberships, chatActions,
|
||||
chatConversations, chatEndpoints, chatMessageLinks, chatPublications,
|
||||
companies, completionContracts, documentRevisions, documents,
|
||||
issueDocuments, type Db, nativeRunFinalizations, nativeRunResults,
|
||||
statusDecisions, statusDecisionEffects, workAssessments, heartbeatRunEvents,
|
||||
heartbeatRuns, issueComments, issueQuestionResponseDeliveries, issueThreadInteractions,
|
||||
issues, toolApplications, toolConnections,
|
||||
} from "@paperclipai/db";
|
||||
|
||||
// This owns only fixture data. Each test retains its worker, clock and quiet
|
||||
// assertions, and must stop the worker before cleanup.
|
||||
export function createChatMilestoneFixtures(db: Db) {
|
||||
const fixtureCompanies: string[] = [];
|
||||
const fixtureUsers: string[] = [];
|
||||
async function create(companyId: string = randomUUID()) {
|
||||
const agentId = randomUUID(), issueId = randomUUID(), endpointId = randomUUID(), conversationId = randomUUID(), commentId = randomUUID(), userId = randomUUID();
|
||||
if (!fixtureCompanies.includes(companyId)) {
|
||||
fixtureCompanies.push(companyId);
|
||||
await db.insert(companies).values({ id: companyId, name: "Milestones", issuePrefix: `M${companyId.slice(0, 7)}` });
|
||||
}
|
||||
fixtureUsers.push(userId);
|
||||
await db.insert(authUsers).values({ id: userId, name: "Owner", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() });
|
||||
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, status: "active", membershipRole: "owner" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Maya", adapterType: "paperclip_runner" });
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: "Chat task", status: "in_progress", assigneeAgentId: agentId, responsibleUserId: userId });
|
||||
const applicationId = randomUUID(), connectionId = randomUUID();
|
||||
await db.insert(toolApplications).values({ id: applicationId, companyId, name: `Chat ${applicationId}`, type: "chat" });
|
||||
await db.insert(toolConnections).values({ id: connectionId, companyId, applicationId, name: "Chat", uid: connectionId, transport: "chat_sdk", connectionPurpose: "channel", enabled: true, status: "active" });
|
||||
await db.insert(chatEndpoints).values({ id: endpointId, companyId, connectionId, assignedAgentId: agentId, publicId: endpointId, provider: "slack", status: "active" });
|
||||
await db.insert(chatConversations).values({ id: conversationId, companyId, endpointId, issueId, externalConversationId: conversationId, externalThreadId: conversationId, externalLabel: "Chat" });
|
||||
await db.insert(issueComments).values({ id: commentId, companyId, issueId, authorType: "user", body: "Start work" });
|
||||
await db.insert(chatMessageLinks).values({ companyId, endpointId, conversationId, commentId, providerMessageId: commentId, direction: "inbound" });
|
||||
return { companyId, agentId, issueId, endpointId, conversationId, commentId, userId };
|
||||
}
|
||||
async function cleanup() {
|
||||
for (const companyId of fixtureCompanies.splice(0)) {
|
||||
await db.delete(activityLog).where(eq(activityLog.companyId, companyId));
|
||||
await db.delete(caseHandoffs).where(eq(caseHandoffs.companyId, companyId));
|
||||
await db.delete(cases).where(eq(cases.companyId, companyId));
|
||||
await db.delete(chatActions).where(eq(chatActions.companyId, companyId));
|
||||
await db.delete(chatMessageLinks).where(eq(chatMessageLinks.companyId, companyId));
|
||||
await db.delete(chatPublications).where(eq(chatPublications.companyId, companyId));
|
||||
await db.delete(chatConversations).where(eq(chatConversations.companyId, companyId));
|
||||
await db.delete(chatEndpoints).where(eq(chatEndpoints.companyId, companyId));
|
||||
await db.delete(issueQuestionResponseDeliveries).where(eq(issueQuestionResponseDeliveries.companyId, companyId));
|
||||
await db.delete(issueThreadInteractions).where(eq(issueThreadInteractions.companyId, companyId));
|
||||
await db.delete(issueComments).where(eq(issueComments.companyId, companyId));
|
||||
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.companyId, companyId));
|
||||
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, companyId));
|
||||
await db.delete(statusDecisionEffects).where(eq(statusDecisionEffects.companyId, companyId));
|
||||
await db.delete(statusDecisions).where(eq(statusDecisions.companyId, companyId));
|
||||
await db.delete(workAssessments).where(eq(workAssessments.companyId, companyId));
|
||||
await db.delete(nativeRunResults).where(eq(nativeRunResults.companyId, companyId));
|
||||
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId));
|
||||
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId));
|
||||
await db.delete(completionContracts).where(eq(completionContracts.companyId, companyId));
|
||||
await db.delete(issueDocuments).where(eq(issueDocuments.companyId, companyId));
|
||||
await db.delete(documentRevisions).where(eq(documentRevisions.companyId, companyId));
|
||||
await db.delete(documents).where(eq(documents.companyId, companyId));
|
||||
await db.delete(issues).where(eq(issues.companyId, companyId));
|
||||
await db.delete(agents).where(eq(agents.companyId, companyId));
|
||||
await db.delete(companyMemberships).where(eq(companyMemberships.companyId, companyId));
|
||||
await db.delete(companies).where(eq(companies.id, companyId));
|
||||
}
|
||||
for (const userId of fixtureUsers.splice(0)) await db.delete(authUsers).where(eq(authUsers.id, userId));
|
||||
}
|
||||
return { create, cleanup };
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { agents, companies, companyMemberships, companySecrets, connectionGrants, heartbeatRuns, issueComments, issues, toolApplications, toolConnectionInstalls, toolConnections, userSecretDefinitions, type Db } from "@paperclipai/db";
|
||||
import { initializeRunIdentity, reserveSteeredIdentity, acceptSteeredIdentity } from "../../services/run-identity.js";
|
||||
import { initializeRunIdentity, reserveSteeredIdentity, reconcileSteeredIdentity } from "../../services/run-identity.js";
|
||||
|
||||
export function githubCredentialsFixture(getDb: () => Db) {
|
||||
async function seed() {
|
||||
@@ -145,7 +145,7 @@ export function githubCredentialsFixture(getDb: () => Db) {
|
||||
...input,
|
||||
messageId: id,
|
||||
});
|
||||
await acceptSteeredIdentity(db, context!);
|
||||
await reconcileSteeredIdentity(db, context!);
|
||||
}
|
||||
return { seed, grant, switchTo };
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
@@ -48,7 +49,8 @@ import { instanceSettingsService } from "../services/instance-settings.js";
|
||||
import { withNativeWorkspaceFinalizationOwnership } from "../services/native-runtime/native-workspace-finalization-ownership.js";
|
||||
import { noticeMetadataReferencesRecoveryAction } from "../services/recovery/successful-run-handoff.js";
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_RECOVERY_ACTION_TEST_DATABASE_URL");
|
||||
const embeddedPostgresSupport = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
function makeRecoveryActionRow(overrides: Record<string, unknown> = {}) {
|
||||
@@ -154,8 +156,8 @@ describeEmbeddedPostgres("issue recovery actions", () => {
|
||||
beforeAll(async () => {
|
||||
operationLogRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-recovery-operation-logs-"));
|
||||
process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = operationLogRoot;
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-issue-recovery-actions-");
|
||||
db = createDb(tempDb.connectionString);
|
||||
if (!external) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-issue-recovery-actions-");
|
||||
db = createDb(external ?? tempDb!.connectionString);
|
||||
}, 30_000);
|
||||
|
||||
afterEach(async () => {
|
||||
@@ -182,6 +184,7 @@ describeEmbeddedPostgres("issue recovery actions", () => {
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await db?.$client.end({ timeout: 1 });
|
||||
await tempDb?.cleanup();
|
||||
if (priorOperationLogRoot === undefined) delete process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
|
||||
else process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = priorOperationLogRoot;
|
||||
@@ -616,8 +619,8 @@ describeEmbeddedPostgres("issue recovery actions", () => {
|
||||
await db.update(agents).set({ status: "paused" }).where(eq(agents.id, coderId));
|
||||
const enqueueWakeup = vi.fn(async () => null);
|
||||
const recovery = recoveryService(db, { enqueueWakeup });
|
||||
const operatorDb = createDb(tempDb!.connectionString, { maxConnections: 1 });
|
||||
const probeDb = createDb(tempDb!.connectionString, { maxConnections: 1 });
|
||||
const operatorDb = createDb(external ?? tempDb!.connectionString, { maxConnections: 1 });
|
||||
const probeDb = createDb(external ?? tempDb!.connectionString, { maxConnections: 1 });
|
||||
let release!: () => void;
|
||||
let ready!: () => void;
|
||||
let holderPid = 0;
|
||||
@@ -766,7 +769,7 @@ describeEmbeddedPostgres("issue recovery actions", () => {
|
||||
.where(eq(issues.id, sourceIssue.id));
|
||||
}
|
||||
const [observed] = await db.select().from(issues).where(eq(issues.id, sourceIssue.id));
|
||||
const singleConnectionDb = createDb(tempDb!.connectionString, { maxConnections: 1 });
|
||||
const singleConnectionDb = createDb(external ?? tempDb!.connectionString, { maxConnections: 1 });
|
||||
let timer: ReturnType<typeof setTimeout> | undefined;
|
||||
try {
|
||||
const work = recoveryService(singleConnectionDb, { enqueueWakeup: async () => null })
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import express from "express";
|
||||
import request from "supertest";
|
||||
import { eq } from "drizzle-orm";
|
||||
@@ -20,18 +21,19 @@ vi.mock("../services/github-fetch.js", async importOriginal => ({
|
||||
...await importOriginal<typeof import("../services/github-fetch.js")>(),
|
||||
ghFetch: vi.fn(async () => new Response(null, { status: 404 })),
|
||||
}));
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_PROJECT_GIT_TEST_DATABASE_URL");
|
||||
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
(support.supported ? describe : describe.skip)("explicit project Git authorization", () => {
|
||||
let db: ReturnType<typeof createDb>, database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
const { seed, grant, switchTo } = githubCredentialsFixture(() => db);
|
||||
beforeAll(async () => {
|
||||
database = await startEmbeddedPostgresTestDatabase("paperclip-project-git-authorization-");
|
||||
db = createDb(database.connectionString);
|
||||
if (!external) database = await startEmbeddedPostgresTestDatabase("paperclip-project-git-authorization-");
|
||||
db = createDb(external ?? database!.connectionString);
|
||||
await db.insert(authUsers).values(["A", "B"].map(id => ({
|
||||
id, name: id, email: `${id}@example.invalid`, createdAt: new Date(), updatedAt: new Date(),
|
||||
})));
|
||||
}, 30_000);
|
||||
afterAll(async () => { await database?.cleanup(); }, 60_000);
|
||||
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await database?.cleanup(); }, 60_000);
|
||||
async function projectAuthorizationFixture() {
|
||||
const input = await seed();
|
||||
const selected = await grant(input, "A");
|
||||
|
||||
@@ -26,6 +26,7 @@ import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { issueThreadInteractionService } from "../services/issue-thread-interactions.js";
|
||||
import {
|
||||
buildQuestionResponseDeliveryEnvelope,
|
||||
@@ -33,7 +34,8 @@ import {
|
||||
questionResponseDeliveryService,
|
||||
} from "../services/question-response-delivery.js";
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_QUESTION_DELIVERY_TEST_DATABASE_URL");
|
||||
const embeddedPostgresSupport = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported
|
||||
? describe
|
||||
: describe.skip;
|
||||
@@ -64,10 +66,10 @@ describeEmbeddedPostgres("question response delivery", () => {
|
||||
> | null = null;
|
||||
|
||||
beforeAll(async () => {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase(
|
||||
"paperclip-question-delivery-",
|
||||
);
|
||||
db = createDb(tempDb.connectionString);
|
||||
if (!externalDatabaseUrl) {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-question-delivery-");
|
||||
}
|
||||
db = createDb(externalDatabaseUrl ?? tempDb!.connectionString);
|
||||
}, 20_000);
|
||||
|
||||
afterEach(async () => {
|
||||
@@ -84,6 +86,7 @@ describeEmbeddedPostgres("question response delivery", () => {
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await db?.$client.end();
|
||||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
subscribeDatabaseWork,
|
||||
costEvents,
|
||||
activityLog,
|
||||
agents,
|
||||
@@ -25,9 +27,11 @@ const mockTelemetryClient = vi.hoisted(() => ({ track: vi.fn() }));
|
||||
vi.mock("../telemetry.ts", () => ({ getTelemetryClient: () => mockTelemetryClient }));
|
||||
|
||||
import { heartbeatService } from "../services/heartbeat.ts";
|
||||
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
|
||||
import { recoveryService } from "../services/recovery/service.ts";
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_STALE_LOCK_TEST_DATABASE_URL");
|
||||
const embeddedPostgresSupport = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
if (!embeddedPostgresSupport.supported) {
|
||||
@@ -43,8 +47,8 @@ describeEmbeddedPostgres("recovery sweepStaleIssueLocks", () => {
|
||||
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
|
||||
beforeAll(async () => {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-stale-lock-sweep-");
|
||||
db = createDb(tempDb.connectionString);
|
||||
if (!external) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-stale-lock-sweep-");
|
||||
db = createDb(external ?? tempDb!.connectionString);
|
||||
}, 20_000);
|
||||
|
||||
afterEach(async () => {
|
||||
@@ -64,6 +68,7 @@ describeEmbeddedPostgres("recovery sweepStaleIssueLocks", () => {
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await db?.$client.end({ timeout: 1 });
|
||||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
@@ -585,8 +590,15 @@ describeEmbeddedPostgres("recovery sweepStaleIssueLocks", () => {
|
||||
it.each(["observed", "workspace_finalizing", "ready_for_assessment", "arbitrating", "retryable_failure"])(
|
||||
"preserves a completed native result in %s after its provider exits", async phase => {
|
||||
const { runningRunId, issueId } = await seedPendingNativeResult(phase);
|
||||
const result = await recoveryService(db, { enqueueWakeup: vi.fn() }).sweepStaleIssueLocks();
|
||||
expect(result).toEqual({ cleared: 0, issueIds: [], terminalizedRunIds: [] });
|
||||
const events: string[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.chatRunMilestones, event => events.push(event));
|
||||
try {
|
||||
const recovery = recoveryService(db, { enqueueWakeup: vi.fn() });
|
||||
for (let pass = 0; pass < 3; pass++) {
|
||||
expect(await recovery.sweepStaleIssueLocks()).toEqual({ cleared: 0, issueIds: [], terminalizedRunIds: [] });
|
||||
}
|
||||
expect(events).toEqual([]);
|
||||
} finally { unsubscribe(); }
|
||||
expect(await db.select({ status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode }).from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, runningRunId))).toEqual([{ status: "running", errorCode: null }]);
|
||||
expect(await db.select({ executionRunId: issues.executionRunId }).from(issues)
|
||||
|
||||
@@ -3,7 +3,7 @@ import { eq, sql } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { agentWakeupRequests, agents, companies, applyPendingMigrations, createDb, closeRegisteredClients, heartbeatRuns, heartbeatRunEvents, issueComments, issueThreadInteractions, issues, secretAccessEvents } from "@paperclipai/db";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { acceptSteeredIdentity, captureRunIdentity, initializeRunIdentity, listRunIdentityContexts, rejectSteeredIdentity, reserveSteeredIdentity } from "../services/run-identity.js";
|
||||
import { acceptSteeredIdentity, reconcileSteeredIdentity, captureRunIdentity, initializeRunIdentity, listRunIdentityContexts, rejectSteeredIdentity, reserveSteeredIdentity } from "../services/run-identity.js";
|
||||
|
||||
const externalDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL;
|
||||
const support = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
|
||||
@@ -143,8 +143,8 @@ const support = externalDatabaseUrl ? { supported: true } : await getEmbeddedPos
|
||||
expect((await captureRunIdentity(db, input, options)).context?.responsibleUserId).toBe("B");
|
||||
expect(startedUnderA.context?.responsibleUserId).toBe("A");
|
||||
const a = await reserveSteeredIdentity(db, { ...input, messageId: input.messageIds[2] });
|
||||
await acceptSteeredIdentity(db, a!);
|
||||
await acceptSteeredIdentity(db, b!);
|
||||
await reconcileSteeredIdentity(db, a!);
|
||||
await reconcileSteeredIdentity(db, b!);
|
||||
expect((await captureRunIdentity(db, input, options)).context?.responsibleUserId).toBe("A");
|
||||
});
|
||||
it("keeps rejected steering unchanged and preserves the originating identity on retry", async () => {
|
||||
@@ -203,7 +203,7 @@ const support = externalDatabaseUrl ? { supported: true } : await getEmbeddedPos
|
||||
sourceIdentityContextId: origin.id, payload: { prompt: "Continue?" } as never,
|
||||
});
|
||||
const b = await reserveSteeredIdentity(db, { ...input, messageId: input.messageIds[1] });
|
||||
await acceptSteeredIdentity(db, b!);
|
||||
await reconcileSteeredIdentity(db, b!);
|
||||
for (const approval of [true, false]) {
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId: input.companyId, agentId: input.agentId, status: "running" });
|
||||
|
||||
@@ -270,7 +270,7 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu
|
||||
await db.update(chatVoiceInboundCalls).set({ expiresAt: new Date(0), updatedAt: new Date(0) }).where(eq(chatVoiceInboundCalls.id, f.call.id));
|
||||
const onError = vi.fn();
|
||||
const maintenance = createChatReconciliationCoordinator({
|
||||
processPendingDeliveries: async () => undefined, projectRunMilestones: async () => 0,
|
||||
processPendingDeliveries: async () => undefined, projectNativeProgress: async () => 0,
|
||||
flushPublications: async () => undefined, onError,
|
||||
});
|
||||
try {
|
||||
|
||||
Reference in new issue
Block a user