mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
refactor(wake-queue): own blocked admission deferral (#22332)
## Thinking Path > - Paperclip runs agents in response to work on a task. > - A person can leave a comment while execution recovery holds that task. > - Wake-queue checks the recovery gate under the task lock. Heartbeat used to choose how to save the blocked input. > - The deferred-work worker later retries the saved receipt by ID. > - This change gives wake-queue responsibility for saving the input before it confirms deferral. ## Linked Issues or Issue Description **Current behavior** The gate and the write that makes a blocked wake recoverable have different owners. Heartbeat also reports an admitted deferral when an update to an existing receipt matches no row. **Proposed behavior** Let the recovery admission operation save the first receipt or update the existing one in the same transaction. Confirm admission only when the write succeeds. Keep replaceable automatic signals on their existing diagnostic path. **Reason and benefit** Changes to recovery admission can now be checked against one rule. A blocked comment must have a durable receipt and remain recoverable across delivery attempts. Related changes include [#22292](https://github.com/paperclipai/paperclip/pull/22292) and [#22290](https://github.com/paperclipai/paperclip/pull/22290). ## What Changed - Added a transaction-bound deferral operation to wake-queue recovery. It handles existing receipts, saved input, and replaceable system signals. - Scoped retry updates to the company, issue, agent, deferred status, and receipt ID. A stale receipt cannot confirm admission or cause a new receipt to be inserted. - Kept case refusals and issue state callbacks in heartbeat. A saved comment uses the existing delivery worker and post-commit effects. ## Verification - Retried a saved comment by receipt ID through the real heartbeat and run-control entry points. While the hold remained, no run started. After the hold cleared, repeated retry delivered the comment to exactly one run. - Forced a rollback after the wake-queue Postgres adapter saved a receipt. The test observed the receipt within the transaction and confirmed its absence after rollback. Tested missing receipts and receipts for another issue or agent. The admission application test covers error propagation and post-commit effects when the transaction fails. - The test-only revision `9f443f9e6b` passed 9 focused PostgreSQL and recovery checks and all 15 issue-wake admission application tests. - The preceding production revision `eb37328472` passed the server TypeScript check, the module boundary check, and 498 tests across five queue and recovery suites. Twelve pairs of ordinary comments issued the same SQL before and after that production change (42 statements for a fresh wake and 35 for a join). - Both updated test files passed in full (249 tests). [CI attempt 3](https://github.com/paperclipai/paperclip/actions/runs/44572369812/attempts/3) passed on `9f443f9e6b` after retrying interrupted jobs. [Standard Greptile review](https://github.com/paperclipai/paperclip/pull/22332#issuecomment-7014835241) scored the same commit 5/5 with no open findings. ## Risks The affected path handles blocked admissions that may race a retry or a reassignment. Existing recovery gates, saved comment IDs, receipt identity, and notifications must survive that race. This change has no migration or new worker. The healthy admission path is expected to issue the same SQL. ## Model Used OpenAI Codex with GPT-6.2 Sol for implementation and verification. Codex directed and reviewed the change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
417802ca87
commit
87ef7097b6
13 files changed
+256
-71
No files matched your search
@@ -313,6 +313,62 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("saves the blocked comment receipt inside admission and rolls it back when admission fails", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const agentId = await seedAgent({ companyId });
|
||||
const issueId = await seedIssue({ companyId, assigneeAgentId: agentId });
|
||||
const authorUserId = randomUUID();
|
||||
const [comment] = await db.insert(issueComments).values({
|
||||
companyId, issueId, authorType: "user", authorUserId, body: "Please continue.",
|
||||
}).returning();
|
||||
const condition = { recoveryActionId: randomUUID(), reason: "execution_recovery", message: "Waiting for recovery." };
|
||||
const failure = new Error("Admission failed before commit");
|
||||
|
||||
await expect(createPostgresIssueWakeAdmission(db)({ companyId, issueId }, async (tx, { deferBlockedWake }) => {
|
||||
await expect(deferBlockedWake({
|
||||
commentId: comment.id, contextSnapshot: { issueId }, condition,
|
||||
coalesce: false, publications: [], nextAction: "Inspect the stopped run.",
|
||||
request: { agentId, source: "automation", triggerDetail: "system", reason: "issue_commented",
|
||||
payload: { issueId, commentId: comment.id }, requestedByActorType: "user",
|
||||
requestedByActorId: authorUserId, idempotencyKey: null },
|
||||
})).resolves.toEqual({ kind: "deferred", admitted: true });
|
||||
const [receipt] = await tx.select().from(agentWakeupRequests);
|
||||
expect(receipt).toMatchObject({ companyId, agentId, status: "deferred_issue_execution", runId: null,
|
||||
payload: { issueId, commentId: comment.id, executionWait: condition,
|
||||
_paperclipWakeContext: { wakeCommentIds: [comment.id] } } });
|
||||
throw failure;
|
||||
})).rejects.toBe(failure);
|
||||
expect(await db.select().from(agentWakeupRequests)).toEqual([]);
|
||||
});
|
||||
|
||||
it.each(["issue", "agent", "missing"])("ignores a recovery retry with a %s receipt mismatch", async mismatch => {
|
||||
const companyId = await seedCompany();
|
||||
const agentId = await seedAgent({ companyId });
|
||||
const issueId = await seedIssue({ companyId, assigneeAgentId: agentId });
|
||||
const receipt = {
|
||||
companyId,
|
||||
issueId: mismatch === "issue" ? await seedIssue({ companyId }) : issueId,
|
||||
agentId: mismatch === "agent" ? await seedAgent({ companyId, name: "Other agent" }) : agentId,
|
||||
};
|
||||
const requestId = await seedDeferredWake(receipt);
|
||||
const [before] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, requestId));
|
||||
|
||||
await createPostgresIssueWakeAdmission(db)({ companyId, issueId }, async (_tx, { deferBlockedWake }) => {
|
||||
const publications: Parameters<typeof deferBlockedWake>[0]["publications"] = [];
|
||||
const outcome = await deferBlockedWake({
|
||||
existingRequestId: mismatch === "missing" ? randomUUID() : requestId, contextSnapshot: {}, coalesce: false, publications,
|
||||
condition: { recoveryActionId: null, reason: "execution_recovery", message: "Waiting." }, nextAction: "Wait.",
|
||||
request: { agentId, source: "automation", triggerDetail: "system", reason: "issue_commented",
|
||||
payload: null, requestedByActorType: "user", requestedByActorId: null, idempotencyKey: null },
|
||||
});
|
||||
expect(outcome).toEqual({ kind: "deferred" });
|
||||
expect(publications).toEqual([]);
|
||||
return outcome;
|
||||
});
|
||||
expect(await db.select().from(agentWakeupRequests)).toEqual([before]);
|
||||
expect(await db.select().from(activityLog)).toEqual([]);
|
||||
});
|
||||
|
||||
it("checks conversation reset eligibility using the row refreshed under the admission lock", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const agentId = await seedAgent({ companyId });
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
import { notifyStrandedAgentStatus } from "../../../services/live-run-recovery-notifications.js";
|
||||
import { notifyDeferredExecutionWork } from "../../../services/deferred-execution-notifications.js";
|
||||
import { writeDeferredExecutionWait } from "../../../services/deferred-execution-wait.js";
|
||||
import { recordExecutionWait } from "../../../services/execution-wait.js";
|
||||
import type { ActivityPublication } from "../../../services/activity-log.js";
|
||||
import { notifyQueuedRunDispatch, notifyRunDispatch } from "../../../services/run-dispatch-notifications.js";
|
||||
import { notifyStatusCardIssueChange } from "../../../services/status-card-notifications.js";
|
||||
import { notifyExternalConversationSettlement } from "../../../services/external-conversation-settlement-notifications.js";
|
||||
@@ -17,6 +20,7 @@ import { currentConversationCommentCondition, isConversation, isConversationRese
|
||||
import { getExecutionBlocker, getExecutionOwnershipBlocker, hasPendingRunCleanup } from "../../../services/execution-blocker.js";
|
||||
import { preserveTaskRemoteWorkspaceWriter } from "../../../services/native-runtime/native-failed-workspace-recovery.js";
|
||||
import type { WakeRecoveryAdmissionStore } from "../application/recovery-admission.js";
|
||||
import { createWakeRecoveryDeferral } from "../application/recovery-deferral.js";
|
||||
import { nativeManualAdmissionHoldsExecution } from "../../../services/native-runtime/native-runner-ownership.js";
|
||||
import { and, asc, eq, inArray, isNull, ne, notInArray, or, sql } from "drizzle-orm";
|
||||
import { readReleaseRun, type ReleaseRun } from "./release-run-read.js";
|
||||
@@ -1332,6 +1336,7 @@ export function createPostgresIssueWakeAdmission(db: Db) {
|
||||
lockedIssue: IssueAdmissionRow | null;
|
||||
lockIssue: () => Promise<IssueAdmissionRow | null>;
|
||||
recovery: WakeRecoveryAdmissionStore<IssueWakeContinuationInput, IssueWakeContinuationResult>;
|
||||
deferBlockedWake: ReturnType<typeof createWakeRecoveryDeferral<ActivityPublication>>;
|
||||
reader: WakeAdmissionReader;
|
||||
writer: WakeAdmissionWriter;
|
||||
fresh: FreshIssueRunStore<HeartbeatRunRow>;
|
||||
@@ -1348,6 +1353,23 @@ export function createPostgresIssueWakeAdmission(db: Db) {
|
||||
return lockedIssue;
|
||||
};
|
||||
await lockIssue();
|
||||
const deferBlockedWake = createWakeRecoveryDeferral<ActivityPublication>(issueId, Boolean(durableChatRequest), {
|
||||
updateWait({ requestId, agentId, condition, publications }) {
|
||||
return writeDeferredExecutionWait(tx as unknown as Db, {
|
||||
companyId, issueId, agentId, requestId,
|
||||
}, condition, { replace: true, publications });
|
||||
},
|
||||
async insertDeferred(request) {
|
||||
await notifyDeferredExecutionWork(tx, { companyId });
|
||||
await tx.insert(agentWakeupRequests).values({ ...request, companyId, status: DEFERRED_WAKE_STATUS });
|
||||
},
|
||||
async recordWait({ request, condition, nextAction, coalesce }) {
|
||||
await recordExecutionWait(tx as unknown as Db, {
|
||||
issueId, condition, coalesce,
|
||||
request: { ...request, companyId, reason: "execution_reconciliation_required", error: nextAction },
|
||||
});
|
||||
},
|
||||
});
|
||||
return evaluate(tx, {
|
||||
lockedIssue,
|
||||
lockIssue,
|
||||
@@ -1383,6 +1405,10 @@ export function createPostgresIssueWakeAdmission(db: Db) {
|
||||
return admitExplicitNativeContinuation({ ...input, db: tx as unknown as Db, companyId, issueId, dryRun, onBlocked });
|
||||
},
|
||||
},
|
||||
deferBlockedWake: async input => {
|
||||
if (!lockedIssue) throw new Error("wake-queue: recovery deferral requires a locked issue");
|
||||
return deferBlockedWake(input);
|
||||
},
|
||||
reader: createWakeAdmissionReader(tx, companyId),
|
||||
writer: createWakeAdmissionWriter(tx, companyId),
|
||||
fresh: createFreshIssueRunStore(tx, companyId),
|
||||
|
||||
@@ -8,18 +8,7 @@ import {
|
||||
} from "../../../lib/queued-comment-context.js";
|
||||
import { hasInteractionContinuationWakeContext, isInteractionResolutionWakePayload } from "../domain/context.js";
|
||||
import { parseObject, readNonEmptyString } from "../domain/values.js";
|
||||
import type { DurableWakeAdmissionReceipt } from "./ports.js";
|
||||
|
||||
type WakeReceiptSeed = {
|
||||
agentId: string;
|
||||
source: string;
|
||||
triggerDetail: string | null;
|
||||
reason: string | null;
|
||||
payload: Record<string, unknown> | null;
|
||||
requestedByActorType: string | null;
|
||||
requestedByActorId: string | null;
|
||||
idempotencyKey: string | null;
|
||||
};
|
||||
import type { DurableWakeAdmissionReceipt, WakeReceiptSeed } from "./ports.js";
|
||||
|
||||
type QueuedRunSeed = {
|
||||
id?: string;
|
||||
|
||||
@@ -321,6 +321,17 @@ export type DurableWakeAdmissionReceipt = {
|
||||
requestedAt: Date;
|
||||
};
|
||||
|
||||
export type WakeReceiptSeed = {
|
||||
agentId: string;
|
||||
source: string;
|
||||
triggerDetail: string | null;
|
||||
reason: string | null;
|
||||
payload: Record<string, unknown> | null;
|
||||
requestedByActorType: string | null;
|
||||
requestedByActorId: string | null;
|
||||
idempotencyKey: string | null;
|
||||
};
|
||||
|
||||
export type CoalescedDeferredAdmissionReceipt =
|
||||
Omit<DurableWakeAdmissionReceipt, "id" | "requestedAt"> &
|
||||
Partial<Pick<DurableWakeAdmissionReceipt, "id" | "requestedAt">> & {
|
||||
|
||||
@@ -15,7 +15,7 @@ describe("wake recovery admission", () => {
|
||||
const prepare = createWakeRecoveryAdmission({
|
||||
readBlocker: async () => ({ blocker, verifiedReset: false }),
|
||||
admitExplicit,
|
||||
}, vi.fn(), vi.fn());
|
||||
}, vi.fn(), vi.fn(), vi.fn());
|
||||
|
||||
const recovery = await prepare(wake);
|
||||
expect(recovery).not.toHaveProperty("admitBusy");
|
||||
@@ -33,7 +33,7 @@ describe("wake recovery admission", () => {
|
||||
const prepare = createWakeRecoveryAdmission({
|
||||
readBlocker: async () => ({ blocker, verifiedReset: false }),
|
||||
admitExplicit,
|
||||
}, admitBusy, admitFresh);
|
||||
}, admitBusy, admitFresh, vi.fn());
|
||||
|
||||
const busy = await (await prepare(wake)).checkBusyAdmission();
|
||||
expect(busy.kind).toBe("allowed");
|
||||
@@ -53,7 +53,7 @@ describe("wake recovery admission", () => {
|
||||
const prepare = createWakeRecoveryAdmission({
|
||||
readBlocker: async () => ({ blocker: null, verifiedReset: false }),
|
||||
admitExplicit,
|
||||
}, vi.fn(), admitFresh);
|
||||
}, vi.fn(), admitFresh, vi.fn());
|
||||
const busy = await (await prepare(wake)).checkBusyAdmission();
|
||||
expect(admitExplicit).not.toHaveBeenCalled();
|
||||
if (busy.kind !== "allowed") throw new Error("Expected busy admission");
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import type { ExecutionBlocker } from "@paperclipai/shared";
|
||||
import type { createWakeRecoveryDeferral } from "./recovery-deferral.js";
|
||||
|
||||
type ContinuationWait = { reason: string; message: string };
|
||||
type AdmissionBlocker = Pick<ExecutionBlocker, "recoveryActionId" | "cause" | "nextAction" | "runId">;
|
||||
@@ -19,10 +20,11 @@ export type WakeRecoveryAdmissionStore<ExplicitInput, ExplicitResult> = {
|
||||
Promise<ExplicitResult | null>;
|
||||
};
|
||||
|
||||
export function createWakeRecoveryAdmission<ExplicitInput extends ExplicitContinuationIntent, ExplicitResult, Busy, Fresh>(
|
||||
export function createWakeRecoveryAdmission<ExplicitInput extends ExplicitContinuationIntent, ExplicitResult, Busy, Fresh, Publication>(
|
||||
store: WakeRecoveryAdmissionStore<ExplicitInput, ExplicitResult>,
|
||||
admitBusy: Busy,
|
||||
admitFresh: Fresh,
|
||||
defer: ReturnType<typeof createWakeRecoveryDeferral<Publication>>,
|
||||
) {
|
||||
return async function prepareRecovery(
|
||||
input: ExplicitInput & WakeRecoveryReadInput & {
|
||||
@@ -36,6 +38,7 @@ export function createWakeRecoveryAdmission<ExplicitInput extends ExplicitContin
|
||||
return {
|
||||
blocker,
|
||||
ownsReconstruction,
|
||||
defer,
|
||||
async checkBusyAdmission() {
|
||||
let wait: ContinuationWait | undefined;
|
||||
const onBlocked = (reason: string, message: string) => { wait = { reason, message }; };
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { createWakeRecoveryDeferral } from "./recovery-deferral.js";
|
||||
|
||||
const condition = { recoveryActionId: "action", reason: "execution_recovery", message: "Waiting for recovery." };
|
||||
const request = {
|
||||
agentId: "agent", source: "assignment", triggerDetail: "system", reason: "issue_commented",
|
||||
payload: { issueId: "other" }, requestedByActorType: "user", requestedByActorId: "user",
|
||||
idempotencyKey: "user-wake",
|
||||
};
|
||||
const input = { request, condition, contextSnapshot: { wakeCommentIds: ["previous", "comment"] },
|
||||
nextAction: "Inspect recovery.", coalesce: false, publications: [] };
|
||||
|
||||
function fixture() {
|
||||
const store = { updateWait: vi.fn(), insertDeferred: vi.fn(), recordWait: vi.fn() };
|
||||
return { ...store, defer: createWakeRecoveryDeferral("issue", false, store) };
|
||||
}
|
||||
|
||||
describe("wake recovery deferral", () => {
|
||||
it("saves a scoped comment receipt before confirming admission", async () => {
|
||||
const f = fixture();
|
||||
f.insertDeferred.mockRejectedValueOnce(new Error("save failed"));
|
||||
await expect(f.defer({ ...input, commentId: "comment" })).rejects.toThrow("save failed");
|
||||
await expect(f.defer({ ...input, commentId: "comment" })).resolves.toEqual({ kind: "deferred", admitted: true });
|
||||
expect(f.insertDeferred).toHaveBeenLastCalledWith({ ...request, payload: {
|
||||
issueId: "issue", commentId: "comment", executionWait: condition,
|
||||
_paperclipWakeContext: { wakeCommentIds: ["previous", "comment"], wakeCommentId: "comment", commentId: "comment" },
|
||||
} });
|
||||
expect(f.recordWait).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([false, true])("updates a requested receipt without replacing it (saved: %s)", async saved => {
|
||||
const f = fixture();
|
||||
f.updateWait.mockResolvedValue(saved);
|
||||
await expect(f.defer({ ...input, existingRequestId: "existing", commentId: "comment" })).resolves.toEqual(
|
||||
saved ? { kind: "deferred", admitted: true } : { kind: "deferred" },
|
||||
);
|
||||
expect(f.updateWait).toHaveBeenCalledExactlyOnceWith({ requestId: "existing", agentId: request.agentId, condition, publications: [] });
|
||||
expect(f.insertDeferred).not.toHaveBeenCalled();
|
||||
expect(f.recordWait).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("records a diagnostic for a blocked signal without saved work", async () => {
|
||||
const f = fixture();
|
||||
await expect(f.defer({ ...input, contextSnapshot: {}, coalesce: true })).resolves.toEqual({ kind: "deferred", admitted: true });
|
||||
expect(f.recordWait).toHaveBeenCalledExactlyOnceWith({ request, condition, nextAction: input.nextAction, coalesce: true });
|
||||
expect(f.insertDeferred).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,56 @@
|
||||
import {
|
||||
DEFERRED_WAKE_CONTEXT_KEY,
|
||||
queuedCommentIdsFromRunContext,
|
||||
withQueuedCommentIdsInWakePayload,
|
||||
} from "../../../lib/queued-comment-context.js";
|
||||
import { hasInteractionContinuationWakeContext } from "../domain/context.js";
|
||||
import { readNonEmptyString } from "../domain/values.js";
|
||||
import type { DurableWakeAdmissionReceipt, WakeReceiptSeed } from "./ports.js";
|
||||
|
||||
type DeferralCondition = { recoveryActionId: string | null; reason: string; message: string };
|
||||
type ReceiptSeed = WakeReceiptSeed & Partial<DurableWakeAdmissionReceipt>;
|
||||
|
||||
export type WakeRecoveryDeferralStore<Publication> = {
|
||||
updateWait(input: { requestId: string; agentId: string; condition: DeferralCondition; publications: Publication[] }): Promise<boolean>;
|
||||
insertDeferred(request: ReceiptSeed): Promise<void>;
|
||||
recordWait(input: { request: ReceiptSeed; condition: DeferralCondition; nextAction: string; coalesce: boolean }): Promise<void>;
|
||||
};
|
||||
|
||||
export function createWakeRecoveryDeferral<Publication>(
|
||||
issueId: string,
|
||||
durableChat: boolean,
|
||||
store: WakeRecoveryDeferralStore<Publication>,
|
||||
) {
|
||||
return async function defer({
|
||||
request, existingRequestId, commentId, contextSnapshot, condition, nextAction, coalesce, publications,
|
||||
}: {
|
||||
request: ReceiptSeed;
|
||||
existingRequestId?: string;
|
||||
commentId?: string | null;
|
||||
contextSnapshot: Record<string, unknown>;
|
||||
condition: DeferralCondition;
|
||||
nextAction: string;
|
||||
coalesce: boolean;
|
||||
publications: Publication[];
|
||||
}): Promise<{ kind: "deferred"; admitted?: true }> {
|
||||
if (existingRequestId) {
|
||||
const saved = await store.updateWait({ requestId: existingRequestId, agentId: request.agentId, condition, publications });
|
||||
return saved ? { kind: "deferred", admitted: true } : { kind: "deferred" };
|
||||
}
|
||||
if (durableChat || commentId || hasInteractionContinuationWakeContext(contextSnapshot) ||
|
||||
readNonEmptyString(contextSnapshot.nativeStatusWakeIntentId)) {
|
||||
const commentIds = [...new Set([...queuedCommentIdsFromRunContext(contextSnapshot), ...(commentId ? [commentId] : [])])];
|
||||
await store.insertDeferred({
|
||||
...request,
|
||||
payload: withQueuedCommentIdsInWakePayload({
|
||||
...request.payload, issueId,
|
||||
[DEFERRED_WAKE_CONTEXT_KEY]: contextSnapshot,
|
||||
executionWait: condition,
|
||||
}, commentIds),
|
||||
});
|
||||
} else {
|
||||
await store.recordWait({ request, condition, nextAction, coalesce });
|
||||
}
|
||||
return { kind: "deferred", admitted: true };
|
||||
};
|
||||
}
|
||||
@@ -15,6 +15,7 @@ import { createQueuedCommentSteeringAdapter, lockBoardQueuedComments, readBoardQ
|
||||
import type { QueuedCommentQueuePostgresAdapterDeps } from "./adapters/queued-comment-postgres.js";
|
||||
import { createFreshIssueRunAdmission } from "./application/fresh-issue-run.js";
|
||||
import { createWakeRecoveryAdmission } from "./application/recovery-admission.js";
|
||||
import type { ActivityPublication } from "../../services/activity-log.js";
|
||||
import { admitIssueWake, createIssueWakePreflight, type IssueWakeAdmissionEffects, type IssueWakeAdmissionOutcome } from "./application/issue-wake-admission.js";
|
||||
import { createAdmitWakeBehindIssueExecution, createReleaseIssueExecution } from "./application/use-cases.js";
|
||||
import {
|
||||
@@ -92,7 +93,8 @@ type LockedIssueAdmission = {
|
||||
lockedIssue: IssueAdmissionRow | null;
|
||||
lockIssue: () => Promise<IssueAdmissionRow | null>;
|
||||
prepareRecovery: ReturnType<typeof createWakeRecoveryAdmission<
|
||||
IssueWakeContinuationInput, IssueWakeContinuationResult, ReturnType<typeof createAdmitWakeBehindIssueExecution>, ReturnType<typeof createFreshIssueRunAdmission<HeartbeatRun>>
|
||||
IssueWakeContinuationInput, IssueWakeContinuationResult, ReturnType<typeof createAdmitWakeBehindIssueExecution>,
|
||||
ReturnType<typeof createFreshIssueRunAdmission<HeartbeatRun>>, ActivityPublication
|
||||
>>;
|
||||
};
|
||||
|
||||
@@ -121,7 +123,7 @@ export function createWakeQueue(db: Db, deps: WakeQueueDeps) {
|
||||
evaluateHeartbeatPolicy: (tx: IssueAdmissionDbTransaction, admission: LockedIssueAdmission) => Promise<IssueWakeAdmissionOutcome<HeartbeatRun>>,
|
||||
effects: IssueWakeAdmissionEffects<HeartbeatRun>,
|
||||
) {
|
||||
return admitIssueWake(() => withIssueAdmission({ companyId, issueId, durableChatRequest }, (tx, { lockedIssue, lockIssue, recovery, reader, writer, fresh }) => {
|
||||
return admitIssueWake(() => withIssueAdmission({ companyId, issueId, durableChatRequest }, (tx, { lockedIssue, lockIssue, recovery, deferBlockedWake, reader, writer, fresh }) => {
|
||||
const admitBusy = createAdmitWakeBehindIssueExecution({ reader, writer, helpers: deps.wakeAdmissionHelpers });
|
||||
const admitFresh = createFreshIssueRunAdmission(fresh);
|
||||
return evaluateHeartbeatPolicy(tx, {
|
||||
@@ -137,7 +139,7 @@ export function createWakeQueue(db: Db, deps: WakeQueueDeps) {
|
||||
throw new Error("wake-queue: admission does not match the locked issue");
|
||||
}
|
||||
return admitFresh(input);
|
||||
}),
|
||||
}, deferBlockedWake),
|
||||
});
|
||||
}), effects);
|
||||
},
|
||||
|
||||
@@ -19,13 +19,18 @@ export class PersistedExecutionWaitRefusal extends HttpError {
|
||||
* reloading. Pass `publications` when the caller's transaction is still open;
|
||||
* they must be published after it commits.
|
||||
*/
|
||||
export async function writeDeferredExecutionWait(db: Db, scope: { companyId: string; requestId: string },
|
||||
export async function writeDeferredExecutionWait(db: Db, scope: { companyId: string; requestId: string; issueId?: string; agentId?: string },
|
||||
condition: Record<string, unknown>, options: { replace?: boolean; publications?: ActivityPublication[] } = {}) {
|
||||
const pending: ActivityPublication[] = [];
|
||||
const receipt = and(
|
||||
eq(agentWakeupRequests.companyId, scope.companyId), eq(agentWakeupRequests.id, scope.requestId),
|
||||
eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
||||
scope.issueId ? sql`${agentWakeupRequests.payload}->>'issueId' = ${scope.issueId}` : undefined,
|
||||
scope.agentId ? eq(agentWakeupRequests.agentId, scope.agentId) : undefined,
|
||||
);
|
||||
const saved = await db.transaction(async tx => {
|
||||
const [current] = await tx.select({ payload: agentWakeupRequests.payload, agentId: agentWakeupRequests.agentId }).from(agentWakeupRequests)
|
||||
.where(and(eq(agentWakeupRequests.companyId, scope.companyId), eq(agentWakeupRequests.id, scope.requestId),
|
||||
eq(agentWakeupRequests.status, "deferred_issue_execution"))).for("update");
|
||||
.where(receipt).for("update");
|
||||
if (!current) return false;
|
||||
const previous = current.payload?.executionWait;
|
||||
const previousReason = previous && typeof previous === "object" && !Array.isArray(previous)
|
||||
@@ -37,8 +42,7 @@ export async function writeDeferredExecutionWait(db: Db, scope: { companyId: str
|
||||
${options.replace ? sql`'{}'::jsonb` : sql`coalesce(${agentWakeupRequests.payload}->'executionWait', '{}'::jsonb)`}
|
||||
|| ${JSON.stringify(condition)}::jsonb)`,
|
||||
updatedAt: new Date(),
|
||||
}).where(and(eq(agentWakeupRequests.companyId, scope.companyId), eq(agentWakeupRequests.id, scope.requestId),
|
||||
eq(agentWakeupRequests.status, "deferred_issue_execution"))).returning({ id: agentWakeupRequests.id });
|
||||
}).where(receipt).returning({ id: agentWakeupRequests.id });
|
||||
const issueId = current.payload?.issueId;
|
||||
if (saved && reasonChanged && typeof issueId === "string" && issueId && typeof condition.reason === "string") {
|
||||
const { publication } = await persistActivity(tx as unknown as Db, { companyId: scope.companyId, actorType: "system",
|
||||
|
||||
@@ -325,7 +325,7 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
|
||||
expect(await db.select().from(heartbeatRuns)).toHaveLength(kind === "fresh" ? 0 : 1);
|
||||
});
|
||||
|
||||
it("defers an ordinary user wake behind a recovery hold committed while acquiring the task lock", async () => {
|
||||
it("delivers a comment saved behind a concurrent recovery hold to one run on repeated retry", async () => {
|
||||
const { user, company, agent, issue, deps, queue } = await waitingConversationFixture();
|
||||
await db.update(issues).set({ conversationAgentId: null, conversationUserId: null, conversationState: null, status: "todo" })
|
||||
.where(eq(issues.id, issue.id));
|
||||
@@ -335,7 +335,7 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
|
||||
const actionId = randomUUID();
|
||||
function enqueue() {
|
||||
return queue.enqueueWakeup(agent.id, {
|
||||
source: "assignment", triggerDetail: "system", reason: "issue_commented",
|
||||
source: "automation", triggerDetail: "system", reason: "issue_commented",
|
||||
requestedByActorType: "user", requestedByActorId: user.id, payload: { issueId: issue.id, commentId: comment.id },
|
||||
});
|
||||
}
|
||||
@@ -364,11 +364,28 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
|
||||
|
||||
await expect(pending).resolves.toBeNull();
|
||||
expect(await db.select().from(heartbeatRuns)).toEqual([]);
|
||||
expect(await db.select().from(agentWakeupRequests)).toEqual([
|
||||
const receipts = await db.select().from(agentWakeupRequests);
|
||||
expect(receipts).toEqual([
|
||||
expect.objectContaining({ status: "deferred_issue_execution", runId: null,
|
||||
payload: expect.objectContaining({ issueId: issue.id, commentId: comment.id,
|
||||
executionWait: expect.objectContaining({ recoveryActionId: actionId }) }) }),
|
||||
]);
|
||||
const [receipt] = receipts;
|
||||
const controlDeps = runControlCallbacks(db);
|
||||
vi.mocked(controlDeps.enqueueWakeup).mockImplementation(queue.enqueueWakeup);
|
||||
const control = createHeartbeatRunControl(db, controlDeps);
|
||||
|
||||
deps.getSchedulingSuppression.mockResolvedValueOnce({ suppressed: false, reason: null });
|
||||
await control.resumeSavedLegacyComments(company.id, receipt.id);
|
||||
expect(await db.select().from(heartbeatRuns)).toEqual([]);
|
||||
|
||||
await db.update(issueRecoveryActions).set({ evidence: {} }).where(eq(issueRecoveryActions.id, actionId));
|
||||
deps.getSchedulingSuppression.mockResolvedValueOnce({ suppressed: false, reason: null });
|
||||
await control.resumeSavedLegacyComments(company.id, receipt.id);
|
||||
await control.resumeSavedLegacyComments(company.id, receipt.id);
|
||||
expect(await db.select().from(heartbeatRuns)).toEqual([expect.objectContaining({
|
||||
contextSnapshot: expect.objectContaining({ wakeCommentIds: [comment.id] }),
|
||||
})]);
|
||||
expect(deps.executeRun).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
|
||||
@@ -4019,49 +4019,21 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
|
||||
) => {
|
||||
if (caseRequest) waitForCaseAdmission("execution_reconciliation_required", executionBlocker.nextAction);
|
||||
opts.issueStateGuard?.onExecutionDeferred?.(executionBlocker.cause);
|
||||
const condition = { recoveryActionId: executionBlocker.recoveryActionId, ...continuationWait };
|
||||
if (executionWaitRequestId) {
|
||||
await writeDeferredExecutionWait(tx as unknown as Db, { companyId: agent.companyId,
|
||||
requestId: executionWaitRequestId }, condition, { replace: true, publications: issueActivityPublications });
|
||||
return { kind: "deferred" as const, admitted: true };
|
||||
}
|
||||
if (durableRequest || wakeCommentId ||
|
||||
hasInteractionContinuationWakeContext(enrichedContextSnapshot) ||
|
||||
readNonEmptyString(enrichedContextSnapshot.nativeStatusWakeIntentId)) {
|
||||
await notifyDeferredExecutionWork(tx, { companyId: agent.companyId });
|
||||
await tx.insert(agentWakeupRequests).values({
|
||||
...durableReceiptFields,
|
||||
companyId: agent.companyId, agentId, source, triggerDetail, reason,
|
||||
payload: withQueuedCommentIdsInWakePayload({
|
||||
...payload,
|
||||
issueId: issue.id,
|
||||
[DEFERRED_WAKE_CONTEXT_KEY]: enrichedContextSnapshot,
|
||||
executionWait: condition,
|
||||
}, [...new Set([
|
||||
...queuedCommentIdsFromRunContext(enrichedContextSnapshot),
|
||||
...(wakeCommentId ? [wakeCommentId] : []),
|
||||
])]),
|
||||
status: "deferred_issue_execution",
|
||||
return recovery.defer({
|
||||
existingRequestId: executionWaitRequestId,
|
||||
commentId: wakeCommentId,
|
||||
contextSnapshot: enrichedContextSnapshot,
|
||||
condition: { recoveryActionId: executionBlocker.recoveryActionId, ...continuationWait },
|
||||
nextAction: executionBlocker.nextAction,
|
||||
coalesce: coalesceExecutionWait,
|
||||
publications: issueActivityPublications,
|
||||
request: {
|
||||
...durableReceiptFields, agentId, source, triggerDetail, reason, payload,
|
||||
requestedByActorType: opts.requestedByActorType ?? null,
|
||||
requestedByActorId: opts.requestedByActorId ?? null,
|
||||
idempotencyKey: opts.idempotencyKey ?? null,
|
||||
});
|
||||
} else {
|
||||
await recordExecutionWait(tx as unknown as Db, {
|
||||
issueId: issue.id, condition, coalesce: coalesceExecutionWait,
|
||||
request: {
|
||||
...durableReceiptFields,
|
||||
companyId: agent.companyId, agentId, source, triggerDetail,
|
||||
reason: "execution_reconciliation_required",
|
||||
error: executionBlocker.nextAction,
|
||||
payload,
|
||||
requestedByActorType: opts.requestedByActorType ?? null,
|
||||
requestedByActorId: opts.requestedByActorId ?? null,
|
||||
idempotencyKey: opts.idempotencyKey ?? null,
|
||||
},
|
||||
});
|
||||
}
|
||||
return { kind: "deferred" as const, admitted: true };
|
||||
},
|
||||
});
|
||||
};
|
||||
const explicitContinuationRunId = randomUUID();
|
||||
const recovery = await prepareRecovery({
|
||||
|
||||
Reference in new issue
Block a user