Own the issue admission lock and dispatch after commit in wake-queue (#22272)

## Thinking Path

> - Paperclip admits work for a task while other agents and users may
also act on it.
> - A fresh run and a wake behind an active execution both require the
same issue lock.
> - Heartbeat previously opened that transaction and passed handles
across the module boundary.
> - The wake-queue module can bind both admission operations to the
locked company and issue.
> - The result of the database commit determines whether to emit a
queued event, dispatch a run, or return a saved result.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The transaction and handoff used by task-bound wake admission.

**Subsystem affected**

Server task admission.

**Current behavior**

Heartbeat opens the transaction, builds a scope for the busy path, and
dispatches based on the result. Its contention handler can reach
buffered side effects after a rollback.

**Proposed behavior**

Wake-queue owns the lock and the durable acknowledgement. The
application hands off a committed result. A rolled-back contention
attempt returns without publishing its buffered effects.

**Additional context**

Depends on #22268 and #22266.

## What Changed

- Bind the busy and fresh admission operations to a single company and
issue in Postgres. Remove the transaction scope object.
- Keep specialized heartbeat gates and continuation writes on the
acquired transaction.
- Buffer the workspace preflight activity publication until the
transaction commits.
- Centralize the handoff for new, coalesced, replayed, deferred, skipped
and durable results.

## Verification

On 1c5a069406, local qualification passed 94 application tests, 33
focused Postgres scenarios, and all four fresh/busy rollback cases.
Server typecheck and the module boundary check passed.

Run the wake-queue application and adapter tests and the heartbeat queue
and comment wake suites. Verify the rollback, replay, chat
acknowledgement, and tenant and issue mismatch cases. Run the server
typecheck and module boundary check.

## Risks

The ordering of commit, saved notification and dispatch is observable. A
recognized chat authorization contention now discards buffered effects
from the rolled-back transaction. Specialized heartbeat policy still
uses the locked database transaction.

## Model Used

OpenAI GPT-6.2 Sol (`gpt-6.2-sol`) implemented the change. OpenAI Codex
directed the work, reviewed the diff, and ran qualification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Nicky LeachandPaperclip authored and GitHub committed 2026-11-18 19:24:40 -08:00
1 parent eca7f85778
commit ebb98120dc
21 files changed
+440 -363

No files matched your search

@@ -1,7 +1,7 @@
import { randomUUID } from "node:crypto";
import { eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { activityLog, agents, agentWakeupRequests, companies, createDb, heartbeatRunEvents, heartbeatRuns, issueComments, issueThreadInteractions, issues, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, type Db } from "@paperclipai/db";
import { activityLog, agents, agentWakeupRequests, companies, createDb, heartbeatRunEvents, heartbeatRuns, issueComments, issueThreadInteractions, issues, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
import { filterRunMessageInteractionDelivery, hasAutomaticInteractionDeliveryHold, preparedRunInteractionDeliveryDenied, recheckQueuedCommentInteractionDelivery,
recheckQueuedRunInteractions, withInteractionDeliveryOrigin } from "./agent-interaction-delivery.js";
@@ -9,7 +9,7 @@ import { createHeartbeatQueue } from "./heartbeat/queue.js";
import { createHeartbeatQueueTestDependencies } from "../__tests__/helpers/heartbeat-queue.js";
import { withPluginInteractionOrigin } from "./plugin-interaction-context.js";
import { recoveryService } from "./recovery/service.js";
import { createAdmissionTransactionScope, createWakeAdmissionWriter } from "../modules/wake-queue/adapters/postgres.js";
import { createWakeAdmissionWriter } from "../modules/wake-queue/adapters/postgres.js";
const support = await getEmbeddedPostgresTestSupport();
describe.skipIf(!support.supported)("pending agent interaction delivery", () => {
@@ -113,8 +113,7 @@ describe.skipIf(!support.supported)("pending agent interaction delivery", () =>
const context = { issueId: f.issueId, wakeReason: reason, ...(assignmentFirst ? { wakeCommentId: messageId, wakeCommentIds: [messageId] } : {}) };
const actorType = assignmentFirst ? "user" : "agent", actorId = assignmentFirst ? "human" : f.sender;
const payload = withInteractionDeliveryOrigin({ issueId: f.issueId, _paperclipWakeContext: context }, actorType, actorId);
await db.transaction(tx => createWakeAdmissionWriter().coalesceIntoActiveExecutionRun(
createAdmissionTransactionScope(f.companyId, tx as unknown as Db), {
await db.transaction(tx => createWakeAdmissionWriter(tx, f.companyId).coalesceIntoActiveExecutionRun({
companyId: f.companyId, agentId: f.recipient, activeExecutionRunId: f.run.id, activeExecutionRunStatus: "queued",
mergedContextSnapshot: { ...context, wakeCommentId: messageId, wakeCommentIds: [messageId] },
source: "assignment", triggerDetail: "system", payload, requestedByActorType: actorType,
+59 -7
View File
@@ -12,11 +12,11 @@ import { createReleaseIssueExecution } from "../../modules/wake-queue/applicatio
import { closeProcessRunAdmission, resetProcessRunAdmissionForTests } from "./run-admission.js";
import { createHeartbeatQueue, DEFERRED_WAKE_CONTEXT_KEY, notifyOtherAgentsQueuedOnSettledIssue, shouldDeferFollowupWakeForSameIssue, shouldQueueFollowupForRunningIssueWake, type HeartbeatQueueDependencies } from "./queue.js";
import { createWakeQueue } from "../../modules/wake-queue/index.js";
import { TransactionScope } from "../../modules/wake-queue/application/ports.js";
import { enrichWakeContextSnapshot, mergeCoalescedContextSnapshot } from "./run-preparation.js";
import { withQueuedCommentIdsInWakePayload, withQueuedCommentIdsInRunContext } from "../issue-queued-comment-queue.js";
import { settleConversationTurn } from "../agent-conversations.js";
import { chatCompletionDeliveryService, recordChatCompletion, recordChatHandoff } from "../chat-completion-delivery.js";
import { createDurableChatWakeupRequest } from "../durable-chat-wakeup.js";
import { instanceSettingsService } from "../instance-settings.js";
import { createHeartbeatRunControl } from "./run-control.js";
import { callbacks as runControlCallbacks } from "./__tests__/run-control-fixtures.js";
@@ -255,6 +255,60 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
return { ...f, agent, service };
}
it.each([
["fresh", "authorization contention"], ["busy", "authorization contention"],
["fresh", "wrong locked issue"], ["busy", "wrong locked issue"],
] as const)("rolls back a %s chat admission on %s without publishing", async (kind, failure) => {
const { user, company, agent, issue, deps, queue } = await waitingConversationFixture();
await db.update(issues).set({ conversationAgentId: null, conversationUserId: null, conversationState: null, status: "todo" })
.where(eq(issues.id, issue.id));
if (kind === "busy") {
const [active] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id,
status: "running", invocationSource: "assignment", contextSnapshot: { issueId: issue.id } }).returning();
await db.update(issues).set({ executionRunId: active.id, executionAgentNameKey: agent.name, executionLockedAt: new Date() })
.where(eq(issues.id, issue.id));
deps.liveRunExecutions.has.mockImplementation((id) => id === active.id);
}
const [comment] = await db.insert(issueComments).values({ companyId: company.id, issueId: issue.id,
authorType: "user", authorUserId: user.id, body: "Please continue" }).returning();
const id = randomUUID();
const authorize = vi.fn(async () => {});
const onAdmitted = vi.fn(async (tx: Db, { runId }: { runId: string | null }) => {
const [receipt] = await tx.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, id));
expect(receipt).toMatchObject({ status: kind === "fresh" ? "queued" : "deferred_issue_execution", runId });
expect(Boolean(runId)).toBe(kind === "fresh");
throw Object.assign(new Error("authorization lock was held"), { code: "55P03" });
});
const request = createDurableChatWakeupRequest({ id, companyId: company.id, issueId: issue.id, agentId: agent.id,
commentId: comment.id, requestedByActorType: "user", requestedByActorId: user.id, requestedAt: new Date(), authorize, onAdmitted });
const [otherIssue] = failure === "wrong locked issue" ? await db.insert(issues).values({
companyId: company.id, assigneeAgentId: agent.id, title: "Another task",
}).returning() : [];
const publishIssueActivity = vi.fn();
const admit = deps.wakeQueue.admitIssueWake.getMockImplementation()!;
deps.wakeQueue.admitIssueWake.mockImplementation((target, evaluate, effects) => admit(target,
(tx, admission) => evaluate(tx, otherIssue ? {
admitFreshIssueRun: (input) => admission.admitFreshIssueRun({ ...input,
issue: { ...input.issue, id: otherIssue.id }, contextSnapshot: { ...input.contextSnapshot, issueId: otherIssue.id } }),
admitWakeBehindIssueExecution: (input) => admission.admitWakeBehindIssueExecution({ ...input,
issueId: otherIssue.id, contextSnapshot: { ...input.contextSnapshot, issueId: otherIssue.id } }),
} : admission), { ...effects, publishIssueActivity }));
const wake = queue.enqueueWakeup(agent.id, {
source: "assignment", triggerDetail: "system", reason: "External chat message received",
requestedByActorType: "user", requestedByActorId: user.id, allowRunCoalescing: false, durableChatRequest: request,
payload: { issueId: issue.id, commentId: comment.id, mutation: "chat_message_received" },
});
if (otherIssue) await expect(wake).rejects.toThrow(/does not match the locked/);
else await expect(wake).resolves.toBeNull();
expect(authorize).toHaveBeenCalledTimes(1);
expect(onAdmitted).toHaveBeenCalledTimes(otherIssue ? 0 : 1);
expect(publishIssueActivity).not.toHaveBeenCalled();
expect(deps.executeRun).not.toHaveBeenCalled();
expect(await db.select().from(agentWakeupRequests)).toEqual([]);
expect(await db.select().from(heartbeatRuns)).toHaveLength(kind === "fresh" ? 0 : 1);
});
it("admits the owner's context compaction maintenance request while its conversation waits", async () => {
const f = await waitingCompactionFixture();
const accepted = await f.service.request({ companyId: f.company.id, issue: f.issue, userId: f.user.id });
@@ -287,8 +341,7 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
mergeCoalescedContextSnapshot, shouldDeferFollowupWakeForSameIssue, shouldQueueFollowupForRunningIssueWake },
recovery: { escalateStrandedAssignedIssue: async () => undefined, escalateStrandedRecoveryIssueInPlace: async () => undefined },
});
f.deps.wakeQueue.createAdmissionTransactionScope.mockImplementation(admission.createAdmissionTransactionScope);
f.deps.wakeQueue.admitWakeBehindIssueExecution.mockImplementation(admission.admitWakeBehindIssueExecution);
f.deps.wakeQueue.admitIssueWake.mockImplementation(admission.admitIssueWake);
let releaseAvailability!: () => void;
const bothAvailable = new Promise<void>(resolve => { releaseAvailability = resolve; });
let arrivals = 0;
@@ -333,7 +386,7 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
const context = enrichWakeContextSnapshot({ contextSnapshot: { ...options.contextSnapshot },
reason: options.reason ?? null, source, triggerDetail, payload: options.payload ?? null });
await db.transaction(async tx => {
await createWakeAdmissionWriter().insertNewDeferredWake(TransactionScope.create(f.company.id, tx), {
await createWakeAdmissionWriter(tx, f.company.id).insertNewDeferredWake({
companyId: f.company.id, agentId, source, triggerDetail,
payload: { ...options.payload, manualUserWake: true, [DEFERRED_WAKE_CONTEXT_KEY]: context.contextSnapshot },
requestedByActorType: options.requestedByActorType ?? null,
@@ -553,7 +606,7 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
}).returning();
await db.transaction(async (tx) => {
const store = createFreshIssueRunStore(TransactionScope.create(local.company.id, tx), local.company.id);
const store = createFreshIssueRunStore(tx, local.company.id);
expect(await store.saveReceipt({
kind: "promote", wakeId: localDeferredWake.id, agentId: foreignAgent.id, payload: null,
})).toBeUndefined();
@@ -889,8 +942,7 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
mergeCoalescedContextSnapshot, shouldDeferFollowupWakeForSameIssue, shouldQueueFollowupForRunningIssueWake },
recovery: { escalateStrandedAssignedIssue: async () => undefined, escalateStrandedRecoveryIssueInPlace: async () => undefined },
});
deps.wakeQueue.createAdmissionTransactionScope.mockImplementation(admission.createAdmissionTransactionScope);
deps.wakeQueue.admitWakeBehindIssueExecution.mockImplementation(admission.admitWakeBehindIssueExecution);
deps.wakeQueue.admitIssueWake.mockImplementation(admission.admitIssueWake);
const queue = createHeartbeatQueue(db, deps);
if (scenario === "running") {
deps.canStartNewExternalBotWork.mockResolvedValue(true);
+64 -107
View File
@@ -91,7 +91,6 @@ import {
DEFERRED_CHAT_WAKE_RETRY_KEY,
assertDurableChatWakeupReceipt,
assertDurableChatWakeupRequest,
withDurableChatAdmission,
authorizeFailedChatRunRetryWake,
FailedChatRunRetryAuthorizationError,
unadmittedChatWakeupCondition,
@@ -219,7 +218,7 @@ import type { budgetService } from "../budgets.js";
import { isBoardOwnedBudgetBlock } from "../budget-block-ownership.js";
import type { instanceSettingsService } from "../instance-settings.js";
import type { createRunDispatch } from "../../modules/run-dispatch/index.js";
import { bindFreshIssueRunAdmission, hasRetainedExternalChatRetry, type PostCommitEffect as WakeQueuePostCommitEffect, type createWakeQueue } from "../../modules/wake-queue/index.js";
import { hasRetainedExternalChatRetry, type PostCommitEffect as WakeQueuePostCommitEffect, type createWakeQueue } from "../../modules/wake-queue/index.js";
import { crashPoint } from "@paperclipai/shared/crash-points";
import { emitSystemTaskBlockedAfterCommit } from "../product-telemetry.js";
@@ -248,7 +247,7 @@ export interface HeartbeatQueueDependencies extends Pick<HeartbeatRetryDependenc
budgets: Pick<ReturnType<typeof budgetService>, "getInvocationBlock">;
instanceSettings: Pick<ReturnType<typeof instanceSettingsService>, "getExperimental">;
runDispatch: Pick<ReturnType<typeof createRunDispatch>, "cancelStaleQueuedRun">;
wakeQueue: Pick<ReturnType<typeof createWakeQueue>, "createAdmissionTransactionScope" | "admitWakeBehindIssueExecution">;
wakeQueue: Pick<ReturnType<typeof createWakeQueue>, "admitIssueWake">;
activeRunExecutions: Set<string>;
activeRunExecutionPromises: Set<Promise<void>>;
activeWakeupPromises: Set<Promise<unknown>>;
@@ -3521,21 +3520,15 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
}
if (issueId) {
// Mention-triggered wakes can request input from another agent, but they must
// still respect the issue execution lock so a second agent cannot start on the
// same issue workspace while the assignee already has a live run.
const agentNameKey = normalizeAgentNameKey(agent.name);
const cancelledRunsToEmit: (typeof heartbeatRuns.$inferSelect)[] = [];
const issueActivityPublications: ActivityPublication[] = [];
const issuePostCommitActions: IssuePostCommitAction[] = [];
const outcome = await db
.transaction(withDurableChatAdmission(durableRequest, async (tx) => {
await tx.execute(
sql`select id from issues where id = ${issueId} and company_id = ${agent.companyId} for update`,
);
return wakeQueue.admitIssueWake(
{ companyId: agent.companyId, issueId, durableChatRequest: durableRequest },
async (tx, { admitWakeBehindIssueExecution, admitFreshIssueRun }) => {
if (executionWaitRequestId) {
const [pending] = await tx.select().from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.id, executionWaitRequestId), eq(agentWakeupRequests.companyId, agent.companyId),
@@ -4730,7 +4723,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
resolvedStrategy,
hasResolvablePriorSessionWorkspace,
},
});
}, issueActivityPublications);
return { kind: "skipped" as const };
}
}
@@ -4802,46 +4795,39 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
return { kind: "coalesced" as const, run: activeExecutionRun };
}
const admissionScope = wakeQueue.createAdmissionTransactionScope(
agent.companyId,
tx as unknown as Db,
);
const admission = await wakeQueue.admitWakeBehindIssueExecution(
admissionScope,
{
companyId: agent.companyId,
issueId: issue.id,
agentId,
agentNameKey,
issueExecutionAgentNameKey: issue.executionAgentNameKey,
activeExecutionRun: {
id: activeExecutionRun.id,
agentId: activeExecutionRun.agentId,
status: activeExecutionRun.status,
contextSnapshot: activeExecutionRun.contextSnapshot,
wakeupRequestId: activeExecutionRun.wakeupRequestId,
},
allowRunCoalescing: isConversation(issue) ? false : opts.allowRunCoalescing,
durableReceipt: receiptRequest
? {
id: receiptRequest.id,
requestedAt: receiptRequest.requestedAt,
}
: undefined,
reason,
liveRunExecutions,
wakeCommentId,
forceFreshSession:
enrichedContextSnapshot.forceFreshSession === true,
contextSnapshot: enrichedContextSnapshot,
source,
triggerDetail,
payload,
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
const admission = await admitWakeBehindIssueExecution({
companyId: agent.companyId,
issueId: issue.id,
agentId,
agentNameKey,
issueExecutionAgentNameKey: issue.executionAgentNameKey,
activeExecutionRun: {
id: activeExecutionRun.id,
agentId: activeExecutionRun.agentId,
status: activeExecutionRun.status,
contextSnapshot: activeExecutionRun.contextSnapshot,
wakeupRequestId: activeExecutionRun.wakeupRequestId,
},
);
allowRunCoalescing: isConversation(issue) ? false : opts.allowRunCoalescing,
durableReceipt: receiptRequest
? {
id: receiptRequest.id,
requestedAt: receiptRequest.requestedAt,
}
: undefined,
reason,
liveRunExecutions,
wakeCommentId,
forceFreshSession:
enrichedContextSnapshot.forceFreshSession === true,
contextSnapshot: enrichedContextSnapshot,
source,
triggerDetail,
payload,
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
});
if (admission.kind === "coalesced") {
return {
@@ -5075,7 +5061,6 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
}, issueActivityPublications);
}
const admitFreshIssueRun = bindFreshIssueRunAdmission(agent.companyId, tx as unknown as Db);
const newRun = await admitFreshIssueRun({
receipt: {
agentId,
@@ -5130,61 +5115,33 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
// transitions to "running" — Fix A (lazy locking).
return { kind: "queued" as const, run: newRun };
}))
.catch((error) => {
if (isExternalChatWaitAuthorizationContention(error))
return { kind: "deferred" as const };
throw error;
});
// The admission has committed. What follows is dispatch of saved rows.
if (outcome.kind !== "deferred" && outcome.kind !== "skipped") admission?.saved();
for (const publication of issueActivityPublications) publishActivity(publication);
await executeIssuePostCommitActions(db, issuePostCommitActions);
// Telemetry for the cancelled runs is best-effort background work.
// Fire it here and never await it: none of the lifecycle work below,
// nor this function's return, depends on it, so a slow telemetry
// lookup must not delay them.
for (const cancelledRun of cancelledRunsToEmit) {
void emitAgentTaskRun(db, cancelledRun);
}
if (outcome.kind === "durable") {
return outcome.receipt.runId ? getRun(outcome.receipt.runId) : null;
}
if (outcome.kind === "deferred" || outcome.kind === "skipped") {
return null;
}
if (outcome.kind === "coalesced") {
await attachQueuedSetupRequest(agent);
await startNextQueuedRunForAgent(agent.id);
return outcome.run;
}
if (outcome.kind === "replayed") {
if (outcome.run.status === "queued") {
await attachQueuedSetupRequest(agent);
await startNextQueuedRunForAgent(agent.id);
}
return outcome.run;
}
const newRun = outcome.run;
publishLiveEvent({
companyId: newRun.companyId,
type: "heartbeat.run.queued",
payload: {
runId: newRun.id,
agentId: newRun.agentId,
invocationSource: newRun.invocationSource,
triggerDetail: newRun.triggerDetail,
wakeupRequestId: newRun.wakeupRequestId,
},
});
await attachQueuedSetupRequest(agent);
await startNextQueuedRunForAgent(agent.id);
return newRun;
{
saved: () => admission?.saved(),
publishIssueActivity() {
for (const publication of issueActivityPublications) publishActivity(publication);
},
executeIssueActions: () => executeIssuePostCommitActions(db, issuePostCommitActions),
cancelledRuns: cancelledRunsToEmit,
emitCancelledRun: (run) => emitAgentTaskRun(db, run),
readRun: getRun,
publishQueuedRun(run) {
publishLiveEvent({
companyId: run.companyId,
type: "heartbeat.run.queued",
payload: {
runId: run.id,
agentId: run.agentId,
invocationSource: run.invocationSource,
triggerDetail: run.triggerDetail,
wakeupRequestId: run.wakeupRequestId,
},
});
},
attachSetupRequest: () => attachQueuedSetupRequest(agent),
dispatch: () => startNextQueuedRunForAgent(agent.id),
},
);
}
if (durableRequest) throw new Error("chat_inbound_wakeup_binding_denied");
@@ -1,7 +1,7 @@
import { randomUUID } from "node:crypto";
import { resolve } from "node:path";
import { and, eq, sql } from "drizzle-orm";
import { agents, companies, completionContracts, databaseWorkPending, environmentLeases, reconcileDatabaseWork, heartbeatRunEvents, heartbeatRuns, issues, issueRelations,
import { agents, agentWakeupRequests, companies, completionContracts, databaseWorkPending, environmentLeases, reconcileDatabaseWork, heartbeatRunEvents, heartbeatRuns, issues, issueRelations,
issueTreeHolds, issueTreeHoldMembers, issueRecoveryActions, nativeRunFinalizations, nativeRunResults, nativeRunnerCheckpoints } from "@paperclipai/db";
import { expect, it, vi } from "vitest";
import type { NativeExecutionInputV1 } from "@paperclipai/paperclip-runner";
@@ -156,11 +156,12 @@ describeEmbeddedPostgres("remote native retry execution admission", () => {
const dependencies = createHeartbeatQueueTestDependencies(f.db);
dependencies.getAgent.mockResolvedValue(agent);
dependencies.instanceSettings.getExperimental.mockImplementation(() => instanceSettingsService(f.db).getExperimental());
dependencies.wakeQueue.admitWakeBehindIssueExecution.mockResolvedValue({ kind: "deferred" });
await createHeartbeatQueue(f.db, dependencies).enqueueWakeup(f.agentId, { source: "assignment", reason: "issue_assigned",
requestedByActorType: "system", payload: { issueId: f.issueId }, contextSnapshot: { issueId: f.issueId } });
expect(await createHeartbeatQueue(f.db, dependencies).enqueueWakeup(f.agentId, { source: "assignment", reason: "issue_assigned",
allowRunCoalescing: false, requestedByActorType: "system", payload: { issueId: f.issueId }, contextSnapshot: { issueId: f.issueId } })).toBeNull();
expect(dependencies.executeRun).not.toHaveBeenCalled();
expect(dependencies.wakeQueue.admitWakeBehindIssueExecution).toHaveBeenCalledOnce();
const [wake] = await f.db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, f.companyId),
sql`${agentWakeupRequests.payload}->>'issueId' = ${f.issueId}`));
expect(wake).toMatchObject({ status: "deferred_issue_execution", runId: null });
expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0])
.toMatchObject({ executionRunId: f.runId, checkoutRunId: f.runId });
});
@@ -1,13 +1,12 @@
import { createAdmissionTransactionScope, createWakeAdmissionReader, createWakeAdmissionWriter } from "../modules/wake-queue/adapters/postgres.js";
import { createAdmitWakeBehindIssueExecution } from "../modules/wake-queue/application/use-cases.js";
import { filterZombieCoalesceTarget, heartbeatService } from "./heartbeat.js";
import { createWakeAdmissionReader, createWakeAdmissionWriter } from "../modules/wake-queue/adapters/postgres.js";
import { heartbeatService } from "./heartbeat.js";
import { mergeCoalescedContextSnapshot } from "./heartbeat/run-preparation.js";
import { recoveryService } from "./recovery/index.js";
import { inspectWorkspaceSource } from "./workspace-source-inspection.js";
import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
import { isPreDispatchExecutionWaitVerified } from "./pre-dispatch-execution-wait.js";
import { heartbeatRunEvents, agents, approvals, budgetPolicies, costEvents, completionContracts, issueApprovals, issueTreeHolds, issueTreeHoldMembers, companies, createDb, environmentLeases, environments, heartbeatRuns, issueComments, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, agentWakeupRequests, workspaceOperations, plugins } from "@paperclipai/db";
import { createHeartbeatQueue, shouldDeferFollowupWakeForSameIssue, shouldQueueFollowupForRunningIssueWake } from "./heartbeat/queue.js";
import { createHeartbeatQueue } from "./heartbeat/queue.js";
import { createHeartbeatQueueTestDependencies } from "../__tests__/helpers/heartbeat-queue.js";
import { recordRequiredWorkspaceRestoreFailure } from "./legacy-execution-recovery.js";
import { admitExplicitNativeContinuation } from "./explicit-native-continuation.js";
@@ -1425,11 +1424,11 @@ describe("automatic retained workspace recovery", () => {
const incoming = { source: "issue.comment", issueId: f.issueId, wakeReason: "issue_commented",
wakeCommentId: followup.id, wakeCommentIds: [followup.id] };
await db.transaction(async tx => {
const scope = createAdmissionTransactionScope(f.companyId, tx as unknown as typeof db);
const reader = createWakeAdmissionReader(tx, f.companyId);
const actor = { companyId: f.companyId, wakeupRequestId: wake.id, requestedByActorType: "user", requestedByActorId: "operator" };
expect(await createWakeAdmissionReader().matchesActiveWakeActor(scope, actor)).toBe(true);
expect(await createWakeAdmissionReader().matchesActiveWakeActor(scope, { ...actor, requestedByActorId: "other-user" })).toBe(false);
await createWakeAdmissionWriter().coalesceIntoActiveExecutionRun(scope, {
expect(await reader.matchesActiveWakeActor(actor)).toBe(true);
expect(await reader.matchesActiveWakeActor({ ...actor, requestedByActorId: "other-user" })).toBe(false);
await createWakeAdmissionWriter(tx, f.companyId).coalesceIntoActiveExecutionRun({
companyId: f.companyId, agentId: original.agentId, activeExecutionRunId: queued.id, activeExecutionRunStatus: queued.status,
mergedContextSnapshot: mergeCoalescedContextSnapshot((await run(queued.id)).contextSnapshot, incoming,
{ preserveExistingInteractionContinuation: true }),
@@ -1454,12 +1453,15 @@ describe("automatic retained workspace recovery", () => {
await db.update(agentWakeupRequests).set({ runId: null }).where(eq(agentWakeupRequests.id, wake.id));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, queued.id));
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id));
const reader = createWakeAdmissionReader(), sameActor = vi.spyOn(reader, "matchesActiveWakeActor");
queueDeps.wakeQueue.createAdmissionTransactionScope.mockImplementation(createAdmissionTransactionScope);
queueDeps.wakeQueue.admitWakeBehindIssueExecution.mockImplementation(createAdmitWakeBehindIssueExecution({
reader, writer: createWakeAdmissionWriter(), helpers: { filterZombieCoalesceTarget, mergeCoalescedContextSnapshot,
shouldDeferFollowupWakeForSameIssue, shouldQueueFollowupForRunningIssueWake },
}));
const admitIssueWake = queueDeps.wakeQueue.admitIssueWake.getMockImplementation()!;
queueDeps.wakeQueue.admitIssueWake.mockImplementation((target, evaluate, effects) =>
admitIssueWake(target, (tx, admission) => evaluate(tx, {
...admission,
admitWakeBehindIssueExecution(input) {
expect(input.durableReceipt).toBeUndefined();
return admission.admitWakeBehindIssueExecution(input);
},
}), effects));
queueDeps.liveRunExecutions.has.mockImplementation(id => t.deps.activeRunExecutions.has(id));
const stopped = await run(f.runId);
if (kind !== "active") {
@@ -1511,8 +1513,6 @@ describe("automatic retained workspace recovery", () => {
const current = await run(admitted!.id);
const [claimedWake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, current.wakeupRequestId!));
expect(current.contextSnapshot?.wakeCommentIds).toEqual(ids);
expect(sameActor).not.toHaveBeenCalled();
expect(queueDeps.wakeQueue.admitWakeBehindIssueExecution.mock.calls.every(([, input]) => input.durableReceipt === undefined)).toBe(true);
expect(queueDeps.executeRun).not.toHaveBeenCalled();
return { ...t, queued: current, wake: claimedWake, followup, expectedIds: ids, latestAuthor: kind === "deferred-different" ? "third-user" : kind === "active" ? "second-user" : "operator" };
}
@@ -1887,10 +1887,10 @@ describe("automatic retained workspace recovery", () => {
}, 30_000);
it("preserves ordinary no-issue deferred merge semantics", async () => {
const f = await seed(), original = await run(f.runId), writer = createWakeAdmissionWriter();
const f = await seed(), original = await run(f.runId);
const [carrier] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: original.agentId,
source: "automation", status: "deferred_issue_execution", payload: { text: "original" }, requestedByActorType: "system" }).returning();
await db.transaction(tx => writer.mergeIntoExistingDeferredWake(createAdmissionTransactionScope(f.companyId, tx as unknown as typeof db), {
await db.transaction(tx => createWakeAdmissionWriter(tx, f.companyId).mergeIntoExistingDeferredWake({
companyId: f.companyId, existingDeferredWakeId: carrier.id, mergedPayload: { text: "merged" }, nextCoalescedCount: 1,
coalescedReceipt: { agentId: original.agentId, source: "automation", triggerDetail: null, reason: null, payload: { text: "saved" }, requestedByActorType: "system", requestedByActorId: null, idempotencyKey: null, runId: null },
}));
@@ -1901,12 +1901,12 @@ describe("automatic retained workspace recovery", () => {
});
it("rolls back saved primary and carrier overwrite if incoming mixed-author receipt persistence fails", async () => {
const f = await seed(), original = await run(f.runId), writer = createWakeAdmissionWriter();
const f = await seed(), original = await run(f.runId);
const [first] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId, authorType: "user", authorUserId: "operator", body: "First saved input" }).returning();
const [second] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId, authorType: "user", authorUserId: "second-user", body: "Second saved input" }).returning();
const [carrier] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: original.agentId,
source: "automation", status: "deferred_issue_execution", payload: { issueId: f.issueId, commentId: first.id }, requestedByActorType: "user", requestedByActorId: "operator" }).returning();
const merge = (tx: Parameters<Parameters<typeof db.transaction>[0]>[0], id?: string) => writer.mergeIntoExistingDeferredWake(createAdmissionTransactionScope(f.companyId, tx as unknown as typeof db), {
const merge = (tx: Parameters<Parameters<typeof db.transaction>[0]>[0], id?: string) => createWakeAdmissionWriter(tx, f.companyId).mergeIntoExistingDeferredWake({
companyId: f.companyId, existingDeferredWakeId: carrier.id, mergedPayload: { issueId: f.issueId, commentId: second.id }, nextCoalescedCount: 1,
coalescedReceipt: { ...(id ? { id } : {}), agentId: original.agentId, source: "automation", triggerDetail: null, reason: "issue_commented", payload: { issueId: f.issueId, commentId: second.id, coalescedIntoWakeupRequestId: "untrusted" }, requestedByActorType: "user", requestedByActorId: "second-user", idempotencyKey: null, runId: null },
});