refactor(wake-queue): enforce recovery before task admission (#22292)

## Thinking Path

> - Paperclip admits assignments and task comments while it holds the
task lock.
> - A task can have a recovery hold. An explicit continuation can
authorize additional work.
> - A wake can join an active execution after a provisional check. A
fresh run must settle the continuation in the transaction that creates
the run.
> - Heartbeat previously received both admission functions before either
check ran.
> - Make the wake-queue module release each function after the
corresponding recovery check passes.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Enforce recovery eligibility at the entry points that attach a wake to
an issue execution or create a fresh run.

**Subsystem affected**

Wake-queue and heartbeat issue admission.

**Current behavior**

Heartbeat selects when to read the blocker and apply the explicit
continuation before invoking the admission writers.

**Proposed behavior**

Wake-queue controls access to its two admission writers after the task
is locked. The same recovery and continuation services decide which work
may proceed.

**Additional context**

Depends on #22289. #22290 changes the storage cost of the blocker check
in a separate layer.

## What Changed

- Bind the recovery read to the issue and transaction selected by
wake-queue. Read the current issue lock again after durable chat
authorization when required.
- Expose the busy writer after the preliminary continuation check
passes. Expose the fresh writer after the transaction applies any
required continuation.
- Preserve heartbeat’s existing ordering for system fallback, budget and
daily limits, task retry, and durable receipts.

## Verification

On bde1778161, 420 tests passed across five suites. Server TypeScript
and the module dependency check passed. Final head f9d44c7728 removes
three inline comment lines. All five affected application and locked
PostgreSQL cases passed again on that commit. End-to-end fixture SQL and
lock counts were identical on the parent and the ownership layer. The
application tests exercise blocked recovery, provisional permission
followed by a refusal, and ordinary fresh admission. The PostgreSQL
tests cover current authorization state, scoped writes, rollback, and
deferred delivery.

GitHub selected middle-stack verification. All applicable checks passed
on f9d44c7728. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/44461026141)
· [Standard Greptile review,
5/5](https://github.com/paperclipai/paperclip/pull/22292#issuecomment-6999993272)

## Risks

An explicit continuation can settle a hold. Settling it before the later
gates could lose work if admission declines the wake. The final check
stays immediately before fresh admission in the same transaction.
Heartbeat still owns specialized receipt and fallback paths.

## Model Used

OpenAI GPT-6.2 Sol (gpt-6.2-sol) implemented the change. OpenAI Codex
directed and qualified it.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
This commit is contained in:
Nicky Leach authored and GitHub committed 2026-11-19 07:23:03 -08:00
1 parent 503268b806
commit 7ffc58a612
9 files changed
+271 -84

No files matched your search

+6 -3
View File
@@ -340,9 +340,12 @@ company and issue lock, the admission transaction, and the handoff after commit
for both fresh task runs and wakes arriving behind an active execution. Wake-queue
checks conversation eligibility before locking the task. The locked task supplies
the attribution fields used to select the responsible user for a fresh run.
Heartbeat uses the transaction for its specialized eligibility gates and
continuation writes. Wake-queue selects eligible saved comment receipts in request order and
binds them to the new run. Chat authorization and its admission callback run in
Wake-queue reads recovery state inside the admission transaction. It checks
continuation eligibility before allowing a wake to join active work, then
settles any explicit continuation before allowing a fresh run. Heartbeat
uses the same transaction for its other eligibility gates and specialized
receipt handling. Wake-queue selects eligible saved comment receipts in request
order and binds them to the new run. Chat authorization and its admission callback run in
the same transaction. Authorization contention defers the wake and rolls back
without publishing buffered effects.
`createHeartbeatQueue` binds these operations without querying or starting work.
@@ -257,6 +257,31 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => {
}
});
it("checks conversation reset eligibility using the row refreshed under the admission lock", async () => {
const companyId = await seedCompany();
const agentId = await seedAgent({ companyId });
const issueId = await seedIssue({ companyId, assigneeAgentId: agentId });
const userId = randomUUID();
const [comment] = await db.insert(issueComments).values({
companyId, issueId, authorType: "user", authorUserId: userId, body: "/new",
}).returning();
await db.insert(issueRecoveryActions).values({
companyId, sourceIssueId: issueId, kind: "active_run_watchdog", cause: "legacy_execution_requires_reconciliation",
fingerprint: randomUUID(), nextAction: "Inspect the stopped run.",
});
await createPostgresIssueWakeAdmission(db)({ companyId, issueId }, async (tx, { lockedIssue, lockIssue, recovery }) => {
const input = { commentId: comment.id, requestedByActorType: "user", requestedByActorId: userId };
expect(lockedIssue?.conversationAgentId).toBeNull();
await expect(recovery.readBlocker(input)).resolves.toMatchObject({ verifiedReset: false, blocker: { cause: "legacy_execution_requires_reconciliation" } });
await tx.update(issues).set({ conversationAgentId: agentId, conversationUserId: userId, conversationState: "active" })
.where(and(eq(issues.companyId, companyId), eq(issues.id, issueId)));
await expect(lockIssue()).resolves.toMatchObject({ conversationAgentId: agentId });
await expect(recovery.readBlocker(input)).resolves.toEqual({ verifiedReset: true, blocker: null });
return { kind: "skipped" };
});
});
it("locks an issue added after the initial admission lock missed", async () => {
const companyId = await seedCompany();
const issueId = randomUUID();
@@ -13,8 +13,10 @@ import { readRunCancellation } from "../../../services/run-cancellation.js";
import { withRunRedactionHistory } from "../../../services/run-secret-redaction.js";
import { CHAT_COMPLETION_WAKE_REASON, isCompletedOnboardingHandoffWake, settleReportedChatCompletions } from "../../../services/chat-completion-delivery.js";
import { earlierDeclinedOutcomeReply } from "../../../services/chat-interaction-outcome-report.js";
import { currentConversationCommentCondition } from "../../../services/agent-conversations.js";
import { currentConversationCommentCondition, isConversation, isConversationReset } from "../../../services/agent-conversations.js";
import { getExecutionBlocker, hasPendingRunCleanup } from "../../../services/execution-blocker.js";
import { preserveTaskRemoteWorkspaceWriter } from "../../../services/native-runtime/native-failed-workspace-recovery.js";
import type { WakeRecoveryAdmissionStore } from "../application/recovery-admission.js";
import { nativeManualAdmissionHoldsExecution } from "../../../services/native-runtime/native-runner-ownership.js";
import { and, asc, eq, inArray, isNull, ne, notInArray, or, sql } from "drizzle-orm";
import { readReleaseRun, type ReleaseRun } from "./release-run-read.js";
@@ -56,7 +58,7 @@ import { executionRetryAccounting } from "../../../services/execution-recovery-a
import { issueService } from "../../../services/issues.js";
import { issueRecoveryActionService } from "../../../services/issue-recovery-actions.js";
import { readContinuationAttempt } from "../../../services/recovery/run-liveness-continuations.js";
import { undeliveredLegacyUserCommentIds } from "../../../services/explicit-native-continuation.js";
import { admitExplicitNativeContinuation, undeliveredLegacyUserCommentIds } from "../../../services/explicit-native-continuation.js";
import { withRecoveryContext } from "../../../services/recovery/status-only-context.js";
import { parseIssueExecutionState } from "../../../services/issue-execution-policy.js";
import {
@@ -1320,6 +1322,8 @@ const issueAdmissionFields = {
};
export type IssueAdmissionRow = Pick<IssueRow, keyof typeof issueAdmissionFields>;
export type IssueWakeContinuationInput = Omit<Parameters<typeof admitExplicitNativeContinuation>[0], "db" | "companyId" | "issueId" | "dryRun" | "onBlocked">;
export type IssueWakeContinuationResult = NonNullable<Awaited<ReturnType<typeof admitExplicitNativeContinuation>>>;
export function createPostgresIssueWakeAdmission(db: Db) {
return async function withIssueAdmission(
@@ -1327,6 +1331,7 @@ export function createPostgresIssueWakeAdmission(db: Db) {
evaluate: (tx: IssueAdmissionDbTransaction, stores: {
lockedIssue: IssueAdmissionRow | null;
lockIssue: () => Promise<IssueAdmissionRow | null>;
recovery: WakeRecoveryAdmissionStore<IssueWakeContinuationInput, IssueWakeContinuationResult>;
reader: WakeAdmissionReader;
writer: WakeAdmissionWriter;
fresh: FreshIssueRunStore<HeartbeatRunRow>;
@@ -1335,15 +1340,41 @@ export function createPostgresIssueWakeAdmission(db: Db) {
try {
return await db.transaction(withDurableChatAdmission(durableChatRequest, async (tx) => {
// Mentions may target another agent, but every admission must share the issue lock.
let lockedIssue: IssueAdmissionRow | null = null;
const lockIssue = async () => {
const [issue] = await tx.select(issueAdmissionFields).from(issues)
.where(and(eq(issues.id, issueId), eq(issues.companyId, companyId))).for("update");
return issue ?? null;
lockedIssue = issue ?? null;
return lockedIssue;
};
const lockedIssue = await lockIssue();
await lockIssue();
return evaluate(tx, {
lockedIssue,
lockIssue,
recovery: {
async readBlocker({ commentId, requestedByActorType, requestedByActorId }) {
const issue = lockedIssue;
const userComment = requestedByActorType === "user" ? commentId : null;
const [resetComment] = isConversation(issue) && userComment && requestedByActorId
? await tx.select({ body: issueComments.body }).from(issueComments).where(and(
eq(issueComments.companyId, companyId), eq(issueComments.issueId, issueId),
eq(issueComments.id, userComment), eq(issueComments.authorUserId, requestedByActorId),
eq(issueComments.authorType, "user"), isNull(issueComments.deletedAt), isNull(issueComments.createdByRunId),
)).limit(1)
: [];
const verifiedReset = Boolean(resetComment && isConversationReset(resetComment.body));
const preserveWorkspace = userComment && issue?.executionWorkspaceId;
if (preserveWorkspace) await preserveTaskRemoteWorkspaceWriter(tx as unknown as Db, companyId, issueId);
const blocker = await getExecutionBlocker(tx as unknown as Db, companyId, issueId, {
conversationResetCommentId: verifiedReset ? userComment : null,
lockedIssue: preserveWorkspace ? undefined : issue ?? undefined,
});
return { blocker, verifiedReset };
},
admitExplicit(input, dryRun, onBlocked) {
return admitExplicitNativeContinuation({ ...input, db: tx as unknown as Db, companyId, issueId, dryRun, onBlocked });
},
},
reader: createWakeAdmissionReader(tx, companyId),
writer: createWakeAdmissionWriter(tx, companyId),
fresh: createFreshIssueRunStore(tx, companyId),
@@ -0,0 +1,64 @@
import { describe, expect, it, vi } from "vitest";
import { createWakeRecoveryAdmission } from "./recovery-admission.js";
const blocker = {
cause: "legacy_execution_requires_reconciliation" as const,
recoveryActionId: "recovery",
runId: "stopped",
nextAction: "Inspect the stopped run.",
};
const wake = { commentId: "message" };
describe("wake recovery admission", () => {
it("withholds both ordinary admission capabilities when recovery is required", async () => {
const admitExplicit = vi.fn().mockResolvedValue(null);
const prepare = createWakeRecoveryAdmission({
readBlocker: async () => ({ blocker, verifiedReset: false }),
admitExplicit,
}, vi.fn(), vi.fn());
const recovery = await prepare(wake);
expect(recovery).not.toHaveProperty("admitBusy");
expect(recovery).not.toHaveProperty("admitFresh");
const result = await recovery.checkBusyAdmission();
expect(result.kind).toBe("blocked");
expect(result).not.toHaveProperty("admitBusy");
expect(result).not.toHaveProperty("checkFreshAdmission");
expect(admitExplicit).toHaveBeenCalledExactlyOnceWith(wake, true, expect.any(Function));
});
it("requires actual fresh continuation after the successful busy dry run", async () => {
const admitExplicit = vi.fn().mockResolvedValueOnce({ previousRunId: "stopped" }).mockResolvedValueOnce(null);
const admitBusy = vi.fn(), admitFresh = vi.fn();
const prepare = createWakeRecoveryAdmission({
readBlocker: async () => ({ blocker, verifiedReset: false }),
admitExplicit,
}, admitBusy, admitFresh);
const busy = await (await prepare(wake)).checkBusyAdmission();
expect(busy.kind).toBe("allowed");
if (busy.kind !== "allowed") throw new Error("Expected busy admission");
expect(busy.admitBusy).toBe(admitBusy);
expect(admitExplicit).toHaveBeenCalledExactlyOnceWith(wake, true, expect.any(Function));
expect(busy).not.toHaveProperty("admitFresh");
const fresh = await busy.checkFreshAdmission();
expect(fresh.kind).toBe("blocked");
expect(fresh).not.toHaveProperty("admitFresh");
expect(admitExplicit).toHaveBeenLastCalledWith(wake, false, expect.any(Function));
});
it("exposes normal fresh admission after the final continuation check", async () => {
const admitExplicit = vi.fn().mockResolvedValue(null);
const admitFresh = vi.fn();
const prepare = createWakeRecoveryAdmission({
readBlocker: async () => ({ blocker: null, verifiedReset: false }),
admitExplicit,
}, vi.fn(), admitFresh);
const busy = await (await prepare(wake)).checkBusyAdmission();
expect(admitExplicit).not.toHaveBeenCalled();
if (busy.kind !== "allowed") throw new Error("Expected busy admission");
const fresh = await busy.checkFreshAdmission();
expect(fresh).toMatchObject({ kind: "allowed", admitFresh, explicitContinuation: null });
expect(admitExplicit).toHaveBeenCalledExactlyOnceWith(wake, false, expect.any(Function));
});
});
@@ -0,0 +1,60 @@
import type { ExecutionBlocker } from "@paperclipai/shared";
type ContinuationWait = { reason: string; message: string };
type AdmissionBlocker = Pick<ExecutionBlocker, "recoveryActionId" | "cause" | "nextAction" | "runId">;
type WakeRecoveryReadInput = {
commentId: string | null;
requestedByActorType?: string | null;
requestedByActorId?: string | null;
};
type ExplicitContinuationIntent = { commentId: string | null; queuedCommentInterruptId?: string };
export type WakeRecoveryAdmissionStore<ExplicitInput, ExplicitResult> = {
readBlocker(input: WakeRecoveryReadInput): Promise<{
blocker: AdmissionBlocker | null;
verifiedReset: boolean;
}>;
admitExplicit(input: ExplicitInput, dryRun: boolean, onBlocked: (reason: string, message: string) => void):
Promise<ExplicitResult | null>;
};
export function createWakeRecoveryAdmission<ExplicitInput extends ExplicitContinuationIntent, ExplicitResult, Busy, Fresh>(
store: WakeRecoveryAdmissionStore<ExplicitInput, ExplicitResult>,
admitBusy: Busy,
admitFresh: Fresh,
) {
return async function prepareRecovery(
input: ExplicitInput & WakeRecoveryReadInput & {
reconciliation?: { sourceRunId: string; actionId: string };
},
) {
const { blocker, verifiedReset } = await store.readBlocker(input);
const ownsReconstruction = Boolean(input.reconciliation && blocker &&
blocker.runId === input.reconciliation.sourceRunId && blocker.recoveryActionId === input.reconciliation.actionId);
return {
blocker,
ownsReconstruction,
async checkBusyAdmission() {
let wait: ContinuationWait | undefined;
const onBlocked = (reason: string, message: string) => { wait = { reason, message }; };
const blocked = () => Boolean(blocker || (wait && !verifiedReset));
if ((blocker || input.queuedCommentInterruptId) && !ownsReconstruction &&
!(await store.admitExplicit(input, true, onBlocked)) && blocked()) {
return { kind: "blocked" as const, wait };
}
return {
kind: "allowed" as const,
admitBusy,
wait,
async checkFreshAdmission() {
const explicitContinuation = await store.admitExplicit(input, false, onBlocked);
if (!explicitContinuation && blocked() && !ownsReconstruction) return { kind: "blocked" as const, wait };
return { kind: "allowed" as const, explicitContinuation, admitFresh, wait };
},
};
},
};
};
}
+10 -7
View File
@@ -7,11 +7,14 @@ import {
type IssueAdmissionDbTransaction,
type IssueAdmissionRow,
type IssueAdmissionTarget,
type IssueWakeContinuationInput,
type IssueWakeContinuationResult,
} from "./adapters/postgres.js";
import { createQueuedCommentIssueLockWriter } from "./adapters/queued-comment-postgres.js";
import { createQueuedCommentSteeringAdapter, lockBoardQueuedComments, readBoardQueuedComments } from "./adapters/queued-comment-board-postgres.js";
import type { QueuedCommentQueuePostgresAdapterDeps } from "./adapters/queued-comment-postgres.js";
import { createFreshIssueRunAdmission } from "./application/fresh-issue-run.js";
import { createWakeRecoveryAdmission } from "./application/recovery-admission.js";
import { admitIssueWake, createIssueWakePreflight, type IssueWakeAdmissionEffects, type IssueWakeAdmissionOutcome } from "./application/issue-wake-admission.js";
import { createAdmitWakeBehindIssueExecution, createReleaseIssueExecution } from "./application/use-cases.js";
import {
@@ -88,8 +91,9 @@ type HeartbeatRun = typeof heartbeatRuns.$inferSelect;
type LockedIssueAdmission = {
lockedIssue: IssueAdmissionRow | null;
lockIssue: () => Promise<IssueAdmissionRow | null>;
admitWakeBehindIssueExecution: ReturnType<typeof createAdmitWakeBehindIssueExecution>;
admitFreshIssueRun: ReturnType<typeof createFreshIssueRunAdmission<HeartbeatRun>>;
prepareRecovery: ReturnType<typeof createWakeRecoveryAdmission<
IssueWakeContinuationInput, IssueWakeContinuationResult, ReturnType<typeof createAdmitWakeBehindIssueExecution>, ReturnType<typeof createFreshIssueRunAdmission<HeartbeatRun>>
>>;
};
/** Composes the issue admission and release workflows with their Postgres locks. */
@@ -117,24 +121,23 @@ export function createWakeQueue(db: Db, deps: WakeQueueDeps) {
evaluateHeartbeatPolicy: (tx: IssueAdmissionDbTransaction, admission: LockedIssueAdmission) => Promise<IssueWakeAdmissionOutcome<HeartbeatRun>>,
effects: IssueWakeAdmissionEffects<HeartbeatRun>,
) {
return admitIssueWake(() => withIssueAdmission({ companyId, issueId, durableChatRequest }, (tx, { lockedIssue, lockIssue, reader, writer, fresh }) => {
return admitIssueWake(() => withIssueAdmission({ companyId, issueId, durableChatRequest }, (tx, { lockedIssue, lockIssue, recovery, reader, writer, fresh }) => {
const admitBusy = createAdmitWakeBehindIssueExecution({ reader, writer, helpers: deps.wakeAdmissionHelpers });
const admitFresh = createFreshIssueRunAdmission(fresh);
return evaluateHeartbeatPolicy(tx, {
lockedIssue,
lockIssue,
async admitWakeBehindIssueExecution(input) {
prepareRecovery: createWakeRecoveryAdmission(recovery, async (input: Parameters<typeof admitBusy>[0]) => {
if (input.companyId !== companyId || input.issueId !== issueId || input.contextSnapshot.issueId !== issueId) {
throw new Error("wake-queue: admission does not match the locked company and issue");
}
return admitBusy(input);
},
async admitFreshIssueRun(input) {
}, async (input: Parameters<typeof admitFresh>[0]) => {
if (input.issue.id !== issueId || input.contextSnapshot.issueId !== issueId) {
throw new Error("wake-queue: admission does not match the locked issue");
}
return admitFresh(input);
},
}),
});
}), effects);
},
+34 -7
View File
@@ -284,10 +284,30 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
deps.wakeQueue.admitIssueWake.mockImplementation((target, evaluate, effects) => admit(target,
(tx, admission) => evaluate(tx, otherIssue ? {
...admission,
admitFreshIssueRun: (input) => admission.admitFreshIssueRun({ ...input,
issue: { ...input.issue, id: otherIssue.id }, contextSnapshot: { ...input.contextSnapshot, issueId: otherIssue.id } }),
admitWakeBehindIssueExecution: (input) => admission.admitWakeBehindIssueExecution({ ...input,
issueId: otherIssue.id, contextSnapshot: { ...input.contextSnapshot, issueId: otherIssue.id } }),
async prepareRecovery(input) {
const recovery = await admission.prepareRecovery(input);
return {
...recovery,
async checkBusyAdmission() {
const busy = await recovery.checkBusyAdmission();
if (busy.kind !== "allowed") return busy;
return {
...busy,
admitBusy: (input) => busy.admitBusy({ ...input, issueId: otherIssue.id,
contextSnapshot: { ...input.contextSnapshot, issueId: otherIssue.id } }),
async checkFreshAdmission() {
const fresh = await busy.checkFreshAdmission();
if (fresh.kind !== "allowed") return fresh;
return {
...fresh,
admitFresh: (input) => fresh.admitFresh({ ...input,
issue: { ...input.issue, id: otherIssue.id }, contextSnapshot: { ...input.contextSnapshot, issueId: otherIssue.id } }),
};
},
};
},
};
},
} : admission), { ...effects, publishIssueActivity }));
const wake = queue.enqueueWakeup(agent.id, {
@@ -1013,9 +1033,16 @@ describe.skipIf(!support.supported)("heartbeat queue database wiring", () => {
});
const busyAdmission = vi.fn();
deps.wakeQueue.admitIssueWake.mockImplementation((target, evaluate, effects) => admission.admitIssueWake(target,
(tx, stores) => evaluate(tx, { ...stores, admitWakeBehindIssueExecution(input) {
busyAdmission(input.activeExecutionRun.id, input.issueExecutionAgentNameKey);
return stores.admitWakeBehindIssueExecution(input);
(tx, stores) => evaluate(tx, { ...stores, async prepareRecovery(input) {
const recovery = await stores.prepareRecovery(input);
return { ...recovery, async checkBusyAdmission() {
const busy = await recovery.checkBusyAdmission();
if (busy.kind !== "allowed") return busy;
return { ...busy, admitBusy(input) {
busyAdmission(input.activeExecutionRun.id, input.issueExecutionAgentNameKey);
return busy.admitBusy(input);
} };
} };
} }), effects));
const queue = createHeartbeatQueue(db, deps);
if (scenario === "running") {
+27 -60
View File
@@ -56,14 +56,12 @@ import { hasWorkspaceRestoreFailure } from "@paperclipai/shared";
import { isVerifiedDependencyInteractionResponse, readQueuedInteractionResponse } from "../queued-interaction-response.js";
import {
isConversation,
isConversationReset,
isWaitingConversation,
} from "../agent-conversations.js";
import { onboardingGreetingNeeded } from "../onboarding-chat.js";
import { ONBOARDING_CHAT_GREETING_REASON } from "../onboarding-chat-prompt.js";
import { legacyControllerClaim } from "../legacy-controller-lease.js";
import {
admitExplicitNativeContinuation,
undeliveredLegacyUserCommentIds,
} from "../explicit-native-continuation.js";
import { canRetryStoppedRun } from "../cancelled-native-startup.js";
@@ -80,7 +78,6 @@ import {
import { claimQueuedNativeReviewRun } from "../native-runtime/native-review-dispatch.js";
import { nativeCleanupHoldsAdmission } from "../native-runtime/native-cleanup-admission-hold.js";
import { adapterExecutionControls } from "../adapter-execution-control.js";
import { preserveTaskRemoteWorkspaceWriter } from "../native-runtime/native-failed-workspace-recovery.js";
import { executionFailureRetryCount } from "../execution-recovery-attempt.js";
import {
slackMentionAllowsRunStart,
@@ -3520,7 +3517,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
return wakeQueue.admitIssueWake(
{ companyId: agent.companyId, issueId, durableChatRequest: durableRequest },
async (tx, { lockedIssue, lockIssue, admitWakeBehindIssueExecution, admitFreshIssueRun }) => {
async (tx, { lockedIssue, lockIssue, prepareRecovery }) => {
if (executionWaitRequestId) {
const [pending] = await tx.select().from(agentWakeupRequests).where(and(
eq(agentWakeupRequests.id, executionWaitRequestId), eq(agentWakeupRequests.companyId, agent.companyId),
@@ -3870,6 +3867,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
}
let reconciledSourceRunId: string | null = null;
let reconciledRecoveryActionId: string | null = null;
let reconciledRestoreRetryCount: number | null = null;
if (executionReconciliationWake) {
const actionId = readNonEmptyString(
@@ -4012,10 +4010,10 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
reconciledRestoreRetryCount = executionFailureRetryCount(reconciledRun!);
}
reconciledSourceRunId = sourceRunId;
if (enrichedContextSnapshot.recoveryActionId === action.id) reconciledRecoveryActionId = action.id;
}
let continuationWait = { reason: "execution_recovery", message: "Waiting for execution recovery. Your message is saved." };
let continuationBlocked = false;
const deferBlockedExecution = async (
executionBlocker: Pick<NonNullable<Awaited<ReturnType<typeof getExecutionBlocker>>>, "recoveryActionId" | "cause" | "nextAction">,
) => {
@@ -4066,32 +4064,21 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
return { kind: "deferred" as const, admitted: true };
};
const explicitContinuationRunId = randomUUID();
const [resetComment] = isConversation(issue) && opts.requestedByActorType === "user" && opts.requestedByActorId && wakeCommentId
? await tx.select({ body: issueComments.body }).from(issueComments).where(and(
eq(issueComments.companyId, issue.companyId), eq(issueComments.issueId, issue.id),
eq(issueComments.id, wakeCommentId), eq(issueComments.authorUserId, opts.requestedByActorId),
eq(issueComments.authorType, "user"), isNull(issueComments.deletedAt), isNull(issueComments.createdByRunId),
)).limit(1)
: [];
const verifiedConversationReset = Boolean(resetComment && isConversationReset(resetComment.body));
const preserveWorkspace = wakeCommentId && opts.requestedByActorType === "user"
&& issue.executionWorkspaceId !== null;
if (preserveWorkspace)
await preserveTaskRemoteWorkspaceWriter(tx as unknown as Db, issue.companyId, issue.id);
const executionBlocker = await getExecutionBlocker(
tx as unknown as Db, issue.companyId, issue.id,
{
conversationResetCommentId: verifiedConversationReset ? wakeCommentId : null,
lockedIssue: preserveWorkspace ? undefined : issue,
},
);
// Prove eligibility without retiring the hold. Later gates can still
// decline this wake; hold retirement and successor creation stay atomic.
// A bound chat request has already rechecked its current principal
// above. Treat its new user message like a board comment, but keep
// failed-run retry actions on their separate exact-request path.
const ownsReconstructionContinuation = reconciledSourceRunId && executionBlocker?.runId === reconciledSourceRunId
&& executionBlocker.recoveryActionId === enrichedContextSnapshot.recoveryActionId;
const recovery = await prepareRecovery({
agentId, actorType: taskRetryActorId ? "user" : opts.requestedByActorType, actorId: taskRetryActorId ?? opts.requestedByActorId,
requestedByActorType: opts.requestedByActorType, requestedByActorId: opts.requestedByActorId,
reason: taskRetryActorId ? "task_retry" : durableRequest && !failedChatRetry ? "issue_commented" : reason,
commentId: wakeCommentId ?? null, failedRunId: opts.failedRunId, successorRunId: explicitContinuationRunId,
queuedCommentInterruptId: opts.queuedCommentInterruptId,
queuedCommentRequestId: opts.queuedCommentRequestId,
taskRetryRequestId: opts.taskRetryRequestId,
...(durableRequest?.resumeDeferred && !failedChatRetry ? { durableChatRequest: durableRequest } : {}),
interactionPayload: payload,
reconciliation: reconciledSourceRunId && reconciledRecoveryActionId ? {
sourceRunId: reconciledSourceRunId, actionId: reconciledRecoveryActionId,
} : undefined,
});
const { blocker: executionBlocker, ownsReconstruction: ownsReconstructionContinuation } = recovery;
const continuationHold = () => executionBlocker ?? {
recoveryActionId: null, cause: "native_continuation_requires_reconciliation" as const, nextAction: continuationWait.message,
};
@@ -4118,19 +4105,9 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
? await undeliveredContinuation(tx as unknown as Db, { companyId: issue.companyId, issueId: issue.id }) : null;
if (continuationToDeliver)
Object.assign(enrichedContextSnapshot, undeliveredContinuationContext(continuationToDeliver, enrichedContextSnapshot));
if ((executionBlocker || opts.queuedCommentInterruptId) && !ownsReconstructionContinuation && !(await admitExplicitNativeContinuation({
db: tx as unknown as Db, companyId: issue.companyId, issueId: issue.id,
agentId, actorType: taskRetryActorId ? "user" : opts.requestedByActorType, actorId: taskRetryActorId ?? opts.requestedByActorId,
reason: taskRetryActorId ? "task_retry" : durableRequest && !failedChatRetry ? "issue_commented" : reason,
commentId: wakeCommentId ?? null, failedRunId: opts.failedRunId, successorRunId: explicitContinuationRunId,
queuedCommentInterruptId: opts.queuedCommentInterruptId,
queuedCommentRequestId: opts.queuedCommentRequestId,
taskRetryRequestId: opts.taskRetryRequestId,
...(durableRequest?.resumeDeferred && !failedChatRetry ? { durableChatRequest: durableRequest } : {}),
interactionPayload: payload,
dryRun: true,
onBlocked: (reason, message) => { continuationBlocked = true; continuationWait = { reason, message }; },
})) && (executionBlocker || (continuationBlocked && !verifiedConversationReset))) {
const busyRecovery = await recovery.checkBusyAdmission();
if (busyRecovery.wait) continuationWait = busyRecovery.wait;
if (busyRecovery.kind === "blocked") {
// A hold left by an unfinished run never refuses an ordinary
// continuation: unless its cause is the person's, the fallback
// settles it in this transaction and this wake becomes the fresh
@@ -4764,7 +4741,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
return { kind: "coalesced" as const, run: activeExecutionRun };
}
const admission = await admitWakeBehindIssueExecution({
const admission = await busyRecovery.admitBusy({
companyId: agent.companyId,
issueId: issue.id,
agentId,
@@ -4999,20 +4976,10 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
return { kind: "skipped" as const };
}
const explicitContinuation = await admitExplicitNativeContinuation({
db: tx as unknown as Db, companyId: issue.companyId, issueId: issue.id,
agentId, actorType: taskRetryActorId ? "user" : opts.requestedByActorType, actorId: taskRetryActorId ?? opts.requestedByActorId,
reason: taskRetryActorId ? "task_retry" : durableRequest && !failedChatRetry ? "issue_commented" : reason,
commentId: wakeCommentId ?? null, failedRunId: opts.failedRunId, successorRunId: explicitContinuationRunId,
queuedCommentInterruptId: opts.queuedCommentInterruptId,
queuedCommentRequestId: opts.queuedCommentRequestId,
taskRetryRequestId: opts.taskRetryRequestId,
...(durableRequest?.resumeDeferred && !failedChatRetry ? { durableChatRequest: durableRequest } : {}),
interactionPayload: payload,
onBlocked: (reason, message) => { continuationBlocked = true; continuationWait = { reason, message }; },
});
if (!explicitContinuation && (executionBlocker || (continuationBlocked && !verifiedConversationReset)) && !ownsReconstructionContinuation)
return deferBlockedExecution(continuationHold());
const freshRecovery = await busyRecovery.checkFreshAdmission();
if (freshRecovery.wait) continuationWait = freshRecovery.wait;
if (freshRecovery.kind === "blocked") return deferBlockedExecution(continuationHold());
const { explicitContinuation } = freshRecovery;
if (explicitContinuation) {
enrichedContextSnapshot.forceFreshSession = true;
enrichedContextSnapshot.previousRunId = explicitContinuation.previousRunId;
@@ -5030,7 +4997,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
}, issueActivityPublications);
}
const newRun = await admitFreshIssueRun({
const newRun = await freshRecovery.admitFresh({
receipt: {
agentId,
source,
@@ -1457,9 +1457,16 @@ describe("automatic retained workspace recovery", () => {
queueDeps.wakeQueue.admitIssueWake.mockImplementation((target, evaluate, effects) =>
admitIssueWake(target, (tx, admission) => evaluate(tx, {
...admission,
admitWakeBehindIssueExecution(input) {
expect(input.durableReceipt).toBeUndefined();
return admission.admitWakeBehindIssueExecution(input);
async prepareRecovery(input) {
const recovery = await admission.prepareRecovery(input);
return { ...recovery, async checkBusyAdmission() {
const busy = await recovery.checkBusyAdmission();
if (busy.kind !== "allowed") return busy;
return { ...busy, admitBusy(input) {
expect(input.durableReceipt).toBeUndefined();
return busy.admitBusy(input);
} };
} };
},
}), effects));
queueDeps.liveRunExecutions.has.mockImplementation(id => t.deps.activeRunExecutions.has(id));