fix(heartbeat): retry operator Stop after a transaction deadlock (#17895)

<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/69/changes/paperclip/69
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Operator Stop must revoke task authority before it stops provider
execution.
> - Its transaction locks the task and then the run. A concurrent writer
can hold the run while it waits for the task.
> - PostgreSQL can abort the Stop transaction with a confirmed deadlock.
The cancel API then returns 500.
> - This pull request retries only that aborted database transaction, at
most three times.
> - Provider Stop and activity publication stay after commit, so the
retry cannot repeat those effects.

## Linked Issues or Issue Description

Refs #17639. Its agent-chat project browser test completed the product
assertions, then failed during cleanup when the cancel API returned 500.
A real PostgreSQL lock-cycle test reproduces the same `40P01` error in
the Stop transaction.

Related: #13911 addresses audit foreign-key lock contention. This change
handles a different concurrent run-writer path without changing the
schema or task/run lock order.

Stack base: #17639. This is a separate correctness repair for the sleep
migration stack, not another polling migration.

## What Changed

- Control the runner timeout regression fixture's clock so it reaches
the pending append before expiry; retain production cancellation and
late-append assertions.

- Retry the Stop authority transaction only after PostgreSQL reports
`40P01`. Make at most three attempts, with short bounded backoff.
- Discard rolled-back local run state, claim flags, and queued activity
publications before each retry. Re-read current authority under fresh
locks.
- Keep provider cancellation and activity publication outside the
retried transaction. Preserve other errors and the existing native
lock-conflict response.
- Add a real two-transaction deadlock test. Add server-side failure
tests for retry exhaustion and errors that must not retry.
- Let run-control tests use a validated disposable PostgreSQL database
URL.

## Verification

Published head: `60ba892ae9fc5faaf8cd0833694b4a66c13323c3`, targeting
master after #17639 merged at
`1ed0af41867ba40fd1d23292d395588d18c2fcbe`. The normal ancestry merge
preserves the complete five-path tested cancellation patch
byte-for-byte, including the review repair, and all four unrelated
current master paths. The production retry logic remains unchanged from
this slice's reviewed implementation; the follow-up adds coverage and
its audit.

- **228 distinct complete cases pass:** all 48 run-control cases and all
180 native-session runtime cases. The two new targeted cases are
included in 48 and are not counted again. Server typechecking, build and
module-boundary checks also pass.
- The new real PostgreSQL cases reach a failed native run's correlated
Stop claim, cancellation fence, board handoff retirement and activity
write before raising `40P01`. One case preserves retry authority and
commits exactly one cancellation activity/live notice. The other changes
coordinator authority between attempts, returns 409, preserves the
failed run and handoff, and performs no provider/workspace cleanup.
- A negative control removes only the publication-list reset. It fails
with two live notices for one durable activity. Production was restored
byte-for-byte before final validation. The existing actual
two-connection deadlock, bounded attempts, non-deadlock rejection and
post-commit provider behavior remain covered.
- Preserve the initial new-assertion typecheck failure (optional
live-event payload) and the corrected final pass. Service-local test
files are typechecked; `src/__tests__` is excluded. The negative-control
harness initially expected an untruncated Vitest assertion line; the
actual named 1-versus-2 failure and restoration were inspected before
correcting its result matcher.
- [Current-parent acceptance and failure
ledger](https://github.com/paperclipai/paperclip/blob/60ba892ae9fc5faaf8cd0833694b4a66c13323c3/doc/plans/2026-10-28-cancellation-deadlock-acceptance.md)
records current scope, commands, isolated fixtures, mutation evidence
and limits. Earlier combined-stack counts retain their original revision
scope. The existing controlled runner timeout/late-append fixture is
retained and passes within the complete 180-case runtime suite.
- This addresses the prior Ponytail request to exercise native claimed
state before deadlock rollback and re-evaluate authority afterward. The
advisory loop-invariant snapshot/named-attempt-bound note remains a
maintainability suggestion, not a runtime defect. No production loop
refactor or new worker duty is introduced.
- **All 58 current checks are clean; Greptile and Ponytail both rate
this exact head 5/5.** Ponytail explicitly confirms that the
native-claim coverage request is resolved and says Ship. The formerly
queued dependency/quality review workflow has completed successfully.
Old `fffc`/parent checks and reviews are not current passes; the former
pending Snyk status is not called successful. Actual failing Ponytail
findings must be fixed. Missing/stopped/paused turns without a verdict
are authorized nonblockers. Active unfinished reviews retain their
original per-head timer, with a 25-minute monitor cutoff respecting the
30-minute maximum.
- No full local repository matrix, live provider execution, external
alarm, physical host/database suspension or savings claim is made. No
schema, dependency, workflow, shared scheduler contract or shared OS
resource changes are introduced.

## Risks

- Persistent contention still returns an error after the third attempt.
This change does not promise to eliminate every possible lock cycle.
- Retrying a write after an ambiguous connection failure would be
unsafe. The retry accepts only PostgreSQL's confirmed deadlock abort
code; it does not accept transport errors.
- No schema or public API contract changes. Current task/run authority
and cancellation eligibility are checked again on each attempt.

## Model Used

OpenAI GPT-6 via Codex, with shell, GitHub CLI, TypeScript, PostgreSQL,
and Playwright tools. The exact serving model ID and context window are
not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-28 11:46:58 -05:00
1 parent 4f8104dec6
commit 4808a3a2b1
5 files changed
+271 -34

No files matched your search

@@ -0,0 +1,77 @@
# Cancellation deadlock retry acceptance
This existing correctness slice (#17895) retries only the operator Stop database
transaction after PostgreSQL reports `40P01`, at most three times. Provider Stop,
workspace settlement, and activity publication remain after commit. This is not
a new polling or scheduling duty.
## Current parent and outstanding review
The previous published head is
`fffc8d0c46fbbeb3da4a23a36e638fefd3b8fe3d`, originally based on native progress
`08d07ead105dc89c2af9b62f4d90bff8cc93540c`. The slice is prepared by a normal
merge of verified native progress `6f365285f5b233a89acc714eadfead5ab145100e`.
All four original changed-file hunks retain their exact content and context.
Only the run-control blob index hashes differ because the parent also changed
that file. The automatic composition is clean and preserves those parent edits.
The prior Ponytail review said Ship with an actionable coverage request: existing
deadlock tests did not reach a correlated native failed run's claimed state before
rollback. The follow-up adds two real PostgreSQL cases in the existing run-control
suite, with no new production behavior:
1. Keep retry authority: claim the correlated Stop, fence the failed native run,
cancel its real board terminal handoff, and persist cancellation activity inside
the first transaction. Raise PostgreSQL `40P01` before commit. Retry succeeds;
exactly one cancellation activity is durable and exactly one live notice is
published. The correlation belongs to the committed attempt.
2. Revoke retry authority: after that aborted transaction rolls back, an independent
coordinator update retires retry eligibility. The second attempt returns 409,
preserves the failed run and uncancelled handoff, publishes no cancellation
activity, and performs no provider or workspace cleanup.
The existing actual two-connection lock-cycle regression and the bounded retry/
non-deadlock cases remain in the complete suite. The new tests exercise claims
and activity produced before a real PostgreSQL transaction abort; they do not
infer correctness from a mocked retry callback alone.
## Evidence and failure accounting
- Both new cases pass against the retained production reset logic.
- Removing only `publications.length = 0` makes the preserved-authority case fail:
two live cancellation notices appear for one committed activity. Production was
restored byte-for-byte to its integrated index before further validation.
`/tmp/sleep-aux-native-retry-mutation-proof.json` records that comparison.
- The negative-control harness initially expected an untruncated assertion line;
Vitest abbreviates that line. Its actual named failure and 1-versus-2 result were
inspected, and the matcher was corrected without rerunning or altering production.
- Typechecking caught an optional live-event payload in the new assertion. The
assertion now uses optional access. Preserve the original failed typecheck at
`/tmp/sleep-aux-current-parent-types.log`; the final typecheck passes at
`/tmp/sleep-aux-current-parent-types-final.log`.
- **228 distinct complete cases pass:** all 48 run-control cases, including the two
additions, and all 180 native-session runtime cases, including the existing
controlled timeout/late-append fixture. The two targeted probes are included in
48 and are not counted again.
- Server typechecking, build, and module-boundary checks pass. `server/tsconfig.json`
excludes `src/__tests__`, but includes service-local test files such as
`run-control.test.ts`; typechecking does not replace test execution. Earlier
statements about tests omitted by production typechecking must be read with
that path boundary.
The three PostgreSQL probes use separate new disposable databases on the owned
local cluster. Full-run results and exit receipts are in
`/tmp/sleep-aux-current-parent-{types-final,build,boundaries,control,runner}.log`.
The targeted positive and mutation logs are
`/tmp/sleep-aux-native-retry-regression.log` and
`/tmp/sleep-aux-native-retry-mutant-red.log`.
## Remaining gates and limits
This is local preparation. The dependency must merge, the normal merge onto its
actual landed master must preserve this tested patch and all master changes, and
fresh exact-head CI plus Greptile 5/5 must pass before protected merge. Prior
review scores and checks on older heads are not current passes. The previous
Snyk status was pending and is not called successful here. No shared OS resource
cleanup, schema change, whole-repository local matrix, live provider execution,
external alarm, physical suspension, or savings qualification is claimed.
@@ -3499,10 +3499,6 @@ describe("executeNativeSession recovery", () => {
});
it("stops and closes a timed-out consumer even when the caller requested a warm session", async () => {
let markAppendStarted = () => {};
const appendStarted = new Promise<void>((resolve) => {
markAppendStarted = resolve;
});
let releaseAppend = () => {};
const appendReleased = new Promise<void>((resolve) => {
releaseAppend = resolve;
@@ -3515,7 +3511,6 @@ describe("executeNativeSession recovery", () => {
let appendCommitted = false;
const appendEvent = vi.fn(
async (_event: PrpEvent, options?: { signal: AbortSignal }) => {
markAppendStarted();
await Promise.race([
appendReleased,
new Promise<never>((_resolve, reject) => {
@@ -3612,28 +3607,39 @@ describe("executeNativeSession recovery", () => {
async completeRun() {},
};
const execution = executeNativeSession({
input,
backend,
controlPlane: port,
runnerInstanceId: "runner-recovery",
controlPlaneInstanceId: "control-recovery",
timeoutMs: 1,
keepSessionOpen: true,
});
const rejection = expect(execution).rejects.toThrow(
"native session timed out",
);
await appendStarted;
await vi.waitFor(() => expect(iteratorTeardown).toHaveBeenCalledOnce());
await rejection;
expect(cancel).toHaveBeenCalledOnce();
expect(close).toHaveBeenCalled();
expect(appendEvent).toHaveBeenCalledOnce();
expect(appendCommitted).toBe(false);
releaseAppend();
await new Promise<void>((resolve) => setImmediate(resolve));
expect(appendCommitted).toBe(false);
// Reach the pending append before expiring the consumer. A real 1 ms
// deadline can expire during setup before the consumer reaches this case.
vi.useFakeTimers();
try {
const execution = executeNativeSession({
input,
backend,
controlPlane: port,
runnerInstanceId: "runner-recovery",
controlPlaneInstanceId: "control-recovery",
timeoutMs: 1,
keepSessionOpen: true,
});
const rejection = expect(execution).rejects.toThrow(
"native session timed out",
);
await vi.advanceTimersByTimeAsync(0);
expect(appendEvent).toHaveBeenCalledOnce();
await vi.advanceTimersByTimeAsync(1);
await rejection;
expect(iteratorTeardown).toHaveBeenCalledOnce();
expect(cancel).toHaveBeenCalledOnce();
expect(close).toHaveBeenCalled();
expect(appendEvent).toHaveBeenCalledOnce();
expect(appendCommitted).toBe(false);
releaseAppend();
await vi.advanceTimersByTimeAsync(0);
expect(appendCommitted).toBe(false);
} finally {
releaseAppend();
releaseTeardown();
vi.useRealTimers();
}
});
it("closes a failed session while retaining an uncancellable event read", async () => {
@@ -1,8 +1,9 @@
import type { Db } from "@paperclipai/db";
import { agents, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns } from "@paperclipai/db";
import { agents, companies, createDb, applyPendingMigrations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns } from "@paperclipai/db";
import { sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, vi, type Mock } from "vitest";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../../__tests__/helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "../../../__tests__/helpers/external-test-database.js";
import { runningProcesses } from "../../../adapters/index.js";
import { terminateLocalService } from "../../local-service-supervisor.js";
import { createHeartbeatRunState } from "../run-state.js";
@@ -65,15 +66,17 @@ export function callbacks(db: Db): TestCallbacks {
failedProcessRunCancellations: new Map<string, { settled: Promise<void>; failed: boolean; error?: unknown }>(),
} satisfies HeartbeatRunControlDependencies;
}
export const runControlPostgresSupport = await getEmbeddedPostgresTestSupport();
const externalTestDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_RUN_CONTROL_TEST_DATABASE_URL");
export const runControlPostgresSupport = externalTestDatabaseUrl ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
if (!runControlPostgresSupport.supported) console.warn(`Skipping run-control database tests: ${runControlPostgresSupport.reason}`);
export function setupRunControlDatabase(name: string, ready: (db: Db) => void) {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: Db;
beforeAll(async () => {
database = await startEmbeddedPostgresTestDatabase(name);
db = createDb(database.connectionString);
if (externalTestDatabaseUrl) await applyPendingMigrations(externalTestDatabaseUrl);
else database = await startEmbeddedPostgresTestDatabase(name);
db = createDb(externalTestDatabaseUrl ?? database!.connectionString);
ready(db);
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
afterAll(async () => {
@@ -1,7 +1,7 @@
import { randomUUID } from "node:crypto";
import type { Db } from "@paperclipai/db";
import { activityLog, agents, agentWakeupRequests, caseEvents, caseHandoffs, cases, companies, environmentLeases, heartbeatRuns, issueComments, issues, nativeRunFinalizations } from "@paperclipai/db";
import { eq, inArray } from "drizzle-orm";
import { eq, inArray, sql } from "drizzle-orm";
import { afterEach, describe, expect, it, vi } from "vitest";
import { runningProcesses } from "../../adapters/index.js";
import {
@@ -14,6 +14,7 @@ import { AdapterStopTimeoutError } from "../adapter-stop-timeout.js";
import { HttpError } from "../../errors.js";
import { captureException } from "../../sentry.js";
import { terminateLocalService } from "../local-service-supervisor.js";
import { publishLiveEvent } from "../live-events.js";
import { createHeartbeatRunState } from "./run-state.js";
import { createHeartbeatRunControl } from "./run-control.js";
import { preserveCancelledNativeWorkspaces } from "../native-runtime/native-failed-workspace-recovery.js";
@@ -173,6 +174,79 @@ describe.skipIf(!runControlPostgresSupport.supported)("heartbeat run-control dat
)).toMatchObject({ status: "cancelled" });
});
it.each([["40P01", 3], ["XX000", 1]] as const)("bounds cancellation transaction attempts for PostgreSQL %s", async (code, attempts) => {
const { company, agent } = await fixture();
const run = await liveRun(company.id, agent.id);
const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Cancellation failure", status: "in_progress",
assigneeAgentId: agent.id, executionRunId: run.id }).returning();
await db.update(heartbeatRuns).set({ issueId: issue.id, contextSnapshot: { issueId: issue.id } }).where(eq(heartbeatRuns.id, run.id));
await db.execute(sql`create sequence run_control_cancel_attempts`);
await db.execute(sql.raw(`create function run_control_cancel_fault() returns trigger language plpgsql as $$
begin
if new.result_json->'executionCancellation'->>'state' = 'requested' then
perform nextval('run_control_cancel_attempts');
raise exception using errcode = '${code}', message = 'Synthetic cancellation transaction failure';
end if;
return new;
end $$`));
await db.execute(sql`create trigger run_control_cancel_fault before update on heartbeat_runs
for each row execute function run_control_cancel_fault()`);
const deps = callbacks(db);
try {
await expect(createHeartbeatRunControl(db, deps).cancelRunInternal(run.id, "Operator Stop", {
resultJson: { cancelledByActorType: "user", cancelledByUserId: "operator" },
})).rejects.toMatchObject({ cause: { code } });
const [counter] = await db.execute<{ count: number }>(sql`select last_value::int as count from run_control_cancel_attempts`);
expect(counter.count).toBe(attempts);
expect(await deps.getRun(run.id)).toMatchObject({ status: "running", resultJson: null });
expect(terminateLocalService).not.toHaveBeenCalled();
expect(deps.appendRunEvent).not.toHaveBeenCalled();
} finally {
await db.execute(sql`drop trigger run_control_cancel_fault on heartbeat_runs`);
await db.execute(sql`drop function run_control_cancel_fault()`);
await db.execute(sql`drop sequence run_control_cancel_attempts`);
}
});
it("finishes operator Stop after a concurrent run writer causes a transaction deadlock", async () => {
const { company, agent } = await fixture();
const run = await liveRun(company.id, agent.id);
const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Stop contention", status: "in_progress",
assigneeAgentId: agent.id, executionRunId: run.id, checkoutRunId: run.id }).returning();
await db.update(heartbeatRuns).set({ issueId: issue.id, contextSnapshot: { issueId: issue.id } }).where(eq(heartbeatRuns.id, run.id));
let enter!: () => void, release!: () => void;
const entered = new Promise<void>(resolve => { enter = resolve; });
const released = new Promise<void>(resolve => { release = resolve; });
let blockerPid = 0;
const writer = db.transaction(async tx => {
// Make Stop's default detector see the cycle first, so this reproduces
// the API-side victim observed in CI rather than a random victim.
await tx.execute(sql`set local deadlock_timeout = '10s'`);
const [backend] = await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`);
blockerPid = backend.pid;
await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)).for("update");
enter(); await released;
await tx.update(issues).set({ updatedAt: new Date() }).where(eq(issues.id, issue.id));
});
await entered;
const deps = callbacks(db);
const stopping = createHeartbeatRunControl(db, deps).cancelRunInternal(run.id, "Operator Stop", {
resultJson: { cancelledByActorType: "user", cancelledByUserId: "operator" },
}).then(value => ({ value, error: undefined }), error => ({ value: undefined, error }));
try {
await vi.waitFor(async () => {
const waiting = await db.execute(sql`select pid from pg_stat_activity
where ${blockerPid} = any(pg_blocking_pids(pid)) and wait_event_type = 'Lock'`);
expect(waiting.length).toBeGreaterThan(0);
}, { timeout: 5000 });
} finally { release(); }
await writer;
const result = await stopping;
expect(result.error).toBeUndefined();
expect(result.value).toMatchObject({ status: "cancelled" });
expect(await deps.getRun(run.id)).toMatchObject({ status: "cancelled" });
});
it.each(["queued", "running"] as const)("preserves case intent when a guarded Stop loses to %s promotion", async status => {
const { company, agent } = await fixture();
const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Promoted case work", status: "todo", assigneeAgentId: agent.id }).returning();
@@ -393,6 +467,63 @@ describe.skipIf(!runControlPostgresSupport.supported)("heartbeat run-control dat
.toMatchObject({ status: "in_progress", phase: "verification", completedAt: null });
});
it.each([false, true])("discards rolled-back native Stop claims and activity before retry (authority revoked=%s)", async revokeAuthority => {
const f = await boardTerminalFixture("running", "native");
const requestId = randomUUID(), deps = callbacks(db);
vi.mocked(publishLiveEvent).mockClear();
const transact = db.transaction.bind(db);
let injected = false;
const transaction = vi.spyOn(db, "transaction").mockImplementationOnce(async (operation, config) => {
try {
return await transact(async tx => {
const result = await operation(tx);
const [claimed] = await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id));
expect(claimed.resultJson?.startupCancellation).toMatchObject({ cancellationRequestId: requestId });
expect(claimed.resultJson?.executionCancellation).toMatchObject({ state: "requested" });
expect((await tx.select().from(caseHandoffs).where(eq(caseHandoffs.id, f.receipt.handoffId)))[0].state).toBe("cancelled");
expect((await tx.select().from(activityLog).where(eq(activityLog.companyId, f.company.id)))
.filter(row => row.action === "case.handoff_cancelled")).toHaveLength(1);
expect(deps.getAgent).not.toHaveBeenCalled();
injected = true;
await tx.execute(sql`do $$ begin raise exception using errcode = '40P01', message = 'Deadlock after native Stop claim'; end $$`);
return result;
}, config);
} catch (error) {
if (injected && revokeAuthority) await db.update(nativeRunFinalizations)
.set({ phase: "terminal_failure", failureCode: "retry_authority_revoked", nextAttemptAt: null })
.where(eq(nativeRunFinalizations.runId, f.run.id));
throw error;
}
});
try {
const stopping = createHeartbeatRunControl(db, deps).cancelRunInternal(f.run.id, "Retry native Stop", {
resultJson: { cancelledByActorType: "user", cancelledByUserId: "operator" },
cancellationRequestId: requestId, cancellationRequestedByUserId: "operator",
});
if (revokeAuthority) {
await expect(stopping).rejects.toMatchObject({ status: 409 });
expect(await deps.getRun(f.run.id)).toMatchObject({ status: "failed", resultJson: null });
expect(deps.getAgent).not.toHaveBeenCalled();
expect(deps.options.closeWarmNativeSessionsForRun).not.toHaveBeenCalled();
expect(deps.envOrchestrator.releaseForRun).not.toHaveBeenCalled();
} else {
await expect(stopping).resolves.toMatchObject({ status: "cancelled" });
expect((await deps.getRun(f.run.id))?.resultJson?.startupCancellation).toMatchObject({ cancellationRequestId: requestId });
}
expect(injected).toBe(true);
expect(transaction.mock.calls.length).toBeGreaterThanOrEqual(2);
expect((await db.select().from(caseHandoffs).where(eq(caseHandoffs.id, f.receipt.handoffId)))[0].state)
.toBe(revokeAuthority ? "awaiting_completion" : "cancelled");
const expected = revokeAuthority ? 0 : 1;
expect((await db.select().from(activityLog).where(eq(activityLog.companyId, f.company.id)))
.filter(row => row.action === "case.handoff_cancelled")).toHaveLength(expected);
expect(vi.mocked(publishLiveEvent).mock.calls.map(([event]) => event)
.filter(event => event.companyId === f.company.id && event.type === "activity.logged"
&& event.payload?.action === "case.handoff_cancelled")).toHaveLength(expected);
expect(terminateLocalService).not.toHaveBeenCalled();
} finally { transaction.mockRestore(); }
});
it("stops current preparing execution despite a prior terminal checkout when cancelling board intent", async () => {
// Queue admission binds the new execution before preparation adopts its
// predecessor's checkout. The execution owner is authoritative in this gap.
+22 -2
View File
@@ -1134,7 +1134,7 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont
let caseHandoffRetryClaimed = false;
if (handoffIssueId && supersedesCaseHandoff && options.resultJson?.caseHandoffYieldStop !== true) {
const publications: ActivityPublication[] = [];
const supersessionApplied = await db.transaction(async tx => {
const applySupersession = () => db.transaction(async tx => {
const [issue] = await tx.select({ id: issues.id, executionRunId: issues.executionRunId, checkoutRunId: issues.checkoutRunId,
assigneeAgentId: issues.assigneeAgentId, assigneeUserId: issues.assigneeUserId }).from(issues)
.where(and(eq(issues.companyId, run!.companyId), eq(issues.id, handoffIssueId))).for("update");
@@ -1189,7 +1189,27 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont
await cancelCaseHandoffsForIssue(tx as unknown as Db, { companyId: run!.companyId, issueId: handoffIssueId,
sourceRunId: runId, includeUnboundTerminalIntent: Boolean(fenced && ownsTask), reason, publications });
return true;
}).catch(rethrowNativeCancellationLockConflict);
});
let supersessionApplied = false;
for (let attempt = 0; ; attempt++) {
const previousRun = run;
try {
supersessionApplied = await applySupersession();
break;
} catch (error) {
const value = parseObject(error);
const deadlock = value.code === "40P01" || parseObject(value.cause).code === "40P01";
if (!deadlock || attempt >= 2) rethrowNativeCancellationLockConflict(error);
// PostgreSQL aborted the entire transaction. Discard its local
// receipts before re-reading current authority under fresh locks.
// Provider Stop and publication happen only after this boundary.
run = previousRun;
cancellationClaimed = false;
caseHandoffRetryClaimed = false;
publications.length = 0;
await new Promise(resolve => setTimeout(resolve, 25 * (attempt + 1)));
}
}
publications.forEach(publishActivity);
if (!supersessionApplied) return null;
}