fix(task-retry): retain chat Retry during identity contention (#18896)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task Retry must preserve a person's request and enforce the current
chat authority.
> - Chat retry preparation takes nonblocking source locks before it can
authorize the saved source.
> - Sustained lock contention could discard the Retry request before it
was saved.
> - This pull request saves one unadmitted request in the existing chat
retry queue.
> - The worker checks the source and current access before normal
admission. The task shows a scheduled retry while it waits.

## Linked Issues or Issue Description

**What happened?**

Task Retry returned HTTP 500 when another transaction held the saved
chat principal's identity lock or the endpoint authorization row. The
source preparation error rolled back the canonical Retry intent. The
same gap existed when Retry had to restore a certified input batch that
had never reached a provider. With the bounded synchronous retries in
#18895, sustained contention returns 409 without a saved intent.

**Expected behavior**

HTTP 202 retains one Retry request. Acceptance does not grant source
authority or mean that a provider started. The existing worker prepares
the request if the source locks become available before its saved
deadline. It must reject changed input, revoked access, Stop, or
superseding work before input adoption, publication, or provider work.

**Steps to reproduce**

1. Prepare a failed task with its original admitted GitHub chat source.
2. Hold that principal's `chat-identity` advisory lock or the endpoint
row on a second PostgreSQL connection.
3. Send the normal task Retry HTTP request. Check the response and
persisted intent.
4. Release the lock. Process the existing chat retry queue and check
that one authorized successor is queued.

**Paperclip version or commit**

Reproduced on master `8771606fd87f69ed4057779e3dce3a9bca8088e6`.

**Deployment mode**

Source checkout, real Express Retry route and embedded PostgreSQL.
External chat transport is a test fixture. The live hosted fix has not
been deployed or verified.

Related: #18895 handles brief admission contention with bounded
synchronous attempts and returns 409 if it remains busy. This change
retains a durable request through sustained identity contention. It uses
the existing background retry owner and adds real HTTP, restored-input,
Stop, publication, restart, and task-presentation coverage.

## What Changed

- Preserve a strict pending-source action only when the identity
try-lock reports contention or the endpoint authorization query reports
lock-unavailable through an explicit Retry callback. Bind it to the
canonical task Retry, company, task, agent, endpoint, conversation, and
principal. Other database and authorization errors keep their existing
behavior.
- Revalidate the canonical intent, current board permission, saved
source, and latest execution before the existing worker can prepare that
action. Ordinary authorization rejects an unprepared action.
- Keep certified input restoration inside source preparation. Recheck
the saved deadline before the prepared payload write. Expiry, a nested
savepoint, or outer rollback cannot commit partial restoration.
- Give source authorization contention one saved 60-second deadline and
a separate counter. Keep its waiting phase until current source
authorization succeeds or the action terminates. It does not spend the
eight delivery attempts. Duplicate clicks, worker restart, and
background sweeps do not renew either limit. Reuse the existing work
notifications and outbox deadlines.
- Persist completion of the authorization wait under the current claim
before delivery. A restart revalidates current authority. Generic
delivery failure preserves the stored phase even when a commit
acknowledgment is lost; it does not grant a wake or renew the saved
deadline, counters, or claim.
- Project pending preparation and prepared but unsettled actions as the
existing scheduled-retry state with the actual outbox deadline. Keep the
exact task, intent, agent, and source binding. Active execution keeps
its normal working state. No UI or API enum change is needed.
- Add regressions and document the admission boundary.

## Verification

- RED: the original real HTTP route returned 500 under a held PostgreSQL
identity lock. The restored-input case also returned 500 while its 60
sibling source and budget cases passed. The current endpoint-row case
returned 409 without a saved intent under the bounded synchronous
fallback.
- RED: the worker failed a valid Retry during an actual nine-second
PostgreSQL lock. Removing only the latest-run guard makes the new
queued-run negative fail. Source preparation could also commit restored
input after its deadline, and an already prepared action could process
after a late lock release. The prepared but unsettled task view lost its
scheduled state.
- GREEN: all 91 selected real PostgreSQL integration cases passed.
Coverage includes the 62-case saved-input matrix; coalescing; restart;
one real heartbeat successor; source, actor, Stop, newer-work and
publication refusals; a real nine-second lock; fixed-deadline
exhaustion; the separate eight delivery attempts; certified restoration
rollback on expiry; prepared-action early and late release; a real
conversation-row wait that crosses the deadline during current
authorization; endpoint-row contention; unknown source; an
already-revoked actor; unchanged unrelated ingress errors; durable-work
idle settlement; actual outbox presentation deadlines; and recovery
after authorization commits, with Stop and a lost commit acknowledgment.
- GREEN: all 75 cases in `task-retry.test.ts`,
`task-interruption-presentation.test.ts`,
`chat-failed-run-retry-boundary.test.ts`, and `chat-action-work.test.ts`
passed without a name filter.
- RED → GREEN: stopping after source authorization committed left a
stale waiting flag, so a fresh worker failed a previously authorized
Retry after the deadline. The fix saves that phase in the same
transaction. A real PostgreSQL commit followed by an injected lost
acknowledgment also reproduced generic settlement restoring the stale
flag. The corrected settlement preserves the stored phase. No wake is
sent on the ambiguous response. Recovery queues once after current
checks; Stop still prevents a successor. The deadline, contention count,
original input, failed run, and publications stay unchanged.
- Module boundary checks pass. Full workspace typecheck and the full
build passed at `d8a4a1a739aa6825714b04691d4166a00991808b`. At this
exact head, both required checks (`ci / verify` and `ci / e2e`) pass.
The inventory has 53 successful, three skipped, and one neutral check,
plus the automatic Paperclip Review in progress. The legacy Snyk status
passes. Greptile is 5/5 on this head, with no new comments or unresolved
threads. A current Paperclip assessment has not been published yet.
- The full local `pnpm test:run` did not pass: its second chat shard hit
a 15-second timeout in the HTTP/restart test after 348 and 293 passing
cases. One bounded diagnostic of the same 642-case shard pair passed,
with that case completing in 606 ms. The matching Linux CI pair passed
too. The original timeout remains unexplained and nonreproduced. No
timeout changed. The earlier stopped run was incomplete, and both
conflicting documentation additions are retained.
- CI chat 3/3 exposed a separate fixture assumption: restoration can
commit before a durable wake receipt exists. One known pre-receipt
delivery failure reproduces the same missing-run error. The corrected
fixture drives the existing worker until the exact canonical intent and
wake agree. It proves one real wake, one successor, exact source/actor
binding, and one restoration. The exact failed shard pair now passes
locally: 635 cases. Workspace typecheck also passes at test-only head
`94d691c9df61b05019b96900d5800a8f1c484e9a`. Production source is
unchanged from the previous verified build. That head completed all
registered checks: 53 successful, two neutral, three skipped, plus the
successful legacy status. Greptile scored 5/5. The subsequent Paperclip
4/5 persistence finding is addressed by the source repair above; its new
exact-head review is pending.
- Compatibility: 111 selected cases passed with #18895 head
`9b5d877161ffcbf16c0fb6920e8155a3620c85b0` in a disposable worktree. The
branch is now synchronized with master `4a83ba4050`, which includes that
change.
- Preserve the old CI failure: at head
`37954951fb66f3b50d0f2a3bec78d6993c479e87`, server shard 6/12 initially
failed the empty organization-audience replacement versus member-removal
assertion in `tool-access-service.test.ts`. The exact race family passed
on unchanged base and head; all 501 file tests and two 30-case controls
passed on base. The one Linux replay passed 119 files and 2,833 cases,
with three skipped. No guard or assertion changed. The initial failure
remains unexplained and nonreproduced; this PR does not claim to fix it.
- The prior external Superagent scan could not start because its
Security Graph returned HTTP 503. Its neutral check was not a completed
security assessment.
- These tests exercise the real Express route, PostgreSQL transactions,
and heartbeat queue. External chat transport is a fixture. They do not
run a live model or prove recovery of an original hosted task.

## Risks

- A still-pending source authorization wait fails after its fixed
60-second contention deadline. A completed wait is saved, but new
contention keeps the original deadline. A healthy lock can exceed that
limit. Other delivery errors retain the eight-attempt budget. Current
authority checks remain required before admission. Inspection does not
reset either limit.
- The new pending phase must never be treated as an authorized chat
source. Normal authorization rejects it until the worker prepares and
binds the exact source.
- The task projection reads the existing linked outbox deadline. It adds
a scoped join and does not create a timer or change a business deadline.
- No schema migration, provider identity change, or new delivery queue
is included.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository editing, tool
use, and test execution. The host did not expose an exact deployment
snapshot ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] Required Paperclip CI gates are green (`ci / verify` and `ci /
e2e`; automatic review progress is recorded above)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-11-04 07:18:48 -06:00
1 parent c5251b4c0e
commit b6352578c5
7 files changed
+970 -43

No files matched your search

+30
View File
@@ -944,6 +944,36 @@ same preparation retains its directory and receipt check. A published workspace
on the same allocation retains its recorded directory. Automatic bootstrap
timeout retries still require a fresh allocation under the policy above.
### Saved chat Retry preparation
Task Retry can meet a busy chat identity or endpoint lock before its source is authorized.
The canonical Retry and its linked chat action then retain one unadmitted request.
The existing chat retry worker prepares that request after the lock is available.
It checks the current board actor, source, principal, conversation, task owner,
and canonical intent before normal admission. Stop, changed input, revoked access,
or newer execution can reject it without adopting input or publishing a notice.
Restoration of a certified unstarted input batch stays in the same transaction as
source preparation. Contention cannot commit a partial restoration.
Pending preparation and a prepared but unsettled action show the existing
retry-scheduled state and the chat outbox's actual retry or stale-claim deadline.
They do not imply that a provider started. Source authorization contention has a separate
60-second deadline saved with the pending request, or at the first contention
for an older action. Waiting on these locks does not consume the eight delivery
attempts. Expiry is checked before source preparation commits and before a
prepared action can leave its authorization wait. A lock that becomes available
after expiry cannot complete a still-pending wait. The authorization transaction
saves completion of that wait under the current claim before wake delivery.
A restart rechecks current source and execution gates; elapsed time alone does
not expire a completed wait. New contention still uses the original deadline.
Generic delivery failure cannot put a completed wait back into the pending
state. A lost commit acknowledgment preserves the stored phase and does not
authorize a wake.
Duplicate clicks and background sweeps cannot renew either limit. Preparation
does not create a generic board wake. Only the endpoint query's lock-unavailable
error can defer through its explicit Retry callback. Other database and
authorization errors keep their existing behavior.
### Runner authentication and ACP startup handshake bounds
The native Runner waits up to the ten-second connection setup budget for the
@@ -20,7 +20,7 @@ import { nativeChatProgressWork } from "../services/chat-native-progress-work.js
import { cloudActorHeaderSourceFromHeaders, resolveCloudTenantActor } from "../middleware/auth.js";
import { cloudTenantPrimaryCompanyId } from "../services/cloud-instance.js";
import { accessService } from "../services/access.js";
import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db";
import { databaseWorkPending, reconcileDatabaseWork, signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db";
import { claimTaskRetryInput, releaseUnadmittedTaskRetryInput } from "../services/chat-channels/task-retry-input.js";
import { restoreNativeChatReviewPresentationInTransaction } from "../services/native-runtime/native-chat-review-presentation.js";
import { notifyChatEndpointConfiguration, notifyChatCompanyConfiguration } from "../services/chat-channels/configuration-work-notifications.js";
@@ -40,6 +40,7 @@ import { createChatRuntime } from "../services/chat-runtime.js";
import { createChatSdkRuntime } from "../services/chat-sdk-runtime.js";
import { readFreshWorkspaceReadiness } from "../services/fresh-workspace-continuation.js";
import { requestTaskRetry, reconcileTaskChatRetryReceipts } from "../services/task-retry.js";
import * as taskRetryModule from "../services/task-retry.js";
import { taskInterruptionPresentation } from "../services/task-interruption-presentation.js";
import { withQueuedCommentIdsInWakePayload } from "../services/issue-queued-comment-queue.js";
import { createSystemRecovery } from "../services/system-recovery-state.js";
@@ -64479,6 +64480,580 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
return { ...f, pending, originalDelivery };
}
async function busyChatRetryHttpFixture(withPending = false) {
const f = await currentGitHubTaskRetryFixture(withPending);
const originalPublications = await readChatPublicationsByEndpointId(f.endpoint.id);
const { issueRoutes } = await import("../routes/issues.js");
const companyId = f.fixture.companyId, issueId = f.issue.id;
const app = express();
app.use(express.json());
app.use((req, _res, next) => { req.actor = boardActor(companyId); next(); });
app.use("/api", issueRoutes(db, createStorageService().storage, { chatRunRetries: f.service }));
app.use(errorHandler);
let release = () => {};
const held = new Promise<void>(resolve => { release = resolve; });
let acquired = () => {};
const locked = new Promise<void>(resolve => { acquired = resolve; });
const holding = db.transaction(async tx => {
await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`chat-identity:${companyId}:${f.action.principalId}`}, 0))`);
acquired();
await held;
});
await locked;
f.wakeup.mockClear();
return { ...f, app, companyId, issueId, originalPublications,
releaseIdentity: async () => { release(); await holding; } };
}
it("retains one HTTP task Retry intent while saved chat identity authorization is busy", async () => {
const f = await busyChatRetryHttpFixture();
let restarted: ReturnType<typeof createService> | undefined;
try {
const { companyId, issueId, app } = f;
const original = (await readHeartbeatRunsById(f.runId))[0];
const originalInput = (await readAgentWakeupRequestsById(f.action.id))[0];
const body = { observedRunId: f.runId, idempotencyKey: randomUUID() };
const responses = await Promise.all([body, body, { ...body, idempotencyKey: randomUUID() }]
.map(value => request(app).post(`/api/issues/${issueId}/retry`).send(value)));
for (const response of responses) expect(response.status, JSON.stringify(response.body)).toBe(202);
expect(responses[0].body.presentation).toMatchObject({ state: "retry_scheduled", canRetry: false });
expect(new Set(responses.map(response => response.body.requestId)).size).toBe(1);
expect(new Set(responses.map(response => response.body.chatActionId)).size).toBe(1);
const response = responses[0];
const [intent] = await readAgentWakeupRequestsById(response.body.requestId);
expect(intent).toMatchObject({ reason: "task_retry", status: "deferred_issue_execution", runId: null });
expect((await request(app).post(`/api/issues/${issueId}/retry`).send(body)).body.requestId).toBe(intent.id);
const [action] = await db.select().from(chatActions).where(eq(chatActions.id, response.body.chatActionId));
expect(action.payload).toMatchObject({ sourceAuthorizationPending: true, taskRetryIntentId: intent.id });
expect(action.payload.sourceScopeSha256).toBeUndefined();
await f.service.processFailedChatRunRetry(action.id);
const [waiting] = await db.select().from(chatActions).where(eq(chatActions.id, action.id));
expect(await taskInterruptionPresentation(db, (await readIssuesById(issueId))[0]))
.toMatchObject({ state: "retry_scheduled", nextRetryAt: waiting.result!.retryAt, canRetry: false });
expect((await readAgentWakeupRequestsById(f.action.id))[0]).toEqual(originalInput);
await heartbeatService(db).resumeTaskRetry(companyId, intent.id);
expect(f.wakeup).not.toHaveBeenCalled();
expect(await readChatPublicationsByEndpointId(f.endpoint.id)).toEqual(f.originalPublications);
expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(0);
expect((await db.select().from(chatActions).where(and(eq(chatActions.companyId, companyId),
eq(chatActions.kind, "failed_run_retry"))))).toHaveLength(1);
await f.service.shutdown();
restarted = createService(new FakeChatSdkRuntime(), f.providerFetch, { wakeup: heartbeatService(db).wakeup });
await f.releaseIdentity();
await vi.waitFor(async () => {
await restarted!.service.processFailedChatRunRetries();
await reconcileTaskChatRetryReceipts(db, { companyId, requestId: intent.id });
expect((await readAgentWakeupRequestsById(intent.id))[0]?.runId).toBeTruthy();
}, { timeout: 5_000, interval: 50 });
const [settled] = await readAgentWakeupRequestsById(intent.id);
const [successor] = await readHeartbeatRunsById(settled.runId!);
expect(successor).toMatchObject({ status: "queued", retryOfRunId: f.runId,
contextSnapshot: { source: "chat:github", taskRetryRequestId: intent.id,
wakeCommentIds: [f.action.payload.commentId] } });
expect(restarted.wakeup).toHaveBeenCalledOnce();
expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(1);
expect((await readHeartbeatRunsById(f.runId))[0]).toEqual(original);
const [authorized] = await db.select().from(chatActions).where(eq(chatActions.id, action.id));
expect(authorized.payload.sourceAuthorizationPending).toBeUndefined();
expect(authorized.payload.sourceScopeSha256).toMatch(/^[a-f0-9]{64}$/);
expect((await request(app).post(`/api/issues/${issueId}/retry`).send(body)).body.requestId).toBe(intent.id);
} finally {
await f.releaseIdentity();
await restarted?.service.shutdown();
await f.service.shutdown();
}
});
it.each(["unchanged", "source_edited", "principal_revoked", "unknown_source", "actor_revoked_before"] as const)("retains HTTP Retry under the endpoint authorization row lock: %s", async mode => {
const f = await busyChatRetryHttpFixture(true);
let release = () => {};
let holding: Promise<unknown> | undefined;
try {
await f.releaseIdentity();
const sourceIds = [f.action.id, f.pending!.id];
const originalInputs = await db.select().from(agentWakeupRequests).where(inArray(agentWakeupRequests.id, sourceIds))
.orderBy(asc(agentWakeupRequests.id));
if (mode === "unknown_source") {
const [source] = await readHeartbeatRunsById(f.runId);
await db.update(heartbeatRuns).set({ contextSnapshot: { ...source.contextSnapshot, source: "chat:unknown" } })
.where(eq(heartbeatRuns.id, f.runId));
}
if (mode === "actor_revoked_before") await db.update(companyMemberships).set({ status: "suspended", updatedAt: new Date() })
.where(and(eq(companyMemberships.companyId, f.companyId), eq(companyMemberships.principalId, "owner-user")));
let acquired = () => {};
const locked = new Promise<void>(resolve => { acquired = resolve; });
const held = new Promise<void>(resolve => { release = resolve; });
holding = db.transaction(async tx => {
await tx.select({ id: chatEndpoints.id }).from(chatEndpoints).where(and(eq(chatEndpoints.companyId, f.companyId),
eq(chatEndpoints.id, f.endpoint.id))).for("no key update");
acquired(); await held;
});
await locked;
const response = await request(f.app).post(`/api/issues/${f.issueId}/retry`)
.send({ observedRunId: f.runId, idempotencyKey: randomUUID() });
if (mode === "unknown_source" || mode === "actor_revoked_before") {
expect(response.status, JSON.stringify(response.body)).toBe(mode === "unknown_source" ? 409 : 404);
expect(await db.select().from(chatActions).where(and(eq(chatActions.companyId, f.companyId),
eq(chatActions.kind, "failed_run_retry")))).toEqual([]);
expect(await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, f.companyId),
eq(agentWakeupRequests.reason, "task_retry")))).toEqual([]);
expect(f.wakeup).not.toHaveBeenCalled();
expect(await readChatPublicationsByEndpointId(f.endpoint.id)).toEqual(f.originalPublications);
expect(await db.select().from(agentWakeupRequests).where(inArray(agentWakeupRequests.id, sourceIds))
.orderBy(asc(agentWakeupRequests.id))).toEqual(originalInputs);
expect((await readHeartbeatRunsByCompanyId(f.companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(0);
return;
}
expect(response.status, JSON.stringify(response.body)).toBe(202);
await f.service.processFailedChatRunRetry(response.body.chatActionId);
expect((await db.select().from(chatActions).where(eq(chatActions.id, response.body.chatActionId)))[0])
.toMatchObject({ status: "issued", payload: { sourceAuthorizationPending: true }, result: { attemptCount: 0 } });
expect(f.wakeup).not.toHaveBeenCalled();
expect(await db.select().from(agentWakeupRequests).where(inArray(agentWakeupRequests.id, sourceIds))
.orderBy(asc(agentWakeupRequests.id))).toEqual(originalInputs);
expect(await readChatPublicationsByEndpointId(f.endpoint.id)).toEqual(f.originalPublications);
if (mode === "source_edited") await db.update(issueComments).set({ body: "Changed saved request",
updatedAt: new Date(Date.now() + 1) }).where(eq(issueComments.id, String(f.pending!.payload.commentId)));
if (mode === "principal_revoked") await db.update(companyMemberships).set({ status: "suspended", updatedAt: new Date() })
.where(and(eq(companyMemberships.companyId, f.companyId), eq(companyMemberships.principalId, "owner-user")));
release(); await holding;
await vi.waitFor(async () => {
await f.service.processFailedChatRunRetries();
const [action] = await db.select().from(chatActions).where(eq(chatActions.id, response.body.chatActionId));
expect(action.status).toBe(mode === "unchanged" ? "processed" : "failed");
}, { timeout: 5_000, interval: 50 });
if (mode === "unchanged") {
await reconcileTaskChatRetryReceipts(db, { companyId: f.companyId, requestId: response.body.requestId });
const [intent] = await readAgentWakeupRequestsById(response.body.requestId);
expect((await readHeartbeatRunsById(intent.runId!))[0]).toMatchObject({ status: "queued", retryOfRunId: f.runId,
contextSnapshot: { wakeCommentIds: [f.pending!.payload.commentId] } });
expect(f.wakeup).toHaveBeenCalledOnce();
expect((await readHeartbeatRunsByCompanyId(f.companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(1);
} else {
expect(f.wakeup).not.toHaveBeenCalled();
expect((await readAgentWakeupRequestsById(response.body.requestId))[0]).toMatchObject({ status: "failed", runId: null });
expect(await db.select().from(agentWakeupRequests).where(inArray(agentWakeupRequests.id, sourceIds))
.orderBy(asc(agentWakeupRequests.id))).toEqual(originalInputs);
expect(await readChatPublicationsByEndpointId(f.endpoint.id)).toEqual(f.originalPublications);
expect((await readHeartbeatRunsByCompanyId(f.companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(0);
}
} finally { release(); await holding; await f.releaseIdentity(); await f.service.shutdown(); }
});
it("keeps a saved HTTP Retry alive through a healthy nine-second identity transaction", async () => {
const f = await busyChatRetryHttpFixture();
try {
const original = (await readHeartbeatRunsById(f.runId))[0];
const originalInput = (await readAgentWakeupRequestsById(f.action.id))[0];
const response = await request(f.app).post(`/api/issues/${f.issueId}/retry`)
.send({ observedRunId: f.runId, idempotencyKey: randomUUID() });
expect(response.status, JSON.stringify(response.body)).toBe(202);
const started = performance.now();
await vi.waitFor(async () => {
await f.service.processFailedChatRunRetries();
expect(performance.now() - started).toBeGreaterThanOrEqual(9_000);
}, { timeout: 12_000, interval: 50 });
const [waiting] = await db.select().from(chatActions).where(eq(chatActions.id, response.body.chatActionId));
expect(waiting.status).toBe("issued");
expect(waiting.result?.attemptCount).toBe(0);
expect(waiting.payload.sourceAuthorizationPending).toBe(true);
expect(f.wakeup).not.toHaveBeenCalled();
expect(await readChatPublicationsByEndpointId(f.endpoint.id)).toEqual(f.originalPublications);
expect((await readAgentWakeupRequestsById(f.action.id))[0]).toEqual(originalInput);
expect((await readHeartbeatRunsByCompanyId(f.companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(0);
await f.releaseIdentity();
await vi.waitFor(async () => {
await f.service.processFailedChatRunRetries();
await reconcileTaskChatRetryReceipts(db, { companyId: f.companyId, requestId: response.body.requestId });
expect((await readAgentWakeupRequestsById(response.body.requestId))[0]?.runId).toBeTruthy();
}, { timeout: 5_000, interval: 50 });
expect(f.wakeup).toHaveBeenCalledOnce();
expect((await readHeartbeatRunsByCompanyId(f.companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(1);
expect((await readHeartbeatRunsById(f.runId))[0]).toEqual(original);
} finally { await f.releaseIdentity(); await f.service.shutdown(); }
}, 30_000);
it("projects prepared but unsettled task Retry using its actual stale-claim deadline", async () => {
const f = await busyChatRetryHttpFixture();
let releaseWake = () => {};
const heldWake = new Promise<void>(resolve => { releaseWake = resolve; });
let processing: ReturnType<typeof f.service.processFailedChatRunRetry> | undefined;
try {
const response = await request(f.app).post(`/api/issues/${f.issueId}/retry`)
.send({ observedRunId: f.runId, idempotencyKey: randomUUID() });
expect(response.status, JSON.stringify(response.body)).toBe(202);
await f.service.processFailedChatRunRetry(response.body.chatActionId);
const [waiting] = await db.select().from(chatActions).where(eq(chatActions.id, response.body.chatActionId));
await f.releaseIdentity();
const wake = heartbeatService(db).wakeup;
f.wakeup.mockImplementation(async (...args) => { await heldWake; return wake(...args); });
await new Promise(resolve => setTimeout(resolve, Math.max(0, Date.parse(String(waiting.result?.retryAt)) - Date.now()) + 1));
processing = f.service.processFailedChatRunRetry(response.body.chatActionId);
await vi.waitFor(() => expect(f.wakeup).toHaveBeenCalledOnce());
const [prepared] = await db.select().from(chatActions).where(eq(chatActions.id, response.body.chatActionId));
expect(prepared.status).toBe("processing");
expect(prepared.payload.sourceAuthorizationPending).toBeUndefined();
expect((await readAgentWakeupRequestsById(response.body.requestId))[0]?.runId).toBeNull();
const deadline = await f.service.nextChatActionAt("failed_run_retry");
expect(deadline).not.toBeNull();
expect(await taskInterruptionPresentation(db, (await readIssuesById(f.issueId))[0]))
.toMatchObject({ state: "retry_scheduled", canRetry: false, nextRetryAt: new Date(deadline!).toISOString() });
releaseWake();
await processing;
await reconcileTaskChatRetryReceipts(db, { companyId: f.companyId, requestId: response.body.requestId });
expect((await readAgentWakeupRequestsById(response.body.requestId))[0]?.runId).toBeTruthy();
expect(await taskInterruptionPresentation(db, (await readIssuesById(f.issueId))[0]))
.toMatchObject({ state: "working", nextRetryAt: null });
} finally { releaseWake(); await processing; await f.releaseIdentity(); await f.service.shutdown(); }
});
it.each(["source_edited", "principal_revoked", "stopped", "mixed_publications", "newer_queued_run"] as const)(
"does not adopt input or deliver a pending identity-contended Retry after %s", async mode => {
const f = await busyChatRetryHttpFixture();
try {
let expectedPublications = f.originalPublications;
const originalInput = (await readAgentWakeupRequestsById(f.action.id))[0];
const response = await request(f.app).post(`/api/issues/${f.issueId}/retry`)
.send({ observedRunId: f.runId, idempotencyKey: randomUUID() });
expect(response.status, JSON.stringify(response.body)).toBe(202);
await f.service.processFailedChatRunRetry(response.body.chatActionId);
if (mode === "source_edited") await db.update(issueComments).set({ body: "Changed saved request",
updatedAt: new Date(Date.now() + 1) }).where(eq(issueComments.id, String(f.action.payload.commentId)));
if (mode === "principal_revoked") await db.update(companyMemberships).set({ status: "suspended", updatedAt: new Date() })
.where(and(eq(companyMemberships.companyId, f.companyId), eq(companyMemberships.principalId, "owner-user")));
if (mode === "stopped") await heartbeatService(db).cancelInvocationsForAgents([f.fixture.assignedAgentId], "operator_stop");
if (mode === "newer_queued_run") {
const [source] = await readHeartbeatRunsById(f.runId);
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.fixture.assignedAgentId,
issueId: f.issueId, nativeIssueId: f.issueId, runtimeMode: "native", status: "queued",
createdAt: new Date(Math.max(Date.now(), source.createdAt.getTime() + 1)),
contextSnapshot: { issueId: f.issueId } });
}
if (mode === "mixed_publications") {
const [comment] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId,
authorAgentId: f.fixture.assignedAgentId, createdByRunId: f.runId, body: "Saved failed result" }).returning();
await db.insert(chatPublications).values((["failed", "delivery_unknown"] as const).map(state => ({
companyId: f.companyId, endpointId: f.endpoint.id, conversationId: f.conversation.id,
issueId: f.issueId, commentId: comment.id, idempotencyKey: `run:${f.runId}:${state}`,
payload: { text: comment.body }, state,
})));
expectedPublications = await readChatPublicationsByEndpointId(f.endpoint.id);
}
const expectedRunIds = (await readHeartbeatRunsByCompanyId(f.companyId)).map(run => run.id).sort();
await f.releaseIdentity();
await vi.waitFor(async () => {
await f.service.processFailedChatRunRetries();
const [action] = await db.select().from(chatActions).where(eq(chatActions.id, response.body.chatActionId));
expect(action.status).toBe("failed");
}, { timeout: 5_000, interval: 50 });
expect(f.wakeup).not.toHaveBeenCalled();
expect(await readChatPublicationsByEndpointId(f.endpoint.id)).toEqual(expectedPublications);
expect(await taskInterruptionPresentation(db, (await readIssuesById(f.issueId))[0]))
.toMatchObject({ state: mode === "newer_queued_run" ? "working" : "failed", nextRetryAt: null });
expect((await readAgentWakeupRequestsById(f.action.id))[0]).toEqual(originalInput);
expect((await readAgentWakeupRequestsById(response.body.requestId))[0]?.runId).toBeNull();
if (mode === "stopped") expect((await readAgentWakeupRequestsById(response.body.requestId))[0]?.status).toBe("cancelled");
expect((await readHeartbeatRunsByCompanyId(f.companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(0);
expect((await readHeartbeatRunsByCompanyId(f.companyId)).map(run => run.id).sort()).toEqual(expectedRunIds);
} finally { await f.releaseIdentity(); await f.service.shutdown(); }
});
it.each([false, true])("expires the saved identity wait without renewal when the lock is released before expiry processing: %s", async released => {
const f = await busyChatRetryHttpFixture();
let restarted: ReturnType<typeof createService> | undefined;
try {
const originalInput = (await readAgentWakeupRequestsById(f.action.id))[0];
const body = { observedRunId: f.runId, idempotencyKey: randomUUID() };
const response = await request(f.app).post(`/api/issues/${f.issueId}/retry`).send(body);
expect(response.status, JSON.stringify(response.body)).toBe(202);
const [saved] = await db.select().from(chatActions).where(eq(chatActions.id, response.body.chatActionId));
const deadline = String(saved.result?.identityContentionDeadlineAt);
expect(Date.parse(deadline)).toBe(saved.createdAt.getTime() + 60_000);
await f.service.processFailedChatRunRetry(saved.id);
const [waiting] = await db.select().from(chatActions).where(eq(chatActions.id, saved.id));
expect(waiting.result).toMatchObject({ attemptCount: 0, identityContentionCount: 1,
identityContentionDeadlineAt: deadline });
expect(Date.parse(String(waiting.result?.retryAt))).toBeLessThanOrEqual(Date.parse(deadline));
for (const value of [body, { ...body, idempotencyKey: randomUUID() }]) {
const duplicate = await request(f.app).post(`/api/issues/${f.issueId}/retry`).send(value);
expect(duplicate.status, JSON.stringify(duplicate.body)).toBe(202);
expect(duplicate.body.requestId).toBe(response.body.requestId);
}
expect((await db.select().from(chatActions).where(eq(chatActions.id, saved.id)))[0].result)
.toEqual(waiting.result);
await f.service.shutdown();
restarted = createService(new FakeChatSdkRuntime(), f.providerFetch, { wakeup: heartbeatService(db).wakeup });
const expiredDeadline = new Date(Date.now() - 1).toISOString();
await db.update(chatActions).set({ result: { ...waiting.result,
identityContentionDeadlineAt: expiredDeadline, retryAt: expiredDeadline } }).where(eq(chatActions.id, saved.id));
if (released) await f.releaseIdentity();
await restarted.service.processFailedChatRunRetries();
const [failed] = await db.select().from(chatActions).where(eq(chatActions.id, saved.id));
expect(failed).toMatchObject({ status: "failed", payload: { sourceAuthorizationPending: true },
result: { code: "failed_chat_retry_contention_expired", attemptCount: 0,
identityContentionDeadlineAt: expiredDeadline } });
expect(failed.result?.retryAt).toBeUndefined();
await f.releaseIdentity();
await restarted.service.processFailedChatRunRetries();
await restarted.service.processFailedChatRunRetry(saved.id);
await heartbeatService(db).resumeTaskRetry(f.companyId, response.body.requestId);
expect((await db.select().from(chatActions).where(eq(chatActions.id, saved.id)))[0]).toEqual(failed);
expect((await readAgentWakeupRequestsById(response.body.requestId))[0]).toMatchObject({ status: "failed", runId: null });
expect((await readAgentWakeupRequestsById(f.action.id))[0]).toEqual(originalInput);
expect(restarted.wakeup).not.toHaveBeenCalled();
expect(f.wakeup).not.toHaveBeenCalled();
expect(await readChatPublicationsByEndpointId(f.endpoint.id)).toEqual(f.originalPublications);
expect((await readHeartbeatRunsByCompanyId(f.companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(0);
} finally { await f.releaseIdentity(); await restarted?.service.shutdown(); await f.service.shutdown(); }
});
it("persists the first identity wait deadline for an older pending action", async () => {
const f = await busyChatRetryHttpFixture();
let restarted: ReturnType<typeof createService> | undefined;
try {
const response = await request(f.app).post(`/api/issues/${f.issueId}/retry`)
.send({ observedRunId: f.runId, idempotencyKey: randomUUID() });
expect(response.status, JSON.stringify(response.body)).toBe(202);
await f.service.processFailedChatRunRetry(response.body.chatActionId);
await db.update(chatActions).set({ result: { code: "failed_chat_retry_contended", attemptCount: 0 } })
.where(eq(chatActions.id, response.body.chatActionId));
const startedAt = Date.now();
await f.service.processFailedChatRunRetry(response.body.chatActionId);
const [waiting] = await db.select().from(chatActions).where(eq(chatActions.id, response.body.chatActionId));
const deadline = Date.parse(String(waiting.result?.identityContentionDeadlineAt));
expect(deadline).toBeGreaterThanOrEqual(startedAt + 60_000);
expect(deadline).toBeLessThanOrEqual(Date.now() + 60_000);
await f.service.shutdown();
restarted = createService(new FakeChatSdkRuntime(), f.providerFetch, { wakeup: heartbeatService(db).wakeup });
await new Promise(resolve => setTimeout(resolve, Math.max(0, Date.parse(String(waiting.result?.retryAt)) - Date.now()) + 1));
await restarted.service.processFailedChatRunRetries();
const [again] = await db.select().from(chatActions).where(eq(chatActions.id, response.body.chatActionId));
expect(again.result).toMatchObject({ attemptCount: 0, identityContentionCount: 2,
identityContentionDeadlineAt: waiting.result?.identityContentionDeadlineAt });
expect(restarted.wakeup).not.toHaveBeenCalled();
} finally { await f.releaseIdentity(); await restarted?.service.shutdown(); await f.service.shutdown(); }
});
it.each(["early_release", "expired_release", "authorization_expired"] as const)("honors a prepared Retry identity wait after %s", async mode => {
const f = await busyChatRetryHttpFixture();
let release = () => {};
let holding: Promise<unknown> | undefined;
let releaseAuthorization = () => {};
let authorizationHolding: Promise<unknown> | undefined;
let processing: Promise<void> | undefined;
try {
const originalInput = (await readAgentWakeupRequestsById(f.action.id))[0];
const response = await request(f.app).post(`/api/issues/${f.issueId}/retry`)
.send({ observedRunId: f.runId, idempotencyKey: randomUUID() });
expect(response.status, JSON.stringify(response.body)).toBe(202);
await f.service.processFailedChatRunRetry(response.body.chatActionId);
const processDue = async () => {
const [current] = await db.select().from(chatActions).where(eq(chatActions.id, response.body.chatActionId));
await new Promise(resolve => setTimeout(resolve, Math.max(0, Date.parse(String(current.result?.retryAt)) - Date.now()) + 1));
await f.service.processFailedChatRunRetry(current.id);
};
await f.releaseIdentity();
f.wakeup.mockRejectedValueOnce(new Error("temporary_wakeup_unavailable"));
await processDue();
const [prepared] = await db.select().from(chatActions).where(eq(chatActions.id, response.body.chatActionId));
expect(prepared.status).toBe("issued");
expect(prepared.payload.sourceAuthorizationPending).toBeUndefined();
expect(prepared.result?.identityContentionPending).toBeUndefined();
let acquired = () => {};
const locked = new Promise<void>(resolve => { acquired = resolve; });
const held = new Promise<void>(resolve => { release = resolve; });
holding = db.transaction(async tx => {
await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`chat-identity:${f.companyId}:${f.action.principalId}`}, 0))`);
acquired(); await held;
});
await locked;
await processDue();
const [waiting] = await db.select().from(chatActions).where(eq(chatActions.id, response.body.chatActionId));
expect(waiting).toMatchObject({ status: "issued", result: { attemptCount: 1, identityContentionCount: 2,
identityContentionDeadlineAt: prepared.result?.identityContentionDeadlineAt } });
expect(f.wakeup).toHaveBeenCalledOnce();
if (mode === "expired_release") {
const expired = new Date(Date.now() - 1).toISOString();
await db.update(chatActions).set({ result: { ...waiting.result,
identityContentionDeadlineAt: expired, retryAt: expired } }).where(eq(chatActions.id, waiting.id));
}
let authorizationDeadline = 0, authorizationPid = 0;
if (mode === "authorization_expired") {
authorizationDeadline = Date.now() + 5_000;
await db.update(chatActions).set({ result: { ...waiting.result,
identityContentionDeadlineAt: new Date(authorizationDeadline).toISOString(), retryAt: new Date(0).toISOString() } })
.where(eq(chatActions.id, waiting.id));
let acquiredAuthorization = () => {};
const authorizationLocked = new Promise<void>(resolve => { acquiredAuthorization = resolve; });
const heldAuthorization = new Promise<void>(resolve => { releaseAuthorization = resolve; });
authorizationHolding = db.transaction(async tx => {
await tx.select({ id: chatConversations.id }).from(chatConversations).where(and(
eq(chatConversations.companyId, f.companyId), eq(chatConversations.id, f.conversation.id))).for("update");
const [{ pid }] = await tx.execute(sql`select pg_backend_pid() as pid`);
authorizationPid = Number(pid);
acquiredAuthorization(); await heldAuthorization;
});
await authorizationLocked;
}
release(); await holding;
if (mode === "authorization_expired") {
processing = processDue();
await vi.waitFor(async () => {
const [{ blocked }] = await db.execute(sql`select exists(select 1 from pg_stat_activity
where ${authorizationPid} = any(pg_blocking_pids(pid))) as blocked`);
expect(blocked).toBe(true);
}, { timeout: 5_000, interval: 25 });
await new Promise(resolve => setTimeout(resolve, Math.max(0, authorizationDeadline - Date.now()) + 1));
releaseAuthorization(); await authorizationHolding;
await processing;
} else await processDue();
const [settled] = await db.select().from(chatActions).where(eq(chatActions.id, waiting.id));
expect(settled.status).toBe(mode === "early_release" ? "processed" : "failed");
expect(settled.result?.identityContentionPending).toBeUndefined();
if (mode !== "early_release") {
expect(settled.result).toMatchObject({ code: "failed_chat_retry_contention_expired", attemptCount: 1 });
expect(f.wakeup).toHaveBeenCalledOnce();
expect((await readAgentWakeupRequestsById(response.body.requestId))[0]).toMatchObject({ status: "failed", runId: null });
expect((await readAgentWakeupRequestsById(f.action.id))[0]).toEqual(originalInput);
expect(await readChatPublicationsByEndpointId(f.endpoint.id)).toEqual(f.originalPublications);
expect((await readHeartbeatRunsByCompanyId(f.companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(0);
} else {
await reconcileTaskChatRetryReceipts(db, { companyId: f.companyId, requestId: response.body.requestId });
expect((await readAgentWakeupRequestsById(response.body.requestId))[0]?.runId).toBeTruthy();
expect(f.wakeup).toHaveBeenCalledTimes(2);
expect((await readHeartbeatRunsByCompanyId(f.companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(1);
}
} finally {
releaseAuthorization(); release(); await authorizationHolding; await processing;
await holding; await f.releaseIdentity(); await f.service.shutdown();
}
});
it.each(["restart", "stopped", "ack_lost"] as const)("recovers the committed authorization phase after %s without renewing its deadline", async mode => {
const f = await busyChatRetryHttpFixture();
let restarted: ReturnType<typeof createService> | undefined;
let releaseWake = () => {};
const heldWake = new Promise<void>(resolve => { releaseWake = resolve; });
let oldProcessing: ReturnType<typeof f.service.processFailedChatRunRetry> | undefined;
let oldShutdown: Promise<void> | undefined;
let restoreBegin = () => {};
try {
const original = (await readHeartbeatRunsById(f.runId))[0];
const originalInput = (await readAgentWakeupRequestsById(f.action.id))[0];
const response = await request(f.app).post(`/api/issues/${f.issueId}/retry`)
.send({ observedRunId: f.runId, idempotencyKey: randomUUID() });
expect(response.status, JSON.stringify(response.body)).toBe(202);
await f.service.processFailedChatRunRetry(response.body.chatActionId);
const [pending] = await db.select().from(chatActions).where(eq(chatActions.id, response.body.chatActionId));
const deadline = new Date(Date.now() + 5_000).toISOString();
await db.update(chatActions).set({ result: { ...pending.result,
identityContentionDeadlineAt: deadline, retryAt: new Date(0).toISOString() } }).where(eq(chatActions.id, pending.id));
let lostAck = false;
if (mode === "ack_lost") {
const begin = db.$client.begin.bind(db.$client);
const fault = vi.spyOn(db.$client, "begin").mockImplementation((async (...args: unknown[]) => {
const value = await Reflect.apply(begin, db.$client, args);
if (!lostAck && value?.intent?.id === pending.id) {
const [committed] = await db.select().from(chatActions).where(eq(chatActions.id, pending.id));
expect(committed.result?.identityContentionPending).toBeUndefined();
const expectedResult = { ...value.intent.result };
delete expectedResult.identityContentionPending;
expect(committed.result).toEqual(expectedResult);
expect(committed.updatedAt).toEqual(value.intent.updatedAt);
lostAck = true;
throw new Error("authorization_commit_ack_lost");
}
return value;
}) as typeof db.$client.begin);
restoreBegin = () => fault.mockRestore();
} else f.wakeup.mockImplementation(async () => { await heldWake; throw new Error("owner_lost_before_wakeup"); });
await f.releaseIdentity();
oldProcessing = f.service.processFailedChatRunRetry(pending.id);
if (mode === "ack_lost") {
await oldProcessing;
restoreBegin();
expect(lostAck).toBe(true);
expect(f.wakeup).not.toHaveBeenCalled();
await reconcileDatabaseWork(db, CHAT_ACTION_QUEUES.failed_run_retry);
} else await vi.waitFor(() => expect(f.wakeup).toHaveBeenCalledOnce());
const [authorized] = await db.select().from(chatActions).where(eq(chatActions.id, pending.id));
if (mode !== "ack_lost") expect(authorized.status).toBe("processing");
expect(authorized.payload.sourceAuthorizationPending).toBeUndefined();
expect((await readAgentWakeupRequestsById(pending.id))[0]).toBeUndefined();
oldShutdown = f.service.shutdown();
await new Promise(resolve => setTimeout(resolve, Math.max(0, Date.parse(deadline) - Date.now()) + 1));
await db.update(chatActions).set({ updatedAt: new Date(Date.now() - 60_001) }).where(eq(chatActions.id, pending.id));
if (mode === "stopped") await heartbeatService(db).cancelInvocationsForAgents([f.fixture.assignedAgentId], "operator_stop");
restarted = createService(new FakeChatSdkRuntime(), f.providerFetch, { wakeup: heartbeatService(db).wakeup });
await restarted.service.processFailedChatRunRetry(pending.id);
await reconcileTaskChatRetryReceipts(db, { companyId: f.companyId, requestId: response.body.requestId });
const [settled] = await db.select().from(chatActions).where(eq(chatActions.id, pending.id));
const [intent] = await readAgentWakeupRequestsById(response.body.requestId);
if (mode !== "stopped") {
expect(settled.status).toBe("processed");
expect(intent.runId).toBeTruthy();
expect(restarted.wakeup).toHaveBeenCalledOnce();
expect((await readHeartbeatRunsByCompanyId(f.companyId)).filter(run => run.retryOfRunId === f.runId)
.map(run => run.id)).toEqual([intent.runId]);
} else {
expect(["issued", "failed"]).toContain(settled.status);
expect(intent).toMatchObject({ status: "cancelled", runId: null });
expect(restarted.wakeup).toHaveBeenCalledOnce();
await expect(restarted.wakeup.mock.results[0].value).resolves.toBeNull();
expect((await readAgentWakeupRequestsById(pending.id))[0]).toBeUndefined();
expect((await readHeartbeatRunsByCompanyId(f.companyId)).filter(run => run.retryOfRunId === f.runId)).toHaveLength(0);
}
expect(authorized.result?.identityContentionPending).toBeUndefined();
expect(settled.result?.identityContentionDeadlineAt).toBe(deadline);
expect(settled.result?.identityContentionCount).toBe(pending.result?.identityContentionCount);
if (mode === "ack_lost") expect(settled.result?.attemptCount).toBe(2);
expect(await readChatPublicationsByEndpointId(f.endpoint.id)).toEqual(f.originalPublications);
expect((await readHeartbeatRunsById(f.runId))[0]).toEqual(original);
expect((await readAgentWakeupRequestsById(f.action.id))[0]).toEqual(originalInput);
releaseWake(); await oldProcessing; await oldShutdown;
expect((await db.select().from(chatActions).where(eq(chatActions.id, pending.id)))[0]).toEqual(settled);
} finally {
restoreBegin();
releaseWake(); await oldProcessing; await oldShutdown;
await f.releaseIdentity(); await restarted?.service.shutdown(); await f.service.shutdown();
}
});
it("exhausts the existing delivery Retry budget after identity authorization without background renewal", async () => {
const f = await busyChatRetryHttpFixture();
try {
const originalInput = (await readAgentWakeupRequestsById(f.action.id))[0];
const response = await request(f.app).post(`/api/issues/${f.issueId}/retry`)
.send({ observedRunId: f.runId, idempotencyKey: randomUUID() });
expect(response.status, JSON.stringify(response.body)).toBe(202);
await f.service.processFailedChatRunRetry(response.body.chatActionId);
const [contended] = await db.select().from(chatActions).where(eq(chatActions.id, response.body.chatActionId));
await f.releaseIdentity();
f.wakeup.mockRejectedValue(new Error("temporary_wakeup_unavailable"));
for (let attempt = 0; attempt < 8; attempt++) {
const [current] = await db.select().from(chatActions).where(eq(chatActions.id, response.body.chatActionId));
if (current.status === "failed") break;
const retryAt = Date.parse(String(current.result?.retryAt));
expect(Number.isFinite(retryAt)).toBe(true);
await new Promise(resolve => setTimeout(resolve, Math.max(0, retryAt - Date.now()) + 1));
await f.service.processFailedChatRunRetries();
}
const [failed] = await db.select().from(chatActions).where(eq(chatActions.id, response.body.chatActionId));
expect(failed).toMatchObject({ status: "failed", result: { attemptCount: 8, identityContentionCount: 1,
identityContentionDeadlineAt: contended.result?.identityContentionDeadlineAt } });
expect(failed.payload.sourceAuthorizationPending).toBeUndefined();
expect(f.wakeup).toHaveBeenCalledTimes(8);
await f.service.processFailedChatRunRetries();
await f.service.processFailedChatRunRetry(failed.id);
await heartbeatService(db).resumeTaskRetry(f.companyId, response.body.requestId);
expect((await db.select().from(chatActions).where(eq(chatActions.id, failed.id)))[0]).toEqual(failed);
expect((await readAgentWakeupRequestsById(response.body.requestId))[0]).toMatchObject({ status: "failed", runId: null });
expect((await readAgentWakeupRequestsById(f.action.id))[0]).toEqual(originalInput);
expect(f.wakeup).toHaveBeenCalledTimes(8);
expect(await readChatPublicationsByEndpointId(f.endpoint.id)).toEqual(f.originalPublications);
} finally { await f.releaseIdentity(); await f.service.shutdown(); }
}, 20_000);
async function successfulGitHubQueuedInputFixture() {
const f = await currentGitHubTaskRetryFixture(true);
const companyId = f.fixture.companyId, issueId = f.issue.id;
@@ -65578,7 +66153,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
});
it.each([
...(["real_budget_ingress", "legacy_budget_ingress", "legacy_prior_response", "legacy_execution", "legacy_execution_log", "legacy_execution_lease", "legacy_execution_profile", "legacy_execution_event", "legacy_execution_controller", "legacy_execution_intent", "legacy_no_prior", "legacy_prior_late", "legacy_prior_unprocessed", "legacy_prior_delivery_unprocessed", "legacy_prior_numeric_app", "legacy_response_conflict", "legacy_prior_wrong_app", "legacy_prior_wrong_agent", "legacy_prior_unlinked", "legacy_prior_retargeted", "legacy_explicit_null", "legacy_explicit_empty", "legacy_explicit_malformed", "legacy_explicit_mismatch", "legacy_current_app_changed", "source_app_changed", "source_destination_changed", "accepted", "multiple_saved", "real_queue", "committed_succeeded", "committed_consumed", "committed_failure", "committed_failed_publication", "committed_unknown_publication", "committed_pending_publication", "committed_mixed_publication", "committed_consumed_publication", "committed_succeeded_publication", "committed_delayed_cleanup", "committed_board", "committed_board_existing", "committed_board_existing_coalesced", "committed_board_existing_source_edited", "committed_board_existing_board_edited", "committed_board_existing_controller", "committed_board_existing_allocation", "committed_board_existing_receipt", "committed_board_existing_delivered", "committed_board_existing_source_owner", "committed_board_existing_source_delivered", "committed_unknown_cleanup", "committed_live_controller", "unreconciled", "source_edited", "source_claimed", "malformed_previous", "endpoint_paused", "principal_revoked", "still_budget_blocked"] as const)
...(["real_budget_ingress", "legacy_budget_ingress", "legacy_prior_response", "legacy_execution", "legacy_execution_log", "legacy_execution_lease", "legacy_execution_profile", "legacy_execution_event", "legacy_execution_controller", "legacy_execution_intent", "legacy_no_prior", "legacy_prior_late", "legacy_prior_unprocessed", "legacy_prior_delivery_unprocessed", "legacy_prior_numeric_app", "legacy_response_conflict", "legacy_prior_wrong_app", "legacy_prior_wrong_agent", "legacy_prior_unlinked", "legacy_prior_retargeted", "legacy_explicit_null", "legacy_explicit_empty", "legacy_explicit_malformed", "legacy_explicit_mismatch", "legacy_current_app_changed", "source_app_changed", "source_destination_changed", "accepted", "multiple_saved", "real_queue", "committed_succeeded", "committed_consumed", "committed_failure", "committed_failed_publication", "committed_unknown_publication", "committed_pending_publication", "committed_mixed_publication", "committed_consumed_publication", "committed_succeeded_publication", "committed_delayed_cleanup", "committed_board", "committed_board_existing", "committed_board_existing_coalesced", "committed_board_existing_identity_contended", "committed_board_existing_identity_expired", "committed_board_existing_source_edited", "committed_board_existing_board_edited", "committed_board_existing_controller", "committed_board_existing_allocation", "committed_board_existing_receipt", "committed_board_existing_delivered", "committed_board_existing_source_owner", "committed_board_existing_source_delivered", "committed_unknown_cleanup", "committed_live_controller", "unreconciled", "source_edited", "source_claimed", "malformed_previous", "endpoint_paused", "principal_revoked", "still_budget_blocked"] as const)
.map(mode => ({ provider: "github" as const, mode })),
])("retries the newer saved $provider request instead of the failed request after a budget hold: $mode", async ({ provider, mode }) => {
const queuedBoard = mode === "committed_board" || mode.startsWith("committed_board_existing");
@@ -66095,7 +66670,38 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
});
const inputIds = [boardWake.id, ...savedInputs.map(source => source.id)];
const before = await db.select().from(agentWakeupRequests).where(inArray(agentWakeupRequests.id, inputIds)).orderBy(asc(agentWakeupRequests.id));
if (mode !== "committed_board_existing" && mode !== "committed_board_existing_coalesced") {
if (mode === "committed_board_existing_identity_contended") {
const events: string[] = [];
const topics = [DELIVERY_QUEUES.deferredExecutionReceipts, CHAT_ACTION_QUEUES.failed_run_retry];
const unsubscribes = topics.map(topic => subscribeDatabaseWork(db, topic, event => events.push(event)));
const rollback = new Error("rollback restored Retry preparation");
let rolledBackActionId: string | undefined;
try {
await expect(db.transaction(async tx => {
events.length = 0;
const staged = await requestTaskRetry({ ...nextInput, db: tx as unknown as Db });
rolledBackActionId = staged.chatActionId ?? undefined;
expect(rolledBackActionId).toBeTruthy();
expect(events).toContain("intent");
expect(events).not.toContain("settled");
expect(await db.select().from(agentWakeupRequests).where(inArray(agentWakeupRequests.id, inputIds))
.orderBy(asc(agentWakeupRequests.id))).toEqual(before);
expect(await tx.select().from(activityLog).where(and(eq(activityLog.companyId, companyId),
eq(activityLog.runId, failed.id), eq(activityLog.action, "issue.retry_inputs_restored")))).toHaveLength(1);
throw rollback;
})).rejects.toBe(rollback);
for (const topic of topics) {
await reconcileDatabaseWork(db, topic);
expect(databaseWorkPending(db, topic)).toBe(false);
}
expect(await db.select().from(chatActions).where(eq(chatActions.id, rolledBackActionId!))).toEqual([]);
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, companyId),
eq(activityLog.runId, failed.id), eq(activityLog.action, "issue.retry_inputs_restored")))).toEqual([]);
expect(await db.select().from(agentWakeupRequests).where(inArray(agentWakeupRequests.id, inputIds))
.orderBy(asc(agentWakeupRequests.id))).toEqual(before);
} finally { unsubscribes.forEach(unsubscribe => unsubscribe()); }
}
if (mode !== "committed_board_existing" && mode !== "committed_board_existing_coalesced" && mode !== "committed_board_existing_identity_contended" && mode !== "committed_board_existing_identity_expired") {
await expect(requestTaskRetry(nextInput)).rejects.toMatchObject({
details: { code: "chat_failed_run_retry_not_authorized" },
});
@@ -66112,14 +66718,98 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
const retryHttp = () => httpRequest(app).post(`/api/issues/${context.issue.id}/retry`).send({
idempotencyKey: nextInput.idempotencyKey, observedRunId: nextInput.observedRunId,
}).expect(202);
const next = (await retryHttp()).body;
const restoredRetryWake = vi.fn(heartbeat.wakeup);
const retryWithIdentityContention = async () => {
if (mode !== "committed_board_existing_identity_contended" && mode !== "committed_board_existing_identity_expired") return retryHttp();
let release = () => {};
const held = new Promise<void>(resolve => { release = resolve; });
let acquired = () => {};
const locked = new Promise<void>(resolve => { acquired = resolve; });
const holding = db.transaction(async tx => {
await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`chat-identity:${companyId}:${saved.principalId}`}, 0))`);
acquired(); await held;
});
try {
await locked;
const response = await retryHttp();
if (mode === "committed_board_existing_identity_expired" || mode === "committed_board_existing_identity_contended")
await context.service.processFailedChatRunRetry(response.body.chatActionId);
if (mode === "committed_board_existing_identity_contended") context.wakeup
.mockImplementation(restoredRetryWake).mockRejectedValueOnce(new Error("retry_admission_unavailable_before_receipt"));
expect(await db.select().from(agentWakeupRequests).where(inArray(agentWakeupRequests.id, inputIds))
.orderBy(asc(agentWakeupRequests.id))).toEqual(before);
return response;
} finally { release(); await holding; }
};
const next = (await retryWithIdentityContention()).body;
expect(next).toMatchObject({ accepted: true });
if (mode === "committed_board_existing_identity_expired") {
const [pending] = await db.select().from(chatActions).where(eq(chatActions.id, next.chatActionId!));
expect(pending.payload).toMatchObject({ sourceAuthorizationPending: true, restoreUnstartedInput: true });
const deadline = Date.now() + 5_000;
await db.update(chatActions).set({ result: { ...pending.result,
identityContentionDeadlineAt: new Date(deadline).toISOString(), retryAt: new Date(0).toISOString() } })
.where(eq(chatActions.id, pending.id));
const realPermission = taskRetryModule.taskRetryMutationAllowed;
const permission = vi.spyOn(taskRetryModule, "taskRetryMutationAllowed").mockImplementation(async (database, input) => {
const allowed = await realPermission(database, input);
if (input.companyId === companyId) {
expect(await database.select().from(activityLog).where(and(eq(activityLog.companyId, companyId),
eq(activityLog.runId, failed.id), eq(activityLog.action, "issue.retry_inputs_restored")))).toHaveLength(1);
await new Promise(resolve => setTimeout(resolve, Math.max(0, deadline - Date.now()) + 1));
}
return allowed;
});
const publications = await readChatPublicationsByEndpointId(context.endpoint.id);
const runIds = (await readHeartbeatRunsByCompanyId(companyId)).map(run => run.id).sort();
context.wakeup.mockClear();
try {
await context.service.processFailedChatRunRetry(pending.id);
expect(permission).toHaveBeenCalled();
expect((await db.select().from(chatActions).where(eq(chatActions.id, pending.id)))[0])
.toMatchObject({ status: "failed", payload: { sourceAuthorizationPending: true },
result: { code: "failed_chat_retry_contention_expired" } });
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, companyId),
eq(activityLog.runId, failed.id), eq(activityLog.action, "issue.retry_inputs_restored")))).toEqual([]);
expect(await db.select().from(agentWakeupRequests).where(inArray(agentWakeupRequests.id, inputIds))
.orderBy(asc(agentWakeupRequests.id))).toEqual(before);
expect(context.wakeup).not.toHaveBeenCalled();
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual(publications);
expect((await readAgentWakeupRequestsById(next.requestId!))[0]).toMatchObject({ status: "failed", runId: null });
expect((await readHeartbeatRunsByCompanyId(companyId)).map(run => run.id).sort()).toEqual(runIds);
} finally { permission.mockRestore(); }
return;
}
expect((await retryHttp()).body.requestId).toBe(next.requestId);
if (mode === "committed_board_existing_identity_contended") await vi.waitFor(async () => {
await context.service.processFailedChatRunRetries();
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, companyId),
eq(activityLog.runId, failed.id), eq(activityLog.action, "issue.retry_inputs_restored")))).toHaveLength(1);
}, { timeout: 5_000, interval: 50 });
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, companyId),
eq(activityLog.runId, failed.id), eq(activityLog.action, "issue.retry_inputs_restored")))).toHaveLength(1);
expect(await context.service.processFailedChatRunRetry(next.chatActionId!)).toMatchObject({ status: "queued" });
[resumed] = await readAgentWakeupRequestsById(next.chatActionId!);
expect(resumed.runId).toBeTruthy();
await vi.waitFor(async () => {
expect(await context.service.processFailedChatRunRetry(next.chatActionId!)).toMatchObject({ status: "queued" });
[resumed] = await readAgentWakeupRequestsById(next.chatActionId!);
expect(resumed).toMatchObject({ companyId, agentId: context.fixture.assignedAgentId,
requestedByActorType: "user", requestedByActorId: "owner-user",
payload: { taskRetryRequestId: next.requestId, retryOfRunId: failed.id } });
expect(resumed.runId).toBeTruthy();
await reconcileTaskChatRetryReceipts(db, { companyId, requestId: next.requestId! });
expect((await readAgentWakeupRequestsById(next.requestId!))[0]?.runId).toBe(resumed.runId);
}, { timeout: 5_000, interval: 50 });
if (mode === "committed_board_existing_identity_contended") {
expect(restoredRetryWake).toHaveBeenCalledOnce();
expect(restoredRetryWake).toHaveBeenCalledWith(context.fixture.assignedAgentId, expect.objectContaining({
taskRetryRequestId: next.requestId, requestedByActorType: "user", requestedByActorId: "owner-user",
contextSnapshot: expect.objectContaining({ taskRetryRequestId: next.requestId,
chatFailedRunRetry: { version: 1, actionId: next.chatActionId, failedRunId: failed.id } }),
}));
expect((await readHeartbeatRunsByCompanyId(companyId)).filter(run =>
run.contextSnapshot?.taskRetryRequestId === next.requestId).map(run => run.id)).toEqual([resumed.runId]);
expect(await db.select().from(activityLog).where(and(eq(activityLog.companyId, companyId),
eq(activityLog.runId, failed.id), eq(activityLog.action, "issue.retry_inputs_restored")))).toHaveLength(1);
}
if (mode === "committed_board_existing_coalesced") {
const restored = await db.select().from(agentWakeupRequests)
.where(inArray(agentWakeupRequests.id, savedInputs.map(source => source.id)));
@@ -13,6 +13,8 @@ import { FAILED_RUN_RETRY_STALE_MS, dueChatAction } from "../chat-action-work.js
import { readChatAuthorizationRun } from "./authorization-run-read.js";
import { selectTaskRetryInput, claimTaskRetryInput, settleTaskRetryInput, releaseUnadmittedTaskRetryInput, taskRetryInputSchema, type TaskRetryInput } from "./task-retry-input.js";
import { queuedCommentIdsFromWakePayload } from "../issue-queued-comment-queue.js";
import { taskRetryMutationAllowed } from "../task-retry.js";
import { z } from "zod";
import type { TaskRetryWakeAuthority } from "./github/task-retry-authorization.js";
import {
@@ -21,6 +23,7 @@ import {
import {
and,
asc,
desc,
eq,
gte,
inArray,
@@ -97,6 +100,7 @@ import { CHAT_CONVERSATION_ID_KEY } from "../../modules/run-dispatch/index.js";
type DbOrTransaction = Db | Parameters<Parameters<Db["transaction"]>[0]>[0];
type DeliveryRow = typeof chatDeliveries.$inferSelect;
type RetryRuntimeFence = { credentialFingerprint: string; generation: number };
const FAILED_RUN_RETRY_IDENTITY_CONTENTION_MS = 60_000;
export type ChatFailedRetrySource = Awaited<ReturnType<ReturnType<typeof createChatFailedRunRetry>["failedChatRetrySource"]>>;
type ChatFailedRunRetryOptions = {
@@ -184,6 +188,14 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
taskRetryInput?: TaskRetryInput;
sourceScopeSha256: string;
};
type PendingRetryBinding = Pick<FailedChatRetrySource, "endpointId" | "conversationId" | "principalId">;
const pendingRetrySourceSchema = z.object({
version: z.literal(1), sourceAuthorizationPending: z.literal(true),
failedRunId: z.string().uuid(), issueId: z.string().uuid(), agentId: z.string().uuid(),
initiatedByUserId: z.string().min(1), taskRetryIntentId: z.string().uuid(),
taskRetryInput: taskRetryInputSchema.optional(),
restoreUnstartedInput: z.literal(true).optional(),
}).strict();
function hashFailedRetrySource(
scope: Omit<FailedChatRetrySource, "sourceScopeSha256">,
): FailedChatRetrySource {
@@ -751,6 +763,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
publication?: boolean;
taskRetryIntentId?: string;
taskRetryInput?: TaskRetryInput;
onIdentityContention?: (binding: PendingRetryBinding) => void;
taskRetryReauthorization?: TaskRetryWakeAuthority;
/** Internal selector for already committed output, never retry admission. */
committedResponse?: {
@@ -1148,11 +1161,17 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
)
)
throw failedChatRetryDenied();
const binding = { endpointId: first.endpointId, conversationId: first.conversationId, principalId: first.principalId };
const deferSourceAuthorization = (): never => {
input.onIdentityContention?.(binding);
throw new NativeChatReviewPresentationContentionError();
};
const endpointContention = input.taskRetryIntentId && !input.publication && !input.committedResponse
? { onEndpointContention: deferSourceAuthorization } : {};
const identity = await tx.execute(
sql`select pg_try_advisory_xact_lock(hashtextextended(${`chat-identity:${input.companyId}:${first.principalId}`}, 0)) as locked`,
);
if (!identity[0]?.locked)
throw new NativeChatReviewPresentationContentionError();
if (!identity[0]?.locked) deferSourceAuthorization();
const actionIds = ordered.map(action => action.id);
const currentActions = await tx.select().from(chatActions).where(and(eq(chatActions.companyId, input.companyId),
inArray(chatActions.id, actionIds))).orderBy(asc(chatActions.id)).for("update", { noWait: true });
@@ -1182,6 +1201,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
const invalidated = new Set(lifecycleRows.map(row => lifecycleKey(row.target, row.context)));
const batch = reauthorization ? await authorizeGitHubTaskRetryBatch(tx, ordered, {
nonblocking: true, terminal: input.publication === true, taskRetry: reauthorization,
...endpointContention,
}).catch(error => { if (isExternalActionAuthorizationChange(error)) throw failedChatRetryDenied(input.taskRetryIntentId ? "source_authorization" : undefined, safeChatWakeAuthorityReason(error)); throw error; }) : null;
const canonicalSources: Array<Record<string, unknown>> = [];
let destination: Awaited<ReturnType<typeof authorizeInboundWakeup>> | null =
@@ -1190,6 +1210,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
destination = batch?.get(action.id) ?? await authorizeInboundWakeup(tx, action, {
nonblocking: true,
terminal: input.publication === true,
...endpointContention,
...(reauthorization ? { taskRetry: reauthorization } : {}),
}).catch((error) => {
if (isExternalActionAuthorizationChange(error))
@@ -1373,6 +1394,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
* The failed run and its canonical Retry remain unchanged history. */
async function restoreUnstartedRetryInput(tx: DbOrTransaction, input: {
companyId: string; issueId: string; agentId: string; failedRunId: string; initiatedByUserId: string;
onIdentityContention?: (binding: PendingRetryBinding) => void;
}) {
const [run] = await tx.select().from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.agentId, input.agentId),
@@ -1455,18 +1477,40 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
retryIntentId?: string;
},
): Promise<{ actionId: string; issueId: string }> {
let taskRetryInput = input.retryIntentId ? await selectTaskRetryInput(tx, input) : undefined;
if (input.retryIntentId && taskRetryInput && !taskRetryInput.sourceActionIds.length)
taskRetryInput = await restoreUnstartedRetryInput(tx, input) ?? taskRetryInput;
const source = await failedChatRetrySource(tx, { ...input, taskRetryIntentId: input.retryIntentId, taskRetryInput });
let taskRetryInput: TaskRetryInput | undefined;
let source: FailedChatRetrySource | null = null;
const pending: { binding: PendingRetryBinding | null; restoreUnstartedInput: boolean } = {
binding: null, restoreUnstartedInput: false,
};
const onIdentityContention = (binding: PendingRetryBinding) => { pending.binding = binding; };
try {
source = await tx.transaction(async sourceTx => {
taskRetryInput = input.retryIntentId ? await selectTaskRetryInput(sourceTx, input) : undefined;
if (input.retryIntentId && taskRetryInput && !taskRetryInput.sourceActionIds.length) {
pending.restoreUnstartedInput = true;
taskRetryInput = await restoreUnstartedRetryInput(sourceTx, { ...input, onIdentityContention }) ?? taskRetryInput;
}
return failedChatRetrySource(sourceTx, { ...input, taskRetryIntentId: input.retryIntentId,
taskRetryInput, onIdentityContention });
});
} catch (error) {
if (!(error instanceof NativeChatReviewPresentationContentionError) || !input.retryIntentId || !pending.binding) throw error;
}
const binding = source ?? pending.binding!;
const payload = source ? { version: 1, ...source, initiatedByUserId: input.initiatedByUserId,
...(input.retryIntentId ? { taskRetryIntentId: input.retryIntentId } : {}) }
: pendingRetrySourceSchema.parse({ version: 1, sourceAuthorizationPending: true,
failedRunId: input.failedRunId, issueId: input.issueId, agentId: input.agentId,
initiatedByUserId: input.initiatedByUserId, taskRetryIntentId: input.retryIntentId, taskRetryInput,
...(pending.restoreUnstartedInput ? { restoreUnstartedInput: true } : {}) });
const providerActionId = input.retryIntentId
? `task-retry:${input.retryIntentId}` : `failed-run-retry:${source.failedRunId}`;
? `task-retry:${input.retryIntentId}` : `failed-run-retry:${source!.failedRunId}`;
const [existing] = await tx
.select()
.from(chatActions)
.where(
and(
eq(chatActions.endpointId, source.endpointId),
eq(chatActions.endpointId, binding.endpointId),
eq(chatActions.providerActionId, providerActionId),
),
)
@@ -1475,12 +1519,16 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
if (
existing.kind !== "failed_run_retry" ||
existing.companyId !== input.companyId ||
existing.payload.sourceScopeSha256 !== source.sourceScopeSha256
(source ? existing.payload.sourceScopeSha256 !== source.sourceScopeSha256
: JSON.stringify(existing.payload) !== JSON.stringify(payload))
)
throw failedChatRetryDenied();
return { actionId: existing.id, issueId: input.issueId };
}
if (!input.retryIntentId) await assertFailedNativeRetryState(tx, source, input.companyId);
if (!input.retryIntentId) {
if (!source) throw failedChatRetryDenied();
await assertFailedNativeRetryState(tx, source, input.companyId);
}
const [busy] = await tx
.select({ id: heartbeatRuns.id })
.from(heartbeatRuns)
@@ -1503,19 +1551,16 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
.insert(chatActions)
.values({
companyId: input.companyId,
endpointId: source.endpointId,
conversationId: source.conversationId,
principalId: source.principalId,
endpointId: binding.endpointId,
conversationId: binding.conversationId,
principalId: binding.principalId,
kind: "failed_run_retry",
providerActionId,
payload: {
version: 1,
...source,
initiatedByUserId: input.initiatedByUserId,
...(input.retryIntentId ? { taskRetryIntentId: input.retryIntentId } : {}),
},
payload,
status: "issued",
result: { code: "failed_chat_retry_queued", attemptCount: 0 },
result: { code: source ? "failed_chat_retry_queued" : "failed_chat_retry_contended", attemptCount: 0,
...(!source ? { retryAt: now.toISOString(), identityContentionCount: 0, identityContentionPending: true,
identityContentionDeadlineAt: new Date(now.getTime() + FAILED_RUN_RETRY_IDENTITY_CONTENTION_MS).toISOString() } : {}) },
createdAt: now,
updatedAt: now,
})
@@ -1530,12 +1575,56 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
details: {
retryActionId: intent.id,
issueId: input.issueId,
commentCount: source.commentIds.length,
...(source ? { commentCount: source.commentIds.length } : { sourceAuthorizationPending: true }),
},
});
return { actionId: intent.id, issueId: input.issueId };
}
function assertIdentityContentionDeadline(action: typeof chatActions.$inferSelect) {
const deadline = action.result?.identityContentionDeadlineAt;
if (deadline !== undefined && !(typeof deadline === "string" && Date.parse(deadline) > Date.now())) {
throw new NativeChatReviewPresentationContentionError();
}
}
async function preparePendingRetrySource(action: typeof chatActions.$inferSelect) {
const pending = pendingRetrySourceSchema.safeParse(action.payload);
if (!pending.success) throw failedChatRetryDenied();
return db.transaction(async tx => {
const input = pending.data;
const restored = input.restoreUnstartedInput ? await restoreUnstartedRetryInput(tx, {
companyId: action.companyId, issueId: input.issueId, agentId: input.agentId,
failedRunId: input.failedRunId, initiatedByUserId: input.initiatedByUserId,
}) : undefined;
const source = await failedChatRetrySource(tx, { companyId: action.companyId,
issueId: input.issueId, agentId: input.agentId, failedRunId: input.failedRunId,
taskRetryIntentId: input.taskRetryIntentId, taskRetryInput: restored ?? input.taskRetryInput });
const retry = await readTaskChatRetryIntent(tx as unknown as Db, { companyId: action.companyId,
issueId: input.issueId, agentId: input.agentId, sourceRunId: input.failedRunId,
intentId: input.taskRetryIntentId, chatActionId: action.id });
if (!retry || retry.wake.status !== "deferred_issue_execution" || retry.wake.runId ||
retry.intent.actorId !== input.initiatedByUserId ||
!await taskRetryMutationAllowed(tx as unknown as Db, { companyId: action.companyId,
issueId: input.issueId, actorId: retry.intent.actorId, actorSource: retry.intent.actorSource })) throw failedChatRetryDenied();
const [latest] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, action.companyId), or(eq(heartbeatRuns.issueId, input.issueId),
eq(heartbeatRuns.nativeIssueId, input.issueId), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${input.issueId}`),
)).orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
if (latest?.id !== input.failedRunId || source.endpointId !== action.endpointId ||
source.conversationId !== action.conversationId || source.principalId !== action.principalId) throw failedChatRetryDenied();
await notifyChatActionWork(tx, "failed_run_retry");
assertIdentityContentionDeadline(action);
const [prepared] = await tx.update(chatActions).set({ payload: { version: 1, ...source,
initiatedByUserId: input.initiatedByUserId, taskRetryIntentId: input.taskRetryIntentId } }).where(and(
eq(chatActions.id, action.id), eq(chatActions.companyId, action.companyId), eq(chatActions.status, "processing"),
eq(chatActions.updatedAt, action.updatedAt), sql`${chatActions.payload}->'sourceAuthorizationPending' = 'true'::jsonb`,
)).returning();
if (!prepared) throw failedChatRetryDenied();
return prepared;
});
}
async function authorizeFailedChatRunRetry(
tx: DbOrTransaction,
input: {
@@ -1545,6 +1634,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
retryActionId: string;
runId?: string;
publication?: boolean;
onIdentityContention?: (binding: PendingRetryBinding) => void;
},
) {
const [snapshot] = await tx
@@ -1564,6 +1654,10 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
snapshot.payload.agentId !== input.agentId
)
throw failedChatRetryDenied();
if (snapshot.payload.sourceAuthorizationPending === true) {
if (["failed", "cancelled"].includes(snapshot.status)) throw failedChatRetryDenied();
throw new NativeChatReviewPresentationContentionError();
}
const source = await failedChatRetrySource(tx, {
...input,
failedRunId: snapshot.payload.failedRunId,
@@ -1773,7 +1867,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
Date.parse(candidate.result.retryAt) > now.getTime())
)
return failedChatRetryReceipt(candidate);
const [claimed] = await db.transaction(async (tx) => {
const [claimedCandidate] = await db.transaction(async (tx) => {
await notifyChatActionWork(tx, "failed_run_retry");
return tx
.update(chatActions)
@@ -1787,9 +1881,26 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
)
.returning();
});
if (!claimed) return failedChatRetryReceipt(candidate);
const attemptCount = Number(candidate.result?.attemptCount ?? 0) + 1;
if (!claimedCandidate) return failedChatRetryReceipt(candidate);
let claimed = claimedCandidate;
const previousAttemptCount = Number(candidate.result?.attemptCount ?? 0);
const attemptCount = previousAttemptCount + 1;
let observedIdentityContention = false;
let contentionResult: Record<string, unknown> = {
...(candidate.result?.identityContentionDeadlineAt !== undefined
? { identityContentionDeadlineAt: candidate.result.identityContentionDeadlineAt } : {}),
...(candidate.result?.identityContentionCount !== undefined
? { identityContentionCount: candidate.result.identityContentionCount } : {}),
...(candidate.result?.identityContentionPending === true ? { identityContentionPending: true } : {}),
};
const settle = async (status: string, result: Record<string, unknown>) => {
const settledResult = { ...contentionResult, ...result };
if (["processed", "failed", "cancelled"].includes(status)) delete settledResult.identityContentionPending;
const durableResult = !observedIdentityContention && settledResult.identityContentionPending === true
? sql`(${JSON.stringify(settledResult)}::jsonb - 'identityContentionPending') || case
when ${chatActions.result}->'identityContentionPending' = 'true'::jsonb
then jsonb_build_object('identityContentionPending', true) else '{}'::jsonb end`
: settledResult;
const [current] = await db.transaction(async (tx) => {
if (typeof claimed.payload.taskRetryIntentId === "string") await notifyTaskRetrySettlement(tx, {
companyId: claimed.companyId, requestId: claimed.payload.taskRetryIntentId,
@@ -1800,7 +1911,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
});
return tx
.update(chatActions)
.set({ status, result, updatedAt: new Date() })
.set({ status, result: durableResult, updatedAt: new Date() })
.where(
and(
eq(chatActions.id, claimed.id),
@@ -1813,15 +1924,31 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
return failedChatRetryReceipt(await reconcileFailedTaskRetry(current ?? claimed));
};
try {
if (claimed.payload.sourceAuthorizationPending === true || contentionResult.identityContentionPending === true)
assertIdentityContentionDeadline(claimed);
if (claimed.payload.sourceAuthorizationPending === true) {
claimed = await preparePendingRetrySource(claimed);
}
const scope = {
companyId: claimed.companyId,
issueId: String(claimed.payload.issueId),
agentId: String(claimed.payload.agentId),
retryActionId: claimed.id,
};
const source = await db.transaction((tx) =>
authorizeFailedChatRunRetry(tx, scope),
);
const source = await db.transaction(async tx => {
const authorized = await authorizeFailedChatRunRetry(tx, scope);
if (contentionResult.identityContentionPending === true) {
await notifyChatActionWork(tx, "failed_run_retry");
assertIdentityContentionDeadline(claimed);
const [advanced] = await tx.update(chatActions).set({
result: sql`coalesce(${chatActions.result}, '{}'::jsonb) - 'identityContentionPending'`,
}).where(and(eq(chatActions.companyId, claimed.companyId), eq(chatActions.id, claimed.id),
eq(chatActions.status, "processing"), eq(chatActions.updatedAt, now))).returning({ id: chatActions.id });
if (!advanced) throw failedChatRetryDenied();
}
return authorized;
});
delete contentionResult.identityContentionPending;
const existing = await db
.select()
.from(agentWakeupRequests)
@@ -1945,6 +2072,24 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
code: "failed_chat_retry_durable", attemptCount,
});
}
if (error instanceof NativeChatReviewPresentationContentionError) {
observedIdentityContention = true;
const contendedAt = Date.now();
const identityContentionDeadlineAt = contentionResult.identityContentionDeadlineAt
?? new Date(contendedAt + FAILED_RUN_RETRY_IDENTITY_CONTENTION_MS).toISOString();
const deadline = typeof identityContentionDeadlineAt === "string" ? Date.parse(identityContentionDeadlineAt) : NaN;
const identityContentionCount = Number(contentionResult.identityContentionCount ?? 0) + 1;
contentionResult = { identityContentionDeadlineAt, identityContentionCount, identityContentionPending: true };
const retryable = Number.isSafeInteger(previousAttemptCount) && previousAttemptCount >= 0 && previousAttemptCount < 8
&& Number.isSafeInteger(identityContentionCount) && identityContentionCount > 0
&& Number.isFinite(deadline) && contendedAt < deadline;
return settle(retryable ? "issued" : "failed", {
code: retryable ? "failed_chat_retry_contended" : "failed_chat_retry_contention_expired",
attemptCount: previousAttemptCount,
...(retryable ? { retryAt: new Date(Math.min(deadline,
contendedAt + Math.min(5_000, identityContentionCount * 250))).toISOString() } : {}),
});
}
const retryable = !denied && attemptCount < 8;
return settle(retryable ? "issued" : "failed", {
code: retryable
@@ -116,10 +116,12 @@ type GitHubRetryBinding = {
export function createGitHubTaskRetryBatchAuthority<T extends GitHubRetryBinding>(authorize: (
tx: DbOrTransaction, action: typeof chatActions.$inferSelect,
notice: { nonblocking: true; terminal: boolean; taskRetry: TaskRetryWakeAuthority },
notice: { nonblocking: true; terminal: boolean; taskRetry: TaskRetryWakeAuthority;
onEndpointContention?: () => never },
) => Promise<T>) {
async function authorizeGitHubTaskRetryBatch(tx: DbOrTransaction, actions: Array<typeof chatActions.$inferSelect>,
notice: { nonblocking: true; terminal: boolean; taskRetry: TaskRetryWakeAuthority }) {
notice: { nonblocking: true; terminal: boolean; taskRetry: TaskRetryWakeAuthority;
onEndpointContention?: () => never }) {
const deny = chatWakeAuthorityDenied;
const first = actions[0];
if (!first) throw deny("batch_scope");
@@ -43,7 +43,8 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions
async function authorizeInboundWakeup(
tx: DbOrTransaction,
action: typeof chatActions.$inferSelect,
notice?: { nonblocking: true; terminal: boolean; taskRetry?: TaskRetryWakeAuthority },
notice?: { nonblocking: true; terminal: boolean; taskRetry?: TaskRetryWakeAuthority;
onEndpointContention?: () => never },
) {
const payload = action.payload;
const deny = chatWakeAuthorityDenied;
@@ -91,7 +92,17 @@ export function createInboundWakeAuthority(db: Db, options: WakeAuthorityOptions
)
.for("no key update", { noWait: true })
.limit(1)
.then((rows) => rows[0] ?? null);
.then((rows) => rows[0] ?? null)
.catch(error => {
if (notice?.onEndpointContention) {
let cause: unknown = error;
for (let depth = 0; depth < 4 && cause && typeof cause === "object"; depth += 1) {
if ("code" in cause && cause.code === "55P03") notice.onEndpointContention();
cause = "cause" in cause ? cause.cause : undefined;
}
}
throw error;
});
const delivery = await tx
.select()
.from(chatDeliveries)
@@ -7,6 +7,38 @@ const run = { id: "run", agentId: "agent", status: "failed", scheduledRetryAt: n
resultJson: { projectAccessDenied: { projectId: "22222222-2222-4222-8222-222222222222" } } };
const agent = { status: "idle", pauseReason: null };
describe("unadmitted chat Retry presentation", () => {
const now = Date.parse("2026-01-01T12:00:00Z");
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
const wait = { id: "retry", requestedAt: new Date(now), sourceRunId: run.id,
preparationStatus: "issued", preparationRetryAt: now + 250 };
const input = { run: failed, agent, wait, paused: false, approval: false };
it.each(["issued", "processing"])("uses the persisted %s outbox deadline without implying provider work", status => {
for (const offset of [0, 20_000]) expect(projectTaskInterruption(issue, {
...input, wait: { ...wait, preparationStatus: status },
}, now + offset)).toMatchObject({ state: "retry_scheduled", canRetry: false,
nextRetryAt: new Date(wait.preparationRetryAt).toISOString() });
});
it.each(["failed", "cancelled"])("keeps terminal %s preparation out of the working state", status => {
expect(projectTaskInterruption(issue, { ...input, wait: { ...wait, preparationStatus: status,
preparationRetryAt: null } }, now)).toMatchObject({ state: "failed", nextRetryAt: null });
});
it("keeps current gates and admitted execution authoritative", () => {
expect(projectTaskInterruption(issue, { ...input, paused: true }, now))
.toMatchObject({ state: "waiting_for_user", userAction: "resume" });
expect(projectTaskInterruption(issue, { ...input, approval: true }, now))
.toMatchObject({ state: "waiting_for_user", userAction: "approval" });
for (const status of ["queued", "running"]) expect(projectTaskInterruption(issue, {
...input, run: { ...failed, id: "successor", status },
}, now)).toMatchObject({ state: "working", nextRetryAt: null });
expect(projectTaskInterruption(issue, { ...input, wait: { ...wait, sourceRunId: "old-run" } }, now))
.toMatchObject({ state: "working", nextRetryAt: null });
});
});
it("explains a persisted Retry wait without exposing the saved payload", () => {
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
const wait = { id: "retry", requestedAt: new Date(), sourceRunId: failed.id,
@@ -1,5 +1,5 @@
import { and, desc, eq, inArray, isNull, sql, type SQL } from "drizzle-orm";
import { agents, agentWakeupRequests, approvals, companies, heartbeatRuns, issueApprovals, issueRecoveryActions,
import { agents, agentWakeupRequests, approvals, chatActions, companies, heartbeatRuns, issueApprovals, issueRecoveryActions,
issues, issueTreeHoldMembers, issueTreeHolds, projects, type Db } from "@paperclipai/db";
import { unionAll } from "drizzle-orm/pg-core";
import { HEARTBEAT_RUN_STATUSES, readProviderFailure, type TaskInterruptionPresentation } from "@paperclipai/shared";
@@ -14,12 +14,15 @@ import { createRunSecretRedactionRegistry } from "./run-secret-redaction.js";
import { instanceSettingsService } from "./instance-settings.js";
import { redactCurrentUserText } from "../log-redaction.js";
import { logger } from "../middleware/logger.js";
import { canonicalExecutionUuid } from "./execution-identity.js";
import { chatActionDeadline } from "./chat-action-work.js";
type ProjectionIssue = Pick<typeof issues.$inferSelect, "id" | "companyId" | "status" | "assigneeAgentId" | "projectId">;
type ProjectionRun = Pick<typeof heartbeatRuns.$inferSelect,
"id" | "status" | "scheduledRetryAt" | "errorCode" | "livenessState" | "resultJson"> &
Partial<Pick<typeof heartbeatRuns.$inferSelect, "agentId" | "error" | "companyId" | "nativeIssueId" | "runtimeMode">>;
type ProjectionWait = { id: string; requestedAt: Date; sourceRunId?: string | null; executionWaitReason?: string | null };
type ProjectionWait = { id: string; requestedAt: Date; sourceRunId?: string | null; executionWaitReason?: string | null;
preparationStatus?: string | null; preparationRetryAt?: number | string | null };
type ProjectionState = {
run?: ProjectionRun;
action?: typeof issueRecoveryActions.$inferSelect;
@@ -105,6 +108,12 @@ export function projectTaskInterruption(issue: ProjectionIssue, input: Projectio
if (!ready) return { ...base, responsibility: "user", state: "waiting_for_user",
userAction: "project_access", projectAccess };
}
if (retryWait?.sourceRunId === run.id && retryWait.preparationStatus) {
const deadline = retryWait.preparationRetryAt == null ? NaN : Number(retryWait.preparationRetryAt);
if (["issued", "processing"].includes(retryWait.preparationStatus) && Number.isFinite(deadline))
return { ...base, state: "retry_scheduled", nextRetryAt: new Date(deadline).toISOString() };
return { ...base, state: "failed", canRetry: retryWait.preparationStatus !== "cancelled" };
}
const retryAt = system?.nextAttemptAt ?? run.scheduledRetryAt?.getTime();
if (retryAt && retryAt > now) return { ...base, state: "retry_scheduled",
nextRetryAt: new Date(retryAt).toISOString(), canRetry: true };
@@ -181,8 +190,16 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
// A new message must not hide an existing Retry's admission explanation.
db.selectDistinctOn([wakeIssue, isRetryWait], { issueId: wakeIssue, id: agentWakeupRequests.id, requestedAt: agentWakeupRequests.requestedAt,
sourceRunId: boundedJsonString(sql`${agentWakeupRequests.payload}#>'{taskRetry,sourceRunId}'`, 160),
executionWaitReason: boundedJsonString(sql`${agentWakeupRequests.payload}#>'{executionWait,reason}'`, 80) })
.from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, companyId), inArray(wakeIssue, ids),
executionWaitReason: boundedJsonString(sql`${agentWakeupRequests.payload}#>'{executionWait,reason}'`, 80),
preparationStatus: chatActions.status, preparationRetryAt: chatActionDeadline("failed_run_retry") })
.from(agentWakeupRequests).leftJoin(chatActions, and(eq(chatActions.companyId, agentWakeupRequests.companyId),
eq(chatActions.id, canonicalExecutionUuid(sql`${agentWakeupRequests.payload}#>>'{taskRetry,chatActionId}'`)),
eq(chatActions.kind, "failed_run_retry"), sql`${chatActions.payload}->>'taskRetryIntentId' = ${agentWakeupRequests.id}::text`,
sql`${chatActions.payload}->>'issueId' = ${wakeIssue}`,
sql`${chatActions.payload}->>'agentId' = ${agentWakeupRequests.agentId}::text`,
sql`${chatActions.payload}->>'failedRunId' = ${agentWakeupRequests.payload}#>>'{taskRetry,sourceRunId}'`,
inArray(chatActions.status, ["issued", "processing", "failed", "cancelled"])))
.where(and(eq(agentWakeupRequests.companyId, companyId), inArray(wakeIssue, ids),
eq(agentWakeupRequests.status, "deferred_issue_execution"), eq(agentWakeupRequests.requestedByActorType, "user")))
.orderBy(wakeIssue, isRetryWait, desc(agentWakeupRequests.requestedAt), desc(agentWakeupRequests.id)),
agentIds.length ? db.select({ id: agents.id, status: agents.status, pauseReason: agents.pauseReason }).from(agents)