mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each task shows an interruption banner. The banner offers Retry when an attempt failed and nothing will move the task forward > - The banner reads the task's newest run. A continuation that the system cancels before it starts can be the newest run, while a different run owns the task and does the work > - In a restart-chaos wave, the banner offered Retry for about 140 seconds while the owning run worked. The task then finished with no click > - A Retry that the user does not need breaks the rule that the needed-Retry rate is 0% > - This pull request makes the banner and the Retry endpoint read the same observed run. That run is the live run that owns the task, or else the newest real attempt > - The benefit is that Retry shows only when nothing will advance the task, and a shown Retry is accepted ## Linked Issues or Issue Description **What happened?** A restart-chaos wave (three release swaps) showed a Retry banner on one task for 141 to 144 seconds. The reason code was `issue_execution_lock_changed`. Nobody clicked Retry, and the task completed. Timeline (UTC): - 14:32:09. Run A starts. - 14:51:12. A user follow-up wakes run B. Run B is queued behind run A. - 14:52:46. Run A succeeds. The system schedules continuation run C (`retryOfRunId` = A, reason `native_agent_blocker_continuation`). Run B starts and takes the task execution lock. - 14:53:16. The dispatch gate cancels C before it starts with `issue_execution_lock_changed`, because run B owns the lock. This is correct de-duplication. - 14:53:17. The banner shows `failed` with Retry. Run C was created after run B, so C is the newest run. The projection reads only the newest run and ignores that run B is running and owns the lock. - 14:55:41. Run B succeeds and the task is done. The banner goes away. Retry was never needed. Run B owned the task for the full banner window. **Expected behavior** The banner does not offer Retry while a run owns the task and is queued or running. A run that the system cancelled before it started because another run took the lock is not a failed attempt. When Retry is shown, pressing it is accepted for the run that the banner shows. **Steps to reproduce** 1. Create a task. Run A succeeds and leaves the task in progress. 2. Queue run B for the same task (for example, a user comment). Create continuation run C after run B. 3. Start run B and set it as the task execution lock owner. 4. Cancel run C before it starts with error code `issue_execution_lock_changed`. 5. Read the task interruption. Before this change: `state: failed`, `canRetry: true`, observed run C. After this change: `state: working`, `canRetry: false`, observed run B. **Paperclip version or commit** `fe0c4aa95d` (master at the time of the wave). ## What Changed - `server/src/services/task-retry-eligibility.ts`: add two shared conditions. - `taskAttemptCondition()`: a run cancelled before it started with `recovery_no_longer_needed`, `issue_closed_before_start` or `issue_execution_lock_changed` is not an attempt. The first two codes were already excluded inline in the projection. `issue_execution_lock_changed` is new. - `liveExecutionOwnerCondition()`: the run is the task's execution lock owner and it is queued or running. - `server/src/services/task-interruption-presentation.ts`: the observed run is the live execution owner when there is one. Otherwise it is the newest attempt, with the same native, first-class and legacy precedence as before. The owner lookup is a primary-key read, so the per-task run scan does not grow. - `server/src/services/task-retry.ts`: Retry admission, the refused chat Retry continuation and the superseded-Retry check use the same attempt filter. Admission also accepts the live execution owner when the request names it. The newest attempt stays acceptable, so the automatic Retry hooks (restart, provider limit, budget resume) keep their behavior. - `heldTaskRetryRefusals` uses the same attempt filter. - New test `server/src/__tests__/task-retry-banner-execution-owner.test.ts`: one state table. Each row asserts the list and detail projection, and when Retry is shown, presses it through `requestTaskRetry`. ### State table (tested as one table) | Row | Runs and lock | Banner | Observed run | Retry press | | --- | --- | --- | --- | --- | | 1 | Continuation scheduled; follow-up running and owns the lock | working, no Retry | follow-up | n/a | | 2 | Continuation cancelled unstarted (`issue_execution_lock_changed`); follow-up running and owns the lock (t032) | working, no Retry | follow-up | n/a | | 3 | Same as row 2, lock pointer cleared mid-transfer | working, no Retry | follow-up | n/a | | 4 | Scheduled continuation overdue past the 2-minute dispatch grace; follow-up running and owns the lock | working, no Retry | follow-up | n/a | | 5 | Newer attempt failed; older queued run owns the lock | working, no Retry | queued owner | automatic Retry of the failed run is still accepted | | 6 | Continuation superseded; follow-up then succeeded | resolved, no Retry | follow-up | n/a | | 7 | Continuation superseded; follow-up then failed | failed, Retry | follow-up | accepted | | 8 | Continuation superseded; predecessor failed; nothing else runs | failed, Retry | predecessor | accepted | | 9 | Continuation started, then lost the lock | failed, Retry | continuation | accepted | | 10 | Lock pointer left on a finished failed run | failed, Retry | that run | accepted | | 11 | Row 2 with native runs (`nativeIssueId`) | working, no Retry | native follow-up | Retry naming the owner is accepted | | 12 | Queued owner past its 15-minute move-forward wait; newer failed attempt | working, Retry (`move_forward`) | queued owner | accepted | Invariants that the table checks: - Retry is never shown while something will advance the task (rows 1 to 6, 11). Row 12 offers only `move_forward` on the owner. - A Retry press that names the live owner is accepted (rows 1, 2, 3, 5, 11, 12). - Retry is always shown when nothing will advance the task (rows 7 to 10). - The shared attempt filter is tested in `heldTaskRetryRefusals`, `retireSupersededTaskRetry` and the refused chat Retry continuation (`person_with_superseded_continuation` in `chat-channels.integration.test.ts`). - When Retry is shown, pressing it is accepted for the observed run (rows 7 to 10). ## Verification - Red: with the service changes removed, rows 1 to 8 fail (8 of 10). Rows 9 and 10 pass on both sides. - Green: `cd server && npx vitest run src/__tests__/task-retry-banner-execution-owner.test.ts` passes 14 of 14. - Related suites pass: `task-interruption-query.test.ts` (includes the query-plan bound test), `task-interruption-automatic-path.test.ts`, `legacy-continuation-authority.test.ts`, `native-closed-session-retry.integration.test.ts`, `interrupted-preparation-restore-retry.test.ts`, and the other server tests that call `taskInterruptionPresentation`, `requestTaskRetry` or `heldTaskRetryRefusals`. - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - Low risk. The change is read-side only. It does not change scheduling, dispatch, the lock, or the run-dispatch gate. - A run cancelled before start with `issue_execution_lock_changed` no longer shows as a failed attempt. If the lock was released and nothing else ran, the banner shows the previous attempt. A failed previous attempt still offers Retry (row 8). - The projection query has one more union branch. It is a primary-key lookup through `issues.execution_run_id`. The existing query-plan bound test passes. - No runner, migration or API contract change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Refs #22306 (the earlier fix for a due scheduled retry shown as failed). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
715 lines
53 KiB
TypeScript
715 lines
53 KiB
TypeScript
import { canActorRetryConversation } from "../lib/conversation-wake-policy.js";
|
|
import { notifyStrandedAgentStatus } from "./live-run-recovery-notifications.js";
|
|
import { notifyExternalConversationSettlement } from "./external-conversation-settlement-notifications.js";
|
|
import { writeExecutionControl } from "./execution-control-notifications.js";
|
|
import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
|
|
import { z } from "zod";
|
|
import { userStopBoundaryAt } from "./run-cancellation.js";
|
|
import { readWorkspaceRecovery } from "./workspace-recovery-receipt.js";
|
|
import { randomUUID } from "node:crypto";
|
|
import { taskRetryRunAncestry } from "./recovery-run-ancestry.js";
|
|
import { and, desc, eq, inArray, isNull, or, sql } from "drizzle-orm";
|
|
import { agentWakeupRequests, chatActions, chatConversations, chatEndpoints, heartbeatRuns, issues, issueComments, runIdentityContexts, issueRecoveryActions, type Db } from "@paperclipai/db";
|
|
import { conflict, forbidden, notFound, HttpError } from "../errors.js";
|
|
import { persistActivity } from "./activity-log.js";
|
|
import { createSystemRecovery, readSystemRecovery, savedFilesUnrecoverable } from "./system-recovery-state.js";
|
|
import { adoptSystemRecovery, updateSystemRecovery } from "./system-recovery.js";
|
|
import { notifyChatActionWork } from "./chat-work-notifications.js";
|
|
import { authorizationService } from "./authorization.js";
|
|
import { preserveNativeFailedWorkspace, preserveTaskRemoteWorkspaceWriter } from "./native-runtime/native-failed-workspace-recovery.js";
|
|
import { NativeWorkspaceFinalizationBusyError } from "./native-runtime/native-workspace-finalization-ownership.js";
|
|
import { adapterExecutionControls } from "./adapter-execution-control.js";
|
|
import { hasRequiredWorkspaceRecovery } from "./workspace-restore-recovery-state.js";
|
|
import { deferredChatInputPredicate, resumeDeferredChatWake, unadmittedChatWakeupCondition } from "./durable-chat-wakeup.js";
|
|
import { notifyDeferredExecutionWork } from "./deferred-execution-notifications.js";
|
|
import { hasPendingRunCleanup } from "./execution-blocker.js";
|
|
import { liveExecutionOwnerCondition, taskAttemptCondition } from "./task-retry-eligibility.js";
|
|
import { undeliveredContinuation } from "./fresh-continuation-retry-fold.js";
|
|
import { PersistedExecutionWaitRefusal } from "./deferred-execution-wait.js";
|
|
import { rearmExhaustedNativeFinalization } from "./native-runtime/exhausted-finalization-retry.js";
|
|
import { NativeChatReviewPresentationContentionError } from "./native-runtime/native-chat-review-presentation.js";
|
|
import { isExternalChatWaitAuthorizationContention } from "./native-runtime/chat-attachment-reuse.js";
|
|
import { selectTaskRetryInput } from "./chat-channels/task-retry-input.js";
|
|
|
|
export async function taskRetryMutationAllowed(db: Db, input: {
|
|
companyId: string; issueId: string; actorId: string; actorSource: TaskRetryIntent["actorSource"];
|
|
}) {
|
|
const [issue] = await db.select({ status: issues.status, assigneeAgentId: issues.assigneeAgentId,
|
|
assigneeUserId: issues.assigneeUserId }).from(issues).where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId)));
|
|
if (!issue) return false;
|
|
const permission = await authorizationService(db).decide({
|
|
actor: { type: "board", userId: input.actorId, source: input.actorSource }, action: "issue:mutate",
|
|
resource: { type: "issue", companyId: input.companyId, issueId: input.issueId, ...issue },
|
|
});
|
|
return permission.allowed;
|
|
}
|
|
|
|
export function assertTaskRetryConversationAllowed(
|
|
issue: { conversationAgentId?: string | null; conversationUserId?: string | null },
|
|
actorId: string,
|
|
) {
|
|
if (!canActorRetryConversation(issue, actorId)) throw forbidden("Only the conversation owner can retry a chat run");
|
|
}
|
|
|
|
const taskRetryIntentSchema = z.object({
|
|
schema: z.literal("paperclip.task-retry.v1"), generationId: z.string().uuid(), sourceRunId: z.string().uuid(),
|
|
actorId: z.string().min(1), actorSource: z.enum(["local_implicit", "session", "board_key", "cloud_tenant"]).default("session"), strategy: z.enum(["evaluate", "continue", "fresh_isolated_workspace", "finalize_result"]),
|
|
chatActionId: z.string().uuid().optional(), canonicalIntentId: z.string().uuid().optional(),
|
|
queuedChatRequestId: z.string().uuid().optional(),
|
|
initiatedBy: z.enum(["restart_recovery", "provider_limit_reset"]).optional(),
|
|
boardMessageOrigin: z.object({ commentId: z.string().uuid(), identityContextId: z.string().uuid() }).optional(),
|
|
});
|
|
export type TaskRetryIntent = z.infer<typeof taskRetryIntentSchema>;
|
|
|
|
/** Dispatch declines a Retry intent for this reason when its own requester,
|
|
* not the task's shared state, prevents it from starting; a click by a user
|
|
* with access replaces it. An edited saved board message
|
|
* (`retry_source_changed`) is terminal: only new input continues. */
|
|
export const DECLINED_TASK_RETRY_REASONS = ["retry_authority_denied"] as const;
|
|
export function isDeclinedTaskRetry(payload: Record<string, unknown> | null | undefined) {
|
|
const wait = payload?.executionWait;
|
|
const reason = wait && typeof wait === "object" && !Array.isArray(wait) ? (wait as Record<string, unknown>).reason : null;
|
|
return DECLINED_TASK_RETRY_REASONS.some(value => value === reason);
|
|
}
|
|
export function readTaskRetryIntent(payload: Record<string, unknown> | null | undefined): TaskRetryIntent | null {
|
|
const result = taskRetryIntentSchema.safeParse(payload?.taskRetry);
|
|
return result.success ? result.data : null;
|
|
}
|
|
|
|
/** A board message has its own accepted user identity, not an external inbound
|
|
* receipt. Certify its exact unchanged source and retained pre-provider evidence;
|
|
* this selects the canonical Retry owner and never substitutes chat authority.
|
|
* Normal queue containment and current permissions still govern dispatch. */
|
|
export async function readTaskRetryBoardMessageOrigin(db: Db, input: {
|
|
companyId: string; issueId: string; agentId: string; sourceRunId: string; actorId: string;
|
|
}) {
|
|
return readBoardMessageRetrySource(db, input, new Set<string>());
|
|
}
|
|
|
|
async function readBoardMessageRetrySource(db: Db, input: {
|
|
companyId: string; issueId: string; agentId: string; sourceRunId: string; actorId: string;
|
|
}, seen: Set<string>): Promise<{ commentId: string; identityContextId: string } | null> {
|
|
if (seen.size >= 20 || seen.has(input.sourceRunId) || adapterExecutionControls.has(input.sourceRunId)) return null;
|
|
seen.add(input.sourceRunId);
|
|
const [task] = await db.select({ id: issues.id }).from(issues).where(and(
|
|
eq(issues.id, input.issueId), eq(issues.companyId, input.companyId), eq(issues.assigneeAgentId, input.agentId),
|
|
sql`${issues.status} not in ('done', 'cancelled')`));
|
|
if (!task) return null;
|
|
const sourceConditions = [
|
|
eq(heartbeatRuns.id, input.sourceRunId), eq(heartbeatRuns.companyId, input.companyId),
|
|
eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.responsibleUserId, input.actorId),
|
|
eq(heartbeatRuns.status, "failed"), eq(heartbeatRuns.runtimeMode, "legacy"), eq(heartbeatRuns.errorCode, "setup_failed"),
|
|
sql`${heartbeatRuns.finishedAt} is not null`,
|
|
sql`(${heartbeatRuns.issueId} is null or ${heartbeatRuns.issueId} = ${input.issueId}::uuid)`,
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${input.issueId}`,
|
|
sql`(${heartbeatRuns.contextSnapshot}->>'taskId' is null or ${heartbeatRuns.contextSnapshot}->>'taskId' = ${input.issueId})`,
|
|
sql`${heartbeatRuns.resultJson}#>>'{executionRecovery,kind}' = 'bootstrap'`,
|
|
sql`${heartbeatRuns.resultJson}#>'{executionRecovery,providerWorkStarted}' = 'false'::jsonb`,
|
|
sql`${heartbeatRuns.resultJson}->>'stopReason' = 'adapter_failed'`,
|
|
sql`not exists (select 1 from jsonb_object_keys(case when jsonb_typeof(${heartbeatRuns.resultJson}) = 'object'
|
|
then ${heartbeatRuns.resultJson} else '{}'::jsonb end) key
|
|
where key not in ('effectiveTimeoutSec', 'executionRecovery', 'stopReason', 'timeoutConfigured', 'timeoutFired', 'timeoutSource', 'failureCause'))`,
|
|
sql`${heartbeatRuns.runnerProfileJson}->'adapterDispatch' = '{"adapterType":"paperclip_runner"}'::jsonb`,
|
|
sql`not exists (select 1 from jsonb_object_keys(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object'
|
|
then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) key
|
|
where key not in ('adapterDispatch', 'aiConnectionNonAssigneeCommentWake', 'chatControlRecoveryAdmission'))`,
|
|
...[heartbeatRuns.nativeIssueId, heartbeatRuns.nativeSessionId, heartbeatRuns.nativePhase,
|
|
heartbeatRuns.runtimeModeResolvedAt, heartbeatRuns.runtimeModeResolverVersion, heartbeatRuns.runtimeModeReason, heartbeatRuns.driverKind, heartbeatRuns.driverVersion,
|
|
heartbeatRuns.completionContractId, heartbeatRuns.runnerInstanceId, heartbeatRuns.sessionIdAfter,
|
|
heartbeatRuns.externalRunId, heartbeatRuns.processPid, heartbeatRuns.processGroupId, heartbeatRuns.processStartedAt,
|
|
heartbeatRuns.logStore, heartbeatRuns.logRef, heartbeatRuns.logBytes, heartbeatRuns.logSha256, heartbeatRuns.stdoutExcerpt,
|
|
heartbeatRuns.stderrExcerpt, heartbeatRuns.lastOutputAt, heartbeatRuns.lastOutputBytes,
|
|
heartbeatRuns.lastOutputStream, heartbeatRuns.exitCode, heartbeatRuns.signal].map(column => sql`${column} is null`),
|
|
sql`${heartbeatRuns.lastOutputSeq} = 0`, sql`not ${heartbeatRuns.logCompressed}`,
|
|
sql`(${heartbeatRuns.usageJson} is null or ${heartbeatRuns.usageJson} - 'ledgerScope' = '{}'::jsonb)`,
|
|
sql`(${heartbeatRuns.controllerLeaseExpiresAt} is null or ${heartbeatRuns.controllerLeaseExpiresAt} <= now())`,
|
|
sql`not exists (select 1 from native_run_results where company_id = ${input.companyId}::uuid and run_id = ${input.sourceRunId}::uuid)`,
|
|
sql`not exists (select 1 from native_run_finalizations where company_id = ${input.companyId}::uuid and run_id = ${input.sourceRunId}::uuid)`,
|
|
sql`not exists (select 1 from environment_leases where company_id = ${input.companyId}::uuid and heartbeat_run_id = ${input.sourceRunId}::uuid)`,
|
|
sql`exists (select 1 from heartbeat_run_events where company_id = ${input.companyId}::uuid and run_id = ${input.sourceRunId}::uuid
|
|
and event_type = 'error' and stream = 'system' and source_event_id is null and message = ${heartbeatRuns.error})`,
|
|
sql`exists (select 1 from heartbeat_run_events where company_id = ${input.companyId}::uuid and run_id = ${input.sourceRunId}::uuid
|
|
and event_type = 'lifecycle' and stream = 'system' and source_event_id is null and message = 'task.run.measured'
|
|
and payload->>'durableSettlement' = 'true' and payload->>'outcome' = 'failed')`,
|
|
sql`not exists (select 1 from heartbeat_run_events where company_id = ${input.companyId}::uuid and run_id = ${input.sourceRunId}::uuid
|
|
and (stream <> 'system' or source_event_id is not null or source_instance_id is not null or source_seq is not null
|
|
or source_payload_sha256 is not null or protocol_schema_version is not null or event_type not in ('error', 'agent_observability.unavailable', 'lifecycle')
|
|
or (event_type = 'lifecycle' and (message <> 'task.run.measured' or payload->>'durableSettlement' is distinct from 'true'))
|
|
or (event_type = 'error' and message is distinct from ${heartbeatRuns.error})))`,
|
|
];
|
|
const [origin] = await db.select({ commentId: issueComments.id, identityContextId: runIdentityContexts.id })
|
|
.from(heartbeatRuns)
|
|
.innerJoin(runIdentityContexts, and(eq(runIdentityContexts.id, heartbeatRuns.activeIdentityContextId),
|
|
eq(runIdentityContexts.companyId, input.companyId), eq(runIdentityContexts.runId, heartbeatRuns.id),
|
|
eq(runIdentityContexts.status, "accepted"), eq(runIdentityContexts.responsibleUserId, input.actorId)))
|
|
.innerJoin(issueComments, and(eq(issueComments.id, runIdentityContexts.messageId),
|
|
eq(issueComments.companyId, input.companyId), eq(issueComments.issueId, input.issueId),
|
|
eq(issueComments.authorType, "user"), eq(issueComments.authorUserId, input.actorId),
|
|
sql`${issueComments.createdByRunId} is null`, sql`${issueComments.deletedAt} is null`,
|
|
eq(issueComments.createdAt, issueComments.updatedAt), sql`nullif(trim(${issueComments.body}), '') is not null`))
|
|
.innerJoin(agentWakeupRequests, and(eq(agentWakeupRequests.id, heartbeatRuns.wakeupRequestId),
|
|
eq(agentWakeupRequests.companyId, input.companyId), eq(agentWakeupRequests.agentId, input.agentId),
|
|
eq(agentWakeupRequests.runId, heartbeatRuns.id), eq(agentWakeupRequests.requestedByActorType, "user"),
|
|
eq(agentWakeupRequests.requestedByActorId, input.actorId), eq(agentWakeupRequests.reason, "issue_reopened_via_comment"), eq(agentWakeupRequests.status, "failed"),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${input.issueId}`,
|
|
sql`${agentWakeupRequests.payload}->>'commentId' = ${issueComments.id}::text`))
|
|
.where(and(...sourceConditions,
|
|
sql`${heartbeatRuns.contextSnapshot}->>'source' = 'issue.comment.reopen'`,
|
|
sql`${heartbeatRuns.contextSnapshot}->>'wakeCommentId' = ${issueComments.id}::text`,
|
|
sql`${heartbeatRuns.contextSnapshot}->'wakeCommentIds' = jsonb_build_array(${issueComments.id}::text)`,
|
|
)).limit(1);
|
|
if (origin) return origin;
|
|
// A repeated preparation failure must retain the same certified board
|
|
// source through its real coalesced intent and failed dispatch receipt.
|
|
// Every predecessor is checked again; cycles and long damaged chains fail closed.
|
|
const [successor] = await db.select({ identityContextId: heartbeatRuns.activeIdentityContextId,
|
|
retryId: sql<unknown>`${heartbeatRuns.contextSnapshot}->'taskRetryRequestId'`,
|
|
previousRunId: sql<unknown>`${heartbeatRuns.contextSnapshot}->'previousRunId'`,
|
|
dispatchKey: agentWakeupRequests.idempotencyKey }).from(heartbeatRuns)
|
|
.innerJoin(agentWakeupRequests, and(eq(agentWakeupRequests.id, heartbeatRuns.wakeupRequestId),
|
|
eq(agentWakeupRequests.companyId, input.companyId), eq(agentWakeupRequests.agentId, input.agentId),
|
|
eq(agentWakeupRequests.runId, heartbeatRuns.id), eq(agentWakeupRequests.status, "failed"),
|
|
eq(agentWakeupRequests.reason, "task_retry"), eq(agentWakeupRequests.requestedByActorType, "user"),
|
|
eq(agentWakeupRequests.requestedByActorId, input.actorId)))
|
|
.where(and(...sourceConditions, sql`${heartbeatRuns.contextSnapshot}->'boardMessageRetry' = 'true'::jsonb`)).limit(1);
|
|
if (!successor?.identityContextId || !z.string().uuid().safeParse(successor.retryId).success ||
|
|
successor.dispatchKey !== `task-retry-dispatch:${successor.retryId}`) return null;
|
|
const [receipt] = await db.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.id, String(successor.retryId)), eq(agentWakeupRequests.companyId, input.companyId),
|
|
eq(agentWakeupRequests.agentId, input.agentId), eq(agentWakeupRequests.runId, input.sourceRunId),
|
|
eq(agentWakeupRequests.status, "coalesced"), eq(agentWakeupRequests.reason, "task_retry"),
|
|
eq(agentWakeupRequests.requestedByActorType, "user"), eq(agentWakeupRequests.requestedByActorId, input.actorId),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${input.issueId}`));
|
|
const intent = readTaskRetryIntent(receipt?.payload);
|
|
if (!intent?.boardMessageOrigin || intent.generationId !== receipt!.id || intent.actorId !== input.actorId ||
|
|
intent.chatActionId || intent.queuedChatRequestId || intent.canonicalIntentId || intent.sourceRunId !== successor.previousRunId) return null;
|
|
const [identity] = await db.select({ id: runIdentityContexts.id }).from(runIdentityContexts).where(and(
|
|
eq(runIdentityContexts.id, successor.identityContextId), eq(runIdentityContexts.companyId, input.companyId),
|
|
eq(runIdentityContexts.runId, input.sourceRunId), eq(runIdentityContexts.responsibleUserId, input.actorId),
|
|
eq(runIdentityContexts.status, "accepted"), eq(runIdentityContexts.cause, "task_retry"),
|
|
eq(runIdentityContexts.parentContextId, intent.boardMessageOrigin.identityContextId),
|
|
sql`${runIdentityContexts.messageId} is null`)).limit(1);
|
|
if (!identity) return null;
|
|
const parent = await readBoardMessageRetrySource(db, { ...input, sourceRunId: intent.sourceRunId }, seen);
|
|
return parent && parent.commentId === intent.boardMessageOrigin.commentId &&
|
|
parent.identityContextId === intent.boardMessageOrigin.identityContextId
|
|
? { commentId: parent.commentId, identityContextId: identity.id } : null;
|
|
}
|
|
|
|
/** Caller holds the issue dispatch lock. Newer work supersedes an unadmitted
|
|
* Retry; retiring its authority does not acknowledge or discard saved input. */
|
|
export async function retireSupersededTaskRetry(db: Db, input: {
|
|
companyId: string; issueId: string; requestId: string; sourceRunId: string;
|
|
}) {
|
|
const [latest] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, input.companyId),
|
|
or(eq(heartbeatRuns.issueId, input.issueId), eq(heartbeatRuns.nativeIssueId, input.issueId),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${input.issueId}`), taskAttemptCondition(),
|
|
)).orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
|
|
if (latest?.id === input.sourceRunId) return false;
|
|
const [wake] = await db.select({ agentId: agentWakeupRequests.agentId }).from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, input.companyId), eq(agentWakeupRequests.id, input.requestId),
|
|
inArray(agentWakeupRequests.status, ["deferred_issue_execution", "queued", "claimed"]),
|
|
)).for("update");
|
|
if (!wake) return false;
|
|
await notifyStrandedAgentStatus(db, { companyId: input.companyId, agentId: wake.agentId });
|
|
await notifyExternalConversationSettlement(db, input);
|
|
const retired = await db.update(agentWakeupRequests).set({ status: "skipped", error: "task_retry_superseded",
|
|
finishedAt: new Date(), updatedAt: new Date() }).where(and(
|
|
eq(agentWakeupRequests.companyId, input.companyId), eq(agentWakeupRequests.id, input.requestId),
|
|
eq(agentWakeupRequests.reason, "task_retry"), sql`${agentWakeupRequests.runId} is null`,
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${input.issueId}`,
|
|
inArray(agentWakeupRequests.status, ["deferred_issue_execution", "queued", "claimed"]),
|
|
)).returning({ id: agentWakeupRequests.id });
|
|
return retired.length > 0;
|
|
}
|
|
|
|
export async function retireTaskRetriesContinuedBy(db: Db, input: {
|
|
companyId: string; issueId: string; agentId: string; sourceRunIds: string[];
|
|
}) {
|
|
if (!input.sourceRunIds.length) return;
|
|
const held = and(eq(agentWakeupRequests.companyId, input.companyId), eq(agentWakeupRequests.agentId, input.agentId),
|
|
eq(agentWakeupRequests.reason, "task_retry"), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
isNull(agentWakeupRequests.runId), sql`${agentWakeupRequests.payload}->>'issueId' = ${input.issueId}`,
|
|
inArray(sql<string>`${agentWakeupRequests.payload}#>>'{taskRetry,sourceRunId}'`, input.sourceRunIds));
|
|
const [heldRetry] = await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(held).limit(1);
|
|
if (!heldRetry) return;
|
|
await notifyStrandedAgentStatus(db, { companyId: input.companyId, agentId: input.agentId });
|
|
await notifyExternalConversationSettlement(db, { companyId: input.companyId, issueId: input.issueId });
|
|
await db.update(agentWakeupRequests).set({ status: "skipped", error: "task_retry_superseded",
|
|
finishedAt: new Date(), updatedAt: new Date() }).where(held);
|
|
}
|
|
|
|
type TaskRetryRequest = {
|
|
db: Db; companyId: string; issueId: string; actorId: string;
|
|
idempotencyKey: string; observedRunId: string | null; actorSource?: string;
|
|
retryAction?: "move_forward";
|
|
/** Paperclip requests the user's Retry itself: after a restart lost an
|
|
* unstarted turn, or once a provider usage limit it waited on has reset. */
|
|
initiatedBy?: "restart_recovery" | "provider_limit_reset";
|
|
prepareChatRetry?: (tx: Db, scope: { companyId: string; issueId: string; agentId: string;
|
|
failedRunId: string; initiatedByUserId: string; retryIntentId?: string }) => Promise<{ actionId: string }>;
|
|
};
|
|
|
|
/** Activity actor for a Retry: the user, or the Paperclip hook acting for them. */
|
|
function automaticRetryActor(input: Pick<TaskRetryRequest, "initiatedBy" | "actorId">) {
|
|
if (!input.initiatedBy) return { actorType: "user" as const, actorId: input.actorId };
|
|
return { actorType: "system" as const,
|
|
actorId: input.initiatedBy === "restart_recovery" ? "restart-recovery" : "provider-limit-reset" };
|
|
}
|
|
|
|
const TASK_RETRY_CONTENTION_ATTEMPTS = 10;
|
|
const TASK_RETRY_CONTENTION_DELAY_MS = 100;
|
|
|
|
function isExactChatReplayRefusal(error: unknown) {
|
|
return error instanceof HttpError && error.status === 409 &&
|
|
(error.details as Record<string, unknown> | undefined)?.code === "chat_failed_run_retry_not_authorized";
|
|
}
|
|
|
|
function isTaskRetryContention(error: unknown) {
|
|
return error instanceof NativeChatReviewPresentationContentionError || isExternalChatWaitAuthorizationContention(error);
|
|
}
|
|
|
|
/** Persist a deliberate click under dispatch's issue-first lock order. This
|
|
* transaction never contacts a provider. The normal wake queue owns execution. */
|
|
export async function requestTaskRetry(input: TaskRetryRequest) {
|
|
for (let attempt = 1; ; attempt += 1) {
|
|
try {
|
|
return await admitTaskRetry(input);
|
|
} catch (error) {
|
|
if (!isTaskRetryContention(error)) throw error;
|
|
// Chat authorization for the same principal is briefly locked. The whole
|
|
// admission rolled back, so a fresh attempt rereads every fact.
|
|
if (attempt >= TASK_RETRY_CONTENTION_ATTEMPTS)
|
|
throw conflict("Retry is busy because chat authorization is in progress. Try again.",
|
|
{ code: "task_retry_busy", retryable: true });
|
|
}
|
|
await new Promise(resolve => setTimeout(resolve, TASK_RETRY_CONTENTION_DELAY_MS));
|
|
}
|
|
}
|
|
|
|
function admitTaskRetry(input: TaskRetryRequest) {
|
|
return writeExecutionControl(input.db, DELIVERY_QUEUES.workspaceRecovery, async tx => {
|
|
const db = tx as unknown as Db;
|
|
const [issue] = await tx.select().from(issues).where(and(eq(issues.companyId, input.companyId),
|
|
eq(issues.id, input.issueId))).for("update");
|
|
if (!issue) throw notFound("Task not found");
|
|
assertTaskRetryConversationAllowed(issue, input.actorId);
|
|
const key = `task-retry:${input.idempotencyKey}`;
|
|
const [sameRequest] = await tx.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, input.companyId), eq(agentWakeupRequests.idempotencyKey, key)));
|
|
if (sameRequest) {
|
|
if (sameRequest.payload?.issueId !== issue.id || sameRequest.requestedByActorId !== input.actorId)
|
|
throw conflict("This retry request belongs to a different task or user.");
|
|
const intent = readTaskRetryIntent(sameRequest.payload);
|
|
const [coalescedRun] = sameRequest.status === "coalesced" && sameRequest.runId
|
|
? await tx.select({ id: heartbeatRuns.id, status: heartbeatRuns.status }).from(heartbeatRuns)
|
|
.where(and(eq(heartbeatRuns.companyId, issue.companyId), eq(heartbeatRuns.id, sameRequest.runId)))
|
|
: [];
|
|
return { requestId: intent?.canonicalIntentId ?? sameRequest.id, accepted: true, stale: false,
|
|
observedRunId: intent?.sourceRunId ?? null, chatActionId: intent?.chatActionId,
|
|
...(coalescedRun?.status === "scheduled_retry" ? { scheduledRetryRunId: coalescedRun.id } : {}) };
|
|
}
|
|
const [latest] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, issue.companyId),
|
|
or(eq(heartbeatRuns.issueId, issue.id), eq(heartbeatRuns.nativeIssueId, issue.id),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`), taskAttemptCondition()))
|
|
.orderBy(...(input.observedRunId ? [sql`case when ${heartbeatRuns.id}::text = ${input.observedRunId}
|
|
and ${liveExecutionOwnerCondition(issue.companyId, issue.id)} then 0 else 1 end`] : []),
|
|
desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
|
|
if (["done", "cancelled"].includes(issue.status) || !issue.assigneeAgentId || !latest ||
|
|
latest.agentId !== issue.assigneeAgentId || input.observedRunId !== latest.id) {
|
|
return { requestId: null, accepted: false, stale: true, observedRunId: latest?.id ?? null, chatActionId: undefined };
|
|
}
|
|
if (input.retryAction === "move_forward" && !["queued", "running"].includes(latest.status) &&
|
|
!(await undeliveredContinuation(tx, { companyId: issue.companyId, issueId: issue.id })))
|
|
return { requestId: null, accepted: false, stale: false, alreadyMovedOn: true, observedRunId: latest.id, chatActionId: undefined };
|
|
let active = (await tx.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, issue.companyId), eq(agentWakeupRequests.agentId, issue.assigneeAgentId),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`,
|
|
sql`${agentWakeupRequests.payload}->'taskRetry'->>'schema' = 'paperclip.task-retry.v1'`,
|
|
inArray(agentWakeupRequests.status, ["deferred_issue_execution", "queued", "claimed"]),
|
|
)).orderBy(desc(agentWakeupRequests.requestedAt)).limit(1)).at(0);
|
|
const activeIntent = readTaskRetryIntent(active?.payload);
|
|
if (active && !active.runId && activeIntent && await retireSupersededTaskRetry(db, {
|
|
companyId: issue.companyId, issueId: issue.id, requestId: active.id, sourceRunId: activeIntent.sourceRunId,
|
|
})) active = undefined;
|
|
if (active && !active.runId && issue.conversationAgentId && active.requestedByActorId !== issue.conversationUserId) {
|
|
// A former owner's unadmitted receipt cannot own the new owner's Retry.
|
|
await tx.update(agentWakeupRequests).set({ status: "failed", finishedAt: new Date(), updatedAt: new Date() })
|
|
.where(eq(agentWakeupRequests.id, active.id));
|
|
active = undefined;
|
|
}
|
|
// An accepted result that only failed to finalize is applied again, never
|
|
// replaced by a new provider run. The receipt coalesces into that run.
|
|
const finalizeId = randomUUID();
|
|
const savedResult = latest.runtimeMode === "native" && latest.status === "succeeded"
|
|
? await rearmExhaustedNativeFinalization(db, { companyId: issue.companyId, issueId: issue.id, runId: latest.id,
|
|
generationId: active?.id ?? finalizeId, actorId: input.actorId })
|
|
: { kind: "not_eligible" as const };
|
|
if (savedResult.kind !== "not_eligible") {
|
|
// A later click joins the saved-result Retry already under way (or applied).
|
|
const canonicalId = (savedResult.kind === "joined" ? savedResult.generationId : null) ?? active?.id ?? null;
|
|
const finalizeIntent: TaskRetryIntent = { schema: "paperclip.task-retry.v1", generationId: canonicalId ?? finalizeId,
|
|
sourceRunId: latest.id, actorId: input.actorId,
|
|
actorSource: taskRetryIntentSchema.shape.actorSource.parse(input.actorSource ?? "session"), strategy: "finalize_result",
|
|
...(canonicalId ? { canonicalIntentId: canonicalId } : {}) };
|
|
const settled = { status: "coalesced", runId: latest.id, finishedAt: new Date(), updatedAt: new Date() } as const;
|
|
// An unadmitted Retry for this run would start a provider turn; it joins the re-finalization instead.
|
|
if (active && !active.runId) await tx.update(agentWakeupRequests).set({ ...settled,
|
|
payload: sql`jsonb_set(${agentWakeupRequests.payload}, '{taskRetry,strategy}', '"finalize_result"'::jsonb)` })
|
|
.where(eq(agentWakeupRequests.id, active.id));
|
|
await tx.insert(agentWakeupRequests).values({ id: finalizeId, companyId: issue.companyId, agentId: issue.assigneeAgentId,
|
|
source: "on_demand", triggerDetail: "manual", reason: "task_retry", idempotencyKey: key,
|
|
requestedByActorType: "user", requestedByActorId: input.actorId, ...settled,
|
|
payload: { issueId: issue.id, taskRetry: finalizeIntent } });
|
|
await persistActivity(db, { companyId: issue.companyId, ...automaticRetryActor(input),
|
|
action: "issue.retry_requested", entityType: "issue", entityId: issue.id, runId: latest.id,
|
|
details: { requestId: canonicalId ?? finalizeId, sourceRunId: latest.id, attachedToExistingWork: Boolean(canonicalId),
|
|
strategy: "finalize_result",
|
|
...(input.initiatedBy ? { initiatedBy: input.initiatedBy, onBehalfOfUserId: input.actorId } : {}) } });
|
|
const finalizing = savedResult.kind === "rearmed" || savedResult.finalizing;
|
|
return { requestId: canonicalId ?? finalizeId, accepted: true, stale: false, observedRunId: latest.id,
|
|
chatActionId: undefined, ...(finalizing ? { finalizeRunId: latest.id } : {}) };
|
|
}
|
|
const busy = ["queued", "running", "scheduled_retry"].includes(latest.status);
|
|
if (!busy) {
|
|
await preserveTaskRemoteWorkspaceWriter(db, issue.companyId, issue.id);
|
|
// Historical deadline failures can predate their preservation record.
|
|
// Register the exact prepared source; this proves neither a stopped
|
|
// executor nor permission to replay its work.
|
|
try { await preserveNativeFailedWorkspace(db, latest); }
|
|
catch (error) { if (!(error instanceof NativeWorkspaceFinalizationBusyError)) throw error; }
|
|
}
|
|
const retrySources = !busy ? await taskRetryRunAncestry({ db, companyId: issue.companyId,
|
|
issueId: issue.id, agentId: issue.assigneeAgentId, sourceRunId: latest.id }) : null;
|
|
const records = await tx.select().from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.companyId, issue.companyId), eq(issueRecoveryActions.sourceIssueId, issue.id),
|
|
inArray(sql`${issueRecoveryActions.evidence}->>'runId'`, retrySources ?? [latest.id]))).for("update");
|
|
for (let index = 0; !busy && index < records.length; index++) {
|
|
const action = records[index];
|
|
if (!hasRequiredWorkspaceRecovery(action.evidence) || readSystemRecovery(action.evidence)) continue;
|
|
await adoptSystemRecovery(db, { companyId: issue.companyId, issueId: issue.id,
|
|
runId: String(action.evidence.runId), actionId: action.id });
|
|
[records[index]] = await tx.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id));
|
|
}
|
|
// An unadmitted request can outlive its exhausted repair. A new deliberate
|
|
// click replaces that generation; a replay of the same key returned above.
|
|
if (active && !busy && !active.runId && records.some(action => {
|
|
const recovery = readSystemRecovery(action.evidence);
|
|
return recovery?.state === "exhausted" && recovery.sourceRunId === action.evidence.runId;
|
|
})) {
|
|
await tx.update(agentWakeupRequests).set({ status: "failed", finishedAt: new Date(), updatedAt: new Date() })
|
|
.where(eq(agentWakeupRequests.id, active.id));
|
|
active = undefined;
|
|
}
|
|
// Dispatch declined this unadmitted intent for a reason bound to the intent
|
|
// itself (its requester's access or saved source), so it can never start.
|
|
// A new deliberate click (new key) replaces it with a fresh attempt under the
|
|
// clicker's current authority instead of joining it. Waits on shared state
|
|
// (cleanup, recovery, an active run) keep coalescing: that intent starts by
|
|
// itself once the gate clears. An automatic hook never replaces a user's click.
|
|
let replacedRequestId: string | undefined;
|
|
if (active && !busy && !active.runId && active.status === "deferred_issue_execution" &&
|
|
!input.initiatedBy && activeIntent && !activeIntent.chatActionId &&
|
|
!activeIntent.queuedChatRequestId && isDeclinedTaskRetry(active.payload)) {
|
|
await tx.update(agentWakeupRequests).set({ status: "skipped", error: "task_retry_replaced",
|
|
finishedAt: new Date(), updatedAt: new Date() }).where(and(eq(agentWakeupRequests.id, active.id),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"), sql`${agentWakeupRequests.runId} is null`));
|
|
replacedRequestId = active.id;
|
|
active = undefined;
|
|
}
|
|
const id = randomUUID();
|
|
const intent: TaskRetryIntent = { schema: "paperclip.task-retry.v1", generationId: active?.id ?? id,
|
|
sourceRunId: latest.id, actorId: input.actorId,
|
|
actorSource: taskRetryIntentSchema.shape.actorSource.parse(input.actorSource ?? "session"), strategy: "evaluate",
|
|
...(active && activeIntent?.chatActionId ? { chatActionId: activeIntent.chatActionId } : {}),
|
|
...(active && activeIntent?.queuedChatRequestId ? { queuedChatRequestId: activeIntent.queuedChatRequestId } : {}),
|
|
...(active ? { canonicalIntentId: active.id } : {}),
|
|
...(input.initiatedBy && !active ? { initiatedBy: input.initiatedBy } : {}) };
|
|
// Preserve external chat authority through its existing admission owner.
|
|
// Generic board Retry must never manufacture a chat-inbound request.
|
|
const [chat] = await tx.select({ id: chatConversations.id }).from(chatConversations)
|
|
.innerJoin(chatEndpoints, eq(chatEndpoints.id, chatConversations.endpointId)).where(and(
|
|
eq(chatConversations.companyId, issue.companyId), eq(chatConversations.issueId, issue.id),
|
|
eq(chatEndpoints.externalExecutionPolicy, "restricted"))).limit(1);
|
|
const boardMessageOrigin = chat && !active && !busy
|
|
? await readTaskRetryBoardMessageOrigin(db, { companyId: issue.companyId,
|
|
issueId: issue.id, agentId: issue.assigneeAgentId, sourceRunId: latest.id, actorId: input.actorId }) : null;
|
|
if (boardMessageOrigin) intent.boardMessageOrigin = boardMessageOrigin;
|
|
if (chat && !active && latest.status === "succeeded") {
|
|
const [savedInput] = await tx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, issue.companyId), eq(agentWakeupRequests.agentId, issue.assigneeAgentId),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"), sql`${agentWakeupRequests.runId} is null`,
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issue.id}`, deferredChatInputPredicate(db),
|
|
)).orderBy(desc(agentWakeupRequests.requestedAt), desc(agentWakeupRequests.id)).limit(1).for("update");
|
|
// A completed turn stays history. Only independently admitted input can
|
|
// continue, with its original identity and current inbound authorization.
|
|
if (savedInput) {
|
|
intent.queuedChatRequestId = savedInput.id;
|
|
await notifyDeferredExecutionWork(tx, { companyId: issue.companyId, requestId: id });
|
|
}
|
|
}
|
|
const savedBoardTurn = Boolean(chat && !active && !busy && !boardMessageOrigin && !intent.queuedChatRequestId &&
|
|
(latest.contextSnapshot?.source === "issue.comment.reopen" || latest.contextSnapshot?.boardMessageRetry === true) &&
|
|
latest.responsibleUserId === input.actorId);
|
|
const hasUnadmittedChatInput = () => taskHasUnadmittedChatInput(db, { companyId: issue.companyId, issueId: issue.id,
|
|
agentId: issue.assigneeAgentId!, failedRunId: latest.id });
|
|
if (savedBoardTurn && (input.initiatedBy || await hasUnadmittedChatInput())) {
|
|
throw conflict("This saved message can't be repeated exactly as it was sent. Retry continues the task from its saved work instead.",
|
|
{ code: "chat_failed_run_retry_not_authorized", reason: "saved_board_message_not_replayable" });
|
|
}
|
|
await tx.insert(agentWakeupRequests).values({ id, companyId: issue.companyId, agentId: issue.assigneeAgentId,
|
|
source: "on_demand", triggerDetail: "manual", reason: "task_retry", idempotencyKey: key,
|
|
requestedByActorType: "user", requestedByActorId: input.actorId,
|
|
status: active || busy ? "coalesced" : "deferred_issue_execution",
|
|
runId: active?.runId ?? (busy ? latest.id : null),
|
|
payload: { issueId: issue.id, taskRetry: intent }, finishedAt: active || busy ? new Date() : null });
|
|
if (chat && !boardMessageOrigin && !intent.queuedChatRequestId && !active && !busy && !savedBoardTurn) {
|
|
if (!input.prepareChatRetry && input.initiatedBy) throw conflict("Retry is unavailable for this connected conversation.");
|
|
const prepareChatRetry = input.prepareChatRetry ?? (() => { throw conflict("Retry is unavailable for this connected conversation.",
|
|
{ code: "chat_failed_run_retry_not_authorized" }); });
|
|
const scope = { companyId: issue.companyId, issueId: issue.id, agentId: issue.assigneeAgentId,
|
|
failedRunId: latest.id, initiatedByUserId: input.actorId, retryIntentId: id };
|
|
const prepared = await tx.transaction(savepoint => prepareChatRetry(savepoint as unknown as Db, scope))
|
|
.catch(async error => {
|
|
if (input.initiatedBy || !isExactChatReplayRefusal(error)) throw error;
|
|
if (await hasUnadmittedChatInput()) throw error;
|
|
return null;
|
|
});
|
|
if (prepared) {
|
|
intent.chatActionId = prepared.actionId;
|
|
await tx.update(agentWakeupRequests).set({ payload: { issueId: issue.id, taskRetry: intent } })
|
|
.where(eq(agentWakeupRequests.id, id));
|
|
}
|
|
}
|
|
if (!busy) {
|
|
for (const action of records) {
|
|
const previous = readSystemRecovery(action.evidence);
|
|
if (!previous || previous.sourceRunId !== action.evidence.runId) continue;
|
|
// A completed preservation still owns its snapshot evidence. Bind the
|
|
// new intent so sibling holds can select that same continuation, without
|
|
// repeating recovery or renewing its attempt budget.
|
|
if (previous.state === "complete" || previous.state === "running" || savedFilesUnrecoverable(previous) || (previous.state === "exhausted"
|
|
&& previous.failure?.retryClass === "use_alternative_strategy" && !previous.freshAttemptRunId)) {
|
|
await updateSystemRecovery(db, { companyId: issue.companyId, issueId: issue.id, runId: previous.sourceRunId,
|
|
actionId: action.id }, previous, { ...previous, intentId: active?.id ?? id });
|
|
continue;
|
|
}
|
|
// Active repair keeps its attempt and cooldown. An exhausted generation
|
|
// gets a new budget only because this click is a new durable intent.
|
|
const next = previous.state === "exhausted"
|
|
? { ...createSystemRecovery(previous.sourceRunId, id), revision: previous.revision + 1,
|
|
preservationOperationIds: previous.preservationOperationIds,
|
|
providerCooldownAt: previous.providerCooldownAt, nextAttemptAt: previous.providerCooldownAt }
|
|
: { ...previous, revision: previous.revision + 1, intentId: active?.id ?? id,
|
|
// Retry may advance our backoff, never a provider's cooldown and
|
|
// never an operation whose physical owner is still running.
|
|
nextAttemptAt: Math.max(Date.now(), previous.providerCooldownAt ?? 0) };
|
|
if (next.generationId !== previous.generationId) await persistActivity(db, {
|
|
companyId: issue.companyId, actorType: "system", actorId: "task-retry", entityType: "issue", entityId: issue.id,
|
|
action: "issue.recovery_generation_replaced", runId: latest.id,
|
|
details: { recoveryActionId: action.id, previousGeneration: previous, nextGenerationId: next.generationId },
|
|
});
|
|
await updateSystemRecovery(db, { companyId: issue.companyId, issueId: issue.id, runId: previous.sourceRunId,
|
|
actionId: action.id }, previous, next);
|
|
}
|
|
}
|
|
await persistActivity(db, { companyId: issue.companyId, ...automaticRetryActor(input),
|
|
action: "issue.retry_requested", entityType: "issue", entityId: issue.id, runId: latest.id,
|
|
details: { requestId: active?.id ?? id, sourceRunId: latest.id, attachedToExistingWork: Boolean(active || busy),
|
|
...(replacedRequestId ? { replacedRequestId } : {}),
|
|
...(input.initiatedBy ? { initiatedBy: input.initiatedBy, onBehalfOfUserId: input.actorId } : {}) } });
|
|
return { requestId: active?.id ?? id, accepted: true, stale: false, observedRunId: latest.id,
|
|
chatActionId: intent.chatActionId,
|
|
...(!active && latest.status === "scheduled_retry" ? { scheduledRetryRunId: latest.id } : {}) };
|
|
}, { companyId: input.companyId });
|
|
}
|
|
|
|
async function taskHasUnadmittedChatInput(db: Db, input: { companyId: string; issueId: string; agentId: string; failedRunId: string }) {
|
|
const [held] = await db.select({ id: issues.id }).from(issues).where(and(eq(issues.id, input.issueId),
|
|
eq(issues.companyId, input.companyId), unadmittedChatWakeupCondition(issues.id, issues.companyId))).limit(1);
|
|
return Boolean(held) || Boolean((await selectTaskRetryInput(db, input))?.sourceActionIds.length);
|
|
}
|
|
|
|
async function continueRefusedTaskChatRetry(db: Db, wake: typeof agentWakeupRequests.$inferSelect,
|
|
intent: TaskRetryIntent, chat: typeof chatActions.$inferSelect) {
|
|
if (intent.canonicalIntentId || intent.queuedChatRequestId || intent.generationId !== wake.id ||
|
|
wake.requestedByActorType !== "user" || wake.requestedByActorId !== intent.actorId ||
|
|
!wake.idempotencyKey?.startsWith("task-retry:") || chat.status !== "failed" ||
|
|
chat.payload.taskRetryIntentId !== wake.id) return false;
|
|
const issueId = String(wake.payload?.issueId);
|
|
return db.transaction(async tx => {
|
|
const [issue] = await tx.select().from(issues).where(and(eq(issues.id, issueId),
|
|
eq(issues.companyId, wake.companyId))).for("update");
|
|
if (!issue || issue.assigneeAgentId !== wake.agentId || ["done", "cancelled"].includes(issue.status)) return false;
|
|
const [latest] = await tx.select({ id: heartbeatRuns.id, finishedAt: heartbeatRuns.finishedAt, status: heartbeatRuns.status,
|
|
startedAt: heartbeatRuns.startedAt, errorCode: heartbeatRuns.errorCode, resultJson: heartbeatRuns.resultJson }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, wake.companyId), or(eq(heartbeatRuns.issueId, issueId), eq(heartbeatRuns.nativeIssueId, issueId),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`), taskAttemptCondition()))
|
|
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
|
|
if (latest?.id !== intent.sourceRunId || !latest.finishedAt) return false;
|
|
if (intent.initiatedBy) {
|
|
const [joined] = await tx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, wake.companyId), eq(agentWakeupRequests.agentId, wake.agentId),
|
|
eq(agentWakeupRequests.reason, "task_retry"), eq(agentWakeupRequests.status, "coalesced"),
|
|
isNull(agentWakeupRequests.runId), eq(agentWakeupRequests.requestedByActorType, "user"),
|
|
eq(agentWakeupRequests.requestedByActorId, intent.actorId),
|
|
sql`${agentWakeupRequests.payload}#>>'{taskRetry,canonicalIntentId}' = ${wake.id}`,
|
|
sql`${agentWakeupRequests.payload}#>'{taskRetry,initiatedBy}' is null`)).limit(1);
|
|
if (!joined) return false;
|
|
}
|
|
const heldBy = latest.status === "cancelled" && !latest.startedAt && latest.errorCode === "execution_reconciliation_required"
|
|
? z.string().uuid().safeParse((latest.resultJson?.executionWait as Record<string, unknown> | undefined)?.recoveryActionId) : null;
|
|
const [blockingAction] = heldBy?.success ? await tx.select({ evidence: issueRecoveryActions.evidence })
|
|
.from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, wake.companyId),
|
|
eq(issueRecoveryActions.sourceIssueId, issueId), eq(issueRecoveryActions.id, heldBy.data))) : [];
|
|
const recoveredRunIds = [intent.sourceRunId, ...(typeof blockingAction?.evidence.runId === "string"
|
|
? [blockingAction.evidence.runId] : [])];
|
|
const recoveries = await tx.select({ evidence: issueRecoveryActions.evidence }).from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.companyId, wake.companyId), eq(issueRecoveryActions.sourceIssueId, issueId),
|
|
inArray(sql<string>`${issueRecoveryActions.evidence}->>'runId'`, recoveredRunIds)));
|
|
if (!recoveries.some(({ evidence }) => {
|
|
const workspace = readWorkspaceRecovery(evidence);
|
|
return readSystemRecovery(evidence)?.state === "complete" && workspace?.status === "recovered" &&
|
|
workspace.runId === evidence.runId && recoveredRunIds.includes(workspace.runId);
|
|
})) return false;
|
|
if (!await taskRetryMutationAllowed(tx as unknown as Db, { companyId: wake.companyId, issueId,
|
|
actorId: intent.actorId, actorSource: intent.actorSource })) return false;
|
|
const [chatReceipt] = await tx.select({ status: agentWakeupRequests.status, runId: agentWakeupRequests.runId })
|
|
.from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, wake.companyId), eq(agentWakeupRequests.id, chat.id)));
|
|
if (chatReceipt && (chatReceipt.runId || !["failed", "skipped", "cancelled"].includes(chatReceipt.status))) return false;
|
|
const [unadmitted] = await tx.select({ id: issues.id }).from(issues).where(and(eq(issues.id, issueId),
|
|
eq(issues.companyId, wake.companyId), unadmittedChatWakeupCondition(issues.id, issues.companyId))).limit(1);
|
|
if (unadmitted) return false;
|
|
const savedInputIds = (await selectTaskRetryInput(tx, { companyId: wake.companyId, issueId, agentId: wake.agentId,
|
|
failedRunId: intent.sourceRunId }))?.sourceActionIds ?? [];
|
|
const stopBoundary = userStopBoundaryAt(latest);
|
|
const savedInputs = savedInputIds.length ? await tx.select({ id: chatActions.id, authorizedAt: issueComments.createdAt })
|
|
.from(chatActions).innerJoin(issueComments, and(eq(issueComments.companyId, chatActions.companyId),
|
|
eq(issueComments.issueId, issueId), sql`${issueComments.id}::text = ${chatActions.payload}->>'commentId'`))
|
|
.where(and(eq(chatActions.companyId, wake.companyId), inArray(chatActions.id, savedInputIds))) : [];
|
|
if (!stopBoundary || savedInputs.length !== savedInputIds.length ||
|
|
savedInputs.some(input => input.authorizedAt > stopBoundary)) return false;
|
|
const { chatActionId: _replaced, initiatedBy: _automatic, ...continuation } = intent;
|
|
await notifyDeferredExecutionWork(tx, { companyId: wake.companyId, requestId: wake.id });
|
|
const [converted] = await tx.update(agentWakeupRequests).set({ updatedAt: new Date(),
|
|
payload: { ...wake.payload, taskRetry: continuation, refusedChatRetry: { chatActionId: chat.id,
|
|
code: typeof chat.result?.code === "string" ? chat.result.code : null } } })
|
|
.where(and(eq(agentWakeupRequests.id, wake.id), eq(agentWakeupRequests.companyId, wake.companyId),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution"), sql`${agentWakeupRequests.runId} is null`,
|
|
sql`${agentWakeupRequests.payload}#>>'{taskRetry,chatActionId}' = ${chat.id}`))
|
|
.returning({ id: agentWakeupRequests.id });
|
|
if (!converted) return false;
|
|
if (savedInputIds.length) await tx.update(agentWakeupRequests).set({ status: "coalesced", finishedAt: new Date(),
|
|
updatedAt: new Date(), payload: sql`coalesce(${agentWakeupRequests.payload}, '{}'::jsonb) || ${JSON.stringify({
|
|
coalescedIntoWakeupRequestId: wake.id })}::jsonb` }).where(and(eq(agentWakeupRequests.companyId, wake.companyId),
|
|
inArray(agentWakeupRequests.id, savedInputIds), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
isNull(agentWakeupRequests.runId)));
|
|
await persistActivity(tx as unknown as Db, { companyId: wake.companyId, actorType: "system", actorId: "task-retry",
|
|
action: "issue.retry_continued_from_saved_work", entityType: "issue", entityId: issueId, runId: intent.sourceRunId,
|
|
details: { requestId: wake.id, sourceRunId: intent.sourceRunId, refusedChatActionId: chat.id,
|
|
...(savedInputIds.length ? { heldInputRequestIds: savedInputIds } : {}) } });
|
|
return true;
|
|
});
|
|
}
|
|
|
|
export async function settleRefusedTaskChatRetry(db: Db, chat: typeof chatActions.$inferSelect) {
|
|
const intentId = chat.payload.taskRetryIntentId;
|
|
if (typeof intentId !== "string") return;
|
|
const unadmitted = and(eq(agentWakeupRequests.companyId, chat.companyId), eq(agentWakeupRequests.id, intentId),
|
|
eq(agentWakeupRequests.reason, "task_retry"), isNull(agentWakeupRequests.runId),
|
|
inArray(agentWakeupRequests.status, ["deferred_issue_execution", "queued", "claimed"]),
|
|
sql`${agentWakeupRequests.payload}#>>'{taskRetry,chatActionId}' = ${chat.id}`);
|
|
const [wake] = await db.select().from(agentWakeupRequests).where(unadmitted);
|
|
const intent = readTaskRetryIntent(wake?.payload);
|
|
if (!wake || intent && await continueRefusedTaskChatRetry(db, wake, intent, chat)) return;
|
|
await db.update(agentWakeupRequests).set({ status: "failed", finishedAt: new Date(), updatedAt: new Date() }).where(unadmitted);
|
|
}
|
|
|
|
/** Chat owns admission and authorization. The task receipt follows that
|
|
* durable outcome; it never launches a parallel generic board execution. */
|
|
export async function reconcileTaskChatRetryReceipts(db: Db, scope: { companyId?: string; requestId?: string; settleOnly?: boolean } = {}) {
|
|
const pending = await db.select().from(agentWakeupRequests).where(and(
|
|
scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined,
|
|
scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined,
|
|
eq(agentWakeupRequests.reason, "task_retry"), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
or(sql`${agentWakeupRequests.payload}#>>'{taskRetry,chatActionId}' is not null`,
|
|
sql`${agentWakeupRequests.payload}#>>'{taskRetry,queuedChatRequestId}' is not null`),
|
|
)).orderBy(agentWakeupRequests.updatedAt).limit(50);
|
|
for (const wake of pending) {
|
|
const intent = readTaskRetryIntent(wake.payload);
|
|
if (!intent || (!intent.chatActionId && !intent.queuedChatRequestId)) continue;
|
|
const actionId = intent.queuedChatRequestId ?? intent.chatActionId!;
|
|
const [chat] = await db.select().from(chatActions).where(and(eq(chatActions.companyId, wake.companyId),
|
|
eq(chatActions.id, actionId), eq(chatActions.kind, intent.queuedChatRequestId ? "inbound_wakeup" : "failed_run_retry"),
|
|
sql`${chatActions.payload}->>'issueId' = ${String(wake.payload?.issueId)}`));
|
|
let [receipt] = chat ? await db.select().from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, wake.companyId), eq(agentWakeupRequests.id, chat.id),
|
|
eq(agentWakeupRequests.agentId, wake.agentId))) : [];
|
|
let inputDenied = false;
|
|
if (intent.queuedChatRequestId && !scope.settleOnly && !receipt?.runId) {
|
|
const issueId = String(wake.payload?.issueId);
|
|
const admission = await db.transaction(async tx => {
|
|
if (intent.generationId !== wake.id || intent.canonicalIntentId ||
|
|
wake.requestedByActorType !== "user" || wake.requestedByActorId !== intent.actorId ||
|
|
!wake.idempotencyKey?.startsWith("task-retry:")) return "denied";
|
|
const [issue] = await tx.select().from(issues).where(and(eq(issues.id, issueId),
|
|
eq(issues.companyId, wake.companyId))).for("update");
|
|
if (!issue || issue.assigneeAgentId !== wake.agentId || ["done", "cancelled"].includes(issue.status)) return "denied";
|
|
if (await retireSupersededTaskRetry(tx as unknown as Db, { companyId: wake.companyId, issueId,
|
|
requestId: wake.id, sourceRunId: intent.sourceRunId })) return "denied";
|
|
if (!await taskRetryMutationAllowed(tx as unknown as Db, { companyId: wake.companyId, issueId,
|
|
actorId: intent.actorId, actorSource: intent.actorSource })) return "denied";
|
|
if (issue.conversationAgentId && issue.conversationUserId !== intent.actorId) return "denied";
|
|
if (adapterExecutionControls.has(intent.sourceRunId) ||
|
|
await hasPendingRunCleanup(tx as unknown as Db, wake.companyId, intent.sourceRunId)) return "waiting";
|
|
return "ready";
|
|
});
|
|
inputDenied = admission === "denied" || !chat || !receipt;
|
|
if (!inputDenied && admission === "ready" && receipt?.status === "deferred_issue_execution") {
|
|
try {
|
|
await resumeDeferredChatWake(db, { companyId: wake.companyId, agentId: wake.agentId, issueId, requestId: actionId });
|
|
} catch (error) {
|
|
if (!(error instanceof PersistedExecutionWaitRefusal)) {
|
|
if (!(error instanceof HttpError) || ![403, 404, 409].includes(error.status)) throw error;
|
|
inputDenied = true;
|
|
}
|
|
}
|
|
[receipt] = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, wake.companyId),
|
|
eq(agentWakeupRequests.id, actionId), eq(agentWakeupRequests.agentId, wake.agentId)));
|
|
}
|
|
inputDenied ||= Boolean(receipt && ["failed", "skipped", "cancelled"].includes(receipt.status));
|
|
}
|
|
// A deferred chat receipt is admitted input, not a completed dispatch.
|
|
// Re-stage that exact action through chat's owner; it rechecks authority
|
|
// and normal queue policy before updating the existing receipt.
|
|
if (!intent.queuedChatRequestId && !scope.settleOnly && chat?.status === "processed" && chat.payload.taskRetryIntentId === wake.id &&
|
|
chat.payload.failedRunId === intent.sourceRunId && receipt?.status === "deferred_issue_execution" &&
|
|
!receipt.runId && receipt.payload?.taskRetryRequestId === wake.id) {
|
|
await db.transaction(async tx => {
|
|
await notifyChatActionWork(tx, "failed_run_retry");
|
|
await tx.update(chatActions).set({ status: "issued", updatedAt: new Date() })
|
|
.where(and(eq(chatActions.id, chat.id), eq(chatActions.companyId, wake.companyId),
|
|
eq(chatActions.status, "processed"), eq(chatActions.updatedAt, chat.updatedAt)));
|
|
});
|
|
}
|
|
if (!intent.queuedChatRequestId && chat && !receipt?.runId && await continueRefusedTaskChatRetry(db, wake, intent, chat)) continue;
|
|
const failed = inputDenied || (chat && ["failed", "cancelled"].includes(chat.status));
|
|
const write = (writer: Pick<Db, "update">) => writer.update(agentWakeupRequests).set({
|
|
...(receipt?.runId ? { status: "coalesced", runId: receipt.runId, finishedAt: new Date() }
|
|
: failed ? { status: "failed", finishedAt: new Date() } : {}),
|
|
updatedAt: new Date(),
|
|
}).where(and(eq(agentWakeupRequests.id, wake.id), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
intent.queuedChatRequestId
|
|
? sql`${agentWakeupRequests.payload}#>>'{taskRetry,queuedChatRequestId}' = ${intent.queuedChatRequestId}`
|
|
: sql`${agentWakeupRequests.payload}#>>'{taskRetry,chatActionId}' = ${intent.chatActionId!}`));
|
|
if (receipt?.runId || failed) await db.transaction(async tx => {
|
|
await notifyStrandedAgentStatus(tx, { companyId: wake.companyId, agentId: wake.agentId });
|
|
await notifyExternalConversationSettlement(tx, { companyId: wake.companyId,
|
|
issueId: typeof wake.payload?.issueId === "string" ? wake.payload.issueId : undefined });
|
|
await write(tx);
|
|
});
|
|
else await write(db);
|
|
}
|
|
}
|