From c6c7a61e2c361ea995c669461eaa3150da55985e Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 12 Nov 2026 22:58:48 -0600 Subject: [PATCH] fix(retry): release a Retry held on a resolved decision (#21267) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When an agent run fails, the task shows a Retry button. Admission can hold an accepted Retry while a person must answer a pending approval or question (`decision_pending`). > - The hold is saved on the deferred Retry receipt. Nothing cleared that saved reason when the person answered, so the task kept saying "Retry is waiting for a pending approval or question" when nothing was left to answer. > - The deferred-receipt worker drops a refused receipt from its schedule. No producer hinted it when the decision resolved, so the held Retry did not start until some unrelated event rescanned the company. > - A Retry was also held by a confirmation that was already open when the failed turn was admitted. That turn had already run past the confirmation, so the Retry added a gate that the original turn did not have. > - A person's comment on the task started a continuation, but the held Retry receipt stayed deferred next to it. > - This pull request makes a resolved decision release the held Retry at once, keeps the projection from showing a resolved decision as the wait, stops holding a Retry on a decision that predates the failed turn, and settles a held Retry when a comment or answer continues the same failed run. > - The benefit is that Retry always works: a person never waits on a decision that is already resolved. ## Linked Issues or Issue Description **What happened?** A board user pressed Retry on blocked tasks. No run started. The task said "Retry is waiting for a pending approval or question to be resolved. Your message is saved." In one task, the only interaction was a `request_confirmation` that the previous successful run created. The user's later comment started the turn that failed. The confirmation stayed pending, so every Retry attached to the same held receipt. After the user rejected the confirmation and sent a new comment, the task still showed the same wait. No run started until a server restart rescanned the deferred receipts about three minutes later. The task projection said `canRetry: true` with the `decision_pending` wait. **Expected behavior** An accepted Retry is never held on a decision that is already resolved. Accepting or rejecting the decision releases a held Retry right away. A person's comment on the blocked task starts work, and the held Retry is settled. The task projection never shows a resolved decision as the Retry wait. **Steps to reproduce** 1. Let an agent run on a remote runner create a `request_confirmation` for the task. 2. Let the runner be lost so the run fails with `native_continuation_requires_reconciliation`, and let saved work recovery complete. 3. Press Retry. The Retry receipt is saved with `executionWait.reason = decision_pending`. 4. Reject or accept the confirmation. 5. Observe that the receipt keeps `decision_pending`, the task keeps showing the wait, and the Retry does not start until an unrelated rescan. **Paperclip version or commit** Staging Core `57aeda190b`. Fixed on top of `origin/master` `53608bd1d6`. **Deployment mode** Hosted cloud (staging), remote sandbox runner. **Agent adapter(s) involved** `paperclip_runner` (native runtime). ## What Changed - `hasPendingContinuationDecision` (`explicit-native-continuation.ts`) is now the single check for a decision that holds a person's continuation. Admission and the task projection both use it. - For a Retry, a pending interaction that was created before the failed source run was created does not hold the Retry. That turn was already admitted while the decision was open. Decisions that the failed run created (for example a plan review) still hold the Retry. - `releaseDecisionHeldExecution` (`deferred-execution-wait.ts`) removes the saved `decision_pending` wait from deferred receipts on the task and hints each receipt to the deferred-receipt worker. It runs in the resolution transaction of every interaction resolution path (through `notifyChatConfirmationResolution`) and in question answers. - The task projection reads the live decision state. A saved `decision_pending` wait whose decision is resolved is not shown. This also repairs receipts saved before this change. - When a comment or an interaction answer continues a failed run, `retireTaskRetriesContinuedBy` (`task-retry.ts`) settles held Retry receipts for the same source run as `task_retry_superseded`. ## Verification - Red/green: `cd server && npx vitest run src/services/governed-wait-runner-loss.test.ts`. The five new tests fail on `origin/master` sources and pass with this change (17/17 pass): - releases the held Retry as soon as the person chooses accept - releases the held Retry as soon as the person chooses reject - never shows a resolved confirmation as the Retry's wait, and both projections offer the same Retry - a person's comment starts work and settles the held Retry once nothing is left to answer - starts a Retry at once when the confirmation was already open before the failed turn started - The existing tests that keep a Retry or a message on hold while a plan review, governed tool action, secret proposal, or connection authorization is pending still pass. - Related suites: deferred execution wait, explicit native continuation, interaction resolution and service, task interruption presentation and query, task retry, deferred receipt work, interaction routes. ## Risks - Medium-low. A Retry now starts when the only pending interaction is older than the failed run. That turn already ran with the interaction open, so the Retry gets no new authority. Governed tool actions still need their own approval before the tool runs. - Interaction resolution now also updates deferred receipts on the same task. The update is limited to receipts with a saved `decision_pending` wait. A receipt that is still blocked by another pending decision is held again on its next admission. - Issue-linked approvals (`approvals` table) do not yet hint the held receipt when they resolve. The projection no longer shows them as the wait after they resolve. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, browser for read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../chat-confirmation-notifications.ts | 2 + .../src/services/deferred-execution-wait.ts | 21 ++- .../services/explicit-native-continuation.ts | 61 ++++++--- .../governed-wait-runner-loss.test.ts | 127 +++++++++++++++++- server/src/services/issue-question-context.ts | 7 +- .../issue-thread-interactions.test.ts | 1 + .../src/services/issue-thread-interactions.ts | 2 + .../task-interruption-presentation.ts | 8 ++ server/src/services/task-retry.ts | 16 +++ 9 files changed, 223 insertions(+), 22 deletions(-) diff --git a/server/src/services/chat-confirmation-notifications.ts b/server/src/services/chat-confirmation-notifications.ts index 02cdfbee8c8..eef8685ebe9 100644 --- a/server/src/services/chat-confirmation-notifications.ts +++ b/server/src/services/chat-confirmation-notifications.ts @@ -1,5 +1,6 @@ import { notifyMergeConfirmationWork } from "./merged-pr-confirmation-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; +import { releaseDecisionHeldExecution } from "./deferred-execution-wait.js"; export type ChatConfirmationScope = { companyId: string; issueId?: string }; @@ -15,4 +16,5 @@ export async function notifyChatConfirmationResolution(tx: object, interaction: if (["request_confirmation", "request_checkbox_confirmation", "suggest_tasks", "connection_intent"].includes(interaction.kind)) { await notifyChatConfirmationWork(tx, interaction); } + await releaseDecisionHeldExecution(tx, interaction); } diff --git a/server/src/services/deferred-execution-wait.ts b/server/src/services/deferred-execution-wait.ts index 08f974c144a..30cd7422b0e 100644 --- a/server/src/services/deferred-execution-wait.ts +++ b/server/src/services/deferred-execution-wait.ts @@ -1,6 +1,6 @@ import { HttpError } from "../errors.js"; -import { and, eq, sql } from "drizzle-orm"; -import { agentWakeupRequests, type Db } from "@paperclipai/db"; +import { and, eq, inArray, sql } from "drizzle-orm"; +import { agentWakeupRequests, issues, type Db } from "@paperclipai/db"; import { notifyDeferredExecutionWork } from "./deferred-execution-notifications.js"; import { persistActivity, publishActivity, type ActivityPublication } from "./activity-log.js"; @@ -52,3 +52,20 @@ export async function writeDeferredExecutionWait(db: Db, scope: { companyId: str else for (const publication of pending) publishActivity(publication); return saved; } + +export async function releaseDecisionHeldExecution(tx: object, scope: { companyId: string; issueId?: string | null }) { + if (!scope.issueId) return; + const db = tx as Pick; + await db.select({ id: issues.id }).from(issues) + .where(and(eq(issues.companyId, scope.companyId), eq(issues.id, scope.issueId))).for("update"); + const held = and(eq(agentWakeupRequests.companyId, scope.companyId), eq(agentWakeupRequests.status, "deferred_issue_execution"), + sql`${agentWakeupRequests.payload}->>'issueId' = ${scope.issueId}`, + sql`${agentWakeupRequests.payload}#>>'{executionWait,reason}' = 'decision_pending'`); + const receipts = await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(held).for("update"); + if (!receipts.length) return; + for (const { id } of receipts) await notifyDeferredExecutionWork(tx, { companyId: scope.companyId, requestId: id }); + await db.update(agentWakeupRequests).set({ + payload: sql`${agentWakeupRequests.payload} - 'executionWait'`, + updatedAt: new Date(), + }).where(and(held, inArray(agentWakeupRequests.id, receipts.map(receipt => receipt.id)))); +} diff --git a/server/src/services/explicit-native-continuation.ts b/server/src/services/explicit-native-continuation.ts index f3f4bc110f8..37c509ded51 100644 --- a/server/src/services/explicit-native-continuation.ts +++ b/server/src/services/explicit-native-continuation.ts @@ -4,7 +4,7 @@ import { assertDurableChatWakeupRequest, assertDurableChatWakeupReceipt, type Du import { notifyExecutionRunWork } from "./execution-control-notifications.js"; import { canSelectFreshWorkspace, freshReadinessSchema, retainedIssueWorkspaceForRetry, type ContinuationReadiness } from "./fresh-workspace-continuation.js"; import { verifiedWorkspaceSnapshotReceipt } from "./workspace-recovery-snapshot.js"; -import { readTaskRetryIntent } from "./task-retry.js"; +import { readTaskRetryIntent, retireTaskRetriesContinuedBy } from "./task-retry.js"; import { isPreDispatchExecutionWaitVerified } from "./pre-dispatch-execution-wait.js"; import { readSystemRecovery } from "./system-recovery-state.js"; import { hasRequiredWorkspaceRecovery } from "./workspace-restore-recovery-state.js"; @@ -21,7 +21,7 @@ import { hasNativeLocalProcessStop, hasHistoricalSuspendedNativeSession, nativeG import { completeTerminatedLocalNativeSessionCleanup, completeTerminatedRemoteNativeSessionCleanup, nativeSessionCleanupBlock } from "../vendor/paperclip-runner/index.js"; import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt, remoteLeaseCleanupScope } from "./remote-execution-termination.js"; import { z } from "zod"; -import { and, desc, eq, gt, inArray, isNull, ne, not, or, sql } from "drizzle-orm"; +import { and, desc, eq, gt, inArray, isNull, ne, not, or, sql, type SQL } from "drizzle-orm"; import { agents, agentWakeupRequests, approvals, issueApprovals, issueThreadInteractions, environmentLeases, heartbeatRuns, issueComments, issueRecoveryActions, @@ -134,6 +134,45 @@ export async function undeliveredLegacyUserCommentIds( return commentIds.filter(id => valid.has(id) && !firstTurn.has(id) && !historyRead.has(id)); } +function continuationDecisionHolds(companyId: string, wait: { issueId: SQL; actorId: SQL; taskRetrySourceRunId: SQL }) { + return sql`(exists (select 1 from ${issueThreadInteractions} + where ${issueThreadInteractions.companyId} = ${companyId} and ${issueThreadInteractions.issueId} = ${wait.issueId} + and ${issueThreadInteractions.status} = 'pending' + and ${activeIssueInteractionCondition({ responsibleUser: wait.actorId })} + and (${wait.taskRetrySourceRunId} is null or (not (${issueThreadInteractions.kind} in ('ask_user_questions', 'connection_intent') + and not (${issueThreadInteractions.payload} ?| array['toolAction', 'secretProposal', 'connectionAuthorization']) + and coalesce(${issueThreadInteractions.payload} #>> '{target,type}', '') <> 'issue_document') + and not exists (select 1 from ${heartbeatRuns} admitted + where admitted.company_id = ${companyId} and admitted.id::text = ${wait.taskRetrySourceRunId} + and admitted.created_at > ${issueThreadInteractions.createdAt})))) + or exists (select 1 from ${issueApprovals} inner join ${approvals} + on ${approvals.id} = ${issueApprovals.approvalId} and ${approvals.companyId} = ${companyId} + where ${issueApprovals.companyId} = ${companyId} and ${issueApprovals.issueId} = ${wait.issueId} + and ${approvals.status} in ('pending', 'revision_requested')))`; +} + +export async function hasPendingContinuationDecision(db: Db, input: { + companyId: string; issueId: string; actorId: string; taskRetrySourceRunId?: string | null; +}): Promise { + const [row] = await db.execute<{ holds: boolean }>(sql`select ${continuationDecisionHolds(input.companyId, { + issueId: sql`${input.issueId}::uuid`, actorId: sql`${input.actorId}::text`, + taskRetrySourceRunId: sql`${input.taskRetrySourceRunId ?? null}::text`, + })} as holds`); + return row?.holds === true; +} + +export async function pendingContinuationDecisionWaits(db: Db, companyId: string, + waits: Array<{ id: string; issueId: string; actorId: string; taskRetrySourceRunId: string | null }>): Promise> { + if (!waits.length) return new Set(); + const rows = await db.execute<{ id: string }>(sql`select held.id from unnest( + ${sql.param(waits.map(wait => wait.id))}::text[], ${sql.param(waits.map(wait => wait.issueId))}::uuid[], + ${sql.param(waits.map(wait => wait.actorId))}::text[], ${sql.param(waits.map(wait => wait.taskRetrySourceRunId))}::text[]) + as held(id, issue_id, actor_id, source_run_id) + where ${continuationDecisionHolds(companyId, { issueId: sql`held.issue_id`, actorId: sql`held.actor_id`, + taskRetrySourceRunId: sql`held.source_run_id` })}`); + return new Set(rows.map(row => row.id)); +} + /** Called under the issue lock, in the transaction that creates the new turn. * A user request authorizes a new conversation, not replay of the failed run. * Unknown action outcomes and all prior records stay intact. @@ -315,19 +354,9 @@ export async function admitExplicitNativeContinuation(input: { } if (!sourceChecks.length && !nativeQueuedSend) return null; if (blocker && blocker.recoveryActionId === null) return null; - const [pendingInteraction] = await db.select({ id: issueThreadInteractions.id }).from(issueThreadInteractions).where(and( - eq(issueThreadInteractions.companyId, companyId), eq(issueThreadInteractions.issueId, issueId), - eq(issueThreadInteractions.status, "pending"), - activeIssueInteractionCondition({ responsibleUserId: actorId }), - taskRetry ? sql`not (${issueThreadInteractions.kind} in ('ask_user_questions', 'connection_intent') - and not (${issueThreadInteractions.payload} ?| array['toolAction', 'secretProposal', 'connectionAuthorization']) - and coalesce(${issueThreadInteractions.payload} #>> '{target,type}', '') <> 'issue_document')` : undefined, - )).limit(1); - const [pendingApproval] = await db.select({ id: approvals.id }).from(issueApprovals).innerJoin(approvals, and( - eq(approvals.id, issueApprovals.approvalId), eq(approvals.companyId, companyId), - )).where(and(eq(issueApprovals.companyId, companyId), eq(issueApprovals.issueId, issueId), - inArray(approvals.status, ["pending", "revision_requested"]))).limit(1); - if (pendingInteraction || pendingApproval) return blocked("decision_pending", "A pending approval or question must be resolved before this message can start."); + if (await hasPendingContinuationDecision(db, { companyId, issueId, actorId, + taskRetrySourceRunId: taskRetry?.sourceRunId ?? null })) + return blocked("decision_pending", "A pending approval or question must be resolved before this message can start."); if (!sourceChecks.length) return null; const sources: Run[] = []; @@ -704,6 +733,8 @@ export async function admitExplicitNativeContinuation(input: { details: { continuation: retry ? "explicit_user_retry" : "explicit_user_message", ...authorization, recoveryActionIds: actions.map(action => action.id), previousRunIds: sources.map(run => run.id) }, }); + if (!input.taskRetryRequestId) await retireTaskRetriesContinuedBy(db, { companyId, issueId, agentId, + sourceRunIds: sources.map(run => run.id) }); if (readiness) await persistActivity(db, { companyId, actorType: "system", actorId: "task-recovery", action: readiness.mode === "reuse_issue_workspace" ? "issue.retry_workspace_reused" : "issue.fresh_workspace_selected", entityType: "issue", entityId: issueId, details: { ...readiness, runId: input.successorRunId } }); diff --git a/server/src/services/governed-wait-runner-loss.test.ts b/server/src/services/governed-wait-runner-loss.test.ts index ba0277a7a3c..796c46b82ad 100644 --- a/server/src/services/governed-wait-runner-loss.test.ts +++ b/server/src/services/governed-wait-runner-loss.test.ts @@ -9,11 +9,13 @@ import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js"; import { heartbeatService } from "./heartbeat.js"; import { requestInterruptedNativeTurnRetry } from "./heartbeat/restart-interrupted-turn.js"; import { getExecutionBlocker } from "./execution-blocker.js"; -import { taskInterruptionPresentation } from "./task-interruption-presentation.js"; +import { taskInterruptionPresentation, taskInterruptionPresentations } from "./task-interruption-presentation.js"; +import { pendingContinuationDecisionWaits } from "./explicit-native-continuation.js"; import { readSystemRecovery } from "./system-recovery-state.js"; import { reconcileRetainedWorkspaces } from "./workspace-recovery.js"; -import { requestTaskRetry } from "./task-retry.js"; +import { readTaskRetryIntent, requestTaskRetry } from "./task-retry.js"; import { buildExecutionContinuation } from "./execution-continuation.js"; +import { issueThreadInteractionService } from "./issue-thread-interactions.js"; import { useWorkspaceRecoveryFixture, action, db, run, seed } from "./__tests__/workspace-recovery-fixture.js"; const targets = vi.hoisted(() => new Map()); @@ -204,4 +206,125 @@ describe("a held task with a pending decision after its remote runner was lost", expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId)))[0].evidence) .not.toHaveProperty("explicitUserContinuation"); }, 60_000); + describe("a Retry held on a confirmation the person resolves", () => { + const heldOnConfirmation = async () => { + const f = await lostRunnerWithPendingDecision("parkedQuestion"); + await db.update(issueThreadInteractions).set({ kind: "request_confirmation", + payload: { version: 1, prompt: "Review sidebar subtask screenshots" } as never }) + .where(eq(issueThreadInteractions.id, f.decision.id)); + const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: "local-board", + actorSource: "local_implicit", idempotencyKey: randomUUID(), observedRunId: f.runId }); + await f.service.resumeTaskRetry(f.companyId, receipt.requestId!); + expect({ successors: (await queuedSuccessors(f.companyId)).length, deferred: await deferredWaits(f.companyId) }) + .toEqual({ successors: 0, deferred: ["decision_pending"] }); + return { ...f, requestId: receipt.requestId! }; + }; + const savedResolution = (f: Awaited>) => + db.update(issueThreadInteractions).set({ status: "rejected", resolvedByUserId: "local-board", resolvedAt: new Date(), + result: { version: 1, outcome: "rejected", reason: "what" } as never }) + .where(eq(issueThreadInteractions.id, f.decision.id)); + const currentIssue = async (issueId: string) => (await db.select().from(issues).where(eq(issues.id, issueId)))[0]; + const retryReceipt = async (id: string) => (await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, id)))[0]; + + it.each(["accept", "reject"] as const)("releases the held Retry as soon as the person chooses %s", async choice => { + const f = await heldOnConfirmation(); + const interactions = issueThreadInteractionService(db); + const issue = await currentIssue(f.issueId); + if (choice === "accept") await interactions.acceptInteraction(issue, f.decision.id, {}, { userId: "local-board" }); + else await interactions.rejectInteraction(issue, f.decision.id, { reason: "what" }, { userId: "local-board" }); + expect((await retryReceipt(f.requestId)).payload).not.toHaveProperty("executionWait"); + expect(await deferredWaits(f.companyId)).not.toContain("decision_pending"); + await f.service.resumeTaskRetry(f.companyId, f.requestId); + const [successor] = await queuedSuccessors(f.companyId); + expect(successor?.contextSnapshot).toMatchObject({ previousRunId: f.runId, taskRetryRequestId: f.requestId }); + }, 60_000); + + it("accepts the confirmation while the held Retry is admitted, without either aborting", async () => { + const f = await heldOnConfirmation(); + const issue = await currentIssue(f.issueId); + const outcomes = await Promise.allSettled([ + issueThreadInteractionService(db).acceptInteraction(issue, f.decision.id, {}, { userId: "local-board" }), + f.service.resumeTaskRetry(f.companyId, f.requestId), + f.service.resumeTaskRetry(f.companyId, f.requestId), + ]); + expect(outcomes.filter(outcome => outcome.status === "rejected")).toEqual([]); + await f.service.resumeTaskRetry(f.companyId, f.requestId); + expect({ successors: (await queuedSuccessors(f.companyId)).length, deferred: await deferredWaits(f.companyId) }) + .toEqual({ successors: 1, deferred: [] }); + }, 60_000); + + it("never shows a resolved confirmation as the Retry's wait, and both projections offer the same Retry", async () => { + const f = await heldOnConfirmation(); + await savedResolution(f); + await db.update(agentWakeupRequests).set({ requestedAt: new Date(Date.now() - 5 * 60_000) }) + .where(eq(agentWakeupRequests.id, f.requestId)); + const interruption = await taskInterruptionPresentation(db, await currentIssue(f.issueId)); + const blocker = await getExecutionBlocker(db, f.companyId, f.issueId); + expect(interruption).toMatchObject({ state: "failed", canRetry: true }); + expect(interruption?.retryWait).toBeUndefined(); + expect(blocker?.canRetry).toBe(interruption?.canRetry); + }, 60_000); + + it("a person's comment starts work and settles the held Retry once nothing is left to answer", async () => { + const f = await heldOnConfirmation(); + await savedResolution(f); + const [comment] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId, + authorType: "user", authorUserId: "local-board", body: "answer me" }).returning(); + await f.service.wakeup(f.agentId, { + source: "automation", triggerDetail: "system", reason: "issue_commented", + payload: { issueId: f.issueId, commentId: comment.id, mutation: "comment" }, + requestedByActorType: "user", requestedByActorId: "local-board", + contextSnapshot: { issueId: f.issueId, taskId: f.issueId, wakeReason: "issue_commented", wakeCommentId: comment.id }, + }); + expect({ successors: (await queuedSuccessors(f.companyId)).length, deferred: await deferredWaits(f.companyId) }) + .toEqual({ successors: 1, deferred: [] }); + expect(await retryReceipt(f.requestId)).toMatchObject({ status: "skipped", error: "task_retry_superseded" }); + }, 60_000); + + it("starts a Retry at once when the confirmation was already open before the failed turn started", async () => { + const f = await lostRunnerWithPendingDecision("parkedQuestion"); + const failed = await run(f.runId); + await db.update(issueThreadInteractions).set({ kind: "request_confirmation", + payload: { version: 1, prompt: "Review sidebar subtask screenshots" } as never, + createdAt: new Date(failed.createdAt.getTime() - 6 * 60_000) }) + .where(eq(issueThreadInteractions.id, f.decision.id)); + const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: "local-board", + actorSource: "local_implicit", idempotencyKey: randomUUID(), observedRunId: f.runId }); + await f.service.resumeTaskRetry(f.companyId, receipt.requestId!); + expect({ successors: (await queuedSuccessors(f.companyId)).length, deferred: await deferredWaits(f.companyId) }) + .toEqual({ successors: 1, deferred: [] }); + }, 60_000); + + it("reads every saved decision wait on a task list in one bounded query", async () => { + const f = await heldOnConfirmation(); + const source = await currentIssue(f.issueId); + const listed = await db.insert(issues).values(Array.from({ length: 40 }, (_, index) => ({ + ...source, id: randomUUID(), identifier: `${source.identifier}-${index}`, issueNumber: (source.issueNumber ?? 0) + 1000 + index, + executionRunId: null, checkoutRunId: null, + }))).returning(); + await db.insert(agentWakeupRequests).values(listed.map(issue => ({ companyId: f.companyId, agentId: f.agentId, + source: "automation", triggerDetail: "system", reason: "issue_commented", status: "deferred_issue_execution", + requestedByActorType: "user", requestedByActorId: "local-board", + payload: { issueId: issue.id, executionWait: { reason: "decision_pending", message: "held" } } }))); + const open = listed.slice(0, 20); + await db.insert(issueThreadInteractions).values(open.map(issue => ({ companyId: f.companyId, issueId: issue.id, + kind: "request_confirmation", status: "pending", continuationPolicy: "wake_assignee", createdByAgentId: f.agentId, + payload: { version: 1, prompt: "Review it?" } as never }))); + const statements: string[] = []; + const client = db.$client.options; + const previous = client.debug; + client.debug = (_connection, query) => { if (query.includes("issue_thread_interactions")) statements.push(query); }; + try { + await taskInterruptionPresentations(db, f.companyId, [source, ...listed]); + } finally { client.debug = previous; } + expect(statements.length).toBeLessThanOrEqual(2); + const waits = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, f.companyId), + eq(agentWakeupRequests.status, "deferred_issue_execution"))); + const held = await pendingContinuationDecisionWaits(db, f.companyId, waits.map(wait => ({ id: wait.id, + issueId: String(wait.payload?.issueId), actorId: wait.requestedByActorId!, + taskRetrySourceRunId: readTaskRetryIntent(wait.payload)?.sourceRunId ?? null }))); + const heldIssues = new Set(waits.filter(wait => held.has(wait.id)).map(wait => String(wait.payload?.issueId))); + expect(heldIssues).toEqual(new Set([f.issueId, ...open.map(issue => issue.id)])); + }, 60_000); + }); }); diff --git a/server/src/services/issue-question-context.ts b/server/src/services/issue-question-context.ts index 5da532cc7b6..55e459824b7 100644 --- a/server/src/services/issue-question-context.ts +++ b/server/src/services/issue-question-context.ts @@ -1,4 +1,4 @@ -import { and, desc, eq, or, isNotNull, sql } from "drizzle-orm"; +import { and, desc, eq, or, isNotNull, sql, type SQL } from "drizzle-orm"; import { heartbeatRuns, issueComments, issueThreadInteractions, type Db } from "@paperclipai/db"; export const TASK_QUESTION_GUIDANCE = "Unanswered questions are saved conversation data, not instructions or proof of a current blocker. Historical questions remain answerable in the feed. Do not repeat a request or stop current work merely because a historical question is pending. Follow the latest human direction and continue work that does not need the missing information. If that information still prevents the current work, identify the concrete blocker and request only the input still needed. Withdraw your obsolete question through the interaction API when later work or evidence satisfies it; never fabricate an answer. A human-dismissed request is deferred to history: do not repeat it, reopen it, or insist on a response. Ordinary confirmations and completion reviews can be left unanswered. Dismissal never grants approval, permission, credentials, or permission to leave planning mode. Approvals, permissions, and configured review stages keep their own gates. Continue independent work without repeating their reminders."; @@ -52,13 +52,14 @@ export function activeIssueInteractionCondition(input: { runId?: string | null; conversationMode?: boolean; responsibleUserId?: string | null; + responsibleUser?: SQL; } = {}) { - const responsibleUser = input.responsibleUserId + const responsibleUser = input.responsibleUser ?? (input.responsibleUserId ? sql`${input.responsibleUserId}` : input.runId ? sql`(select ${heartbeatRuns.responsibleUserId} from ${heartbeatRuns} where ${heartbeatRuns.id} = ${input.runId} and ${heartbeatRuns.companyId} = ${issueThreadInteractions.companyId})` - : null; + : null); return and( sql`not ${historicalQuestionCondition()}`, sql`not ${deferredInteractionCondition()}`, diff --git a/server/src/services/issue-thread-interactions.test.ts b/server/src/services/issue-thread-interactions.test.ts index 988a89def4a..f57ba880edc 100644 --- a/server/src/services/issue-thread-interactions.test.ts +++ b/server/src/services/issue-thread-interactions.test.ts @@ -68,6 +68,7 @@ function createFakeDb(args: { selectCallCount += 1; const rows = selectCallCount === 1 ? [interactionRow] : (args.parentRows ?? []); return createSelectChain(table => { + if (getTableName(table as never) === "agent_wakeup_requests") return []; if (getTableName(table as never) === "issues") return [{ id: interactionRow.issueId, companyId: interactionRow.companyId, status: "in_progress", assigneeAgentId: interactionRow.createdByAgentId, conversationAgentId: null, diff --git a/server/src/services/issue-thread-interactions.ts b/server/src/services/issue-thread-interactions.ts index 6472672859b..05bf3018c14 100644 --- a/server/src/services/issue-thread-interactions.ts +++ b/server/src/services/issue-thread-interactions.ts @@ -7,6 +7,7 @@ import { notifyMergeConfirmationWork, type MergeConfirmationScope } from "./merg import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { notifyIssueReviewState } from "./issue-review-state-notifications.js"; import { notifyChatConfirmationResolution } from "./chat-confirmation-notifications.js"; +import { releaseDecisionHeldExecution } from "./deferred-execution-wait.js"; import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js"; import { notifyToolReviewChange } from "./tool-review-work-notifications.js"; import { projectGitHubInteractionErrors } from "./tool-access/github-action-error-projection.js"; @@ -5685,6 +5686,7 @@ export function issueThreadInteractionService( } await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor, [JSON.stringify(input)]); await notifyChatRunMilestones(tx, current); + await releaseDecisionHeldExecution(tx, current); await mutationOptions.beforeResolveInTransaction?.(tx); const resolvedAt = new Date(); const [row] = await tx diff --git a/server/src/services/task-interruption-presentation.ts b/server/src/services/task-interruption-presentation.ts index 19e933cd2eb..f941f789b7c 100644 --- a/server/src/services/task-interruption-presentation.ts +++ b/server/src/services/task-interruption-presentation.ts @@ -489,6 +489,7 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i // Keep at most two candidates per task: latest Retry and latest other input. // A new message must not hide an existing Retry's admission explanation. db.selectDistinctOn([wakeIssue, isRetryWait], { issueId: wakeIssue, id: agentWakeupRequests.id, requestedAt: agentWakeupRequests.requestedAt, + requestedByActorId: agentWakeupRequests.requestedByActorId, sourceRunId: boundedJsonString(sql`${agentWakeupRequests.payload}#>'{taskRetry,sourceRunId}'`, 160), executionWaitReason: boundedJsonString(sql`to_jsonb(${unpricedUsageWait(sql`${agentWakeupRequests.payload}#>>'{executionWait,reason}'`, sql`${agentWakeupRequests.payload}#>>'{executionWait,message}'`)})`, 80), @@ -627,6 +628,13 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i // that authority, even before dispatch retires the old receipt. Retained // ordinary input is independent and must still be shown as waiting. const currentWaits = waits.filter(wait => !wait.sourceRunId || wait.sourceRunId === runMap.get(wait.issueId)?.id); + const decisionWaits = currentWaits.filter(wait => wait.executionWaitReason === "decision_pending"); + if (decisionWaits.length) { + const { pendingContinuationDecisionWaits } = await import("./explicit-native-continuation.js"); + const held = await pendingContinuationDecisionWaits(db, companyId, decisionWaits.flatMap(wait => wait.requestedByActorId + ? [{ id: wait.id, issueId: wait.issueId, actorId: wait.requestedByActorId, taskRetrySourceRunId: wait.sourceRunId }] : [])); + for (const wait of decisionWaits) if (!held.has(wait.id)) wait.executionWaitReason = null; + } const retryWaitMap = byIssue(currentWaits.filter(wait => wait.sourceRunId)); const retryIssueIds = [...retryWaitMap.keys()]; const retryDependencyMap = new Map(); diff --git a/server/src/services/task-retry.ts b/server/src/services/task-retry.ts index 94cd50580e8..8d76021d72d 100644 --- a/server/src/services/task-retry.ts +++ b/server/src/services/task-retry.ts @@ -225,6 +225,22 @@ export async function retireSupersededTaskRetry(db: Db, input: { return retired.length > 0; } +export async function retireTaskRetriesContinuedBy(db: Db, input: { + companyId: string; issueId: string; agentId: string; sourceRunIds: string[]; +}) { + if (!input.sourceRunIds.length) return; + const held = and(eq(agentWakeupRequests.companyId, input.companyId), eq(agentWakeupRequests.agentId, input.agentId), + eq(agentWakeupRequests.reason, "task_retry"), eq(agentWakeupRequests.status, "deferred_issue_execution"), + isNull(agentWakeupRequests.runId), sql`${agentWakeupRequests.payload}->>'issueId' = ${input.issueId}`, + inArray(sql`${agentWakeupRequests.payload}#>>'{taskRetry,sourceRunId}'`, input.sourceRunIds)); + const [heldRetry] = await db.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(held).limit(1); + if (!heldRetry) return; + await notifyStrandedAgentStatus(db, { companyId: input.companyId, agentId: input.agentId }); + await notifyExternalConversationSettlement(db, { companyId: input.companyId, issueId: input.issueId }); + await db.update(agentWakeupRequests).set({ status: "skipped", error: "task_retry_superseded", + finishedAt: new Date(), updatedAt: new Date() }).where(held); +} + type TaskRetryRequest = { db: Db; companyId: string; issueId: string; actorId: string; idempotencyKey: string; observedRunId: string | null; actorSource?: string;