From 4fe34df748be3990825cef61f16edaf5d18a7b2f Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 13 Nov 2026 02:27:22 -0600 Subject: [PATCH] Do not show Retry on a parent with open subtasks while its queued message is settled (#21320) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The task view shows a "Retry" notice when a task needs a person to start it again. > - The product rule is that a task that continues by itself never shows this notice, not even for a few seconds. > - A parent task that waits for its subtasks showed the notice for 3.4 seconds after its run succeeded. > - The cause is the queued-message grace in the task interruption presentation: it does not apply to `blocked` tasks. > - #21200 kept `blocked` out of that grace on purpose, so this pull request adds only the exact observed shape: a parent with open subtasks and a queued task message. > - The benefit is that a parent that waits for its subtasks no longer shows a false failure, and every other blocked task keeps its Retry. ## Linked Issues or Issue Description Refs #21264, #21200 (added the queued-message grace and kept `blocked` out of it) **What happened?** A Codex parent task ran across a two-server release swap. Its run waited for two subtasks and then succeeded with the disposition `blocked`. The task status became `blocked`. A user comment (a deferred wake, status `deferred_issue_execution`) had been queued 13 minutes earlier, while the run was active. For 3.4 seconds after the run finished, the task presentation was `state: "failed"`, `canRetry: true`. Then the release drain cancelled the wake as "Queued messages were already delivered to a completed run", and the presentation changed to `resolved`. The subtasks finished later. Their `issue_children_completed` wake continued the parent, and it completed without help. The cause is in `projectTaskInterruptionState` in `server/src/services/task-interruption-presentation.ts`. The `queuedDuringRun` check shows a succeeded run with user input queued before it finished as `working` for up to 2 minutes. It applies only to `todo`, `in_progress` and `in_review`. For a `blocked` task the presentation fell through to the last line: `failed` with `canRetry: true`. The workspace finalization deferral (`native_workspace_finalization_busy`) in the same window does not change the presentation. The presentation does not read it. **How this differs from what #21200 narrowed** #21200 first applied the grace to every status. Its commit `a3fae9bb67` removed `blocked` because input queued on a blocked task is not promoted by itself, and because a recovery that gives up hands its hold to the board and must end with Retry. This change does not add `blocked` back. It adds one narrow case, and all of these must be true: - The latest run **succeeded**. A failed, interrupted or given-up run is not affected. - The task is `blocked` **and has open subtasks of its own** (not `done` or `cancelled`). Their completion wakes the parent again (`issue_children_completed`). - The pending item is an ordinary **task comment** wake (`commentId` or `wakeCommentIds` in its payload), not a Retry request (`taskRetry.sourceRunId`). - The comment was queued before the run finished, the run finished less than 2 minutes ago, and saved-work recovery is not exhausted. These are the existing bounds. A blocked task that waits for a person or a board decision has no open subtasks, so it keeps Retry as before. **Expected behavior** After the run succeeds, the parent shows `working` with no Retry while its queued message is settled. Then it shows `resolved` (message already delivered) or `working` (message promoted to a new run). If nothing settles the message in 2 minutes, Retry shows. **Steps to reproduce** 1. Assign a parent task that creates subtasks and waits for them. 2. Post a comment on the parent while its run is active. 3. Let the run succeed with the disposition `blocked`. 4. Read the task presentation in the first second after the run finishes. It shows `failed` with Retry. **Paperclip version or commit** master `bbafa78fe2`. The wave that found it ran on `62d673cc3f`. ## What Changed - `server/src/services/task-interruption-presentation.ts`: - The batched wait read also reads whether the wake carries a task comment (`queuedComment`). - One company-scoped read finds which `blocked` tasks with a succeeded latest run and a queued comment have open subtasks (`waitingOnOpenSubtasks`). It runs only for those tasks. - The `queuedDuringRun` grace also applies to such a task. The `todo`, `in_progress` and `in_review` behavior does not change. - `server/src/__tests__/task-interruption-automatic-path.test.ts`, new embedded-Postgres tests: - The exact observed state: a `blocked` parent with two open subtasks, a run that succeeded 1 second ago, a user comment wake queued 13 minutes earlier. Shows `working`, no Retry. - The same parent after the wake is cancelled as already delivered: `resolved`. - The same parent after the wake is promoted to a new run: `working` on the new run. - Guard: the same parent after the 2-minute grace: `failed` with Retry. - Guard: a `blocked` task with a queued comment and no subtasks, or only `done` or `cancelled` subtasks (waiting for a person or a board decision): `failed` with Retry, unchanged. - Guard: a parent with open subtasks whose queued wake carries no comment: `failed` with Retry, unchanged. - Guard: a parent with open subtasks and a queued comment whose run failed and whose recovery hold went to the board: `failed` with Retry. - Guard: a parent with open subtasks and a queued comment whose saved-work recovery is exhausted: `failed` with Retry. - A parent whose run succeeded with no queued message stays `resolved` when its subtasks are open, blocked or cancelled. - The existing #21200 assertion (a `blocked` task with queued input and no subtasks shows Retry) is kept unchanged. ## Verification - `cd server && npx vitest run src/__tests__/task-interruption-automatic-path.test.ts` - Red on master `bbafa78fe2`: "shows a parent whose run just succeeded waiting on its subtasks as working while its queued message is settled" returns `state: "failed"`, `canRetry: true`. This is the value that the wave recorded. The other 15 tests pass on master. - Green with this change: 16 of 16. - A plain `blocked` inclusion fails 3 of the guards: the #21200 assertion, "still offers Retry on a blocked task with a queued message and no open subtasks" and "still offers Retry on a parent with open subtasks when its queued wake carries no message". This shows that the narrow condition is necessary. - `task-interruption-presentation.test.ts`, `task-interruption-query.test.ts`: green (116 tests with the file above). - `cd server && npx tsc --noEmit -p .`: 0 errors. - Not verified: no new chaos wave. The flash was seen once in a loaded release-swap wave. The server test reproduces its saved state. ## Risks - Low risk. The change is presentation only. It does not change wake promotion, dispatch or Retry. - A parent with open subtasks and a stranded queued comment now shows `working` for up to 2 minutes after its run succeeded, and then Retry. Before, it showed Retry at once. - One extra indexed read (`issues_company_parent_idx`) runs only for blocked tasks with a succeeded latest run and a queued comment. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude Code with tool use (shell, file edits, test execution). Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../task-interruption-automatic-path.test.ts | 101 ++++++++++++++++++ .../task-interruption-presentation.ts | 20 +++- 2 files changed, 117 insertions(+), 4 deletions(-) diff --git a/server/src/__tests__/task-interruption-automatic-path.test.ts b/server/src/__tests__/task-interruption-automatic-path.test.ts index 17443fd2171..74f0da150a3 100644 --- a/server/src/__tests__/task-interruption-automatic-path.test.ts +++ b/server/src/__tests__/task-interruption-automatic-path.test.ts @@ -4,6 +4,7 @@ import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { agents, agentWakeupRequests, applyPendingMigrations, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations } from "@paperclipai/db"; import { taskInterruptionPresentation } from "../services/task-interruption-presentation.js"; +import { createSystemRecovery } from "../services/system-recovery-state.js"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; @@ -96,6 +97,106 @@ describe.skipIf(!support.supported)("Retry is offered only when no automatic pat expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ canRetry: true }); }); + async function seedParentWaitingOnSubtasks(childStatus = "in_progress") { + const { companyId, agentId, issue } = await seedTask("blocked"); + for (const title of ["Subtask 1", "Subtask 2"]) + await db.insert(issues).values({ companyId, title, status: childStatus, assigneeAgentId: agentId, parentId: issue.id }); + const runId = randomUUID(), finishedAt = new Date(Date.now() - 1_000); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue", + status: "succeeded", runtimeMode: "native", startedAt: new Date(Date.now() - 25 * 60_000), finishedAt }); + return { companyId, agentId, issue, runId }; + } + + async function seedCommentQueuedDuringRun(companyId: string, agentId: string, issueId: string) { + const commentId = randomUUID(); + const [wake] = await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "automation", + reason: "issue_execution_deferred", triggerDetail: "system", status: "deferred_issue_execution", + requestedByActorType: "user", requestedByActorId: "local-board", requestedAt: new Date(Date.now() - 13 * 60_000), + payload: { issueId, mutation: "comment", commentId, + _paperclipWakeContext: { source: "issue.comment", issueId, wakeReason: "issue_commented", wakeCommentIds: [commentId] } }, + }).returning(); + return wake!; + } + + it("shows a parent whose run just succeeded waiting on its subtasks as working while its queued message is settled", async () => { + const { companyId, agentId, issue, runId } = await seedParentWaitingOnSubtasks(); + await seedCommentQueuedDuringRun(companyId, agentId, issue.id); + expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ responsibility: "system", state: "working", + canRetry: false, observedRunId: runId }); + }); + + it("shows the parent as resolved once its queued message is cancelled as already delivered", async () => { + const { companyId, agentId, issue } = await seedParentWaitingOnSubtasks(); + const wake = await seedCommentQueuedDuringRun(companyId, agentId, issue.id); + await db.update(agentWakeupRequests).set({ status: "cancelled", finishedAt: new Date(), + error: "Queued messages were already delivered to a completed run" }).where(eq(agentWakeupRequests.id, wake.id)); + expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "resolved", canRetry: false }); + }); + + it("shows the parent as working once its queued message is promoted to a new run", async () => { + const { companyId, agentId, issue } = await seedParentWaitingOnSubtasks(); + const wake = await seedCommentQueuedDuringRun(companyId, agentId, issue.id); + const nextRunId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: nextRunId, companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue", + status: "queued", runtimeMode: "native", wakeupRequestId: wake.id }); + await db.update(agentWakeupRequests).set({ status: "queued", runId: nextRunId }).where(eq(agentWakeupRequests.id, wake.id)); + expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "working", canRetry: false, observedRunId: nextRunId }); + }); + + it("offers Retry on the parent once its queued message is not settled within the grace", async () => { + const { companyId, agentId, issue, runId } = await seedParentWaitingOnSubtasks(); + await seedCommentQueuedDuringRun(companyId, agentId, issue.id); + await db.update(heartbeatRuns).set({ finishedAt: new Date(Date.now() - 3 * 60_000) }).where(eq(heartbeatRuns.id, runId)); + expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true }); + }); + + it("still offers Retry on a blocked task with a queued message and no open subtasks", async () => { + for (const childStatus of [null, "done", "cancelled"]) { + const { companyId, agentId, issue } = await seedTask("blocked"); + if (childStatus) await db.insert(issues).values({ companyId, title: "Finished subtask", status: childStatus, + assigneeAgentId: agentId, parentId: issue.id }); + await db.insert(heartbeatRuns).values({ companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue", + status: "succeeded", runtimeMode: "native", finishedAt: new Date(Date.now() - 1_000) }); + await seedCommentQueuedDuringRun(companyId, agentId, issue.id); + expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true }); + } + }); + + it("still offers Retry on a parent with open subtasks when its queued wake carries no message", async () => { + const { companyId, agentId, issue } = await seedParentWaitingOnSubtasks(); + await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "on_demand", reason: "issue_assigned", + status: "deferred_issue_execution", requestedByActorType: "user", requestedAt: new Date(Date.now() - 13 * 60_000), + payload: { issueId: issue.id } }); + expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true }); + }); + + it("still offers Retry on a parent with open subtasks and a queued message when its run failed and recovery gave up", async () => { + const { companyId, agentId, issue, runId, hold } = await seedSwapEndedRun(new Date(Date.now() - 5_000)); + await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, issue.id)); + await db.insert(issues).values({ companyId, title: "Open subtask", status: "in_progress", assigneeAgentId: agentId, parentId: issue.id }); + await seedCommentQueuedDuringRun(companyId, agentId, issue.id); + await db.update(issueRecoveryActions).set({ ownerType: "board" }).where(eq(issueRecoveryActions.id, hold.id)); + const blocked = { ...issue, status: "blocked" }; + expect(await taskInterruptionPresentation(db, blocked)).toMatchObject({ state: "failed", canRetry: true, observedRunId: runId }); + }); + + it("still offers Retry on a parent with open subtasks and a queued message when saved-work recovery is exhausted", async () => { + const { companyId, agentId, issue, runId } = await seedParentWaitingOnSubtasks(); + await seedCommentQueuedDuringRun(companyId, agentId, issue.id); + await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issue.id, kind: "active_run_watchdog", + status: "escalated", ownerType: "system", returnOwnerAgentId: agentId, cause: "native_restart_recovery_blocked", + fingerprint: `exhausted:${runId}`, nextAction: "Retry to continue.", maxAttempts: 3, + evidence: { runId, systemRecovery: { ...createSystemRecovery(runId), state: "exhausted", attempts: 3, retriesUsed: 3 } } }); + expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true }); + }); + + it("does not show a failed attempt on a parent whose run succeeded with no queued message", async () => { + for (const childStatus of ["in_progress", "blocked", "cancelled"]) { + const { issue } = await seedParentWaitingOnSubtasks(childStatus); + expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "resolved", canRetry: false }); + } + }); + it("does not offer Retry on a parent whose run succeeded and that waits for its subtasks", async () => { const { companyId, agentId, issue } = await seedTask(); await db.insert(issues).values({ companyId, title: "Subtask", status: "in_progress", assigneeAgentId: agentId, parentId: issue.id }); diff --git a/server/src/services/task-interruption-presentation.ts b/server/src/services/task-interruption-presentation.ts index 01a72a947c4..d0b9c1dd35f 100644 --- a/server/src/services/task-interruption-presentation.ts +++ b/server/src/services/task-interruption-presentation.ts @@ -1,7 +1,7 @@ import { STORAGE_UNAVAILABLE_WAIT_RETRY_REASON } from "../storage/storage-unavailable.js"; import { AGENT_DIRECTORY_WAIT_RETRY_REASON } from "./execution-recovery-attempt.js"; import { boundSetupCommandFailure, readSetupCommandFailure } from "./workspace-setup-command-failure.js"; -import { and, desc, eq, inArray, isNull, not, sql, type SQL } from "drizzle-orm"; +import { and, desc, eq, inArray, isNull, not, notInArray, sql, type SQL } from "drizzle-orm"; import { unidentifiedLocalLaunchRefusals } from "./native-local-process-stop.js"; import { nativeSessionCleanupBlock } from "../vendor/paperclip-runner/index.js"; import { agents, agentWakeupRequests, approvals, chatActions, companies, environmentLeases, heartbeatRuns, issueApprovals, issueRecoveryActions, @@ -65,7 +65,8 @@ type ProjectionRun = Pick now - 10_000 || queuedDuringRun) && system?.state !== "exhausted") return { ...base, state: "working" }; @@ -496,7 +499,10 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i executionWaitReason: boundedJsonString(sql`to_jsonb(${unpricedUsageWait(sql`${agentWakeupRequests.payload}#>>'{executionWait,reason}'`, sql`${agentWakeupRequests.payload}#>>'{executionWait,message}'`)})`, 80), preparationStatus: chatActions.status, preparationRetryAt: chatActionDeadline("failed_run_retry"), - preparationUpdatedAt: chatActions.updatedAt }) + preparationUpdatedAt: chatActions.updatedAt, + queuedComment: sql`coalesce(jsonb_typeof(${agentWakeupRequests.payload}->'commentId') = 'string' + or jsonb_array_length(case when jsonb_typeof(${agentWakeupRequests.payload}#>'{_paperclipWakeContext,wakeCommentIds}') = 'array' + then ${agentWakeupRequests.payload}#>'{_paperclipWakeContext,wakeCommentIds}' end) > 0, false)` }) .from(agentWakeupRequests).leftJoin(chatActions, and(eq(chatActions.companyId, agentWakeupRequests.companyId), eq(chatActions.id, canonicalExecutionUuid(sql`${agentWakeupRequests.payload}#>>'{taskRetry,chatActionId}'`)), eq(chatActions.kind, "failed_run_retry"), sql`${chatActions.payload}->>'taskRetryIntentId' = ${agentWakeupRequests.id}::text`, @@ -670,6 +676,11 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i for (const [waitingIssueId, row] of readiness) if (row.unresolvedBlockerIssueIds.length) waitingOnBlockers.add(waitingIssueId); } + const parentIds = selected.filter(issue => issue.status === "blocked" && runMap.get(issue.id)?.status === "succeeded" + && currentWaits.some(wait => wait.issueId === issue.id && wait.queuedComment)).map(issue => issue.id); + const waitingOnOpenSubtasks = new Set(parentIds.length ? (await db.selectDistinct({ parentId: issues.parentId }).from(issues) + .where(and(eq(issues.companyId, companyId), inArray(issues.parentId, parentIds), notInArray(issues.status, ["done", "cancelled"])))) + .flatMap(row => row.parentId ? [row.parentId] : []) : []); for (const wait of currentWaits) { const current = waitMap.get(wait.issueId); if (!current || wait.requestedAt > current.requestedAt || @@ -736,6 +747,7 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i admissionWait: admissionWaitMap.get(issue.id), retryDependencies: retryDependencyMap.get(issue.id), waitingOnBlockers: waitingOnBlockers.has(issue.id), + waitingOnOpenSubtasks: waitingOnOpenSubtasks.has(issue.id), cleanupSettling: cleanupSettling.has(issue.id), }, now); result.set(issue.id, presentation);