mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
Do not show Retry on a parent with open subtasks while its queued message is settled (#21320)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The task view shows a "Retry" notice when a task needs a person to start it again. > - The product rule is that a task that continues by itself never shows this notice, not even for a few seconds. > - A parent task that waits for its subtasks showed the notice for 3.4 seconds after its run succeeded. > - The cause is the queued-message grace in the task interruption presentation: it does not apply to `blocked` tasks. > - #21200 kept `blocked` out of that grace on purpose, so this pull request adds only the exact observed shape: a parent with open subtasks and a queued task message. > - The benefit is that a parent that waits for its subtasks no longer shows a false failure, and every other blocked task keeps its Retry. ## Linked Issues or Issue Description Refs #21264, #21200 (added the queued-message grace and kept `blocked` out of it) **What happened?** A Codex parent task ran across a two-server release swap. Its run waited for two subtasks and then succeeded with the disposition `blocked`. The task status became `blocked`. A user comment (a deferred wake, status `deferred_issue_execution`) had been queued 13 minutes earlier, while the run was active. For 3.4 seconds after the run finished, the task presentation was `state: "failed"`, `canRetry: true`. Then the release drain cancelled the wake as "Queued messages were already delivered to a completed run", and the presentation changed to `resolved`. The subtasks finished later. Their `issue_children_completed` wake continued the parent, and it completed without help. The cause is in `projectTaskInterruptionState` in `server/src/services/task-interruption-presentation.ts`. The `queuedDuringRun` check shows a succeeded run with user input queued before it finished as `working` for up to 2 minutes. It applies only to `todo`, `in_progress` and `in_review`. For a `blocked` task the presentation fell through to the last line: `failed` with `canRetry: true`. The workspace finalization deferral (`native_workspace_finalization_busy`) in the same window does not change the presentation. The presentation does not read it. **How this differs from what #21200 narrowed** #21200 first applied the grace to every status. Its commit `a3fae9bb67` removed `blocked` because input queued on a blocked task is not promoted by itself, and because a recovery that gives up hands its hold to the board and must end with Retry. This change does not add `blocked` back. It adds one narrow case, and all of these must be true: - The latest run **succeeded**. A failed, interrupted or given-up run is not affected. - The task is `blocked` **and has open subtasks of its own** (not `done` or `cancelled`). Their completion wakes the parent again (`issue_children_completed`). - The pending item is an ordinary **task comment** wake (`commentId` or `wakeCommentIds` in its payload), not a Retry request (`taskRetry.sourceRunId`). - The comment was queued before the run finished, the run finished less than 2 minutes ago, and saved-work recovery is not exhausted. These are the existing bounds. A blocked task that waits for a person or a board decision has no open subtasks, so it keeps Retry as before. **Expected behavior** After the run succeeds, the parent shows `working` with no Retry while its queued message is settled. Then it shows `resolved` (message already delivered) or `working` (message promoted to a new run). If nothing settles the message in 2 minutes, Retry shows. **Steps to reproduce** 1. Assign a parent task that creates subtasks and waits for them. 2. Post a comment on the parent while its run is active. 3. Let the run succeed with the disposition `blocked`. 4. Read the task presentation in the first second after the run finishes. It shows `failed` with Retry. **Paperclip version or commit** master `bbafa78fe2`. The wave that found it ran on `62d673cc3f`. ## What Changed - `server/src/services/task-interruption-presentation.ts`: - The batched wait read also reads whether the wake carries a task comment (`queuedComment`). - One company-scoped read finds which `blocked` tasks with a succeeded latest run and a queued comment have open subtasks (`waitingOnOpenSubtasks`). It runs only for those tasks. - The `queuedDuringRun` grace also applies to such a task. The `todo`, `in_progress` and `in_review` behavior does not change. - `server/src/__tests__/task-interruption-automatic-path.test.ts`, new embedded-Postgres tests: - The exact observed state: a `blocked` parent with two open subtasks, a run that succeeded 1 second ago, a user comment wake queued 13 minutes earlier. Shows `working`, no Retry. - The same parent after the wake is cancelled as already delivered: `resolved`. - The same parent after the wake is promoted to a new run: `working` on the new run. - Guard: the same parent after the 2-minute grace: `failed` with Retry. - Guard: a `blocked` task with a queued comment and no subtasks, or only `done` or `cancelled` subtasks (waiting for a person or a board decision): `failed` with Retry, unchanged. - Guard: a parent with open subtasks whose queued wake carries no comment: `failed` with Retry, unchanged. - Guard: a parent with open subtasks and a queued comment whose run failed and whose recovery hold went to the board: `failed` with Retry. - Guard: a parent with open subtasks and a queued comment whose saved-work recovery is exhausted: `failed` with Retry. - A parent whose run succeeded with no queued message stays `resolved` when its subtasks are open, blocked or cancelled. - The existing #21200 assertion (a `blocked` task with queued input and no subtasks shows Retry) is kept unchanged. ## Verification - `cd server && npx vitest run src/__tests__/task-interruption-automatic-path.test.ts` - Red on master `bbafa78fe2`: "shows a parent whose run just succeeded waiting on its subtasks as working while its queued message is settled" returns `state: "failed"`, `canRetry: true`. This is the value that the wave recorded. The other 15 tests pass on master. - Green with this change: 16 of 16. - A plain `blocked` inclusion fails 3 of the guards: the #21200 assertion, "still offers Retry on a blocked task with a queued message and no open subtasks" and "still offers Retry on a parent with open subtasks when its queued wake carries no message". This shows that the narrow condition is necessary. - `task-interruption-presentation.test.ts`, `task-interruption-query.test.ts`: green (116 tests with the file above). - `cd server && npx tsc --noEmit -p .`: 0 errors. - Not verified: no new chaos wave. The flash was seen once in a loaded release-swap wave. The server test reproduces its saved state. ## Risks - Low risk. The change is presentation only. It does not change wake promotion, dispatch or Retry. - A parent with open subtasks and a stranded queued comment now shows `working` for up to 2 minutes after its run succeeded, and then Retry. Before, it showed Retry at once. - One extra indexed read (`issues_company_parent_idx`) runs only for blocked tasks with a succeeded latest run and a queued comment. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude Code with tool use (shell, file edits, test execution). Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d9a4bb23fe
commit
4fe34df748
2 files changed
+117
-4
No files matched your search
@@ -4,6 +4,7 @@ import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { agents, agentWakeupRequests, applyPendingMigrations, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS,
|
||||
heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations } from "@paperclipai/db";
|
||||
import { taskInterruptionPresentation } from "../services/task-interruption-presentation.js";
|
||||
import { createSystemRecovery } from "../services/system-recovery-state.js";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
|
||||
@@ -96,6 +97,106 @@ describe.skipIf(!support.supported)("Retry is offered only when no automatic pat
|
||||
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ canRetry: true });
|
||||
});
|
||||
|
||||
async function seedParentWaitingOnSubtasks(childStatus = "in_progress") {
|
||||
const { companyId, agentId, issue } = await seedTask("blocked");
|
||||
for (const title of ["Subtask 1", "Subtask 2"])
|
||||
await db.insert(issues).values({ companyId, title, status: childStatus, assigneeAgentId: agentId, parentId: issue.id });
|
||||
const runId = randomUUID(), finishedAt = new Date(Date.now() - 1_000);
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue",
|
||||
status: "succeeded", runtimeMode: "native", startedAt: new Date(Date.now() - 25 * 60_000), finishedAt });
|
||||
return { companyId, agentId, issue, runId };
|
||||
}
|
||||
|
||||
async function seedCommentQueuedDuringRun(companyId: string, agentId: string, issueId: string) {
|
||||
const commentId = randomUUID();
|
||||
const [wake] = await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "automation",
|
||||
reason: "issue_execution_deferred", triggerDetail: "system", status: "deferred_issue_execution",
|
||||
requestedByActorType: "user", requestedByActorId: "local-board", requestedAt: new Date(Date.now() - 13 * 60_000),
|
||||
payload: { issueId, mutation: "comment", commentId,
|
||||
_paperclipWakeContext: { source: "issue.comment", issueId, wakeReason: "issue_commented", wakeCommentIds: [commentId] } },
|
||||
}).returning();
|
||||
return wake!;
|
||||
}
|
||||
|
||||
it("shows a parent whose run just succeeded waiting on its subtasks as working while its queued message is settled", async () => {
|
||||
const { companyId, agentId, issue, runId } = await seedParentWaitingOnSubtasks();
|
||||
await seedCommentQueuedDuringRun(companyId, agentId, issue.id);
|
||||
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ responsibility: "system", state: "working",
|
||||
canRetry: false, observedRunId: runId });
|
||||
});
|
||||
|
||||
it("shows the parent as resolved once its queued message is cancelled as already delivered", async () => {
|
||||
const { companyId, agentId, issue } = await seedParentWaitingOnSubtasks();
|
||||
const wake = await seedCommentQueuedDuringRun(companyId, agentId, issue.id);
|
||||
await db.update(agentWakeupRequests).set({ status: "cancelled", finishedAt: new Date(),
|
||||
error: "Queued messages were already delivered to a completed run" }).where(eq(agentWakeupRequests.id, wake.id));
|
||||
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "resolved", canRetry: false });
|
||||
});
|
||||
|
||||
it("shows the parent as working once its queued message is promoted to a new run", async () => {
|
||||
const { companyId, agentId, issue } = await seedParentWaitingOnSubtasks();
|
||||
const wake = await seedCommentQueuedDuringRun(companyId, agentId, issue.id);
|
||||
const nextRunId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: nextRunId, companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue",
|
||||
status: "queued", runtimeMode: "native", wakeupRequestId: wake.id });
|
||||
await db.update(agentWakeupRequests).set({ status: "queued", runId: nextRunId }).where(eq(agentWakeupRequests.id, wake.id));
|
||||
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "working", canRetry: false, observedRunId: nextRunId });
|
||||
});
|
||||
|
||||
it("offers Retry on the parent once its queued message is not settled within the grace", async () => {
|
||||
const { companyId, agentId, issue, runId } = await seedParentWaitingOnSubtasks();
|
||||
await seedCommentQueuedDuringRun(companyId, agentId, issue.id);
|
||||
await db.update(heartbeatRuns).set({ finishedAt: new Date(Date.now() - 3 * 60_000) }).where(eq(heartbeatRuns.id, runId));
|
||||
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true });
|
||||
});
|
||||
|
||||
it("still offers Retry on a blocked task with a queued message and no open subtasks", async () => {
|
||||
for (const childStatus of [null, "done", "cancelled"]) {
|
||||
const { companyId, agentId, issue } = await seedTask("blocked");
|
||||
if (childStatus) await db.insert(issues).values({ companyId, title: "Finished subtask", status: childStatus,
|
||||
assigneeAgentId: agentId, parentId: issue.id });
|
||||
await db.insert(heartbeatRuns).values({ companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue",
|
||||
status: "succeeded", runtimeMode: "native", finishedAt: new Date(Date.now() - 1_000) });
|
||||
await seedCommentQueuedDuringRun(companyId, agentId, issue.id);
|
||||
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("still offers Retry on a parent with open subtasks when its queued wake carries no message", async () => {
|
||||
const { companyId, agentId, issue } = await seedParentWaitingOnSubtasks();
|
||||
await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "on_demand", reason: "issue_assigned",
|
||||
status: "deferred_issue_execution", requestedByActorType: "user", requestedAt: new Date(Date.now() - 13 * 60_000),
|
||||
payload: { issueId: issue.id } });
|
||||
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true });
|
||||
});
|
||||
|
||||
it("still offers Retry on a parent with open subtasks and a queued message when its run failed and recovery gave up", async () => {
|
||||
const { companyId, agentId, issue, runId, hold } = await seedSwapEndedRun(new Date(Date.now() - 5_000));
|
||||
await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, issue.id));
|
||||
await db.insert(issues).values({ companyId, title: "Open subtask", status: "in_progress", assigneeAgentId: agentId, parentId: issue.id });
|
||||
await seedCommentQueuedDuringRun(companyId, agentId, issue.id);
|
||||
await db.update(issueRecoveryActions).set({ ownerType: "board" }).where(eq(issueRecoveryActions.id, hold.id));
|
||||
const blocked = { ...issue, status: "blocked" };
|
||||
expect(await taskInterruptionPresentation(db, blocked)).toMatchObject({ state: "failed", canRetry: true, observedRunId: runId });
|
||||
});
|
||||
|
||||
it("still offers Retry on a parent with open subtasks and a queued message when saved-work recovery is exhausted", async () => {
|
||||
const { companyId, agentId, issue, runId } = await seedParentWaitingOnSubtasks();
|
||||
await seedCommentQueuedDuringRun(companyId, agentId, issue.id);
|
||||
await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issue.id, kind: "active_run_watchdog",
|
||||
status: "escalated", ownerType: "system", returnOwnerAgentId: agentId, cause: "native_restart_recovery_blocked",
|
||||
fingerprint: `exhausted:${runId}`, nextAction: "Retry to continue.", maxAttempts: 3,
|
||||
evidence: { runId, systemRecovery: { ...createSystemRecovery(runId), state: "exhausted", attempts: 3, retriesUsed: 3 } } });
|
||||
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "failed", canRetry: true });
|
||||
});
|
||||
|
||||
it("does not show a failed attempt on a parent whose run succeeded with no queued message", async () => {
|
||||
for (const childStatus of ["in_progress", "blocked", "cancelled"]) {
|
||||
const { issue } = await seedParentWaitingOnSubtasks(childStatus);
|
||||
expect(await taskInterruptionPresentation(db, issue)).toMatchObject({ state: "resolved", canRetry: false });
|
||||
}
|
||||
});
|
||||
|
||||
it("does not offer Retry on a parent whose run succeeded and that waits for its subtasks", async () => {
|
||||
const { companyId, agentId, issue } = await seedTask();
|
||||
await db.insert(issues).values({ companyId, title: "Subtask", status: "in_progress", assigneeAgentId: agentId, parentId: issue.id });
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { STORAGE_UNAVAILABLE_WAIT_RETRY_REASON } from "../storage/storage-unavailable.js";
|
||||
import { AGENT_DIRECTORY_WAIT_RETRY_REASON } from "./execution-recovery-attempt.js";
|
||||
import { boundSetupCommandFailure, readSetupCommandFailure } from "./workspace-setup-command-failure.js";
|
||||
import { and, desc, eq, inArray, isNull, not, sql, type SQL } from "drizzle-orm";
|
||||
import { and, desc, eq, inArray, isNull, not, notInArray, sql, type SQL } from "drizzle-orm";
|
||||
import { unidentifiedLocalLaunchRefusals } from "./native-local-process-stop.js";
|
||||
import { nativeSessionCleanupBlock } from "../vendor/paperclip-runner/index.js";
|
||||
import { agents, agentWakeupRequests, approvals, chatActions, companies, environmentLeases, heartbeatRuns, issueApprovals, issueRecoveryActions,
|
||||
@@ -65,7 +65,8 @@ type ProjectionRun = Pick<typeof heartbeatRuns.$inferSelect,
|
||||
type ProjectionWait = { id: string; requestedAt: Date; sourceRunId?: string | null; executionWaitReason?: string | null;
|
||||
preparationStatus?: string | null; preparationRetryAt?: number | string | null; preparationUpdatedAt?: Date | null;
|
||||
/** When the user pressed Interrupt to deliver this queued message now. */
|
||||
queuedInterruptRequestedAt?: string | null };
|
||||
queuedInterruptRequestedAt?: string | null;
|
||||
queuedComment?: boolean };
|
||||
type ProjectionState = {
|
||||
run?: ProjectionRun;
|
||||
action?: typeof issueRecoveryActions.$inferSelect;
|
||||
@@ -89,6 +90,7 @@ type ProjectionState = {
|
||||
retryDependencies?: RetryDependencies;
|
||||
/** The task waits on unresolved blocker tasks, a wait that does not depend on its last run. */
|
||||
waitingOnBlockers?: boolean;
|
||||
waitingOnOpenSubtasks?: boolean;
|
||||
/** A live owner is settling this run's session cleanup gate: an unreleased
|
||||
* remote environment, or a scheduled close in this process. Without one,
|
||||
* nothing settles the gate by itself, so Retry is the way forward. */
|
||||
@@ -345,7 +347,8 @@ function projectTaskInterruptionState(issue: ProjectionIssue, input: ProjectionS
|
||||
canRetry: heldRetryOffersRetry(reason, input.cleanupSettling), retryWait: { reason,
|
||||
message: retryWaitMessages[reason] ?? "Retry is waiting for execution checks. Your message is saved." } };
|
||||
}
|
||||
const queuedDuringRun = Boolean(wait && run.status === "succeeded" && ["todo", "in_progress", "in_review"].includes(issue.status) && run.finishedAt && wait.requestedAt.getTime() <= run.finishedAt.getTime()
|
||||
const queuedDuringRun = Boolean(wait && run.status === "succeeded" && (["todo", "in_progress", "in_review"].includes(issue.status)
|
||||
|| (issue.status === "blocked" && input.waitingOnOpenSubtasks && !wait.sourceRunId && wait.queuedComment)) && run.finishedAt && wait.requestedAt.getTime() <= run.finishedAt.getTime()
|
||||
&& now - run.finishedAt.getTime() < QUEUED_INPUT_DISPATCH_GRACE_MS);
|
||||
if (wait && (wait.requestedAt.getTime() > now - 10_000 || queuedDuringRun) && system?.state !== "exhausted")
|
||||
return { ...base, state: "working" };
|
||||
@@ -496,7 +499,10 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
|
||||
executionWaitReason: boundedJsonString(sql`to_jsonb(${unpricedUsageWait(sql`${agentWakeupRequests.payload}#>>'{executionWait,reason}'`,
|
||||
sql`${agentWakeupRequests.payload}#>>'{executionWait,message}'`)})`, 80),
|
||||
preparationStatus: chatActions.status, preparationRetryAt: chatActionDeadline("failed_run_retry"),
|
||||
preparationUpdatedAt: chatActions.updatedAt })
|
||||
preparationUpdatedAt: chatActions.updatedAt,
|
||||
queuedComment: sql<boolean>`coalesce(jsonb_typeof(${agentWakeupRequests.payload}->'commentId') = 'string'
|
||||
or jsonb_array_length(case when jsonb_typeof(${agentWakeupRequests.payload}#>'{_paperclipWakeContext,wakeCommentIds}') = 'array'
|
||||
then ${agentWakeupRequests.payload}#>'{_paperclipWakeContext,wakeCommentIds}' end) > 0, false)` })
|
||||
.from(agentWakeupRequests).leftJoin(chatActions, and(eq(chatActions.companyId, agentWakeupRequests.companyId),
|
||||
eq(chatActions.id, canonicalExecutionUuid(sql`${agentWakeupRequests.payload}#>>'{taskRetry,chatActionId}'`)),
|
||||
eq(chatActions.kind, "failed_run_retry"), sql`${chatActions.payload}->>'taskRetryIntentId' = ${agentWakeupRequests.id}::text`,
|
||||
@@ -670,6 +676,11 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
|
||||
for (const [waitingIssueId, row] of readiness)
|
||||
if (row.unresolvedBlockerIssueIds.length) waitingOnBlockers.add(waitingIssueId);
|
||||
}
|
||||
const parentIds = selected.filter(issue => issue.status === "blocked" && runMap.get(issue.id)?.status === "succeeded"
|
||||
&& currentWaits.some(wait => wait.issueId === issue.id && wait.queuedComment)).map(issue => issue.id);
|
||||
const waitingOnOpenSubtasks = new Set(parentIds.length ? (await db.selectDistinct({ parentId: issues.parentId }).from(issues)
|
||||
.where(and(eq(issues.companyId, companyId), inArray(issues.parentId, parentIds), notInArray(issues.status, ["done", "cancelled"]))))
|
||||
.flatMap(row => row.parentId ? [row.parentId] : []) : []);
|
||||
for (const wait of currentWaits) {
|
||||
const current = waitMap.get(wait.issueId);
|
||||
if (!current || wait.requestedAt > current.requestedAt ||
|
||||
@@ -736,6 +747,7 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
|
||||
admissionWait: admissionWaitMap.get(issue.id),
|
||||
retryDependencies: retryDependencyMap.get(issue.id),
|
||||
waitingOnBlockers: waitingOnBlockers.has(issue.id),
|
||||
waitingOnOpenSubtasks: waitingOnOpenSubtasks.has(issue.id),
|
||||
cleanupSettling: cleanupSettling.has(issue.id),
|
||||
}, now);
|
||||
result.set(issue.id, presentation);
|
||||
|
||||
Reference in new issue
Block a user