From b2b2c1e60b6293d2bf9e591498bfd4f1c05137a6 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 16 Nov 2026 19:59:48 -0600 Subject: [PATCH] fix(issues): release a done blocker whose ended run already finalized (#21948) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A parent task can wait for its subtasks through "blocks" relations. When the last blocker is done, the parent wakes with `issue_children_completed` or `issue_blockers_resolved` > - A done blocker with workspace work also passes a workspace-finalize barrier. The barrier holds dependents until the blocker's run saves its workspace > - The barrier reads the blocker's latest workspace operation. A save that starts inside the run's finalize and ends after it, or any operation that the same run records after its finalize, makes the barrier look unsaved > - That run has ended, so it records no further finalize. Nothing clears the barrier, every wake for the parent becomes a skipped `issue_dependencies_blocked` wait, and the periodic backstop also sees the blocker as unresolved > - This pull request treats a settled operation as saved when its run has ended and that run recorded a successful finalize for the blocker issue > - The benefit is that the parent gets its children-completed run, and parents that are stuck now are released by the existing dependency backstop with no manual step ## Linked Issues or Issue Description Refs #19219, #17206, #18665. #19219 releases the barrier when a terminal run recorded no finalize at all. This change is narrower: the run must have recorded a successful finalize for the blocker issue. A failed finalize with no later success still holds the gate (#17206 owns that recovery). **What happened?** A parent task created two subtasks. Subtask 1 finished and its run saved the workspace. Then the parent run bound both subtasks as blockers and moved the parent to `blocked` (`durable_dependency_blocker_bound`). Subtask 2 finished 30 seconds later. The `issue_children_completed` wake for the parent was skipped and recorded as `issue_dependencies_blocked` with `unresolvedBlockerIssueIds` set to subtask 1, which had been done for 2.5 minutes. The parent showed `pendingFinalizeBlockerIssueIds: [subtask 1]` and `canRetry: false`. No later wake came. The parent stayed blocked until a person changed it by hand. **Expected behavior** When every blocker is done and each blocker's run has finished with a successful workspace finalize, the parent gets one children-completed run. **Steps to reproduce** 1. Create a parent with two subtasks on separate execution workspaces. 2. Finish subtask 1. Its run records a successful `workspace_finalize`, and a `remote_workspace_snapshot` of the same run ends after the finalize ends. The incident run had a failed copy-back that was retried during a restart overlap. 3. Bind both subtasks as blockers of the parent, then block the parent. 4. Finish subtask 2 and dispatch the native `issue_children_completed` intent. 5. Before this change, the wake is skipped as `issue_dependencies_blocked` and no run starts. **Paperclip version or commit** master `6205b24d54`. ## What Changed - `pendingFinalizeBlockerCondition` (`server/src/services/issues.ts`) has one more release rule. The latest blocker operation is settled (`succeeded`, `failed` or `skipped`). Its heartbeat run is terminal. That run recorded a successful `workspace_finalize` for the blocker issue on the same workspace (or with no workspace id). Readiness, the queue dependency gate, blocker attention and the periodic backstop all read this condition. - Live runs keep the old behavior. A save that ends after its finalize, or an operation that starts after it, still holds dependents while the run is running. - Foreign-run operations, unfinished operations and failed finalizes with no success still hold dependents. ## Verification - New regression test `heartbeat-dependency-scheduling.test.ts` › "runs the parent's children-completed wake when a child it bound after completion saved after its finalize". It replays the incident order: child 1 done and finalized, its snapshot ends after the finalize, the parent binds both children and blocks, child 2 finishes, the native children-completed intent dispatches. Red before the fix (no run starts; the wake becomes `issue_dependencies_blocked`). Green after the fix (a run starts with the intent's context, and no dependency wait is recorded). - `issues-service.test.ts` nested-snapshot matrix: `late-finish` and `later-operation` now run with a live owner run (still pending) and with a terminal owner run (released). `covered`, `foreign-run`, `unfinished` and `failed-finalize` stay as before. - `cd server && npx vitest run src/__tests__/heartbeat-dependency-scheduling.test.ts src/__tests__/issues-service.test.ts` passes (207 tests). - `pnpm verify` and `tsc --noEmit` for `server` pass locally. ## Risks - Low risk. The change only releases a barrier for a run that has already ended after a successful finalize for the issue. That run cannot write another finalize, so the old rule held dependents forever with no path to release them. - An operation that a terminal run recorded after its finalize is no longer a reason to hold dependents. Such work can never be saved by that run. Holding the parent forever does not protect it. - No schema, API or UI contract change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test execution) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../heartbeat-dependency-scheduling.test.ts | 85 +++++++++++++++++++ server/src/__tests__/issues-service.test.ts | 21 +++-- server/src/services/issues.ts | 18 ++++ 3 files changed, 118 insertions(+), 6 deletions(-) diff --git a/server/src/__tests__/heartbeat-dependency-scheduling.test.ts b/server/src/__tests__/heartbeat-dependency-scheduling.test.ts index 134ce9cc8cc..e7d912cf9b9 100644 --- a/server/src/__tests__/heartbeat-dependency-scheduling.test.ts +++ b/server/src/__tests__/heartbeat-dependency-scheduling.test.ts @@ -277,6 +277,91 @@ describeEmbeddedPostgres("heartbeat dependency-aware queued run selection", () = } }, 45_000); + it("runs the parent's children-completed wake when a child it bound after completion saved after its finalize", async () => { + const companyId = randomUUID(); + const agentId = randomUUID(); + const parentId = randomUUID(); + const firstChildId = randomUUID(); + const secondChildId = randomUUID(); + const firstWorkspaceId = randomUUID(); + const secondWorkspaceId = randomUUID(); + const firstRunId = randomUUID(); + const secondRunId = randomUUID(); + const decisionId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Child completion", + issuePrefix: `C${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`, + requireBoardApprovalForNewAgents: false, defaultResponsibleUserId: "responsible-user" }); + await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: "responsible-user", status: "active", membershipRole: "member" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Parent", role: "engineer", + status: "active", adapterType: "codex_local", adapterConfig: {}, permissions: {}, + runtimeConfig: { heartbeat: { wakeOnDemand: true, maxConcurrentRuns: 1 } } }); + await db.insert(executionWorkspaces).values([firstWorkspaceId, secondWorkspaceId].map((id) => ({ + id, companyId, mode: "shared_workspace", strategyType: "project_primary", name: "Child workspace", status: "active", + }))); + await db.insert(issues).values([ + { id: parentId, companyId, title: "Parent", status: "in_progress", + assigneeAgentId: agentId, responsibleUserId: "responsible-user" }, + { id: firstChildId, companyId, parentId, title: "First child", status: "done", + assigneeAgentId: agentId, executionWorkspaceId: firstWorkspaceId, responsibleUserId: "responsible-user", + completedAt: new Date("2026-11-16T23:33:00.000Z") }, + { id: secondChildId, companyId, parentId, title: "Second child", status: "in_progress", + assigneeAgentId: agentId, executionWorkspaceId: secondWorkspaceId, responsibleUserId: "responsible-user" }, + ]); + await db.insert(heartbeatRuns).values([ + { id: firstRunId, companyId, agentId, issueId: firstChildId, status: "succeeded", + startedAt: new Date("2026-11-16T23:28:28.000Z"), finishedAt: new Date("2026-11-16T23:32:59.430Z") }, + { id: secondRunId, companyId, agentId, issueId: secondChildId, status: "running", + startedAt: new Date("2026-11-16T23:31:17.000Z") }, + ]); + await db.insert(workspaceOperations).values([ + { companyId, executionWorkspaceId: firstWorkspaceId, issueId: firstChildId, heartbeatRunId: firstRunId, + phase: "workspace_finalize", status: "succeeded", + startedAt: new Date("2026-11-16T23:32:57.000Z"), finishedAt: new Date("2026-11-16T23:32:59.000Z") }, + { companyId, executionWorkspaceId: firstWorkspaceId, issueId: firstChildId, heartbeatRunId: firstRunId, + phase: "remote_workspace_snapshot", status: "succeeded", + startedAt: new Date("2026-11-16T23:32:58.000Z"), finishedAt: new Date("2026-11-16T23:32:59.200Z") }, + ]); + await db.insert(issueRelations).values([firstChildId, secondChildId].map((blockerId) => ({ + companyId, issueId: blockerId, relatedIssueId: parentId, type: "blocks" as const, createdByAgentId: agentId, + }))); + await db.update(issues).set({ status: "blocked", blockedTransitionAt: new Date("2026-11-16T23:34:52.374Z") }) + .where(eq(issues.id, parentId)); + await db.insert(workspaceOperations).values({ companyId, executionWorkspaceId: secondWorkspaceId, + issueId: secondChildId, heartbeatRunId: secondRunId, phase: "workspace_finalize", status: "succeeded", + startedAt: new Date("2026-11-16T23:35:20.000Z"), finishedAt: new Date("2026-11-16T23:35:22.000Z") }); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date("2026-11-16T23:35:23.250Z") }) + .where(eq(heartbeatRuns.id, secondRunId)); + await db.update(issues).set({ status: "done", completedAt: new Date("2026-11-16T23:35:23.218Z") }) + .where(eq(issues.id, secondChildId)); + const [intent] = await db.insert(agentWakeupRequests).values({ companyId, agentId, + source: "automation", triggerDetail: "system", reason: "issue_children_completed", + requestedByActorType: "system", requestedByActorId: "native-status-committer", + idempotencyKey: `issue_children_completed:${parentId}:${secondChildId}:${decisionId}`, + payload: { issueId: parentId, taskId: parentId, _paperclipWakeContext: { + issueId: parentId, taskId: parentId, source: "native_status_decision", + nativeChildCompletionDecisionId: decisionId, completedChildIssueId: secondChildId, + childIssueIds: [firstChildId, secondChildId], + } }, + }).returning(); + + await heartbeat.dispatchPendingNativeStatusWakeups({ companyId }); + expect(await waitForCondition(async () => mockAdapterExecute.mock.calls.length === 1, 10_000)).toBe(true); + await heartbeat.drainActiveRunExecutions(); + + const dependencyWaits = await db.select().from(agentWakeupRequests).where(and( + eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.reason, "issue_dependencies_blocked"), + )); + expect(dependencyWaits).toEqual([]); + const parentRuns = await db.select().from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, companyId), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${parentId}`, + )); + const childrenCompletedRun = parentRuns.find((run) => run.contextSnapshot?.nativeStatusWakeIntentId === intent!.id); + expect(childrenCompletedRun?.contextSnapshot).toMatchObject({ + wakeReason: "issue_children_completed", nativeStatusWakeIntentId: intent!.id, + completedChildIssueId: secondChildId, + }); + }, 45_000); + it("keeps a saved native continuation behind its active predecessor without retagging that run", async () => { const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(); let finishPredecessor!: () => void; diff --git a/server/src/__tests__/issues-service.test.ts b/server/src/__tests__/issues-service.test.ts index 75aa6921574..bde1ad87d66 100644 --- a/server/src/__tests__/issues-service.test.ts +++ b/server/src/__tests__/issues-service.test.ts @@ -4773,13 +4773,23 @@ const dependencyTestDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_ISSU }); }); - it.each(["covered", "foreign-run", "unfinished", "failed-finalize", "late-finish", "later-operation"])( - "requires a successful owning finalize to cover a nested snapshot: %s", async kind => { + it.each([ + ["covered", "succeeded", true], + ["foreign-run", "succeeded", false], + ["unfinished", "succeeded", false], + ["failed-finalize", "succeeded", false], + ["late-finish", "running", false], + ["later-operation", "running", false], + ["late-finish", "succeeded", true], + ["later-operation", "succeeded", true], + ] as const)( + "requires a successful owning finalize to cover a nested snapshot: %s (run %s)", async (kind, runStatus, ready) => { const f = await seedSharedWorkspaceDependency(); const runId = randomUUID(), foreignRunId = randomUUID(); - await db.insert(heartbeatRuns).values([runId, foreignRunId].map(id => ({ - id, companyId: f.companyId, agentId: f.assigneeAgentId, status: "succeeded", - }))); + await db.insert(heartbeatRuns).values([ + { id: runId, companyId: f.companyId, agentId: f.assigneeAgentId, status: runStatus }, + { id: foreignRunId, companyId: f.companyId, agentId: f.assigneeAgentId, status: "succeeded" }, + ]); const started = new Date("2026-05-23T22:00:00.000Z"); const finished = new Date("2026-05-23T22:00:02.000Z"); await db.insert(workspaceOperations).values([ @@ -4797,7 +4807,6 @@ const dependencyTestDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_ISSU startedAt: new Date(finished.getTime() + 100), finishedAt: new Date(finished.getTime() + 200), }] : []), ]); - const ready = kind === "covered"; expect((await svc.listWakeableBlockedDependents(f.blockerId)).map(issue => issue.id)) .toEqual(ready ? [f.dependentId] : []); await expect(svc.getDependencyReadiness(f.dependentId)).resolves.toMatchObject({ diff --git a/server/src/services/issues.ts b/server/src/services/issues.ts index 0bdcf07c2af..3d14f402788 100644 --- a/server/src/services/issues.ts +++ b/server/src/services/issues.ts @@ -2625,6 +2625,24 @@ function pendingFinalizeBlockerCondition(companyId: string, issueId: SQL, worksp and recovery_finalize.started_at >= coalesce(latest_blocker_operation.finished_at, latest_blocker_operation.started_at) ) ) + and not ( + latest_blocker_operation.status in ('succeeded', 'failed', 'skipped') + and exists ( + select 1 from heartbeat_runs settled_owner_run + where settled_owner_run.company_id = ${companyId}::uuid + and settled_owner_run.id = latest_blocker_operation.heartbeat_run_id + and settled_owner_run.status in (${sql.join([...TERMINAL_HEARTBEAT_RUN_STATUSES].map(status => sql`${status}`), sql`, `)}) + ) + and exists ( + select 1 from workspace_operations settled_run_finalize + where settled_run_finalize.company_id = ${companyId}::uuid + and settled_run_finalize.issue_id = ${issueId} + and settled_run_finalize.heartbeat_run_id = latest_blocker_operation.heartbeat_run_id + and settled_run_finalize.phase = 'workspace_finalize' and settled_run_finalize.status = 'succeeded' + and (settled_run_finalize.execution_workspace_id is null + or settled_run_finalize.execution_workspace_id = ${workspaceId}) + ) + ) )`; }