From 6be53f0831fe342081fc7a5eb11346e1541d5f4d Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 19 Nov 2026 14:37:57 -0600 Subject: [PATCH] fix(issues): keep a held task blocked when its agent tries to resume it (#22245) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run stops with an unknown outcome, recovery can record an execution hold for the task. Wake admission refuses every wake for the task until the system continues it from saved work or a person uses Retry. Recovery sets the task to "Blocked", so a person can see it. > - An agent can move that task out of "Blocked". It can change the status, send `reopen` or `resume`, or check the task out. None of these routes checks the execution hold, so the hold stays. > - The task then looks workable, but its agent cannot start. Its scheduled follow-up check (monitor) comes due, and the scheduler tries it on every tick. Wake admission refuses each try. Agents re-arm the check, and the check stays overdue for days. > - #22215 makes these tasks show as "Blocked" in lists. It does not stop agents from making the state. > - This pull request refuses an agent request that moves a held task out of "Blocked". The response is 409 and gives the next action of the hold. > - The benefit is that a held task stays "Blocked" until its hold is released. Its follow-up checks wait and do not retry on every tick, and the agent gets the correct reason. ## Linked Issues or Issue Description (B) No issue exists. Bug report: **What happened?** Scheduled follow-up checks on agent tasks stayed overdue for many days. Each due check was skipped with `execution_reconciliation_required`. One skip record merged more than 25,000 attempts. The tasks were in `in_progress` or `in_review`, but an execution hold from a failed run refused every wake. The agent re-armed the checks, and the checks never started. An agent can move a task out of `blocked` while such a hold exists. The resume guard and checkout do not check the hold. **What did you expect to happen?** A task that wake admission holds stays `blocked` until the hold is released. An agent that tries to move it gets an error that tells it what must happen first. **Steps to reproduce** 1. Make a `blocked` task that is assigned to an agent. 2. Add a recovery record for the task with status `resolved`, cause `provider_failure_meaning_unverified` and evidence `automaticRecovery.replay = "blocked"`. `getExecutionBlocker` returns this hold. 3. As the assigned agent, send `PATCH /api/issues/:id` with `{ "status": "in_progress" }`. You can also send `reopen` or `resume`, or `POST /api/issues/:id/checkout` with `expectedStatuses: ["blocked"]`. 4. See that the request succeeds and the task leaves `blocked`. The hold stays, so wake admission still refuses the task. 5. Give the task an execution policy with a review stage. As the agent, send `PATCH /api/issues/:id` with `{ "status": "done" }`. The execution policy changes the status to `in_review` and assigns the reviewer. The task leaves `blocked`, and the hold stays. **Paperclip version or commit** `master` at deacaf727c. Refs #22215 (display status for the same holds). Related open PRs: #14590 parks monitor wakes on held tasks and retires a no-replay hold when a person restores the status. #14156 changes how a settled no-replay hold is released. #19786 keeps the rule that only a board user retires an automatic no-replay hold. This pull request adds the matching rule for agents and task status. ## What Changed - `server/src/routes/issues.ts`: new `assertAgentMayLeaveBlocked`. It gets the status that the request will save. When an agent moves a `blocked` task to a status other than `blocked`, `done` or `cancelled`, it reads `getExecutionBlocker`. If a recovery hold applies, it returns 409 with `recoveryActionId`, `cause` and `nextAction`. The task does not change. - `PATCH /issues/:id` calls the check one time, after the execution policy transition and before any write. It uses the final status. This covers a status change, `reopen`, `resume`, and a `done` request that a review or approval stage changes to `in_review`. - `POST /issues/:id/comments` calls the check when the comment moves the task to `todo` (`reopen` or `resume`). The native `comment_on_task` tool uses the comments route. Agent comments without `reopen` or `resume` do not move a `blocked` task. - `POST /issues/:id/checkout` calls the check with `in_progress` when `expectedStatuses` includes `blocked`. - The check applies only to recovery holds (a blocker with a `recoveryActionId`). A failed legacy run whose process or lease is still alive also blocks wakes (`execution_owner_active`), but that lock clears without a person. If the check refused it, the task would stay `blocked` after the lock clears, and nothing would release it. - `server/src/services/execution-blocker.ts`: `getExecutionBlocker` gets a `recoveryHoldOnly` option. With this option, it does not return the live-run lock first, so it finds a recovery hold that exists at the same time. The check uses this option. Other callers do not change. - Behavior for people does not change. A board user can still change the status, send a message or use Retry. An agent can still close a held task. - `server/src/__tests__/issue-recovery-actions.test.ts`: new tests with a real database. They seed a `blocked` task with a resolved no-replay hold and try each agent path, including `done` with a review stage. A second test shows that an agent can still cancel the task, and can move it to `todo` after the hold stops blocking replay. A third test shows that a live previous run without a recovery hold does not hold the agent. A case with both a live previous run and a recovery hold shows that the agent is refused. ## Verification - Failing before the fix (first commit): `pnpm --dir server exec vitest run src/__tests__/issue-recovery-actions.test.ts -t "no-replay hold when its agent"`. The tests "keeps a task blocked behind a no-replay hold when its agent tries a status change / reopen / resume comment / checkout" failed with `expected 200 to be 409` (`201` for the comment). - Failing before the fix (second commit): the new case "... tries a completion that its review stage hands to a reviewer" failed with `expected 200 to be 409`. The route accepted `done` and started to wake the reviewer. - Failing before the fix (third commit): the new case "... tries a status change while a live previous run also owns the task" failed with `expected 200 to be 409`. - Passing after the fix: `vitest run src/__tests__/issue-recovery-actions.test.ts -t "no-replay hold|live previous run|close a held task"`, 8 of 8 pass. - Affected suites on the current head: `vitest run` on `issue-recovery-actions`, `issue-comment-reopen-routes`, `issue-update-comment-wakeup-routes`, `issue-stale-execution-lock-routes`, `issue-closed-workspace-routes`, `low-trust-red-team-routes`, `issue-agent-mutation-ownership-routes`, `issue-execution-policy-routes`, `issue-status-version-route-service`, `heartbeat-held-comment-wake`, `legacy-continuation-authority` and `issue-display-status`. 12 files, 703 tests pass (with `--hookTimeout=240000`, because a setup hook is slow in the local sandbox). - Typecheck: after `pnpm --filter @paperclipai/paperclip-runner run build:typescript`, `node server/scripts/typecheck-layers.mjs` checks all server layers and test trees, with no errors in server files. The local install has only the server dependencies, so files in `cli/`, `ui/` and other packages could not resolve their modules. CI runs the full typecheck. - Not run locally: the full `pnpm verify` selection is too large for the local sandbox. CI runs it. ## Risks - Behavior change for agents: an agent that tries to move a held task out of `blocked` now gets 409. Wake admission already refused that task, so the agent could not run it. - The system still continues a hold that it can settle (fresh continuation from saved work and the assigned-work sweep). That path does not need an agent to unblock the task. - An agent `reopen` or `resume` comment on a held task is refused, and the comment is not saved. The agent can post the comment without `reopen` or `resume`. - Cost: one `getExecutionBlocker` read, only when an agent tries to move a `blocked` task. - Not covered: tasks that are already in this state, and recovery paths that record a hold without setting the task to `blocked`. These still show as "Blocked" through #22215. #14590 changes monitor retries for them. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../__tests__/issue-recovery-actions.test.ts | 73 +++++++++++++++++++ server/src/routes/issues.ts | 24 ++++++ server/src/services/execution-blocker.ts | 3 +- 3 files changed, 99 insertions(+), 1 deletion(-) diff --git a/server/src/__tests__/issue-recovery-actions.test.ts b/server/src/__tests__/issue-recovery-actions.test.ts index 36c2d056587..18dbd995d89 100644 --- a/server/src/__tests__/issue-recovery-actions.test.ts +++ b/server/src/__tests__/issue-recovery-actions.test.ts @@ -48,6 +48,8 @@ import { caseHandoffService } from "../services/case-handoffs.js"; import { instanceSettingsService } from "../services/instance-settings.js"; import { withNativeWorkspaceFinalizationOwnership } from "../services/native-runtime/native-workspace-finalization-ownership.js"; import { noticeMetadataReferencesRecoveryAction } from "../services/recovery/successful-run-handoff.js"; +import { getExecutionBlocker } from "../services/execution-blocker.js"; +import { CONVERSATION_CONTINUATION_POLICY } from "../services/conversation-continuation.js"; const external = resolveExternalTestDatabaseUrl("PAPERCLIP_RECOVERY_ACTION_TEST_DATABASE_URL"); const embeddedPostgresSupport = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport(); @@ -2148,6 +2150,77 @@ describeEmbeddedPostgres("issue recovery actions", () => { expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action!.id)))[0]).toEqual(recorded); }); + async function seedHeldBlockedTask() { + const fixture = await seedCompany(); + const failedRunId = randomUUID(), agentRunId = randomUUID(); + await seedHeartbeatRun({ companyId: fixture.companyId, agentId: fixture.coderId, runId: failedRunId, + issueId: fixture.sourceIssueId, status: "failed" }); + await seedHeartbeatRun({ companyId: fixture.companyId, agentId: fixture.coderId, runId: agentRunId }); + await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, fixture.sourceIssueId)); + const [hold] = await db.insert(issueRecoveryActions).values({ + companyId: fixture.companyId, sourceIssueId: fixture.sourceIssueId, kind: "active_run_watchdog", + status: "resolved", outcome: "blocked", ownerType: "board", returnOwnerAgentId: fixture.coderId, + cause: "provider_failure_meaning_unverified", fingerprint: failedRunId, + nextAction: "Preserve recorded work without replay.", + evidence: { runId: failedRunId, automaticRecovery: { replay: "blocked", actionOutcome: "unknown" } }, + }).returning(); + const agent = createApp({ type: "agent", source: "agent_jwt", companyId: fixture.companyId, + agentId: fixture.coderId, runId: agentRunId }); + return { ...fixture, hold: hold!, agent }; + } + + async function seedLiveOwnerRun(f: Awaited>) { + await db.insert(heartbeatRuns).values({ id: randomUUID(), companyId: f.companyId, agentId: f.coderId, + invocationSource: "manual", status: "failed", processPid: process.pid, contextSnapshot: { issueId: f.sourceIssueId }, + resultJson: { conversationContinuation: CONVERSATION_CONTINUATION_POLICY } }); + expect(await getExecutionBlocker(db, f.companyId, f.sourceIssueId)) + .toMatchObject({ cause: "execution_owner_active", recoveryActionId: null }); + } + + it.each([ + ["status change", (f: Awaited>) => + request(f.agent).patch(`/api/issues/${f.sourceIssueId}`).send({ status: "in_progress" })], + ["reopen", (f: Awaited>) => + request(f.agent).patch(`/api/issues/${f.sourceIssueId}`).send({ reopen: true, comment: "Reopening the case." })], + ["resume comment", (f: Awaited>) => + request(f.agent).post(`/api/issues/${f.sourceIssueId}/comments`).send({ body: "Continuing the case.", resume: true })], + ["checkout", (f: Awaited>) => + request(f.agent).post(`/api/issues/${f.sourceIssueId}/checkout`).send({ agentId: f.coderId, expectedStatuses: ["blocked"] })], + ["completion that its review stage hands to a reviewer", async (f: Awaited>) => { + await db.update(issues).set({ executionPolicy: { mode: "normal", commentRequired: false, + stages: [{ id: randomUUID(), type: "review", participants: [{ type: "agent", agentId: f.managerId }] }] } }) + .where(eq(issues.id, f.sourceIssueId)); + return request(f.agent).patch(`/api/issues/${f.sourceIssueId}`).send({ status: "done" }); + }], + ["status change while a live previous run also owns the task", async (f: Awaited>) => { + await seedLiveOwnerRun(f); + return request(f.agent).patch(`/api/issues/${f.sourceIssueId}`).send({ status: "todo" }); + }], + ])("keeps a task blocked behind a no-replay hold when its agent tries a %s", async (_entry, send) => { + const f = await seedHeldBlockedTask(); + const response = await send(f); + expect(response.status).toBe(409); + expect(response.body.details).toMatchObject({ recoveryActionId: f.hold.id, nextAction: expect.stringContaining(f.hold.nextAction) }); + expect((await db.select().from(issues).where(eq(issues.id, f.sourceIssueId)))[0]!.status).toBe("blocked"); + }); + + it("lets the agent close a held task, or move it on once the hold no longer blocks replay", async () => { + const closed = await seedHeldBlockedTask(); + await request(closed.agent).patch(`/api/issues/${closed.sourceIssueId}`).send({ status: "cancelled" }).expect(200); + const released = await seedHeldBlockedTask(); + await db.update(issueRecoveryActions).set({ evidence: { runId: released.hold.evidence.runId } }) + .where(eq(issueRecoveryActions.id, released.hold.id)); + await request(released.agent).patch(`/api/issues/${released.sourceIssueId}`).send({ status: "todo" }).expect(200); + expect((await db.select().from(issues).where(eq(issues.id, released.sourceIssueId)))[0]!.status).toBe("todo"); + }); + + it("does not hold a blocked task while only a live previous run still owns it", async () => { + const f = await seedHeldBlockedTask(); + await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.hold.id)); + await seedLiveOwnerRun(f); + await request(f.agent).patch(`/api/issues/${f.sourceIssueId}`).send({ status: "todo" }).expect(200); + }); + async function seedReconciledDelivery() { const fixture = await seedCompany(); const { companyId, coderId, sourceIssueId } = fixture; diff --git a/server/src/routes/issues.ts b/server/src/routes/issues.ts index 0da962ebb8e..0df3332f67d 100644 --- a/server/src/routes/issues.ts +++ b/server/src/routes/issues.ts @@ -7280,6 +7280,22 @@ export function issueRoutes( return false; } + async function assertAgentMayLeaveBlocked( + req: Request, + res: Response, + issue: { id: string; companyId: string; status: string }, + nextStatus: string, + ) { + if (req.actor.type !== "agent" || issue.status !== "blocked" || nextStatus === "blocked" || isClosedIssueStatus(nextStatus)) return true; + const hold = await getExecutionBlocker(db, issue.companyId, issue.id, { recoveryHoldOnly: true }); + if (!hold?.recoveryActionId) return true; + res.status(409).json({ + error: `Issue stays blocked until its execution recovery hold is released. ${hold.nextAction}`, + details: { issueId: issue.id, recoveryActionId: hold.recoveryActionId, cause: hold.cause, nextAction: hold.nextAction }, + }); + return false; + } + async function assertExplicitResumeIntentAllowed( req: Request, res: Response, @@ -15179,6 +15195,7 @@ export function issueRoutes( } const nextStatus = updateFields.status ?? existing.status; + if (!(await assertAgentMayLeaveBlocked(req, res, existing, nextStatus))) return; if (updateFields.unblockDescriptor && nextStatus !== "blocked") { throw unprocessable("unblockDescriptor requires blocked status"); } @@ -17116,6 +17133,7 @@ export function issueRoutes( const checkoutRunId = requireAgentRunId(req, res); if (req.actor.type === "agent" && !checkoutRunId) return; + if (req.body.expectedStatuses.includes("blocked") && !(await assertAgentMayLeaveBlocked(req, res, issue, "in_progress"))) return; // Reopen the closed isolated workspace only after the run-id gate passes. A // rejected checkout must not rebuild and republish the workspace as active. @@ -19507,6 +19525,12 @@ export function issueRoutes( .json({ error: "Issue follow-up blocked by unresolved blockers" }); return; } + if ( + effectiveMoveToTodoRequested && + !hasUnresolvedFirstClassBlockers && + !(await assertAgentMayLeaveBlocked(req, res, issue, "todo")) + ) + return; if ( !(await assertCrossIssueInfluenceWithinRunCap( req, diff --git a/server/src/services/execution-blocker.ts b/server/src/services/execution-blocker.ts index fdc644f4418..f73e3794f4e 100644 --- a/server/src/services/execution-blocker.ts +++ b/server/src/services/execution-blocker.ts @@ -339,6 +339,7 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st lockedIssue?: Pick; /** The native run being dispatched; its own scheduled resume does not block it. */ resumingNativeRunId?: string | null; + recoveryHoldOnly?: boolean; }): Promise { const lockedIssue = options?.lockedIssue; const conversation = lockedIssue?.id === issueId && lockedIssue.companyId === companyId @@ -370,7 +371,7 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st eq(issueComments.id, conversation.conversationBoundaryCommentId), )).limit(1) : []; - const ownership = await getExecutionOwnershipBlocker(db, companyId, issueId); + const ownership = options?.recoveryHoldOnly ? null : await getExecutionOwnershipBlocker(db, companyId, issueId); if (ownership) return ownership; const [action] = restoreHold ? [restoreHold] : await db.select().from(issueRecoveryActions).where(and( eq(issueRecoveryActions.companyId, companyId),