From 763adbea09418be3c43ce1f296a3174da6ebc978 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 8 Nov 2026 20:55:36 -0600 Subject: [PATCH] fix(execution): a pause's stop never blocks the task behind an incident with no control (#20143) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A budget hard stop pauses the agent, which stops its running work > - In the restart-chaos H1 scenario, the stopped task went `blocked` behind a board action that said "inspect and reconcile" and offered no button, and it stayed there after the board resumed the budget > - A pause is an intentional stop, not a failure, so it must not leave the task with no way forward > - This pull request treats a pause's stop as intentional: the task keeps its status and no incident opens, so the normal Retry works > - The benefit is that a budget or agent pause never strands a task behind a control-less incident ## Linked Issues or Issue Description Refs #20113, #20101 **What happened?** Restart-chaos wave w-20261108T210118-s9802 (H1). At the budget pause, task CHA-1 went `blocked` (21:02:27.870) with an active board `active_run_watchdog` action, cause `native_continuation_requires_reconciliation`, next action "Inspect the original failure and reconcile the previous execution before continuing." After raise_budget_and_resume it stayed blocked, `canRetry=false`, with no Retry. The run was cancelled by the agent lifecycle pause (`cancellation: { reason: "Agent lifecycle stop requested", initiator: system, expected: true }`). The wake-queue release (`recordNativeTerminalRecoveryIfNeeded`) blocks every cancelled native run that is not an acknowledged user Stop. **Expected behavior** A pause's stop keeps the task's status and opens no incident; the task can be continued. **Steps to reproduce** Pause an agent (budget hold or company budget pause) while its native run is active. Before this change, the task is blocked behind a board action with no control. ## What Changed - New `server/src/services/governance-pause-stop.ts`: `isGovernancePauseStop(run)` matches an expected system cancellation whose reason is the budget pause stop, or the agent lifecycle stop with a `pauseStop` stamp. - `wake-queue/adapters/postgres.ts` `recordNativeTerminalRecoveryIfNeeded`: returns early for such a run, so the release keeps the task's status and opens no reconciliation incident. - `agent-lifecycle-driver.ts`: the `pausing` host step stamps `resultJson.pauseStop = { holds, companyPauseReason }` on the agent's active runs before stopping them (one update, company pause reason by subquery). Termination and cleanup share the lifecycle stop reason but write no stamp, so they still open the incident. ## Verification - `wake-queue/adapters/postgres.test.ts`: new cases. A run stopped by the lifecycle pause or by the budget pause keeps the task `in_progress` with no incident (both red on master). Agent termination, and another system stop (company archive), still open the incident. - `agent-lifecycle/__tests__/lifecycle.test.ts`: "records what a pause stopped each active run for". `driver-scope.test.ts` updated for the stamp's single update. - `wake-queue` (107), `lifecycle` (68+), `driver-scope`, `companies-service`: 199 passed. Server typecheck (`typecheck-layers`, low-memory): no errors. - Not run: the H1 chaos rerun (after the follow-up continuation PR). ## Risks - A native run that a pause stopped no longer opens a reconciliation incident. Resuming it is the normal Retry, which continues the conversation. Termination, archive and failures still open one. - One extra update per pause attempt on the agent's active runs. It is idempotent (skips runs that already have the stamp) and not on any turn path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../__tests__/driver-scope.test.ts | 9 +- .../__tests__/lifecycle.test.ts | 13 +++ .../wake-queue/adapters/postgres.test.ts | 90 ++++++++++++++++++- .../modules/wake-queue/adapters/postgres.ts | 13 +++ server/src/services/agent-lifecycle-driver.ts | 17 +++- server/src/services/governance-pause-stop.ts | 29 ++++++ 6 files changed, 165 insertions(+), 6 deletions(-) create mode 100644 server/src/services/governance-pause-stop.ts diff --git a/server/src/modules/agent-lifecycle/__tests__/driver-scope.test.ts b/server/src/modules/agent-lifecycle/__tests__/driver-scope.test.ts index 2f1999565b7..da514753201 100644 --- a/server/src/modules/agent-lifecycle/__tests__/driver-scope.test.ts +++ b/server/src/modules/agent-lifecycle/__tests__/driver-scope.test.ts @@ -8,10 +8,13 @@ import { createLifecycleDriver } from "../../../services/agent-lifecycle-driver. describe("lifecycle host stop scope", () => { it("pauses only the requested agent without loading its company peers", async () => { - const db = { select: vi.fn(() => { throw new Error("No peer lookup expected"); }) } as unknown as Db; + // The pause stamps its cause on the agent's own runs (one update), never a peer lookup. + const update = vi.fn(() => ({ set: () => ({ where: async () => [] }) })); + const db = { select: vi.fn(() => { throw new Error("No peer lookup expected"); }), update } as unknown as Db; const driver = createLifecycleDriver(db, {} as never); - expect(await driver.runHost({ id: "target", companyId: "company", lifecycleState: "pausing", - lifecycleVersion: 2, lifecycleRequiredPluginIds: [], lifecycleError: null, lifecycleOperation: null })).toBe("complete"); + expect(await driver.runHost({ id: "target", companyId: "company", lifecycleState: "pausing", lifecycleHolds: ["manual"], + lifecycleVersion: 2, lifecycleRequiredPluginIds: [], lifecycleError: null, lifecycleOperation: null } as never)).toBe("complete"); + expect(update).toHaveBeenCalledOnce(); // A pause keeps the messages it deferred (work-intent outbox). expect(stop).toHaveBeenCalledWith(["target"], "Agent lifecycle stop requested", { preservePauseDeferred: true }); expect(db.select).not.toHaveBeenCalled(); diff --git a/server/src/modules/agent-lifecycle/__tests__/lifecycle.test.ts b/server/src/modules/agent-lifecycle/__tests__/lifecycle.test.ts index e55f627582a..ea0b6ba6da7 100644 --- a/server/src/modules/agent-lifecycle/__tests__/lifecycle.test.ts +++ b/server/src/modules/agent-lifecycle/__tests__/lifecycle.test.ts @@ -371,6 +371,19 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu expect(await db.select().from(principalPermissionGrants).where(eq(principalPermissionGrants.companyId, companyId))).toHaveLength(0); }); + // H1 bug B: a pause's stop is told apart from termination by the cause the + // pause records on each run it stops. + it("records what a pause stopped each active run for", async () => { + const agent = await hire(); + await db.update(companies).set({ status: "paused", pauseReason: "budget" }).where(eq(companies.id, companyId)); + await db.update(agents).set({ lifecycleState: "pausing", lifecycleHolds: ["company_paused"] }).where(eq(agents.id, agent.id)); + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id, status: "queued" }).returning(); + await createLifecycleDriver(db, {} as never).runHost(await current(agent.id) as never); + const [stopped] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)); + expect(stopped.status).toBe("cancelled"); + expect(stopped.resultJson).toMatchObject({ pauseStop: { holds: ["company_paused"], companyPauseReason: "budget" } }); + }); + it("runs the saved harness test and keeps a failed configuration out of ready", async () => { const driver = createLifecycleDriver(db, {} as never); const work = worker(driver.runHost); diff --git a/server/src/modules/wake-queue/adapters/postgres.test.ts b/server/src/modules/wake-queue/adapters/postgres.test.ts index 831d8228d3d..761624744fa 100644 --- a/server/src/modules/wake-queue/adapters/postgres.test.ts +++ b/server/src/modules/wake-queue/adapters/postgres.test.ts @@ -3,7 +3,7 @@ import { DELIVERY_QUEUES } from "../../../services/delivery-work-notifications.j import { instanceSettingsService } from "../../../services/instance-settings.js"; import { randomBytes, randomUUID } from "node:crypto"; import { and, eq } from "drizzle-orm"; -import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; import type { Db } from "@paperclipai/db"; import { activityLog, @@ -754,6 +754,94 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => { expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wakeId)))[0].status).toBe("deferred_issue_execution"); }); + // H1 bug B: a budget pause stopped the agent's run, and the release blocked + // the task behind a reconciliation incident with no control. A pause is an + // intentional stop; resuming continues the task. + it.each(["Agent lifecycle stop requested", "Cancelled due to budget pause", "Cancelled due to agent pause"])( + "keeps a native task's status and opens no incident when a pause stopped its run (%s)", async (reason) => { + const companyId = await seedCompany(); + const agentId = await seedAgent({ companyId }); + const issueId = await seedIssue({ companyId, assigneeAgentId: agentId, status: "in_progress" }); + const runId = await seedRun({ companyId, agentId, status: "cancelled", contextSnapshot: { issueId } }); + await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issueId, errorCode: "cancelled", + resultJson: { cancellation: { reason, source: "control_plane", expected: true, initiator: { type: "system" }, + recordedAt: new Date().toISOString() }, + ...(reason === "Agent lifecycle stop requested" ? { pauseStop: { holds: ["budget"], companyPauseReason: null } } : {}) } }).where(eq(heartbeatRuns.id, runId)); + await db.update(issues).set({ executionRunId: runId, checkoutRunId: runId }).where(eq(issues.id, issueId)); + const adapter = createPostgresWakeQueueAdapter(db, stubDeps); + await adapter.withIssueExecutionLock({ companyId, runId, now: new Date() }, async () => undefined as never); + expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]).toMatchObject({ status: "in_progress" }); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0); + }); + + // Review: the full release (createReleaseIssueExecution), not only the + // native incident step, must leave a paused agent's task unblocked. + it.each(["native", "legacy"] as const)("keeps a %s task unblocked through the full release when its agent's pause stopped the run", async (runtimeMode) => { + const companyId = await seedCompany(); + const agentId = await seedAgent({ companyId }); + await db.update(agents).set({ status: "paused", lifecycleState: "pausing", lifecycleHolds: ["manual"] }).where(eq(agents.id, agentId)); + const issueId = await seedIssue({ companyId, assigneeAgentId: agentId, status: "in_progress" }); + const runId = await seedRun({ companyId, agentId, status: "cancelled", contextSnapshot: { issueId } }); + await db.update(heartbeatRuns).set({ runtimeMode, ...(runtimeMode === "native" ? { nativeIssueId: issueId } : {}), + errorCode: "cancelled", resultJson: { cancellation: { reason: "Agent lifecycle stop requested", source: "control_plane", + expected: true, initiator: { type: "system" } }, pauseStop: { holds: ["manual"], companyPauseReason: null } } }) + .where(eq(heartbeatRuns.id, runId)); + await db.update(issues).set({ executionRunId: runId, checkoutRunId: runId }).where(eq(issues.id, issueId)); + const escalateStrandedAssignedIssue = vi.fn(async () => undefined); + const release = createReleaseIssueExecution({ issueLock: createPostgresWakeQueueAdapter(db, stubDeps), + recovery: { escalateStrandedAssignedIssue, escalateStrandedRecoveryIssueInPlace: async () => undefined }, + }); + await release({ companyId, runId, now: new Date() }); + expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]).toMatchObject({ status: "in_progress" }); + expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0); + expect(escalateStrandedAssignedIssue).not.toHaveBeenCalled(); + }); + + it("still opens a reconciliation incident when termination replaced the pause that stamped the run", async () => { + const companyId = await seedCompany(); + const agentId = await seedAgent({ companyId }); + await db.update(agents).set({ lifecycleState: "terminating" }).where(eq(agents.id, agentId)); + const issueId = await seedIssue({ companyId, assigneeAgentId: agentId, status: "in_progress" }); + const runId = await seedRun({ companyId, agentId, status: "cancelled", contextSnapshot: { issueId } }); + await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issueId, errorCode: "cancelled", + resultJson: { cancellation: { reason: "Agent lifecycle stop requested", source: "control_plane", expected: true, + initiator: { type: "system" } }, pauseStop: { holds: ["budget"], companyPauseReason: null } } }) + .where(eq(heartbeatRuns.id, runId)); + const adapter = createPostgresWakeQueueAdapter(db, stubDeps); + await adapter.withIssueExecutionLock({ companyId, runId, now: new Date() }, async () => undefined as never); + expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)))[0]) + .toMatchObject({ cause: "native_continuation_requires_reconciliation" }); + }); + + it("still opens a reconciliation incident when agent termination stopped the run", async () => { + // Termination and cleanup use the lifecycle stop reason too, with no pauseStop. + const companyId = await seedCompany(); + const agentId = await seedAgent({ companyId }); + const issueId = await seedIssue({ companyId, assigneeAgentId: agentId, status: "in_progress" }); + const runId = await seedRun({ companyId, agentId, status: "cancelled", contextSnapshot: { issueId } }); + await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issueId, errorCode: "cancelled", + resultJson: { cancellation: { reason: "Agent lifecycle stop requested", source: "control_plane", + expected: true, initiator: { type: "system" } } } }).where(eq(heartbeatRuns.id, runId)); + const adapter = createPostgresWakeQueueAdapter(db, stubDeps); + await adapter.withIssueExecutionLock({ companyId, runId, now: new Date() }, async () => undefined as never); + expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)))[0]) + .toMatchObject({ cause: "native_continuation_requires_reconciliation" }); + }); + + it("still opens a reconciliation incident for a native run cancelled by another system stop", async () => { + const companyId = await seedCompany(); + const agentId = await seedAgent({ companyId }); + const issueId = await seedIssue({ companyId, assigneeAgentId: agentId, status: "in_progress" }); + const runId = await seedRun({ companyId, agentId, status: "cancelled", contextSnapshot: { issueId } }); + await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issueId, errorCode: "cancelled", + resultJson: { cancellation: { reason: "Cancelled because the company was archived", source: "control_plane", + expected: true, initiator: { type: "system" } } } }).where(eq(heartbeatRuns.id, runId)); + const adapter = createPostgresWakeQueueAdapter(db, stubDeps); + await adapter.withIssueExecutionLock({ companyId, runId, now: new Date() }, async () => undefined as never); + expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)))[0]) + .toMatchObject({ cause: "native_continuation_requires_reconciliation" }); + }); + it.each(["in_progress", "blocked"])("preserves recovery ownership and queued messages when a native task fails from %s", async (status) => { const companyId = await seedCompany(); const agentId = await seedAgent({ companyId }); diff --git a/server/src/modules/wake-queue/adapters/postgres.ts b/server/src/modules/wake-queue/adapters/postgres.ts index c3bbf5219a8..d59c9340a0a 100644 --- a/server/src/modules/wake-queue/adapters/postgres.ts +++ b/server/src/modules/wake-queue/adapters/postgres.ts @@ -17,6 +17,7 @@ import { getExecutionBlocker, hasPendingRunCleanup } from "../../../services/exe import { nativeManualAdmissionHoldsExecution } from "../../../services/native-runtime/native-runner-ownership.js"; import { and, asc, eq, inArray, isNull, notInArray, or, sql } from "drizzle-orm"; import { readReleaseRun, type ReleaseRun } from "./release-run-read.js"; +import { isGovernancePauseStop } from "../../../services/governance-pause-stop.js"; import type { Db } from "@paperclipai/db"; import { deliveredQueuedCommentIds } from "../../../services/queued-comment-delivery.js"; import { extractIssueReferenceIdentifiers, isUuidLike } from "@paperclipai/shared"; @@ -124,6 +125,9 @@ export function isOperatorStoppedRunRow(row: Pick = new Set([ + AGENT_LIFECYCLE_STOP_REASON, BUDGET_PAUSE_STOP_REASON, AGENT_PAUSE_STOP_REASON]); + +const record = (value: unknown): Record => + value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; + +export function isGovernancePauseStop(run: { status: string; resultJson?: Record | null }): boolean { + if (run.status !== "cancelled") return false; + const cancellation = record(run.resultJson?.cancellation); + if (record(cancellation.initiator).type !== "system" || cancellation.expected !== true + || typeof cancellation.reason !== "string" || !GOVERNANCE_PAUSE_STOP_REASONS.has(cancellation.reason)) return false; + // Termination and cleanup issue the same lifecycle stop reason. Only the + // pausing path records pauseStop, so require it for that reason. + return cancellation.reason !== AGENT_LIFECYCLE_STOP_REASON + || Object.keys(record(run.resultJson?.pauseStop)).length > 0; +}