fix(execution): a pause's stop never blocks the task behind an incident with no control (#20143)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A budget hard stop pauses the agent, which stops its running work
> - In the restart-chaos H1 scenario, the stopped task went `blocked`
behind a board action that said "inspect and reconcile" and offered no
button, and it stayed there after the board resumed the budget
> - A pause is an intentional stop, not a failure, so it must not leave
the task with no way forward
> - This pull request treats a pause's stop as intentional: the task
keeps its status and no incident opens, so the normal Retry works
> - The benefit is that a budget or agent pause never strands a task
behind a control-less incident

## Linked Issues or Issue Description

Refs #20113, #20101

**What happened?**
Restart-chaos wave w-20261108T210118-s9802 (H1). At the budget pause,
task CHA-1 went `blocked` (21:02:27.870) with an active board
`active_run_watchdog` action, cause
`native_continuation_requires_reconciliation`, next action "Inspect the
original failure and reconcile the previous execution before
continuing." After raise_budget_and_resume it stayed blocked,
`canRetry=false`, with no Retry. The run was cancelled by the agent
lifecycle pause (`cancellation: { reason: "Agent lifecycle stop
requested", initiator: system, expected: true }`). The wake-queue
release (`recordNativeTerminalRecoveryIfNeeded`) blocks every cancelled
native run that is not an acknowledged user Stop.

**Expected behavior**
A pause's stop keeps the task's status and opens no incident; the task
can be continued.

**Steps to reproduce**
Pause an agent (budget hold or company budget pause) while its native
run is active. Before this change, the task is blocked behind a board
action with no control.

## What Changed

- New `server/src/services/governance-pause-stop.ts`:
`isGovernancePauseStop(run)` matches an expected system cancellation
whose reason is the budget pause stop, or the agent lifecycle stop with
a `pauseStop` stamp.
- `wake-queue/adapters/postgres.ts`
`recordNativeTerminalRecoveryIfNeeded`: returns early for such a run, so
the release keeps the task's status and opens no reconciliation
incident.
- `agent-lifecycle-driver.ts`: the `pausing` host step stamps
`resultJson.pauseStop = { holds, companyPauseReason }` on the agent's
active runs before stopping them (one update, company pause reason by
subquery). Termination and cleanup share the lifecycle stop reason but
write no stamp, so they still open the incident.

## Verification

- `wake-queue/adapters/postgres.test.ts`: new cases. A run stopped by
the lifecycle pause or by the budget pause keeps the task `in_progress`
with no incident (both red on master). Agent termination, and another
system stop (company archive), still open the incident.
- `agent-lifecycle/__tests__/lifecycle.test.ts`: "records what a pause
stopped each active run for". `driver-scope.test.ts` updated for the
stamp's single update.
- `wake-queue` (107), `lifecycle` (68+), `driver-scope`,
`companies-service`: 199 passed. Server typecheck (`typecheck-layers`,
low-memory): no errors.
- Not run: the H1 chaos rerun (after the follow-up continuation PR).

## Risks

- A native run that a pause stopped no longer opens a reconciliation
incident. Resuming it is the normal Retry, which continues the
conversation. Termination, archive and failures still open one.
- One extra update per pause attempt on the agent's active runs. It is
idempotent (skips runs that already have the stamp) and not on any turn
path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/modules/agent-lifecycle/__tests__/driver-scope.test.ts

@@ -8,10 +8,13 @@ import { createLifecycleDriver } from "../../../services/agent-lifecycle-driver.
 
 describe("lifecycle host stop scope", () => {
   it("pauses only the requested agent without loading its company peers", async () => {
-    const db = { select: vi.fn(() => { throw new Error("No peer lookup expected"); }) } as unknown as Db;
+    // The pause stamps its cause on the agent's own runs (one update), never a peer lookup.
+    const update = vi.fn(() => ({ set: () => ({ where: async () => [] }) }));
+    const db = { select: vi.fn(() => { throw new Error("No peer lookup expected"); }), update } as unknown as Db;
     const driver = createLifecycleDriver(db, {} as never);
-    expect(await driver.runHost({ id: "target", companyId: "company", lifecycleState: "pausing",
-      lifecycleVersion: 2, lifecycleRequiredPluginIds: [], lifecycleError: null, lifecycleOperation: null })).toBe("complete");
+    expect(await driver.runHost({ id: "target", companyId: "company", lifecycleState: "pausing", lifecycleHolds: ["manual"],
+      lifecycleVersion: 2, lifecycleRequiredPluginIds: [], lifecycleError: null, lifecycleOperation: null } as never)).toBe("complete");
+    expect(update).toHaveBeenCalledOnce();
     // A pause keeps the messages it deferred (work-intent outbox).
     expect(stop).toHaveBeenCalledWith(["target"], "Agent lifecycle stop requested", { preservePauseDeferred: true });
     expect(db.select).not.toHaveBeenCalled();

server/src/modules/agent-lifecycle/__tests__/lifecycle.test.ts

@@ -371,6 +371,19 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu
     expect(await db.select().from(principalPermissionGrants).where(eq(principalPermissionGrants.companyId, companyId))).toHaveLength(0);
   });
 
+  // H1 bug B: a pause's stop is told apart from termination by the cause the
+  // pause records on each run it stops.
+  it("records what a pause stopped each active run for", async () => {
+    const agent = await hire();
+    await db.update(companies).set({ status: "paused", pauseReason: "budget" }).where(eq(companies.id, companyId));
+    await db.update(agents).set({ lifecycleState: "pausing", lifecycleHolds: ["company_paused"] }).where(eq(agents.id, agent.id));
+    const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id, status: "queued" }).returning();
+    await createLifecycleDriver(db, {} as never).runHost(await current(agent.id) as never);
+    const [stopped] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
+    expect(stopped.status).toBe("cancelled");
+    expect(stopped.resultJson).toMatchObject({ pauseStop: { holds: ["company_paused"], companyPauseReason: "budget" } });
+  });
+
   it("runs the saved harness test and keeps a failed configuration out of ready", async () => {
     const driver = createLifecycleDriver(db, {} as never);
     const work = worker(driver.runHost);

server/src/modules/wake-queue/adapters/postgres.test.ts

@@ -3,7 +3,7 @@ import { DELIVERY_QUEUES } from "../../../services/delivery-work-notifications.j
 import { instanceSettingsService } from "../../../services/instance-settings.js";
 import { randomBytes, randomUUID } from "node:crypto";
 import { and, eq } from "drizzle-orm";
-import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
+import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
 import type { Db } from "@paperclipai/db";
 import {
   activityLog,
@@ -754,6 +754,94 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => {
     expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wakeId)))[0].status).toBe("deferred_issue_execution");
   });
 
+  // H1 bug B: a budget pause stopped the agent's run, and the release blocked
+  // the task behind a reconciliation incident with no control. A pause is an
+  // intentional stop; resuming continues the task.
+  it.each(["Agent lifecycle stop requested", "Cancelled due to budget pause", "Cancelled due to agent pause"])(
+    "keeps a native task's status and opens no incident when a pause stopped its run (%s)", async (reason) => {
+    const companyId = await seedCompany();
+    const agentId = await seedAgent({ companyId });
+    const issueId = await seedIssue({ companyId, assigneeAgentId: agentId, status: "in_progress" });
+    const runId = await seedRun({ companyId, agentId, status: "cancelled", contextSnapshot: { issueId } });
+    await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issueId, errorCode: "cancelled",
+      resultJson: { cancellation: { reason, source: "control_plane", expected: true, initiator: { type: "system" },
+        recordedAt: new Date().toISOString() },
+        ...(reason === "Agent lifecycle stop requested" ? { pauseStop: { holds: ["budget"], companyPauseReason: null } } : {}) } }).where(eq(heartbeatRuns.id, runId));
+    await db.update(issues).set({ executionRunId: runId, checkoutRunId: runId }).where(eq(issues.id, issueId));
+    const adapter = createPostgresWakeQueueAdapter(db, stubDeps);
+    await adapter.withIssueExecutionLock({ companyId, runId, now: new Date() }, async () => undefined as never);
+    expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]).toMatchObject({ status: "in_progress" });
+    expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0);
+  });
+
+  // Review: the full release (createReleaseIssueExecution), not only the
+  // native incident step, must leave a paused agent's task unblocked.
+  it.each(["native", "legacy"] as const)("keeps a %s task unblocked through the full release when its agent's pause stopped the run", async (runtimeMode) => {
+    const companyId = await seedCompany();
+    const agentId = await seedAgent({ companyId });
+    await db.update(agents).set({ status: "paused", lifecycleState: "pausing", lifecycleHolds: ["manual"] }).where(eq(agents.id, agentId));
+    const issueId = await seedIssue({ companyId, assigneeAgentId: agentId, status: "in_progress" });
+    const runId = await seedRun({ companyId, agentId, status: "cancelled", contextSnapshot: { issueId } });
+    await db.update(heartbeatRuns).set({ runtimeMode, ...(runtimeMode === "native" ? { nativeIssueId: issueId } : {}),
+      errorCode: "cancelled", resultJson: { cancellation: { reason: "Agent lifecycle stop requested", source: "control_plane",
+        expected: true, initiator: { type: "system" } }, pauseStop: { holds: ["manual"], companyPauseReason: null } } })
+      .where(eq(heartbeatRuns.id, runId));
+    await db.update(issues).set({ executionRunId: runId, checkoutRunId: runId }).where(eq(issues.id, issueId));
+    const escalateStrandedAssignedIssue = vi.fn(async () => undefined);
+    const release = createReleaseIssueExecution({ issueLock: createPostgresWakeQueueAdapter(db, stubDeps),
+      recovery: { escalateStrandedAssignedIssue, escalateStrandedRecoveryIssueInPlace: async () => undefined },
+    });
+    await release({ companyId, runId, now: new Date() });
+    expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]).toMatchObject({ status: "in_progress" });
+    expect(await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))).toHaveLength(0);
+    expect(escalateStrandedAssignedIssue).not.toHaveBeenCalled();
+  });
+
+  it("still opens a reconciliation incident when termination replaced the pause that stamped the run", async () => {
+    const companyId = await seedCompany();
+    const agentId = await seedAgent({ companyId });
+    await db.update(agents).set({ lifecycleState: "terminating" }).where(eq(agents.id, agentId));
+    const issueId = await seedIssue({ companyId, assigneeAgentId: agentId, status: "in_progress" });
+    const runId = await seedRun({ companyId, agentId, status: "cancelled", contextSnapshot: { issueId } });
+    await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issueId, errorCode: "cancelled",
+      resultJson: { cancellation: { reason: "Agent lifecycle stop requested", source: "control_plane", expected: true,
+        initiator: { type: "system" } }, pauseStop: { holds: ["budget"], companyPauseReason: null } } })
+      .where(eq(heartbeatRuns.id, runId));
+    const adapter = createPostgresWakeQueueAdapter(db, stubDeps);
+    await adapter.withIssueExecutionLock({ companyId, runId, now: new Date() }, async () => undefined as never);
+    expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)))[0])
+      .toMatchObject({ cause: "native_continuation_requires_reconciliation" });
+  });
+
+  it("still opens a reconciliation incident when agent termination stopped the run", async () => {
+    // Termination and cleanup use the lifecycle stop reason too, with no pauseStop.
+    const companyId = await seedCompany();
+    const agentId = await seedAgent({ companyId });
+    const issueId = await seedIssue({ companyId, assigneeAgentId: agentId, status: "in_progress" });
+    const runId = await seedRun({ companyId, agentId, status: "cancelled", contextSnapshot: { issueId } });
+    await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issueId, errorCode: "cancelled",
+      resultJson: { cancellation: { reason: "Agent lifecycle stop requested", source: "control_plane",
+        expected: true, initiator: { type: "system" } } } }).where(eq(heartbeatRuns.id, runId));
+    const adapter = createPostgresWakeQueueAdapter(db, stubDeps);
+    await adapter.withIssueExecutionLock({ companyId, runId, now: new Date() }, async () => undefined as never);
+    expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)))[0])
+      .toMatchObject({ cause: "native_continuation_requires_reconciliation" });
+  });
+
+  it("still opens a reconciliation incident for a native run cancelled by another system stop", async () => {
+    const companyId = await seedCompany();
+    const agentId = await seedAgent({ companyId });
+    const issueId = await seedIssue({ companyId, assigneeAgentId: agentId, status: "in_progress" });
+    const runId = await seedRun({ companyId, agentId, status: "cancelled", contextSnapshot: { issueId } });
+    await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: issueId, errorCode: "cancelled",
+      resultJson: { cancellation: { reason: "Cancelled because the company was archived", source: "control_plane",
+        expected: true, initiator: { type: "system" } } } }).where(eq(heartbeatRuns.id, runId));
+    const adapter = createPostgresWakeQueueAdapter(db, stubDeps);
+    await adapter.withIssueExecutionLock({ companyId, runId, now: new Date() }, async () => undefined as never);
+    expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)))[0])
+      .toMatchObject({ cause: "native_continuation_requires_reconciliation" });
+  });
+
   it.each(["in_progress", "blocked"])("preserves recovery ownership and queued messages when a native task fails from %s", async (status) => {
     const companyId = await seedCompany();
     const agentId = await seedAgent({ companyId });

server/src/modules/wake-queue/adapters/postgres.ts

@@ -17,6 +17,7 @@ import { getExecutionBlocker, hasPendingRunCleanup } from "../../../services/exe
 import { nativeManualAdmissionHoldsExecution } from "../../../services/native-runtime/native-runner-ownership.js";
 import { and, asc, eq, inArray, isNull, notInArray, or, sql } from "drizzle-orm";
 import { readReleaseRun, type ReleaseRun } from "./release-run-read.js";
+import { isGovernancePauseStop } from "../../../services/governance-pause-stop.js";
 import type { Db } from "@paperclipai/db";
 import { deliveredQueuedCommentIds } from "../../../services/queued-comment-delivery.js";
 import { extractIssueReferenceIdentifiers, isUuidLike } from "@paperclipai/shared";
@@ -124,6 +125,9 @@ export function isOperatorStoppedRunRow(row: Pick<HeartbeatRunRow, "status" | "e
   const cancellation = readRunCancellation(row.resultJson);
   if (cancellation?.source === "queued_message") return false;
   if (cancellation?.source === "operator" && cancellation.expected) return true;
+  // A pause's stop (budget, agent or lifecycle pause) is intentional too: the
+  // release must not treat the paused agent's task as stranded work.
+  if (isGovernancePauseStop(row)) return true;
   if (row.errorCode && OPERATOR_STOP_ERROR_CODES.has(row.errorCode)) return true;
   const actorType = parseObject(row.resultJson).cancelledByActorType;
   return actorType === "user" || actorType === "board";
@@ -780,6 +784,15 @@ async function recordNativeTerminalRecoveryIfNeeded(tx: Db, run: ReleaseRun, iss
     issue.assigneeAgentId === run.agentId &&
     !["done", "cancelled"].includes(issue.status);
   if (!applies || isAcknowledgedNativeStop(run) || isAcknowledgedNativeReassignmentStop(run)) return false;
+  // A pause's stop is intentional: the task keeps its status and resuming
+  // continues it. It is never a reconciliation incident with no control. A
+  // termination that replaced the pause shares its stop reason, so the agent
+  // must still be pausable, not terminating or terminated.
+  if (isGovernancePauseStop(run)) {
+    const [agent] = await tx.select({ lifecycleState: agents.lifecycleState }).from(agents)
+      .where(and(eq(agents.id, run.agentId), eq(agents.companyId, issue.companyId)));
+    if (agent && !["terminating", "cleaning_up", "terminated", "rejected"].includes(agent.lifecycleState)) return false;
+  }
 
   const existing = await tx
     .select({ id: issueRecoveryActions.id, evidence: issueRecoveryActions.evidence })

server/src/services/agent-lifecycle-driver.ts

@@ -1,6 +1,7 @@
-import { and, eq, inArray } from "drizzle-orm";
+import { and, eq, inArray, sql } from "drizzle-orm";
+import { CANCELLABLE_HEARTBEAT_RUN_STATUSES } from "./heartbeat/recovery.js";
 import { promoteDeferredWorkIntents } from "./work-intent-outbox.js";
-import { agents, environments, plugins, pluginCompanySettings, type Db } from "@paperclipai/db";
+import { agents, companies, environments, heartbeatRuns, plugins, pluginCompanySettings, type Db } from "@paperclipai/db";
 import { instanceSettingsService } from "./instance-settings.js";
 import { pluginHasAgentEnvironments, pluginServesAgentEnvironment } from "./plugin-environment-driver.js";
 import { heartbeatService } from "./heartbeat.js";
@@ -62,6 +63,18 @@ export function createLifecycleDriver(db: Db, manager: PluginWorkerManager): Lif
             await db.select().from(agents).where(eq(agents.companyId, agent.companyId)))];
         const reason = agent.lifecycleHolds?.includes("company_archived")
           ? "Cancelled because the company was archived" : "Agent lifecycle stop requested";
+        // A pause records why it stopped each run (pauseStop). It marks the
+        // stop as a pause, not termination or cleanup, which share this reason.
+        if (agent.lifecycleState === "pausing") {
+          const pauseStop = sql`jsonb_build_object('pauseStop', jsonb_build_object(
+            'holds', ${JSON.stringify(agent.lifecycleHolds ?? [])}::jsonb,
+            'companyPauseReason', (select ${companies.pauseReason} from ${companies} where ${companies.id} = ${agent.companyId})))`;
+          await db.update(heartbeatRuns).set({
+            resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${pauseStop}`,
+          }).where(and(eq(heartbeatRuns.companyId, agent.companyId), eq(heartbeatRuns.agentId, agent.id),
+            inArray(heartbeatRuns.status, [...CANCELLABLE_HEARTBEAT_RUN_STATUSES]),
+            sql`not (coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) ? 'pauseStop')`));
+        }
         if (!await heartbeat.stopInvocationsForAgents(ids, reason,
           { preservePauseDeferred: agent.lifecycleState === "pausing" })) return "pending";
       }

server/src/services/governance-pause-stop.ts

@@ -0,0 +1,29 @@
+/**
+ * A governance pause stop: the control plane cancelled the run because its
+ * agent was paused (the direct pause route, or the agent lifecycle stop that a
+ * budget, company or manual pause issues) or because a budget hard stop fenced it. The stop is
+ * intentional, not a failure: the task keeps its status, and resuming
+ * continues the work (budget-resume-continuation.ts). It never becomes a
+ * reconciliation incident that has no control.
+ */
+export const AGENT_LIFECYCLE_STOP_REASON = "Agent lifecycle stop requested";
+export const BUDGET_PAUSE_STOP_REASON = "Cancelled due to budget pause";
+/** The direct pause route stops the agent's runs with this reason before the lifecycle worker runs. */
+export const AGENT_PAUSE_STOP_REASON = "Cancelled due to agent pause";
+
+const GOVERNANCE_PAUSE_STOP_REASONS: ReadonlySet<string> = new Set([
+  AGENT_LIFECYCLE_STOP_REASON, BUDGET_PAUSE_STOP_REASON, AGENT_PAUSE_STOP_REASON]);
+
+const record = (value: unknown): Record<string, unknown> =>
+  value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
+
+export function isGovernancePauseStop(run: { status: string; resultJson?: Record<string, unknown> | null }): boolean {
+  if (run.status !== "cancelled") return false;
+  const cancellation = record(run.resultJson?.cancellation);
+  if (record(cancellation.initiator).type !== "system" || cancellation.expected !== true
+    || typeof cancellation.reason !== "string" || !GOVERNANCE_PAUSE_STOP_REASONS.has(cancellation.reason)) return false;
+  // Termination and cleanup issue the same lifecycle stop reason. Only the
+  // pausing path records pauseStop, so require it for that reason.
+  return cancellation.reason !== AGENT_LIFECYCLE_STOP_REASON
+    || Object.keys(record(run.resultJson?.pauseStop)).length > 0;
+}