fix(native): a governance pause Stop with no turn result stays a cancellation (#20186)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A budget hard stop pauses the company; the pause's lifecycle Stop
cancels the agents' running native turns
> - When a Codex turn is interrupted this way, it ends without a
semantic result
> - The native executor treated that missing result as a retryable
interruption, so it rewrote the cancelled run to `failed` and scheduled
an exact resume
> - The pause held that recovery, and Raise budget and resume continues
only runs that are still cancelled, so the task stayed blocked with no
new run
> - This pull request makes the executor treat an acknowledged
governance pause Stop like a person's Stop: the run stays cancelled
> - The benefit is that Raise budget and resume continues the task
(#20144), with no manual Retry

## Linked Issues or Issue Description

Refs #20144 (resume continues budget-stopped work), #20143 (governance
pause stops are not failures), #20113

**What happened?**
Restart-chaos wave w-20261109T051242-s9802 (H1, seed 9802), task t03
(Codex, fault K2). The company budget crossed its hard limit at
05:19:39.9. The pause's lifecycle Stop ("Agent lifecycle stop
requested") cancelled run 59e976bc at 05:19:40.05, and Codex
acknowledged the interrupt. The turn ended without a semantic result,
and at 05:19:41.27 the executor rewrote the run to `failed` /
`native_session_interrupted`, with an agent-owned `active_run_watchdog`
recovery (exact checkpoint resume). The task went to `blocked`. After
`raise_budget_and_resume` at 05:20:20, no run started for 14 minutes.

**Expected behavior**
A pause's Stop is intentional. The run stays cancelled with its pause
record, and resuming continues the task once.

**Steps to reproduce**
Pause a company for budget while a Codex native turn runs, so the Stop
interrupts the turn before it records a result. Before this change, the
run becomes `failed` with a scheduled exact resume instead of staying
`cancelled`.

## What Changed

- `native-session-executor.ts`: the no-semantic-result branch now also
accepts an acknowledged governance pause Stop (`isGovernancePauseStop`
with an acknowledged native Stop receipt) as an intentional stop. It
settles the coordinator as `native_retry_cancelled` and raises
`NativeCancellationPendingRecoveryError`, as it already does for a
person's Stop and for a reassignment. The run is not rewritten to
`failed`, and no recovery action is created.
- The Stop's cancellation record is written by its fence before
dispatch, so the check does not depend on whether the terminal write has
landed.

## Verification

- `native-session-executor.test.ts`: new cases for "Agent lifecycle stop
requested" with a `pauseStop`, and for "Cancelled due to budget pause".
Each was red before the fix (it reached the retryable-failure write) and
is green now. A control case shows that a system lifecycle Stop without
`pauseStop` (termination) still takes the retry path.
- Full `native-session-executor.test.ts`: 1309 passed. Server typecheck
(`typecheck-layers`, low-memory): no errors.
- Not run: the H1 chaos rerun. Phase 3 runs it after merge.

## Risks

- A governance pause Stop that lands during a turn no longer schedules
an exact resume of that turn. The task continues on resume through a new
Retry (#20144) instead.
- Termination and cleanup use the same lifecycle Stop reason. They are
excluded because only the pausing path records `pauseStop`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-09 00:42:25 -06:00
1 parent 7c65a2f479
commit 3964ee1fca
2 files changed
+108

No files matched your search

@@ -13556,6 +13556,101 @@ describe("native session bounded recovery", () => {
expect(state.upsertRecoveryAction).not.toHaveBeenCalled();
});
// H1 (restart chaos): a budget pause's lifecycle Stop interrupted a Codex
// turn, which ended without a semantic result. That Stop must stay a
// cancellation (resume continues it), never an interrupted-run retry the
// pause then strands.
it.each(["Agent lifecycle stop requested", "Cancelled due to budget pause"])("does not turn an acknowledged governance pause Stop (%s) into a retry", async (reason) => {
const updates: Array<{ table: unknown; values: Record<string, unknown> }> = [];
const stop: Record<string, unknown> = {};
state.execute.mockReset().mockImplementationOnce(async () => {
Object.assign(stop, {
cancellation: { reason, source: "control_plane", expected: true, initiator: { type: "system" },
recordedAt: new Date().toISOString() },
...(reason === "Agent lifecycle stop requested" ? { pauseStop: { holds: ["company_paused"], companyPauseReason: "budget" } } : {}),
nativeCancellation: {
schema: "paperclip.native-cancellation.v1", ...execution.binding, scope: "run", reasonCode: "cancellation_run_only",
dispatched: true, dispatchState: "acknowledged", intentAuditId: "intent", acknowledgementAuditId: "ack",
} });
throw new Error("native_finalization_missing: session returned no semantic result");
});
state.upsertRecoveryAction.mockClear();
await expect(executePaperclipNativeSession({
db: leaseDb(execution, {}, stop, updates), execution, runnerInstanceId: "governance-stop",
})).rejects.toThrow("native_cancellation_pending_recovery");
expect(updates.some(update => update.table === heartbeatRuns && update.values.status === "failed")).toBe(false);
expect(updates.some(update => update.table === nativeRunFinalizations && update.values.failureCode === "native_retry_cancelled")).toBe(true);
expect(state.upsertRecoveryAction).not.toHaveBeenCalled();
});
it("waits for an in-flight governance pause Stop to save its acknowledgement before judging a missing result", async () => {
let releaseCleanup!: () => void, endTurn!: () => void, opened!: () => void;
const turnEnded = new Promise<void>(resolve => { endTurn = resolve; });
const sessionOpen = new Promise<void>(resolve => { opened = resolve; });
state.cancel.mockReset().mockReturnValue({ cleanup: new Promise<void>(resolve => { releaseCleanup = resolve; }) });
state.execute.mockReset().mockImplementationOnce(async (options) => {
await options.onSession({ cancel: state.cancel });
opened();
await turnEnded;
throw new Error("native_finalization_missing: session returned no semantic result");
});
state.upsertRecoveryAction.mockClear();
const runResultJson: Record<string, unknown> = {};
const updates: Array<{ table: unknown; values: Record<string, unknown> }> = [];
const outcome = executePaperclipNativeSession({ db: leaseDb(execution, {}, runResultJson, updates), execution, runnerInstanceId: "pause-stop-in-flight" })
.catch(error => error);
// The pause's fence recorded its cancellation before dispatch.
const persistence = cancellationDb({ resultJson: {
cancellation: { reason: "Agent lifecycle stop requested", source: "control_plane", expected: true,
initiator: { type: "system" }, recordedAt: new Date().toISOString() },
pauseStop: { holds: ["company_paused"], companyPauseReason: "budget" } } });
const transaction = persistence.db.transaction.bind(persistence.db);
// The executor reads the same durable run row the Stop writes.
(persistence.db as unknown as { transaction: typeof transaction }).transaction = (async (operation: Parameters<typeof transaction>[0]) => {
const result = await transaction(operation);
Object.assign(runResultJson, persistence.getResultJson());
return result;
}) as typeof transaction;
await sessionOpen;
const stopping = cancelNativeSession(execution.binding.runId, "Agent lifecycle stop requested", { db: persistence.db, scope: "run" });
try {
await vi.waitFor(() => expect(state.cancel).toHaveBeenCalledOnce());
// The interrupted turn ends before the Stop saves its acknowledgement.
endTurn();
await new Promise(resolve => setTimeout(resolve, 20));
expect(updates.some(update => update.table === heartbeatRuns && update.values.status === "failed")).toBe(false);
releaseCleanup();
await stopping;
expect(await outcome).toMatchObject({ name: "NativeCancellationPendingRecoveryError" });
expect(updates.some(update => update.table === heartbeatRuns && update.values.status === "failed")).toBe(false);
expect(updates.some(update => update.table === nativeRunFinalizations && update.values.failureCode === "native_retry_cancelled")).toBe(true);
expect(state.upsertRecoveryAction).not.toHaveBeenCalled();
} finally { releaseCleanup(); endTurn(); await outcome; }
});
it("still retries a missing result after a system Stop that is not a governance pause", async () => {
const updates: Array<{ table: unknown; values: Record<string, unknown> }> = [];
const stop: Record<string, unknown> = {};
state.execute.mockReset().mockImplementationOnce(async () => {
Object.assign(stop, {
cancellation: { reason: "Agent lifecycle stop requested", source: "control_plane", expected: true,
initiator: { type: "system" }, recordedAt: new Date().toISOString() },
nativeCancellation: {
schema: "paperclip.native-cancellation.v1", ...execution.binding, scope: "run", reasonCode: "cancellation_run_only",
dispatched: true, dispatchState: "acknowledged", intentAuditId: "intent", acknowledgementAuditId: "ack",
} });
throw new Error("native_finalization_missing: session returned no semantic result");
});
state.upsertRecoveryAction.mockReset().mockResolvedValue({});
const service = vi.spyOn(issueServiceModule, "issueService").mockReturnValue({ update: vi.fn(async () => null) } as unknown as ReturnType<typeof issueServiceModule.issueService>);
try {
await expect(executePaperclipNativeSession({
db: leaseDb(execution, {}, stop, updates), execution, runnerInstanceId: "termination-stop",
})).rejects.not.toThrow("native_cancellation_pending_recovery");
expect(updates.some(update => update.table === nativeRunFinalizations && update.values.failureCode === "native_retry_cancelled")).toBe(false);
} finally { service.mockRestore(); }
});
it.each(["pending", "acknowledged"])("preserves a %s Stop when cancellation wins before the first turn", async (dispatchState) => {
const updates: Array<{ table: unknown; values: Record<string, unknown> }> = [];
const stop: Record<string, unknown> = {};
@@ -1,4 +1,5 @@
import { remoteRecoveryLeaseMismatch } from "./remote-recovery-lease-mismatch.js";
import { isGovernancePauseStop } from "../governance-pause-stop.js";
import { nativeLeaseHolder, notifyNativeTokenLeaseClaimed } from "./native-token-lease-settlement.js";
import { captureRemoteNativeProviderArtifacts } from "./native-provider-artifact-capture.js";
import { NativeSessionDiagnostics, nativeSessionDiagnostics, type NativeDiagnosticLease } from "./native-session-diagnostics.js";
@@ -11581,9 +11582,14 @@ async function executePaperclipNativeSessionWithinScope(
}
trace.activate(taskSettleScope);
liveQuestions.close();
const activeAtFailure = activeNativeSessions.get(input.execution.binding.runId);
activeNativeSessions.delete(input.execution.binding.runId);
clearSteeringDeliveries(input.execution.binding.runId);
clearNativeRuntimeRequestResolutions(input.execution.binding.runId);
// A Stop interrupts the provider before it saves its acknowledgement.
// When the interrupted turn ends first, wait for that Stop to settle, so
// the checks below judge its saved receipt, not a failure.
if (activeAtFailure?.cancelRequested) await activeAtFailure.cancellationSettled?.catch(() => undefined);
// Stop before paperclip_finish is normal. Bounded provider teardown has
// finished; a missing result must not overwrite cancellation or trigger
// another turn to perform completion bookkeeping.
@@ -11608,7 +11614,14 @@ async function executePaperclipNativeSessionWithinScope(
stopIntent.scope === "run" &&
typeof stopIntent.intentAuditId === "string" && stopIntent.intentAuditId.length > 0 &&
["pending", "acknowledged"].includes(String(stopIntent.dispatchState));
// A governance pause (budget, company or agent pause) Stop is the same
// kind of intentional stop. The fence records its cancellation before
// dispatch, so judge it whether or not the terminal write has landed;
// resume continues it (budget-resume-continuation.ts).
const governancePauseStop = !!stoppedRun && hasAcknowledgedNativeRunStopReceipt(stoppedRun)
&& isGovernancePauseStop({ status: "cancelled", resultJson: stoppedRun.resultJson });
if (stoppedRun && (pendingStartupStop || hasAcknowledgedNativeStopIntent(stoppedRun) || hasAcknowledgedNativeReassignmentStopIntent(stoppedRun)
|| governancePauseStop
|| (error instanceof NativeCancellationPendingRecoveryError && hasAcknowledgedNativeRunStopReceipt(stoppedRun)))) {
const [settled] = await writeExecutionControl(input.db, [DELIVERY_QUEUES.executionFinalization, DELIVERY_QUEUES.accounting], tx => tx.update(nativeRunFinalizations).set({
phase: "terminal_failure", failureCode: "native_retry_cancelled", nextAttemptAt: null,