From 3964ee1fcac4cdfb9981f1ddb82b3f4777189ebc Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 9 Nov 2026 00:42:25 -0600 Subject: [PATCH] fix(native): a governance pause Stop with no turn result stays a cancellation (#20186) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A budget hard stop pauses the company; the pause's lifecycle Stop cancels the agents' running native turns > - When a Codex turn is interrupted this way, it ends without a semantic result > - The native executor treated that missing result as a retryable interruption, so it rewrote the cancelled run to `failed` and scheduled an exact resume > - The pause held that recovery, and Raise budget and resume continues only runs that are still cancelled, so the task stayed blocked with no new run > - This pull request makes the executor treat an acknowledged governance pause Stop like a person's Stop: the run stays cancelled > - The benefit is that Raise budget and resume continues the task (#20144), with no manual Retry ## Linked Issues or Issue Description Refs #20144 (resume continues budget-stopped work), #20143 (governance pause stops are not failures), #20113 **What happened?** Restart-chaos wave w-20261109T051242-s9802 (H1, seed 9802), task t03 (Codex, fault K2). The company budget crossed its hard limit at 05:19:39.9. The pause's lifecycle Stop ("Agent lifecycle stop requested") cancelled run 59e976bc at 05:19:40.05, and Codex acknowledged the interrupt. The turn ended without a semantic result, and at 05:19:41.27 the executor rewrote the run to `failed` / `native_session_interrupted`, with an agent-owned `active_run_watchdog` recovery (exact checkpoint resume). The task went to `blocked`. After `raise_budget_and_resume` at 05:20:20, no run started for 14 minutes. **Expected behavior** A pause's Stop is intentional. The run stays cancelled with its pause record, and resuming continues the task once. **Steps to reproduce** Pause a company for budget while a Codex native turn runs, so the Stop interrupts the turn before it records a result. Before this change, the run becomes `failed` with a scheduled exact resume instead of staying `cancelled`. ## What Changed - `native-session-executor.ts`: the no-semantic-result branch now also accepts an acknowledged governance pause Stop (`isGovernancePauseStop` with an acknowledged native Stop receipt) as an intentional stop. It settles the coordinator as `native_retry_cancelled` and raises `NativeCancellationPendingRecoveryError`, as it already does for a person's Stop and for a reassignment. The run is not rewritten to `failed`, and no recovery action is created. - The Stop's cancellation record is written by its fence before dispatch, so the check does not depend on whether the terminal write has landed. ## Verification - `native-session-executor.test.ts`: new cases for "Agent lifecycle stop requested" with a `pauseStop`, and for "Cancelled due to budget pause". Each was red before the fix (it reached the retryable-failure write) and is green now. A control case shows that a system lifecycle Stop without `pauseStop` (termination) still takes the retry path. - Full `native-session-executor.test.ts`: 1309 passed. Server typecheck (`typecheck-layers`, low-memory): no errors. - Not run: the H1 chaos rerun. Phase 3 runs it after merge. ## Risks - A governance pause Stop that lands during a turn no longer schedules an exact resume of that turn. The task continues on resume through a new Retry (#20144) instead. - Termination and cleanup use the same lifecycle Stop reason. They are excluded because only the pausing path records `pauseStop`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../native-session-executor.test.ts | 95 +++++++++++++++++++ .../native-runtime/native-session-executor.ts | 13 +++ 2 files changed, 108 insertions(+) diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index c1b67f1737f..d96451efb43 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -13556,6 +13556,101 @@ describe("native session bounded recovery", () => { expect(state.upsertRecoveryAction).not.toHaveBeenCalled(); }); + // H1 (restart chaos): a budget pause's lifecycle Stop interrupted a Codex + // turn, which ended without a semantic result. That Stop must stay a + // cancellation (resume continues it), never an interrupted-run retry the + // pause then strands. + it.each(["Agent lifecycle stop requested", "Cancelled due to budget pause"])("does not turn an acknowledged governance pause Stop (%s) into a retry", async (reason) => { + const updates: Array<{ table: unknown; values: Record }> = []; + const stop: Record = {}; + state.execute.mockReset().mockImplementationOnce(async () => { + Object.assign(stop, { + cancellation: { reason, source: "control_plane", expected: true, initiator: { type: "system" }, + recordedAt: new Date().toISOString() }, + ...(reason === "Agent lifecycle stop requested" ? { pauseStop: { holds: ["company_paused"], companyPauseReason: "budget" } } : {}), + nativeCancellation: { + schema: "paperclip.native-cancellation.v1", ...execution.binding, scope: "run", reasonCode: "cancellation_run_only", + dispatched: true, dispatchState: "acknowledged", intentAuditId: "intent", acknowledgementAuditId: "ack", + } }); + throw new Error("native_finalization_missing: session returned no semantic result"); + }); + state.upsertRecoveryAction.mockClear(); + await expect(executePaperclipNativeSession({ + db: leaseDb(execution, {}, stop, updates), execution, runnerInstanceId: "governance-stop", + })).rejects.toThrow("native_cancellation_pending_recovery"); + expect(updates.some(update => update.table === heartbeatRuns && update.values.status === "failed")).toBe(false); + expect(updates.some(update => update.table === nativeRunFinalizations && update.values.failureCode === "native_retry_cancelled")).toBe(true); + expect(state.upsertRecoveryAction).not.toHaveBeenCalled(); + }); + + it("waits for an in-flight governance pause Stop to save its acknowledgement before judging a missing result", async () => { + let releaseCleanup!: () => void, endTurn!: () => void, opened!: () => void; + const turnEnded = new Promise(resolve => { endTurn = resolve; }); + const sessionOpen = new Promise(resolve => { opened = resolve; }); + state.cancel.mockReset().mockReturnValue({ cleanup: new Promise(resolve => { releaseCleanup = resolve; }) }); + state.execute.mockReset().mockImplementationOnce(async (options) => { + await options.onSession({ cancel: state.cancel }); + opened(); + await turnEnded; + throw new Error("native_finalization_missing: session returned no semantic result"); + }); + state.upsertRecoveryAction.mockClear(); + const runResultJson: Record = {}; + const updates: Array<{ table: unknown; values: Record }> = []; + const outcome = executePaperclipNativeSession({ db: leaseDb(execution, {}, runResultJson, updates), execution, runnerInstanceId: "pause-stop-in-flight" }) + .catch(error => error); + // The pause's fence recorded its cancellation before dispatch. + const persistence = cancellationDb({ resultJson: { + cancellation: { reason: "Agent lifecycle stop requested", source: "control_plane", expected: true, + initiator: { type: "system" }, recordedAt: new Date().toISOString() }, + pauseStop: { holds: ["company_paused"], companyPauseReason: "budget" } } }); + const transaction = persistence.db.transaction.bind(persistence.db); + // The executor reads the same durable run row the Stop writes. + (persistence.db as unknown as { transaction: typeof transaction }).transaction = (async (operation: Parameters[0]) => { + const result = await transaction(operation); + Object.assign(runResultJson, persistence.getResultJson()); + return result; + }) as typeof transaction; + await sessionOpen; + const stopping = cancelNativeSession(execution.binding.runId, "Agent lifecycle stop requested", { db: persistence.db, scope: "run" }); + try { + await vi.waitFor(() => expect(state.cancel).toHaveBeenCalledOnce()); + // The interrupted turn ends before the Stop saves its acknowledgement. + endTurn(); + await new Promise(resolve => setTimeout(resolve, 20)); + expect(updates.some(update => update.table === heartbeatRuns && update.values.status === "failed")).toBe(false); + releaseCleanup(); + await stopping; + expect(await outcome).toMatchObject({ name: "NativeCancellationPendingRecoveryError" }); + expect(updates.some(update => update.table === heartbeatRuns && update.values.status === "failed")).toBe(false); + expect(updates.some(update => update.table === nativeRunFinalizations && update.values.failureCode === "native_retry_cancelled")).toBe(true); + expect(state.upsertRecoveryAction).not.toHaveBeenCalled(); + } finally { releaseCleanup(); endTurn(); await outcome; } + }); + + it("still retries a missing result after a system Stop that is not a governance pause", async () => { + const updates: Array<{ table: unknown; values: Record }> = []; + const stop: Record = {}; + state.execute.mockReset().mockImplementationOnce(async () => { + Object.assign(stop, { + cancellation: { reason: "Agent lifecycle stop requested", source: "control_plane", expected: true, + initiator: { type: "system" }, recordedAt: new Date().toISOString() }, + nativeCancellation: { + schema: "paperclip.native-cancellation.v1", ...execution.binding, scope: "run", reasonCode: "cancellation_run_only", + dispatched: true, dispatchState: "acknowledged", intentAuditId: "intent", acknowledgementAuditId: "ack", + } }); + throw new Error("native_finalization_missing: session returned no semantic result"); + }); + state.upsertRecoveryAction.mockReset().mockResolvedValue({}); + const service = vi.spyOn(issueServiceModule, "issueService").mockReturnValue({ update: vi.fn(async () => null) } as unknown as ReturnType); + try { + await expect(executePaperclipNativeSession({ + db: leaseDb(execution, {}, stop, updates), execution, runnerInstanceId: "termination-stop", + })).rejects.not.toThrow("native_cancellation_pending_recovery"); + expect(updates.some(update => update.table === nativeRunFinalizations && update.values.failureCode === "native_retry_cancelled")).toBe(false); + } finally { service.mockRestore(); } + }); + it.each(["pending", "acknowledged"])("preserves a %s Stop when cancellation wins before the first turn", async (dispatchState) => { const updates: Array<{ table: unknown; values: Record }> = []; const stop: Record = {}; diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index ed4ffb7cae8..a908267d1c0 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -1,4 +1,5 @@ import { remoteRecoveryLeaseMismatch } from "./remote-recovery-lease-mismatch.js"; +import { isGovernancePauseStop } from "../governance-pause-stop.js"; import { nativeLeaseHolder, notifyNativeTokenLeaseClaimed } from "./native-token-lease-settlement.js"; import { captureRemoteNativeProviderArtifacts } from "./native-provider-artifact-capture.js"; import { NativeSessionDiagnostics, nativeSessionDiagnostics, type NativeDiagnosticLease } from "./native-session-diagnostics.js"; @@ -11581,9 +11582,14 @@ async function executePaperclipNativeSessionWithinScope( } trace.activate(taskSettleScope); liveQuestions.close(); + const activeAtFailure = activeNativeSessions.get(input.execution.binding.runId); activeNativeSessions.delete(input.execution.binding.runId); clearSteeringDeliveries(input.execution.binding.runId); clearNativeRuntimeRequestResolutions(input.execution.binding.runId); + // A Stop interrupts the provider before it saves its acknowledgement. + // When the interrupted turn ends first, wait for that Stop to settle, so + // the checks below judge its saved receipt, not a failure. + if (activeAtFailure?.cancelRequested) await activeAtFailure.cancellationSettled?.catch(() => undefined); // Stop before paperclip_finish is normal. Bounded provider teardown has // finished; a missing result must not overwrite cancellation or trigger // another turn to perform completion bookkeeping. @@ -11608,7 +11614,14 @@ async function executePaperclipNativeSessionWithinScope( stopIntent.scope === "run" && typeof stopIntent.intentAuditId === "string" && stopIntent.intentAuditId.length > 0 && ["pending", "acknowledged"].includes(String(stopIntent.dispatchState)); + // A governance pause (budget, company or agent pause) Stop is the same + // kind of intentional stop. The fence records its cancellation before + // dispatch, so judge it whether or not the terminal write has landed; + // resume continues it (budget-resume-continuation.ts). + const governancePauseStop = !!stoppedRun && hasAcknowledgedNativeRunStopReceipt(stoppedRun) + && isGovernancePauseStop({ status: "cancelled", resultJson: stoppedRun.resultJson }); if (stoppedRun && (pendingStartupStop || hasAcknowledgedNativeStopIntent(stoppedRun) || hasAcknowledgedNativeReassignmentStopIntent(stoppedRun) + || governancePauseStop || (error instanceof NativeCancellationPendingRecoveryError && hasAcknowledgedNativeRunStopReceipt(stoppedRun)))) { const [settled] = await writeExecutionControl(input.db, [DELIVERY_QUEUES.executionFinalization, DELIVERY_QUEUES.accounting], tx => tx.update(nativeRunFinalizations).set({ phase: "terminal_failure", failureCode: "native_retry_cancelled", nextAttemptAt: null,