mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 12:37:55 +01:00
fix(native): a governance pause Stop with no turn result stays a cancellation (#20186)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A budget hard stop pauses the company; the pause's lifecycle Stop cancels the agents' running native turns > - When a Codex turn is interrupted this way, it ends without a semantic result > - The native executor treated that missing result as a retryable interruption, so it rewrote the cancelled run to `failed` and scheduled an exact resume > - The pause held that recovery, and Raise budget and resume continues only runs that are still cancelled, so the task stayed blocked with no new run > - This pull request makes the executor treat an acknowledged governance pause Stop like a person's Stop: the run stays cancelled > - The benefit is that Raise budget and resume continues the task (#20144), with no manual Retry ## Linked Issues or Issue Description Refs #20144 (resume continues budget-stopped work), #20143 (governance pause stops are not failures), #20113 **What happened?** Restart-chaos wave w-20261109T051242-s9802 (H1, seed 9802), task t03 (Codex, fault K2). The company budget crossed its hard limit at 05:19:39.9. The pause's lifecycle Stop ("Agent lifecycle stop requested") cancelled run 59e976bc at 05:19:40.05, and Codex acknowledged the interrupt. The turn ended without a semantic result, and at 05:19:41.27 the executor rewrote the run to `failed` / `native_session_interrupted`, with an agent-owned `active_run_watchdog` recovery (exact checkpoint resume). The task went to `blocked`. After `raise_budget_and_resume` at 05:20:20, no run started for 14 minutes. **Expected behavior** A pause's Stop is intentional. The run stays cancelled with its pause record, and resuming continues the task once. **Steps to reproduce** Pause a company for budget while a Codex native turn runs, so the Stop interrupts the turn before it records a result. Before this change, the run becomes `failed` with a scheduled exact resume instead of staying `cancelled`. ## What Changed - `native-session-executor.ts`: the no-semantic-result branch now also accepts an acknowledged governance pause Stop (`isGovernancePauseStop` with an acknowledged native Stop receipt) as an intentional stop. It settles the coordinator as `native_retry_cancelled` and raises `NativeCancellationPendingRecoveryError`, as it already does for a person's Stop and for a reassignment. The run is not rewritten to `failed`, and no recovery action is created. - The Stop's cancellation record is written by its fence before dispatch, so the check does not depend on whether the terminal write has landed. ## Verification - `native-session-executor.test.ts`: new cases for "Agent lifecycle stop requested" with a `pauseStop`, and for "Cancelled due to budget pause". Each was red before the fix (it reached the retryable-failure write) and is green now. A control case shows that a system lifecycle Stop without `pauseStop` (termination) still takes the retry path. - Full `native-session-executor.test.ts`: 1309 passed. Server typecheck (`typecheck-layers`, low-memory): no errors. - Not run: the H1 chaos rerun. Phase 3 runs it after merge. ## Risks - A governance pause Stop that lands during a turn no longer schedules an exact resume of that turn. The task continues on resume through a new Retry (#20144) instead. - Termination and cleanup use the same lifecycle Stop reason. They are excluded because only the pausing path records `pauseStop`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7c65a2f479
commit
3964ee1fca
2 files changed
+108
No files matched your search
@@ -13556,6 +13556,101 @@ describe("native session bounded recovery", () => {
|
||||
expect(state.upsertRecoveryAction).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
// H1 (restart chaos): a budget pause's lifecycle Stop interrupted a Codex
|
||||
// turn, which ended without a semantic result. That Stop must stay a
|
||||
// cancellation (resume continues it), never an interrupted-run retry the
|
||||
// pause then strands.
|
||||
it.each(["Agent lifecycle stop requested", "Cancelled due to budget pause"])("does not turn an acknowledged governance pause Stop (%s) into a retry", async (reason) => {
|
||||
const updates: Array<{ table: unknown; values: Record<string, unknown> }> = [];
|
||||
const stop: Record<string, unknown> = {};
|
||||
state.execute.mockReset().mockImplementationOnce(async () => {
|
||||
Object.assign(stop, {
|
||||
cancellation: { reason, source: "control_plane", expected: true, initiator: { type: "system" },
|
||||
recordedAt: new Date().toISOString() },
|
||||
...(reason === "Agent lifecycle stop requested" ? { pauseStop: { holds: ["company_paused"], companyPauseReason: "budget" } } : {}),
|
||||
nativeCancellation: {
|
||||
schema: "paperclip.native-cancellation.v1", ...execution.binding, scope: "run", reasonCode: "cancellation_run_only",
|
||||
dispatched: true, dispatchState: "acknowledged", intentAuditId: "intent", acknowledgementAuditId: "ack",
|
||||
} });
|
||||
throw new Error("native_finalization_missing: session returned no semantic result");
|
||||
});
|
||||
state.upsertRecoveryAction.mockClear();
|
||||
await expect(executePaperclipNativeSession({
|
||||
db: leaseDb(execution, {}, stop, updates), execution, runnerInstanceId: "governance-stop",
|
||||
})).rejects.toThrow("native_cancellation_pending_recovery");
|
||||
expect(updates.some(update => update.table === heartbeatRuns && update.values.status === "failed")).toBe(false);
|
||||
expect(updates.some(update => update.table === nativeRunFinalizations && update.values.failureCode === "native_retry_cancelled")).toBe(true);
|
||||
expect(state.upsertRecoveryAction).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("waits for an in-flight governance pause Stop to save its acknowledgement before judging a missing result", async () => {
|
||||
let releaseCleanup!: () => void, endTurn!: () => void, opened!: () => void;
|
||||
const turnEnded = new Promise<void>(resolve => { endTurn = resolve; });
|
||||
const sessionOpen = new Promise<void>(resolve => { opened = resolve; });
|
||||
state.cancel.mockReset().mockReturnValue({ cleanup: new Promise<void>(resolve => { releaseCleanup = resolve; }) });
|
||||
state.execute.mockReset().mockImplementationOnce(async (options) => {
|
||||
await options.onSession({ cancel: state.cancel });
|
||||
opened();
|
||||
await turnEnded;
|
||||
throw new Error("native_finalization_missing: session returned no semantic result");
|
||||
});
|
||||
state.upsertRecoveryAction.mockClear();
|
||||
const runResultJson: Record<string, unknown> = {};
|
||||
const updates: Array<{ table: unknown; values: Record<string, unknown> }> = [];
|
||||
const outcome = executePaperclipNativeSession({ db: leaseDb(execution, {}, runResultJson, updates), execution, runnerInstanceId: "pause-stop-in-flight" })
|
||||
.catch(error => error);
|
||||
// The pause's fence recorded its cancellation before dispatch.
|
||||
const persistence = cancellationDb({ resultJson: {
|
||||
cancellation: { reason: "Agent lifecycle stop requested", source: "control_plane", expected: true,
|
||||
initiator: { type: "system" }, recordedAt: new Date().toISOString() },
|
||||
pauseStop: { holds: ["company_paused"], companyPauseReason: "budget" } } });
|
||||
const transaction = persistence.db.transaction.bind(persistence.db);
|
||||
// The executor reads the same durable run row the Stop writes.
|
||||
(persistence.db as unknown as { transaction: typeof transaction }).transaction = (async (operation: Parameters<typeof transaction>[0]) => {
|
||||
const result = await transaction(operation);
|
||||
Object.assign(runResultJson, persistence.getResultJson());
|
||||
return result;
|
||||
}) as typeof transaction;
|
||||
await sessionOpen;
|
||||
const stopping = cancelNativeSession(execution.binding.runId, "Agent lifecycle stop requested", { db: persistence.db, scope: "run" });
|
||||
try {
|
||||
await vi.waitFor(() => expect(state.cancel).toHaveBeenCalledOnce());
|
||||
// The interrupted turn ends before the Stop saves its acknowledgement.
|
||||
endTurn();
|
||||
await new Promise(resolve => setTimeout(resolve, 20));
|
||||
expect(updates.some(update => update.table === heartbeatRuns && update.values.status === "failed")).toBe(false);
|
||||
releaseCleanup();
|
||||
await stopping;
|
||||
expect(await outcome).toMatchObject({ name: "NativeCancellationPendingRecoveryError" });
|
||||
expect(updates.some(update => update.table === heartbeatRuns && update.values.status === "failed")).toBe(false);
|
||||
expect(updates.some(update => update.table === nativeRunFinalizations && update.values.failureCode === "native_retry_cancelled")).toBe(true);
|
||||
expect(state.upsertRecoveryAction).not.toHaveBeenCalled();
|
||||
} finally { releaseCleanup(); endTurn(); await outcome; }
|
||||
});
|
||||
|
||||
it("still retries a missing result after a system Stop that is not a governance pause", async () => {
|
||||
const updates: Array<{ table: unknown; values: Record<string, unknown> }> = [];
|
||||
const stop: Record<string, unknown> = {};
|
||||
state.execute.mockReset().mockImplementationOnce(async () => {
|
||||
Object.assign(stop, {
|
||||
cancellation: { reason: "Agent lifecycle stop requested", source: "control_plane", expected: true,
|
||||
initiator: { type: "system" }, recordedAt: new Date().toISOString() },
|
||||
nativeCancellation: {
|
||||
schema: "paperclip.native-cancellation.v1", ...execution.binding, scope: "run", reasonCode: "cancellation_run_only",
|
||||
dispatched: true, dispatchState: "acknowledged", intentAuditId: "intent", acknowledgementAuditId: "ack",
|
||||
} });
|
||||
throw new Error("native_finalization_missing: session returned no semantic result");
|
||||
});
|
||||
state.upsertRecoveryAction.mockReset().mockResolvedValue({});
|
||||
const service = vi.spyOn(issueServiceModule, "issueService").mockReturnValue({ update: vi.fn(async () => null) } as unknown as ReturnType<typeof issueServiceModule.issueService>);
|
||||
try {
|
||||
await expect(executePaperclipNativeSession({
|
||||
db: leaseDb(execution, {}, stop, updates), execution, runnerInstanceId: "termination-stop",
|
||||
})).rejects.not.toThrow("native_cancellation_pending_recovery");
|
||||
expect(updates.some(update => update.table === nativeRunFinalizations && update.values.failureCode === "native_retry_cancelled")).toBe(false);
|
||||
} finally { service.mockRestore(); }
|
||||
});
|
||||
|
||||
it.each(["pending", "acknowledged"])("preserves a %s Stop when cancellation wins before the first turn", async (dispatchState) => {
|
||||
const updates: Array<{ table: unknown; values: Record<string, unknown> }> = [];
|
||||
const stop: Record<string, unknown> = {};
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { remoteRecoveryLeaseMismatch } from "./remote-recovery-lease-mismatch.js";
|
||||
import { isGovernancePauseStop } from "../governance-pause-stop.js";
|
||||
import { nativeLeaseHolder, notifyNativeTokenLeaseClaimed } from "./native-token-lease-settlement.js";
|
||||
import { captureRemoteNativeProviderArtifacts } from "./native-provider-artifact-capture.js";
|
||||
import { NativeSessionDiagnostics, nativeSessionDiagnostics, type NativeDiagnosticLease } from "./native-session-diagnostics.js";
|
||||
@@ -11581,9 +11582,14 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
}
|
||||
trace.activate(taskSettleScope);
|
||||
liveQuestions.close();
|
||||
const activeAtFailure = activeNativeSessions.get(input.execution.binding.runId);
|
||||
activeNativeSessions.delete(input.execution.binding.runId);
|
||||
clearSteeringDeliveries(input.execution.binding.runId);
|
||||
clearNativeRuntimeRequestResolutions(input.execution.binding.runId);
|
||||
// A Stop interrupts the provider before it saves its acknowledgement.
|
||||
// When the interrupted turn ends first, wait for that Stop to settle, so
|
||||
// the checks below judge its saved receipt, not a failure.
|
||||
if (activeAtFailure?.cancelRequested) await activeAtFailure.cancellationSettled?.catch(() => undefined);
|
||||
// Stop before paperclip_finish is normal. Bounded provider teardown has
|
||||
// finished; a missing result must not overwrite cancellation or trigger
|
||||
// another turn to perform completion bookkeeping.
|
||||
@@ -11608,7 +11614,14 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
stopIntent.scope === "run" &&
|
||||
typeof stopIntent.intentAuditId === "string" && stopIntent.intentAuditId.length > 0 &&
|
||||
["pending", "acknowledged"].includes(String(stopIntent.dispatchState));
|
||||
// A governance pause (budget, company or agent pause) Stop is the same
|
||||
// kind of intentional stop. The fence records its cancellation before
|
||||
// dispatch, so judge it whether or not the terminal write has landed;
|
||||
// resume continues it (budget-resume-continuation.ts).
|
||||
const governancePauseStop = !!stoppedRun && hasAcknowledgedNativeRunStopReceipt(stoppedRun)
|
||||
&& isGovernancePauseStop({ status: "cancelled", resultJson: stoppedRun.resultJson });
|
||||
if (stoppedRun && (pendingStartupStop || hasAcknowledgedNativeStopIntent(stoppedRun) || hasAcknowledgedNativeReassignmentStopIntent(stoppedRun)
|
||||
|| governancePauseStop
|
||||
|| (error instanceof NativeCancellationPendingRecoveryError && hasAcknowledgedNativeRunStopReceipt(stoppedRun)))) {
|
||||
const [settled] = await writeExecutionControl(input.db, [DELIVERY_QUEUES.executionFinalization, DELIVERY_QUEUES.accounting], tx => tx.update(nativeRunFinalizations).set({
|
||||
phase: "terminal_failure", failureCode: "native_retry_cancelled", nextAttemptAt: null,
|
||||
|
||||
Reference in new issue
Block a user