From 06a4a07781494b7d98dc500d0357b39aad997ea9 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Fri, 13 Nov 2026 13:47:52 -0800 Subject: [PATCH] fix(native): settle a Stop seen at budget admission as that cancellation (#21483) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs prepare a sandbox, then reserve budget, then start provider work > - An operator can press Stop while a native run still prepares > - Budget admission reads the recorded Stop on the locked run row and throws a generic 409 "Run cancellation already requested" > - Heartbeat treats that throw as a run failure, so an expected Stop becomes a failed run and a `run_failure` error event > - This pull request makes admission report the Stop through the existing native startup Stop contract > - The benefit is that a Stop during preparation ends as a cancellation, with no false failure, no retry, and no stranded Stop intent ## Linked Issues or Issue Description **What happened?** An operator stopped a native run while it prepared. `reserveRunBudget` found `resultJson.cancellation` / `startupCancellation` on the locked run row and threw `conflict("Run cancellation already requested")`. Heartbeat sent this error to `failRun`. The run ended `failed` with `adapter_failed` or `native_continuation_requires_reconciliation`, and Sentry got a `run_failure` event with `cancellationExpected: true`. The pending native Stop intent stayed `pending`. Restart recovery then logged `native_stop_requires_existing_remote_session` for a session that never existed. Two staging Sentry groups show this: 20 events since 2026-11-10. **Expected behavior** A Stop that the server records before provider work starts ends the run as `cancelled`. It does not report a failure, it does not schedule a retry, and it acknowledges the Stop intent. **Steps to reproduce** 1. Start a native run on a task. 2. Press Stop while the run prepares, before budget admission. 3. See that the run ends `failed` with the error "Run cancellation already requested". No public issue or pull request covers this. A GitHub search for "cancellation already requested" found no related fix. ## What Changed - `reserveRunBudget` now throws the existing `NativeCancellationPendingRecoveryError` for a native run when it finds a recorded Stop. Heartbeat already settles this error through `cancelRunInternal("Recovered durable native run cancellation")`. This is the same path as a Stop that native startup sees when the session opens. - Legacy runs keep the 409 conflict. The incident evidence is for native runs only. - The final dispatch gate does not change. It already treats a recorded Stop as a lost race, not as a failure. - New regression file `server/src/__tests__/heartbeat-native-stop-before-admission.test.ts`. ## Verification - `npx vitest run src/__tests__/heartbeat-native-stop-before-admission.test.ts` (in `server/`, embedded Postgres): - Before the fix: the test "settles an operator Stop recorded before budget admission" fails. The run ends `status: "failed"`, `error: "Run cancellation already requested"`. This is the staging result. - After the fix: 2/2 pass. The run ends `cancelled`, the provider session never opens, no retry run exists, any task hold records `originalFailureCode: "cancelled"`, and the native Stop intent becomes `acknowledged`. - The second test is a safety case. A real `heartbeat.cancelRun` that settles during admission still ends `cancelled`, and the Stop request resolves. - Focused suites pass, 13 files and 304 tests: native pre-dispatch retry, admission and cleanup, budget enforcement scoping, closed native receipt accounting, native recovery lease reacquire, native cleanup admission, native startup cleanup recovery, native cancellation request, native Stop pending, run-dispatch Postgres adapter, companies service, execution preparation timing. - `pnpm typecheck` in `server/` passes. - `pnpm verify` selects a broad set because the change is under heartbeat. CI supplies that coverage. - Not verified: a live staging Stop. Staging gets this fix after merge through the normal automatic deploy. ## Risks - Low risk. The change applies only when admission finds a recorded Stop on a native run. Before this change that run always failed. - `budget-reservations.ts` now imports an error class from `native-session-executor.ts`. The executor does not import `budget-reservations`, so this adds no import cycle. - A user Stop with `dispatched: false` can still put an `active_run_watchdog` hold on the task. That is existing behavior for Stops that reach no provider (#18820 keeps the strict dispatched receipt). This PR does not change it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- ...tbeat-native-stop-before-admission.test.ts | 156 ++++++++++++++++++ server/src/services/budget-reservations.ts | 6 +- 2 files changed, 161 insertions(+), 1 deletion(-) create mode 100644 server/src/__tests__/heartbeat-native-stop-before-admission.test.ts diff --git a/server/src/__tests__/heartbeat-native-stop-before-admission.test.ts b/server/src/__tests__/heartbeat-native-stop-before-admission.test.ts new file mode 100644 index 00000000000..b06a12a1424 --- /dev/null +++ b/server/src/__tests__/heartbeat-native-stop-before-admission.test.ts @@ -0,0 +1,156 @@ +import { randomUUID } from "node:crypto"; +import { and, eq, sql } from "drizzle-orm"; +import { activityLog, agents, authUsers, companyMemberships, createDb, heartbeatRuns, issueComments, issueRecoveryActions, + issues } from "@paperclipai/db"; +import { describe, expect, it, vi } from "vitest"; +import { heartbeatService, type HeartbeatServiceOptions } from "../services/heartbeat.js"; +import { instanceSettingsService } from "../services/instance-settings.js"; +import { closeIdleWarmNativeSessionsForRestart } from "../services/native-runtime/native-session-executor.js"; +import { nativeRestartFixture, waitForCondition } from "./helpers/native-restart-fixture.js"; +import { getRecoveryPostgresSupport } from "./helpers/recovery-postgres-fixture.js"; + +const admission = vi.hoisted(() => ({ beforeReserve: undefined as ((runId: string) => Promise) | undefined })); +vi.mock("../services/budget-reservations.js", async importOriginal => { + const actual = await importOriginal(); + return { + ...actual, + reserveRunBudget: async (...args: Parameters) => { + await admission.beforeReserve?.(args[2]); + return actual.reserveRunBudget(...args); + }, + }; +}); + +const support = await getRecoveryPostgresSupport({}); + +function nativeBackend(db: ReturnType, opened: string[]): + NonNullable { + const capabilities = { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }; + return execution => ({ + descriptor: async () => { + const [run] = await db.select({ runnerInstanceId: heartbeatRuns.runnerInstanceId }).from(heartbeatRuns) + .where(eq(heartbeatRuns.id, execution.binding.runId)); + if (!run?.runnerInstanceId) throw new Error("Normal native admission did not persist a runner identity"); + return { kind: "mock", name: "native-stop-before-admission", version: "1", capabilities, + runtimeContextCapabilities: { instructions: "native", skills: "native", mcp: "native" } }; + }, + openSession: async () => { + opened.push(execution.binding.runId); + throw new Error("A stopped run opened a provider session"); + }, + recoverSession: async () => { + opened.push(execution.binding.runId); + throw new Error("A stopped run recovered a provider session"); + }, + }); +} + +describe.skipIf(!support.supported)("heartbeat native Stop before budget admission", () => { + const fixture = nativeRestartFixture({}); + + async function startTask(db: ReturnType, heartbeat: ReturnType) { + const { companyId, agentId } = fixture.current; + const userId = randomUUID(); + await db.insert(authUsers).values({ id: userId, name: "Task owner", email: `${userId}@example.invalid`, + createdAt: new Date(), updatedAt: new Date() }); + await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, membershipRole: "owner" }); + await db.update(agents).set({ status: "idle", adapterConfig: { provider: "codex" } }).where(eq(agents.id, agentId)); + await instanceSettingsService(db).updateExperimental({ enableAgentChat: true, enableNativeRunner: true }); + const [issue] = await db.insert(issues).values({ companyId, title: "Stop while preparing", status: "in_progress", + assigneeAgentId: agentId, responsibleUserId: userId, conversationAgentId: agentId, conversationUserId: userId, + conversationState: "active" }).returning(); + const [message] = await db.insert(issueComments).values({ companyId, issueId: issue!.id, + authorType: "user", authorUserId: userId, body: "Start." }).returning(); + const run = await heartbeat.wakeup(agentId, { source: "on_demand", triggerDetail: "manual", reason: "issue_commented", + requestedByActorType: "user", requestedByActorId: userId, manualUserWake: true, + payload: { issueId: issue!.id, commentId: message!.id }, + contextSnapshot: { issueId: issue!.id, taskKey: issue!.id, wakeCommentId: message!.id } }); + expect(run).not.toBeNull(); + await waitForCondition("the run to settle", () => heartbeat.getTaskDrainStatus().quiescent); + return { issueId: issue!.id, runId: run!.id }; + } + + async function expectSettledAsStop(db: ReturnType, runId: string, issueId: string, opened: string[], + reason: string) { + const { companyId } = fixture.current; + await waitForCondition("the stopped run to settle", async () => { + const [current] = await db.select({ status: heartbeatRuns.status }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + return current?.status !== "running" && current?.status !== "queued"; + }); + const [stopped] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + expect(stopped).toMatchObject({ status: "cancelled", errorCode: "cancelled", error: reason }); + expect(stopped?.resultJson ?? {}).not.toHaveProperty("nativePreDispatchFailure"); + expect(opened).toEqual([]); + expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), + eq(heartbeatRuns.retryOfRunId, runId)))).toHaveLength(0); + for (const action of await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId))) + expect(action.evidence).toMatchObject({ runId, originalFailureCode: "cancelled" }); + return stopped!; + } + + it("settles an operator Stop recorded before budget admission as that cancellation, not a run failure", async () => { + const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 }); + const opened: string[] = []; + const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db, opened) }); + try { + admission.beforeReserve = async runId => { + admission.beforeReserve = undefined; + const recordedAt = new Date().toISOString(); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + const intentId = `native-cancellation:${randomUUID()}`; + const [audit] = await db.insert(activityLog).values({ companyId: run!.companyId, actorType: "system", + actorId: "native-session-cancellation", action: "native.cancellation_intent_recorded", entityType: "heartbeat_run", + entityId: runId, agentId: run!.agentId, runId, details: { intentId, scope: "run", + reasonCode: "cancellation_run_only", effects: ["release_run_resources"] } }).returning(); + await db.update(heartbeatRuns).set({ resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({ + nativeCancellation: { schema: "paperclip.native-cancellation.v1", intentId, intentAuditId: audit!.id, + companyId: run!.companyId, runId, issueId: run!.nativeIssueId, scope: "run", reasonCode: "cancellation_run_only", + effects: ["release_run_resources"], dispatchState: "pending", dispatched: false, decisionId: null, + priorCoordinatorDecisionId: null, recordedAt }, + cancelledByActorType: "user", cancelledByUserId: "operator", + executionCancellation: { state: "requested", requestedAt: recordedAt }, + cancellation: { reason: "Cancelled by a board operator", source: "operator", expected: true, + initiator: { id: "operator", type: "user" }, recordedAt }, + startupCancellation: { errorCode: "cancelled", requestedAt: recordedAt, beforeNativeSelection: false }, + })}::jsonb` }).where(eq(heartbeatRuns.id, runId)); + }; + const { issueId, runId } = await startTask(db, heartbeat); + const stopped = await expectSettledAsStop(db, runId, issueId, opened, "Recovered durable native run cancellation"); + expect(stopped.resultJson?.nativeCancellation).toMatchObject({ dispatchState: "acknowledged", dispatched: false }); + } finally { + admission.beforeReserve = undefined; + await heartbeat.drainActiveRunExecutions(); + await closeIdleWarmNativeSessionsForRestart(); + await db.$client.end(); + } + }, 90_000); + + it("leaves an operator Stop that settles during admission cancelled", async () => { + const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 }); + const opened: string[] = []; + const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db, opened) }); + let stop: Promise | undefined; + try { + admission.beforeReserve = async runId => { + admission.beforeReserve = undefined; + stop = heartbeat.cancelRun(runId, "Cancelled by a board operator", { + resultJson: { cancelledByActorType: "user", cancelledByUserId: "operator" }, + }); + await waitForCondition("the Stop to be recorded", async () => { + const [current] = await db.select({ resultJson: heartbeatRuns.resultJson }).from(heartbeatRuns) + .where(eq(heartbeatRuns.id, runId)); + return Boolean(current?.resultJson?.cancellation || current?.resultJson?.startupCancellation); + }); + }; + const { issueId, runId } = await startTask(db, heartbeat); + await expect(stop).resolves.toMatchObject({ id: runId, status: "cancelled" }); + await expectSettledAsStop(db, runId, issueId, opened, "Cancelled by a board operator"); + } finally { + admission.beforeReserve = undefined; + await stop?.catch(() => undefined); + await heartbeat.drainActiveRunExecutions(); + await closeIdleWarmNativeSessionsForRestart(); + await db.$client.end(); + } + }, 90_000); +}); diff --git a/server/src/services/budget-reservations.ts b/server/src/services/budget-reservations.ts index 49c47eb14a1..7e5bcbe6d47 100644 --- a/server/src/services/budget-reservations.ts +++ b/server/src/services/budget-reservations.ts @@ -99,7 +99,11 @@ export async function reserveRunBudget(db: Db, companyId: string, runId: string, const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, companyId))).for("update"); if (!run) throw notFound("Run not found"); const stopRecovery = await readClaimedNativeStopRecovery(tx, run, recoveryLeaseOwner); - if ((run.resultJson?.cancellation || run.resultJson?.startupCancellation) && !stopRecovery) throw conflict("Run cancellation already requested"); + if ((run.resultJson?.cancellation || run.resultJson?.startupCancellation) && !stopRecovery) { + if (run.runtimeMode !== "native") throw conflict("Run cancellation already requested"); + const { NativeCancellationPendingRecoveryError } = await import("./native-runtime/native-session-executor.js"); + throw new NativeCancellationPendingRecoveryError(); + } if (run.costAccountedAt || !["queued", "running"].includes(run.status)) throw conflict("Run can no longer start provider work"); const [existing] = await tx.select().from(budgetReservations).where(and(eq(budgetReservations.companyId, companyId), eq(budgetReservations.runId, runId))); if (existing) {