fix(native): settle a Stop seen at budget admission as that cancellation (#21483)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs prepare a sandbox, then reserve budget, then start
provider work
> - An operator can press Stop while a native run still prepares
> - Budget admission reads the recorded Stop on the locked run row and
throws a generic 409 "Run cancellation already requested"
> - Heartbeat treats that throw as a run failure, so an expected Stop
becomes a failed run and a `run_failure` error event
> - This pull request makes admission report the Stop through the
existing native startup Stop contract
> - The benefit is that a Stop during preparation ends as a
cancellation, with no false failure, no retry, and no stranded Stop
intent

## Linked Issues or Issue Description

**What happened?**
An operator stopped a native run while it prepared. `reserveRunBudget`
found `resultJson.cancellation` / `startupCancellation` on the locked
run row and threw `conflict("Run cancellation already requested")`.
Heartbeat sent this error to `failRun`. The run ended `failed` with
`adapter_failed` or `native_continuation_requires_reconciliation`, and
Sentry got a `run_failure` event with `cancellationExpected: true`. The
pending native Stop intent stayed `pending`. Restart recovery then
logged `native_stop_requires_existing_remote_session` for a session that
never existed. Two staging Sentry groups show this: 20 events since
2026-11-10.

**Expected behavior**
A Stop that the server records before provider work starts ends the run
as `cancelled`. It does not report a failure, it does not schedule a
retry, and it acknowledges the Stop intent.

**Steps to reproduce**
1. Start a native run on a task.
2. Press Stop while the run prepares, before budget admission.
3. See that the run ends `failed` with the error "Run cancellation
already requested".

No public issue or pull request covers this. A GitHub search for
"cancellation already requested" found no related fix.

## What Changed

- `reserveRunBudget` now throws the existing
`NativeCancellationPendingRecoveryError` for a native run when it finds
a recorded Stop. Heartbeat already settles this error through
`cancelRunInternal("Recovered durable native run cancellation")`. This
is the same path as a Stop that native startup sees when the session
opens.
- Legacy runs keep the 409 conflict. The incident evidence is for native
runs only.
- The final dispatch gate does not change. It already treats a recorded
Stop as a lost race, not as a failure.
- New regression file
`server/src/__tests__/heartbeat-native-stop-before-admission.test.ts`.

## Verification

- `npx vitest run
src/__tests__/heartbeat-native-stop-before-admission.test.ts` (in
`server/`, embedded Postgres):
- Before the fix: the test "settles an operator Stop recorded before
budget admission" fails. The run ends `status: "failed"`, `error: "Run
cancellation already requested"`. This is the staging result.
- After the fix: 2/2 pass. The run ends `cancelled`, the provider
session never opens, no retry run exists, any task hold records
`originalFailureCode: "cancelled"`, and the native Stop intent becomes
`acknowledged`.
- The second test is a safety case. A real `heartbeat.cancelRun` that
settles during admission still ends `cancelled`, and the Stop request
resolves.
- Focused suites pass, 13 files and 304 tests: native pre-dispatch
retry, admission and cleanup, budget enforcement scoping, closed native
receipt accounting, native recovery lease reacquire, native cleanup
admission, native startup cleanup recovery, native cancellation request,
native Stop pending, run-dispatch Postgres adapter, companies service,
execution preparation timing.
- `pnpm typecheck` in `server/` passes.
- `pnpm verify` selects a broad set because the change is under
heartbeat. CI supplies that coverage.
- Not verified: a live staging Stop. Staging gets this fix after merge
through the normal automatic deploy.

## Risks

- Low risk. The change applies only when admission finds a recorded Stop
on a native run. Before this change that run always failed.
- `budget-reservations.ts` now imports an error class from
`native-session-executor.ts`. The executor does not import
`budget-reservations`, so this adds no import cycle.
- A user Stop with `dispatched: false` can still put an
`active_run_watchdog` hold on the task. That is existing behavior for
Stops that reach no provider (#18820 keeps the strict dispatched
receipt). This PR does not change it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution,
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Devin FoleyandClaude Opus 5.5 authored and GitHub committed 2026-11-13 13:47:52 -08:00
1 parent 5df1ada9c1
commit 06a4a07781
2 files changed
+161 -1

No files matched your search

@@ -0,0 +1,156 @@
import { randomUUID } from "node:crypto";
import { and, eq, sql } from "drizzle-orm";
import { activityLog, agents, authUsers, companyMemberships, createDb, heartbeatRuns, issueComments, issueRecoveryActions,
issues } from "@paperclipai/db";
import { describe, expect, it, vi } from "vitest";
import { heartbeatService, type HeartbeatServiceOptions } from "../services/heartbeat.js";
import { instanceSettingsService } from "../services/instance-settings.js";
import { closeIdleWarmNativeSessionsForRestart } from "../services/native-runtime/native-session-executor.js";
import { nativeRestartFixture, waitForCondition } from "./helpers/native-restart-fixture.js";
import { getRecoveryPostgresSupport } from "./helpers/recovery-postgres-fixture.js";
const admission = vi.hoisted(() => ({ beforeReserve: undefined as ((runId: string) => Promise<void>) | undefined }));
vi.mock("../services/budget-reservations.js", async importOriginal => {
const actual = await importOriginal<typeof import("../services/budget-reservations.js")>();
return {
...actual,
reserveRunBudget: async (...args: Parameters<typeof actual.reserveRunBudget>) => {
await admission.beforeReserve?.(args[2]);
return actual.reserveRunBudget(...args);
},
};
});
const support = await getRecoveryPostgresSupport({});
function nativeBackend(db: ReturnType<typeof createDb>, opened: string[]):
NonNullable<HeartbeatServiceOptions["nativeSessionBackendFactory"]> {
const capabilities = { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true };
return execution => ({
descriptor: async () => {
const [run] = await db.select({ runnerInstanceId: heartbeatRuns.runnerInstanceId }).from(heartbeatRuns)
.where(eq(heartbeatRuns.id, execution.binding.runId));
if (!run?.runnerInstanceId) throw new Error("Normal native admission did not persist a runner identity");
return { kind: "mock", name: "native-stop-before-admission", version: "1", capabilities,
runtimeContextCapabilities: { instructions: "native", skills: "native", mcp: "native" } };
},
openSession: async () => {
opened.push(execution.binding.runId);
throw new Error("A stopped run opened a provider session");
},
recoverSession: async () => {
opened.push(execution.binding.runId);
throw new Error("A stopped run recovered a provider session");
},
});
}
describe.skipIf(!support.supported)("heartbeat native Stop before budget admission", () => {
const fixture = nativeRestartFixture({});
async function startTask(db: ReturnType<typeof createDb>, heartbeat: ReturnType<typeof heartbeatService>) {
const { companyId, agentId } = fixture.current;
const userId = randomUUID();
await db.insert(authUsers).values({ id: userId, name: "Task owner", email: `${userId}@example.invalid`,
createdAt: new Date(), updatedAt: new Date() });
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, membershipRole: "owner" });
await db.update(agents).set({ status: "idle", adapterConfig: { provider: "codex" } }).where(eq(agents.id, agentId));
await instanceSettingsService(db).updateExperimental({ enableAgentChat: true, enableNativeRunner: true });
const [issue] = await db.insert(issues).values({ companyId, title: "Stop while preparing", status: "in_progress",
assigneeAgentId: agentId, responsibleUserId: userId, conversationAgentId: agentId, conversationUserId: userId,
conversationState: "active" }).returning();
const [message] = await db.insert(issueComments).values({ companyId, issueId: issue!.id,
authorType: "user", authorUserId: userId, body: "Start." }).returning();
const run = await heartbeat.wakeup(agentId, { source: "on_demand", triggerDetail: "manual", reason: "issue_commented",
requestedByActorType: "user", requestedByActorId: userId, manualUserWake: true,
payload: { issueId: issue!.id, commentId: message!.id },
contextSnapshot: { issueId: issue!.id, taskKey: issue!.id, wakeCommentId: message!.id } });
expect(run).not.toBeNull();
await waitForCondition("the run to settle", () => heartbeat.getTaskDrainStatus().quiescent);
return { issueId: issue!.id, runId: run!.id };
}
async function expectSettledAsStop(db: ReturnType<typeof createDb>, runId: string, issueId: string, opened: string[],
reason: string) {
const { companyId } = fixture.current;
await waitForCondition("the stopped run to settle", async () => {
const [current] = await db.select({ status: heartbeatRuns.status }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
return current?.status !== "running" && current?.status !== "queued";
});
const [stopped] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
expect(stopped).toMatchObject({ status: "cancelled", errorCode: "cancelled", error: reason });
expect(stopped?.resultJson ?? {}).not.toHaveProperty("nativePreDispatchFailure");
expect(opened).toEqual([]);
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.retryOfRunId, runId)))).toHaveLength(0);
for (const action of await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, issueId)))
expect(action.evidence).toMatchObject({ runId, originalFailureCode: "cancelled" });
return stopped!;
}
it("settles an operator Stop recorded before budget admission as that cancellation, not a run failure", async () => {
const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 });
const opened: string[] = [];
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db, opened) });
try {
admission.beforeReserve = async runId => {
admission.beforeReserve = undefined;
const recordedAt = new Date().toISOString();
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
const intentId = `native-cancellation:${randomUUID()}`;
const [audit] = await db.insert(activityLog).values({ companyId: run!.companyId, actorType: "system",
actorId: "native-session-cancellation", action: "native.cancellation_intent_recorded", entityType: "heartbeat_run",
entityId: runId, agentId: run!.agentId, runId, details: { intentId, scope: "run",
reasonCode: "cancellation_run_only", effects: ["release_run_resources"] } }).returning();
await db.update(heartbeatRuns).set({ resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({
nativeCancellation: { schema: "paperclip.native-cancellation.v1", intentId, intentAuditId: audit!.id,
companyId: run!.companyId, runId, issueId: run!.nativeIssueId, scope: "run", reasonCode: "cancellation_run_only",
effects: ["release_run_resources"], dispatchState: "pending", dispatched: false, decisionId: null,
priorCoordinatorDecisionId: null, recordedAt },
cancelledByActorType: "user", cancelledByUserId: "operator",
executionCancellation: { state: "requested", requestedAt: recordedAt },
cancellation: { reason: "Cancelled by a board operator", source: "operator", expected: true,
initiator: { id: "operator", type: "user" }, recordedAt },
startupCancellation: { errorCode: "cancelled", requestedAt: recordedAt, beforeNativeSelection: false },
})}::jsonb` }).where(eq(heartbeatRuns.id, runId));
};
const { issueId, runId } = await startTask(db, heartbeat);
const stopped = await expectSettledAsStop(db, runId, issueId, opened, "Recovered durable native run cancellation");
expect(stopped.resultJson?.nativeCancellation).toMatchObject({ dispatchState: "acknowledged", dispatched: false });
} finally {
admission.beforeReserve = undefined;
await heartbeat.drainActiveRunExecutions();
await closeIdleWarmNativeSessionsForRestart();
await db.$client.end();
}
}, 90_000);
it("leaves an operator Stop that settles during admission cancelled", async () => {
const db = createDb(fixture.current.temporary.connectionString, { maxConnections: 4 });
const opened: string[] = [];
const heartbeat = heartbeatService(db, { nativeSessionBackendFactory: nativeBackend(db, opened) });
let stop: Promise<unknown> | undefined;
try {
admission.beforeReserve = async runId => {
admission.beforeReserve = undefined;
stop = heartbeat.cancelRun(runId, "Cancelled by a board operator", {
resultJson: { cancelledByActorType: "user", cancelledByUserId: "operator" },
});
await waitForCondition("the Stop to be recorded", async () => {
const [current] = await db.select({ resultJson: heartbeatRuns.resultJson }).from(heartbeatRuns)
.where(eq(heartbeatRuns.id, runId));
return Boolean(current?.resultJson?.cancellation || current?.resultJson?.startupCancellation);
});
};
const { issueId, runId } = await startTask(db, heartbeat);
await expect(stop).resolves.toMatchObject({ id: runId, status: "cancelled" });
await expectSettledAsStop(db, runId, issueId, opened, "Cancelled by a board operator");
} finally {
admission.beforeReserve = undefined;
await stop?.catch(() => undefined);
await heartbeat.drainActiveRunExecutions();
await closeIdleWarmNativeSessionsForRestart();
await db.$client.end();
}
}, 90_000);
});
+5 -1
View File
@@ -99,7 +99,11 @@ export async function reserveRunBudget(db: Db, companyId: string, runId: string,
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, companyId))).for("update");
if (!run) throw notFound("Run not found");
const stopRecovery = await readClaimedNativeStopRecovery(tx, run, recoveryLeaseOwner);
if ((run.resultJson?.cancellation || run.resultJson?.startupCancellation) && !stopRecovery) throw conflict("Run cancellation already requested");
if ((run.resultJson?.cancellation || run.resultJson?.startupCancellation) && !stopRecovery) {
if (run.runtimeMode !== "native") throw conflict("Run cancellation already requested");
const { NativeCancellationPendingRecoveryError } = await import("./native-runtime/native-session-executor.js");
throw new NativeCancellationPendingRecoveryError();
}
if (run.costAccountedAt || !["queued", "running"].includes(run.status)) throw conflict("Run can no longer start provider work");
const [existing] = await tx.select().from(budgetReservations).where(and(eq(budgetReservations.companyId, companyId), eq(budgetReservations.runId, runId)));
if (existing) {