fix(runs): a person's Stop wins over lease-release teardown (#21300)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can stop an agent run with Stop (cancel). The run must then
end as cancelled
> - A run that a person stops just as it starts tears down its
environment lease at the same moment
> - The lease-release teardown wrote `interrupted`
(`lease_released_before_terminal`). It recognized only an acknowledged
Stop, and it decided from a run snapshot that it read before the Stop
request landed
> - An interrupted run looks like an infrastructure failure, and it is a
recovery candidate. That is against the person's decision
> - This pull request makes a recorded person Stop win over the teardown
in both orderings
> - The benefit is that a person's Stop always ends the run as cancelled

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `steering.stop-versus-pause`, the fixture
cancelled a resumed run as it reached native running. The Stop request
was recorded at 04:02:11.049. The teardown wrote `interrupted` with
`lease_released_before_terminal` at 04:02:11.083. The native
cancellation intent was still `pending`, so the teardown did not see it
as a Stop. The eval needs `cancelled`.

**Steps to reproduce**
1. Start a native run.
2. Record a person Stop request on the run (the cancel route does this
first).
3. Let the lease-release teardown run before the cancel finalizer writes
`cancelled`.
4. The run ends `interrupted`.

**Expected behavior**
The run ends `cancelled`. This is true when the Stop is recorded before
the teardown reads the run, and when it is recorded between that read
and the write.

## What Changed

- `server/src/services/acknowledged-native-stop.ts`:
`hasRecordedUserStopRequest` and `recordedUserStopRequestSql` recognize
a person Stop that the cancel route recorded (`cancelledByActorType:
"user"`, a user id, and a requested or acknowledged execution
cancellation or a startup cancellation fence).
- `server/src/services/heartbeat/run-lifecycle.ts`:
`terminalizeRunOnLeaseRelease` re-reads the run before it decides. A
recorded person Stop now gives `cancelled`. The `interrupted` write is
fenced on "no person Stop recorded", so a Stop that lands between the
read and the write makes the write fail. The function then reads again
and writes `cancelled`.
-
`server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts`:
two orderings (Stop recorded before the teardown read; Stop recorded
after the read, with the stale snapshot) and a control (an agent-only
request still reports `interrupted`).

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-run-lease-release-terminalization.test.ts` —
both new ordering tests fail on master (`interrupted`,
`lease_released_before_terminal`) and pass here.
- `cd server && npx vitest run
src/__tests__/heartbeat-run-lease-release-terminalization.test.ts
src/__tests__/heartbeat-run-terminalize-before-release.test.ts
src/services/heartbeat/run-lifecycle.test.ts` (67 passed).
- `cd server && npx tsc --noEmit` passes.
- Paid E2E: see the combined-branch run in the comments.

## Risks

- Low. Only the lease-release teardown changes, and only when a person
Stop is recorded on the run. The cancel path already wrote `cancelled`
for the same run when it won the race.
- The teardown now makes one more read of the run row. This is one
indexed primary-key read per teardown.
- Previous-release saved state: no persisted shape changes. Runs saved
by the previous release carry the same Stop fields.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-12 23:52:13 -06:00
1 parent bc71c23e0a
commit f82ec65800
3 files changed
+171 -46

No files matched your search

@@ -1,5 +1,5 @@
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
costEvents,
@@ -167,6 +167,96 @@ describeEmbeddedPostgres("heartbeat terminalizeRunOnLeaseRelease", () => {
expect(terminal).toMatchObject({ status: "cancelled", error: null, errorCode: null });
});
function personCancelRequest(input: { companyId: string; issueId: string; runId: string }) {
const requestedAt = new Date().toISOString();
return {
cancellation: { reason: "Cancelled by a board operator", source: "operator", expected: true,
initiator: { id: "board-user", type: "user" }, recordedAt: requestedAt },
cancelledByUserId: "board-user",
cancelledByActorType: "user",
executionCancellation: { state: "requested", requestedAt },
startupCancellation: { errorCode: "cancelled", requestedAt, beforeNativeSelection: false },
nativeCancellation: {
schema: "paperclip.native-cancellation.v1", runId: input.runId, companyId: input.companyId, issueId: input.issueId,
scope: "run", reasonCode: "cancellation_run_only", effects: ["release_run_resources"], dispatched: false,
dispatchState: "pending", intentId: `native-cancellation:${randomUUID()}`, intentAuditId: randomUUID(),
recordedAt: requestedAt,
},
};
}
it("ends a run cancelled when a person's Stop was recorded before teardown read the run", async () => {
const { companyId, issueId, runId } = await seed({ issueStatus: "in_progress", runStatus: "running" });
const [run] = await db.update(heartbeatRuns).set({
nativeIssueId: issueId,
resultJson: personCancelRequest({ companyId, issueId, runId }),
}).where(eq(heartbeatRuns.id, runId)).returning();
const terminal = await heartbeatService(db).terminalizeRunOnLeaseRelease(run!);
expect(terminal).toMatchObject({ status: "cancelled", errorCode: null });
const row = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).then((rows) => rows[0]);
expect(row?.status).toBe("cancelled");
expect(row?.resultJson).toMatchObject({ cancelledByActorType: "user", cancelledByUserId: "board-user" });
});
it("ends a run cancelled when a person's Stop lands after teardown read the run", async () => {
const { companyId, issueId, runId, run: staleRun } = await seed({ issueStatus: "in_progress", runStatus: "running" });
await db.update(heartbeatRuns).set({
nativeIssueId: issueId,
resultJson: personCancelRequest({ companyId, issueId, runId }),
}).where(eq(heartbeatRuns.id, runId));
const terminal = await heartbeatService(db).terminalizeRunOnLeaseRelease(staleRun);
expect(terminal).toMatchObject({ status: "cancelled", errorCode: null });
const row = await db.select({ status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode })
.from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).then((rows) => rows[0]);
expect(row).toEqual({ status: "cancelled", errorCode: null });
});
it("ends a run cancelled when a person's Stop commits while teardown's write waits", async () => {
const { companyId, issueId, runId, run: staleRun } = await seed({ issueStatus: "in_progress", runStatus: "running" });
const other = createDb(tempDb!.connectionString);
let locked!: () => void, release!: () => void;
const lockHeld = new Promise<void>(resolve => { locked = resolve; });
const gate = new Promise<void>(resolve => { release = resolve; });
try {
const stop = other.transaction(async tx => {
await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("update");
locked();
await gate;
await tx.update(heartbeatRuns).set({ nativeIssueId: issueId, resultJson: personCancelRequest({ companyId, issueId, runId }) })
.where(eq(heartbeatRuns.id, runId));
});
await lockHeld;
const teardown = heartbeatService(db).terminalizeRunOnLeaseRelease(staleRun);
await vi.waitFor(async () => {
const waiting = await db.execute(sql`select 1 from pg_stat_activity
where wait_event_type = 'Lock' and datname = current_database() and query ilike '%heartbeat_runs%'`);
expect(waiting.length).toBeGreaterThan(0);
}, { timeout: 10_000 });
release();
await stop;
expect(await teardown).toMatchObject({ status: "cancelled", errorCode: null });
} finally {
release();
await other.$client.end();
}
});
it("still reports interruption when only an agent requested the stop", async () => {
const { companyId, issueId, runId } = await seed({ issueStatus: "in_progress", runStatus: "running" });
const request = personCancelRequest({ companyId, issueId, runId });
const [run] = await db.update(heartbeatRuns).set({
resultJson: { ...request, cancelledByActorType: "agent", cancelledByUserId: undefined },
}).where(eq(heartbeatRuns.id, runId)).returning();
const terminal = await heartbeatService(db).terminalizeRunOnLeaseRelease(run!);
expect(terminal).toMatchObject({ status: "interrupted", errorCode: "lease_released_before_terminal" });
});
it("preserves an acknowledged agent termination Stop when teardown wins the finalizer race", async () => {
const { companyId, issueId, runId } = await seed({ issueStatus: "in_progress", runStatus: "running" });
const [run] = await db.update(heartbeatRuns).set({
@@ -1,4 +1,4 @@
import { and, desc, eq, inArray, isNotNull } from "drizzle-orm";
import { and, desc, eq, inArray, isNotNull, sql } from "drizzle-orm";
import { environmentLeases, heartbeatRunEvents, heartbeatRuns, type Db } from "@paperclipai/db";
import { parseObject } from "../adapters/utils.js";
import { resolveProviderApiPlacement } from "./native-runtime/provider-api-placement.js";
@@ -34,6 +34,23 @@ export function hasAcknowledgedNativeStopIntent(run: Parameters<typeof hasAcknow
hasAcknowledgedNativeRunStopReceipt(run);
}
export function hasRecordedUserStopRequest(run: { resultJson: Record<string, unknown> | null }): boolean {
const result = run.resultJson;
if (result?.cancelledByActorType !== "user" || typeof result.cancelledByUserId !== "string" || !result.cancelledByUserId) return false;
const state = parseObject(result.executionCancellation).state;
const startup = result.startupCancellation;
return state === "requested" || state === "acknowledged" ||
(Boolean(startup) && typeof startup === "object" && !Array.isArray(startup));
}
export function recordedUserStopRequestSql() {
const result = heartbeatRuns.resultJson;
return sql`(coalesce(${result}->>'cancelledByActorType', '') = 'user'
and coalesce(${result}->>'cancelledByUserId', '') <> ''
and (coalesce(${result}->'executionCancellation'->>'state', '') in ('requested', 'acknowledged')
or jsonb_typeof(${result}->'startupCancellation') = 'object'))`;
}
export function hasAcknowledgedNativeLifecycleStopIntent(run: Parameters<typeof hasAcknowledgedNativeRunStopReceipt>[0]): boolean {
return isAgentLifecycleStopRecord(run.resultJson) && hasAcknowledgedNativeRunStopReceipt(run);
}
+62 -44
View File
@@ -50,6 +50,8 @@ import {
hasAcknowledgedNativeReassignmentStopIntent,
hasAcknowledgedNativeLifecycleStopIntent,
hasAcknowledgedNativeStopIntent,
hasRecordedUserStopRequest,
recordedUserStopRequestSql,
} from "../acknowledged-native-stop.js";
import { connectionIntentService } from "../connection-intents.js";
import {
@@ -1376,54 +1378,70 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl
// released. It never overwrites a status that another path already made
// terminal.
async function terminalizeRunOnLeaseRelease(
run: typeof heartbeatRuns.$inferSelect,
staleRun: typeof heartbeatRuns.$inferSelect,
): Promise<typeof heartbeatRuns.$inferSelect> {
if (isNativeRunnerOwnershipHeld(run)) return run;
if (isHeartbeatRunTerminalStatus(run.status)) return run;
if (run.status !== "running" && run.status !== "queued") return run;
if (isNativeRunnerOwnershipHeld(staleRun)) return staleRun;
if (isHeartbeatRunTerminalStatus(staleRun.status)) return staleRun;
if (staleRun.status !== "running" && staleRun.status !== "queued") return staleRun;
let run = staleRun;
let write: Awaited<ReturnType<typeof setRunStatusFromLive>> | null = null;
let terminalStatus: "succeeded" | "cancelled" | "interrupted" = "interrupted";
let issueId: string | null = null;
for (let attempt = 0; attempt < 2; attempt++) {
const current = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id))
.then((rows) => rows[0] ?? null);
if (!current) return run;
run = current;
if (isNativeRunnerOwnershipHeld(run)) return run;
if (isHeartbeatRunTerminalStatus(run.status)) return run;
if (run.status !== "running" && run.status !== "queued") return run;
// Choose the terminal status that reflects the true outcome. When the issue
// already reached a terminal status, the run reached its goal, so use the
// matching terminal run status. Otherwise the teardown cut the run short,
// so use "interrupted".
const issueId = readNonEmptyString(
parseObject(run.contextSnapshot).issueId,
);
let terminalStatus: "succeeded" | "cancelled" | "interrupted" =
"interrupted";
if (issueId) {
const issueStatus = await db
.select({ status: issues.status })
.from(issues)
.where(eq(issues.id, issueId))
.then((rows) => rows[0]?.status ?? null);
if (issueStatus === "done") terminalStatus = "succeeded";
else if (issueStatus === "cancelled") terminalStatus = "cancelled";
// Choose the terminal status that reflects the true outcome. When the issue
// already reached a terminal status, the run reached its goal, so use the
// matching terminal run status. Otherwise the teardown cut the run short,
// so use "interrupted".
issueId = readNonEmptyString(parseObject(run.contextSnapshot).issueId);
terminalStatus = "interrupted";
if (issueId) {
const issueStatus = await db
.select({ status: issues.status })
.from(issues)
.where(eq(issues.id, issueId))
.then((rows) => rows[0]?.status ?? null);
if (issueStatus === "done") terminalStatus = "succeeded";
else if (issueStatus === "cancelled") terminalStatus = "cancelled";
}
// Teardown can beat the cancellation finalizer. Preserve the acknowledged
// user intent instead of reporting an infrastructure interruption.
if (hasRecordedUserStopRequest(run) || hasAcknowledgedNativeStopIntent(run) || hasAcknowledgedNativeReassignmentStopIntent(run)
|| hasAcknowledgedNativeLifecycleStopIntent(run)) terminalStatus = "cancelled";
// Match both "running" and "queued". A queued run has released its lease but
// never reached "running", so a running-only update would miss it and leave
// a phantom live run behind.
write = await setRunStatusFromLive(
run.id,
terminalStatus,
["running", "queued"],
{
finishedAt: run.finishedAt ?? new Date(),
error: run.error ?? (terminalStatus === "interrupted"
? `run terminalized on environment lease release: heartbeat_runs.status was still ${run.status} at teardown` : null),
errorCode:
run.errorCode ??
(terminalStatus === "interrupted"
? "lease_released_before_terminal"
: null),
},
undefined,
terminalStatus === "cancelled" ? undefined : not(recordedUserStopRequestSql()),
);
if (write.updated) break;
if (!write.run || isHeartbeatRunTerminalStatus(write.run.status)) break;
}
// Teardown can beat the cancellation finalizer. Preserve the acknowledged
// user intent instead of reporting an infrastructure interruption.
if (hasAcknowledgedNativeStopIntent(run) || hasAcknowledgedNativeReassignmentStopIntent(run)
|| hasAcknowledgedNativeLifecycleStopIntent(run)) terminalStatus = "cancelled";
if (!write) return run;
const message = `run terminalized on environment lease release: heartbeat_runs.status was still ${run.status} at teardown`;
// Match both "running" and "queued". A queued run has released its lease but
// never reached "running", so a running-only update would miss it and leave
// a phantom live run behind.
const write = await setRunStatusFromLive(
run.id,
terminalStatus,
["running", "queued"],
{
finishedAt: run.finishedAt ?? new Date(),
error: run.error ?? (terminalStatus === "interrupted" ? message : null),
errorCode:
run.errorCode ??
(terminalStatus === "interrupted"
? "lease_released_before_terminal"
: null),
},
);
if (!write.updated) {
// Another path already finalized the run. Keep that terminal outcome.
return write.run ?? run;