mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
fix(runs): a person's Stop wins over lease-release teardown (#21300)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person can stop an agent run with Stop (cancel). The run must then end as cancelled > - A run that a person stops just as it starts tears down its environment lease at the same moment > - The lease-release teardown wrote `interrupted` (`lease_released_before_terminal`). It recognized only an acknowledged Stop, and it decided from a run snapshot that it read before the Stop request landed > - An interrupted run looks like an infrastructure failure, and it is a recovery candidate. That is against the person's decision > - This pull request makes a recorded person Stop win over the teardown in both orderings > - The benefit is that a person's Stop always ends the run as cancelled ## Linked Issues or Issue Description **What happened?** In the paid chat eval `steering.stop-versus-pause`, the fixture cancelled a resumed run as it reached native running. The Stop request was recorded at 04:02:11.049. The teardown wrote `interrupted` with `lease_released_before_terminal` at 04:02:11.083. The native cancellation intent was still `pending`, so the teardown did not see it as a Stop. The eval needs `cancelled`. **Steps to reproduce** 1. Start a native run. 2. Record a person Stop request on the run (the cancel route does this first). 3. Let the lease-release teardown run before the cancel finalizer writes `cancelled`. 4. The run ends `interrupted`. **Expected behavior** The run ends `cancelled`. This is true when the Stop is recorded before the teardown reads the run, and when it is recorded between that read and the write. ## What Changed - `server/src/services/acknowledged-native-stop.ts`: `hasRecordedUserStopRequest` and `recordedUserStopRequestSql` recognize a person Stop that the cancel route recorded (`cancelledByActorType: "user"`, a user id, and a requested or acknowledged execution cancellation or a startup cancellation fence). - `server/src/services/heartbeat/run-lifecycle.ts`: `terminalizeRunOnLeaseRelease` re-reads the run before it decides. A recorded person Stop now gives `cancelled`. The `interrupted` write is fenced on "no person Stop recorded", so a Stop that lands between the read and the write makes the write fail. The function then reads again and writes `cancelled`. - `server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts`: two orderings (Stop recorded before the teardown read; Stop recorded after the read, with the stale snapshot) and a control (an agent-only request still reports `interrupted`). ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-run-lease-release-terminalization.test.ts` — both new ordering tests fail on master (`interrupted`, `lease_released_before_terminal`) and pass here. - `cd server && npx vitest run src/__tests__/heartbeat-run-lease-release-terminalization.test.ts src/__tests__/heartbeat-run-terminalize-before-release.test.ts src/services/heartbeat/run-lifecycle.test.ts` (67 passed). - `cd server && npx tsc --noEmit` passes. - Paid E2E: see the combined-branch run in the comments. ## Risks - Low. Only the lease-release teardown changes, and only when a person Stop is recorded on the run. The cancel path already wrote `cancelled` for the same run when it won the race. - The teardown now makes one more read of the run row. This is one indexed primary-key read per teardown. - Previous-release saved state: no persisted shape changes. Runs saved by the previous release carry the same Stop fields. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
bc71c23e0a
commit
f82ec65800
3 files changed
+171
-46
No files matched your search
@@ -1,5 +1,5 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
costEvents,
|
||||
@@ -167,6 +167,96 @@ describeEmbeddedPostgres("heartbeat terminalizeRunOnLeaseRelease", () => {
|
||||
expect(terminal).toMatchObject({ status: "cancelled", error: null, errorCode: null });
|
||||
});
|
||||
|
||||
function personCancelRequest(input: { companyId: string; issueId: string; runId: string }) {
|
||||
const requestedAt = new Date().toISOString();
|
||||
return {
|
||||
cancellation: { reason: "Cancelled by a board operator", source: "operator", expected: true,
|
||||
initiator: { id: "board-user", type: "user" }, recordedAt: requestedAt },
|
||||
cancelledByUserId: "board-user",
|
||||
cancelledByActorType: "user",
|
||||
executionCancellation: { state: "requested", requestedAt },
|
||||
startupCancellation: { errorCode: "cancelled", requestedAt, beforeNativeSelection: false },
|
||||
nativeCancellation: {
|
||||
schema: "paperclip.native-cancellation.v1", runId: input.runId, companyId: input.companyId, issueId: input.issueId,
|
||||
scope: "run", reasonCode: "cancellation_run_only", effects: ["release_run_resources"], dispatched: false,
|
||||
dispatchState: "pending", intentId: `native-cancellation:${randomUUID()}`, intentAuditId: randomUUID(),
|
||||
recordedAt: requestedAt,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
it("ends a run cancelled when a person's Stop was recorded before teardown read the run", async () => {
|
||||
const { companyId, issueId, runId } = await seed({ issueStatus: "in_progress", runStatus: "running" });
|
||||
const [run] = await db.update(heartbeatRuns).set({
|
||||
nativeIssueId: issueId,
|
||||
resultJson: personCancelRequest({ companyId, issueId, runId }),
|
||||
}).where(eq(heartbeatRuns.id, runId)).returning();
|
||||
|
||||
const terminal = await heartbeatService(db).terminalizeRunOnLeaseRelease(run!);
|
||||
|
||||
expect(terminal).toMatchObject({ status: "cancelled", errorCode: null });
|
||||
const row = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).then((rows) => rows[0]);
|
||||
expect(row?.status).toBe("cancelled");
|
||||
expect(row?.resultJson).toMatchObject({ cancelledByActorType: "user", cancelledByUserId: "board-user" });
|
||||
});
|
||||
|
||||
it("ends a run cancelled when a person's Stop lands after teardown read the run", async () => {
|
||||
const { companyId, issueId, runId, run: staleRun } = await seed({ issueStatus: "in_progress", runStatus: "running" });
|
||||
await db.update(heartbeatRuns).set({
|
||||
nativeIssueId: issueId,
|
||||
resultJson: personCancelRequest({ companyId, issueId, runId }),
|
||||
}).where(eq(heartbeatRuns.id, runId));
|
||||
|
||||
const terminal = await heartbeatService(db).terminalizeRunOnLeaseRelease(staleRun);
|
||||
|
||||
expect(terminal).toMatchObject({ status: "cancelled", errorCode: null });
|
||||
const row = await db.select({ status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode })
|
||||
.from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).then((rows) => rows[0]);
|
||||
expect(row).toEqual({ status: "cancelled", errorCode: null });
|
||||
});
|
||||
|
||||
it("ends a run cancelled when a person's Stop commits while teardown's write waits", async () => {
|
||||
const { companyId, issueId, runId, run: staleRun } = await seed({ issueStatus: "in_progress", runStatus: "running" });
|
||||
const other = createDb(tempDb!.connectionString);
|
||||
let locked!: () => void, release!: () => void;
|
||||
const lockHeld = new Promise<void>(resolve => { locked = resolve; });
|
||||
const gate = new Promise<void>(resolve => { release = resolve; });
|
||||
try {
|
||||
const stop = other.transaction(async tx => {
|
||||
await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("update");
|
||||
locked();
|
||||
await gate;
|
||||
await tx.update(heartbeatRuns).set({ nativeIssueId: issueId, resultJson: personCancelRequest({ companyId, issueId, runId }) })
|
||||
.where(eq(heartbeatRuns.id, runId));
|
||||
});
|
||||
await lockHeld;
|
||||
const teardown = heartbeatService(db).terminalizeRunOnLeaseRelease(staleRun);
|
||||
await vi.waitFor(async () => {
|
||||
const waiting = await db.execute(sql`select 1 from pg_stat_activity
|
||||
where wait_event_type = 'Lock' and datname = current_database() and query ilike '%heartbeat_runs%'`);
|
||||
expect(waiting.length).toBeGreaterThan(0);
|
||||
}, { timeout: 10_000 });
|
||||
release();
|
||||
await stop;
|
||||
expect(await teardown).toMatchObject({ status: "cancelled", errorCode: null });
|
||||
} finally {
|
||||
release();
|
||||
await other.$client.end();
|
||||
}
|
||||
});
|
||||
|
||||
it("still reports interruption when only an agent requested the stop", async () => {
|
||||
const { companyId, issueId, runId } = await seed({ issueStatus: "in_progress", runStatus: "running" });
|
||||
const request = personCancelRequest({ companyId, issueId, runId });
|
||||
const [run] = await db.update(heartbeatRuns).set({
|
||||
resultJson: { ...request, cancelledByActorType: "agent", cancelledByUserId: undefined },
|
||||
}).where(eq(heartbeatRuns.id, runId)).returning();
|
||||
|
||||
const terminal = await heartbeatService(db).terminalizeRunOnLeaseRelease(run!);
|
||||
|
||||
expect(terminal).toMatchObject({ status: "interrupted", errorCode: "lease_released_before_terminal" });
|
||||
});
|
||||
|
||||
it("preserves an acknowledged agent termination Stop when teardown wins the finalizer race", async () => {
|
||||
const { companyId, issueId, runId } = await seed({ issueStatus: "in_progress", runStatus: "running" });
|
||||
const [run] = await db.update(heartbeatRuns).set({
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { and, desc, eq, inArray, isNotNull } from "drizzle-orm";
|
||||
import { and, desc, eq, inArray, isNotNull, sql } from "drizzle-orm";
|
||||
import { environmentLeases, heartbeatRunEvents, heartbeatRuns, type Db } from "@paperclipai/db";
|
||||
import { parseObject } from "../adapters/utils.js";
|
||||
import { resolveProviderApiPlacement } from "./native-runtime/provider-api-placement.js";
|
||||
@@ -34,6 +34,23 @@ export function hasAcknowledgedNativeStopIntent(run: Parameters<typeof hasAcknow
|
||||
hasAcknowledgedNativeRunStopReceipt(run);
|
||||
}
|
||||
|
||||
export function hasRecordedUserStopRequest(run: { resultJson: Record<string, unknown> | null }): boolean {
|
||||
const result = run.resultJson;
|
||||
if (result?.cancelledByActorType !== "user" || typeof result.cancelledByUserId !== "string" || !result.cancelledByUserId) return false;
|
||||
const state = parseObject(result.executionCancellation).state;
|
||||
const startup = result.startupCancellation;
|
||||
return state === "requested" || state === "acknowledged" ||
|
||||
(Boolean(startup) && typeof startup === "object" && !Array.isArray(startup));
|
||||
}
|
||||
|
||||
export function recordedUserStopRequestSql() {
|
||||
const result = heartbeatRuns.resultJson;
|
||||
return sql`(coalesce(${result}->>'cancelledByActorType', '') = 'user'
|
||||
and coalesce(${result}->>'cancelledByUserId', '') <> ''
|
||||
and (coalesce(${result}->'executionCancellation'->>'state', '') in ('requested', 'acknowledged')
|
||||
or jsonb_typeof(${result}->'startupCancellation') = 'object'))`;
|
||||
}
|
||||
|
||||
export function hasAcknowledgedNativeLifecycleStopIntent(run: Parameters<typeof hasAcknowledgedNativeRunStopReceipt>[0]): boolean {
|
||||
return isAgentLifecycleStopRecord(run.resultJson) && hasAcknowledgedNativeRunStopReceipt(run);
|
||||
}
|
||||
|
||||
@@ -50,6 +50,8 @@ import {
|
||||
hasAcknowledgedNativeReassignmentStopIntent,
|
||||
hasAcknowledgedNativeLifecycleStopIntent,
|
||||
hasAcknowledgedNativeStopIntent,
|
||||
hasRecordedUserStopRequest,
|
||||
recordedUserStopRequestSql,
|
||||
} from "../acknowledged-native-stop.js";
|
||||
import { connectionIntentService } from "../connection-intents.js";
|
||||
import {
|
||||
@@ -1376,54 +1378,70 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl
|
||||
// released. It never overwrites a status that another path already made
|
||||
// terminal.
|
||||
async function terminalizeRunOnLeaseRelease(
|
||||
run: typeof heartbeatRuns.$inferSelect,
|
||||
staleRun: typeof heartbeatRuns.$inferSelect,
|
||||
): Promise<typeof heartbeatRuns.$inferSelect> {
|
||||
if (isNativeRunnerOwnershipHeld(run)) return run;
|
||||
if (isHeartbeatRunTerminalStatus(run.status)) return run;
|
||||
if (run.status !== "running" && run.status !== "queued") return run;
|
||||
if (isNativeRunnerOwnershipHeld(staleRun)) return staleRun;
|
||||
if (isHeartbeatRunTerminalStatus(staleRun.status)) return staleRun;
|
||||
if (staleRun.status !== "running" && staleRun.status !== "queued") return staleRun;
|
||||
let run = staleRun;
|
||||
let write: Awaited<ReturnType<typeof setRunStatusFromLive>> | null = null;
|
||||
let terminalStatus: "succeeded" | "cancelled" | "interrupted" = "interrupted";
|
||||
let issueId: string | null = null;
|
||||
for (let attempt = 0; attempt < 2; attempt++) {
|
||||
const current = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (!current) return run;
|
||||
run = current;
|
||||
if (isNativeRunnerOwnershipHeld(run)) return run;
|
||||
if (isHeartbeatRunTerminalStatus(run.status)) return run;
|
||||
if (run.status !== "running" && run.status !== "queued") return run;
|
||||
|
||||
// Choose the terminal status that reflects the true outcome. When the issue
|
||||
// already reached a terminal status, the run reached its goal, so use the
|
||||
// matching terminal run status. Otherwise the teardown cut the run short,
|
||||
// so use "interrupted".
|
||||
const issueId = readNonEmptyString(
|
||||
parseObject(run.contextSnapshot).issueId,
|
||||
);
|
||||
let terminalStatus: "succeeded" | "cancelled" | "interrupted" =
|
||||
"interrupted";
|
||||
if (issueId) {
|
||||
const issueStatus = await db
|
||||
.select({ status: issues.status })
|
||||
.from(issues)
|
||||
.where(eq(issues.id, issueId))
|
||||
.then((rows) => rows[0]?.status ?? null);
|
||||
if (issueStatus === "done") terminalStatus = "succeeded";
|
||||
else if (issueStatus === "cancelled") terminalStatus = "cancelled";
|
||||
// Choose the terminal status that reflects the true outcome. When the issue
|
||||
// already reached a terminal status, the run reached its goal, so use the
|
||||
// matching terminal run status. Otherwise the teardown cut the run short,
|
||||
// so use "interrupted".
|
||||
issueId = readNonEmptyString(parseObject(run.contextSnapshot).issueId);
|
||||
terminalStatus = "interrupted";
|
||||
if (issueId) {
|
||||
const issueStatus = await db
|
||||
.select({ status: issues.status })
|
||||
.from(issues)
|
||||
.where(eq(issues.id, issueId))
|
||||
.then((rows) => rows[0]?.status ?? null);
|
||||
if (issueStatus === "done") terminalStatus = "succeeded";
|
||||
else if (issueStatus === "cancelled") terminalStatus = "cancelled";
|
||||
}
|
||||
|
||||
// Teardown can beat the cancellation finalizer. Preserve the acknowledged
|
||||
// user intent instead of reporting an infrastructure interruption.
|
||||
if (hasRecordedUserStopRequest(run) || hasAcknowledgedNativeStopIntent(run) || hasAcknowledgedNativeReassignmentStopIntent(run)
|
||||
|| hasAcknowledgedNativeLifecycleStopIntent(run)) terminalStatus = "cancelled";
|
||||
|
||||
// Match both "running" and "queued". A queued run has released its lease but
|
||||
// never reached "running", so a running-only update would miss it and leave
|
||||
// a phantom live run behind.
|
||||
write = await setRunStatusFromLive(
|
||||
run.id,
|
||||
terminalStatus,
|
||||
["running", "queued"],
|
||||
{
|
||||
finishedAt: run.finishedAt ?? new Date(),
|
||||
error: run.error ?? (terminalStatus === "interrupted"
|
||||
? `run terminalized on environment lease release: heartbeat_runs.status was still ${run.status} at teardown` : null),
|
||||
errorCode:
|
||||
run.errorCode ??
|
||||
(terminalStatus === "interrupted"
|
||||
? "lease_released_before_terminal"
|
||||
: null),
|
||||
},
|
||||
undefined,
|
||||
terminalStatus === "cancelled" ? undefined : not(recordedUserStopRequestSql()),
|
||||
);
|
||||
if (write.updated) break;
|
||||
if (!write.run || isHeartbeatRunTerminalStatus(write.run.status)) break;
|
||||
}
|
||||
|
||||
// Teardown can beat the cancellation finalizer. Preserve the acknowledged
|
||||
// user intent instead of reporting an infrastructure interruption.
|
||||
if (hasAcknowledgedNativeStopIntent(run) || hasAcknowledgedNativeReassignmentStopIntent(run)
|
||||
|| hasAcknowledgedNativeLifecycleStopIntent(run)) terminalStatus = "cancelled";
|
||||
|
||||
if (!write) return run;
|
||||
const message = `run terminalized on environment lease release: heartbeat_runs.status was still ${run.status} at teardown`;
|
||||
// Match both "running" and "queued". A queued run has released its lease but
|
||||
// never reached "running", so a running-only update would miss it and leave
|
||||
// a phantom live run behind.
|
||||
const write = await setRunStatusFromLive(
|
||||
run.id,
|
||||
terminalStatus,
|
||||
["running", "queued"],
|
||||
{
|
||||
finishedAt: run.finishedAt ?? new Date(),
|
||||
error: run.error ?? (terminalStatus === "interrupted" ? message : null),
|
||||
errorCode:
|
||||
run.errorCode ??
|
||||
(terminalStatus === "interrupted"
|
||||
? "lease_released_before_terminal"
|
||||
: null),
|
||||
},
|
||||
);
|
||||
if (!write.updated) {
|
||||
// Another path already finalized the run. Keep that terminal outcome.
|
||||
return write.run ?? run;
|
||||
|
||||
Reference in new issue
Block a user