Files
PaperClipAI/server/src/services/acknowledged-native-stop.ts
T
DottaandClaude Opus 5.5 67403baaa4 fix(native): an agent termination Stop with no turn result stays a cancellation (#21093)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person terminates an agent, Paperclip stops the agent's
running native turn
> - Two paths can record that Stop: the terminate route ("Cancelled
because the agent was terminated ...") and the lifecycle worker ("Agent
lifecycle stop requested"). Both are system-initiated and expected
> - When the stopped turn ends without a semantic result, the native
executor keeps the run cancelled only for a person's Stop, a
reassignment, or a governance pause. A termination Stop fell through, so
the run ended `failed` with `native_session_interrupted`
> - Whether the run ended cancelled or failed depended on which path won
the race
> - This pull request treats an acknowledged termination Stop like a
person's Stop in the executor and in the lease-release terminalization
> - The benefit is that a terminated agent's interrupted run ends
cancelled, whichever path records the Stop first

## Linked Issues or Issue Description

Refs #20186 (the same treatment for a governance pause Stop)

**What happened?**
A person chose Terminate for an agent while its native turn ran. The
lifecycle Stop recorded an acknowledged native cancellation (initiator
system). The turn ended without a semantic result. The executor then
wrote the run as `failed` with `native_finalization_missing: session
returned no semantic result`.

**Expected behavior**
The run ends `cancelled`. A termination is an intentional Stop, not a
failure to retry.

**Steps to reproduce**
Terminate an agent while its native turn runs, so that the Stop
interrupts the turn before it records a result. Before this change, the
run became `failed` when the lifecycle worker recorded the Stop first.

## What Changed

- `governance-pause-stop.ts`: adds `AGENT_TERMINATED_STOP_REASON`,
`COMPANY_ARCHIVED_STOP_REASON` and `isAgentLifecycleStopRecord`. The
predicate accepts a system-initiated, expected control-plane
cancellation whose reason is a lifecycle, termination or company-archive
Stop.
- `acknowledged-native-stop.ts`: adds
`hasAcknowledgedNativeLifecycleStopIntent`, which requires that record
and an acknowledged, dispatched native Stop receipt.
- `native-session-executor.ts`: the no-semantic-result branch accepts
that intent, so it settles the coordinator as `native_retry_cancelled`
and raises `NativeCancellationPendingRecoveryError`, as for a person's
Stop.
- `heartbeat/run-lifecycle.ts`: lease-release terminalization keeps such
a run `cancelled`.
- The terminate route and the lifecycle driver use the shared reason
constants. The reason text does not change.

## Verification

- `native-session-executor.test.ts`: a new case for each reason
(lifecycle Stop, terminate route, company archive). Each was red before
the change and is green now. The earlier control case now uses a system
Stop that is neither a pause nor a termination, and it still takes the
retry path. Full file: 1414 passed.
- `heartbeat-run-lease-release-terminalization.test.ts`: a new case for
a lifecycle Stop when teardown wins the finalizer race. It was red
before the change (`interrupted`) and is green now.
- Product evidence: a paid chat-eval run of `agents.retire-agent` on a
branch that combines this change with three related fixes. The result is
in a PR comment.

## Risks

- A termination or company-archive Stop that interrupts a turn no longer
schedules an exact resume of that turn. The agent is leaving service, so
no resume can run.
- Other system Stops keep their current behavior.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:19:21 -06:00

116 lines
8.0 KiB
TypeScript

import { and, desc, eq, inArray, isNotNull } from "drizzle-orm";
import { environmentLeases, heartbeatRunEvents, heartbeatRuns, type Db } from "@paperclipai/db";
import { parseObject } from "../adapters/utils.js";
import { resolveProviderApiPlacement } from "./native-runtime/provider-api-placement.js";
import { hasNativeLocalProcessStop } from "./native-local-process-stop.js";
import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
import { validatePrpEvent } from "../vendor/paperclip-runner/index.js";
import { nativeSha256 } from "./native-runtime/canonical.js";
import { isConfirmedCursorCloudTerminalObservation } from "./native-runtime/cursor-cloud-runtime.js";
import { isAgentLifecycleStopRecord } from "./governance-pause-stop.js";
export function hasAcknowledgedNativeRunStopRequest(run: {
id: string; companyId: string; status: string; nativeIssueId: string | null;
resultJson: Record<string, unknown> | null;
}): boolean {
const result = run.resultJson;
const intent = result?.nativeCancellation as Record<string, unknown> | undefined;
return intent?.schema === "paperclip.native-cancellation.v1" && intent.runId === run.id &&
intent.companyId === run.companyId && intent.issueId === run.nativeIssueId &&
intent.scope === "run" && intent.reasonCode === "cancellation_run_only" &&
intent.dispatchState === "acknowledged" && typeof intent.dispatched === "boolean" &&
typeof intent.intentAuditId === "string" && Boolean(intent.intentAuditId) &&
typeof intent.acknowledgementAuditId === "string" && Boolean(intent.acknowledgementAuditId);
}
export function hasAcknowledgedNativeRunStopReceipt(run: Parameters<typeof hasAcknowledgedNativeRunStopRequest>[0]): boolean {
return hasAcknowledgedNativeRunStopRequest(run) &&
(run.resultJson?.nativeCancellation as Record<string, unknown>).dispatched === true;
}
export function hasAcknowledgedNativeStopIntent(run: Parameters<typeof hasAcknowledgedNativeRunStopReceipt>[0]): boolean {
return run.resultJson?.cancelledByActorType === "user" &&
typeof run.resultJson.cancelledByUserId === "string" && Boolean(run.resultJson.cancelledByUserId) &&
hasAcknowledgedNativeRunStopReceipt(run);
}
export function hasAcknowledgedNativeLifecycleStopIntent(run: Parameters<typeof hasAcknowledgedNativeRunStopReceipt>[0]): boolean {
return isAgentLifecycleStopRecord(run.resultJson) && hasAcknowledgedNativeRunStopReceipt(run);
}
/** An intentional handoff retains task state and deferred work for its successor. */
export function hasAcknowledgedNativeReassignmentStopIntent(run: Parameters<typeof hasAcknowledgedNativeRunStopReceipt>[0]): boolean {
return run.resultJson?.reassignmentStopRequested === true && hasAcknowledgedNativeRunStopReceipt(run);
}
export function isAcknowledgedNativeReassignmentStop(run: Parameters<typeof hasAcknowledgedNativeRunStopReceipt>[0]): boolean {
return run.status === "cancelled" && hasAcknowledgedNativeReassignmentStopIntent(run);
}
export function isAcknowledgedNativeStop(run: Parameters<typeof hasAcknowledgedNativeStopIntent>[0]): boolean {
return run.status === "cancelled" && hasAcknowledgedNativeStopIntent(run);
}
/** API-owned execution has no Computer lease. Its provider terminal and retired
* local controller must both be recorded before publishing shared Stop proof. */
export async function acknowledgedNativeApiStopExecutionHasStopped(db: Db, run: Pick<typeof heartbeatRuns.$inferSelect,
"id" | "companyId" | "agentId" | "nativeIssueId" | "status" | "resultJson" | "processPid" | "processGroupId" |
"runtimeMode" | "driverKind" | "runnerProfileJson" | "nativeSessionId" | "runnerInstanceId">): Promise<boolean> {
if (run.runtimeMode !== "native" || run.driverKind !== "cursor_cloud_api" || !isAcknowledgedNativeStop(run) ||
run.processPid !== null || run.processGroupId !== null || !run.nativeIssueId || !run.nativeSessionId || !run.runnerInstanceId) return false;
const intent = parseObject(run.resultJson?.nativeCancellation);
const requestedAt = typeof intent.recordedAt === "string" ? Date.parse(intent.recordedAt) : NaN;
if (!Number.isFinite(requestedAt) || !Array.isArray(intent.effects) || intent.effects.length !== 1 ||
intent.effects[0] !== "release_run_resources" ||
!String(intent.intentAuditId ?? "").trim() || !String(intent.acknowledgementAuditId ?? "").trim()) return false;
const profile = parseObject(run.runnerProfileJson);
const execution = parseObject(profile.nativeExecutionInput);
const binding = parseObject(execution.binding), session = parseObject(execution.session);
if (profile.executionPlacement === undefined || profile.backend !== "cursor_cloud_api" ||
binding.companyId !== run.companyId || binding.runId !== run.id || binding.issueId !== run.nativeIssueId ||
binding.agentId !== run.agentId || session.normalizedSessionId !== run.nativeSessionId || session.driverKind !== "cursor_cloud_api") return false;
try {
if (!resolveProviderApiPlacement({ run, agent: { adapterType: "process", adapterConfig: {} } })) return false;
} catch { return false; }
const [lease] = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id),
)).limit(1);
if (lease || !await hasNativeLocalProcessStop(db, run.companyId, run.id)) return false;
const [terminal] = await db.select().from(heartbeatRunEvents).where(and(
eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id),
isNotNull(heartbeatRunEvents.sourceEventId),
inArray(heartbeatRunEvents.eventType, ["turn.started", "turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"]),
)).orderBy(desc(heartbeatRunEvents.seq)).limit(1);
const validated = validatePrpEvent(parseObject(terminal?.payload).prpEvent);
if (!terminal || !validated.ok || terminal.sourcePayloadSha256 !== nativeSha256(validated.event)) return false;
const event = validated.event;
return isConfirmedCursorCloudTerminalObservation({ eventType: terminal.eventType, payload: event.payload }) && terminal.agentId === run.agentId &&
event.schema === "paperclip.prp.event.v1" && [1, 2].includes(Number(event.schemaVersion)) &&
terminal.protocolSchemaVersion === event.schemaVersion && event.eventType === terminal.eventType &&
event.sourceKind === "runner" && event.runId === run.id && event.normalizedSessionId === run.nativeSessionId &&
event.sourceInstanceId === run.runnerInstanceId && terminal.sourceInstanceId === event.sourceInstanceId &&
terminal.sourceEventId === event.sourceEventId && terminal.sourceSeq === event.sourceSeq &&
Number.isSafeInteger(event.sourceSeq) && Number(event.sourceSeq) > 0 &&
typeof event.turnId === "string" && Boolean(event.turnId);
}
/** A Stop acknowledgement alone does not prove provider cleanup completed. */
export async function acknowledgedNativeStopExecutionHasStopped(db: Db, run: Pick<typeof heartbeatRuns.$inferSelect,
"id" | "companyId" | "nativeIssueId" | "status" | "resultJson" | "processPid" | "processGroupId">) {
if (!isAcknowledgedNativeStop(run)) return false;
const leases = await db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id),
));
// Provider PIDs are meaningful only on their own host. A recovered sandbox
// queued for destroy keeps its verified stop; the destroy is only cleanup.
if (leases.some(lease => lease.provider !== "local"))
return leases.every(lease => hasRemoteTerminationReceipt(lease) || hasRecoveredCleanupStopReceipt(lease));
if (leases.some(lease => !lease.releasedAt || lease.cleanupStatus === "failed")) return false;
if (!run.processPid && !run.processGroupId) return hasNativeLocalProcessStop(db, run.companyId, run.id);
const absent = (pid: number) => {
try { process.kill(pid, 0); return false; }
catch (error) { return (error as NodeJS.ErrnoException).code === "ESRCH"; }
};
return (!run.processPid || absent(run.processPid)) && (!run.processGroupId || absent(-run.processGroupId));
}