From 67403baaa49e78c6ee6c9bdd8599990fcaa606ba Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 12 Nov 2026 06:19:21 -0600 Subject: [PATCH] fix(native): an agent termination Stop with no turn result stays a cancellation (#21093) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a person terminates an agent, Paperclip stops the agent's running native turn > - Two paths can record that Stop: the terminate route ("Cancelled because the agent was terminated ...") and the lifecycle worker ("Agent lifecycle stop requested"). Both are system-initiated and expected > - When the stopped turn ends without a semantic result, the native executor keeps the run cancelled only for a person's Stop, a reassignment, or a governance pause. A termination Stop fell through, so the run ended `failed` with `native_session_interrupted` > - Whether the run ended cancelled or failed depended on which path won the race > - This pull request treats an acknowledged termination Stop like a person's Stop in the executor and in the lease-release terminalization > - The benefit is that a terminated agent's interrupted run ends cancelled, whichever path records the Stop first ## Linked Issues or Issue Description Refs #20186 (the same treatment for a governance pause Stop) **What happened?** A person chose Terminate for an agent while its native turn ran. The lifecycle Stop recorded an acknowledged native cancellation (initiator system). The turn ended without a semantic result. The executor then wrote the run as `failed` with `native_finalization_missing: session returned no semantic result`. **Expected behavior** The run ends `cancelled`. A termination is an intentional Stop, not a failure to retry. **Steps to reproduce** Terminate an agent while its native turn runs, so that the Stop interrupts the turn before it records a result. Before this change, the run became `failed` when the lifecycle worker recorded the Stop first. ## What Changed - `governance-pause-stop.ts`: adds `AGENT_TERMINATED_STOP_REASON`, `COMPANY_ARCHIVED_STOP_REASON` and `isAgentLifecycleStopRecord`. The predicate accepts a system-initiated, expected control-plane cancellation whose reason is a lifecycle, termination or company-archive Stop. - `acknowledged-native-stop.ts`: adds `hasAcknowledgedNativeLifecycleStopIntent`, which requires that record and an acknowledged, dispatched native Stop receipt. - `native-session-executor.ts`: the no-semantic-result branch accepts that intent, so it settles the coordinator as `native_retry_cancelled` and raises `NativeCancellationPendingRecoveryError`, as for a person's Stop. - `heartbeat/run-lifecycle.ts`: lease-release terminalization keeps such a run `cancelled`. - The terminate route and the lifecycle driver use the shared reason constants. The reason text does not change. ## Verification - `native-session-executor.test.ts`: a new case for each reason (lifecycle Stop, terminate route, company archive). Each was red before the change and is green now. The earlier control case now uses a system Stop that is neither a pause nor a termination, and it still takes the retry path. Full file: 1414 passed. - `heartbeat-run-lease-release-terminalization.test.ts`: a new case for a lifecycle Stop when teardown wins the finalizer race. It was red before the change (`interrupted`) and is green now. - Product evidence: a paid chat-eval run of `agents.retire-agent` on a branch that combines this change with three related fixes. The result is in a PR comment. ## Risks - A termination or company-archive Stop that interrupts a turn no longer schedules an exact resume of that turn. The agent is leaving service, so no resume can run. - Other system Stops keep their current behavior. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- ...-run-lease-release-terminalization.test.ts | 18 +++++++++++ server/src/routes/agents.ts | 3 +- .../src/services/acknowledged-native-stop.ts | 5 +++ server/src/services/agent-lifecycle-driver.ts | 3 +- server/src/services/governance-pause-stop.ts | 13 ++++++++ .../src/services/heartbeat/run-lifecycle.ts | 4 ++- .../native-session-executor.test.ts | 32 +++++++++++++++++-- .../native-runtime/native-session-executor.ts | 4 +-- 8 files changed, 74 insertions(+), 8 deletions(-) diff --git a/server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts b/server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts index 872da4596e4..422f044097e 100644 --- a/server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts +++ b/server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts @@ -167,6 +167,24 @@ describeEmbeddedPostgres("heartbeat terminalizeRunOnLeaseRelease", () => { expect(terminal).toMatchObject({ status: "cancelled", error: null, errorCode: null }); }); + it("preserves an acknowledged agent termination Stop when teardown wins the finalizer race", async () => { + const { companyId, issueId, runId } = await seed({ issueStatus: "in_progress", runStatus: "running" }); + const [run] = await db.update(heartbeatRuns).set({ + nativeIssueId: issueId, + resultJson: { + cancellation: { reason: "Agent lifecycle stop requested", source: "control_plane", expected: true, + initiator: { type: "system" }, recordedAt: new Date().toISOString() }, + nativeCancellation: { + schema: "paperclip.native-cancellation.v1", runId, companyId, issueId, scope: "run", + reasonCode: "cancellation_run_only", dispatched: true, dispatchState: "acknowledged", + intentAuditId: randomUUID(), acknowledgementAuditId: randomUUID(), + }, + }, + }).where(eq(heartbeatRuns.id, runId)).returning(); + const terminal = await heartbeatService(db).terminalizeRunOnLeaseRelease(run!); + expect(terminal).toMatchObject({ status: "cancelled", error: null, errorCode: null }); + }); + it("forces a still-queued run to interrupted when the lease releases before it starts", async () => { // A queued run holds a lease but never reached "running". The teardown // released the lease, so the run must not stay queued and show a phantom diff --git a/server/src/routes/agents.ts b/server/src/routes/agents.ts index ac4b5f90d5d..0775ce81a34 100644 --- a/server/src/routes/agents.ts +++ b/server/src/routes/agents.ts @@ -167,6 +167,7 @@ import { skillVersionSelectionMap } from "../services/runtime-skill-selections.j import { isFixedClaudeOAuthBinding, secretService } from "../services/secrets.js"; import { authorizationDeniedDetails } from "../services/authorization.js"; import { providerTraceStore } from "../services/provider-trace-store.js"; +import { AGENT_TERMINATED_STOP_REASON } from "../services/governance-pause-stop.js"; import { persistReprojectedWorkspaceDiffs, projectCodexWorkspaceDiffsFromTrace, @@ -6053,7 +6054,7 @@ export function agentRoutes( const invalidOrgChainDescendantIds = listInvalidOrgChainDescendantIds(id, companyAgentRows); const cancellation = await heartbeat.cancelInvocationsForAgents( [id, ...invalidOrgChainDescendantIds], - "Cancelled because the agent was terminated or became invalid-org-chain under a terminated manager", + AGENT_TERMINATED_STOP_REASON, ); await logActivity(db, { diff --git a/server/src/services/acknowledged-native-stop.ts b/server/src/services/acknowledged-native-stop.ts index 6ad77cc89f9..25beefee409 100644 --- a/server/src/services/acknowledged-native-stop.ts +++ b/server/src/services/acknowledged-native-stop.ts @@ -7,6 +7,7 @@ import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt } from "./r import { validatePrpEvent } from "../vendor/paperclip-runner/index.js"; import { nativeSha256 } from "./native-runtime/canonical.js"; import { isConfirmedCursorCloudTerminalObservation } from "./native-runtime/cursor-cloud-runtime.js"; +import { isAgentLifecycleStopRecord } from "./governance-pause-stop.js"; export function hasAcknowledgedNativeRunStopRequest(run: { id: string; companyId: string; status: string; nativeIssueId: string | null; @@ -33,6 +34,10 @@ export function hasAcknowledgedNativeStopIntent(run: Parameters[0]): boolean { + return isAgentLifecycleStopRecord(run.resultJson) && hasAcknowledgedNativeRunStopReceipt(run); +} + /** An intentional handoff retains task state and deferred work for its successor. */ export function hasAcknowledgedNativeReassignmentStopIntent(run: Parameters[0]): boolean { return run.resultJson?.reassignmentStopRequested === true && hasAcknowledgedNativeRunStopReceipt(run); diff --git a/server/src/services/agent-lifecycle-driver.ts b/server/src/services/agent-lifecycle-driver.ts index 00cd438faf6..20fe17f001b 100644 --- a/server/src/services/agent-lifecycle-driver.ts +++ b/server/src/services/agent-lifecycle-driver.ts @@ -10,6 +10,7 @@ import type { PluginWorkerManager } from "./plugin-worker-manager.js"; import type { LifecycleDriver } from "../modules/agent-lifecycle/index.js"; import { agentHarnessVerificationService } from "./agent-harness-verification.js"; import { agentInstanceSecretService } from "./agent-instance-secrets.js"; +import { AGENT_LIFECYCLE_STOP_REASON, COMPANY_ARCHIVED_STOP_REASON } from "./governance-pause-stop.js"; export function createLifecycleDriver(db: Db, manager: PluginWorkerManager): LifecycleDriver { const { verify } = agentHarnessVerificationService(db, manager); @@ -62,7 +63,7 @@ export function createLifecycleDriver(db: Db, manager: PluginWorkerManager): Lif : [agent.id, ...listInvalidOrgChainDescendantIds(agent.id, await db.select().from(agents).where(eq(agents.companyId, agent.companyId)))]; const reason = agent.lifecycleHolds?.includes("company_archived") - ? "Cancelled because the company was archived" : "Agent lifecycle stop requested"; + ? COMPANY_ARCHIVED_STOP_REASON : AGENT_LIFECYCLE_STOP_REASON; // Record what this pause stopped the runs for, so a budget resume // continues only the work its own budget stopped, never a manual pause. if (agent.lifecycleState === "pausing") { diff --git a/server/src/services/governance-pause-stop.ts b/server/src/services/governance-pause-stop.ts index 04a1e5a5616..3c0e00fe29d 100644 --- a/server/src/services/governance-pause-stop.ts +++ b/server/src/services/governance-pause-stop.ts @@ -10,6 +10,9 @@ export const AGENT_LIFECYCLE_STOP_REASON = "Agent lifecycle stop requested"; export const BUDGET_PAUSE_STOP_REASON = "Cancelled due to budget pause"; /** The direct pause route stops the agent's runs with this reason before the lifecycle worker runs. */ export const AGENT_PAUSE_STOP_REASON = "Cancelled due to agent pause"; +export const AGENT_TERMINATED_STOP_REASON = + "Cancelled because the agent was terminated or became invalid-org-chain under a terminated manager"; +export const COMPANY_ARCHIVED_STOP_REASON = "Cancelled because the company was archived"; const GOVERNANCE_PAUSE_STOP_REASONS: ReadonlySet = new Set([ AGENT_LIFECYCLE_STOP_REASON, BUDGET_PAUSE_STOP_REASON, AGENT_PAUSE_STOP_REASON]); @@ -27,3 +30,13 @@ export function isGovernancePauseStop(run: { status: string; resultJson?: Record return cancellation.reason !== AGENT_LIFECYCLE_STOP_REASON || Object.keys(record(run.resultJson?.pauseStop)).length > 0; } + +const AGENT_TERMINATION_STOP_REASONS: ReadonlySet = new Set([ + AGENT_LIFECYCLE_STOP_REASON, AGENT_TERMINATED_STOP_REASON, COMPANY_ARCHIVED_STOP_REASON]); + +export function isAgentLifecycleStopRecord(resultJson: Record | null | undefined): boolean { + const cancellation = record(resultJson?.cancellation); + return record(cancellation.initiator).type === "system" && cancellation.source === "control_plane" + && cancellation.expected === true && typeof cancellation.reason === "string" + && AGENT_TERMINATION_STOP_REASONS.has(cancellation.reason); +} diff --git a/server/src/services/heartbeat/run-lifecycle.ts b/server/src/services/heartbeat/run-lifecycle.ts index da232abd907..f4837170077 100644 --- a/server/src/services/heartbeat/run-lifecycle.ts +++ b/server/src/services/heartbeat/run-lifecycle.ts @@ -48,6 +48,7 @@ import { isWaitingConversation } from "../agent-conversations.js"; import { PROCESS_IDENTITY_RECORDED } from "../native-local-process-stop.js"; import { hasAcknowledgedNativeReassignmentStopIntent, + hasAcknowledgedNativeLifecycleStopIntent, hasAcknowledgedNativeStopIntent, } from "../acknowledged-native-stop.js"; import { connectionIntentService } from "../connection-intents.js"; @@ -1402,7 +1403,8 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl // Teardown can beat the cancellation finalizer. Preserve the acknowledged // user intent instead of reporting an infrastructure interruption. - if (hasAcknowledgedNativeStopIntent(run) || hasAcknowledgedNativeReassignmentStopIntent(run)) terminalStatus = "cancelled"; + if (hasAcknowledgedNativeStopIntent(run) || hasAcknowledgedNativeReassignmentStopIntent(run) + || hasAcknowledgedNativeLifecycleStopIntent(run)) terminalStatus = "cancelled"; const message = `run terminalized on environment lease release: heartbeat_runs.status was still ${run.status} at teardown`; // Match both "running" and "queued". A queued run has released its lease but diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 32025817354..9e87d7bad89 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -14496,12 +14496,38 @@ describe("native session bounded recovery", () => { } finally { releaseCleanup(); endTurn(); await outcome; } }); - it("still retries a missing result after a system Stop that is not a governance pause", async () => { + it.each([ + "Agent lifecycle stop requested", + "Cancelled because the agent was terminated or became invalid-org-chain under a terminated manager", + "Cancelled because the company was archived", + ])("settles an acknowledged agent termination Stop (%s) with no turn result as a cancellation", async (reason) => { const updates: Array<{ table: unknown; values: Record }> = []; const stop: Record = {}; state.execute.mockReset().mockImplementationOnce(async () => { Object.assign(stop, { - cancellation: { reason: "Agent lifecycle stop requested", source: "control_plane", expected: true, + cancellation: { reason, source: "control_plane", expected: true, initiator: { type: "system" }, + recordedAt: new Date().toISOString() }, + nativeCancellation: { + schema: "paperclip.native-cancellation.v1", ...execution.binding, scope: "run", reasonCode: "cancellation_run_only", + dispatched: true, dispatchState: "acknowledged", intentAuditId: "intent", acknowledgementAuditId: "ack", + } }); + throw new Error("native_finalization_missing: session returned no semantic result"); + }); + state.upsertRecoveryAction.mockClear(); + await expect(executePaperclipNativeSession({ + db: leaseDb(execution, {}, stop, updates), execution, runnerInstanceId: "termination-stop", + })).rejects.toThrow("native_cancellation_pending_recovery"); + expect(updates.some(update => update.table === heartbeatRuns && update.values.status === "failed")).toBe(false); + expect(updates.some(update => update.table === nativeRunFinalizations && update.values.failureCode === "native_retry_cancelled")).toBe(true); + expect(state.upsertRecoveryAction).not.toHaveBeenCalled(); + }); + + it("still retries a missing result after a system Stop that is neither a pause nor a termination", async () => { + const updates: Array<{ table: unknown; values: Record }> = []; + const stop: Record = {}; + state.execute.mockReset().mockImplementationOnce(async () => { + Object.assign(stop, { + cancellation: { reason: "Superseded by a newer execution", source: "control_plane", expected: true, initiator: { type: "system" }, recordedAt: new Date().toISOString() }, nativeCancellation: { schema: "paperclip.native-cancellation.v1", ...execution.binding, scope: "run", reasonCode: "cancellation_run_only", @@ -14513,7 +14539,7 @@ describe("native session bounded recovery", () => { const service = vi.spyOn(issueServiceModule, "issueService").mockReturnValue({ update: vi.fn(async () => null) } as unknown as ReturnType); try { await expect(executePaperclipNativeSession({ - db: leaseDb(execution, {}, stop, updates), execution, runnerInstanceId: "termination-stop", + db: leaseDb(execution, {}, stop, updates), execution, runnerInstanceId: "other-system-stop", })).rejects.not.toThrow("native_cancellation_pending_recovery"); expect(updates.some(update => update.table === nativeRunFinalizations && update.values.failureCode === "native_retry_cancelled")).toBe(false); } finally { service.mockRestore(); } diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index fd67d8f80b9..461462c6b7d 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -119,7 +119,7 @@ import { imageGenerationBindingForRun } from "./generate-image-tool.js"; import { readVerifiedRemoteWorkspaceFile } from "./remote-deliverable-file.js"; import { copyBackCodexAuth } from "@paperclipai/adapter-codex-local/server"; import { nativeCompletionFeedback } from "./native-completion-feedback.js"; -import { hasAcknowledgedNativeReassignmentStopIntent, hasAcknowledgedNativeStopIntent, hasAcknowledgedNativeRunStopReceipt } from "../acknowledged-native-stop.js"; +import { hasAcknowledgedNativeLifecycleStopIntent, hasAcknowledgedNativeReassignmentStopIntent, hasAcknowledgedNativeStopIntent, hasAcknowledgedNativeRunStopReceipt } from "../acknowledged-native-stop.js"; import { stoppedCodexTurnIsTextOnly } from "./stopped-codex-turn.js"; import { prepareVerifiedRemoteProviderPack, useVerifiedPreinstalledRemoteProviderPack, shouldUseBundledAcpxImageAssets } from "./remote-provider-pack.js"; import { uploadVerifiedProviderPack } from "./provider-pack-upload.js"; @@ -12709,7 +12709,7 @@ async function executePaperclipNativeSessionAttempt( const governancePauseStop = !!stoppedRun && hasAcknowledgedNativeRunStopReceipt(stoppedRun) && isGovernancePauseStop({ status: "cancelled", resultJson: stoppedRun.resultJson }); if (stoppedRun && (pendingStartupStop || hasAcknowledgedNativeStopIntent(stoppedRun) || hasAcknowledgedNativeReassignmentStopIntent(stoppedRun) - || governancePauseStop + || governancePauseStop || hasAcknowledgedNativeLifecycleStopIntent(stoppedRun) || (error instanceof NativeCancellationPendingRecoveryError && hasAcknowledgedNativeRunStopReceipt(stoppedRun)))) { const [settled] = await writeExecutionControl(input.db, [DELIVERY_QUEUES.executionFinalization, DELIVERY_QUEUES.accounting], tx => tx.update(nativeRunFinalizations).set({ phase: "terminal_failure", failureCode: "native_retry_cancelled", nextAttemptAt: null,