mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
fix(native): an agent termination Stop with no turn result stays a cancellation (#21093)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person terminates an agent, Paperclip stops the agent's
running native turn
> - Two paths can record that Stop: the terminate route ("Cancelled
because the agent was terminated ...") and the lifecycle worker ("Agent
lifecycle stop requested"). Both are system-initiated and expected
> - When the stopped turn ends without a semantic result, the native
executor keeps the run cancelled only for a person's Stop, a
reassignment, or a governance pause. A termination Stop fell through, so
the run ended `failed` with `native_session_interrupted`
> - Whether the run ended cancelled or failed depended on which path won
the race
> - This pull request treats an acknowledged termination Stop like a
person's Stop in the executor and in the lease-release terminalization
> - The benefit is that a terminated agent's interrupted run ends
cancelled, whichever path records the Stop first
## Linked Issues or Issue Description
Refs #20186 (the same treatment for a governance pause Stop)
**What happened?**
A person chose Terminate for an agent while its native turn ran. The
lifecycle Stop recorded an acknowledged native cancellation (initiator
system). The turn ended without a semantic result. The executor then
wrote the run as `failed` with `native_finalization_missing: session
returned no semantic result`.
**Expected behavior**
The run ends `cancelled`. A termination is an intentional Stop, not a
failure to retry.
**Steps to reproduce**
Terminate an agent while its native turn runs, so that the Stop
interrupts the turn before it records a result. Before this change, the
run became `failed` when the lifecycle worker recorded the Stop first.
## What Changed
- `governance-pause-stop.ts`: adds `AGENT_TERMINATED_STOP_REASON`,
`COMPANY_ARCHIVED_STOP_REASON` and `isAgentLifecycleStopRecord`. The
predicate accepts a system-initiated, expected control-plane
cancellation whose reason is a lifecycle, termination or company-archive
Stop.
- `acknowledged-native-stop.ts`: adds
`hasAcknowledgedNativeLifecycleStopIntent`, which requires that record
and an acknowledged, dispatched native Stop receipt.
- `native-session-executor.ts`: the no-semantic-result branch accepts
that intent, so it settles the coordinator as `native_retry_cancelled`
and raises `NativeCancellationPendingRecoveryError`, as for a person's
Stop.
- `heartbeat/run-lifecycle.ts`: lease-release terminalization keeps such
a run `cancelled`.
- The terminate route and the lifecycle driver use the shared reason
constants. The reason text does not change.
## Verification
- `native-session-executor.test.ts`: a new case for each reason
(lifecycle Stop, terminate route, company archive). Each was red before
the change and is green now. The earlier control case now uses a system
Stop that is neither a pause nor a termination, and it still takes the
retry path. Full file: 1414 passed.
- `heartbeat-run-lease-release-terminalization.test.ts`: a new case for
a lifecycle Stop when teardown wins the finalizer race. It was red
before the change (`interrupted`) and is green now.
- Product evidence: a paid chat-eval run of `agents.retire-agent` on a
branch that combines this change with three related fixes. The result is
in a PR comment.
## Risks
- A termination or company-archive Stop that interrupts a turn no longer
schedules an exact resume of that turn. The agent is leaving service, so
no resume can run.
- Other system Stops keep their current behavior.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
ca62ca9e2a
commit
67403baaa4
8 files changed
+74
-8
No files matched your search
@@ -167,6 +167,24 @@ describeEmbeddedPostgres("heartbeat terminalizeRunOnLeaseRelease", () => {
|
||||
expect(terminal).toMatchObject({ status: "cancelled", error: null, errorCode: null });
|
||||
});
|
||||
|
||||
it("preserves an acknowledged agent termination Stop when teardown wins the finalizer race", async () => {
|
||||
const { companyId, issueId, runId } = await seed({ issueStatus: "in_progress", runStatus: "running" });
|
||||
const [run] = await db.update(heartbeatRuns).set({
|
||||
nativeIssueId: issueId,
|
||||
resultJson: {
|
||||
cancellation: { reason: "Agent lifecycle stop requested", source: "control_plane", expected: true,
|
||||
initiator: { type: "system" }, recordedAt: new Date().toISOString() },
|
||||
nativeCancellation: {
|
||||
schema: "paperclip.native-cancellation.v1", runId, companyId, issueId, scope: "run",
|
||||
reasonCode: "cancellation_run_only", dispatched: true, dispatchState: "acknowledged",
|
||||
intentAuditId: randomUUID(), acknowledgementAuditId: randomUUID(),
|
||||
},
|
||||
},
|
||||
}).where(eq(heartbeatRuns.id, runId)).returning();
|
||||
const terminal = await heartbeatService(db).terminalizeRunOnLeaseRelease(run!);
|
||||
expect(terminal).toMatchObject({ status: "cancelled", error: null, errorCode: null });
|
||||
});
|
||||
|
||||
it("forces a still-queued run to interrupted when the lease releases before it starts", async () => {
|
||||
// A queued run holds a lease but never reached "running". The teardown
|
||||
// released the lease, so the run must not stay queued and show a phantom
|
||||
|
||||
@@ -167,6 +167,7 @@ import { skillVersionSelectionMap } from "../services/runtime-skill-selections.j
|
||||
import { isFixedClaudeOAuthBinding, secretService } from "../services/secrets.js";
|
||||
import { authorizationDeniedDetails } from "../services/authorization.js";
|
||||
import { providerTraceStore } from "../services/provider-trace-store.js";
|
||||
import { AGENT_TERMINATED_STOP_REASON } from "../services/governance-pause-stop.js";
|
||||
import {
|
||||
persistReprojectedWorkspaceDiffs,
|
||||
projectCodexWorkspaceDiffsFromTrace,
|
||||
@@ -6053,7 +6054,7 @@ export function agentRoutes(
|
||||
const invalidOrgChainDescendantIds = listInvalidOrgChainDescendantIds(id, companyAgentRows);
|
||||
const cancellation = await heartbeat.cancelInvocationsForAgents(
|
||||
[id, ...invalidOrgChainDescendantIds],
|
||||
"Cancelled because the agent was terminated or became invalid-org-chain under a terminated manager",
|
||||
AGENT_TERMINATED_STOP_REASON,
|
||||
);
|
||||
|
||||
await logActivity(db, {
|
||||
|
||||
@@ -7,6 +7,7 @@ import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt } from "./r
|
||||
import { validatePrpEvent } from "../vendor/paperclip-runner/index.js";
|
||||
import { nativeSha256 } from "./native-runtime/canonical.js";
|
||||
import { isConfirmedCursorCloudTerminalObservation } from "./native-runtime/cursor-cloud-runtime.js";
|
||||
import { isAgentLifecycleStopRecord } from "./governance-pause-stop.js";
|
||||
|
||||
export function hasAcknowledgedNativeRunStopRequest(run: {
|
||||
id: string; companyId: string; status: string; nativeIssueId: string | null;
|
||||
@@ -33,6 +34,10 @@ export function hasAcknowledgedNativeStopIntent(run: Parameters<typeof hasAcknow
|
||||
hasAcknowledgedNativeRunStopReceipt(run);
|
||||
}
|
||||
|
||||
export function hasAcknowledgedNativeLifecycleStopIntent(run: Parameters<typeof hasAcknowledgedNativeRunStopReceipt>[0]): boolean {
|
||||
return isAgentLifecycleStopRecord(run.resultJson) && hasAcknowledgedNativeRunStopReceipt(run);
|
||||
}
|
||||
|
||||
/** An intentional handoff retains task state and deferred work for its successor. */
|
||||
export function hasAcknowledgedNativeReassignmentStopIntent(run: Parameters<typeof hasAcknowledgedNativeRunStopReceipt>[0]): boolean {
|
||||
return run.resultJson?.reassignmentStopRequested === true && hasAcknowledgedNativeRunStopReceipt(run);
|
||||
|
||||
@@ -10,6 +10,7 @@ import type { PluginWorkerManager } from "./plugin-worker-manager.js";
|
||||
import type { LifecycleDriver } from "../modules/agent-lifecycle/index.js";
|
||||
import { agentHarnessVerificationService } from "./agent-harness-verification.js";
|
||||
import { agentInstanceSecretService } from "./agent-instance-secrets.js";
|
||||
import { AGENT_LIFECYCLE_STOP_REASON, COMPANY_ARCHIVED_STOP_REASON } from "./governance-pause-stop.js";
|
||||
|
||||
export function createLifecycleDriver(db: Db, manager: PluginWorkerManager): LifecycleDriver {
|
||||
const { verify } = agentHarnessVerificationService(db, manager);
|
||||
@@ -62,7 +63,7 @@ export function createLifecycleDriver(db: Db, manager: PluginWorkerManager): Lif
|
||||
: [agent.id, ...listInvalidOrgChainDescendantIds(agent.id,
|
||||
await db.select().from(agents).where(eq(agents.companyId, agent.companyId)))];
|
||||
const reason = agent.lifecycleHolds?.includes("company_archived")
|
||||
? "Cancelled because the company was archived" : "Agent lifecycle stop requested";
|
||||
? COMPANY_ARCHIVED_STOP_REASON : AGENT_LIFECYCLE_STOP_REASON;
|
||||
// Record what this pause stopped the runs for, so a budget resume
|
||||
// continues only the work its own budget stopped, never a manual pause.
|
||||
if (agent.lifecycleState === "pausing") {
|
||||
|
||||
@@ -10,6 +10,9 @@ export const AGENT_LIFECYCLE_STOP_REASON = "Agent lifecycle stop requested";
|
||||
export const BUDGET_PAUSE_STOP_REASON = "Cancelled due to budget pause";
|
||||
/** The direct pause route stops the agent's runs with this reason before the lifecycle worker runs. */
|
||||
export const AGENT_PAUSE_STOP_REASON = "Cancelled due to agent pause";
|
||||
export const AGENT_TERMINATED_STOP_REASON =
|
||||
"Cancelled because the agent was terminated or became invalid-org-chain under a terminated manager";
|
||||
export const COMPANY_ARCHIVED_STOP_REASON = "Cancelled because the company was archived";
|
||||
|
||||
const GOVERNANCE_PAUSE_STOP_REASONS: ReadonlySet<string> = new Set([
|
||||
AGENT_LIFECYCLE_STOP_REASON, BUDGET_PAUSE_STOP_REASON, AGENT_PAUSE_STOP_REASON]);
|
||||
@@ -27,3 +30,13 @@ export function isGovernancePauseStop(run: { status: string; resultJson?: Record
|
||||
return cancellation.reason !== AGENT_LIFECYCLE_STOP_REASON
|
||||
|| Object.keys(record(run.resultJson?.pauseStop)).length > 0;
|
||||
}
|
||||
|
||||
const AGENT_TERMINATION_STOP_REASONS: ReadonlySet<string> = new Set([
|
||||
AGENT_LIFECYCLE_STOP_REASON, AGENT_TERMINATED_STOP_REASON, COMPANY_ARCHIVED_STOP_REASON]);
|
||||
|
||||
export function isAgentLifecycleStopRecord(resultJson: Record<string, unknown> | null | undefined): boolean {
|
||||
const cancellation = record(resultJson?.cancellation);
|
||||
return record(cancellation.initiator).type === "system" && cancellation.source === "control_plane"
|
||||
&& cancellation.expected === true && typeof cancellation.reason === "string"
|
||||
&& AGENT_TERMINATION_STOP_REASONS.has(cancellation.reason);
|
||||
}
|
||||
@@ -48,6 +48,7 @@ import { isWaitingConversation } from "../agent-conversations.js";
|
||||
import { PROCESS_IDENTITY_RECORDED } from "../native-local-process-stop.js";
|
||||
import {
|
||||
hasAcknowledgedNativeReassignmentStopIntent,
|
||||
hasAcknowledgedNativeLifecycleStopIntent,
|
||||
hasAcknowledgedNativeStopIntent,
|
||||
} from "../acknowledged-native-stop.js";
|
||||
import { connectionIntentService } from "../connection-intents.js";
|
||||
@@ -1402,7 +1403,8 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl
|
||||
|
||||
// Teardown can beat the cancellation finalizer. Preserve the acknowledged
|
||||
// user intent instead of reporting an infrastructure interruption.
|
||||
if (hasAcknowledgedNativeStopIntent(run) || hasAcknowledgedNativeReassignmentStopIntent(run)) terminalStatus = "cancelled";
|
||||
if (hasAcknowledgedNativeStopIntent(run) || hasAcknowledgedNativeReassignmentStopIntent(run)
|
||||
|| hasAcknowledgedNativeLifecycleStopIntent(run)) terminalStatus = "cancelled";
|
||||
|
||||
const message = `run terminalized on environment lease release: heartbeat_runs.status was still ${run.status} at teardown`;
|
||||
// Match both "running" and "queued". A queued run has released its lease but
|
||||
|
||||
@@ -14496,12 +14496,38 @@ describe("native session bounded recovery", () => {
|
||||
} finally { releaseCleanup(); endTurn(); await outcome; }
|
||||
});
|
||||
|
||||
it("still retries a missing result after a system Stop that is not a governance pause", async () => {
|
||||
it.each([
|
||||
"Agent lifecycle stop requested",
|
||||
"Cancelled because the agent was terminated or became invalid-org-chain under a terminated manager",
|
||||
"Cancelled because the company was archived",
|
||||
])("settles an acknowledged agent termination Stop (%s) with no turn result as a cancellation", async (reason) => {
|
||||
const updates: Array<{ table: unknown; values: Record<string, unknown> }> = [];
|
||||
const stop: Record<string, unknown> = {};
|
||||
state.execute.mockReset().mockImplementationOnce(async () => {
|
||||
Object.assign(stop, {
|
||||
cancellation: { reason: "Agent lifecycle stop requested", source: "control_plane", expected: true,
|
||||
cancellation: { reason, source: "control_plane", expected: true, initiator: { type: "system" },
|
||||
recordedAt: new Date().toISOString() },
|
||||
nativeCancellation: {
|
||||
schema: "paperclip.native-cancellation.v1", ...execution.binding, scope: "run", reasonCode: "cancellation_run_only",
|
||||
dispatched: true, dispatchState: "acknowledged", intentAuditId: "intent", acknowledgementAuditId: "ack",
|
||||
} });
|
||||
throw new Error("native_finalization_missing: session returned no semantic result");
|
||||
});
|
||||
state.upsertRecoveryAction.mockClear();
|
||||
await expect(executePaperclipNativeSession({
|
||||
db: leaseDb(execution, {}, stop, updates), execution, runnerInstanceId: "termination-stop",
|
||||
})).rejects.toThrow("native_cancellation_pending_recovery");
|
||||
expect(updates.some(update => update.table === heartbeatRuns && update.values.status === "failed")).toBe(false);
|
||||
expect(updates.some(update => update.table === nativeRunFinalizations && update.values.failureCode === "native_retry_cancelled")).toBe(true);
|
||||
expect(state.upsertRecoveryAction).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("still retries a missing result after a system Stop that is neither a pause nor a termination", async () => {
|
||||
const updates: Array<{ table: unknown; values: Record<string, unknown> }> = [];
|
||||
const stop: Record<string, unknown> = {};
|
||||
state.execute.mockReset().mockImplementationOnce(async () => {
|
||||
Object.assign(stop, {
|
||||
cancellation: { reason: "Superseded by a newer execution", source: "control_plane", expected: true,
|
||||
initiator: { type: "system" }, recordedAt: new Date().toISOString() },
|
||||
nativeCancellation: {
|
||||
schema: "paperclip.native-cancellation.v1", ...execution.binding, scope: "run", reasonCode: "cancellation_run_only",
|
||||
@@ -14513,7 +14539,7 @@ describe("native session bounded recovery", () => {
|
||||
const service = vi.spyOn(issueServiceModule, "issueService").mockReturnValue({ update: vi.fn(async () => null) } as unknown as ReturnType<typeof issueServiceModule.issueService>);
|
||||
try {
|
||||
await expect(executePaperclipNativeSession({
|
||||
db: leaseDb(execution, {}, stop, updates), execution, runnerInstanceId: "termination-stop",
|
||||
db: leaseDb(execution, {}, stop, updates), execution, runnerInstanceId: "other-system-stop",
|
||||
})).rejects.not.toThrow("native_cancellation_pending_recovery");
|
||||
expect(updates.some(update => update.table === nativeRunFinalizations && update.values.failureCode === "native_retry_cancelled")).toBe(false);
|
||||
} finally { service.mockRestore(); }
|
||||
|
||||
@@ -119,7 +119,7 @@ import { imageGenerationBindingForRun } from "./generate-image-tool.js";
|
||||
import { readVerifiedRemoteWorkspaceFile } from "./remote-deliverable-file.js";
|
||||
import { copyBackCodexAuth } from "@paperclipai/adapter-codex-local/server";
|
||||
import { nativeCompletionFeedback } from "./native-completion-feedback.js";
|
||||
import { hasAcknowledgedNativeReassignmentStopIntent, hasAcknowledgedNativeStopIntent, hasAcknowledgedNativeRunStopReceipt } from "../acknowledged-native-stop.js";
|
||||
import { hasAcknowledgedNativeLifecycleStopIntent, hasAcknowledgedNativeReassignmentStopIntent, hasAcknowledgedNativeStopIntent, hasAcknowledgedNativeRunStopReceipt } from "../acknowledged-native-stop.js";
|
||||
import { stoppedCodexTurnIsTextOnly } from "./stopped-codex-turn.js";
|
||||
import { prepareVerifiedRemoteProviderPack, useVerifiedPreinstalledRemoteProviderPack, shouldUseBundledAcpxImageAssets } from "./remote-provider-pack.js";
|
||||
import { uploadVerifiedProviderPack } from "./provider-pack-upload.js";
|
||||
@@ -12709,7 +12709,7 @@ async function executePaperclipNativeSessionAttempt(
|
||||
const governancePauseStop = !!stoppedRun && hasAcknowledgedNativeRunStopReceipt(stoppedRun)
|
||||
&& isGovernancePauseStop({ status: "cancelled", resultJson: stoppedRun.resultJson });
|
||||
if (stoppedRun && (pendingStartupStop || hasAcknowledgedNativeStopIntent(stoppedRun) || hasAcknowledgedNativeReassignmentStopIntent(stoppedRun)
|
||||
|| governancePauseStop
|
||||
|| governancePauseStop || hasAcknowledgedNativeLifecycleStopIntent(stoppedRun)
|
||||
|| (error instanceof NativeCancellationPendingRecoveryError && hasAcknowledgedNativeRunStopReceipt(stoppedRun)))) {
|
||||
const [settled] = await writeExecutionControl(input.db, [DELIVERY_QUEUES.executionFinalization, DELIVERY_QUEUES.accounting], tx => tx.update(nativeRunFinalizations).set({
|
||||
phase: "terminal_failure", failureCode: "native_retry_cancelled", nextAttemptAt: null,
|
||||
|
||||
Reference in new issue
Block a user