mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 04:21:24 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person terminates an agent, Paperclip stops the agent's
running native turn
> - Two paths can record that Stop: the terminate route ("Cancelled
because the agent was terminated ...") and the lifecycle worker ("Agent
lifecycle stop requested"). Both are system-initiated and expected
> - When the stopped turn ends without a semantic result, the native
executor keeps the run cancelled only for a person's Stop, a
reassignment, or a governance pause. A termination Stop fell through, so
the run ended `failed` with `native_session_interrupted`
> - Whether the run ended cancelled or failed depended on which path won
the race
> - This pull request treats an acknowledged termination Stop like a
person's Stop in the executor and in the lease-release terminalization
> - The benefit is that a terminated agent's interrupted run ends
cancelled, whichever path records the Stop first
## Linked Issues or Issue Description
Refs #20186 (the same treatment for a governance pause Stop)
**What happened?**
A person chose Terminate for an agent while its native turn ran. The
lifecycle Stop recorded an acknowledged native cancellation (initiator
system). The turn ended without a semantic result. The executor then
wrote the run as `failed` with `native_finalization_missing: session
returned no semantic result`.
**Expected behavior**
The run ends `cancelled`. A termination is an intentional Stop, not a
failure to retry.
**Steps to reproduce**
Terminate an agent while its native turn runs, so that the Stop
interrupts the turn before it records a result. Before this change, the
run became `failed` when the lifecycle worker recorded the Stop first.
## What Changed
- `governance-pause-stop.ts`: adds `AGENT_TERMINATED_STOP_REASON`,
`COMPANY_ARCHIVED_STOP_REASON` and `isAgentLifecycleStopRecord`. The
predicate accepts a system-initiated, expected control-plane
cancellation whose reason is a lifecycle, termination or company-archive
Stop.
- `acknowledged-native-stop.ts`: adds
`hasAcknowledgedNativeLifecycleStopIntent`, which requires that record
and an acknowledged, dispatched native Stop receipt.
- `native-session-executor.ts`: the no-semantic-result branch accepts
that intent, so it settles the coordinator as `native_retry_cancelled`
and raises `NativeCancellationPendingRecoveryError`, as for a person's
Stop.
- `heartbeat/run-lifecycle.ts`: lease-release terminalization keeps such
a run `cancelled`.
- The terminate route and the lifecycle driver use the shared reason
constants. The reason text does not change.
## Verification
- `native-session-executor.test.ts`: a new case for each reason
(lifecycle Stop, terminate route, company archive). Each was red before
the change and is green now. The earlier control case now uses a system
Stop that is neither a pause nor a termination, and it still takes the
retry path. Full file: 1414 passed.
- `heartbeat-run-lease-release-terminalization.test.ts`: a new case for
a lifecycle Stop when teardown wins the finalizer race. It was red
before the change (`interrupted`) and is green now.
- Product evidence: a paid chat-eval run of `agents.retire-agent` on a
branch that combines this change with three related fixes. The result is
in a PR comment.
## Risks
- A termination or company-archive Stop that interrupts a turn no longer
schedules an exact resume of that turn. The agent is leaving service, so
no resume can run.
- Other system Stops keep their current behavior.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
90 lines
6.5 KiB
TypeScript
90 lines
6.5 KiB
TypeScript
import { and, eq, inArray, sql } from "drizzle-orm";
|
|
import { CANCELLABLE_HEARTBEAT_RUN_STATUSES } from "./heartbeat/recovery.js";
|
|
import { promoteDeferredWorkIntents } from "./work-intent-outbox.js";
|
|
import { agents, companies, environments, heartbeatRuns, plugins, pluginCompanySettings, type Db } from "@paperclipai/db";
|
|
import { instanceSettingsService } from "./instance-settings.js";
|
|
import { pluginHasAgentEnvironments, pluginServesAgentEnvironment } from "./plugin-environment-driver.js";
|
|
import { heartbeatService } from "./heartbeat.js";
|
|
import { listInvalidOrgChainDescendantIds } from "./agent-invokability.js";
|
|
import type { PluginWorkerManager } from "./plugin-worker-manager.js";
|
|
import type { LifecycleDriver } from "../modules/agent-lifecycle/index.js";
|
|
import { agentHarnessVerificationService } from "./agent-harness-verification.js";
|
|
import { agentInstanceSecretService } from "./agent-instance-secrets.js";
|
|
import { AGENT_LIFECYCLE_STOP_REASON, COMPANY_ARCHIVED_STOP_REASON } from "./governance-pause-stop.js";
|
|
|
|
export function createLifecycleDriver(db: Db, manager: PluginWorkerManager): LifecycleDriver {
|
|
const { verify } = agentHarnessVerificationService(db, manager);
|
|
const heartbeat = heartbeatService(db, { pluginWorkerManager: manager });
|
|
return {
|
|
async onReady(agent) {
|
|
await db.transaction(tx => promoteDeferredWorkIntents(tx, { companyId: agent.companyId, trigger: "pause_lifted" }));
|
|
await heartbeat.resumeQueuedRunsForAgent(agent.companyId, agent.id);
|
|
},
|
|
async requiredPluginIds(agent) {
|
|
const selected = agent.lifecycleRequiredPluginIds;
|
|
const installed = await db.select().from(plugins).where(inArray(plugins.status, ["ready", "error", "installed", "upgrade_pending"]));
|
|
const disabled = await db.select({ pluginId: pluginCompanySettings.pluginId }).from(pluginCompanySettings)
|
|
.where(and(eq(pluginCompanySettings.companyId, agent.companyId), eq(pluginCompanySettings.enabled, false)));
|
|
const candidates = installed.filter(plugin => !disabled.some(row => row.pluginId === plugin.id) &&
|
|
plugin.manifestJson.agentLifecycle && plugin.manifestJson.capabilities.includes("agents.lifecycle.manage"));
|
|
const [row] = await db.select({ environmentId: agents.defaultEnvironmentId }).from(agents).where(eq(agents.id, agent.id));
|
|
const environmentId = row?.environmentId ?? (await instanceSettingsService(db).get()).defaultEnvironmentId;
|
|
const [environment] = environmentId ? await db.select().from(environments).where(eq(environments.id, environmentId)) : [];
|
|
if (selected !== null) {
|
|
const added = candidates.filter(plugin => pluginServesAgentEnvironment(plugin, environment) && !selected.includes(plugin.id));
|
|
return [...selected, ...added.map(plugin => plugin.id)];
|
|
}
|
|
return candidates.filter(plugin => !pluginHasAgentEnvironments(plugin) || pluginServesAgentEnvironment(plugin, environment)).map(plugin => plugin.id);
|
|
},
|
|
async runPlugin(agent, pluginId) {
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, pluginId));
|
|
const [settings] = await db.select().from(pluginCompanySettings).where(and(
|
|
eq(pluginCompanySettings.pluginId, pluginId), eq(pluginCompanySettings.companyId, agent.companyId)));
|
|
if (settings?.enabled === false || !plugin || plugin.status !== "ready" || !plugin.manifestJson.agentLifecycle || !plugin.manifestJson.capabilities.includes("agents.lifecycle.manage")) {
|
|
throw Object.assign(new Error("A required lifecycle plugin is unavailable"), { code: "required_plugin_unavailable" });
|
|
}
|
|
const operationId = agent.lifecycleOperation!.id;
|
|
const secrets = agentInstanceSecretService(db);
|
|
const instanceSecrets = ["preparing", "resuming"].includes(agent.lifecycleState)
|
|
? await secrets.read({ companyId: agent.companyId, agentId: agent.id, pluginId }) : {};
|
|
const request = { companyId: agent.companyId, agentId: agent.id, operationId, version: agent.lifecycleVersion, phase: agent.lifecycleState,
|
|
instanceSecrets };
|
|
const result = await manager.call(plugin.id, "agentLifecycle", request, 30_000);
|
|
if (!result || result.operationId !== operationId || result.version !== agent.lifecycleVersion || !["complete", "pending"].includes(result.status)) {
|
|
throw Object.assign(new Error("Invalid lifecycle result"), { code: "invalid_lifecycle_result" });
|
|
}
|
|
if (result.instanceSecrets !== undefined) await secrets.apply(request, pluginId, result.instanceSecrets, agent.lifecycleOperation?.leaseOwner);
|
|
return result.status;
|
|
},
|
|
async runHost(agent) {
|
|
if (agent.lifecycleState === "verifying") return verify(agent);
|
|
if (["pausing", "terminating", "cleaning_up"].includes(agent.lifecycleState)) {
|
|
const ids = agent.lifecycleState === "pausing" ? [agent.id]
|
|
: [agent.id, ...listInvalidOrgChainDescendantIds(agent.id,
|
|
await db.select().from(agents).where(eq(agents.companyId, agent.companyId)))];
|
|
const reason = agent.lifecycleHolds?.includes("company_archived")
|
|
? COMPANY_ARCHIVED_STOP_REASON : AGENT_LIFECYCLE_STOP_REASON;
|
|
// Record what this pause stopped the runs for, so a budget resume
|
|
// continues only the work its own budget stopped, never a manual pause.
|
|
if (agent.lifecycleState === "pausing") {
|
|
// The cause recorded when the pause started survives a resume that
|
|
// already removed the holds before this Stop ran.
|
|
const cause = agent.lifecycleOperation?.pauseCause;
|
|
const pauseStop = sql`jsonb_build_object('pauseStop', jsonb_build_object(
|
|
'holds', ${JSON.stringify(cause?.holds ?? agent.lifecycleHolds ?? [])}::jsonb,
|
|
'companyPauseReason', ${cause ? sql`${cause.companyPauseReason}::text`
|
|
: sql`(select ${companies.pauseReason} from ${companies} where ${companies.id} = ${agent.companyId})`}))`;
|
|
await db.update(heartbeatRuns).set({
|
|
resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${pauseStop}`,
|
|
}).where(and(eq(heartbeatRuns.companyId, agent.companyId), eq(heartbeatRuns.agentId, agent.id),
|
|
inArray(heartbeatRuns.status, [...CANCELLABLE_HEARTBEAT_RUN_STATUSES]),
|
|
sql`not (coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) ? 'pauseStop')`));
|
|
}
|
|
if (!await heartbeat.stopInvocationsForAgents(ids, reason,
|
|
{ preservePauseDeferred: agent.lifecycleState === "pausing" })) return "pending";
|
|
}
|
|
return "complete";
|
|
},
|
|
};
|
|
}
|