mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 06:10:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A budget hard stop cancels the scope's running work; the board resolves it with Raise budget and resume > - In the restart-chaos H1 scenario, the cancelled tasks did not continue after resume and each needed a manual Retry > - Resume is the board's explicit decision to continue, so the work it stopped should continue once, as if the board clicked Retry > - This pull request files that Retry automatically, keyed by the cancelled run, once the run's stop has settled > - The benefit is that a budget raise restores the work without manual Retries, without repeating completed work, and without charging the run's attempt budget ## Linked Issues or Issue Description Refs #20143 (bug B; this PR builds on it), #20113 (bug A), #20101 (superseded: split into #20143 and this PR) **What happened?** Restart-chaos waves w-20261108T195944-s42 and w-20261108T210118-s9802 (H1). After `raise_budget_and_resume`, CHA-3's budget-cancelled run (enforcement stop) and CHA-1's run (agent lifecycle pause from the budget hold) stayed cancelled until someone clicked Retry. **Expected behavior** Resuming continues each task its budget stopped, exactly once, with no attempt charged, an activity entry, and no repeat of work that already continued. A run a person stopped stays stopped. **Steps to reproduce** Set a budget below spend while a task runs; let the stop cancel the run; resolve with Raise budget and resume. Before this change, no new run is queued. ## What Changed - New table `budget_resume_claims` (migration 0451). One row per cancelled run, outside the run's result JSON, so a Stop that settles during the resume cannot erase it. - `markBudgetCancelledWorkForResume` (called by `resolveIncident` before the incident closes): claims every task run its budget stopped since the incident opened. That covers runs budget enforcement fenced, including stops still in flight, and runs the agent lifecycle pause stopped for this budget (`pauseStop`, carried from the lifecycle operation's `pauseCause` so a resume before the Stop keeps the link). Runs a person stopped are excluded. Two indexed passes led by `(company_id, status)`. - `deliverBudgetResumeContinuations`: once a claimed run's stop settles, files the resolving board member's Retry (`requestTaskRetry`, idempotency key `budget-resume:<run>`, with the member's session source and the chat Retry callback). The Retry path checks, under the task lock, that the run is still the task's latest and the task is open, so continued or completed work is never repeated. Claim settlement and its activity entry commit together; retryable refusals and errors retry. - Delivery runs in the always-on budget-enforcement worker after each pass's Stop delivery. Its schedule includes deliverable claims and the earliest stale-claim expiry. The execution-continuation worker is a backstop. - `agent.paused` / `agent.resumed` logging is in #20113. `budget.cancelled_work_resume_requested` (incident) and `budget.cancelled_work_continued` / `_not_continued` (task) are recorded here. ## Verification - New `budget-resume-continuation.test.ts` (Postgres, real `resolveIncident` → claims → `requestTaskRetry` → real `resumeTaskRetry` into a queued run): - each task continues once, including a blocked task; repeat delivery or resolve saves nothing new; - a stop still in flight at resume, whose terminal write replaces the result JSON, is still continued; - a 30-task scope is fully continued; - a lifecycle-pause stop caused by the budget is continued, and one caused by a manual pause is not; - continued work, done tasks, a run that finished before its stop, and person-stopped runs (including ones enforcement also fenced) are not continued. - `lifecycle.test.ts`: the pause records its cause on the operation and on stopped runs, including when the budget is resumed before the pause's Stop. `budget-enforcement-worker.test.ts`: delivery after Stop delivery, and a wake at claim expiry. - 287 tests across the affected suites passed. Server typecheck (`typecheck-layers`, low-memory): no errors. - Not run: the H1 chaos rerun (Phase 3 runs it after merge). ## Risks - Resume now continues budget-stopped tasks automatically. A board that wants a task stopped can Stop it again; person-stopped runs are never continued. - A new table and one index (migration 0451). > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
89 lines
6.4 KiB
TypeScript
89 lines
6.4 KiB
TypeScript
import { and, eq, inArray, sql } from "drizzle-orm";
|
|
import { CANCELLABLE_HEARTBEAT_RUN_STATUSES } from "./heartbeat/recovery.js";
|
|
import { promoteDeferredWorkIntents } from "./work-intent-outbox.js";
|
|
import { agents, companies, environments, heartbeatRuns, plugins, pluginCompanySettings, type Db } from "@paperclipai/db";
|
|
import { instanceSettingsService } from "./instance-settings.js";
|
|
import { pluginHasAgentEnvironments, pluginServesAgentEnvironment } from "./plugin-environment-driver.js";
|
|
import { heartbeatService } from "./heartbeat.js";
|
|
import { listInvalidOrgChainDescendantIds } from "./agent-invokability.js";
|
|
import type { PluginWorkerManager } from "./plugin-worker-manager.js";
|
|
import type { LifecycleDriver } from "../modules/agent-lifecycle/index.js";
|
|
import { agentHarnessVerificationService } from "./agent-harness-verification.js";
|
|
import { agentInstanceSecretService } from "./agent-instance-secrets.js";
|
|
|
|
export function createLifecycleDriver(db: Db, manager: PluginWorkerManager): LifecycleDriver {
|
|
const { verify } = agentHarnessVerificationService(db, manager);
|
|
const heartbeat = heartbeatService(db, { pluginWorkerManager: manager });
|
|
return {
|
|
async onReady(agent) {
|
|
await db.transaction(tx => promoteDeferredWorkIntents(tx, { companyId: agent.companyId, trigger: "pause_lifted" }));
|
|
await heartbeat.resumeQueuedRunsForAgent(agent.companyId, agent.id);
|
|
},
|
|
async requiredPluginIds(agent) {
|
|
const selected = agent.lifecycleRequiredPluginIds;
|
|
const installed = await db.select().from(plugins).where(inArray(plugins.status, ["ready", "error", "installed", "upgrade_pending"]));
|
|
const disabled = await db.select({ pluginId: pluginCompanySettings.pluginId }).from(pluginCompanySettings)
|
|
.where(and(eq(pluginCompanySettings.companyId, agent.companyId), eq(pluginCompanySettings.enabled, false)));
|
|
const candidates = installed.filter(plugin => !disabled.some(row => row.pluginId === plugin.id) &&
|
|
plugin.manifestJson.agentLifecycle && plugin.manifestJson.capabilities.includes("agents.lifecycle.manage"));
|
|
const [row] = await db.select({ environmentId: agents.defaultEnvironmentId }).from(agents).where(eq(agents.id, agent.id));
|
|
const environmentId = row?.environmentId ?? (await instanceSettingsService(db).get()).defaultEnvironmentId;
|
|
const [environment] = environmentId ? await db.select().from(environments).where(eq(environments.id, environmentId)) : [];
|
|
if (selected !== null) {
|
|
const added = candidates.filter(plugin => pluginServesAgentEnvironment(plugin, environment) && !selected.includes(plugin.id));
|
|
return [...selected, ...added.map(plugin => plugin.id)];
|
|
}
|
|
return candidates.filter(plugin => !pluginHasAgentEnvironments(plugin) || pluginServesAgentEnvironment(plugin, environment)).map(plugin => plugin.id);
|
|
},
|
|
async runPlugin(agent, pluginId) {
|
|
const [plugin] = await db.select().from(plugins).where(eq(plugins.id, pluginId));
|
|
const [settings] = await db.select().from(pluginCompanySettings).where(and(
|
|
eq(pluginCompanySettings.pluginId, pluginId), eq(pluginCompanySettings.companyId, agent.companyId)));
|
|
if (settings?.enabled === false || !plugin || plugin.status !== "ready" || !plugin.manifestJson.agentLifecycle || !plugin.manifestJson.capabilities.includes("agents.lifecycle.manage")) {
|
|
throw Object.assign(new Error("A required lifecycle plugin is unavailable"), { code: "required_plugin_unavailable" });
|
|
}
|
|
const operationId = agent.lifecycleOperation!.id;
|
|
const secrets = agentInstanceSecretService(db);
|
|
const instanceSecrets = ["preparing", "resuming"].includes(agent.lifecycleState)
|
|
? await secrets.read({ companyId: agent.companyId, agentId: agent.id, pluginId }) : {};
|
|
const request = { companyId: agent.companyId, agentId: agent.id, operationId, version: agent.lifecycleVersion, phase: agent.lifecycleState,
|
|
instanceSecrets };
|
|
const result = await manager.call(plugin.id, "agentLifecycle", request, 30_000);
|
|
if (!result || result.operationId !== operationId || result.version !== agent.lifecycleVersion || !["complete", "pending"].includes(result.status)) {
|
|
throw Object.assign(new Error("Invalid lifecycle result"), { code: "invalid_lifecycle_result" });
|
|
}
|
|
if (result.instanceSecrets !== undefined) await secrets.apply(request, pluginId, result.instanceSecrets, agent.lifecycleOperation?.leaseOwner);
|
|
return result.status;
|
|
},
|
|
async runHost(agent) {
|
|
if (agent.lifecycleState === "verifying") return verify(agent);
|
|
if (["pausing", "terminating", "cleaning_up"].includes(agent.lifecycleState)) {
|
|
const ids = agent.lifecycleState === "pausing" ? [agent.id]
|
|
: [agent.id, ...listInvalidOrgChainDescendantIds(agent.id,
|
|
await db.select().from(agents).where(eq(agents.companyId, agent.companyId)))];
|
|
const reason = agent.lifecycleHolds?.includes("company_archived")
|
|
? "Cancelled because the company was archived" : "Agent lifecycle stop requested";
|
|
// Record what this pause stopped the runs for, so a budget resume
|
|
// continues only the work its own budget stopped, never a manual pause.
|
|
if (agent.lifecycleState === "pausing") {
|
|
// The cause recorded when the pause started survives a resume that
|
|
// already removed the holds before this Stop ran.
|
|
const cause = agent.lifecycleOperation?.pauseCause;
|
|
const pauseStop = sql`jsonb_build_object('pauseStop', jsonb_build_object(
|
|
'holds', ${JSON.stringify(cause?.holds ?? agent.lifecycleHolds ?? [])}::jsonb,
|
|
'companyPauseReason', ${cause ? sql`${cause.companyPauseReason}::text`
|
|
: sql`(select ${companies.pauseReason} from ${companies} where ${companies.id} = ${agent.companyId})`}))`;
|
|
await db.update(heartbeatRuns).set({
|
|
resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${pauseStop}`,
|
|
}).where(and(eq(heartbeatRuns.companyId, agent.companyId), eq(heartbeatRuns.agentId, agent.id),
|
|
inArray(heartbeatRuns.status, [...CANCELLABLE_HEARTBEAT_RUN_STATUSES]),
|
|
sql`not (coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) ? 'pauseStop')`));
|
|
}
|
|
if (!await heartbeat.stopInvocationsForAgents(ids, reason,
|
|
{ preservePauseDeferred: agent.lifecycleState === "pausing" })) return "pending";
|
|
}
|
|
return "complete";
|
|
},
|
|
};
|
|
}
|