mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(heartbeat): deliver each accepted follow-up exactly once through restarts and admission holds (#19741)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can comment, click Retry, or rely on automatic recovery while an agent works on a task > - Restart-chaos waves showed accepted follow-ups being dropped, or delivered twice, around restarts and temporary company admission holds (accounting still settling after a restart) > - A dropped comment means the user's request is never acted on. A duplicate reopens work that is already done > - This pull request makes accepted work wait through a temporary hold and resume when it clears. It also stops a comment from being promoted again after a completed run already received it > - The benefit is that each accepted follow-up is delivered exactly once, eventually, unless the task is cancelled or a reached hard stop applies ## Linked Issues or Issue Description Refs #19702 (never-started tasks during a temporary hold), #19700 (bounded accounting hold), #14910 (stale reopen from a deferred wake; no file overlap). **What happened?** Restart-chaos waves found these shapes: 1. **Comment wake cancelled.** A comment was saved behind a run and promoted when the run ended. A temporary company hold refused the unstarted queued run, and claim cancelled it terminally. The recovery wake that followed was cancelled the same way, and the task rested `blocked` without the comment. 2. **Dead Retry.** A Retry click during the hold stayed `deferred_issue_execution` after the hold cleared. Nothing woke it, and the issue held no execution lock. 3. **Refused assignment.** A new task's first wake was refused during the hold. Nothing woke stranded-work recovery when the hold cleared, so the task stayed `todo` with no run. 4. **Silent comment after recovery.** A comment was saved behind a run that restart recovery finalized. A company-scoped hint waited out the 30 s deferred-receipt rescan, so the task looked done with the message unread. 5. **Comment delivered twice.** A run whose provider process was lost had a saved comment. Its retry started after the comment, received it in its prompt (task history), did the work, and finished the task. The saved wake was then promoted anyway, which reopened the done task. **Expected behavior** An accepted follow-up (comment, Retry, recovery) is delivered exactly once, eventually, unless the task is cancelled. A temporary refusal never ends in a terminal cancel, skip, or block. Reaching a hard stop remains the board's decision. ## What Changed - `heartbeat/queue.ts`: on a budget block at claim, an unstarted queued run stays queued when the block is temporary. A reached hard stop or blocked unpriced usage still cancels it. - `recovery/service.ts`: stranded-work recovery waits out a temporary hold for any task. It does not escalate to `blocked`. This extends #19702 beyond never-started tasks. - `budget-block-ownership.ts`: a shared `isBoardOwnedBudgetBlock`, moved out of the recovery service. `budgets.ts` is unchanged. - `run-cost-accounting.ts`: settling a run's accounting wakes queued-run dispatch, the deferred-receipt worker, and stranded-work recovery. Held work waits passively; this is what resumes it. - `deferred-execution-notifications.ts`: new `notifyDeferredWakesForIssue`. `native-run-projection.ts` (a run reaching a terminal status) and `environments.ts` (lease release) now hint each saved wake of the issue by id, so delivery does not wait out the company rescan interval. - `wake-queue/adapters/postgres.ts`: release treats a saved comment as delivered only when a succeeded run of the wake's agent on that issue started after the comment's latest revision, and either received the comment as its own wake input (`wakeCommentIds`) or was a retry of the interrupted turn whose task history included it. In that case it cancels the saved wake with "Queued messages were already delivered to a completed run" and leaves the task as it is. Task history alone never consumes a message saved for its own turn. One query checks the whole batch. - `run-cost-accounting.ts`: settling late receipt debt also sends the hold-release notifications. - `doc/execution-semantics.md`: documents the rule. ## Verification Red on `master` and green with this change, on embedded Postgres: - `server/src/__tests__/heartbeat-held-comment-wake.test.ts`: - A promoted comment refused by the hold stays pending, with no cancelled run and no `blocked` task. It is delivered once after the hold clears through real `accountRunCost`. - A Retry clicked during the hold starts exactly once after accounting settles, which wakes its owner. - A new task whose assignment was refused during the hold starts once accounting settles, which wakes stranded-work recovery. - A comment that a later succeeded retry already received as history, or that a later run received as wake input, is consumed without reopening the task. - Guard tests (green on both): - A comment is still delivered in each of these cases: the later run never received it, it was edited after that run started, it was posted while the run was already running, or a non-retry run only saw it as history. - A task whose continuation was refused waits out the hold. - A reached hard stop is still refused. - `server/src/__tests__/native-run-projection-work.test.ts`: projecting a recovered run terminal hints the saved comment's own wake (red on `master`). - Related suites pass locally (14 files, 482 tests): wake-queue, queued comments routes, refused assignment wake, native run projection. - Live replays (local, unmasked, kept instances): - `--replay w-20261107T031005-s7501/t04` (Claude C1, K1): PASS. The comment's run was created at 03:56:24 during the hold, stayed queued, started at 04:06:24 when the hold cleared, and succeeded. It is the only run carrying the comment. - `--replay w-20261107T002911-s7401/t01` (before the per-wake hint): it reproduced shape 4. Run 592d772f was killed 250 ms after `run.result.accepted`, released and finalized at restart, and the task rested `done` with the comment still saved. Calling the existing sweep on the kept database promoted it, which confirmed that the missing piece was the trigger. ## Risks - A queued run can now sit queued for the length of a temporary hold, which #19700 bounds. It does not hold the issue's execution lock, so later comments still coalesce into it. - The delivered-comment rule depends on the run's recorded prompt context. A succeeded run that received the comment in its history counts as delivery, even if the agent did not act on it. This matches exactly-once delivery semantics. - Settling accounting now emits three extra notifications per run. They are coalesced per company. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in Claude Code with tool use (shell, file editing, local test execution, live restart-chaos replays). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
18b6b71d83
commit
e620ce3678
13 files changed
+429
-35
No files matched your search
@@ -0,0 +1,19 @@
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { budgetPolicies, type Db } from "@paperclipai/db";
|
||||
import { policyBlocks } from "./budgets.js";
|
||||
|
||||
/** A reached hard stop (spend at the limit, or blocked unpriced usage) is a
|
||||
* board decision. Any other budget block (completed runs awaiting accounting,
|
||||
* a native run recovering its accounting, a hold awaiting reconciliation)
|
||||
* clears by itself. Read only: the admission check already reconciled. */
|
||||
export async function isBoardOwnedBudgetBlock(
|
||||
db: Db,
|
||||
companyId: string,
|
||||
block: { scopeType: string; scopeId: string },
|
||||
) {
|
||||
const policies = await db.select().from(budgetPolicies).where(and(
|
||||
eq(budgetPolicies.companyId, companyId), eq(budgetPolicies.scopeType, block.scopeType),
|
||||
eq(budgetPolicies.scopeId, block.scopeId), eq(budgetPolicies.isActive, true)));
|
||||
for (const policy of policies) if (await policyBlocks(db, policy)) return true;
|
||||
return false;
|
||||
}
|
||||
@@ -1,5 +1,5 @@
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { sql } from "drizzle-orm";
|
||||
import { agentWakeupRequests, type Db } from "@paperclipai/db";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
|
||||
export type DeferredExecutionScope = { companyId: string; requestId?: string };
|
||||
@@ -17,6 +17,18 @@ export async function notifyDeferredExecutionWork(tx: object, scope: DeferredExe
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.deferredExecutionReceipts, hint(scope));
|
||||
}
|
||||
|
||||
/** The last gate holding one issue's saved wakes cleared (its run reached a
|
||||
* terminal status or released its environment). Hint each saved wake by id: a
|
||||
* company hint can wait out the worker's reconstruction interval, which left
|
||||
* a delivered-looking task with an unread message. */
|
||||
export async function notifyDeferredWakesForIssue(tx: object, scope: { companyId: string; issueId: string }) {
|
||||
const wakes = await (tx as Pick<Db, "select">).select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(and(
|
||||
eq(agentWakeupRequests.companyId, scope.companyId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
||||
sql`${agentWakeupRequests.payload}->>'issueId' = ${scope.issueId}`,
|
||||
)).limit(25);
|
||||
for (const wake of wakes) await notifyDeferredExecutionWork(tx, { companyId: scope.companyId, requestId: wake.id });
|
||||
}
|
||||
|
||||
export async function notifyNativeStatusWakeWork(tx: object, scope: DeferredExecutionScope) {
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.nativeStatusWakeIntents, hint(scope));
|
||||
}
|
||||
|
||||
@@ -4,7 +4,7 @@ import { notifyNativeEnvironmentSettlement } from "./native-environment-settleme
|
||||
import { notifyRetainedNativeSessionCleanup } from "./retained-native-session-cleanup-notifications.js";
|
||||
import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js";
|
||||
import { notifyEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
import { notifyDeferredExecutionWork } from "./deferred-execution-notifications.js";
|
||||
import { notifyDeferredExecutionWork, notifyDeferredWakesForIssue } from "./deferred-execution-notifications.js";
|
||||
import { withoutCleanupVerification } from "./cleanup-verification-proof.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
|
||||
@@ -1771,6 +1771,7 @@ export function environmentService(db: Db) {
|
||||
// Messages saved behind a finished run wait for its cleanup to
|
||||
// settle. Reconsider them through normal deferred admission.
|
||||
await notifyDeferredExecutionWork(tx, { companyId: previous.companyId });
|
||||
if (previous.issueId) await notifyDeferredWakesForIssue(tx, { companyId: previous.companyId, issueId: previous.issueId });
|
||||
}
|
||||
if (previous.creationOperationId && previous.status === "pending_cleanup" && options?.cleanupStatus === "success" && status !== "pending_cleanup") {
|
||||
await tx.update(sandboxCreationOperations).set({ state: "resolved", updatedAt: now })
|
||||
@@ -1905,12 +1906,15 @@ export function environmentService(db: Db) {
|
||||
): Promise<EnvironmentLease[]> => {
|
||||
const now = new Date();
|
||||
const rows = await db.transaction(async tx => {
|
||||
const [run] = await tx.select({ companyId: heartbeatRuns.companyId }).from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, heartbeatRunId)).limit(1);
|
||||
const [run] = await tx.select({ companyId: heartbeatRuns.companyId,
|
||||
issueId: sql<string | null>`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId')` })
|
||||
.from(heartbeatRuns).where(eq(heartbeatRuns.id, heartbeatRunId)).limit(1);
|
||||
if (run) {
|
||||
await notifyEnvironmentCleanup(tx, { companyId: run.companyId, runId: heartbeatRunId });
|
||||
await notifyRetainedNativeSessionCleanup(tx, { companyId: run.companyId, runId: heartbeatRunId });
|
||||
await notifyDeferredExecutionWork(tx, { companyId: run.companyId });
|
||||
if (run.issueId && /^[0-9a-f-]{36}$/i.test(run.issueId))
|
||||
await notifyDeferredWakesForIssue(tx, { companyId: run.companyId, issueId: run.issueId });
|
||||
}
|
||||
return tx.update(environmentLeases)
|
||||
.set({
|
||||
|
||||
@@ -186,6 +186,7 @@ import type { HeartbeatRecoveryDependencies, createHeartbeatRecovery } from "./r
|
||||
import { executeIssuePostCommitActions, type IssuePostCommitAction, type issueService } from "../issues.js";
|
||||
import type { issueTreeControlService } from "../issue-tree-control.js";
|
||||
import type { budgetService } from "../budgets.js";
|
||||
import { isBoardOwnedBudgetBlock } from "../budget-block-ownership.js";
|
||||
import type { instanceSettingsService } from "../instance-settings.js";
|
||||
import type { createRunDispatch } from "../../modules/run-dispatch/index.js";
|
||||
import type { PostCommitEffect as WakeQueuePostCommitEffect, createWakeQueue } from "../../modules/wake-queue/index.js";
|
||||
@@ -908,6 +909,11 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
|
||||
);
|
||||
if (budgetBlock) {
|
||||
if (caseHandoffReceipt) return null;
|
||||
// A temporary hold (accounting that is still settling) only delays
|
||||
// accepted work. Cancelling the unstarted run would drop the user's
|
||||
// message, Retry, or recovery it carries; keep it queued. Settling
|
||||
// accounting re-notifies run dispatch. A reached hard stop is the board's.
|
||||
if (!run.startedAt && !(await isBoardOwnedBudgetBlock(db, run.companyId, budgetBlock))) return null;
|
||||
await cancelRunInternal(run.id, budgetBlock.reason);
|
||||
return null;
|
||||
}
|
||||
|
||||
@@ -14,6 +14,7 @@ import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-pre
|
||||
import { notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js";
|
||||
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
|
||||
import { notifyDeferredWakesForIssue } from "../deferred-execution-notifications.js";
|
||||
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
|
||||
import { emitAgentTaskRun } from "../agent-task-run-telemetry.js";
|
||||
import { isNewTerminalRunFailure, reportRunFailure } from "../run-failure-report.js";
|
||||
@@ -203,7 +204,10 @@ export async function projectCommittedRun(input: {
|
||||
})
|
||||
.where(needsRepair)
|
||||
.returning();
|
||||
if (after) await settleNativeTerminalWake(tx, after);
|
||||
if (after) {
|
||||
await settleNativeTerminalWake(tx, after);
|
||||
await notifyDeferredWakesForIssue(tx, { companyId: input.run.companyId, issueId: input.coordinator.issueId });
|
||||
}
|
||||
return after ? { before, after } : null;
|
||||
});
|
||||
|
||||
|
||||
@@ -117,7 +117,8 @@ import {
|
||||
} from "../activity-log.js";
|
||||
import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
|
||||
import { emitAgentTaskRun } from "../agent-task-run-telemetry.js";
|
||||
import { budgetService, policyBlocks } from "../budgets.js";
|
||||
import { budgetService } from "../budgets.js";
|
||||
import { isBoardOwnedBudgetBlock } from "../budget-block-ownership.js";
|
||||
import { unadmittedChatWakeupCondition } from "../durable-chat-wakeup.js";
|
||||
import { issueRecoveryActionService } from "../issue-recovery-actions.js";
|
||||
import { PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS } from "../system-recovery-state.js";
|
||||
@@ -2157,21 +2158,6 @@ export function recoveryService(
|
||||
return Boolean(await readInvocationBudgetBlock(issue, agentId));
|
||||
}
|
||||
|
||||
/** A reached hard stop (spend at the limit, or blocked unpriced usage) is a
|
||||
* board decision. Any other budget block (completed runs awaiting accounting,
|
||||
* a native run recovering its accounting, a hold awaiting reconciliation)
|
||||
* clears by itself. Read only: the admission check already reconciled. */
|
||||
async function isBoardOwnedBudgetBlock(
|
||||
companyId: string,
|
||||
block: { scopeType: string; scopeId: string },
|
||||
) {
|
||||
const policies = await db.select().from(budgetPolicies).where(and(
|
||||
eq(budgetPolicies.companyId, companyId), eq(budgetPolicies.scopeType, block.scopeType),
|
||||
eq(budgetPolicies.scopeId, block.scopeId), eq(budgetPolicies.isActive, true)));
|
||||
for (const policy of policies) if (await policyBlocks(db, policy)) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
async function reconcileUnassignedBlockingIssues(opts?: { companyId?: string }) {
|
||||
const candidates = await db
|
||||
.select({
|
||||
@@ -4846,12 +4832,11 @@ export function recoveryService(
|
||||
}
|
||||
}
|
||||
const budgetBlock = await readInvocationBudgetBlock(issue, agentId);
|
||||
// A task that never started has nothing to recover: a temporary accounting
|
||||
// hold only delays its first admission. Leave it `todo` so the assigned-todo
|
||||
// liveness dispatch below admits it once the hold clears. A reached hard
|
||||
// stop still goes to the board.
|
||||
if (budgetBlock && issue.status === "todo" && !latestRun
|
||||
&& !(await isBoardOwnedBudgetBlock(issue.companyId, budgetBlock))) {
|
||||
// A temporary hold (accounting still settling) only delays the task's
|
||||
// start or continuation; it is not a reason to escalate. Leave the task
|
||||
// as it is: settling accounting wakes this recovery again, which then
|
||||
// admits it. A reached hard stop still goes to the board.
|
||||
if (budgetBlock && !(await isBoardOwnedBudgetBlock(db, issue.companyId, budgetBlock))) {
|
||||
result.skipped += 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -1,4 +1,7 @@
|
||||
import { notifyStrandedAgentStatus } from "./live-run-recovery-notifications.js";
|
||||
import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js";
|
||||
import { notifyRunDispatch } from "./run-dispatch-notifications.js";
|
||||
import { notifyIssueRecovery } from "./issue-recovery-notifications.js";
|
||||
import { usdToUnits, unitsToCents } from "@paperclipai/shared";
|
||||
import { and, asc, eq, inArray, isNull, or, sql } from "drizzle-orm";
|
||||
import { budgetReservations, agents, environmentLeases, heartbeatRunEvents, heartbeatRuns, nativeRunFinalizations, runUsageReceipts, issues, projects, decisionInvocations, promptInjectionInvocations, type Db } from "@paperclipai/db";
|
||||
@@ -117,6 +120,15 @@ const nativeProviderWorkNeverStarted = sql`${heartbeatRuns.runtimeMode} = 'nativ
|
||||
* grace plus a margin, a terminal run whose recovery has settled no longer
|
||||
* holds new company work: its debt settles as unpriced, flagged so the
|
||||
* blocking policy still counts it and the board sees which run it was. */
|
||||
/** A run's accounting settled: an admission hold may have cleared. Held work
|
||||
* (queued runs, saved messages and Retries, and tasks whose start or
|
||||
* continuation was refused) waits passively, so wake its owners. */
|
||||
async function notifyAccountingHoldRelease(tx: object, companyId: string) {
|
||||
await notifyRunDispatch(tx, { companyId });
|
||||
await notifyDeferredExecutionEligibility(tx, { companyId });
|
||||
await notifyIssueRecovery(tx, { companyId });
|
||||
}
|
||||
|
||||
export const ACCOUNTING_CLOSURE_TIMEOUT_MS = REMOTE_RUNNER_RECONNECT_GRACE_MS + 3 * 60_000;
|
||||
/** `at` is one shared cutoff, so the actionable scan and the wake query
|
||||
* cannot both miss a run whose bound falls between them. */
|
||||
@@ -205,9 +217,13 @@ export async function accountRunCost(db: Db, runId: string, hooks: BudgetService
|
||||
let [run] = await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("update");
|
||||
if (!run?.costAccountingPending || !terminalStatuses.includes(run.status)) return false;
|
||||
if (run.costAccountedAt) {
|
||||
if (object(run.usageJson).accountingLateReceiptPending === true)
|
||||
if (object(run.usageJson).accountingLateReceiptPending === true) {
|
||||
// Clearing late debt can release an admission hold too.
|
||||
await notifyAccountingHoldRelease(tx, run.companyId);
|
||||
return reconcileLateRunUsageReceiptInTransaction(tx, run, publications);
|
||||
}
|
||||
await notifyStrandedAgentStatus(tx, { companyId: run.companyId, agentId: run.agentId });
|
||||
await notifyAccountingHoldRelease(tx, run.companyId);
|
||||
await tx.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, run.id));
|
||||
await queueBudgetEvaluation(tx, { companyId: run.companyId, agentId: run.agentId,
|
||||
projectId: text(object(object(run.usageJson).ledgerScope).projectId) });
|
||||
@@ -249,6 +265,7 @@ export async function accountRunCost(db: Db, runId: string, hooks: BudgetService
|
||||
// Acknowledgement is part of the same transaction. Budget checks inside
|
||||
// this transaction must not mistake this receipt for missing accounting.
|
||||
await notifyStrandedAgentStatus(tx, { companyId: run.companyId, agentId: run.agentId });
|
||||
await notifyAccountingHoldRelease(tx, run.companyId);
|
||||
await tx.update(heartbeatRuns).set({ costAccountingPending: false, costAccountedAt: new Date() }).where(eq(heartbeatRuns.id, run.id));
|
||||
await tx.update(budgetReservations).set({ state: preProviderFailure ? "released" : "settled", settledAt: new Date() })
|
||||
.where(and(eq(budgetReservations.companyId, run.companyId), eq(budgetReservations.runId, run.id), eq(budgetReservations.state, "held")));
|
||||
@@ -326,6 +343,7 @@ export async function accountRunCost(db: Db, runId: string, hooks: BudgetService
|
||||
await queueBudgetEvaluation(tx, { companyId: run.companyId, agentId: run.agentId, projectId: text(scope.projectId) });
|
||||
// Late usage reported by a later run can arrive before this charge posts.
|
||||
const lateDebt = usage.accountingLateReceiptPending === true;
|
||||
if (!lateDebt) await notifyAccountingHoldRelease(tx, run.companyId);
|
||||
await tx.update(heartbeatRuns).set({ costAccountingPending: lateDebt, costAccountedAt: new Date(), accountingProjectionVersion: "v2",
|
||||
accountingLastError: lateDebt ? "late_usage_requires_reconciliation" : null, accountingLastAttemptAt: new Date(),
|
||||
// A complete receipt arriving later may price this charge (see usage-receipts).
|
||||
|
||||
Reference in new issue
Block a user