Files
PaperClipAI/server/src/services/work-intent-outbox.ts
T
DottaandClaude Opus 5.5 e0e6462a73 fix(heartbeat): keep an agent's explicit resume comment that a stale continuation dropped (#20856)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents coordinate on tasks through comments. A comment posted with
`resume: true` is an explicit request for more work; on a finished task
it reopens the task, for a person or for another agent.
> - A comment that arrives while the assignee's run is still settling
queues its own run. If the settling run then closes the task, that
queued run is cancelled before dispatch as a stale continuation
(`continuation_task_ownership_changed`).
> - For a person's comment, the outbox saves the unread message first
and the release drain reopens the task (chaos s9401/t72). For another
agent's explicit `resume: true` comment, nothing was saved. The request
was dropped and the task stayed `done`.
> - This pull request saves that agent comment too, with its explicit
resume intent. The existing release-drain rule for an agent's explicit
resume reopens the task and delivers the comment once.
> - The benefit is that a parent agent's "do this next turn" request is
not lost to a timing race. An agent comment without `resume: true` stays
communicative, as before.

## Linked Issues or Issue Description

**What happened?**
On a hosted staging instance, a parent agent posted a comment with
`resume: true` on its child's task: "a changed requirement for your NEXT
turn". The child's provider turn had finished 90 s earlier, but its run
was still settling (workspace copy-back). The comment queued a new run.
40 s later the settling run applied its `done` decision. The queued run
was then cancelled with `continuation_task_ownership_changed`. The task
stayed `done` and no run ever read the comment. The cancelled run itself
is not shown as a failure (the task is terminal, so task chat hides it,
and Sentry skips expected cancellations), so nothing told anyone the
request was lost.

**Steps to reproduce**
1. Run an agent on a task. While its run is still settling, post a
comment from another agent with `resume: true`.
2. Let the run close the task (`done`).
3. The comment's queued run is cancelled as a stale continuation. The
task is not reopened and the comment is never delivered.

**Expected behavior**
The task reopens for the explicit resume request, and the comment is
delivered exactly once, as for a person's comment.

## What Changed

- `deferUndeliveredCommentsOfStaleRun` finds, per owed comment, the
comment's own `issue.comment_added` activity record: written by its
author (another agent, not the assignee) with `resumeIntent: true`.
Merged runs, whose wake is rewritten as comments join, therefore keep
each comment's own intent.
- `saveUndeliveredUserComments` also admits those comments
(`explicitResumeCommentIds`). When only such comments remain, the
deferred receipt records the agent as requester and carries
`resumeIntent`/`followUpRequested` in its payload and wake context. The
release drain's existing rule for an agent's explicit resume on a `done`
task then reopens and delivers it. Person-comment behaviour is
unchanged.
- `doc/run-log-events.md` describes the rule.

## Verification

- RED on `origin/master` (`1d67e309e0`): `pnpm --filter
@paperclipai/server exec vitest run
src/__tests__/heartbeat-held-comment-wake.test.ts -t "explicit resume
comment"` fails: `expected [] to have a length of 1 but got +0` (the
comment is never delivered after the stale cancellation).
- GREEN: the whole file passes (29/29), including two-requester
merged-run tests in both orders, the new negative test (`does not reopen
a finished task for another agent's comment without resume intent`) and
the existing person-comment and exactly-once tests.
- Related suites `ownership-reconciliation-report`,
`execution-continuation`, `chat-wake-conversation-batch` plus this file:
118/118 tests pass locally. One `afterAll` cleanup hook in
`ownership-reconciliation-report` timed out under local machine load; CI
runs the suite fresh.
- `pnpm --filter @paperclipai/server exec tsc --noEmit` passes.
- Not done: no restart-chaos cell. The harness posts follow-ups as the
board user, which s9401/t72 already covers; an agent-authored `resume:
true` comment needs a two-agent workload the harness does not have.

## Risks

- Low risk. Only a comment whose own wake was an agent's explicit
`resume: true` qualifies, only on a task an agent run closed, and only
when no run read the comment. The assignee's own comments never qualify.
- The ownership reconciler's durable backstop
(`checkStaleDroppedComments`) still covers person comments only. If the
fast path fails, an agent's resume comment is not recovered later.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:57:47 -06:00

185 lines
11 KiB
TypeScript

import { and, eq, inArray, isNotNull, isNull, ne, or, sql } from "drizzle-orm";
import { agentWakeupRequests, issueComments, issues, type Db } from "@paperclipai/db";
import { notifyDeferredExecutionEligibility, notifyDeferredExecutionWork, notifyDeferredWakesForIssue } from "./deferred-execution-notifications.js";
import { withQueuedCommentIdsInWakePayload } from "./issue-queued-comment-queue.js";
import { deliveredQueuedCommentIds } from "./queued-comment-delivery.js";
import { notifyIssueRecovery } from "./issue-recovery-notifications.js";
import { notifyRunDispatch } from "./run-dispatch-notifications.js";
import { persistActivity, publishActivity, type ActivityPublication } from "./activity-log.js";
/**
* Work-intent outbox (durable ownership plan, section E; invariant 4).
*
* An accepted work intent (assignment, comment, Retry, recovery) ends in one
* of a closed set of terminal outcomes. Every other refusal is temporary: the
* intent is saved as `deferred_issue_execution` with an `executionWait` that
* names the refusal and the hold-release event that clears it. Each such event
* calls `promoteDeferredWorkIntents`, and the ownership reconciler calls it as
* a backstop for a missed event. Promotion re-runs normal admission; it never
* bypasses a gate, and the existing compare-and-swap promotion plus the
* per-comment consumption check keep delivery exactly once.
*/
/** The only terminal outcomes of an accepted work intent. */
export const WORK_INTENT_TERMINAL_OUTCOMES = ["delivered", "consumed", "task_cancelled", "hard_stop"] as const;
export type WorkIntentTerminalOutcome = (typeof WORK_INTENT_TERMINAL_OUTCOMES)[number];
/** Hold-release events. Each promotes the intents its hold deferred. */
export const WORK_INTENT_HOLD_TRIGGERS = [
"accounting_settled", "lock_released", "pause_lifted", "tree_hold_released", "scheduling_resumed", "reconciler_backstop",
] as const;
export type WorkIntentHoldTrigger = (typeof WORK_INTENT_HOLD_TRIGGERS)[number];
/** Temporary refusals of an accepted intent, and the event that clears each.
* A refusal not listed here is either a terminal outcome (a reached hard
* stop, a cancelled or reassigned task) or a signal that was never accepted
* work (a disabled timer, no actionable work). */
export const TEMPORARY_REFUSAL_TRIGGERS: Readonly<Record<string, WorkIntentHoldTrigger>> = {
"budget.blocked": "pause_lifted",
"company.inactive": "pause_lifted",
"agent.not_invokable": "pause_lifted",
"issue_tree_hold_active": "tree_hold_released",
"heartbeat.scheduling_suppressed": "scheduling_resumed",
/** A user message whose wake was lost (the reconciler saved it). */
wake_lost: "reconciler_backstop",
};
export function temporaryRefusalTrigger(reason: string): WorkIntentHoldTrigger | null {
return TEMPORARY_REFUSAL_TRIGGERS[reason] ?? null;
}
/** The `executionWait` saved on a deferred intent. */
export function deferredWorkIntentWait(reason: string, message: string, details: Record<string, unknown> = {}) {
return { ...details, reason, trigger: temporaryRefusalTrigger(reason), message };
}
/** Saved intents back off 30 s after each refusal (deferred-receipt-work.ts).
* A hold-release event makes the intents it may clear due at once. */
const DEFERRED_RETRY_BACKOFF = sql`interval '30 seconds'`;
/**
* The one promotion path for deferred work intents. Call it inside a
* transaction, normally the one that releases a hold (accounting settled, a lock released, a
* pause lifted). It wakes queued-run dispatch, the saved-receipt worker and
* stranded-work recovery for the company, and hints each saved intent of the
* issue by id. All notifications publish on commit.
*/
export async function promoteDeferredWorkIntents(tx: object, input: {
companyId: string; trigger: WorkIntentHoldTrigger; issueId?: string | null;
}) {
// Register the work signals before the write they announce.
await notifyRunDispatch(tx, { companyId: input.companyId });
await notifyDeferredExecutionEligibility(tx, { companyId: input.companyId });
await notifyIssueRecovery(tx, { companyId: input.companyId });
if (input.issueId) await notifyDeferredWakesForIssue(tx, { companyId: input.companyId, issueId: input.issueId });
const db = tx as Pick<Db, "update">;
await db.update(agentWakeupRequests).set({
updatedAt: sql`least(${agentWakeupRequests.updatedAt}, now() - ${DEFERRED_RETRY_BACKOFF})`,
}).where(and(
eq(agentWakeupRequests.companyId, input.companyId),
eq(agentWakeupRequests.status, "deferred_issue_execution"),
sql`${agentWakeupRequests.payload}->'executionWait' is not null`,
sql`${agentWakeupRequests.updatedAt} > now() - ${DEFERRED_RETRY_BACKOFF}`,
input.issueId ? sql`${agentWakeupRequests.payload}->>'issueId' = ${input.issueId}` : undefined,
));
}
const DEFERRED_WAKE_CONTEXT_KEY = "_paperclipWakeContext";
/** SQL: the latest recorded `done` transition of the task was not made by a
* person. A task a person closed (or with no recorded close) is never
* reopened for an earlier message. */
export function taskClosedByAgentSql(companyId: unknown, issueId: unknown) {
return sql`coalesce((select closed.actor_type from activity_log closed
where closed.company_id = ${companyId} and closed.entity_type = 'issue' and closed.entity_id = ${issueId}::text
and closed.action = 'issue.updated'
and (closed.details->>'toStatus' = 'done' or closed.details->>'status' = 'done')
order by closed.created_at desc limit 1), 'user') <> 'user'`;
}
/**
* Record the missing outbox entry for user messages whose wake was lost or
* ended without delivering them: one deferred receipt of the user that
* carries the undelivered comments. Only the task's assigned agent, and only
* live user comments that no completed run received, are saved. Statuses the
* caller allows (for example `done`, which the release drain reopens for a
* user's message) gate it. Normal admission and the release drain deliver
* the receipt once. Returns the receipt id, or null when nothing is owed.
*
* `explicitResumeCommentIds` also admits those comments of another agent: each
* was posted with `resume: true`, which reopens finished work as a person's
* comment does. When only such comments remain, the receipt names the latest
* one's author as requester and keeps the explicit resume intent, so the
* release drain's existing agent-resume rule reopens the task. The assignee's
* own comments never qualify.
*/
export async function saveUndeliveredUserComments(db: Db, input: {
companyId: string; issueId: string; agentId: string; commentIds: string[];
allowedStatuses: readonly string[]; requestedByActorId?: string | null;
source?: string | null; triggerDetail?: string | null;
/** Names why the receipt waits and what promotes it. */
executionWait?: Record<string, unknown>;
reopenAgentClosedDone?: boolean;
explicitResumeCommentIds?: readonly string[];
}): Promise<string | null> {
if (input.commentIds.length === 0) return null;
let publication: ActivityPublication | undefined;
const savedId = await db.transaction(async tx => {
const [issue] = await tx.select({ status: issues.status, assigneeAgentId: issues.assigneeAgentId,
conversationAgentId: issues.conversationAgentId }).from(issues)
.where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("update");
// A reassigned or cancelled task ends the intent; a conversation keeps
// its own inbound authorization path.
if (!issue || issue.assigneeAgentId !== input.agentId || issue.conversationAgentId ||
!input.allowedStatuses.includes(issue.status)) return null;
if (issue.status === "done") {
const [closed] = await tx.execute<{ byAgent: boolean }>(sql`select ${taskClosedByAgentSql(input.companyId, input.issueId)} as "byAgent"`);
if (closed?.byAgent !== true) return null;
}
const resumeIds = input.explicitResumeCommentIds?.filter(id => input.commentIds.includes(id)) ?? [];
// A direct human comment: no agent author and no creating run.
const human = and(isNotNull(issueComments.authorUserId), isNull(issueComments.authorAgentId), isNull(issueComments.createdByRunId));
const live = await tx.select({ id: issueComments.id, authorUserId: issueComments.authorUserId,
authorAgentId: issueComments.authorAgentId }).from(issueComments).where(and(
eq(issueComments.companyId, input.companyId), eq(issueComments.issueId, input.issueId),
inArray(issueComments.id, input.commentIds), isNull(issueComments.deletedAt),
resumeIds.length ? or(human, and(inArray(issueComments.id, resumeIds), isNotNull(issueComments.authorAgentId),
ne(issueComments.authorAgentId, input.agentId))) : human));
const delivered = await deliveredQueuedCommentIds(tx as unknown as Db, { companyId: input.companyId,
issueId: input.issueId, agentId: input.agentId, commentIds: live.map(row => row.id) });
const remaining = input.commentIds.filter(id => live.some(row => row.id === id) && !delivered.has(id));
if (remaining.length === 0) return null;
const latest = remaining.at(-1)!;
const resumingAgentId = live.find(row => row.id === latest)?.authorAgentId ?? null;
const agentResume = resumingAgentId !== null && remaining.every(id => live.find(row => row.id === id)?.authorAgentId);
const resumeIntent = agentResume ? { resumeIntent: true, followUpRequested: true } : {};
await notifyDeferredExecutionWork(tx, { companyId: input.companyId });
await notifyDeferredWakesForIssue(tx, { companyId: input.companyId, issueId: input.issueId });
const [saved] = await tx.insert(agentWakeupRequests).values({
companyId: input.companyId, agentId: input.agentId, source: input.source ?? "automation",
triggerDetail: input.triggerDetail ?? "system", reason: "issue_commented", status: "deferred_issue_execution",
requestedByActorType: agentResume ? "agent" : "user",
requestedByActorId: agentResume ? resumingAgentId : input.requestedByActorId
?? [...remaining].reverse().map(id => live.find(row => row.id === id)?.authorUserId).find(Boolean) ?? null,
payload: withQueuedCommentIdsInWakePayload({ issueId: input.issueId, mutation: "comment", commentId: latest,
...resumeIntent,
...(input.executionWait ? { executionWait: input.executionWait } : {}),
[DEFERRED_WAKE_CONTEXT_KEY]: { issueId: input.issueId, taskId: input.issueId, wakeReason: "issue_commented",
source: "issue.comment", wakeCommentIds: remaining, wakeCommentId: latest, ...resumeIntent } }, remaining),
}).returning({ id: agentWakeupRequests.id });
if (saved && issue.status === "done" && input.reopenAgentClosedDone) {
const { issueService } = await import("./issues.js");
const updated = await issueService(tx as unknown as Db).updateForCompany(input.issueId, input.companyId,
{ status: "todo", executionState: null }, tx);
if (!updated) throw new Error("undelivered_comment_task_changed");
publication = (await persistActivity(tx as unknown as Db, { companyId: input.companyId, actorType: "system",
actorId: "ownership-reconciler", agentId: input.agentId, action: "issue.updated", entityType: "issue",
entityId: input.issueId,
details: { status: "todo", reopened: true, reopenedFrom: "done", source: "undelivered_comment_backstop" } })).publication;
}
return saved?.id ?? null;
});
if (publication) publishActivity(publication);
return savedId;
}