Files
PaperClipAI/server/src/services/issue-dependency-wakeups.ts
T
DottaandPaperclip 5d25117ffa fix(heartbeat): recover deferred dependency completion wakes (#19457)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Tasks wait for other tasks through saved dependency edges and wake
requests.
> - A dependency can finish while its assignee answers a question about
the blocked task.
> - Paperclip saves the completion wake behind that active turn.
> - Recovery did not find that saved wake unless it also contained a
comment or an interaction response.
> - This pull request restores that wake through the existing release
and dispatch checks.
> - The task can then continue after the active turn and its cleanup
finish.

## Linked Issues or Issue Description

**What happened?**

A task stayed blocked after its dependency finished. The completion wake
was saved while an audit turn was active. The wake stayed deferred after
that turn ended. It had no comment or interaction response, so both
recovery queries and the input guard excluded it.

**Expected behavior**

Recover the saved dependency wake when the prior turn and its cleanup
finish. Start one dependency continuation under the current execution
rules.

**Steps to reproduce**

1. Run a question response on a dependency-blocked task.
2. Finish the dependency while the response is active.
3. Save the completion wake behind that response.
4. Finish the response and its environment cleanup, or restart the
workers.
5. Observe that the saved wake has no run.

**Paperclip version or commit**

Reproduced against `70d23612dd`. Rechecked against `2f35eb1080` with the
final regression fixture.

**Deployment mode**

Source checkout on Linux x64, Node 25.9.0, pnpm 9.15.4, and isolated
embedded PostgreSQL.

Related work: #16673 added recovery for saved messages. #17288 and
#16966 address assignment wakes after cleanup. This change covers
dependency completion receipts with no message.

## What Changed

- Include saved `issue_blockers_resolved` receipts in worker
reconstruction and stranded queue selection.
- Let these receipts pass the input guard without a comment or
interaction response.
- Use the existing release transaction and dispatch checks. Add no
timer, migration, or direct task status change.
- Add process-backed regression cases for restart, cleanup release,
cancellation, agent pause, pending cleanup, and unresolved blockers.
- Update the dependency wake recovery document.

- Share the dependency-wake payload predicate and SQL fragment across
reconstruction and stranded receipt admission, so these eligibility
checks have one owner.

## Verification

- The final regression fails on the base code: the receipt stays
`deferred_issue_execution` and has no successor. The same test passes
with the fix.
- Six focused cases pass. They use real PostgreSQL and a real process
adapter. They check saved output, one dependency continuation, clean
task locks, repeat recovery, and execution gates. They do not call a
model provider.
- Three nearby suites pass: 125 tests across queue wiring, dependency
scheduling, and dependency recovery.
- Final-head server layered typechecking passes with
`PAPERCLIP_TYPECHECK_LOW_MEMORY=1`. Whitespace and module boundary
checks pass. Two parallel checker attempts reached the sandbox memory
limit; the completed low-memory check covers all source and test layers.
- The full deferred-receipt suite passes: 45 tests. Combined with the
nearby suites, 170 local tests pass. These results cover head
`8f9d3aa6ac19a6d2532bd869eb8adda7cf122a5a`. Required CI remains in
progress.
- Local workspace typechecking and build stop at the Runner Rust step
because Cargo is absent. The full local test suite was not run. Required
CI must supply broad coverage.
- The hosted incident has not been recovered or deployed from this
branch.

- Review follow-up at `182385e9bc433adac06440997de7e4cfbf889a72`: normal
merge of master `0d03610d2f73b7cd799547e43929bb2303319024`; no conflict
or history rewrite. All 54 receipt/helper cases pass on macOS arm64,
Node 26.4.0, real embedded PostgreSQL and process adapters. The six
dependency continuation and safety cases also pass separately. Module
boundaries and whitespace checks pass. No assertion, timeout, execution
gate or retry policy changed for this extraction.
- Current shared, SDK and adapter-utils source builds pass. The first
broad local server typecheck read stale borrowed adapter declarations;
rebuilding them exposed nominal type duplication between the borrowed
workspace and this checkout. It is not counted as a server typecheck
pass. All 30 SDK and 169 adapter runtime JavaScript outputs match the
previously tested artifacts. Final revision
`182385e9bc433adac06440997de7e4cfbf889a72` now passes all 55 applicable
CI checks (three skipped, one neutral review), including clean-workspace
typecheck, build and broad tests. [CI
result](https://github.com/paperclipai/paperclip/actions/runs/42262173960).
Greptile and Paperclip review both give 5/5 with no open findings.

## Risks

- Recovery now considers a saved dependency receipt with no message.
Current ownership, source finalization, cleanup, pause, cancellation,
and dependency checks still control execution.
- The scan remains bounded and uses existing transaction notifications
and restart reconstruction. The integration checks verify that it
becomes quiet after settlement.

## Model Used

- OpenAI Codex, based on GPT-6. The exact provider model ID and context
size are not exposed in this session. Used reasoning, tool calls,
TypeScript edits, and test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 11:49:38 -06:00

281 lines
10 KiB
TypeScript

import { and, eq, inArray, sql } from "drizzle-orm";
import { createHash } from "node:crypto";
import type { Db } from "@paperclipai/db";
import { agentWakeupRequests } from "@paperclipai/db";
export const ISSUE_BLOCKERS_RESOLVED_WAKE_REASON = "issue_blockers_resolved";
export const issueBlockersResolvedWakePayloadSql =
sql`${agentWakeupRequests.payload}#>>'{_paperclipWakeContext,wakeReason}' = ${ISSUE_BLOCKERS_RESOLVED_WAKE_REASON}`;
export function isIssueBlockersResolvedWakePayload(payload: Record<string, unknown> | null | undefined): boolean {
const context = payload?._paperclipWakeContext;
return context !== null && typeof context === "object" && !Array.isArray(context)
&& (context as Record<string, unknown>).wakeReason === ISSUE_BLOCKERS_RESOLVED_WAKE_REASON;
}
/**
* Whether moving an issue into `blocked` should immediately wake its assignee
* because every recorded blocker is already resolved ("blockers restored").
*
* The assignee agent blocking its own issue is declaring a wait on something
* outside the blocker graph (a pending confirmation, a human action). Waking it
* straight back made it re-block in a loop: every re-block stamps a new
* `blockedTransitionAt`, so the ready-state dedup key never matched and the
* agent was woken again on each run (five wakes in four minutes observed).
* Board or other-actor transitions and assignee changes still wake, and the
* dependency backstop still covers a genuinely stranded issue.
*/
export function shouldWakeOnRestoredBlockedDependency(input: {
previousStatus: string;
nextStatus: string;
previousAssigneeAgentId: string | null;
nextAssigneeAgentId: string | null;
blockerSetEdited: boolean;
actorType: string;
actorAgentId: string | null;
}): boolean {
if (input.nextStatus !== "blocked" || !input.nextAssigneeAgentId) return false;
const assigneeChanged = input.previousAssigneeAgentId !== input.nextAssigneeAgentId;
if (input.previousStatus === "blocked" && !input.blockerSetEdited && !assigneeChanged) return false;
const selfBlockedByAssignee =
input.actorType === "agent" &&
input.actorAgentId !== null &&
input.actorAgentId === input.nextAssigneeAgentId &&
!assigneeChanged;
return !selfBlockedByAssignee;
}
// A wake counts as "already delivered or in flight for the current ready state"
// for these statuses. The level-triggered state key uses this full set so that
// one wake for a ready state suppresses further wakes for the SAME state. This
// bounds reconciliation: after one wake, later passes find the completed row.
const IDEMPOTENT_DEPENDENCY_WAKE_STATUSES = [
"queued",
"deferred_issue_execution",
"claimed",
"completed",
] as const;
// A wake counts as "still in flight" for these statuses. The `completed` status
// is not in this set on purpose. Dependency readiness is level-triggered, so a
// historical completed per-edge wake must never suppress a new wake for the
// current ready state. The dedup uses this set only for the legacy per-edge key
// and for old no-cycle state keys that are still queued after a deploy.
const IN_FLIGHT_DEPENDENCY_WAKE_STATUSES = [
"queued",
"deferred_issue_execution",
"claimed",
] as const;
const IDEMPOTENT_DEPENDENCY_WAKE_STATUS_SET = new Set<string>(IDEMPOTENT_DEPENDENCY_WAKE_STATUSES);
const IN_FLIGHT_DEPENDENCY_WAKE_STATUS_SET = new Set<string>(IN_FLIGHT_DEPENDENCY_WAKE_STATUSES);
export type IssueBlockersResolvedWakeCycleInput = Date | string | null | undefined;
export type IssueBlockersResolvedReadyStateInput = {
dependentIssueId: string;
blockerIssueIds: string[];
blockedTransitionAt?: IssueBlockersResolvedWakeCycleInput;
};
/**
* Canonical blocked-cycle stamp for the dependency-ready state key.
* `blockedTransitionAt` is UTC ISO-8601, or `none` when the dependent has no
* recorded transition into `blocked`.
*/
export function formatIssueBlockersResolvedWakeCycle(
blockedTransitionAt: IssueBlockersResolvedWakeCycleInput,
): string {
if (blockedTransitionAt == null || blockedTransitionAt === "") return "none";
const parsed = blockedTransitionAt instanceof Date
? blockedTransitionAt
: new Date(blockedTransitionAt);
if (Number.isNaN(parsed.getTime())) return "none";
return parsed.toISOString();
}
function uniqueSortedBlockerIssueIds(blockerIssueIds: string[]): string[] {
return [...new Set(blockerIssueIds.filter(Boolean))].sort();
}
function hashBlockerReadyStateDigest(sortedBlockerIssueIds: string[], cycle: string | null): string {
const payload = cycle == null
? sortedBlockerIssueIds.join(",")
: `${sortedBlockerIssueIds.join(",")}\n${cycle}`;
return createHash("sha256").update(payload).digest("hex").slice(0, 32);
}
function buildStateKey(dependentIssueId: string, digest: string, blockerCount: number): string {
return [
ISSUE_BLOCKERS_RESOLVED_WAKE_REASON,
"state",
dependentIssueId,
String(blockerCount),
digest,
].join(":");
}
/**
* Legacy per-edge idempotency key. One key encodes a single resolved blocker
* edge `issue_blockers_resolved:{dependentIssueId}:{resolvedBlockerIssueId}`.
* The dedup keeps this format only to read wake rows written before the
* level-triggered state key existed.
*/
export function buildIssueBlockersResolvedWakeIdempotencyKey(input: {
dependentIssueId: string;
resolvedBlockerIssueId: string;
}) {
return [
ISSUE_BLOCKERS_RESOLVED_WAKE_REASON,
input.dependentIssueId,
input.resolvedBlockerIssueId,
].join(":");
}
/**
* Pre-cycle level-triggered key. Rows written before the ready state included
* `blockedTransitionAt` hashed only the sorted blocker ids. Lookup still reads
* this format so an in-flight deploy-overlap wake can suppress a duplicate.
*/
export function buildIssueBlockersResolvedWakeStateKeyWithoutCycle(input: {
dependentIssueId: string;
blockerIssueIds: string[];
}) {
const sortedBlockerIssueIds = uniqueSortedBlockerIssueIds(input.blockerIssueIds);
return buildStateKey(
input.dependentIssueId,
hashBlockerReadyStateDigest(sortedBlockerIssueIds, null),
sortedBlockerIssueIds.length,
);
}
/**
* Level-triggered idempotency key. One key encodes the full set of blockers that
* defines the current dependency-ready state plus the dependent's current
* blocked cycle (`blockedTransitionAt`, or `none`). Two wakes for the same ready
* state share the key. A wake from an earlier blocked cycle has a different
* cycle stamp, so it produces a different key and never suppresses the current
* wake. All three emit paths (route-time, finalize-time, periodic backstop) use
* this key so they share one idempotency rule.
*/
export function buildIssueBlockersResolvedWakeStateKey(input: IssueBlockersResolvedReadyStateInput) {
const sortedBlockerIssueIds = uniqueSortedBlockerIssueIds(input.blockerIssueIds);
const cycle = formatIssueBlockersResolvedWakeCycle(input.blockedTransitionAt);
return buildStateKey(
input.dependentIssueId,
hashBlockerReadyStateDigest(sortedBlockerIssueIds, cycle),
sortedBlockerIssueIds.length,
);
}
function parseWakeCycleDate(blockedTransitionAt: IssueBlockersResolvedWakeCycleInput): Date | null {
if (blockedTransitionAt == null || blockedTransitionAt === "") return null;
const parsed = blockedTransitionAt instanceof Date
? blockedTransitionAt
: new Date(blockedTransitionAt);
if (Number.isNaN(parsed.getTime())) return null;
return parsed;
}
function wakeCoversIssueBlockersResolvedReadyState(
wake: {
status: string;
idempotencyKey: string | null;
requestedAt: Date;
},
keys: {
cycleKey: string;
oldStateKey: string;
legacyKeys: Set<string>;
blockedTransitionAt: Date | null;
},
): boolean {
const idempotencyKey = wake.idempotencyKey;
if (!idempotencyKey) return false;
if (idempotencyKey === keys.cycleKey) {
return IDEMPOTENT_DEPENDENCY_WAKE_STATUS_SET.has(wake.status);
}
if (idempotencyKey === keys.oldStateKey) {
if (IN_FLIGHT_DEPENDENCY_WAKE_STATUS_SET.has(wake.status)) return true;
if (wake.status !== "completed") return false;
if (!keys.blockedTransitionAt) return true;
return wake.requestedAt.getTime() >= keys.blockedTransitionAt.getTime();
}
if (keys.legacyKeys.has(idempotencyKey)) {
return IN_FLIGHT_DEPENDENCY_WAKE_STATUS_SET.has(wake.status);
}
return false;
}
/**
* Find a wake that already covers the current dependency-ready state of the
* dependent issue. The check is level-triggered and cycle-aware:
*
* - The cycle-aware state key matches a wake in any idempotent status
* (including `completed`). This suppresses a duplicate for the SAME ready
* state, including the current blocked cycle.
* - The old no-cycle state key matches in-flight statuses (deploy overlap),
* or a `completed` wake whose `requestedAt` is at or after the current
* `blockedTransitionAt` (same cycle). A completed old-key wake from a
* previous cycle does not suppress.
* - Each legacy per-edge key matches only a wake that is still in flight.
*
* Returns the first matching wake or `null`.
*/
export async function findExistingIssueBlockersResolvedWakeForReadyState(
db: Db,
input: {
companyId: string;
dependentIssueId: string;
blockerIssueIds: string[];
blockedTransitionAt?: IssueBlockersResolvedWakeCycleInput;
},
) {
const cycleKey = buildIssueBlockersResolvedWakeStateKey(input);
const oldStateKey = buildIssueBlockersResolvedWakeStateKeyWithoutCycle(input);
const legacyKeyList = [
...new Set(
input.blockerIssueIds
.filter(Boolean)
.map((resolvedBlockerIssueId) =>
buildIssueBlockersResolvedWakeIdempotencyKey({
dependentIssueId: input.dependentIssueId,
resolvedBlockerIssueId,
}),
),
),
];
const lookupKeys = [...new Set([cycleKey, oldStateKey, ...legacyKeyList])];
const blockedTransitionAt = parseWakeCycleDate(input.blockedTransitionAt);
const rows = await db
.select({
id: agentWakeupRequests.id,
status: agentWakeupRequests.status,
idempotencyKey: agentWakeupRequests.idempotencyKey,
requestedAt: agentWakeupRequests.requestedAt,
})
.from(agentWakeupRequests)
.where(
and(
eq(agentWakeupRequests.companyId, input.companyId),
inArray(agentWakeupRequests.idempotencyKey, lookupKeys),
),
);
const covering = rows.find((row) =>
wakeCoversIssueBlockersResolvedReadyState(row, {
cycleKey,
oldStateKey,
legacyKeys: new Set(legacyKeyList),
blockedTransitionAt,
}),
);
return covering ?? null;
}