mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 14:19:00 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent that delegates waits for its children, and is woken when
they finish (`issue_children_completed`)
> - Restart chaos t06 ended with both children done and the parent
blocked forever
> - The children finished while the parent's run was still going, so the
wake was deferred as a follow-up. That run then ended with the agent
setting the parent blocked, and the deferred wake was never started
> - The durable re-drivers for deferred wakes cover only
`issue_blockers_resolved`, so nothing else picks up a missed
`issue_children_completed`
> - This pull request makes both re-drivers cover children-completed
wakes too
> - The benefit is that a parent always hears that its children
finished, even when the release drain misses the deferred wake
## Linked Issues or Issue Description
Refs #19831 (durable ownership plan). Restart chaos wave
`w-20261109T041248-s9501`, task t06 (claude, M1, K2), Daytona per-turn,
master fa7950121d.
**Problem:** parent CHA-10 (all times UTC):
- 04:29:57: run 9cf49f52 started.
- 04:32:35 and 04:32:43: its children CHA-13 and CHA-14 finished.
- 04:32:43: `issue_children_completed` merged into the running turn.
- 04:32:53: that wake was deferred as a follow-up
(`issue_execution_deferred`, `deferred_issue_execution`, no run).
- 04:33:13: the turn ended with the agent setting the parent `blocked`,
a decision made before it saw the children's results.
The deferred wake stayed deferred for the rest of the wave, including
three later restarts. The parent sat blocked with no Retry. The
deferred-receipt worker (`deferred-receipt-work.ts`) and the
stranded-queue resume (`queue.ts`) select deferred dependency wakes with
`issueBlockersResolvedWakePayloadSql`, which matches only
`issue_blockers_resolved`.
## What Changed
- `server/src/services/issue-dependency-wakeups.ts`:
`dependencyResolvedWakePayloadSql` and `isDependencyResolvedWakePayload`
replace the blockers-only predicates. They match both
`issue_blockers_resolved` and `issue_children_completed` in
`_paperclipWakeContext.wakeReason`.
- The deferred-receipt worker and the stranded-queue resume use them, so
a deferred children-completed wake is started once the parent has no
live run, the same way a blockers-resolved one is.
- No code comments added.
## Verification
Red/green:
- `deferred-execution-receipt-work.test.ts` "starts a deferred
children-completed wake left behind when its parent's run blocked the
parent (restart chaos t06)". It sets up a blocked parent whose last run
succeeded, a done child, and the deferred `issue_children_completed`
receipt as t06 left it.
- On master: the receipt stays `deferred_issue_execution` (red).
- With this PR: the receipt completes with exactly one
`issue_children_completed` run.
- `deferred-execution-receipt-work` and
`heartbeat-dependency-scheduling` pass (60 tests). `tsc --noEmit -p
server` is clean.
## Risks
- The workers now also select deferred children-completed wakes.
Admission is unchanged: they still pass the queue's own gates, and they
start only when the issue has no live run, the same as blockers-resolved
wakes.
- No schema change.
## Model Used
Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with
tool use (shell, file editing).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above (#20069, #20108)
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
user docs affected)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (pending)
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(pending)
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
286 lines
11 KiB
TypeScript
286 lines
11 KiB
TypeScript
import { and, eq, inArray, sql } from "drizzle-orm";
|
|
import { createHash } from "node:crypto";
|
|
import type { Db } from "@paperclipai/db";
|
|
import { agentWakeupRequests } from "@paperclipai/db";
|
|
|
|
export const ISSUE_BLOCKERS_RESOLVED_WAKE_REASON = "issue_blockers_resolved";
|
|
|
|
export const ISSUE_CHILDREN_COMPLETED_WAKE_REASON = "issue_children_completed";
|
|
|
|
const DEPENDENCY_RESOLVED_WAKE_REASONS = [ISSUE_BLOCKERS_RESOLVED_WAKE_REASON, ISSUE_CHILDREN_COMPLETED_WAKE_REASON];
|
|
|
|
export const dependencyResolvedWakePayloadSql =
|
|
sql`${agentWakeupRequests.payload}#>>'{_paperclipWakeContext,wakeReason}' in (${sql.join(
|
|
DEPENDENCY_RESOLVED_WAKE_REASONS.map(reason => sql`${reason}`), sql`, `)})`;
|
|
|
|
export function isDependencyResolvedWakePayload(payload: Record<string, unknown> | null | undefined): boolean {
|
|
const context = payload?._paperclipWakeContext;
|
|
return context !== null && typeof context === "object" && !Array.isArray(context)
|
|
&& DEPENDENCY_RESOLVED_WAKE_REASONS.includes(String((context as Record<string, unknown>).wakeReason));
|
|
}
|
|
|
|
/**
|
|
* Whether moving an issue into `blocked` should immediately wake its assignee
|
|
* because every recorded blocker is already resolved ("blockers restored").
|
|
*
|
|
* The assignee agent blocking its own issue is declaring a wait on something
|
|
* outside the blocker graph (a pending confirmation, a human action). Waking it
|
|
* straight back made it re-block in a loop: every re-block stamps a new
|
|
* `blockedTransitionAt`, so the ready-state dedup key never matched and the
|
|
* agent was woken again on each run (five wakes in four minutes observed).
|
|
* Board or other-actor transitions and assignee changes still wake, and the
|
|
* dependency backstop still covers a genuinely stranded issue.
|
|
*/
|
|
export function shouldWakeOnRestoredBlockedDependency(input: {
|
|
previousStatus: string;
|
|
nextStatus: string;
|
|
previousAssigneeAgentId: string | null;
|
|
nextAssigneeAgentId: string | null;
|
|
blockerSetEdited: boolean;
|
|
actorType: string;
|
|
actorAgentId: string | null;
|
|
}): boolean {
|
|
if (input.nextStatus !== "blocked" || !input.nextAssigneeAgentId) return false;
|
|
const assigneeChanged = input.previousAssigneeAgentId !== input.nextAssigneeAgentId;
|
|
if (input.previousStatus === "blocked" && !input.blockerSetEdited && !assigneeChanged) return false;
|
|
const selfBlockedByAssignee =
|
|
input.actorType === "agent" &&
|
|
input.actorAgentId !== null &&
|
|
input.actorAgentId === input.nextAssigneeAgentId &&
|
|
!assigneeChanged;
|
|
return !selfBlockedByAssignee;
|
|
}
|
|
|
|
// A wake counts as "already delivered or in flight for the current ready state"
|
|
// for these statuses. The level-triggered state key uses this full set so that
|
|
// one wake for a ready state suppresses further wakes for the SAME state. This
|
|
// bounds reconciliation: after one wake, later passes find the completed row.
|
|
const IDEMPOTENT_DEPENDENCY_WAKE_STATUSES = [
|
|
"queued",
|
|
"deferred_issue_execution",
|
|
"claimed",
|
|
"completed",
|
|
] as const;
|
|
|
|
// A wake counts as "still in flight" for these statuses. The `completed` status
|
|
// is not in this set on purpose. Dependency readiness is level-triggered, so a
|
|
// historical completed per-edge wake must never suppress a new wake for the
|
|
// current ready state. The dedup uses this set only for the legacy per-edge key
|
|
// and for old no-cycle state keys that are still queued after a deploy.
|
|
const IN_FLIGHT_DEPENDENCY_WAKE_STATUSES = [
|
|
"queued",
|
|
"deferred_issue_execution",
|
|
"claimed",
|
|
] as const;
|
|
|
|
const IDEMPOTENT_DEPENDENCY_WAKE_STATUS_SET = new Set<string>(IDEMPOTENT_DEPENDENCY_WAKE_STATUSES);
|
|
const IN_FLIGHT_DEPENDENCY_WAKE_STATUS_SET = new Set<string>(IN_FLIGHT_DEPENDENCY_WAKE_STATUSES);
|
|
|
|
export type IssueBlockersResolvedWakeCycleInput = Date | string | null | undefined;
|
|
|
|
export type IssueBlockersResolvedReadyStateInput = {
|
|
dependentIssueId: string;
|
|
blockerIssueIds: string[];
|
|
blockedTransitionAt?: IssueBlockersResolvedWakeCycleInput;
|
|
};
|
|
|
|
/**
|
|
* Canonical blocked-cycle stamp for the dependency-ready state key.
|
|
* `blockedTransitionAt` is UTC ISO-8601, or `none` when the dependent has no
|
|
* recorded transition into `blocked`.
|
|
*/
|
|
export function formatIssueBlockersResolvedWakeCycle(
|
|
blockedTransitionAt: IssueBlockersResolvedWakeCycleInput,
|
|
): string {
|
|
if (blockedTransitionAt == null || blockedTransitionAt === "") return "none";
|
|
const parsed = blockedTransitionAt instanceof Date
|
|
? blockedTransitionAt
|
|
: new Date(blockedTransitionAt);
|
|
if (Number.isNaN(parsed.getTime())) return "none";
|
|
return parsed.toISOString();
|
|
}
|
|
|
|
function uniqueSortedBlockerIssueIds(blockerIssueIds: string[]): string[] {
|
|
return [...new Set(blockerIssueIds.filter(Boolean))].sort();
|
|
}
|
|
|
|
function hashBlockerReadyStateDigest(sortedBlockerIssueIds: string[], cycle: string | null): string {
|
|
const payload = cycle == null
|
|
? sortedBlockerIssueIds.join(",")
|
|
: `${sortedBlockerIssueIds.join(",")}\n${cycle}`;
|
|
return createHash("sha256").update(payload).digest("hex").slice(0, 32);
|
|
}
|
|
|
|
function buildStateKey(dependentIssueId: string, digest: string, blockerCount: number): string {
|
|
return [
|
|
ISSUE_BLOCKERS_RESOLVED_WAKE_REASON,
|
|
"state",
|
|
dependentIssueId,
|
|
String(blockerCount),
|
|
digest,
|
|
].join(":");
|
|
}
|
|
|
|
/**
|
|
* Legacy per-edge idempotency key. One key encodes a single resolved blocker
|
|
* edge `issue_blockers_resolved:{dependentIssueId}:{resolvedBlockerIssueId}`.
|
|
* The dedup keeps this format only to read wake rows written before the
|
|
* level-triggered state key existed.
|
|
*/
|
|
export function buildIssueBlockersResolvedWakeIdempotencyKey(input: {
|
|
dependentIssueId: string;
|
|
resolvedBlockerIssueId: string;
|
|
}) {
|
|
return [
|
|
ISSUE_BLOCKERS_RESOLVED_WAKE_REASON,
|
|
input.dependentIssueId,
|
|
input.resolvedBlockerIssueId,
|
|
].join(":");
|
|
}
|
|
|
|
/**
|
|
* Pre-cycle level-triggered key. Rows written before the ready state included
|
|
* `blockedTransitionAt` hashed only the sorted blocker ids. Lookup still reads
|
|
* this format so an in-flight deploy-overlap wake can suppress a duplicate.
|
|
*/
|
|
export function buildIssueBlockersResolvedWakeStateKeyWithoutCycle(input: {
|
|
dependentIssueId: string;
|
|
blockerIssueIds: string[];
|
|
}) {
|
|
const sortedBlockerIssueIds = uniqueSortedBlockerIssueIds(input.blockerIssueIds);
|
|
return buildStateKey(
|
|
input.dependentIssueId,
|
|
hashBlockerReadyStateDigest(sortedBlockerIssueIds, null),
|
|
sortedBlockerIssueIds.length,
|
|
);
|
|
}
|
|
|
|
/**
|
|
* Level-triggered idempotency key. One key encodes the full set of blockers that
|
|
* defines the current dependency-ready state plus the dependent's current
|
|
* blocked cycle (`blockedTransitionAt`, or `none`). Two wakes for the same ready
|
|
* state share the key. A wake from an earlier blocked cycle has a different
|
|
* cycle stamp, so it produces a different key and never suppresses the current
|
|
* wake. All three emit paths (route-time, finalize-time, periodic backstop) use
|
|
* this key so they share one idempotency rule.
|
|
*/
|
|
export function buildIssueBlockersResolvedWakeStateKey(input: IssueBlockersResolvedReadyStateInput) {
|
|
const sortedBlockerIssueIds = uniqueSortedBlockerIssueIds(input.blockerIssueIds);
|
|
const cycle = formatIssueBlockersResolvedWakeCycle(input.blockedTransitionAt);
|
|
return buildStateKey(
|
|
input.dependentIssueId,
|
|
hashBlockerReadyStateDigest(sortedBlockerIssueIds, cycle),
|
|
sortedBlockerIssueIds.length,
|
|
);
|
|
}
|
|
|
|
function parseWakeCycleDate(blockedTransitionAt: IssueBlockersResolvedWakeCycleInput): Date | null {
|
|
if (blockedTransitionAt == null || blockedTransitionAt === "") return null;
|
|
const parsed = blockedTransitionAt instanceof Date
|
|
? blockedTransitionAt
|
|
: new Date(blockedTransitionAt);
|
|
if (Number.isNaN(parsed.getTime())) return null;
|
|
return parsed;
|
|
}
|
|
|
|
function wakeCoversIssueBlockersResolvedReadyState(
|
|
wake: {
|
|
status: string;
|
|
idempotencyKey: string | null;
|
|
requestedAt: Date;
|
|
},
|
|
keys: {
|
|
cycleKey: string;
|
|
oldStateKey: string;
|
|
legacyKeys: Set<string>;
|
|
blockedTransitionAt: Date | null;
|
|
},
|
|
): boolean {
|
|
const idempotencyKey = wake.idempotencyKey;
|
|
if (!idempotencyKey) return false;
|
|
|
|
if (idempotencyKey === keys.cycleKey) {
|
|
return IDEMPOTENT_DEPENDENCY_WAKE_STATUS_SET.has(wake.status);
|
|
}
|
|
|
|
if (idempotencyKey === keys.oldStateKey) {
|
|
if (IN_FLIGHT_DEPENDENCY_WAKE_STATUS_SET.has(wake.status)) return true;
|
|
if (wake.status !== "completed") return false;
|
|
if (!keys.blockedTransitionAt) return true;
|
|
return wake.requestedAt.getTime() >= keys.blockedTransitionAt.getTime();
|
|
}
|
|
|
|
if (keys.legacyKeys.has(idempotencyKey)) {
|
|
return IN_FLIGHT_DEPENDENCY_WAKE_STATUS_SET.has(wake.status);
|
|
}
|
|
|
|
return false;
|
|
}
|
|
|
|
/**
|
|
* Find a wake that already covers the current dependency-ready state of the
|
|
* dependent issue. The check is level-triggered and cycle-aware:
|
|
*
|
|
* - The cycle-aware state key matches a wake in any idempotent status
|
|
* (including `completed`). This suppresses a duplicate for the SAME ready
|
|
* state, including the current blocked cycle.
|
|
* - The old no-cycle state key matches in-flight statuses (deploy overlap),
|
|
* or a `completed` wake whose `requestedAt` is at or after the current
|
|
* `blockedTransitionAt` (same cycle). A completed old-key wake from a
|
|
* previous cycle does not suppress.
|
|
* - Each legacy per-edge key matches only a wake that is still in flight.
|
|
*
|
|
* Returns the first matching wake or `null`.
|
|
*/
|
|
export async function findExistingIssueBlockersResolvedWakeForReadyState(
|
|
db: Db,
|
|
input: {
|
|
companyId: string;
|
|
dependentIssueId: string;
|
|
blockerIssueIds: string[];
|
|
blockedTransitionAt?: IssueBlockersResolvedWakeCycleInput;
|
|
},
|
|
) {
|
|
const cycleKey = buildIssueBlockersResolvedWakeStateKey(input);
|
|
const oldStateKey = buildIssueBlockersResolvedWakeStateKeyWithoutCycle(input);
|
|
const legacyKeyList = [
|
|
...new Set(
|
|
input.blockerIssueIds
|
|
.filter(Boolean)
|
|
.map((resolvedBlockerIssueId) =>
|
|
buildIssueBlockersResolvedWakeIdempotencyKey({
|
|
dependentIssueId: input.dependentIssueId,
|
|
resolvedBlockerIssueId,
|
|
}),
|
|
),
|
|
),
|
|
];
|
|
const lookupKeys = [...new Set([cycleKey, oldStateKey, ...legacyKeyList])];
|
|
const blockedTransitionAt = parseWakeCycleDate(input.blockedTransitionAt);
|
|
|
|
const rows = await db
|
|
.select({
|
|
id: agentWakeupRequests.id,
|
|
status: agentWakeupRequests.status,
|
|
idempotencyKey: agentWakeupRequests.idempotencyKey,
|
|
requestedAt: agentWakeupRequests.requestedAt,
|
|
})
|
|
.from(agentWakeupRequests)
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.companyId, input.companyId),
|
|
inArray(agentWakeupRequests.idempotencyKey, lookupKeys),
|
|
),
|
|
);
|
|
|
|
const covering = rows.find((row) =>
|
|
wakeCoversIssueBlockersResolvedReadyState(row, {
|
|
cycleKey,
|
|
oldStateKey,
|
|
legacyKeys: new Set(legacyKeyList),
|
|
blockedTransitionAt,
|
|
}),
|
|
);
|
|
return covering ?? null;
|
|
}
|