Files
PaperClipAI/server/src/modules/run-dispatch/domain/wake-context.ts
T
DottaandClaude Opus 5.5 c49ab6b9ee feat(issues): wake the parent once when a child or blocker stalls on the board (#20098)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent task delegates to children and waits on blockers. Its agent
learns of progress only through wakes
> - Today only completion (`issue_children_completed`) and resolution
(`issue_blockers_resolved`) wake a waiting issue
> - In restart chaos `w-20261108T181313-s9701` t05, a blocker stopped on
a recovery action that needs the board, and its parent sat silently
blocked for 50+ minutes
> - This pull request wakes the waiting agent once per such stall, tells
it what it can do, and shows the stall on the parent's page with the
child's Retry
> - The benefit is that a stuck child or blocker no longer silently
strands its parent

## Linked Issues or Issue Description

Refs #19831 (durable ownership plan; restart chaos campaign). Restart
chaos wave `w-20261108T181313-s9701`, task t05 (M1, claude).

**Problem**

From the kept database:
- CHA-8 `blocks` CHA-5.
- CHA-8's board-owned recovery action (`native_session_retry_exhausted`)
was created at 18:44:02.909 and resolved with outcome `blocked` at
18:44:08, when CHA-8 moved to `blocked` for the board.
- CHA-5's only wake in the whole wave was its `issue_assigned` at
18:20:48. Its agent never heard about CHA-8.

**Expected behavior**
- When a child or blocker stalls on the board, the waiting agent is
woken once. It can then re-delegate, escalate, or block with a reason.
- The board sees the stall on the waiting task, with the child's Retry.

## What Changed

- **Shared:**
- `ISSUE_BLOCKER_STALLED_WAKE_REASON` (`issue_blocker_stalled`),
`ISSUE_BLOCKER_STALL_RELATIONS`, and the types
`IssueBlockerStallRelation` and `IssueBlockerStalledWakeContext`;
  - `Issue.stalledDependencies`.
- **Server, `services/recovery/blocker-stall-wakeups.ts`:**
- A stall is an issue that is not done or cancelled, whose latest
recovery action needs the board. That means either open
(`active`/`escalated`) and board-owned or escalated, or board-owned and
resolved as `blocked` while the issue is still `blocked`. The second
case is what t05 needed: recovery resolves its action that way when it
hands the issue to the board.
- Waiters are the parent, and every issue the stalled one `blocks`, when
they are open, visible and agent-assigned.
- Each waiter gets one wake per stall. The idempotency key
`issue_blocker_stalled:{waiter}:{stalled}:{recoveryActionId}` is checked
through the indexed `(company_id, payload->>'issueId')` path. A later
stall has a new action id, so it wakes again.
  - Each wake writes activity `issue.blocker_stalled_wake`.
- The pass runs from the recovery-action sweep
(`reconcileActiveRecoveryActions`) and from the per-issue blocked sweep
(`reconcileResolvedDependencyWakeBackstop`, which visits each blocked
issue). No new worker.
- A waiter's agent that cannot read the stalled issue (`issue:read`,
which honors private visibility) is woken without its title, identifier
or recovery details; the activity entry withholds them too.
- **Database:** migration `0450` adds the partial unique index
`agent_wakeup_requests_issue_blocker_stalled_uq` on `(company_id,
idempotency_key)` for `issue_blocker_stalled:` keys in delivered or
pending statuses, so two concurrent sweeps cannot both deliver a stall.
It is built `CONCURRENTLY IF NOT EXISTS` outside a transaction (listed
in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`).
- **Queue:** `allowsDependencyBlockedWake` (in
`run-dispatch/domain/wake-context.ts`) lets a stall wake run while the
waiter's blockers are unresolved, at both dependency gates. Without it
the wake was parked until the stalled blocker resolved, which is never.
Pause holds are unchanged.
- **Agent guidance:** the wake payload carries `blockerStall`, and
`renderPaperclipWakePrompt` adds a "Stalled dependency" section. It
names the task and its recovery action and the board's next action, then
the three choices: re-delegate, escalate with a comment, or block with a
reason naming the stalled task.
- **Other reason lists:** `issue_blocker_stalled` joins the
responsible-user wake reasons, the conversation execution-wake reasons
(it never drives a chat turn) and the wake diagnostic's known reasons.
- **Board surface:**
- `GET /issues/:id` returns `stalledDependencies`, readable issues only
(`readableIssueEdgeIds`).
- `StalledDependencyNotice` shows on the issue's chat notices, next to
`IssueBlockedNotice`: each stalled child or blocker, its board next
action, a link, and the child's own Retry (`useTaskRetry`), offered only
when the child's interruption allows it.

## Verification

Red/green:
- `server/src/__tests__/issue-blocker-stall-wakeups.test.ts` (14 tests).
  - On master: no `issue_blocker_stalled` wake is raised.
- With this PR: the parent is woken once per stall, and re-woken after a
new stall. A blocker's dependent is woken.
- The t05 shape (an action resolved as `blocked`) wakes once, and
nothing more after a Retry.
- No wake for an agent-owned action, a done, unassigned or hidden
waiter, or an action resolved another way.
  - `listStalledDependencies` lists children and blockers.
- `heartbeat-dependency-scheduling.test.ts` "admits an
issue_blocker_stalled wake while a blocker is unresolved". On master the
wake was parked (`null`); once admitted, the run-start gate cancelled
it. With this PR it runs with `dependencyBlockedInteraction`. The file:
14 passed.
- `packages/adapter-utils` `server-utils.test.ts`: the new prompt test
failed on master (no `blockerStall`). The file: 141 passed.
- `ui/src/components/StalledDependencyNotice.test.tsx`: 3 passed.
`IssueChatThread.test.tsx`: 109 passed.

Regression and static checks:
- Nearby server suites (recovery durable wait, dependency wake routes,
recovery actions, run-preparation): 149 passed.
- `tsc --noEmit` passes for `packages/shared`, `server` and `ui/src`.
- `pnpm check:token-gates` is clean.

Review round 1 (Greptile, Superagent), each with a regression:
- A backlog waiter is never woken. The waiter's status and assignee are
guarded at enqueue.
- One stall per run: no coalescing, so a merged run cannot keep only the
last stall's context.
- Only a delivered or pending wake ends a stall. A refused one
(wake-on-demand off) is retried after 10 minutes, and an agent that
takes no wakes is skipped without writing a row.
- Company-wide passes resume after the last stall a capped pass reached.
The per-issue sweep also checks the issue's own children and blockers,
which covers an unassigned stalled blocker.
- The dependency gates no longer trust the reason string.
`verifyBlockerStallWake` requires a system request, the stalled issue's
current stall action, and a real child or blocker edge. An
agent-requested wake with the same context is parked (tested).
- The notice shows a refused Retry's reason, and a lost response as
"Taking longer than usual…", with Retry still available (tested).

Review round 2 (Greptile), each with a regression:
- Race: two passes claiming one stall leave one delivered row (unique
index; the loser's insert is skipped).
- The company recovery sweep selects companies whose only stall is a
resolved-`blocked` board action, so an unassigned stalled blocker with a
todo waiter is reached.
- Private stalled issue: the waiter's wake and activity carry no title,
identifier or recovery details.
- `GET /issues/:id` always returns `stalledDependencies`, so a cleared
stall replaces the cached list.
- The notice shows a failed child read with a Reload button (tested).
- `pnpm --filter @paperclipai/db check:migrations` passes.

Review round 3 (Greptile), each with a regression:
- A stall wake that fails to enqueue is retried: the recovery-action
pass reports `blockerStallWakeFailures` and the dependency pass counts
it in `enqueueFailed`, so the worker throws and retries at its backoff
(`issue-recovery-work-lifecycle.test.ts`).
- The notice drops a stall once the child's refreshed status has moved
on (`StalledDependencyNotice.test.tsx`, 7 tests).

## Risks

- A deploy wakes the waiters of every issue that is stalled today, once
each. That is intended (they are stranded now), and it's bounded at 200
stalls per pass.
- A stall wake may run while blockers are unresolved, like a comment
wake. It is bounded to one per stall, and pause holds still apply.
- The agent skill (`skills/paperclip`, CODEOWNER-owned) does not yet
list the new reason. The wake prompt carries the guidance.
- One additive index, built concurrently; no persisted run state change.
Existing rows cannot violate it before deploy, since no
`issue_blocker_stalled:` keys exist yet.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 18:26:07 -06:00

208 lines
9.2 KiB
TypeScript

// Shared wake-reason and retry-reason constants, and the pure classifiers
// that read them off a run's context snapshot. `heartbeat.ts` and this
// module's own Postgres adapter both decide on the same context snapshot
// shape, so this file is their one shared source for it: a second,
// independently maintained copy in either file could silently drift out of
// sync with the other and change only one of the two gates that read it.
function parseObject(value: unknown): Record<string, unknown> {
return typeof value === "object" && value !== null && !Array.isArray(value)
? (value as Record<string, unknown>)
: {};
}
function readNonEmptyString(value: unknown): string | null {
return typeof value === "string" && value.trim().length > 0 ? value : null;
}
export const MAX_TURN_CONTINUATION_RETRY_REASON = "max_turns_continuation";
export const WORKSPACE_BUSY_RETRY_REASON = "workspace_busy";
export const AI_CONNECTION_POOL_WAIT_RETRY_REASON = "ai_connection_pool_wait";
export const AI_CONNECTION_BUSY_RETRY_REASON = "ai_connection_busy";
export const RUNNER_REVISION_WAIT_RETRY_REASON = "runner_revision_wait";
export const INTERACTION_CONTINUATION_INFRA_RETRY_REASON = "interaction_continuation_infra_retry";
export const INTERACTION_CONTINUATION_INFRA_WAKE_REASON = "interaction_continuation_infra_retry";
export const WAKE_COMMENT_IDS_KEY = "wakeCommentIds";
export const RESOLVED_INTERACTION_CONTINUATION_STATUSES = new Set([
"accepted",
"answered",
"cancelled",
"rejected",
]);
// A run silent for this long no longer counts as the shared-workspace holder
// it was waiting behind. Kept here so the deferral gate and the parked-wait
// check in this module's adapter read one value.
export const WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS = 60 * 60 * 1000;
// Fallback re-check for a parked workspace wait. The holder ending wakes the
// waiter directly; this only bounds a missed wake.
export const WORKSPACE_BUSY_PARKED_RECHECK_MS = 5 * 60 * 1000;
// Re-check for a wait behind an ended remote writer whose save or sandbox
// release has not settled. Settling has no single completion point to wake
// from, so the parked run re-checks on this cadence. Each check creates no run.
// It reads the workspace, the writer run, its task, and the writer's leases
// (one row per sandbox lease of that run, so a small set) in the promotion
// transaction. The checks last only while the wait exists, up to the cutoff.
export const WORKSPACE_BUSY_SETTLING_RECHECK_MS = 60 * 1000;
export const WORKSPACE_BUSY_WAIT_KEY = "workspaceBusyWait";
/** What a workspace-busy retry is waiting behind, recorded at deferral. */
export type WorkspaceBusyWait = {
/** `execution:<id>` or `project:<id>`; waiters on one key are admitted in arrival order. */
key: string;
holderRunId: string;
/** ISO time of the first deferral for this key; carried across re-deferrals. */
waitingSince: string;
/**
* The task's workspace binding when it deferred. The wait applies only while
* the binding is unchanged; a remote reservation can defer before it binds.
*/
issueExecutionWorkspaceId: string | null;
issueProjectWorkspaceId: string | null;
};
export function workspaceBusyWaitKey(input: {
executionWorkspaceId: string | null;
projectWorkspaceId: string | null;
}): string | null {
if (input.executionWorkspaceId) return `execution:${input.executionWorkspaceId}`;
if (input.projectWorkspaceId) return `project:${input.projectWorkspaceId}`;
return null;
}
export function readWorkspaceBusyWait(
retryReason: string | null | undefined,
contextSnapshot: Record<string, unknown> | null | undefined,
): WorkspaceBusyWait | null {
if (retryReason !== WORKSPACE_BUSY_RETRY_REASON) return null;
const wait = parseObject(parseObject(contextSnapshot)[WORKSPACE_BUSY_WAIT_KEY]);
const key = readNonEmptyString(wait.key);
const holderRunId = readNonEmptyString(wait.holderRunId);
const waitingSince = readNonEmptyString(wait.waitingSince);
if (!key || !holderRunId || !waitingSince || Number.isNaN(Date.parse(waitingSince))) return null;
if (!("issueExecutionWorkspaceId" in wait) || !("issueProjectWorkspaceId" in wait)) return null;
return {
key, holderRunId, waitingSince,
issueExecutionWorkspaceId: readNonEmptyString(wait.issueExecutionWorkspaceId),
issueProjectWorkspaceId: readNonEmptyString(wait.issueProjectWorkspaceId),
};
}
/**
* True for a resource-wait retry whose original run did not
* execute under assignee-ship (a comment or review-participant wake). Such a
* retry has an expected assignee mismatch, so the scheduled-retry gate and
* the queued-run staleness check must not treat it as a reassignment.
*/
export function isNonAssigneeWorkspaceBusyRetry(
retryReason: string | null | undefined,
contextSnapshot: Record<string, unknown>,
): boolean {
return (
(retryReason === WORKSPACE_BUSY_RETRY_REASON &&
contextSnapshot.workspaceBusyDeferredWhileAssignee === false) ||
((retryReason === AI_CONNECTION_BUSY_RETRY_REASON || retryReason === AI_CONNECTION_POOL_WAIT_RETRY_REASON) &&
contextSnapshot.aiConnectionBusyDeferredWhileAssignee === false)
);
}
export function extractWakeCommentIds(
contextSnapshot: Record<string, unknown> | null | undefined,
): string[] {
const raw = contextSnapshot?.[WAKE_COMMENT_IDS_KEY];
if (!Array.isArray(raw)) return [];
const out: string[] = [];
for (const entry of raw) {
const value = readNonEmptyString(entry);
if (!value || out.includes(value)) continue;
out.push(value);
}
return out;
}
export function deriveCommentId(
contextSnapshot: Record<string, unknown> | null | undefined,
payload?: Record<string, unknown> | null,
): string | null {
const batchedCommentId = extractWakeCommentIds(contextSnapshot).at(-1);
return (
batchedCommentId ??
readNonEmptyString(contextSnapshot?.wakeCommentId) ??
readNonEmptyString(contextSnapshot?.commentId) ??
readNonEmptyString(payload?.commentId) ??
null
);
}
/**
* The external chat conversation a wake answers. Several conversations can
* share one issue (for example GitHub review finding threads), but a reviewed
* chat run can attest only one, so wakes for different conversations must
* never share a batch.
*/
export const CHAT_CONVERSATION_ID_KEY = "chatConversationId";
export function readWakeChatConversationId(contextSnapshot: unknown): string | null {
return readNonEmptyString(parseObject(contextSnapshot)[CHAT_CONVERSATION_ID_KEY]);
}
/** True when both wakes answer a chat conversation and they are different ones. */
export function wakeChatConversationsConflict(existing: unknown, incoming: unknown): boolean {
const existingId = readWakeChatConversationId(existing);
const incomingId = readWakeChatConversationId(incoming);
return existingId !== null && incomingId !== null && existingId !== incomingId;
}
/**
* `allowedWakeReasons` is the issue-tree-control module's own set of wake
* reasons that excuse an interaction wake. This file stays free of service
* imports, so the caller passes the set in rather than this function reading
* it from the service directly.
*/
export function allowsIssueInteractionWake(
contextSnapshot: Record<string, unknown> | null | undefined,
allowedWakeReasons: ReadonlySet<string>,
): boolean {
const wakeReason = readNonEmptyString(contextSnapshot?.wakeReason);
if (!wakeReason || !allowedWakeReasons.has(wakeReason)) return false;
return Boolean(deriveCommentId(contextSnapshot));
}
/**
* Whether a wake may run while its issue still has unresolved blockers: a
* verified comment interaction, or a stalled child or blocker
* (`issue_blocker_stalled`, once per stall), which the assignee must be able
* to re-delegate, escalate or block on while that dependency is stuck. The
* caller verifies the stall against server state (`verifyBlockerStallWake`);
* the wake reason string alone never qualifies. Pause holds are separate and
* do not use this rule.
*/
export function allowsDependencyBlockedWake(
contextSnapshot: Record<string, unknown> | null | undefined,
interactionWakeReasons: ReadonlySet<string>,
verifiedBlockerStall: boolean,
): boolean {
return allowsIssueInteractionWake(contextSnapshot, interactionWakeReasons) || verifiedBlockerStall;
}
export function isResolvedInteractionContinuationWakeContext(contextSnapshot: unknown): boolean {
const context = parseObject(contextSnapshot);
const interactionId = readNonEmptyString(context.interactionId);
const interactionStatus = readNonEmptyString(context.interactionStatus);
if (!interactionId || !interactionStatus) return false;
const retiredConnection = interactionStatus === "expired"
&& context.interactionKind === "connection_intent"
&& context.connectionIntentResolution === "existing_connection";
if (!retiredConnection && !RESOLVED_INTERACTION_CONTINUATION_STATUSES.has(interactionStatus)) return false;
const mutation = readNonEmptyString(context.mutation);
const wakeReason = readNonEmptyString(context.wakeReason);
const retryReason = readNonEmptyString(context.retryReason);
return (
(mutation === "interaction" && wakeReason === "issue_commented") ||
(mutation === "interaction" && retryReason === RUNNER_REVISION_WAIT_RETRY_REASON) ||
wakeReason === INTERACTION_CONTINUATION_INFRA_WAKE_REASON ||
retryReason === INTERACTION_CONTINUATION_INFRA_RETRY_REASON
);
}