fix(task-watchdogs): deduplicate unchanged stopped-state wakes (#10207)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Task watchdogs review issue subtrees when no run or queued wake
keeps work live
> - Pending human interactions and approvals are valid stopped states
that still need one watchdog review
> - The existing fingerprint included volatile activity timestamps, so
unchanged stopped trees could wake repeatedly after comments, documents,
work products, or sibling completions
> - This pull request fingerprints only review-material leaf and wait
state, persists the reviewed snapshot, and suppresses shrink-only
repeats
> - The benefit is one review per materially new stopped state without
weakening liveness classification or hiding human waits

## Linked Issues or Issue Description

### What happened?

Task-watchdog stop fingerprints changed for metadata-only activity and
completed siblings, producing duplicate wakes after an unchanged stop
had already been reviewed.

### Expected behavior

Pending interactions and approvals remain classified as stopped, but a
reviewed stopped state only wakes again when waits, non-terminal leaves,
status, assignment, or blockers gain material changes.

### Steps to reproduce

1. Review a stopped watched subtree with a pending human wait or
multiple non-terminal stopped leaves.
2. Add only comment/document/work-product activity, or complete one
stopped sibling without changing the wait set.
3. Observe a duplicate wake from the timestamp-heavy fingerprint.

Related public work: Refs #9452 for overlapping task-watchdog service
edits and #10043 for related no-op fingerprint suppression.

## What Changed

- Added fingerprint v2 over non-terminal material leaves plus
subtree-wide pending wait ids, excluding volatile timestamps while
retaining them in wake context.
- Added nullable observed/reviewed JSONB stop snapshots and shrink-only
reviewed-state suppression with legacy exact-fingerprint fallback.
- Added pending interaction kinds and approval ids to watchdog wake
context, review comments, and comment metadata.
- Added classifier and scheduler coverage for waiting-leaf liveness,
metadata stability, sibling shrink suppression, material changes,
snapshot promotion, legacy rows, and unchanged idempotency keys.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/task-watchdogs-classifier.test.ts
src/__tests__/task-watchdogs-scheduler.test.ts` — 2 files, 36 tests
passed.
- `pnpm --filter @paperclipai/server typecheck` — passed.
- `git diff --check origin/master...HEAD` — passed.

## Risks

- Fingerprint version 2 intentionally re-fingerprints every currently
stopped watched tree once after deployment, causing a one-time wake
burst before the new reviewed snapshots are established.
- Migration `0191_task_watchdog_stop_snapshots.sql` only adds two
nullable JSONB columns with no backfill; legacy rows continue
exact-fingerprint behavior until a post-deploy review promotes a
snapshot.
- PR #9452 edits the same service file; whichever lands second may need
a trivial rebase.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex, exact model ID `gpt-5.6-sol`, high reasoning mode, with
repository tool use and code execution. The runtime did not expose a
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-07-25 08:34:41 -05:00
1 parent 4e00818574
commit c481be44e3
7 files changed
+507 -18

No files matched your search

+174 -12
View File
@@ -87,6 +87,7 @@ export type TaskWatchdogClassifierWaitingPath = {
companyId: string;
issueId: string;
id?: string | null;
kind?: string | null;
status: string;
};
@@ -99,7 +100,9 @@ export type TaskWatchdogClassifierRelation = {
export type TaskWatchdogClassifierConfig = Pick<
IssueWatchdogSummary,
"companyId" | "issueId" | "lastReviewedFingerprint"
>;
> & {
lastReviewedStopSnapshot?: TaskWatchdogStopSnapshot | null;
};
export type TaskWatchdogStoppedLeaf = {
issueId: string;
@@ -117,6 +120,34 @@ export type TaskWatchdogStoppedLeaf = {
latestWorkProductAt: string | null;
};
export type TaskWatchdogMaterialLeaf = Pick<
TaskWatchdogStoppedLeaf,
| "issueId"
| "status"
| "assigneeAgentId"
| "assigneeUserId"
| "blockerIssueIds"
| "pendingInteractionIds"
| "pendingApprovalIds"
>;
export type TaskWatchdogWaitsByIssueId = Record<string, {
pendingInteractionIds: string[];
pendingApprovalIds: string[];
}>;
export type TaskWatchdogStopSnapshot = {
version: 2;
fingerprint: string;
materialLeaves: TaskWatchdogMaterialLeaf[];
waitsByIssueId: TaskWatchdogWaitsByIssueId;
};
type TaskWatchdogPendingInteractionsByIssueId = Record<string, Array<{
id: string;
kind: string | null;
}>>;
export type TaskWatchdogClassifierResult =
| {
state: "not_applicable";
@@ -141,6 +172,8 @@ export type TaskWatchdogClassifierResult =
includedIssueIds: string[];
stopFingerprint: string;
stoppedLeaves: TaskWatchdogStoppedLeaf[];
stopSnapshot: TaskWatchdogStopSnapshot;
pendingInteractionsByIssueId: TaskWatchdogPendingInteractionsByIssueId;
}
| {
state: "stopped";
@@ -148,6 +181,8 @@ export type TaskWatchdogClassifierResult =
includedIssueIds: string[];
stopFingerprint: string;
stoppedLeaves: TaskWatchdogStoppedLeaf[];
stopSnapshot: TaskWatchdogStopSnapshot;
pendingInteractionsByIssueId: TaskWatchdogPendingInteractionsByIssueId;
};
export type TaskWatchdogClassifierInput = {
@@ -269,17 +304,70 @@ function waitingPathIds(
function stableStopFingerprint(input: {
companyId: string;
watchedIssueId: string;
leaves: TaskWatchdogStoppedLeaf[];
materialLeaves: TaskWatchdogMaterialLeaf[];
waitsByIssueId: TaskWatchdogWaitsByIssueId;
}) {
const payload = JSON.stringify({
version: 1,
version: 2,
companyId: input.companyId,
watchedIssueId: input.watchedIssueId,
leaves: input.leaves,
materialLeaves: input.materialLeaves,
waitsByIssueId: input.waitsByIssueId,
});
return `task_watchdog_stop:${createHash("sha256").update(payload).digest("hex")}`;
}
function materialLeaf(leaf: TaskWatchdogStoppedLeaf): TaskWatchdogMaterialLeaf {
return {
issueId: leaf.issueId,
status: leaf.status,
assigneeAgentId: leaf.assigneeAgentId,
assigneeUserId: leaf.assigneeUserId,
blockerIssueIds: leaf.blockerIssueIds,
pendingInteractionIds: leaf.pendingInteractionIds,
pendingApprovalIds: leaf.pendingApprovalIds,
};
}
function parseStopSnapshot(value: unknown): TaskWatchdogStopSnapshot | null {
if (!value || typeof value !== "object") return null;
const candidate = value as Partial<TaskWatchdogStopSnapshot>;
if (
candidate.version !== 2 ||
typeof candidate.fingerprint !== "string" ||
!Array.isArray(candidate.materialLeaves) ||
!candidate.waitsByIssueId ||
typeof candidate.waitsByIssueId !== "object"
) return null;
return candidate as TaskWatchdogStopSnapshot;
}
// Snapshots loaded from jsonb columns come back with Postgres's normalized key
// order, so equality checks against freshly built snapshots must not depend on
// object key order.
function canonicalJson(value: unknown): string {
return JSON.stringify(value, (_key, val) =>
val && typeof val === "object" && !Array.isArray(val)
? Object.fromEntries(
Object.entries(val as Record<string, unknown>).sort(([left], [right]) =>
left < right ? -1 : left > right ? 1 : 0
),
)
: val);
}
function isShrinkOfReviewedSnapshot(
current: TaskWatchdogStopSnapshot,
reviewed: TaskWatchdogStopSnapshot | null | undefined,
) {
if (!reviewed || canonicalJson(current.waitsByIssueId) !== canonicalJson(reviewed.waitsByIssueId)) return false;
const reviewedLeaves = new Map(reviewed.materialLeaves.map((leaf) => [leaf.issueId, leaf]));
return current.materialLeaves.every((leaf) => {
const previous = reviewedLeaves.get(leaf.issueId);
return previous != null && canonicalJson(previous) === canonicalJson(leaf);
});
}
export function classifyTaskWatchdogSubtree(input: TaskWatchdogClassifierInput): TaskWatchdogClassifierResult {
const issuesById = new Map(input.issues.map((issue) => [issue.id, issue]));
const root = issuesById.get(input.watchdog.issueId);
@@ -380,8 +468,25 @@ export function classifyTaskWatchdogSubtree(input: TaskWatchdogClassifierInput):
blockersByIssueId.set(relation.blockedIssueId, list);
}
const nonTerminalIssues = included
.filter((issue) => !isTerminalIssueStatus(issue.status))
.sort((left, right) => left.id.localeCompare(right.id));
const waitsByIssueId = Object.fromEntries(nonTerminalIssues
.map((issue) => [issue.id, {
pendingInteractionIds: waitingPathIds(input.pendingInteractions, input.watchdog.companyId, issue.id),
pendingApprovalIds: waitingPathIds(input.pendingApprovals, input.watchdog.companyId, issue.id),
}] as const)
.filter(([, waits]) => waits.pendingInteractionIds.length > 0 || waits.pendingApprovalIds.length > 0));
const pendingInteractionsByIssueId = Object.fromEntries(nonTerminalIssues
.map((issue) => [issue.id, (input.pendingInteractions ?? [])
.filter((path) => path.companyId === input.watchdog.companyId && path.issueId === issue.id)
.map((path) => ({ id: path.id ?? `${path.status}:${path.issueId}`, kind: path.kind ?? null }))
.sort((left, right) => left.id.localeCompare(right.id))] as const)
.filter(([, waits]) => waits.length > 0));
const leaves = included
.filter((issue) => (includedChildrenByParentId.get(issue.id) ?? []).length === 0)
.filter((issue) => !isTerminalIssueStatus(issue.status))
.sort((left, right) => left.id.localeCompare(right.id))
.map((issue) => ({
issueId: issue.id,
@@ -398,19 +503,32 @@ export function classifyTaskWatchdogSubtree(input: TaskWatchdogClassifierInput):
latestDocumentAt: optionalIso(issue.latestDocumentAt),
latestWorkProductAt: optionalIso(issue.latestWorkProductAt),
}));
const materialLeaves = leaves.map(materialLeaf);
const stopFingerprint = stableStopFingerprint({
companyId: input.watchdog.companyId,
watchedIssueId: input.watchdog.issueId,
leaves,
materialLeaves,
waitsByIssueId,
});
const currentStopSnapshot: TaskWatchdogStopSnapshot = {
version: 2,
fingerprint: stopFingerprint,
materialLeaves,
waitsByIssueId,
};
if (input.watchdog.lastReviewedFingerprint === stopFingerprint) {
if (
input.watchdog.lastReviewedFingerprint === stopFingerprint ||
isShrinkOfReviewedSnapshot(currentStopSnapshot, input.watchdog.lastReviewedStopSnapshot)
) {
return {
state: "already_reviewed",
reason: "The current stopped subtree fingerprint was already reviewed by the watchdog.",
includedIssueIds: includedIds,
stopFingerprint,
stoppedLeaves: leaves,
stopSnapshot: currentStopSnapshot,
pendingInteractionsByIssueId,
};
}
@@ -420,6 +538,8 @@ export function classifyTaskWatchdogSubtree(input: TaskWatchdogClassifierInput):
includedIssueIds: includedIds,
stopFingerprint,
stoppedLeaves: leaves,
stopSnapshot: currentStopSnapshot,
pendingInteractionsByIssueId,
};
}
@@ -501,11 +621,20 @@ function buildStoppedFingerprintComment(input: {
sourceIssue: Pick<IssueRow, "identifier" | "id">;
stopFingerprint: string;
stoppedLeaves: TaskWatchdogStoppedLeaf[];
pendingInteractionsByIssueId: TaskWatchdogPendingInteractionsByIssueId;
resumed: boolean;
}) {
const leafLines = input.stoppedLeaves.slice(0, 12).map((leaf) =>
`- ${leaf.identifier ?? leaf.issueId}: ${leaf.status} (updated ${leaf.updatedAt})`
);
const shortId = (id: string) => id.length > 8 ? `${id.slice(0, 8)}…` : id;
const leafLines = input.stoppedLeaves.slice(0, 12).map((leaf) => {
const interactionKinds = new Map(
(input.pendingInteractionsByIssueId[leaf.issueId] ?? []).map((wait) => [wait.id, wait.kind]),
);
const waits = [
...leaf.pendingInteractionIds.map((id) => `${interactionKinds.get(id) ?? "interaction"} ${shortId(id)}`),
...leaf.pendingApprovalIds.map((id) => `approval ${shortId(id)}`),
];
return `- ${leaf.identifier ?? leaf.issueId}: ${leaf.status}${waits.length > 0 ? ` (pending ${waits.join(", ")})` : ""}`;
});
const more = input.stoppedLeaves.length > leafLines.length
? `\n- ...and ${input.stoppedLeaves.length - leafLines.length} more stopped leaves`
: "";
@@ -524,8 +653,13 @@ function buildStoppedFingerprintComment(input: {
function stoppedFingerprintMetadata(input: {
sourceIssueId: string;
stopFingerprint: string;
waitsByIssueId: TaskWatchdogWaitsByIssueId;
resumed: boolean;
}) {
const pendingWaitCount = Object.values(input.waitsByIssueId).reduce(
(count, waits) => count + waits.pendingInteractionIds.length + waits.pendingApprovalIds.length,
0,
);
return {
version: 1 as const,
sections: [
@@ -534,6 +668,7 @@ function stoppedFingerprintMetadata(input: {
rows: [
{ type: "text" as const, label: "Watched issue", text: input.sourceIssueId },
{ type: "text" as const, label: "Stopped fingerprint", text: input.stopFingerprint },
{ type: "text" as const, label: "Pending waits", text: String(pendingWaitCount) },
{ type: "text" as const, label: "Resume intent", text: input.resumed ? "true" : "false" },
],
},
@@ -557,6 +692,10 @@ function watchdogWakeContext(input: {
watchedIssueIdentifier: input.sourceIssue.identifier,
watchedIssueTitle: input.sourceIssue.title,
stopFingerprint: input.classification.stopFingerprint,
pendingInteractions: input.classification.pendingInteractionsByIssueId,
pendingApprovals: Object.fromEntries(Object.entries(input.classification.stopSnapshot.waitsByIssueId)
.filter(([, waits]) => waits.pendingApprovalIds.length > 0)
.map(([issueId, waits]) => [issueId, waits.pendingApprovalIds])),
capabilities: {
targetScope: {
watchedIssueId: input.sourceIssue.id,
@@ -877,6 +1016,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
companyId: issueThreadInteractions.companyId,
issueId: issueThreadInteractions.issueId,
id: issueThreadInteractions.id,
kind: issueThreadInteractions.kind,
status: issueThreadInteractions.status,
})
.from(issueThreadInteractions)
@@ -953,7 +1093,10 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
const completedRunIssueIds = await collectCompletedRunIssueIds(companyId, freshIssueIds);
return {
watchdog: summarizeIssueWatchdog(watchdog),
watchdog: {
...summarizeIssueWatchdog(watchdog),
lastReviewedStopSnapshot: parseStopSnapshot(watchdog.lastReviewedStopSnapshot),
},
issues: issueRows.map((issue) => ({
...issue,
latestCommentAt: latestCommentByIssueId.get(issue.id) ?? null,
@@ -1136,11 +1279,20 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
: false;
if (!isWatchdogReviewDisposition(watchdogIssue, hasPendingReviewPath)) return watchdog;
const reviewedFingerprint = reviewedFingerprintForWatchdogIssue(watchdogIssue);
if (!reviewedFingerprint || watchdog.lastReviewedFingerprint === reviewedFingerprint) return watchdog;
if (!reviewedFingerprint) return watchdog;
const observedSnapshot = parseStopSnapshot(watchdog.lastObservedStopSnapshot);
const reviewedStopSnapshot = observedSnapshot?.fingerprint === reviewedFingerprint
? observedSnapshot
: null;
if (
watchdog.lastReviewedFingerprint === reviewedFingerprint &&
canonicalJson(parseStopSnapshot(watchdog.lastReviewedStopSnapshot)) === canonicalJson(reviewedStopSnapshot)
) return watchdog;
const [updated] = await db
.update(issueWatchdogs)
.set({
lastReviewedFingerprint: reviewedFingerprint,
lastReviewedStopSnapshot: reviewedStopSnapshot,
lastCompletedAt: new Date(),
updatedAt: new Date(),
})
@@ -1161,6 +1313,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
watchdogIssueId: watchdogIssue.id,
reviewedFingerprint,
lastObservedFingerprint: watchdog.lastObservedFingerprint,
reviewedStopSnapshot,
watchdogIssueStatus: watchdogIssue.status,
},
});
@@ -1214,6 +1367,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
sourceIssue: input.sourceIssue,
stopFingerprint: input.classification.stopFingerprint,
stoppedLeaves: input.classification.stoppedLeaves,
pendingInteractionsByIssueId: input.classification.pendingInteractionsByIssueId,
resumed: true,
}),
{ runId: input.runId ?? null },
@@ -1222,6 +1376,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
metadata: stoppedFingerprintMetadata({
sourceIssueId: input.sourceIssue.id,
stopFingerprint: input.classification.stopFingerprint,
waitsByIssueId: input.classification.stopSnapshot.waitsByIssueId,
resumed: true,
}),
},
@@ -1263,6 +1418,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
sourceIssue: input.sourceIssue,
stopFingerprint: input.classification.stopFingerprint,
stoppedLeaves: input.classification.stoppedLeaves,
pendingInteractionsByIssueId: input.classification.pendingInteractionsByIssueId,
resumed: false,
}),
{ runId: input.runId ?? null },
@@ -1271,6 +1427,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
metadata: stoppedFingerprintMetadata({
sourceIssueId: input.sourceIssue.id,
stopFingerprint: input.classification.stopFingerprint,
waitsByIssueId: input.classification.stopSnapshot.waitsByIssueId,
resumed: false,
}),
},
@@ -1305,6 +1462,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
.set({
watchdogIssueId: existingWatchdogIssueId,
lastObservedFingerprint: classification.stopFingerprint,
lastObservedStopSnapshot: classification.stopSnapshot,
updatedAt: new Date(),
})
.where(eq(issueWatchdogs.id, watchdog.id));
@@ -1324,13 +1482,15 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
if (await sameFingerprintWatchdogReviewIsStillOpen(existingWatchdogIssue, classification.stopFingerprint)) {
if (
watchdog.watchdogIssueId !== existingWatchdogIssue!.id ||
watchdog.lastObservedFingerprint !== classification.stopFingerprint
watchdog.lastObservedFingerprint !== classification.stopFingerprint ||
canonicalJson(parseStopSnapshot(watchdog.lastObservedStopSnapshot)) !== canonicalJson(classification.stopSnapshot)
) {
await db
.update(issueWatchdogs)
.set({
watchdogIssueId: existingWatchdogIssue!.id,
lastObservedFingerprint: classification.stopFingerprint,
lastObservedStopSnapshot: classification.stopSnapshot,
updatedAt: new Date(),
})
.where(eq(issueWatchdogs.id, watchdog.id));
@@ -1354,6 +1514,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
.set({
watchdogIssueId: watchdogIssue.id,
lastObservedFingerprint: classification.stopFingerprint,
lastObservedStopSnapshot: classification.stopSnapshot,
lastTriggeredAt: now,
triggerCount: sql`${issueWatchdogs.triggerCount} + 1`,
updatedAt: now,
@@ -1374,6 +1535,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
watchdogId: watchdog.id,
watchdogIssueId: watchdogIssue.id,
stopFingerprint: classification.stopFingerprint,
stopSnapshot: classification.stopSnapshot,
stoppedLeaves: classification.stoppedLeaves,
},
});