fix(watchdogs): reject incomplete subtree scans (#15063)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task watchdogs verify stopped issue subtrees and restore valid work
paths.
> - A watchdog must read the full subtree before it can prove that all
paths stopped.
> - The current query stops after 100 levels and does not report omitted
children.
> - A running child beyond this limit can be absent from both review
dispatch and mutation checks.
> - This pull request reports an incomplete scan and blocks both
decisions.
> - Complete subtrees keep their current behavior and stop fingerprints.

## Linked Issues or Issue Description

Fixes: #15061

Related PRs: #11327 changes subtree loading but keeps the same depth
limit without a completeness result. #10331 changes mutation freshness
after watchdog writes. Neither patch covers this boundary. This patch
does not take code from those proposals.

## What Changed

- Keep the existing 100-level traversal limit.
- Check for an eligible child beyond the boundary in the same SQL
statement as the subtree read.
- Apply the same company, visibility, harness, and watchdog-origin
filters to this check.
- Return `incomplete` without a stop fingerprint when the scan omits
eligible children.
- Skip review dispatch and reject watchdog mutation revalidation for an
incomplete scan.
- Report incomplete scans through their own count and watched issue IDs.
Log each newly incomplete issue at startup or during periodic
reconciliation. Suppress unchanged results. Allow a new warning after
the issue recovers and becomes incomplete again.
- Add real PostgreSQL regressions for dispatch, a previously valid
mutation scope, the complete boundary leaf, and excluded children.
- Document the incomplete-scan behavior.

## Verification

- Base: `569c7203aa24b95440682983ce7940ba1d4247bd`. Head:
`06f68199656f979ef5eb4897851f0fbcbca592d3`.
- Before the fix, two real PostgreSQL regressions fail. The service
dispatches a review for a partial subtree. It also accepts a prior scope
after a running depth-100 child appears. The other 19 scheduler checks
pass.
- On the exact final head, `pnpm -r typecheck` and `pnpm build` pass on
isolated Ubuntu with Node 24.19.0, pnpm 9.15.4, and runner Rust 1.97.1.
- All 75 related tests pass. Run `pnpm --filter @paperclipai/server exec
vitest run src/__tests__/task-watchdogs-classifier.test.ts
src/__tests__/task-watchdogs-scheduler.test.ts
src/__tests__/server-startup-feedback-export.test.ts
src/__tests__/issue-watchdogs-routes.test.ts --maxWorkers=1`.
- Full `test:run` coverage passes across 29 logical groups: all official
stable-runner lanes plus supplemental service and script tests. The
selection covers all 900 server source test files without duplicate or
omitted selections. It also includes UI, CLI, and other workspace
projects.
- A separate validation tree applies both this PR and #15067 to upstream
`994d6edcdd4e15d5f9cc5cf8c135ac599104b86a` without conflicts. Type
checks, the build, and all 98 related tests across five files pass. Both
public PR heads remain unchanged.
- Four full server lanes required test-process environment corrections.
The host DNS search suffix resolved a `.invalid` fixture to an address.
Ancestor project authentication settings affected hire fixtures. The old
path-isolation tool also failed test imports with `EFAULT`. The exact
base has the same failures: 8 failed and 128 passed assertions without
isolation, then three failed imports with the old tool. All 136 checks
pass on both the base and this head with the corrected isolation. All
four original full lanes pass with the corrected tool, a resolver
without a search suffix, empty ancestor project settings, and the
standard Node path. No assertion or wait limit changed. The host
configuration remains unchanged.
- Earlier interrupted Ubuntu and Docker runs are retained and are not
counted as passes. This exact-head Ubuntu run passes the original chat,
port-conflict, and serialized lanes. The causes of the earlier host-loss
and timing failures are not fully established.
- The startup import regression in an earlier revision is fixed by a
diagnostics module with no database dependencies. All 21 startup checks
pass.
- Exact-head CI, including the added scans after marking the PR ready:
54 successful checks and 2 skipped checks. Greptile: 5/5, no outstanding
findings or follow-ups. Both review threads are resolved.
- The final audit rechecked issue authors, assignees, comments, 206 open
watchdog search results, root functions and fields, source history, and
related diffs. The latest upstream is
`994d6edcdd4e15d5f9cc5cf8c135ac599104b86a`. Its new patches do not cover
this root. No competing root repair was found within the audited scope.
The issue has no assignee or new comment.
- No model provider, production database, or user instance was used.
This verifies service behavior, not a full HTTP authorization bypass.

## Risks

- A subtree with eligible children beyond 100 levels now waits instead
of running a review. This is a deliberate safety boundary. It remains
blocked until the scan can include the full subtree.
- The boundary check adds a child-existence lookup to the existing
recursive query.
- No schema migration is required. Complete subtrees retain their
fingerprint format.
- This does not add unlimited traversal or close the separate race
between a subtree snapshot and later source mutations.
- The configuration lifecycle defect in #15060 is handled by separate PR
#15067. This branch does not include its schema or configuration
changes.

## Model Used

- OpenAI Codex, based on GPT-6, assisted with source analysis,
implementation, and test execution. The exact serving model ID and
context window are not exposed in this session. The agent used
reasoning, shell tools, GitHub CLI, and real PostgreSQL tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
This commit is contained in:
fengqing authored and GitHub committed 2026-10-26 20:53:47 -07:00
1 parent 61ecc93033
commit 05218571c3
7 files changed
+191 -7

No files matched your search

+1 -1
View File
@@ -84,7 +84,7 @@ Paperclip runs a watchdog reconciliation tick at server startup, at the end of e
For each active watchdog the tick:
1. **Walks the watched subtree.** Starts at the configured issue and follows `parent_id` downward, excluding every issue whose `originKind = 'task_watchdog'` and everything below it. This excludes the watchdog's own review tasks so it cannot trigger itself.
1. **Walks the watched subtree.** Starts at the configured issue and follows `parent_id` downward, excluding every issue whose `originKind = 'task_watchdog'` and everything below it. This excludes the watchdog's own review tasks so it cannot trigger itself. The scan includes at most 100 levels, with the watched issue at level 1. If an eligible child remains beyond that limit, the result is **incomplete**. Paperclip skips review dispatch and rejects watchdog mutation revalidation until it can read the full subtree. A leaf at level 100 does not by itself make the scan incomplete. Reconciliation reports these cases separately through `incomplete` and `incompleteIssueIds`. Startup and periodic reconciliation warn when a watched issue first becomes incomplete. The warning contains the newly affected issue IDs. An unchanged incomplete issue does not produce repeated warnings. Once a successful scan no longer reports that issue as incomplete, a later incomplete scan can warn again.
2. **Checks for live paths.** If any included issue has a live run (`queued`, `running`, `scheduled_retry`), a queued wake request, or a scheduled retry, the subtree is **live** and the watchdog does not fire.
3. **Computes a stop fingerprint.** A SHA-256 hash over the stopped leaves' identifiers, statuses, blockers, pending interactions, and the current watchdog configuration. The configuration is part of the fingerprint, so changing the agent or instructions invalidates the previously reviewed state.
4. **Compares against `lastReviewedFingerprint`.** Match → suppress (the watchdog already saw this exact stopped state). New → proceed.
@@ -1,6 +1,8 @@
import { describe, expect, it } from "vitest";
import { describe, expect, it, vi } from "vitest";
import { classifyTaskWatchdogSubtree, type TaskWatchdogClassifierIssue } from "../services/task-watchdogs.ts";
import { createIncompleteTaskWatchdogScanReporter } from "../services/task-watchdog-diagnostics.ts";
const companyId = "company-1";
const sourceId = "source-1";
const childId = "child-1";
@@ -35,6 +37,45 @@ function classify(overrides: Partial<Parameters<typeof classifyTaskWatchdogSubtr
}
describe("task watchdog subtree classifier", () => {
it("reports an unchanged incomplete scan once across startup and a day of periodic ticks", () => {
const warn = vi.fn();
const report = createIncompleteTaskWatchdogScanReporter(warn);
report({ incomplete: 2, incompleteIssueIds: [sourceId, childId] }, "startup");
for (let tick = 0; tick < 2880; tick += 1) {
report({ incomplete: 2, incompleteIssueIds: [childId, sourceId] }, "periodic");
}
expect(warn).toHaveBeenCalledTimes(1);
expect(warn).toHaveBeenCalledWith(
{ incomplete: 2, newIncompleteIssueIds: [sourceId, childId] },
"startup task-watchdog reconciliation found incomplete subtree scans",
);
});
it("reports new incomplete issues and reports a resolved issue if it becomes incomplete again", () => {
const warn = vi.fn();
const report = createIncompleteTaskWatchdogScanReporter(warn);
report({ incomplete: 1, incompleteIssueIds: [sourceId] }, "startup");
report({ incomplete: 2, incompleteIssueIds: [sourceId, childId] }, "periodic");
expect(warn).toHaveBeenLastCalledWith(
{ incomplete: 2, newIncompleteIssueIds: [childId] },
"periodic task-watchdog reconciliation found incomplete subtree scans",
);
report({ incomplete: 0, incompleteIssueIds: [] }, "periodic");
expect(warn).toHaveBeenCalledTimes(2);
report({ incomplete: 1, incompleteIssueIds: [sourceId] }, "periodic");
expect(warn).toHaveBeenCalledTimes(3);
expect(warn).toHaveBeenLastCalledWith(
{ incomplete: 1, newIncompleteIssueIds: [sourceId] },
"periodic task-watchdog reconciliation found incomplete subtree scans",
);
});
it("does not produce a stopped fingerprint from an incomplete scan", () => {
const result = classify({ subtreeIncomplete: true });
expect(result.state).toBe("incomplete");
expect(result).not.toHaveProperty("stopFingerprint");
});
it("suppresses watchdog wakeups while watched subtree work has a live path", () => {
const result = classify({
issues: [
@@ -437,6 +437,81 @@ describeEmbeddedPostgres("task watchdog scheduler", () => {
expect(wakes).toHaveLength(1);
});
async function seedDeepSubtree() {
const companyId = await seedCompany();
const sourceId = await seedIssue(companyId, { status: "blocked", issueNumber: 1000 });
let boundaryId = sourceId;
for (let depth = 1; depth < 100; depth += 1) {
boundaryId = await seedIssue(companyId, {
parentId: boundaryId, status: "blocked", identifier: `DEEP-${depth}`, issueNumber: 1000 + depth,
});
}
const agentId = await seedAgent(companyId);
const watchdog = await seedWatchdog(companyId, sourceId, agentId);
return { companyId, sourceId, boundaryId, agentId, watchdog, ...createService() };
}
it("rejects an incomplete subtree before dispatching a stopped review", async () => {
const x = await seedDeepSubtree();
const omittedId = await seedIssue(x.companyId, {
parentId: x.boundaryId, status: "blocked", issueNumber: 1100,
});
await db.insert(heartbeatRuns).values({
companyId: x.companyId, agentId: x.agentId, status: "running",
contextSnapshot: { issueId: omittedId },
});
expect(await x.service.reconcileTaskWatchdogs({ companyId: x.companyId }))
.toMatchObject({ checked: 1, triggered: 0, incomplete: 1, incompleteIssueIds: [x.sourceId], skipped: 0 });
expect(x.wakes).toHaveLength(0);
const [persisted] = await db.select().from(issueWatchdogs).where(eq(issueWatchdogs.id, x.watchdog.id));
expect(persisted!.watchdogIssueId).toBeNull();
expect(persisted!.lastObservedFingerprint).toBeNull();
});
it("allows a complete boundary leaf but revokes its scope when an omitted child appears", async () => {
const x = await seedDeepSubtree();
expect((await x.service.reconcileTaskWatchdogs({ companyId: x.companyId })).triggered).toBe(1);
const [persisted] = await db.select().from(issueWatchdogs).where(eq(issueWatchdogs.id, x.watchdog.id));
const scope = {
kind: "watchdog" as const, watchdogId: x.watchdog.id, companyId: x.companyId,
watchedIssueId: x.sourceId, stopFingerprint: persisted!.lastObservedFingerprint,
};
expect((await x.service.revalidateMutationScope(scope)).allowed).toBe(true);
const omittedId = await seedIssue(x.companyId, {
parentId: x.boundaryId, status: "blocked", issueNumber: 1100,
});
await db.insert(heartbeatRuns).values({
companyId: x.companyId, agentId: x.agentId, status: "running",
contextSnapshot: { issueId: omittedId },
});
const incomplete = await x.service.revalidateMutationScope(scope);
expect(incomplete.allowed).toBe(false);
expect(incomplete.classification?.state).toBe("incomplete");
expect(incomplete.reason).toContain("incomplete");
expect(await x.service.reconcileForIssueAndAncestors(x.companyId, x.sourceId))
.toMatchObject({ triggered: 0, incomplete: 1, incompleteIssueIds: [x.sourceId], skipped: 0 });
expect(x.wakes).toHaveLength(1);
await db.update(issues).set({ parentId: x.sourceId }).where(eq(issues.id, omittedId));
const visible = await x.service.revalidateMutationScope(scope);
expect(visible.allowed).toBe(false);
expect(visible.classification?.state).toBe("live");
});
it("does not treat excluded boundary children as a truncated watched subtree", async () => {
const x = await seedDeepSubtree();
const hiddenId = await seedIssue(x.companyId, { parentId: x.boundaryId, issueNumber: 1100 });
await db.update(issues).set({ hiddenAt: new Date() }).where(eq(issues.id, hiddenId));
const harnessId = await seedIssue(x.companyId, { parentId: x.boundaryId, issueNumber: 1103 });
await db.update(issues).set({ harnessKind: "test" }).where(eq(issues.id, harnessId));
await seedIssue(x.companyId, {
parentId: x.boundaryId, issueNumber: 1101, originKind: "task_watchdog", originId: x.boundaryId,
});
const otherCompanyId = await seedCompany();
await seedIssue(otherCompanyId, { parentId: x.boundaryId, issueNumber: 1102 });
expect((await x.service.reconcileTaskWatchdogs({ companyId: x.companyId })).triggered).toBe(1);
expect(x.wakes).toHaveLength(1);
});
it("reconciles ancestor watchdogs for a descendant issue mutation", async () => {
const companyId = await seedCompany();
const sourceId = await seedIssue(companyId, { identifier: "WDOG-ANCESTOR", status: "done" });
+6
View File
@@ -85,6 +85,7 @@ import detectPort from "detect-port";
import { createApp } from "./app.js";
import { loadConfig } from "./config.js";
import { logger } from "./middleware/logger.js";
import { createIncompleteTaskWatchdogScanReporter } from "./services/task-watchdog-diagnostics.js";
import { setStartupRecoveryPhase } from "./startup-recovery-state.js";
import {
StartupRefusalError,
@@ -927,6 +928,9 @@ async function startServerWithDatabaseTeardown(
}
};
const pluginWorkerManager = createPluginWorkerManager();
const reportIncompleteWatchdogScans = createIncompleteTaskWatchdogScanReporter(
(details, message) => logger.warn(details, message),
);
const heartbeat = config.heartbeatSchedulerEnabled
? heartbeatService(db as any, { pluginWorkerManager })
: null;
@@ -1733,6 +1737,7 @@ async function startServerWithDatabaseTeardown(
}
const taskWatchdogsReconciled = await heartbeat.reconcileTaskWatchdogs();
reportIncompleteWatchdogScans(taskWatchdogsReconciled, "startup");
if (taskWatchdogsReconciled.triggered > 0) {
logger.warn(
{ ...taskWatchdogsReconciled },
@@ -1940,6 +1945,7 @@ async function startServerWithDatabaseTeardown(
})
.then(async () => {
const reconciled = await heartbeat.reconcileTaskWatchdogs();
reportIncompleteWatchdogScans(reconciled, "periodic");
if (reconciled.triggered > 0) {
logger.warn({ ...reconciled }, "periodic task-watchdog reconciliation triggered watchdog work");
}
+4
View File
@@ -528,6 +528,8 @@ function noopTaskWatchdogService(): TaskWatchdogService {
live: 0,
pendingFirstRun: 0,
alreadyReviewed: 0,
incomplete: 0,
incompleteIssueIds: [],
skipped: 0,
watchdogIssueIds: [],
}),
@@ -535,6 +537,8 @@ function noopTaskWatchdogService(): TaskWatchdogService {
checked: 0,
triggered: 0,
pendingFirstRun: 0,
incomplete: 0,
incompleteIssueIds: [],
skipped: 0,
watchdogIssueIds: [],
}),
@@ -0,0 +1,16 @@
export function createIncompleteTaskWatchdogScanReporter(
warn: (details: { incomplete: number; newIncompleteIssueIds: string[] }, message: string) => void,
) {
let previousIncompleteIssueIds = new Set<string>();
return (result: { incomplete: number; incompleteIssueIds: string[] }, phase: "startup" | "periodic") => {
const currentIssueIds = new Set(result.incompleteIssueIds);
const newIncompleteIssueIds = [...currentIssueIds].filter((id) => !previousIncompleteIssueIds.has(id));
previousIncompleteIssueIds = currentIssueIds;
if (newIncompleteIssueIds.length === 0) return;
warn(
{ incomplete: result.incomplete, newIncompleteIssueIds },
`${phase} task-watchdog reconciliation found incomplete subtree scans`,
);
};
}
+47 -5
View File
@@ -149,6 +149,11 @@ type TaskWatchdogPendingInteractionsByIssueId = Record<string, Array<{
}>>;
export type TaskWatchdogClassifierResult =
| {
state: "incomplete";
reason: string;
includedIssueIds: string[];
}
| {
state: "not_applicable";
reason: string;
@@ -188,6 +193,7 @@ export type TaskWatchdogClassifierResult =
export type TaskWatchdogClassifierInput = {
watchdog: TaskWatchdogClassifierConfig;
issues: TaskWatchdogClassifierIssue[];
subtreeIncomplete?: boolean;
activeRuns?: TaskWatchdogClassifierPath[];
queuedWakeRequests?: TaskWatchdogClassifierPath[];
blockers?: TaskWatchdogClassifierRelation[];
@@ -382,6 +388,15 @@ export function classifyTaskWatchdogSubtree(input: TaskWatchdogClassifierInput):
};
}
// A partial scan cannot prove that every execution path has stopped.
if (input.subtreeIncomplete) {
return {
state: "incomplete",
reason: "Watched subtree scan is incomplete because it exceeds the traversal depth limit.",
includedIssueIds: [...issuesById.keys()],
};
}
const childrenByParentId = new Map<string, TaskWatchdogClassifierIssue[]>();
for (const issue of input.issues) {
if (issue.companyId !== input.watchdog.companyId || !issue.parentId) continue;
@@ -927,20 +942,35 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
assignee_user_id AS "assigneeUserId",
origin_kind AS "originKind",
updated_at AS "updatedAt",
created_at AS "createdAt"
created_at AS "createdAt",
EXISTS (
SELECT 1
FROM watched_issues boundary
JOIN issues child ON child.parent_id = boundary.id
WHERE boundary.depth = ${TASK_WATCHDOG_SUBTREE_MAX_DEPTH - 1}
AND child.company_id = ${companyId}
AND child.hidden_at IS NULL
AND child.harness_kind IS NULL
AND child.origin_kind <> ${TASK_WATCHDOG_ORIGIN_KIND}
) AS "subtreeIncomplete"
FROM watched_issues
`);
return (Array.isArray(rows) ? rows : []) as TaskWatchdogClassifierIssue[];
const issueRows = (Array.isArray(rows) ? rows : []) as Array<
TaskWatchdogClassifierIssue & { subtreeIncomplete: boolean }
>;
return { issues: issueRows, incomplete: issueRows.some((row) => row.subtreeIncomplete) };
}
async function collectClassifierInput(companyId: string, watchdog: IssueWatchdogRow) {
const issueRows = await loadWatchdogSubtreeIssues(companyId, watchdog.issueId);
const subtree = await loadWatchdogSubtreeIssues(companyId, watchdog.issueId);
const issueRows = subtree.issues;
const subtreeIssueIds = issueRows.map((issue) => issue.id);
if (subtreeIssueIds.length === 0) {
if (subtreeIssueIds.length === 0 || subtree.incomplete) {
return {
watchdog: summarizeIssueWatchdog(watchdog),
issues: [],
issues: issueRows,
subtreeIncomplete: subtree.incomplete,
activeRuns: [],
queuedWakeRequests: [],
blockers: [],
@@ -1665,6 +1695,8 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
allowed: false as const,
reason: classification.state === "stopped"
? "Task-watchdog review is stale because the watched subtree stop fingerprint changed; refresh the source state before mutating it."
: classification.state === "incomplete"
? "Task-watchdog review cannot authorize mutations because the watched subtree scan is incomplete."
: "Task-watchdog review is stale because the watched subtree now has a live, waiting, already-reviewed, or not-applicable path; refresh the source state before mutating it.",
classification,
};
@@ -1763,6 +1795,8 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
live: 0,
pendingFirstRun: 0,
alreadyReviewed: 0,
incomplete: 0,
incompleteIssueIds: [] as string[],
skipped: 0,
watchdogIssueIds: [] as string[],
};
@@ -1782,6 +1816,9 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
result.pendingFirstRun += 1;
} else if (evaluated.state === "already_reviewed") {
result.alreadyReviewed += 1;
} else if (evaluated.state === "incomplete") {
result.incomplete += 1;
result.incompleteIssueIds.push(row.issueId);
} else {
result.skipped += 1;
}
@@ -1799,6 +1836,8 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
checked: 0,
triggered: 0,
pendingFirstRun: 0,
incomplete: 0,
incompleteIssueIds: [] as string[],
skipped: 0,
watchdogIssueIds: [] as string[],
};
@@ -1810,6 +1849,9 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
result.watchdogIssueIds.push(evaluated.watchdogIssueId);
} else if (evaluated.state === "pending_first_run") {
result.pendingFirstRun += 1;
} else if (evaluated.state === "incomplete") {
result.incomplete += 1;
result.incompleteIssueIds.push(row.issueId);
} else if (
evaluated.state === "watchdog_review_open" ||
evaluated.state === "watchdog_live" ||