Files
PaperClipAI/doc/TASK-WATCHDOG.md
T
fengqing 05218571c3 fix(watchdogs): reject incomplete subtree scans (#15063)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task watchdogs verify stopped issue subtrees and restore valid work
paths.
> - A watchdog must read the full subtree before it can prove that all
paths stopped.
> - The current query stops after 100 levels and does not report omitted
children.
> - A running child beyond this limit can be absent from both review
dispatch and mutation checks.
> - This pull request reports an incomplete scan and blocks both
decisions.
> - Complete subtrees keep their current behavior and stop fingerprints.

## Linked Issues or Issue Description

Fixes: #15061

Related PRs: #11327 changes subtree loading but keeps the same depth
limit without a completeness result. #10331 changes mutation freshness
after watchdog writes. Neither patch covers this boundary. This patch
does not take code from those proposals.

## What Changed

- Keep the existing 100-level traversal limit.
- Check for an eligible child beyond the boundary in the same SQL
statement as the subtree read.
- Apply the same company, visibility, harness, and watchdog-origin
filters to this check.
- Return `incomplete` without a stop fingerprint when the scan omits
eligible children.
- Skip review dispatch and reject watchdog mutation revalidation for an
incomplete scan.
- Report incomplete scans through their own count and watched issue IDs.
Log each newly incomplete issue at startup or during periodic
reconciliation. Suppress unchanged results. Allow a new warning after
the issue recovers and becomes incomplete again.
- Add real PostgreSQL regressions for dispatch, a previously valid
mutation scope, the complete boundary leaf, and excluded children.
- Document the incomplete-scan behavior.

## Verification

- Base: `569c7203aa24b95440682983ce7940ba1d4247bd`. Head:
`06f68199656f979ef5eb4897851f0fbcbca592d3`.
- Before the fix, two real PostgreSQL regressions fail. The service
dispatches a review for a partial subtree. It also accepts a prior scope
after a running depth-100 child appears. The other 19 scheduler checks
pass.
- On the exact final head, `pnpm -r typecheck` and `pnpm build` pass on
isolated Ubuntu with Node 24.19.0, pnpm 9.15.4, and runner Rust 1.97.1.
- All 75 related tests pass. Run `pnpm --filter @paperclipai/server exec
vitest run src/__tests__/task-watchdogs-classifier.test.ts
src/__tests__/task-watchdogs-scheduler.test.ts
src/__tests__/server-startup-feedback-export.test.ts
src/__tests__/issue-watchdogs-routes.test.ts --maxWorkers=1`.
- Full `test:run` coverage passes across 29 logical groups: all official
stable-runner lanes plus supplemental service and script tests. The
selection covers all 900 server source test files without duplicate or
omitted selections. It also includes UI, CLI, and other workspace
projects.
- A separate validation tree applies both this PR and #15067 to upstream
`994d6edcdd4e15d5f9cc5cf8c135ac599104b86a` without conflicts. Type
checks, the build, and all 98 related tests across five files pass. Both
public PR heads remain unchanged.
- Four full server lanes required test-process environment corrections.
The host DNS search suffix resolved a `.invalid` fixture to an address.
Ancestor project authentication settings affected hire fixtures. The old
path-isolation tool also failed test imports with `EFAULT`. The exact
base has the same failures: 8 failed and 128 passed assertions without
isolation, then three failed imports with the old tool. All 136 checks
pass on both the base and this head with the corrected isolation. All
four original full lanes pass with the corrected tool, a resolver
without a search suffix, empty ancestor project settings, and the
standard Node path. No assertion or wait limit changed. The host
configuration remains unchanged.
- Earlier interrupted Ubuntu and Docker runs are retained and are not
counted as passes. This exact-head Ubuntu run passes the original chat,
port-conflict, and serialized lanes. The causes of the earlier host-loss
and timing failures are not fully established.
- The startup import regression in an earlier revision is fixed by a
diagnostics module with no database dependencies. All 21 startup checks
pass.
- Exact-head CI, including the added scans after marking the PR ready:
54 successful checks and 2 skipped checks. Greptile: 5/5, no outstanding
findings or follow-ups. Both review threads are resolved.
- The final audit rechecked issue authors, assignees, comments, 206 open
watchdog search results, root functions and fields, source history, and
related diffs. The latest upstream is
`994d6edcdd4e15d5f9cc5cf8c135ac599104b86a`. Its new patches do not cover
this root. No competing root repair was found within the audited scope.
The issue has no assignee or new comment.
- No model provider, production database, or user instance was used.
This verifies service behavior, not a full HTTP authorization bypass.

## Risks

- A subtree with eligible children beyond 100 levels now waits instead
of running a review. This is a deliberate safety boundary. It remains
blocked until the scan can include the full subtree.
- The boundary check adds a child-existence lookup to the existing
recursive query.
- No schema migration is required. Complete subtrees retain their
fingerprint format.
- This does not add unlimited traversal or close the separate race
between a subtree snapshot and later source mutations.
- The configuration lifecycle defect in #15060 is handled by separate PR
#15067. This branch does not include its schema or configuration
changes.

## Model Used

- OpenAI Codex, based on GPT-6, assisted with source analysis,
implementation, and test execution. The exact serving model ID and
context window are not exposed in this session. The agent used
reasoning, shell tools, GitHub CLI, and real PostgreSQL tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-10-26 20:53:47 -07:00

15 KiB
Raw Blame History

Task Watchdog

Table of contents


A task watchdog is an agent you assign to verify a stopped issue tree and put it back into motion when stopping was a mistake. You configure it on a single issue, and it watches that issue plus its non-watchdog descendants. When every leaf in that subtree comes to rest — done, cancelled, blocked, in review, or waiting on an interaction — and there is no live continuation path, Paperclip wakes the watchdog agent to read the evidence and decide whether the stop is legitimate.

Watchdogs are opt-in per issue. There is no global "watch everything" mode.


Why it exists

Agents can stop work for the wrong reasons: misreading a blocker, accepting a stale plan confirmation, declaring "done" without proof, leaving an issue in_review with no real reviewer, or running into a recoverable failure and giving up. None of those failures wake anyone on their own — the tree just sits.

A task watchdog gives you a second pass on stopped work without rerunning the original assignee. It is verification-shaped, not execution-shaped: the watchdog reads what other agents claimed, checks it against the evidence in the thread, and either accepts the stop or restores a live path.

It is not an output-silence monitor for active runs. That is a separate mechanism — the silent active-run watchdog described in doc/execution-semantics.md §12. Task watchdog only fires when the whole watched subtree has come to rest.


Mental model

Three concepts share the word "watchdog" inside Paperclip. Keep them separate:

Concept What it watches When it fires
Task watchdog (this doc) A configured issue + its non-watchdog descendants The whole watched subtree has stopped and the stop is new
Silent active-run watchdog A single still-running process The process has produced no output for the threshold window
Liveness recovery Any agent-owned in_progress issue with no live path Stalled work detected during the periodic recovery scan

The task watchdog is configured by you (or by an agent on your behalf). The other two run automatically on every project.


Configuration

A watchdog has three fields:

Field Required Notes
Watched issue yes The issue you attach the watchdog to. Configured implicitly via the issue you edit.
Watchdog agent yes Any same-company, invokable agent. Cannot be paused, terminated, or budget-blocked.
Instructions no Free-form text (trimmed; empty becomes null). Can narrow focus; cannot expand authority.

A single watched issue holds at most one active watchdog. Re-assigning the agent or editing instructions invalidates the previously reviewed state and forces a fresh evaluation on the next scan.

From the UI

Two surfaces edit the watchdog:

  • New issue dialog — the three-dot menu reveals a Watchdog row. Pick an agent and (optionally) type instructions. The chip in the dialog footer shows the chosen agent and a snippet of the instructions. The watchdog is created together with the issue.
  • Issue properties — the Watchdog row sits next to Monitor. Empty state reads Set watchdog; configured state shows the watchdog agent icon and name plus a truncated instructions preview. Click the row to open the editor popover with an agent picker, an instructions textarea ("What should the watchdog watch for and how should it keep work moving?"), a Remove button, and a Set watchdog / Update button. When a watchdog run has produced a child review task, the row shows a small badge linking to that task.

From the API

GET    /api/issues/:issueId/watchdog
PUT    /api/issues/:issueId/watchdog   { "agentId": "...", "instructions": "..." | null }
DELETE /api/issues/:issueId/watchdog

PUT is upsert. DELETE disables the row (it is not hard-deleted; the table keeps the history for audit). All three routes require write access to the watched issue and produce activity records (issue.watchdog_created, issue.watchdog_updated, issue.watchdog_removed) with the run id and actor.


How a scan works

Paperclip runs a watchdog reconciliation tick at server startup, at the end of each heartbeat cycle, and on demand after any mutation that could change the watched subtree (status, blockers, assignment, interactions). The tick is per-company and only walks active rows.

For each active watchdog the tick:

  1. Walks the watched subtree. Starts at the configured issue and follows parent_id downward, excluding every issue whose originKind = 'task_watchdog' and everything below it. This excludes the watchdog's own review tasks so it cannot trigger itself. The scan includes at most 100 levels, with the watched issue at level 1. If an eligible child remains beyond that limit, the result is incomplete. Paperclip skips review dispatch and rejects watchdog mutation revalidation until it can read the full subtree. A leaf at level 100 does not by itself make the scan incomplete. Reconciliation reports these cases separately through incomplete and incompleteIssueIds. Startup and periodic reconciliation warn when a watched issue first becomes incomplete. The warning contains the newly affected issue IDs. An unchanged incomplete issue does not produce repeated warnings. Once a successful scan no longer reports that issue as incomplete, a later incomplete scan can warn again.
  2. Checks for live paths. If any included issue has a live run (queued, running, scheduled_retry), a queued wake request, or a scheduled retry, the subtree is live and the watchdog does not fire.
  3. Computes a stop fingerprint. A SHA-256 hash over the stopped leaves' identifiers, statuses, blockers, pending interactions, and the current watchdog configuration. The configuration is part of the fingerprint, so changing the agent or instructions invalidates the previously reviewed state.
  4. Compares against lastReviewedFingerprint. Match → suppress (the watchdog already saw this exact stopped state). New → proceed.
  5. Ensures a review task exists. Creates (or reopens) one child issue with originKind = 'task_watchdog' and originId = watchedIssueId. Idempotent per watchdog — only one review task is ever live at a time.
  6. Wakes the watchdog agent. Sends a wake with wakeReason = task_watchdog_stopped_subtree, the stop fingerprint, the leaf summaries, the default mandate, and any custom instructions. The idempotency key is (watchdogId, stopFingerprint), so retries cannot stack duplicate wakes.

When the subtree changes between scans (someone restarts work, adds a blocker, or accepts an interaction) the stop fingerprint changes too, and the watchdog will be woken again for the new state — even if the previous run already disposed of an earlier fingerprint.


What the watchdog agent does

On wake, the watchdog agent reads a fixed default mandate plus your custom instructions. The mandate explicitly tells it to:

  • Treat every stopped leaf as a claim that must be verified against comments, documents, work products, screenshots, tests, blockers, and review state. Do not accept "I could not" or "waiting for approval" as automatically valid.
  • Leave genuinely-complete leaves alone, with a short note on what was checked.
  • If a leaf is not genuinely complete, restore a live path: reopen the issue, reassign, comment actionable instructions, create a follow-up child issue inside the watched subtree, or resolve an interaction that the ordinary agent audience permits.
  • If the blocker is real, leave a valid waiting disposition that names the unblock owner and the next action.

The mandate also enforces safety constraints that custom instructions cannot override:

  • Stay inside the watched subtree. No cross-company mutations, no mutations outside the watched issue and its non-watchdog descendants.
  • No impersonating formal approvals, accepting spend or hiring decisions, using an interaction response as downstream authority, or bypassing execution-policy stages that require a typed reviewer or approver.
  • No creating another watchdog for the watched subtree. No waking itself. Exactly one reusable review task per watched issue.
  • Custom instructions can narrow focus or veto specific shortcuts. They cannot grant authority the server does not already give the watchdog.

The formal authority contract (the full list of allowed and disallowed mutations, plus the ordinary interaction resolver contract) is in doc/SPEC-implementation.md §§9.8.1–9.9.

Writing custom instructions

Custom instructions are most useful when they tell the watchdog what evidence to look at and what shortcuts to refuse. Examples:

Before accepting any leaf as done, check that there is a corresponding green CI run linked in the comments. If there isn't, reopen the leaf and ask for one.

When an interaction is not_creator or human_only, preserve that restriction. Do not treat the watchdog role as an override.

If a leaf is blocked on the marketing team, accept the wait but make sure the unblock owner is named in the blocker reason.

What custom instructions cannot do: grant authority outside the watched subtree, approve formal governance decisions, widen an interaction's resolver audience, or override safety constraints. The server enforces this regardless of what the instructions say.


Scope enforcement

Every watchdog-originated mutation is gated by a server-side scope check derived from the agent run's contextSnapshot.taskWatchdog field. The check resolves to a { kind: "watchdog", watchdogId, companyId, watchedIssueId, watchdogIssueId } envelope and rejects:

  • mutations on issues outside the watched subtree (parent-chain walk, depth-limited)
  • mutations on issues whose company id does not match the watchdog's company
  • interaction resolution outside the ordinary effective audience, named addressee, company/issue scope, run-attribution, low-trust/task-bridge, target-staleness, or exact-once checks
  • changes to the watchdog configuration itself (a watchdog cannot edit its own row or create another watchdog)
  • direct edits to active-run output or execution-policy decisions that require a typed participant

The check is wired into the issue update, status change, blocker, assignment, and interaction routes. Any disallowed mutation is rejected at the route layer; the watchdog agent must take a different path (comment, in-subtree follow-up issue, leave a valid waiting state, escalate to a human owner).


Origin and badges

Watchdog-generated review tasks carry originKind = 'task_watchdog' and originId = watchedIssueId. The UI surfaces this in three places:

  • The properties row on the watched issue shows a small task badge linking to the active review task whenever one exists.
  • The review task itself carries an origin badge distinguishing it from manually created child issues.
  • The board's audit activity feed labels every watchdog-driven mutation with the watchdog id, source issue id, watchdog issue id, run id, and stop fingerprint.

These origin markers are also what excludes the review task from future scans. The walk-down ignores them, so the watchdog cannot scan or trigger on its own review tasks.


When not to use a watchdog

A task watchdog is useful when:

  • the work has many leaves and you want a second pass before trusting "all green"
  • the original assignee tends to declare done too fast, or accept plans you would not accept
  • the tree is important enough that a missed false-stop is worth an extra agent run

It is not the right tool for:

  • monitoring a single running process for silence — that is the silent active-run watchdog, automatic, no configuration
  • liveness recovery on stalled agent-owned issues without an explicit recovery surface — that is automatic too
  • formal approvals or security-sensitive downstream actions — an interaction response cannot authorize those
  • replacing a human reviewer on a typed execution-policy stage — the watchdog cannot bypass typed participants

If what you actually want is "wake me when this is done," use a routine or an issue-thread interaction with continuationPolicy: wake_assignee, not a watchdog.


Reference

Topic File
Authority contract (formal) doc/SPEC-implementation.md §9.9
Execution semantics (formal) doc/execution-semantics.md §11
Silent active-run watchdog doc/execution-semantics.md §12
Database schema packages/db/src/schema/issue_watchdogs.ts
Server service server/src/services/task-watchdogs.ts
Scope enforcement server/src/services/task-watchdog-scope.ts
Wake context + default mandate packages/adapter-utils/src/server-utils.ts (WATCHDOG_DEFAULT_MANDATE)
HTTP routes server/src/routes/issues.ts (GET/PUT/DELETE /issues/:id/watchdog)
Properties UI ui/src/components/IssueProperties.tsx (Watchdog row)
New-issue dialog UI ui/src/components/NewIssueDialog.tsx

And that's all folks!

              ,.
             (_|,.
            ,' /, )_______   _
         __j o``-'        `.'-)'
         (")                 \'
          `-j                |
            `-._(           /
               |_\  |--^.  /
              /_]'|_| /_)_/
                 /_]'  /_]'