Files
PaperClipAI/server/src/services/task-watchdogs.ts
T
DottaandPaperclip 124b2cd0ad fix(server): certify background deadlines for idle tenants (#19001)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Background services own future checks, retries, and recovery work.
> - An idle tenant can stop only when a durable wake covers its
remaining duties.
> - The current alarm catalog covers several domains but leaves these
owners blocked.
> - A saved future timestamp alone cannot prove that reconstruction or
admitted work has settled.
> - This pull request binds durable domain snapshots to the actual
registered worker revisions.
> - The tenant can hand off supported future duties and retain all
checks for uncertain or active work.

## Linked Issues or Issue Description

No matching public issue was found. Related work: [domain deadline
handoff #18656](https://github.com/paperclipai/paperclip/pull/18656),
[owned blocker diagnostics
#18632](https://github.com/paperclipai/paperclip/pull/18632), and
[completed history proof
#18646](https://github.com/paperclipai/paperclip/pull/18646). The
separate [native idle-session change
#18988](https://github.com/paperclipai/paperclip/pull/18988) covers
runtime retirement. [Far-future retry re-probes
#13534](https://github.com/paperclipai/paperclip/pull/13534) changes
recovery responsiveness after an upstream success; this change preserves
existing retry decisions.

**What happened?**

Future status-card, external-object, skill-sync, watchdog, retry, and
quota duties lacked complete owner-bound coverage in the idle deadline
proof. A completed quota task could also retain its monitor policy and
timestamp and remain blocked after the scheduler had disarmed it.

**Expected behavior**

A settled registered owner can hand off its exact supported future
deadline. Active work, dirty reconstruction, transaction uncertainty,
changed revisions, claims, unknown state, and unsupported duties keep
the tenant awake. Completed history retains its data and stops recurring
work when the owner has no remaining duty.

**Steps to reproduce**

1. Produce a supported future duty through its normal service.
2. Let the registered worker finish reconstruction and admitted work.
3. Request an owned idle report with the negotiated domain deadline
catalog.
4. Check the saved earliest deadline and generation before durable alarm
registration.
5. Stop and reconstruct the worker. Check that ordinary admission and
claims resume useful work once due.
6. Complete a quota task. Keep its saved policy and timestamp. Check
that linked unfinished work still blocks sleep and that settled history
does not.

**Paperclip version or commit**

The owned patch was qualified at
`699c8569fa34338dd75cdc5731381ec1e2efdb45`. Publication head
`491fac368ee72eaac8ae105d53e6bf6b305d8a6a` normally merges actual master
`039e0dad2c51c80251e1302a19c5566e039c357a`. All 43 owned files are
byte-identical. All 29 new upstream paths are preserved. The later
normal current-master integration is
`8c912e1ba4fc66397a7341ade59373af8075068f`, described separately below.

**Deployment mode**

Source checkout with real PostgreSQL 17 and a controlled local Node
process. No deployed alarm, tenant suspension, database suspension, or
fleet result is claimed.

## What Changed

- Add registered-worker reconciliation revisions and invalidate them on
stop.
- Keep status-card, external-object, and skill deadline interpretation
with their owners.
- Reuse the watchdog classifier against the supplied database snapshot.
- Validate the exact saved retry run and paired queued wake.
- Validate legacy quota source bindings and the current assignee or
reviewer policy.
- Preserve common issue scope, dependency, sibling, and state checks for
covered waits.
- Compose normalized owner snapshots within the existing bounded
deadline proof and recheck revisions after inspection.
- Bind actual service and worker instances from server startup. Missing
bindings preserve existing blockers.
- Retain completed quota policy and timestamps while independently
rejecting linked unfinished runs or wakes.
- Block unconsumed budget evaluation inputs through the owning queue
predicate. Accounting policy and settlement stay unchanged.
- Bind completed quota history through fixed-size UUID-array parameters,
preserve exact company/task/source associations, and guard external
references before the run-ID join.
- Batch watchdog subtrees and shared classifier facts without changing
the 64-query or two-second proof budget; preserve exact company pairing
for first-run evidence.
- Extend production-path regression controls and document supported
coverage and limits.

## Verification

Current integration head: `688abc6cac6016b9fb50e5832f81e2975ed5d49f`.
Fresh exact-head checks completed with **58 successes, three intentional
skips and one neutral review**, including aggregate `ci / verify`. The
complete current Greptile review is **5/5**. The complete current
Ponytail verdict is **Ship (4/5)**, with only optional cleanup of an
unused MCP proof input; both prior correctness findings are resolved.
All review threads and nested pages were inspected. The normal master
merges preserve the registered retry deadline owner and current startup
completion log after the same readiness await. The earlier focused
evidence below remains tied to its tested source revisions.


Current-master integration `8c912e1ba4fc66397a7341ade59373af8075068f`:

- Normally merge actual master
`46fb933b0155d183ce64a8bdc4950d1b58c926f6`. Every nonoverlapping tree
blob retains its owning parent across 14,044 paths. All 42 owner-only
files and 15 critical binding/proof/readiness files remain
byte-identical. Three automatic overlaps preserve the existing watchdog
binding, conservative unavailable-reader fallback and optional actual
owner arguments. The two scoped-deadline conflicts retain the entire
upstream resource-reservation/mid-pass admission/abort algorithm inside
the existing reconciliation revision/running/failure fence, with both
parent test sets.
- Actual TypeScript 7.0.2 server noEmit passed in 9.434 seconds, normal
source declaration layers in 12.102 seconds, and four selected
test/helper/config layers extending the actual CI configuration in 3.167
seconds. No compiler option weakening, heap override, donor declarations
or SDK replacement stub.
- Focused PG0 controls passed: 102 scheduler/proof/owner controls in
16.451 seconds, including all 21 scoped-deadline cases; 121 startup
controls; and all 107 unchanged instance-settings route cases in 6.456
seconds. The latter first hit a local prewire-guard
listener-registration race. Only the ignored guard was corrected to
register the same actual OS-bound server port immediately after native
listen returns. Closed, unknown, foreign, offhost and PG destinations
remain blocked. Original failure evidence is retained; product code and
assertions were unchanged.
- The exact current UI policy command now reports zero findings (0.362
seconds); its seven scanner controls pass. The earlier dependent PR used
an old scanner/source baseline under a newly enforcing trusted workflow
and reported 397 findings. Carrying the genuine current master resolves
that mismatch without editing unrelated UI sites or relaxing the gate.
- Every meaningful worker checked its executing PID/thread/preload hash
and matching receipt. PG0 controls opened no PostgreSQL sockets; all
owned processes exited. Original dependency-cache/DNS, ignored-harness
syntax/type and guard failures remain separate from product results.
- This is fresh semantic and focused runtime qualification, not a local
whole-repository or current-head PostgreSQL acceptance rerun. The
upstream workspace-busy parked-retry/terminal-holder notification path
remains unchanged by this merge and needs its separate current-graph
producer check. Historical six-case/scale results below stay tied to
their measured sources. A matching controlled default-cadence follow-up
is in #19191; no fresh current-head two-cycle or deployed suspension
claim follows from this merge.
- All 59 exact-head check contexts now pass (55 success, one neutral,
three intentional skips), plus the legacy security status. The complete
edited Greptile review is 5/5 for this exact commit, with all
root/nested pages inspected and no unresolved threads. Ponytail stopped
without completing its new turn; it supplied no new verdict or finding.
Prior actual findings remain addressed. Earlier scores/checks below
belong to their explicit earlier commits.


On frozen clean merged source `699c8569fa`:

- Real PostgreSQL acceptance: **6/6 passed**,83.483 seconds wall. Actual
retry producer creates the exact saved run/wake; stopped and
reconstructed owner resumes through current admission, one durable
start, real local process output and normal task completion. Includes
cancellation, drain arrival, retained quota history with queued-run/wake
negatives, watchdog review settlement and budget-input
empty/pending/consumed control using the actual owner.
- Affected coordinator, deadline, scoped-deadline, recovery and
composed-proof controls: **78/78 in five files passed**,32.114 seconds
wall, worker 1/no file parallelism.
- `pnpm exec tsc --noEmit -p server/tsconfig.json`: **passed**,17.467
seconds. Native TypeScript 7.0.2 and Node 24.22.2.
- PostgreSQL tests use one dedicated test database, matching
fixture/admin URLs, pool limit four, one worker, and no embedded
fallback. Every run recorded unchanged source hashes and a clean
checkout.
- The eight-second real-clock retry is a domain execution check. It is
below the Cloud minimum wake lead and is not a positive accepted hosted
sleep cycle.
- The completed quota-history product failure and original
fixture/protocol failures were preserved; the owning-reader fix passed a
real PostgreSQL red/green regression before the fresh merged six-case
pass.
- After runtime completion, the dedicated database had zero other
clients and no owned runtime processes. All three qualification process
PIDs exited. The coordinator independently audits and drops the
database.
- Comment-only cleanup has equal strict TypeScript parser ASTs across 15
files. The final lexical audit finds no added comments relative to
actual master. `git diff --check` passed.
- Earlier retry fixtures omitted responsible-user membership, advanced
JavaScript time past the real PostgreSQL lease clock, or produced stdout
without normal task completion. Corrected fixtures use real membership,
an eight-second real deadline, and ordinary task comment/completion. The
last case exposed legitimate disposition repair rather than duplicate
retry promotion.
- The quota and watchdog controls first ran before admitted work or the
existing 30-second reconstruction duty settled. The stronger tests
require an interim blocked report and then exact settled-owner proof.
Production deadlines were not changed. The separate
completed-quota-history defect was fixed in its reader and verified with
linked unfinished-run and wake controls.
- The status/object and skill owners reviewed the actual final startup,
route and post-backup bindings. Their domain source evidence carries
where token-equivalent. The master coordinator coalescing change was
reviewed separately. Historical domain PostgreSQL results are not
presented as fresh full-suite results on this head.

The later master merge adds retry predecessor cleanup checks under the
existing issue claim lock. The source audit confirms unchanged saved
deadlines and run/wake identities. The three actual retry
lifecycle/cancellation/drain cases passed on the batched candidate with
this parent (31.817 seconds). Those local dependency controls preceded
the current exact-head remote run. Native backup preservation and
TestOps changes from master are retained. No pass is inferred for
changed dependency behavior solely from unchanged owned files.

Review repair published as `643f1e7400`:

- The original reader returned `unknown` after seven actual watchdog
reviews settled across seven companies. The unchanged composed-report
regression passed after batching; the final reader passed in 61.841
seconds, including active queued-work and stopped-owner rejection.
- A separate two-company red/green control rejects terminal-run context
from another company as first-run grace evidence. Shared facts are
serialized once, with per-root IDs and review summaries.
- The affected recovery, proof and diagnostic unit controls passed 47/47
(14.575 seconds). Startup owner-contract fixtures passed 121/121 (10.589
seconds).
- Diagnostic PostgreSQL controls passed7/7 (12.406 seconds), including
the normal budget-input producer and its exact blocker reason. Initial
empty-state setup uses the same cleanup as per-case teardown when a
dedicated test database is reused.
- Actual repository `pnpm exec tsc --noEmit -p server/tsconfig.json`
passed 12.710 seconds with no heap override. Final source has no added
code comments; whitespace checks pass.
- The original query-budget failure, two-company discriminator, initial
cross-suite fixture failure and an interrupted incorrect local-driver
selector are retained separately. The interrupted run is not a pass.
Independent cleanup found zero owned clients or runtime processes.

Quota-scale repair published as `6696a37f8a`:

- The exact 10,000-row completed-history case reproduced
`MAX_PARAMETERS_EXCEEDED`:90,002 bound parameters exceeded the driver's
65,534 limit, and the composed report returned `unknown` despite settled
owners. Original failure and actual driver diagnostics are retained.
- Fixed pending-work queries use three typed UUID-array parameters plus
the result limit, keeping exact company/issue/source tuple bindings. The
primary run-ID column stays uncast; a guarded CASE validates the raw
external reference before conversion. The 10,000-row cap and
64-query/two-second proof budget are unchanged.
- The same 10,000-row composed-report case passed 18.860 seconds, with
unfinished run/wake rejection, malformed-reference safety and
conservative 10,001-row overflow. Its fixture derives retained history
from the actual quota producer and normal task completion, expands valid
persisted issue/source pairs, then starts the actual registered owners.
- The existing actual quota producer case passed 14.187 seconds,
including exact future deadline, retained completion history, unfinished
work, real foreign-company source references and noncanonical/malformed
references.
- Affected owner/proof controls passed 22/22 (26.444 seconds).
Production server noEmit passed 16.039 seconds.
- CI on 643 identified TS7034/TS7005 in the settled-watchdog fixture's
initially untyped collection. It is now explicitly typed. A separate
affected-entry semantic check extending the repository's actual
`server/src/__tests__/tsconfig.json` passed 15.093 seconds: six affected
tests, two owner files, ambient declarations and current imported
source, with unchanged strict/module/lib/path settings. The production
server config excludes that test directory and is not presented as
test-layer proof. No custom suppression or replacement SDK stub was
added.
- All runs were sequential with pool max 4, worker 1,
source/config/command hashes and cleared inherited
provider/LIVE/other-database variables. Cleanup found zero owned
database clients or local runtime processes. At prior head
`6696a37f8a5718cd0f79054969a0084462d41d47`, all required CI gates pass.
The exact-head inventory has 55 successes, one neutral review and three
intentional skips. Greptile is 5/5. The complete latest Ponytail verdict
confirms the quota finding is fixed; its only remaining note is optional
unused MCP proof state, not a behavior defect. All three review threads
are resolved. The earlier 643 result remains historical.

The repository-wide typecheck, tests, build and browser gates passed in
the prior `6696a37f8a5718cd0f79054969a0084462d41d47` CI run. The new
`8c912e1ba4fc66397a7341ade59373af8075068f` full CI also passes. They
were not run as a full local workspace command in the focused PostgreSQL
lease. Local evidence above uses bounded external databases and does not
claim embedded-cluster, live-provider or deployed-alarm qualification.

## Risks

- An unsupported or uncertain owner state must remain blocked. An
optimistic exemption could lose work while compute is stopped.
- Changes during reconstruction or inspection must invalidate the proof.
A memory-only timer is not durable alarm coverage.
- Current authorization, budgets, cancellation, accounting, and resource
ownership still apply on dispatch. The proof does not grant execution
authority.
- The supported writer model uses in-process transaction signals and
startup/resume reconstruction. Independent SQL writers and other
replicas need an explicit wake contract.
- Native, review, coalesced, and taskless retry variants remain
conservative until their owner cases qualify. Reviewer quota and hosted
alarm qualification remain open.
- The shared catalog remains opt-in. Legacy callers and missing owner
bindings retain their old blockers.

## Model Used

OpenAI `gpt-6.1-sol` was the configured model for the delegated service
implementations and reviews. OpenAI Codex, based on GPT-6, performed
managing review and integration. Both used repository tools and code
execution. The exact served backend snapshots and context-window limits
were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run tests locally and they pass on the final merged head
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green — exact current-head run passes
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups —
complete current-head review and all paginated threads inspected
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 13:48:55 -06:00

1960 lines
74 KiB
TypeScript

import type { DeadlineReconciliationSnapshot } from "./deadline-reconciliation.js";
import { idleDeadlineRowLimit, type IdleDeadlineSnapshot } from "./idle-deadline-types.js";
import { notifyStatusCardIssueChange } from "./status-card-notifications.js";
import { notifyIssueRecovery } from "./issue-recovery-notifications.js";
import { createHash } from "node:crypto";
import { and, asc, eq, gte, inArray, isNull, or, sql } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
agentWakeupRequests,
agents,
approvals,
heartbeatRuns,
issueComments,
issueDocuments,
issueApprovals,
issueRelations,
issues,
issueThreadInteractions,
issueWatchdogs,
issueWorkProducts,
} from "@paperclipai/db";
import type { IssueWatchdog, IssueWatchdogSummary } from "@paperclipai/shared";
import { conflict, notFound } from "../errors.js";
import { parseObject } from "../adapters/utils.js";
import { logActivity } from "./activity-log.js";
import { evaluateAgentInvokabilityFromDb } from "./agent-invokability.js";
import { issueService } from "./issues.js";
import { visibleIssueCondition } from "./issue-visibility.js";
import { TASK_WATCHDOG_ORIGIN_KIND } from "./task-watchdog-scope.js";
const TASK_WATCHDOG_STOP_FINGERPRINT_PREFIX = "task_watchdog_stop:";
const TASK_WATCHDOG_SUBTREE_MAX_DEPTH = 100;
const TASK_WATCHDOG_LIVE_RUN_STATUSES = ["queued", "running", "scheduled_retry"] as const;
const TASK_WATCHDOG_WAKE_REQUEST_STATUSES = ["queued", "deferred_issue_execution"] as const;
const TASK_WATCHDOG_TERMINAL_ISSUE_STATUSES = ["done", "cancelled"] as const;
const TASK_WATCHDOG_TERMINAL_RUN_STATUSES = ["succeeded", "interrupted", "failed", "cancelled", "timed_out"] as const;
// Grace window after an issue is created/assigned during which its first
// assignment run/wake may have been enqueued but is not yet visible to a
// watchdog evaluation (the eval can race the issue's own assignment run).
// Within this window a non-terminal issue that has never completed a run is
// treated as not-yet-stopped so the evaluation does not produce a
// false-positive stopped-subtree review. The periodic watchdog reconciler
// re-evaluates after the window, so a genuinely idle issue still triggers.
const TASK_WATCHDOG_FIRST_RUN_GRACE_MS = 15_000;
type ActorFields = {
agentId?: string | null;
userId?: string | null;
runId?: string | null;
};
export type IssueWatchdogUpsertInput = {
agentId: string;
instructions?: string | null;
actor?: ActorFields;
};
type IssueWatchdogRow = typeof issueWatchdogs.$inferSelect;
type IssueRow = typeof issues.$inferSelect;
export type TaskWatchdogClassifierIssue = Pick<
IssueRow,
| "id"
| "companyId"
| "identifier"
| "title"
| "status"
| "parentId"
| "assigneeAgentId"
| "assigneeUserId"
| "originKind"
| "updatedAt"
> & {
// Optional so existing callers/tests that do not care about the first-run
// grace window keep working; the pending-first-run guard is skipped when
// it (or `evaluatedAt`) is absent.
createdAt?: Date | string | null;
latestCommentAt?: Date | string | null;
latestDocumentAt?: Date | string | null;
latestWorkProductAt?: Date | string | null;
};
export type TaskWatchdogClassifierPath = {
companyId: string;
issueId: string | null;
agentId?: string | null;
status: string;
};
export type TaskWatchdogClassifierWaitingPath = {
companyId: string;
issueId: string;
id?: string | null;
kind?: string | null;
status: string;
};
export type TaskWatchdogClassifierRelation = {
companyId: string;
blockerIssueId: string;
blockedIssueId: string;
};
export type TaskWatchdogClassifierConfig = Pick<
IssueWatchdogSummary,
"companyId" | "issueId" | "lastReviewedFingerprint"
> & {
lastReviewedStopSnapshot?: TaskWatchdogStopSnapshot | null;
};
export type TaskWatchdogStoppedLeaf = {
issueId: string;
identifier: string | null;
title: string;
status: string;
assigneeAgentId: string | null;
assigneeUserId: string | null;
blockerIssueIds: string[];
pendingInteractionIds: string[];
pendingApprovalIds: string[];
updatedAt: string;
latestCommentAt: string | null;
latestDocumentAt: string | null;
latestWorkProductAt: string | null;
};
export type TaskWatchdogMaterialLeaf = Pick<
TaskWatchdogStoppedLeaf,
| "issueId"
| "status"
| "assigneeAgentId"
| "assigneeUserId"
| "blockerIssueIds"
| "pendingInteractionIds"
| "pendingApprovalIds"
>;
export type TaskWatchdogWaitsByIssueId = Record<string, {
pendingInteractionIds: string[];
pendingApprovalIds: string[];
}>;
export type TaskWatchdogStopSnapshot = {
version: 2;
fingerprint: string;
materialLeaves: TaskWatchdogMaterialLeaf[];
waitsByIssueId: TaskWatchdogWaitsByIssueId;
};
type TaskWatchdogPendingInteractionsByIssueId = Record<string, Array<{
id: string;
kind: string | null;
}>>;
export type TaskWatchdogClassifierResult =
| {
state: "incomplete";
reason: string;
includedIssueIds: string[];
}
| {
state: "not_applicable";
reason: string;
includedIssueIds: string[];
}
| {
state: "live";
reason: string;
includedIssueIds: string[];
liveIssueIds: string[];
}
| {
state: "pending_first_run";
reason: string;
includedIssueIds: string[];
pendingIssueIds: string[];
}
| {
state: "already_reviewed";
reason: string;
includedIssueIds: string[];
stopFingerprint: string;
stoppedLeaves: TaskWatchdogStoppedLeaf[];
stopSnapshot: TaskWatchdogStopSnapshot;
pendingInteractionsByIssueId: TaskWatchdogPendingInteractionsByIssueId;
}
| {
state: "stopped";
reason: string;
includedIssueIds: string[];
stopFingerprint: string;
stoppedLeaves: TaskWatchdogStoppedLeaf[];
stopSnapshot: TaskWatchdogStopSnapshot;
pendingInteractionsByIssueId: TaskWatchdogPendingInteractionsByIssueId;
};
export type TaskWatchdogClassifierInput = {
watchdog: TaskWatchdogClassifierConfig;
issues: TaskWatchdogClassifierIssue[];
subtreeIncomplete?: boolean;
activeRuns?: TaskWatchdogClassifierPath[];
queuedWakeRequests?: TaskWatchdogClassifierPath[];
blockers?: TaskWatchdogClassifierRelation[];
pendingInteractions?: TaskWatchdogClassifierWaitingPath[];
pendingApprovals?: TaskWatchdogClassifierWaitingPath[];
// Timestamp the evaluation reads its snapshot at. When provided together
// with a positive `firstRunGraceMs`, the classifier suppresses a
// stopped-subtree verdict for issues created within the grace window that
// have never completed a run (their first assignment run/wake may not yet
// be visible). Omit to disable the guard (legacy behavior).
evaluatedAt?: Date | string | null;
firstRunGraceMs?: number | null;
// Ids of included issues that have at least one run in a terminal status.
// Such issues are never treated as "pending first run" — they have
// demonstrably executed, so a stop is genuine rather than a snapshot race.
completedRunIssueIds?: string[];
};
type TaskWatchdogWakeupOptions = {
source?: "timer" | "assignment" | "on_demand" | "automation";
triggerDetail?: "manual" | "ping" | "callback" | "system";
reason?: string | null;
payload?: Record<string, unknown> | null;
idempotencyKey?: string | null;
requestedByActorType?: "user" | "agent" | "system";
requestedByActorId?: string | null;
contextSnapshot?: Record<string, unknown>;
};
type TaskWatchdogWakeup = (
agentId: string,
opts?: TaskWatchdogWakeupOptions,
) => Promise<{ id: string } | null>;
export type TaskWatchdogServiceDeps = {
enqueueWakeup?: TaskWatchdogWakeup;
};
function normalizeInstructions(value: string | null | undefined): string | null {
if (value == null) return null;
const trimmed = value.trim();
return trimmed.length > 0 ? trimmed : null;
}
export function summarizeIssueWatchdog(row: IssueWatchdogRow): IssueWatchdogSummary {
return {
id: row.id,
companyId: row.companyId,
issueId: row.issueId,
watchdogAgentId: row.watchdogAgentId,
instructions: row.instructions,
status: row.status as IssueWatchdogSummary["status"],
watchdogIssueId: row.watchdogIssueId,
lastObservedFingerprint: row.lastObservedFingerprint,
lastReviewedFingerprint: row.lastReviewedFingerprint,
lastTriggeredAt: row.lastTriggeredAt,
lastCompletedAt: row.lastCompletedAt,
triggerCount: row.triggerCount,
createdAt: row.createdAt,
updatedAt: row.updatedAt,
};
}
function toIssueWatchdog(row: IssueWatchdogRow): IssueWatchdog {
return {
...summarizeIssueWatchdog(row),
createdByAgentId: row.createdByAgentId,
createdByUserId: row.createdByUserId,
createdByRunId: row.createdByRunId,
updatedByAgentId: row.updatedByAgentId,
updatedByUserId: row.updatedByUserId,
updatedByRunId: row.updatedByRunId,
};
}
function issueUpdatedAtIso(issue: Pick<TaskWatchdogClassifierIssue, "updatedAt">) {
return issue.updatedAt instanceof Date
? issue.updatedAt.toISOString()
: new Date(String(issue.updatedAt)).toISOString();
}
function optionalIso(value: Date | string | null | undefined): string | null {
if (value == null) return null;
const ms = value instanceof Date ? value.getTime() : new Date(value).getTime();
if (!Number.isFinite(ms)) return null;
return new Date(ms).toISOString();
}
function toEpochMs(value: Date | string | null | undefined): number | null {
if (value == null) return null;
const ms = value instanceof Date ? value.getTime() : new Date(value).getTime();
return Number.isFinite(ms) ? ms : null;
}
function pathIssueIds(paths: TaskWatchdogClassifierPath[] | undefined, companyId: string) {
return new Set(
(paths ?? [])
.filter((path) => path.companyId === companyId && typeof path.issueId === "string" && path.issueId.length > 0)
.map((path) => path.issueId as string),
);
}
function waitingPathIds(
paths: TaskWatchdogClassifierWaitingPath[] | undefined,
companyId: string,
issueId: string,
) {
return (paths ?? [])
.filter((path) => path.companyId === companyId && path.issueId === issueId)
.map((path) => path.id ?? `${path.status}:${path.issueId}`)
.sort();
}
function stableStopFingerprint(input: {
companyId: string;
watchedIssueId: string;
materialLeaves: TaskWatchdogMaterialLeaf[];
waitsByIssueId: TaskWatchdogWaitsByIssueId;
}) {
const payload = JSON.stringify({
version: 2,
companyId: input.companyId,
watchedIssueId: input.watchedIssueId,
materialLeaves: input.materialLeaves,
waitsByIssueId: input.waitsByIssueId,
});
return `task_watchdog_stop:${createHash("sha256").update(payload).digest("hex")}`;
}
function materialLeaf(leaf: TaskWatchdogStoppedLeaf): TaskWatchdogMaterialLeaf {
return {
issueId: leaf.issueId,
status: leaf.status,
assigneeAgentId: leaf.assigneeAgentId,
assigneeUserId: leaf.assigneeUserId,
blockerIssueIds: leaf.blockerIssueIds,
pendingInteractionIds: leaf.pendingInteractionIds,
pendingApprovalIds: leaf.pendingApprovalIds,
};
}
function parseStopSnapshot(value: unknown): TaskWatchdogStopSnapshot | null {
if (!value || typeof value !== "object") return null;
const candidate = value as Partial<TaskWatchdogStopSnapshot>;
if (
candidate.version !== 2 ||
typeof candidate.fingerprint !== "string" ||
!Array.isArray(candidate.materialLeaves) ||
!candidate.waitsByIssueId ||
typeof candidate.waitsByIssueId !== "object"
) return null;
return candidate as TaskWatchdogStopSnapshot;
}
// Snapshots loaded from jsonb columns come back with Postgres's normalized key
// order, so equality checks against freshly built snapshots must not depend on
// object key order.
function canonicalJson(value: unknown): string {
return JSON.stringify(value, (_key, val) =>
val && typeof val === "object" && !Array.isArray(val)
? Object.fromEntries(
Object.entries(val as Record<string, unknown>).sort(([left], [right]) =>
left < right ? -1 : left > right ? 1 : 0
),
)
: val);
}
function isShrinkOfReviewedSnapshot(
current: TaskWatchdogStopSnapshot,
reviewed: TaskWatchdogStopSnapshot | null | undefined,
) {
if (!reviewed || canonicalJson(current.waitsByIssueId) !== canonicalJson(reviewed.waitsByIssueId)) return false;
const reviewedLeaves = new Map(reviewed.materialLeaves.map((leaf) => [leaf.issueId, leaf]));
return current.materialLeaves.every((leaf) => {
const previous = reviewedLeaves.get(leaf.issueId);
return previous != null && canonicalJson(previous) === canonicalJson(leaf);
});
}
export function classifyTaskWatchdogSubtree(input: TaskWatchdogClassifierInput): TaskWatchdogClassifierResult {
const issuesById = new Map(input.issues.map((issue) => [issue.id, issue]));
const root = issuesById.get(input.watchdog.issueId);
if (!root || root.companyId !== input.watchdog.companyId) {
return { state: "not_applicable", reason: "Watched issue is missing.", includedIssueIds: [] };
}
if (root.originKind === TASK_WATCHDOG_ORIGIN_KIND) {
return {
state: "not_applicable",
reason: "Task watchdog origin issues cannot themselves be watched.",
includedIssueIds: [],
};
}
// A partial scan cannot prove that every execution path has stopped.
if (input.subtreeIncomplete) {
return {
state: "incomplete",
reason: "Watched subtree scan is incomplete because it exceeds the traversal depth limit.",
includedIssueIds: [...issuesById.keys()],
};
}
const childrenByParentId = new Map<string, TaskWatchdogClassifierIssue[]>();
for (const issue of input.issues) {
if (issue.companyId !== input.watchdog.companyId || !issue.parentId) continue;
const list = childrenByParentId.get(issue.parentId) ?? [];
list.push(issue);
childrenByParentId.set(issue.parentId, list);
}
for (const children of childrenByParentId.values()) {
children.sort((left, right) => left.id.localeCompare(right.id));
}
const included: TaskWatchdogClassifierIssue[] = [];
const visit = (issue: TaskWatchdogClassifierIssue) => {
if (issue.originKind === TASK_WATCHDOG_ORIGIN_KIND) return;
included.push(issue);
for (const child of childrenByParentId.get(issue.id) ?? []) {
visit(child);
}
};
visit(root);
if (included.length === 0) {
return { state: "not_applicable", reason: "Watched subtree has no non-watchdog issues.", includedIssueIds: [] };
}
const includedIds = included.map((issue) => issue.id);
const includedIdSet = new Set(includedIds);
const liveIssueIds = [
...pathIssueIds(input.activeRuns, input.watchdog.companyId),
...pathIssueIds(input.queuedWakeRequests, input.watchdog.companyId),
].filter((issueId) => includedIdSet.has(issueId));
const uniqueLiveIssueIds = [...new Set(liveIssueIds)].sort();
if (uniqueLiveIssueIds.length > 0) {
return {
state: "live",
reason: "At least one issue in the watched subtree has a live run, queued wake, or scheduled retry.",
includedIssueIds: includedIds,
liveIssueIds: uniqueLiveIssueIds,
};
}
// Pending-first-run guard: a watchdog evaluation triggered as part of issue
// (or watchdog) creation can read its snapshot before the issue's own
// assignment run/wake is committed/visible, making an actively-starting
// subtree look idle. Suppress the stopped verdict for non-terminal issues
// created within the first-run grace window that have never completed a run.
const evaluatedAtMs = toEpochMs(input.evaluatedAt);
const graceMs = input.firstRunGraceMs ?? 0;
if (evaluatedAtMs != null && graceMs > 0) {
const completedRunIssueIds = new Set(input.completedRunIssueIds ?? []);
const pendingIssueIds = included
.filter((issue) => {
if (isTerminalIssueStatus(issue.status)) return false;
if (completedRunIssueIds.has(issue.id)) return false;
const createdAtMs = toEpochMs(issue.createdAt);
if (createdAtMs == null) return false;
return evaluatedAtMs - createdAtMs < graceMs;
})
.map((issue) => issue.id)
.sort();
if (pendingIssueIds.length > 0) {
return {
state: "pending_first_run",
reason:
"A watched issue was created within the first-run grace window and has not yet completed a run; deferring evaluation until its first assignment run/wake is observable.",
includedIssueIds: includedIds,
pendingIssueIds,
};
}
}
const includedChildrenByParentId = new Map<string, string[]>();
for (const issue of included) {
if (!issue.parentId || !includedIdSet.has(issue.parentId)) continue;
const list = includedChildrenByParentId.get(issue.parentId) ?? [];
list.push(issue.id);
includedChildrenByParentId.set(issue.parentId, list);
}
const blockersByIssueId = new Map<string, string[]>();
for (const relation of input.blockers ?? []) {
if (relation.companyId !== input.watchdog.companyId) continue;
if (!includedIdSet.has(relation.blockedIssueId)) continue;
const list = blockersByIssueId.get(relation.blockedIssueId) ?? [];
list.push(relation.blockerIssueId);
blockersByIssueId.set(relation.blockedIssueId, list);
}
const nonTerminalIssues = included
.filter((issue) => !isTerminalIssueStatus(issue.status))
.sort((left, right) => left.id.localeCompare(right.id));
const waitsByIssueId = Object.fromEntries(nonTerminalIssues
.map((issue) => [issue.id, {
pendingInteractionIds: waitingPathIds(input.pendingInteractions, input.watchdog.companyId, issue.id),
pendingApprovalIds: waitingPathIds(input.pendingApprovals, input.watchdog.companyId, issue.id),
}] as const)
.filter(([, waits]) => waits.pendingInteractionIds.length > 0 || waits.pendingApprovalIds.length > 0));
const pendingInteractionsByIssueId = Object.fromEntries(nonTerminalIssues
.map((issue) => [issue.id, (input.pendingInteractions ?? [])
.filter((path) => path.companyId === input.watchdog.companyId && path.issueId === issue.id)
.map((path) => ({ id: path.id ?? `${path.status}:${path.issueId}`, kind: path.kind ?? null }))
.sort((left, right) => left.id.localeCompare(right.id))] as const)
.filter(([, waits]) => waits.length > 0));
const leaves = included
.filter((issue) => (includedChildrenByParentId.get(issue.id) ?? []).length === 0)
.filter((issue) => !isTerminalIssueStatus(issue.status))
.sort((left, right) => left.id.localeCompare(right.id))
.map((issue) => ({
issueId: issue.id,
identifier: issue.identifier,
title: issue.title,
status: issue.status,
assigneeAgentId: issue.assigneeAgentId,
assigneeUserId: issue.assigneeUserId,
blockerIssueIds: [...new Set(blockersByIssueId.get(issue.id) ?? [])].sort(),
pendingInteractionIds: waitingPathIds(input.pendingInteractions, input.watchdog.companyId, issue.id),
pendingApprovalIds: waitingPathIds(input.pendingApprovals, input.watchdog.companyId, issue.id),
updatedAt: issueUpdatedAtIso(issue),
latestCommentAt: optionalIso(issue.latestCommentAt),
latestDocumentAt: optionalIso(issue.latestDocumentAt),
latestWorkProductAt: optionalIso(issue.latestWorkProductAt),
}));
const materialLeaves = leaves.map(materialLeaf);
const stopFingerprint = stableStopFingerprint({
companyId: input.watchdog.companyId,
watchedIssueId: input.watchdog.issueId,
materialLeaves,
waitsByIssueId,
});
const currentStopSnapshot: TaskWatchdogStopSnapshot = {
version: 2,
fingerprint: stopFingerprint,
materialLeaves,
waitsByIssueId,
};
if (
input.watchdog.lastReviewedFingerprint === stopFingerprint ||
isShrinkOfReviewedSnapshot(currentStopSnapshot, input.watchdog.lastReviewedStopSnapshot)
) {
return {
state: "already_reviewed",
reason: "The current stopped subtree fingerprint was already reviewed by the watchdog.",
includedIssueIds: includedIds,
stopFingerprint,
stoppedLeaves: leaves,
stopSnapshot: currentStopSnapshot,
pendingInteractionsByIssueId,
};
}
return {
state: "stopped",
reason: "No issue in the watched subtree has a live execution path.",
includedIssueIds: includedIds,
stopFingerprint,
stoppedLeaves: leaves,
stopSnapshot: currentStopSnapshot,
pendingInteractionsByIssueId,
};
}
async function assertWatchedIssue(dbOrTx: any, companyId: string, issueId: string) {
const issue = await dbOrTx
.select({ id: issues.id, companyId: issues.companyId })
.from(issues)
.where(and(eq(issues.id, issueId), eq(issues.companyId, companyId)))
.then((rows: Array<{ id: string; companyId: string }>) => rows[0] ?? null);
if (!issue) throw notFound("Issue not found");
return issue;
}
async function assertWatchdogAgentInvokable(dbOrTx: any, companyId: string, agentId: string) {
const agent = await dbOrTx
.select({
id: agents.id,
companyId: agents.companyId,
name: agents.name,
reportsTo: agents.reportsTo,
status: agents.status,
})
.from(agents)
.where(eq(agents.id, agentId))
.then((rows: Array<{
id: string;
companyId: string;
name: string;
reportsTo: string | null;
status: string;
}>) => rows[0] ?? null);
if (!agent || agent.companyId !== companyId) {
throw notFound("Watchdog agent not found");
}
const invokability = await evaluateAgentInvokabilityFromDb(dbOrTx as Db, agent);
if (!invokability.invokable) {
throw conflict("Cannot assign watchdog to an agent that is not invokable", invokability);
}
return agent;
}
function readNonEmptyString(value: unknown): string | null {
return typeof value === "string" && value.trim().length > 0 ? value : null;
}
function issueIdFromRunContext(contextSnapshot: unknown) {
const context = parseObject(contextSnapshot);
return readNonEmptyString(context.issueId) ?? readNonEmptyString(context.taskId);
}
function issueIdFromWakePayload(payload: unknown) {
const parsed = parseObject(payload);
const nested = parseObject(parsed._paperclipWakeContext);
return readNonEmptyString(parsed.issueId) ??
readNonEmptyString(parsed.taskId) ??
readNonEmptyString(nested.issueId) ??
readNonEmptyString(nested.taskId);
}
function normalizeStopFingerprint(value: string | null | undefined) {
const trimmed = value?.trim();
return trimmed?.startsWith(TASK_WATCHDOG_STOP_FINGERPRINT_PREFIX) ? trimmed : null;
}
function stopFingerprintFromText(value: string | null | undefined) {
const match = value?.match(/task_watchdog_stop:[a-f0-9]+/i);
return normalizeStopFingerprint(match?.[0] ?? null);
}
function reviewedFingerprintForWatchdogIssue(issue: Pick<IssueRow, "originFingerprint" | "description">) {
return normalizeStopFingerprint(issue.originFingerprint) ?? stopFingerprintFromText(issue.description);
}
function taskWatchdogWakeIdempotencyKey(watchdogId: string, stopFingerprint: string) {
return `task_watchdog:${watchdogId}:${stopFingerprint}`;
}
function buildStoppedFingerprintComment(input: {
sourceIssue: Pick<IssueRow, "identifier" | "id">;
stopFingerprint: string;
stoppedLeaves: TaskWatchdogStoppedLeaf[];
pendingInteractionsByIssueId: TaskWatchdogPendingInteractionsByIssueId;
resumed: boolean;
}) {
const shortId = (id: string) => id.length > 8 ? `${id.slice(0, 8)}…` : id;
const leafLines = input.stoppedLeaves.slice(0, 12).map((leaf) => {
const interactionKinds = new Map(
(input.pendingInteractionsByIssueId[leaf.issueId] ?? []).map((wait) => [wait.id, wait.kind]),
);
const waits = [
...leaf.pendingInteractionIds.map((id) => `${interactionKinds.get(id) ?? "interaction"} ${shortId(id)}`),
...leaf.pendingApprovalIds.map((id) => `approval ${shortId(id)}`),
];
return `- ${leaf.identifier ?? leaf.issueId}: ${leaf.status}${waits.length > 0 ? ` (pending ${waits.join(", ")})` : ""}`;
});
const more = input.stoppedLeaves.length > leafLines.length
? `\n- ...and ${input.stoppedLeaves.length - leafLines.length} more stopped leaves`
: "";
return [
input.resumed ? "Task watchdog resumed for stopped subtree." : "Task watchdog started for stopped subtree.",
"",
`Watched issue: ${input.sourceIssue.identifier ?? input.sourceIssue.id}`,
`Stopped fingerprint: \`${input.stopFingerprint}\``,
"",
"Stopped leaves:",
...(leafLines.length > 0 ? leafLines : ["- No leaf issues found."]),
more,
].filter((line) => line !== "").join("\n");
}
function stoppedFingerprintMetadata(input: {
sourceIssueId: string;
stopFingerprint: string;
waitsByIssueId: TaskWatchdogWaitsByIssueId;
resumed: boolean;
}) {
const pendingWaitCount = Object.values(input.waitsByIssueId).reduce(
(count, waits) => count + waits.pendingInteractionIds.length + waits.pendingApprovalIds.length,
0,
);
return {
version: 1 as const,
sections: [
{
title: "Task Watchdog",
rows: [
{ type: "text" as const, label: "Watched issue", text: input.sourceIssueId },
{ type: "text" as const, label: "Stopped fingerprint", text: input.stopFingerprint },
{ type: "text" as const, label: "Pending waits", text: String(pendingWaitCount) },
{ type: "text" as const, label: "Resume intent", text: input.resumed ? "true" : "false" },
],
},
],
};
}
function watchdogWakeContext(input: {
watchdog: IssueWatchdogRow;
watchdogIssue: IssueRow;
sourceIssue: IssueRow;
classification: Extract<TaskWatchdogClassifierResult, { state: "stopped" }>;
}) {
return {
issueId: input.watchdogIssue.id,
taskId: input.watchdogIssue.id,
wakeReason: "task_watchdog_stopped_subtree",
source: TASK_WATCHDOG_ORIGIN_KIND,
taskWatchdog: {
watchedIssueId: input.sourceIssue.id,
watchedIssueIdentifier: input.sourceIssue.identifier,
watchedIssueTitle: input.sourceIssue.title,
stopFingerprint: input.classification.stopFingerprint,
// The wake normalizer only reads `taskWatchdog`, so board instructions
// must live here. A top-level copy is silently dropped and the mandate
// renders "No board-supplied watchdog instructions" instead.
customInstructions: input.watchdog.instructions,
// Same contract: the mandate's "Terminal / stopped leaves to verify"
// section reads `terminalLeafSummaries` here, keyed on `id`, not the
// top-level `stoppedLeaves` array keyed on `issueId`.
terminalLeafSummaries: input.classification.stoppedLeaves.map((leaf) => ({
id: leaf.issueId,
identifier: leaf.identifier,
title: leaf.title,
status: leaf.status,
})),
pendingInteractions: input.classification.pendingInteractionsByIssueId,
pendingApprovals: Object.fromEntries(Object.entries(input.classification.stopSnapshot.waitsByIssueId)
.filter(([, waits]) => waits.pendingApprovalIds.length > 0)
.map(([issueId, waits]) => [issueId, waits.pendingApprovalIds])),
capabilities: {
targetScope: {
watchedIssueId: input.sourceIssue.id,
watchedIssueIdentifier: input.sourceIssue.identifier,
watchdogIssueId: input.watchdogIssue.id,
includeNonWatchdogDescendants: true,
excludedOriginKinds: [TASK_WATCHDOG_ORIGIN_KIND],
},
operations: [
"comment_on_watched_subtree_issues",
"transition_watched_subtree_issue_status",
"reassign_watched_subtree_issues",
"create_child_issues_under_non_watchdog_watched_subtree",
"create_product_bug_followups_outside_watched_subtree",
"resolve_issue_thread_interactions_through_ordinary_audience_policy",
"update_reusable_watchdog_issue",
],
deniedOperations: [
"create_visible_probe_issues_or_throwaway_tasks",
"create_product_bug_followups_as_source_tree_children",
"mutate_task_watchdog_descendants",
"mutate_outside_watched_subtree",
"resolve_human_only_interactions_or_security_sensitive_approvals",
"create_nested_task_watchdogs",
],
},
},
watchdogId: input.watchdog.id,
watchedIssueId: input.sourceIssue.id,
watchedIssueIdentifier: input.sourceIssue.identifier,
stopFingerprint: input.classification.stopFingerprint,
stoppedLeaves: input.classification.stoppedLeaves,
customInstructions: input.watchdog.instructions,
resumeIntent: true,
followUpRequested: true,
};
}
function isTerminalIssueStatus(status: string) {
return TASK_WATCHDOG_TERMINAL_ISSUE_STATUSES.includes(
status as (typeof TASK_WATCHDOG_TERMINAL_ISSUE_STATUSES)[number],
);
}
function isWatchdogReviewDisposition(issue: Pick<
IssueRow,
"status" | "assigneeUserId" | "executionState" | "monitorNextCheckAt"
>, hasPendingReviewPath: boolean) {
if (issue.status === "done" || issue.status === "blocked") return true;
if (issue.status !== "in_review") return false;
return Boolean(issue.assigneeUserId || issue.executionState || issue.monitorNextCheckAt || hasPendingReviewPath);
}
function isUniqueConstraintConflict(error: unknown, constraintName: string) {
const queue: unknown[] = [error];
const messages: string[] = [];
let hasUniqueCode = false;
let hasConstraint = false;
for (const candidate of queue) {
if (!candidate || typeof candidate !== "object") continue;
const typed = candidate as {
code?: string;
constraint?: string;
constraint_name?: string;
cause?: unknown;
message?: string;
};
if (typed.code === "23505") hasUniqueCode = true;
if (typed.constraint === constraintName || typed.constraint_name === constraintName) hasConstraint = true;
if (typed.message) messages.push(typed.message);
if (typed.cause) queue.push(typed.cause);
}
const message = messages.join("\n");
return (hasUniqueCode || message.includes("duplicate key value violates unique constraint")) &&
(hasConstraint || message.includes(constraintName));
}
function isActiveTaskWatchdogUniqueConflict(error: unknown) {
return isUniqueConstraintConflict(error, "issues_active_task_watchdog_uq");
}
function isIssueWatchdogUniqueConflict(error: unknown) {
return isUniqueConstraintConflict(error, "issue_watchdogs_company_issue_uq");
}
async function updateIssueWatchdogRow(
dbOrTx: any,
existing: IssueWatchdogRow,
input: IssueWatchdogUpsertInput,
now: Date,
) {
const [updated] = await dbOrTx
.update(issueWatchdogs)
.set({
watchdogAgentId: input.agentId,
instructions: normalizeInstructions(input.instructions),
status: "active",
updatedByAgentId: input.actor?.agentId ?? null,
updatedByUserId: input.actor?.userId ?? null,
updatedByRunId: input.actor?.runId ?? null,
updatedAt: now,
})
.where(eq(issueWatchdogs.id, existing.id))
.returning();
return updated;
}
async function upsertIssueWatchdogInTransaction(
dbOrTx: any,
companyId: string,
issueId: string,
input: IssueWatchdogUpsertInput,
): Promise<{ watchdog: IssueWatchdog; created: boolean }> {
await assertWatchedIssue(dbOrTx, companyId, issueId);
await assertWatchdogAgentInvokable(dbOrTx, companyId, input.agentId);
const now = new Date();
const existing = await dbOrTx
.select()
.from(issueWatchdogs)
.where(and(eq(issueWatchdogs.companyId, companyId), eq(issueWatchdogs.issueId, issueId)))
.then((rows: IssueWatchdogRow[]) => rows[0] ?? null);
await notifyIssueRecovery(dbOrTx, { companyId });
if (existing) {
const updated = await updateIssueWatchdogRow(dbOrTx, existing, input, now);
return { watchdog: toIssueWatchdog(updated), created: false };
}
const insertResult: { row: IssueWatchdogRow; created: boolean } = await dbOrTx.transaction(async (tx: Db) => tx
.insert(issueWatchdogs)
.values({
companyId,
issueId,
watchdogAgentId: input.agentId,
instructions: normalizeInstructions(input.instructions),
status: "active",
createdByAgentId: input.actor?.agentId ?? null,
createdByUserId: input.actor?.userId ?? null,
createdByRunId: input.actor?.runId ?? null,
updatedByAgentId: input.actor?.agentId ?? null,
updatedByUserId: input.actor?.userId ?? null,
updatedByRunId: input.actor?.runId ?? null,
createdAt: now,
updatedAt: now,
})
.returning()
.then((rows: IssueWatchdogRow[]) => ({ row: rows[0], created: true })))
.catch(async (error: unknown) => {
if (!isIssueWatchdogUniqueConflict(error)) throw error;
const winner = await dbOrTx
.select()
.from(issueWatchdogs)
.where(and(eq(issueWatchdogs.companyId, companyId), eq(issueWatchdogs.issueId, issueId)))
.then((rows: IssueWatchdogRow[]) => rows[0] ?? null);
if (!winner) throw error;
const updated = await updateIssueWatchdogRow(dbOrTx, winner, input, now);
return { row: updated, created: false };
});
return { watchdog: toIssueWatchdog(insertResult.row), created: insertResult.created };
}
export function upsertIssueWatchdogForIssue(
dbOrTx: Db | Parameters<Parameters<Db["transaction"]>[0]>[0],
companyId: string,
issueId: string,
input: IssueWatchdogUpsertInput,
): Promise<{ watchdog: IssueWatchdog; created: boolean }> {
return dbOrTx.transaction(tx => upsertIssueWatchdogInTransaction(tx, companyId, issueId, input));
}
export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) {
const issuesSvc = issueService(db);
async function loadWatchdogSubtreeIssues(companyId: string, watchedIssueId: string, reader: Pick<Db, "select" | "execute"> = db, batch?: IssueWatchdogRow[]) {
const roots = batch ?? [{ id: watchedIssueId, companyId, issueId: watchedIssueId }];
const rows = await reader.execute(sql`
WITH RECURSIVE roots(watchdog_id, root_company_id, root_issue_id) AS (
VALUES ${sql.join(roots.map(root => sql`(${root.id}::uuid, ${root.companyId}::uuid, ${root.issueId}::uuid)`), sql`, `)}
), watched_issues AS (
SELECT
id,
company_id,
identifier,
title,
status,
parent_id,
assignee_agent_id,
assignee_user_id,
origin_kind,
updated_at,
created_at,
0 AS depth,
roots.watchdog_id
FROM issues
JOIN roots ON issues.id = roots.root_issue_id AND issues.company_id = roots.root_company_id
WHERE hidden_at IS NULL
AND harness_kind IS NULL
UNION ALL
SELECT
child.id,
child.company_id,
child.identifier,
child.title,
child.status,
child.parent_id,
child.assignee_agent_id,
child.assignee_user_id,
child.origin_kind,
child.updated_at,
child.created_at,
watched_issues.depth + 1,
watched_issues.watchdog_id
FROM issues child
JOIN watched_issues ON child.parent_id = watched_issues.id
WHERE child.company_id = watched_issues.company_id
AND child.hidden_at IS NULL
AND child.harness_kind IS NULL
AND child.origin_kind <> ${TASK_WATCHDOG_ORIGIN_KIND}
AND watched_issues.depth < ${TASK_WATCHDOG_SUBTREE_MAX_DEPTH - 1}
)
SELECT
watchdog_id AS "watchdogId",
id,
company_id AS "companyId",
identifier,
title,
status,
parent_id AS "parentId",
assignee_agent_id AS "assigneeAgentId",
assignee_user_id AS "assigneeUserId",
origin_kind AS "originKind",
updated_at AS "updatedAt",
created_at AS "createdAt",
EXISTS (
SELECT 1
FROM watched_issues boundary
JOIN issues child ON child.parent_id = boundary.id
WHERE boundary.watchdog_id = watched_issues.watchdog_id
AND boundary.depth = ${TASK_WATCHDOG_SUBTREE_MAX_DEPTH - 1}
AND child.company_id = watched_issues.company_id
AND child.hidden_at IS NULL
AND child.harness_kind IS NULL
AND child.origin_kind <> ${TASK_WATCHDOG_ORIGIN_KIND}
) AS "subtreeIncomplete"
FROM watched_issues
${batch ? sql`LIMIT ${idleDeadlineRowLimit + 1}` : sql``}
`);
const issueRows = (Array.isArray(rows) ? rows : []) as Array<
TaskWatchdogClassifierIssue & { watchdogId: string; subtreeIncomplete: boolean }
>;
return { issues: issueRows, incomplete: issueRows.some((row) => row.subtreeIncomplete) };
}
async function collectClassifierInput(companyId: string, watchdog: IssueWatchdogRow, reader: Pick<Db, "select" | "execute"> = db, at = new Date(),
suppliedSubtree?: { issues: TaskWatchdogClassifierIssue[]; incomplete: boolean }, companyIds = [companyId]) {
const subtree = suppliedSubtree ?? await loadWatchdogSubtreeIssues(companyId, watchdog.issueId, reader);
const issueRows = subtree.issues;
const subtreeIssueIds = issueRows.map((issue) => issue.id);
if (subtreeIssueIds.length === 0 || subtree.incomplete) {
return {
watchdog: summarizeIssueWatchdog(watchdog),
issues: issueRows,
subtreeIncomplete: subtree.incomplete,
activeRuns: [],
queuedWakeRequests: [],
blockers: [],
pendingInteractions: [],
pendingApprovals: [],
evaluatedAt: at,
firstRunGraceMs: TASK_WATCHDOG_FIRST_RUN_GRACE_MS,
completedRunIssueIds: [],
} satisfies TaskWatchdogClassifierInput;
}
const [
activeRunRows,
activeIssueRunRows,
wakeRows,
blockerRows,
interactionRows,
approvalRows,
commentActivityRows,
documentActivityRows,
workProductActivityRows,
] = await Promise.all([
reader
.select({
companyId: heartbeatRuns.companyId,
agentId: heartbeatRuns.agentId,
status: heartbeatRuns.status,
contextSnapshot: heartbeatRuns.contextSnapshot,
})
.from(heartbeatRuns)
.where(and(
inArray(heartbeatRuns.companyId, companyIds),
inArray(heartbeatRuns.status, [...TASK_WATCHDOG_LIVE_RUN_STATUSES]),
or(
inArray(sql`${heartbeatRuns.contextSnapshot}->>'issueId'`, subtreeIssueIds),
inArray(sql`${heartbeatRuns.contextSnapshot}->>'taskId'`, subtreeIssueIds),
),
)),
reader
.select({
companyId: issues.companyId,
agentId: heartbeatRuns.agentId,
status: heartbeatRuns.status,
issueId: issues.id,
})
.from(issues)
.innerJoin(heartbeatRuns, eq(issues.executionRunId, heartbeatRuns.id))
.where(and(
inArray(issues.companyId, companyIds),
inArray(issues.id, subtreeIssueIds),
visibleIssueCondition(),
inArray(heartbeatRuns.status, [...TASK_WATCHDOG_LIVE_RUN_STATUSES]),
)),
reader
.select({
companyId: agentWakeupRequests.companyId,
agentId: agentWakeupRequests.agentId,
status: agentWakeupRequests.status,
payload: agentWakeupRequests.payload,
})
.from(agentWakeupRequests)
.where(and(
inArray(agentWakeupRequests.companyId, companyIds),
inArray(agentWakeupRequests.status, [...TASK_WATCHDOG_WAKE_REQUEST_STATUSES]),
or(
inArray(sql`${agentWakeupRequests.payload}->>'issueId'`, subtreeIssueIds),
inArray(sql`${agentWakeupRequests.payload}->>'taskId'`, subtreeIssueIds),
inArray(sql`${agentWakeupRequests.payload}->'_paperclipWakeContext'->>'issueId'`, subtreeIssueIds),
inArray(sql`${agentWakeupRequests.payload}->'_paperclipWakeContext'->>'taskId'`, subtreeIssueIds),
),
)),
reader
.select({
companyId: issueRelations.companyId,
blockerIssueId: issueRelations.issueId,
blockedIssueId: issueRelations.relatedIssueId,
})
.from(issueRelations)
.where(and(
inArray(issueRelations.companyId, companyIds),
eq(issueRelations.type, "blocks"),
inArray(issueRelations.relatedIssueId, subtreeIssueIds),
)),
reader
.select({
companyId: issueThreadInteractions.companyId,
issueId: issueThreadInteractions.issueId,
id: issueThreadInteractions.id,
kind: issueThreadInteractions.kind,
status: issueThreadInteractions.status,
})
.from(issueThreadInteractions)
.where(and(
inArray(issueThreadInteractions.companyId, companyIds),
inArray(issueThreadInteractions.issueId, subtreeIssueIds),
eq(issueThreadInteractions.status, "pending"),
)),
reader
.select({
companyId: issueApprovals.companyId,
issueId: issueApprovals.issueId,
id: approvals.id,
status: approvals.status,
})
.from(issueApprovals)
.innerJoin(approvals, eq(issueApprovals.approvalId, approvals.id))
.where(and(
inArray(issueApprovals.companyId, companyIds),
inArray(issueApprovals.issueId, subtreeIssueIds),
inArray(approvals.status, ["pending", "revision_requested"]),
)),
reader
.select({
issueId: issueComments.issueId,
latestAt: sql<Date | null>`MAX(${issueComments.updatedAt})`,
})
.from(issueComments)
.where(and(
inArray(issueComments.companyId, companyIds),
inArray(issueComments.issueId, subtreeIssueIds),
isNull(issueComments.deletedAt),
))
.groupBy(issueComments.issueId),
reader
.select({
issueId: issueDocuments.issueId,
latestAt: sql<Date | null>`MAX(${issueDocuments.updatedAt})`,
})
.from(issueDocuments)
.where(and(
inArray(issueDocuments.companyId, companyIds),
inArray(issueDocuments.issueId, subtreeIssueIds),
))
.groupBy(issueDocuments.issueId),
reader
.select({
issueId: issueWorkProducts.issueId,
latestAt: sql<Date | null>`MAX(${issueWorkProducts.updatedAt})`,
})
.from(issueWorkProducts)
.where(and(
inArray(issueWorkProducts.companyId, companyIds),
inArray(issueWorkProducts.issueId, subtreeIssueIds),
))
.groupBy(issueWorkProducts.issueId),
]);
const latestCommentByIssueId = new Map(commentActivityRows.map((row) => [row.issueId, row.latestAt]));
const latestDocumentByIssueId = new Map(documentActivityRows.map((row) => [row.issueId, row.latestAt]));
const latestWorkProductByIssueId = new Map(workProductActivityRows.map((row) => [row.issueId, row.latestAt]));
const evaluatedAt = at;
const evaluatedAtMs = evaluatedAt.getTime();
// Only the issues created within the first-run grace window can be racing
// their own assignment run; scope the (potentially expensive) terminal-run
// lookup to those few issues so the common path stays a no-op.
const freshIssueIds = issueRows
.filter((row) => {
if (isTerminalIssueStatus(row.status)) return false;
const createdAtMs = toEpochMs(row.createdAt);
return createdAtMs != null && evaluatedAtMs - createdAtMs < TASK_WATCHDOG_FIRST_RUN_GRACE_MS;
})
.map((row) => row.id);
const completedRunIssueIds = await collectCompletedRunIssueIds(companyIds, freshIssueIds, reader,
new Map(issueRows.map(issue => [issue.id, issue.companyId])));
return {
watchdog: {
...summarizeIssueWatchdog(watchdog),
lastReviewedStopSnapshot: parseStopSnapshot(watchdog.lastReviewedStopSnapshot),
},
issues: issueRows.map((issue) => ({
...issue,
latestCommentAt: latestCommentByIssueId.get(issue.id) ?? null,
latestDocumentAt: latestDocumentByIssueId.get(issue.id) ?? null,
latestWorkProductAt: latestWorkProductByIssueId.get(issue.id) ?? null,
})),
activeRuns: activeRunRows.map((row) => ({
companyId: row.companyId,
agentId: row.agentId,
status: row.status,
issueId: issueIdFromRunContext(row.contextSnapshot),
})).concat(activeIssueRunRows),
queuedWakeRequests: wakeRows.map((row) => ({
companyId: row.companyId,
agentId: row.agentId,
status: row.status,
issueId: issueIdFromWakePayload(row.payload),
})),
blockers: blockerRows,
pendingInteractions: interactionRows,
pendingApprovals: approvalRows,
evaluatedAt,
firstRunGraceMs: TASK_WATCHDOG_FIRST_RUN_GRACE_MS,
completedRunIssueIds,
} satisfies TaskWatchdogClassifierInput;
}
// Returns the subset of `issueIds` that already have at least one run in a
// terminal status. Such issues have demonstrably executed, so a stopped
// subtree is genuine and must not be masked by the pending-first-run guard.
async function collectCompletedRunIssueIds(companyId: string | string[], issueIds: string[], reader: Pick<Db, "select" | "execute"> = db,
issueCompanyIds?: ReadonlyMap<string, string>) {
if (issueIds.length === 0) return [];
const candidates = new Set(issueIds);
const [contextRuns, executionRuns] = await Promise.all([
reader
.select({ companyId: heartbeatRuns.companyId, contextSnapshot: heartbeatRuns.contextSnapshot })
.from(heartbeatRuns)
.where(and(
inArray(heartbeatRuns.companyId, typeof companyId === "string" ? [companyId] : companyId),
inArray(heartbeatRuns.status, [...TASK_WATCHDOG_TERMINAL_RUN_STATUSES]),
or(
inArray(sql`${heartbeatRuns.contextSnapshot}->>'issueId'`, issueIds),
inArray(sql`${heartbeatRuns.contextSnapshot}->>'taskId'`, issueIds),
),
)),
reader
.select({ issueId: issues.id })
.from(issues)
.innerJoin(heartbeatRuns, and(eq(issues.executionRunId, heartbeatRuns.id), eq(heartbeatRuns.companyId, issues.companyId)))
.where(and(
inArray(issues.companyId, typeof companyId === "string" ? [companyId] : companyId),
inArray(issues.id, issueIds),
inArray(heartbeatRuns.status, [...TASK_WATCHDOG_TERMINAL_RUN_STATUSES]),
)),
]);
const completed = new Set<string>();
for (const row of contextRuns) {
const issueId = issueIdFromRunContext(row.contextSnapshot);
if (issueId && candidates.has(issueId) && (!issueCompanyIds || issueCompanyIds.get(issueId) === row.companyId)) completed.add(issueId);
}
for (const row of executionRuns) {
completed.add(row.issueId);
}
return [...completed];
}
async function findTaskWatchdogIssue(companyId: string, watchedIssueId: string) {
return db
.select()
.from(issues)
.where(and(
eq(issues.companyId, companyId),
eq(issues.originKind, TASK_WATCHDOG_ORIGIN_KIND),
eq(issues.originId, watchedIssueId),
visibleIssueCondition(),
))
.orderBy(asc(issues.createdAt), asc(issues.id))
.limit(1)
.then((rows) => rows[0] ?? null);
}
async function hasLivePathForIssue(companyId: string, issueId: string) {
const [run, issueRun, wake] = await Promise.all([
db
.select({ id: heartbeatRuns.id })
.from(heartbeatRuns)
.where(and(
eq(heartbeatRuns.companyId, companyId),
inArray(heartbeatRuns.status, [...TASK_WATCHDOG_LIVE_RUN_STATUSES]),
sql`(${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}
OR ${heartbeatRuns.contextSnapshot}->>'taskId' = ${issueId})`,
))
.limit(1)
.then((rows) => rows[0] ?? null),
db
.select({ id: heartbeatRuns.id })
.from(issues)
.innerJoin(heartbeatRuns, eq(issues.executionRunId, heartbeatRuns.id))
.where(and(
eq(issues.companyId, companyId),
eq(issues.id, issueId),
inArray(heartbeatRuns.status, [...TASK_WATCHDOG_LIVE_RUN_STATUSES]),
))
.limit(1)
.then((rows) => rows[0] ?? null),
db
.select({ id: agentWakeupRequests.id })
.from(agentWakeupRequests)
.where(and(
eq(agentWakeupRequests.companyId, companyId),
inArray(agentWakeupRequests.status, [...TASK_WATCHDOG_WAKE_REQUEST_STATUSES]),
sql`(${agentWakeupRequests.payload}->>'issueId' = ${issueId}
OR ${agentWakeupRequests.payload}->>'taskId' = ${issueId}
OR ${agentWakeupRequests.payload}->'_paperclipWakeContext'->>'issueId' = ${issueId}
OR ${agentWakeupRequests.payload}->'_paperclipWakeContext'->>'taskId' = ${issueId})`,
))
.limit(1)
.then((rows) => rows[0] ?? null),
]);
return Boolean(run || issueRun || wake);
}
async function sameFingerprintWatchdogReviewIsStillOpen(
watchdogIssue: IssueRow | null,
stopFingerprint: string,
) {
if (!watchdogIssue) return false;
if (watchdogIssue.originFingerprint !== stopFingerprint) return false;
if (isTerminalIssueStatus(watchdogIssue.status) || watchdogIssue.status === "backlog") return false;
if (watchdogIssue.status === "in_review") {
const hasPendingReviewPath = await watchdogIssueHasPendingReviewPath(watchdogIssue.companyId, watchdogIssue.id);
return isWatchdogReviewDisposition(watchdogIssue, hasPendingReviewPath);
}
return true;
}
async function watchdogIssueNeedsFreshWake(watchdogIssue: IssueRow) {
if (watchdogIssue.status !== "in_review") return false;
const hasPendingReviewPath = await watchdogIssueHasPendingReviewPath(watchdogIssue.companyId, watchdogIssue.id);
return !isWatchdogReviewDisposition(watchdogIssue, hasPendingReviewPath);
}
async function watchdogIssueHasPendingReviewPath(companyId: string, issueId: string) {
const [interaction, approval] = await Promise.all([
db
.select({ id: issueThreadInteractions.id })
.from(issueThreadInteractions)
.where(and(
eq(issueThreadInteractions.companyId, companyId),
eq(issueThreadInteractions.issueId, issueId),
eq(issueThreadInteractions.status, "pending"),
))
.limit(1)
.then((rows) => rows[0] ?? null),
db
.select({ id: approvals.id })
.from(issueApprovals)
.innerJoin(approvals, eq(issueApprovals.approvalId, approvals.id))
.where(and(
eq(issueApprovals.companyId, companyId),
eq(issueApprovals.issueId, issueId),
inArray(approvals.status, ["pending", "revision_requested"]),
))
.limit(1)
.then((rows) => rows[0] ?? null),
]);
return Boolean(interaction || approval);
}
async function markTerminalWatchdogIssueReviewed(watchdog: IssueWatchdogRow, opts: { runId?: string | null } = {}) {
if (!watchdog.watchdogIssueId || !watchdog.lastObservedFingerprint) return watchdog;
const watchdogIssue = await db
.select()
.from(issues)
.where(and(eq(issues.companyId, watchdog.companyId), eq(issues.id, watchdog.watchdogIssueId)))
.then((rows) => rows[0] ?? null);
if (!watchdogIssue) return watchdog;
const hasPendingReviewPath = watchdogIssue.status === "in_review"
? await watchdogIssueHasPendingReviewPath(watchdog.companyId, watchdogIssue.id)
: false;
if (!isWatchdogReviewDisposition(watchdogIssue, hasPendingReviewPath)) return watchdog;
const reviewedFingerprint = reviewedFingerprintForWatchdogIssue(watchdogIssue);
if (!reviewedFingerprint) return watchdog;
const observedSnapshot = parseStopSnapshot(watchdog.lastObservedStopSnapshot);
const reviewedStopSnapshot = observedSnapshot?.fingerprint === reviewedFingerprint
? observedSnapshot
: null;
if (
watchdog.lastReviewedFingerprint === reviewedFingerprint &&
canonicalJson(parseStopSnapshot(watchdog.lastReviewedStopSnapshot)) === canonicalJson(reviewedStopSnapshot)
) return watchdog;
const [updated] = await db
.update(issueWatchdogs)
.set({
lastReviewedFingerprint: reviewedFingerprint,
lastReviewedStopSnapshot: reviewedStopSnapshot,
lastCompletedAt: new Date(),
updatedAt: new Date(),
})
.where(eq(issueWatchdogs.id, watchdog.id))
.returning();
await logActivity(db, {
companyId: watchdog.companyId,
actorType: "system",
actorId: "system",
agentId: watchdog.watchdogAgentId,
runId: opts.runId ?? null,
action: "issue.task_watchdog_fingerprint_reviewed",
entityType: "issue",
entityId: watchdog.issueId,
details: {
source: "task_watchdogs.review_disposition",
watchdogId: watchdog.id,
watchdogIssueId: watchdogIssue.id,
reviewedFingerprint,
lastObservedFingerprint: watchdog.lastObservedFingerprint,
reviewedStopSnapshot,
watchdogIssueStatus: watchdogIssue.status,
},
});
return updated ?? watchdog;
}
async function ensureReusableWatchdogIssue(input: {
watchdog: IssueWatchdogRow;
sourceIssue: IssueRow;
classification: Extract<TaskWatchdogClassifierResult, { state: "stopped" }>;
runId?: string | null;
}) {
const existing = input.watchdog.watchdogIssueId
? await db
.select()
.from(issues)
.where(and(
eq(issues.companyId, input.watchdog.companyId),
eq(issues.id, input.watchdog.watchdogIssueId),
visibleIssueCondition(),
))
.then((rows) => rows[0] ?? null)
: null;
const fallback = existing ?? await findTaskWatchdogIssue(input.watchdog.companyId, input.sourceIssue.id);
if (fallback) {
const shouldReopen = isTerminalIssueStatus(fallback.status) ||
fallback.status === "backlog" ||
await watchdogIssueNeedsFreshWake(fallback);
const watchdogIssue = shouldReopen
? await issuesSvc.update(fallback.id, {
status: "todo",
assigneeAgentId: input.watchdog.watchdogAgentId,
parentId: input.sourceIssue.id,
parentDepthPolicy: "allow_nested",
projectId: input.sourceIssue.projectId,
goalId: input.sourceIssue.goalId,
billingCode: input.sourceIssue.billingCode,
originFingerprint: input.classification.stopFingerprint,
}) ?? fallback
: fallback;
if (!shouldReopen && watchdogIssue.originFingerprint !== input.classification.stopFingerprint) {
await db.transaction(async tx => {
await notifyStatusCardIssueChange(tx, input.watchdog.companyId);
await tx
.update(issues)
.set({ originFingerprint: input.classification.stopFingerprint, updatedAt: new Date() })
.where(and(eq(issues.companyId, input.watchdog.companyId), eq(issues.id, watchdogIssue.id)));
});
watchdogIssue.originFingerprint = input.classification.stopFingerprint;
}
await issuesSvc.addComment(
watchdogIssue.id,
buildStoppedFingerprintComment({
sourceIssue: input.sourceIssue,
stopFingerprint: input.classification.stopFingerprint,
stoppedLeaves: input.classification.stoppedLeaves,
pendingInteractionsByIssueId: input.classification.pendingInteractionsByIssueId,
resumed: true,
}),
{ runId: input.runId ?? null },
{
authorType: "system",
metadata: stoppedFingerprintMetadata({
sourceIssueId: input.sourceIssue.id,
stopFingerprint: input.classification.stopFingerprint,
waitsByIssueId: input.classification.stopSnapshot.waitsByIssueId,
resumed: true,
}),
},
);
return watchdogIssue;
}
const created = await issuesSvc.create(input.sourceIssue.companyId, {
title: `Watchdog review for ${input.sourceIssue.identifier ?? input.sourceIssue.title}`,
description: [
"Task watchdog review issue.",
"",
`Watched issue: ${input.sourceIssue.identifier ?? input.sourceIssue.id}`,
`Stopped fingerprint: ${input.classification.stopFingerprint}`,
"",
"The watchdog agent should verify the stopped subtree and either confirm the disposition or restore a valid live path.",
].join("\n"),
status: "todo",
priority: input.sourceIssue.priority,
parentId: input.sourceIssue.id,
// Watchdog review issues and their serialized follow-ups are located by
// parent; they are system machinery, not agent-authored decomposition.
parentDepthPolicy: "allow_nested",
projectId: input.sourceIssue.projectId,
goalId: input.sourceIssue.goalId,
assigneeAgentId: input.watchdog.watchdogAgentId,
originKind: TASK_WATCHDOG_ORIGIN_KIND,
originId: input.sourceIssue.id,
originFingerprint: input.classification.stopFingerprint,
billingCode: input.sourceIssue.billingCode,
inheritExecutionWorkspaceFromIssueId: input.sourceIssue.id,
})
.catch(async (error: unknown) => {
if (!isActiveTaskWatchdogUniqueConflict(error)) throw error;
const winner = await findTaskWatchdogIssue(input.watchdog.companyId, input.sourceIssue.id);
if (!winner) throw error;
return winner;
});
await issuesSvc.addComment(
created.id,
buildStoppedFingerprintComment({
sourceIssue: input.sourceIssue,
stopFingerprint: input.classification.stopFingerprint,
stoppedLeaves: input.classification.stoppedLeaves,
pendingInteractionsByIssueId: input.classification.pendingInteractionsByIssueId,
resumed: false,
}),
{ runId: input.runId ?? null },
{
authorType: "system",
metadata: stoppedFingerprintMetadata({
sourceIssueId: input.sourceIssue.id,
stopFingerprint: input.classification.stopFingerprint,
waitsByIssueId: input.classification.stopSnapshot.waitsByIssueId,
resumed: false,
}),
},
);
return created;
}
async function evaluateWatchdog(row: IssueWatchdogRow, opts: { runId?: string | null; onDeferredDeadline?: (at: number) => void } = {}) {
const watchdog = await markTerminalWatchdogIssueReviewed(row, opts);
const sourceIssue = await db
.select()
.from(issues)
.where(and(eq(issues.companyId, watchdog.companyId), eq(issues.id, watchdog.issueId), visibleIssueCondition()))
.then((rows) => rows[0] ?? null);
if (!sourceIssue || sourceIssue.originKind === TASK_WATCHDOG_ORIGIN_KIND) {
return { state: "skipped" as const, reason: "watched_issue_not_applicable" };
}
const input = await collectClassifierInput(watchdog.companyId, watchdog);
const classification = classifyTaskWatchdogSubtree(input);
if (classification.state === "pending_first_run") {
const pending = new Set(classification.pendingIssueIds);
for (const issue of input.issues) if (pending.has(issue.id)) {
const createdAt = toEpochMs(issue.createdAt);
if (createdAt !== null) opts.onDeferredDeadline?.(createdAt + TASK_WATCHDOG_FIRST_RUN_GRACE_MS);
}
}
if (classification.state !== "stopped") {
return { state: classification.state, reason: classification.reason, classification };
}
const existingWatchdogIssueId = watchdog.watchdogIssueId ?? (await findTaskWatchdogIssue(
watchdog.companyId,
sourceIssue.id,
))?.id ?? null;
if (existingWatchdogIssueId && await hasLivePathForIssue(watchdog.companyId, existingWatchdogIssueId)) {
await db
.update(issueWatchdogs)
.set({
watchdogIssueId: existingWatchdogIssueId,
lastObservedFingerprint: classification.stopFingerprint,
lastObservedStopSnapshot: classification.stopSnapshot,
updatedAt: new Date(),
})
.where(eq(issueWatchdogs.id, watchdog.id));
return { state: "watchdog_live" as const, classification, watchdogIssueId: existingWatchdogIssueId };
}
const existingWatchdogIssue = existingWatchdogIssueId
? await db
.select()
.from(issues)
.where(and(
eq(issues.companyId, watchdog.companyId),
eq(issues.id, existingWatchdogIssueId),
visibleIssueCondition(),
))
.then((rows) => rows[0] ?? null)
: null;
if (await sameFingerprintWatchdogReviewIsStillOpen(existingWatchdogIssue, classification.stopFingerprint)) {
if (
watchdog.watchdogIssueId !== existingWatchdogIssue!.id ||
watchdog.lastObservedFingerprint !== classification.stopFingerprint ||
canonicalJson(parseStopSnapshot(watchdog.lastObservedStopSnapshot)) !== canonicalJson(classification.stopSnapshot)
) {
await db
.update(issueWatchdogs)
.set({
watchdogIssueId: existingWatchdogIssue!.id,
lastObservedFingerprint: classification.stopFingerprint,
lastObservedStopSnapshot: classification.stopSnapshot,
updatedAt: new Date(),
})
.where(eq(issueWatchdogs.id, watchdog.id));
}
return {
state: "watchdog_review_open" as const,
classification,
watchdogIssueId: existingWatchdogIssue!.id,
};
}
const watchdogIssue = await ensureReusableWatchdogIssue({
watchdog,
sourceIssue,
classification,
runId: opts.runId ?? null,
});
const now = new Date();
await db
.update(issueWatchdogs)
.set({
watchdogIssueId: watchdogIssue.id,
lastObservedFingerprint: classification.stopFingerprint,
lastObservedStopSnapshot: classification.stopSnapshot,
lastTriggeredAt: now,
triggerCount: sql`${issueWatchdogs.triggerCount} + 1`,
updatedAt: now,
})
.where(eq(issueWatchdogs.id, watchdog.id));
await logActivity(db, {
companyId: sourceIssue.companyId,
actorType: "system",
actorId: "system",
agentId: watchdog.watchdogAgentId,
runId: opts.runId ?? null,
action: "issue.task_watchdog_triggered",
entityType: "issue",
entityId: sourceIssue.id,
details: {
source: "task_watchdogs.evaluate",
watchdogId: watchdog.id,
watchdogIssueId: watchdogIssue.id,
stopFingerprint: classification.stopFingerprint,
stopSnapshot: classification.stopSnapshot,
stoppedLeaves: classification.stoppedLeaves,
},
});
const context = watchdogWakeContext({
watchdog,
watchdogIssue,
sourceIssue,
classification,
});
const wake = deps.enqueueWakeup
? await deps.enqueueWakeup(watchdog.watchdogAgentId, {
source: "automation",
triggerDetail: "system",
reason: "task_watchdog_stopped_subtree",
payload: context,
contextSnapshot: context,
idempotencyKey: taskWatchdogWakeIdempotencyKey(watchdog.id, classification.stopFingerprint),
requestedByActorType: "system",
requestedByActorId: null,
})
: null;
return {
state: "triggered" as const,
classification,
watchdogIssueId: watchdogIssue.id,
wakeupRunId: wake?.id ?? null,
};
}
async function listActiveWatchdogsForCompany(companyId?: string | null, watchdogId?: string) {
return db
.select()
.from(issueWatchdogs)
.where(and(
eq(issueWatchdogs.status, "active"),
...(companyId ? [eq(issueWatchdogs.companyId, companyId)] : []),
watchdogId ? eq(issueWatchdogs.id, watchdogId) : undefined,
));
}
async function activeWatchdogsForIssueAndAncestors(companyId: string, issueId: string) {
const ancestorRows = await db.execute(sql`
WITH RECURSIVE ancestors(id, parent_id, depth) AS (
SELECT id, parent_id, 0
FROM issues
WHERE company_id = ${companyId}
AND id = ${issueId}
AND hidden_at IS NULL
AND harness_kind IS NULL
UNION ALL
SELECT parent.id, parent.parent_id, ancestors.depth + 1
FROM issues parent
JOIN ancestors ON parent.id = ancestors.parent_id
WHERE parent.company_id = ${companyId}
AND parent.hidden_at IS NULL
AND parent.harness_kind IS NULL
AND ancestors.depth < ${TASK_WATCHDOG_SUBTREE_MAX_DEPTH - 1}
)
SELECT id FROM ancestors
`);
const ancestorIds = (Array.isArray(ancestorRows) ? ancestorRows : [])
.map((row) => typeof row === "object" && row !== null ? (row as Record<string, unknown>).id : null)
.filter((id): id is string => typeof id === "string");
if (ancestorIds.length === 0) return [];
return db
.select()
.from(issueWatchdogs)
.where(and(
eq(issueWatchdogs.companyId, companyId),
eq(issueWatchdogs.status, "active"),
inArray(issueWatchdogs.issueId, ancestorIds),
));
}
async function revalidateMutationScope(scope: {
kind: "watchdog";
watchdogId: string;
companyId: string;
watchedIssueId: string;
stopFingerprint: string | null;
}) {
if (!scope.stopFingerprint) {
return {
allowed: false as const,
reason: "Task-watchdog run context is missing the stopped fingerprint required for mutation revalidation.",
};
}
const watchdog = await db
.select()
.from(issueWatchdogs)
.where(and(
eq(issueWatchdogs.id, scope.watchdogId),
eq(issueWatchdogs.companyId, scope.companyId),
eq(issueWatchdogs.issueId, scope.watchedIssueId),
eq(issueWatchdogs.status, "active"),
))
.then((rows) => rows[0] ?? null);
if (!watchdog) {
return {
allowed: false as const,
reason: "Task-watchdog run context is not backed by an active persisted watchdog.",
};
}
const input = await collectClassifierInput(watchdog.companyId, watchdog);
const classification = classifyTaskWatchdogSubtree(input);
if (classification.state === "stopped" && classification.stopFingerprint === scope.stopFingerprint) {
return { allowed: true as const, classification };
}
return {
allowed: false as const,
reason: classification.state === "stopped"
? "Task-watchdog review is stale because the watched subtree stop fingerprint changed; refresh the source state before mutating it."
: classification.state === "incomplete"
? "Task-watchdog review cannot authorize mutations because the watched subtree scan is incomplete."
: "Task-watchdog review is stale because the watched subtree now has a live, waiting, already-reviewed, or not-applicable path; refresh the source state before mutating it.",
classification,
};
}
return {
readIdleSchedule: async (tx: Pick<Db, "select" | "execute">, now: number,
owner: () => DeadlineReconciliationSnapshot, checkBudget: () => void = () => {}): Promise<IdleDeadlineSnapshot> => {
const before = owner();
const state: unknown[] = [before];
const blocked = (): IdleDeadlineSnapshot => ({ blocked: true, state, deadlines: [] });
if (!before.reconciled) return blocked();
const rows = await tx.select().from(issueWatchdogs).orderBy(asc(issueWatchdogs.id)).limit(idleDeadlineRowLimit + 1);
state.push(rows);
if (rows.length > idleDeadlineRowLimit) return blocked();
const deadlines: number[] = [];
if (rows.some(row => row.status !== "active" && row.status !== "disabled")) return blocked();
const active = rows.filter(row => row.status === "active");
if (active.length) {
checkBudget();
const first = active[0];
const subtrees = await loadWatchdogSubtreeIssues(first.companyId, first.issueId, tx, active);
if (subtrees.issues.length > idleDeadlineRowLimit || subtrees.incomplete) return blocked();
const uniqueIssues = [...new Map(subtrees.issues.map(issue => [issue.id, issue])).values()];
const shared = await collectClassifierInput(first.companyId, first, tx, new Date(now),
{ issues: uniqueIssues, incomplete: false }, [...new Set(active.map(row => row.companyId))]);
const { watchdog: _watchdog, evaluatedAt: _evaluatedAt, ...sharedFacts } = shared;
state.push(sharedFacts);
for (const row of active) {
checkBudget();
const includedIds = new Set(subtrees.issues.filter(issue => issue.watchdogId === row.id).map(issue => issue.id));
const input = { ...shared, watchdog: { ...summarizeIssueWatchdog(row),
lastReviewedStopSnapshot: parseStopSnapshot(row.lastReviewedStopSnapshot) },
issues: shared.issues.filter(issue => includedIds.has(issue.id)) };
state.push({ watchdog: input.watchdog, issueIds: [...includedIds] });
const classification = classifyTaskWatchdogSubtree(input);
if (classification.state === "incomplete" || classification.state === "stopped") return blocked();
if (classification.state === "pending_first_run") {
for (const issue of input.issues) if (classification.pendingIssueIds.includes(issue.id)) {
const created = toEpochMs(issue.createdAt);
if (created === null || created + TASK_WATCHDOG_FIRST_RUN_GRACE_MS <= now) return blocked();
deadlines.push(created + TASK_WATCHDOG_FIRST_RUN_GRACE_MS);
}
}
}
}
const after = owner();
if (!after.reconciled || before.revision !== after.revision) return blocked();
return { blocked: false, state, deadlines };
},
getActiveForIssue: async (companyId: string, issueId: string): Promise<IssueWatchdog | null> => {
const row = await db
.select()
.from(issueWatchdogs)
.where(and(
eq(issueWatchdogs.companyId, companyId),
eq(issueWatchdogs.issueId, issueId),
eq(issueWatchdogs.status, "active"),
))
.then((rows) => rows[0] ?? null);
return row ? toIssueWatchdog(row) : null;
},
listActiveSummariesForIssues: async (
companyId: string,
issueIds: string[],
dbOrTx: any = db,
): Promise<Map<string, IssueWatchdogSummary>> => {
if (issueIds.length === 0) return new Map();
const rows = await dbOrTx
.select()
.from(issueWatchdogs)
.where(and(
eq(issueWatchdogs.companyId, companyId),
inArray(issueWatchdogs.issueId, [...new Set(issueIds)]),
eq(issueWatchdogs.status, "active"),
));
return new Map(rows.map((row: IssueWatchdogRow) => [row.issueId, summarizeIssueWatchdog(row)]));
},
upsertForIssue: async (
companyId: string,
issueId: string,
input: IssueWatchdogUpsertInput,
): Promise<{ watchdog: IssueWatchdog; created: boolean }> => {
return upsertIssueWatchdogForIssue(db, companyId, issueId, input);
},
disableForIssue: async (
companyId: string,
issueId: string,
actor: ActorFields = {},
): Promise<IssueWatchdog | null> => {
return db.transaction(async tx => {
await assertWatchedIssue(tx, companyId, issueId);
const existing = await tx
.select()
.from(issueWatchdogs)
.where(and(eq(issueWatchdogs.companyId, companyId), eq(issueWatchdogs.issueId, issueId)))
.then((rows) => rows[0] ?? null);
if (!existing || existing.status === "disabled") return null;
await notifyIssueRecovery(tx, { companyId });
const [updated] = await tx
.update(issueWatchdogs)
.set({
status: "disabled",
updatedByAgentId: actor.agentId ?? null,
updatedByUserId: actor.userId ?? null,
updatedByRunId: actor.runId ?? null,
updatedAt: new Date(),
})
.where(eq(issueWatchdogs.id, existing.id))
.returning();
return toIssueWatchdog(updated);
});
},
reconcileTaskWatchdogs: async (opts: {
companyId?: string | null;
runId?: string | null;
watchdogId?: string;
onDeferredDeadline?: (at: number) => void;
issueCreatedAtGte?: Date | null;
} = {}) => {
let rows = await listActiveWatchdogsForCompany(opts.companyId ?? null, opts.watchdogId);
if (opts.issueCreatedAtGte) {
const watchdogIssueIds = [...new Set(rows.map((row) => row.issueId))];
const eligibleIssueIds = new Set(
watchdogIssueIds.length === 0
? []
: (await db
.select({ id: issues.id })
.from(issues)
.where(and(
inArray(issues.id, watchdogIssueIds),
gte(issues.createdAt, opts.issueCreatedAtGte),
)))
.map((issue) => issue.id),
);
rows = rows.filter((row) => eligibleIssueIds.has(row.issueId));
}
const result = {
checked: 0,
triggered: 0,
live: 0,
pendingFirstRun: 0,
alreadyReviewed: 0,
incomplete: 0,
incompleteIssueIds: [] as string[],
skipped: 0,
watchdogIssueIds: [] as string[],
};
for (const row of rows) {
result.checked += 1;
const evaluated = await evaluateWatchdog(row, { runId: opts.runId ?? null, onDeferredDeadline: opts.onDeferredDeadline });
if (evaluated.state === "triggered") {
result.triggered += 1;
result.watchdogIssueIds.push(evaluated.watchdogIssueId);
} else if (
evaluated.state === "live" ||
evaluated.state === "watchdog_live" ||
evaluated.state === "watchdog_review_open"
) {
result.live += 1;
} else if (evaluated.state === "pending_first_run") {
result.pendingFirstRun += 1;
} else if (evaluated.state === "already_reviewed") {
result.alreadyReviewed += 1;
} else if (evaluated.state === "incomplete") {
result.incomplete += 1;
result.incompleteIssueIds.push(row.issueId);
} else {
result.skipped += 1;
}
}
return result;
},
reconcileForIssueAndAncestors: async (
companyId: string,
issueId: string,
opts: { runId?: string | null } = {},
) => {
const rows = await activeWatchdogsForIssueAndAncestors(companyId, issueId);
const result = {
checked: 0,
triggered: 0,
pendingFirstRun: 0,
incomplete: 0,
incompleteIssueIds: [] as string[],
skipped: 0,
watchdogIssueIds: [] as string[],
};
for (const row of rows) {
result.checked += 1;
const evaluated = await evaluateWatchdog(row, { runId: opts.runId ?? null });
if (evaluated.state === "triggered") {
result.triggered += 1;
result.watchdogIssueIds.push(evaluated.watchdogIssueId);
} else if (evaluated.state === "pending_first_run") {
result.pendingFirstRun += 1;
} else if (evaluated.state === "incomplete") {
result.incomplete += 1;
result.incompleteIssueIds.push(row.issueId);
} else if (
evaluated.state === "watchdog_review_open" ||
evaluated.state === "watchdog_live" ||
evaluated.state === "live"
) {
// Existing review work is already open for this stopped state.
} else {
result.skipped += 1;
}
}
return result;
},
revalidateMutationScope,
};
}