mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-28 20:21:50 +01:00
fix(task-watchdog): rearm conditional review acceptances
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
aed4478c81
commit
1f9f99d2c6
16 files changed
+573
-16
No files matched your search
@@ -171,6 +171,11 @@ Blocked issues should stay idle while blockers remain unresolved. Paperclip shou
|
||||
|
||||
`cancelled` is terminal for the blocker issue itself, but it does not satisfy the dependency. A cancelled blocker edge remains unresolved until the edge is removed or replaced, and Paperclip must surface blocker attention on the dependent regardless of whether that dependent is currently displayed as `blocked`, `todo`, `backlog`, or another non-terminal agent-owned status.
|
||||
|
||||
Blocker hygiene is a write-time and transition-time contract, not a rendering hint:
|
||||
|
||||
- **Write-time:** setting an issue to `blocked` requires at least one unresolved blocker edge or a first-class external owner/action recorded on the issue. A bare `blocked` with neither is invalid.
|
||||
- **Transition-time:** any transition that leaves a `blocked` issue with zero unresolved blocker edges must queue the idempotent `issue_blockers_resolved` wake for that issue.
|
||||
|
||||
If a parent is truly waiting on a child, model that with blockers. Do not rely on the parent/child relationship alone.
|
||||
|
||||
## 7. Accepted-Plan Decomposition
|
||||
@@ -241,6 +246,17 @@ Every later run that encounters the same accepted-plan fingerprint must consult
|
||||
|
||||
Concurrent accepted-plan runs are therefore idempotent relative to the fingerprint. Creating multiple child trees for the same `(sourceIssueId, acceptedPlanRevisionId)` pair is a product bug.
|
||||
|
||||
### Plan shape: terminal gates, not phase funnels
|
||||
|
||||
Decomposition shape determines how much of the tree one slow leaf can freeze.
|
||||
|
||||
- **Umbrellas block on their terminal gate, not on every phase.** The umbrella issue should carry a blocker edge to its final acceptance gate (the QA sign-off or review rollup issue), not edges to all N phases. Per-phase dependencies belong between the phases themselves; duplicating them onto the umbrella adds edges that delay the umbrella's wake without encoding any additional dependency.
|
||||
- **Avoid single-leaf funnels.** A shape where every open issue transitively waits on one leaf means that leaf's internal recovery latency stalls the entire tree, and later-phase owners can neither start nor observe progress. Keep independent phases on independent branches with their own gates so they run and report in parallel; serialize only where a real dependency exists.
|
||||
|
||||
Held against the three coordination invariants: productive work continues (independent branches proceed while one leaf recovers); only real blockers stop work (the umbrella and each phase carry only edges that encode true dependencies); no infinite loops (fewer, more truthful edges means fewer resolution wakes, with wake idempotency unchanged).
|
||||
|
||||
The same guidance is mirrored in the `paperclip-converting-plans-to-tasks` skill, which is the operational surface planners actually read at decomposition time.
|
||||
|
||||
## 8. Non-Terminal Issue Liveness Contract
|
||||
|
||||
For agent-owned, non-terminal issues, Paperclip should never leave work in a state where nobody is responsible for the next move and nothing will wake or surface it.
|
||||
@@ -451,12 +467,16 @@ Because `serviceName` and `notes` remain visible in issue activity and wake cont
|
||||
|
||||
Monitor bounds are enforced. Paperclip rejects attempts to re-arm a monitor whose `timeoutAt` or `maxAttempts` is already exhausted. When a scheduled monitor reaches an exhausted bound at trigger time, Paperclip clears it and follows `recoveryPolicy`: `wake_owner` queues a bounded recovery wake for the assignee, `create_recovery_issue` opens visible issue-backed recovery work, and `escalate_to_board` records a board-visible escalation comment/activity.
|
||||
|
||||
A monitor may also be armed through a delegated monitor/wake proposal from a run actor that does not own the issue (see "Cross-issue coordination", section 6). The proposal is consumed under the target owner's policy and counts against the same `timeoutAt`/`maxAttempts` bounds as a self-armed monitor. This replaces the disposable "monitor X" polling-issue chain, which is a relay-issue anti-pattern: the check belongs on the watched issue's own monitor, not on a stream of one-shot checker issues.
|
||||
|
||||
Use `blocked` instead of a monitor when no Paperclip assignee owns a responsible polling path. In that case, name the external owner/action or create first-class recovery/blocker work.
|
||||
|
||||
### `blocked`
|
||||
|
||||
This is explicit waiting state.
|
||||
|
||||
`blocked` with zero unresolved blocker edges and no first-class external owner/action is invalid at write time, and any transition that clears an issue's last unresolved edge must queue the `issue_blockers_resolved` wake (section 6). A blocked issue that reaches zero unresolved edges without waking is a product bug, not a cosmetic gap: nothing else will ever pick it up. The stalled-`blocked` classification below is likewise a contract the product must surface through recovery and visibility paths, not just a documented category.
|
||||
|
||||
A healthy `blocked` issue has an explicit waiting path:
|
||||
|
||||
- first-class blockers exist, and each unresolved leaf has a valid action path under this contract
|
||||
@@ -565,7 +585,7 @@ The reusable watchdog issue is a child of the watched source issue for audit and
|
||||
|
||||
### Stopped-subtree evaluation
|
||||
|
||||
Task watchdog evaluation is conservative. If any included issue has a live run, queued wake, or scheduled retry that should fire without intervention, the subtree is live and the task watchdog does not run.
|
||||
Task watchdog evaluation is conservative. If any included issue has a live run, queued wake, scheduled retry, or scheduled issue monitor that should fire without intervention, the subtree is live and the task watchdog does not run. A future `monitorNextCheckAt` on an included issue is a live/waiting path for this classifier: the monitor owner holds a bounded continuation, so the subtree is not stopped. Because monitor state is a fingerprint input, consuming, clearing, or losing that monitor changes durable subtree state and re-enters evaluation.
|
||||
|
||||
If no included issue has a live path, Paperclip computes a stop fingerprint from durable subtree state, including at least:
|
||||
|
||||
@@ -576,7 +596,7 @@ If no included issue has a live path, Paperclip computes a stop fingerprint from
|
||||
- terminal or cancelled leaf evidence
|
||||
- the watchdog configuration revision, including watchdog agent and instructions changes
|
||||
|
||||
If the fingerprint equals the watchdog's last reviewed fingerprint, Paperclip suppresses another watchdog wake. If the fingerprint is new, Paperclip creates or reopens the reusable watchdog issue and wakes the configured watchdog agent with the source issue, watchdog config, stop fingerprint, leaf summary, default mandate, custom instructions, and server-derived capability metadata that names the allowed operations, denied operations, reusable watchdog issue, and non-watchdog target scope.
|
||||
If the fingerprint equals the watchdog's last reviewed fingerprint and that review's acceptance is still in force (see "Terminal and conditional acceptance" below), Paperclip suppresses another watchdog wake. If the fingerprint is new, or the prior acceptance was conditional and has expired, Paperclip creates or reopens the reusable watchdog issue and wakes the configured watchdog agent with the source issue, watchdog config, stop fingerprint, leaf summary, default mandate, custom instructions, and server-derived capability metadata that names the allowed operations, denied operations, reusable watchdog issue, and non-watchdog target scope.
|
||||
|
||||
Changing the watchdog agent or custom instructions invalidates the reviewed fingerprint and forces a fresh evaluation even if the subtree state did not otherwise change.
|
||||
|
||||
@@ -607,6 +627,21 @@ A task watchdog may resolve only eligible `request_confirmation` plan confirmati
|
||||
|
||||
The watchdog cannot resolve `request_checkbox_confirmation`, `ask_user_questions`, `suggest_tasks`, linked approvals, execution-policy decisions unless it is the typed participant outside watchdog capacity, or document comments written as freeform approval.
|
||||
|
||||
### Terminal and conditional acceptance
|
||||
|
||||
Every watchdog review that leaves the subtree stopped records which kind of acceptance it is. The reviewed fingerprint suppresses re-wakes only as far as that acceptance reaches.
|
||||
|
||||
- **Terminal acceptance** asserts that the stopped state is a legitimate final or waiting disposition. It must name the explicit waiting path: a human owner, a pending interaction or linked approval, or a first-class blocker whose unresolved leaf is itself live or explicitly waiting under section 8. A terminal acceptance suppresses re-wake for the identical fingerprint indefinitely.
|
||||
- **Conditional acceptance** asserts that work is live or a bounded continuation is scheduled. It must name its liveness source: a run id, a monitor next-check time, or a queued wake id. It expires when that source resolves or at a bounded re-arm deadline, whichever comes first. The default re-arm deadline is 30 minutes, aligned with the silent active-run watchdog's `continue` re-arm window in section 12. After expiry, a still-stopped subtree re-wakes the watchdog even when the stop fingerprint is unchanged.
|
||||
|
||||
An acceptance that names no waiting path and no liveness source is treated as conditional with the default re-arm deadline. A verdict the server rejected as stale during the review — because the subtree turned live mid-review — must not seal the fingerprint at all; the next scan evaluates from current state.
|
||||
|
||||
### Bounded same-fingerprint reviews
|
||||
|
||||
Consecutive watchdog reviews of an identical stop fingerprint are bounded (default 3). Reaching the bound means the watchdog is cycling without changing the subtree. At the bound, the machinery must stop silently re-reviewing and instead escalate to an explicit human/board interaction on the watched source issue that presents the stop evidence and the watchdog's prior verdicts.
|
||||
|
||||
Terminal acceptances do not count toward this bound — they already suppress identical fingerprints — so legitimately waiting trees neither churn nor escalate. Re-arm wakes for an expired conditional acceptance are deduplicated per review cycle, not per fingerprint alone, so a re-arm on an unchanged fingerprint is still delivered.
|
||||
|
||||
### Completion and fingerprint updates
|
||||
|
||||
The watchdog's reviewed fingerprint should update only after the watchdog issue reaches a valid disposition:
|
||||
@@ -616,6 +651,8 @@ The watchdog's reviewed fingerprint should update only after the watchdog issue
|
||||
- `blocked` with first-class blockers or a named external owner/action
|
||||
- a watchdog mutation that restores live work, where the subsequent source-subtree mutation naturally changes the stop fingerprint
|
||||
|
||||
When the disposition leaves the subtree stopped, the review must also record whether the acceptance is terminal or conditional, and for a conditional acceptance the named liveness source and expiry, as defined above.
|
||||
|
||||
If the watchdog moved work forward, Paperclip should not mark the old fingerprint as permanently acceptable just because the watchdog issue completed. The next scan should observe the changed subtree state and either suppress because work is live or compute a new stopped fingerprint later.
|
||||
|
||||
Task watchdogs must not silently mark source work done from prose comments, must not duplicate child trees for the same accepted plan revision, and must not create another task-watchdog issue for the same source issue.
|
||||
@@ -643,6 +680,8 @@ Watchdog decisions are explicit operator/recovery-owner decisions:
|
||||
|
||||
Operators should prefer `snooze` for known time-bounded quiet periods. `continue` is only a short acknowledgement of the current evidence; if the run remains silent after the re-arm window, the periodic watchdog scan can create or update review work again.
|
||||
|
||||
The 30-minute `continue` re-arm window and the task watchdog's conditional-acceptance re-arm deadline in section 11 are the same principle: an acknowledgement that the current state is acceptable is time-bounded unless it names a legitimate terminal or waiting path. Only terminal acceptances suppress indefinitely.
|
||||
|
||||
The board can record watchdog decisions. The assigned owner of an issue-backed watchdog evaluation can also record them. Other agents cannot.
|
||||
|
||||
### Source-aware watchdog folding
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
ALTER TABLE "issue_watchdogs" ADD COLUMN IF NOT EXISTS "review_acceptance_kind" text;
|
||||
--> statement-breakpoint
|
||||
ALTER TABLE "issue_watchdogs" ADD COLUMN IF NOT EXISTS "review_expires_at" timestamp with time zone;
|
||||
--> statement-breakpoint
|
||||
ALTER TABLE "issue_watchdogs" ADD COLUMN IF NOT EXISTS "same_fingerprint_review_count" integer DEFAULT 0 NOT NULL;
|
||||
--> statement-breakpoint
|
||||
DO $$
|
||||
BEGIN
|
||||
IF NOT EXISTS (
|
||||
SELECT 1
|
||||
FROM pg_constraint
|
||||
WHERE conname = 'issue_watchdogs_review_acceptance_kind_check'
|
||||
AND conrelid = 'issue_watchdogs'::regclass
|
||||
) THEN
|
||||
ALTER TABLE "issue_watchdogs"
|
||||
ADD CONSTRAINT "issue_watchdogs_review_acceptance_kind_check"
|
||||
CHECK ("review_acceptance_kind" IS NULL OR "review_acceptance_kind" IN ('terminal', 'conditional'));
|
||||
END IF;
|
||||
END $$;
|
||||
@@ -1338,6 +1338,13 @@
|
||||
"when": 1784916886226,
|
||||
"tag": "0192_task_watchdog_stop_snapshots",
|
||||
"breakpoints": true
|
||||
},
|
||||
{
|
||||
"idx": 193,
|
||||
"version": "7",
|
||||
"when": 1785165132988,
|
||||
"tag": "0193_task_watchdog_conditional_acceptance",
|
||||
"breakpoints": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,5 +1,5 @@
|
||||
import { sql } from "drizzle-orm";
|
||||
import { index, integer, jsonb, pgTable, text, timestamp, uniqueIndex, uuid } from "drizzle-orm/pg-core";
|
||||
import { check, index, integer, jsonb, pgTable, text, timestamp, uniqueIndex, uuid } from "drizzle-orm/pg-core";
|
||||
import { agents } from "./agents.js";
|
||||
import { companies } from "./companies.js";
|
||||
import { heartbeatRuns } from "./heartbeat_runs.js";
|
||||
@@ -19,6 +19,9 @@ export const issueWatchdogs = pgTable(
|
||||
lastReviewedFingerprint: text("last_reviewed_fingerprint"),
|
||||
lastObservedStopSnapshot: jsonb("last_observed_stop_snapshot"),
|
||||
lastReviewedStopSnapshot: jsonb("last_reviewed_stop_snapshot"),
|
||||
reviewAcceptanceKind: text("review_acceptance_kind"),
|
||||
reviewExpiresAt: timestamp("review_expires_at", { withTimezone: true }),
|
||||
sameFingerprintReviewCount: integer("same_fingerprint_review_count").notNull().default(0),
|
||||
lastTriggeredAt: timestamp("last_triggered_at", { withTimezone: true }),
|
||||
lastCompletedAt: timestamp("last_completed_at", { withTimezone: true }),
|
||||
triggerCount: integer("trigger_count").notNull().default(0),
|
||||
@@ -38,5 +41,9 @@ export const issueWatchdogs = pgTable(
|
||||
companyWatchdogIssueIdx: uniqueIndex("issue_watchdogs_company_watchdog_issue_uq")
|
||||
.on(table.companyId, table.watchdogIssueId)
|
||||
.where(sql`${table.watchdogIssueId} is not null`),
|
||||
reviewAcceptanceKindCheck: check(
|
||||
"issue_watchdogs_review_acceptance_kind_check",
|
||||
sql`${table.reviewAcceptanceKind} is null or ${table.reviewAcceptanceKind} in ('terminal', 'conditional')`,
|
||||
),
|
||||
}),
|
||||
);
|
||||
@@ -846,6 +846,7 @@ export type {
|
||||
IssueProductivityReviewTrigger,
|
||||
IssueRecoveryAction,
|
||||
IssueWatchdog,
|
||||
IssueWatchdogReviewAcceptanceKind,
|
||||
IssueWatchdogStatus,
|
||||
IssueWatchdogSummary,
|
||||
SuccessfulRunHandoffState,
|
||||
|
||||
@@ -655,6 +655,7 @@ export type {
|
||||
IssueAttachment,
|
||||
IssueLabel,
|
||||
IssueWatchdog,
|
||||
IssueWatchdogReviewAcceptanceKind,
|
||||
IssueWatchdogStatus,
|
||||
IssueWatchdogSummary,
|
||||
} from "./issue.js";
|
||||
|
||||
@@ -682,6 +682,7 @@ export interface IssueExecutionDecision {
|
||||
}
|
||||
|
||||
export type IssueWatchdogStatus = "active" | "disabled";
|
||||
export type IssueWatchdogReviewAcceptanceKind = "terminal" | "conditional";
|
||||
|
||||
export interface IssueWatchdogSummary {
|
||||
id: string;
|
||||
@@ -693,6 +694,9 @@ export interface IssueWatchdogSummary {
|
||||
watchdogIssueId: string | null;
|
||||
lastObservedFingerprint: string | null;
|
||||
lastReviewedFingerprint: string | null;
|
||||
reviewAcceptanceKind: IssueWatchdogReviewAcceptanceKind | null;
|
||||
reviewExpiresAt: Date | null;
|
||||
sameFingerprintReviewCount: number;
|
||||
lastTriggeredAt: Date | null;
|
||||
lastCompletedAt: Date | null;
|
||||
triggerCount: number;
|
||||
|
||||
@@ -50,6 +50,34 @@ describe("task watchdog subtree classifier", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("treats a future issue monitor as a live path", () => {
|
||||
const result = classify({
|
||||
issues: [issue({ monitorNextCheckAt: new Date("2026-06-17T20:30:00.000Z") })],
|
||||
evaluatedAt: new Date("2026-06-17T20:00:00.000Z"),
|
||||
});
|
||||
|
||||
expect(result).toMatchObject({
|
||||
state: "live",
|
||||
liveIssueIds: [sourceId],
|
||||
});
|
||||
});
|
||||
|
||||
it("includes non-live monitor state in the stopped fingerprint", () => {
|
||||
const first = classify({
|
||||
issues: [issue({ monitorNextCheckAt: new Date("2026-06-17T19:00:00.000Z") })],
|
||||
evaluatedAt: new Date("2026-06-17T20:00:00.000Z"),
|
||||
});
|
||||
const second = classify({
|
||||
issues: [issue({ monitorNextCheckAt: new Date("2026-06-17T19:30:00.000Z") })],
|
||||
evaluatedAt: new Date("2026-06-17T20:00:00.000Z"),
|
||||
});
|
||||
|
||||
expect(first.state).toBe("stopped");
|
||||
expect(second.state).toBe("stopped");
|
||||
if (first.state !== "stopped" || second.state !== "stopped") return;
|
||||
expect(first.stopFingerprint).not.toBe(second.stopFingerprint);
|
||||
});
|
||||
|
||||
it("treats terminal and waiting leaves as stopped work that needs verification", () => {
|
||||
const result = classify({
|
||||
issues: [
|
||||
|
||||
@@ -50,6 +50,7 @@ describeEmbeddedPostgres("task watchdog scheduler", () => {
|
||||
await db.delete(issueWorkProducts);
|
||||
await db.delete(issueDocuments);
|
||||
await db.delete(documents);
|
||||
await db.delete(issueThreadInteractions);
|
||||
await db.delete(issueComments);
|
||||
await db.delete(heartbeatRuns);
|
||||
await db.delete(agentWakeupRequests);
|
||||
@@ -108,6 +109,7 @@ describeEmbeddedPostgres("task watchdog scheduler", () => {
|
||||
originId: overrides.originId,
|
||||
originFingerprint: overrides.originFingerprint,
|
||||
updatedAt: overrides.updatedAt,
|
||||
monitorNextCheckAt: overrides.monitorNextCheckAt,
|
||||
// Default to an "established" issue (created well before the first-run
|
||||
// grace window) so the pending-first-run guard does not defer it. Tests
|
||||
// exercising the create-race pass an explicit recent `createdAt`.
|
||||
@@ -357,6 +359,28 @@ describeEmbeddedPostgres("task watchdog scheduler", () => {
|
||||
expect(watchdogIssues).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("does not trigger while a watched issue has a future monitor", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const sourceId = await seedIssue(companyId, {
|
||||
identifier: "WDOG-MONITOR",
|
||||
status: "in_progress",
|
||||
monitorNextCheckAt: new Date(Date.now() + 60 * 60 * 1000),
|
||||
});
|
||||
const agentId = await seedAgent(companyId);
|
||||
await seedWatchdog(companyId, sourceId, agentId);
|
||||
const { service, wakes } = createService();
|
||||
|
||||
const result = await service.reconcileTaskWatchdogs({ companyId });
|
||||
|
||||
expect(result).toMatchObject({ checked: 1, triggered: 0, live: 1 });
|
||||
expect(wakes).toHaveLength(0);
|
||||
const watchdogIssues = await db
|
||||
.select({ id: issues.id })
|
||||
.from(issues)
|
||||
.where(and(eq(issues.companyId, companyId), eq(issues.originKind, "task_watchdog")));
|
||||
expect(watchdogIssues).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("does not keep the source live for runs under a nested task-watchdog issue", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const sourceId = await seedIssue(companyId, { identifier: "WDOG-NEST", status: "done" });
|
||||
@@ -402,6 +426,134 @@ describeEmbeddedPostgres("task watchdog scheduler", () => {
|
||||
expect(wakes).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("re-wakes an identical fingerprint after conditional acceptance expires", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const sourceId = await seedIssue(companyId, { identifier: "WDOG-REARM", status: "done" });
|
||||
const agentId = await seedAgent(companyId);
|
||||
await seedWatchdog(companyId, sourceId, agentId);
|
||||
const { service, wakes } = createService();
|
||||
|
||||
await service.reconcileTaskWatchdogs({ companyId });
|
||||
const [initial] = await db.select().from(issueWatchdogs).where(eq(issueWatchdogs.issueId, sourceId));
|
||||
await db.update(issues).set({ status: "done", updatedAt: new Date() }).where(eq(issues.id, initial!.watchdogIssueId!));
|
||||
|
||||
const accepted = await service.reconcileTaskWatchdogs({ companyId });
|
||||
expect(accepted).toMatchObject({ checked: 1, triggered: 0, alreadyReviewed: 1 });
|
||||
const [conditionallyReviewed] = await db.select().from(issueWatchdogs).where(eq(issueWatchdogs.id, initial!.id));
|
||||
expect(conditionallyReviewed).toMatchObject({
|
||||
reviewAcceptanceKind: "conditional",
|
||||
sameFingerprintReviewCount: 1,
|
||||
});
|
||||
|
||||
await db.update(issueWatchdogs).set({ reviewExpiresAt: new Date(Date.now() - 1_000) }).where(eq(issueWatchdogs.id, initial!.id));
|
||||
const rearmed = await service.reconcileTaskWatchdogs({ companyId });
|
||||
|
||||
expect(rearmed).toMatchObject({ checked: 1, triggered: 1 });
|
||||
expect(wakes).toHaveLength(2);
|
||||
expect(wakes[0]?.opts?.idempotencyKey).not.toBe(wakes[1]?.opts?.idempotencyKey);
|
||||
expect(wakes[1]?.opts?.idempotencyKey).toContain(":1");
|
||||
});
|
||||
|
||||
it("escalates instead of silently re-reviewing at the identical-fingerprint bound", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const sourceId = await seedIssue(companyId, { identifier: "WDOG-ESCALATE", status: "done" });
|
||||
const agentId = await seedAgent(companyId);
|
||||
await seedWatchdog(companyId, sourceId, agentId);
|
||||
const { service, wakes } = createService();
|
||||
|
||||
await service.reconcileTaskWatchdogs({ companyId });
|
||||
const [watchdog] = await db.select().from(issueWatchdogs).where(eq(issueWatchdogs.issueId, sourceId));
|
||||
const completedAt = new Date();
|
||||
await db.update(issues).set({ status: "done", updatedAt: new Date(completedAt.getTime() - 1_000) })
|
||||
.where(eq(issues.id, watchdog!.watchdogIssueId!));
|
||||
await db.update(issueWatchdogs).set({
|
||||
lastReviewedFingerprint: watchdog!.lastObservedFingerprint,
|
||||
reviewAcceptanceKind: "conditional",
|
||||
reviewExpiresAt: new Date(completedAt.getTime() - 1_000),
|
||||
sameFingerprintReviewCount: 3,
|
||||
lastCompletedAt: completedAt,
|
||||
}).where(eq(issueWatchdogs.id, watchdog!.id));
|
||||
|
||||
const result = await service.reconcileTaskWatchdogs({ companyId });
|
||||
|
||||
expect(result).toMatchObject({ checked: 1, triggered: 0, escalated: 1 });
|
||||
expect(wakes).toHaveLength(1);
|
||||
const interactions = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.issueId, sourceId));
|
||||
expect(interactions).toHaveLength(1);
|
||||
expect(interactions[0]).toMatchObject({ kind: "ask_user_questions", status: "pending" });
|
||||
|
||||
const repeated = await service.reconcileTaskWatchdogs({ companyId });
|
||||
expect(repeated).toMatchObject({ checked: 1, triggered: 0, escalated: 1 });
|
||||
expect(wakes).toHaveLength(1);
|
||||
const repeatedInteractions = await db.select().from(issueThreadInteractions)
|
||||
.where(eq(issueThreadInteractions.issueId, sourceId));
|
||||
expect(repeatedInteractions).toHaveLength(1);
|
||||
|
||||
await db.update(issueThreadInteractions).set({
|
||||
status: "accepted",
|
||||
result: { version: 1, outcome: "accepted" },
|
||||
resolvedAt: new Date(),
|
||||
updatedAt: new Date(),
|
||||
}).where(eq(issueThreadInteractions.id, repeatedInteractions[0]!.id));
|
||||
|
||||
const escalatedAfterResolution = await service.reconcileTaskWatchdogs({ companyId });
|
||||
|
||||
expect(escalatedAfterResolution).toMatchObject({ checked: 1, triggered: 0, escalated: 1 });
|
||||
const interactionsAfterResolution = await db.select().from(issueThreadInteractions)
|
||||
.where(eq(issueThreadInteractions.issueId, sourceId));
|
||||
expect(interactionsAfterResolution).toHaveLength(2);
|
||||
expect(interactionsAfterResolution.filter((interaction) => interaction.status === "pending")).toHaveLength(1);
|
||||
expect(wakes).toHaveLength(1);
|
||||
|
||||
await db.update(issues).set({
|
||||
monitorNextCheckAt: new Date(Date.now() + 60_000),
|
||||
updatedAt: new Date(),
|
||||
}).where(eq(issues.id, sourceId));
|
||||
|
||||
const liveAfterMonitor = await service.reconcileTaskWatchdogs({ companyId });
|
||||
|
||||
expect(liveAfterMonitor).toMatchObject({ checked: 1, live: 1, triggered: 0, escalated: 0 });
|
||||
expect(wakes).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("keeps terminal acceptance suppressed for an identical fingerprint", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const sourceId = await seedIssue(companyId, { identifier: "WDOG-TERMINAL", status: "done" });
|
||||
const agentId = await seedAgent(companyId);
|
||||
await seedWatchdog(companyId, sourceId, agentId);
|
||||
const { service, wakes } = createService();
|
||||
|
||||
await service.reconcileTaskWatchdogs({ companyId });
|
||||
const [watchdog] = await db.select().from(issueWatchdogs).where(eq(issueWatchdogs.issueId, sourceId));
|
||||
await db.insert(heartbeatRuns).values({
|
||||
companyId,
|
||||
agentId,
|
||||
status: "succeeded",
|
||||
invocationSource: "assignment",
|
||||
contextSnapshot: {
|
||||
issueId: watchdog!.watchdogIssueId!,
|
||||
taskWatchdog: {
|
||||
stopFingerprint: watchdog!.lastObservedFingerprint!,
|
||||
reviewAcceptance: { kind: "terminal" },
|
||||
},
|
||||
},
|
||||
});
|
||||
await db.update(issues).set({ status: "done", updatedAt: new Date() }).where(eq(issues.id, watchdog!.watchdogIssueId!));
|
||||
|
||||
const firstSuppressed = await service.reconcileTaskWatchdogs({ companyId });
|
||||
const secondSuppressed = await service.reconcileTaskWatchdogs({ companyId });
|
||||
|
||||
expect(firstSuppressed).toMatchObject({ triggered: 0, alreadyReviewed: 1 });
|
||||
expect(secondSuppressed).toMatchObject({ triggered: 0, alreadyReviewed: 1 });
|
||||
expect(wakes).toHaveLength(1);
|
||||
const [reviewed] = await db.select().from(issueWatchdogs).where(eq(issueWatchdogs.id, watchdog!.id));
|
||||
expect(reviewed).toMatchObject({
|
||||
reviewAcceptanceKind: "terminal",
|
||||
reviewExpiresAt: null,
|
||||
sameFingerprintReviewCount: 0,
|
||||
});
|
||||
});
|
||||
|
||||
it("marks a completed watchdog fingerprint reviewed, then reuses the same issue for a later stopped state", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const sourceId = await seedIssue(companyId, { identifier: "WDOG-3", status: "done" });
|
||||
@@ -658,6 +810,58 @@ describeEmbeddedPostgres("task watchdog scheduler", () => {
|
||||
]));
|
||||
});
|
||||
|
||||
it("does not seal a fingerprint after the server rejected that review as stale", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const sourceId = await seedIssue(companyId, { identifier: "WDOG-STALE-REVIEW", status: "blocked" });
|
||||
const agentId = await seedAgent(companyId);
|
||||
await seedWatchdog(companyId, sourceId, agentId);
|
||||
const { service, wakes } = createService();
|
||||
|
||||
await service.reconcileTaskWatchdogs({ companyId });
|
||||
const [watchdog] = await db.select().from(issueWatchdogs).where(eq(issueWatchdogs.issueId, sourceId));
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({
|
||||
id: runId,
|
||||
companyId,
|
||||
agentId,
|
||||
status: "running",
|
||||
invocationSource: "assignment",
|
||||
contextSnapshot: {
|
||||
issueId: watchdog!.watchdogIssueId!,
|
||||
taskWatchdog: {
|
||||
watchedIssueId: sourceId,
|
||||
stopFingerprint: watchdog!.lastObservedFingerprint!,
|
||||
},
|
||||
},
|
||||
});
|
||||
await db.update(issues).set({ status: "in_review", updatedAt: new Date(Date.now() + 1_000) })
|
||||
.where(eq(issues.id, sourceId));
|
||||
|
||||
const rejected = await service.revalidateMutationScope({
|
||||
kind: "watchdog",
|
||||
watchdogId: watchdog!.id,
|
||||
companyId,
|
||||
watchedIssueId: sourceId,
|
||||
stopFingerprint: watchdog!.lastObservedFingerprint!,
|
||||
runId,
|
||||
});
|
||||
expect(rejected.allowed).toBe(false);
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, runId));
|
||||
await db.update(issues).set({ status: "done", updatedAt: new Date(Date.now() + 2_000) })
|
||||
.where(eq(issues.id, watchdog!.watchdogIssueId!));
|
||||
|
||||
const result = await service.reconcileTaskWatchdogs({ companyId });
|
||||
|
||||
expect(result).toMatchObject({ checked: 1, triggered: 1 });
|
||||
expect(wakes).toHaveLength(2);
|
||||
const [updatedWatchdog] = await db.select().from(issueWatchdogs).where(eq(issueWatchdogs.id, watchdog!.id));
|
||||
expect(updatedWatchdog?.lastReviewedFingerprint).toBeNull();
|
||||
const [reviewRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
expect(reviewRun?.contextSnapshot).toMatchObject({
|
||||
taskWatchdog: { reviewRejectedAsStale: true },
|
||||
});
|
||||
});
|
||||
|
||||
it("revalidates a stale watchdog review as live when the source gets a fresh run path", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const sourceId = await seedIssue(companyId, { identifier: "WDOG-LIVE-REVALIDATE", status: "blocked" });
|
||||
|
||||
@@ -331,6 +331,7 @@ function noopTaskWatchdogService(): TaskWatchdogService {
|
||||
live: 0,
|
||||
pendingFirstRun: 0,
|
||||
alreadyReviewed: 0,
|
||||
escalated: 0,
|
||||
skipped: 0,
|
||||
watchdogIssueIds: [],
|
||||
}),
|
||||
|
||||
@@ -1134,6 +1134,7 @@ export function issueThreadInteractionService(db: Db) {
|
||||
issue: { id: string; companyId: string },
|
||||
input: CreateIssueThreadInteraction,
|
||||
actor: InteractionActor,
|
||||
opts: { touchIssue?: boolean } = {},
|
||||
) => {
|
||||
const data = normalizeCreateInteractionInput(createIssueThreadInteractionSchema.parse(input));
|
||||
|
||||
@@ -1230,7 +1231,9 @@ export function issueThreadInteractionService(db: Db) {
|
||||
return hydrateInteraction(existing);
|
||||
}
|
||||
|
||||
await touchIssue(db, issue.id);
|
||||
if (opts.touchIssue !== false) {
|
||||
await touchIssue(db, issue.id);
|
||||
}
|
||||
return hydrateInteraction(created);
|
||||
},
|
||||
|
||||
|
||||
@@ -28,6 +28,7 @@ export type TaskWatchdogMutationScope =
|
||||
watchedIssueId: string;
|
||||
watchdogIssueId: string | null;
|
||||
stopFingerprint: string | null;
|
||||
runId: string;
|
||||
};
|
||||
|
||||
function isPlainRecord(value: unknown): value is Record<string, unknown> {
|
||||
@@ -118,6 +119,7 @@ export async function resolveTaskWatchdogMutationScope(
|
||||
watchedIssueId: watchdog.issueId,
|
||||
watchdogIssueId: watchdog.watchdogIssueId ?? null,
|
||||
stopFingerprint: taskWatchdog.stopFingerprint,
|
||||
runId: run.id,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { and, asc, eq, gte, inArray, isNull, or, sql } from "drizzle-orm";
|
||||
import { and, asc, desc, eq, gte, inArray, isNull, or, sql } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import {
|
||||
agentWakeupRequests,
|
||||
@@ -21,6 +21,7 @@ import { parseObject } from "../adapters/utils.js";
|
||||
import { logActivity } from "./activity-log.js";
|
||||
import { evaluateAgentInvokabilityFromDb } from "./agent-invokability.js";
|
||||
import { issueService } from "./issues.js";
|
||||
import { issueThreadInteractionService } from "./issue-thread-interactions.js";
|
||||
import { visibleIssueCondition } from "./issue-visibility.js";
|
||||
import { TASK_WATCHDOG_ORIGIN_KIND } from "./task-watchdog-scope.js";
|
||||
|
||||
@@ -38,6 +39,8 @@ const TASK_WATCHDOG_TERMINAL_RUN_STATUSES = ["succeeded", "interrupted", "failed
|
||||
// false-positive stopped-subtree review. The periodic watchdog reconciler
|
||||
// re-evaluates after the window, so a genuinely idle issue still triggers.
|
||||
const TASK_WATCHDOG_FIRST_RUN_GRACE_MS = 15_000;
|
||||
const TASK_WATCHDOG_CONDITIONAL_REARM_MS = 30 * 60 * 1000;
|
||||
const TASK_WATCHDOG_SAME_FINGERPRINT_REVIEW_LIMIT = 3;
|
||||
|
||||
type ActorFields = {
|
||||
agentId?: string | null;
|
||||
@@ -74,6 +77,7 @@ export type TaskWatchdogClassifierIssue = Pick<
|
||||
latestCommentAt?: Date | string | null;
|
||||
latestDocumentAt?: Date | string | null;
|
||||
latestWorkProductAt?: Date | string | null;
|
||||
monitorNextCheckAt?: Date | string | null;
|
||||
};
|
||||
|
||||
export type TaskWatchdogClassifierPath = {
|
||||
@@ -102,7 +106,7 @@ export type TaskWatchdogClassifierConfig = Pick<
|
||||
"companyId" | "issueId" | "lastReviewedFingerprint"
|
||||
> & {
|
||||
lastReviewedStopSnapshot?: TaskWatchdogStopSnapshot | null;
|
||||
};
|
||||
} & Partial<Pick<IssueWatchdogSummary, "reviewAcceptanceKind" | "reviewExpiresAt">>;
|
||||
|
||||
export type TaskWatchdogStoppedLeaf = {
|
||||
issueId: string;
|
||||
@@ -118,6 +122,7 @@ export type TaskWatchdogStoppedLeaf = {
|
||||
latestCommentAt: string | null;
|
||||
latestDocumentAt: string | null;
|
||||
latestWorkProductAt: string | null;
|
||||
monitorNextCheckAt: string | null;
|
||||
};
|
||||
|
||||
export type TaskWatchdogMaterialLeaf = Pick<
|
||||
@@ -129,6 +134,7 @@ export type TaskWatchdogMaterialLeaf = Pick<
|
||||
| "blockerIssueIds"
|
||||
| "pendingInteractionIds"
|
||||
| "pendingApprovalIds"
|
||||
| "monitorNextCheckAt"
|
||||
>;
|
||||
|
||||
export type TaskWatchdogWaitsByIssueId = Record<string, {
|
||||
@@ -243,6 +249,9 @@ export function summarizeIssueWatchdog(row: IssueWatchdogRow): IssueWatchdogSumm
|
||||
watchdogIssueId: row.watchdogIssueId,
|
||||
lastObservedFingerprint: row.lastObservedFingerprint,
|
||||
lastReviewedFingerprint: row.lastReviewedFingerprint,
|
||||
reviewAcceptanceKind: row.reviewAcceptanceKind as IssueWatchdogSummary["reviewAcceptanceKind"],
|
||||
reviewExpiresAt: row.reviewExpiresAt,
|
||||
sameFingerprintReviewCount: row.sameFingerprintReviewCount,
|
||||
lastTriggeredAt: row.lastTriggeredAt,
|
||||
lastCompletedAt: row.lastCompletedAt,
|
||||
triggerCount: row.triggerCount,
|
||||
@@ -326,6 +335,7 @@ function materialLeaf(leaf: TaskWatchdogStoppedLeaf): TaskWatchdogMaterialLeaf {
|
||||
blockerIssueIds: leaf.blockerIssueIds,
|
||||
pendingInteractionIds: leaf.pendingInteractionIds,
|
||||
pendingApprovalIds: leaf.pendingApprovalIds,
|
||||
monitorNextCheckAt: leaf.monitorNextCheckAt,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -411,6 +421,13 @@ export function classifyTaskWatchdogSubtree(input: TaskWatchdogClassifierInput):
|
||||
const liveIssueIds = [
|
||||
...pathIssueIds(input.activeRuns, input.watchdog.companyId),
|
||||
...pathIssueIds(input.queuedWakeRequests, input.watchdog.companyId),
|
||||
...included
|
||||
.filter((issue) => {
|
||||
const monitorNextCheckAtMs = toEpochMs(issue.monitorNextCheckAt);
|
||||
const evaluatedAtMs = toEpochMs(input.evaluatedAt) ?? Date.now();
|
||||
return monitorNextCheckAtMs != null && monitorNextCheckAtMs > evaluatedAtMs;
|
||||
})
|
||||
.map((issue) => issue.id),
|
||||
].filter((issueId) => includedIdSet.has(issueId));
|
||||
const uniqueLiveIssueIds = [...new Set(liveIssueIds)].sort();
|
||||
if (uniqueLiveIssueIds.length > 0) {
|
||||
@@ -502,6 +519,7 @@ export function classifyTaskWatchdogSubtree(input: TaskWatchdogClassifierInput):
|
||||
latestCommentAt: optionalIso(issue.latestCommentAt),
|
||||
latestDocumentAt: optionalIso(issue.latestDocumentAt),
|
||||
latestWorkProductAt: optionalIso(issue.latestWorkProductAt),
|
||||
monitorNextCheckAt: optionalIso(issue.monitorNextCheckAt),
|
||||
}));
|
||||
const materialLeaves = leaves.map(materialLeaf);
|
||||
const stopFingerprint = stableStopFingerprint({
|
||||
@@ -517,9 +535,16 @@ export function classifyTaskWatchdogSubtree(input: TaskWatchdogClassifierInput):
|
||||
waitsByIssueId,
|
||||
};
|
||||
|
||||
const reviewExpiresAtMs = toEpochMs(input.watchdog.reviewExpiresAt);
|
||||
const evaluatedAtMsForReview = toEpochMs(input.evaluatedAt) ?? Date.now();
|
||||
const acceptanceStillInForce = input.watchdog.reviewAcceptanceKind !== "conditional"
|
||||
|| (reviewExpiresAtMs != null && reviewExpiresAtMs > evaluatedAtMsForReview);
|
||||
if (
|
||||
input.watchdog.lastReviewedFingerprint === stopFingerprint ||
|
||||
isShrinkOfReviewedSnapshot(currentStopSnapshot, input.watchdog.lastReviewedStopSnapshot)
|
||||
acceptanceStillInForce &&
|
||||
(
|
||||
input.watchdog.lastReviewedFingerprint === stopFingerprint ||
|
||||
isShrinkOfReviewedSnapshot(currentStopSnapshot, input.watchdog.lastReviewedStopSnapshot)
|
||||
)
|
||||
) {
|
||||
return {
|
||||
state: "already_reviewed",
|
||||
@@ -585,6 +610,34 @@ function readNonEmptyString(value: unknown): string | null {
|
||||
return typeof value === "string" && value.trim().length > 0 ? value : null;
|
||||
}
|
||||
|
||||
function taskWatchdogReviewAcceptance(value: unknown, now: Date) {
|
||||
const root = parseObject(value);
|
||||
const taskWatchdog = parseObject(root.taskWatchdog);
|
||||
const acceptance = parseObject(taskWatchdog.reviewAcceptance);
|
||||
const kind = acceptance.kind === "terminal" ? "terminal" : "conditional";
|
||||
if (kind === "terminal") {
|
||||
return { kind, expiresAt: null } as const;
|
||||
}
|
||||
|
||||
const defaultExpiresAt = new Date(now.getTime() + TASK_WATCHDOG_CONDITIONAL_REARM_MS);
|
||||
const source = parseObject(acceptance.livenessSource);
|
||||
const namedExpiry = [acceptance.expiresAt, source.expiresAt, source.nextCheckAt]
|
||||
.map((candidate) => toEpochMs(candidate as Date | string | null | undefined))
|
||||
.filter((candidate): candidate is number => candidate != null && candidate > now.getTime())
|
||||
.sort((left, right) => left - right)[0];
|
||||
return {
|
||||
kind,
|
||||
expiresAt: namedExpiry == null
|
||||
? defaultExpiresAt
|
||||
: new Date(Math.min(namedExpiry, defaultExpiresAt.getTime())),
|
||||
} as const;
|
||||
}
|
||||
|
||||
function taskWatchdogReviewWasRejectedAsStale(value: unknown) {
|
||||
const root = parseObject(value);
|
||||
return parseObject(root.taskWatchdog).reviewRejectedAsStale === true;
|
||||
}
|
||||
|
||||
function issueIdFromRunContext(contextSnapshot: unknown) {
|
||||
const context = parseObject(contextSnapshot);
|
||||
return readNonEmptyString(context.issueId) ?? readNonEmptyString(context.taskId);
|
||||
@@ -613,8 +666,16 @@ function reviewedFingerprintForWatchdogIssue(issue: Pick<IssueRow, "originFinger
|
||||
return normalizeStopFingerprint(issue.originFingerprint) ?? stopFingerprintFromText(issue.description);
|
||||
}
|
||||
|
||||
function taskWatchdogWakeIdempotencyKey(watchdogId: string, stopFingerprint: string) {
|
||||
return `task_watchdog:${watchdogId}:${stopFingerprint}`;
|
||||
function taskWatchdogWakeIdempotencyKey(
|
||||
watchdogId: string,
|
||||
stopFingerprint: string,
|
||||
sameFingerprintReviewCount: number,
|
||||
) {
|
||||
return `task_watchdog:${watchdogId}:${stopFingerprint}:${sameFingerprintReviewCount}`;
|
||||
}
|
||||
|
||||
function isTaskWatchdogEscalationIdempotencyKey(value: string | null | undefined, watchdogId: string) {
|
||||
return value?.startsWith(`task_watchdog_escalation:${watchdogId}:`) === true;
|
||||
}
|
||||
|
||||
function buildStoppedFingerprintComment(input: {
|
||||
@@ -874,6 +935,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
|
||||
assignee_user_id,
|
||||
origin_kind,
|
||||
updated_at,
|
||||
monitor_next_check_at,
|
||||
created_at,
|
||||
0 AS depth
|
||||
FROM issues
|
||||
@@ -893,6 +955,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
|
||||
child.assignee_user_id,
|
||||
child.origin_kind,
|
||||
child.updated_at,
|
||||
child.monitor_next_check_at,
|
||||
child.created_at,
|
||||
watched_issues.depth + 1
|
||||
FROM issues child
|
||||
@@ -914,6 +977,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
|
||||
assignee_user_id AS "assigneeUserId",
|
||||
origin_kind AS "originKind",
|
||||
updated_at AS "updatedAt",
|
||||
monitor_next_check_at AS "monitorNextCheckAt",
|
||||
created_at AS "createdAt"
|
||||
FROM watched_issues
|
||||
`);
|
||||
@@ -1018,6 +1082,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
|
||||
id: issueThreadInteractions.id,
|
||||
kind: issueThreadInteractions.kind,
|
||||
status: issueThreadInteractions.status,
|
||||
idempotencyKey: issueThreadInteractions.idempotencyKey,
|
||||
})
|
||||
.from(issueThreadInteractions)
|
||||
.where(and(
|
||||
@@ -1116,7 +1181,15 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
|
||||
issueId: issueIdFromWakePayload(row.payload),
|
||||
})),
|
||||
blockers: blockerRows,
|
||||
pendingInteractions: interactionRows,
|
||||
pendingInteractions: interactionRows
|
||||
.filter((row) => !isTaskWatchdogEscalationIdempotencyKey(row.idempotencyKey, watchdog.id))
|
||||
.map((row) => ({
|
||||
companyId: row.companyId,
|
||||
issueId: row.issueId,
|
||||
id: row.id,
|
||||
kind: row.kind,
|
||||
status: row.status,
|
||||
})),
|
||||
pendingApprovals: approvalRows,
|
||||
evaluatedAt,
|
||||
firstRunGraceMs: TASK_WATCHDOG_FIRST_RUN_GRACE_MS,
|
||||
@@ -1280,21 +1353,53 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
|
||||
if (!isWatchdogReviewDisposition(watchdogIssue, hasPendingReviewPath)) return watchdog;
|
||||
const reviewedFingerprint = reviewedFingerprintForWatchdogIssue(watchdogIssue);
|
||||
if (!reviewedFingerprint) return watchdog;
|
||||
if (watchdog.lastCompletedAt && watchdog.lastCompletedAt.getTime() >= watchdogIssue.updatedAt.getTime()) {
|
||||
return watchdog;
|
||||
}
|
||||
const observedSnapshot = parseStopSnapshot(watchdog.lastObservedStopSnapshot);
|
||||
const reviewedStopSnapshot = observedSnapshot?.fingerprint === reviewedFingerprint
|
||||
? observedSnapshot
|
||||
: null;
|
||||
const reviewRun = await db
|
||||
.select({ contextSnapshot: heartbeatRuns.contextSnapshot, resultJson: heartbeatRuns.resultJson })
|
||||
.from(heartbeatRuns)
|
||||
.where(and(
|
||||
eq(heartbeatRuns.companyId, watchdog.companyId),
|
||||
eq(heartbeatRuns.agentId, watchdog.watchdogAgentId),
|
||||
sql`(${heartbeatRuns.contextSnapshot}->>'issueId' = ${watchdogIssue.id}
|
||||
OR ${heartbeatRuns.contextSnapshot}->>'taskId' = ${watchdogIssue.id})`,
|
||||
sql`${heartbeatRuns.contextSnapshot}->'taskWatchdog'->>'stopFingerprint' = ${reviewedFingerprint}`,
|
||||
))
|
||||
.orderBy(desc(heartbeatRuns.createdAt))
|
||||
.limit(1)
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (taskWatchdogReviewWasRejectedAsStale(reviewRun?.contextSnapshot)) return watchdog;
|
||||
const now = new Date();
|
||||
const resultTaskWatchdog = parseObject(parseObject(reviewRun?.resultJson).taskWatchdog);
|
||||
const acceptance = taskWatchdogReviewAcceptance(
|
||||
Object.keys(resultTaskWatchdog).length > 0 ? reviewRun?.resultJson : reviewRun?.contextSnapshot,
|
||||
now,
|
||||
);
|
||||
const sameFingerprintReviewCount = acceptance.kind === "terminal"
|
||||
? 0
|
||||
: watchdog.lastReviewedFingerprint === reviewedFingerprint
|
||||
? watchdog.sameFingerprintReviewCount + 1
|
||||
: 1;
|
||||
if (
|
||||
watchdog.lastReviewedFingerprint === reviewedFingerprint &&
|
||||
canonicalJson(parseStopSnapshot(watchdog.lastReviewedStopSnapshot)) === canonicalJson(reviewedStopSnapshot)
|
||||
canonicalJson(parseStopSnapshot(watchdog.lastReviewedStopSnapshot)) === canonicalJson(reviewedStopSnapshot) &&
|
||||
acceptance.kind === "terminal"
|
||||
) return watchdog;
|
||||
const [updated] = await db
|
||||
.update(issueWatchdogs)
|
||||
.set({
|
||||
lastReviewedFingerprint: reviewedFingerprint,
|
||||
lastReviewedStopSnapshot: reviewedStopSnapshot,
|
||||
lastCompletedAt: new Date(),
|
||||
updatedAt: new Date(),
|
||||
reviewAcceptanceKind: acceptance.kind,
|
||||
reviewExpiresAt: acceptance.expiresAt,
|
||||
sameFingerprintReviewCount,
|
||||
lastCompletedAt: now,
|
||||
updatedAt: now,
|
||||
})
|
||||
.where(eq(issueWatchdogs.id, watchdog.id))
|
||||
.returning();
|
||||
@@ -1312,6 +1417,9 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
|
||||
watchdogId: watchdog.id,
|
||||
watchdogIssueId: watchdogIssue.id,
|
||||
reviewedFingerprint,
|
||||
reviewAcceptanceKind: acceptance.kind,
|
||||
reviewExpiresAt: acceptance.expiresAt?.toISOString() ?? null,
|
||||
sameFingerprintReviewCount,
|
||||
lastObservedFingerprint: watchdog.lastObservedFingerprint,
|
||||
reviewedStopSnapshot,
|
||||
watchdogIssueStatus: watchdogIssue.status,
|
||||
@@ -1445,7 +1553,6 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
|
||||
if (!sourceIssue || sourceIssue.originKind === TASK_WATCHDOG_ORIGIN_KIND) {
|
||||
return { state: "skipped" as const, reason: "watched_issue_not_applicable" };
|
||||
}
|
||||
|
||||
const input = await collectClassifierInput(watchdog.companyId, watchdog);
|
||||
const classification = classifyTaskWatchdogSubtree(input);
|
||||
if (classification.state !== "stopped") {
|
||||
@@ -1502,6 +1609,99 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
|
||||
};
|
||||
}
|
||||
|
||||
const shouldEscalateSameFingerprintReview =
|
||||
watchdog.lastReviewedFingerprint === classification.stopFingerprint
|
||||
&& watchdog.reviewAcceptanceKind === "conditional"
|
||||
&& watchdog.sameFingerprintReviewCount >= TASK_WATCHDOG_SAME_FINGERPRINT_REVIEW_LIMIT;
|
||||
if (shouldEscalateSameFingerprintReview) {
|
||||
const legacyEscalationIdempotencyKey = `task_watchdog_escalation:${watchdog.id}:${classification.stopFingerprint}`;
|
||||
const escalationIdempotencyPrefix = `${legacyEscalationIdempotencyKey}:`;
|
||||
const matchingEscalations = await db
|
||||
.select({
|
||||
id: issueThreadInteractions.id,
|
||||
status: issueThreadInteractions.status,
|
||||
})
|
||||
.from(issueThreadInteractions)
|
||||
.where(and(
|
||||
eq(issueThreadInteractions.companyId, sourceIssue.companyId),
|
||||
eq(issueThreadInteractions.issueId, sourceIssue.id),
|
||||
or(
|
||||
eq(issueThreadInteractions.idempotencyKey, legacyEscalationIdempotencyKey),
|
||||
sql`${issueThreadInteractions.idempotencyKey} like ${`${escalationIdempotencyPrefix}%`}`,
|
||||
),
|
||||
));
|
||||
const pendingEscalation = matchingEscalations.find((row) => row.status === "pending") ?? null;
|
||||
if (pendingEscalation) {
|
||||
return {
|
||||
state: "escalated" as const,
|
||||
classification,
|
||||
interactionId: pendingEscalation.id,
|
||||
};
|
||||
}
|
||||
const escalationIdempotencyKey = `${escalationIdempotencyPrefix}${matchingEscalations.length + 1}`;
|
||||
const interaction = await issueThreadInteractionService(db).create(
|
||||
sourceIssue,
|
||||
{
|
||||
kind: "ask_user_questions",
|
||||
idempotencyKey: escalationIdempotencyKey,
|
||||
title: "Task watchdog needs a board decision",
|
||||
summary: "The watchdog reviewed the same stopped subtree three times without a durable state change.",
|
||||
continuationPolicy: "wake_assignee",
|
||||
payload: {
|
||||
version: 1,
|
||||
title: "How should this stopped task tree proceed?",
|
||||
submitLabel: "Send decision",
|
||||
supersedeOnUserComment: true,
|
||||
questions: [{
|
||||
id: "watchdog_next_step",
|
||||
prompt: "The same stopped state has reached the watchdog review limit. What should happen next?",
|
||||
helpText: `Stopped fingerprint: ${classification.stopFingerprint}`,
|
||||
selectionMode: "single",
|
||||
required: true,
|
||||
options: [
|
||||
{ id: "resume", label: "Resume work", description: "Reopen or reassign the stopped work." },
|
||||
{ id: "accept", label: "Accept stopped state", description: "Treat the current state as terminally acceptable." },
|
||||
{ id: "investigate", label: "Investigate", description: "Assign a human investigation before more watchdog reviews." },
|
||||
],
|
||||
}],
|
||||
},
|
||||
},
|
||||
{},
|
||||
// The escalation card is watchdog metadata; touching the watched issue
|
||||
// would mutate stopped-state fingerprint inputs and cause self re-arms.
|
||||
{ touchIssue: false },
|
||||
);
|
||||
await db
|
||||
.update(issueWatchdogs)
|
||||
.set({
|
||||
lastObservedFingerprint: classification.stopFingerprint,
|
||||
updatedAt: new Date(),
|
||||
})
|
||||
.where(eq(issueWatchdogs.id, watchdog.id));
|
||||
await logActivity(db, {
|
||||
companyId: sourceIssue.companyId,
|
||||
actorType: "system",
|
||||
actorId: "system",
|
||||
agentId: watchdog.watchdogAgentId,
|
||||
runId: opts.runId ?? null,
|
||||
action: "issue.task_watchdog_escalated",
|
||||
entityType: "issue",
|
||||
entityId: sourceIssue.id,
|
||||
details: {
|
||||
source: "task_watchdogs.evaluate",
|
||||
watchdogId: watchdog.id,
|
||||
stopFingerprint: classification.stopFingerprint,
|
||||
sameFingerprintReviewCount: watchdog.sameFingerprintReviewCount,
|
||||
interactionId: interaction.id,
|
||||
},
|
||||
});
|
||||
return {
|
||||
state: "escalated" as const,
|
||||
classification,
|
||||
interactionId: interaction.id,
|
||||
};
|
||||
}
|
||||
|
||||
const watchdogIssue = await ensureReusableWatchdogIssue({
|
||||
watchdog,
|
||||
sourceIssue,
|
||||
@@ -1553,7 +1753,11 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
|
||||
reason: "task_watchdog_stopped_subtree",
|
||||
payload: context,
|
||||
contextSnapshot: context,
|
||||
idempotencyKey: taskWatchdogWakeIdempotencyKey(watchdog.id, classification.stopFingerprint),
|
||||
idempotencyKey: taskWatchdogWakeIdempotencyKey(
|
||||
watchdog.id,
|
||||
classification.stopFingerprint,
|
||||
watchdog.sameFingerprintReviewCount,
|
||||
),
|
||||
requestedByActorType: "system",
|
||||
requestedByActorId: null,
|
||||
})
|
||||
@@ -1617,6 +1821,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
|
||||
companyId: string;
|
||||
watchedIssueId: string;
|
||||
stopFingerprint: string | null;
|
||||
runId?: string | null;
|
||||
}) {
|
||||
if (!scope.stopFingerprint) {
|
||||
return {
|
||||
@@ -1648,6 +1853,31 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
|
||||
return { allowed: true as const, classification };
|
||||
}
|
||||
|
||||
const run = scope.runId ? await db
|
||||
.select({ contextSnapshot: heartbeatRuns.contextSnapshot })
|
||||
.from(heartbeatRuns)
|
||||
.where(and(eq(heartbeatRuns.id, scope.runId), eq(heartbeatRuns.companyId, scope.companyId)))
|
||||
.then((rows) => rows[0] ?? null) : null;
|
||||
if (run) {
|
||||
const contextSnapshot = parseObject(run.contextSnapshot);
|
||||
const taskWatchdog = parseObject(contextSnapshot.taskWatchdog);
|
||||
await db
|
||||
.update(heartbeatRuns)
|
||||
.set({
|
||||
contextSnapshot: {
|
||||
...contextSnapshot,
|
||||
taskWatchdog: {
|
||||
...taskWatchdog,
|
||||
reviewRejectedAsStale: true,
|
||||
rejectedCurrentState: classification.state,
|
||||
rejectedAt: new Date().toISOString(),
|
||||
},
|
||||
},
|
||||
updatedAt: new Date(),
|
||||
})
|
||||
.where(eq(heartbeatRuns.id, scope.runId!));
|
||||
}
|
||||
|
||||
return {
|
||||
allowed: false as const,
|
||||
reason: classification.state === "stopped"
|
||||
@@ -1750,6 +1980,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
|
||||
live: 0,
|
||||
pendingFirstRun: 0,
|
||||
alreadyReviewed: 0,
|
||||
escalated: 0,
|
||||
skipped: 0,
|
||||
watchdogIssueIds: [] as string[],
|
||||
};
|
||||
@@ -1759,6 +1990,8 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {})
|
||||
if (evaluated.state === "triggered") {
|
||||
result.triggered += 1;
|
||||
result.watchdogIssueIds.push(evaluated.watchdogIssueId);
|
||||
} else if (evaluated.state === "escalated") {
|
||||
result.escalated += 1;
|
||||
} else if (
|
||||
evaluated.state === "live" ||
|
||||
evaluated.state === "watchdog_live" ||
|
||||
|
||||
@@ -20,6 +20,7 @@ For the **mechanics** of recording a plan (issue document with key `plan`, comme
|
||||
- **Take responsibility.** Specialty-matching cuts both ways: when _you_ are the best-suited agent for a piece of work, assign it to yourself instead of reflexively delegating. Don't hand off to avoid load.
|
||||
- **Use the dependency tree.** Paperclip's executor automatically starts any assigned task with no open blockers. Parent/child issue nesting is structure, not execution blocking. Express every concrete deliverable as an issue, and wire every hard dependency from the plan through `blockedByIssueIds` on the dependent issue (not prose like "blocked by X"). When a blocker reaches `done`, dependents auto-wake.
|
||||
- **Order, then parallelize.** Sequence work by real dependencies, not by personal preference. Independent branches of the graph should start in parallel. Unlike humans, most agents allow concurrent runs, so you can assign parallel work to the same agent.
|
||||
- **Shape the graph: terminal gates, not funnels.** Block an umbrella/rollup issue on its terminal acceptance gate (the final QA or review issue), not on every phase — per-phase dependencies belong between the phases themselves, and duplicating them onto the umbrella only delays its wake. Avoid single-leaf funnels where every open issue transitively waits on one leaf: one slow leaf then stalls the whole tree, and later-phase owners can neither start nor observe progress. Keep independent branches independent; serialize only where a real dependency exists. (Normative reference: `doc/execution-semantics.md`, "Plan shape: terminal gates, not phase funnels".)
|
||||
- **Enough is enough.** Plans exist to unblock execution, not replace it. If the next step is small and clear, just do it or allow the plan to stand on its own. Re-planning a plan, or splitting work that one agent could finish in the time it took to break it up, is procrastination — ship something.
|
||||
|
||||
## When converting an accepted plan into tasks
|
||||
@@ -34,6 +35,7 @@ After creating the tasks, re-fetch the created issues or otherwise verify the is
|
||||
- [ ] Every concrete deliverable is an issue (or named as a known follow-up).
|
||||
- [ ] Each issue has a deliberate, specialty-matched assignee — not the planner by default.
|
||||
- [ ] Each issue's real blockers are declared via `blockedByIssueIds`.
|
||||
- [ ] Umbrella/rollup issues block on their terminal gate, not on every phase; no single-leaf funnels.
|
||||
- [ ] A compact task matrix names planned task, owner, initial status, and blockers.
|
||||
- [ ] Tasks without blockers have an explicit reason they can start immediately.
|
||||
- [ ] Created issues were re-fetched or otherwise verified before closing the source planning issue.
|
||||
|
||||
@@ -64,6 +64,9 @@ describe("buildIssuePropertiesPanelKey", () => {
|
||||
watchdogIssueId: null,
|
||||
lastObservedFingerprint: null,
|
||||
lastReviewedFingerprint: null,
|
||||
reviewAcceptanceKind: null,
|
||||
reviewExpiresAt: null,
|
||||
sameFingerprintReviewCount: 0,
|
||||
lastTriggeredAt: null,
|
||||
lastCompletedAt: null,
|
||||
triggerCount: 0,
|
||||
|
||||
@@ -28,6 +28,9 @@ function watchdog(overrides: Partial<IssueWatchdogSummary> = {}): IssueWatchdogS
|
||||
watchdogIssueId: null,
|
||||
lastObservedFingerprint: null,
|
||||
lastReviewedFingerprint: null,
|
||||
reviewAcceptanceKind: null,
|
||||
reviewExpiresAt: null,
|
||||
sameFingerprintReviewCount: 0,
|
||||
lastTriggeredAt: null,
|
||||
lastCompletedAt: null,
|
||||
triggerCount: 0,
|
||||
|
||||
Reference in new issue
Block a user