Files
PaperClipAI/server/src/__tests__/issue-recovery-deadlines.test.ts
T
f33c4cf6f7 feat(links): cache rich link status in the database and load it on idle (#21746)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task text contains links to pull requests and other work.
> - A link has useful status that should be visible when a task opens.
> - The current cache lives in the server process and the browser.
> - A new browser or server restart has no stored status to show.
> - This pull request stores link status in the database and reads it in
batches.
> - Idle page work updates stale status without a hover or a delay to
task data.

## Linked Issues or Issue Description

**Problem or motivation**

PR status is unknown in a new browser until the user points to each
link. The process cache cannot supply saved status after a server
restart. Related work: #20593 and #20654. A search found no other PR for
database link caching.

**Proposed solution**

Store status by company, viewer and canonical resource URL. Read saved
status in one indexed batch query. After document load and core request
settlement, use browser idle time to update missing or stale status. Use
the same query for hover and idle work.

**Alternatives considered**

Adding provider calls to task reads would delay task loading.
Company-wide reuse of personal GitHub results would cross viewer access
boundaries. A periodic server worker would create work when no page
needs the result.

**Roadmap alignment**

This extends the shipped rich-link feature. It does not duplicate a
roadmap project or add a new provider integration.

## What Changed

- Add an additive, repeat-safe `link_previews` migration. Store channel,
resource kind, free-form status, JSON metadata and freshness. The table
has no task relationship or GitHub status enum.
- Bound each company/viewer cache to 1,000 rows across concurrent server
instances. Remove rows expired for over 30 days during writes. Limit
provider lookups to 600 requests per minute per scope/process. Keep
cached batch reads available during backoff.
- Add a database store owned by the link service. Reuse canonical
results across tasks. Keep company and viewer isolation. Save hover and
idle results. Preserve last known status during an outage and cache
unavailable results.
- Add a board-only batch GET endpoint, including viewer-role access. It
reads at most 50 URLs with one indexed select and makes no provider
calls. Both endpoints use private, no-store HTTP responses.
- Add a page coordinator for mounted rich links. Reuse browser data,
share pending reads, and defer provider work until document load, core
request settlement and browser idle time. Limit automatic work to 200
unique links and three requests in flight. Remove queued work when links
leave the page. Keep encoded batch URLs below 6 KB.
- Keep exact navigation URLs, status styling, anchored cards and
keyboard access. Add a page-load Storybook case and browser regression.

The link service owns cache policy and provider calls. The browser
coordinator owns page timing. The shared contract adds only the batch
limit and response shape. The database adds storage because process and
browser caches cannot serve a new browser session. Plugin resolver
registration is outside this change; storage accepts future provider
channels and status values.

## Verification

- **All 93 affected tests pass:** 40 link service, real PostgreSQL and
HTTP cases, plus 53 OpenAPI cases. Two SQL regressions and one HTTP case
failed before the storage repair. Concurrent writers now retain at most
1,000 rows per company/viewer; cleanup removes long-expired rows and
keeps recent stale status. Request backoff preserves cached reads and
isolates companies/viewers. The earlier eight concurrent hydration
regressions also pass. Focused store/service/unit types pass. Commands:
`pnpm exec vitest run server/src/__tests__/link-previews.test.ts
server/src/__tests__/link-preview-store.integration.test.ts
server/src/__tests__/link-preview-routes.test.ts --maxWorkers=1` and
`pnpm exec vitest run server/src/__tests__/openapi-routes.test.ts
--maxWorkers=1`.
- **All 50 affected server tests passed before the later server-only
repairs** on the master-integrated source: 20 recovery/deadline cases,
one authenticated review-counts HTTP case, 10 link-service cases, 13
real PostgreSQL cache cases and six link-route cases. Command: `pnpm
exec vitest run server/src/__tests__/issue-recovery-deadlines.test.ts
server/src/__tests__/tool-connection-review-counts-http.test.ts
server/src/__tests__/link-previews.test.ts
server/src/__tests__/link-preview-store.integration.test.ts
server/src/__tests__/link-preview-routes.test.ts --maxWorkers=1`.
- Reproduced the CI watchdog assertion locally twice before the test
repair. Ordinary assignment recovery can win `wakeReason` while the
watchdog wake coalesces into the same successful run with its full
mandate. The test now verifies the persisted watchdog identity, watched
issue and stopped fingerprint, plus exactly one review run. All 21
recovery/HTTP cases passed before integrating master; all 50 affected
cases pass after integration. No runtime behavior was changed for this
repair.
- **Fresh static Storybook build and all 22 browser cases pass**, with
two workers and retries disabled. They cover states, anchored hover,
keyboard access, exact navigation, duplicate requests, reload
persistence, expiry and saved-status hydration followed by one idle
refresh. The first browser attempt could not launch because this session
lacked Chromium; installing the expected browser resolved setup. That
attempt had no behavioral verdict.
- The cache repair remains covered by four PostgreSQL red/green
regressions for expired memory and newer durable status during provider
failure/timeout, plus check-time precedence and failed-read recovery.
Earlier UI/database/focused server type and migration
safety/compatibility/replay checks remain unchanged-input evidence.
- Master `77ddf260bf9a3de3dcb8456a60f4f809d67401fd` is integrated
cleanly. Link migration 0489 and its snapshot chain are unchanged. SDK
source validation checks 545 inputs with no stale source identities or
changed runtime/lock pins. Whitespace validation passes.
- `pnpm verify --explain` returns exit 2 and requires broad CI. No full
local suite or deployed acceptance is claimed. Earlier local full-server
type checks exceeded memory; current-head CI supplies broad
type/build/test coverage.
- [Previous CI
43900336895](https://github.com/paperclipai/paperclip/actions/runs/43900336895)
passed full types/build but failed on embedded PostgreSQL startup
(`ECONNREFUSED`), the watchdog assertion above, and unrelated plan-task
browser cleanup (activity-log foreign key on company deletion). One
failed-jobs retry was requested. Its browser shard 7 now passes without
code changes; the exact deletion-race cause remains unproven. The
watchdog assertion is repaired in this revision. The previous failures
remain recorded, not counted as an overall pass.
- [Prior CI
43905482848](https://github.com/paperclipai/paperclip/actions/runs/43905482848)
passed full types, build and all eight browser shards. Several test jobs
were cancelled; the run has no overall pass. Its concurrent update
finding is repaired. [Next CI
43909197530](https://github.com/paperclipai/paperclip/actions/runs/43909197530)
also passed types, build and all browser shards, but several server jobs
were cancelled. Greptile and Paperclip Review gave that revision 5/5.
Its security scan found unbounded retained cache rows. The quota,
retention and request-limit repair above addresses that finding. All ten
review threads are resolved.
- [Current CI
43912947278](https://github.com/paperclipai/paperclip/actions/runs/43912947278),
fresh code reviews and a fresh security scan are pending on
`335d89ef1668dcdeb66bb2059512a7c2da29e273`. Auto-merge is disabled until
the current revision clears its gates.

Published head: `335d89ef1668dcdeb66bb2059512a7c2da29e273`. Local head:
`b37b87c704a2ba8a65233c689a6a743f4011ce14`. Both have tree
`643ebd19b9729b1e6a6bb695442269c3db32be3a`; GitHub tree creation
verified equality.

## Risks

- Each company/viewer retains at most 1,000 rows. Writes remove excess
rows and entries whose freshness expired over 30 days ago. Inactive
scopes remain bounded and are cleaned on the next write. Company
deletion removes all rows. No periodic worker runs. Scope writes use a
transaction lock; the existing deadline and concurrency budget cover
that work.
- Last known status can be stale until idle or hover work completes.
Provider failures keep that status through the negative-cache window.
- Automatic enrichment adds bounded provider traffic. It runs only for
mounted supported links, once per mount, with the existing timeout and
server concurrency limit.
- Cache storage is viewer-scoped because provider results can be
private. New viewers load their own results. This prevents reuse across
users with different repository access.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, repository tools, code
execution and browser automation. The runtime does not expose the exact
serving model ID or context-window size. No subagents were used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
2026-11-16 14:33:39 -06:00

500 lines
33 KiB
TypeScript

import { setImmediate as immediate } from "node:timers/promises";
import { idleWorkSnapshot } from "../services/task-admission.js";
import { randomUUID } from "node:crypto";
import { and, eq, inArray, sql } from "drizzle-orm";
import { expect, it, vi } from "vitest";
import { agents, budgetPolicies, heartbeatRuns, issueRecoveryActions, issueRelations, issues, issueWatchdogs } from "@paperclipai/db";
import { heartbeatService } from "../services/heartbeat.js";
import { issueService } from "../services/issues.js";
import { issueRecoveryActionService } from "../services/issue-recovery-actions.js";
import { registerIssueRecoveryWork } from "../services/recovery/work.js";
import { changedTaskCondition } from "../services/recovery/changed-tasks.js";
import { recoveryService } from "../services/recovery/service.js";
import { ISSUE_RECOVERY_BACKSTOP_MS, ISSUE_RECOVERY_MAX_ISSUE_IDS, notifyIssueRecovery, resolveIssueGraph }
from "../services/issue-recovery-notifications.js";
import { notifyNativeStatusReassessment } from "../services/native-status-reassessment-notifications.js";
import { notifyExecutionRunWork } from "../services/execution-control-notifications.js";
import { notifyRunDispatch } from "../services/run-dispatch-notifications.js";
import { notifyExternalConversationSettlement } from "../services/external-conversation-settlement-notifications.js";
import { notifyChatInboundWakeNotices } from "../services/chat-inbound-wake-notice-notifications.js";
import { promoteDeferredWorkIntents } from "../services/work-intent-outbox.js";
import { taskWatchdogService } from "../services/task-watchdogs.js";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js";
describeIssueRecovery("issue recovery production producers and deadlines", () => {
const state = issueRecoveryFixture();
const { start, compete, fixture, completingProvider, settled, quiet, finish, completed } = state;
it("keeps empty, configured idle, and terminal history quiet", async () => {
await fixture(); await start(); await quiet();
});
it("creates a blocking child through the ordinary root service transaction", async () => {
const f = await fixture();
const parent = await issueService(state.db).create(f.companyId, { title: "Parent", status: "todo" });
const child = await issueService(state.db).createChild(parent.id, { title: "Blocking child", status: "todo", blockParentUntilDone: true });
expect(child.parentBlockerAdded).toBe(true);
expect(await state.db.select().from(issueRelations).where(and(eq(issueRelations.companyId, f.companyId),
eq(issueRelations.issueId, child.issue.id), eq(issueRelations.relatedIssueId, parent.id)))).toHaveLength(1);
});
it("bounds subsidiary scans for100 assignment candidates and coalesces repeated company writes", async () => {
const f = await fixture(), foreign = await fixture();
await state.db.update(agents).set({ lifecycleState: "preparing" }).where(eq(agents.id, f.agentId));
await state.db.insert(issues).values(Array.from({ length: 100 }, (_, index) => ({ companyId: f.companyId,
title: `Waiting setup ${index}`, status: "todo", assigneeAgentId: f.agentId })));
const [source] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Done source", status: "done" }).returning();
const action = await issueRecoveryActionService(state.db).upsertSourceScoped({ companyId: f.companyId, sourceIssueId: source.id,
kind: "stranded_assigned_issue", ownerType: "agent", ownerAgentId: f.agentId, cause: "stranded_assigned_issue",
fingerprint: "query-budget", nextAction: "Restore work", wakePolicy: { type: "wake_owner" }, evidence: {} });
const scans = { blockers: 0, actions: 0, loads: 0 };
const originalSelect = state.db.select.bind(state.db);
vi.spyOn(state.db, "select").mockImplementation(((fields?: Record<string, unknown>) => {
const query = originalSelect(fields as never); const from = query.from.bind(query);
query.from = ((table: unknown) => {
if (table === issues && fields?.id === issues.id && fields.companyId === issues.companyId && Object.keys(fields).length === 2) scans.loads++;
if (table === issueRelations && fields?.createdByAgentId === issues.createdByAgentId) scans.blockers++;
if (table === issueRecoveryActions && fields?.action === issueRecoveryActions) scans.actions++;
return from(table as never);
}) as typeof query.from;
return query;
}) as typeof state.db.select);
await start(); await settled();
expect(scans.actions).toBe(1); expect(scans.blockers).toBe(0); expect(scans.loads).toBeLessThanOrEqual(11);
expect((await originalSelect().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id)))[0].status).toBe("resolved");
const [foreignSource] = await state.db.insert(issues).values({ companyId: foreign.companyId, title: "Foreign done", status: "done" }).returning();
const [foreignAction] = await state.db.insert(issueRecoveryActions).values({ companyId: foreign.companyId, sourceIssueId: foreignSource.id,
kind: "stranded_assigned_issue", ownerType: "board", cause: "stranded_assigned_issue", fingerprint: "foreign-query-budget", nextAction: "Inspect" }).returning();
await state.db.transaction(tx => notifyIssueRecovery(tx, { companyId: f.companyId })); await settled();
const before = { ...scans };
for (let index = 0; index < 100; index++) await state.db.transaction(tx => notifyIssueRecovery(tx, { companyId: f.companyId }));
await new Promise(resolve => setTimeout(resolve, 100));
expect(scans.loads).toBe(before.loads);
await settled();
expect(scans.actions).toBe(before.actions); expect(scans.blockers).toBe(before.blockers);
expect(scans.loads).toBeLessThanOrEqual(before.loads + 4);
expect((await originalSelect().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, foreignAction.id)))[0].status).toBe("active");
expect(await originalSelect().from(heartbeatRuns)).toHaveLength(0); await quiet();
});
async function recordedRecovery(owner: Partial<Parameters<typeof registerIssueRecoveryWork>[2]>,
options?: Parameters<typeof registerIssueRecoveryWork>[4]) {
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
state.coordinator = createDeliveryWorkCoordinator({ owner: state.db, canRun: () => true, onError: error => state.errors.push(error) });
const register = state.coordinator.register.bind(state.coordinator);
state.coordinator.register = (queue, task) => register(queue, { ...task, run: signal => {
const pass = task.run(signal); state.passes.add(pass);
void pass.then(() => state.passes.delete(pass), () => state.passes.delete(pass));
return pass;
} });
state.worker = registerIssueRecoveryWork(state.db, state.coordinator, {
stranded: async () => {}, orphanBlockers: async () => {}, recoveryActions: async () => {},
dependencies: async () => ({ enqueueFailed: 0 }), locks: async () => {},
watchdogs: async () => ({ incomplete: 0 }), schedulingSuppressed: async () => false, ...owner,
}, () => true, options);
await state.worker.ready;
}
async function settleWork(options: { backstopPending?: boolean } = {}) {
for (let step = 0; step < 120; step++) {
await vi.advanceTimersByTimeAsync(1000);
await Promise.allSettled([...state.passes]); await immediate();
const next = state.coordinator.nextWakeAt();
const settledNow = next === null || (options.backstopPending === true && next > Date.now() + 60_000);
if (settledNow && idleWorkSnapshot().active === 0) return;
}
throw new Error("Recovery did not settle its committed work within two virtual minutes");
}
async function runGraph(companyId: string, agentId: string) {
const [other] = await state.db.insert(agents).values({ companyId, name: "Other agent", role: "engineer", status: "idle",
adapterType: "process", adapterConfig: {} }).returning({ id: agents.id });
const [parent] = await state.db.insert(issues).values({ companyId, title: "Parent", status: "in_progress", assigneeAgentId: agentId })
.returning({ id: issues.id });
const [task] = await state.db.insert(issues).values({ companyId, title: "Run task", status: "in_progress", assigneeAgentId: agentId,
parentId: parent.id }).returning({ id: issues.id });
const [dependent] = await state.db.insert(issues).values({ companyId, title: "Waits on the run task", status: "todo",
assigneeAgentId: other.id }).returning({ id: issues.id });
const [sibling] = await state.db.insert(issues).values({ companyId, title: "Next task of the agent", status: "todo",
assigneeAgentId: agentId }).returning({ id: issues.id });
await state.db.insert(issueRelations).values({ companyId, issueId: task.id, relatedIssueId: dependent.id, type: "blocks" });
const [run] = await state.db.insert(heartbeatRuns).values({ companyId, agentId, status: "failed", finishedAt: new Date(),
contextSnapshot: { issueId: task.id } }).returning({ id: heartbeatRuns.id });
const backlog = async (count: number) => (await state.db.insert(issues).values(Array.from({ length: count }, (_, index) => ({
companyId, title: `Backlog ${index}`, status: "todo", assigneeAgentId: other.id, createdAt: new Date(), updatedAt: new Date(),
}))).returning({ id: issues.id })).map(row => row.id);
return { runId: run.id, taskId: task.id, siblingId: sibling.id, backlog, affected: [parent.id, task.id, dependent.id].sort() };
}
it("limits an ordinary comment to its issue among one thousand open assignments", async () => {
const f = await fixture();
const rows = await state.db.insert(issues).values(Array.from({ length: 1000 }, (_, index) => ({
companyId: f.companyId, title: `Open work ${index}`, status: "todo", assigneeAgentId: f.agentId,
}))).returning({ id: issues.id });
const checked: string[] = [];
await recordedRecovery({ stranded: async scope => { checked.push(scope.issueId); } });
await settleWork();
expect(new Set(checked).size).toBe(1000);
checked.length = 0;
await issueService(state.db).addComment(rows[0].id, "New evidence on this task", { userId: "recovery-user" });
await settleWork({ backstopPending: true });
expect(checked).toEqual([rows[0].id]);
checked.length = 0;
await state.db.transaction(tx => notifyIssueRecovery(tx, { companyId: f.companyId }));
await settleWork({ backstopPending: true });
expect(new Set(checked).size).toBe(1000);
await vi.advanceTimersByTimeAsync(ISSUE_RECOVERY_BACKSTOP_MS);
await settleWork();
await quiet();
});
it("re-checks only the tasks a run lifecycle event can affect, whatever the backlog", async () => {
const f = await fixture();
const graph = await runGraph(f.companyId, f.agentId);
await graph.backlog(1000);
const checked: string[] = [];
const assigned: string[] = [];
await recordedRecovery({
stranded: async scope => { checked.push(scope.issueId); },
assignedWork: {
listSuspects: (scope, cursor, limit) => recoveryService(state.db, { enqueueWakeup: async () => null })
.assignedWork.listSuspects(scope, cursor, limit),
checkIssue: async scope => { assigned.push(scope.issueId); },
},
}, { backstopMs: 60 * 60_000 });
await settleWork({ backstopPending: true });
expect(new Set(checked).size).toBe(1004);
expect(checked).toContain(graph.siblingId);
async function lifecycleEvent(producer: (tx: object) => Promise<unknown>, standalone?: () => Promise<unknown>) {
checked.length = 0; assigned.length = 0;
const statements: string[] = [], prior = state.db.$client.options.debug;
state.db.$client.options.debug = (_connection, query) => { statements.push(query); };
try {
if (standalone) await standalone(); else await state.db.transaction(tx => producer(tx));
await settleWork({ backstopPending: true });
} finally { state.db.$client.options.debug = prior; }
expect([...new Set(checked)].sort()).toEqual(graph.affected);
expect(assigned.every(id => graph.affected.includes(id))).toBe(true);
await vi.advanceTimersByTimeAsync(31_000);
return statements.length;
}
const finished = () => (tx: object) => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: graph.runId });
const first = await lifecycleEvent(finished());
await lifecycleEvent(tx => notifyExecutionRunWork(tx, { companyId: f.companyId, runId: graph.runId, agentId: f.agentId }));
await lifecycleEvent(tx => notifyRunDispatch(tx, { companyId: f.companyId, issueId: graph.taskId }));
await lifecycleEvent(tx => promoteDeferredWorkIntents(tx, { companyId: f.companyId, trigger: "lock_released", issueId: graph.taskId }));
await lifecycleEvent(tx => promoteDeferredWorkIntents(tx, { companyId: f.companyId, trigger: "accounting_settled", runId: graph.runId }));
const recoveryActions = issueRecoveryActionService(state.db);
await lifecycleEvent(async () => {}, () => recoveryActions.upsertSourceScoped({ companyId: f.companyId, sourceIssueId: graph.taskId,
kind: "stranded_assigned_issue", ownerType: "board", cause: "stranded_assigned_issue", fingerprint: "run-failed",
nextAction: "Choose the next step", evidence: {} }));
await lifecycleEvent(async () => {}, () => recoveryActions.resolveActiveForIssue({ companyId: f.companyId, sourceIssueId: graph.taskId,
status: "resolved", outcome: "restored", resolutionNote: "Recovered" }));
await graph.backlog(4000);
expect(await lifecycleEvent(finished())).toBe(first);
});
it("re-checks a run's own tasks for its settlement, lease and dispatch writers, and nothing for a run without a task", async () => {
const f = await fixture();
const graph = await runGraph(f.companyId, f.agentId);
await graph.backlog(300);
const [taskless] = await state.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "succeeded",
finishedAt: new Date(), contextSnapshot: { source: "scheduler" } }).returning({ id: heartbeatRuns.id });
const checked: string[] = [];
const assigned: string[] = [];
await recordedRecovery({
stranded: async scope => { checked.push(scope.issueId); },
assignedWork: {
listSuspects: (scope, cursor, limit) => recoveryService(state.db, { enqueueWakeup: async () => null })
.assignedWork.listSuspects(scope, cursor, limit),
checkIssue: async scope => { assigned.push(scope.issueId); },
},
}, { backstopMs: 60 * 60_000 });
await settleWork({ backstopPending: true });
const writers = (runId: string): Array<(tx: object) => Promise<unknown>> => [
tx => notifyRunDispatch(tx, { companyId: f.companyId, runId }),
tx => notifyExecutionRunWork(tx, { companyId: f.companyId, runId }),
tx => notifyExternalConversationSettlement(tx, { companyId: f.companyId, runId }),
tx => notifyChatInboundWakeNotices(tx, { companyId: f.companyId, runId }),
tx => promoteDeferredWorkIntents(tx, { companyId: f.companyId, trigger: "lock_released", issueId: null, runId }),
];
for (const [runId, expected] of [[graph.runId, graph.affected], [taskless.id, []]] as Array<[string, string[]]>) {
for (const writer of writers(runId)) {
checked.length = 0; assigned.length = 0;
await state.db.transaction(tx => writer(tx));
await settleWork({ backstopPending: true });
expect([...new Set(checked)].sort()).toEqual(expected);
expect(assigned.every(id => expected.includes(id))).toBe(true);
await vi.advanceTimersByTimeAsync(31_000);
}
}
checked.length = 0; assigned.length = 0;
await issueService(state.db).addComment(graph.siblingId, "A note on the next task", { userId: "recovery-user" });
await settleWork({ backstopPending: true });
expect(checked).toEqual([graph.siblingId]);
expect(assigned.every(id => id === graph.siblingId)).toBe(true);
});
it("counts an agent's own run heartbeat as no change to its other tasks, and a resume as a change", async () => {
const f = await fixture();
const old = new Date(Date.now() - 60 * 60_000);
const tasks = await state.db.insert(issues).values(Array.from({ length: 5 }, (_, index) => ({ companyId: f.companyId,
title: `Waiting ${index}`, status: "todo", assigneeAgentId: f.agentId, createdAt: old, updatedAt: old }))).returning({ id: issues.id });
const since = new Date(Date.now() - 5 * 60_000);
const changed = async () => (await state.db.select({ id: issues.id }).from(issues)
.where(and(eq(issues.companyId, f.companyId), changedTaskCondition(since)))).map(row => row.id).sort();
const heartbeatAt = new Date();
await state.db.update(agents).set({ status: "idle", lastHeartbeatAt: heartbeatAt, updatedAt: heartbeatAt }).where(eq(agents.id, f.agentId));
expect(await changed()).toEqual([]);
await state.db.update(agents).set({ status: "idle", updatedAt: new Date(heartbeatAt.getTime() + 1000) }).where(eq(agents.id, f.agentId));
expect(await changed()).toEqual(tasks.map(row => row.id).sort());
});
it("counts an agent's run start as no change to its other tasks, and a pause during the run as a change", async () => {
const f = await fixture();
const old = new Date(Date.now() - 60 * 60_000);
await state.db.update(agents).set({ lastHeartbeatAt: old, updatedAt: old }).where(eq(agents.id, f.agentId));
const tasks = await state.db.insert(issues).values(Array.from({ length: 5 }, (_, index) => ({ companyId: f.companyId,
title: `Settled ${index}`, status: "in_review", assigneeAgentId: f.agentId, createdAt: old, updatedAt: old }))).returning({ id: issues.id });
const since = new Date(Date.now() - 5 * 60_000);
const changed = async () => (await state.db.select({ id: issues.id }).from(issues)
.where(and(eq(issues.companyId, f.companyId), changedTaskCondition(since)))).map(row => row.id).sort();
// The write of execution start in heartbeat.ts: a status flip with no heartbeat time.
await state.db.update(agents).set({ status: "running", updatedAt: new Date() }).where(eq(agents.id, f.agentId));
expect(await changed()).toEqual([]);
await state.db.update(agents).set({ status: "paused", updatedAt: new Date() }).where(eq(agents.id, f.agentId));
expect(await changed()).toEqual(tasks.map(row => row.id).sort());
});
it("resolves a run without a task to no tasks and a missing task to the whole company", async () => {
const f = await fixture();
const [taskless] = await state.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running",
contextSnapshot: { source: "scheduler" } }).returning({ id: heartbeatRuns.id });
expect(await resolveIssueGraph(state.db, { companyId: f.companyId, runId: taskless.id })).toEqual([]);
expect(await resolveIssueGraph(state.db, { companyId: f.companyId, runId: randomUUID() })).toEqual([]);
expect(await resolveIssueGraph(state.db, { companyId: f.companyId, issueId: randomUUID() })).toBeNull();
});
it("resolves a run's task graph through index lookups, not the backlog", async () => {
const f = await fixture();
const graph = await runGraph(f.companyId, f.agentId);
await graph.backlog(2000);
await state.db.execute(sql`analyze issues`);
await state.db.execute(sql`analyze heartbeat_runs`);
const [read] = await state.db.transaction(async tx => {
expect(await resolveIssueGraph(tx as unknown as typeof state.db, { companyId: f.companyId, runId: graph.runId })).toEqual(graph.affected);
return tx.execute<{ rows: number }>(sql`select (coalesce(seq_tup_read, 0) + coalesce(idx_tup_fetch, 0))::int as rows
from pg_stat_xact_user_tables where relname = 'issues'`);
});
expect(read.rows).toBeLessThan(50);
});
it("finds a stranded task outside the affected set within the backstop bound", async () => {
const f = await fixture();
const graph = await runGraph(f.companyId, f.agentId);
await graph.backlog(50);
const checkedAt = new Map<string, number>();
await recordedRecovery({ stranded: async scope => { checkedAt.set(scope.issueId, Date.now()); } });
await settleWork({ backstopPending: true });
const fullPassAt = Date.now();
const [unrelated] = await graph.backlog(1);
checkedAt.clear();
await state.db.transaction(tx => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: graph.runId }));
await settleWork({ backstopPending: true });
expect(checkedAt.has(unrelated)).toBe(false);
expect(state.coordinator.nextWakeAt()).toBeLessThanOrEqual(fullPassAt + ISSUE_RECOVERY_BACKSTOP_MS);
await vi.advanceTimersByTimeAsync(ISSUE_RECOVERY_BACKSTOP_MS);
await settleWork();
expect(checkedAt.get(unrelated)! - fullPassAt).toBeLessThanOrEqual(ISSUE_RECOVERY_BACKSTOP_MS + 5_000);
checkedAt.clear();
await vi.advanceTimersByTimeAsync(20 * 60_000);
expect(checkedAt.size).toBe(0);
const [later] = await graph.backlog(1);
const eventAt = Date.now();
await state.db.transaction(tx => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: graph.runId }));
await settleWork();
expect(checkedAt.get(later)! - eventAt).toBeLessThanOrEqual(5_000);
await quiet();
});
async function advanceWithoutWaiting(ms: number) {
const until = Date.now() + ms;
while (Date.now() < until) { await vi.advanceTimersByTimeAsync(Math.min(1000, until - Date.now())); await immediate(); }
}
it("keeps a saved deadline of a task that a changed-tasks backstop pass does not load", async () => {
const f = await fixture();
const graph = await runGraph(f.companyId, f.agentId);
const [target] = await graph.backlog(20);
const checkedAt: number[] = [];
await recordedRecovery({ stranded: async scope => {
if (scope.issueId !== target) return;
checkedAt.push(Date.now());
scope.onDeferredDeadline(Date.now() + 16 * 60_000);
} });
await settleWork({ backstopPending: true });
for (let round = 0; round < 2; round++) {
await vi.advanceTimersByTimeAsync(ISSUE_RECOVERY_BACKSTOP_MS + 60_000);
await state.db.transaction(tx => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: graph.runId }));
await settleWork({ backstopPending: true });
}
const savedFor = checkedAt.at(-1)! + 16 * 60_000;
checkedAt.length = 0;
await vi.advanceTimersByTimeAsync(savedFor - Date.now() + 2_000);
await settleWork({ backstopPending: true });
expect(checkedAt.length).toBeGreaterThan(0);
expect(checkedAt[0]! - savedFor).toBeLessThanOrEqual(5_000);
});
it("starts the next changed-tasks window from the start of a slow backstop scan", async () => {
const f = await fixture();
const graph = await runGraph(f.companyId, f.agentId);
const backlog = (await graph.backlog(250)).sort();
const behind = backlog[0]!;
let slow = false, changedBehind = false;
const checkedAt = new Map<string, number[]>();
await recordedRecovery({ stranded: async scope => {
checkedAt.set(scope.issueId, [...checkedAt.get(scope.issueId) ?? [], Date.now()]);
if (!slow) return;
vi.setSystemTime(Date.now() + 20_000);
if (scope.issueId === behind && !changedBehind) {
changedBehind = true;
await state.db.update(issues).set({ updatedAt: new Date() }).where(eq(issues.id, behind));
}
} }, { fullBackstopMs: 24 * 60 * 60_000 });
await settleWork({ backstopPending: true });
await vi.advanceTimersByTimeAsync(ISSUE_RECOVERY_BACKSTOP_MS + 60_000);
await state.db.update(issues).set({ updatedAt: new Date() }).where(inArray(issues.id, backlog));
await vi.advanceTimersByTimeAsync(ISSUE_RECOVERY_BACKSTOP_MS + 60_000);
slow = true;
checkedAt.clear();
const scanStartedAt = Date.now();
await state.db.transaction(tx => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: graph.runId }));
await settleWork({ backstopPending: true });
slow = false;
expect(changedBehind).toBe(true);
expect(backlog.every(id => checkedAt.has(id))).toBe(true);
expect(Date.now() - scanStartedAt).toBeGreaterThan(2 * ISSUE_RECOVERY_BACKSTOP_MS);
checkedAt.clear();
await vi.advanceTimersByTimeAsync(ISSUE_RECOVERY_BACKSTOP_MS + 60_000);
await state.db.transaction(tx => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: graph.runId }));
await settleWork({ backstopPending: true });
expect(checkedAt.has(behind)).toBe(true);
}, 300_000);
it("keeps the company pass for hold releases that can admit any task", async () => {
const f = await fixture();
const graph = await runGraph(f.companyId, f.agentId);
const backlog = await graph.backlog(150);
const checked: string[] = [];
await recordedRecovery({ stranded: async scope => { checked.push(scope.issueId); } });
await settleWork({ backstopPending: true });
const everything = new Set([...graph.affected, graph.siblingId, ...backlog]);
await state.db.insert(budgetPolicies).values({ companyId: f.companyId, scopeType: "company", scopeId: f.companyId,
metric: "billed_cents", windowKind: "calendar_month_utc", amount: 1000, hardStopEnabled: true, isActive: true });
for (const input of [{ trigger: "accounting_settled" as const, runId: graph.runId }, { trigger: "pause_lifted" as const }]) {
checked.length = 0;
await state.db.transaction(tx => promoteDeferredWorkIntents(tx, { companyId: f.companyId, ...input }));
await settleWork({ backstopPending: true });
expect(new Set(checked)).toEqual(everything);
await vi.advanceTimersByTimeAsync(31_000);
}
});
it("falls back to the company pass when a run affects more tasks than a hint names", async () => {
const f = await fixture();
const graph = await runGraph(f.companyId, f.agentId);
const backlog = await graph.backlog(200);
const checked: string[] = [];
await recordedRecovery({ stranded: async scope => { checked.push(scope.issueId); } });
await settleWork({ backstopPending: true });
const dependents = await graph.backlog(ISSUE_RECOVERY_MAX_ISSUE_IDS);
await state.db.insert(issueRelations).values(dependents.map(id => ({ companyId: f.companyId, issueId: graph.taskId,
relatedIssueId: id, type: "blocks" as const })));
checked.length = 0;
await state.db.transaction(tx => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: graph.runId }));
await settleWork({ backstopPending: true });
expect(new Set(checked)).toEqual(new Set([...graph.affected, graph.siblingId, ...backlog, ...dependents]));
});
it("checks only blocked tasks whose blockers are done, however many still wait", async () => {
const f = await fixture();
const [openBlocker, doneBlocker] = await state.db.insert(issues).values([
{ companyId: f.companyId, title: "Open blocker", status: "in_progress", assigneeAgentId: f.agentId },
{ companyId: f.companyId, title: "Done blocker", status: "done" },
]).returning({ id: issues.id });
const waiting = await state.db.insert(issues).values(Array.from({ length: 300 }, (_, index) => ({
companyId: f.companyId, title: `Waiting ${index}`, status: "blocked", assigneeAgentId: f.agentId,
}))).returning({ id: issues.id });
const [ready] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Ready", status: "blocked",
assigneeAgentId: f.agentId }).returning({ id: issues.id });
await state.db.insert(issueRelations).values([
...waiting.map(row => ({ companyId: f.companyId, issueId: openBlocker.id, relatedIssueId: row.id, type: "blocks" as const })),
{ companyId: f.companyId, issueId: doneBlocker.id, relatedIssueId: ready.id, type: "blocks" as const },
]);
const checked: string[] = [];
await recordedRecovery({ dependencies: async scope => { checked.push(scope.issueId); return { enqueueFailed: 0 }; } });
await settleWork();
expect(checked).toEqual([ready.id]);
checked.length = 0;
await state.db.transaction(tx => notifyIssueRecovery(tx, { companyId: f.companyId }));
await settleWork();
expect(checked).toEqual([ready.id]);
checked.length = 0;
await issueService(state.db).update(openBlocker.id, { status: "done" });
await settleWork({ backstopPending: true });
expect(new Set(checked)).toEqual(new Set([ready.id, ...waiting.map(row => row.id)]));
await vi.advanceTimersByTimeAsync(ISSUE_RECOVERY_BACKSTOP_MS);
await settleWork();
await quiet();
});
it("uses ordinary watchdog configuration and subtree closure to dispatch a reusable review", async () => {
const f = await fixture();
const source = await issueService(state.db).create(f.companyId, { title: "Finished subtree", status: "done" });
const failures = await completingProvider(f, () => taskWatchdogService(state.db).disableForIssue(f.companyId, source.id));
await start(); await quiet();
await compete();
const watchdog = await taskWatchdogService(state.db).upsertForIssue(f.companyId, source.id, { agentId: f.agentId });
let reviewId: string | undefined;
await vi.waitFor(async () => {
const [row] = await state.db.select().from(issueWatchdogs).where(eq(issueWatchdogs.id, watchdog.watchdog.id));
expect(row.watchdogIssueId).toBeTruthy(); reviewId = row.watchdogIssueId!;
}, { timeout: 15_000 });
const review = await completed(reviewId!);
expect(failures).toEqual([]);
// Ordinary assignment recovery can win the wake reason while the watchdog
// wake coalesces into that run. Verify its mandate and one dispatch instead.
expect(review.contextSnapshot).toMatchObject({
watchdogId: watchdog.watchdog.id,
taskWatchdog: {
watchedIssueId: source.id,
stopFingerprint: expect.stringMatching(/^task_watchdog_stop:/),
},
});
expect(await state.db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${reviewId}`))).toHaveLength(1);
await finish(reviewId!);
await taskWatchdogService(state.db).disableForIssue(f.companyId, source.id);
if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); await quiet();
}, 90_000);
it("bounds ordinary writes to their company and reconstructs foreign stale debt after restart", async () => {
const selected = await fixture(); await start(); await quiet();
const foreign = await fixture();
const [run] = await state.db.insert(heartbeatRuns).values({ companyId: foreign.companyId, agentId: foreign.agentId,
status: "failed", finishedAt: new Date() }).returning();
const [stale] = await state.db.insert(issues).values({ companyId: foreign.companyId, title: "Foreign stale lock", status: "done",
checkoutRunId: run.id }).returning();
await issueService(state.db).create(selected.companyId, { title: "Selected change", status: "done" }); await quiet();
expect((await state.db.select().from(issues).where(eq(issues.id, stale.id)))[0].checkoutRunId).toBe(run.id);
await state.worker.stop(); await state.dispatch.stop(); await state.cancellation.stop(); await state.coordinator.stop();
state.heartbeat = heartbeatService(state.db); await start(); await quiet();
expect((await state.db.select().from(issues).where(eq(issues.id, stale.id)))[0].checkoutRunId).toBeNull();
});
it("retains a fresh subtree's exact first-run grace then disarms on disable", async () => {
const f = await fixture();
const source = await issueService(state.db).create(f.companyId, { title: "Fresh unassigned subtree", status: "todo" });
await start(); await quiet();
await taskWatchdogService(state.db).upsertForIssue(f.companyId, source.id, { agentId: f.agentId });
await vi.waitFor(() => expect(state.coordinator.nextWakeAt()).toBe(source.createdAt.getTime() + 15_000), { timeout: 5_000 });
expect(await state.db.select().from(issues).where(eq(issues.originKind, "task_watchdog"))).toHaveLength(0);
await taskWatchdogService(state.db).disableForIssue(f.companyId, source.id);
if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); await quiet();
});
it("waits for a live foreign controller's exact stored lease before clearing a terminal lock", async () => {
const f = await fixture();
const source = await issueService(state.db).create(f.companyId, { title: "Remote owner", status: "done" });
const lease = new Date(Date.now() + 5_000);
const [run] = await state.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running",
runtimeMode: "legacy", controllerBootId: randomUUID(), controllerLeaseExpiresAt: lease,
contextSnapshot: { issueId: source.id }, processPid: 2_000_000_000, startedAt: new Date() }).returning();
await state.db.update(issues).set({ executionRunId: run.id }).where(eq(issues.id, source.id));
await start();
await vi.waitFor(() => expect(state.coordinator.nextWakeAt()).toBe(lease.getTime()), { timeout: 3_000 });
expect((await state.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].status).toBe("running");
await vi.waitFor(async () => expect((await state.db.select().from(issues).where(eq(issues.id, source.id)))[0].executionRunId).toBeNull(), { timeout: 12_000 });
expect((await state.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].status).toBe("succeeded");
if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); await quiet();
}, 90_000);
});