mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task text contains links to pull requests and other work. > - A link has useful status that should be visible when a task opens. > - The current cache lives in the server process and the browser. > - A new browser or server restart has no stored status to show. > - This pull request stores link status in the database and reads it in batches. > - Idle page work updates stale status without a hover or a delay to task data. ## Linked Issues or Issue Description **Problem or motivation** PR status is unknown in a new browser until the user points to each link. The process cache cannot supply saved status after a server restart. Related work: #20593 and #20654. A search found no other PR for database link caching. **Proposed solution** Store status by company, viewer and canonical resource URL. Read saved status in one indexed batch query. After document load and core request settlement, use browser idle time to update missing or stale status. Use the same query for hover and idle work. **Alternatives considered** Adding provider calls to task reads would delay task loading. Company-wide reuse of personal GitHub results would cross viewer access boundaries. A periodic server worker would create work when no page needs the result. **Roadmap alignment** This extends the shipped rich-link feature. It does not duplicate a roadmap project or add a new provider integration. ## What Changed - Add an additive, repeat-safe `link_previews` migration. Store channel, resource kind, free-form status, JSON metadata and freshness. The table has no task relationship or GitHub status enum. - Bound each company/viewer cache to 1,000 rows across concurrent server instances. Remove rows expired for over 30 days during writes. Limit provider lookups to 600 requests per minute per scope/process. Keep cached batch reads available during backoff. - Add a database store owned by the link service. Reuse canonical results across tasks. Keep company and viewer isolation. Save hover and idle results. Preserve last known status during an outage and cache unavailable results. - Add a board-only batch GET endpoint, including viewer-role access. It reads at most 50 URLs with one indexed select and makes no provider calls. Both endpoints use private, no-store HTTP responses. - Add a page coordinator for mounted rich links. Reuse browser data, share pending reads, and defer provider work until document load, core request settlement and browser idle time. Limit automatic work to 200 unique links and three requests in flight. Remove queued work when links leave the page. Keep encoded batch URLs below 6 KB. - Keep exact navigation URLs, status styling, anchored cards and keyboard access. Add a page-load Storybook case and browser regression. The link service owns cache policy and provider calls. The browser coordinator owns page timing. The shared contract adds only the batch limit and response shape. The database adds storage because process and browser caches cannot serve a new browser session. Plugin resolver registration is outside this change; storage accepts future provider channels and status values. ## Verification - **All 93 affected tests pass:** 40 link service, real PostgreSQL and HTTP cases, plus 53 OpenAPI cases. Two SQL regressions and one HTTP case failed before the storage repair. Concurrent writers now retain at most 1,000 rows per company/viewer; cleanup removes long-expired rows and keeps recent stale status. Request backoff preserves cached reads and isolates companies/viewers. The earlier eight concurrent hydration regressions also pass. Focused store/service/unit types pass. Commands: `pnpm exec vitest run server/src/__tests__/link-previews.test.ts server/src/__tests__/link-preview-store.integration.test.ts server/src/__tests__/link-preview-routes.test.ts --maxWorkers=1` and `pnpm exec vitest run server/src/__tests__/openapi-routes.test.ts --maxWorkers=1`. - **All 50 affected server tests passed before the later server-only repairs** on the master-integrated source: 20 recovery/deadline cases, one authenticated review-counts HTTP case, 10 link-service cases, 13 real PostgreSQL cache cases and six link-route cases. Command: `pnpm exec vitest run server/src/__tests__/issue-recovery-deadlines.test.ts server/src/__tests__/tool-connection-review-counts-http.test.ts server/src/__tests__/link-previews.test.ts server/src/__tests__/link-preview-store.integration.test.ts server/src/__tests__/link-preview-routes.test.ts --maxWorkers=1`. - Reproduced the CI watchdog assertion locally twice before the test repair. Ordinary assignment recovery can win `wakeReason` while the watchdog wake coalesces into the same successful run with its full mandate. The test now verifies the persisted watchdog identity, watched issue and stopped fingerprint, plus exactly one review run. All 21 recovery/HTTP cases passed before integrating master; all 50 affected cases pass after integration. No runtime behavior was changed for this repair. - **Fresh static Storybook build and all 22 browser cases pass**, with two workers and retries disabled. They cover states, anchored hover, keyboard access, exact navigation, duplicate requests, reload persistence, expiry and saved-status hydration followed by one idle refresh. The first browser attempt could not launch because this session lacked Chromium; installing the expected browser resolved setup. That attempt had no behavioral verdict. - The cache repair remains covered by four PostgreSQL red/green regressions for expired memory and newer durable status during provider failure/timeout, plus check-time precedence and failed-read recovery. Earlier UI/database/focused server type and migration safety/compatibility/replay checks remain unchanged-input evidence. - Master `77ddf260bf9a3de3dcb8456a60f4f809d67401fd` is integrated cleanly. Link migration 0489 and its snapshot chain are unchanged. SDK source validation checks 545 inputs with no stale source identities or changed runtime/lock pins. Whitespace validation passes. - `pnpm verify --explain` returns exit 2 and requires broad CI. No full local suite or deployed acceptance is claimed. Earlier local full-server type checks exceeded memory; current-head CI supplies broad type/build/test coverage. - [Previous CI 43900336895](https://github.com/paperclipai/paperclip/actions/runs/43900336895) passed full types/build but failed on embedded PostgreSQL startup (`ECONNREFUSED`), the watchdog assertion above, and unrelated plan-task browser cleanup (activity-log foreign key on company deletion). One failed-jobs retry was requested. Its browser shard 7 now passes without code changes; the exact deletion-race cause remains unproven. The watchdog assertion is repaired in this revision. The previous failures remain recorded, not counted as an overall pass. - [Prior CI 43905482848](https://github.com/paperclipai/paperclip/actions/runs/43905482848) passed full types, build and all eight browser shards. Several test jobs were cancelled; the run has no overall pass. Its concurrent update finding is repaired. [Next CI 43909197530](https://github.com/paperclipai/paperclip/actions/runs/43909197530) also passed types, build and all browser shards, but several server jobs were cancelled. Greptile and Paperclip Review gave that revision 5/5. Its security scan found unbounded retained cache rows. The quota, retention and request-limit repair above addresses that finding. All ten review threads are resolved. - [Current CI 43912947278](https://github.com/paperclipai/paperclip/actions/runs/43912947278), fresh code reviews and a fresh security scan are pending on `335d89ef1668dcdeb66bb2059512a7c2da29e273`. Auto-merge is disabled until the current revision clears its gates. Published head: `335d89ef1668dcdeb66bb2059512a7c2da29e273`. Local head: `b37b87c704a2ba8a65233c689a6a743f4011ce14`. Both have tree `643ebd19b9729b1e6a6bb695442269c3db32be3a`; GitHub tree creation verified equality. ## Risks - Each company/viewer retains at most 1,000 rows. Writes remove excess rows and entries whose freshness expired over 30 days ago. Inactive scopes remain bounded and are cleaned on the next write. Company deletion removes all rows. No periodic worker runs. Scope writes use a transaction lock; the existing deadline and concurrency budget cover that work. - Last known status can be stale until idle or hover work completes. Provider failures keep that status through the negative-cache window. - Automatic enrichment adds bounded provider traffic. It runs only for mounted supported links, once per mount, with the existing timeout and server concurrency limit. - Cache storage is viewer-scoped because provider results can be private. New viewers load their own results. This prevents reuse across users with different repository access. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools, code execution and browser automation. The runtime does not expose the exact serving model ID or context-window size. No subagents were used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
500 lines
33 KiB
TypeScript
500 lines
33 KiB
TypeScript
import { setImmediate as immediate } from "node:timers/promises";
|
|
import { idleWorkSnapshot } from "../services/task-admission.js";
|
|
import { randomUUID } from "node:crypto";
|
|
import { and, eq, inArray, sql } from "drizzle-orm";
|
|
import { expect, it, vi } from "vitest";
|
|
import { agents, budgetPolicies, heartbeatRuns, issueRecoveryActions, issueRelations, issues, issueWatchdogs } from "@paperclipai/db";
|
|
import { heartbeatService } from "../services/heartbeat.js";
|
|
import { issueService } from "../services/issues.js";
|
|
import { issueRecoveryActionService } from "../services/issue-recovery-actions.js";
|
|
import { registerIssueRecoveryWork } from "../services/recovery/work.js";
|
|
import { changedTaskCondition } from "../services/recovery/changed-tasks.js";
|
|
import { recoveryService } from "../services/recovery/service.js";
|
|
import { ISSUE_RECOVERY_BACKSTOP_MS, ISSUE_RECOVERY_MAX_ISSUE_IDS, notifyIssueRecovery, resolveIssueGraph }
|
|
from "../services/issue-recovery-notifications.js";
|
|
import { notifyNativeStatusReassessment } from "../services/native-status-reassessment-notifications.js";
|
|
import { notifyExecutionRunWork } from "../services/execution-control-notifications.js";
|
|
import { notifyRunDispatch } from "../services/run-dispatch-notifications.js";
|
|
import { notifyExternalConversationSettlement } from "../services/external-conversation-settlement-notifications.js";
|
|
import { notifyChatInboundWakeNotices } from "../services/chat-inbound-wake-notice-notifications.js";
|
|
import { promoteDeferredWorkIntents } from "../services/work-intent-outbox.js";
|
|
import { taskWatchdogService } from "../services/task-watchdogs.js";
|
|
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
|
import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js";
|
|
|
|
describeIssueRecovery("issue recovery production producers and deadlines", () => {
|
|
const state = issueRecoveryFixture();
|
|
const { start, compete, fixture, completingProvider, settled, quiet, finish, completed } = state;
|
|
it("keeps empty, configured idle, and terminal history quiet", async () => {
|
|
await fixture(); await start(); await quiet();
|
|
});
|
|
it("creates a blocking child through the ordinary root service transaction", async () => {
|
|
const f = await fixture();
|
|
const parent = await issueService(state.db).create(f.companyId, { title: "Parent", status: "todo" });
|
|
const child = await issueService(state.db).createChild(parent.id, { title: "Blocking child", status: "todo", blockParentUntilDone: true });
|
|
expect(child.parentBlockerAdded).toBe(true);
|
|
expect(await state.db.select().from(issueRelations).where(and(eq(issueRelations.companyId, f.companyId),
|
|
eq(issueRelations.issueId, child.issue.id), eq(issueRelations.relatedIssueId, parent.id)))).toHaveLength(1);
|
|
});
|
|
it("bounds subsidiary scans for100 assignment candidates and coalesces repeated company writes", async () => {
|
|
const f = await fixture(), foreign = await fixture();
|
|
await state.db.update(agents).set({ lifecycleState: "preparing" }).where(eq(agents.id, f.agentId));
|
|
await state.db.insert(issues).values(Array.from({ length: 100 }, (_, index) => ({ companyId: f.companyId,
|
|
title: `Waiting setup ${index}`, status: "todo", assigneeAgentId: f.agentId })));
|
|
const [source] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Done source", status: "done" }).returning();
|
|
const action = await issueRecoveryActionService(state.db).upsertSourceScoped({ companyId: f.companyId, sourceIssueId: source.id,
|
|
kind: "stranded_assigned_issue", ownerType: "agent", ownerAgentId: f.agentId, cause: "stranded_assigned_issue",
|
|
fingerprint: "query-budget", nextAction: "Restore work", wakePolicy: { type: "wake_owner" }, evidence: {} });
|
|
const scans = { blockers: 0, actions: 0, loads: 0 };
|
|
const originalSelect = state.db.select.bind(state.db);
|
|
vi.spyOn(state.db, "select").mockImplementation(((fields?: Record<string, unknown>) => {
|
|
const query = originalSelect(fields as never); const from = query.from.bind(query);
|
|
query.from = ((table: unknown) => {
|
|
if (table === issues && fields?.id === issues.id && fields.companyId === issues.companyId && Object.keys(fields).length === 2) scans.loads++;
|
|
if (table === issueRelations && fields?.createdByAgentId === issues.createdByAgentId) scans.blockers++;
|
|
if (table === issueRecoveryActions && fields?.action === issueRecoveryActions) scans.actions++;
|
|
return from(table as never);
|
|
}) as typeof query.from;
|
|
return query;
|
|
}) as typeof state.db.select);
|
|
await start(); await settled();
|
|
expect(scans.actions).toBe(1); expect(scans.blockers).toBe(0); expect(scans.loads).toBeLessThanOrEqual(11);
|
|
expect((await originalSelect().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id)))[0].status).toBe("resolved");
|
|
const [foreignSource] = await state.db.insert(issues).values({ companyId: foreign.companyId, title: "Foreign done", status: "done" }).returning();
|
|
const [foreignAction] = await state.db.insert(issueRecoveryActions).values({ companyId: foreign.companyId, sourceIssueId: foreignSource.id,
|
|
kind: "stranded_assigned_issue", ownerType: "board", cause: "stranded_assigned_issue", fingerprint: "foreign-query-budget", nextAction: "Inspect" }).returning();
|
|
await state.db.transaction(tx => notifyIssueRecovery(tx, { companyId: f.companyId })); await settled();
|
|
const before = { ...scans };
|
|
for (let index = 0; index < 100; index++) await state.db.transaction(tx => notifyIssueRecovery(tx, { companyId: f.companyId }));
|
|
await new Promise(resolve => setTimeout(resolve, 100));
|
|
expect(scans.loads).toBe(before.loads);
|
|
await settled();
|
|
expect(scans.actions).toBe(before.actions); expect(scans.blockers).toBe(before.blockers);
|
|
expect(scans.loads).toBeLessThanOrEqual(before.loads + 4);
|
|
expect((await originalSelect().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, foreignAction.id)))[0].status).toBe("active");
|
|
expect(await originalSelect().from(heartbeatRuns)).toHaveLength(0); await quiet();
|
|
});
|
|
async function recordedRecovery(owner: Partial<Parameters<typeof registerIssueRecoveryWork>[2]>,
|
|
options?: Parameters<typeof registerIssueRecoveryWork>[4]) {
|
|
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
|
|
state.coordinator = createDeliveryWorkCoordinator({ owner: state.db, canRun: () => true, onError: error => state.errors.push(error) });
|
|
const register = state.coordinator.register.bind(state.coordinator);
|
|
state.coordinator.register = (queue, task) => register(queue, { ...task, run: signal => {
|
|
const pass = task.run(signal); state.passes.add(pass);
|
|
void pass.then(() => state.passes.delete(pass), () => state.passes.delete(pass));
|
|
return pass;
|
|
} });
|
|
state.worker = registerIssueRecoveryWork(state.db, state.coordinator, {
|
|
stranded: async () => {}, orphanBlockers: async () => {}, recoveryActions: async () => {},
|
|
dependencies: async () => ({ enqueueFailed: 0 }), locks: async () => {},
|
|
watchdogs: async () => ({ incomplete: 0 }), schedulingSuppressed: async () => false, ...owner,
|
|
}, () => true, options);
|
|
await state.worker.ready;
|
|
}
|
|
async function settleWork(options: { backstopPending?: boolean } = {}) {
|
|
for (let step = 0; step < 120; step++) {
|
|
await vi.advanceTimersByTimeAsync(1000);
|
|
await Promise.allSettled([...state.passes]); await immediate();
|
|
const next = state.coordinator.nextWakeAt();
|
|
const settledNow = next === null || (options.backstopPending === true && next > Date.now() + 60_000);
|
|
if (settledNow && idleWorkSnapshot().active === 0) return;
|
|
}
|
|
throw new Error("Recovery did not settle its committed work within two virtual minutes");
|
|
}
|
|
async function runGraph(companyId: string, agentId: string) {
|
|
const [other] = await state.db.insert(agents).values({ companyId, name: "Other agent", role: "engineer", status: "idle",
|
|
adapterType: "process", adapterConfig: {} }).returning({ id: agents.id });
|
|
const [parent] = await state.db.insert(issues).values({ companyId, title: "Parent", status: "in_progress", assigneeAgentId: agentId })
|
|
.returning({ id: issues.id });
|
|
const [task] = await state.db.insert(issues).values({ companyId, title: "Run task", status: "in_progress", assigneeAgentId: agentId,
|
|
parentId: parent.id }).returning({ id: issues.id });
|
|
const [dependent] = await state.db.insert(issues).values({ companyId, title: "Waits on the run task", status: "todo",
|
|
assigneeAgentId: other.id }).returning({ id: issues.id });
|
|
const [sibling] = await state.db.insert(issues).values({ companyId, title: "Next task of the agent", status: "todo",
|
|
assigneeAgentId: agentId }).returning({ id: issues.id });
|
|
await state.db.insert(issueRelations).values({ companyId, issueId: task.id, relatedIssueId: dependent.id, type: "blocks" });
|
|
const [run] = await state.db.insert(heartbeatRuns).values({ companyId, agentId, status: "failed", finishedAt: new Date(),
|
|
contextSnapshot: { issueId: task.id } }).returning({ id: heartbeatRuns.id });
|
|
const backlog = async (count: number) => (await state.db.insert(issues).values(Array.from({ length: count }, (_, index) => ({
|
|
companyId, title: `Backlog ${index}`, status: "todo", assigneeAgentId: other.id, createdAt: new Date(), updatedAt: new Date(),
|
|
}))).returning({ id: issues.id })).map(row => row.id);
|
|
return { runId: run.id, taskId: task.id, siblingId: sibling.id, backlog, affected: [parent.id, task.id, dependent.id].sort() };
|
|
}
|
|
it("limits an ordinary comment to its issue among one thousand open assignments", async () => {
|
|
const f = await fixture();
|
|
const rows = await state.db.insert(issues).values(Array.from({ length: 1000 }, (_, index) => ({
|
|
companyId: f.companyId, title: `Open work ${index}`, status: "todo", assigneeAgentId: f.agentId,
|
|
}))).returning({ id: issues.id });
|
|
const checked: string[] = [];
|
|
await recordedRecovery({ stranded: async scope => { checked.push(scope.issueId); } });
|
|
await settleWork();
|
|
expect(new Set(checked).size).toBe(1000);
|
|
checked.length = 0;
|
|
await issueService(state.db).addComment(rows[0].id, "New evidence on this task", { userId: "recovery-user" });
|
|
await settleWork({ backstopPending: true });
|
|
expect(checked).toEqual([rows[0].id]);
|
|
checked.length = 0;
|
|
await state.db.transaction(tx => notifyIssueRecovery(tx, { companyId: f.companyId }));
|
|
await settleWork({ backstopPending: true });
|
|
expect(new Set(checked).size).toBe(1000);
|
|
await vi.advanceTimersByTimeAsync(ISSUE_RECOVERY_BACKSTOP_MS);
|
|
await settleWork();
|
|
await quiet();
|
|
});
|
|
it("re-checks only the tasks a run lifecycle event can affect, whatever the backlog", async () => {
|
|
const f = await fixture();
|
|
const graph = await runGraph(f.companyId, f.agentId);
|
|
await graph.backlog(1000);
|
|
const checked: string[] = [];
|
|
const assigned: string[] = [];
|
|
await recordedRecovery({
|
|
stranded: async scope => { checked.push(scope.issueId); },
|
|
assignedWork: {
|
|
listSuspects: (scope, cursor, limit) => recoveryService(state.db, { enqueueWakeup: async () => null })
|
|
.assignedWork.listSuspects(scope, cursor, limit),
|
|
checkIssue: async scope => { assigned.push(scope.issueId); },
|
|
},
|
|
}, { backstopMs: 60 * 60_000 });
|
|
await settleWork({ backstopPending: true });
|
|
expect(new Set(checked).size).toBe(1004);
|
|
expect(checked).toContain(graph.siblingId);
|
|
async function lifecycleEvent(producer: (tx: object) => Promise<unknown>, standalone?: () => Promise<unknown>) {
|
|
checked.length = 0; assigned.length = 0;
|
|
const statements: string[] = [], prior = state.db.$client.options.debug;
|
|
state.db.$client.options.debug = (_connection, query) => { statements.push(query); };
|
|
try {
|
|
if (standalone) await standalone(); else await state.db.transaction(tx => producer(tx));
|
|
await settleWork({ backstopPending: true });
|
|
} finally { state.db.$client.options.debug = prior; }
|
|
expect([...new Set(checked)].sort()).toEqual(graph.affected);
|
|
expect(assigned.every(id => graph.affected.includes(id))).toBe(true);
|
|
await vi.advanceTimersByTimeAsync(31_000);
|
|
return statements.length;
|
|
}
|
|
const finished = () => (tx: object) => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: graph.runId });
|
|
const first = await lifecycleEvent(finished());
|
|
await lifecycleEvent(tx => notifyExecutionRunWork(tx, { companyId: f.companyId, runId: graph.runId, agentId: f.agentId }));
|
|
await lifecycleEvent(tx => notifyRunDispatch(tx, { companyId: f.companyId, issueId: graph.taskId }));
|
|
await lifecycleEvent(tx => promoteDeferredWorkIntents(tx, { companyId: f.companyId, trigger: "lock_released", issueId: graph.taskId }));
|
|
await lifecycleEvent(tx => promoteDeferredWorkIntents(tx, { companyId: f.companyId, trigger: "accounting_settled", runId: graph.runId }));
|
|
const recoveryActions = issueRecoveryActionService(state.db);
|
|
await lifecycleEvent(async () => {}, () => recoveryActions.upsertSourceScoped({ companyId: f.companyId, sourceIssueId: graph.taskId,
|
|
kind: "stranded_assigned_issue", ownerType: "board", cause: "stranded_assigned_issue", fingerprint: "run-failed",
|
|
nextAction: "Choose the next step", evidence: {} }));
|
|
await lifecycleEvent(async () => {}, () => recoveryActions.resolveActiveForIssue({ companyId: f.companyId, sourceIssueId: graph.taskId,
|
|
status: "resolved", outcome: "restored", resolutionNote: "Recovered" }));
|
|
await graph.backlog(4000);
|
|
expect(await lifecycleEvent(finished())).toBe(first);
|
|
});
|
|
it("re-checks a run's own tasks for its settlement, lease and dispatch writers, and nothing for a run without a task", async () => {
|
|
const f = await fixture();
|
|
const graph = await runGraph(f.companyId, f.agentId);
|
|
await graph.backlog(300);
|
|
const [taskless] = await state.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "succeeded",
|
|
finishedAt: new Date(), contextSnapshot: { source: "scheduler" } }).returning({ id: heartbeatRuns.id });
|
|
const checked: string[] = [];
|
|
const assigned: string[] = [];
|
|
await recordedRecovery({
|
|
stranded: async scope => { checked.push(scope.issueId); },
|
|
assignedWork: {
|
|
listSuspects: (scope, cursor, limit) => recoveryService(state.db, { enqueueWakeup: async () => null })
|
|
.assignedWork.listSuspects(scope, cursor, limit),
|
|
checkIssue: async scope => { assigned.push(scope.issueId); },
|
|
},
|
|
}, { backstopMs: 60 * 60_000 });
|
|
await settleWork({ backstopPending: true });
|
|
const writers = (runId: string): Array<(tx: object) => Promise<unknown>> => [
|
|
tx => notifyRunDispatch(tx, { companyId: f.companyId, runId }),
|
|
tx => notifyExecutionRunWork(tx, { companyId: f.companyId, runId }),
|
|
tx => notifyExternalConversationSettlement(tx, { companyId: f.companyId, runId }),
|
|
tx => notifyChatInboundWakeNotices(tx, { companyId: f.companyId, runId }),
|
|
tx => promoteDeferredWorkIntents(tx, { companyId: f.companyId, trigger: "lock_released", issueId: null, runId }),
|
|
];
|
|
for (const [runId, expected] of [[graph.runId, graph.affected], [taskless.id, []]] as Array<[string, string[]]>) {
|
|
for (const writer of writers(runId)) {
|
|
checked.length = 0; assigned.length = 0;
|
|
await state.db.transaction(tx => writer(tx));
|
|
await settleWork({ backstopPending: true });
|
|
expect([...new Set(checked)].sort()).toEqual(expected);
|
|
expect(assigned.every(id => expected.includes(id))).toBe(true);
|
|
await vi.advanceTimersByTimeAsync(31_000);
|
|
}
|
|
}
|
|
checked.length = 0; assigned.length = 0;
|
|
await issueService(state.db).addComment(graph.siblingId, "A note on the next task", { userId: "recovery-user" });
|
|
await settleWork({ backstopPending: true });
|
|
expect(checked).toEqual([graph.siblingId]);
|
|
expect(assigned.every(id => id === graph.siblingId)).toBe(true);
|
|
});
|
|
it("counts an agent's own run heartbeat as no change to its other tasks, and a resume as a change", async () => {
|
|
const f = await fixture();
|
|
const old = new Date(Date.now() - 60 * 60_000);
|
|
const tasks = await state.db.insert(issues).values(Array.from({ length: 5 }, (_, index) => ({ companyId: f.companyId,
|
|
title: `Waiting ${index}`, status: "todo", assigneeAgentId: f.agentId, createdAt: old, updatedAt: old }))).returning({ id: issues.id });
|
|
const since = new Date(Date.now() - 5 * 60_000);
|
|
const changed = async () => (await state.db.select({ id: issues.id }).from(issues)
|
|
.where(and(eq(issues.companyId, f.companyId), changedTaskCondition(since)))).map(row => row.id).sort();
|
|
const heartbeatAt = new Date();
|
|
await state.db.update(agents).set({ status: "idle", lastHeartbeatAt: heartbeatAt, updatedAt: heartbeatAt }).where(eq(agents.id, f.agentId));
|
|
expect(await changed()).toEqual([]);
|
|
await state.db.update(agents).set({ status: "idle", updatedAt: new Date(heartbeatAt.getTime() + 1000) }).where(eq(agents.id, f.agentId));
|
|
expect(await changed()).toEqual(tasks.map(row => row.id).sort());
|
|
});
|
|
it("counts an agent's run start as no change to its other tasks, and a pause during the run as a change", async () => {
|
|
const f = await fixture();
|
|
const old = new Date(Date.now() - 60 * 60_000);
|
|
await state.db.update(agents).set({ lastHeartbeatAt: old, updatedAt: old }).where(eq(agents.id, f.agentId));
|
|
const tasks = await state.db.insert(issues).values(Array.from({ length: 5 }, (_, index) => ({ companyId: f.companyId,
|
|
title: `Settled ${index}`, status: "in_review", assigneeAgentId: f.agentId, createdAt: old, updatedAt: old }))).returning({ id: issues.id });
|
|
const since = new Date(Date.now() - 5 * 60_000);
|
|
const changed = async () => (await state.db.select({ id: issues.id }).from(issues)
|
|
.where(and(eq(issues.companyId, f.companyId), changedTaskCondition(since)))).map(row => row.id).sort();
|
|
// The write of execution start in heartbeat.ts: a status flip with no heartbeat time.
|
|
await state.db.update(agents).set({ status: "running", updatedAt: new Date() }).where(eq(agents.id, f.agentId));
|
|
expect(await changed()).toEqual([]);
|
|
await state.db.update(agents).set({ status: "paused", updatedAt: new Date() }).where(eq(agents.id, f.agentId));
|
|
expect(await changed()).toEqual(tasks.map(row => row.id).sort());
|
|
});
|
|
it("resolves a run without a task to no tasks and a missing task to the whole company", async () => {
|
|
const f = await fixture();
|
|
const [taskless] = await state.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running",
|
|
contextSnapshot: { source: "scheduler" } }).returning({ id: heartbeatRuns.id });
|
|
expect(await resolveIssueGraph(state.db, { companyId: f.companyId, runId: taskless.id })).toEqual([]);
|
|
expect(await resolveIssueGraph(state.db, { companyId: f.companyId, runId: randomUUID() })).toEqual([]);
|
|
expect(await resolveIssueGraph(state.db, { companyId: f.companyId, issueId: randomUUID() })).toBeNull();
|
|
});
|
|
it("resolves a run's task graph through index lookups, not the backlog", async () => {
|
|
const f = await fixture();
|
|
const graph = await runGraph(f.companyId, f.agentId);
|
|
await graph.backlog(2000);
|
|
await state.db.execute(sql`analyze issues`);
|
|
await state.db.execute(sql`analyze heartbeat_runs`);
|
|
const [read] = await state.db.transaction(async tx => {
|
|
expect(await resolveIssueGraph(tx as unknown as typeof state.db, { companyId: f.companyId, runId: graph.runId })).toEqual(graph.affected);
|
|
return tx.execute<{ rows: number }>(sql`select (coalesce(seq_tup_read, 0) + coalesce(idx_tup_fetch, 0))::int as rows
|
|
from pg_stat_xact_user_tables where relname = 'issues'`);
|
|
});
|
|
expect(read.rows).toBeLessThan(50);
|
|
});
|
|
it("finds a stranded task outside the affected set within the backstop bound", async () => {
|
|
const f = await fixture();
|
|
const graph = await runGraph(f.companyId, f.agentId);
|
|
await graph.backlog(50);
|
|
const checkedAt = new Map<string, number>();
|
|
await recordedRecovery({ stranded: async scope => { checkedAt.set(scope.issueId, Date.now()); } });
|
|
await settleWork({ backstopPending: true });
|
|
const fullPassAt = Date.now();
|
|
const [unrelated] = await graph.backlog(1);
|
|
checkedAt.clear();
|
|
await state.db.transaction(tx => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: graph.runId }));
|
|
await settleWork({ backstopPending: true });
|
|
expect(checkedAt.has(unrelated)).toBe(false);
|
|
expect(state.coordinator.nextWakeAt()).toBeLessThanOrEqual(fullPassAt + ISSUE_RECOVERY_BACKSTOP_MS);
|
|
await vi.advanceTimersByTimeAsync(ISSUE_RECOVERY_BACKSTOP_MS);
|
|
await settleWork();
|
|
expect(checkedAt.get(unrelated)! - fullPassAt).toBeLessThanOrEqual(ISSUE_RECOVERY_BACKSTOP_MS + 5_000);
|
|
checkedAt.clear();
|
|
await vi.advanceTimersByTimeAsync(20 * 60_000);
|
|
expect(checkedAt.size).toBe(0);
|
|
const [later] = await graph.backlog(1);
|
|
const eventAt = Date.now();
|
|
await state.db.transaction(tx => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: graph.runId }));
|
|
await settleWork();
|
|
expect(checkedAt.get(later)! - eventAt).toBeLessThanOrEqual(5_000);
|
|
await quiet();
|
|
});
|
|
async function advanceWithoutWaiting(ms: number) {
|
|
const until = Date.now() + ms;
|
|
while (Date.now() < until) { await vi.advanceTimersByTimeAsync(Math.min(1000, until - Date.now())); await immediate(); }
|
|
}
|
|
it("keeps a saved deadline of a task that a changed-tasks backstop pass does not load", async () => {
|
|
const f = await fixture();
|
|
const graph = await runGraph(f.companyId, f.agentId);
|
|
const [target] = await graph.backlog(20);
|
|
const checkedAt: number[] = [];
|
|
await recordedRecovery({ stranded: async scope => {
|
|
if (scope.issueId !== target) return;
|
|
checkedAt.push(Date.now());
|
|
scope.onDeferredDeadline(Date.now() + 16 * 60_000);
|
|
} });
|
|
await settleWork({ backstopPending: true });
|
|
for (let round = 0; round < 2; round++) {
|
|
await vi.advanceTimersByTimeAsync(ISSUE_RECOVERY_BACKSTOP_MS + 60_000);
|
|
await state.db.transaction(tx => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: graph.runId }));
|
|
await settleWork({ backstopPending: true });
|
|
}
|
|
const savedFor = checkedAt.at(-1)! + 16 * 60_000;
|
|
checkedAt.length = 0;
|
|
await vi.advanceTimersByTimeAsync(savedFor - Date.now() + 2_000);
|
|
await settleWork({ backstopPending: true });
|
|
expect(checkedAt.length).toBeGreaterThan(0);
|
|
expect(checkedAt[0]! - savedFor).toBeLessThanOrEqual(5_000);
|
|
});
|
|
it("starts the next changed-tasks window from the start of a slow backstop scan", async () => {
|
|
const f = await fixture();
|
|
const graph = await runGraph(f.companyId, f.agentId);
|
|
const backlog = (await graph.backlog(250)).sort();
|
|
const behind = backlog[0]!;
|
|
let slow = false, changedBehind = false;
|
|
const checkedAt = new Map<string, number[]>();
|
|
await recordedRecovery({ stranded: async scope => {
|
|
checkedAt.set(scope.issueId, [...checkedAt.get(scope.issueId) ?? [], Date.now()]);
|
|
if (!slow) return;
|
|
vi.setSystemTime(Date.now() + 20_000);
|
|
if (scope.issueId === behind && !changedBehind) {
|
|
changedBehind = true;
|
|
await state.db.update(issues).set({ updatedAt: new Date() }).where(eq(issues.id, behind));
|
|
}
|
|
} }, { fullBackstopMs: 24 * 60 * 60_000 });
|
|
await settleWork({ backstopPending: true });
|
|
await vi.advanceTimersByTimeAsync(ISSUE_RECOVERY_BACKSTOP_MS + 60_000);
|
|
await state.db.update(issues).set({ updatedAt: new Date() }).where(inArray(issues.id, backlog));
|
|
await vi.advanceTimersByTimeAsync(ISSUE_RECOVERY_BACKSTOP_MS + 60_000);
|
|
slow = true;
|
|
checkedAt.clear();
|
|
const scanStartedAt = Date.now();
|
|
await state.db.transaction(tx => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: graph.runId }));
|
|
await settleWork({ backstopPending: true });
|
|
slow = false;
|
|
expect(changedBehind).toBe(true);
|
|
expect(backlog.every(id => checkedAt.has(id))).toBe(true);
|
|
expect(Date.now() - scanStartedAt).toBeGreaterThan(2 * ISSUE_RECOVERY_BACKSTOP_MS);
|
|
checkedAt.clear();
|
|
await vi.advanceTimersByTimeAsync(ISSUE_RECOVERY_BACKSTOP_MS + 60_000);
|
|
await state.db.transaction(tx => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: graph.runId }));
|
|
await settleWork({ backstopPending: true });
|
|
expect(checkedAt.has(behind)).toBe(true);
|
|
}, 300_000);
|
|
it("keeps the company pass for hold releases that can admit any task", async () => {
|
|
const f = await fixture();
|
|
const graph = await runGraph(f.companyId, f.agentId);
|
|
const backlog = await graph.backlog(150);
|
|
const checked: string[] = [];
|
|
await recordedRecovery({ stranded: async scope => { checked.push(scope.issueId); } });
|
|
await settleWork({ backstopPending: true });
|
|
const everything = new Set([...graph.affected, graph.siblingId, ...backlog]);
|
|
await state.db.insert(budgetPolicies).values({ companyId: f.companyId, scopeType: "company", scopeId: f.companyId,
|
|
metric: "billed_cents", windowKind: "calendar_month_utc", amount: 1000, hardStopEnabled: true, isActive: true });
|
|
for (const input of [{ trigger: "accounting_settled" as const, runId: graph.runId }, { trigger: "pause_lifted" as const }]) {
|
|
checked.length = 0;
|
|
await state.db.transaction(tx => promoteDeferredWorkIntents(tx, { companyId: f.companyId, ...input }));
|
|
await settleWork({ backstopPending: true });
|
|
expect(new Set(checked)).toEqual(everything);
|
|
await vi.advanceTimersByTimeAsync(31_000);
|
|
}
|
|
});
|
|
it("falls back to the company pass when a run affects more tasks than a hint names", async () => {
|
|
const f = await fixture();
|
|
const graph = await runGraph(f.companyId, f.agentId);
|
|
const backlog = await graph.backlog(200);
|
|
const checked: string[] = [];
|
|
await recordedRecovery({ stranded: async scope => { checked.push(scope.issueId); } });
|
|
await settleWork({ backstopPending: true });
|
|
const dependents = await graph.backlog(ISSUE_RECOVERY_MAX_ISSUE_IDS);
|
|
await state.db.insert(issueRelations).values(dependents.map(id => ({ companyId: f.companyId, issueId: graph.taskId,
|
|
relatedIssueId: id, type: "blocks" as const })));
|
|
checked.length = 0;
|
|
await state.db.transaction(tx => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: graph.runId }));
|
|
await settleWork({ backstopPending: true });
|
|
expect(new Set(checked)).toEqual(new Set([...graph.affected, graph.siblingId, ...backlog, ...dependents]));
|
|
});
|
|
it("checks only blocked tasks whose blockers are done, however many still wait", async () => {
|
|
const f = await fixture();
|
|
const [openBlocker, doneBlocker] = await state.db.insert(issues).values([
|
|
{ companyId: f.companyId, title: "Open blocker", status: "in_progress", assigneeAgentId: f.agentId },
|
|
{ companyId: f.companyId, title: "Done blocker", status: "done" },
|
|
]).returning({ id: issues.id });
|
|
const waiting = await state.db.insert(issues).values(Array.from({ length: 300 }, (_, index) => ({
|
|
companyId: f.companyId, title: `Waiting ${index}`, status: "blocked", assigneeAgentId: f.agentId,
|
|
}))).returning({ id: issues.id });
|
|
const [ready] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Ready", status: "blocked",
|
|
assigneeAgentId: f.agentId }).returning({ id: issues.id });
|
|
await state.db.insert(issueRelations).values([
|
|
...waiting.map(row => ({ companyId: f.companyId, issueId: openBlocker.id, relatedIssueId: row.id, type: "blocks" as const })),
|
|
{ companyId: f.companyId, issueId: doneBlocker.id, relatedIssueId: ready.id, type: "blocks" as const },
|
|
]);
|
|
const checked: string[] = [];
|
|
await recordedRecovery({ dependencies: async scope => { checked.push(scope.issueId); return { enqueueFailed: 0 }; } });
|
|
await settleWork();
|
|
expect(checked).toEqual([ready.id]);
|
|
checked.length = 0;
|
|
await state.db.transaction(tx => notifyIssueRecovery(tx, { companyId: f.companyId }));
|
|
await settleWork();
|
|
expect(checked).toEqual([ready.id]);
|
|
checked.length = 0;
|
|
await issueService(state.db).update(openBlocker.id, { status: "done" });
|
|
await settleWork({ backstopPending: true });
|
|
expect(new Set(checked)).toEqual(new Set([ready.id, ...waiting.map(row => row.id)]));
|
|
await vi.advanceTimersByTimeAsync(ISSUE_RECOVERY_BACKSTOP_MS);
|
|
await settleWork();
|
|
await quiet();
|
|
});
|
|
it("uses ordinary watchdog configuration and subtree closure to dispatch a reusable review", async () => {
|
|
const f = await fixture();
|
|
const source = await issueService(state.db).create(f.companyId, { title: "Finished subtree", status: "done" });
|
|
const failures = await completingProvider(f, () => taskWatchdogService(state.db).disableForIssue(f.companyId, source.id));
|
|
await start(); await quiet();
|
|
await compete();
|
|
const watchdog = await taskWatchdogService(state.db).upsertForIssue(f.companyId, source.id, { agentId: f.agentId });
|
|
let reviewId: string | undefined;
|
|
await vi.waitFor(async () => {
|
|
const [row] = await state.db.select().from(issueWatchdogs).where(eq(issueWatchdogs.id, watchdog.watchdog.id));
|
|
expect(row.watchdogIssueId).toBeTruthy(); reviewId = row.watchdogIssueId!;
|
|
}, { timeout: 15_000 });
|
|
const review = await completed(reviewId!);
|
|
expect(failures).toEqual([]);
|
|
// Ordinary assignment recovery can win the wake reason while the watchdog
|
|
// wake coalesces into that run. Verify its mandate and one dispatch instead.
|
|
expect(review.contextSnapshot).toMatchObject({
|
|
watchdogId: watchdog.watchdog.id,
|
|
taskWatchdog: {
|
|
watchedIssueId: source.id,
|
|
stopFingerprint: expect.stringMatching(/^task_watchdog_stop:/),
|
|
},
|
|
});
|
|
expect(await state.db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
|
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${reviewId}`))).toHaveLength(1);
|
|
await finish(reviewId!);
|
|
await taskWatchdogService(state.db).disableForIssue(f.companyId, source.id);
|
|
if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); await quiet();
|
|
}, 90_000);
|
|
it("bounds ordinary writes to their company and reconstructs foreign stale debt after restart", async () => {
|
|
const selected = await fixture(); await start(); await quiet();
|
|
const foreign = await fixture();
|
|
const [run] = await state.db.insert(heartbeatRuns).values({ companyId: foreign.companyId, agentId: foreign.agentId,
|
|
status: "failed", finishedAt: new Date() }).returning();
|
|
const [stale] = await state.db.insert(issues).values({ companyId: foreign.companyId, title: "Foreign stale lock", status: "done",
|
|
checkoutRunId: run.id }).returning();
|
|
await issueService(state.db).create(selected.companyId, { title: "Selected change", status: "done" }); await quiet();
|
|
expect((await state.db.select().from(issues).where(eq(issues.id, stale.id)))[0].checkoutRunId).toBe(run.id);
|
|
await state.worker.stop(); await state.dispatch.stop(); await state.cancellation.stop(); await state.coordinator.stop();
|
|
state.heartbeat = heartbeatService(state.db); await start(); await quiet();
|
|
expect((await state.db.select().from(issues).where(eq(issues.id, stale.id)))[0].checkoutRunId).toBeNull();
|
|
});
|
|
it("retains a fresh subtree's exact first-run grace then disarms on disable", async () => {
|
|
const f = await fixture();
|
|
const source = await issueService(state.db).create(f.companyId, { title: "Fresh unassigned subtree", status: "todo" });
|
|
await start(); await quiet();
|
|
await taskWatchdogService(state.db).upsertForIssue(f.companyId, source.id, { agentId: f.agentId });
|
|
await vi.waitFor(() => expect(state.coordinator.nextWakeAt()).toBe(source.createdAt.getTime() + 15_000), { timeout: 5_000 });
|
|
expect(await state.db.select().from(issues).where(eq(issues.originKind, "task_watchdog"))).toHaveLength(0);
|
|
await taskWatchdogService(state.db).disableForIssue(f.companyId, source.id);
|
|
if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); await quiet();
|
|
});
|
|
it("waits for a live foreign controller's exact stored lease before clearing a terminal lock", async () => {
|
|
const f = await fixture();
|
|
const source = await issueService(state.db).create(f.companyId, { title: "Remote owner", status: "done" });
|
|
const lease = new Date(Date.now() + 5_000);
|
|
const [run] = await state.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running",
|
|
runtimeMode: "legacy", controllerBootId: randomUUID(), controllerLeaseExpiresAt: lease,
|
|
contextSnapshot: { issueId: source.id }, processPid: 2_000_000_000, startedAt: new Date() }).returning();
|
|
await state.db.update(issues).set({ executionRunId: run.id }).where(eq(issues.id, source.id));
|
|
await start();
|
|
await vi.waitFor(() => expect(state.coordinator.nextWakeAt()).toBe(lease.getTime()), { timeout: 3_000 });
|
|
expect((await state.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].status).toBe("running");
|
|
await vi.waitFor(async () => expect((await state.db.select().from(issues).where(eq(issues.id, source.id)))[0].executionRunId).toBeNull(), { timeout: 12_000 });
|
|
expect((await state.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].status).toBe("succeeded");
|
|
if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); await quiet();
|
|
}, 90_000);
|
|
});
|