diff --git a/doc/plans/2026-10-30-deferred-execution-sleep.md b/doc/plans/2026-10-30-deferred-execution-sleep.md new file mode 100644 index 00000000000..9c90025abe6 --- /dev/null +++ b/doc/plans/2026-10-30-deferred-execution-sleep.md @@ -0,0 +1,54 @@ +# Deferred execution receipt ownership + +This slice removes the periodic `resumeQueuedRuns` sweep after queue dispatch migrated separately. It also delegates native status wake intents out of the periodic orphan reaper. Existing queue admission, authorization, task recovery, cancellation, resource and controller checks remain the mutation authority. + +| Duty | Durable pending truth | Committing producers | Deadline and exit | +| --- | --- | --- | --- | +| Task Retry | deferred task_retry wake with certified actor/generation | requestTaskRetry through execution control; execution, issue, agent, company and budget eligibility | immediate; current admission links canonical intent to one dispatch/run, supersession or terminal policy retires it; blocked input awaits its actual gate mutation | +| Saved comments and interaction responses | deferred wake carrying saved comment IDs or interaction mutation | wake admission insert; merge only genuinely new comment IDs or interaction identity; issue/lock/assignment/dependency and execution eligibility | immediate after terminal source; original release owner revalidates comment/card authority, process/cleanup evidence and issue claim | +| Execution wait | deferred wake with executionWait | original admission/restored source receipt; actual recovery action, execution and eligibility changes | original updatedAt + 30 seconds, then current owner revalidation; unchanged blocked receipt carries no recurring scan | +| Interrupt retry | deferred wake with queuedCommentInterrupt | authorized route persists interrupt marker; cancellation lifecycle and cleanup/eligibility transitions | original updatedAt + 30 seconds; cancelled legacy run naming exact queue bypasses initial grace as before; owner retains explicit interrupt and stop evidence | +| Cancelled legacy queue | cancelled legacy run result names exact still-deferred queue | run lifecycle terminal/control commits; original interrupt intent | immediate restart reconstruction; original release/admission owner settles or leaves saved input behind its actual blocker | +| Task chat receipt settlement | deferred task_retry with taskRetry.chatActionId | failed chat retry action settlement | immediate settlement-only pass folds linked action result; it cannot restage its own processed-but-deferred chat action. Startup/actual eligibility retains normal re-admission policy | +| Native status wake intent | system/native-status-committer wake queued/claimed without run | status committer transaction; actual execution/issue/agent/company/budget eligibility | immediate queued intent; claimedAt + 60 seconds for crash recovery; error updatedAt + 30 seconds. Existing deterministic dispatcher identity/claims preserve per-intent crash recovery; original intent becomes linked/coalesced/skipped while a separate blocked dispatch receipt owns saved admission | + +Three scoped owners reconstruct only their actual durable receipt predicates, with keyset pages of 100. Normal signals contain normalized company/request identity. Instance scheduler settings alone may request instance-wide reconstruction. Repeated source changes coalesce reconstruction for 30 seconds; already-known stored deadlines preempt that cooldown. Pending checks read memory only. Shutdown uses the existing coordinator and physical execution tracking; task drain reconciles producer transactions and releases the settled intent hold while retaining work for resume. Warm standby and disabled registration do not execute domain work. + +The generic run-dispatch notifier and broad issue review-state notifier deliberately do not fan out deferred eligibility. Queue bookkeeping may write them while processing its own saved input. Separate eligibility commits and settlement-only chat notifications avoid a self-restaging loop. + +C2 remains separate and mandatory: active output silence deadlines, remaining orphan/process/controller/remote authority and resource debts, legacy shutdown settlements, stranded agent status, and final aggregate chain removal. Retained workspace and migrated native finalization workers remain their existing owners. + +Review controls cover queue edits that move an already loaded grace deadline. Each due check re-reads current stored grace or claim time before admission. Subtree hold release, subtree restore, and member archival publish eligibility inside their own transactions. Member archival preserves the cancelled Retry tombstone created by an earlier user reassignment. + +A resume that first records a new execution-wait reason arms its existing 30-second grace once. Writing the same closed-gate reason again does not publish another hint. A clock obligation, an eligibility change, and an infrastructure failure therefore retain separate exits: stored deadline, owning committed producer, and bounded failure backoff. Refreshing a diagnostic timestamp alone does not create permanent polling behind a deliberate pause. + +Persisted budget and agent refusals are normalized by the existing deferred-wait owner. The wait writer reports whether its locked conditional update saved the receipt, and only that successful path can produce the internal `PersistedExecutionWaitRefusal` subtype. It remains an HTTP409 with the original message and details. Receipt recovery parks that outcome instead of creating failure debt; the committed first/changed reason still grants its one stored30-second grace. Unrelated409s, failed writes and unexpected provider/infrastructure errors retain failure backoff. This adds no admission policy and changes no native/task-chat ownership. + +Production-path controls use actual task Retry, heartbeat, PostgreSQL budget/lifecycle services and a child process writing saved stdout. The lifecycle host completion boundary is controlled explicitly. One set of real budget/agent refusal fixtures consumes its finite grace and becomes quiet for25 minutes of timer advancement with zero driver SQL and no execution. Independent warm-worker fixtures keep the real PostgreSQL clock throughout admission, actual gate-producer release, controller authority and child-process exit, proving exactly one saved execution. Long-idle and provider-release evidence are separate; no clock reset or restart is used to claim warm recovery. A PostgreSQL trigger fault proves failed wait persistence retains recovery backoff, and an unrelated unbranded409 remains a failure. + +Project budget admission locks and re-reads the company-scoped project before publishing pause intent. An already-paused project emits no new eligibility hint and preserves its pause reason and timestamp, including a manual pause. The actual transition still publishes dispatch and deferred eligibility intent before updating the project in the accounting transaction. Concurrent admission checks serialize through that authority. Project-scoped saved Retry controls cover its one grace followed by25 minutes with zero driver SQL, plus a separate real-clock budget increase through one succeeded process with saved stdout. + + +Native receipt checks share an opaque dispatcher created by the existing queue owner for exactly one bounded worker pass. Its canonical company/agent/issue/review scope map is the same map used by the manual dispatcher; the native worker explicitly admits at most100due entries per pass, matching the existing default batch while retaining company fairness. The next pass creates a fresh map. This restores same-pass grouping without introducing global or cross-invocation deduplication, changing claim authority, or consuming later notices. Each receipt still re-reads its own moved claim/error deadline before dispatch. + +Actual PostgreSQL controls reconstruct two saved native intents into one dispatch and shared run with real process stdout, cover the100-intent boundary, and keep another issue and company separate. A real issue-row lock holds the first admission while a separate transaction commits a new notice whose requestedAt is explicitly older than admission. The fixed current batch cannot consume that new identity; a fresh pass creates a distinct run, preserving the late notice even though its ordering metadata is old. Once all executions and stored deadlines settle,25virtual minutes produce zero driver SQL. The canonical dispatch body, including parsing, claims, deterministic receipt recovery and error handling, is unchanged apart from moving its map into the owner-created batch closure. + + +### Browser fixture producer boundary + +The legacy recovery browser fixture used to insert a deferred request through a +separate SQL connection after server startup, then wait for the removed poll. +That is neither saved startup history nor a supported in-process producer. The +automatic browser case now seeds while the company is paused, verifies the saved +request and sole failed source run, and reactivates through the ordinary company +API. No Retry or duplicate message is sent. It requires exactly one new run, +one persisted agent response, the original queued comment in the process prompt, +and the response after page reload. Startup reconstruction remains covered by +the PostgreSQL case that persists history before worker registration. + +The corrected automatic path passes three consecutive built-UI/browser runs +(2.1 minutes total). All six legacy continuation browser paths pass together +(1.6 minutes), including task/inbox Retry, Retry after reload, ordinary message, +queued Retry and automatic reactivation. The fixture also reads the configured +external PostgreSQL connection when present, so local proof can reuse a dedicated +test database instead of starting another database process. diff --git a/doc/plans/2026-10-30-issue-recovery-sleep.md b/doc/plans/2026-10-30-issue-recovery-sleep.md new file mode 100644 index 00000000000..f6a8bcaba11 --- /dev/null +++ b/doc/plans/2026-10-30-issue-recovery-sleep.md @@ -0,0 +1,32 @@ +# Issue recovery without periodic scans + +The issue recovery owners reconstruct their current obligations once at startup. Ordinary comment writes invalidate only the affected issue for assignment and lock recovery. Lifecycle, graph, run, configuration and eligibility changes invalidate their company. The coordinator keeps scopes, per-issue deadlines, and per-entry failures in memory. Its pending check does not query the database. + +| Owner | Durable truth and producer boundary | Deadline or event | Exit | +| --- | --- | --- | --- | +| Stranded assignment | Visible assigned todo/in-progress/review issues; issue create/update/checkout/release; run lifecycle and execution control; agent/company/budget/configuration; interaction, approval and subtree hold changes | A committed eligibility change. A transient continuation uses the latest finish plus the existing exponential delay. | An execution path, durable wait, escalation, or current ineligibility. | +| Orphan blocker assignment | Agent-created unassigned todo/blocked blockers with a nonterminal dependent; issue, relation, agent and company changes | Once per changed company, independent of assignment candidates. | Assigned, no longer blocking, or creator cannot execute until its next eligibility change. | +| Active recovery actions | Active/escalated action records, including actions on completed sources; action, issue, run and wait changes | Once per changed company, independent of assignment candidates. | Resolved, restored path, explicit hold or a separately owned retry. | +| Resolved dependency | Blocked assigned issues and company-scoped blocker relations; blocker status or relation changes through issue service and import; run/control and pause gate changes | The committing change. A failed enqueue retains that issue with error backoff. | Existing idempotent wake, live path, unresolved dependency or deliberate self-block. | +| Task watchdog | Active issue watchdogs and their current source subtree; normal watchdog upsert/disable; issue/comment/document/work-product and wait changes | Exact source creation plus the existing 15-second first-run grace. An incomplete scan retains the watchdog with error backoff. | Live path, open/reviewed reusable review, disabled configuration, or ineligible source. | +| Stale issue lock | Existing checkout/execution run references; issue and run lifecycle changes | A committing ownership change, or the fresh owning controller's saved lease expiry. | No stale reference, a valid owner, or the existing guarded terminalization and lock clear. | + +Local comment hints do not broaden into an assignment check for every open task. A real PostgreSQL regression starts with1,000 open assigned issues, writes one comment through `issueService.addComment`, and observes exactly one assignment check; a subsequent company eligibility notification still checks all1,000. Restoring the previous comment hint makes that same regression check all1,000 after the comment. The test substitutes the expensive assignment decision only to count its invocations; it uses the real transaction, notification, coordinator and paged SQL. Separate producer acceptance cases exercise real recovery decisions and process execution. + +Parent/descendant disposition repair and shared execution references make status and run changes broader than a comment. Their existing notification boundaries deliberately retain company scope. Orphan blockers, active recovery actions, resolved dependencies and watchdogs also keep their company/subtree duties; an issue comment does not drop those independent obligations. The scope serializer selects known primitive fields and never serializes an entire insert or update object. + +Reconstruction uses keyset pages of at most100rows. Repeated company hints coalesce reconstruction for30seconds. Already-known entry deadlines preempt that spacing without an early probe on each comment. Scheduling suppression is read once per pass. A failed read keeps scoped backoff debt. The two subsidiary company repairs do not run inside the per-issue loop. Legacy direct aggregate API calls retain both repairs. + +The owner methods accept company and issue/watchdog identity. A narrowed lock check still reads other same-company references before using terminal-issue authority. A stale terminal alias cannot end an active issue's shared run. The controller lease owner supplies its fresh expiry; the existing write predicate revalidates the observed boot ID and database-clock expiry. + +Producer coverage uses the existing run-dispatch, execution-control, issue-review-state, native status reassessment and external conversation settlement boundaries. Direct gaps have explicit signals inside their owning transaction: document writes/import, attachment removal, blocker import, recovery-action adoption/mutation, conversation pause release, Telegram failure comment replacement, native artifact title/metadata updates and company deletion. Metadata-only comment classification that does not change timestamp, visibility or execution eligibility needs no additional watchdog activation. Approval links and decisions use their existing issue-review/continuation effects. Work-product create/update/delete/import uses its existing native status boundary. + +The new owners replace only these four startup and periodic calls. Deferred comments, silent active runs and residual orphan/resource recovery retain their existing backstop until the next slice. Every previously migrated native-owner skip remains unchanged. + +The ordinary dependency and watchdog process controls request completion through a loopback test-operator boundary before exiting. That boundary validates the actual run/company/agent and uses issue service; the watchdog control also disables its source through watchdog service. These cases prove normal completion without creating new disposition debt. The separate incomplete-process control prints an execution proof but cannot apply a task disposition, so the test operator completes that task through issue service after the existing policy saves a retry. If the existing continuation policy already saved a disposition-repair retry, the test preserves its actual timer and stored deadline on the real clock. Cleanup joins both competing coordinators before inspecting cancellation. A separate acceptance case deliberately waits for a production-created retry and verifies that exact row becomes cancelled, then all registered deadlines settle before the twenty-minute virtual idle measurement. Its120-second budget covers the existing60-second retry,30-second reconstruction and real I/O; the ordinary dependency test retains90seconds. Installing fake timers after an owner armed a real timer left that timer waiting on wall time despite virtual advances, so cleanup does not switch clocks while any owner obligation remains. The final quiet proof requires no registered deadlines, no tracked in-flight work and no owner SQL. It does not claim a saved future retry is already drained. + +Active action repair has no separate grace or lease countdown. Its delayed disposition repair first writes a scheduled-retry run, then mirrors the date into action wakePolicy.retryAt and timeoutAt. The dispatch owner owns that saved run deadline. Other skipped gates wait for their existing execution, wait, pause, budget, or finalizer producer; the company action worker does not add a polling fallback. + +Agent Chat is an instance-wide eligibility gate for saved conversation comments. An effective disabled-to-enabled transition publishes the existing stranded-assignment topic before the settings write in the same transaction. The worker reconstructs eligible conversation assignments after commit; other issue owners are not activated by this flag. Effective unchanged values, unrelated flags and disabling do not publish assignment hints. A rolled-back enable leaves the accepted comment parked. + +The real PostgreSQL control accepts a valid conversation comment while chat is enabled, disables chat before registering recovery, and proves the outbox stays quiet. Committing the ordinary settings enable alone launches one real process. That process calls a controlled loopback reply boundary which writes the agent comment through issue service; normal conversation completion consumes the message and returns to waiting. Rollback, unchanged/unrelated settings and disable-before-dispatch controls retain input without launching a run. Idle checks inspect actual PostgreSQL statements over twenty virtual minutes, including transactional queries. diff --git a/server/src/__tests__/budgets-service.test.ts b/server/src/__tests__/budgets-service.test.ts index 0ea5a94dfc6..1b2495cf7b6 100644 --- a/server/src/__tests__/budgets-service.test.ts +++ b/server/src/__tests__/budgets-service.test.ts @@ -1,4 +1,5 @@ import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../services/delivery-work-notifications.js"; import { randomUUID } from "node:crypto"; import { eq } from "drizzle-orm"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; @@ -264,6 +265,31 @@ describeEmbeddedPostgres("budgetService release gate enforcement", () => { }); }); + it.each([false, true])("only signals a real project budget pause with concurrent admission checks (already paused: %s)", async alreadyPaused => { + const { companyId, agentId, projectId } = await createBudgetFixture(); + await db.insert(budgetPolicies).values({ companyId, scopeType: "project", scopeId: projectId, + metric: "billed_cents", windowKind: "lifetime", amount: 100, hardStopEnabled: true, notifyEnabled: false }); + await insertCostEvent({ companyId, agentId, projectId, costCents: 125 }); + const pausedAt = new Date("2020-01-01T00:00:00.000Z"); + if (alreadyPaused) await db.update(projects).set({ pausedAt, pauseReason: "manual" }).where(eq(projects.id, projectId)); + const dispatch = vi.fn(), deferred = vi.fn(); + const unsubscribe = [subscribeDeliveryWork(db, DELIVERY_QUEUES.queuedRunDispatch, dispatch), + subscribeDeliveryWork(db, DELIVERY_QUEUES.deferredExecutionReceipts, deferred)]; + try { + const service = budgetService(db); + const blocks = await Promise.all(Array.from({ length: 4 }, () => service.getInvocationBlock(companyId, agentId, { projectId }))); + expect(blocks.every(block => block?.scopeType === "project" && block.scopeId === projectId)).toBe(true); + const [paused] = await db.select().from(projects).where(eq(projects.id, projectId)); + expect(paused).toMatchObject({ pauseReason: alreadyPaused ? "manual" : "budget", pausedAt: alreadyPaused ? pausedAt : expect.any(Date) }); + expect(dispatch).toHaveBeenCalledTimes(alreadyPaused ? 0 : 1); + expect(deferred).toHaveBeenCalledTimes(alreadyPaused ? 0 : 1); + await Promise.all(Array.from({ length: 4 }, () => service.getInvocationBlock(companyId, agentId, { projectId }))); + expect((await db.select().from(projects).where(eq(projects.id, projectId)))[0]).toEqual(paused); + expect(dispatch).toHaveBeenCalledTimes(alreadyPaused ? 0 : 1); + expect(deferred).toHaveBeenCalledTimes(alreadyPaused ? 0 : 1); + } finally { unsubscribe.forEach(stop => stop()); } + }); + it("hard-stops project work until a valid budget raise resumes it and overview reconciles ledger spend", async () => { const { companyId, agentId, projectId } = await createBudgetFixture(); const cancelWorkForScope = vi.fn().mockResolvedValue(undefined); diff --git a/server/src/__tests__/company-portability.test.ts b/server/src/__tests__/company-portability.test.ts index fdf00b828af..f6dbbf552b6 100644 --- a/server/src/__tests__/company-portability.test.ts +++ b/server/src/__tests__/company-portability.test.ts @@ -4168,6 +4168,7 @@ describe("company portability", () => { const insertedRelationValues: Array> = []; const monitorUpdates: Array> = []; const db = { + transaction: async (write: (tx: unknown) => Promise) => write(db), insert: () => ({ values: (rows: Array>) => ({ onConflictDoNothing: async () => { diff --git a/server/src/__tests__/deferred-execution-receipt-work.test.ts b/server/src/__tests__/deferred-execution-receipt-work.test.ts new file mode 100644 index 00000000000..e902c734429 --- /dev/null +++ b/server/src/__tests__/deferred-execution-receipt-work.test.ts @@ -0,0 +1,938 @@ +import { createServer } from "node:http"; +import { conflict, HttpError } from "../errors.js"; +import { setImmediate as immediate } from "node:timers/promises"; +import { setTimeout as realTimeout, clearTimeout as clearRealTimeout } from "node:timers"; +import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../services/delivery-work-notifications.js"; +import { instanceSettingsService } from "../services/instance-settings.js"; +import { budgetService } from "../services/budgets.js"; +import { configureAgentLifecycle, createAgentLifecycle } from "../services/agent-lifecycle.js"; +import { accessService } from "../services/access.js"; +import { PersistedExecutionWaitRefusal, writeDeferredExecutionWait } from "../services/deferred-execution-wait.js"; +import { createQueuedCommentIssueLockWriter } from "../modules/wake-queue/adapters/queued-comment-postgres.js"; +import { createHeartbeatQueue } from "../services/heartbeat/queue.js"; +import { createHeartbeatQueueTestDependencies } from "./helpers/heartbeat-queue.js"; +import { issueTreeControlService } from "../services/issue-tree-control.js"; +import { randomUUID } from "node:crypto"; +import { and, eq, sql } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { agents, agentWakeupRequests, authUsers, chatActions, chatEndpoints, toolApplications, toolConnections, companies, companyMemberships, costEvents, createDb, projects, reconcileDatabaseWork, heartbeatRuns, issueComments, issues } from "@paperclipai/db"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { notifyDeferredExecutionEligibility, notifyDeferredExecutionWork, notifyNativeStatusWakeWork, notifyTaskRetrySettlement } from "../services/deferred-execution-notifications.js"; +import { registerDeferredReceiptWork } from "../services/heartbeat/deferred-receipt-work.js"; +import { heartbeatService } from "../services/heartbeat.js"; +import { requestTaskRetry } from "../services/task-retry.js"; +import { issueService } from "../services/issues.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; + +vi.setConfig({ testTimeout: 90_000 }); +// Vitest's waitFor advances fake time on every poll. Keep PostgreSQL leases +// and actual child processes on wall time until an explicit idle deadline step. +function waitForReal(assertion: () => unknown | Promise, { timeout = 1000 } = {}) { + return new Promise((resolve, reject) => { + let settled = false, lastError: unknown; + let retry: ReturnType | undefined; + const deadline = realTimeout(() => finish(lastError ?? new Error("Expectation did not complete before its deadline")), timeout); + function finish(error?: unknown) { + if (settled) return; + settled = true; clearRealTimeout(deadline); if (retry) clearRealTimeout(retry); + if (error) reject(error); else resolve(); + } + async function attempt() { + try { await assertion(); finish(); } + catch (error) { lastError = error; if (!settled) retry = realTimeout(attempt, 50); } + } + void attempt(); + }); +} +describe("deferred receipt durable dispatch", () => { + let temporary: Awaited> | undefined; + let db: ReturnType; + let heartbeat: ReturnType; + let coordinator: ReturnType; + let drain = false; + const errors: unknown[] = []; + const errorTimes: number[] = []; + const statements: string[] = []; + const passes = new Set>(); + const extraStops: Array<() => Promise> = []; + const lifecycleWorkers = new Set>(); + beforeAll(async () => { + const external = resolveExternalTestDatabaseUrl("PAPERCLIP_DEFERRED_RECEIPT_TEST_DATABASE_URL"); + if (!external) temporary = await startEmbeddedPostgresTestDatabase("deferred-receipts-"); + db = createDb(external ?? temporary!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); + db.$client.options.onnotice = () => {}; + db.$client.options.debug = (_connection, query) => { statements.push(query); }; + await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`))); + }); + beforeEach(async () => { + await db.execute(sql`truncate companies cascade`); drain = false; errors.length = 0; errorTimes.length = 0; + // Install one clock before any worker arms a timer. It follows wall time + // during actual process execution; accelerated steps only settle idle work. + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + heartbeat = heartbeatService(db); + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !drain, canReconcile: () => true, + onError: error => { errors.push(error); errorTimes.push(Date.now()); } }); + const register = coordinator.register.bind(coordinator); + coordinator.register = (queue, task) => register(queue, { ...task, run: signal => { + const pass = Promise.resolve(task.run(signal)); passes.add(pass); + void pass.then(() => passes.delete(pass), () => passes.delete(pass)); return pass; + } }); + }); + afterEach(async () => { + await coordinator.stop(); await heartbeat.drainActiveRunExecutions(); + for (const stop of extraStops.splice(0)) await stop(); + await Promise.all([...lifecycleWorkers].map(worker => worker.stop())); lifecycleWorkers.clear(); + vi.useRealTimers(); vi.restoreAllMocks(); expect(errors).toEqual([]); + }); + afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); + async function fixture() { + const companyId = randomUUID(), actorId = "deferred-receipt-user"; + await db.insert(authUsers).values({ id: actorId, name: "Receipt operator", email: `${actorId}@example.test`, + createdAt: new Date(), updatedAt: new Date() }).onConflictDoNothing(); + await db.insert(companies).values({ id: companyId, name: "Deferred dispatch", issuePrefix: `DE${companyId.slice(0, 6)}`, + defaultResponsibleUserId: actorId, requireBoardApprovalForNewAgents: false }); + await db.insert(companyMemberships).values({ companyId, principalId: actorId, principalType: "user", status: "active", membershipRole: "owner" }); + const [agent] = await db.insert(agents).values({ companyId, name: "Receipt process", adapterType: "process", status: "idle", + adapterConfig: { command: process.execPath, args: ["-e", "console.log('receipt-proof')"], cwd: process.cwd() }, + runtimeConfig: { heartbeat: { enabled: true, intervalSec: 0, wakeOnDemand: true } } }).returning(); + const [issue] = await db.insert(issues).values({ companyId, title: "Resume saved task", status: "todo", assigneeAgentId: agent.id }).returning(); + const [source] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id, issueId: issue.id, + status: "failed", finishedAt: new Date(), responsibleUserId: actorId, errorCode: "process_failed", + contextSnapshot: { issueId: issue.id }, resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } } }).returning(); + return { companyId, agent, issue, source, actorId }; + } + async function quiet() { + await Promise.allSettled([...passes]); await heartbeat.drainActiveRunExecutions(); + await waitForReal(async () => { + if (vi.isFakeTimers()) { + // Fake-clock cases have no live execution. Settle finite cleanup + // deadlines explicitly instead of advancing at waitFor's wall speed. + await vi.advanceTimersByTimeAsync(1000); + await Promise.allSettled([...passes]); await immediate(); + } + expect(coordinator.nextWakeAt()).toBeNull(); expect(idleWorkSnapshot().active).toBe(0); + }, { timeout: 40_000 }); + const reads = vi.spyOn(db, "select"), executes = vi.spyOn(db, "execute"); + const selected = reads.mock.calls.length, executed = executes.mock.calls.length; + const count = statements.length; + await vi.advanceTimersByTimeAsync(25 * 60_000); + expect(statements).toHaveLength(count); + expect(reads).toHaveBeenCalledTimes(selected); expect(executes).toHaveBeenCalledTimes(executed); + } + async function finish(issueId: string) { + await issueService(db).update(issueId, { status: "done" }); + await waitForReal(async () => { + await Promise.allSettled([...passes]); await heartbeat.drainActiveRunExecutions(); + expect(await db.select().from(heartbeatRuns).where(sql`${heartbeatRuns.status} in ('queued', 'running')`)).toHaveLength(0); + }, { timeout: 15_000 }); + const saved = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.status, "scheduled_retry")); + if (saved.length) { + const due = Math.max(...saved.map(run => run.scheduledRetryAt!.getTime())); + while (Date.now() <= due) { + await vi.advanceTimersByTimeAsync(Math.min(500, due - Date.now() + 1)); + await Promise.allSettled([...passes]); await immediate(); + } + } + await waitForReal(async () => { + // A committed cancellation may be coalesced behind an already-armed + // worker deadline. Follow that deadline rather than a guessed 500ms step. + const next = coordinator.nextWakeAt(); + await vi.advanceTimersByTimeAsync(next === null ? 500 : Math.max(1, next - Date.now())); + await Promise.allSettled([...passes]); await immediate(); + await heartbeat.drainActiveRunExecutions(); + const remaining = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.status, "scheduled_retry")); + expect(remaining, JSON.stringify({ next: coordinator.nextWakeAt(), at: Date.now(), + runs: remaining.map(run => ({ id: run.id, at: run.scheduledRetryAt, error: run.errorCode })) })).toHaveLength(0); + }); + } + it("is quiet after empty reconstruction and terminal receipt history", async () => { + const f = await fixture(); + await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agent.id, + source: "on_demand", status: "completed", reason: "task_retry", payload: { issueId: f.issue.id } }); + await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready; await quiet(); + }); + it("dispatches an ordinary committed task Retry into saved process output", async () => { + const f = await fixture(); + await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready; + await heartbeat.registerRunDispatchWork(coordinator, () => !drain).ready; + await heartbeat.registerScheduledRetryCancellationWork(coordinator, () => !drain).ready; + const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issue.id, actorId: f.actorId, + observedRunId: f.source.id, idempotencyKey: randomUUID() }); + expect(receipt.accepted).toBe(true); + let runId: string | undefined; + await waitForReal(async () => { + const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!)); + expect(wake.runId, JSON.stringify(wake)).toBeTruthy(); runId = wake.runId!; + }, { timeout: 15_000 }); + await waitForReal(async () => { + await heartbeat.drainActiveRunExecutions(); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId!)); + expect(run.status, JSON.stringify({ error: run.error, code: run.errorCode })).toBe("succeeded"); + expect(run.stdoutExcerpt).toContain("receipt-proof"); + }, { timeout: 15_000 }); + await finish(f.issue.id); + const linked = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.reason, "task_retry"), sql`${agentWakeupRequests.runId} is not null`)); + expect(linked).toHaveLength(2); + expect(new Set(linked.map(row => row.runId))).toEqual(new Set([runId])); + expect(linked.map(row => row.idempotencyKey)).toContain(`task-retry-dispatch:${receipt.requestId}`); + expect(linked.find(row => row.id === receipt.requestId)?.status).toBe("coalesced"); + expect(await db.select().from(heartbeatRuns).where(sql`${heartbeatRuns.contextSnapshot}->>'taskRetryRequestId' = ${receipt.requestId}`)).toHaveLength(1); + await quiet(); + }); + async function blockedRetry(gate: "budget" | "project budget" | "paused agent") { + const f = await fixture(), lifecycle = createAgentLifecycle(db), budgets = budgetService(db); + // Host lifecycle completion is controlled; all state, admission and the + // eventual process adapter execution use their production implementations. + const lifecycleWorker = configureAgentLifecycle(db, { requiredPluginIds: async () => [], + runHost: async () => "complete", runPlugin: async () => "complete" }); + lifecycleWorkers.add(lifecycleWorker); + const projectId = gate === "project budget" ? randomUUID() : null; + if (projectId) { + await db.insert(projects).values({ id: projectId, companyId: f.companyId, name: "Retry budget project" }); + await db.update(issues).set({ projectId }).where(eq(issues.id, f.issue.id)); + } + const policy = { scopeType: projectId ? "project" as const : "agent" as const, scopeId: projectId ?? f.agent.id, amount: 100, + metric: "billed_cents" as const, windowKind: "lifetime" as const, hardStopEnabled: true, notifyEnabled: false }; + if (gate !== "paused agent") { + await db.insert(costEvents).values({ companyId: f.companyId, agentId: f.agent.id, projectId, + provider: "test", model: "receipt-proof", costCents: 200, occurredAt: new Date() }); + await budgets.upsertPolicy(f.companyId, policy, f.actorId); + } else await lifecycle.pauseAgent(f.agent.id); + await lifecycleWorker.process(f.agent.id); + await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready; + await heartbeat.registerRunDispatchWork(coordinator, () => !drain).ready; + await heartbeat.registerScheduledRetryCancellationWork(coordinator, () => !drain).ready; + const request = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issue.id, actorId: f.actorId, + observedRunId: f.source.id, idempotencyKey: randomUUID() }); + expect(request.accepted).toBe(true); + const read = async () => (await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, request.requestId!)))[0]; + await waitForReal(async () => expect((await read()).payload?.executionWait).toMatchObject({ + reason: gate !== "paused agent" ? "budget.blocked" : "agent.not_invokable", + }), { timeout: 10_000 }); + const refusals: unknown[] = []; + await heartbeat.resumeExecutionWaitComments({ companyId: f.companyId, requestId: request.requestId!, onError: error => refusals.push(error) }); + expect(refusals).toHaveLength(1); + expect(refusals[0]).toBeInstanceOf(HttpError); + expect(refusals[0]).toMatchObject({ status: 409, + details: gate !== "paused agent" ? { scopeType: policy.scopeType, scopeId: policy.scopeId } : { status: "paused", reason: "paused" } }); + return { f, lifecycle, budgets, policy, lifecycleWorker, request, read }; + } + + it.each(["budget", "project budget", "paused agent"] as const)("parks real task Retry behind unchanged %s without recurring SQL", async gate => { + const { request, read } = await blockedRetry(gate); + // The clock was installed before worker registration and every timer. + // No execution exists while we observe actual driver SQL across25min. + for (let n = 0; n < 3; n++) { + await vi.advanceTimersByTimeAsync(31_000); await Promise.allSettled([...passes]); await immediate(); + } + expect(errors).toEqual([]); + await quiet(); + expect(await read()).toMatchObject({ status: "deferred_issue_execution", runId: null }); + expect(await db.select().from(heartbeatRuns).where(sql`${heartbeatRuns.contextSnapshot}->>'taskRetryRequestId' = ${request.requestId}`)).toHaveLength(0); + }); + + it.each(["budget", "project budget", "paused agent"] as const)("releases real %s Retry into exactly one saved process execution", async gate => { + const { f, lifecycle, budgets, policy, lifecycleWorker, request, read } = await blockedRetry(gate); + // Keep the clock advancing with wall time through gate release and actual + // child-process exit. Accelerate only after that execution has settled. + if (gate !== "paused agent") await budgets.upsertPolicy(f.companyId, { ...policy, amount: 1000 }, f.actorId); + else await lifecycle.resumeAgent(f.agent.id); + await lifecycleWorker.process(f.agent.id); + let runId: string | null = null; + await waitForReal(async () => { + const wake = await read(); expect(wake.runId).toBeTruthy(); runId = wake.runId; + }, { timeout: 40_000 }); + await waitForReal(async () => { + await heartbeat.drainActiveRunExecutions(); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId!)); + expect(run.status, JSON.stringify({ error: run.error, code: run.errorCode })).toBe("succeeded"); + expect(run.stdoutExcerpt).toContain("receipt-proof"); + }, { timeout: 15_000 }); + expect(await db.select().from(heartbeatRuns).where(sql`${heartbeatRuns.contextSnapshot}->>'taskRetryRequestId' = ${request.requestId}`)).toHaveLength(1); + await finish(f.issue.id); await quiet(); + }); + + it.each(["missing", "cancelled"] as const)("preserves ordinary409 when a %s receipt cannot save the gate", async status => { + const f = await fixture(), requestId = randomUUID(); + if (status === "cancelled") await db.insert(agentWakeupRequests).values({ id: requestId, companyId: f.companyId, + agentId: f.agent.id, source: "on_demand", status, payload: { issueId: f.issue.id } }); + const deps = createHeartbeatQueueTestDependencies(db), queue = createHeartbeatQueue(db, deps); + deps.getAgent.mockResolvedValue(f.agent); + deps.instanceSettings.getExperimental.mockImplementation(() => instanceSettingsService(db).getExperimental()); + for (const gate of ["budget", "agent"] as const) { + deps.budgets.getInvocationBlock.mockResolvedValue(gate === "budget" + ? { scopeType: "agent", scopeId: f.agent.id, scopeName: f.agent.name, reason: "Budget blocked" } : null); + deps.getAgentInvokability.mockResolvedValue({ invokable: false, message: "Paused", reason: "paused", details: {}, invalidOrgChain: false }); + const error = await queue.enqueueWakeup(f.agent.id, { source: "on_demand", reason: "task_retry", + requestedByActorType: "user", requestedByActorId: f.actorId, payload: { issueId: f.issue.id }, + contextSnapshot: { issueId: f.issue.id, responsibleUserId: f.actorId } }, requestId).catch(cause => cause); + expect(error).toBeInstanceOf(HttpError); + expect(error).not.toBeInstanceOf(PersistedExecutionWaitRefusal); + expect(error).toMatchObject({ status: 409, message: gate === "budget" ? "Budget blocked" : "Paused" }); + } + const rows = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, requestId)); + expect(rows).toHaveLength(status === "missing" ? 0 : 1); + if (status === "cancelled") expect(rows[0]).toMatchObject({ status: "cancelled", runId: null, payload: { issueId: f.issue.id } }); + }); + + it("retains failure backoff for a real wait-persistence database error", async () => { + // This case observes the millisecond before retry. Freeze before any + // fixture or worker can arm a timer; it never starts a child process. + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: false }); + const f = await fixture(); + await createAgentLifecycle(db).pauseAgent(f.agent.id); + const request = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issue.id, actorId: f.actorId, + observedRunId: f.source.id, idempotencyKey: randomUUID() }); + await db.execute(sql`create function c1_refusal_fault() returns trigger language plpgsql as $$ + begin if new.reason = 'task_retry' and new.payload ? 'executionWait' then + raise exception 'C1 injected wait persistence failure'; end if; return new; end $$`); + await db.execute(sql`create trigger c1_refusal_fault before update on agent_wakeup_requests + for each row execute function c1_refusal_fault()`); + try { + await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready; + await waitForReal(() => expect(errors).toHaveLength(1)); + const failure = errors[0] as AggregateError; + expect(failure).toBeInstanceOf(AggregateError); + expect(String(failure.errors[0]?.errors[0]?.cause?.message)).toContain("C1 injected wait persistence failure"); + const due = errorTimes[0] + 30_000; + // Other owners may publish a coalesced deadline after their real SQL + // settles. Run those saved deadlines without reaching the failed retry. + await waitForReal(async () => { + await Promise.allSettled([...passes]); await immediate(); + const next = coordinator.nextWakeAt(); + if (next !== null && next < due) { + await vi.advanceTimersByTimeAsync(Math.max(1, next - Date.now() + 1)); + await Promise.allSettled([...passes]); await immediate(); + } + expect(errors).toHaveLength(1); + expect(coordinator.nextWakeAt()).toBe(due); + }); + expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, request.requestId!)))[0].payload?.executionWait).toBeUndefined(); + await vi.advanceTimersByTimeAsync(due - Date.now() - 1); + expect(errors).toHaveLength(1); + await db.execute(sql`drop trigger c1_refusal_fault on agent_wakeup_requests`); + await db.execute(sql`drop function c1_refusal_fault()`); + errors.length = 0; + for (let n = 0; n < 3; n++) { + await vi.advanceTimersByTimeAsync(31_000); await Promise.allSettled([...passes]); await immediate(); + } + expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, request.requestId!)))[0].payload?.executionWait) + .toMatchObject({ reason: "agent.not_invokable" }); + await quiet(); + } finally { + await db.execute(sql`drop trigger if exists c1_refusal_fault on agent_wakeup_requests`); + await db.execute(sql`drop function if exists c1_refusal_fault()`); + } + }); + + it("does not park an unrelated HTTP409 as a persisted refusal", async () => { + const f = await fixture(); + const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agent.id, + source: "on_demand", reason: "task_retry", status: "deferred_issue_execution" }).returning(); + const failure = conflict("Unexpected admission conflict"); + let fail = true, failedAt = 0; + const resume = vi.fn(async (scope: { onError?: (error: unknown) => void }) => { + if (fail) { failedAt = Date.now(); scope.onError?.(failure); } + }); + await registerDeferredReceiptWork(db, coordinator, { resume, settleTaskChat: async () => {}, createNativeBatch: () => async () => {}, + schedulingSuppressed: async () => false }, () => !drain).ready; + expect(errors).toHaveLength(1); + expect((errors[0] as AggregateError).errors[0].errors).toEqual([failure]); + expect(coordinator.nextWakeAt()).toBe(failedAt + 30_000); + expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id)))[0].status).toBe("deferred_issue_execution"); + errors.length = 0; fail = false; + await vi.advanceTimersByTimeAsync(30_000); + await waitForReal(() => expect(resume).toHaveBeenCalledTimes(2)); + await quiet(); + }); + + async function nativeCohort(f: Awaited>, count = 2, completeAfter = 1) { + const failures: unknown[] = []; + let delivered = 0; + const listener = createServer(async (request, response) => { + try { + let body = ""; + for await (const chunk of request) body += chunk; + const { runId } = JSON.parse(body) as { runId: string }; + const [run] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), + eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.agentId, f.agent.id))); + const issueId = run?.contextSnapshot?.issueId; + const [target] = typeof issueId === "string" ? await db.select().from(issues).where(and(eq(issues.id, issueId), + eq(issues.companyId, f.companyId), eq(issues.assigneeAgentId, f.agent.id))) : []; + if (!run || !target) throw new Error("Unknown native cohort run"); + delivered++; + if (delivered >= completeAfter) await issueService(db).update(target.id, { status: "done" }); + else await issueService(db).addComment(target.id, "This native batch was handled.", { agentId: f.agent.id, runId: run.id }); + response.end("ok"); + } catch (error) { failures.push(error); response.statusCode = 500; response.end("completion failed"); } + }); + await new Promise(resolve => listener.listen(0, "127.0.0.1", resolve)); + extraStops.push(async () => { listener.closeAllConnections(); await new Promise(resolve => listener.close(() => resolve())); }); + const port = (listener.address() as { port: number }).port; + const script = `fetch("http://127.0.0.1:${port}/complete", {method:"POST",body:JSON.stringify({runId:process.env.PAPERCLIP_RUN_ID})}).then(async response=>{if(!response.ok)throw new Error(await response.text());console.log("receipt-proof")}).catch(error=>{console.error(error);process.exitCode=1})`; + await db.update(agents).set({ adapterConfig: { command: process.execPath, args: ["-e", script], cwd: process.cwd() } }) + .where(eq(agents.id, f.agent.id)); + const at = Date.now(); + const intents = await db.insert(agentWakeupRequests).values(Array.from({ length: count }, (_, index) => ({ + reason: index % 2 ? "issue_blockers_resolved" : "issue_children_completed", + companyId: f.companyId, agentId: f.agent.id, source: "automation", triggerDetail: "system", + status: "queued", requestedByActorType: "system", requestedByActorId: "native-status-committer", + requestedAt: new Date(at - (count - index) * 1000), idempotencyKey: `native-cohort:${index}:${f.issue.id}`, + payload: { issueId: f.issue.id, taskId: f.issue.id, + _paperclipWakeContext: { issueId: f.issue.id, taskId: f.issue.id, source: "native_status_decision" } }, + }))).returning(); + return { intents, failures }; + } + async function nativeCohortResult(f: Awaited>, cohort: Awaited>, + owners = [heartbeat]) { + await waitForReal(async () => { + const rows = await db.select().from(agentWakeupRequests).where(sql`${agentWakeupRequests.id} in ${cohort.intents.map(row => row.id)}`); + expect(rows.map(row => row.status)).toEqual(cohort.intents.map(() => "coalesced")); + }, { timeout: 15_000 }); + const dispatched = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, f.companyId), + sql`${agentWakeupRequests.requestedByActorId} like 'native-status-wake-dispatch:%'`, + sql`${agentWakeupRequests.payload}->>'issueId' = ${f.issue.id}`)); + expect(dispatched.map(row => ({ id: row.id, runId: row.runId, status: row.status }))).toHaveLength(1); + const [first, second] = await db.select().from(agentWakeupRequests) + .where(sql`${agentWakeupRequests.id} in ${cohort.intents.map(row => row.id)}`); + expect(first.runId).toBeTruthy(); expect(second.runId).toBe(first.runId); + expect((await db.select().from(agentWakeupRequests).where(sql`${agentWakeupRequests.id} in ${cohort.intents.map(row => row.id)}`)) + .every(row => row.runId === first.runId)).toBe(true); + await waitForReal(async () => { + await Promise.all(owners.map(owner => owner.drainActiveRunExecutions())); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, first.runId!)); + expect(run.status, JSON.stringify({ error: run.error, code: run.errorCode })).toBe("succeeded"); + expect(run.stdoutExcerpt).toContain("receipt-proof"); + }, { timeout: 15_000 }); + expect(cohort.failures).toEqual([]); + return first.runId!; + } + it("coalesces two due native status intents through the registered receipt worker", async () => { + const f = await fixture(), cohort = await nativeCohort(f); + await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready; + await nativeCohortResult(f, cohort); await quiet(); + }); + it("preserves one dispatch for the canonical hundred-intent native batch", async () => { + const f = await fixture(), cohort = await nativeCohort(f, 100); + await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready; + await nativeCohortResult(f, cohort); await quiet(); + }); + it("keeps other issues and companies separate within one native worker batch", async () => { + const f = await fixture(), cohort = await nativeCohort(f), foreign = await fixture(), foreignCohort = await nativeCohort(foreign); + const [otherIssue] = await db.insert(issues).values({ companyId: f.companyId, assigneeAgentId: f.agent.id, + status: "todo", title: "Separate native issue" }).returning(); + const otherIntents = await db.insert(agentWakeupRequests).values(cohort.intents.map((_, index) => ({ + companyId: f.companyId, agentId: f.agent.id, source: "automation", triggerDetail: "system", + reason: index ? "issue_blockers_resolved" : "issue_children_completed", status: "queued", + requestedByActorType: "system", requestedByActorId: "native-status-committer", + idempotencyKey: `other-native-issue:${index}:${otherIssue.id}`, + payload: { issueId: otherIssue.id, taskId: otherIssue.id, + _paperclipWakeContext: { issueId: otherIssue.id, taskId: otherIssue.id, source: "native_status_decision" } }, + }))).returning(); + await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready; + const first = await nativeCohortResult(f, cohort); + const other = await nativeCohortResult({ ...f, issue: otherIssue }, { intents: otherIntents, failures: cohort.failures }); + const third = await nativeCohortResult(foreign, foreignCohort); + expect(new Set([first, other, third]).size).toBe(3); await quiet(); + }); + it("dispatches a notice committed during held admission in a fresh native pass even with old requestedAt", async () => { + const f = await fixture(), cohort = await nativeCohort(f, 2, 2); + let allowLate!: () => void, began!: () => void, releaseIssue!: () => void, locked!: () => void; + const lateGate = new Promise(resolve => { allowLate = resolve; }); + const lateBegan = new Promise(resolve => { began = resolve; }); + const issueGate = new Promise(resolve => { releaseIssue = resolve; }); + const issueLocked = new Promise(resolve => { locked = resolve; }); + let late: typeof agentWakeupRequests.$inferSelect | undefined; + const lateRequestedAt = new Date(Date.now() - 60_000); + const lateTransaction = db.transaction(async tx => { + await tx.execute(sql`select 1`); began(); await lateGate; + const id = randomUUID(); + await notifyNativeStatusWakeWork(tx, { companyId: f.companyId, requestId: id }); + [late] = await tx.insert(agentWakeupRequests).values({ id, companyId: f.companyId, agentId: f.agent.id, + source: "automation", triggerDetail: "system", reason: "issue_children_completed", status: "queued", + requestedByActorType: "system", requestedByActorId: "native-status-committer", requestedAt: lateRequestedAt, + idempotencyKey: `late-native:${id}`, + payload: { issueId: f.issue.id, taskId: f.issue.id, + _paperclipWakeContext: { issueId: f.issue.id, taskId: f.issue.id, source: "native_status_decision" } }, + }).returning(); + }); + await lateBegan; + const issueLock = db.transaction(async tx => { + await tx.select().from(issues).where(eq(issues.id, f.issue.id)).for("update"); locked(); await issueGate; + }); + await issueLocked; + const registration = heartbeat.registerDeferredReceiptWork(coordinator, () => !drain); + try { + await waitForReal(async () => expect((await db.select().from(agentWakeupRequests) + .where(sql`${agentWakeupRequests.id} in ${cohort.intents.map(row => row.id)}`)).some(row => row.status === "claimed")).toBe(true), { timeout: 5000 }); + const [claimed] = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, f.companyId), + eq(agentWakeupRequests.status, "claimed"))); + allowLate(); await lateTransaction; + expect(late!.requestedAt.getTime()).toBeLessThanOrEqual(claimed.claimedAt!.getTime()); + } finally { allowLate(); releaseIssue(); await Promise.all([lateTransaction, issueLock]); } + await registration.ready; + let lateRunId: string | null = null; + await waitForReal(async () => { + const [saved] = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, f.companyId), + eq(agentWakeupRequests.requestedByActorType, "system"), + eq(agentWakeupRequests.requestedByActorId, `native-status-wake-dispatch:${late!.id}`))); + expect(saved?.runId, JSON.stringify(saved)).toBeTruthy(); lateRunId = saved.runId; + }, { timeout: 45_000 }); + const originals = await db.select().from(agentWakeupRequests).where(sql`${agentWakeupRequests.id} in ${cohort.intents.map(row => row.id)}`); + expect(originals.every(row => row.runId === originals[0].runId)).toBe(true); + expect(lateRunId).not.toBe(originals[0].runId); + await waitForReal(async () => { + await heartbeat.drainActiveRunExecutions(); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, lateRunId!)); + expect(run.status, JSON.stringify({ error: run.error, code: run.errorCode })).toBe("succeeded"); + expect(run.stdoutExcerpt).toContain("receipt-proof"); + expect(run.contextSnapshot?.nativeStatusWakeIntentId).toBe(late!.id); + }, { timeout: 15_000 }); + expect(cohort.failures).toEqual([]); + expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, late!.id)))[0].status).toBe("coalesced"); + expect(await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, f.companyId), + sql`${agentWakeupRequests.requestedByActorId} like 'native-status-wake-dispatch:%'`))).toHaveLength(2); + await quiet(); + }); + it.each(["native claim", "execution wait"])("reconstructs an exact fresh %s deadline without probing early", async kind => { + const f = await fixture(), updatedAt = new Date(), native = kind === "native claim"; + const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agent.id, + source: "automation", status: native ? "claimed" : "deferred_issue_execution", claimedAt: updatedAt, updatedAt, + payload: native ? {} : { executionWait: { reason: "recovery" } }, + requestedByActorType: "system", requestedByActorId: native ? "native-status-committer" : null }).returning(); + const check = vi.fn(async (scope: { companyId?: string; requestId?: string }) => { + await db.update(agentWakeupRequests).set({ status: "skipped" }).where(eq(agentWakeupRequests.id, scope.requestId!)); + }); + await registerDeferredReceiptWork(db, coordinator, { resume: check, settleTaskChat: vi.fn(), createNativeBatch: () => check, + schedulingSuppressed: async () => false }, () => !drain).ready; + const due = updatedAt.getTime() + (native ? 60_000 : 30_000); + expect(coordinator.nextWakeAt()).toBe(due); expect(check).not.toHaveBeenCalled(); + await vi.advanceTimersByTimeAsync(due - Date.now() - 1); expect(check).not.toHaveBeenCalled(); + await vi.advanceTimersByTimeAsync(1); + await waitForReal(() => expect(check).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ companyId: f.companyId, requestId: wake.id }))); + await quiet(); + }); + it("reconciles the first drained producer and waits for resume without domain SQL", async () => { + const f = await fixture(); drain = true; + const resume = vi.fn(async () => {}), dispatchNative = vi.fn(async () => {}); + await registerDeferredReceiptWork(db, coordinator, { resume, settleTaskChat: vi.fn(), createNativeBatch: () => dispatchNative, + schedulingSuppressed: async () => false }, () => !drain).ready; + await db.transaction(async tx => { + await notifyDeferredExecutionWork(tx, { companyId: f.companyId }); + await tx.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agent.id, source: "on_demand", + status: "deferred_issue_execution", reason: "task_retry" }); + }); + await vi.advanceTimersByTimeAsync(1000); + expect(resume).not.toHaveBeenCalled(); expect(dispatchNative).not.toHaveBeenCalled(); + await waitForReal(() => expect(idleWorkSnapshot().active).toBe(0)); + drain = false; await vi.advanceTimersByTimeAsync(30_000); + await waitForReal(() => expect(resume).toHaveBeenCalledTimes(1)); await quiet(); + }); + it.each([ + { name: "saved comment", payload: { _paperclipWakeContext: { wakeCommentIds: ["saved-comment"] } } }, + { name: "interaction", payload: { mutation: "interaction" } }, + { name: "execution wait", payload: { executionWait: { reason: "recovery" } } }, + { name: "interrupt", payload: { queuedCommentInterrupt: { actorId: "operator" } } }, + ])("activates only the committed company's $name receipt after an empty start", async ({ payload }) => { + const f = await fixture(), foreign = await fixture(); + const resume = vi.fn(async () => {}); + await registerDeferredReceiptWork(db, coordinator, { resume, settleTaskChat: vi.fn(), createNativeBatch: () => vi.fn(), + schedulingSuppressed: async () => false }, () => !drain).ready; + const [other] = await db.insert(agentWakeupRequests).values({ companyId: foreign.companyId, agentId: foreign.agent.id, + source: "on_demand", status: "deferred_issue_execution", payload }).returning(); + let requestId: string; + await db.transaction(async tx => { + await tx.transaction(async nested => { + await notifyDeferredExecutionWork(nested, { companyId: f.companyId }); + const [row] = await nested.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agent.id, + source: "on_demand", status: "deferred_issue_execution", payload, updatedAt: new Date(Date.now() - 31_000) }).returning(); + requestId = row.id; + }); + expect(resume).not.toHaveBeenCalled(); + }); + await waitForReal(() => expect(resume).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ companyId: f.companyId, requestId })), { timeout: 5_000 }); + expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, other.id)))[0].status).toBe("deferred_issue_execution"); + await quiet(); + }); + it("reconstructs a cancelled legacy run's exact deferred queue even without comment payload", async () => { + const f = await fixture(); + const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agent.id, + source: "on_demand", status: "deferred_issue_execution" }).returning(); + await db.update(heartbeatRuns).set({ status: "cancelled", resultJson: { queuedCommentInterruptQueueId: wake.id } }) + .where(eq(heartbeatRuns.id, f.source.id)); + const resume = vi.fn(async () => {}); + await registerDeferredReceiptWork(db, coordinator, { resume, settleTaskChat: vi.fn(), createNativeBatch: () => vi.fn(), + schedulingSuppressed: async () => false }, () => !drain).ready; + expect(resume).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ companyId: f.companyId, requestId: wake.id })); + await quiet(); + }); + it("parks an accepted conversation Retry while chat is disabled and resumes it on enablement", async () => { + const settings = instanceSettingsService(db), original = await settings.getExperimental(); + await settings.updateExperimental({ enableAgentChat: true }); + try { + const f = await fixture(); + await db.update(issues).set({ status: "in_review", conversationAgentId: f.agent.id, + conversationUserId: f.actorId, conversationState: "waiting" }).where(eq(issues.id, f.issue.id)); + const request = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issue.id, + actorId: f.actorId, observedRunId: f.source.id, idempotencyKey: randomUUID() }); + expect(request.accepted).toBe(true); + await settings.updateExperimental({ enableAgentChat: false }); + await expect(requestTaskRetry({ db, companyId: f.companyId, issueId: f.issue.id, + actorId: f.actorId, observedRunId: f.source.id, idempotencyKey: randomUUID() })) + .rejects.toMatchObject({ status: 404, message: "Agent Chat is disabled" }); + const priorCursor = new Date(Date.now() - 31_000); + await db.update(agentWakeupRequests).set({ updatedAt: priorCursor }).where(eq(agentWakeupRequests.id, request.requestId!)); + await heartbeat.resumeExecutionWaitComments({ companyId: f.companyId }); + const [parked] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, request.requestId!)); + expect(parked.updatedAt.getTime()).toBeGreaterThan(priorCursor.getTime()); + expect(parked).toMatchObject({ status: "deferred_issue_execution", runId: null }); + await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready; + await heartbeat.registerRunDispatchWork(coordinator, () => !drain).ready; + await Promise.allSettled([...passes]); + expect(errors).toEqual([]); + const read = async () => (await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, request.requestId!)))[0]; + expect(await read()).toMatchObject({ status: "deferred_issue_execution", runId: null }); + await waitForReal(async () => { + await Promise.allSettled([...passes]); + expect(coordinator.nextWakeAt()).toBeNull(); expect(idleWorkSnapshot().active).toBe(0); + }, { timeout: 40_000 }); + const idleStatements = statements.length; + await new Promise(resolve => setTimeout(resolve, 31_000)); + expect(statements).toHaveLength(idleStatements); + await settings.updateExperimental({ enableAgentChat: true }); + let runId: string | null = null; + await waitForReal(async () => { + const saved = await read(); expect(saved.runId).toBeTruthy(); runId = saved.runId; + }, { timeout: 5_000 }); + await waitForReal(async () => { + await heartbeat.drainActiveRunExecutions(); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId!)); + expect(run.status).toBe("succeeded"); expect(run.stdoutExcerpt).toContain("receipt-proof"); + }, { timeout: 15_000 }); + expect(await db.select().from(heartbeatRuns).where(sql`${heartbeatRuns.contextSnapshot}->>'taskRetryRequestId' = ${request.requestId}`)).toHaveLength(1); + await heartbeat.drainActiveRunExecutions(); + } finally { await settings.updateExperimental({ enableAgentChat: original.enableAgentChat }); } + }); + it("resumes saved conversation input only after chat is re-enabled and its transaction commits", async () => { + const settings = instanceSettingsService(db), original = await settings.getExperimental(); + await settings.updateExperimental({ enableAgentChat: true }); + try { + const f = await fixture(); + await db.update(issues).set({ status: "in_review", conversationAgentId: f.agent.id, + conversationUserId: f.actorId, conversationState: "waiting" }).where(eq(issues.id, f.issue.id)); + const comment = await issueService(db).addComment(f.issue.id, "Continue with this saved input", { userId: f.actorId }); + const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agent.id, + source: "on_demand", status: "deferred_issue_execution", reason: "issue_commented", + requestedByActorType: "user", requestedByActorId: f.actorId, updatedAt: new Date(Date.now() - 31_000), + payload: { issueId: f.issue.id, queuedCommentInterrupt: { actorId: f.actorId }, + _paperclipWakeContext: { issueId: f.issue.id, wakeCommentIds: [comment.id] } } }).returning(); + await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date(Date.now() - 31_000), + resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false }, + queuedCommentInterruptQueueId: wake.id } }).where(eq(heartbeatRuns.id, f.source.id)); + await settings.updateExperimental({ enableAgentChat: false }); + await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready; + await heartbeat.registerRunDispatchWork(coordinator, () => !drain).ready; + await waitForReal(async () => { + await Promise.allSettled([...passes]); + expect(coordinator.nextWakeAt()).toBeNull(); expect(idleWorkSnapshot().active).toBe(0); + }, { timeout: 5_000 }); + const idleStatements = statements.length; + await new Promise(resolve => setTimeout(resolve, 31_000)); + expect(statements).toHaveLength(idleStatements); + const read = async () => (await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id)))[0]; + expect(await read()).toMatchObject({ status: "deferred_issue_execution", runId: null }); + await expect(db.transaction(async tx => { + await instanceSettingsService(tx as unknown as typeof db).updateExperimental({ enableAgentChat: true }); + throw new Error("rolled back chat enablement"); + })).rejects.toThrow("rolled back chat enablement"); + await Promise.allSettled([...passes]); + expect(await read()).toMatchObject({ status: "deferred_issue_execution", runId: null }); + expect((await settings.getExperimental()).enableAgentChat).toBe(false); + await settings.updateExperimental({ enableAgentChat: true }); + let runId: string | null = null; + await waitForReal(async () => { + const saved = await read(); expect(saved.runId).toBeTruthy(); runId = saved.runId; + }, { timeout: 5_000 }); + await waitForReal(async () => { + await heartbeat.drainActiveRunExecutions(); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId!)); + expect(run.status).toBe("succeeded"); expect(run.stdoutExcerpt).toContain("receipt-proof"); + }, { timeout: 15_000 }); + const runs = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), + sql`${heartbeatRuns.id} <> ${f.source.id}`)); + expect(runs).toHaveLength(1); + expect(runs[0].contextSnapshot?.wakeCommentIds).toEqual([comment.id]); + await quiet(); + } finally { await settings.updateExperimental({ enableAgentChat: original.enableAgentChat }); } + }); + it("settles a processed chat receipt without restaging it until an actual eligibility commit", async () => { + const f = await fixture(); + await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready; + const [application] = await db.insert(toolApplications).values({ companyId: f.companyId, name: "Receipt chat", type: "mcp_http" }).returning(); + const [connection] = await db.insert(toolConnections).values({ companyId: f.companyId, applicationId: application.id, + uid: randomUUID(), name: "Receipt chat", transport: "rest_api" }).returning(); + const [endpoint] = await db.insert(chatEndpoints).values({ companyId: f.companyId, connectionId: connection.id, + provider: "slack", publicId: randomUUID(), assignedAgentId: f.agent.id }).returning(); + const intentId = randomUUID(), actionId = randomUUID(), previous = new Date(Date.now() - 60_000); + await db.insert(agentWakeupRequests).values([ + { id: intentId, companyId: f.companyId, agentId: f.agent.id, source: "on_demand", reason: "task_retry", + status: "deferred_issue_execution", updatedAt: previous, payload: { issueId: f.issue.id, + taskRetry: { schema: "paperclip.task-retry.v1", generationId: intentId, sourceRunId: f.source.id, + actorId: f.actorId, strategy: "evaluate", chatActionId: actionId } } }, + { id: actionId, companyId: f.companyId, agentId: f.agent.id, source: "automation", status: "deferred_issue_execution", + payload: { issueId: f.issue.id, taskRetryRequestId: intentId } }, + ]); + await db.transaction(async tx => { + await notifyTaskRetrySettlement(tx, { companyId: f.companyId, requestId: intentId }); + await tx.insert(chatActions).values({ id: actionId, companyId: f.companyId, endpointId: endpoint.id, + providerActionId: randomUUID(), kind: "failed_run_retry", status: "processed", updatedAt: new Date(), + payload: { issueId: f.issue.id, taskRetryIntentId: intentId, failedRunId: f.source.id } }); + }); + await waitForReal(async () => { + const [intent] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, intentId)); + expect(intent.updatedAt.getTime()).toBeGreaterThan(previous.getTime()); + }, { timeout: 5_000 }); + expect((await db.select().from(chatActions).where(eq(chatActions.id, actionId)))[0].status).toBe("processed"); + await db.transaction(tx => notifyDeferredExecutionEligibility(tx, { companyId: f.companyId })); + await waitForReal(async () => expect((await db.select().from(chatActions).where(eq(chatActions.id, actionId)))[0].status) + .toBe("issued"), { timeout: 5_000 }); + await quiet(); + }); + it("retains a moved interrupt grace deadline after an ordinary saved queue reorder", async () => { + const f = await fixture(), initialAt = new Date(); + const comments = await db.insert(issueComments).values(["first", "second"].map(body => ({ + companyId: f.companyId, issueId: f.issue.id, body, authorType: "user" as const, authorUserId: f.actorId, + }))).returning(); + const payload = { issueId: f.issue.id, queuedCommentInterrupt: { actorId: f.actorId }, + _paperclipWakeContext: { wakeCommentIds: comments.map(row => row.id) } }; + const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agent.id, + source: "on_demand", status: "deferred_issue_execution", payload, updatedAt: initialAt }).returning(); + const deps = createHeartbeatQueueTestDependencies(db), queue = createHeartbeatQueue(db, deps); + await registerDeferredReceiptWork(db, coordinator, { resume: scope => queue.resumeDeferredExecutionReceipts(scope), + settleTaskChat: async () => {}, createNativeBatch: () => async () => {}, schedulingSuppressed: async () => false }, () => !drain).ready; + await vi.advanceTimersByTimeAsync(initialAt.getTime() + 10_000 - Date.now()); + const movedAt = new Date(); + const writer = createQueuedCommentIssueLockWriter(db, { syncCommentReferences: async () => {}, + deleteCommentReferenceSource: async () => {}, syncCommentExternalObjectsSafely: async () => {} }); + await writer.withLockedQueue({ issue: f.issue, queueId: wake.id, + actor: { actorType: "user", actorId: f.actorId, agentId: null, runId: null, agentApiKeyId: null } }, + async (_locked, tx) => { await tx.updateWakeQueuedCommentIds({ wakeId: wake.id, payload, + ids: comments.map(row => row.id).reverse(), updatedAt: movedAt }); }); + await vi.advanceTimersByTimeAsync(initialAt.getTime() + 30_000 - Date.now()); + await waitForReal(() => expect(idleWorkSnapshot().active).toBe(0)); + expect(deps.resumeQueuedCommentInterrupt).not.toHaveBeenCalled(); + expect(coordinator.nextWakeAt()).toBe(movedAt.getTime() + 30_000); + await vi.advanceTimersByTimeAsync(movedAt.getTime() + 30_000 - Date.now()); + await waitForReal(() => expect(deps.resumeQueuedCommentInterrupt).toHaveBeenCalledExactlyOnceWith(f.companyId, wake.id)); + await quiet(); + }); + it("reactivates a held task Retry when the authorized subtree hold is released without wakeAgents", async () => { + const f = await fixture(), tree = issueTreeControlService(db); + const actor = { actorType: "user" as const, actorId: f.actorId, userId: f.actorId }; + const hold = await tree.createHold(f.companyId, f.issue.id, { mode: "pause", actor }); + await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready; + await heartbeat.registerRunDispatchWork(coordinator, () => !drain).ready; + await heartbeat.registerScheduledRetryCancellationWork(coordinator, () => !drain).ready; + const request = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issue.id, actorId: f.actorId, + observedRunId: f.source.id, idempotencyKey: randomUUID() }); + expect(request.requestId).toBeTruthy(); + await waitForReal(async () => { + const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, request.requestId!)); + expect(wake.payload?.executionWait).toMatchObject({ reason: "issue_tree_hold_active" }); + expect(wake.runId).toBeNull(); + }, { timeout: 5_000 }); + await waitForReal(() => { expect(coordinator.nextWakeAt()).toBeNull(); expect(idleWorkSnapshot().active).toBe(0); }, { timeout: 40_000 }); + await tree.releaseHold(f.companyId, f.issue.id, hold.hold.id, { actor, reason: "Resume saved Retry" }); + let runId: string | null = null; + await waitForReal(async () => { + const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, request.requestId!)); + expect(wake.runId, JSON.stringify(wake)).toBeTruthy(); runId = wake.runId; + }, { timeout: 40_000 }); + // Observe this Retry before closing the task and draining its follow-ups. + await waitForReal(async () => { + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId!)); + expect(run.status, JSON.stringify({ error: run.error, code: run.errorCode })).toBe("succeeded"); + expect(run.stdoutExcerpt).toContain("receipt-proof"); + }, { timeout: 15_000 }); + expect(await db.select().from(heartbeatRuns).where(sql`${heartbeatRuns.contextSnapshot}->>'taskRetryRequestId' = ${request.requestId}`)).toHaveLength(1); + await finish(f.issue.id); await quiet(); + }); + it("keeps a moved native dispatch error deadline instead of retrying at its old time", async () => { + const f = await fixture(), initialAt = new Date(); + const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agent.id, + source: "automation", status: "queued", error: "Transient dispatch failure", updatedAt: initialAt, + requestedByActorType: "system", requestedByActorId: "native-status-committer" }).returning(); + const dispatchNative = vi.fn(async () => {}); + await registerDeferredReceiptWork(db, coordinator, { resume: async () => {}, settleTaskChat: async () => {}, + createNativeBatch: () => dispatchNative, schedulingSuppressed: async () => false }, () => !drain).ready; + await vi.advanceTimersByTimeAsync(initialAt.getTime() + 10_000 - Date.now()); + const movedAt = new Date(); + await db.update(agentWakeupRequests).set({ updatedAt: movedAt }).where(eq(agentWakeupRequests.id, wake.id)); + await vi.advanceTimersByTimeAsync(initialAt.getTime() + 30_000 - Date.now()); + await waitForReal(() => expect(idleWorkSnapshot().active).toBe(0)); + expect(dispatchNative).not.toHaveBeenCalled(); expect(coordinator.nextWakeAt()).toBe(movedAt.getTime() + 30_000); + await vi.advanceTimersByTimeAsync(movedAt.getTime() + 30_000 - Date.now()); + await waitForReal(() => expect(dispatchNative).toHaveBeenCalledTimes(1)); await quiet(); + }); + it("arms one new wait reason grace and stays quiet when that same gate is still closed", async () => { + const f = await fixture(); let reason = "operator_pause"; + const resume = vi.fn(async (scope: { companyId?: string; requestId?: string }) => { + await writeDeferredExecutionWait(db, { companyId: scope.companyId!, requestId: scope.requestId! }, { reason }); + }); + await registerDeferredReceiptWork(db, coordinator, { resume, settleTaskChat: async () => {}, createNativeBatch: () => async () => {}, + schedulingSuppressed: async () => false }, () => !drain).ready; + const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agent.id, + source: "on_demand", status: "deferred_issue_execution", reason: "task_retry" }).returning(); + for (const nextReason of ["operator_pause", "budget_pause"]) { + reason = nextReason; + await writeDeferredExecutionWait(db, { companyId: f.companyId, requestId: wake.id }, { reason }); + const [current] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id)); + const due = current.updatedAt.getTime() + 30_000; + await vi.advanceTimersByTimeAsync(1_000); + await waitForReal(() => expect(coordinator.nextWakeAt()).toBe(due)); + const checked = resume.mock.calls.length; + await vi.advanceTimersByTimeAsync(due - Date.now()); + await waitForReal(() => expect(resume).toHaveBeenCalledTimes(checked + 1)); + await quiet(); + } + expect(resume).toHaveBeenCalledTimes(2); + }); + it.each([false, true])("reactivates a saved Retry through ordinary restore cancelled task (held unrelated Retry: %s)", async heldFollowup => { + const f = await fixture(), actor = { actorType: "user" as const, actorId: f.actorId, userId: f.actorId }; + const request = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issue.id, actorId: f.actorId, + observedRunId: f.source.id, idempotencyKey: randomUUID() }); + const tree = issueTreeControlService(db); + const cancel = await tree.createHold(f.companyId, f.issue.id, { mode: "cancel", actor }); + await tree.cancelIssueStatusesForHold(f.companyId, f.issue.id, cancel.hold.id); + const restore = await tree.createHold(f.companyId, f.issue.id, { mode: "restore", actor }); + const commitEligibility = () => tree.restoreIssueStatusesForHold(f.companyId, f.issue.id, restore.hold.id, { actor }); + await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready; + await heartbeat.registerRunDispatchWork(coordinator, () => !drain).ready; + await heartbeat.registerScheduledRetryCancellationWork(coordinator, () => !drain).ready; + await waitForReal(() => { expect(coordinator.nextWakeAt()).toBeNull(); expect(idleWorkSnapshot().active).toBe(0); }, { timeout: 5_000 }); + expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, request.requestId!)))[0].runId).toBeNull(); + let releaseFollowup: (() => void) | undefined; + let draining: Promise | undefined; + let drained = false; + const providerErrors: unknown[] = []; + try { + if (heldFollowup) { + const other = await fixture(); + const release = new Promise(resolve => { releaseFollowup = resolve; }); + let admitted = false; + const listener = createServer(async (incoming, response) => { + try { + let body = ""; + for await (const chunk of incoming) body += chunk; + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, body)); + expect(run).toMatchObject({ companyId: other.companyId, agentId: other.agent.id, status: "running" }); + admitted = true; + await release; + await issueService(db).update(other.issue.id, { status: "done" }); + response.end("ok"); + } catch (error) { providerErrors.push(error); response.statusCode = 500; response.end("failed"); } + }); + extraStops.push(async () => { + const closed = new Promise(resolve => listener.close(() => resolve())); + listener.closeAllConnections(); await closed; + }); + await new Promise(resolve => listener.listen(0, "127.0.0.1", resolve)); + const port = (listener.address() as { port: number }).port; + const script = `fetch("http://127.0.0.1:${port}", {method:"POST",body:process.env.PAPERCLIP_RUN_ID}).then(async response=>{if(!response.ok)throw new Error(await response.text());console.log("held-retry-proof")}).catch(error=>{console.error(error);process.exitCode=1})`; + await db.update(agents).set({ adapterConfig: { command: process.execPath, args: ["-e", script], cwd: process.cwd() } }).where(eq(agents.id, other.agent.id)); + await requestTaskRetry({ db, companyId: other.companyId, issueId: other.issue.id, actorId: other.actorId, + observedRunId: other.source.id, idempotencyKey: randomUUID() }); + await waitForReal(() => expect(admitted).toBe(true), { timeout: 15_000 }); + } + await commitEligibility(); + let runId: string | null = null; + await waitForReal(async () => { + const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, request.requestId!)); + expect(wake.runId, JSON.stringify(wake)).toBeTruthy(); runId = wake.runId; + }, { timeout: 15_000 }); + if (heldFollowup) { + await waitForReal(async () => { + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId!)); + expect(run.status).toBe("succeeded"); expect(run.stdoutExcerpt).toContain("receipt-proof"); + }, { timeout: 15_000 }); + draining = heartbeat.drainActiveRunExecutions().then(() => { drained = true; }); + await immediate(); + expect(drained).toBe(false); + } + // Observe this Retry before closing the task and draining unrelated runs. + await waitForReal(async () => { + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId!)); + expect(run.status, JSON.stringify({ error: run.error, code: run.errorCode })).toBe("succeeded"); + expect(run.stdoutExcerpt).toContain("receipt-proof"); + }, { timeout: 15_000 }); + expect(await db.select().from(heartbeatRuns).where(sql`${heartbeatRuns.contextSnapshot}->>'taskRetryRequestId' = ${request.requestId}`)).toHaveLength(1); + if (heldFollowup) expect(drained).toBe(false); + releaseFollowup?.(); + await finish(f.issue.id); + await draining; + expect(providerErrors).toEqual([]); + await quiet(); + } finally { + releaseFollowup?.(); + await draining; + } + }); + it("member archive publishes committed eligibility while preserving a cancelled Retry tombstone", async () => { + const f = await fixture(), departing = `departing-${randomUUID()}`; + const request = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issue.id, actorId: f.actorId, + observedRunId: f.source.id, idempotencyKey: randomUUID() }); + await db.insert(authUsers).values({ id: departing, name: "Departing member", email: `${departing}@example.test`, + createdAt: new Date(), updatedAt: new Date() }); + const [member] = await db.insert(companyMemberships).values({ companyId: f.companyId, principalId: departing, + principalType: "user", status: "active", membershipRole: "member" }).returning(); + await issueService(db).update(f.issue.id, { assigneeUserId: departing, assigneeAgentId: null }); + const readIntent = async () => (await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, request.requestId!)))[0]; + expect(await readIntent()).toMatchObject({ status: "cancelled", runId: null }); + const deferred = vi.fn(), native = vi.fn(); + const unsubscribes = [subscribeDeliveryWork(db, DELIVERY_QUEUES.deferredExecutionReceipts, deferred), + subscribeDeliveryWork(db, DELIVERY_QUEUES.nativeStatusWakeIntents, native)]; + try { + await expect(db.transaction(async tx => { + await accessService(tx as unknown as typeof db).archiveMember(f.companyId, member.id, + { reassignment: { assigneeAgentId: f.agent.id } }); + expect(deferred).not.toHaveBeenCalled(); expect(native).not.toHaveBeenCalled(); + throw new Error("roll back member archive"); + })).rejects.toThrow("roll back member archive"); + expect(deferred).toHaveBeenCalledTimes(1); expect(native).toHaveBeenCalledTimes(1); + expect((await db.select().from(issues).where(eq(issues.id, f.issue.id)))[0].assigneeUserId).toBe(departing); + await reconcileDatabaseWork(db, DELIVERY_QUEUES.deferredExecutionReceipts); + await reconcileDatabaseWork(db, DELIVERY_QUEUES.nativeStatusWakeIntents); + deferred.mockClear(); native.mockClear(); + await db.transaction(async tx => { + await accessService(tx as unknown as typeof db).archiveMember(f.companyId, member.id, + { reassignment: { assigneeAgentId: f.agent.id } }); + expect(deferred).not.toHaveBeenCalled(); expect(native).not.toHaveBeenCalled(); + }); + expect(deferred).toHaveBeenCalledTimes(1); expect(native).toHaveBeenCalledTimes(1); + expect((await db.select().from(issues).where(eq(issues.id, f.issue.id)))[0]).toMatchObject({ + assigneeUserId: null, assigneeAgentId: f.agent.id, + }); + expect((await db.select().from(companyMemberships).where(eq(companyMemberships.id, member.id)))[0].status).toBe("archived"); + expect(await readIntent()).toMatchObject({ status: "cancelled", runId: null }); + expect(await db.select().from(heartbeatRuns).where(sql`${heartbeatRuns.contextSnapshot}->>'taskRetryRequestId' = ${request.requestId}`)).toHaveLength(0); + } finally { for (const unsubscribe of unsubscribes) unsubscribe(); } + await quiet(); + }); + it("outer rollback and nested savepoint rollback do not dispatch receipt intent", async () => { + const f = await fixture(); const dispatchNative = vi.fn(async () => {}); + await registerDeferredReceiptWork(db, coordinator, { resume: vi.fn(), settleTaskChat: vi.fn(), createNativeBatch: () => dispatchNative, + schedulingSuppressed: async () => false }, () => !drain).ready; + await expect(db.transaction(async tx => { + await notifyNativeStatusWakeWork(tx, { companyId: f.companyId }); + throw new Error("outer rollback"); + })).rejects.toThrow("outer rollback"); + await db.transaction(async tx => { + await expect(tx.transaction(async nested => { + await notifyNativeStatusWakeWork(nested, { companyId: f.companyId }); throw new Error("nested rollback"); + })).rejects.toThrow("nested rollback"); + }); + await quiet(); expect(dispatchNative).not.toHaveBeenCalled(); + }); +}); diff --git a/server/src/__tests__/issue-recovery-deadlines.test.ts b/server/src/__tests__/issue-recovery-deadlines.test.ts new file mode 100644 index 00000000000..c155007f253 --- /dev/null +++ b/server/src/__tests__/issue-recovery-deadlines.test.ts @@ -0,0 +1,525 @@ +import { createServer } from "node:http"; +import { setImmediate as immediate } from "node:timers/promises"; +import { idleWorkSnapshot } from "../services/task-admission.js"; +import { randomUUID } from "node:crypto"; +import { and, eq, sql } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { agents, authUsers, companies, companyMemberships, createDb, heartbeatRuns, issueRecoveryActions, issueRelations, issues, issueWatchdogs, issueComments } from "@paperclipai/db"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { heartbeatService } from "../services/heartbeat.js"; +import { instanceSettingsService } from "../services/instance-settings.js"; +import { undeliveredConversationComments } from "../services/agent-conversations.js"; +import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../services/delivery-work-notifications.js"; +import { issueService } from "../services/issues.js"; +import { issueRecoveryActionService } from "../services/issue-recovery-actions.js"; +import { registerIssueRecoveryWork } from "../services/recovery/work.js"; +import { notifyIssueRecovery } from "../services/issue-recovery-notifications.js"; +import { taskWatchdogService } from "../services/task-watchdogs.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; + +vi.setConfig({ testTimeout: 90_000 }); + +const external = resolveExternalTestDatabaseUrl("PAPERCLIP_ISSUE_RECOVERY_TEST_DATABASE_URL"); +const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("issue recovery production producers and deadlines", () => { + let database: Awaited> | undefined; + let db: ReturnType; + let heartbeat: ReturnType; + let coordinator: ReturnType; + let worker: ReturnType["registerIssueStateRecoveryWork"]>; + let dispatch: ReturnType["registerRunDispatchWork"]>; + let cancellation: ReturnType["registerScheduledRetryCancellationWork"]>; + let draining = false; + const tasks = new Map number | null; hasPending: () => Promise }>(); + const errors: unknown[] = []; + const passes = new Set>(); + const extraStops: Array<() => Promise> = []; + const extraDeadlines: Array<() => number | null> = []; + beforeAll(async () => { + if (!external) database = await startEmbeddedPostgresTestDatabase("issue-recovery-work-"); + db = createDb(external ?? database!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); + await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`))); + }, 90_000); + beforeEach(async () => { + await db.execute(sql`TRUNCATE companies CASCADE`); + heartbeat = heartbeatService(db); draining = false; errors.length = 0; + }); + afterEach(async () => { + for (const stop of extraStops.splice(0)) await stop(); + extraDeadlines.length = 0; + await worker?.stop(); await dispatch?.stop(); await cancellation?.stop(); await coordinator?.stop(); + await heartbeat.drainActiveRunExecutions(); + vi.useRealTimers(); vi.restoreAllMocks(); + expect(errors).toEqual([]); + }); + afterAll(async () => { await db?.$client.end({ timeout: 1 }); await database?.cleanup(); }); + async function start() { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, canReconcile: () => true, + onError: error => errors.push(error) }); + tasks.clear(); + const register = coordinator.register.bind(coordinator); + coordinator.register = (queue, task) => { + tasks.set(queue, task); + return register(queue, { ...task, run: signal => { + const work = task.run(signal); passes.add(work); + void work.then(() => passes.delete(work), () => passes.delete(work)); + return work; + } }); + }; + worker = heartbeat.registerIssueStateRecoveryWork(coordinator, () => !draining); + dispatch = heartbeat.registerRunDispatchWork(coordinator, () => !draining); + cancellation = heartbeat.registerScheduledRetryCancellationWork(coordinator, () => !draining); + await Promise.all([worker.ready, dispatch.ready, cancellation.ready]); + } + async function compete() { + const other = heartbeatService(db); + const competing = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const register = competing.register.bind(competing); + competing.register = (queue, task) => register(queue, { ...task, run: signal => { + const pass = task.run(signal); passes.add(pass); + void pass.then(() => passes.delete(pass), () => passes.delete(pass)); + return pass; + } }); + const work = other.registerIssueStateRecoveryWork(competing, () => true); + extraStops.push(() => competing.stop()); extraDeadlines.push(competing.nextWakeAt); await work.ready; + } + async function fixture() { + const companyId = randomUUID(); + await db.insert(authUsers).values({ id: "recovery-user", name: "Recovery operator", email: "recovery-user@example.test", createdAt: new Date(), updatedAt: new Date() }).onConflictDoNothing(); + await db.insert(companies).values({ id: companyId, name: "Recovery", issuePrefix: `R${companyId.slice(0, 7)}`, + requireBoardApprovalForNewAgents: false, defaultResponsibleUserId: "recovery-user" }); + await db.insert(companyMemberships).values({ companyId, principalId: "recovery-user", principalType: "user", status: "active", membershipRole: "owner" }); + const [agent] = await db.insert(agents).values({ companyId, name: "Recovery process", role: "engineer", status: "idle", + adapterType: "process", adapterConfig: { command: process.execPath, args: ["-e", "console.log('recovery-proof')"], cwd: process.cwd() }, + runtimeConfig: { heartbeat: { enabled: true, intervalSec: 0, wakeOnDemand: true } }, + }).returning(); + return { companyId, agentId: agent.id }; + } + async function completingProvider(f: { companyId: string; agentId: string }, beforeComplete?: () => Promise) { + const failures: unknown[] = []; + const listener = createServer(async (request, response) => { + try { + let body = ""; + for await (const chunk of request) body += chunk; + const { runId } = JSON.parse(body) as { runId: string }; + const [run] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), + eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.agentId, f.agentId))); + if (!run?.contextSnapshot?.issueId) throw new Error("Unknown controlled task run"); + await beforeComplete?.(); + await issueService(db).update(String(run.contextSnapshot.issueId), { status: "done" }); + response.end("ok"); + } catch (error) { failures.push(error); response.statusCode = 500; response.end("completion failed"); } + }); + await new Promise(resolve => listener.listen(0, "127.0.0.1", resolve)); + extraStops.push(async () => { listener.closeAllConnections(); await new Promise(resolve => listener.close(() => resolve())); }); + const port = (listener.address() as { port: number }).port; + const script = `fetch("http://127.0.0.1:${port}/complete", {method:"POST", body:JSON.stringify({runId:process.env.PAPERCLIP_RUN_ID})}).then(async response=>{if(!response.ok)throw new Error(await response.text());console.log("recovery-proof")}).catch(error=>{console.error(error);process.exitCode=1})`; + await db.update(agents).set({ adapterConfig: { command: process.execPath, args: ["-e", script], cwd: process.cwd() } }) + .where(eq(agents.id, f.agentId)); + return failures; + } + async function settled() { + await vi.waitFor(async () => { + await Promise.allSettled([...passes]); + await heartbeat.drainActiveRunExecutions(); + const wake = coordinator.nextWakeAt(); + const detail = wake === null ? "" : JSON.stringify({ now: Date.now(), tasks: await Promise.all([...tasks].map(async ([queue, task]) => ({ queue, at: task.nextRunAt?.(), pending: await task.hasPending() }))), + runs: (await db.select().from(heartbeatRuns)).map(run => ({ id: run.id, status: run.status, error: run.errorCode, lease: run.controllerLeaseExpiresAt, reason: run.contextSnapshot?.wakeReason })) }); + expect(wake, detail).toBeNull(); + expect(extraDeadlines.map(next => next())).toEqual(extraDeadlines.map(() => null)); + expect(idleWorkSnapshot().active).toBe(0); + }, { timeout: 40_000 }); + } + async function quiet() { + await settled(); + expect(await db.select().from(heartbeatRuns).where(sql`${heartbeatRuns.status} in ('queued', 'running')`)).toHaveLength(0); + const reads = vi.spyOn(db, "select"); const executes = vi.spyOn(db, "execute"); + const selected = reads.mock.calls.length, executed = executes.mock.calls.length; + const statements: string[] = [], priorDebug = db.$client.options.debug; + db.$client.options.debug = (_connection, query) => { statements.push(query); }; + if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); + try { + await vi.advanceTimersByTimeAsync(20 * 60_000); + expect(reads).toHaveBeenCalledTimes(selected); expect(executes).toHaveBeenCalledTimes(executed); + expect(statements).toEqual([]); + } finally { vi.useRealTimers(); db.$client.options.debug = priorDebug; reads.mockRestore(); executes.mockRestore(); } + } + async function finish(issueId: string) { + const retries = () => db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.status, "scheduled_retry"), + sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`)); + const before = await retries(); + await issueService(db).update(issueId, { status: "done" }); + // Closing one blocker can legitimately start its dependent. Wait for this + // task's dispatch and execution; quiet() verifies the whole fixture later. + await vi.waitFor(async () => { + await Promise.allSettled([...passes]); + await heartbeat.drainActiveRunExecutions(); + expect(await db.select().from(heartbeatRuns).where(and( + sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`, + sql`${heartbeatRuns.status} in ('queued', 'running')`, + ))).toHaveLength(0); + }, { timeout: 15_000 }); + const saved = [...new Map([...before, ...await retries()].map(run => [run.id, run])).values()]; + if (saved.length > 0) { + const trace: unknown[] = []; + await vi.waitFor(async () => { + await Promise.allSettled([...passes]); + await heartbeat.drainActiveRunExecutions(); + const remaining = await retries(); + if (remaining.length > 0) trace.push({ now: Date.now(), + deadlines: [coordinator.nextWakeAt(), ...extraDeadlines.map(next => next())], + remaining: remaining.map(run => ({ id: run.id, at: run.scheduledRetryAt })) }); + expect(remaining, JSON.stringify(trace)).toHaveLength(0); + }, { timeout: Math.max(...saved.map(run => run.scheduledRetryAt!.getTime())) - Date.now() + 30_000 }); + for (const retry of saved) { + const [settledRetry] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, retry.id)); + expect(settledRetry.status).toBe("cancelled"); + } + } + } + async function completed(issueId: string) { + let runId: string | undefined; + await vi.waitFor(async () => { + const runs = await db.select().from(heartbeatRuns).where(sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`); + const run = runs.find(row => row.status === "running" || row.status === "succeeded"); + expect(run).toBeDefined(); runId = run!.id; + }, { timeout: 15_000 }); + await vi.waitFor(async () => { + await heartbeat.drainActiveRunExecutions(); + const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId!)); + expect(current.status, JSON.stringify({ error: current.error, code: current.errorCode, stderr: current.stderrExcerpt })).toBe("succeeded"); + }, { timeout: 15_000 }); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId!)); + expect(run.stdoutExcerpt).toContain("recovery-proof"); + return run; + } + async function savedConversation(withMessage = true) { + const f = await fixture(), settings = instanceSettingsService(db); + await settings.updateExperimental({ enableAgentChat: true, enableConferenceRoomChat: false }); + const replies: string[] = [], failures: unknown[] = []; + const listener = createServer(async (request, response) => { + try { + let body = ""; + for await (const chunk of request) body += chunk; + const { runId } = JSON.parse(body) as { runId: string }; + const [run] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), + eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.agentId, f.agentId))); + if (!run) throw new Error("Unknown controlled conversation run"); + await issueService(db).addComment(String(run.contextSnapshot?.issueId), "Saved chat delivered", { + agentId: f.agentId, runId: run.id, + }); + replies.push(run.id); response.end("ok"); + } catch (error) { failures.push(error); response.statusCode = 500; response.end("controlled reply failed"); } + }); + await new Promise(resolve => listener.listen(0, "127.0.0.1", resolve)); + extraStops.push(async () => { listener.closeAllConnections(); await new Promise(resolve => listener.close(() => resolve())); }); + const port = (listener.address() as { port: number }).port; + const script = `fetch("http://127.0.0.1:${port}/reply", {method:"POST", body:JSON.stringify({runId:process.env.PAPERCLIP_RUN_ID})}).then(async response=>{if(!response.ok)throw new Error(await response.text());console.log("recovery-proof")}).catch(error=>{console.error(error);process.exitCode=1})`; + await db.update(agents).set({ adapterConfig: { command: process.execPath, args: ["-e", script], cwd: process.cwd() } }) + .where(eq(agents.id, f.agentId)); + const chat = await issueService(db).create(f.companyId, { title: "Saved conversation", status: "in_review", + conversationAgentId: f.agentId, conversationUserId: "recovery-user", assigneeAgentId: f.agentId, + conversationState: "waiting" }); + const message = withMessage ? await issueService(db).addComment(chat.id, "Keep this message", { userId: "recovery-user" }, { clientRequestId: randomUUID() }) : null; + await settings.updateExperimental({ enableAgentChat: false }); + await start(); await quiet(); + expect(replies).toEqual([]); + return { ...f, settings, chat, message, replies, failures }; + } + it("dispatches a saved conversation only from the committed Agent Chat enable transition", async () => { + const f = await savedConversation(); + expect(await undeliveredConversationComments(db, f.companyId, f.chat.id)).toHaveLength(1); + await f.settings.updateExperimental({ enableAgentChat: true }); + expect(coordinator.nextWakeAt()).not.toBeNull(); + await vi.waitFor(async () => expect(await db.select().from(heartbeatRuns) + .where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1), { timeout: 40_000 }); + const run = await completed(f.chat.id); + expect(f.replies).toEqual([run.id]); expect(f.failures).toEqual([]); + expect(await undeliveredConversationComments(db, f.companyId, f.chat.id)).toEqual([]); + expect((await db.select().from(issues).where(eq(issues.id, f.chat.id)))[0]) + .toMatchObject({ conversationState: "waiting", executionRunId: null }); + expect(await db.select().from(issueComments).where(and(eq(issueComments.createdByRunId, run.id), + eq(issueComments.authorAgentId, f.agentId)))).toHaveLength(1); + await quiet(); + let hints = 0; + const stop = subscribeDeliveryWork(db, DELIVERY_QUEUES.strandedAssignments, () => { hints++; }); + try { await f.settings.updateExperimental({ enableAgentChat: true }); expect(hints).toBe(0); await quiet(); } + finally { stop(); } + expect(f.replies).toEqual([run.id]); + }); + it("keeps disabled saved conversations parked after rollback, unrelated flags and unchanged values", async () => { + const f = await savedConversation(); + await expect(db.transaction(async tx => { + await instanceSettingsService(tx as unknown as typeof db).updateExperimental({ enableAgentChat: true }); + throw new Error("chat flag rollback"); + })).rejects.toThrow("chat flag rollback"); + await quiet(); + expect((await f.settings.getExperimental()).enableAgentChat).toBe(false); + let hints = 0; + const stop = subscribeDeliveryWork(db, DELIVERY_QUEUES.strandedAssignments, () => { hints++; }); + try { + await f.settings.updateExperimental({ enableConferenceRoomChat: true }); + await f.settings.updateExperimental({ enableAgentChat: false }); + expect(hints).toBe(0); await quiet(); + } finally { stop(); } + expect(f.replies).toEqual([]); expect(f.failures).toEqual([]); + expect(await db.select().from(heartbeatRuns)).toEqual([]); + expect(await undeliveredConversationComments(db, f.companyId, f.chat.id)).toHaveLength(1); + }); + it("does not dispatch an accepted saved message after Agent Chat is disabled", async () => { + const f = await savedConversation(false); + await f.settings.updateExperimental({ enableAgentChat: true }); await quiet(); + draining = true; + await issueService(db).addComment(f.chat.id, "Retain while disabled", { userId: "recovery-user" }, { clientRequestId: randomUUID() }); + let hints = 0; + const stop = subscribeDeliveryWork(db, DELIVERY_QUEUES.strandedAssignments, () => { hints++; }); + try { await f.settings.updateExperimental({ enableAgentChat: false }); expect(hints).toBe(0); } + finally { stop(); } + draining = false; await quiet(); + expect(f.replies).toEqual([]); expect(f.failures).toEqual([]); + expect(await db.select().from(heartbeatRuns)).toEqual([]); + expect(await undeliveredConversationComments(db, f.companyId, f.chat.id)).toHaveLength(1); + }); + it("keeps empty, configured idle, and terminal history quiet", async () => { + await fixture(); await start(); await quiet(); + }); + it("creates a blocking child through the ordinary root service transaction", async () => { + const f = await fixture(); + const parent = await issueService(db).create(f.companyId, { title: "Parent", status: "todo" }); + const child = await issueService(db).createChild(parent.id, { title: "Blocking child", status: "todo", blockParentUntilDone: true }); + expect(child.parentBlockerAdded).toBe(true); + expect(await db.select().from(issueRelations).where(and(eq(issueRelations.companyId, f.companyId), + eq(issueRelations.issueId, child.issue.id), eq(issueRelations.relatedIssueId, parent.id)))).toHaveLength(1); + }); + it.each(["startup", "committed"])("restores blocker-only and completed-action debt with no assignment candidates (%s)", async mode => { + const f = await fixture(); + if (mode === "committed") { await start(); await quiet(); } + const blocker = await issueService(db).create(f.companyId, { title: "Orphan blocker", status: "blocked", createdByAgentId: f.agentId }); + const dependent = await issueService(db).create(f.companyId, { title: "Blocked dependent", status: "blocked", blockedByIssueIds: [blocker.id] }); + const completedSource = await issueService(db).create(f.companyId, { title: "Completed source", status: "done" }); + const action = await issueRecoveryActionService(db).upsertSourceScoped({ companyId: f.companyId, sourceIssueId: completedSource.id, + kind: "stranded_assigned_issue", ownerType: "agent", ownerAgentId: f.agentId, cause: "stranded_assigned_issue", + fingerprint: "source-terminal", nextAction: "Restore work", wakePolicy: { type: "wake_owner" }, evidence: {} }); + if (mode === "startup") await start(); + await vi.waitFor(async () => { + expect((await db.select().from(issues).where(eq(issues.id, blocker.id)))[0].assigneeAgentId).toBe(f.agentId); + expect((await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id)))[0].status).toBe("resolved"); + }, { timeout: 40_000 }); + await finish(blocker.id); await finish(dependent.id); await quiet(); + }); + it("bounds subsidiary scans for100 assignment candidates and coalesces repeated company writes", async () => { + const f = await fixture(), foreign = await fixture(); + await db.update(agents).set({ lifecycleState: "preparing" }).where(eq(agents.id, f.agentId)); + await db.insert(issues).values(Array.from({ length: 100 }, (_, index) => ({ companyId: f.companyId, + title: `Waiting setup ${index}`, status: "todo", assigneeAgentId: f.agentId }))); + const [source] = await db.insert(issues).values({ companyId: f.companyId, title: "Done source", status: "done" }).returning(); + const action = await issueRecoveryActionService(db).upsertSourceScoped({ companyId: f.companyId, sourceIssueId: source.id, + kind: "stranded_assigned_issue", ownerType: "agent", ownerAgentId: f.agentId, cause: "stranded_assigned_issue", + fingerprint: "query-budget", nextAction: "Restore work", wakePolicy: { type: "wake_owner" }, evidence: {} }); + const scans = { blockers: 0, actions: 0, loads: 0 }; + const originalSelect = db.select.bind(db); + vi.spyOn(db, "select").mockImplementation(((fields?: Record) => { + const query = originalSelect(fields as never); const from = query.from.bind(query); + query.from = ((table: unknown) => { + if (table === issues && fields?.id === issues.id && fields.companyId === issues.companyId && Object.keys(fields).length === 2) scans.loads++; + if (table === issueRelations && fields?.createdByAgentId === issues.createdByAgentId) scans.blockers++; + if (table === issueRecoveryActions && fields?.action === issueRecoveryActions) scans.actions++; + return from(table as never); + }) as typeof query.from; + return query; + }) as typeof db.select); + await start(); await settled(); + expect(scans.actions).toBe(1); expect(scans.blockers).toBe(0); expect(scans.loads).toBeLessThanOrEqual(8); + expect((await originalSelect().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, action.id)))[0].status).toBe("resolved"); + const [foreignSource] = await db.insert(issues).values({ companyId: foreign.companyId, title: "Foreign done", status: "done" }).returning(); + const [foreignAction] = await db.insert(issueRecoveryActions).values({ companyId: foreign.companyId, sourceIssueId: foreignSource.id, + kind: "stranded_assigned_issue", ownerType: "board", cause: "stranded_assigned_issue", fingerprint: "foreign-query-budget", nextAction: "Inspect" }).returning(); + const before = { ...scans }; + for (let index = 0; index < 100; index++) await db.transaction(tx => notifyIssueRecovery(tx, { companyId: f.companyId })); + await new Promise(resolve => setTimeout(resolve, 100)); + expect(scans.loads).toBe(before.loads); + await settled(); + expect(scans.actions).toBe(before.actions); expect(scans.blockers).toBe(before.blockers); + expect(scans.loads).toBeLessThanOrEqual(before.loads + 4); + expect((await originalSelect().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, foreignAction.id)))[0].status).toBe("active"); + expect(await originalSelect().from(heartbeatRuns)).toHaveLength(0); await quiet(); + }); + it("limits an ordinary comment to its issue among one thousand open assignments", async () => { + const f = await fixture(); + const rows = await db.insert(issues).values(Array.from({ length: 1000 }, (_, index) => ({ + companyId: f.companyId, title: `Open work ${index}`, status: "todo", assigneeAgentId: f.agentId, + }))).returning({ id: issues.id }); + const checked: string[] = []; + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const register = coordinator.register.bind(coordinator); + coordinator.register = (queue, task) => register(queue, { ...task, run: signal => { + const pass = task.run(signal); passes.add(pass); + void pass.then(() => passes.delete(pass), () => passes.delete(pass)); + return pass; + } }); + // Observe the actual paged reconstruction boundary without executing one + // thousand adapters. The producer, transaction, coordinator and SQL are real; + // the other acceptance cases exercise the resulting recovery decisions. + worker = registerIssueRecoveryWork(db, coordinator, { + stranded: async scope => { checked.push(scope.issueId); }, + orphanBlockers: async () => {}, recoveryActions: async () => {}, + dependencies: async () => ({ enqueueFailed: 0 }), locks: async () => {}, + watchdogs: async () => ({ incomplete: 0 }), schedulingSuppressed: async () => false, + }, () => true); + await worker.ready; + async function settleWork() { + for (let step = 0; step < 120; step++) { + await vi.advanceTimersByTimeAsync(1000); + await Promise.allSettled([...passes]); await immediate(); + if (coordinator.nextWakeAt() === null && idleWorkSnapshot().active === 0) return; + } + throw new Error("Recovery did not settle its committed work within two virtual minutes"); + } + await settleWork(); + expect(new Set(checked).size).toBe(1000); + checked.length = 0; + await issueService(db).addComment(rows[0].id, "New evidence on this task", { userId: "recovery-user" }); + await settleWork(); + expect(checked).toEqual([rows[0].id]); + // Company eligibility changes still reconstruct all assignments. Narrowing + // comment hints must not hide unrelated work after an operator resumes it. + checked.length = 0; + await db.transaction(tx => notifyIssueRecovery(tx, { companyId: f.companyId })); + await settleWork(); + expect(new Set(checked).size).toBe(1000); + await quiet(); + }); + it("dispatches an ordinary new assignment after commit and returns to quiet after completion", async () => { + const f = await fixture(); await start(); await quiet(); + const issue = await issueService(db).create(f.companyId, { title: "Recover assignment", status: "todo", assigneeAgentId: f.agentId }); + await completed(issue.id); + await finish(issue.id); await quiet(); + }, 90_000); + it("reacts to blocker resolution once and dispatches the durable dependency wake", async () => { + const f = await fixture(); + const blocker = await issueService(db).create(f.companyId, { title: "Blocker", status: "todo" }); + const dependent = await issueService(db).create(f.companyId, { title: "Dependent", status: "blocked", assigneeAgentId: f.agentId, + blockedByIssueIds: [blocker.id] }); + const failures = await completingProvider(f); + await start(); await quiet(); + await compete(); + await issueService(db).update(blocker.id, { status: "done" }); + const run = await completed(dependent.id); + expect(run.contextSnapshot?.wakeReason).toBe("issue_blockers_resolved"); + expect(failures).toEqual([]); + await finish(dependent.id); await quiet(); + expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), + sql`${heartbeatRuns.contextSnapshot}->>'wakeReason' = 'issue_blockers_resolved'`))).toHaveLength(1); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1); + }, 90_000); + it("settles saved disposition debt on its original real-clock timers before measuring idle", async () => { + const f = await fixture(); + const blocker = await issueService(db).create(f.companyId, { title: "Saved debt blocker", status: "todo" }); + const issue = await issueService(db).create(f.companyId, { title: "Saved disposition debt", status: "blocked", + assigneeAgentId: f.agentId, blockedByIssueIds: [blocker.id] }); + await start(); await compete(); + await issueService(db).update(blocker.id, { status: "done" }); + await completed(issue.id); + let retryId: string | undefined; + await vi.waitFor(async () => { + const [retry] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.status, "scheduled_retry"), + sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issue.id}`)); + expect(retry).toBeDefined(); retryId = retry!.id; + }, { timeout: 40_000 }); + await finish(issue.id); await quiet(); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, retryId!)))[0].status).toBe("cancelled"); + }, 120_000); + it("uses ordinary watchdog configuration and subtree closure to dispatch a reusable review", async () => { + const f = await fixture(); + const source = await issueService(db).create(f.companyId, { title: "Finished subtree", status: "done" }); + const failures = await completingProvider(f, () => taskWatchdogService(db).disableForIssue(f.companyId, source.id)); + await start(); await quiet(); + await compete(); + const watchdog = await taskWatchdogService(db).upsertForIssue(f.companyId, source.id, { agentId: f.agentId }); + let reviewId: string | undefined; + await vi.waitFor(async () => { + const [row] = await db.select().from(issueWatchdogs).where(eq(issueWatchdogs.id, watchdog.watchdog.id)); + expect(row.watchdogIssueId).toBeTruthy(); reviewId = row.watchdogIssueId!; + }, { timeout: 15_000 }); + await completed(reviewId!); + expect(failures).toEqual([]); + expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), + sql`${heartbeatRuns.contextSnapshot}->>'wakeReason' = 'task_watchdog_stopped_subtree'`))).toHaveLength(1); + await finish(reviewId!); + await taskWatchdogService(db).disableForIssue(f.companyId, source.id); + if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); await quiet(); + }, 90_000); + it("bounds ordinary writes to their company and reconstructs foreign stale debt after restart", async () => { + const selected = await fixture(); await start(); await quiet(); + const foreign = await fixture(); + const [run] = await db.insert(heartbeatRuns).values({ companyId: foreign.companyId, agentId: foreign.agentId, + status: "failed", finishedAt: new Date() }).returning(); + const [stale] = await db.insert(issues).values({ companyId: foreign.companyId, title: "Foreign stale lock", status: "done", + checkoutRunId: run.id }).returning(); + await issueService(db).create(selected.companyId, { title: "Selected change", status: "done" }); await quiet(); + expect((await db.select().from(issues).where(eq(issues.id, stale.id)))[0].checkoutRunId).toBe(run.id); + await worker.stop(); await dispatch.stop(); await cancellation.stop(); await coordinator.stop(); + heartbeat = heartbeatService(db); await start(); await quiet(); + expect((await db.select().from(issues).where(eq(issues.id, stale.id)))[0].checkoutRunId).toBeNull(); + }); + it("fences nested assignment writes until outer commit and ignores rollback", async () => { + const f = await fixture(); await start(); await quiet(); + let committedId = ""; + await db.transaction(async outer => { + await outer.transaction(async nested => { + const issue = await issueService(db).create(f.companyId, { title: "Nested commit", status: "todo", assigneeAgentId: f.agentId }, nested); + committedId = issue.id; + }); + await new Promise(resolve => setTimeout(resolve, 100)); + expect(await db.select().from(heartbeatRuns)).toHaveLength(0); + }); + await completed(committedId); await finish(committedId); await settled(); + await expect(db.transaction(async outer => { + await outer.transaction(async nested => { + await issueService(db).create(f.companyId, { title: "Nested rollback", status: "todo", assigneeAgentId: f.agentId }, nested); + }); + throw new Error("rollback control"); + })).rejects.toThrow("rollback control"); + await quiet(); + expect(await db.select().from(issues).where(eq(issues.title, "Nested rollback"))).toHaveLength(0); + }, 90_000); + it("retains the first committed assignment during drain and executes on resume", async () => { + const f = await fixture(); await start(); await quiet(); draining = true; + const issue = await issueService(db).create(f.companyId, { title: "Drained assignment", status: "todo", assigneeAgentId: f.agentId }); + await new Promise(resolve => setTimeout(resolve, 100)); + const reads = vi.spyOn(db, "select"); + if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); + try { await vi.advanceTimersByTimeAsync(90_000); expect(reads).not.toHaveBeenCalled(); } + finally { vi.useRealTimers(); reads.mockRestore(); } + expect(await db.select().from(heartbeatRuns)).toHaveLength(0); + draining = false; worker.wake(); dispatch.wake(); + await completed(issue.id); await finish(issue.id); await quiet(); + }, 90_000); + it("retains a fresh subtree's exact first-run grace then disarms on disable", async () => { + const f = await fixture(); + const source = await issueService(db).create(f.companyId, { title: "Fresh unassigned subtree", status: "todo" }); + await start(); await quiet(); + await taskWatchdogService(db).upsertForIssue(f.companyId, source.id, { agentId: f.agentId }); + await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBe(source.createdAt.getTime() + 15_000), { timeout: 5_000 }); + expect(await db.select().from(issues).where(eq(issues.originKind, "task_watchdog"))).toHaveLength(0); + await taskWatchdogService(db).disableForIssue(f.companyId, source.id); + if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); await quiet(); + }); + it("waits for a live foreign controller's exact stored lease before clearing a terminal lock", async () => { + const f = await fixture(); + const source = await issueService(db).create(f.companyId, { title: "Remote owner", status: "done" }); + const lease = new Date(Date.now() + 5_000); + const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "running", + runtimeMode: "legacy", controllerBootId: randomUUID(), controllerLeaseExpiresAt: lease, + contextSnapshot: { issueId: source.id }, processPid: 2_000_000_000, startedAt: new Date() }).returning(); + await db.update(issues).set({ executionRunId: run.id }).where(eq(issues.id, source.id)); + await start(); + await vi.waitFor(() => expect(coordinator.nextWakeAt()).toBe(lease.getTime()), { timeout: 3_000 }); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].status).toBe("running"); + await vi.waitFor(async () => expect((await db.select().from(issues).where(eq(issues.id, source.id)))[0].executionRunId).toBeNull(), { timeout: 12_000 }); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].status).toBe("succeeded"); + if (!vi.isFakeTimers()) vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); await quiet(); + }, 90_000); +}); diff --git a/server/src/__tests__/issue-recovery-work-lifecycle.test.ts b/server/src/__tests__/issue-recovery-work-lifecycle.test.ts new file mode 100644 index 00000000000..52f465daa8e --- /dev/null +++ b/server/src/__tests__/issue-recovery-work-lifecycle.test.ts @@ -0,0 +1,149 @@ +import { PgDialect } from "drizzle-orm/pg-core"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { issueRecoveryActions, issueWatchdogs } from "@paperclipai/db"; +import { installDatabaseWorkSignals } from "../../../packages/db/src/work-signals.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { notifyIssueRecovery } from "../services/issue-recovery-notifications.js"; +import { registerIssueRecoveryWork } from "../services/recovery/work.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; + +const stops: Array<() => Promise> = []; +afterEach(async () => { for (const stop of stops.splice(0)) await stop(); vi.useRealTimers(); }); +function fixture() { + vi.useFakeTimers(); vi.setSystemTime(1_000_000); + const dialect = new PgDialect(); + const rows = { issues: [] as Array<{ id: string; companyId: string }>, watchdogs: [] as Array<{ id: string; companyId: string }>, blockers: [] as Array<{ id: string; companyId: string }>, actions: [] as Array<{ id: string; companyId: string }> }; + const execute = vi.fn(async (query: any) => { + if (dialect.sqlToQuery(query).sql.includes("pg_current_xact_id")) return [{ xid: "42" }]; + throw new Error("unexpected SQL"); + }); + const load = vi.fn((table: unknown, distinct = false) => distinct + ? table === issueRecoveryActions ? rows.actions : rows.blockers + : table === issueWatchdogs ? rows.watchdogs : rows.issues); + const query = (distinct = false) => ({ from: (table: unknown) => ({ where: () => ({ orderBy: () => ({ limit: async () => load(table, distinct) }) }) }) }); + const db: any = installDatabaseWorkSignals({ execute, + select: () => query(), selectDistinct: () => query(true), + transaction: async (callback: (tx: any) => Promise) => callback({ execute, select: db.select, selectDistinct: db.selectDistinct, transaction: db.transaction }), + }); + const state = { draining: false, standby: false, suppressed: false, deadline: null as number | null, fail: false, incomplete: false }; + const check = vi.fn(async (scope: { onDeferredDeadline(at: number): void }) => { + if (state.fail) throw new Error("database unavailable"); + if (state.deadline !== null && state.deadline > Date.now()) scope.onDeferredDeadline(state.deadline); + }); + const orphanBlockers = vi.fn(async () => {}), recoveryActions = vi.fn(async () => {}); + const suppression = vi.fn(async () => state.suppressed); + const owner = { orphanBlockers, recoveryActions, stranded: check, dependencies: async (scope: any) => { await check(scope); return { enqueueFailed: 0 }; }, + locks: check, watchdogs: async (scope: any) => { await check(scope); return { incomplete: state.incomplete ? 1 : 0 }; }, + schedulingSuppressed: suppression }; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !state.standby, + canReconcile: () => !state.standby, onError: vi.fn() }); + const worker = registerIssueRecoveryWork(db, coordinator, owner, () => !state.draining && !state.standby); + stops.push(() => coordinator.stop()); + return { db, rows, state, check, load, coordinator, worker, orphanBlockers, recoveryActions, suppression, + notify: () => db.transaction((tx: any) => notifyIssueRecovery(tx, { companyId: "selected" })), + settle: () => vi.advanceTimersByTimeAsync(1_000) }; +} +describe("issue recovery obligation lifecycle", () => { + it("has no timer, SQL, or owner probe after empty reconstruction", async () => { + const f = fixture(); await f.worker.ready; + expect(f.coordinator.nextWakeAt()).toBeNull(); const loaded = f.load.mock.calls.length; + await vi.advanceTimersByTimeAsync(20 * 60_000); + expect(f.load).toHaveBeenCalledTimes(loaded); expect(f.check).not.toHaveBeenCalled(); + }); + it("reads scheduling suppression once per pass instead of once per issue", async () => { + const f = fixture(); + f.rows.issues.push(...Array.from({ length: 25 }, (_, index) => ({ id: `issue-${index}`, companyId: "selected" }))); + await f.worker.ready; + expect(f.check).toHaveBeenCalledTimes(75); + expect(f.suppression).toHaveBeenCalledTimes(6); + expect(f.coordinator.nextWakeAt()).toBeNull(); + }); + it("backs off a failed suppression read without probing an empty or failed owner repeatedly", async () => { + const f = fixture(); f.suppression.mockRejectedValueOnce(new Error("settings unavailable")); + await f.worker.ready; const reads = f.suppression.mock.calls.length; + expect(f.coordinator.nextWakeAt()).toBe(1_030_000); + await vi.advanceTimersByTimeAsync(29_999); expect(f.suppression).toHaveBeenCalledTimes(reads); + await vi.advanceTimersByTimeAsync(1); expect(f.suppression).toHaveBeenCalledTimes(reads + 1); + expect(f.coordinator.nextWakeAt()).toBeNull(); + }); + it("normalizes a wide scope with cyclic SQL objects before publishing work hints", async () => { + const f = fixture(); await f.worker.ready; + const wide = { companyId: "selected", endpointId: issueRecoveryActions.id, table: issueRecoveryActions }; + await f.db.transaction((tx: object) => notifyIssueRecovery(tx, wide)); await f.settle(); + expect(f.coordinator.nextWakeAt()).toBeNull(); + const loads = f.load.mock.calls.length; + await vi.advanceTimersByTimeAsync(20 * 60_000); expect(f.load).toHaveBeenCalledTimes(loads); + }); + it("runs orphan blocker and action duties without assignment entries", async () => { + const f = fixture(); f.rows.blockers.push({ id: "selected", companyId: "selected" }); + f.rows.actions.push({ id: "selected", companyId: "selected" }); await f.worker.ready; + expect(f.check).not.toHaveBeenCalled(); + expect(f.orphanBlockers).toHaveBeenCalledExactlyOnceWith({ companyId: "selected" }); + expect(f.recoveryActions).toHaveBeenCalledExactlyOnceWith({ companyId: "selected" }); + expect(f.coordinator.nextWakeAt()).toBeNull(); + }); + it("coalesces repeated company reconstruction without delaying an existing lease deadline", async () => { + const f = fixture(); f.rows.issues.push({ id: "issue", companyId: "selected" }); + f.state.deadline = 1_015_000; await f.worker.ready; + await f.notify(); await f.settle(); const loaded = f.load.mock.calls.length, checked = f.check.mock.calls.length; + for (let index = 0; index < 100; index++) { await f.notify(); await vi.advanceTimersByTimeAsync(10); } + expect(f.load).toHaveBeenCalledTimes(loaded); expect(f.check).toHaveBeenCalledTimes(checked); + await vi.advanceTimersByTimeAsync(13_000); + expect(f.state.deadline).toBe(1_015_000); + expect(f.coordinator.nextWakeAt()).toBe(1_031_000); + await vi.advanceTimersByTimeAsync(16_000); + expect(f.load.mock.calls.length).toBeGreaterThan(loaded); + expect(f.coordinator.nextWakeAt()).toBeNull(); + }); + it("retains exact grace and lease deadlines without early owner probes", async () => { + const f = fixture(); f.rows.issues.push({ id: "issue", companyId: "selected" }); + f.rows.watchdogs.push({ id: "watchdog", companyId: "selected" }); f.state.deadline = 1_015_000; + await f.worker.ready; expect(f.coordinator.nextWakeAt()).toBe(1_015_000); + const checked = f.check.mock.calls.length; + await vi.advanceTimersByTimeAsync(14_999); expect(f.check).toHaveBeenCalledTimes(checked); + await vi.advanceTimersByTimeAsync(1); expect(f.check.mock.calls.length).toBe(checked * 2); + expect(f.coordinator.nextWakeAt()).toBeNull(); + }); + it("releases settled producer holds during domain drain while coordinator remains open", async () => { + const f = fixture(); await f.worker.ready; f.state.draining = true; + f.rows.issues.push({ id: "issue", companyId: "selected" }); + await f.notify(); await f.settle(); const loaded = f.load.mock.calls.length; + await vi.advanceTimersByTimeAsync(90_000); + expect(f.load).toHaveBeenCalledTimes(loaded); expect(f.check).not.toHaveBeenCalled(); + expect(idleWorkSnapshot().active).toBe(0); + f.state.draining = false; f.worker.wake(); await f.settle(); + expect(f.check).toHaveBeenCalled(); expect(f.coordinator.nextWakeAt()).toBeNull(); + }); + it("retries concrete failed entries and then disarms without reloading the company", async () => { + const f = fixture(); f.rows.issues.push({ id: "issue", companyId: "selected" }); f.state.fail = true; + await f.worker.ready; const loaded = f.load.mock.calls.length; + expect(f.coordinator.nextWakeAt()).toBe(1_030_000); + await vi.advanceTimersByTimeAsync(29_999); const checked = f.check.mock.calls.length; + f.state.fail = false; await vi.advanceTimersByTimeAsync(1); + expect(f.check.mock.calls.length).toBe(checked * 2); expect(f.load).toHaveBeenCalledTimes(loaded); + expect(f.coordinator.nextWakeAt()).toBeNull(); + }); + it("keeps an incomplete watchdog subtree as concrete retry debt", async () => { + const f = fixture(); f.rows.watchdogs.push({ id: "watchdog", companyId: "selected" }); f.state.incomplete = true; + await f.worker.ready; expect(f.coordinator.nextWakeAt()).toBe(1_030_000); + const loaded = f.load.mock.calls.length; f.state.incomplete = false; + await vi.advanceTimersByTimeAsync(30_000); + expect(f.check).toHaveBeenCalledTimes(2); expect(f.load).toHaveBeenCalledTimes(loaded); + expect(f.coordinator.nextWakeAt()).toBeNull(); + }); + it("keeps a concurrent producer dirty until the in-flight owner has completed", async () => { + const f = fixture(); f.rows.issues.push({ id: "issue", companyId: "selected" }); await f.worker.ready; + let release!: () => void; const held = new Promise(resolve => { release = resolve; }); + f.check.mockImplementationOnce(() => held); + await f.notify(); await f.settle(); await f.notify(); release(); await f.settle(); await vi.advanceTimersByTimeAsync(30_000); + expect(f.coordinator.nextWakeAt()).toBeNull(); + expect(f.check.mock.calls.length).toBeGreaterThan(3); + }); + it("retains startup work on standby and cancels future deadlines on shutdown", async () => { + const f = fixture(); f.state.standby = true; f.rows.watchdogs.push({ id: "watchdog", companyId: "selected" }); + await f.worker.ready; await vi.advanceTimersByTimeAsync(90_000); expect(f.load).not.toHaveBeenCalled(); + f.state.standby = false; f.state.deadline = Date.now() + 60_000; f.worker.wake(); await f.settle(); + expect(f.check).toHaveBeenCalledOnce(); await f.worker.stop(); + await vi.advanceTimersByTimeAsync(120_000); expect(f.check).toHaveBeenCalledOnce(); + }); +}); diff --git a/server/src/__tests__/issue-tree-control-service.test.ts b/server/src/__tests__/issue-tree-control-service.test.ts index 923f399d3bd..e98efb18309 100644 --- a/server/src/__tests__/issue-tree-control-service.test.ts +++ b/server/src/__tests__/issue-tree-control-service.test.ts @@ -41,7 +41,7 @@ describeEmbeddedPostgres("issueTreeControlService", () => { beforeAll(async () => { if (externalUrl) { await applyPendingMigrations(externalUrl); - db = createDb(externalUrl); + db = createDb(externalUrl, { maxConnections: 6 }); return; } tempDb = await startEmbeddedPostgresTestDatabase("paperclip-issue-tree-control-"); diff --git a/server/src/__tests__/scoped-deadline-work.test.ts b/server/src/__tests__/scoped-deadline-work.test.ts index 7c6792c287b..3d838bb6c96 100644 --- a/server/src/__tests__/scoped-deadline-work.test.ts +++ b/server/src/__tests__/scoped-deadline-work.test.ts @@ -10,7 +10,7 @@ afterEach(async () => { await coordinator?.stop(); vi.useRealTimers(); }); it("preserves a subsecond replacement deadline when the owner does not require pass spacing", async () => { vi.useFakeTimers(); const onError = vi.fn(); - const owner = installDatabaseWorkSignals({ execute: vi.fn(async () => []) }); + const owner = installDatabaseWorkSignals({ execute: vi.fn(async () => []), transaction: vi.fn(async () => undefined) }); coordinator = createDeliveryWorkCoordinator({ owner, canRun: () => true, onError }); const now = Date.now(); let dueAt = now + 60_000; @@ -43,7 +43,7 @@ type Entry = { id: string; companyId: string; issueId: string; dueAt: number }; function concurrentFixture(entries: Entry[], limit = 4, concurrent = true) { vi.useFakeTimers(); const onError = vi.fn(); - const owner = installDatabaseWorkSignals({ execute: vi.fn(async () => []) }); + const owner = installDatabaseWorkSignals({ execute: vi.fn(async () => []), transaction: vi.fn(async () => undefined) }); let canRun = true; let releaseAll = false; const releases = new Map void>(); @@ -177,7 +177,7 @@ it("stops starting checks and awaits every admitted check during shutdown", asyn it("keeps failures isolated and retries their deadline without repeating successful checks", async () => { vi.useFakeTimers(); - const owner = installDatabaseWorkSignals({ execute: vi.fn(async () => []) }); + const owner = installDatabaseWorkSignals({ execute: vi.fn(async () => []), transaction: vi.fn(async () => undefined) }); const onError = vi.fn(); coordinator = createDeliveryWorkCoordinator({ owner, canRun: () => true, onError }); const entries = [entry("1", "shared"), entry("2", "shared"), entry("3")]; @@ -200,3 +200,98 @@ it("keeps failures isolated and retries their deadline without repeating success expect(load).toHaveBeenCalledOnce(); expect(coordinator.nextWakeAt()).toBeNull(); }); + + +it("joins one beforePass across keyed parallel checks while retaining per-task order", async () => { + vi.useFakeTimers(); + const owner = installDatabaseWorkSignals({ execute: vi.fn(async () => []), transaction: vi.fn(async () => undefined) }); + const onError = vi.fn(), beforePass = vi.fn(async () => {}); + const started: string[] = [], completed = new Set(), releases = new Map void>(); + let releaseAll = false; + const entries = [entry("first", "shared"), entry("second", "shared"), entry("independent", "other")]; + const load = vi.fn(async () => entries.filter(row => !completed.has(row.id))); + coordinator = createDeliveryWorkCoordinator({ owner, canRun: () => true, onError }); + const worker = registerScopedDeadlineWork(coordinator, { + queue: DELIVERY_QUEUES.nativeResponsePresentation, minimumPassIntervalMs: 0, + concurrentChecks: { limit: 2, key: (row: Entry) => `${row.companyId}:${row.issueId}` }, + reconstructionIntervalMs: 30_000, beforePass, + decodeScope: hint => hint === "*" ? undefined : hint, + matches: (row, companyId?: string) => companyId === undefined || row.companyId === companyId, + load, + check: async row => { + started.push(row.id); + if (!releaseAll) await new Promise(resolve => releases.set(row.id, resolve)); + completed.add(row.id); + return null; + }, + }, () => true); + try { + await vi.advanceTimersByTimeAsync(0); + expect(started).toEqual(["first", "independent"]); + expect(beforePass).toHaveBeenCalledOnce(); + worker.invalidate("company"); + await vi.advanceTimersByTimeAsync(0); + expect(beforePass).toHaveBeenCalledOnce(); + releases.get("independent")!(); + await vi.advanceTimersByTimeAsync(0); + expect(started).not.toContain("second"); + releases.get("first")!(); + await vi.advanceTimersByTimeAsync(0); + expect(started).toEqual(["first", "independent", "second"]); + expect(beforePass).toHaveBeenCalledOnce(); + } finally { + releaseAll = true; for (const release of releases.values()) release(); await worker.ready; + } + await vi.advanceTimersByTimeAsync(1); + expect(beforePass).toHaveBeenCalledTimes(2); + expect(load).toHaveBeenCalledTimes(2); + expect(coordinator.nextWakeAt()).toBeNull(); + await vi.advanceTimersByTimeAsync(120_000); + expect(load).toHaveBeenCalledTimes(2); expect(beforePass).toHaveBeenCalledTimes(2); + expect(onError).not.toHaveBeenCalled(); +}); + +it("keeps an exact deadline and late reconstruction debt with concurrency enabled across drain", async () => { + vi.useFakeTimers(); + const owner = installDatabaseWorkSignals({ execute: vi.fn(async () => []), transaction: vi.fn(async () => undefined) }); + const onError = vi.fn(), beforePass = vi.fn(async () => {}); + let canRun = true, completed = false, entered!: () => void, release!: () => void; + const loading = new Promise(resolve => { entered = resolve; }); + const held = new Promise(resolve => { release = resolve; }); + const now = Date.now(), dueAt = now + 5_000; + const candidate = { ...entry("saved", "task"), dueAt }; + const load = vi.fn(async () => { if (load.mock.calls.length === 2) { entered(); await held; } return completed ? [] : [candidate]; }); + const check = vi.fn(async () => { completed = true; return null; }); + coordinator = createDeliveryWorkCoordinator({ owner, canRun: () => canRun, onError }); + const worker = registerScopedDeadlineWork(coordinator, { + queue: DELIVERY_QUEUES.nativeResponsePresentation, minimumPassIntervalMs: 0, + concurrentChecks: { limit: 2, key: (row: Entry) => row.issueId }, + reconstructionIntervalMs: 30_000, beforePass, + decodeScope: hint => hint === "*" ? undefined : hint, + matches: (row, companyId?: string) => companyId === undefined || row.companyId === companyId, + load, check, + }, () => canRun); + await worker.ready; + worker.invalidate("company"); + try { + await vi.advanceTimersByTimeAsync(1); await loading; + worker.invalidate("company"); + } finally { release(); } + await vi.advanceTimersByTimeAsync(0); + expect(load).toHaveBeenCalledTimes(2); + expect(coordinator.nextWakeAt()).toBe(dueAt); + await vi.advanceTimersByTimeAsync(4_999); + expect(check).toHaveBeenCalledOnce(); expect(load).toHaveBeenCalledTimes(2); + expect(coordinator.nextWakeAt()).toBe(now + 30_001); + canRun = false; worker.wake(); + await vi.advanceTimersByTimeAsync(20_000); + expect(load).toHaveBeenCalledTimes(2); expect(check).toHaveBeenCalledOnce(); + canRun = true; worker.wake(); + await vi.advanceTimersByTimeAsync(5_001); + expect(load).toHaveBeenCalledTimes(3); expect(check).toHaveBeenCalledOnce(); + expect(beforePass).toHaveBeenCalledTimes(4); + expect(coordinator.nextWakeAt()).toBeNull(); + await vi.advanceTimersByTimeAsync(120_000); + expect(load).toHaveBeenCalledTimes(3); expect(check).toHaveBeenCalledOnce(); + expect(onError).not.toHaveBeenCalled(); +}); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index e00c8519df7..55d261f91c0 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -134,6 +134,8 @@ const { })), reconcileHotRestartAdoption: vi.fn(async () => ({ mode: "none" })), reapOrphanedRuns: vi.fn(async () => ({ reaped: 0, runIds: [] })), + registerIssueStateRecoveryWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), + registerDeferredReceiptWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), registerRunDispatchWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), registerScheduledRetryCancellationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), registerStoppedInstructionCollectionWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), @@ -927,7 +929,10 @@ describe("startServer feedback export wiring", () => { it.each([false, true])("registers retry and queued dispatch only when maintenance is enabled (%s)", async enabled => { loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled })); await startServer(); await settleStartupHousekeeping(); + expect(heartbeatServiceMock.registerIssueStateRecoveryWork).toHaveBeenCalledTimes(enabled ? 1 : 0); expect(heartbeatServiceMock.registerRunDispatchWork).toHaveBeenCalledTimes(enabled ? 1 : 0); + expect(heartbeatServiceMock.registerDeferredReceiptWork).toHaveBeenCalledTimes(enabled ? 1 : 0); + expect(heartbeatServiceMock.resumeQueuedRuns).not.toHaveBeenCalled(); if (enabled) { const [coordinator, admitted] = heartbeatServiceMock.registerRunDispatchWork.mock.calls[0]; const app = await createAppMock.mock.results.at(-1)!.value; @@ -1334,12 +1339,12 @@ describe("startServer feedback export wiring", () => { await startServer(); await settleStartupHousekeeping(); expect(heartbeatServiceMock.reconcileCostAccounting).toHaveBeenCalledTimes(1); - expect(heartbeatServiceMock.sweepStaleIssueLocks).toHaveBeenCalledTimes(1); + expect(heartbeatServiceMock.sweepStaleIssueLocks).not.toHaveBeenCalled(); expect(intervalCallback).not.toBeNull(); intervalCallback?.(); await new Promise((resolve) => setImmediate(resolve)); - expect(heartbeatServiceMock.sweepStaleIssueLocks).toHaveBeenCalledTimes(2); - expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true, skipRetainedNativeSessionCleanup: true, skipNativeWorkspaceFinalization: true, skipNativeResultCommit: true, skipNativeRunProjection: true, skipNativeRecoveryNotices: true, skipExternalConversationSettlement: true, skipExternalAgentSettlement: true, skipNativeEnvironmentSettlement: true }); + expect(heartbeatServiceMock.sweepStaleIssueLocks).not.toHaveBeenCalled(); + expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipNativeStatusWakeIntents: true, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true, skipRetainedNativeSessionCleanup: true, skipNativeWorkspaceFinalization: true, skipNativeResultCommit: true, skipNativeRunProjection: true, skipNativeRecoveryNotices: true, skipExternalConversationSettlement: true, skipExternalAgentSettlement: true, skipNativeEnvironmentSettlement: true }); expect(heartbeatServiceMock.reconcileCostAccounting).toHaveBeenCalledTimes(1); expect(heartbeatServiceMock.promoteDueScheduledRetries).not.toHaveBeenCalled(); expect(retiredDetector).not.toHaveBeenCalled(); diff --git a/server/src/__tests__/task-watchdogs-classifier.test.ts b/server/src/__tests__/task-watchdogs-classifier.test.ts index 16ef800c088..71865c75dfe 100644 --- a/server/src/__tests__/task-watchdogs-classifier.test.ts +++ b/server/src/__tests__/task-watchdogs-classifier.test.ts @@ -1,8 +1,6 @@ -import { describe, expect, it, vi } from "vitest"; +import { describe, expect, it } from "vitest"; import { classifyTaskWatchdogSubtree, type TaskWatchdogClassifierIssue } from "../services/task-watchdogs.ts"; -import { createIncompleteTaskWatchdogScanReporter } from "../services/task-watchdog-diagnostics.ts"; - const companyId = "company-1"; const sourceId = "source-1"; const childId = "child-1"; @@ -37,39 +35,6 @@ function classify(overrides: Partial { - it("reports an unchanged incomplete scan once across startup and a day of periodic ticks", () => { - const warn = vi.fn(); - const report = createIncompleteTaskWatchdogScanReporter(warn); - report({ incomplete: 2, incompleteIssueIds: [sourceId, childId] }, "startup"); - for (let tick = 0; tick < 2880; tick += 1) { - report({ incomplete: 2, incompleteIssueIds: [childId, sourceId] }, "periodic"); - } - expect(warn).toHaveBeenCalledTimes(1); - expect(warn).toHaveBeenCalledWith( - { incomplete: 2, newIncompleteIssueIds: [sourceId, childId] }, - "startup task-watchdog reconciliation found incomplete subtree scans", - ); - }); - - it("reports new incomplete issues and reports a resolved issue if it becomes incomplete again", () => { - const warn = vi.fn(); - const report = createIncompleteTaskWatchdogScanReporter(warn); - report({ incomplete: 1, incompleteIssueIds: [sourceId] }, "startup"); - report({ incomplete: 2, incompleteIssueIds: [sourceId, childId] }, "periodic"); - expect(warn).toHaveBeenLastCalledWith( - { incomplete: 2, newIncompleteIssueIds: [childId] }, - "periodic task-watchdog reconciliation found incomplete subtree scans", - ); - report({ incomplete: 0, incompleteIssueIds: [] }, "periodic"); - expect(warn).toHaveBeenCalledTimes(2); - report({ incomplete: 1, incompleteIssueIds: [sourceId] }, "periodic"); - expect(warn).toHaveBeenCalledTimes(3); - expect(warn).toHaveBeenLastCalledWith( - { incomplete: 1, newIncompleteIssueIds: [sourceId] }, - "periodic task-watchdog reconciliation found incomplete subtree scans", - ); - }); - it("does not produce a stopped fingerprint from an incomplete scan", () => { const result = classify({ subtreeIncomplete: true }); expect(result.state).toBe("incomplete"); diff --git a/server/src/index.ts b/server/src/index.ts index cd8bb951946..fffac2a7972 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -109,7 +109,6 @@ import { createApp } from "./app.js"; import { initializeCloudRuntimeConfig } from "./services/cloud-runtime-config.js"; import { loadCloudBootstrapConfig, loadConfig, type Config } from "./config.js"; import { logger } from "./middleware/logger.js"; -import { createIncompleteTaskWatchdogScanReporter } from "./services/task-watchdog-diagnostics.js"; import { setStartupRecoveryPhase } from "./startup-recovery-state.js"; import { StartupRefusalError, @@ -973,9 +972,6 @@ async function startServerWithDatabaseTeardown( } }; const pluginWorkerManager = createPluginWorkerManager(); - const reportIncompleteWatchdogScans = createIncompleteTaskWatchdogScanReporter( - (details, message) => logger.warn(details, message), - ); const heartbeat = config.heartbeatSchedulerEnabled ? heartbeatService(db as any, { pluginWorkerManager }) : null; @@ -1778,7 +1774,6 @@ async function startServerWithDatabaseTeardown( } } - await timeStartupRecovery("queued_runs", () => heartbeat.resumeQueuedRuns()); const recoveredGoalActions = await timeStartupRecovery("goal_actions", () => heartbeat.recoverPendingSessionGoalActions()); if ( recoveredGoalActions.enqueued > 0 || @@ -1796,47 +1791,10 @@ async function startServerWithDatabaseTeardown( "startup session-goal recovery resumed durable agent goals", ); } - const reconciled = await heartbeat.reconcileStrandedAssignedIssues(); - if ( - reconciled.assignmentDispatched > 0 || - reconciled.dispatchRequeued > 0 || - reconciled.continuationRequeued > 0 || - reconciled.successfulRunHandoffEscalated > 0 || - reconciled.successfulRunHandoffRetried > 0 || - reconciled.escalated > 0 - ) { - logger.warn( - { ...reconciled }, - "startup heartbeat recovery changed assigned issue state", - ); - } - - const dependencyWakesReconciled = await heartbeat.reconcileResolvedDependencyWakes(); - if (dependencyWakesReconciled.healed > 0) { - logger.warn( - { ...dependencyWakesReconciled }, - "startup dependency-wake reconciliation restored task execution paths", - ); - } - - const taskWatchdogsReconciled = await heartbeat.reconcileTaskWatchdogs(); - reportIncompleteWatchdogScans(taskWatchdogsReconciled, "startup"); - if (taskWatchdogsReconciled.triggered > 0) { - logger.warn( - { ...taskWatchdogsReconciled }, - "startup task-watchdog reconciliation triggered watchdog work", - ); - } - const scanned = await heartbeat.scanSilentActiveRuns(); if (scanned.created > 0 || scanned.escalated > 0) { logger.warn({ ...scanned }, "startup active-run output watchdog created review work"); } - - const swept = await heartbeat.sweepStaleIssueLocks(); - if (swept.cleared > 0) { - logger.warn({ ...swept }, "startup stale-lock sweeper cleared issue locks"); - } })().catch((err) => { logger.error({ err }, "startup heartbeat recovery failed"); throw err; @@ -1845,6 +1803,10 @@ async function startServerWithDatabaseTeardown( await startupHeartbeatRecovery; } + if (config.heartbeatSchedulerEnabled) await heartbeat.registerIssueStateRecoveryWork(deliveryWork, + () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready; + if (config.heartbeatSchedulerEnabled) await heartbeat.registerDeferredReceiptWork(deliveryWork, + () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready; if (config.heartbeatSchedulerEnabled) await heartbeat.registerRunDispatchWork(deliveryWork, () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready; if (config.heartbeatSchedulerEnabled) await heartbeat.registerScheduledRetryCancellationWork(deliveryWork, @@ -2048,49 +2010,13 @@ async function startServerWithDatabaseTeardown( // Periodically reap orphaned runs (5-min staleness threshold) and make sure // persisted queued work is still being driven forward. runPeriodicRecovery(() => heartbeat - .reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true, skipRetainedNativeSessionCleanup: true, skipNativeWorkspaceFinalization: true, skipNativeResultCommit: true, skipNativeRunProjection: true, skipNativeRecoveryNotices: true, skipExternalConversationSettlement: true, skipExternalAgentSettlement: true, skipNativeEnvironmentSettlement: true }) - .then(async () => { - await heartbeat.resumeQueuedRuns(); - const reconciled = await heartbeat.reconcileStrandedAssignedIssues({ skipExternalConversationSettlement: true }); - if ( - reconciled.assignmentDispatched > 0 || - reconciled.dispatchRequeued > 0 || - reconciled.continuationRequeued > 0 || - reconciled.successfulRunHandoffEscalated > 0 || - reconciled.successfulRunHandoffRetried > 0 || - reconciled.escalated > 0 - ) { - logger.warn( - { ...reconciled }, - "periodic heartbeat recovery changed assigned issue state", - ); - } - }) - .then(async () => { - const reconciled = await heartbeat.reconcileResolvedDependencyWakes(); - if (reconciled.healed > 0) { - logger.warn({ ...reconciled }, "periodic dependency-wake reconciliation restored task execution paths"); - } - }) - .then(async () => { - const reconciled = await heartbeat.reconcileTaskWatchdogs(); - reportIncompleteWatchdogScans(reconciled, "periodic"); - if (reconciled.triggered > 0) { - logger.warn({ ...reconciled }, "periodic task-watchdog reconciliation triggered watchdog work"); - } - }) + .reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipNativeStatusWakeIntents: true, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true, skipRetainedNativeSessionCleanup: true, skipNativeWorkspaceFinalization: true, skipNativeResultCommit: true, skipNativeRunProjection: true, skipNativeRecoveryNotices: true, skipExternalConversationSettlement: true, skipExternalAgentSettlement: true, skipNativeEnvironmentSettlement: true }) .then(async () => { const scanned = await heartbeat.scanSilentActiveRuns(); if (scanned.created > 0 || scanned.escalated > 0) { logger.warn({ ...scanned }, "periodic active-run output watchdog created review work"); } }) - .then(async () => { - const swept = await heartbeat.sweepStaleIssueLocks(); - if (swept.cleared > 0) { - logger.warn({ ...swept }, "periodic stale-lock sweeper cleared issue locks"); - } - }) .catch((err) => { logger.error({ err }, "periodic heartbeat recovery failed"); })); diff --git a/server/src/modules/agent-lifecycle/__tests__/verification-retry.test.ts b/server/src/modules/agent-lifecycle/__tests__/verification-retry.test.ts index cc76c705d32..93cc524613d 100644 --- a/server/src/modules/agent-lifecycle/__tests__/verification-retry.test.ts +++ b/server/src/modules/agent-lifecycle/__tests__/verification-retry.test.ts @@ -19,8 +19,8 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu let companyId: string; const workers: Array<{ stop(): Promise }> = []; beforeAll(async () => { - if (external) { await applyPendingMigrations(external); db = createDb(external); } - else { database = await startEmbeddedPostgresTestDatabase("paperclip-verification-retry-"); db = createDb(database.connectionString); } + if (external) { await applyPendingMigrations(external); db = createDb(external, { maxConnections: 6 }); } + else { database = await startEmbeddedPostgresTestDatabase("paperclip-verification-retry-"); db = createDb(database.connectionString, { maxConnections: 6 }); } }, 90_000); afterAll(async () => { if (external) await closeRegisteredClients(external); else await database?.cleanup(); }); beforeEach(async () => { @@ -76,12 +76,22 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu it("permits an explicit retry of the same operation and completes verification before readiness", async () => { const { agent, work, runHost } = await failed(); - runHost.mockResolvedValue("complete"); - await createAgentLifecycle(db).retry(agent.id); - expect((await current(agent.id)).lifecycleOperation).not.toHaveProperty("retryRequired"); - expect((await current(agent.id)).lifecycleOperation).not.toHaveProperty("verificationFailures"); - expect((await current(agent.id)).lifecycleOperation!.id).toBe(agent.lifecycleOperation!.id); - await work.process(agent.id); + let release!: () => void; + const verification = new Promise(resolve => { release = resolve; }); + runHost.mockImplementation(async () => { await verification; return "complete"; }); + let processing: Promise | undefined; + try { + await createAgentLifecycle(db).retry(agent.id); + processing = work.process(agent.id); + const retried = await current(agent.id); + expect(retried).toMatchObject({ lifecycleState: "verifying", status: "paused" }); + expect(retried.lifecycleOperation).not.toHaveProperty("retryRequired"); + expect(retried.lifecycleOperation).not.toHaveProperty("verificationFailures"); + expect(retried.lifecycleOperation!.id).toBe(agent.lifecycleOperation!.id); + } finally { + release(); + await processing; + } expect(await current(agent.id)).toMatchObject({ lifecycleState: "ready", status: "idle", lifecycleError: null }); }); diff --git a/server/src/modules/run-dispatch/adapters/postgres.ts b/server/src/modules/run-dispatch/adapters/postgres.ts index b05f94e1a4b..5acbfcf6884 100644 --- a/server/src/modules/run-dispatch/adapters/postgres.ts +++ b/server/src/modules/run-dispatch/adapters/postgres.ts @@ -1,3 +1,4 @@ +import { notifyDeferredExecutionWork } from "../../../services/deferred-execution-notifications.js"; import { hasRequiredWorkspaceRecovery } from "../../../services/workspace-restore-recovery-state.js"; import { readNativeWorkspaceSyncReference } from "../../../services/native-runtime/native-workspace-sync.js"; import { isPreDispatchExecutionWaitVerified } from "../../../services/pre-dispatch-execution-wait.js"; @@ -996,6 +997,7 @@ export function createPostgresRunDispatchAdapter( recordedCommentId && authors.get(recordedCommentId) === actorId ? [recordedCommentId] : []; })); if (commentIds.some(id => !recordedCommentIds.has(id))) return null; + await notifyDeferredExecutionWork(tx, { companyId: run.companyId, requestId: wake.id }); const [deferred] = await tx.update(agentWakeupRequests).set({ status: "deferred_issue_execution", runId: null, claimedAt: null, finishedAt: null, error: null, payload: withQueuedCommentIdsInWakePayload({ ...payload, _paperclipWakeContext: context, diff --git a/server/src/modules/wake-queue/adapters/postgres.test.ts b/server/src/modules/wake-queue/adapters/postgres.test.ts index 068a21f2420..09fbb1749ac 100644 --- a/server/src/modules/wake-queue/adapters/postgres.test.ts +++ b/server/src/modules/wake-queue/adapters/postgres.test.ts @@ -1,3 +1,4 @@ +import { resolveExternalTestDatabaseUrl } from "../../../__tests__/helpers/external-test-database.js"; import { DELIVERY_QUEUES } from "../../../services/delivery-work-notifications.js"; import { instanceSettingsService } from "../../../services/instance-settings.js"; import { randomBytes, randomUUID } from "node:crypto"; @@ -37,7 +38,8 @@ import type { TransactionScope } from "../application/ports.js"; // a foreign-company row is invisible to a read, and a deferred-status // compare-and-set that affects no row leaves no other trace. The decision // branching itself is proven against plain facts in `domain/policy.test.ts`. -const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const external = resolveExternalTestDatabaseUrl("PAPERCLIP_WAKE_QUEUE_TEST_DATABASE_URL"); +const embeddedPostgresSupport = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; if (!embeddedPostgresSupport.supported) { @@ -57,8 +59,8 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => { }; beforeAll(async () => { - tempDb = await startEmbeddedPostgresTestDatabase("paperclip-wake-queue-postgres-adapter-"); - db = createDb(tempDb.connectionString); + if (!external) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-wake-queue-postgres-adapter-"); + db = createDb(external ?? tempDb!.connectionString, { maxConnections: 6 }); }, 20_000); afterEach(async () => { @@ -78,6 +80,7 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => { }); afterAll(async () => { + await db?.$client.end({ timeout: 1 }); await tempDb?.cleanup(); }); diff --git a/server/src/modules/wake-queue/adapters/postgres.ts b/server/src/modules/wake-queue/adapters/postgres.ts index 1df4420615f..e7a5f12f3f2 100644 --- a/server/src/modules/wake-queue/adapters/postgres.ts +++ b/server/src/modules/wake-queue/adapters/postgres.ts @@ -1,3 +1,4 @@ +import { notifyDeferredExecutionWork } from "../../../services/deferred-execution-notifications.js"; import { notifyRunDispatch } from "../../../services/run-dispatch-notifications.js"; import { notifyExternalConversationSettlement } from "../../../services/external-conversation-settlement-notifications.js"; import { notifyNativeStatusReassessment } from "../../../services/native-status-reassessment-notifications.js"; @@ -1010,6 +1011,11 @@ export function createWakeAdmissionWriter(): WakeAdmissionWriter { )).for("update"); if (!prior) throw new Error("wake-queue: the deferred wake to merge into was not found for this company"); const priorPayload = parseObject(prior.payload), issueId = readNonEmptyString(priorPayload.issueId); + const previousComments = new Set(queuedCommentIdsFromWakePayload(priorPayload)); + const newComments = queuedCommentIdsFromWakePayload(input.mergedPayload).some(id => !previousComments.has(id)); + const newInteraction = readNonEmptyString(input.mergedPayload.interactionId); + if (newComments || newInteraction && newInteraction !== readNonEmptyString(priorPayload.interactionId)) + await notifyDeferredExecutionWork(tx, { companyId: input.companyId, requestId: prior.id }); const priorCommentId = readNonEmptyString(priorPayload.commentId), priorActor = readNonEmptyString(prior.requestedByActorId); const [priorComment] = prior.requestedByActorType === "user" && priorActor && issueId && isUuidLike(issueId) && priorCommentId && isUuidLike(priorCommentId) ? await tx.select({ id: issueComments.id, body: issueComments.body }).from(issueComments).where(and( @@ -1081,6 +1087,7 @@ export function createWakeAdmissionWriter(): WakeAdmissionWriter { async insertNewDeferredWake(scope, input) { const tx = requireAdmissionTx(scope, input.companyId); + await notifyDeferredExecutionWork(tx, { companyId: input.companyId }); await tx.insert(agentWakeupRequests).values({ ...input.durableReceipt, companyId: input.companyId, diff --git a/server/src/routes/issues.ts b/server/src/routes/issues.ts index 70cead48cfa..9be40978e26 100644 --- a/server/src/routes/issues.ts +++ b/server/src/routes/issues.ts @@ -1,3 +1,4 @@ +import { notifyDeferredExecutionWork } from "../services/deferred-execution-notifications.js"; import { notifyNativeRecoveryNotices } from "../services/native-recovery-notice-notifications.js"; import { notifyNativeStatusReassessment } from "../services/native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../services/unsafe-workspace-export-notifications.js"; @@ -16687,6 +16688,7 @@ export function issueRoutes( } // The click is durable fresh user intent, including when the message // predates a failed run's stop. Keep its content and original attribution. + await notifyDeferredExecutionWork(tx, { companyId: issue.companyId, requestId: locked.wake.id }); await tx.update(agentWakeupRequests).set({ payload: { ...readObject(locked.wake.payload), queuedCommentInterrupt: { actorId: actor.actorId, requestedAt: new Date().toISOString(), diff --git a/server/src/services/access.ts b/server/src/services/access.ts index aa0f9dcc828..32086736eb2 100644 --- a/server/src/services/access.ts +++ b/server/src/services/access.ts @@ -1,3 +1,4 @@ +import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; @@ -743,6 +744,7 @@ export function accessService(db: Db) { sql`${issues.status} not in ('done', 'cancelled')`, ); await notifyNativeStatusReassessment(tx, { companyId }); + await notifyDeferredExecutionEligibility(tx, { companyId }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); const resetInProgress = await tx diff --git a/server/src/services/agent-configuration.ts b/server/src/services/agent-configuration.ts index dc10efaa082..2d30da27cb7 100644 --- a/server/src/services/agent-configuration.ts +++ b/server/src/services/agent-configuration.ts @@ -1,3 +1,4 @@ +import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js"; import { notifyRunDispatch } from "./run-dispatch-notifications.js"; import { notifySkillVersionPinChange } from "./skill-version-retention-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; @@ -85,6 +86,7 @@ export async function prepareAgentConfiguration(db: Db, existing: AgentConfigura ); } await notifyRunDispatch(db, { companyId: existing.companyId }); + await notifyDeferredExecutionEligibility(db, { companyId: existing.companyId }); if (normalizedPatch.runtimeConfig !== undefined || normalizedPatch.reportsTo !== undefined) { await notifyDeliveryWork(db, DELIVERY_QUEUES.agentTimers); } diff --git a/server/src/services/agent-conversations.ts b/server/src/services/agent-conversations.ts index a6b70361945..d491c104684 100644 --- a/server/src/services/agent-conversations.ts +++ b/server/src/services/agent-conversations.ts @@ -1,3 +1,4 @@ +import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; import { FAST_RESPONSE_AGENT_GUIDANCE, fastResponseHistoryBody } from "@paperclipai/shared"; @@ -547,6 +548,7 @@ export async function resumeConversationForReset(db: Db, comment: typeof issueCo eq(issues.id, comment.issueId), eq(issues.companyId, comment.companyId), )).for("update"); if (!isConversation(issue) || comment.conversationSessionGeneration != null) return; + await notifyIssueRecovery(tx, { companyId: issue.companyId }); const released = await tx.update(issueTreeHolds).set({ status: "released", releasedAt: new Date(), updatedAt: new Date(), releasedByActorType: "user", releasedByUserId: comment.authorUserId, diff --git a/server/src/services/agent-lifecycle.ts b/server/src/services/agent-lifecycle.ts index 59357fcc5de..8bb46882256 100644 --- a/server/src/services/agent-lifecycle.ts +++ b/server/src/services/agent-lifecycle.ts @@ -1,3 +1,4 @@ +import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js"; import { notifyRunDispatch } from "./run-dispatch-notifications.js"; import { notifyNativeReviewResponse } from "./native-review-response-notifications.js"; import { isAgentStatusInvokable } from "@paperclipai/shared"; @@ -41,6 +42,7 @@ export function createAgentLifecycleEffects(_hooks: BudgetServiceHooks = {}): li async beforeStatusChange(tx, companyId, agentId, before, after) { if (before !== after) { await notifyRunDispatch(tx, { companyId }); + await notifyDeferredExecutionEligibility(tx, { companyId }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); } if (!isAgentStatusInvokable(before) && isAgentStatusInvokable(after)) { diff --git a/server/src/services/budgets.ts b/server/src/services/budgets.ts index 21ae37eda03..9a18c8069a6 100644 --- a/server/src/services/budgets.ts +++ b/server/src/services/budgets.ts @@ -1,3 +1,4 @@ +import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js"; import { notifyRunDispatch } from "./run-dispatch-notifications.js"; import { notifyApprovalWork } from "./approval-work-notifications.js"; import { notifyCompanyRetentionWork } from "./retention-work-notifications.js"; @@ -301,7 +302,13 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica } if (policy.scopeType === "project") { + // Admission rechecks the same hard stop. Lock current eligibility before + // publishing intent so a no-op pause cannot reactivate its own receipts. + const [project] = await db.select({ pausedAt: projects.pausedAt }).from(projects) + .where(and(eq(projects.id, policy.scopeId), eq(projects.companyId, policy.companyId))).for("update"); + if (!project || project.pausedAt !== null) return; await notifyRunDispatch(db, { companyId: policy.companyId }); + await notifyDeferredExecutionEligibility(db, { companyId: policy.companyId }); await db .update(projects) .set({ @@ -309,13 +316,14 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica pausedAt: now, updatedAt: now, }) - .where(and(eq(projects.id, policy.scopeId), isNull(projects.pausedAt))); + .where(and(eq(projects.id, policy.scopeId), eq(projects.companyId, policy.companyId), isNull(projects.pausedAt))); return; } const scope = await resolveScopeRecord(db, "company", policy.scopeId); if (scope.paused) return; await notifyRunDispatch(db, { companyId: policy.companyId }); + await notifyDeferredExecutionEligibility(db, { companyId: policy.companyId }); await notifyCompanyRetentionWork(db, policy.companyId); await notifyDeliveryWork(db, DELIVERY_QUEUES.agentTimers); await notifyDeliveryWork(db, DELIVERY_QUEUES.routineSchedules); @@ -363,6 +371,7 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica if (policy.scopeType === "project") { await notifyRunDispatch(db, { companyId: policy.companyId }); + await notifyDeferredExecutionEligibility(db, { companyId: policy.companyId }); await db .update(projects) .set({ @@ -378,6 +387,7 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica const scope = await resolveScopeRecord(db, "company", policy.scopeId); if (scope.pauseReason !== "budget") return; await notifyRunDispatch(db, { companyId: policy.companyId }); + await notifyDeferredExecutionEligibility(db, { companyId: policy.companyId }); await notifyCompanyRetentionWork(db, policy.companyId); await notifyDeliveryWork(db, DELIVERY_QUEUES.agentTimers); await notifyDeliveryWork(db, DELIVERY_QUEUES.routineSchedules); diff --git a/server/src/services/chat-channels/failed-run-retry.ts b/server/src/services/chat-channels/failed-run-retry.ts index c324a6f3c34..f8b8c3ebeab 100644 --- a/server/src/services/chat-channels/failed-run-retry.ts +++ b/server/src/services/chat-channels/failed-run-retry.ts @@ -1,3 +1,4 @@ +import { notifyDeferredExecutionWork, notifyTaskRetrySettlement } from "../deferred-execution-notifications.js"; import { notifyNativeResponsePresentation } from "../native-response-presentation-notifications.js"; import { notifyChatInboundWakeNotices } from "../chat-inbound-wake-notice-notifications.js"; import { createFailedRetryNotices } from "./failed-retry-notices.js"; @@ -1424,6 +1425,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti // Source authorization above checked every external receipt and its owner. // Reset only this certified unstarted batch under the task's admission lock. const skippedIds = source.taskRetryInput?.skippedActionIds ?? []; + await notifyDeferredExecutionWork(tx, { companyId: input.companyId }); await tx.update(agentWakeupRequests).set({ status: "deferred_issue_execution", runId: null, finishedAt: null, payload: sql`${agentWakeupRequests.payload} - 'coalescedIntoWakeupRequestId'`, updatedAt: new Date(), }).where(and(eq(agentWakeupRequests.companyId, input.companyId), eq(agentWakeupRequests.agentId, input.agentId), @@ -1784,6 +1786,9 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti const attemptCount = Number(candidate.result?.attemptCount ?? 0) + 1; const settle = async (status: string, result: Record) => { const [current] = await db.transaction(async (tx) => { + if (typeof claimed.payload.taskRetryIntentId === "string") await notifyTaskRetrySettlement(tx, { + companyId: claimed.companyId, requestId: claimed.payload.taskRetryIntentId, + }); await notifyChatActionWork(tx, "failed_run_retry"); if (status === "processed") await notifyNativeResponsePresentation(tx, { companyId: claimed.companyId, issueId: String(claimed.payload.issueId), diff --git a/server/src/services/chat-channels/task-retry-input.ts b/server/src/services/chat-channels/task-retry-input.ts index 71743f42ad4..8257a8ec079 100644 --- a/server/src/services/chat-channels/task-retry-input.ts +++ b/server/src/services/chat-channels/task-retry-input.ts @@ -1,3 +1,4 @@ +import { notifyDeferredExecutionWork } from "../deferred-execution-notifications.js"; import { z } from "zod"; import { and, asc, eq, gt, inArray, isNull, or, sql } from "drizzle-orm"; import { agentWakeupRequests, chatActions, chatConversations, chatEndpoints, heartbeatRuns, issues, type Db } from "@paperclipai/db"; @@ -91,6 +92,7 @@ export async function releaseUnadmittedTaskRetryInput(db: Db, input: { companyId )).for("update"); if (receipt?.runId) return; await notifyChatInboundWakeNotices(tx, { companyId: input.companyId, issueId: issue.id }); + await notifyDeferredExecutionWork(tx, { companyId: input.companyId }); await tx.update(agentWakeupRequests).set({ status: "deferred_issue_execution", finishedAt: null, payload: sql`${agentWakeupRequests.payload} - 'coalescedIntoWakeupRequestId'`, updatedAt: new Date(), }).where(and(eq(agentWakeupRequests.companyId, input.companyId), eq(agentWakeupRequests.status, "coalesced"), diff --git a/server/src/services/chat-channels/telegram/attachment-failure.ts b/server/src/services/chat-channels/telegram/attachment-failure.ts index 58d5d4a2677..785b3116baf 100644 --- a/server/src/services/chat-channels/telegram/attachment-failure.ts +++ b/server/src/services/chat-channels/telegram/attachment-failure.ts @@ -1,3 +1,4 @@ +import { notifyIssueRecovery } from "../../issue-recovery-notifications.js"; import { createProviderEffects, type ProviderEffectTarget } from "../provider-effects.js"; import type { EndpointRow, RuntimeContext, InboundRuntimeContext } from "../runtime-types.js"; @@ -78,6 +79,7 @@ export function createTelegramAttachmentFailure(db: Db, options: TelegramAttachm })(), ); const effect = await db.transaction(async (tx) => { + await notifyIssueRecovery(tx, { companyId: input.endpoint.companyId }); // Replace the optimistic attachment placeholder with an explicit safe // Paperclip-visible failure. The message link stays authoritative for // exact provider redelivery, while no agent wakeup is enqueued for an diff --git a/server/src/services/companies.ts b/server/src/services/companies.ts index a9a8f59998c..f0522ee4ddb 100644 --- a/server/src/services/companies.ts +++ b/server/src/services/companies.ts @@ -1,3 +1,4 @@ +import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js"; import { notifyRunDispatch } from "./run-dispatch-notifications.js"; import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js"; import { notifyCompanyRetentionWork } from "./retention-work-notifications.js"; @@ -334,6 +335,7 @@ export function companyService(db: Db, _budgetHooks: BudgetServiceHooks = {}) { if (companyPatch.status !== undefined && companyPatch.status !== existing.status) { await notifyRunDispatch(tx, { companyId: id }); + await notifyDeferredExecutionEligibility(tx, { companyId: id }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); await notifyDeliveryWork(tx, DELIVERY_QUEUES.routineSchedules); @@ -491,6 +493,7 @@ export function companyService(db: Db, _budgetHooks: BudgetServiceHooks = {}) { if (!wasAlreadyArchived) { await notifyRunDispatch(tx, { companyId: id }); + await notifyDeferredExecutionEligibility(tx, { companyId: id }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); await notifyDeliveryWork(tx, DELIVERY_QUEUES.routineSchedules); diff --git a/server/src/services/company-deletion.ts b/server/src/services/company-deletion.ts index 7ab88aadfa9..26bc4e081a1 100644 --- a/server/src/services/company-deletion.ts +++ b/server/src/services/company-deletion.ts @@ -1,3 +1,4 @@ +import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "./cancelled-workspace-preservation-notifications.js"; import { notifyInstructionCollection } from "./instruction-collection-notifications.js"; @@ -71,6 +72,7 @@ export async function deleteCompany(db: Db, id: string) { const [existing] = await tx.select({ id: companies.id }).from(companies) .where(eq(companies.id, id)).for("no key update"); if (!existing) return null; + await notifyIssueRecovery(tx, { companyId: id }); await notifyChatCompanyConfiguration(tx, id); await chatCompanyDeletion.deleteCompanyData(tx, id); await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); diff --git a/server/src/services/company-portability.ts b/server/src/services/company-portability.ts index dc7dca311d6..585b8fcf5f4 100644 --- a/server/src/services/company-portability.ts +++ b/server/src/services/company-portability.ts @@ -1,3 +1,4 @@ +import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import { createAgentLifecycle } from "./agent-lifecycle.js"; import { agentAppearanceSchema, agentHarnessType, paperclipRunnerProfileForHarness, agentRunner, type AgentRunnerChoice } from "@paperclipai/shared"; import { resolveNewAgentRunnerForCompany } from "./agent-runner-selection.js"; @@ -6465,7 +6466,9 @@ export function companyPortabilityService(db: Db, storage?: StorageService) { } if (relationRows.length > 0) { const relationCompanyId = targetCompany.id; - await db + await db.transaction(async tx => { + await notifyIssueRecovery(tx, { companyId: relationCompanyId }); + await tx .insert(issueRelations) .values(relationRows.map((row) => ({ companyId: relationCompanyId, @@ -6476,6 +6479,7 @@ export function companyPortabilityService(db: Db, storage?: StorageService) { createdByUserId: actorUserId ?? null, }))) .onConflictDoNothing(); + }); } } diff --git a/server/src/services/deferred-execution-notifications.ts b/server/src/services/deferred-execution-notifications.ts new file mode 100644 index 00000000000..b82cea725a5 --- /dev/null +++ b/server/src/services/deferred-execution-notifications.ts @@ -0,0 +1,34 @@ +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; + +export type DeferredExecutionScope = { companyId: string; requestId?: string }; +export type DeferredReceiptRecoveryOptions = { + companyId?: string; + requestId?: string; + onError?: (error: unknown) => void; +}; + +function hint(scope: DeferredExecutionScope) { + return JSON.stringify({ companyId: scope.companyId, requestId: scope.requestId }); +} + +export async function notifyDeferredExecutionWork(tx: object, scope: DeferredExecutionScope) { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.deferredExecutionReceipts, hint(scope)); +} + +export async function notifyNativeStatusWakeWork(tx: object, scope: DeferredExecutionScope) { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.nativeStatusWakeIntents, hint(scope)); +} + +export async function notifyTaskRetrySettlement(tx: object, scope: DeferredExecutionScope) { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.taskRetryReceiptSettlement, hint(scope)); +} + +export async function notifyDeferredExecutionEligibility(tx: object, scope?: { companyId: string }) { + if (scope) { + await notifyDeferredExecutionWork(tx, scope); + await notifyNativeStatusWakeWork(tx, scope); + } else { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.deferredExecutionReceipts, "*"); + await notifyDeliveryWork(tx, DELIVERY_QUEUES.nativeStatusWakeIntents, "*"); + } +} diff --git a/server/src/services/deferred-execution-wait.ts b/server/src/services/deferred-execution-wait.ts new file mode 100644 index 00000000000..6924cb4e1ae --- /dev/null +++ b/server/src/services/deferred-execution-wait.ts @@ -0,0 +1,33 @@ +import { HttpError } from "../errors.js"; +import { and, eq, sql } from "drizzle-orm"; +import { agentWakeupRequests, type Db } from "@paperclipai/db"; +import { notifyDeferredExecutionWork } from "./deferred-execution-notifications.js"; + +/** Admission has saved this refusal on the caller's durable deferred receipt. + * HTTP callers still receive the existing 409; receipt recovery waits for its + * owning gate producer instead of treating the refusal as infrastructure debt. */ +export class PersistedExecutionWaitRefusal extends HttpError { + constructor(message: string, details?: unknown) { super(409, message, details); } +} + +export async function writeDeferredExecutionWait(db: Db, scope: { companyId: string; requestId: string }, + condition: Record, options: { replace?: boolean } = {}) { + return db.transaction(async tx => { + const [current] = await tx.select({ payload: agentWakeupRequests.payload }).from(agentWakeupRequests) + .where(and(eq(agentWakeupRequests.companyId, scope.companyId), eq(agentWakeupRequests.id, scope.requestId), + eq(agentWakeupRequests.status, "deferred_issue_execution"))).for("update"); + if (!current) return false; + const previous = current.payload?.executionWait; + const previousReason = previous && typeof previous === "object" && !Array.isArray(previous) + ? (previous as Record).reason : undefined; + if (!previous || previousReason !== condition.reason) await notifyDeferredExecutionWork(tx, scope); + const [saved] = await tx.update(agentWakeupRequests).set({ + payload: sql`jsonb_set(coalesce(${agentWakeupRequests.payload}, '{}'::jsonb), '{executionWait}', + ${options.replace ? sql`'{}'::jsonb` : sql`coalesce(${agentWakeupRequests.payload}->'executionWait', '{}'::jsonb)`} + || ${JSON.stringify(condition)}::jsonb)`, + updatedAt: new Date(), + }).where(and(eq(agentWakeupRequests.companyId, scope.companyId), eq(agentWakeupRequests.id, scope.requestId), + eq(agentWakeupRequests.status, "deferred_issue_execution"))).returning({ id: agentWakeupRequests.id }); + return Boolean(saved); + }); +} diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index 7388bdb8861..79f0f5d1d3d 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -2,6 +2,15 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db"; /** Topics for existing durable queues; register intent before their writes. */ export const DELIVERY_QUEUES = { + deferredExecutionReceipts: "deferred-execution-receipts", + taskRetryReceiptSettlement: "task-retry-receipt-settlement", + nativeStatusWakeIntents: "native-status-wake-intents", + orphanBlockers: "orphan-blockers", + activeRecoveryActions: "active-recovery-actions", + strandedAssignments: "stranded-assignments", + resolvedDependencies: "resolved-dependencies", + taskWatchdogs: "task-watchdogs", + staleIssueLocks: "stale-issue-locks", scheduledRetryDispatch: "scheduled-retry-dispatch", queuedRunDispatch: "queued-run-dispatch", runEvidenceMigration: "run-evidence-migration", diff --git a/server/src/services/documents.ts b/server/src/services/documents.ts index 493c495e007..814c68eeea7 100644 --- a/server/src/services/documents.ts +++ b/server/src/services/documents.ts @@ -1,3 +1,4 @@ +import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import { randomUUID } from "node:crypto"; import { and, asc, desc, eq, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; @@ -275,6 +276,7 @@ export function documentService(db: Db) { .where(eq(issueDocuments.issueId, issue.id)); const fallbackKey = nextAvailableDocumentKey(key, issueDocumentKeys.map((row) => row.key)); + await notifyIssueRecovery(tx, { companyId: issue.companyId }); const [document] = await tx .insert(documents) .values({ @@ -376,6 +378,7 @@ export function documentService(db: Db) { }); } + await notifyIssueRecovery(tx, { companyId: issue.companyId }); const nextRevisionNumber = existing.latestRevisionNumber + 1; const [revision] = await tx .insert(documentRevisions) @@ -438,6 +441,7 @@ export function documentService(db: Db) { throw conflict("Document does not exist yet", { key }); } + await notifyIssueRecovery(tx, { companyId: issue.companyId }); const [document] = await tx .insert(documents) .values({ @@ -592,6 +596,9 @@ export function documentService(db: Db) { }); } await db.transaction(async (tx) => { + for (const companyId of new Set(rows.map(row => row.companyId))) { + await notifyIssueRecovery(tx, { companyId }); + } await insertRowsInChunks(tx, documents, documentRows); await insertRowsInChunks(tx, documentRevisions, revisionRows); await insertRowsInChunks(tx, issueDocuments, linkRows); @@ -644,6 +651,7 @@ export function documentService(db: Db) { }); } + await notifyIssueRecovery(tx, { companyId: existing.companyId }); const now = new Date(); const nextRevisionNumber = existing.latestRevisionNumber + 1; const [restoredRevision] = await tx @@ -722,6 +730,7 @@ export function documentService(db: Db) { }; } + await notifyIssueRecovery(tx, { companyId: existing.companyId }); const now = new Date(); await tx .update(documents) @@ -769,6 +778,7 @@ export function documentService(db: Db) { }; } + await notifyIssueRecovery(tx, { companyId: existing.companyId }); const now = new Date(); await tx .update(documents) @@ -817,6 +827,7 @@ export function documentService(db: Db) { }); } + await notifyIssueRecovery(tx, { companyId: existing.companyId }); await tx.delete(issueDocuments).where(eq(issueDocuments.documentId, existing.id)); await tx.delete(documents).where(eq(documents.id, existing.id)); diff --git a/server/src/services/execution-control-notifications.ts b/server/src/services/execution-control-notifications.ts index 1839176769f..7d9530f5197 100644 --- a/server/src/services/execution-control-notifications.ts +++ b/server/src/services/execution-control-notifications.ts @@ -1,3 +1,4 @@ +import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js"; import { notifyRunDispatch } from "./run-dispatch-notifications.js"; import type { Db } from "@paperclipai/db"; import { DELIVERY_QUEUES, notifyDeliveryWork, type DeliveryQueue } from "./delivery-work-notifications.js"; @@ -5,7 +6,10 @@ import { DELIVERY_QUEUES, notifyDeliveryWork, type DeliveryQueue } from "./deliv export type ExecutionWorkScope = { companyId: string; agentId?: string; runId?: string }; export async function notifyExecutionRunWork(transaction: object, scope?: ExecutionWorkScope) { - if (scope) await notifyRunDispatch(transaction, scope); + if (scope) { + await notifyRunDispatch(transaction, scope); + await notifyDeferredExecutionEligibility(transaction, scope); + } for (const queue of [DELIVERY_QUEUES.executionStatus, DELIVERY_QUEUES.executionFinalization, DELIVERY_QUEUES.executionReplacement, DELIVERY_QUEUES.executionDisposition, DELIVERY_QUEUES.issueMonitors]) { await notifyDeliveryWork(transaction, queue); @@ -20,6 +24,7 @@ export function writeExecutionControl( ): Promise { return db.transaction(async transaction => { await notifyRunDispatch(transaction, scope); + if (scope) await notifyDeferredExecutionEligibility(transaction, scope); await notifyDeliveryWork(transaction, DELIVERY_QUEUES.issueMonitors); for (const topic of typeof queue === "string" ? [queue] : queue) await notifyDeliveryWork(transaction, topic); return write(transaction as unknown as Db); diff --git a/server/src/services/external-conversation-settlement-notifications.ts b/server/src/services/external-conversation-settlement-notifications.ts index 46f4493ff2a..d5e4cdd506c 100644 --- a/server/src/services/external-conversation-settlement-notifications.ts +++ b/server/src/services/external-conversation-settlement-notifications.ts @@ -1,7 +1,9 @@ +import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import type { ExternalConversationSettlementScope } from "./slack-conversation-lifecycle.js"; -export function notifyExternalConversationSettlement(tx: object, scope?: ExternalConversationSettlementScope) { +export async function notifyExternalConversationSettlement(tx: object, scope?: ExternalConversationSettlementScope) { + await notifyIssueRecovery(tx, scope); return notifyDeliveryWork(tx, DELIVERY_QUEUES.externalConversationSettlement, scope ? JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId }) : "*"); } diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 0fdefcf3109..43cd368ea00 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1,3 +1,6 @@ +import { registerDeferredReceiptWork } from "./heartbeat/deferred-receipt-work.js"; +import { reconcileTaskChatRetryReceipts } from "./task-retry.js"; +import { registerIssueRecoveryWork } from "./recovery/work.js"; import { registerRunDispatchWork } from "./heartbeat/dispatch-work.js"; import { NATIVE_RETAINED_OWNER_WAIT } from "./native-runtime/native-controller-recovery.js"; import { externalConversationSettlementWork } from "./external-conversation-settlement-work.js"; @@ -1618,15 +1621,15 @@ export function heartbeatService( void reportRunFailure(db, updated, failureReport); } - async function reconcileStrandedAssignedIssues(opts?: { skipExternalConversationSettlement?: boolean }) { + async function reconcileStrandedAssignedIssues(opts?: Parameters[0]) { return recovery.reconcileStrandedAssignedIssues({ - skipExternalConversationSettlement: opts?.skipExternalConversationSettlement, + ...opts, issueCreatedAtGte: await getWorktreeExecutionCutoff(), }); } - async function sweepStaleIssueLocks() { - return recovery.sweepStaleIssueLocks(); + async function sweepStaleIssueLocks(opts?: Parameters[0]) { + return recovery.sweepStaleIssueLocks(opts); } function issueIdFromRunContext(contextSnapshot: unknown) { @@ -1656,10 +1659,7 @@ export function heartbeatService( }); } - async function reconcileTaskWatchdogs(opts?: { - companyId?: string | null; - runId?: string | null; - }) { + async function reconcileTaskWatchdogs(opts?: Parameters[0]) { return taskWatchdogs.reconcileTaskWatchdogs({ ...opts, issueCreatedAtGte: await getWorktreeExecutionCutoff(), @@ -1684,10 +1684,7 @@ export function heartbeatService( return recovery.buildRunOutputSilence(run, now); } - async function reconcileResolvedDependencyWakes(opts?: { - runId?: string | null; - companyId?: string | null; - }) { + async function reconcileResolvedDependencyWakes(opts?: Parameters[0]) { return recovery.reconcileResolvedDependencyWakeBackstop(opts); } @@ -4236,6 +4233,23 @@ export function heartbeatService( computeTaskDrain, applyTaskDrain, + registerDeferredReceiptWork: (coordinator: Parameters[1], canRun: () => boolean) => + registerDeferredReceiptWork(db, coordinator, { + resume: scope => heartbeatQueue.resumeDeferredExecutionReceipts(scope), + settleTaskChat: scope => reconcileTaskChatRetryReceipts(db, { ...scope, settleOnly: true }), + createNativeBatch: heartbeatQueue.createNativeStatusWakeBatch, + schedulingSuppressed: async () => (await getSchedulingSuppression(true)).suppressed, + }, () => canRun() && readTaskDrain(new Date()) === null), + registerIssueStateRecoveryWork: (coordinator: Parameters[1], canRun: () => boolean) => + registerIssueRecoveryWork(db, coordinator, { + stranded: scope => reconcileStrandedAssignedIssues({ ...scope, skipExternalConversationSettlement: true, skipRelatedRecovery: true }), + orphanBlockers: recovery.reconcileUnassignedBlockingIssues, + recoveryActions: recovery.reconcileActiveRecoveryActions, + dependencies: reconcileResolvedDependencyWakes, + locks: sweepStaleIssueLocks, + watchdogs: reconcileTaskWatchdogs, + schedulingSuppressed: async () => (await getSchedulingSuppression(true)).suppressed, + }, () => canRun() && readTaskDrain(new Date()) === null), registerRunDispatchWork: (coordinator: Parameters[0], canRun: () => boolean) => registerRunDispatchWork(coordinator, { promoteDueScheduledRetries, diff --git a/server/src/services/heartbeat/deferred-receipt-work.ts b/server/src/services/heartbeat/deferred-receipt-work.ts new file mode 100644 index 00000000000..9cca7528228 --- /dev/null +++ b/server/src/services/heartbeat/deferred-receipt-work.ts @@ -0,0 +1,128 @@ +import { PersistedExecutionWaitRefusal } from "../deferred-execution-wait.js"; +import { and, asc, eq, gt, inArray, isNull, or, sql } from "drizzle-orm"; +import { agentWakeupRequests, type Db } from "@paperclipai/db"; +import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; +import type { DeferredExecutionScope, DeferredReceiptRecoveryOptions } from "../deferred-execution-notifications.js"; +import { registerScopedDeadlineWork } from "../scoped-deadline-work.js"; + +type Entry = { id: string; companyId: string; dueAt: number }; +type NativeIntentOptions = DeferredExecutionScope & { + onDeferredDeadline(at: number): void; + onError(error: unknown): void; +}; + +export function registerDeferredReceiptWork( + db: Db, + coordinator: ReturnType, + owner: { + resume(scope: DeferredReceiptRecoveryOptions): Promise; + settleTaskChat(scope: DeferredExecutionScope): Promise; + createNativeBatch(): (scope: NativeIntentOptions) => Promise; + schedulingSuppressed(): Promise; + }, + canRun: () => boolean, +) { + function register(kind: "deferred" | "task_chat" | "native") { + const cursors = new Map(); + let enabled = false; + let dispatchNative: ReturnType; + const queue = kind === "deferred" ? DELIVERY_QUEUES.deferredExecutionReceipts + : kind === "task_chat" ? DELIVERY_QUEUES.taskRetryReceiptSettlement : DELIVERY_QUEUES.nativeStatusWakeIntents; + return registerScopedDeadlineWork(coordinator, { + queue, + batchSize: kind === "native" ? 100 : undefined, + reconstructionIntervalMs: 30_000, + beforePass: async () => { + enabled = !(await owner.schedulingSuppressed()); + if (kind === "native") dispatchNative = owner.createNativeBatch(); + }, + decodeScope: hint => hint === "*" ? undefined : JSON.parse(hint), + matches: (entry, scope) => !scope || entry.companyId === scope.companyId && (!scope.requestId || entry.id === scope.requestId), + load: async scope => { + const key = scope ? JSON.stringify({ companyId: scope.companyId, requestId: scope.requestId }) : "*"; + if (!enabled) { cursors.delete(key); return []; } + const cursor = cursors.get(key); + const rows = await db.select({ id: agentWakeupRequests.id, companyId: agentWakeupRequests.companyId, + status: agentWakeupRequests.status, updatedAt: agentWakeupRequests.updatedAt, + claimedAt: agentWakeupRequests.claimedAt, error: agentWakeupRequests.error, + delayed: sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt' is not null + or ${agentWakeupRequests.payload}->'executionWait' is not null`, + interruptedLegacy: sql`exists (select 1 from heartbeat_runs r + where r.company_id = ${agentWakeupRequests.companyId} and r.agent_id = ${agentWakeupRequests.agentId} + and r.runtime_mode = 'legacy' and r.status = 'cancelled' + and r.result_json->>'queuedCommentInterruptQueueId' = ${agentWakeupRequests.id}::text)`, + }).from(agentWakeupRequests).where(and( + scope ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined, + scope?.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined, + cursor ? gt(agentWakeupRequests.id, cursor) : undefined, + kind === "native" ? and( + eq(agentWakeupRequests.requestedByActorType, "system"), + eq(agentWakeupRequests.requestedByActorId, "native-status-committer"), + inArray(agentWakeupRequests.status, ["queued", "claimed"]), isNull(agentWakeupRequests.runId), + ) : and(eq(agentWakeupRequests.status, "deferred_issue_execution"), + kind === "task_chat" ? and(eq(agentWakeupRequests.reason, "task_retry"), + sql`${agentWakeupRequests.payload}#>>'{taskRetry,chatActionId}' is not null`) : or( + eq(agentWakeupRequests.reason, "task_retry"), + sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt' is not null`, + sql`${agentWakeupRequests.payload}->'executionWait' is not null`, + sql`jsonb_typeof(${agentWakeupRequests.payload}#>'{_paperclipWakeContext,wakeCommentIds}') = 'array' + and ${agentWakeupRequests.payload}#>'{_paperclipWakeContext,wakeCommentIds}' <> '[]'::jsonb`, + sql`${agentWakeupRequests.payload}->>'mutation' = 'interaction'`, + sql`exists (select 1 from heartbeat_runs r where r.company_id = ${agentWakeupRequests.companyId} + and r.runtime_mode = 'legacy' and r.status = 'cancelled' + and r.result_json->>'queuedCommentInterruptQueueId' = ${agentWakeupRequests.id}::text)`, + )), + )).orderBy(asc(agentWakeupRequests.id)).limit(100); + const complete = rows.length < 100; + if (complete) cursors.delete(key); else cursors.set(key, rows.at(-1)!.id); + return { rows: rows.map(row => ({ id: row.id, companyId: row.companyId, + dueAt: kind === "native" && row.status === "claimed" && row.claimedAt ? row.claimedAt.getTime() + 60_000 + : kind === "native" && row.error ? row.updatedAt.getTime() + 30_000 + : kind === "deferred" && row.delayed && !row.interruptedLegacy ? row.updatedAt.getTime() + 30_000 + : Date.now(), + })), complete }; + }, + check: async entry => { + if (!enabled) return null; + if (kind !== "task_chat") { + const [current] = await db.select({ status: agentWakeupRequests.status, payload: agentWakeupRequests.payload, + updatedAt: agentWakeupRequests.updatedAt, claimedAt: agentWakeupRequests.claimedAt, error: agentWakeupRequests.error, + interruptedLegacy: sql`exists (select 1 from heartbeat_runs r + where r.company_id = ${agentWakeupRequests.companyId} and r.agent_id = ${agentWakeupRequests.agentId} + and r.runtime_mode = 'legacy' and r.status = 'cancelled' + and r.result_json->>'queuedCommentInterruptQueueId' = ${agentWakeupRequests.id}::text)`, + }).from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, entry.companyId), + eq(agentWakeupRequests.id, entry.id))); + if (!current) return null; + const due = kind === "native" && current.status === "claimed" && current.claimedAt + ? current.claimedAt.getTime() + 60_000 + : kind === "native" && current.status === "queued" && current.error ? current.updatedAt.getTime() + 30_000 + : kind === "deferred" && current.status === "deferred_issue_execution" && !current.interruptedLegacy + && (current.payload?.queuedCommentInterrupt || current.payload?.executionWait) + ? current.updatedAt.getTime() + 30_000 : null; + if (due !== null && due > Date.now()) return { ...entry, dueAt: due }; + } + const errors: unknown[] = []; + let next: number | null = null; + const scope = { companyId: entry.companyId, requestId: entry.id }; + if (kind === "native") await dispatchNative({ ...scope, + onDeferredDeadline: at => { next = next === null ? at : Math.min(next, at); }, + onError: error => { errors.push(error); }, + }); + else if (kind === "task_chat") await owner.settleTaskChat(scope); + else { + const failed = (error: unknown) => { if (!(error instanceof PersistedExecutionWaitRefusal)) errors.push(error); }; + try { await owner.resume({ ...scope, onError: failed }); } + catch (error) { failed(error); } + } + if (errors.length) throw new AggregateError(errors, `Deferred receipt ${entry.id} could not settle`); + return next === null ? null : { ...entry, dueAt: next }; + }, + }, canRun); + } + const workers = [register("deferred"), register("task_chat"), register("native")]; + return { ready: Promise.all(workers.map(worker => worker.ready)), + wake() { for (const worker of workers) worker.invalidate(); }, + async stop() { await Promise.all(workers.map(worker => worker.stop())); } }; +} diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts index 64ca0b84ffd..b5e6d25e1be 100644 --- a/server/src/services/heartbeat/queue.ts +++ b/server/src/services/heartbeat/queue.ts @@ -1,3 +1,6 @@ +import { PersistedExecutionWaitRefusal, writeDeferredExecutionWait } from "../deferred-execution-wait.js"; +import { notifyDeferredExecutionWork } from "../deferred-execution-notifications.js"; +import type { DeferredReceiptRecoveryOptions } from "../deferred-execution-notifications.js"; import { notifyRunDispatch } from "../run-dispatch-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js"; import { notifyExternalAgentSettlement } from "../../modules/external-agents/index.js"; @@ -222,7 +225,7 @@ export interface HeartbeatQueueDependencies extends Pick void; cancelRunInternal: (runId: string, reason?: string, options?: { errorCode?: string }) => Promise; cancelActiveForAgentInternal: (agentId: string, reason: string) => Promise; - resumeExecutionWaitComments: () => Promise; + resumeExecutionWaitComments: (scope?: DeferredReceiptRecoveryOptions) => Promise; resumeQueuedCommentInterrupt: (companyId: string, queueId: string) => Promise; resumeSavedLegacyComments: (companyId: string, queueId: string) => Promise; formatIssueIdentifierLink: (identifier: string | null, fallback: string) => string; @@ -1851,13 +1854,15 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende return cancelled; } - async function resumeQueuedRuns() { + async function resumeDeferredExecutionReceipts(scope: DeferredReceiptRecoveryOptions = {}) { if ((await getSchedulingSuppression()).suppressed) return; - await resumeExecutionWaitComments(); + await resumeExecutionWaitComments(scope); const cutoff = await getWorktreeExecutionCutoff(); const pendingInterrupts = await db.select({ id: agentWakeupRequests.id, companyId: agentWakeupRequests.companyId }) .from(agentWakeupRequests).innerJoin(companies, eq(companies.id, agentWakeupRequests.companyId)) - .where(and(eq(agentWakeupRequests.status, "deferred_issue_execution"), + .where(and(scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined, + scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined, + eq(agentWakeupRequests.status, "deferred_issue_execution"), eq(companies.status, "active"), sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt' is not null`, lte(agentWakeupRequests.updatedAt, new Date(Date.now() - 30_000)), @@ -1868,6 +1873,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende eq(agentWakeupRequests.id, wake.id), eq(agentWakeupRequests.status, "deferred_issue_execution"), )); await resumeQueuedCommentInterrupt(wake.companyId, wake.id).catch(err => { + scope.onError?.(err); logger.warn({ err, queueId: wake.id }, "failed to resume interrupted comment queue"); }); } @@ -1881,7 +1887,9 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende or(eq(issues.assigneeAgentId, agentWakeupRequests.agentId), sql`${agentWakeupRequests.payload} #>> '{_paperclipWakeContext,wakeReason}' = 'issue_comment_mentioned'`))) .innerJoin(companies, and(eq(companies.id, issues.companyId), eq(companies.status, "active"))) - .where(and(eq(agentWakeupRequests.status, "deferred_issue_execution"), + .where(and(scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined, + scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined, + eq(agentWakeupRequests.status, "deferred_issue_execution"), isNull(issues.executionRunId), or(and( sql`jsonb_typeof(${agentWakeupRequests.payload} #> '{_paperclipWakeContext,wakeCommentIds}') = 'array'`, @@ -1920,7 +1928,8 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende if (latest.runtimeMode !== "legacy" && latest.runtimeMode !== "native") continue; if (wake.idempotencyKey?.startsWith("chat-inbound:")) { await resumeDeferredChatWake(db, { companyId: wake.companyId, agentId: wake.agentId, - issueId: String(wake.payload?.issueId), requestId: wake.id }).catch(() => { + issueId: String(wake.payload?.issueId), requestId: wake.id }).catch(err => { + scope.onError?.(err); logger.warn({ requestId: wake.id }, "saved chat source could not resume after recovery"); }); continue; @@ -1932,6 +1941,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende latest.errorCode === "execution_reconciliation_required"; if ((latest.status !== "cancelled" || cancelledAdmission) && await getExecutionBlocker(db, wake.companyId, String(wake.payload?.issueId))) { await resumeSavedLegacyComments(wake.companyId, wake.id).catch(err => { + scope.onError?.(err); logger.warn({ err, queueId: wake.id }, "failed to deliver saved legacy comment after recovery stopped"); }); } @@ -1939,6 +1949,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende await releaseIssueExecutionAndPromote(latest, { suppressImmediateRecovery: true, resumeStrandedQueue: true }).then(() => { if (latest.runtimeMode === "native") releasedNativeSources.add(releaseSource); }).catch(err => { + scope.onError?.(err); logger.warn({ err, queueId: wake.id }, "failed to promote stranded comments"); }); } @@ -1956,6 +1967,8 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende )) .innerJoin(companies, eq(companies.id, heartbeatRuns.companyId)) .where(and( + scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined, + scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined, eq(agentWakeupRequests.status, "deferred_issue_execution"), eq(heartbeatRuns.status, "cancelled"), eq(heartbeatRuns.runtimeMode, "legacy"), @@ -1964,10 +1977,15 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende )); for (const run of interruptedQueues) { await releaseIssueExecutionAndPromote(run, { suppressImmediateRecovery: true }).catch((err) => { + scope.onError?.(err); logger.error({ err, runId: run.id }, "failed to retry interrupted comment queue"); }); } + } + + async function resumeQueuedRuns() { + await resumeDeferredExecutionReceipts(); await resumeQueuedRunsAfterDrain(); } @@ -2501,23 +2519,17 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende skipReason: string, patch: Partial = {}, waitCondition?: Record, - ) => { + ): Promise<{ created: boolean; waitSaved?: boolean }> => { // A temporary refusal must leave the handoff retryable, without burning // its one durable receipt ID on a skipped wake. if (caseRequest) return { created: false }; if (executionWaitRequestId) { - await db.update(agentWakeupRequests).set({ - payload: sql`jsonb_set(coalesce(${agentWakeupRequests.payload}, '{}'::jsonb), '{executionWait}', - coalesce(${agentWakeupRequests.payload}->'executionWait', '{}'::jsonb) || ${JSON.stringify({ - reason: skipReason, message: patch.error ?? (skipReason === "issue_tree_hold_active" - ? "This task is paused. Resume it to send your saved message." - : "Waiting for task execution to be enabled. Your message is saved."), - })}::jsonb)`, - updatedAt: new Date(), - }).where(and(eq(agentWakeupRequests.id, executionWaitRequestId), - eq(agentWakeupRequests.companyId, agent.companyId), eq(agentWakeupRequests.agentId, agentId), - eq(agentWakeupRequests.status, "deferred_issue_execution"))); - return { created: false }; + const waitSaved = await writeDeferredExecutionWait(db, { companyId: agent.companyId, requestId: executionWaitRequestId }, { + reason: skipReason, message: patch.error ?? (skipReason === "issue_tree_hold_active" + ? "This task is paused. Resume it to send your saved message." + : "Waiting for task execution to be enabled. Your message is saved."), + }); + return { created: false, waitSaved }; } const request = { ...durableReceiptFields, @@ -2789,31 +2801,30 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende }, ); if (budgetBlock) { - await writeSkippedRequest("budget.blocked", { error: budgetBlock.reason }, { + const wait = await writeSkippedRequest("budget.blocked", { error: budgetBlock.reason }, { scopeType: budgetBlock.scopeType, scopeId: budgetBlock.scopeId, }); if (caseRequest) return null; - throw conflict(budgetBlock.reason, { - scopeType: budgetBlock.scopeType, - scopeId: budgetBlock.scopeId, - }); + const details = { scopeType: budgetBlock.scopeType, scopeId: budgetBlock.scopeId }; + throw wait.waitSaved ? new PersistedExecutionWaitRefusal(budgetBlock.reason, details) + : conflict(budgetBlock.reason, details); } // Setup delays execution, but must not discard work accepted by a caller. const invokability = await getAgentInvokability(isAgentAwaitingSetup(agent) ? { ...agent, status: "idle" } : agent); if (!invokability.invokable) { if (caseRequest) return null; + let waitSaved = false; if (opts.requestedByActorType !== "user" || executionWaitRequestId) { - await writeSkippedRequest("agent.not_invokable", { + const wait = await writeSkippedRequest("agent.not_invokable", { error: invokability.message, }, { status: agent.status, reason: invokability.reason }); + waitSaved = wait.waitSaved === true; } - throw conflict(invokability.message, { - status: agent.status, - reason: invokability.reason, - invalidOrgChain: invokability.invalidOrgChain, - ...invokability.details, - }); + const details = { status: agent.status, reason: invokability.reason, + invalidOrgChain: invokability.invalidOrgChain, ...invokability.details }; + throw waitSaved ? new PersistedExecutionWaitRefusal(invokability.message, details) + : conflict(invokability.message, details); } const policy = parseHeartbeatPolicy(agent); @@ -3404,15 +3415,14 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende opts.issueStateGuard?.onExecutionDeferred?.(executionBlocker.cause); const condition = { recoveryActionId: executionBlocker.recoveryActionId, ...continuationWait }; if (executionWaitRequestId) { - await tx.update(agentWakeupRequests).set({ - payload: sql`jsonb_set(coalesce(${agentWakeupRequests.payload}, '{}'::jsonb), '{executionWait}', ${JSON.stringify(condition)}::jsonb)`, - updatedAt: new Date(), - }).where(eq(agentWakeupRequests.id, executionWaitRequestId)); + await writeDeferredExecutionWait(tx as unknown as Db, { companyId: agent.companyId, + requestId: executionWaitRequestId }, condition, { replace: true }); return { kind: "deferred" as const, admitted: true }; } if (durableRequest || wakeCommentId || hasInteractionContinuationWakeContext(enrichedContextSnapshot) || readNonEmptyString(enrichedContextSnapshot.nativeStatusWakeIntentId)) { + await notifyDeferredExecutionWork(tx, { companyId: agent.companyId }); await tx.insert(agentWakeupRequests).values({ ...durableReceiptFields, companyId: agent.companyId, agentId, source, triggerDetail, reason, @@ -4030,6 +4040,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende })); } if (followUp) { + if (!followUp.consumed) await notifyDeferredExecutionWork(tx, { companyId: agent.companyId }); const assignmentId = followUp.assignmentId; const pending = followUp.consumed ? undefined : pendingFollowUps.find(wake => wake.payload?.[followUpPayloadKey] === assignmentId && @@ -4763,12 +4774,27 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende * wake was inserted but before the intent was linked to it. Dispatch failure * only requeues the intent; it never changes the provider run outcome. */ - async function dispatchPendingNativeStatusWakeups( - input: { - companyId?: string; - limit?: number; - staleClaimMs?: number; - } = {}, + type NativeStatusWakeDispatchInput = { + companyId?: string; + limit?: number; + staleClaimMs?: number; + requestId?: string; + onDeferredDeadline?: (at: number) => void; + onError?: (error: unknown) => void; + }; + type NativeStatusWakeDelivery = { runId: string | null; status: string }; + + function createNativeStatusWakeBatch() { + const deliveredByIssueScope = new Map(); + return (input: NativeStatusWakeDispatchInput = {}) => dispatchNativeStatusWakeBatch(input, deliveredByIssueScope); + } + + function dispatchPendingNativeStatusWakeups(input: NativeStatusWakeDispatchInput = {}) { + return createNativeStatusWakeBatch()(input); + } + + async function dispatchNativeStatusWakeBatch(input: NativeStatusWakeDispatchInput, + deliveredByIssueScope: Map, ) { const now = new Date(); const staleClaimMs = Math.max(1_000, input.staleClaimMs ?? 60_000); @@ -4780,6 +4806,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende input.companyId ? eq(agentWakeupRequests.companyId, input.companyId) : undefined, + input.requestId ? eq(agentWakeupRequests.id, input.requestId) : undefined, eq(agentWakeupRequests.requestedByActorType, "system"), eq(agentWakeupRequests.requestedByActorId, "native-status-committer"), inArray(agentWakeupRequests.status, ["queued", "claimed"]), @@ -4792,10 +4819,6 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende let dispatched = 0; let recovered = 0; let deferred = 0; - const deliveredByIssueScope = new Map< - string, - { runId: string | null; status: string } - >(); for (const candidate of candidates) { const payload = parseObject(candidate.payload); @@ -4856,6 +4879,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende candidate.claimedAt && now.getTime() - candidate.claimedAt.getTime() < staleClaimMs ) { + input.onDeferredDeadline?.(candidate.claimedAt.getTime() + staleClaimMs); deferred += 1; continue; } @@ -5030,6 +5054,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende if (wakeRun) dispatched += 1; else deferred += 1; } catch (error) { + input.onError?.(error); await db .update(agentWakeupRequests) .set({ @@ -5066,6 +5091,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende return { enqueueWakeup, dispatchPendingNativeStatusWakeups, + createNativeStatusWakeBatch, startNextQueuedRunForAgent, countRunningRunsForAgent, releaseRunClaimedJustBeforeSuppression, @@ -5073,6 +5099,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende withChatControlRecoveryGate, trackWakeup, resumeQueuedRuns, + resumeDeferredExecutionReceipts, resumeQueuedRunsAfterDrain, parseHeartbeatPolicy, nextTimerHeartbeatAt, diff --git a/server/src/services/heartbeat/recovery.ts b/server/src/services/heartbeat/recovery.ts index d4e4d65c599..7855e6e4f5b 100644 --- a/server/src/services/heartbeat/recovery.ts +++ b/server/src/services/heartbeat/recovery.ts @@ -2009,7 +2009,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD let settledProcessesDiscovered = false; - async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean; skipNativeResponsePresentation?: boolean; skipNativeReviewResponse?: boolean; skipNativeStatusReassessment?: boolean; skipRetainedNativeSessionCleanup?: boolean; skipNativeWorkspaceFinalization?: boolean; skipNativeResultCommit?: boolean; skipNativeRunProjection?: boolean; skipNativeRecoveryNotices?: boolean; skipExternalConversationSettlement?: boolean; skipExternalAgentSettlement?: boolean; skipNativeEnvironmentSettlement?: boolean }) { + async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipNativeStatusWakeIntents?: boolean; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean; skipNativeResponsePresentation?: boolean; skipNativeReviewResponse?: boolean; skipNativeStatusReassessment?: boolean; skipRetainedNativeSessionCleanup?: boolean; skipNativeWorkspaceFinalization?: boolean; skipNativeResultCommit?: boolean; skipNativeRunProjection?: boolean; skipNativeRecoveryNotices?: boolean; skipExternalConversationSettlement?: boolean; skipExternalAgentSettlement?: boolean; skipNativeEnvironmentSettlement?: boolean }) { if (!opts?.skipCancelledWorkspacePreservation) await preserveCancelledNativeWorkspaces(db); const staleThresholdMs = opts?.staleThresholdMs ?? 0; const now = new Date(); @@ -2053,7 +2053,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD }); if (!opts?.skipRetainedNativeSessionCleanup) scheduleRetainedNativeSessionCleanup(); await reconcileRetainedWorkspaces(db, environmentRuntime, resumeRemoteStopComments); - await dispatchPendingNativeStatusWakeups().catch((error) => { + if (!opts?.skipNativeStatusWakeIntents) await dispatchPendingNativeStatusWakeups().catch((error) => { logger.warn( { err: error }, "failed to dispatch persisted native status wake intents before orphan reaping", diff --git a/server/src/services/heartbeat/run-control.ts b/server/src/services/heartbeat/run-control.ts index 820f1e50c61..c5948f03a08 100644 --- a/server/src/services/heartbeat/run-control.ts +++ b/server/src/services/heartbeat/run-control.ts @@ -1,3 +1,6 @@ +import { instanceSettingsService } from "../instance-settings.js"; +import { writeDeferredExecutionWait } from "../deferred-execution-wait.js"; +import type { DeferredReceiptRecoveryOptions } from "../deferred-execution-notifications.js"; import { observeStoppedWorkspaceRecovery } from "../workspace-recovery.js"; import { notifyScheduledRetryCancellation } from "../scheduled-retry-cancellation-notifications.js"; import { notifyChatInboundWakeNotices } from "../chat-inbound-wake-notice-notifications.js"; @@ -616,12 +619,7 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont onBlocked: (reason, message) => { wait = { reason, message }; }, }); if (!admitted) { - await db.update(agentWakeupRequests).set({ - payload: sql`jsonb_set(coalesce(${agentWakeupRequests.payload}, '{}'::jsonb), '{executionWait}', - coalesce(${agentWakeupRequests.payload}->'executionWait', '{}'::jsonb) || ${JSON.stringify(wait)}::jsonb)`, - updatedAt: new Date(), - }).where(and(eq(agentWakeupRequests.id, wake.id), eq(agentWakeupRequests.companyId, run.companyId), - eq(agentWakeupRequests.status, "deferred_issue_execution"))); + await writeDeferredExecutionWait(db, { companyId: run.companyId, requestId: wake.id }, wait); continue; } } @@ -777,21 +775,31 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont idempotencyKey: `task-retry-dispatch:${wake.id}` }, wake.id); } - async function resumeExecutionWaitComments() { + async function resumeExecutionWaitComments(scope: DeferredReceiptRecoveryOptions = {}) { if ((await getSchedulingSuppression()).suppressed) return; - await reconcileTaskChatRetryReceipts(db); - const retries = await db.select({ id: agentWakeupRequests.id, companyId: agentWakeupRequests.companyId }) + await reconcileTaskChatRetryReceipts(db, scope); + const retries = await db.select({ id: agentWakeupRequests.id, companyId: agentWakeupRequests.companyId, + conversationAgentId: issues.conversationAgentId }) .from(agentWakeupRequests).innerJoin(issues, and(eq(issues.companyId, agentWakeupRequests.companyId), sql`${issues.id}::text = ${agentWakeupRequests.payload}->>'issueId'`, eq(issues.assigneeAgentId, agentWakeupRequests.agentId))) .innerJoin(companies, and(eq(companies.id, issues.companyId), eq(companies.status, "active"))) - .where(and(eq(agentWakeupRequests.reason, "task_retry"), eq(agentWakeupRequests.status, "deferred_issue_execution"), + .where(and(scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined, + scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined, + eq(agentWakeupRequests.reason, "task_retry"), eq(agentWakeupRequests.status, "deferred_issue_execution"), sql`${agentWakeupRequests.payload}->'taskRetry'->>'chatActionId' is null`, notInArray(issues.status, ["done", "cancelled"]))) .orderBy(asc(agentWakeupRequests.updatedAt)).limit(50); + const chatEnabled = !retries.some(retry => retry.conversationAgentId) + || (await instanceSettingsService(db).getExperimental()).enableAgentChat; for (const retry of retries) { + // Accepted chat work waits for the flag's committed eligibility signal. + // Public Retry still rejects new requests while the feature is disabled. await db.update(agentWakeupRequests).set({ updatedAt: new Date() }).where(eq(agentWakeupRequests.id, retry.id)); - await resumeTaskRetry(retry.companyId, retry.id).catch(err => logger.warn({ err, requestId: retry.id }, "Task retry dispatch deferred")); + if (retry.conversationAgentId && !chatEnabled) continue; + await resumeTaskRetry(retry.companyId, retry.id).catch(err => { + scope.onError?.(err); logger.warn({ err, requestId: retry.id }, "Task retry dispatch deferred"); + }); } const waits = await db.select({ wake: agentWakeupRequests }) .from(agentWakeupRequests) @@ -799,7 +807,9 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont sql`${issues.id}::text = ${agentWakeupRequests.payload}->>'issueId'`, eq(issues.assigneeAgentId, agentWakeupRequests.agentId))) .innerJoin(companies, and(eq(companies.id, issues.companyId), eq(companies.status, "active"))) - .where(and(exists(db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and( + .where(and(scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined, + scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined, + exists(db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and( eq(issueRecoveryActions.companyId, issues.companyId), eq(issueRecoveryActions.sourceIssueId, issues.id), executionBlockerPredicate(), ))), eq(agentWakeupRequests.status, "deferred_issue_execution"), @@ -831,11 +841,13 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont if (!run || run.companyId !== wake.companyId || run.agentId !== wake.agentId) continue; if (canContinueCancelledRun(run)) { await resumeSavedComments(wake.companyId, wake.id).catch(err => { + scope.onError?.(err); logger.warn({ err, runId: run.id }, "failed to resume saved input after provider cancellation"); }); continue; } await resumeRemoteStopComments(run, wake.id).catch(err => { + scope.onError?.(err); logger.warn({ err, runId: run.id }, "failed to resume saved execution-wait message"); }); } diff --git a/server/src/services/instance-settings.ts b/server/src/services/instance-settings.ts index 174c39d6125..9bedfa20df7 100644 --- a/server/src/services/instance-settings.ts +++ b/server/src/services/instance-settings.ts @@ -1,3 +1,4 @@ +import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js"; import { notifyRunDispatch } from "./run-dispatch-notifications.js"; import { notifySkillVersionRetentionReconstruction } from "./skill-version-retention-notifications.js"; import { notifyNativeEnvironmentSettlement } from "./native-environment-settlement-notifications.js"; @@ -624,14 +625,19 @@ export function instanceSettingsService(db: Db, options: InstanceSettingsService } } } + const chatBecameEnabled = !toExperimentalView(current.experimental).enableAgentChat + && toExperimentalView(nextExperimental).enableAgentChat; const now = new Date(); const [updated] = await db.transaction(async tx => { + if (chatBecameEnabled) await notifyDeferredExecutionEligibility(tx); await notifyDeliveryWork(tx, DELIVERY_QUEUES.mcpEvents, "*"); await notifyDeliveryWork(tx, DELIVERY_QUEUES.dotEvents, "*"); await notifyDeliveryWork(tx, DELIVERY_QUEUES.grokDispatch); + if (chatBecameEnabled) await notifyDeliveryWork(tx, DELIVERY_QUEUES.strandedAssignments, "*"); if (patch.enableWorktreeRunExecution !== undefined) await notifyScheduledRetryCancellation(tx); if (patch.enableWorktreeRunExecution !== undefined) { await notifyRunDispatch(tx); + await notifyDeferredExecutionEligibility(tx); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); await notifyExternalAgentSettlement(tx); await notifyInstructionCommit(tx); diff --git a/server/src/services/issue-comment-writes.ts b/server/src/services/issue-comment-writes.ts index e24e78e9855..39d8f6f891c 100644 --- a/server/src/services/issue-comment-writes.ts +++ b/server/src/services/issue-comment-writes.ts @@ -1,3 +1,4 @@ +import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import type { PgInsertValue } from "drizzle-orm/pg-core"; import { issueComments, type Db } from "@paperclipai/db"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; @@ -6,6 +7,8 @@ type WriteDb = Db | Parameters[0]>[0]; export function insertIssueComment(db: WriteDb, input: PgInsertValue & { companyId: string }) { return db.transaction(async tx => { + await notifyIssueRecovery(tx, { companyId: input.companyId, + issueId: typeof input.issueId === "string" ? input.issueId : undefined }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.dotEvents, `company:${input.companyId}`); return tx.insert(issueComments).values(input).returning(); }); diff --git a/server/src/services/issue-recovery-actions.ts b/server/src/services/issue-recovery-actions.ts index 48fb001c0b4..316eb413afe 100644 --- a/server/src/services/issue-recovery-actions.ts +++ b/server/src/services/issue-recovery-actions.ts @@ -1,3 +1,4 @@ +import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import { notifyNativeRecoveryNotices } from "./native-recovery-notice-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; import { writeExecutionControl } from "./execution-control-notifications.js"; @@ -322,6 +323,7 @@ export function issueRecoveryActionService(db: DbOrTransaction) { ): Promise { try { const created = await db.transaction(async (tx) => { + await notifyIssueRecovery(tx, { companyId: input.companyId }); await notifyNativeRecoveryNotices(tx, { companyId: input.companyId, issueId: input.sourceIssueId }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionDisposition); await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionContinuation); @@ -413,6 +415,7 @@ export function issueRecoveryActionService(db: DbOrTransaction) { Math.min(nextAttemptCount, effectiveMaxAttempts), ); const [exhausted] = await writeExecutionControl(db, [DELIVERY_QUEUES.executionDisposition, DELIVERY_QUEUES.executionContinuation], async tx => { + await notifyIssueRecovery(tx, { companyId: input.companyId }); await notifyNativeRecoveryNotices(tx, { companyId: input.companyId, issueId: input.sourceIssueId }); if ((!input.preserveExistingOwner && existing.cause !== input.cause) || (input.evidence?.runId !== undefined && input.evidence.runId !== existing.evidence?.runId)) { @@ -474,6 +477,7 @@ export function issueRecoveryActionService(db: DbOrTransaction) { return toReadModel(exhausted); } const [updated] = await writeExecutionControl(db, [DELIVERY_QUEUES.executionDisposition, DELIVERY_QUEUES.executionContinuation], async tx => { + await notifyIssueRecovery(tx, { companyId: input.companyId }); await notifyNativeRecoveryNotices(tx, { companyId: input.companyId, issueId: input.sourceIssueId }); if ((!input.preserveExistingOwner && existing.cause !== input.cause) || (input.evidence?.runId !== undefined && input.evidence.runId !== existing.evidence?.runId)) { @@ -548,6 +552,7 @@ export function issueRecoveryActionService(db: DbOrTransaction) { try { const [created] = await writeExecutionControl(db, [DELIVERY_QUEUES.executionDisposition, DELIVERY_QUEUES.executionContinuation], async tx => { + await notifyIssueRecovery(tx, { companyId: input.companyId }); await notifyNativeRecoveryNotices(tx, { companyId: input.companyId, issueId: input.sourceIssueId }); await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: input.sourceIssueId }); return tx @@ -595,6 +600,7 @@ export function issueRecoveryActionService(db: DbOrTransaction) { const pending = await tx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions) .where(and(...predicates)).limit(1); if (!pending.length) return []; + await notifyIssueRecovery(tx, { companyId: input.companyId }); await notifyNativeRecoveryNotices(tx, { companyId: input.companyId, issueId: input.sourceIssueId }); await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: input.sourceIssueId }); return tx diff --git a/server/src/services/issue-recovery-notifications.ts b/server/src/services/issue-recovery-notifications.ts new file mode 100644 index 00000000000..7b709960831 --- /dev/null +++ b/server/src/services/issue-recovery-notifications.ts @@ -0,0 +1,16 @@ +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; + +/** An issue hint is reserved for local facts such as comments. Lifecycle, + * graph, run and eligibility changes retain company scope because their + * consequences can include ancestors and shared execution references. */ +export type IssueRecoveryScope = { companyId: string; issueId?: string }; + +export async function notifyIssueRecovery(transaction: object, scope?: IssueRecoveryScope) { + const companyHint = scope ? JSON.stringify({ companyId: scope.companyId }) : "*"; + const issueHint = scope ? JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId }) : "*"; + for (const queue of [DELIVERY_QUEUES.orphanBlockers, DELIVERY_QUEUES.activeRecoveryActions, DELIVERY_QUEUES.strandedAssignments, DELIVERY_QUEUES.resolvedDependencies, + DELIVERY_QUEUES.taskWatchdogs, DELIVERY_QUEUES.staleIssueLocks]) { + await notifyDeliveryWork(transaction, queue, + queue === DELIVERY_QUEUES.strandedAssignments || queue === DELIVERY_QUEUES.staleIssueLocks ? issueHint : companyHint); + } +} diff --git a/server/src/services/issue-tree-control.ts b/server/src/services/issue-tree-control.ts index 0d7633f846a..2c05df61ceb 100644 --- a/server/src/services/issue-tree-control.ts +++ b/server/src/services/issue-tree-control.ts @@ -1,3 +1,5 @@ +import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js"; +import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; @@ -758,6 +760,8 @@ export function issueTreeControlService(db: Db) { const releaseReason = input.reason ?? "Subtree resume applied."; const { hold: resumeHold } = await db.transaction(async (tx) => { + await notifyIssueRecovery(tx, { companyId }); + await notifyDeferredExecutionEligibility(tx, { companyId }); const [createdHold] = await tx .insert(issueTreeHolds) .values({ @@ -836,6 +840,8 @@ export function issueTreeControlService(db: Db) { } const { hold, members } = await db.transaction(async (tx) => { + await notifyIssueRecovery(tx, { companyId }); + await notifyDeferredExecutionEligibility(tx, { companyId }); const [createdHold] = await tx .insert(issueTreeHolds) .values({ @@ -1000,6 +1006,7 @@ export function issueTreeControlService(db: Db) { const now = new Date(); const releasedCancelHoldIds = activeCancelHolds.map((hold) => hold.id); const updatedIssues = await db.transaction(async (tx) => { + await notifyDeferredExecutionEligibility(tx, { companyId }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); const restored: TreeStatusUpdateResult["updatedIssues"] = []; @@ -1177,7 +1184,10 @@ export function issueTreeControlService(db: Db) { throw conflict("Issue tree hold is already released"); } - const [updated] = await db + const [updated] = await db.transaction(async tx => { + await notifyIssueRecovery(tx, { companyId }); + await notifyDeferredExecutionEligibility(tx, { companyId }); + return tx .update(issueTreeHolds) .set({ status: "released", @@ -1195,6 +1205,7 @@ export function issueTreeControlService(db: Db) { }) .where(and(eq(issueTreeHolds.id, holdId), eq(issueTreeHolds.companyId, companyId))) .returning(); + }); const members = await db .select() diff --git a/server/src/services/issues.ts b/server/src/services/issues.ts index cc9293db472..34829069c0a 100644 --- a/server/src/services/issues.ts +++ b/server/src/services/issues.ts @@ -1,3 +1,5 @@ +import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js"; +import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import { notifyRunDispatch } from "./run-dispatch-notifications.js"; import { notifyExternalConversationSettlement } from "./external-conversation-settlement-notifications.js"; import { notifyNativeRecoveryNotices } from "./native-recovery-notice-notifications.js"; @@ -7628,6 +7630,8 @@ export function issueService(db: Db) { await assertNoBlockingCycles(companyId, issueId, deduped, dbOrTx); } + await notifyIssueRecovery(dbOrTx, { companyId }); + await notifyDeferredExecutionEligibility(dbOrTx, { companyId }); await dbOrTx .delete(issueRelations) .where( @@ -9676,27 +9680,30 @@ export function issueService(db: Db) { }); if (blockParentUntilDone) { - const existingBlockers = await db - .select({ blockerIssueId: issueRelations.issueId }) - .from(issueRelations) - .where( - and( - eq(issueRelations.companyId, parent.companyId), - eq(issueRelations.relatedIssueId, parent.id), - eq(issueRelations.type, "blocks"), - ), + await db.transaction(async tx => { + const existingBlockers = await tx + .select({ blockerIssueId: issueRelations.issueId }) + .from(issueRelations) + .where( + and( + eq(issueRelations.companyId, parent.companyId), + eq(issueRelations.relatedIssueId, parent.id), + eq(issueRelations.type, "blocks"), + ), + ); + await syncBlockedByIssueIds( + parent.id, + parent.companyId, + [ + ...new Set([ + ...existingBlockers.map((row) => row.blockerIssueId), + child.id, + ]), + ], + { agentId: actorAgentId ?? null, userId: actorUserId ?? null }, + tx, ); - await syncBlockedByIssueIds( - parent.id, - parent.companyId, - [ - ...new Set([ - ...existingBlockers.map((row) => row.blockerIssueId), - child.id, - ]), - ], - { agentId: actorAgentId ?? null, userId: actorUserId ?? null }, - ); + }); [child] = await withIssueRelationSummaries( parent.companyId, [child], @@ -10608,6 +10615,7 @@ export function issueService(db: Db) { } if (values.monitorNextCheckAt) await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); await notifyRunDispatch(tx, { companyId: values.companyId }); + await notifyDeferredExecutionEligibility(tx, { companyId: values.companyId }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); const [issue] = await tx.insert(issues).values(values).returning(); await attachOwnedDraftImages(tx, issue, issue.description, { userId: issueData.createdByUserId, agentId: issueData.createdByAgentId }); @@ -11495,8 +11503,11 @@ export function issueService(db: Db) { await notifyTerminalWorkspaceWork(tx, `issue:${id}`); } if (patch.status !== undefined || patch.assigneeAgentId !== undefined || patch.assigneeUserId !== undefined - || patch.hiddenAt !== undefined || patch.conversationAgentId !== undefined) { + || patch.hiddenAt !== undefined || patch.conversationAgentId !== undefined || patch.parentId !== undefined + || patch.executionState !== undefined || patch.executionRunId !== undefined || patch.checkoutRunId !== undefined + || blockedByIssueIds !== undefined) { await notifyRunDispatch(tx, { companyId: receiptExisting.companyId }); + await notifyDeferredExecutionEligibility(tx, { companyId: receiptExisting.companyId }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); } if (patch.monitorNextCheckAt !== undefined || (receiptExisting.monitorNextCheckAt && @@ -11962,6 +11973,7 @@ export function issueService(db: Db) { await notifyDecisionTargetIssueRemoval(tx, id); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); await notifyRunDispatch(tx, owner ? { companyId: owner.companyId } : undefined); + if (owner) await notifyDeferredExecutionEligibility(tx, owner); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); removedIssue = await tx .delete(issues) @@ -12091,6 +12103,7 @@ export function issueService(db: Db) { const updated = await db.transaction(async tx => { await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); await notifyRunDispatch(tx, { companyId: issueCompany.companyId }); + await notifyDeferredExecutionEligibility(tx, { companyId: issueCompany.companyId }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); return tx .update(issues) @@ -12221,6 +12234,7 @@ export function issueService(db: Db) { const adopted = await db.transaction(async tx => { await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); await notifyRunDispatch(tx, { companyId: issueCompany.companyId }); + await notifyDeferredExecutionEligibility(tx, { companyId: issueCompany.companyId }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); return tx .update(issues) @@ -12477,6 +12491,7 @@ export function issueService(db: Db) { await notifyNativeStatusReassessment(tx, { companyId: existing.companyId, issueId: id }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); await notifyRunDispatch(tx, { companyId: existing.companyId }); + await notifyDeferredExecutionEligibility(tx, { companyId: existing.companyId }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); const updated = await tx .update(issues) @@ -12538,6 +12553,7 @@ export function issueService(db: Db) { } await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); await notifyRunDispatch(tx, { companyId: existing.companyId }); + await notifyDeferredExecutionEligibility(tx, { companyId: existing.companyId }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); const updated = await tx .update(issues) @@ -13816,6 +13832,7 @@ export function issueService(db: Db) { .then((rows) => rows[0] ?? null); if (!existing) return null; + await notifyIssueRecovery(tx, { companyId: existing.companyId }); await tx.delete(issueAttachments).where(eq(issueAttachments.id, id)); await tx.delete(assets).where(eq(assets.id, existing.assetId)); return existing; diff --git a/server/src/services/legacy-controller-lease.ts b/server/src/services/legacy-controller-lease.ts index 55a8439debc..22fa078f459 100644 --- a/server/src/services/legacy-controller-lease.ts +++ b/server/src/services/legacy-controller-lease.ts @@ -77,14 +77,18 @@ async function renewOnLivenessPool( }); } -export async function hasLiveLegacyController(db: Db, run: Run): Promise { - if (run.runtimeMode === "native" || !run.controllerBootId) return false; - const [owner] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( +export async function liveLegacyControllerExpiresAt(db: Db, run: Run): Promise { + if (run.runtimeMode === "native" || !run.controllerBootId) return null; + const [owner] = await db.select({ expiresAt: heartbeatRuns.controllerLeaseExpiresAt }).from(heartbeatRuns).where(and( eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.status, "running"), gt(heartbeatRuns.controllerLeaseExpiresAt, sql`clock_timestamp()`), )); - return Boolean(owner); + return owner?.expiresAt ?? null; +} + +export async function hasLiveLegacyController(db: Db, run: Run): Promise { + return (await liveLegacyControllerExpiresAt(db, run)) !== null; } export type RevokedLegacyControllerState = Pick; diff --git a/server/src/services/native-runtime/chat-attachment-reuse.test.ts b/server/src/services/native-runtime/chat-attachment-reuse.test.ts index ccd4bbe30d6..d0de5ab04c9 100644 --- a/server/src/services/native-runtime/chat-attachment-reuse.test.ts +++ b/server/src/services/native-runtime/chat-attachment-reuse.test.ts @@ -73,7 +73,7 @@ describe("native same-conversation chat attachment reuse", () => { beforeAll(async () => { const external = resolveExternalTestDatabaseUrl("PAPERCLIP_CHAT_REUSE_TEST_DATABASE_URL"); if (!external) temporary = await startEmbeddedPostgresTestDatabase("native-chat-reuse-"); - db = createDb(external ?? temporary!.connectionString); + db = createDb(external ?? temporary!.connectionString, { maxConnections: 6 }); const storageRoot = await mkdtemp( path.join(tmpdir(), "paperclip-chat-reuse-"), ); diff --git a/server/src/services/native-runtime/chat-attachment-reuse.ts b/server/src/services/native-runtime/chat-attachment-reuse.ts index cc7990e9b7a..6cc44299ee7 100644 --- a/server/src/services/native-runtime/chat-attachment-reuse.ts +++ b/server/src/services/native-runtime/chat-attachment-reuse.ts @@ -1,3 +1,4 @@ +import { notifyIssueRecovery } from "../issue-recovery-notifications.js"; import { createHash } from "node:crypto"; import { and, desc, eq, inArray, isNull, sql } from "drizzle-orm"; @@ -1582,7 +1583,9 @@ export async function prepareReusedChatAttachment(input: { if (!workProduct) { throw new Error("paperclip_runner_chat_attachment_origin_not_persisted"); } - await input.db + await input.db.transaction(async tx => { + await notifyIssueRecovery(tx, { companyId: input.binding.companyId }); + await tx .update(issueWorkProducts) .set({ title: input.title, @@ -1602,6 +1605,7 @@ export async function prepareReusedChatAttachment(input: { eq(issueWorkProducts.createdByRunId, input.binding.runId), ), ); + }); const comment = await issueService(input.db).addComment( input.binding.issueId, `Prepared ${input.title} for this response.`, diff --git a/server/src/services/native-runtime/native-runner-file-handoff.test.ts b/server/src/services/native-runtime/native-runner-file-handoff.test.ts index 0abae30539f..e302c432190 100644 --- a/server/src/services/native-runtime/native-runner-file-handoff.test.ts +++ b/server/src/services/native-runtime/native-runner-file-handoff.test.ts @@ -1,3 +1,4 @@ +import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js"; import type { PrpStructuredRunResult } from "../../vendor/paperclip-runner/index.js"; import { nativeCompletionFeedback } from "./native-completion-feedback.js"; import { createHash, randomUUID } from "node:crypto"; @@ -61,10 +62,9 @@ describe("native runner file handoff", () => { const runId = "00000000-0000-4000-8000-000000009104"; beforeAll(async () => { - temporary = await startEmbeddedPostgresTestDatabase( - "native-runner-file-handoff-", - ); - db = createDb(temporary.connectionString); + const external = resolveExternalTestDatabaseUrl("PAPERCLIP_NATIVE_HANDOFF_TEST_DATABASE_URL"); + if (!external) temporary = await startEmbeddedPostgresTestDatabase("native-runner-file-handoff-"); + db = createDb(external ?? temporary!.connectionString, { maxConnections: 6 }); temporaryRoot = await mkdtemp( path.join(tmpdir(), "paperclip-native-file-handoff-"), ); diff --git a/server/src/services/native-runtime/native-runner-file-handoff.ts b/server/src/services/native-runtime/native-runner-file-handoff.ts index 6f2c32beec8..420380500ea 100644 --- a/server/src/services/native-runtime/native-runner-file-handoff.ts +++ b/server/src/services/native-runtime/native-runner-file-handoff.ts @@ -1,3 +1,4 @@ +import { notifyIssueRecovery } from "../issue-recovery-notifications.js"; import { createHash, randomUUID } from "node:crypto"; import { execFile } from "node:child_process"; import { constants } from "node:fs"; @@ -1265,7 +1266,9 @@ export async function prepareNativeRunnerFileHandoff(input: { ) { throw new Error("paperclip_runner_file_handoff_origin_not_persisted"); } - await input.db + await input.db.transaction(async tx => { + await notifyIssueRecovery(tx, { companyId: input.binding.companyId }); + await tx .update(issueWorkProducts) .set({ title: verified.title, updatedAt: new Date() }) .where( @@ -1276,6 +1279,7 @@ export async function prepareNativeRunnerFileHandoff(input: { eq(issueWorkProducts.createdByRunId, input.binding.runId), ), ); + }); const comment = await issueService(input.db).addComment( input.binding.issueId, `Prepared ${verified.title} for this response.`, diff --git a/server/src/services/native-runtime/native-workspace-finalizer.test.ts b/server/src/services/native-runtime/native-workspace-finalizer.test.ts index 59270a2bb65..53063f345e9 100644 --- a/server/src/services/native-runtime/native-workspace-finalizer.test.ts +++ b/server/src/services/native-runtime/native-workspace-finalizer.test.ts @@ -131,7 +131,7 @@ describe("native workspace finalization recovery", () => { const external = resolveExternalTestDatabaseUrl("PAPERCLIP_NATIVE_WORKSPACE_FINALIZER_TEST_DATABASE_URL"); if (!external) temporary = await startEmbeddedPostgresTestDatabase("paperclip-native-workspace-finalizer-"); connectionString = external ?? temporary!.connectionString; - db = createDb(connectionString); + db = createDb(connectionString, { maxConnections: 6 }); await db.insert(companies).values([ { id: companyId, name: "Native workspace owner", issuePrefix: "NWO" }, @@ -452,7 +452,7 @@ describe("native workspace finalization recovery", () => { const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Live copyback" }); const live = await withNativeWorkspaceFinalizationOwnership({ db, companyId, runId: seeded.runId }, async (ownership) => { await ownership.assertHeld(); - expect(await resumeNativeWorkspaceFinalization({ db: createDb(connectionString), runId: seeded.runId })).toBeNull(); + expect(await resumeNativeWorkspaceFinalization({ db: createDb(connectionString, { maxConnections: 6 }), runId: seeded.runId })).toBeNull(); expect(await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, seeded.runId))).toEqual([]); return "copied"; }); @@ -476,7 +476,7 @@ describe("native workspace finalization recovery", () => { try { await entered; await db.execute(sql`select pg_terminate_backend(pid) from pg_locks where locktype = 'advisory' and database = (select oid from pg_database where datname = current_database()) and pid <> pg_backend_pid()`); - const second = await withNativeWorkspaceFinalizationOwnership({ db: createDb(connectionString), companyId, runId: seeded.runId }, async () => "must not run"); + const second = await withNativeWorkspaceFinalizationOwnership({ db: createDb(connectionString, { maxConnections: 6 }), companyId, runId: seeded.runId }, async () => "must not run"); expect(second).toEqual({ acquired: false, reason: "busy" }); const [held] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)); expect(held.runnerProfileJson?.nativeWorkspaceFinalizationOwner).toBeTruthy(); @@ -489,7 +489,7 @@ describe("native workspace finalization recovery", () => { it("recovers its own joined copyback receipt after the application pool disconnects during cleanup", async () => { const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Reconnect joined copyback" }); - const interruptedDb = createDb(connectionString); + const interruptedDb = createDb(connectionString, { maxConnections: 6 }); let joined = false; await expect(withNativeWorkspaceFinalizationOwnership({ db: interruptedDb, companyId, runId: seeded.runId }, async () => { await fs.writeFile(path.join(workspaceRoot, "joined-copyback.txt"), "durable work"); @@ -507,7 +507,7 @@ describe("native workspace finalization recovery", () => { .toEqual({ acquired: false, reason: "owner_unverified" }); } await db.update(heartbeatRuns).set({ runnerProfileJson: beforeReconnect.runnerProfileJson }).where(eq(heartbeatRuns.id, seeded.runId)); - expect(await withNativeWorkspaceFinalizationOwnership({ db: createDb(connectionString), companyId, runId: seeded.runId }, async () => "resumed")) + expect(await withNativeWorkspaceFinalizationOwnership({ db: createDb(connectionString, { maxConnections: 6 }), companyId, runId: seeded.runId }, async () => "resumed")) .toEqual({ acquired: true, value: "resumed" }); }); diff --git a/server/src/services/native-runtime/status-decision-committer.ts b/server/src/services/native-runtime/status-decision-committer.ts index a43868dd25f..3ea3096b710 100644 --- a/server/src/services/native-runtime/status-decision-committer.ts +++ b/server/src/services/native-runtime/status-decision-committer.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusWakeWork } from "../deferred-execution-notifications.js"; import { notifyExternalAgentSettlement } from "../../modules/external-agents/index.js"; import { lockCurrentNativeWorkAssessmentEvidence } from "./work-assessments.js"; import { hasNativeWorkspaceFinalizationOwner } from "./native-workspace-finalization-recovery-state.js"; @@ -478,6 +479,7 @@ async function enqueueWake(input: { .limit(1) .then((rows) => rows[0] ?? null); if (existing) return existing.id; + await notifyNativeStatusWakeWork(input.tx, { companyId: input.companyId }); const inserted = await input.tx .insert(agentWakeupRequests) .values({ diff --git a/server/src/services/native-status-reassessment-notifications.ts b/server/src/services/native-status-reassessment-notifications.ts index fb845523bb4..d3b543d9914 100644 --- a/server/src/services/native-status-reassessment-notifications.ts +++ b/server/src/services/native-status-reassessment-notifications.ts @@ -1,8 +1,10 @@ +import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; export type NativeStatusReassessmentScope = { companyId: string; issueId?: string; runId?: string; agentId?: string }; -export function notifyNativeStatusReassessment(tx: object, scope: NativeStatusReassessmentScope) { +export async function notifyNativeStatusReassessment(tx: object, scope: NativeStatusReassessmentScope) { + await notifyIssueRecovery(tx, scope ? { companyId: scope.companyId } : undefined); return notifyDeliveryWork(tx, DELIVERY_QUEUES.nativeStatusReassessment, JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId, runId: scope.runId, agentId: scope.agentId })); } diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index e3596791942..07e8605bd73 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -27,7 +27,7 @@ import { decideLegacyContinuation, legacyDispositionEpisode, legacyDispositionFingerprint, LEGACY_DISPOSITION_REPAIR_INSTRUCTION, type LegacyDispositionEpisode, } from "./legacy-continuation.js"; -import { CONTROLLER_LEASE_LOST_FAILURE_CODE, hasLiveLegacyController } from "../legacy-controller-lease.js"; +import { CONTROLLER_LEASE_LOST_FAILURE_CODE, hasLiveLegacyController, liveLegacyControllerExpiresAt } from "../legacy-controller-lease.js"; import { isCompletedOnboardingHandoffWake } from "../chat-completion-delivery.js"; import { instanceSettingsService } from "../instance-settings.js"; import { isWaitingConversation, settleConversationTurnForRun, deliverConversationComments } from "../agent-conversations.js"; @@ -245,6 +245,7 @@ type ResolvedDependencyWakeBackstopOptions = { runId?: string | null; companyId?: string | null; blockerIssueId?: string | null; + issueId?: string; source?: ResolvedDependencyWakeBackstopSource; }; @@ -2202,7 +2203,7 @@ export function recoveryService( return Boolean(budgetBlock); } - async function reconcileUnassignedBlockingIssues() { + async function reconcileUnassignedBlockingIssues(opts?: { companyId?: string }) { const candidates = await db .select({ id: issues.id, @@ -2216,6 +2217,8 @@ export function recoveryService( .where( and( eq(issueRelations.type, "blocks"), + eq(issueRelations.companyId, issues.companyId), + opts?.companyId ? eq(issues.companyId, opts.companyId) : undefined, inArray(issues.status, ["todo", "blocked"]), isNull(issues.assigneeAgentId), isNull(issues.assigneeUserId), @@ -3500,7 +3503,7 @@ export function recoveryService( return Boolean(run || wake); } - async function reconcileActiveRecoveryActions() { + async function reconcileActiveRecoveryActions(opts?: { companyId?: string }) { const rows = await db .select({ action: issueRecoveryActions, @@ -3514,7 +3517,8 @@ export function recoveryService( eq(issues.companyId, issueRecoveryActions.companyId), ), ) - .where(inArray(issueRecoveryActions.status, ["active", "escalated"])); + .where(and(inArray(issueRecoveryActions.status, ["active", "escalated"]), + opts?.companyId ? eq(issueRecoveryActions.companyId, opts.companyId) : undefined)); const result = { requeued: 0, @@ -4526,14 +4530,20 @@ export function recoveryService( } async function reconcileStrandedAssignedIssues(opts?: { + companyId?: string; + issueId?: string; + onDeferredDeadline?: (at: number) => void; issueCreatedAtGte?: Date | null; skipExternalConversationSettlement?: boolean; + skipRelatedRecovery?: boolean; }) { const candidates = await db .select(recoveryIssueFields) .from(issues) .where( and( + opts?.companyId ? eq(issues.companyId, opts.companyId) : undefined, + opts?.issueId ? eq(issues.id, opts.issueId) : undefined, isNull(issues.assigneeUserId), inArray(issues.status, ["todo", "in_progress", "in_review"]), or( @@ -5714,6 +5724,7 @@ export function recoveryService( classification.baseBackoffMs * Math.pow(2, Math.max(0, consecutive - 1)); if (elapsed < requiredDelay) { + opts?.onDeferredDeadline?.(latestFinishedAt.getTime() + requiredDelay); result.skipped += 1; continue; } @@ -5773,16 +5784,18 @@ export function recoveryService( } } - const orphanBlockerRecovery = await reconcileUnassignedBlockingIssues(); - result.orphanBlockersAssigned = orphanBlockerRecovery.assigned; - result.skipped += orphanBlockerRecovery.skipped; - result.issueIds.push(...orphanBlockerRecovery.issueIds); + if (!opts?.skipRelatedRecovery) { + const orphanBlockerRecovery = await reconcileUnassignedBlockingIssues(opts); + result.orphanBlockersAssigned = orphanBlockerRecovery.assigned; + result.skipped += orphanBlockerRecovery.skipped; + result.issueIds.push(...orphanBlockerRecovery.issueIds); - const activeRecovery = await reconcileActiveRecoveryActions(); - result.continuationRequeued += activeRecovery.requeued; - result.escalated += activeRecovery.escalated; - result.skipped += activeRecovery.skipped; - result.issueIds.push(...activeRecovery.issueIds); + const activeRecovery = await reconcileActiveRecoveryActions(opts); + result.continuationRequeued += activeRecovery.requeued; + result.escalated += activeRecovery.escalated; + result.skipped += activeRecovery.skipped; + result.issueIds.push(...activeRecovery.issueIds); + } result.issueIds = [...new Set(result.issueIds)]; return result; @@ -5815,7 +5828,7 @@ export function recoveryService( source === "workspace.finalize" ? "workspace_finalize_reconciliation" : "issue_graph_liveness_reconciliation"; - const useCursor = !opts?.blockerIssueId; + const useCursor = !opts?.blockerIssueId && !opts?.issueId; const queryCandidates = (afterIssueId: string | null) => { const filters = [ @@ -5825,6 +5838,7 @@ export function recoveryService( sql`${issues.assigneeAgentId} is not null`, ]; if (opts?.companyId) filters.push(eq(issues.companyId, opts.companyId)); + if (opts?.issueId) filters.push(eq(issues.id, opts.issueId)); if (afterIssueId) filters.push(gt(issues.id, afterIssueId)); if (opts?.blockerIssueId) { @@ -6115,6 +6129,7 @@ export function recoveryService( // fallback below. Without it, a terminal issue named in the run context // snapshot would still terminalize the shared run and defeat the guard. runReferencedByActiveIssue?: boolean; + onDeferredDeadline?: (at: number) => void; }, ): Promise<{ terminalized: boolean; status: string }> { // Act only on a run in "running" status. A "queued" run has no process yet, @@ -6130,8 +6145,11 @@ export function recoveryService( // Another controller may own a sandbox run whose PID has no meaning on // this host. Its live lease owns both execution and finalization, even if // the issue is already terminal or this process has no in-memory handle. - if (await hasLiveLegacyController(db, run)) + const controllerDeadline = await liveLegacyControllerExpiresAt(db, run); + if (controllerDeadline) { + options?.onDeferredDeadline?.(controllerDeadline.getTime()); return { terminalized: false, status: run.status }; + } const pid = run.processPid ?? null; const processGroupId = run.processGroupId ?? null; @@ -6352,7 +6370,7 @@ export function recoveryService( // still claims to be live but can no longer reach a terminal status on its // own, so a stuck "running" run can no longer block the sweep. Idempotent and // safe: clears at most one row's worth of lock columns per candidate. - async function sweepStaleIssueLocks() { + async function sweepStaleIssueLocks(opts?: { companyId?: string; issueId?: string; onDeferredDeadline?: (at: number) => void }) { const result = { cleared: 0, issueIds: [] as string[], @@ -6369,7 +6387,9 @@ export function recoveryService( }) .from(issues) .where( - sql`(${issues.checkoutRunId} is not null or ${issues.executionRunId} is not null)`, + and(sql`(${issues.checkoutRunId} is not null or ${issues.executionRunId} is not null)`, + opts?.companyId ? eq(issues.companyId, opts.companyId) : undefined, + opts?.issueId ? eq(issues.id, opts.issueId) : undefined), ); const referencedRunIds = [ @@ -6384,7 +6404,8 @@ export function recoveryService( ? await db .select() .from(heartbeatRuns) - .where(inArray(heartbeatRuns.id, referencedRunIds)) + .where(and(inArray(heartbeatRuns.id, referencedRunIds), + opts?.companyId ? eq(heartbeatRuns.companyId, opts.companyId) : undefined)) : []; const runStatusById = new Map(); for (const row of runRows) runStatusById.set(row.id, row.status); @@ -6395,8 +6416,15 @@ export function recoveryService( // the same run id in a stale lock column. The terminal reference alone must // not terminalize a run that an active issue still owns, so exclude these // runs from the issue-terminal authority below. + const ownershipReferences = opts?.issueId && referencedRunIds.length > 0 + ? await db.select({ companyId: issues.companyId, status: issues.status, + checkoutRunId: issues.checkoutRunId, executionRunId: issues.executionRunId }).from(issues).where(and( + opts.companyId ? eq(issues.companyId, opts.companyId) : undefined, + or(inArray(issues.checkoutRunId, referencedRunIds), inArray(issues.executionRunId, referencedRunIds)), + )) + : candidates; const runIdsReferencedByActiveIssue = new Set(); - for (const issue of candidates) { + for (const issue of ownershipReferences) { if (issue.status === "done" || issue.status === "cancelled") continue; for (const runId of [issue.checkoutRunId, issue.executionRunId]) { if (runId) runIdsReferencedByActiveIssue.add(runId); @@ -6437,6 +6465,7 @@ export function recoveryService( referencingIssueTerminalStatus: issueTerminalStatusByRunId.get(row.id) ?? null, runReferencedByActiveIssue: runIdsReferencedByActiveIssue.has(row.id), + onDeferredDeadline: opts?.onDeferredDeadline, }); runStatusById.set(row.id, outcome.status); if (outcome.terminalized) result.terminalizedRunIds.push(row.id); @@ -6540,6 +6569,8 @@ export function recoveryService( recordWatchdogDecision, scanSilentActiveRuns, reconcileStrandedAssignedIssues, + reconcileUnassignedBlockingIssues, + reconcileActiveRecoveryActions, reconcileLegacyContinuation, legacyRepairDispatchBlock, sweepStaleIssueLocks, diff --git a/server/src/services/recovery/work.ts b/server/src/services/recovery/work.ts new file mode 100644 index 00000000000..407e4f1eb6d --- /dev/null +++ b/server/src/services/recovery/work.ts @@ -0,0 +1,113 @@ +import { registerTaskWatchdogWork } from "../task-watchdog-work.js"; +import { and, asc, eq, gt, inArray, isNotNull, isNull, or, sql } from "drizzle-orm"; +import { issueRecoveryActions, issueRelations, issues, type Db } from "@paperclipai/db"; +import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; +import { DELIVERY_QUEUES, type DeliveryQueue } from "../delivery-work-notifications.js"; +import type { IssueRecoveryScope } from "../issue-recovery-notifications.js"; +import { registerScopedDeadlineWork } from "../scoped-deadline-work.js"; + +type Entry = { id: string; companyId: string; dueAt: number }; +type Scope = { companyId: string; issueId: string; onDeferredDeadline: (at: number) => void }; +const PAGE_SIZE = 100; + +export function registerIssueRecoveryWork( + db: Db, + coordinator: ReturnType, + owner: { + stranded(scope: Scope): Promise; + orphanBlockers(scope: { companyId: string }): Promise; + recoveryActions(scope: { companyId: string }): Promise; + dependencies(scope: Scope): Promise<{ enqueueFailed: number }>; + locks(scope: Scope): Promise; + watchdogs(scope: { companyId: string; watchdogId: string; onDeferredDeadline: (at: number) => void }): Promise<{ incomplete: number }>; + schedulingSuppressed(): Promise; + }, + canRun: () => boolean, +) { + function register(queue: DeliveryQueue, statuses: string[] | null, check: (scope: Scope) => Promise) { + const cursors = new Map(); + let enabled = false; + return registerScopedDeadlineWork(coordinator, { + queue, + reconstructionIntervalMs: 30_000, + beforePass: async () => { enabled = !(await owner.schedulingSuppressed()); }, + decodeScope: hint => hint === "*" ? undefined : JSON.parse(hint) as IssueRecoveryScope, + matches: (entry, scope) => !scope || (entry.companyId === scope.companyId && (!scope.issueId || entry.id === scope.issueId)), + load: async scope => { + const key = scope ? JSON.stringify([scope.companyId, scope.issueId]) : "*"; + if (!enabled) { cursors.delete(key); return []; } + const cursor = cursors.get(key); + const rows = await db.select({ id: issues.id, companyId: issues.companyId }).from(issues).where(and( + scope ? eq(issues.companyId, scope.companyId) : undefined, + scope?.issueId ? eq(issues.id, scope.issueId) : undefined, + cursor ? gt(issues.id, cursor) : undefined, + statuses ? inArray(issues.status, statuses) : or(isNotNull(issues.checkoutRunId), isNotNull(issues.executionRunId)), + statuses ? isNull(issues.hiddenAt) : undefined, + statuses ? isNull(issues.assigneeUserId) : undefined, + statuses ? or(isNotNull(issues.assigneeAgentId), eq(issues.status, "in_review")) : undefined, + )).orderBy(asc(issues.id)).limit(PAGE_SIZE); + const complete = rows.length < PAGE_SIZE; + if (complete) cursors.delete(key); else cursors.set(key, rows.at(-1)!.id); + return { rows: rows.map(row => ({ ...row, dueAt: Date.now() })), complete }; + }, + check: async entry => { + if (!enabled) return null; + let next: number | null = null; + await check({ companyId: entry.companyId, issueId: entry.id, + onDeferredDeadline: at => { next = next === null ? at : Math.min(next, at); } }); + return next === null ? null : { ...entry, dueAt: next }; + }, + }, canRun); + } + function registerCompany(queue: DeliveryQueue, kind: "blockers" | "actions", check: (scope: { companyId: string }) => Promise) { + const cursors = new Map(); + let enabled = false; + return registerScopedDeadlineWork(coordinator, { + queue, + reconstructionIntervalMs: 30_000, + beforePass: async () => { enabled = !(await owner.schedulingSuppressed()); }, + decodeScope: hint => hint === "*" ? undefined : JSON.parse(hint) as IssueRecoveryScope, + matches: (entry, scope) => !scope || entry.companyId === scope.companyId, + load: async scope => { + const key = scope?.companyId ?? "*"; + if (!enabled) { cursors.delete(key); return []; } + const cursor = cursors.get(key); + const rows = kind === "actions" + ? await db.selectDistinct({ id: issueRecoveryActions.companyId, companyId: issueRecoveryActions.companyId }) + .from(issueRecoveryActions).where(and(inArray(issueRecoveryActions.status, ["active", "escalated"]), + scope ? eq(issueRecoveryActions.companyId, scope.companyId) : undefined, + cursor ? gt(issueRecoveryActions.companyId, cursor) : undefined)) + .orderBy(asc(issueRecoveryActions.companyId)).limit(PAGE_SIZE) + : await db.selectDistinct({ id: issues.companyId, companyId: issues.companyId }).from(issues).where(and( + inArray(issues.status, ["todo", "blocked"]), isNull(issues.assigneeAgentId), isNull(issues.assigneeUserId), + isNotNull(issues.createdByAgentId), + scope ? eq(issues.companyId, scope.companyId) : undefined, + cursor ? gt(issues.companyId, cursor) : undefined, + sql`exists (select 1 from ${issueRelations} r inner join ${issues} dependent + on dependent.id = r.related_issue_id and dependent.company_id = r.company_id + where r.company_id = ${issues.companyId} and r.issue_id = ${issues.id} + and r.type = 'blocks' and dependent.status not in ('done', 'cancelled'))`, + )).orderBy(asc(issues.companyId)).limit(PAGE_SIZE); + const complete = rows.length < PAGE_SIZE; + if (complete) cursors.delete(key); else cursors.set(key, rows.at(-1)!.id); + return { rows: rows.map(row => ({ ...row, dueAt: Date.now() })), complete }; + }, + check: async entry => { if (enabled) await check({ companyId: entry.companyId }); return null; }, + }, canRun); + } + const assignments = register(DELIVERY_QUEUES.strandedAssignments, ["todo", "in_progress", "in_review"], owner.stranded); + const orphanBlockers = registerCompany(DELIVERY_QUEUES.orphanBlockers, "blockers", owner.orphanBlockers); + const recoveryActions = registerCompany(DELIVERY_QUEUES.activeRecoveryActions, "actions", owner.recoveryActions); + const dependencies = register(DELIVERY_QUEUES.resolvedDependencies, ["blocked"], async scope => { + const result = await owner.dependencies(scope); + if (result.enqueueFailed > 0) throw new Error("Resolved dependency wake remains undelivered"); + }); + const locks = register(DELIVERY_QUEUES.staleIssueLocks, null, owner.locks); + const watchdogs = registerTaskWatchdogWork(db, coordinator, { + reconcile: owner.watchdogs, enabled: async () => !(await owner.schedulingSuppressed()), + }, canRun); + const workers = [assignments, orphanBlockers, recoveryActions, dependencies, locks, watchdogs]; + return { ready: Promise.all(workers.map(worker => worker.ready)), + wake() { for (const worker of workers) worker.invalidate(); }, + async stop() { await Promise.all(workers.map(worker => worker.stop())); } }; +} diff --git a/server/src/services/run-dispatch-notifications.ts b/server/src/services/run-dispatch-notifications.ts index 276edacab23..4fca35c1d2a 100644 --- a/server/src/services/run-dispatch-notifications.ts +++ b/server/src/services/run-dispatch-notifications.ts @@ -1,8 +1,10 @@ +import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; export type RunDispatchScope = { companyId: string }; export async function notifyRunDispatch(transaction: object, scope?: RunDispatchScope) { + await notifyIssueRecovery(transaction, scope ? { companyId: scope.companyId } : undefined); const hint = scope ? JSON.stringify({ companyId: scope.companyId }) : "*"; await notifyDeliveryWork(transaction, DELIVERY_QUEUES.queuedRunDispatch, hint); } diff --git a/server/src/services/scoped-deadline-work.ts b/server/src/services/scoped-deadline-work.ts index 8c94a863ccd..fd90a9b372e 100644 --- a/server/src/services/scoped-deadline-work.ts +++ b/server/src/services/scoped-deadline-work.ts @@ -13,6 +13,8 @@ export function registerScopedDeadlineWork( batchSize?: number; minimumPassIntervalMs?: number; concurrentChecks?: { limit: number; key(entry: Entry): string }; + reconstructionIntervalMs?: number; + beforePass?(): Promise; decodeScope(hint: string): Scope | undefined; matches(entry: Entry, scope?: Scope): boolean; load(scope?: Scope, signal?: AbortSignal): Promise; @@ -29,6 +31,7 @@ export function registerScopedDeadlineWork( const entries = new Map(); const failures = new Map(); const scopeFailures = new Map(); + const reconstructAfter = new Map(); let earliest = 0; let lastCompany: string | null = null; function failed(map: typeof failures, id: string) { @@ -38,13 +41,14 @@ export function registerScopedDeadlineWork( function dirtyScopes() { return [...dirty].filter(hint => hint !== "*" || !scopeFailures.has(hint)) .filter(hint => !scans.has(hint) || scopeFailures.has(hint)) + .filter(hint => (reconstructAfter.get(hint) ?? 0) <= Date.now() || scopeFailures.has(hint)) .map(hint => { try { return options.decodeScope(hint); } catch { return undefined; } }); } function next() { if (!canRun()) return dirty.size > 0 || entries.size > 0 ? Date.now() + 30_000 : null; let nextAt: number | null = null; const take = (at: number) => { nextAt = nextAt === null ? at : Math.min(nextAt, at); }; - for (const hint of dirty) take(Math.max(earliest, scopeFailures.get(hint)?.retryAt ?? 0)); + for (const hint of dirty) take(Math.max(earliest, reconstructAfter.get(hint) ?? 0, scopeFailures.get(hint)?.retryAt ?? 0)); const unavailable = dirtyScopes(); for (const entry of entries.values()) if (!unavailable.some(scope => options.matches(entry, scope))) take(Math.max(earliest, entry.dueAt, failures.get(entry.id)?.retryAt ?? 0)); @@ -58,9 +62,18 @@ export function registerScopedDeadlineWork( } }, run: async signal => { if (!canRun() || Date.now() < earliest) return; + if ((options.reconstructionIntervalMs ?? 0) > 0) for (const [hint, until] of reconstructAfter) { + if (until <= Date.now()) reconstructAfter.delete(hint); + } + const ready = [...dirty].filter(hint => Math.max(reconstructAfter.get(hint) ?? 0, scopeFailures.get(hint)?.retryAt ?? 0) <= Date.now()); + if (ready.length === 0 && ![...entries.values()].some(entry => Math.max(entry.dueAt, failures.get(entry.id)?.retryAt ?? 0) <= Date.now())) return; earliest = Date.now() + (options.minimumPassIntervalMs ?? 1000); + try { await options.beforePass?.(); } catch (error) { + for (const hint of ready) failed(scopeFailures, hint); + for (const entry of entries.values()) if (entry.dueAt <= Date.now()) failed(failures, entry.id); + throw error; + } const errors: unknown[] = []; - const ready = [...dirty].filter(hint => (scopeFailures.get(hint)?.retryAt ?? 0) <= Date.now()); const requested = ready.includes("*") ? ["*", ...ready.filter(hint => hint !== "*")].slice(0, 25) : ready.slice(0, 25); @@ -85,6 +98,9 @@ export function registerScopedDeadlineWork( entries.delete(entry.id); failures.delete(entry.id); } scans.delete(hint); + if ((options.reconstructionIntervalMs ?? 0) > 0) { + reconstructAfter.set(hint, Date.now() + options.reconstructionIntervalMs!); + } if ((versions.get(hint) ?? 0) !== scan.version) dirty.add(hint); else versions.delete(hint); if (!scope) scopeFailures.clear(); else scopeFailures.delete(hint); } catch (error) { diff --git a/server/src/services/system-recovery.ts b/server/src/services/system-recovery.ts index a554a765c4c..347593dafed 100644 --- a/server/src/services/system-recovery.ts +++ b/server/src/services/system-recovery.ts @@ -1,3 +1,4 @@ +import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import { and, eq, sql } from "drizzle-orm"; import { heartbeatRuns, issueComments, issueRecoveryActions, type Db } from "@paperclipai/db"; import { writeExecutionControl } from "./execution-control-notifications.js"; @@ -11,6 +12,7 @@ export async function adoptSystemRecovery(db: Db, source: WorkspaceRecoverySourc .where(and(eq(issueRecoveryActions.companyId, source.companyId), eq(issueRecoveryActions.id, source.actionId), eq(issueRecoveryActions.sourceIssueId, source.issueId))).for("update"); if (!existing || existing.evidence.systemRecovery) return; + await notifyIssueRecovery(tx, { companyId: source.companyId }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.workspaceRecovery); const state = createSystemRecovery(source.runId); // Atomic, once per source. An exhausted previous execution budget is history; diff --git a/server/src/services/task-retry.ts b/server/src/services/task-retry.ts index 17075c34eb9..ba5959b3eb6 100644 --- a/server/src/services/task-retry.ts +++ b/server/src/services/task-retry.ts @@ -339,8 +339,10 @@ export async function requestTaskRetry(input: { /** Chat owns admission and authorization. The task receipt follows that * durable outcome; it never launches a parallel generic board execution. */ -export async function reconcileTaskChatRetryReceipts(db: Db) { +export async function reconcileTaskChatRetryReceipts(db: Db, scope: { companyId?: string; requestId?: string; settleOnly?: boolean } = {}) { const pending = await db.select().from(agentWakeupRequests).where(and( + scope.companyId ? eq(agentWakeupRequests.companyId, scope.companyId) : undefined, + scope.requestId ? eq(agentWakeupRequests.id, scope.requestId) : undefined, eq(agentWakeupRequests.reason, "task_retry"), eq(agentWakeupRequests.status, "deferred_issue_execution"), sql`${agentWakeupRequests.payload}#>>'{taskRetry,chatActionId}' is not null`, )).orderBy(agentWakeupRequests.updatedAt).limit(50); @@ -356,7 +358,7 @@ export async function reconcileTaskChatRetryReceipts(db: Db) { // A deferred chat receipt is admitted input, not a completed dispatch. // Re-stage that exact action through chat's owner; it rechecks authority // and normal queue policy before updating the existing receipt. - if (chat?.status === "processed" && chat.payload.taskRetryIntentId === wake.id && + if (!scope.settleOnly && chat?.status === "processed" && chat.payload.taskRetryIntentId === wake.id && chat.payload.failedRunId === intent.sourceRunId && receipt?.status === "deferred_issue_execution" && !receipt.runId && receipt.payload?.taskRetryRequestId === wake.id) { await db.transaction(async tx => { diff --git a/server/src/services/task-watchdog-diagnostics.ts b/server/src/services/task-watchdog-diagnostics.ts deleted file mode 100644 index ea08eebe2b4..00000000000 --- a/server/src/services/task-watchdog-diagnostics.ts +++ /dev/null @@ -1,16 +0,0 @@ -export function createIncompleteTaskWatchdogScanReporter( - warn: (details: { incomplete: number; newIncompleteIssueIds: string[] }, message: string) => void, -) { - let previousIncompleteIssueIds = new Set(); - return (result: { incomplete: number; incompleteIssueIds: string[] }, phase: "startup" | "periodic") => { - const currentIssueIds = new Set(result.incompleteIssueIds); - const newIncompleteIssueIds = [...currentIssueIds].filter((id) => !previousIncompleteIssueIds.has(id)); - previousIncompleteIssueIds = currentIssueIds; - if (newIncompleteIssueIds.length === 0) return; - warn( - { incomplete: result.incomplete, newIncompleteIssueIds }, - `${phase} task-watchdog reconciliation found incomplete subtree scans`, - ); - }; -} - diff --git a/server/src/services/task-watchdog-work.ts b/server/src/services/task-watchdog-work.ts new file mode 100644 index 00000000000..98794472da2 --- /dev/null +++ b/server/src/services/task-watchdog-work.ts @@ -0,0 +1,49 @@ +import { and, asc, eq, gt } from "drizzle-orm"; +import { issueWatchdogs, type Db } from "@paperclipai/db"; +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import type { IssueRecoveryScope } from "./issue-recovery-notifications.js"; +import { registerScopedDeadlineWork } from "./scoped-deadline-work.js"; + +type Entry = { id: string; companyId: string; dueAt: number }; + +export function registerTaskWatchdogWork( + db: Db, + coordinator: ReturnType, + owner: { + enabled(): Promise; + reconcile(scope: { companyId: string; watchdogId: string; onDeferredDeadline: (at: number) => void }): Promise<{ incomplete: number }>; + }, + canRun: () => boolean, +) { + const cursors = new Map(); + let enabled = false; + return registerScopedDeadlineWork(coordinator, { + queue: DELIVERY_QUEUES.taskWatchdogs, + reconstructionIntervalMs: 30_000, + beforePass: async () => { enabled = await owner.enabled(); }, + decodeScope: hint => hint === "*" ? undefined : JSON.parse(hint) as IssueRecoveryScope, + matches: (entry, scope) => !scope || entry.companyId === scope.companyId, + load: async scope => { + const key = scope?.companyId ?? "*"; + if (!enabled) { cursors.delete(key); return []; } + const cursor = cursors.get(key); + const rows = await db.select({ id: issueWatchdogs.id, companyId: issueWatchdogs.companyId }) + .from(issueWatchdogs).where(and(eq(issueWatchdogs.status, "active"), + scope ? eq(issueWatchdogs.companyId, scope.companyId) : undefined, + cursor ? gt(issueWatchdogs.id, cursor) : undefined)) + .orderBy(asc(issueWatchdogs.id)).limit(100); + const complete = rows.length < 100; + if (complete) cursors.delete(key); else cursors.set(key, rows.at(-1)!.id); + return { rows: rows.map(row => ({ ...row, dueAt: Date.now() })), complete }; + }, + check: async entry => { + if (!enabled) return null; + let next: number | null = null; + const result = await owner.reconcile({ companyId: entry.companyId, watchdogId: entry.id, + onDeferredDeadline: at => { next = next === null ? at : Math.min(next, at); } }); + if (result.incomplete > 0) throw new Error(`Task watchdog ${entry.id} in company ${entry.companyId} has an incomplete subtree`); + return next === null ? null : { ...entry, dueAt: next }; + }, + }, canRun); +} diff --git a/server/src/services/task-watchdogs.ts b/server/src/services/task-watchdogs.ts index cf8d92d39df..dd1aa980d3c 100644 --- a/server/src/services/task-watchdogs.ts +++ b/server/src/services/task-watchdogs.ts @@ -1,3 +1,4 @@ +import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import { createHash } from "node:crypto"; import { and, asc, eq, gte, inArray, isNull, or, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; @@ -831,7 +832,7 @@ async function updateIssueWatchdogRow( return updated; } -export async function upsertIssueWatchdogForIssue( +async function upsertIssueWatchdogInTransaction( dbOrTx: any, companyId: string, issueId: string, @@ -847,12 +848,13 @@ export async function upsertIssueWatchdogForIssue( .where(and(eq(issueWatchdogs.companyId, companyId), eq(issueWatchdogs.issueId, issueId))) .then((rows: IssueWatchdogRow[]) => rows[0] ?? null); + await notifyIssueRecovery(dbOrTx, { companyId }); if (existing) { const updated = await updateIssueWatchdogRow(dbOrTx, existing, input, now); return { watchdog: toIssueWatchdog(updated), created: false }; } - const insertResult: { row: IssueWatchdogRow; created: boolean } = await dbOrTx + const insertResult: { row: IssueWatchdogRow; created: boolean } = await dbOrTx.transaction(async (tx: Db) => tx .insert(issueWatchdogs) .values({ companyId, @@ -870,7 +872,7 @@ export async function upsertIssueWatchdogForIssue( updatedAt: now, }) .returning() - .then((rows: IssueWatchdogRow[]) => ({ row: rows[0], created: true })) + .then((rows: IssueWatchdogRow[]) => ({ row: rows[0], created: true }))) .catch(async (error: unknown) => { if (!isIssueWatchdogUniqueConflict(error)) throw error; const winner = await dbOrTx @@ -885,6 +887,15 @@ export async function upsertIssueWatchdogForIssue( return { watchdog: toIssueWatchdog(insertResult.row), created: insertResult.created }; } +export function upsertIssueWatchdogForIssue( + dbOrTx: Db | Parameters[0]>[0], + companyId: string, + issueId: string, + input: IssueWatchdogUpsertInput, +): Promise<{ watchdog: IssueWatchdog; created: boolean }> { + return dbOrTx.transaction(tx => upsertIssueWatchdogInTransaction(tx, companyId, issueId, input)); +} + export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) { const issuesSvc = issueService(db); @@ -1478,7 +1489,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) return created; } - async function evaluateWatchdog(row: IssueWatchdogRow, opts: { runId?: string | null } = {}) { + async function evaluateWatchdog(row: IssueWatchdogRow, opts: { runId?: string | null; onDeferredDeadline?: (at: number) => void } = {}) { const watchdog = await markTerminalWatchdogIssueReviewed(row, opts); const sourceIssue = await db .select() @@ -1491,6 +1502,13 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) const input = await collectClassifierInput(watchdog.companyId, watchdog); const classification = classifyTaskWatchdogSubtree(input); + if (classification.state === "pending_first_run") { + const pending = new Set(classification.pendingIssueIds); + for (const issue of input.issues) if (pending.has(issue.id)) { + const createdAt = toEpochMs(issue.createdAt); + if (createdAt !== null) opts.onDeferredDeadline?.(createdAt + TASK_WATCHDOG_FIRST_RUN_GRACE_MS); + } + } if (classification.state !== "stopped") { return { state: classification.state, reason: classification.reason, classification }; } @@ -1610,13 +1628,14 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) }; } - async function listActiveWatchdogsForCompany(companyId?: string | null) { + async function listActiveWatchdogsForCompany(companyId?: string | null, watchdogId?: string) { return db .select() .from(issueWatchdogs) .where(and( eq(issueWatchdogs.status, "active"), ...(companyId ? [eq(issueWatchdogs.companyId, companyId)] : []), + watchdogId ? eq(issueWatchdogs.id, watchdogId) : undefined, )); } @@ -1746,14 +1765,16 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) issueId: string, actor: ActorFields = {}, ): Promise => { - await assertWatchedIssue(db, companyId, issueId); - const existing = await db + return db.transaction(async tx => { + await assertWatchedIssue(tx, companyId, issueId); + const existing = await tx .select() .from(issueWatchdogs) .where(and(eq(issueWatchdogs.companyId, companyId), eq(issueWatchdogs.issueId, issueId))) .then((rows) => rows[0] ?? null); if (!existing || existing.status === "disabled") return null; - const [updated] = await db + await notifyIssueRecovery(tx, { companyId }); + const [updated] = await tx .update(issueWatchdogs) .set({ status: "disabled", @@ -1765,14 +1786,17 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) .where(eq(issueWatchdogs.id, existing.id)) .returning(); return toIssueWatchdog(updated); + }); }, reconcileTaskWatchdogs: async (opts: { companyId?: string | null; runId?: string | null; + watchdogId?: string; + onDeferredDeadline?: (at: number) => void; issueCreatedAtGte?: Date | null; } = {}) => { - let rows = await listActiveWatchdogsForCompany(opts.companyId ?? null); + let rows = await listActiveWatchdogsForCompany(opts.companyId ?? null, opts.watchdogId); if (opts.issueCreatedAtGte) { const watchdogIssueIds = [...new Set(rows.map((row) => row.issueId))]; const eligibleIssueIds = new Set( @@ -1802,7 +1826,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) }; for (const row of rows) { result.checked += 1; - const evaluated = await evaluateWatchdog(row, { runId: opts.runId ?? null }); + const evaluated = await evaluateWatchdog(row, { runId: opts.runId ?? null, onDeferredDeadline: opts.onDeferredDeadline }); if (evaluated.state === "triggered") { result.triggered += 1; result.watchdogIssueIds.push(evaluated.watchdogIssueId); diff --git a/tests/e2e/agent-chat-projects.spec.ts b/tests/e2e/agent-chat-projects.spec.ts index 0b43071ecd0..46ea6ca7844 100644 --- a/tests/e2e/agent-chat-projects.spec.ts +++ b/tests/e2e/agent-chat-projects.spec.ts @@ -467,17 +467,10 @@ test("plan approval hands the preserved revision to an assigned project task", a expect(task.parentId).toBeNull(); expect(task.projectId).toBeTruthy(); expect(task.assigneeAgentId).toBe(f.agent.id); - const completion = page.getByRole("region", { name: "Conversation messages" }) - .locator("p") - .filter({ hasText: /^Delegated work completed:/ }); - await expect(completion).toHaveCount(1); - const completedTask = completion.getByRole("link").filter({ hasText: task.identifier }); - await expect(completedTask).toBeVisible(); - await expect(completedTask).toContainText(task.identifier); - await expect(completedTask).toHaveAttribute( - "href", - `/${f.company.issuePrefix}/issues/${task.identifier}`, - ); + const completion = page.getByText("Delegated work completed:", { exact: false }); + await expect(completion.getByRole("link").and(page.locator( + `a[href="/${f.company.issuePrefix}/issues/${task.identifier}"]`, + ))).toBeVisible(); const plan = await json( await request.get(`/api/issues/${task.id}/documents/plan`), ); diff --git a/tests/e2e/legacy-failure-continuation.spec.ts b/tests/e2e/legacy-failure-continuation.spec.ts index ed998017eae..46ce639e5b5 100644 --- a/tests/e2e/legacy-failure-continuation.spec.ts +++ b/tests/e2e/legacy-failure-continuation.spec.ts @@ -11,14 +11,15 @@ async function json(response: APIResponse) { return response.json(); } -for (const action of ["task_retry", "retry_after_reload", "inbox_retry", "message", "queued_retry", "automatic_message"] as const) { +for (const action of ["task_retry", "retry_after_reload", "inbox_retry", "message", "queued_retry", "automatic_message_on_reactivation"] as const) { test(`legacy startup hold: ${action} reaches a new agent response`, async ({ page, request }) => { test.setTimeout(120_000); const root = await mkdtemp(path.join(os.tmpdir(), "legacy-recovery-browser-")); const config = JSON.parse(await readFile(process.env.PAPERCLIP_E2E_SERVER_CONFIG!, "utf8")); // Use the running test server's actual port, including fallback allocation. - const pid = await readFile(path.join(config.database.embeddedPostgresDataDir, "postmaster.pid"), "utf8"); - const url = `postgres://paperclip:paperclip@127.0.0.1:${pid.split("\n")[3]}/paperclip`; + const url = config.database.mode === "postgres" ? config.database.connectionString + : `postgres://paperclip:paperclip@127.0.0.1:${(await readFile( + path.join(config.database.embeddedPostgresDataDir, "postmaster.pid"), "utf8")).split("\n")[3]}/paperclip`; const db = createDb(url); const company = await json(await request.post("/api/companies", { data: { name: `Legacy recovery ${action} ${Date.now()}` } })); try { @@ -37,8 +38,11 @@ for (const action of ["task_retry", "retry_after_reload", "inbox_retry", "messag title: "Continue after startup failure", description: "Answer the pending follow-up once.", status: "backlog", assigneeAgentId: agent.id, } })); + if (action === "automatic_message_on_reactivation") { + await json(await request.patch(`/api/companies/${company.id}`, { data: { status: "paused" } })); + } const sourceRunId = randomUUID(); - // Seed the historical incident, then exercise all recovery through the UI. + // Seed the historical incident, then exercise ordinary recovery entry points. // No adapter.invoke or new dispatch identity exists on this pre-upgrade run. await db.insert(heartbeatRuns).values({ id: sourceRunId, companyId: company.id, agentId: agent.id, status: "failed", runtimeMode: "legacy", processPid: action === "queued_retry" ? process.pid : 999999999, @@ -58,7 +62,7 @@ for (const action of ["task_retry", "retry_after_reload", "inbox_retry", "messag await db.insert(companyMemberships).values({ companyId: company.id, principalType: "user", principalId: "original-board", membershipRole: "operator", status: "active" }); } - if (action === "queued_retry" || action === "automatic_message") { + if (action === "queued_retry" || action === "automatic_message_on_reactivation") { const commentId = randomUUID(); // Reproduce a real user comment saved while the failed run was active, // including queues originally created by a system wake. @@ -104,8 +108,19 @@ for (const action of ["task_retry", "retry_after_reload", "inbox_retry", "messag retry.click(), ]); expect(response.ok(), `${response.status()} ${await response.text()}`).toBe(true); - } else if (action === "automatic_message") { - // No Retry, duplicate message, or status change: the saved input runs. + } else if (action === "automatic_message_on_reactivation") { + // This fixture writes history through a separate SQL connection after boot. + // Reactivation is the supported producer that makes saved work eligible; + // startup reconstruction itself is covered before registration in the PG suite. + const pending = await db.select().from(agentWakeupRequests).where(and( + eq(agentWakeupRequests.companyId, company.id), eq(agentWakeupRequests.agentId, agent.id), + eq(agentWakeupRequests.status, "deferred_issue_execution"), + )); + expect(pending).toHaveLength(1); + const before = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, company.id)); + expect(before).toHaveLength(1); + await json(await request.patch(`/api/companies/${company.id}`, { data: { status: "active" } })); + // No Retry or duplicate message: the existing saved input runs once. } else if (action === "message") { await page.getByRole("textbox", { name: "editable markdown" }).fill("Please continue the pending follow-up."); await page.getByRole("button", { name: "Send", exact: true }).click(); @@ -133,7 +148,7 @@ for (const action of ["task_retry", "retry_after_reload", "inbox_retry", "messag expect(runs.filter(run => run.id !== sourceRunId)).toHaveLength(1); expect(runs.find(run => run.id === sourceRunId)).toMatchObject({ status: "failed", resultJson: null }); const prompts = await readFile(path.join(root, "prompts"), "utf8"); - if (action === "queued_retry" || action === "automatic_message") { + if (action === "queued_retry" || action === "automatic_message_on_reactivation") { expect(prompts).toContain("Approved"); await expect(page.getByRole("button", { name: "Interrupt", exact: true })).toHaveCount(0); }