From 3afb2c215b8226fd2928dbf6513eaa06fcde1601 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 16 Nov 2026 22:15:40 -0600 Subject: [PATCH] fix(server): name the blocking database sessions whenever a run's event writes stall (#21956) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server stores the events that a runner sends for each run, and it records the identity of the runner process > - Sometimes the server stops storing a run's events for minutes, and the server log stays quiet > - In two restart-chaos cases, the identity write of the run started and then did not commit for about 197 s and 480 s > - The existing database report only fires when the reattach phase ends at its cap, so it did not fire for the cold-start case > - This pull request logs the pending writes and the blocking database sessions whenever any run write stays pending for 30 s > - The benefit is that the next stall names the session and the query that hold the run, so the fix can target one proven holder ## Linked Issues or Issue Description Refs #21946 **What happened?** Restart-chaos waves on master showed two stalls in event storage. 1. Cold start of an OpenCode run, with no restart. The runner emitted `session.started` at 22:13:23.7Z. The server stored it at 22:16:40.97Z, about 197 s later. The second `native.process_identity_recorded` write of the run started at 22:13:23.706Z, and `runner.turn.submit` took 197270 ms. A second wave showed the same shape (about 191 s). The server log had nothing in that window. 2. After a hard kill, every API GET timed out from 22:35:24Z to 22:43:23Z, server CPU was 0%, and no run events were stored. The process identity transaction of the run began at 22:35:22.6Z and committed at about 22:43:23Z. **Expected behavior** When the server cannot store a run's events, the server log names the cause: which run writes are pending, and which database sessions hold or wait for locks. **Steps to reproduce** 1. Start a transaction that locks a `heartbeat_runs` row (`for update`) and stays idle. 2. Record a process identity for that run (`persistHeartbeatRunProcessMetadata`). 3. Before this change, nothing is logged while the write waits. After this change, a warning names the idle holder and the waiting statement after 30 s. The new database test does this with a 300 ms threshold. **Relevant logs or output** Example warning from the new test: `runner event intake stalled; pending run writes, database sessions and lock waits` with `operations: [{stage: "process_identity", pendingMs: 301}]` and `waits: [{state: "idle in transaction", query: "select id from heartbeat_runs where id = ? ... for update"}, {waitEventType: "Lock", blockedBy: []}]`. ## What Changed - New `server/src/services/run-intake-stall-watch.ts`. It keeps an in-memory list of pending run operations: event receipts (`PaperclipControlPlanePort.appendEvent`), event writes (`appendHeartbeatRunEvent`, `appendHeartbeatRunEventBatch`), and process identity writes (`persistHeartbeatRunProcessMetadata`). - When an operation stays pending for 30 s, the watch logs one warning per stalled run. The warning has: the pending operations and their age, the pending count and oldest age per run, the pool size, the database sessions per application name and state, the lock waits and holders (`pg_blocking_pids`, query without literals, cut to 160 characters), and the event-loop lag. Reports are at most one per 30 s. A second line logs when a reported operation finishes. - The watch arms a timer only while an operation is pending. It reads the database only when it reports, through the private liveness pool with a 5 s statement timeout. - The server names its main pool `paperclip:` unless `DATABASE_APPLICATION_NAME` is set. This shows which server process owns each session after a restart. - `readDatabaseSessionWaits` moves from the reattach phase to `server/src/services/database-session-waits.ts`. The reattach phase re-exports it. - `@paperclipai/db` exports `databaseClientOptionsFromEnv` and `DatabaseClientOptions`. - `doc/DATABASE.md` describes the new log lines and the pool name. - I did not add `idle_in_transaction_session_timeout` to the main pool. Some code holds a transaction open on purpose, and a blind timeout could abort valid work. The new report names idle-in-transaction holders instead. - These are server log lines only. This change does not touch Telemetry or OpenTelemetry. ## Verification - `cd server && npx vitest run src/services/run-intake-stall-watch.test.ts` (5 tests). The database test holds a run row in an idle transaction, starts a process identity write, and checks that the warning names the holder pid, its query, the waiting statement, and the pool name. It is red without the tracking in `run-lifecycle.ts` and `heartbeat-run-events.ts` (`expected [] to have a length of 1`), and green with it. - Unit tests with fake timers check one report per stalled run, the 30 s throttle, the clear line, and that no timer stays armed when nothing is pending. - `cd server && npx vitest run src/services/controller-process-identity.test.ts` checks the pool name and the `DATABASE_APPLICATION_NAME` override. - `npx tsc --noEmit` passes for `server` and `packages/db`. - `pnpm verify` ran locally on the affected tests. ## Risks - Low risk. Each run event adds one map insert and one map delete. No database work happens while writes finish within 30 s. - The main pool `application_name` changes from `paperclip` to `paperclip:`. A dashboard that filters on exactly `paperclip` must use a prefix match, or the operator can set `DATABASE_APPLICATION_NAME`. - A report reads `pg_stat_activity` on the liveness pool. A report during a full database stall can fail; the warning then has `databaseReadError` and still lists the pending operations. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/DATABASE.md | 14 +- docs/deploy/database.md | 4 +- packages/db/src/index.ts | 2 + server/src/index.ts | 8 +- .../controller-process-identity.test.ts | 14 +- .../services/controller-process-identity.ts | 5 + server/src/services/database-session-waits.ts | 77 ++++++ server/src/services/heartbeat-run-events.ts | 30 ++- .../src/services/heartbeat/reattach-phase.ts | 53 +--- .../src/services/heartbeat/run-lifecycle.ts | 5 +- .../src/services/native-local-process-stop.ts | 4 +- ...ative-run-coordinator-store-intake.test.ts | 26 ++ .../native-run-coordinator-store.ts | 11 + .../paperclip-control-plane-port.ts | 7 + .../services/run-intake-stall-watch.test.ts | 245 ++++++++++++++++++ server/src/services/run-intake-stall-watch.ts | 156 +++++++++++ 16 files changed, 602 insertions(+), 59 deletions(-) create mode 100644 server/src/services/database-session-waits.ts create mode 100644 server/src/services/native-runtime/native-run-coordinator-store-intake.test.ts create mode 100644 server/src/services/run-intake-stall-watch.test.ts create mode 100644 server/src/services/run-intake-stall-watch.ts diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 7a2458af677..e37932a4a89 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -134,9 +134,21 @@ DATABASE_POOL_MAX=25 # connection pool size; default: 10 DATABASE_IDLE_TIMEOUT_SECONDS=60 # close idle pooled connections; default: 60 (0 = keep open) DATABASE_CONNECT_TIMEOUT_SECONDS=10 # default: 30 DATABASE_MAX_LIFETIME_SECONDS=1800 # recycle a pooled connection after this long; default: 30-60 min (random) -DATABASE_APPLICATION_NAME=paperclip # application_name in pg_stat_activity; default: paperclip +DATABASE_APPLICATION_NAME=paperclip # application_name in pg_stat_activity; default for the server's main pool: paperclip: ``` +The server names its main pool `paperclip:`, so after a restart the +sessions of each server process can be told apart in `pg_stat_activity`. When a +run's event, process identity, or event-write operation stays unfinished for +30 seconds, the server logs one warning per stalled run ("runner event intake +stalled"). The warning lists the pending operations of each run, the pool size, +the database sessions per application name and state, and the sessions that +wait for a lock or hold one (`pg_blocking_pids`, query text without literals, +cut to 160 characters). It also logs the event-loop lag. A second line +("runner event intake stall cleared") records when the operation finished. The +check runs only while an operation is pending, and it reads `pg_stat_activity` +only when it reports a stall. + `withDedicatedDbConnection` gives an action exclusive use of one session outside the main pool (application name `paperclip-workspace-finalization-lock`). Each `createDb` owner keeps released single-connection clients idle for reuse: always diff --git a/docs/deploy/database.md b/docs/deploy/database.md index 7d9ea38fe3e..57cb92f5c4f 100644 --- a/docs/deploy/database.md +++ b/docs/deploy/database.md @@ -73,7 +73,7 @@ statements longer than 8 KB. If a prepared statement fails because the pooler's stale after a schema change or was lost, Paperclip stops preparing that statement and retries it once unprepared. -Related optional client tuning: `DATABASE_POOL_MAX`, `DATABASE_IDLE_TIMEOUT_SECONDS`, `DATABASE_CONNECT_TIMEOUT_SECONDS`, `DATABASE_MAX_LIFETIME_SECONDS`, `DATABASE_APPLICATION_NAME`. Driver defaults apply when unset, except that idle pooled connections close after 60 seconds (`DATABASE_IDLE_TIMEOUT_SECONDS=0` keeps them open) and the pool reports `application_name=paperclip`. See [Connection pool settings](#connection-pool-settings). +Related optional client tuning: `DATABASE_POOL_MAX`, `DATABASE_IDLE_TIMEOUT_SECONDS`, `DATABASE_CONNECT_TIMEOUT_SECONDS`, `DATABASE_MAX_LIFETIME_SECONDS`, `DATABASE_APPLICATION_NAME`. Driver defaults apply when unset, except that idle pooled connections close after 60 seconds (`DATABASE_IDLE_TIMEOUT_SECONDS=0` keeps them open) and the server's main pool reports `application_name=paperclip:` (other pools use names that start with `paperclip`). See [Connection pool settings](#connection-pool-settings). ## Connection Pool Settings @@ -85,7 +85,7 @@ The server opens one postgres.js pool for its own queries (and a second one when | `DATABASE_IDLE_TIMEOUT_SECONDS` | `60` | Close a pooled connection after this much idle time. `0` keeps idle connections open forever (the driver default). | | `DATABASE_CONNECT_TIMEOUT_SECONDS` | `30` (driver) | Give up on a connection attempt after this long. | | `DATABASE_MAX_LIFETIME_SECONDS` | 30–60 min, randomized (driver) | Recycle a pooled connection once it is this old. | -| `DATABASE_APPLICATION_NAME` | `paperclip` | Value of `application_name` in `pg_stat_activity`, so you can find Paperclip's backends: `SELECT * FROM pg_stat_activity WHERE application_name = 'paperclip';` | +| `DATABASE_APPLICATION_NAME` | `paperclip:` (server main pool) | Value of `application_name` in `pg_stat_activity`. The boot id tells the sessions of each server process apart after a restart. Find Paperclip's backends with `SELECT * FROM pg_stat_activity WHERE application_name LIKE 'paperclip%';` | | `DATABASE_DEDICATED_CONNECTION_MAX` | `24` | Dedicated lock connections in use at once per process before new callers wait. Nested callers do not wait. | | `DATABASE_DEDICATED_CONNECTION_WAIT_MS` | `30000` | After this wait, a queued caller gets a dedicated connection over the limit. | | `DATABASE_PREPARED_STATEMENTS` | Unset | `true` opts parameterized ORM queries into preparation. `false` disables preparation. Use `false` when the pooler does not support named statements. Unset preserves existing behavior. | diff --git a/packages/db/src/index.ts b/packages/db/src/index.ts index c5b68087976..958482a0a7c 100644 --- a/packages/db/src/index.ts +++ b/packages/db/src/index.ts @@ -1,5 +1,7 @@ export { createDb, + databaseClientOptionsFromEnv, + type DatabaseClientOptions, withDedicatedDbConnection, closeDedicatedDbConnections, DEDICATED_CONNECTION_IDLE_MAX, diff --git a/server/src/index.ts b/server/src/index.ts index 0cb0eb45c61..0fb6d43c6d4 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -136,6 +136,8 @@ import { } from "@paperclipai/db"; import detectPort from "detect-port"; import { createApp } from "./app.js"; +import { serverDatabaseClientOptions } from "./services/controller-process-identity.js"; +import { installRunIntakeStallWatch, stopRunIntakeStallWatch } from "./services/run-intake-stall-watch.js"; import { drainHttpRequestWork } from "./middleware/idle-admission.js"; import { initializeCloudRuntimeConfig } from "./services/cloud-runtime-config.js"; import { loadCloudBootstrapConfig, loadConfig, type Config } from "./config.js"; @@ -537,7 +539,7 @@ async function startServerWithDatabaseTeardown( const migrationUrl = startupConfig.databaseMigrationUrl ?? startupConfig.databaseUrl; migrationSummary = await ensureMigrations(migrationUrl, "PostgreSQL"); - db = createDb(startupConfig.databaseUrl); + db = createDb(startupConfig.databaseUrl, serverDatabaseClientOptions()); pluginMigrationDb = startupConfig.databaseMigrationUrl ? createDb(startupConfig.databaseMigrationUrl) : db; logger.info("Using external PostgreSQL via DATABASE_URL/config"); activeDatabaseConnectionString = startupConfig.databaseUrl; @@ -744,7 +746,7 @@ async function startServerWithDatabaseTeardown( autoApply: shouldAutoApplyFirstRunMigrations, }); - db = createDb(embeddedConnectionString); + db = createDb(embeddedConnectionString, serverDatabaseClientOptions()); pluginMigrationDb = db; logger.info("Embedded PostgreSQL ready"); activeDatabaseConnectionString = embeddedConnectionString; @@ -757,10 +759,12 @@ async function startServerWithDatabaseTeardown( // by the fail-loud startup path, so no exit leaves pooled backends behind. const closeDatabaseClients = async () => { setSentryRunLinkResolver(null); + stopRunIntakeStallWatch(); const clients = pluginMigrationDb === db ? [db] : [db, pluginMigrationDb]; await Promise.all([...clients.map((client) => endDatabaseClient(client, 5)), closeLivenessDb(db, 5), closeCheckpointDb(db, 5), closeFileStorageDb(db, 5), closeDedicatedDbConnections(db, 5)]); }; startupDatabase.close = closeDatabaseClients; + installRunIntakeStallWatch(db); startupTimeline.mark("database"); setSentryRunLinkResolver(createSentryRunLinkResolver(db)); await migrateCloudConfiguration(db as any); diff --git a/server/src/services/controller-process-identity.test.ts b/server/src/services/controller-process-identity.test.ts index 2fd8b9a465f..ea4ee5dde81 100644 --- a/server/src/services/controller-process-identity.test.ts +++ b/server/src/services/controller-process-identity.test.ts @@ -1,11 +1,23 @@ import { spawn } from "node:child_process"; import { once } from "node:events"; import { afterEach, describe, expect, it, vi } from "vitest"; -import { processExists, processStartsDefinitelyDiffer } from "./controller-process-identity.js"; +import { currentControllerBootId, processExists, processStartsDefinitelyDiffer, serverDatabaseClientOptions } from "./controller-process-identity.js"; import { isPidAlive, isProcessGroupAlive } from "./local-service-supervisor.js"; import { readProcessStartedAt } from "./hot-restart.js"; import { recoveryProcessMayBeAlive } from "./native-runtime/native-session-executor.js"; +describe("server database client options", () => { + it("names the main pool after this boot so its database sessions can be told apart from an earlier server's", () => { + const options = serverDatabaseClientOptions({ DATABASE_POOL_MAX: "12" }); + expect(options).toEqual({ applicationName: `paperclip:${currentControllerBootId()}`, maxConnections: 12 }); + expect(options.applicationName!.length).toBeLessThanOrEqual(63); + }); + + it("keeps an operator's DATABASE_APPLICATION_NAME", () => { + expect(serverDatabaseClientOptions({ DATABASE_APPLICATION_NAME: "paperclip-web" }).applicationName).toBe("paperclip-web"); + }); +}); + describe("shared process identity proof", () => { afterEach(() => vi.restoreAllMocks()); diff --git a/server/src/services/controller-process-identity.ts b/server/src/services/controller-process-identity.ts index a5649a2f51a..2fd6ffc5941 100644 --- a/server/src/services/controller-process-identity.ts +++ b/server/src/services/controller-process-identity.ts @@ -2,6 +2,7 @@ import { createHash, randomUUID } from "node:crypto"; import { execFileSync } from "node:child_process"; import { readFileSync, readlinkSync } from "node:fs"; import { hostname } from "node:os"; +import { databaseClientOptionsFromEnv, type DatabaseClientOptions } from "@paperclipai/db"; import { readProcessStartedAt } from "./hot-restart.js"; import { currentWorkspaceWorker, type WorkspaceWorker } from "./native-runtime/native-workspace-platform-proof.js"; @@ -127,6 +128,10 @@ export function currentControllerBootId(): string { return bootId; } +export function serverDatabaseClientOptions(env: NodeJS.ProcessEnv = process.env): DatabaseClientOptions { + return { applicationName: `paperclip:${bootId}`, ...databaseClientOptionsFromEnv(env) }; +} + const stampListeners = new Set<() => void>(); /** Called (synchronously, without I/O) whenever this process stamps ownership of a record. */ export function onControllerStamp(listener: () => void): () => void { diff --git a/server/src/services/database-session-waits.ts b/server/src/services/database-session-waits.ts new file mode 100644 index 00000000000..eab20ff0fd3 --- /dev/null +++ b/server/src/services/database-session-waits.ts @@ -0,0 +1,77 @@ +import { sql } from "drizzle-orm"; +import { livenessDb, type Db } from "@paperclipai/db"; + +export type DatabaseSessionWait = { + pid: number; + applicationName: string | null; + state: string | null; + waitEventType: string | null; + waitEvent: string | null; + blockedBy: number[]; + transactionSeconds: number | null; + stateSeconds: number | null; + query: string | null; +}; + +export type DatabaseSessionCount = { applicationName: string; state: string; sessions: number }; + +const SESSION_WAITS_SQL = sql` + with sessions as ( + select pid, application_name, state, wait_event_type, wait_event, pg_blocking_pids(pid) as blocked_by, + xact_start, state_change, query + from pg_stat_activity + where datname = current_database() and pid <> pg_backend_pid() and backend_type = 'client backend' + ), blockers as (select distinct unnest(blocked_by) as pid from sessions) + select pid, application_name, state, wait_event_type, wait_event, blocked_by, + extract(epoch from clock_timestamp() - xact_start)::int as transaction_seconds, + extract(epoch from clock_timestamp() - state_change)::int as state_seconds, + left(regexp_replace(regexp_replace(query, '''(?:[^'']|'''')*''', '?', 'g'), '\\m[0-9][0-9.]*\\M', '?', 'g'), 160) as query + from sessions + where cardinality(blocked_by) > 0 or pid in (select pid from blockers) + or state like 'idle in transaction%' + or (xact_start is not null and clock_timestamp() - xact_start > interval '30 seconds') + order by xact_start nulls last + limit 50`; + +function sessionWait(row: Record): DatabaseSessionWait { + return { + pid: Number(row.pid), + applicationName: (row.application_name as string | null) ?? null, + state: (row.state as string | null) ?? null, + waitEventType: (row.wait_event_type as string | null) ?? null, + waitEvent: (row.wait_event as string | null) ?? null, + blockedBy: Array.isArray(row.blocked_by) ? row.blocked_by.map(Number) : [], + transactionSeconds: row.transaction_seconds === null ? null : Number(row.transaction_seconds), + stateSeconds: row.state_seconds === null ? null : Number(row.state_seconds), + query: (row.query as string | null) ?? null, + }; +} + +export async function readDatabaseSessionWaits(db: Db): Promise { + return livenessDb(db).transaction(async tx => { + await tx.execute(sql`select set_config('statement_timeout', '5000', true)`); + const rows = await tx.execute>(SESSION_WAITS_SQL); + return rows.map(sessionWait); + }); +} + +export async function readDatabaseSessionReport(db: Db, statementTimeoutMs = 5_000): Promise<{ + sessions: DatabaseSessionCount[]; waits: DatabaseSessionWait[]; +}> { + return livenessDb(db).transaction(async tx => { + await tx.execute(sql`select set_config('statement_timeout', ${String(statementTimeoutMs)}, true)`); + const counts = await tx.execute>(sql` + select coalesce(application_name, '') as application_name, coalesce(state, '') as state, count(*)::int as sessions + from pg_stat_activity + where datname = current_database() and pid <> pg_backend_pid() and backend_type = 'client backend' + group by 1, 2 + order by 3 desc + limit 40`); + const rows = await tx.execute>(SESSION_WAITS_SQL); + return { + sessions: counts.map(row => ({ applicationName: String(row.application_name), state: String(row.state), + sessions: Number(row.sessions) })), + waits: rows.map(sessionWait), + }; + }); +} diff --git a/server/src/services/heartbeat-run-events.ts b/server/src/services/heartbeat-run-events.ts index dc355a9a960..f248d3824cd 100644 --- a/server/src/services/heartbeat-run-events.ts +++ b/server/src/services/heartbeat-run-events.ts @@ -13,6 +13,7 @@ import { heartbeatRunEvents, heartbeatRuns, primeDatabaseWorkTransactionId } fro import { DATABASE_WORK_CHANNEL } from "@paperclipai/db/work-signals"; import { nativeSha256 } from "./native-runtime/canonical.js"; import { isUniqueViolation } from "../db-errors.js"; +import { trackRunIntake } from "./run-intake-stall-watch.js"; export const RETRY_EXHAUSTION_MESSAGE_PREFIX = "Bounded retry exhausted"; @@ -88,6 +89,14 @@ export async function allocateHeartbeatRunEventSeq( db: Db, runId: string, runUpdate: HeartbeatRunEventRunUpdate = {}, +): Promise { + return trackRunIntake(runId, "write", "event_sequence", () => reserveRunEventSeq(db, runId, runUpdate)); +} + +async function reserveRunEventSeq( + db: Db, + runId: string, + runUpdate: HeartbeatRunEventRunUpdate, ): Promise { const [updated] = await db .update(heartbeatRuns) @@ -118,6 +127,13 @@ function runEventWakes(input: AppendHeartbeatRunEventInput): RunEventWakes { export async function appendHeartbeatRunEvent( db: Db, input: AppendHeartbeatRunEventInput, +): Promise { + return trackRunIntake(input.runId, "write", input.eventType, () => appendRunEvent(db, input)); +} + +async function appendRunEvent( + db: Db, + input: AppendHeartbeatRunEventInput, ): Promise { // Deduplicated receipts return without allocating, so they cannot carry run changes. if (input.runUpdate && (input.nativeSource || input.retryExhaustion)) throw new Error("heartbeat_run_event_run_update_unsupported"); @@ -178,13 +194,23 @@ export async function appendHeartbeatRunEventBatch( ): Promise { const first = inputs[0]; if (!first) return []; + return trackRunIntake(first.runId, "write", `${first.eventType} x${inputs.length}`, + () => appendRunEventBatch(db, inputs, assertWritable)); +} + +async function appendRunEventBatch( + db: Db, + inputs: readonly AppendHeartbeatRunEventInput[], + assertWritable?: () => void, +): Promise { + const first = inputs[0]!; const source = first.nativeSource; if (!source || inputs.some(input => !input.nativeSource || input.runUpdate || input.retryExhaustion || input.companyId !== first.companyId || input.runId !== first.runId || input.agentId !== first.agentId || input.nativeSource.sourceInstanceId !== source.sourceInstanceId)) { throw new Error("heartbeat_run_event_batch_unsupported"); } - if (inputs.length === 1) return [await appendHeartbeatRunEvent(db, assertWritable ? { ...first, assertWritable } : first)]; + if (inputs.length === 1) return [await appendRunEvent(db, assertWritable ? { ...first, assertWritable } : first)]; const wakes = inputs.map(runEventWakes); if (!wakes.some(wake => Object.values(wake).some(Boolean)) && !is(db, PgTransaction)) { assertWritable?.(); @@ -596,7 +622,7 @@ async function appendLockedEvent( await notifyNativeResponsePresentation(tx, { companyId: run.companyId, runId: input.runId }); } if (wakes.retention) await notifyAttentionRetentionWork(tx, { companyId: input.companyId, sourceKind: "failed_run" }); - const seq = await allocateHeartbeatRunEventSeq(tx, input.runId, input.runUpdate); + const seq = await reserveRunEventSeq(tx, input.runId, input.runUpdate ?? {}); const [row] = await tx.insert(heartbeatRunEvents).values({ companyId: input.companyId, runId: input.runId, diff --git a/server/src/services/heartbeat/reattach-phase.ts b/server/src/services/heartbeat/reattach-phase.ts index 09f43e180f7..09ea191127e 100644 --- a/server/src/services/heartbeat/reattach-phase.ts +++ b/server/src/services/heartbeat/reattach-phase.ts @@ -1,5 +1,6 @@ -import { and, eq, inArray, isNotNull, isNull, ne, notLike, or, sql } from "drizzle-orm"; -import { dedicatedDbConnectionStats, heartbeatRuns, livenessDb, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import { and, eq, inArray, isNotNull, isNull, ne, notLike, or } from "drizzle-orm"; +import { dedicatedDbConnectionStats, heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import { readDatabaseSessionWaits } from "../database-session-waits.js"; import { logger } from "../../middleware/logger.js"; import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js"; @@ -21,6 +22,8 @@ import { type ReattachPhaseSummary, } from "./run-admission.js"; +export { readDatabaseSessionWaits, type DatabaseSessionWait } from "../database-session-waits.js"; + export const REATTACH_PHASE_DEFAULT_CAP_MS = 180_000; export const REATTACH_PHASE_MIN_CAP_MS = 5_000; export const REATTACH_PHASE_MAX_CAP_MS = 600_000; @@ -246,52 +249,6 @@ export function registerReattachPhaseWork( return current.worker; } -export type DatabaseSessionWait = { - pid: number; - applicationName: string | null; - state: string | null; - waitEventType: string | null; - waitEvent: string | null; - blockedBy: number[]; - transactionSeconds: number | null; - stateSeconds: number | null; - query: string | null; -}; - -export async function readDatabaseSessionWaits(db: Db): Promise { - return livenessDb(db).transaction(async tx => { - await tx.execute(sql`select set_config('statement_timeout', '5000', true)`); - const rows = await tx.execute>(sql` - with sessions as ( - select pid, application_name, state, wait_event_type, wait_event, pg_blocking_pids(pid) as blocked_by, - xact_start, state_change, query - from pg_stat_activity - where datname = current_database() and pid <> pg_backend_pid() and backend_type = 'client backend' - ), blockers as (select distinct unnest(blocked_by) as pid from sessions) - select pid, application_name, state, wait_event_type, wait_event, blocked_by, - extract(epoch from clock_timestamp() - xact_start)::int as transaction_seconds, - extract(epoch from clock_timestamp() - state_change)::int as state_seconds, - left(regexp_replace(regexp_replace(query, '''(?:[^'']|'''')*''', '?', 'g'), '\\m[0-9][0-9.]*\\M', '?', 'g'), 160) as query - from sessions - where cardinality(blocked_by) > 0 or pid in (select pid from blockers) - or state like 'idle in transaction%' - or (xact_start is not null and clock_timestamp() - xact_start > interval '30 seconds') - order by xact_start nulls last - limit 50`); - return rows.map(row => ({ - pid: Number(row.pid), - applicationName: (row.application_name as string | null) ?? null, - state: (row.state as string | null) ?? null, - waitEventType: (row.wait_event_type as string | null) ?? null, - waitEvent: (row.wait_event as string | null) ?? null, - blockedBy: Array.isArray(row.blocked_by) ? row.blocked_by.map(Number) : [], - transactionSeconds: row.transaction_seconds === null ? null : Number(row.transaction_seconds), - stateSeconds: row.state_seconds === null ? null : Number(row.state_seconds), - query: (row.query as string | null) ?? null, - })); - }); -} - function authenticatedButUnreached(summary: ReattachPhaseSummary): string[] { return summary.runs.filter(run => run.outcome === "outstanding" && run.authenticatedMs !== null).map(run => run.runId); } diff --git a/server/src/services/heartbeat/run-lifecycle.ts b/server/src/services/heartbeat/run-lifecycle.ts index 103dfe8f709..8e444d2c935 100644 --- a/server/src/services/heartbeat/run-lifecycle.ts +++ b/server/src/services/heartbeat/run-lifecycle.ts @@ -48,6 +48,7 @@ import { cancellationResultJson } from "../run-cancellation.js"; import { CONTROLLER_LEASE_LOST_FAILURE_CODE } from "../legacy-controller-lease.js"; import { isWaitingConversation } from "../agent-conversations.js"; import { PROCESS_IDENTITY_RECORDED } from "../native-local-process-stop.js"; +import { trackRunIntake } from "../run-intake-stall-watch.js"; import { hasAcknowledgedNativeReassignmentStopIntent, hasAcknowledgedNativeLifecycleStopIntent, @@ -625,7 +626,7 @@ export async function persistHeartbeatRunProcessMetadata( ); const startedAt = new Date(observedStartedAt ?? meta.startedAt); const processStartedAt = Number.isNaN(startedAt.getTime()) ? new Date() : startedAt; - return db.transaction(async tx => { + return trackRunIntake(runId, "process_identity", PROCESS_IDENTITY_RECORDED, () => db.transaction(async tx => { const [binding] = await tx .select({ companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId }) .from(heartbeatRuns) @@ -654,7 +655,7 @@ export async function persistHeartbeatRunProcessMetadata( }); return tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)) .then((rows) => rows[0] ?? null); - }); + })); } type SkillTestHeartbeatCompletion = { diff --git a/server/src/services/native-local-process-stop.ts b/server/src/services/native-local-process-stop.ts index d377b91632c..81085f7119e 100644 --- a/server/src/services/native-local-process-stop.ts +++ b/server/src/services/native-local-process-stop.ts @@ -11,6 +11,7 @@ import { NATIVE_SESSION_PROVIDER_EVENT_TYPES } from "./native-runtime/native-ses import os from "node:os"; import { currentProcessSpaceHash, UNKNOWN_PROCESS_SPACE } from "./controller-process-identity.js"; import { proveOwnerGone } from "./owner-proof.js"; +import { beginRunIntake } from "./run-intake-stall-watch.js"; import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED } from "./native-process-start-events.js"; export { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED }; @@ -593,6 +594,7 @@ export async function releaseOwnedSettledNativeRunProcessesAtShutdown(db: Db, op if (stopped.length === 0) return { candidates: candidates.length, stillRunning: candidates.length, released: 0 }; const recorded = JSON.stringify(stopped.map(run => ({ id: run.id, pid: run.processPid, pgid: run.processGroupId, started: run.processStartedAt, started_iso: run.processStartedAtIso }))); + const intake = stopped.map(run => beginRunIntake(run.id, "write", LOCAL_PROCESS_STOPPED)); const released = await db.transaction(async transaction => { const tx = transaction as unknown as Db; await tx.execute(sql.raw(`set local statement_timeout = ${SHUTDOWN_SETTLED_PROCESS_STATEMENT_TIMEOUT_MS}`)); @@ -628,7 +630,7 @@ export async function releaseOwnedSettledNativeRunProcessesAtShutdown(db: Db, op Array<{ companyId: string; runId: string; agentId: string }>; if (rows.length > 0) await notifyInstructionCopyStopEvidenceForRuns(tx, rows); return rows; - }); + }).finally(() => { for (const done of intake) done(); }); for (const run of released) pendingSettledProcessReleases.delete(run.runId); return { candidates: candidates.length, stillRunning: candidates.length - stopped.length, released: released.length }; } diff --git a/server/src/services/native-runtime/native-run-coordinator-store-intake.test.ts b/server/src/services/native-runtime/native-run-coordinator-store-intake.test.ts new file mode 100644 index 00000000000..43d1708ed7b --- /dev/null +++ b/server/src/services/native-runtime/native-run-coordinator-store-intake.test.ts @@ -0,0 +1,26 @@ +import type { Db } from "@paperclipai/db"; +import { describe, expect, it, vi } from "vitest"; + +const tracked = vi.hoisted(() => [] as Array<{ runId: string; stage: string; label: string }>); +vi.mock("../run-intake-stall-watch.js", async importOriginal => { + const actual = await importOriginal(); + return { + ...actual, + trackRunIntake: (runId: string, stage: string, label: string, work: () => Promise) => { + tracked.push({ runId, stage, label }); + return work(); + }, + }; +}); + +const { NativeRunCoordinatorStore } = await import("./native-run-coordinator-store.js"); + +describe("native run coordinator store intake tracking", () => { + it("registers each appended runner event with the run intake stall watch", async () => { + const store = new NativeRunCoordinatorStore({} as Db, + { runId: "run-coordinator" } as ConstructorParameters[1]); + await expect(store.appendEvent({ eventType: "session.started" } as Parameters[0])) + .rejects.toThrow("native_event_schema_invalid"); + expect(tracked).toEqual([{ runId: "run-coordinator", stage: "received", label: "session.started" }]); + }); +}); diff --git a/server/src/services/native-runtime/native-run-coordinator-store.ts b/server/src/services/native-runtime/native-run-coordinator-store.ts index 6bbff21cfdb..6bd417f04ae 100644 --- a/server/src/services/native-runtime/native-run-coordinator-store.ts +++ b/server/src/services/native-runtime/native-run-coordinator-store.ts @@ -10,6 +10,7 @@ import { createHash } from "node:crypto"; import { and, desc, eq, inArray, lt } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; +import { trackRunIntake } from "../run-intake-stall-watch.js"; import { heartbeatRunEvents, heartbeatRuns, @@ -207,6 +208,16 @@ export class NativeRunCoordinatorStore { readonly disposition: "committed" | "duplicate"; readonly cursor: number; readonly highestContiguousSourceSeq: number; + }> { + const eventType = (value as { eventType?: unknown }).eventType; + return trackRunIntake(this.#binding.runId, "received", typeof eventType === "string" ? eventType : "unknown", + () => this.#appendEvent(value)); + } + + async #appendEvent(value: PrpEvent): Promise<{ + readonly disposition: "committed" | "duplicate"; + readonly cursor: number; + readonly highestContiguousSourceSeq: number; }> { const validated = validatePrpEvent(value); if (!validated.ok) throw new Error("native_event_schema_invalid"); diff --git a/server/src/services/native-runtime/paperclip-control-plane-port.ts b/server/src/services/native-runtime/paperclip-control-plane-port.ts index 578b82bd420..d495d94668a 100644 --- a/server/src/services/native-runtime/paperclip-control-plane-port.ts +++ b/server/src/services/native-runtime/paperclip-control-plane-port.ts @@ -36,6 +36,7 @@ import { redactRunnerApiEvent, redactRunnerApiPayloads } from "./runner-api-log- import { createNativeControllerRecovery, NATIVE_CONTROLLER_RECOVERY_MAX_BYTES, type NativeControllerCompatibility } from "./native-controller-recovery.js"; import { eligibleNativeTaskSessionPublication, publishNativeTaskSessionInTransaction, NATIVE_TASK_SESSION_PUBLICATION_KEY, type NativeTaskSessionPublicationContext, type NativeTaskSessionPublicationIssue } from "./native-task-session-publication.js"; +import { trackRunIntake } from "../run-intake-stall-watch.js"; import { lockNativeRunnerCheckpointOwner, nativeRunnerCheckpointOwnerLock, validateNativeRunnerCheckpointOwner, inspectNativeRunnerUpgradeHeader, type NativeRunnerCheckpointOwner } from "./native-runner-checkpoint.js"; import { crashPoint } from "@paperclipai/shared/crash-points"; import { NATIVE_PROVIDER_DURABLE_EVENT_TYPES } from "./native-retired-backup-source.js"; @@ -399,6 +400,12 @@ export class PaperclipControlPlanePort implements ControlPlanePort { } async appendEvent(value: NativeRunEvent | PrpEvent, options?: { signal: AbortSignal; deferPostCommitObservers?: true; streaming?: true }) { + const eventType = (value as { eventType?: unknown }).eventType; + return trackRunIntake(this.#binding.runId, "received", typeof eventType === "string" ? eventType : "unknown", + () => this.#appendEvent(value, options)); + } + + async #appendEvent(value: NativeRunEvent | PrpEvent, options?: { signal: AbortSignal; deferPostCommitObservers?: true; streaming?: true }) { const persistenceStartedAt = Date.now(), persistenceClock = performance.now(); this.#assertActive(options); if (!isPrpEvent(value)) throw new Error("native_legacy_event_not_supported"); diff --git a/server/src/services/run-intake-stall-watch.test.ts b/server/src/services/run-intake-stall-watch.test.ts new file mode 100644 index 00000000000..18a28782998 --- /dev/null +++ b/server/src/services/run-intake-stall-watch.test.ts @@ -0,0 +1,245 @@ +import type { Db } from "@paperclipai/db"; +import { agents, closeLivenessDb, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns } from "@paperclipai/db"; +import { sql } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { logger } from "../middleware/logger.js"; +import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; +import type { DatabaseSessionWait } from "./database-session-waits.js"; +import { allocateHeartbeatRunEventSeq, appendHeartbeatRunEvent } from "./heartbeat-run-events.js"; +import { persistHeartbeatRunProcessMetadata } from "./heartbeat/run-lifecycle.js"; +import { beginRunIntake, installRunIntakeStallWatch, stopRunIntakeStallWatch } from "./run-intake-stall-watch.js"; + +const STALL_MESSAGE = "runner event intake stalled; pending run writes, database sessions and lock waits"; +const CLEARED_MESSAGE = "runner event intake stall cleared"; + +type StallReport = { + eventLoopLagMs: number; + operations: Array<{ runId: string; stage: string; label: string; pendingMs: number }>; + runs: Array<{ runId: string; pending: number; oldestPendingMs: number; stages: Record }>; + pool: { maxConnections: number | null; pendingIntake: number }; + sessions: Array<{ applicationName: string; state: string; sessions: number }>; + waits: DatabaseSessionWait[]; + databaseReadError?: string; +}; + +function stallReports(warn: { mock: { calls: unknown[][] } }): StallReport[] { + return warn.mock.calls.filter(call => call[1] === STALL_MESSAGE).map(call => call[0] as StallReport); +} + +afterEach(() => { + stopRunIntakeStallWatch(); + vi.restoreAllMocks(); + vi.useRealTimers(); +}); + +describe("run intake stall watch", () => { + it("does nothing until it is installed", () => { + const done = beginRunIntake("run", "received", "session.started"); + expect(done).toBeTypeOf("function"); + done(); + }); + + it("reports a stalled run once, throttles the next report, and logs when the stalled write finishes", async () => { + vi.useFakeTimers(); + const warn = vi.spyOn(logger, "warn").mockImplementation(() => logger); + const info = vi.spyOn(logger, "info").mockImplementation(() => logger); + const unreachable = { transaction: () => Promise.reject(new Error("database unreachable")), $client: { options: { max: 7 } } } as unknown as Db; + installRunIntakeStallWatch(unreachable, { stallMs: 1_000, now: () => Date.now() }); + const first = beginRunIntake("run-a", "process_identity", "native.process_identity_recorded"); + vi.advanceTimersByTime(100); + const queued = beginRunIntake("run-a", "received", "session.started"); + const healthy = beginRunIntake("run-b", "write", "item.completed"); + vi.advanceTimersByTime(50); + healthy(); + vi.advanceTimersByTime(50); + const other = beginRunIntake("run-c", "write", "turn.started"); + await vi.advanceTimersByTimeAsync(950); + let reports = stallReports(warn); + expect(reports).toHaveLength(1); + expect(reports[0]!.operations).toEqual([ + expect.objectContaining({ runId: "run-a", stage: "process_identity", label: "native.process_identity_recorded" }), + ]); + expect(reports[0]!.runs[0]).toMatchObject({ runId: "run-a", pending: 2, stages: { process_identity: 1, received: 1 } }); + expect(reports[0]!.pool).toMatchObject({ maxConnections: 7, pendingIntake: 3 }); + expect(reports[0]!.databaseReadError).toBe("database unreachable"); + + await vi.advanceTimersByTimeAsync(800); + expect(stallReports(warn)).toHaveLength(1); + await vi.advanceTimersByTimeAsync(100); + reports = stallReports(warn); + expect(reports).toHaveLength(2); + expect(reports[1]!.operations).toEqual([expect.objectContaining({ runId: "run-c", stage: "write" })]); + + queued(); + expect(info.mock.calls.filter(call => call[1] === CLEARED_MESSAGE)).toHaveLength(0); + first(); + other(); + expect(info.mock.calls.filter(call => call[1] === CLEARED_MESSAGE).map(call => call[0])).toEqual([ + expect.objectContaining({ runId: "run-a", stage: "process_identity" }), + expect.objectContaining({ runId: "run-c", stage: "write" }), + ]); + await vi.advanceTimersByTimeAsync(5_000); + expect(stallReports(warn)).toHaveLength(2); + expect(vi.getTimerCount()).toBe(0); + }); + + it("reports pending writes when the database read hangs, and stops without waiting for it", async () => { + vi.useFakeTimers(); + const warn = vi.spyOn(logger, "warn").mockImplementation(() => logger); + const hung = { transaction: () => new Promise(() => undefined), $client: { options: { max: 3 } } } as unknown as Db; + installRunIntakeStallWatch(hung, { stallMs: 1_000, reportTimeoutMs: 500, now: () => Date.now() }); + const done = beginRunIntake("run-hung", "write", "session.started"); + await vi.advanceTimersByTimeAsync(1_000); + expect(stallReports(warn)).toHaveLength(0); + await vi.advanceTimersByTimeAsync(500); + const [report] = stallReports(warn); + expect(report!.operations).toEqual([expect.objectContaining({ runId: "run-hung", stage: "write" })]); + expect(report!.databaseReadError).toBe("database session read timed out after 500 ms"); + done(); + const second = beginRunIntake("run-later", "write", "turn.started"); + await vi.advanceTimersByTimeAsync(1_200); + stopRunIntakeStallWatch(); + second(); + await vi.advanceTimersByTimeAsync(500); + expect(vi.getTimerCount()).toBe(0); + expect(stallReports(warn)).toHaveLength(2); + }); + + it("names every stalled run across reports when more runs stall than one report lists", async () => { + vi.useFakeTimers(); + const warn = vi.spyOn(logger, "warn").mockImplementation(() => logger); + const unreachable = { transaction: () => Promise.reject(new Error("database unreachable")), $client: {} } as unknown as Db; + installRunIntakeStallWatch(unreachable, { stallMs: 1_000, now: () => Date.now() }); + const runIds = Array.from({ length: 25 }, (_, index) => `run-${index}`); + const pending = runIds.map(runId => beginRunIntake(runId, "write", "item.completed")); + await vi.advanceTimersByTimeAsync(1_000); + expect(stallReports(warn)).toHaveLength(1); + expect(stallReports(warn)[0]!.operations).toHaveLength(20); + await vi.advanceTimersByTimeAsync(1_000); + const reports = stallReports(warn); + expect(reports).toHaveLength(2); + expect(reports[1]!.operations).toHaveLength(5); + expect(reports.flatMap(report => report.operations.map(operation => operation.runId)).sort()).toEqual([...runIds].sort()); + for (const done of pending) done(); + }); + + it("keeps no timer once no run write is pending", async () => { + vi.useFakeTimers(); + installRunIntakeStallWatch({} as Db, { stallMs: 1_000, now: () => Date.now() }); + expect(vi.getTimerCount()).toBe(0); + beginRunIntake("run", "write", "item.delta")(); + await vi.advanceTimersByTimeAsync(1_000); + expect(vi.getTimerCount()).toBe(0); + }); +}); + +const external = resolveExternalTestDatabaseUrl("PAPERCLIP_QUEUE_TEST_DATABASE_URL"); +const support = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport(); +if (!support.supported) console.warn(`Skipping run intake stall database tests: ${support.reason}`); +describe.skipIf(!support.supported)("run intake stall watch on a database", () => { + let database: Awaited> | undefined; + let db: Db; + beforeAll(async () => { + if (!external) database = await startEmbeddedPostgresTestDatabase("paperclip-run-intake-stall"); + db = createDb(external ?? database!.connectionString, { maxConnections: 4, applicationName: "paperclip:stall-test-boot" }); + }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); + afterAll(async () => { + if (db) await closeLivenessDb(db, 1); + await db?.$client.end(); + await database?.cleanup(); + }); + + async function seedRun(prefix: string) { + const [company] = await db.insert(companies).values({ name: `Intake ${prefix}`, issuePrefix: prefix }).returning(); + const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Runner agent", status: "running", + adapterType: "paperclip_runner", adapterConfig: {} }).returning(); + const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, status: "running", + runtimeMode: "native", invocationSource: "assignment", contextSnapshot: {} }).returning(); + return run!; + } + + it("names the idle transaction holding a run row when that run's process identity write waits behind it", async () => { + const run = await seedRun("RIS"); + const warn = vi.spyOn(logger, "warn"); + const info = vi.spyOn(logger, "info"); + installRunIntakeStallWatch(db, { stallMs: 300 }); + let release!: () => void; + const held = new Promise(resolve => { release = resolve; }); + let locked!: (pid: number) => void; + const holderPid = new Promise(resolve => { locked = resolve; }); + const holder = db.transaction(async tx => { + const [backend] = await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`); + await tx.execute(sql.raw(`select id from heartbeat_runs where id = '${run.id}' and next_event_seq >= 0 for update`)); + locked(Number(backend!.pid)); + await held; + }); + const holderBackend = await holderPid; + const identity = persistHeartbeatRunProcessMetadata(db, run.id, + { pid: 563, processGroupId: null, startedAt: new Date().toISOString(), targetKind: "remote" }); + try { + await vi.waitFor(() => expect(stallReports(warn)).toHaveLength(1), { timeout: 5_000, interval: 20 }); + const [report] = stallReports(warn); + expect(report!.operations).toEqual([ + expect.objectContaining({ runId: run.id, stage: "process_identity", label: "native.process_identity_recorded" }), + ]); + expect(report!.runs[0]).toMatchObject({ runId: run.id, stages: { process_identity: 1, write: 1 } }); + expect(report!.pool.maxConnections).toBe(4); + expect(report!.sessions).toEqual(expect.arrayContaining([ + expect.objectContaining({ applicationName: "paperclip:stall-test-boot", state: "idle in transaction" }), + ])); + expect(report!.waits).toEqual(expect.arrayContaining([ + expect.objectContaining({ pid: holderBackend, state: "idle in transaction", applicationName: "paperclip:stall-test-boot" }), + expect.objectContaining({ waitEventType: "Lock", blockedBy: [holderBackend] }), + ])); + expect(JSON.stringify(report!.waits)).not.toContain(run.id); + expect(report!.waits.find(wait => wait.pid === holderBackend)?.query).toContain("where id = ? and next_event_seq >= ? for update"); + } finally { + release(); + await holder; + } + await expect(identity).resolves.toMatchObject({ processPid: 563 }); + expect(info.mock.calls.filter(call => call[1] === CLEARED_MESSAGE).map(call => call[0])).toEqual([ + expect.objectContaining({ runId: run.id, stage: "process_identity" }), + ]); + }); + + it("reports a direct event sequence reservation that waits behind a held run row", async () => { + const run = await seedRun("RIQ"); + const warn = vi.spyOn(logger, "warn"); + installRunIntakeStallWatch(db, { stallMs: 300 }); + let release!: () => void; + const held = new Promise(resolve => { release = resolve; }); + let locked!: () => void; + const isLocked = new Promise(resolve => { locked = resolve; }); + const holder = db.transaction(async tx => { + await tx.execute(sql.raw(`select id from heartbeat_runs where id = '${run.id}' for update`)); + locked(); + await held; + }); + await isLocked; + const reserved = allocateHeartbeatRunEventSeq(db, run.id); + try { + await vi.waitFor(() => expect(stallReports(warn)).toHaveLength(1), { timeout: 5_000, interval: 20 }); + expect(stallReports(warn)[0]!.operations).toEqual([ + expect.objectContaining({ runId: run.id, stage: "write", label: "event_sequence" }), + ]); + } finally { + release(); + await holder; + } + await expect(reserved).resolves.toBeTypeOf("number"); + }); + + it("stays silent while run event writes keep committing", async () => { + const run = await seedRun("RIH"); + const warn = vi.spyOn(logger, "warn"); + installRunIntakeStallWatch(db, { stallMs: 200 }); + for (let index = 0; index < 5; index += 1) { + await appendHeartbeatRunEvent(db, { companyId: run.companyId, runId: run.id, agentId: run.agentId, + eventType: "lifecycle", stream: "system", level: "info", message: `step ${index}` }); + } + await new Promise(resolve => setTimeout(resolve, 400)); + expect(stallReports(warn)).toHaveLength(0); + }); +}); diff --git a/server/src/services/run-intake-stall-watch.ts b/server/src/services/run-intake-stall-watch.ts new file mode 100644 index 00000000000..36d5ff2f80c --- /dev/null +++ b/server/src/services/run-intake-stall-watch.ts @@ -0,0 +1,156 @@ +import { dedicatedDbConnectionStats, type Db } from "@paperclipai/db"; +import { logger } from "../middleware/logger.js"; +import { readDatabaseSessionReport } from "./database-session-waits.js"; + +export const RUN_INTAKE_STALL_MS = 30_000; +export const RUN_INTAKE_REPORT_TIMEOUT_MS = 10_000; +const REPORTED_OPERATIONS_MAX = 20; +const REPORTED_RUNS_MAX = 20; + +export type RunIntakeStage = "received" | "write" | "process_identity"; + +type Operation = { runId: string; stage: RunIntakeStage; label: string; startedAtMs: number; stalled: boolean; listed: boolean }; + +type Watch = { + db: Db; + stallMs: number; + reportTimeoutMs: number; + now: () => number; + operations: Map; + nextToken: number; + timer: ReturnType | null; + dueAtMs: number; + lastReportAtMs: number | null; + report: Promise | null; +}; + +let watch: Watch | null = null; + +const settled = () => undefined; + +export function installRunIntakeStallWatch(db: Db, + options: { stallMs?: number; reportTimeoutMs?: number; now?: () => number } = {}): void { + if (watch?.timer) clearTimeout(watch.timer); + watch = { db, stallMs: options.stallMs ?? RUN_INTAKE_STALL_MS, reportTimeoutMs: options.reportTimeoutMs ?? RUN_INTAKE_REPORT_TIMEOUT_MS, + now: options.now ?? Date.now, operations: new Map(), + nextToken: 0, timer: null, dueAtMs: 0, lastReportAtMs: null, report: null }; +} + +export function stopRunIntakeStallWatch(): void { + const current = watch; + watch = null; + if (!current) return; + if (current.timer) clearTimeout(current.timer); + current.timer = null; +} + +export function beginRunIntake(runId: string, stage: RunIntakeStage, label: string): () => void { + const current = watch; + if (!current) return settled; + const token = current.nextToken++; + const operation: Operation = { runId, stage, label, startedAtMs: current.now(), stalled: false, listed: false }; + current.operations.set(token, operation); + if (!current.timer && !current.report) arm(current, operation.startedAtMs + current.stallMs); + return () => { + if (!current.operations.delete(token) || !operation.listed) return; + logger.info({ runId, stage, label, pendingMs: current.now() - operation.startedAtMs }, + "runner event intake stall cleared"); + }; +} + +export async function trackRunIntake(runId: string, stage: RunIntakeStage, label: string, work: () => Promise): Promise { + const done = beginRunIntake(runId, stage, label); + try { return await work(); } + finally { done(); } +} + +function arm(current: Watch, dueAtMs: number) { + const now = current.now(); + current.dueAtMs = Math.max(dueAtMs, now); + current.timer = setTimeout(() => check(current), current.dueAtMs - now); + current.timer.unref?.(); +} + +function nextDueAtMs(current: Watch): number | null { + for (const operation of current.operations.values()) { + if (!operation.stalled) return operation.startedAtMs + current.stallMs; + } + return null; +} + +function check(current: Watch) { + current.timer = null; + if (watch !== current) return; + const now = current.now(); + const eventLoopLagMs = Math.max(0, now - current.dueAtMs); + if (current.lastReportAtMs !== null && now - current.lastReportAtMs < current.stallMs) { + arm(current, current.lastReportAtMs + current.stallMs); + return; + } + const stalledRuns = new Set(); + for (const operation of current.operations.values()) if (operation.stalled) stalledRuns.add(operation.runId); + const newlyStalled: Operation[] = []; + for (const operation of current.operations.values()) { + if (operation.stalled) continue; + if (now - operation.startedAtMs < current.stallMs) break; + if (stalledRuns.has(operation.runId)) { operation.stalled = true; continue; } + if (newlyStalled.length >= REPORTED_OPERATIONS_MAX) break; + operation.stalled = true; + stalledRuns.add(operation.runId); + newlyStalled.push(operation); + } + if (newlyStalled.length === 0) { + const due = nextDueAtMs(current); + if (due !== null) arm(current, due); + return; + } + current.lastReportAtMs = now; + const report: Promise = reportStall(current, newlyStalled, eventLoopLagMs) + .catch(error => logger.warn({ err: error }, "runner event intake stall report failed")) + .finally(() => { + if (current.report === report) current.report = null; + if (watch !== current || current.timer) return; + const due = nextDueAtMs(current); + if (due !== null) arm(current, due); + }); + current.report = report; +} + +function pendingByRun(current: Watch, now: number) { + const runs = new Map> }>(); + for (const operation of current.operations.values()) { + let run = runs.get(operation.runId); + if (!run) { + run = { runId: operation.runId, pending: 0, oldestPendingMs: now - operation.startedAtMs, oldestLabel: operation.label, stages: {} }; + runs.set(operation.runId, run); + } + run.pending += 1; + run.stages[operation.stage] = (run.stages[operation.stage] ?? 0) + 1; + } + return [...runs.values()].sort((left, right) => right.oldestPendingMs - left.oldestPendingMs).slice(0, REPORTED_RUNS_MAX); +} + +async function reportStall(current: Watch, stalled: Operation[], eventLoopLagMs: number) { + const now = current.now(); + for (const operation of stalled) operation.listed = true; + const operations = stalled.map(operation => ({ runId: operation.runId, stage: operation.stage, label: operation.label, + pendingMs: now - operation.startedAtMs })); + const client = current.db.$client as { options?: { max?: unknown } } | undefined; + const pool = { maxConnections: typeof client?.options?.max === "number" ? client.options.max : null, + pendingIntake: current.operations.size, dedicated: dedicatedDbConnectionStats(current.db) }; + const runs = pendingByRun(current, now); + let expire: ReturnType | undefined; + const expired = new Promise((_resolve, reject) => { + expire = setTimeout(() => reject(new Error(`database session read timed out after ${current.reportTimeoutMs} ms`)), + current.reportTimeoutMs); + expire.unref?.(); + }); + const database = await Promise.race([readDatabaseSessionReport(current.db), expired]).then( + report => ({ ...report, error: null as string | null }), + error => ({ sessions: [], waits: [], error: error instanceof Error ? error.message.slice(0, 160) : "unknown" }), + ).finally(() => clearTimeout(expire)); + logger.warn({ stallThresholdMs: current.stallMs, eventLoopLagMs, operations, runs, pool, sessions: database.sessions, + waits: database.waits, ...(database.error ? { databaseReadError: database.error } : {}) }, + "runner event intake stalled; pending run writes, database sessions and lock waits"); +}