fix(server): name the blocking database sessions whenever a run's event writes stall (#21956)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server stores the events that a runner sends for each run, and
it records the identity of the runner process
> - Sometimes the server stops storing a run's events for minutes, and
the server log stays quiet
> - In two restart-chaos cases, the identity write of the run started
and then did not commit for about 197 s and 480 s
> - The existing database report only fires when the reattach phase ends
at its cap, so it did not fire for the cold-start case
> - This pull request logs the pending writes and the blocking database
sessions whenever any run write stays pending for 30 s
> - The benefit is that the next stall names the session and the query
that hold the run, so the fix can target one proven holder

## Linked Issues or Issue Description

Refs #21946

**What happened?**
Restart-chaos waves on master showed two stalls in event storage.
1. Cold start of an OpenCode run, with no restart. The runner emitted
`session.started` at 22:13:23.7Z. The server stored it at 22:16:40.97Z,
about 197 s later. The second `native.process_identity_recorded` write
of the run started at 22:13:23.706Z, and `runner.turn.submit` took
197270 ms. A second wave showed the same shape (about 191 s). The server
log had nothing in that window.
2. After a hard kill, every API GET timed out from 22:35:24Z to
22:43:23Z, server CPU was 0%, and no run events were stored. The process
identity transaction of the run began at 22:35:22.6Z and committed at
about 22:43:23Z.

**Expected behavior**
When the server cannot store a run's events, the server log names the
cause: which run writes are pending, and which database sessions hold or
wait for locks.

**Steps to reproduce**
1. Start a transaction that locks a `heartbeat_runs` row (`for update`)
and stays idle.
2. Record a process identity for that run
(`persistHeartbeatRunProcessMetadata`).
3. Before this change, nothing is logged while the write waits. After
this change, a warning names the idle holder and the waiting statement
after 30 s. The new database test does this with a 300 ms threshold.

**Relevant logs or output**
Example warning from the new test:
`runner event intake stalled; pending run writes, database sessions and
lock waits` with `operations: [{stage: "process_identity", pendingMs:
301}]` and `waits: [{state: "idle in transaction", query: "select id
from heartbeat_runs where id = ? ... for update"}, {waitEventType:
"Lock", blockedBy: [<holder pid>]}]`.

## What Changed

- New `server/src/services/run-intake-stall-watch.ts`. It keeps an
in-memory list of pending run operations: event receipts
(`PaperclipControlPlanePort.appendEvent`), event writes
(`appendHeartbeatRunEvent`, `appendHeartbeatRunEventBatch`), and process
identity writes (`persistHeartbeatRunProcessMetadata`).
- When an operation stays pending for 30 s, the watch logs one warning
per stalled run. The warning has: the pending operations and their age,
the pending count and oldest age per run, the pool size, the database
sessions per application name and state, the lock waits and holders
(`pg_blocking_pids`, query without literals, cut to 160 characters), and
the event-loop lag. Reports are at most one per 30 s. A second line logs
when a reported operation finishes.
- The watch arms a timer only while an operation is pending. It reads
the database only when it reports, through the private liveness pool
with a 5 s statement timeout.
- The server names its main pool `paperclip:<boot id>` unless
`DATABASE_APPLICATION_NAME` is set. This shows which server process owns
each session after a restart.
- `readDatabaseSessionWaits` moves from the reattach phase to
`server/src/services/database-session-waits.ts`. The reattach phase
re-exports it.
- `@paperclipai/db` exports `databaseClientOptionsFromEnv` and
`DatabaseClientOptions`.
- `doc/DATABASE.md` describes the new log lines and the pool name.
- I did not add `idle_in_transaction_session_timeout` to the main pool.
Some code holds a transaction open on purpose, and a blind timeout could
abort valid work. The new report names idle-in-transaction holders
instead.
- These are server log lines only. This change does not touch Telemetry
or OpenTelemetry.

## Verification

- `cd server && npx vitest run
src/services/run-intake-stall-watch.test.ts` (5 tests). The database
test holds a run row in an idle transaction, starts a process identity
write, and checks that the warning names the holder pid, its query, the
waiting statement, and the pool name. It is red without the tracking in
`run-lifecycle.ts` and `heartbeat-run-events.ts` (`expected [] to have a
length of 1`), and green with it.
- Unit tests with fake timers check one report per stalled run, the 30 s
throttle, the clear line, and that no timer stays armed when nothing is
pending.
- `cd server && npx vitest run
src/services/controller-process-identity.test.ts` checks the pool name
and the `DATABASE_APPLICATION_NAME` override.
- `npx tsc --noEmit` passes for `server` and `packages/db`.
- `pnpm verify` ran locally on the affected tests.

## Risks

- Low risk. Each run event adds one map insert and one map delete. No
database work happens while writes finish within 30 s.
- The main pool `application_name` changes from `paperclip` to
`paperclip:<boot id>`. A dashboard that filters on exactly `paperclip`
must use a prefix match, or the operator can set
`DATABASE_APPLICATION_NAME`.
- A report reads `pg_stat_activity` on the liveness pool. A report
during a full database stall can fail; the warning then has
`databaseReadError` and still lists the pending operations.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-16 22:15:40 -06:00
1 parent 7b8df907d3
commit 3afb2c215b
16 files changed
+602 -59

No files matched your search

+13 -1
View File
@@ -134,9 +134,21 @@ DATABASE_POOL_MAX=25 # connection pool size; default: 10
DATABASE_IDLE_TIMEOUT_SECONDS=60 # close idle pooled connections; default: 60 (0 = keep open)
DATABASE_CONNECT_TIMEOUT_SECONDS=10 # default: 30
DATABASE_MAX_LIFETIME_SECONDS=1800 # recycle a pooled connection after this long; default: 30-60 min (random)
DATABASE_APPLICATION_NAME=paperclip # application_name in pg_stat_activity; default: paperclip
DATABASE_APPLICATION_NAME=paperclip # application_name in pg_stat_activity; default for the server's main pool: paperclip:<boot id>
```
The server names its main pool `paperclip:<boot id>`, so after a restart the
sessions of each server process can be told apart in `pg_stat_activity`. When a
run's event, process identity, or event-write operation stays unfinished for
30 seconds, the server logs one warning per stalled run ("runner event intake
stalled"). The warning lists the pending operations of each run, the pool size,
the database sessions per application name and state, and the sessions that
wait for a lock or hold one (`pg_blocking_pids`, query text without literals,
cut to 160 characters). It also logs the event-loop lag. A second line
("runner event intake stall cleared") records when the operation finished. The
check runs only while an operation is pending, and it reads `pg_stat_activity`
only when it reports a stall.
`withDedicatedDbConnection` gives an action exclusive use of one session outside
the main pool (application name `paperclip-workspace-finalization-lock`). Each
`createDb` owner keeps released single-connection clients idle for reuse: always
+2 -2
View File
@@ -73,7 +73,7 @@ statements longer than 8 KB. If a prepared statement fails because the pooler's
stale after a schema change or was lost, Paperclip stops preparing that
statement and retries it once unprepared.
Related optional client tuning: `DATABASE_POOL_MAX`, `DATABASE_IDLE_TIMEOUT_SECONDS`, `DATABASE_CONNECT_TIMEOUT_SECONDS`, `DATABASE_MAX_LIFETIME_SECONDS`, `DATABASE_APPLICATION_NAME`. Driver defaults apply when unset, except that idle pooled connections close after 60 seconds (`DATABASE_IDLE_TIMEOUT_SECONDS=0` keeps them open) and the pool reports `application_name=paperclip`. See [Connection pool settings](#connection-pool-settings).
Related optional client tuning: `DATABASE_POOL_MAX`, `DATABASE_IDLE_TIMEOUT_SECONDS`, `DATABASE_CONNECT_TIMEOUT_SECONDS`, `DATABASE_MAX_LIFETIME_SECONDS`, `DATABASE_APPLICATION_NAME`. Driver defaults apply when unset, except that idle pooled connections close after 60 seconds (`DATABASE_IDLE_TIMEOUT_SECONDS=0` keeps them open) and the server's main pool reports `application_name=paperclip:<boot id>` (other pools use names that start with `paperclip`). See [Connection pool settings](#connection-pool-settings).
## Connection Pool Settings
@@ -85,7 +85,7 @@ The server opens one postgres.js pool for its own queries (and a second one when
| `DATABASE_IDLE_TIMEOUT_SECONDS` | `60` | Close a pooled connection after this much idle time. `0` keeps idle connections open forever (the driver default). |
| `DATABASE_CONNECT_TIMEOUT_SECONDS` | `30` (driver) | Give up on a connection attempt after this long. |
| `DATABASE_MAX_LIFETIME_SECONDS` | 30–60 min, randomized (driver) | Recycle a pooled connection once it is this old. |
| `DATABASE_APPLICATION_NAME` | `paperclip` | Value of `application_name` in `pg_stat_activity`, so you can find Paperclip's backends: `SELECT * FROM pg_stat_activity WHERE application_name = 'paperclip';` |
| `DATABASE_APPLICATION_NAME` | `paperclip:<boot id>` (server main pool) | Value of `application_name` in `pg_stat_activity`. The boot id tells the sessions of each server process apart after a restart. Find Paperclip's backends with `SELECT * FROM pg_stat_activity WHERE application_name LIKE 'paperclip%';` |
| `DATABASE_DEDICATED_CONNECTION_MAX` | `24` | Dedicated lock connections in use at once per process before new callers wait. Nested callers do not wait. |
| `DATABASE_DEDICATED_CONNECTION_WAIT_MS` | `30000` | After this wait, a queued caller gets a dedicated connection over the limit. |
| `DATABASE_PREPARED_STATEMENTS` | Unset | `true` opts parameterized ORM queries into preparation. `false` disables preparation. Use `false` when the pooler does not support named statements. Unset preserves existing behavior. |
+2
View File
@@ -1,5 +1,7 @@
export {
createDb,
databaseClientOptionsFromEnv,
type DatabaseClientOptions,
withDedicatedDbConnection,
closeDedicatedDbConnections,
DEDICATED_CONNECTION_IDLE_MAX,
+6 -2
View File
@@ -136,6 +136,8 @@ import {
} from "@paperclipai/db";
import detectPort from "detect-port";
import { createApp } from "./app.js";
import { serverDatabaseClientOptions } from "./services/controller-process-identity.js";
import { installRunIntakeStallWatch, stopRunIntakeStallWatch } from "./services/run-intake-stall-watch.js";
import { drainHttpRequestWork } from "./middleware/idle-admission.js";
import { initializeCloudRuntimeConfig } from "./services/cloud-runtime-config.js";
import { loadCloudBootstrapConfig, loadConfig, type Config } from "./config.js";
@@ -537,7 +539,7 @@ async function startServerWithDatabaseTeardown(
const migrationUrl = startupConfig.databaseMigrationUrl ?? startupConfig.databaseUrl;
migrationSummary = await ensureMigrations(migrationUrl, "PostgreSQL");
db = createDb(startupConfig.databaseUrl);
db = createDb(startupConfig.databaseUrl, serverDatabaseClientOptions());
pluginMigrationDb = startupConfig.databaseMigrationUrl ? createDb(startupConfig.databaseMigrationUrl) : db;
logger.info("Using external PostgreSQL via DATABASE_URL/config");
activeDatabaseConnectionString = startupConfig.databaseUrl;
@@ -744,7 +746,7 @@ async function startServerWithDatabaseTeardown(
autoApply: shouldAutoApplyFirstRunMigrations,
});
db = createDb(embeddedConnectionString);
db = createDb(embeddedConnectionString, serverDatabaseClientOptions());
pluginMigrationDb = db;
logger.info("Embedded PostgreSQL ready");
activeDatabaseConnectionString = embeddedConnectionString;
@@ -757,10 +759,12 @@ async function startServerWithDatabaseTeardown(
// by the fail-loud startup path, so no exit leaves pooled backends behind.
const closeDatabaseClients = async () => {
setSentryRunLinkResolver(null);
stopRunIntakeStallWatch();
const clients = pluginMigrationDb === db ? [db] : [db, pluginMigrationDb];
await Promise.all([...clients.map((client) => endDatabaseClient(client, 5)), closeLivenessDb(db, 5), closeCheckpointDb(db, 5), closeFileStorageDb(db, 5), closeDedicatedDbConnections(db, 5)]);
};
startupDatabase.close = closeDatabaseClients;
installRunIntakeStallWatch(db);
startupTimeline.mark("database");
setSentryRunLinkResolver(createSentryRunLinkResolver(db));
await migrateCloudConfiguration(db as any);
@@ -1,11 +1,23 @@
import { spawn } from "node:child_process";
import { once } from "node:events";
import { afterEach, describe, expect, it, vi } from "vitest";
import { processExists, processStartsDefinitelyDiffer } from "./controller-process-identity.js";
import { currentControllerBootId, processExists, processStartsDefinitelyDiffer, serverDatabaseClientOptions } from "./controller-process-identity.js";
import { isPidAlive, isProcessGroupAlive } from "./local-service-supervisor.js";
import { readProcessStartedAt } from "./hot-restart.js";
import { recoveryProcessMayBeAlive } from "./native-runtime/native-session-executor.js";
describe("server database client options", () => {
it("names the main pool after this boot so its database sessions can be told apart from an earlier server's", () => {
const options = serverDatabaseClientOptions({ DATABASE_POOL_MAX: "12" });
expect(options).toEqual({ applicationName: `paperclip:${currentControllerBootId()}`, maxConnections: 12 });
expect(options.applicationName!.length).toBeLessThanOrEqual(63);
});
it("keeps an operator's DATABASE_APPLICATION_NAME", () => {
expect(serverDatabaseClientOptions({ DATABASE_APPLICATION_NAME: "paperclip-web" }).applicationName).toBe("paperclip-web");
});
});
describe("shared process identity proof", () => {
afterEach(() => vi.restoreAllMocks());
@@ -2,6 +2,7 @@ import { createHash, randomUUID } from "node:crypto";
import { execFileSync } from "node:child_process";
import { readFileSync, readlinkSync } from "node:fs";
import { hostname } from "node:os";
import { databaseClientOptionsFromEnv, type DatabaseClientOptions } from "@paperclipai/db";
import { readProcessStartedAt } from "./hot-restart.js";
import { currentWorkspaceWorker, type WorkspaceWorker } from "./native-runtime/native-workspace-platform-proof.js";
@@ -127,6 +128,10 @@ export function currentControllerBootId(): string {
return bootId;
}
export function serverDatabaseClientOptions(env: NodeJS.ProcessEnv = process.env): DatabaseClientOptions {
return { applicationName: `paperclip:${bootId}`, ...databaseClientOptionsFromEnv(env) };
}
const stampListeners = new Set<() => void>();
/** Called (synchronously, without I/O) whenever this process stamps ownership of a record. */
export function onControllerStamp(listener: () => void): () => void {
@@ -0,0 +1,77 @@
import { sql } from "drizzle-orm";
import { livenessDb, type Db } from "@paperclipai/db";
export type DatabaseSessionWait = {
pid: number;
applicationName: string | null;
state: string | null;
waitEventType: string | null;
waitEvent: string | null;
blockedBy: number[];
transactionSeconds: number | null;
stateSeconds: number | null;
query: string | null;
};
export type DatabaseSessionCount = { applicationName: string; state: string; sessions: number };
const SESSION_WAITS_SQL = sql`
with sessions as (
select pid, application_name, state, wait_event_type, wait_event, pg_blocking_pids(pid) as blocked_by,
xact_start, state_change, query
from pg_stat_activity
where datname = current_database() and pid <> pg_backend_pid() and backend_type = 'client backend'
), blockers as (select distinct unnest(blocked_by) as pid from sessions)
select pid, application_name, state, wait_event_type, wait_event, blocked_by,
extract(epoch from clock_timestamp() - xact_start)::int as transaction_seconds,
extract(epoch from clock_timestamp() - state_change)::int as state_seconds,
left(regexp_replace(regexp_replace(query, '''(?:[^'']|'''')*''', '?', 'g'), '\\m[0-9][0-9.]*\\M', '?', 'g'), 160) as query
from sessions
where cardinality(blocked_by) > 0 or pid in (select pid from blockers)
or state like 'idle in transaction%'
or (xact_start is not null and clock_timestamp() - xact_start > interval '30 seconds')
order by xact_start nulls last
limit 50`;
function sessionWait(row: Record<string, unknown>): DatabaseSessionWait {
return {
pid: Number(row.pid),
applicationName: (row.application_name as string | null) ?? null,
state: (row.state as string | null) ?? null,
waitEventType: (row.wait_event_type as string | null) ?? null,
waitEvent: (row.wait_event as string | null) ?? null,
blockedBy: Array.isArray(row.blocked_by) ? row.blocked_by.map(Number) : [],
transactionSeconds: row.transaction_seconds === null ? null : Number(row.transaction_seconds),
stateSeconds: row.state_seconds === null ? null : Number(row.state_seconds),
query: (row.query as string | null) ?? null,
};
}
export async function readDatabaseSessionWaits(db: Db): Promise<DatabaseSessionWait[]> {
return livenessDb(db).transaction(async tx => {
await tx.execute(sql`select set_config('statement_timeout', '5000', true)`);
const rows = await tx.execute<Record<string, unknown>>(SESSION_WAITS_SQL);
return rows.map(sessionWait);
});
}
export async function readDatabaseSessionReport(db: Db, statementTimeoutMs = 5_000): Promise<{
sessions: DatabaseSessionCount[]; waits: DatabaseSessionWait[];
}> {
return livenessDb(db).transaction(async tx => {
await tx.execute(sql`select set_config('statement_timeout', ${String(statementTimeoutMs)}, true)`);
const counts = await tx.execute<Record<string, unknown>>(sql`
select coalesce(application_name, '') as application_name, coalesce(state, '') as state, count(*)::int as sessions
from pg_stat_activity
where datname = current_database() and pid <> pg_backend_pid() and backend_type = 'client backend'
group by 1, 2
order by 3 desc
limit 40`);
const rows = await tx.execute<Record<string, unknown>>(SESSION_WAITS_SQL);
return {
sessions: counts.map(row => ({ applicationName: String(row.application_name), state: String(row.state),
sessions: Number(row.sessions) })),
waits: rows.map(sessionWait),
};
});
}
+28 -2
View File
@@ -13,6 +13,7 @@ import { heartbeatRunEvents, heartbeatRuns, primeDatabaseWorkTransactionId } fro
import { DATABASE_WORK_CHANNEL } from "@paperclipai/db/work-signals";
import { nativeSha256 } from "./native-runtime/canonical.js";
import { isUniqueViolation } from "../db-errors.js";
import { trackRunIntake } from "./run-intake-stall-watch.js";
export const RETRY_EXHAUSTION_MESSAGE_PREFIX = "Bounded retry exhausted";
@@ -88,6 +89,14 @@ export async function allocateHeartbeatRunEventSeq(
db: Db,
runId: string,
runUpdate: HeartbeatRunEventRunUpdate = {},
): Promise<number> {
return trackRunIntake(runId, "write", "event_sequence", () => reserveRunEventSeq(db, runId, runUpdate));
}
async function reserveRunEventSeq(
db: Db,
runId: string,
runUpdate: HeartbeatRunEventRunUpdate,
): Promise<number> {
const [updated] = await db
.update(heartbeatRuns)
@@ -118,6 +127,13 @@ function runEventWakes(input: AppendHeartbeatRunEventInput): RunEventWakes {
export async function appendHeartbeatRunEvent(
db: Db,
input: AppendHeartbeatRunEventInput,
): Promise<AppendHeartbeatRunEventResult> {
return trackRunIntake(input.runId, "write", input.eventType, () => appendRunEvent(db, input));
}
async function appendRunEvent(
db: Db,
input: AppendHeartbeatRunEventInput,
): Promise<AppendHeartbeatRunEventResult> {
// Deduplicated receipts return without allocating, so they cannot carry run changes.
if (input.runUpdate && (input.nativeSource || input.retryExhaustion)) throw new Error("heartbeat_run_event_run_update_unsupported");
@@ -178,13 +194,23 @@ export async function appendHeartbeatRunEventBatch(
): Promise<AppendHeartbeatRunEventResult[]> {
const first = inputs[0];
if (!first) return [];
return trackRunIntake(first.runId, "write", `${first.eventType} x${inputs.length}`,
() => appendRunEventBatch(db, inputs, assertWritable));
}
async function appendRunEventBatch(
db: Db,
inputs: readonly AppendHeartbeatRunEventInput[],
assertWritable?: () => void,
): Promise<AppendHeartbeatRunEventResult[]> {
const first = inputs[0]!;
const source = first.nativeSource;
if (!source || inputs.some(input => !input.nativeSource || input.runUpdate || input.retryExhaustion
|| input.companyId !== first.companyId || input.runId !== first.runId || input.agentId !== first.agentId
|| input.nativeSource.sourceInstanceId !== source.sourceInstanceId)) {
throw new Error("heartbeat_run_event_batch_unsupported");
}
if (inputs.length === 1) return [await appendHeartbeatRunEvent(db, assertWritable ? { ...first, assertWritable } : first)];
if (inputs.length === 1) return [await appendRunEvent(db, assertWritable ? { ...first, assertWritable } : first)];
const wakes = inputs.map(runEventWakes);
if (!wakes.some(wake => Object.values(wake).some(Boolean)) && !is(db, PgTransaction)) {
assertWritable?.();
@@ -596,7 +622,7 @@ async function appendLockedEvent(
await notifyNativeResponsePresentation(tx, { companyId: run.companyId, runId: input.runId });
}
if (wakes.retention) await notifyAttentionRetentionWork(tx, { companyId: input.companyId, sourceKind: "failed_run" });
const seq = await allocateHeartbeatRunEventSeq(tx, input.runId, input.runUpdate);
const seq = await reserveRunEventSeq(tx, input.runId, input.runUpdate ?? {});
const [row] = await tx.insert(heartbeatRunEvents).values({
companyId: input.companyId,
runId: input.runId,
@@ -1,5 +1,6 @@
import { and, eq, inArray, isNotNull, isNull, ne, notLike, or, sql } from "drizzle-orm";
import { dedicatedDbConnectionStats, heartbeatRuns, livenessDb, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { and, eq, inArray, isNotNull, isNull, ne, notLike, or } from "drizzle-orm";
import { dedicatedDbConnectionStats, heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { readDatabaseSessionWaits } from "../database-session-waits.js";
import { logger } from "../../middleware/logger.js";
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js";
@@ -21,6 +22,8 @@ import {
type ReattachPhaseSummary,
} from "./run-admission.js";
export { readDatabaseSessionWaits, type DatabaseSessionWait } from "../database-session-waits.js";
export const REATTACH_PHASE_DEFAULT_CAP_MS = 180_000;
export const REATTACH_PHASE_MIN_CAP_MS = 5_000;
export const REATTACH_PHASE_MAX_CAP_MS = 600_000;
@@ -246,52 +249,6 @@ export function registerReattachPhaseWork(
return current.worker;
}
export type DatabaseSessionWait = {
pid: number;
applicationName: string | null;
state: string | null;
waitEventType: string | null;
waitEvent: string | null;
blockedBy: number[];
transactionSeconds: number | null;
stateSeconds: number | null;
query: string | null;
};
export async function readDatabaseSessionWaits(db: Db): Promise<DatabaseSessionWait[]> {
return livenessDb(db).transaction(async tx => {
await tx.execute(sql`select set_config('statement_timeout', '5000', true)`);
const rows = await tx.execute<Record<string, unknown>>(sql`
with sessions as (
select pid, application_name, state, wait_event_type, wait_event, pg_blocking_pids(pid) as blocked_by,
xact_start, state_change, query
from pg_stat_activity
where datname = current_database() and pid <> pg_backend_pid() and backend_type = 'client backend'
), blockers as (select distinct unnest(blocked_by) as pid from sessions)
select pid, application_name, state, wait_event_type, wait_event, blocked_by,
extract(epoch from clock_timestamp() - xact_start)::int as transaction_seconds,
extract(epoch from clock_timestamp() - state_change)::int as state_seconds,
left(regexp_replace(regexp_replace(query, '''(?:[^'']|'''')*''', '?', 'g'), '\\m[0-9][0-9.]*\\M', '?', 'g'), 160) as query
from sessions
where cardinality(blocked_by) > 0 or pid in (select pid from blockers)
or state like 'idle in transaction%'
or (xact_start is not null and clock_timestamp() - xact_start > interval '30 seconds')
order by xact_start nulls last
limit 50`);
return rows.map(row => ({
pid: Number(row.pid),
applicationName: (row.application_name as string | null) ?? null,
state: (row.state as string | null) ?? null,
waitEventType: (row.wait_event_type as string | null) ?? null,
waitEvent: (row.wait_event as string | null) ?? null,
blockedBy: Array.isArray(row.blocked_by) ? row.blocked_by.map(Number) : [],
transactionSeconds: row.transaction_seconds === null ? null : Number(row.transaction_seconds),
stateSeconds: row.state_seconds === null ? null : Number(row.state_seconds),
query: (row.query as string | null) ?? null,
}));
});
}
function authenticatedButUnreached(summary: ReattachPhaseSummary): string[] {
return summary.runs.filter(run => run.outcome === "outstanding" && run.authenticatedMs !== null).map(run => run.runId);
}
@@ -48,6 +48,7 @@ import { cancellationResultJson } from "../run-cancellation.js";
import { CONTROLLER_LEASE_LOST_FAILURE_CODE } from "../legacy-controller-lease.js";
import { isWaitingConversation } from "../agent-conversations.js";
import { PROCESS_IDENTITY_RECORDED } from "../native-local-process-stop.js";
import { trackRunIntake } from "../run-intake-stall-watch.js";
import {
hasAcknowledgedNativeReassignmentStopIntent,
hasAcknowledgedNativeLifecycleStopIntent,
@@ -625,7 +626,7 @@ export async function persistHeartbeatRunProcessMetadata(
);
const startedAt = new Date(observedStartedAt ?? meta.startedAt);
const processStartedAt = Number.isNaN(startedAt.getTime()) ? new Date() : startedAt;
return db.transaction(async tx => {
return trackRunIntake(runId, "process_identity", PROCESS_IDENTITY_RECORDED, () => db.transaction(async tx => {
const [binding] = await tx
.select({ companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId })
.from(heartbeatRuns)
@@ -654,7 +655,7 @@ export async function persistHeartbeatRunProcessMetadata(
});
return tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId))
.then((rows) => rows[0] ?? null);
});
}));
}
type SkillTestHeartbeatCompletion = {
@@ -11,6 +11,7 @@ import { NATIVE_SESSION_PROVIDER_EVENT_TYPES } from "./native-runtime/native-ses
import os from "node:os";
import { currentProcessSpaceHash, UNKNOWN_PROCESS_SPACE } from "./controller-process-identity.js";
import { proveOwnerGone } from "./owner-proof.js";
import { beginRunIntake } from "./run-intake-stall-watch.js";
import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED } from "./native-process-start-events.js";
export { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED };
@@ -593,6 +594,7 @@ export async function releaseOwnedSettledNativeRunProcessesAtShutdown(db: Db, op
if (stopped.length === 0) return { candidates: candidates.length, stillRunning: candidates.length, released: 0 };
const recorded = JSON.stringify(stopped.map(run => ({ id: run.id, pid: run.processPid, pgid: run.processGroupId,
started: run.processStartedAt, started_iso: run.processStartedAtIso })));
const intake = stopped.map(run => beginRunIntake(run.id, "write", LOCAL_PROCESS_STOPPED));
const released = await db.transaction(async transaction => {
const tx = transaction as unknown as Db;
await tx.execute(sql.raw(`set local statement_timeout = ${SHUTDOWN_SETTLED_PROCESS_STATEMENT_TIMEOUT_MS}`));
@@ -628,7 +630,7 @@ export async function releaseOwnedSettledNativeRunProcessesAtShutdown(db: Db, op
Array<{ companyId: string; runId: string; agentId: string }>;
if (rows.length > 0) await notifyInstructionCopyStopEvidenceForRuns(tx, rows);
return rows;
});
}).finally(() => { for (const done of intake) done(); });
for (const run of released) pendingSettledProcessReleases.delete(run.runId);
return { candidates: candidates.length, stillRunning: candidates.length - stopped.length, released: released.length };
}
@@ -0,0 +1,26 @@
import type { Db } from "@paperclipai/db";
import { describe, expect, it, vi } from "vitest";
const tracked = vi.hoisted(() => [] as Array<{ runId: string; stage: string; label: string }>);
vi.mock("../run-intake-stall-watch.js", async importOriginal => {
const actual = await importOriginal<typeof import("../run-intake-stall-watch.js")>();
return {
...actual,
trackRunIntake: (runId: string, stage: string, label: string, work: () => Promise<unknown>) => {
tracked.push({ runId, stage, label });
return work();
},
};
});
const { NativeRunCoordinatorStore } = await import("./native-run-coordinator-store.js");
describe("native run coordinator store intake tracking", () => {
it("registers each appended runner event with the run intake stall watch", async () => {
const store = new NativeRunCoordinatorStore({} as Db,
{ runId: "run-coordinator" } as ConstructorParameters<typeof NativeRunCoordinatorStore>[1]);
await expect(store.appendEvent({ eventType: "session.started" } as Parameters<typeof store.appendEvent>[0]))
.rejects.toThrow("native_event_schema_invalid");
expect(tracked).toEqual([{ runId: "run-coordinator", stage: "received", label: "session.started" }]);
});
});
@@ -10,6 +10,7 @@ import { createHash } from "node:crypto";
import { and, desc, eq, inArray, lt } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import { trackRunIntake } from "../run-intake-stall-watch.js";
import {
heartbeatRunEvents,
heartbeatRuns,
@@ -207,6 +208,16 @@ export class NativeRunCoordinatorStore {
readonly disposition: "committed" | "duplicate";
readonly cursor: number;
readonly highestContiguousSourceSeq: number;
}> {
const eventType = (value as { eventType?: unknown }).eventType;
return trackRunIntake(this.#binding.runId, "received", typeof eventType === "string" ? eventType : "unknown",
() => this.#appendEvent(value));
}
async #appendEvent(value: PrpEvent): Promise<{
readonly disposition: "committed" | "duplicate";
readonly cursor: number;
readonly highestContiguousSourceSeq: number;
}> {
const validated = validatePrpEvent(value);
if (!validated.ok) throw new Error("native_event_schema_invalid");
@@ -36,6 +36,7 @@ import { redactRunnerApiEvent, redactRunnerApiPayloads } from "./runner-api-log-
import { createNativeControllerRecovery, NATIVE_CONTROLLER_RECOVERY_MAX_BYTES, type NativeControllerCompatibility } from "./native-controller-recovery.js";
import { eligibleNativeTaskSessionPublication, publishNativeTaskSessionInTransaction, NATIVE_TASK_SESSION_PUBLICATION_KEY,
type NativeTaskSessionPublicationContext, type NativeTaskSessionPublicationIssue } from "./native-task-session-publication.js";
import { trackRunIntake } from "../run-intake-stall-watch.js";
import { lockNativeRunnerCheckpointOwner, nativeRunnerCheckpointOwnerLock, validateNativeRunnerCheckpointOwner, inspectNativeRunnerUpgradeHeader, type NativeRunnerCheckpointOwner } from "./native-runner-checkpoint.js";
import { crashPoint } from "@paperclipai/shared/crash-points";
import { NATIVE_PROVIDER_DURABLE_EVENT_TYPES } from "./native-retired-backup-source.js";
@@ -399,6 +400,12 @@ export class PaperclipControlPlanePort implements ControlPlanePort {
}
async appendEvent(value: NativeRunEvent | PrpEvent, options?: { signal: AbortSignal; deferPostCommitObservers?: true; streaming?: true }) {
const eventType = (value as { eventType?: unknown }).eventType;
return trackRunIntake(this.#binding.runId, "received", typeof eventType === "string" ? eventType : "unknown",
() => this.#appendEvent(value, options));
}
async #appendEvent(value: NativeRunEvent | PrpEvent, options?: { signal: AbortSignal; deferPostCommitObservers?: true; streaming?: true }) {
const persistenceStartedAt = Date.now(), persistenceClock = performance.now();
this.#assertActive(options);
if (!isPrpEvent(value)) throw new Error("native_legacy_event_not_supported");
@@ -0,0 +1,245 @@
import type { Db } from "@paperclipai/db";
import { agents, closeLivenessDb, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns } from "@paperclipai/db";
import { sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import { logger } from "../middleware/logger.js";
import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
import type { DatabaseSessionWait } from "./database-session-waits.js";
import { allocateHeartbeatRunEventSeq, appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
import { persistHeartbeatRunProcessMetadata } from "./heartbeat/run-lifecycle.js";
import { beginRunIntake, installRunIntakeStallWatch, stopRunIntakeStallWatch } from "./run-intake-stall-watch.js";
const STALL_MESSAGE = "runner event intake stalled; pending run writes, database sessions and lock waits";
const CLEARED_MESSAGE = "runner event intake stall cleared";
type StallReport = {
eventLoopLagMs: number;
operations: Array<{ runId: string; stage: string; label: string; pendingMs: number }>;
runs: Array<{ runId: string; pending: number; oldestPendingMs: number; stages: Record<string, number> }>;
pool: { maxConnections: number | null; pendingIntake: number };
sessions: Array<{ applicationName: string; state: string; sessions: number }>;
waits: DatabaseSessionWait[];
databaseReadError?: string;
};
function stallReports(warn: { mock: { calls: unknown[][] } }): StallReport[] {
return warn.mock.calls.filter(call => call[1] === STALL_MESSAGE).map(call => call[0] as StallReport);
}
afterEach(() => {
stopRunIntakeStallWatch();
vi.restoreAllMocks();
vi.useRealTimers();
});
describe("run intake stall watch", () => {
it("does nothing until it is installed", () => {
const done = beginRunIntake("run", "received", "session.started");
expect(done).toBeTypeOf("function");
done();
});
it("reports a stalled run once, throttles the next report, and logs when the stalled write finishes", async () => {
vi.useFakeTimers();
const warn = vi.spyOn(logger, "warn").mockImplementation(() => logger);
const info = vi.spyOn(logger, "info").mockImplementation(() => logger);
const unreachable = { transaction: () => Promise.reject(new Error("database unreachable")), $client: { options: { max: 7 } } } as unknown as Db;
installRunIntakeStallWatch(unreachable, { stallMs: 1_000, now: () => Date.now() });
const first = beginRunIntake("run-a", "process_identity", "native.process_identity_recorded");
vi.advanceTimersByTime(100);
const queued = beginRunIntake("run-a", "received", "session.started");
const healthy = beginRunIntake("run-b", "write", "item.completed");
vi.advanceTimersByTime(50);
healthy();
vi.advanceTimersByTime(50);
const other = beginRunIntake("run-c", "write", "turn.started");
await vi.advanceTimersByTimeAsync(950);
let reports = stallReports(warn);
expect(reports).toHaveLength(1);
expect(reports[0]!.operations).toEqual([
expect.objectContaining({ runId: "run-a", stage: "process_identity", label: "native.process_identity_recorded" }),
]);
expect(reports[0]!.runs[0]).toMatchObject({ runId: "run-a", pending: 2, stages: { process_identity: 1, received: 1 } });
expect(reports[0]!.pool).toMatchObject({ maxConnections: 7, pendingIntake: 3 });
expect(reports[0]!.databaseReadError).toBe("database unreachable");
await vi.advanceTimersByTimeAsync(800);
expect(stallReports(warn)).toHaveLength(1);
await vi.advanceTimersByTimeAsync(100);
reports = stallReports(warn);
expect(reports).toHaveLength(2);
expect(reports[1]!.operations).toEqual([expect.objectContaining({ runId: "run-c", stage: "write" })]);
queued();
expect(info.mock.calls.filter(call => call[1] === CLEARED_MESSAGE)).toHaveLength(0);
first();
other();
expect(info.mock.calls.filter(call => call[1] === CLEARED_MESSAGE).map(call => call[0])).toEqual([
expect.objectContaining({ runId: "run-a", stage: "process_identity" }),
expect.objectContaining({ runId: "run-c", stage: "write" }),
]);
await vi.advanceTimersByTimeAsync(5_000);
expect(stallReports(warn)).toHaveLength(2);
expect(vi.getTimerCount()).toBe(0);
});
it("reports pending writes when the database read hangs, and stops without waiting for it", async () => {
vi.useFakeTimers();
const warn = vi.spyOn(logger, "warn").mockImplementation(() => logger);
const hung = { transaction: () => new Promise(() => undefined), $client: { options: { max: 3 } } } as unknown as Db;
installRunIntakeStallWatch(hung, { stallMs: 1_000, reportTimeoutMs: 500, now: () => Date.now() });
const done = beginRunIntake("run-hung", "write", "session.started");
await vi.advanceTimersByTimeAsync(1_000);
expect(stallReports(warn)).toHaveLength(0);
await vi.advanceTimersByTimeAsync(500);
const [report] = stallReports(warn);
expect(report!.operations).toEqual([expect.objectContaining({ runId: "run-hung", stage: "write" })]);
expect(report!.databaseReadError).toBe("database session read timed out after 500 ms");
done();
const second = beginRunIntake("run-later", "write", "turn.started");
await vi.advanceTimersByTimeAsync(1_200);
stopRunIntakeStallWatch();
second();
await vi.advanceTimersByTimeAsync(500);
expect(vi.getTimerCount()).toBe(0);
expect(stallReports(warn)).toHaveLength(2);
});
it("names every stalled run across reports when more runs stall than one report lists", async () => {
vi.useFakeTimers();
const warn = vi.spyOn(logger, "warn").mockImplementation(() => logger);
const unreachable = { transaction: () => Promise.reject(new Error("database unreachable")), $client: {} } as unknown as Db;
installRunIntakeStallWatch(unreachable, { stallMs: 1_000, now: () => Date.now() });
const runIds = Array.from({ length: 25 }, (_, index) => `run-${index}`);
const pending = runIds.map(runId => beginRunIntake(runId, "write", "item.completed"));
await vi.advanceTimersByTimeAsync(1_000);
expect(stallReports(warn)).toHaveLength(1);
expect(stallReports(warn)[0]!.operations).toHaveLength(20);
await vi.advanceTimersByTimeAsync(1_000);
const reports = stallReports(warn);
expect(reports).toHaveLength(2);
expect(reports[1]!.operations).toHaveLength(5);
expect(reports.flatMap(report => report.operations.map(operation => operation.runId)).sort()).toEqual([...runIds].sort());
for (const done of pending) done();
});
it("keeps no timer once no run write is pending", async () => {
vi.useFakeTimers();
installRunIntakeStallWatch({} as Db, { stallMs: 1_000, now: () => Date.now() });
expect(vi.getTimerCount()).toBe(0);
beginRunIntake("run", "write", "item.delta")();
await vi.advanceTimersByTimeAsync(1_000);
expect(vi.getTimerCount()).toBe(0);
});
});
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_QUEUE_TEST_DATABASE_URL");
const support = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
if (!support.supported) console.warn(`Skipping run intake stall database tests: ${support.reason}`);
describe.skipIf(!support.supported)("run intake stall watch on a database", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: Db;
beforeAll(async () => {
if (!external) database = await startEmbeddedPostgresTestDatabase("paperclip-run-intake-stall");
db = createDb(external ?? database!.connectionString, { maxConnections: 4, applicationName: "paperclip:stall-test-boot" });
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
afterAll(async () => {
if (db) await closeLivenessDb(db, 1);
await db?.$client.end();
await database?.cleanup();
});
async function seedRun(prefix: string) {
const [company] = await db.insert(companies).values({ name: `Intake ${prefix}`, issuePrefix: prefix }).returning();
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Runner agent", status: "running",
adapterType: "paperclip_runner", adapterConfig: {} }).returning();
const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, status: "running",
runtimeMode: "native", invocationSource: "assignment", contextSnapshot: {} }).returning();
return run!;
}
it("names the idle transaction holding a run row when that run's process identity write waits behind it", async () => {
const run = await seedRun("RIS");
const warn = vi.spyOn(logger, "warn");
const info = vi.spyOn(logger, "info");
installRunIntakeStallWatch(db, { stallMs: 300 });
let release!: () => void;
const held = new Promise<void>(resolve => { release = resolve; });
let locked!: (pid: number) => void;
const holderPid = new Promise<number>(resolve => { locked = resolve; });
const holder = db.transaction(async tx => {
const [backend] = await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`);
await tx.execute(sql.raw(`select id from heartbeat_runs where id = '${run.id}' and next_event_seq >= 0 for update`));
locked(Number(backend!.pid));
await held;
});
const holderBackend = await holderPid;
const identity = persistHeartbeatRunProcessMetadata(db, run.id,
{ pid: 563, processGroupId: null, startedAt: new Date().toISOString(), targetKind: "remote" });
try {
await vi.waitFor(() => expect(stallReports(warn)).toHaveLength(1), { timeout: 5_000, interval: 20 });
const [report] = stallReports(warn);
expect(report!.operations).toEqual([
expect.objectContaining({ runId: run.id, stage: "process_identity", label: "native.process_identity_recorded" }),
]);
expect(report!.runs[0]).toMatchObject({ runId: run.id, stages: { process_identity: 1, write: 1 } });
expect(report!.pool.maxConnections).toBe(4);
expect(report!.sessions).toEqual(expect.arrayContaining([
expect.objectContaining({ applicationName: "paperclip:stall-test-boot", state: "idle in transaction" }),
]));
expect(report!.waits).toEqual(expect.arrayContaining([
expect.objectContaining({ pid: holderBackend, state: "idle in transaction", applicationName: "paperclip:stall-test-boot" }),
expect.objectContaining({ waitEventType: "Lock", blockedBy: [holderBackend] }),
]));
expect(JSON.stringify(report!.waits)).not.toContain(run.id);
expect(report!.waits.find(wait => wait.pid === holderBackend)?.query).toContain("where id = ? and next_event_seq >= ? for update");
} finally {
release();
await holder;
}
await expect(identity).resolves.toMatchObject({ processPid: 563 });
expect(info.mock.calls.filter(call => call[1] === CLEARED_MESSAGE).map(call => call[0])).toEqual([
expect.objectContaining({ runId: run.id, stage: "process_identity" }),
]);
});
it("reports a direct event sequence reservation that waits behind a held run row", async () => {
const run = await seedRun("RIQ");
const warn = vi.spyOn(logger, "warn");
installRunIntakeStallWatch(db, { stallMs: 300 });
let release!: () => void;
const held = new Promise<void>(resolve => { release = resolve; });
let locked!: () => void;
const isLocked = new Promise<void>(resolve => { locked = resolve; });
const holder = db.transaction(async tx => {
await tx.execute(sql.raw(`select id from heartbeat_runs where id = '${run.id}' for update`));
locked();
await held;
});
await isLocked;
const reserved = allocateHeartbeatRunEventSeq(db, run.id);
try {
await vi.waitFor(() => expect(stallReports(warn)).toHaveLength(1), { timeout: 5_000, interval: 20 });
expect(stallReports(warn)[0]!.operations).toEqual([
expect.objectContaining({ runId: run.id, stage: "write", label: "event_sequence" }),
]);
} finally {
release();
await holder;
}
await expect(reserved).resolves.toBeTypeOf("number");
});
it("stays silent while run event writes keep committing", async () => {
const run = await seedRun("RIH");
const warn = vi.spyOn(logger, "warn");
installRunIntakeStallWatch(db, { stallMs: 200 });
for (let index = 0; index < 5; index += 1) {
await appendHeartbeatRunEvent(db, { companyId: run.companyId, runId: run.id, agentId: run.agentId,
eventType: "lifecycle", stream: "system", level: "info", message: `step ${index}` });
}
await new Promise(resolve => setTimeout(resolve, 400));
expect(stallReports(warn)).toHaveLength(0);
});
});
@@ -0,0 +1,156 @@
import { dedicatedDbConnectionStats, type Db } from "@paperclipai/db";
import { logger } from "../middleware/logger.js";
import { readDatabaseSessionReport } from "./database-session-waits.js";
export const RUN_INTAKE_STALL_MS = 30_000;
export const RUN_INTAKE_REPORT_TIMEOUT_MS = 10_000;
const REPORTED_OPERATIONS_MAX = 20;
const REPORTED_RUNS_MAX = 20;
export type RunIntakeStage = "received" | "write" | "process_identity";
type Operation = { runId: string; stage: RunIntakeStage; label: string; startedAtMs: number; stalled: boolean; listed: boolean };
type Watch = {
db: Db;
stallMs: number;
reportTimeoutMs: number;
now: () => number;
operations: Map<number, Operation>;
nextToken: number;
timer: ReturnType<typeof setTimeout> | null;
dueAtMs: number;
lastReportAtMs: number | null;
report: Promise<void> | null;
};
let watch: Watch | null = null;
const settled = () => undefined;
export function installRunIntakeStallWatch(db: Db,
options: { stallMs?: number; reportTimeoutMs?: number; now?: () => number } = {}): void {
if (watch?.timer) clearTimeout(watch.timer);
watch = { db, stallMs: options.stallMs ?? RUN_INTAKE_STALL_MS, reportTimeoutMs: options.reportTimeoutMs ?? RUN_INTAKE_REPORT_TIMEOUT_MS,
now: options.now ?? Date.now, operations: new Map(),
nextToken: 0, timer: null, dueAtMs: 0, lastReportAtMs: null, report: null };
}
export function stopRunIntakeStallWatch(): void {
const current = watch;
watch = null;
if (!current) return;
if (current.timer) clearTimeout(current.timer);
current.timer = null;
}
export function beginRunIntake(runId: string, stage: RunIntakeStage, label: string): () => void {
const current = watch;
if (!current) return settled;
const token = current.nextToken++;
const operation: Operation = { runId, stage, label, startedAtMs: current.now(), stalled: false, listed: false };
current.operations.set(token, operation);
if (!current.timer && !current.report) arm(current, operation.startedAtMs + current.stallMs);
return () => {
if (!current.operations.delete(token) || !operation.listed) return;
logger.info({ runId, stage, label, pendingMs: current.now() - operation.startedAtMs },
"runner event intake stall cleared");
};
}
export async function trackRunIntake<T>(runId: string, stage: RunIntakeStage, label: string, work: () => Promise<T>): Promise<T> {
const done = beginRunIntake(runId, stage, label);
try { return await work(); }
finally { done(); }
}
function arm(current: Watch, dueAtMs: number) {
const now = current.now();
current.dueAtMs = Math.max(dueAtMs, now);
current.timer = setTimeout(() => check(current), current.dueAtMs - now);
current.timer.unref?.();
}
function nextDueAtMs(current: Watch): number | null {
for (const operation of current.operations.values()) {
if (!operation.stalled) return operation.startedAtMs + current.stallMs;
}
return null;
}
function check(current: Watch) {
current.timer = null;
if (watch !== current) return;
const now = current.now();
const eventLoopLagMs = Math.max(0, now - current.dueAtMs);
if (current.lastReportAtMs !== null && now - current.lastReportAtMs < current.stallMs) {
arm(current, current.lastReportAtMs + current.stallMs);
return;
}
const stalledRuns = new Set<string>();
for (const operation of current.operations.values()) if (operation.stalled) stalledRuns.add(operation.runId);
const newlyStalled: Operation[] = [];
for (const operation of current.operations.values()) {
if (operation.stalled) continue;
if (now - operation.startedAtMs < current.stallMs) break;
if (stalledRuns.has(operation.runId)) { operation.stalled = true; continue; }
if (newlyStalled.length >= REPORTED_OPERATIONS_MAX) break;
operation.stalled = true;
stalledRuns.add(operation.runId);
newlyStalled.push(operation);
}
if (newlyStalled.length === 0) {
const due = nextDueAtMs(current);
if (due !== null) arm(current, due);
return;
}
current.lastReportAtMs = now;
const report: Promise<void> = reportStall(current, newlyStalled, eventLoopLagMs)
.catch(error => logger.warn({ err: error }, "runner event intake stall report failed"))
.finally(() => {
if (current.report === report) current.report = null;
if (watch !== current || current.timer) return;
const due = nextDueAtMs(current);
if (due !== null) arm(current, due);
});
current.report = report;
}
function pendingByRun(current: Watch, now: number) {
const runs = new Map<string, { runId: string; pending: number; oldestPendingMs: number; oldestLabel: string;
stages: Partial<Record<RunIntakeStage, number>> }>();
for (const operation of current.operations.values()) {
let run = runs.get(operation.runId);
if (!run) {
run = { runId: operation.runId, pending: 0, oldestPendingMs: now - operation.startedAtMs, oldestLabel: operation.label, stages: {} };
runs.set(operation.runId, run);
}
run.pending += 1;
run.stages[operation.stage] = (run.stages[operation.stage] ?? 0) + 1;
}
return [...runs.values()].sort((left, right) => right.oldestPendingMs - left.oldestPendingMs).slice(0, REPORTED_RUNS_MAX);
}
async function reportStall(current: Watch, stalled: Operation[], eventLoopLagMs: number) {
const now = current.now();
for (const operation of stalled) operation.listed = true;
const operations = stalled.map(operation => ({ runId: operation.runId, stage: operation.stage, label: operation.label,
pendingMs: now - operation.startedAtMs }));
const client = current.db.$client as { options?: { max?: unknown } } | undefined;
const pool = { maxConnections: typeof client?.options?.max === "number" ? client.options.max : null,
pendingIntake: current.operations.size, dedicated: dedicatedDbConnectionStats(current.db) };
const runs = pendingByRun(current, now);
let expire: ReturnType<typeof setTimeout> | undefined;
const expired = new Promise<never>((_resolve, reject) => {
expire = setTimeout(() => reject(new Error(`database session read timed out after ${current.reportTimeoutMs} ms`)),
current.reportTimeoutMs);
expire.unref?.();
});
const database = await Promise.race([readDatabaseSessionReport(current.db), expired]).then(
report => ({ ...report, error: null as string | null }),
error => ({ sessions: [], waits: [], error: error instanceof Error ? error.message.slice(0, 160) : "unknown" }),
).finally(() => clearTimeout(expire));
logger.warn({ stallThresholdMs: current.stallMs, eventLoopLagMs, operations, runs, pool, sessions: database.sessions,
waits: database.waits, ...(database.error ? { databaseReadError: database.error } : {}) },
"runner event intake stalled; pending run writes, database sessions and lock waits");
}