mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
fix(server): name the blocking database sessions whenever a run's event writes stall (#21956)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server stores the events that a runner sends for each run, and it records the identity of the runner process > - Sometimes the server stops storing a run's events for minutes, and the server log stays quiet > - In two restart-chaos cases, the identity write of the run started and then did not commit for about 197 s and 480 s > - The existing database report only fires when the reattach phase ends at its cap, so it did not fire for the cold-start case > - This pull request logs the pending writes and the blocking database sessions whenever any run write stays pending for 30 s > - The benefit is that the next stall names the session and the query that hold the run, so the fix can target one proven holder ## Linked Issues or Issue Description Refs #21946 **What happened?** Restart-chaos waves on master showed two stalls in event storage. 1. Cold start of an OpenCode run, with no restart. The runner emitted `session.started` at 22:13:23.7Z. The server stored it at 22:16:40.97Z, about 197 s later. The second `native.process_identity_recorded` write of the run started at 22:13:23.706Z, and `runner.turn.submit` took 197270 ms. A second wave showed the same shape (about 191 s). The server log had nothing in that window. 2. After a hard kill, every API GET timed out from 22:35:24Z to 22:43:23Z, server CPU was 0%, and no run events were stored. The process identity transaction of the run began at 22:35:22.6Z and committed at about 22:43:23Z. **Expected behavior** When the server cannot store a run's events, the server log names the cause: which run writes are pending, and which database sessions hold or wait for locks. **Steps to reproduce** 1. Start a transaction that locks a `heartbeat_runs` row (`for update`) and stays idle. 2. Record a process identity for that run (`persistHeartbeatRunProcessMetadata`). 3. Before this change, nothing is logged while the write waits. After this change, a warning names the idle holder and the waiting statement after 30 s. The new database test does this with a 300 ms threshold. **Relevant logs or output** Example warning from the new test: `runner event intake stalled; pending run writes, database sessions and lock waits` with `operations: [{stage: "process_identity", pendingMs: 301}]` and `waits: [{state: "idle in transaction", query: "select id from heartbeat_runs where id = ? ... for update"}, {waitEventType: "Lock", blockedBy: [<holder pid>]}]`. ## What Changed - New `server/src/services/run-intake-stall-watch.ts`. It keeps an in-memory list of pending run operations: event receipts (`PaperclipControlPlanePort.appendEvent`), event writes (`appendHeartbeatRunEvent`, `appendHeartbeatRunEventBatch`), and process identity writes (`persistHeartbeatRunProcessMetadata`). - When an operation stays pending for 30 s, the watch logs one warning per stalled run. The warning has: the pending operations and their age, the pending count and oldest age per run, the pool size, the database sessions per application name and state, the lock waits and holders (`pg_blocking_pids`, query without literals, cut to 160 characters), and the event-loop lag. Reports are at most one per 30 s. A second line logs when a reported operation finishes. - The watch arms a timer only while an operation is pending. It reads the database only when it reports, through the private liveness pool with a 5 s statement timeout. - The server names its main pool `paperclip:<boot id>` unless `DATABASE_APPLICATION_NAME` is set. This shows which server process owns each session after a restart. - `readDatabaseSessionWaits` moves from the reattach phase to `server/src/services/database-session-waits.ts`. The reattach phase re-exports it. - `@paperclipai/db` exports `databaseClientOptionsFromEnv` and `DatabaseClientOptions`. - `doc/DATABASE.md` describes the new log lines and the pool name. - I did not add `idle_in_transaction_session_timeout` to the main pool. Some code holds a transaction open on purpose, and a blind timeout could abort valid work. The new report names idle-in-transaction holders instead. - These are server log lines only. This change does not touch Telemetry or OpenTelemetry. ## Verification - `cd server && npx vitest run src/services/run-intake-stall-watch.test.ts` (5 tests). The database test holds a run row in an idle transaction, starts a process identity write, and checks that the warning names the holder pid, its query, the waiting statement, and the pool name. It is red without the tracking in `run-lifecycle.ts` and `heartbeat-run-events.ts` (`expected [] to have a length of 1`), and green with it. - Unit tests with fake timers check one report per stalled run, the 30 s throttle, the clear line, and that no timer stays armed when nothing is pending. - `cd server && npx vitest run src/services/controller-process-identity.test.ts` checks the pool name and the `DATABASE_APPLICATION_NAME` override. - `npx tsc --noEmit` passes for `server` and `packages/db`. - `pnpm verify` ran locally on the affected tests. ## Risks - Low risk. Each run event adds one map insert and one map delete. No database work happens while writes finish within 30 s. - The main pool `application_name` changes from `paperclip` to `paperclip:<boot id>`. A dashboard that filters on exactly `paperclip` must use a prefix match, or the operator can set `DATABASE_APPLICATION_NAME`. - A report reads `pg_stat_activity` on the liveness pool. A report during a full database stall can fail; the warning then has `databaseReadError` and still lists the pending operations. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7b8df907d3
commit
3afb2c215b
16 files changed
+602
-59
No files matched your search
+13
-1
@@ -134,9 +134,21 @@ DATABASE_POOL_MAX=25 # connection pool size; default: 10
|
||||
DATABASE_IDLE_TIMEOUT_SECONDS=60 # close idle pooled connections; default: 60 (0 = keep open)
|
||||
DATABASE_CONNECT_TIMEOUT_SECONDS=10 # default: 30
|
||||
DATABASE_MAX_LIFETIME_SECONDS=1800 # recycle a pooled connection after this long; default: 30-60 min (random)
|
||||
DATABASE_APPLICATION_NAME=paperclip # application_name in pg_stat_activity; default: paperclip
|
||||
DATABASE_APPLICATION_NAME=paperclip # application_name in pg_stat_activity; default for the server's main pool: paperclip:<boot id>
|
||||
```
|
||||
|
||||
The server names its main pool `paperclip:<boot id>`, so after a restart the
|
||||
sessions of each server process can be told apart in `pg_stat_activity`. When a
|
||||
run's event, process identity, or event-write operation stays unfinished for
|
||||
30 seconds, the server logs one warning per stalled run ("runner event intake
|
||||
stalled"). The warning lists the pending operations of each run, the pool size,
|
||||
the database sessions per application name and state, and the sessions that
|
||||
wait for a lock or hold one (`pg_blocking_pids`, query text without literals,
|
||||
cut to 160 characters). It also logs the event-loop lag. A second line
|
||||
("runner event intake stall cleared") records when the operation finished. The
|
||||
check runs only while an operation is pending, and it reads `pg_stat_activity`
|
||||
only when it reports a stall.
|
||||
|
||||
`withDedicatedDbConnection` gives an action exclusive use of one session outside
|
||||
the main pool (application name `paperclip-workspace-finalization-lock`). Each
|
||||
`createDb` owner keeps released single-connection clients idle for reuse: always
|
||||
|
||||
@@ -73,7 +73,7 @@ statements longer than 8 KB. If a prepared statement fails because the pooler's
|
||||
stale after a schema change or was lost, Paperclip stops preparing that
|
||||
statement and retries it once unprepared.
|
||||
|
||||
Related optional client tuning: `DATABASE_POOL_MAX`, `DATABASE_IDLE_TIMEOUT_SECONDS`, `DATABASE_CONNECT_TIMEOUT_SECONDS`, `DATABASE_MAX_LIFETIME_SECONDS`, `DATABASE_APPLICATION_NAME`. Driver defaults apply when unset, except that idle pooled connections close after 60 seconds (`DATABASE_IDLE_TIMEOUT_SECONDS=0` keeps them open) and the pool reports `application_name=paperclip`. See [Connection pool settings](#connection-pool-settings).
|
||||
Related optional client tuning: `DATABASE_POOL_MAX`, `DATABASE_IDLE_TIMEOUT_SECONDS`, `DATABASE_CONNECT_TIMEOUT_SECONDS`, `DATABASE_MAX_LIFETIME_SECONDS`, `DATABASE_APPLICATION_NAME`. Driver defaults apply when unset, except that idle pooled connections close after 60 seconds (`DATABASE_IDLE_TIMEOUT_SECONDS=0` keeps them open) and the server's main pool reports `application_name=paperclip:<boot id>` (other pools use names that start with `paperclip`). See [Connection pool settings](#connection-pool-settings).
|
||||
|
||||
## Connection Pool Settings
|
||||
|
||||
@@ -85,7 +85,7 @@ The server opens one postgres.js pool for its own queries (and a second one when
|
||||
| `DATABASE_IDLE_TIMEOUT_SECONDS` | `60` | Close a pooled connection after this much idle time. `0` keeps idle connections open forever (the driver default). |
|
||||
| `DATABASE_CONNECT_TIMEOUT_SECONDS` | `30` (driver) | Give up on a connection attempt after this long. |
|
||||
| `DATABASE_MAX_LIFETIME_SECONDS` | 30–60 min, randomized (driver) | Recycle a pooled connection once it is this old. |
|
||||
| `DATABASE_APPLICATION_NAME` | `paperclip` | Value of `application_name` in `pg_stat_activity`, so you can find Paperclip's backends: `SELECT * FROM pg_stat_activity WHERE application_name = 'paperclip';` |
|
||||
| `DATABASE_APPLICATION_NAME` | `paperclip:<boot id>` (server main pool) | Value of `application_name` in `pg_stat_activity`. The boot id tells the sessions of each server process apart after a restart. Find Paperclip's backends with `SELECT * FROM pg_stat_activity WHERE application_name LIKE 'paperclip%';` |
|
||||
| `DATABASE_DEDICATED_CONNECTION_MAX` | `24` | Dedicated lock connections in use at once per process before new callers wait. Nested callers do not wait. |
|
||||
| `DATABASE_DEDICATED_CONNECTION_WAIT_MS` | `30000` | After this wait, a queued caller gets a dedicated connection over the limit. |
|
||||
| `DATABASE_PREPARED_STATEMENTS` | Unset | `true` opts parameterized ORM queries into preparation. `false` disables preparation. Use `false` when the pooler does not support named statements. Unset preserves existing behavior. |
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
export {
|
||||
createDb,
|
||||
databaseClientOptionsFromEnv,
|
||||
type DatabaseClientOptions,
|
||||
withDedicatedDbConnection,
|
||||
closeDedicatedDbConnections,
|
||||
DEDICATED_CONNECTION_IDLE_MAX,
|
||||
|
||||
+6
-2
@@ -136,6 +136,8 @@ import {
|
||||
} from "@paperclipai/db";
|
||||
import detectPort from "detect-port";
|
||||
import { createApp } from "./app.js";
|
||||
import { serverDatabaseClientOptions } from "./services/controller-process-identity.js";
|
||||
import { installRunIntakeStallWatch, stopRunIntakeStallWatch } from "./services/run-intake-stall-watch.js";
|
||||
import { drainHttpRequestWork } from "./middleware/idle-admission.js";
|
||||
import { initializeCloudRuntimeConfig } from "./services/cloud-runtime-config.js";
|
||||
import { loadCloudBootstrapConfig, loadConfig, type Config } from "./config.js";
|
||||
@@ -537,7 +539,7 @@ async function startServerWithDatabaseTeardown(
|
||||
const migrationUrl = startupConfig.databaseMigrationUrl ?? startupConfig.databaseUrl;
|
||||
migrationSummary = await ensureMigrations(migrationUrl, "PostgreSQL");
|
||||
|
||||
db = createDb(startupConfig.databaseUrl);
|
||||
db = createDb(startupConfig.databaseUrl, serverDatabaseClientOptions());
|
||||
pluginMigrationDb = startupConfig.databaseMigrationUrl ? createDb(startupConfig.databaseMigrationUrl) : db;
|
||||
logger.info("Using external PostgreSQL via DATABASE_URL/config");
|
||||
activeDatabaseConnectionString = startupConfig.databaseUrl;
|
||||
@@ -744,7 +746,7 @@ async function startServerWithDatabaseTeardown(
|
||||
autoApply: shouldAutoApplyFirstRunMigrations,
|
||||
});
|
||||
|
||||
db = createDb(embeddedConnectionString);
|
||||
db = createDb(embeddedConnectionString, serverDatabaseClientOptions());
|
||||
pluginMigrationDb = db;
|
||||
logger.info("Embedded PostgreSQL ready");
|
||||
activeDatabaseConnectionString = embeddedConnectionString;
|
||||
@@ -757,10 +759,12 @@ async function startServerWithDatabaseTeardown(
|
||||
// by the fail-loud startup path, so no exit leaves pooled backends behind.
|
||||
const closeDatabaseClients = async () => {
|
||||
setSentryRunLinkResolver(null);
|
||||
stopRunIntakeStallWatch();
|
||||
const clients = pluginMigrationDb === db ? [db] : [db, pluginMigrationDb];
|
||||
await Promise.all([...clients.map((client) => endDatabaseClient(client, 5)), closeLivenessDb(db, 5), closeCheckpointDb(db, 5), closeFileStorageDb(db, 5), closeDedicatedDbConnections(db, 5)]);
|
||||
};
|
||||
startupDatabase.close = closeDatabaseClients;
|
||||
installRunIntakeStallWatch(db);
|
||||
startupTimeline.mark("database");
|
||||
setSentryRunLinkResolver(createSentryRunLinkResolver(db));
|
||||
await migrateCloudConfiguration(db as any);
|
||||
|
||||
@@ -1,11 +1,23 @@
|
||||
import { spawn } from "node:child_process";
|
||||
import { once } from "node:events";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { processExists, processStartsDefinitelyDiffer } from "./controller-process-identity.js";
|
||||
import { currentControllerBootId, processExists, processStartsDefinitelyDiffer, serverDatabaseClientOptions } from "./controller-process-identity.js";
|
||||
import { isPidAlive, isProcessGroupAlive } from "./local-service-supervisor.js";
|
||||
import { readProcessStartedAt } from "./hot-restart.js";
|
||||
import { recoveryProcessMayBeAlive } from "./native-runtime/native-session-executor.js";
|
||||
|
||||
describe("server database client options", () => {
|
||||
it("names the main pool after this boot so its database sessions can be told apart from an earlier server's", () => {
|
||||
const options = serverDatabaseClientOptions({ DATABASE_POOL_MAX: "12" });
|
||||
expect(options).toEqual({ applicationName: `paperclip:${currentControllerBootId()}`, maxConnections: 12 });
|
||||
expect(options.applicationName!.length).toBeLessThanOrEqual(63);
|
||||
});
|
||||
|
||||
it("keeps an operator's DATABASE_APPLICATION_NAME", () => {
|
||||
expect(serverDatabaseClientOptions({ DATABASE_APPLICATION_NAME: "paperclip-web" }).applicationName).toBe("paperclip-web");
|
||||
});
|
||||
});
|
||||
|
||||
describe("shared process identity proof", () => {
|
||||
afterEach(() => vi.restoreAllMocks());
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@ import { createHash, randomUUID } from "node:crypto";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { readFileSync, readlinkSync } from "node:fs";
|
||||
import { hostname } from "node:os";
|
||||
import { databaseClientOptionsFromEnv, type DatabaseClientOptions } from "@paperclipai/db";
|
||||
import { readProcessStartedAt } from "./hot-restart.js";
|
||||
import { currentWorkspaceWorker, type WorkspaceWorker } from "./native-runtime/native-workspace-platform-proof.js";
|
||||
|
||||
@@ -127,6 +128,10 @@ export function currentControllerBootId(): string {
|
||||
return bootId;
|
||||
}
|
||||
|
||||
export function serverDatabaseClientOptions(env: NodeJS.ProcessEnv = process.env): DatabaseClientOptions {
|
||||
return { applicationName: `paperclip:${bootId}`, ...databaseClientOptionsFromEnv(env) };
|
||||
}
|
||||
|
||||
const stampListeners = new Set<() => void>();
|
||||
/** Called (synchronously, without I/O) whenever this process stamps ownership of a record. */
|
||||
export function onControllerStamp(listener: () => void): () => void {
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
import { sql } from "drizzle-orm";
|
||||
import { livenessDb, type Db } from "@paperclipai/db";
|
||||
|
||||
export type DatabaseSessionWait = {
|
||||
pid: number;
|
||||
applicationName: string | null;
|
||||
state: string | null;
|
||||
waitEventType: string | null;
|
||||
waitEvent: string | null;
|
||||
blockedBy: number[];
|
||||
transactionSeconds: number | null;
|
||||
stateSeconds: number | null;
|
||||
query: string | null;
|
||||
};
|
||||
|
||||
export type DatabaseSessionCount = { applicationName: string; state: string; sessions: number };
|
||||
|
||||
const SESSION_WAITS_SQL = sql`
|
||||
with sessions as (
|
||||
select pid, application_name, state, wait_event_type, wait_event, pg_blocking_pids(pid) as blocked_by,
|
||||
xact_start, state_change, query
|
||||
from pg_stat_activity
|
||||
where datname = current_database() and pid <> pg_backend_pid() and backend_type = 'client backend'
|
||||
), blockers as (select distinct unnest(blocked_by) as pid from sessions)
|
||||
select pid, application_name, state, wait_event_type, wait_event, blocked_by,
|
||||
extract(epoch from clock_timestamp() - xact_start)::int as transaction_seconds,
|
||||
extract(epoch from clock_timestamp() - state_change)::int as state_seconds,
|
||||
left(regexp_replace(regexp_replace(query, '''(?:[^'']|'''')*''', '?', 'g'), '\\m[0-9][0-9.]*\\M', '?', 'g'), 160) as query
|
||||
from sessions
|
||||
where cardinality(blocked_by) > 0 or pid in (select pid from blockers)
|
||||
or state like 'idle in transaction%'
|
||||
or (xact_start is not null and clock_timestamp() - xact_start > interval '30 seconds')
|
||||
order by xact_start nulls last
|
||||
limit 50`;
|
||||
|
||||
function sessionWait(row: Record<string, unknown>): DatabaseSessionWait {
|
||||
return {
|
||||
pid: Number(row.pid),
|
||||
applicationName: (row.application_name as string | null) ?? null,
|
||||
state: (row.state as string | null) ?? null,
|
||||
waitEventType: (row.wait_event_type as string | null) ?? null,
|
||||
waitEvent: (row.wait_event as string | null) ?? null,
|
||||
blockedBy: Array.isArray(row.blocked_by) ? row.blocked_by.map(Number) : [],
|
||||
transactionSeconds: row.transaction_seconds === null ? null : Number(row.transaction_seconds),
|
||||
stateSeconds: row.state_seconds === null ? null : Number(row.state_seconds),
|
||||
query: (row.query as string | null) ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
export async function readDatabaseSessionWaits(db: Db): Promise<DatabaseSessionWait[]> {
|
||||
return livenessDb(db).transaction(async tx => {
|
||||
await tx.execute(sql`select set_config('statement_timeout', '5000', true)`);
|
||||
const rows = await tx.execute<Record<string, unknown>>(SESSION_WAITS_SQL);
|
||||
return rows.map(sessionWait);
|
||||
});
|
||||
}
|
||||
|
||||
export async function readDatabaseSessionReport(db: Db, statementTimeoutMs = 5_000): Promise<{
|
||||
sessions: DatabaseSessionCount[]; waits: DatabaseSessionWait[];
|
||||
}> {
|
||||
return livenessDb(db).transaction(async tx => {
|
||||
await tx.execute(sql`select set_config('statement_timeout', ${String(statementTimeoutMs)}, true)`);
|
||||
const counts = await tx.execute<Record<string, unknown>>(sql`
|
||||
select coalesce(application_name, '') as application_name, coalesce(state, '') as state, count(*)::int as sessions
|
||||
from pg_stat_activity
|
||||
where datname = current_database() and pid <> pg_backend_pid() and backend_type = 'client backend'
|
||||
group by 1, 2
|
||||
order by 3 desc
|
||||
limit 40`);
|
||||
const rows = await tx.execute<Record<string, unknown>>(SESSION_WAITS_SQL);
|
||||
return {
|
||||
sessions: counts.map(row => ({ applicationName: String(row.application_name), state: String(row.state),
|
||||
sessions: Number(row.sessions) })),
|
||||
waits: rows.map(sessionWait),
|
||||
};
|
||||
});
|
||||
}
|
||||
@@ -13,6 +13,7 @@ import { heartbeatRunEvents, heartbeatRuns, primeDatabaseWorkTransactionId } fro
|
||||
import { DATABASE_WORK_CHANNEL } from "@paperclipai/db/work-signals";
|
||||
import { nativeSha256 } from "./native-runtime/canonical.js";
|
||||
import { isUniqueViolation } from "../db-errors.js";
|
||||
import { trackRunIntake } from "./run-intake-stall-watch.js";
|
||||
|
||||
export const RETRY_EXHAUSTION_MESSAGE_PREFIX = "Bounded retry exhausted";
|
||||
|
||||
@@ -88,6 +89,14 @@ export async function allocateHeartbeatRunEventSeq(
|
||||
db: Db,
|
||||
runId: string,
|
||||
runUpdate: HeartbeatRunEventRunUpdate = {},
|
||||
): Promise<number> {
|
||||
return trackRunIntake(runId, "write", "event_sequence", () => reserveRunEventSeq(db, runId, runUpdate));
|
||||
}
|
||||
|
||||
async function reserveRunEventSeq(
|
||||
db: Db,
|
||||
runId: string,
|
||||
runUpdate: HeartbeatRunEventRunUpdate,
|
||||
): Promise<number> {
|
||||
const [updated] = await db
|
||||
.update(heartbeatRuns)
|
||||
@@ -118,6 +127,13 @@ function runEventWakes(input: AppendHeartbeatRunEventInput): RunEventWakes {
|
||||
export async function appendHeartbeatRunEvent(
|
||||
db: Db,
|
||||
input: AppendHeartbeatRunEventInput,
|
||||
): Promise<AppendHeartbeatRunEventResult> {
|
||||
return trackRunIntake(input.runId, "write", input.eventType, () => appendRunEvent(db, input));
|
||||
}
|
||||
|
||||
async function appendRunEvent(
|
||||
db: Db,
|
||||
input: AppendHeartbeatRunEventInput,
|
||||
): Promise<AppendHeartbeatRunEventResult> {
|
||||
// Deduplicated receipts return without allocating, so they cannot carry run changes.
|
||||
if (input.runUpdate && (input.nativeSource || input.retryExhaustion)) throw new Error("heartbeat_run_event_run_update_unsupported");
|
||||
@@ -178,13 +194,23 @@ export async function appendHeartbeatRunEventBatch(
|
||||
): Promise<AppendHeartbeatRunEventResult[]> {
|
||||
const first = inputs[0];
|
||||
if (!first) return [];
|
||||
return trackRunIntake(first.runId, "write", `${first.eventType} x${inputs.length}`,
|
||||
() => appendRunEventBatch(db, inputs, assertWritable));
|
||||
}
|
||||
|
||||
async function appendRunEventBatch(
|
||||
db: Db,
|
||||
inputs: readonly AppendHeartbeatRunEventInput[],
|
||||
assertWritable?: () => void,
|
||||
): Promise<AppendHeartbeatRunEventResult[]> {
|
||||
const first = inputs[0]!;
|
||||
const source = first.nativeSource;
|
||||
if (!source || inputs.some(input => !input.nativeSource || input.runUpdate || input.retryExhaustion
|
||||
|| input.companyId !== first.companyId || input.runId !== first.runId || input.agentId !== first.agentId
|
||||
|| input.nativeSource.sourceInstanceId !== source.sourceInstanceId)) {
|
||||
throw new Error("heartbeat_run_event_batch_unsupported");
|
||||
}
|
||||
if (inputs.length === 1) return [await appendHeartbeatRunEvent(db, assertWritable ? { ...first, assertWritable } : first)];
|
||||
if (inputs.length === 1) return [await appendRunEvent(db, assertWritable ? { ...first, assertWritable } : first)];
|
||||
const wakes = inputs.map(runEventWakes);
|
||||
if (!wakes.some(wake => Object.values(wake).some(Boolean)) && !is(db, PgTransaction)) {
|
||||
assertWritable?.();
|
||||
@@ -596,7 +622,7 @@ async function appendLockedEvent(
|
||||
await notifyNativeResponsePresentation(tx, { companyId: run.companyId, runId: input.runId });
|
||||
}
|
||||
if (wakes.retention) await notifyAttentionRetentionWork(tx, { companyId: input.companyId, sourceKind: "failed_run" });
|
||||
const seq = await allocateHeartbeatRunEventSeq(tx, input.runId, input.runUpdate);
|
||||
const seq = await reserveRunEventSeq(tx, input.runId, input.runUpdate ?? {});
|
||||
const [row] = await tx.insert(heartbeatRunEvents).values({
|
||||
companyId: input.companyId,
|
||||
runId: input.runId,
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { and, eq, inArray, isNotNull, isNull, ne, notLike, or, sql } from "drizzle-orm";
|
||||
import { dedicatedDbConnectionStats, heartbeatRuns, livenessDb, nativeRunFinalizations, type Db } from "@paperclipai/db";
|
||||
import { and, eq, inArray, isNotNull, isNull, ne, notLike, or } from "drizzle-orm";
|
||||
import { dedicatedDbConnectionStats, heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db";
|
||||
import { readDatabaseSessionWaits } from "../database-session-waits.js";
|
||||
import { logger } from "../../middleware/logger.js";
|
||||
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js";
|
||||
@@ -21,6 +22,8 @@ import {
|
||||
type ReattachPhaseSummary,
|
||||
} from "./run-admission.js";
|
||||
|
||||
export { readDatabaseSessionWaits, type DatabaseSessionWait } from "../database-session-waits.js";
|
||||
|
||||
export const REATTACH_PHASE_DEFAULT_CAP_MS = 180_000;
|
||||
export const REATTACH_PHASE_MIN_CAP_MS = 5_000;
|
||||
export const REATTACH_PHASE_MAX_CAP_MS = 600_000;
|
||||
@@ -246,52 +249,6 @@ export function registerReattachPhaseWork(
|
||||
return current.worker;
|
||||
}
|
||||
|
||||
export type DatabaseSessionWait = {
|
||||
pid: number;
|
||||
applicationName: string | null;
|
||||
state: string | null;
|
||||
waitEventType: string | null;
|
||||
waitEvent: string | null;
|
||||
blockedBy: number[];
|
||||
transactionSeconds: number | null;
|
||||
stateSeconds: number | null;
|
||||
query: string | null;
|
||||
};
|
||||
|
||||
export async function readDatabaseSessionWaits(db: Db): Promise<DatabaseSessionWait[]> {
|
||||
return livenessDb(db).transaction(async tx => {
|
||||
await tx.execute(sql`select set_config('statement_timeout', '5000', true)`);
|
||||
const rows = await tx.execute<Record<string, unknown>>(sql`
|
||||
with sessions as (
|
||||
select pid, application_name, state, wait_event_type, wait_event, pg_blocking_pids(pid) as blocked_by,
|
||||
xact_start, state_change, query
|
||||
from pg_stat_activity
|
||||
where datname = current_database() and pid <> pg_backend_pid() and backend_type = 'client backend'
|
||||
), blockers as (select distinct unnest(blocked_by) as pid from sessions)
|
||||
select pid, application_name, state, wait_event_type, wait_event, blocked_by,
|
||||
extract(epoch from clock_timestamp() - xact_start)::int as transaction_seconds,
|
||||
extract(epoch from clock_timestamp() - state_change)::int as state_seconds,
|
||||
left(regexp_replace(regexp_replace(query, '''(?:[^'']|'''')*''', '?', 'g'), '\\m[0-9][0-9.]*\\M', '?', 'g'), 160) as query
|
||||
from sessions
|
||||
where cardinality(blocked_by) > 0 or pid in (select pid from blockers)
|
||||
or state like 'idle in transaction%'
|
||||
or (xact_start is not null and clock_timestamp() - xact_start > interval '30 seconds')
|
||||
order by xact_start nulls last
|
||||
limit 50`);
|
||||
return rows.map(row => ({
|
||||
pid: Number(row.pid),
|
||||
applicationName: (row.application_name as string | null) ?? null,
|
||||
state: (row.state as string | null) ?? null,
|
||||
waitEventType: (row.wait_event_type as string | null) ?? null,
|
||||
waitEvent: (row.wait_event as string | null) ?? null,
|
||||
blockedBy: Array.isArray(row.blocked_by) ? row.blocked_by.map(Number) : [],
|
||||
transactionSeconds: row.transaction_seconds === null ? null : Number(row.transaction_seconds),
|
||||
stateSeconds: row.state_seconds === null ? null : Number(row.state_seconds),
|
||||
query: (row.query as string | null) ?? null,
|
||||
}));
|
||||
});
|
||||
}
|
||||
|
||||
function authenticatedButUnreached(summary: ReattachPhaseSummary): string[] {
|
||||
return summary.runs.filter(run => run.outcome === "outstanding" && run.authenticatedMs !== null).map(run => run.runId);
|
||||
}
|
||||
|
||||
@@ -48,6 +48,7 @@ import { cancellationResultJson } from "../run-cancellation.js";
|
||||
import { CONTROLLER_LEASE_LOST_FAILURE_CODE } from "../legacy-controller-lease.js";
|
||||
import { isWaitingConversation } from "../agent-conversations.js";
|
||||
import { PROCESS_IDENTITY_RECORDED } from "../native-local-process-stop.js";
|
||||
import { trackRunIntake } from "../run-intake-stall-watch.js";
|
||||
import {
|
||||
hasAcknowledgedNativeReassignmentStopIntent,
|
||||
hasAcknowledgedNativeLifecycleStopIntent,
|
||||
@@ -625,7 +626,7 @@ export async function persistHeartbeatRunProcessMetadata(
|
||||
);
|
||||
const startedAt = new Date(observedStartedAt ?? meta.startedAt);
|
||||
const processStartedAt = Number.isNaN(startedAt.getTime()) ? new Date() : startedAt;
|
||||
return db.transaction(async tx => {
|
||||
return trackRunIntake(runId, "process_identity", PROCESS_IDENTITY_RECORDED, () => db.transaction(async tx => {
|
||||
const [binding] = await tx
|
||||
.select({ companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId })
|
||||
.from(heartbeatRuns)
|
||||
@@ -654,7 +655,7 @@ export async function persistHeartbeatRunProcessMetadata(
|
||||
});
|
||||
return tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
});
|
||||
}));
|
||||
}
|
||||
|
||||
type SkillTestHeartbeatCompletion = {
|
||||
|
||||
@@ -11,6 +11,7 @@ import { NATIVE_SESSION_PROVIDER_EVENT_TYPES } from "./native-runtime/native-ses
|
||||
import os from "node:os";
|
||||
import { currentProcessSpaceHash, UNKNOWN_PROCESS_SPACE } from "./controller-process-identity.js";
|
||||
import { proveOwnerGone } from "./owner-proof.js";
|
||||
import { beginRunIntake } from "./run-intake-stall-watch.js";
|
||||
|
||||
import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED } from "./native-process-start-events.js";
|
||||
export { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED };
|
||||
@@ -593,6 +594,7 @@ export async function releaseOwnedSettledNativeRunProcessesAtShutdown(db: Db, op
|
||||
if (stopped.length === 0) return { candidates: candidates.length, stillRunning: candidates.length, released: 0 };
|
||||
const recorded = JSON.stringify(stopped.map(run => ({ id: run.id, pid: run.processPid, pgid: run.processGroupId,
|
||||
started: run.processStartedAt, started_iso: run.processStartedAtIso })));
|
||||
const intake = stopped.map(run => beginRunIntake(run.id, "write", LOCAL_PROCESS_STOPPED));
|
||||
const released = await db.transaction(async transaction => {
|
||||
const tx = transaction as unknown as Db;
|
||||
await tx.execute(sql.raw(`set local statement_timeout = ${SHUTDOWN_SETTLED_PROCESS_STATEMENT_TIMEOUT_MS}`));
|
||||
@@ -628,7 +630,7 @@ export async function releaseOwnedSettledNativeRunProcessesAtShutdown(db: Db, op
|
||||
Array<{ companyId: string; runId: string; agentId: string }>;
|
||||
if (rows.length > 0) await notifyInstructionCopyStopEvidenceForRuns(tx, rows);
|
||||
return rows;
|
||||
});
|
||||
}).finally(() => { for (const done of intake) done(); });
|
||||
for (const run of released) pendingSettledProcessReleases.delete(run.runId);
|
||||
return { candidates: candidates.length, stillRunning: candidates.length - stopped.length, released: released.length };
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
|
||||
const tracked = vi.hoisted(() => [] as Array<{ runId: string; stage: string; label: string }>);
|
||||
vi.mock("../run-intake-stall-watch.js", async importOriginal => {
|
||||
const actual = await importOriginal<typeof import("../run-intake-stall-watch.js")>();
|
||||
return {
|
||||
...actual,
|
||||
trackRunIntake: (runId: string, stage: string, label: string, work: () => Promise<unknown>) => {
|
||||
tracked.push({ runId, stage, label });
|
||||
return work();
|
||||
},
|
||||
};
|
||||
});
|
||||
|
||||
const { NativeRunCoordinatorStore } = await import("./native-run-coordinator-store.js");
|
||||
|
||||
describe("native run coordinator store intake tracking", () => {
|
||||
it("registers each appended runner event with the run intake stall watch", async () => {
|
||||
const store = new NativeRunCoordinatorStore({} as Db,
|
||||
{ runId: "run-coordinator" } as ConstructorParameters<typeof NativeRunCoordinatorStore>[1]);
|
||||
await expect(store.appendEvent({ eventType: "session.started" } as Parameters<typeof store.appendEvent>[0]))
|
||||
.rejects.toThrow("native_event_schema_invalid");
|
||||
expect(tracked).toEqual([{ runId: "run-coordinator", stage: "received", label: "session.started" }]);
|
||||
});
|
||||
});
|
||||
@@ -10,6 +10,7 @@ import { createHash } from "node:crypto";
|
||||
import { and, desc, eq, inArray, lt } from "drizzle-orm";
|
||||
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { trackRunIntake } from "../run-intake-stall-watch.js";
|
||||
import {
|
||||
heartbeatRunEvents,
|
||||
heartbeatRuns,
|
||||
@@ -207,6 +208,16 @@ export class NativeRunCoordinatorStore {
|
||||
readonly disposition: "committed" | "duplicate";
|
||||
readonly cursor: number;
|
||||
readonly highestContiguousSourceSeq: number;
|
||||
}> {
|
||||
const eventType = (value as { eventType?: unknown }).eventType;
|
||||
return trackRunIntake(this.#binding.runId, "received", typeof eventType === "string" ? eventType : "unknown",
|
||||
() => this.#appendEvent(value));
|
||||
}
|
||||
|
||||
async #appendEvent(value: PrpEvent): Promise<{
|
||||
readonly disposition: "committed" | "duplicate";
|
||||
readonly cursor: number;
|
||||
readonly highestContiguousSourceSeq: number;
|
||||
}> {
|
||||
const validated = validatePrpEvent(value);
|
||||
if (!validated.ok) throw new Error("native_event_schema_invalid");
|
||||
|
||||
@@ -36,6 +36,7 @@ import { redactRunnerApiEvent, redactRunnerApiPayloads } from "./runner-api-log-
|
||||
import { createNativeControllerRecovery, NATIVE_CONTROLLER_RECOVERY_MAX_BYTES, type NativeControllerCompatibility } from "./native-controller-recovery.js";
|
||||
import { eligibleNativeTaskSessionPublication, publishNativeTaskSessionInTransaction, NATIVE_TASK_SESSION_PUBLICATION_KEY,
|
||||
type NativeTaskSessionPublicationContext, type NativeTaskSessionPublicationIssue } from "./native-task-session-publication.js";
|
||||
import { trackRunIntake } from "../run-intake-stall-watch.js";
|
||||
import { lockNativeRunnerCheckpointOwner, nativeRunnerCheckpointOwnerLock, validateNativeRunnerCheckpointOwner, inspectNativeRunnerUpgradeHeader, type NativeRunnerCheckpointOwner } from "./native-runner-checkpoint.js";
|
||||
import { crashPoint } from "@paperclipai/shared/crash-points";
|
||||
import { NATIVE_PROVIDER_DURABLE_EVENT_TYPES } from "./native-retired-backup-source.js";
|
||||
@@ -399,6 +400,12 @@ export class PaperclipControlPlanePort implements ControlPlanePort {
|
||||
}
|
||||
|
||||
async appendEvent(value: NativeRunEvent | PrpEvent, options?: { signal: AbortSignal; deferPostCommitObservers?: true; streaming?: true }) {
|
||||
const eventType = (value as { eventType?: unknown }).eventType;
|
||||
return trackRunIntake(this.#binding.runId, "received", typeof eventType === "string" ? eventType : "unknown",
|
||||
() => this.#appendEvent(value, options));
|
||||
}
|
||||
|
||||
async #appendEvent(value: NativeRunEvent | PrpEvent, options?: { signal: AbortSignal; deferPostCommitObservers?: true; streaming?: true }) {
|
||||
const persistenceStartedAt = Date.now(), persistenceClock = performance.now();
|
||||
this.#assertActive(options);
|
||||
if (!isPrpEvent(value)) throw new Error("native_legacy_event_not_supported");
|
||||
|
||||
@@ -0,0 +1,245 @@
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { agents, closeLivenessDb, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns } from "@paperclipai/db";
|
||||
import { sql } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
|
||||
import type { DatabaseSessionWait } from "./database-session-waits.js";
|
||||
import { allocateHeartbeatRunEventSeq, appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
|
||||
import { persistHeartbeatRunProcessMetadata } from "./heartbeat/run-lifecycle.js";
|
||||
import { beginRunIntake, installRunIntakeStallWatch, stopRunIntakeStallWatch } from "./run-intake-stall-watch.js";
|
||||
|
||||
const STALL_MESSAGE = "runner event intake stalled; pending run writes, database sessions and lock waits";
|
||||
const CLEARED_MESSAGE = "runner event intake stall cleared";
|
||||
|
||||
type StallReport = {
|
||||
eventLoopLagMs: number;
|
||||
operations: Array<{ runId: string; stage: string; label: string; pendingMs: number }>;
|
||||
runs: Array<{ runId: string; pending: number; oldestPendingMs: number; stages: Record<string, number> }>;
|
||||
pool: { maxConnections: number | null; pendingIntake: number };
|
||||
sessions: Array<{ applicationName: string; state: string; sessions: number }>;
|
||||
waits: DatabaseSessionWait[];
|
||||
databaseReadError?: string;
|
||||
};
|
||||
|
||||
function stallReports(warn: { mock: { calls: unknown[][] } }): StallReport[] {
|
||||
return warn.mock.calls.filter(call => call[1] === STALL_MESSAGE).map(call => call[0] as StallReport);
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
stopRunIntakeStallWatch();
|
||||
vi.restoreAllMocks();
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
describe("run intake stall watch", () => {
|
||||
it("does nothing until it is installed", () => {
|
||||
const done = beginRunIntake("run", "received", "session.started");
|
||||
expect(done).toBeTypeOf("function");
|
||||
done();
|
||||
});
|
||||
|
||||
it("reports a stalled run once, throttles the next report, and logs when the stalled write finishes", async () => {
|
||||
vi.useFakeTimers();
|
||||
const warn = vi.spyOn(logger, "warn").mockImplementation(() => logger);
|
||||
const info = vi.spyOn(logger, "info").mockImplementation(() => logger);
|
||||
const unreachable = { transaction: () => Promise.reject(new Error("database unreachable")), $client: { options: { max: 7 } } } as unknown as Db;
|
||||
installRunIntakeStallWatch(unreachable, { stallMs: 1_000, now: () => Date.now() });
|
||||
const first = beginRunIntake("run-a", "process_identity", "native.process_identity_recorded");
|
||||
vi.advanceTimersByTime(100);
|
||||
const queued = beginRunIntake("run-a", "received", "session.started");
|
||||
const healthy = beginRunIntake("run-b", "write", "item.completed");
|
||||
vi.advanceTimersByTime(50);
|
||||
healthy();
|
||||
vi.advanceTimersByTime(50);
|
||||
const other = beginRunIntake("run-c", "write", "turn.started");
|
||||
await vi.advanceTimersByTimeAsync(950);
|
||||
let reports = stallReports(warn);
|
||||
expect(reports).toHaveLength(1);
|
||||
expect(reports[0]!.operations).toEqual([
|
||||
expect.objectContaining({ runId: "run-a", stage: "process_identity", label: "native.process_identity_recorded" }),
|
||||
]);
|
||||
expect(reports[0]!.runs[0]).toMatchObject({ runId: "run-a", pending: 2, stages: { process_identity: 1, received: 1 } });
|
||||
expect(reports[0]!.pool).toMatchObject({ maxConnections: 7, pendingIntake: 3 });
|
||||
expect(reports[0]!.databaseReadError).toBe("database unreachable");
|
||||
|
||||
await vi.advanceTimersByTimeAsync(800);
|
||||
expect(stallReports(warn)).toHaveLength(1);
|
||||
await vi.advanceTimersByTimeAsync(100);
|
||||
reports = stallReports(warn);
|
||||
expect(reports).toHaveLength(2);
|
||||
expect(reports[1]!.operations).toEqual([expect.objectContaining({ runId: "run-c", stage: "write" })]);
|
||||
|
||||
queued();
|
||||
expect(info.mock.calls.filter(call => call[1] === CLEARED_MESSAGE)).toHaveLength(0);
|
||||
first();
|
||||
other();
|
||||
expect(info.mock.calls.filter(call => call[1] === CLEARED_MESSAGE).map(call => call[0])).toEqual([
|
||||
expect.objectContaining({ runId: "run-a", stage: "process_identity" }),
|
||||
expect.objectContaining({ runId: "run-c", stage: "write" }),
|
||||
]);
|
||||
await vi.advanceTimersByTimeAsync(5_000);
|
||||
expect(stallReports(warn)).toHaveLength(2);
|
||||
expect(vi.getTimerCount()).toBe(0);
|
||||
});
|
||||
|
||||
it("reports pending writes when the database read hangs, and stops without waiting for it", async () => {
|
||||
vi.useFakeTimers();
|
||||
const warn = vi.spyOn(logger, "warn").mockImplementation(() => logger);
|
||||
const hung = { transaction: () => new Promise(() => undefined), $client: { options: { max: 3 } } } as unknown as Db;
|
||||
installRunIntakeStallWatch(hung, { stallMs: 1_000, reportTimeoutMs: 500, now: () => Date.now() });
|
||||
const done = beginRunIntake("run-hung", "write", "session.started");
|
||||
await vi.advanceTimersByTimeAsync(1_000);
|
||||
expect(stallReports(warn)).toHaveLength(0);
|
||||
await vi.advanceTimersByTimeAsync(500);
|
||||
const [report] = stallReports(warn);
|
||||
expect(report!.operations).toEqual([expect.objectContaining({ runId: "run-hung", stage: "write" })]);
|
||||
expect(report!.databaseReadError).toBe("database session read timed out after 500 ms");
|
||||
done();
|
||||
const second = beginRunIntake("run-later", "write", "turn.started");
|
||||
await vi.advanceTimersByTimeAsync(1_200);
|
||||
stopRunIntakeStallWatch();
|
||||
second();
|
||||
await vi.advanceTimersByTimeAsync(500);
|
||||
expect(vi.getTimerCount()).toBe(0);
|
||||
expect(stallReports(warn)).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("names every stalled run across reports when more runs stall than one report lists", async () => {
|
||||
vi.useFakeTimers();
|
||||
const warn = vi.spyOn(logger, "warn").mockImplementation(() => logger);
|
||||
const unreachable = { transaction: () => Promise.reject(new Error("database unreachable")), $client: {} } as unknown as Db;
|
||||
installRunIntakeStallWatch(unreachable, { stallMs: 1_000, now: () => Date.now() });
|
||||
const runIds = Array.from({ length: 25 }, (_, index) => `run-${index}`);
|
||||
const pending = runIds.map(runId => beginRunIntake(runId, "write", "item.completed"));
|
||||
await vi.advanceTimersByTimeAsync(1_000);
|
||||
expect(stallReports(warn)).toHaveLength(1);
|
||||
expect(stallReports(warn)[0]!.operations).toHaveLength(20);
|
||||
await vi.advanceTimersByTimeAsync(1_000);
|
||||
const reports = stallReports(warn);
|
||||
expect(reports).toHaveLength(2);
|
||||
expect(reports[1]!.operations).toHaveLength(5);
|
||||
expect(reports.flatMap(report => report.operations.map(operation => operation.runId)).sort()).toEqual([...runIds].sort());
|
||||
for (const done of pending) done();
|
||||
});
|
||||
|
||||
it("keeps no timer once no run write is pending", async () => {
|
||||
vi.useFakeTimers();
|
||||
installRunIntakeStallWatch({} as Db, { stallMs: 1_000, now: () => Date.now() });
|
||||
expect(vi.getTimerCount()).toBe(0);
|
||||
beginRunIntake("run", "write", "item.delta")();
|
||||
await vi.advanceTimersByTimeAsync(1_000);
|
||||
expect(vi.getTimerCount()).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_QUEUE_TEST_DATABASE_URL");
|
||||
const support = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
|
||||
if (!support.supported) console.warn(`Skipping run intake stall database tests: ${support.reason}`);
|
||||
describe.skipIf(!support.supported)("run intake stall watch on a database", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let db: Db;
|
||||
beforeAll(async () => {
|
||||
if (!external) database = await startEmbeddedPostgresTestDatabase("paperclip-run-intake-stall");
|
||||
db = createDb(external ?? database!.connectionString, { maxConnections: 4, applicationName: "paperclip:stall-test-boot" });
|
||||
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
|
||||
afterAll(async () => {
|
||||
if (db) await closeLivenessDb(db, 1);
|
||||
await db?.$client.end();
|
||||
await database?.cleanup();
|
||||
});
|
||||
|
||||
async function seedRun(prefix: string) {
|
||||
const [company] = await db.insert(companies).values({ name: `Intake ${prefix}`, issuePrefix: prefix }).returning();
|
||||
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Runner agent", status: "running",
|
||||
adapterType: "paperclip_runner", adapterConfig: {} }).returning();
|
||||
const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, status: "running",
|
||||
runtimeMode: "native", invocationSource: "assignment", contextSnapshot: {} }).returning();
|
||||
return run!;
|
||||
}
|
||||
|
||||
it("names the idle transaction holding a run row when that run's process identity write waits behind it", async () => {
|
||||
const run = await seedRun("RIS");
|
||||
const warn = vi.spyOn(logger, "warn");
|
||||
const info = vi.spyOn(logger, "info");
|
||||
installRunIntakeStallWatch(db, { stallMs: 300 });
|
||||
let release!: () => void;
|
||||
const held = new Promise<void>(resolve => { release = resolve; });
|
||||
let locked!: (pid: number) => void;
|
||||
const holderPid = new Promise<number>(resolve => { locked = resolve; });
|
||||
const holder = db.transaction(async tx => {
|
||||
const [backend] = await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`);
|
||||
await tx.execute(sql.raw(`select id from heartbeat_runs where id = '${run.id}' and next_event_seq >= 0 for update`));
|
||||
locked(Number(backend!.pid));
|
||||
await held;
|
||||
});
|
||||
const holderBackend = await holderPid;
|
||||
const identity = persistHeartbeatRunProcessMetadata(db, run.id,
|
||||
{ pid: 563, processGroupId: null, startedAt: new Date().toISOString(), targetKind: "remote" });
|
||||
try {
|
||||
await vi.waitFor(() => expect(stallReports(warn)).toHaveLength(1), { timeout: 5_000, interval: 20 });
|
||||
const [report] = stallReports(warn);
|
||||
expect(report!.operations).toEqual([
|
||||
expect.objectContaining({ runId: run.id, stage: "process_identity", label: "native.process_identity_recorded" }),
|
||||
]);
|
||||
expect(report!.runs[0]).toMatchObject({ runId: run.id, stages: { process_identity: 1, write: 1 } });
|
||||
expect(report!.pool.maxConnections).toBe(4);
|
||||
expect(report!.sessions).toEqual(expect.arrayContaining([
|
||||
expect.objectContaining({ applicationName: "paperclip:stall-test-boot", state: "idle in transaction" }),
|
||||
]));
|
||||
expect(report!.waits).toEqual(expect.arrayContaining([
|
||||
expect.objectContaining({ pid: holderBackend, state: "idle in transaction", applicationName: "paperclip:stall-test-boot" }),
|
||||
expect.objectContaining({ waitEventType: "Lock", blockedBy: [holderBackend] }),
|
||||
]));
|
||||
expect(JSON.stringify(report!.waits)).not.toContain(run.id);
|
||||
expect(report!.waits.find(wait => wait.pid === holderBackend)?.query).toContain("where id = ? and next_event_seq >= ? for update");
|
||||
} finally {
|
||||
release();
|
||||
await holder;
|
||||
}
|
||||
await expect(identity).resolves.toMatchObject({ processPid: 563 });
|
||||
expect(info.mock.calls.filter(call => call[1] === CLEARED_MESSAGE).map(call => call[0])).toEqual([
|
||||
expect.objectContaining({ runId: run.id, stage: "process_identity" }),
|
||||
]);
|
||||
});
|
||||
|
||||
it("reports a direct event sequence reservation that waits behind a held run row", async () => {
|
||||
const run = await seedRun("RIQ");
|
||||
const warn = vi.spyOn(logger, "warn");
|
||||
installRunIntakeStallWatch(db, { stallMs: 300 });
|
||||
let release!: () => void;
|
||||
const held = new Promise<void>(resolve => { release = resolve; });
|
||||
let locked!: () => void;
|
||||
const isLocked = new Promise<void>(resolve => { locked = resolve; });
|
||||
const holder = db.transaction(async tx => {
|
||||
await tx.execute(sql.raw(`select id from heartbeat_runs where id = '${run.id}' for update`));
|
||||
locked();
|
||||
await held;
|
||||
});
|
||||
await isLocked;
|
||||
const reserved = allocateHeartbeatRunEventSeq(db, run.id);
|
||||
try {
|
||||
await vi.waitFor(() => expect(stallReports(warn)).toHaveLength(1), { timeout: 5_000, interval: 20 });
|
||||
expect(stallReports(warn)[0]!.operations).toEqual([
|
||||
expect.objectContaining({ runId: run.id, stage: "write", label: "event_sequence" }),
|
||||
]);
|
||||
} finally {
|
||||
release();
|
||||
await holder;
|
||||
}
|
||||
await expect(reserved).resolves.toBeTypeOf("number");
|
||||
});
|
||||
|
||||
it("stays silent while run event writes keep committing", async () => {
|
||||
const run = await seedRun("RIH");
|
||||
const warn = vi.spyOn(logger, "warn");
|
||||
installRunIntakeStallWatch(db, { stallMs: 200 });
|
||||
for (let index = 0; index < 5; index += 1) {
|
||||
await appendHeartbeatRunEvent(db, { companyId: run.companyId, runId: run.id, agentId: run.agentId,
|
||||
eventType: "lifecycle", stream: "system", level: "info", message: `step ${index}` });
|
||||
}
|
||||
await new Promise(resolve => setTimeout(resolve, 400));
|
||||
expect(stallReports(warn)).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,156 @@
|
||||
import { dedicatedDbConnectionStats, type Db } from "@paperclipai/db";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import { readDatabaseSessionReport } from "./database-session-waits.js";
|
||||
|
||||
export const RUN_INTAKE_STALL_MS = 30_000;
|
||||
export const RUN_INTAKE_REPORT_TIMEOUT_MS = 10_000;
|
||||
const REPORTED_OPERATIONS_MAX = 20;
|
||||
const REPORTED_RUNS_MAX = 20;
|
||||
|
||||
export type RunIntakeStage = "received" | "write" | "process_identity";
|
||||
|
||||
type Operation = { runId: string; stage: RunIntakeStage; label: string; startedAtMs: number; stalled: boolean; listed: boolean };
|
||||
|
||||
type Watch = {
|
||||
db: Db;
|
||||
stallMs: number;
|
||||
reportTimeoutMs: number;
|
||||
now: () => number;
|
||||
operations: Map<number, Operation>;
|
||||
nextToken: number;
|
||||
timer: ReturnType<typeof setTimeout> | null;
|
||||
dueAtMs: number;
|
||||
lastReportAtMs: number | null;
|
||||
report: Promise<void> | null;
|
||||
};
|
||||
|
||||
let watch: Watch | null = null;
|
||||
|
||||
const settled = () => undefined;
|
||||
|
||||
export function installRunIntakeStallWatch(db: Db,
|
||||
options: { stallMs?: number; reportTimeoutMs?: number; now?: () => number } = {}): void {
|
||||
if (watch?.timer) clearTimeout(watch.timer);
|
||||
watch = { db, stallMs: options.stallMs ?? RUN_INTAKE_STALL_MS, reportTimeoutMs: options.reportTimeoutMs ?? RUN_INTAKE_REPORT_TIMEOUT_MS,
|
||||
now: options.now ?? Date.now, operations: new Map(),
|
||||
nextToken: 0, timer: null, dueAtMs: 0, lastReportAtMs: null, report: null };
|
||||
}
|
||||
|
||||
export function stopRunIntakeStallWatch(): void {
|
||||
const current = watch;
|
||||
watch = null;
|
||||
if (!current) return;
|
||||
if (current.timer) clearTimeout(current.timer);
|
||||
current.timer = null;
|
||||
}
|
||||
|
||||
export function beginRunIntake(runId: string, stage: RunIntakeStage, label: string): () => void {
|
||||
const current = watch;
|
||||
if (!current) return settled;
|
||||
const token = current.nextToken++;
|
||||
const operation: Operation = { runId, stage, label, startedAtMs: current.now(), stalled: false, listed: false };
|
||||
current.operations.set(token, operation);
|
||||
if (!current.timer && !current.report) arm(current, operation.startedAtMs + current.stallMs);
|
||||
return () => {
|
||||
if (!current.operations.delete(token) || !operation.listed) return;
|
||||
logger.info({ runId, stage, label, pendingMs: current.now() - operation.startedAtMs },
|
||||
"runner event intake stall cleared");
|
||||
};
|
||||
}
|
||||
|
||||
export async function trackRunIntake<T>(runId: string, stage: RunIntakeStage, label: string, work: () => Promise<T>): Promise<T> {
|
||||
const done = beginRunIntake(runId, stage, label);
|
||||
try { return await work(); }
|
||||
finally { done(); }
|
||||
}
|
||||
|
||||
function arm(current: Watch, dueAtMs: number) {
|
||||
const now = current.now();
|
||||
current.dueAtMs = Math.max(dueAtMs, now);
|
||||
current.timer = setTimeout(() => check(current), current.dueAtMs - now);
|
||||
current.timer.unref?.();
|
||||
}
|
||||
|
||||
function nextDueAtMs(current: Watch): number | null {
|
||||
for (const operation of current.operations.values()) {
|
||||
if (!operation.stalled) return operation.startedAtMs + current.stallMs;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function check(current: Watch) {
|
||||
current.timer = null;
|
||||
if (watch !== current) return;
|
||||
const now = current.now();
|
||||
const eventLoopLagMs = Math.max(0, now - current.dueAtMs);
|
||||
if (current.lastReportAtMs !== null && now - current.lastReportAtMs < current.stallMs) {
|
||||
arm(current, current.lastReportAtMs + current.stallMs);
|
||||
return;
|
||||
}
|
||||
const stalledRuns = new Set<string>();
|
||||
for (const operation of current.operations.values()) if (operation.stalled) stalledRuns.add(operation.runId);
|
||||
const newlyStalled: Operation[] = [];
|
||||
for (const operation of current.operations.values()) {
|
||||
if (operation.stalled) continue;
|
||||
if (now - operation.startedAtMs < current.stallMs) break;
|
||||
if (stalledRuns.has(operation.runId)) { operation.stalled = true; continue; }
|
||||
if (newlyStalled.length >= REPORTED_OPERATIONS_MAX) break;
|
||||
operation.stalled = true;
|
||||
stalledRuns.add(operation.runId);
|
||||
newlyStalled.push(operation);
|
||||
}
|
||||
if (newlyStalled.length === 0) {
|
||||
const due = nextDueAtMs(current);
|
||||
if (due !== null) arm(current, due);
|
||||
return;
|
||||
}
|
||||
current.lastReportAtMs = now;
|
||||
const report: Promise<void> = reportStall(current, newlyStalled, eventLoopLagMs)
|
||||
.catch(error => logger.warn({ err: error }, "runner event intake stall report failed"))
|
||||
.finally(() => {
|
||||
if (current.report === report) current.report = null;
|
||||
if (watch !== current || current.timer) return;
|
||||
const due = nextDueAtMs(current);
|
||||
if (due !== null) arm(current, due);
|
||||
});
|
||||
current.report = report;
|
||||
}
|
||||
|
||||
function pendingByRun(current: Watch, now: number) {
|
||||
const runs = new Map<string, { runId: string; pending: number; oldestPendingMs: number; oldestLabel: string;
|
||||
stages: Partial<Record<RunIntakeStage, number>> }>();
|
||||
for (const operation of current.operations.values()) {
|
||||
let run = runs.get(operation.runId);
|
||||
if (!run) {
|
||||
run = { runId: operation.runId, pending: 0, oldestPendingMs: now - operation.startedAtMs, oldestLabel: operation.label, stages: {} };
|
||||
runs.set(operation.runId, run);
|
||||
}
|
||||
run.pending += 1;
|
||||
run.stages[operation.stage] = (run.stages[operation.stage] ?? 0) + 1;
|
||||
}
|
||||
return [...runs.values()].sort((left, right) => right.oldestPendingMs - left.oldestPendingMs).slice(0, REPORTED_RUNS_MAX);
|
||||
}
|
||||
|
||||
async function reportStall(current: Watch, stalled: Operation[], eventLoopLagMs: number) {
|
||||
const now = current.now();
|
||||
for (const operation of stalled) operation.listed = true;
|
||||
const operations = stalled.map(operation => ({ runId: operation.runId, stage: operation.stage, label: operation.label,
|
||||
pendingMs: now - operation.startedAtMs }));
|
||||
const client = current.db.$client as { options?: { max?: unknown } } | undefined;
|
||||
const pool = { maxConnections: typeof client?.options?.max === "number" ? client.options.max : null,
|
||||
pendingIntake: current.operations.size, dedicated: dedicatedDbConnectionStats(current.db) };
|
||||
const runs = pendingByRun(current, now);
|
||||
let expire: ReturnType<typeof setTimeout> | undefined;
|
||||
const expired = new Promise<never>((_resolve, reject) => {
|
||||
expire = setTimeout(() => reject(new Error(`database session read timed out after ${current.reportTimeoutMs} ms`)),
|
||||
current.reportTimeoutMs);
|
||||
expire.unref?.();
|
||||
});
|
||||
const database = await Promise.race([readDatabaseSessionReport(current.db), expired]).then(
|
||||
report => ({ ...report, error: null as string | null }),
|
||||
error => ({ sessions: [], waits: [], error: error instanceof Error ? error.message.slice(0, 160) : "unknown" }),
|
||||
).finally(() => clearTimeout(expire));
|
||||
logger.warn({ stallThresholdMs: current.stallMs, eventLoopLagMs, operations, runs, pool, sessions: database.sessions,
|
||||
waits: database.waits, ...(database.error ? { databaseReadError: database.error } : {}) },
|
||||
"runner event intake stalled; pending run writes, database sessions and lock waits");
|
||||
}
|
||||
Reference in new issue
Block a user