fix(server): name the blocking database sessions whenever a run's event writes stall (#21956)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server stores the events that a runner sends for each run, and
it records the identity of the runner process
> - Sometimes the server stops storing a run's events for minutes, and
the server log stays quiet
> - In two restart-chaos cases, the identity write of the run started
and then did not commit for about 197 s and 480 s
> - The existing database report only fires when the reattach phase ends
at its cap, so it did not fire for the cold-start case
> - This pull request logs the pending writes and the blocking database
sessions whenever any run write stays pending for 30 s
> - The benefit is that the next stall names the session and the query
that hold the run, so the fix can target one proven holder

## Linked Issues or Issue Description

Refs #21946

**What happened?**
Restart-chaos waves on master showed two stalls in event storage.
1. Cold start of an OpenCode run, with no restart. The runner emitted
`session.started` at 22:13:23.7Z. The server stored it at 22:16:40.97Z,
about 197 s later. The second `native.process_identity_recorded` write
of the run started at 22:13:23.706Z, and `runner.turn.submit` took
197270 ms. A second wave showed the same shape (about 191 s). The server
log had nothing in that window.
2. After a hard kill, every API GET timed out from 22:35:24Z to
22:43:23Z, server CPU was 0%, and no run events were stored. The process
identity transaction of the run began at 22:35:22.6Z and committed at
about 22:43:23Z.

**Expected behavior**
When the server cannot store a run's events, the server log names the
cause: which run writes are pending, and which database sessions hold or
wait for locks.

**Steps to reproduce**
1. Start a transaction that locks a `heartbeat_runs` row (`for update`)
and stays idle.
2. Record a process identity for that run
(`persistHeartbeatRunProcessMetadata`).
3. Before this change, nothing is logged while the write waits. After
this change, a warning names the idle holder and the waiting statement
after 30 s. The new database test does this with a 300 ms threshold.

**Relevant logs or output**
Example warning from the new test:
`runner event intake stalled; pending run writes, database sessions and
lock waits` with `operations: [{stage: "process_identity", pendingMs:
301}]` and `waits: [{state: "idle in transaction", query: "select id
from heartbeat_runs where id = ? ... for update"}, {waitEventType:
"Lock", blockedBy: [<holder pid>]}]`.

## What Changed

- New `server/src/services/run-intake-stall-watch.ts`. It keeps an
in-memory list of pending run operations: event receipts
(`PaperclipControlPlanePort.appendEvent`), event writes
(`appendHeartbeatRunEvent`, `appendHeartbeatRunEventBatch`), and process
identity writes (`persistHeartbeatRunProcessMetadata`).
- When an operation stays pending for 30 s, the watch logs one warning
per stalled run. The warning has: the pending operations and their age,
the pending count and oldest age per run, the pool size, the database
sessions per application name and state, the lock waits and holders
(`pg_blocking_pids`, query without literals, cut to 160 characters), and
the event-loop lag. Reports are at most one per 30 s. A second line logs
when a reported operation finishes.
- The watch arms a timer only while an operation is pending. It reads
the database only when it reports, through the private liveness pool
with a 5 s statement timeout.
- The server names its main pool `paperclip:<boot id>` unless
`DATABASE_APPLICATION_NAME` is set. This shows which server process owns
each session after a restart.
- `readDatabaseSessionWaits` moves from the reattach phase to
`server/src/services/database-session-waits.ts`. The reattach phase
re-exports it.
- `@paperclipai/db` exports `databaseClientOptionsFromEnv` and
`DatabaseClientOptions`.
- `doc/DATABASE.md` describes the new log lines and the pool name.
- I did not add `idle_in_transaction_session_timeout` to the main pool.
Some code holds a transaction open on purpose, and a blind timeout could
abort valid work. The new report names idle-in-transaction holders
instead.
- These are server log lines only. This change does not touch Telemetry
or OpenTelemetry.

## Verification

- `cd server && npx vitest run
src/services/run-intake-stall-watch.test.ts` (5 tests). The database
test holds a run row in an idle transaction, starts a process identity
write, and checks that the warning names the holder pid, its query, the
waiting statement, and the pool name. It is red without the tracking in
`run-lifecycle.ts` and `heartbeat-run-events.ts` (`expected [] to have a
length of 1`), and green with it.
- Unit tests with fake timers check one report per stalled run, the 30 s
throttle, the clear line, and that no timer stays armed when nothing is
pending.
- `cd server && npx vitest run
src/services/controller-process-identity.test.ts` checks the pool name
and the `DATABASE_APPLICATION_NAME` override.
- `npx tsc --noEmit` passes for `server` and `packages/db`.
- `pnpm verify` ran locally on the affected tests.

## Risks

- Low risk. Each run event adds one map insert and one map delete. No
database work happens while writes finish within 30 s.
- The main pool `application_name` changes from `paperclip` to
`paperclip:<boot id>`. A dashboard that filters on exactly `paperclip`
must use a prefix match, or the operator can set
`DATABASE_APPLICATION_NAME`.
- A report reads `pg_stat_activity` on the liveness pool. A report
during a full database stall can fail; the warning then has
`databaseReadError` and still lists the pending operations.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-16 22:15:40 -06:00
1 parent 7b8df907d3
commit 3afb2c215b
16 files changed
+602 -59

No files matched your search

+6 -2
View File
@@ -136,6 +136,8 @@ import {
} from "@paperclipai/db";
import detectPort from "detect-port";
import { createApp } from "./app.js";
import { serverDatabaseClientOptions } from "./services/controller-process-identity.js";
import { installRunIntakeStallWatch, stopRunIntakeStallWatch } from "./services/run-intake-stall-watch.js";
import { drainHttpRequestWork } from "./middleware/idle-admission.js";
import { initializeCloudRuntimeConfig } from "./services/cloud-runtime-config.js";
import { loadCloudBootstrapConfig, loadConfig, type Config } from "./config.js";
@@ -537,7 +539,7 @@ async function startServerWithDatabaseTeardown(
const migrationUrl = startupConfig.databaseMigrationUrl ?? startupConfig.databaseUrl;
migrationSummary = await ensureMigrations(migrationUrl, "PostgreSQL");
db = createDb(startupConfig.databaseUrl);
db = createDb(startupConfig.databaseUrl, serverDatabaseClientOptions());
pluginMigrationDb = startupConfig.databaseMigrationUrl ? createDb(startupConfig.databaseMigrationUrl) : db;
logger.info("Using external PostgreSQL via DATABASE_URL/config");
activeDatabaseConnectionString = startupConfig.databaseUrl;
@@ -744,7 +746,7 @@ async function startServerWithDatabaseTeardown(
autoApply: shouldAutoApplyFirstRunMigrations,
});
db = createDb(embeddedConnectionString);
db = createDb(embeddedConnectionString, serverDatabaseClientOptions());
pluginMigrationDb = db;
logger.info("Embedded PostgreSQL ready");
activeDatabaseConnectionString = embeddedConnectionString;
@@ -757,10 +759,12 @@ async function startServerWithDatabaseTeardown(
// by the fail-loud startup path, so no exit leaves pooled backends behind.
const closeDatabaseClients = async () => {
setSentryRunLinkResolver(null);
stopRunIntakeStallWatch();
const clients = pluginMigrationDb === db ? [db] : [db, pluginMigrationDb];
await Promise.all([...clients.map((client) => endDatabaseClient(client, 5)), closeLivenessDb(db, 5), closeCheckpointDb(db, 5), closeFileStorageDb(db, 5), closeDedicatedDbConnections(db, 5)]);
};
startupDatabase.close = closeDatabaseClients;
installRunIntakeStallWatch(db);
startupTimeline.mark("database");
setSentryRunLinkResolver(createSentryRunLinkResolver(db));
await migrateCloudConfiguration(db as any);
@@ -1,11 +1,23 @@
import { spawn } from "node:child_process";
import { once } from "node:events";
import { afterEach, describe, expect, it, vi } from "vitest";
import { processExists, processStartsDefinitelyDiffer } from "./controller-process-identity.js";
import { currentControllerBootId, processExists, processStartsDefinitelyDiffer, serverDatabaseClientOptions } from "./controller-process-identity.js";
import { isPidAlive, isProcessGroupAlive } from "./local-service-supervisor.js";
import { readProcessStartedAt } from "./hot-restart.js";
import { recoveryProcessMayBeAlive } from "./native-runtime/native-session-executor.js";
describe("server database client options", () => {
it("names the main pool after this boot so its database sessions can be told apart from an earlier server's", () => {
const options = serverDatabaseClientOptions({ DATABASE_POOL_MAX: "12" });
expect(options).toEqual({ applicationName: `paperclip:${currentControllerBootId()}`, maxConnections: 12 });
expect(options.applicationName!.length).toBeLessThanOrEqual(63);
});
it("keeps an operator's DATABASE_APPLICATION_NAME", () => {
expect(serverDatabaseClientOptions({ DATABASE_APPLICATION_NAME: "paperclip-web" }).applicationName).toBe("paperclip-web");
});
});
describe("shared process identity proof", () => {
afterEach(() => vi.restoreAllMocks());
@@ -2,6 +2,7 @@ import { createHash, randomUUID } from "node:crypto";
import { execFileSync } from "node:child_process";
import { readFileSync, readlinkSync } from "node:fs";
import { hostname } from "node:os";
import { databaseClientOptionsFromEnv, type DatabaseClientOptions } from "@paperclipai/db";
import { readProcessStartedAt } from "./hot-restart.js";
import { currentWorkspaceWorker, type WorkspaceWorker } from "./native-runtime/native-workspace-platform-proof.js";
@@ -127,6 +128,10 @@ export function currentControllerBootId(): string {
return bootId;
}
export function serverDatabaseClientOptions(env: NodeJS.ProcessEnv = process.env): DatabaseClientOptions {
return { applicationName: `paperclip:${bootId}`, ...databaseClientOptionsFromEnv(env) };
}
const stampListeners = new Set<() => void>();
/** Called (synchronously, without I/O) whenever this process stamps ownership of a record. */
export function onControllerStamp(listener: () => void): () => void {
@@ -0,0 +1,77 @@
import { sql } from "drizzle-orm";
import { livenessDb, type Db } from "@paperclipai/db";
export type DatabaseSessionWait = {
pid: number;
applicationName: string | null;
state: string | null;
waitEventType: string | null;
waitEvent: string | null;
blockedBy: number[];
transactionSeconds: number | null;
stateSeconds: number | null;
query: string | null;
};
export type DatabaseSessionCount = { applicationName: string; state: string; sessions: number };
const SESSION_WAITS_SQL = sql`
with sessions as (
select pid, application_name, state, wait_event_type, wait_event, pg_blocking_pids(pid) as blocked_by,
xact_start, state_change, query
from pg_stat_activity
where datname = current_database() and pid <> pg_backend_pid() and backend_type = 'client backend'
), blockers as (select distinct unnest(blocked_by) as pid from sessions)
select pid, application_name, state, wait_event_type, wait_event, blocked_by,
extract(epoch from clock_timestamp() - xact_start)::int as transaction_seconds,
extract(epoch from clock_timestamp() - state_change)::int as state_seconds,
left(regexp_replace(regexp_replace(query, '''(?:[^'']|'''')*''', '?', 'g'), '\\m[0-9][0-9.]*\\M', '?', 'g'), 160) as query
from sessions
where cardinality(blocked_by) > 0 or pid in (select pid from blockers)
or state like 'idle in transaction%'
or (xact_start is not null and clock_timestamp() - xact_start > interval '30 seconds')
order by xact_start nulls last
limit 50`;
function sessionWait(row: Record<string, unknown>): DatabaseSessionWait {
return {
pid: Number(row.pid),
applicationName: (row.application_name as string | null) ?? null,
state: (row.state as string | null) ?? null,
waitEventType: (row.wait_event_type as string | null) ?? null,
waitEvent: (row.wait_event as string | null) ?? null,
blockedBy: Array.isArray(row.blocked_by) ? row.blocked_by.map(Number) : [],
transactionSeconds: row.transaction_seconds === null ? null : Number(row.transaction_seconds),
stateSeconds: row.state_seconds === null ? null : Number(row.state_seconds),
query: (row.query as string | null) ?? null,
};
}
export async function readDatabaseSessionWaits(db: Db): Promise<DatabaseSessionWait[]> {
return livenessDb(db).transaction(async tx => {
await tx.execute(sql`select set_config('statement_timeout', '5000', true)`);
const rows = await tx.execute<Record<string, unknown>>(SESSION_WAITS_SQL);
return rows.map(sessionWait);
});
}
export async function readDatabaseSessionReport(db: Db, statementTimeoutMs = 5_000): Promise<{
sessions: DatabaseSessionCount[]; waits: DatabaseSessionWait[];
}> {
return livenessDb(db).transaction(async tx => {
await tx.execute(sql`select set_config('statement_timeout', ${String(statementTimeoutMs)}, true)`);
const counts = await tx.execute<Record<string, unknown>>(sql`
select coalesce(application_name, '') as application_name, coalesce(state, '') as state, count(*)::int as sessions
from pg_stat_activity
where datname = current_database() and pid <> pg_backend_pid() and backend_type = 'client backend'
group by 1, 2
order by 3 desc
limit 40`);
const rows = await tx.execute<Record<string, unknown>>(SESSION_WAITS_SQL);
return {
sessions: counts.map(row => ({ applicationName: String(row.application_name), state: String(row.state),
sessions: Number(row.sessions) })),
waits: rows.map(sessionWait),
};
});
}
+28 -2
View File
@@ -13,6 +13,7 @@ import { heartbeatRunEvents, heartbeatRuns, primeDatabaseWorkTransactionId } fro
import { DATABASE_WORK_CHANNEL } from "@paperclipai/db/work-signals";
import { nativeSha256 } from "./native-runtime/canonical.js";
import { isUniqueViolation } from "../db-errors.js";
import { trackRunIntake } from "./run-intake-stall-watch.js";
export const RETRY_EXHAUSTION_MESSAGE_PREFIX = "Bounded retry exhausted";
@@ -88,6 +89,14 @@ export async function allocateHeartbeatRunEventSeq(
db: Db,
runId: string,
runUpdate: HeartbeatRunEventRunUpdate = {},
): Promise<number> {
return trackRunIntake(runId, "write", "event_sequence", () => reserveRunEventSeq(db, runId, runUpdate));
}
async function reserveRunEventSeq(
db: Db,
runId: string,
runUpdate: HeartbeatRunEventRunUpdate,
): Promise<number> {
const [updated] = await db
.update(heartbeatRuns)
@@ -118,6 +127,13 @@ function runEventWakes(input: AppendHeartbeatRunEventInput): RunEventWakes {
export async function appendHeartbeatRunEvent(
db: Db,
input: AppendHeartbeatRunEventInput,
): Promise<AppendHeartbeatRunEventResult> {
return trackRunIntake(input.runId, "write", input.eventType, () => appendRunEvent(db, input));
}
async function appendRunEvent(
db: Db,
input: AppendHeartbeatRunEventInput,
): Promise<AppendHeartbeatRunEventResult> {
// Deduplicated receipts return without allocating, so they cannot carry run changes.
if (input.runUpdate && (input.nativeSource || input.retryExhaustion)) throw new Error("heartbeat_run_event_run_update_unsupported");
@@ -178,13 +194,23 @@ export async function appendHeartbeatRunEventBatch(
): Promise<AppendHeartbeatRunEventResult[]> {
const first = inputs[0];
if (!first) return [];
return trackRunIntake(first.runId, "write", `${first.eventType} x${inputs.length}`,
() => appendRunEventBatch(db, inputs, assertWritable));
}
async function appendRunEventBatch(
db: Db,
inputs: readonly AppendHeartbeatRunEventInput[],
assertWritable?: () => void,
): Promise<AppendHeartbeatRunEventResult[]> {
const first = inputs[0]!;
const source = first.nativeSource;
if (!source || inputs.some(input => !input.nativeSource || input.runUpdate || input.retryExhaustion
|| input.companyId !== first.companyId || input.runId !== first.runId || input.agentId !== first.agentId
|| input.nativeSource.sourceInstanceId !== source.sourceInstanceId)) {
throw new Error("heartbeat_run_event_batch_unsupported");
}
if (inputs.length === 1) return [await appendHeartbeatRunEvent(db, assertWritable ? { ...first, assertWritable } : first)];
if (inputs.length === 1) return [await appendRunEvent(db, assertWritable ? { ...first, assertWritable } : first)];
const wakes = inputs.map(runEventWakes);
if (!wakes.some(wake => Object.values(wake).some(Boolean)) && !is(db, PgTransaction)) {
assertWritable?.();
@@ -596,7 +622,7 @@ async function appendLockedEvent(
await notifyNativeResponsePresentation(tx, { companyId: run.companyId, runId: input.runId });
}
if (wakes.retention) await notifyAttentionRetentionWork(tx, { companyId: input.companyId, sourceKind: "failed_run" });
const seq = await allocateHeartbeatRunEventSeq(tx, input.runId, input.runUpdate);
const seq = await reserveRunEventSeq(tx, input.runId, input.runUpdate ?? {});
const [row] = await tx.insert(heartbeatRunEvents).values({
companyId: input.companyId,
runId: input.runId,
@@ -1,5 +1,6 @@
import { and, eq, inArray, isNotNull, isNull, ne, notLike, or, sql } from "drizzle-orm";
import { dedicatedDbConnectionStats, heartbeatRuns, livenessDb, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { and, eq, inArray, isNotNull, isNull, ne, notLike, or } from "drizzle-orm";
import { dedicatedDbConnectionStats, heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { readDatabaseSessionWaits } from "../database-session-waits.js";
import { logger } from "../../middleware/logger.js";
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js";
@@ -21,6 +22,8 @@ import {
type ReattachPhaseSummary,
} from "./run-admission.js";
export { readDatabaseSessionWaits, type DatabaseSessionWait } from "../database-session-waits.js";
export const REATTACH_PHASE_DEFAULT_CAP_MS = 180_000;
export const REATTACH_PHASE_MIN_CAP_MS = 5_000;
export const REATTACH_PHASE_MAX_CAP_MS = 600_000;
@@ -246,52 +249,6 @@ export function registerReattachPhaseWork(
return current.worker;
}
export type DatabaseSessionWait = {
pid: number;
applicationName: string | null;
state: string | null;
waitEventType: string | null;
waitEvent: string | null;
blockedBy: number[];
transactionSeconds: number | null;
stateSeconds: number | null;
query: string | null;
};
export async function readDatabaseSessionWaits(db: Db): Promise<DatabaseSessionWait[]> {
return livenessDb(db).transaction(async tx => {
await tx.execute(sql`select set_config('statement_timeout', '5000', true)`);
const rows = await tx.execute<Record<string, unknown>>(sql`
with sessions as (
select pid, application_name, state, wait_event_type, wait_event, pg_blocking_pids(pid) as blocked_by,
xact_start, state_change, query
from pg_stat_activity
where datname = current_database() and pid <> pg_backend_pid() and backend_type = 'client backend'
), blockers as (select distinct unnest(blocked_by) as pid from sessions)
select pid, application_name, state, wait_event_type, wait_event, blocked_by,
extract(epoch from clock_timestamp() - xact_start)::int as transaction_seconds,
extract(epoch from clock_timestamp() - state_change)::int as state_seconds,
left(regexp_replace(regexp_replace(query, '''(?:[^'']|'''')*''', '?', 'g'), '\\m[0-9][0-9.]*\\M', '?', 'g'), 160) as query
from sessions
where cardinality(blocked_by) > 0 or pid in (select pid from blockers)
or state like 'idle in transaction%'
or (xact_start is not null and clock_timestamp() - xact_start > interval '30 seconds')
order by xact_start nulls last
limit 50`);
return rows.map(row => ({
pid: Number(row.pid),
applicationName: (row.application_name as string | null) ?? null,
state: (row.state as string | null) ?? null,
waitEventType: (row.wait_event_type as string | null) ?? null,
waitEvent: (row.wait_event as string | null) ?? null,
blockedBy: Array.isArray(row.blocked_by) ? row.blocked_by.map(Number) : [],
transactionSeconds: row.transaction_seconds === null ? null : Number(row.transaction_seconds),
stateSeconds: row.state_seconds === null ? null : Number(row.state_seconds),
query: (row.query as string | null) ?? null,
}));
});
}
function authenticatedButUnreached(summary: ReattachPhaseSummary): string[] {
return summary.runs.filter(run => run.outcome === "outstanding" && run.authenticatedMs !== null).map(run => run.runId);
}
@@ -48,6 +48,7 @@ import { cancellationResultJson } from "../run-cancellation.js";
import { CONTROLLER_LEASE_LOST_FAILURE_CODE } from "../legacy-controller-lease.js";
import { isWaitingConversation } from "../agent-conversations.js";
import { PROCESS_IDENTITY_RECORDED } from "../native-local-process-stop.js";
import { trackRunIntake } from "../run-intake-stall-watch.js";
import {
hasAcknowledgedNativeReassignmentStopIntent,
hasAcknowledgedNativeLifecycleStopIntent,
@@ -625,7 +626,7 @@ export async function persistHeartbeatRunProcessMetadata(
);
const startedAt = new Date(observedStartedAt ?? meta.startedAt);
const processStartedAt = Number.isNaN(startedAt.getTime()) ? new Date() : startedAt;
return db.transaction(async tx => {
return trackRunIntake(runId, "process_identity", PROCESS_IDENTITY_RECORDED, () => db.transaction(async tx => {
const [binding] = await tx
.select({ companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId })
.from(heartbeatRuns)
@@ -654,7 +655,7 @@ export async function persistHeartbeatRunProcessMetadata(
});
return tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId))
.then((rows) => rows[0] ?? null);
});
}));
}
type SkillTestHeartbeatCompletion = {
@@ -11,6 +11,7 @@ import { NATIVE_SESSION_PROVIDER_EVENT_TYPES } from "./native-runtime/native-ses
import os from "node:os";
import { currentProcessSpaceHash, UNKNOWN_PROCESS_SPACE } from "./controller-process-identity.js";
import { proveOwnerGone } from "./owner-proof.js";
import { beginRunIntake } from "./run-intake-stall-watch.js";
import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED } from "./native-process-start-events.js";
export { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED };
@@ -593,6 +594,7 @@ export async function releaseOwnedSettledNativeRunProcessesAtShutdown(db: Db, op
if (stopped.length === 0) return { candidates: candidates.length, stillRunning: candidates.length, released: 0 };
const recorded = JSON.stringify(stopped.map(run => ({ id: run.id, pid: run.processPid, pgid: run.processGroupId,
started: run.processStartedAt, started_iso: run.processStartedAtIso })));
const intake = stopped.map(run => beginRunIntake(run.id, "write", LOCAL_PROCESS_STOPPED));
const released = await db.transaction(async transaction => {
const tx = transaction as unknown as Db;
await tx.execute(sql.raw(`set local statement_timeout = ${SHUTDOWN_SETTLED_PROCESS_STATEMENT_TIMEOUT_MS}`));
@@ -628,7 +630,7 @@ export async function releaseOwnedSettledNativeRunProcessesAtShutdown(db: Db, op
Array<{ companyId: string; runId: string; agentId: string }>;
if (rows.length > 0) await notifyInstructionCopyStopEvidenceForRuns(tx, rows);
return rows;
});
}).finally(() => { for (const done of intake) done(); });
for (const run of released) pendingSettledProcessReleases.delete(run.runId);
return { candidates: candidates.length, stillRunning: candidates.length - stopped.length, released: released.length };
}
@@ -0,0 +1,26 @@
import type { Db } from "@paperclipai/db";
import { describe, expect, it, vi } from "vitest";
const tracked = vi.hoisted(() => [] as Array<{ runId: string; stage: string; label: string }>);
vi.mock("../run-intake-stall-watch.js", async importOriginal => {
const actual = await importOriginal<typeof import("../run-intake-stall-watch.js")>();
return {
...actual,
trackRunIntake: (runId: string, stage: string, label: string, work: () => Promise<unknown>) => {
tracked.push({ runId, stage, label });
return work();
},
};
});
const { NativeRunCoordinatorStore } = await import("./native-run-coordinator-store.js");
describe("native run coordinator store intake tracking", () => {
it("registers each appended runner event with the run intake stall watch", async () => {
const store = new NativeRunCoordinatorStore({} as Db,
{ runId: "run-coordinator" } as ConstructorParameters<typeof NativeRunCoordinatorStore>[1]);
await expect(store.appendEvent({ eventType: "session.started" } as Parameters<typeof store.appendEvent>[0]))
.rejects.toThrow("native_event_schema_invalid");
expect(tracked).toEqual([{ runId: "run-coordinator", stage: "received", label: "session.started" }]);
});
});
@@ -10,6 +10,7 @@ import { createHash } from "node:crypto";
import { and, desc, eq, inArray, lt } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import { trackRunIntake } from "../run-intake-stall-watch.js";
import {
heartbeatRunEvents,
heartbeatRuns,
@@ -207,6 +208,16 @@ export class NativeRunCoordinatorStore {
readonly disposition: "committed" | "duplicate";
readonly cursor: number;
readonly highestContiguousSourceSeq: number;
}> {
const eventType = (value as { eventType?: unknown }).eventType;
return trackRunIntake(this.#binding.runId, "received", typeof eventType === "string" ? eventType : "unknown",
() => this.#appendEvent(value));
}
async #appendEvent(value: PrpEvent): Promise<{
readonly disposition: "committed" | "duplicate";
readonly cursor: number;
readonly highestContiguousSourceSeq: number;
}> {
const validated = validatePrpEvent(value);
if (!validated.ok) throw new Error("native_event_schema_invalid");
@@ -36,6 +36,7 @@ import { redactRunnerApiEvent, redactRunnerApiPayloads } from "./runner-api-log-
import { createNativeControllerRecovery, NATIVE_CONTROLLER_RECOVERY_MAX_BYTES, type NativeControllerCompatibility } from "./native-controller-recovery.js";
import { eligibleNativeTaskSessionPublication, publishNativeTaskSessionInTransaction, NATIVE_TASK_SESSION_PUBLICATION_KEY,
type NativeTaskSessionPublicationContext, type NativeTaskSessionPublicationIssue } from "./native-task-session-publication.js";
import { trackRunIntake } from "../run-intake-stall-watch.js";
import { lockNativeRunnerCheckpointOwner, nativeRunnerCheckpointOwnerLock, validateNativeRunnerCheckpointOwner, inspectNativeRunnerUpgradeHeader, type NativeRunnerCheckpointOwner } from "./native-runner-checkpoint.js";
import { crashPoint } from "@paperclipai/shared/crash-points";
import { NATIVE_PROVIDER_DURABLE_EVENT_TYPES } from "./native-retired-backup-source.js";
@@ -399,6 +400,12 @@ export class PaperclipControlPlanePort implements ControlPlanePort {
}
async appendEvent(value: NativeRunEvent | PrpEvent, options?: { signal: AbortSignal; deferPostCommitObservers?: true; streaming?: true }) {
const eventType = (value as { eventType?: unknown }).eventType;
return trackRunIntake(this.#binding.runId, "received", typeof eventType === "string" ? eventType : "unknown",
() => this.#appendEvent(value, options));
}
async #appendEvent(value: NativeRunEvent | PrpEvent, options?: { signal: AbortSignal; deferPostCommitObservers?: true; streaming?: true }) {
const persistenceStartedAt = Date.now(), persistenceClock = performance.now();
this.#assertActive(options);
if (!isPrpEvent(value)) throw new Error("native_legacy_event_not_supported");
@@ -0,0 +1,245 @@
import type { Db } from "@paperclipai/db";
import { agents, closeLivenessDb, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns } from "@paperclipai/db";
import { sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import { logger } from "../middleware/logger.js";
import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
import type { DatabaseSessionWait } from "./database-session-waits.js";
import { allocateHeartbeatRunEventSeq, appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
import { persistHeartbeatRunProcessMetadata } from "./heartbeat/run-lifecycle.js";
import { beginRunIntake, installRunIntakeStallWatch, stopRunIntakeStallWatch } from "./run-intake-stall-watch.js";
const STALL_MESSAGE = "runner event intake stalled; pending run writes, database sessions and lock waits";
const CLEARED_MESSAGE = "runner event intake stall cleared";
type StallReport = {
eventLoopLagMs: number;
operations: Array<{ runId: string; stage: string; label: string; pendingMs: number }>;
runs: Array<{ runId: string; pending: number; oldestPendingMs: number; stages: Record<string, number> }>;
pool: { maxConnections: number | null; pendingIntake: number };
sessions: Array<{ applicationName: string; state: string; sessions: number }>;
waits: DatabaseSessionWait[];
databaseReadError?: string;
};
function stallReports(warn: { mock: { calls: unknown[][] } }): StallReport[] {
return warn.mock.calls.filter(call => call[1] === STALL_MESSAGE).map(call => call[0] as StallReport);
}
afterEach(() => {
stopRunIntakeStallWatch();
vi.restoreAllMocks();
vi.useRealTimers();
});
describe("run intake stall watch", () => {
it("does nothing until it is installed", () => {
const done = beginRunIntake("run", "received", "session.started");
expect(done).toBeTypeOf("function");
done();
});
it("reports a stalled run once, throttles the next report, and logs when the stalled write finishes", async () => {
vi.useFakeTimers();
const warn = vi.spyOn(logger, "warn").mockImplementation(() => logger);
const info = vi.spyOn(logger, "info").mockImplementation(() => logger);
const unreachable = { transaction: () => Promise.reject(new Error("database unreachable")), $client: { options: { max: 7 } } } as unknown as Db;
installRunIntakeStallWatch(unreachable, { stallMs: 1_000, now: () => Date.now() });
const first = beginRunIntake("run-a", "process_identity", "native.process_identity_recorded");
vi.advanceTimersByTime(100);
const queued = beginRunIntake("run-a", "received", "session.started");
const healthy = beginRunIntake("run-b", "write", "item.completed");
vi.advanceTimersByTime(50);
healthy();
vi.advanceTimersByTime(50);
const other = beginRunIntake("run-c", "write", "turn.started");
await vi.advanceTimersByTimeAsync(950);
let reports = stallReports(warn);
expect(reports).toHaveLength(1);
expect(reports[0]!.operations).toEqual([
expect.objectContaining({ runId: "run-a", stage: "process_identity", label: "native.process_identity_recorded" }),
]);
expect(reports[0]!.runs[0]).toMatchObject({ runId: "run-a", pending: 2, stages: { process_identity: 1, received: 1 } });
expect(reports[0]!.pool).toMatchObject({ maxConnections: 7, pendingIntake: 3 });
expect(reports[0]!.databaseReadError).toBe("database unreachable");
await vi.advanceTimersByTimeAsync(800);
expect(stallReports(warn)).toHaveLength(1);
await vi.advanceTimersByTimeAsync(100);
reports = stallReports(warn);
expect(reports).toHaveLength(2);
expect(reports[1]!.operations).toEqual([expect.objectContaining({ runId: "run-c", stage: "write" })]);
queued();
expect(info.mock.calls.filter(call => call[1] === CLEARED_MESSAGE)).toHaveLength(0);
first();
other();
expect(info.mock.calls.filter(call => call[1] === CLEARED_MESSAGE).map(call => call[0])).toEqual([
expect.objectContaining({ runId: "run-a", stage: "process_identity" }),
expect.objectContaining({ runId: "run-c", stage: "write" }),
]);
await vi.advanceTimersByTimeAsync(5_000);
expect(stallReports(warn)).toHaveLength(2);
expect(vi.getTimerCount()).toBe(0);
});
it("reports pending writes when the database read hangs, and stops without waiting for it", async () => {
vi.useFakeTimers();
const warn = vi.spyOn(logger, "warn").mockImplementation(() => logger);
const hung = { transaction: () => new Promise(() => undefined), $client: { options: { max: 3 } } } as unknown as Db;
installRunIntakeStallWatch(hung, { stallMs: 1_000, reportTimeoutMs: 500, now: () => Date.now() });
const done = beginRunIntake("run-hung", "write", "session.started");
await vi.advanceTimersByTimeAsync(1_000);
expect(stallReports(warn)).toHaveLength(0);
await vi.advanceTimersByTimeAsync(500);
const [report] = stallReports(warn);
expect(report!.operations).toEqual([expect.objectContaining({ runId: "run-hung", stage: "write" })]);
expect(report!.databaseReadError).toBe("database session read timed out after 500 ms");
done();
const second = beginRunIntake("run-later", "write", "turn.started");
await vi.advanceTimersByTimeAsync(1_200);
stopRunIntakeStallWatch();
second();
await vi.advanceTimersByTimeAsync(500);
expect(vi.getTimerCount()).toBe(0);
expect(stallReports(warn)).toHaveLength(2);
});
it("names every stalled run across reports when more runs stall than one report lists", async () => {
vi.useFakeTimers();
const warn = vi.spyOn(logger, "warn").mockImplementation(() => logger);
const unreachable = { transaction: () => Promise.reject(new Error("database unreachable")), $client: {} } as unknown as Db;
installRunIntakeStallWatch(unreachable, { stallMs: 1_000, now: () => Date.now() });
const runIds = Array.from({ length: 25 }, (_, index) => `run-${index}`);
const pending = runIds.map(runId => beginRunIntake(runId, "write", "item.completed"));
await vi.advanceTimersByTimeAsync(1_000);
expect(stallReports(warn)).toHaveLength(1);
expect(stallReports(warn)[0]!.operations).toHaveLength(20);
await vi.advanceTimersByTimeAsync(1_000);
const reports = stallReports(warn);
expect(reports).toHaveLength(2);
expect(reports[1]!.operations).toHaveLength(5);
expect(reports.flatMap(report => report.operations.map(operation => operation.runId)).sort()).toEqual([...runIds].sort());
for (const done of pending) done();
});
it("keeps no timer once no run write is pending", async () => {
vi.useFakeTimers();
installRunIntakeStallWatch({} as Db, { stallMs: 1_000, now: () => Date.now() });
expect(vi.getTimerCount()).toBe(0);
beginRunIntake("run", "write", "item.delta")();
await vi.advanceTimersByTimeAsync(1_000);
expect(vi.getTimerCount()).toBe(0);
});
});
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_QUEUE_TEST_DATABASE_URL");
const support = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
if (!support.supported) console.warn(`Skipping run intake stall database tests: ${support.reason}`);
describe.skipIf(!support.supported)("run intake stall watch on a database", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: Db;
beforeAll(async () => {
if (!external) database = await startEmbeddedPostgresTestDatabase("paperclip-run-intake-stall");
db = createDb(external ?? database!.connectionString, { maxConnections: 4, applicationName: "paperclip:stall-test-boot" });
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
afterAll(async () => {
if (db) await closeLivenessDb(db, 1);
await db?.$client.end();
await database?.cleanup();
});
async function seedRun(prefix: string) {
const [company] = await db.insert(companies).values({ name: `Intake ${prefix}`, issuePrefix: prefix }).returning();
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Runner agent", status: "running",
adapterType: "paperclip_runner", adapterConfig: {} }).returning();
const [run] = await db.insert(heartbeatRuns).values({ companyId: company.id, agentId: agent.id, status: "running",
runtimeMode: "native", invocationSource: "assignment", contextSnapshot: {} }).returning();
return run!;
}
it("names the idle transaction holding a run row when that run's process identity write waits behind it", async () => {
const run = await seedRun("RIS");
const warn = vi.spyOn(logger, "warn");
const info = vi.spyOn(logger, "info");
installRunIntakeStallWatch(db, { stallMs: 300 });
let release!: () => void;
const held = new Promise<void>(resolve => { release = resolve; });
let locked!: (pid: number) => void;
const holderPid = new Promise<number>(resolve => { locked = resolve; });
const holder = db.transaction(async tx => {
const [backend] = await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`);
await tx.execute(sql.raw(`select id from heartbeat_runs where id = '${run.id}' and next_event_seq >= 0 for update`));
locked(Number(backend!.pid));
await held;
});
const holderBackend = await holderPid;
const identity = persistHeartbeatRunProcessMetadata(db, run.id,
{ pid: 563, processGroupId: null, startedAt: new Date().toISOString(), targetKind: "remote" });
try {
await vi.waitFor(() => expect(stallReports(warn)).toHaveLength(1), { timeout: 5_000, interval: 20 });
const [report] = stallReports(warn);
expect(report!.operations).toEqual([
expect.objectContaining({ runId: run.id, stage: "process_identity", label: "native.process_identity_recorded" }),
]);
expect(report!.runs[0]).toMatchObject({ runId: run.id, stages: { process_identity: 1, write: 1 } });
expect(report!.pool.maxConnections).toBe(4);
expect(report!.sessions).toEqual(expect.arrayContaining([
expect.objectContaining({ applicationName: "paperclip:stall-test-boot", state: "idle in transaction" }),
]));
expect(report!.waits).toEqual(expect.arrayContaining([
expect.objectContaining({ pid: holderBackend, state: "idle in transaction", applicationName: "paperclip:stall-test-boot" }),
expect.objectContaining({ waitEventType: "Lock", blockedBy: [holderBackend] }),
]));
expect(JSON.stringify(report!.waits)).not.toContain(run.id);
expect(report!.waits.find(wait => wait.pid === holderBackend)?.query).toContain("where id = ? and next_event_seq >= ? for update");
} finally {
release();
await holder;
}
await expect(identity).resolves.toMatchObject({ processPid: 563 });
expect(info.mock.calls.filter(call => call[1] === CLEARED_MESSAGE).map(call => call[0])).toEqual([
expect.objectContaining({ runId: run.id, stage: "process_identity" }),
]);
});
it("reports a direct event sequence reservation that waits behind a held run row", async () => {
const run = await seedRun("RIQ");
const warn = vi.spyOn(logger, "warn");
installRunIntakeStallWatch(db, { stallMs: 300 });
let release!: () => void;
const held = new Promise<void>(resolve => { release = resolve; });
let locked!: () => void;
const isLocked = new Promise<void>(resolve => { locked = resolve; });
const holder = db.transaction(async tx => {
await tx.execute(sql.raw(`select id from heartbeat_runs where id = '${run.id}' for update`));
locked();
await held;
});
await isLocked;
const reserved = allocateHeartbeatRunEventSeq(db, run.id);
try {
await vi.waitFor(() => expect(stallReports(warn)).toHaveLength(1), { timeout: 5_000, interval: 20 });
expect(stallReports(warn)[0]!.operations).toEqual([
expect.objectContaining({ runId: run.id, stage: "write", label: "event_sequence" }),
]);
} finally {
release();
await holder;
}
await expect(reserved).resolves.toBeTypeOf("number");
});
it("stays silent while run event writes keep committing", async () => {
const run = await seedRun("RIH");
const warn = vi.spyOn(logger, "warn");
installRunIntakeStallWatch(db, { stallMs: 200 });
for (let index = 0; index < 5; index += 1) {
await appendHeartbeatRunEvent(db, { companyId: run.companyId, runId: run.id, agentId: run.agentId,
eventType: "lifecycle", stream: "system", level: "info", message: `step ${index}` });
}
await new Promise(resolve => setTimeout(resolve, 400));
expect(stallReports(warn)).toHaveLength(0);
});
});
@@ -0,0 +1,156 @@
import { dedicatedDbConnectionStats, type Db } from "@paperclipai/db";
import { logger } from "../middleware/logger.js";
import { readDatabaseSessionReport } from "./database-session-waits.js";
export const RUN_INTAKE_STALL_MS = 30_000;
export const RUN_INTAKE_REPORT_TIMEOUT_MS = 10_000;
const REPORTED_OPERATIONS_MAX = 20;
const REPORTED_RUNS_MAX = 20;
export type RunIntakeStage = "received" | "write" | "process_identity";
type Operation = { runId: string; stage: RunIntakeStage; label: string; startedAtMs: number; stalled: boolean; listed: boolean };
type Watch = {
db: Db;
stallMs: number;
reportTimeoutMs: number;
now: () => number;
operations: Map<number, Operation>;
nextToken: number;
timer: ReturnType<typeof setTimeout> | null;
dueAtMs: number;
lastReportAtMs: number | null;
report: Promise<void> | null;
};
let watch: Watch | null = null;
const settled = () => undefined;
export function installRunIntakeStallWatch(db: Db,
options: { stallMs?: number; reportTimeoutMs?: number; now?: () => number } = {}): void {
if (watch?.timer) clearTimeout(watch.timer);
watch = { db, stallMs: options.stallMs ?? RUN_INTAKE_STALL_MS, reportTimeoutMs: options.reportTimeoutMs ?? RUN_INTAKE_REPORT_TIMEOUT_MS,
now: options.now ?? Date.now, operations: new Map(),
nextToken: 0, timer: null, dueAtMs: 0, lastReportAtMs: null, report: null };
}
export function stopRunIntakeStallWatch(): void {
const current = watch;
watch = null;
if (!current) return;
if (current.timer) clearTimeout(current.timer);
current.timer = null;
}
export function beginRunIntake(runId: string, stage: RunIntakeStage, label: string): () => void {
const current = watch;
if (!current) return settled;
const token = current.nextToken++;
const operation: Operation = { runId, stage, label, startedAtMs: current.now(), stalled: false, listed: false };
current.operations.set(token, operation);
if (!current.timer && !current.report) arm(current, operation.startedAtMs + current.stallMs);
return () => {
if (!current.operations.delete(token) || !operation.listed) return;
logger.info({ runId, stage, label, pendingMs: current.now() - operation.startedAtMs },
"runner event intake stall cleared");
};
}
export async function trackRunIntake<T>(runId: string, stage: RunIntakeStage, label: string, work: () => Promise<T>): Promise<T> {
const done = beginRunIntake(runId, stage, label);
try { return await work(); }
finally { done(); }
}
function arm(current: Watch, dueAtMs: number) {
const now = current.now();
current.dueAtMs = Math.max(dueAtMs, now);
current.timer = setTimeout(() => check(current), current.dueAtMs - now);
current.timer.unref?.();
}
function nextDueAtMs(current: Watch): number | null {
for (const operation of current.operations.values()) {
if (!operation.stalled) return operation.startedAtMs + current.stallMs;
}
return null;
}
function check(current: Watch) {
current.timer = null;
if (watch !== current) return;
const now = current.now();
const eventLoopLagMs = Math.max(0, now - current.dueAtMs);
if (current.lastReportAtMs !== null && now - current.lastReportAtMs < current.stallMs) {
arm(current, current.lastReportAtMs + current.stallMs);
return;
}
const stalledRuns = new Set<string>();
for (const operation of current.operations.values()) if (operation.stalled) stalledRuns.add(operation.runId);
const newlyStalled: Operation[] = [];
for (const operation of current.operations.values()) {
if (operation.stalled) continue;
if (now - operation.startedAtMs < current.stallMs) break;
if (stalledRuns.has(operation.runId)) { operation.stalled = true; continue; }
if (newlyStalled.length >= REPORTED_OPERATIONS_MAX) break;
operation.stalled = true;
stalledRuns.add(operation.runId);
newlyStalled.push(operation);
}
if (newlyStalled.length === 0) {
const due = nextDueAtMs(current);
if (due !== null) arm(current, due);
return;
}
current.lastReportAtMs = now;
const report: Promise<void> = reportStall(current, newlyStalled, eventLoopLagMs)
.catch(error => logger.warn({ err: error }, "runner event intake stall report failed"))
.finally(() => {
if (current.report === report) current.report = null;
if (watch !== current || current.timer) return;
const due = nextDueAtMs(current);
if (due !== null) arm(current, due);
});
current.report = report;
}
function pendingByRun(current: Watch, now: number) {
const runs = new Map<string, { runId: string; pending: number; oldestPendingMs: number; oldestLabel: string;
stages: Partial<Record<RunIntakeStage, number>> }>();
for (const operation of current.operations.values()) {
let run = runs.get(operation.runId);
if (!run) {
run = { runId: operation.runId, pending: 0, oldestPendingMs: now - operation.startedAtMs, oldestLabel: operation.label, stages: {} };
runs.set(operation.runId, run);
}
run.pending += 1;
run.stages[operation.stage] = (run.stages[operation.stage] ?? 0) + 1;
}
return [...runs.values()].sort((left, right) => right.oldestPendingMs - left.oldestPendingMs).slice(0, REPORTED_RUNS_MAX);
}
async function reportStall(current: Watch, stalled: Operation[], eventLoopLagMs: number) {
const now = current.now();
for (const operation of stalled) operation.listed = true;
const operations = stalled.map(operation => ({ runId: operation.runId, stage: operation.stage, label: operation.label,
pendingMs: now - operation.startedAtMs }));
const client = current.db.$client as { options?: { max?: unknown } } | undefined;
const pool = { maxConnections: typeof client?.options?.max === "number" ? client.options.max : null,
pendingIntake: current.operations.size, dedicated: dedicatedDbConnectionStats(current.db) };
const runs = pendingByRun(current, now);
let expire: ReturnType<typeof setTimeout> | undefined;
const expired = new Promise<never>((_resolve, reject) => {
expire = setTimeout(() => reject(new Error(`database session read timed out after ${current.reportTimeoutMs} ms`)),
current.reportTimeoutMs);
expire.unref?.();
});
const database = await Promise.race([readDatabaseSessionReport(current.db), expired]).then(
report => ({ ...report, error: null as string | null }),
error => ({ sessions: [], waits: [], error: error instanceof Error ? error.message.slice(0, 160) : "unknown" }),
).finally(() => clearTimeout(expire));
logger.warn({ stallThresholdMs: current.stallMs, eventLoopLagMs, operations, runs, pool, sessions: database.sessions,
waits: database.waits, ...(database.error ? { databaseReadError: database.error } : {}) },
"runner event intake stalled; pending run writes, database sessions and lock waits");
}