mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server stores the events that a runner sends for each run, and it records the identity of the runner process > - Sometimes the server stops storing a run's events for minutes, and the server log stays quiet > - In two restart-chaos cases, the identity write of the run started and then did not commit for about 197 s and 480 s > - The existing database report only fires when the reattach phase ends at its cap, so it did not fire for the cold-start case > - This pull request logs the pending writes and the blocking database sessions whenever any run write stays pending for 30 s > - The benefit is that the next stall names the session and the query that hold the run, so the fix can target one proven holder ## Linked Issues or Issue Description Refs #21946 **What happened?** Restart-chaos waves on master showed two stalls in event storage. 1. Cold start of an OpenCode run, with no restart. The runner emitted `session.started` at 22:13:23.7Z. The server stored it at 22:16:40.97Z, about 197 s later. The second `native.process_identity_recorded` write of the run started at 22:13:23.706Z, and `runner.turn.submit` took 197270 ms. A second wave showed the same shape (about 191 s). The server log had nothing in that window. 2. After a hard kill, every API GET timed out from 22:35:24Z to 22:43:23Z, server CPU was 0%, and no run events were stored. The process identity transaction of the run began at 22:35:22.6Z and committed at about 22:43:23Z. **Expected behavior** When the server cannot store a run's events, the server log names the cause: which run writes are pending, and which database sessions hold or wait for locks. **Steps to reproduce** 1. Start a transaction that locks a `heartbeat_runs` row (`for update`) and stays idle. 2. Record a process identity for that run (`persistHeartbeatRunProcessMetadata`). 3. Before this change, nothing is logged while the write waits. After this change, a warning names the idle holder and the waiting statement after 30 s. The new database test does this with a 300 ms threshold. **Relevant logs or output** Example warning from the new test: `runner event intake stalled; pending run writes, database sessions and lock waits` with `operations: [{stage: "process_identity", pendingMs: 301}]` and `waits: [{state: "idle in transaction", query: "select id from heartbeat_runs where id = ? ... for update"}, {waitEventType: "Lock", blockedBy: [<holder pid>]}]`. ## What Changed - New `server/src/services/run-intake-stall-watch.ts`. It keeps an in-memory list of pending run operations: event receipts (`PaperclipControlPlanePort.appendEvent`), event writes (`appendHeartbeatRunEvent`, `appendHeartbeatRunEventBatch`), and process identity writes (`persistHeartbeatRunProcessMetadata`). - When an operation stays pending for 30 s, the watch logs one warning per stalled run. The warning has: the pending operations and their age, the pending count and oldest age per run, the pool size, the database sessions per application name and state, the lock waits and holders (`pg_blocking_pids`, query without literals, cut to 160 characters), and the event-loop lag. Reports are at most one per 30 s. A second line logs when a reported operation finishes. - The watch arms a timer only while an operation is pending. It reads the database only when it reports, through the private liveness pool with a 5 s statement timeout. - The server names its main pool `paperclip:<boot id>` unless `DATABASE_APPLICATION_NAME` is set. This shows which server process owns each session after a restart. - `readDatabaseSessionWaits` moves from the reattach phase to `server/src/services/database-session-waits.ts`. The reattach phase re-exports it. - `@paperclipai/db` exports `databaseClientOptionsFromEnv` and `DatabaseClientOptions`. - `doc/DATABASE.md` describes the new log lines and the pool name. - I did not add `idle_in_transaction_session_timeout` to the main pool. Some code holds a transaction open on purpose, and a blind timeout could abort valid work. The new report names idle-in-transaction holders instead. - These are server log lines only. This change does not touch Telemetry or OpenTelemetry. ## Verification - `cd server && npx vitest run src/services/run-intake-stall-watch.test.ts` (5 tests). The database test holds a run row in an idle transaction, starts a process identity write, and checks that the warning names the holder pid, its query, the waiting statement, and the pool name. It is red without the tracking in `run-lifecycle.ts` and `heartbeat-run-events.ts` (`expected [] to have a length of 1`), and green with it. - Unit tests with fake timers check one report per stalled run, the 30 s throttle, the clear line, and that no timer stays armed when nothing is pending. - `cd server && npx vitest run src/services/controller-process-identity.test.ts` checks the pool name and the `DATABASE_APPLICATION_NAME` override. - `npx tsc --noEmit` passes for `server` and `packages/db`. - `pnpm verify` ran locally on the affected tests. ## Risks - Low risk. Each run event adds one map insert and one map delete. No database work happens while writes finish within 30 s. - The main pool `application_name` changes from `paperclip` to `paperclip:<boot id>`. A dashboard that filters on exactly `paperclip` must use a prefix match, or the operator can set `DATABASE_APPLICATION_NAME`. - A report reads `pg_stat_activity` on the liveness pool. A report during a full database stall can fail; the warning then has `databaseReadError` and still lists the pending operations. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
222 lines
9.9 KiB
TypeScript
222 lines
9.9 KiB
TypeScript
import { createHash, randomUUID } from "node:crypto";
|
|
import { execFileSync } from "node:child_process";
|
|
import { readFileSync, readlinkSync } from "node:fs";
|
|
import { hostname } from "node:os";
|
|
import { databaseClientOptionsFromEnv, type DatabaseClientOptions } from "@paperclipai/db";
|
|
import { readProcessStartedAt } from "./hot-restart.js";
|
|
import { currentWorkspaceWorker, type WorkspaceWorker } from "./native-runtime/native-workspace-platform-proof.js";
|
|
|
|
/**
|
|
* Identity of the server process that started a provider side effect, saved
|
|
* with its durable record. After a restart, recovery uses it to prove that the
|
|
* starter has exited, so the side effect cannot still be in flight there.
|
|
*/
|
|
export type ControllerDeployment = { worker: WorkspaceWorker; exclusiveVolume: boolean; volumeId: string | null };
|
|
export type ControllerProcess = {
|
|
bootId: string; pid: number; hostHash: string; processStartedAt: string | null; deployment?: ControllerDeployment;
|
|
};
|
|
|
|
export function currentControllerDeployment(env: NodeJS.ProcessEnv = process.env): ControllerDeployment | null {
|
|
const worker = currentWorkspaceWorker(env);
|
|
if (!worker) return null;
|
|
const volumeId = env.RAILWAY_VOLUME_ID?.trim() || null;
|
|
return { worker, exclusiveVolume: Boolean(volumeId && env.RAILWAY_VOLUME_MOUNT_PATH?.trim()), volumeId };
|
|
}
|
|
|
|
function deploymentField(): { deployment?: ControllerDeployment } {
|
|
const deployment = currentControllerDeployment();
|
|
return deployment ? { deployment } : {};
|
|
}
|
|
|
|
function readDeployment(value: unknown): ControllerDeployment | null {
|
|
if (!value || typeof value !== "object" || Array.isArray(value)) return null;
|
|
const record = value as Record<string, unknown>;
|
|
const worker = record.worker as Record<string, unknown> | null | undefined;
|
|
if (!worker || typeof worker !== "object" || worker.provider !== "railway") return null;
|
|
const ids = [worker.projectId, worker.environmentId, worker.serviceId, worker.deploymentId];
|
|
if (ids.some(id => typeof id !== "string" || id.length === 0)) return null;
|
|
return { worker: { provider: "railway", projectId: ids[0] as string, environmentId: ids[1] as string,
|
|
serviceId: ids[2] as string, deploymentId: ids[3] as string },
|
|
exclusiveVolume: record.exclusiveVolume === true,
|
|
volumeId: typeof record.volumeId === "string" ? record.volumeId : null };
|
|
}
|
|
|
|
/**
|
|
* The one boot id of this server process. Native coordinator claims
|
|
* (`native_run_finalizations.controller_boot_id`), legacy controller leases
|
|
* (`heartbeat_runs.controller_boot_id`), sandbox creation intents and teardown
|
|
* claims all stamp this value, so "owned by this process" means the same thing
|
|
* in every table. Records written by an older release carry that release's
|
|
* random per-module ids; those never equal this value, exactly as a record
|
|
* from any earlier boot never did.
|
|
*/
|
|
const bootId = randomUUID();
|
|
export const UNKNOWN_PROCESS_SPACE = "unknown";
|
|
|
|
/**
|
|
* Names the space in which PIDs are comparable: one kernel boot and, on Linux,
|
|
* one PID namespace. Containers that share a hostname but not a PID namespace
|
|
* get different values. When the space cannot be read, no record is judged.
|
|
* A redeploy onto another host or namespace also never matches; such records
|
|
* keep their original deadlines.
|
|
*/
|
|
function processSpaceHash(): string {
|
|
try {
|
|
const parts = [hostname()];
|
|
if (process.platform === "linux") {
|
|
parts.push(readFileSync("/proc/sys/kernel/random/boot_id", "utf8").trim(), readlinkSync("/proc/self/ns/pid"));
|
|
} else if (process.platform === "darwin") {
|
|
// One bounded, read-only kernel identity observation per module. Rounded
|
|
// boot-time estimates can collide across hosts and cannot authorize PID
|
|
// comparisons. Old weak hashes simply stop matching this stronger proof.
|
|
const bootSession = execFileSync("/usr/sbin/sysctl", ["-n", "kern.bootsessionuuid"], {
|
|
encoding: "utf8", timeout: 1_500, maxBuffer: 4_096, windowsHide: true,
|
|
}).trim().toLowerCase();
|
|
if (!/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/.test(bootSession)) return UNKNOWN_PROCESS_SPACE;
|
|
parts.push(bootSession);
|
|
} else {
|
|
return UNKNOWN_PROCESS_SPACE;
|
|
}
|
|
if (parts.some(part => !part)) return UNKNOWN_PROCESS_SPACE;
|
|
return createHash("sha256").update(parts.join("\0")).digest("hex").slice(0, 32);
|
|
} catch {
|
|
return UNKNOWN_PROCESS_SPACE;
|
|
}
|
|
}
|
|
|
|
const hostHash = processSpaceHash();
|
|
|
|
/** OS-observed start time of any PID (epoch ms), or null when it cannot be read. */
|
|
export async function observedProcessStartedAt(pid: number): Promise<number | null> {
|
|
try {
|
|
const startedAt = Date.parse(await readProcessStartedAt(pid) ?? "");
|
|
return Number.isFinite(startedAt) ? startedAt : null;
|
|
} catch {
|
|
return null;
|
|
}
|
|
}
|
|
|
|
// This process's own start time never changes, so it is read from the OS once.
|
|
// On macOS the read runs `ps -o lstart`; uncached, it ran on every native
|
|
// claim, preparation check and lease renewal. A failed read is not cached, so
|
|
// a transient failure is retried by the next caller.
|
|
let ownStartedAt: Promise<number | null> | undefined;
|
|
/** The start time from the last resolved read, for `currentControllerProcessStamp`. */
|
|
let resolvedProcessStartedAt: string | null = null;
|
|
|
|
/** OS-observed start time of this server process (epoch ms), read at most once
|
|
* successfully per process. Null when the OS cannot report it. */
|
|
export function currentProcessStartedAtMs(): Promise<number | null> {
|
|
if (!ownStartedAt) {
|
|
const read = observedProcessStartedAt(process.pid);
|
|
ownStartedAt = read;
|
|
void read.then(value => {
|
|
if (value === null && ownStartedAt === read) ownStartedAt = undefined;
|
|
// Any resolved read also feeds the no-I/O stamp.
|
|
if (value !== null) resolvedProcessStartedAt = new Date(value).toISOString();
|
|
});
|
|
}
|
|
return ownStartedAt;
|
|
}
|
|
|
|
/** Test seam: forget the cached start time so a test can stub the OS read. */
|
|
export function resetCurrentProcessStartedAtForTests(): void {
|
|
ownStartedAt = undefined;
|
|
}
|
|
|
|
export function currentControllerBootId(): string {
|
|
return bootId;
|
|
}
|
|
|
|
export function serverDatabaseClientOptions(env: NodeJS.ProcessEnv = process.env): DatabaseClientOptions {
|
|
return { applicationName: `paperclip:${bootId}`, ...databaseClientOptionsFromEnv(env) };
|
|
}
|
|
|
|
const stampListeners = new Set<() => void>();
|
|
/** Called (synchronously, without I/O) whenever this process stamps ownership of a record. */
|
|
export function onControllerStamp(listener: () => void): () => void {
|
|
stampListeners.add(listener);
|
|
return () => { stampListeners.delete(listener); };
|
|
}
|
|
function notifyStamp() {
|
|
for (const listener of stampListeners) {
|
|
try { listener(); } catch { /* a listener never blocks a stamp */ }
|
|
}
|
|
}
|
|
|
|
export async function currentControllerProcess(): Promise<ControllerProcess> {
|
|
const startedAt = await currentProcessStartedAtMs();
|
|
const iso = startedAt === null ? null : new Date(startedAt).toISOString();
|
|
resolvedProcessStartedAt = iso;
|
|
notifyStamp();
|
|
return { bootId, pid: process.pid, hostHash, processStartedAt: iso, ...deploymentField() };
|
|
}
|
|
|
|
/**
|
|
* This process's identity without any I/O, for stamps written on the turn
|
|
* path. The start time is present only once an earlier call resolved it. A
|
|
* stamp without one still proves exit when its PID is gone in this PID space;
|
|
* a live PID then stays ambiguous, which fails closed.
|
|
*/
|
|
export function currentControllerProcessStamp(): ControllerProcess {
|
|
notifyStamp();
|
|
return { bootId, pid: process.pid, hostHash, processStartedAt: resolvedProcessStartedAt, ...deploymentField() };
|
|
}
|
|
|
|
export function readControllerProcess(value: unknown): ControllerProcess | null {
|
|
if (!value || typeof value !== "object" || Array.isArray(value)) return null;
|
|
const record = value as Record<string, unknown>;
|
|
if (typeof record.bootId !== "string" || typeof record.hostHash !== "string" ||
|
|
typeof record.pid !== "number" || !Number.isInteger(record.pid) || record.pid <= 0) return null;
|
|
const deployment = readDeployment(record.deployment);
|
|
return { bootId: record.bootId, pid: record.pid, hostHash: record.hostHash,
|
|
processStartedAt: typeof record.processStartedAt === "string" ? record.processStartedAt : null,
|
|
...(deployment ? { deployment } : {}) };
|
|
}
|
|
|
|
/**
|
|
* True when some process owns the PID in this PID space. EPERM means the
|
|
* process exists but belongs to another user, so it counts as alive.
|
|
*/
|
|
export function processExists(pid: number): boolean {
|
|
try {
|
|
process.kill(pid, 0);
|
|
return true;
|
|
} catch (error) {
|
|
return (error as NodeJS.ErrnoException | undefined)?.code === "EPERM";
|
|
}
|
|
}
|
|
|
|
/** Process start times closer than this may be the same process. */
|
|
export const PROCESS_START_PRECISION_MS = 1_000;
|
|
|
|
/**
|
|
* True only when two start times for one PID prove different processes.
|
|
* `ps -o lstart` on macOS and BSD has whole-second precision, while a recorded
|
|
* start can come from a millisecond clock, so a sub-second gap is ambiguous
|
|
* and fails closed (same process). A missing or unparsable time proves nothing.
|
|
*/
|
|
export function processStartsDefinitelyDiffer(
|
|
recorded: Date | string | null | undefined,
|
|
observed: Date | string | null | undefined,
|
|
): boolean {
|
|
const left = recorded instanceof Date ? recorded.getTime() : Date.parse(recorded ?? "");
|
|
const right = observed instanceof Date ? observed.getTime() : Date.parse(observed ?? "");
|
|
return Number.isFinite(left) && Number.isFinite(right) && Math.abs(left - right) >= PROCESS_START_PRECISION_MS;
|
|
}
|
|
|
|
/** This process's PID-space hash. An unreadable space is stored as
|
|
* "unknown", which `sameProcessSpace` never accepts. */
|
|
export function currentProcessSpaceHash(): string {
|
|
return hostHash;
|
|
}
|
|
|
|
/**
|
|
* Whether a PID recorded under `recorded` can be judged in this process.
|
|
* True only for the same readable PID space. A different or unreadable space
|
|
* is never proof: a dead PID here says nothing about a process elsewhere.
|
|
*/
|
|
export function sameProcessSpace(recorded: string | null | undefined): boolean {
|
|
return typeof recorded === "string" && hostHash !== UNKNOWN_PROCESS_SPACE && recorded === hostHash;
|
|
}
|