Files
PaperClipAI/packages/shared/src/telemetry/client.ts
T
DottaandClaude Opus 5.5 55b692d532 feat(telemetry): send Cloud server telemetry directly to PostHog (#22006)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server sends first-party telemetry events through a collector.
The collector anonymizes the events and copies them into PostHog.
> - Paperclip Cloud is our own service. It needs real IDs (workspace,
user, agent, task) so that a person can go from a PostHog event to the
exact workspace and task.
> - The collector path hashes these IDs. It also cannot link a server
event to the browser session that caused it.
> - This pull request adds a server-owned direct sender for Cloud only.
Self-hosted installs keep the collector path with no change.
> - The benefit is real IDs and session links on Cloud, and no change to
self-hosted privacy.

## Linked Issues or Issue Description

**Subsystem affected**

Server telemetry (`packages/shared/src/telemetry`,
`server/src/telemetry.ts`).

**Problem or motivation**

All Cloud telemetry goes through the collector. The collector hashes the
install ID, agent IDs and task IDs. So, on Cloud, you cannot find the
workspace, user or task of an event. Server events also cannot show in a
browser session replay.

**Proposed solution**

On a Cloud-managed server that has `PAPERCLIP_POSTHOG_PROJECT_TOKEN`,
`PAPERCLIP_CLOUD_STACK_ID` and `PAPERCLIP_POSTHOG_ENVIRONMENT`, send
each telemetry batch straight to PostHog with `posthog-node`. Use the
same event shape that the telemetry backend loader makes today, and add
the real IDs. Keep every other install on the collector.

**Alternatives considered**

Send real IDs through the collector. We did not do this, because the
collector exists to anonymize self-hosted data.

**Roadmap alignment**

This is Cloud analytics infrastructure. It does not change a core
product feature.

## What Changed

- Add `server/src/services/cloud-posthog-config.ts`.
`resolveCloudPostHogConfig()` returns `null` unless the instance is
Cloud-managed and has a token and a stack ID.
`isPostHogDirectDeliveryEnabled()` also requires an environment and the
kill switch on (`PAPERCLIP_POSTHOG_DIRECT` is not `0`).
- Add a narrow `TelemetryTransport` seam to `TelemetryClient` in
`packages/shared`. With a transport, each flushed batch goes to the
transport and never to the collector. Without one, the client does what
it did before. A transport can capture per-event context at track time.
The context does not go into the collector envelope.
- Typed helpers hash `task_id` unless the transport sets
`hashesPrivateRefs: false`. Private `source_ref` values (for example a
private repository name) are customer content, so they stay hashed on
every path, Cloud included, with `source_ref_hashed: true`.
- Add `server/src/services/posthog-direct/`:
- `event-shape.ts` builds PostHog messages with the loader's event
names, metadata keys and key order, the deterministic event uuid
formula, and the unknown-enum fallback. It drops events that the backend
quarantines (unregistered names).
- `enum-fallbacks.generated.ts` holds the backend fallback rules. Enum
values come from the shared generated contract.
- `transport.ts` loads `posthog-node` with dynamic `import()` only on
the first Cloud send.
- `request-context.ts` is Cloud-only Express middleware. It reads only
`X-POSTHOG-SESSION-ID` and `X-POSTHOG-WINDOW-ID`. It ignores
`X-POSTHOG-DISTINCT-ID`. The acting user comes only from the
authenticated request actor. Context stops when the response finishes,
so later background work does not use that user.
- `server/src/telemetry.ts` selects exactly one sender per process. On
start, the Cloud sender sends `$groupidentify` (group `workspace`, key =
stack ID) and one `cloud.workspace_started` event with `install_hash`
(sha256 of the raw install ID). Both go straight to `posthog-node` and
are not in `PAPERCLIP_EVENTS`.
- `distinct_id` is the acting user ID, else `workspace:<stackId>`.
Events without an acting user keep `$process_person_profile: false`.
GeoIP stays off on all server events.
- Shutdown flushes the client and then shuts down `posthog-node`.
- Add `scripts/generate-posthog-direct-fixtures.mjs`. It runs the
telemetry backend's own ingest and loader on one envelope and writes the
golden fixture and the fallback table. The script header tells how to
regenerate them.
- Update `packages/shared/src/telemetry/README.md` with a "Cloud Direct
Delivery" section about the privacy boundary.
- Add `posthog-node@5.38.2` to `server/package.json`. This PR does not
change `pnpm-lock.yaml`, because the lockfile refresh workflow owns it.

Intended differences from the loader output on Cloud: `distinct_id`
(user or workspace, not the HMAC install ID), raw `agent_id`, raw
`task_id` (private `source_ref` stays hashed), and these added
properties: `stack_id`, `company_id` and `user_id` when known,
`environment`, `$groups`, `$session_id` and `$window_id`. The uuid
formula is the same. The uuid is different only when the dimensions are
different (raw `task_id`).

## Verification

- `pnpm verify` returned exit 2 (a global CI input changed: the new
script). I ran focused checks. Required CI supplies the broad coverage.
- Focused tests pass: `packages/shared` `src/telemetry` (6 files, 42
tests). Server: `posthog-direct-telemetry`,
`posthog-direct-event-shape`, `cloud-posthog-config`,
`shared-telemetry-events`, `heartbeat-status-preserving-telemetry`,
`issue-thread-interactions-telemetry`, `native-run-finalizer-telemetry`,
`agent-task-run-telemetry`, `error-handler` (9 files, 115 tests), and
`server-startup-feedback-export` (133 tests).
- `tsc --noEmit` passes for `packages/shared`, `server` and `cli`.
- New tests:
- Config matrix: not Cloud, no token, no stack ID, no environment, kill
switch, browser and replay flags.
- Self-hosted: `posthog-node` is never loaded, the collector gets the
batch, and the envelope bytes are the same as before (with `task_id`
hashed). Tracing headers are ignored.
- Cloud: a private `source_ref` stays hashed and the raw repository name
never reaches the sender.
- Cloud: one batch goes only to PostHog. Real IDs, `$groups`,
`environment` on every event. `cloud.workspace_started` reaches the
PostHog sender with `install_hash`. No other event has `install_hash`.
- Golden: our builder output equals the backend loader output, except
the intended differences above. Key order is the same.
- `distinct_id`: user, agent and anonymous requests. A spoofed
`X-POSTHOG-DISTINCT-ID` never changes `distinct_id` or `user_id`. A bad
session header is dropped. Work after the response ends does not keep
the user (red without the response-end guard, green with it).
- Kill switch and a missing environment both fall back to the collector.
- Local run with a real server and a local capture server (no real
PostHog project):
- Cloud mode (authenticated, local Postgres, `PAPERCLIP_POSTHOG_HOST`
set to the capture server): the capture server got `$groupidentify`,
`cloud.workspace_started`, `server.started` and `goal.created` on
`/batch/`. `goal.created` from a Cloud tenant user had `distinct_id` =
that user, `user_id`, `company_id`, `$session_id`, `$window_id`,
`$groups` and no `$process_person_profile`. The spoofed distinct ID
header was not used. Nothing went to `/ingest`. `SIGTERM` ran the
`telemetry_flush` step and sent the queued events.
- Self-hosted mode (same token variables but no Cloud signal): the
capture server got only `/ingest` with the normal envelope. Nothing went
to `/batch/`.
- Runtime dependency note: `posthog-node` loads with dynamic `import()`
only on Cloud with direct delivery on. The load happens after startup,
outside the boot path. The unit test mocks the package to count loads.
The local run used the real package.

## Risks

- Medium. This is a telemetry change, so a privacy review is requested.
Self-hosted output does not change. The direct path is off unless all
three Cloud variables are set, so a merge does nothing until fleet
config sets them.
- Cloud events with an acting user create PostHog person profiles keyed
by the Paperclip user ID. This is intended under the Cloud terms.
- The fallback table can drift if the backend adds a new normalization
rule. A test fails when a rule does not match the shared enum contract.
Enum values come from the shared contract, so new enum values do not
drift.
- `posthog-node` keeps network failures in its own queue and drops
events that PostHog rejects. The client does not resend these, so events
are not sent twice.
- `pnpm-lock.yaml` is not in this PR. The lockfile refresh workflow adds
`posthog-node` after merge.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local server runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 08:54:32 -06:00

568 lines
22 KiB
TypeScript

import { createHash } from "node:crypto";
import type {
TelemetryConfig,
TelemetryDimensions,
TelemetryEvent,
TelemetryEventDimensions,
TelemetryEventEnvelope,
TelemetryEventName,
TelemetryState,
} from "./types.js";
import { type ResolvedTelemetryCaps, resolveCaps } from "./config.js";
import { PAPERCLIP_EVENTS } from "./generated/paperclip-telemetry.js";
const DEFAULT_ENDPOINTS = [
"https://telemetry.paperclip.ing/ingest",
"https://rusqrrg391.execute-api.us-east-1.amazonaws.com/ingest",
] as const;
// Queue-pressure valve: auto-flush once this many events are buffered. This is
// an in-memory backpressure trigger, independent of the wire caps that
// `chunkForSend` enforces on each POST.
const BATCH_SIZE = 50;
const SEND_TIMEOUT_MS = 5_000;
/**
* Deterministic, key-stable JSON serialization used to derive a content-stable
* `batchId`. Object keys are sorted so two structurally-equal event sets always
* produce the same string regardless of insertion order. Mirrors the
* `stableStringify` exemplar in `external-objects-server.ts`.
*/
function stableStringify(value: unknown): string {
if (Array.isArray(value)) {
return `[${value.map((entry) => stableStringify(entry)).join(",")}]`;
}
if (value && typeof value === "object") {
const record = value as Record<string, unknown>;
return `{${Object.keys(record)
.sort()
.map((key) => `${JSON.stringify(key)}:${stableStringify(record[key])}`)
.join(",")}}`;
}
return JSON.stringify(value);
}
/**
* Parses a `Retry-After` header into milliseconds — defensively, only when
* present and a non-negative number of seconds. The telemetry backend emits no
* `Retry-After` today, so this never fires in practice; it exists so a future
* server hint is honored rather than ignored. The HTTP-date form is
* intentionally not parsed.
*/
function parseRetryAfterMs(response: { headers?: { get?(name: string): string | null } }): number | undefined {
const raw = response.headers?.get?.("retry-after");
if (!raw) return undefined;
const seconds = Number(raw);
if (!Number.isFinite(seconds) || seconds < 0) return undefined;
return Math.round(seconds * 1000);
}
type TrackArgs<K extends TelemetryEventName> =
keyof TelemetryEventDimensions<K> extends never
? [dimensions?: TelemetryEventDimensions<K>]
: [dimensions: TelemetryEventDimensions<K>];
// Length of the truncated hex `batchId`. 32 hex chars = 128 bits — the
// collision-safe floor. Do not lower below 32.
const BATCH_ID_HEX_LENGTH = 32;
/**
* A chunk awaiting retry. `events` + `batchId` are frozen at first send and
* re-sent verbatim on every attempt — re-mixing events would change the content
* the server hashed under this id and trigger a 409. `attempt` is 1-based (the
* value of the attempt about to be made); `nextAttemptAt` is an epoch-ms gate.
* `timerId` is this batch's scheduled retry wake-up (if any) — held on the record
* so it can be cancelled when the batch is evicted from the bounded store,
* otherwise the timer would keep firing for a batch that no longer exists.
*/
/** Outcome of delivering one batch, as classified by the client's retry logic. */
export type TelemetrySendResult =
| { kind: "ok" }
| { kind: "status"; status: number; retryAfterMs?: number }
| { kind: "network" };
/**
* One wire-ready batch handed to a custom transport. `events` pairs each event
* with the opaque context the transport captured when the event was tracked.
*/
export interface TelemetryOutgoingBatch {
app: string;
schemaVersion: string;
installId: string;
version: string;
batchId: string;
events: ReadonlyArray<{ event: TelemetryEvent; context: unknown }>;
}
/**
* Narrow seam for a host that delivers telemetry somewhere other than the
* default collector. When a client has a transport, every flushed batch goes
* to `send` and never to the collector endpoints, so each batch has exactly
* one sender. Without a transport the client behaves exactly as before.
*/
export interface TelemetryTransport {
/**
* Whether typed helpers hash private task ids before emission. The default
* collector path always hashes. Private source refs stay hashed on every
* path, because they are customer content rather than identifiers.
*/
readonly hashesPrivateRefs: boolean;
/** Captures per-event context at track time (for example the acting request). */
captureEventContext?(): unknown;
send(batch: TelemetryOutgoingBatch): Promise<TelemetrySendResult>;
}
interface PendingBatch {
events: TelemetryEvent[];
batchId: string;
attempt: number;
nextAttemptAt: number;
timerId?: ReturnType<typeof setTimeout>;
}
export class TelemetryClient {
private queue: TelemetryEvent[] = [];
private readonly config: TelemetryConfig;
private readonly caps: ResolvedTelemetryCaps;
private readonly stateFactory: () => TelemetryState;
private readonly version: string;
private readonly random: () => number;
private state: TelemetryState | null = null;
private flushInterval: ReturnType<typeof setInterval> | null = null;
// In-memory pending-retry store (best-effort; never persisted). Bounded by
// `maxPendingRetryBatches`. Insertion order == age (oldest at the front).
private pending: PendingBatch[] = [];
private readonly retryTimers = new Set<ReturnType<typeof setTimeout>>();
private readonly transport: TelemetryTransport | null;
// Per-event transport context, kept off the event object so the collector
// envelope stays byte-identical whether or not a transport is configured.
private readonly eventContexts = new WeakMap<TelemetryEvent, unknown>();
constructor(
config: TelemetryConfig,
stateFactory: () => TelemetryState,
version: string,
// Injectable RNG for backoff jitter — defaults to `Math.random`; tests pass
// a seeded function for deterministic backoff. Callers keep the 3-arg form.
random: () => number = Math.random,
// Optional custom delivery. Omitted on self-hosted installs, which keep the
// default collector path unchanged.
transport?: TelemetryTransport,
) {
this.config = config;
this.caps = resolveCaps(config);
this.stateFactory = stateFactory;
this.version = version;
this.random = random;
this.transport = transport ?? null;
}
/**
* Tracks first-party Paperclip telemetry events registered in the generated
* backend event schema.
*/
track<K extends TelemetryEventName>(eventName: K, ...args: TrackArgs<K>): void {
if (!this.isRegisteredEventName(eventName)) return;
const [dimensions] = args;
this.enqueue(eventName, dimensions);
}
/**
* Whether `track` would enqueue this first-party event name rather than drop
* it as unregistered. Lets a caller with an expensive payload (e.g. one that
* needs database reads) skip building dimensions for a proposed event the
* client would discard anyway.
*/
isRegisteredEventName(eventName: string): boolean {
return Object.hasOwn(PAPERCLIP_EVENTS, eventName);
}
/**
* Tracks plugin telemetry bridge events whose names are built dynamically
* from third-party plugin input. The backend accepts only explicitly
* registered plugin events.
*/
trackDynamic(eventName: string, dimensions?: TelemetryDimensions): void {
this.enqueue(eventName, dimensions);
}
private enqueue(eventName: string, dimensions?: object): void {
if (!this.config.enabled) return;
this.getState(); // ensure state is initialised (side-effect: creates state file on first call)
const event: TelemetryEvent = {
name: eventName,
occurredAt: new Date().toISOString(),
dimensions: { ...dimensions } as TelemetryDimensions,
};
if (this.transport?.captureEventContext) {
this.eventContexts.set(event, this.transport.captureEventContext());
}
this.queue.push(event);
if (this.queue.length >= BATCH_SIZE) {
void this.flush();
}
}
async flush(): Promise<void> {
if (!this.config.enabled) return;
// Re-send any due retries first, then send freshly-queued events.
await this.drainPending();
if (this.queue.length === 0) return;
const events = this.queue.splice(0);
for (const chunk of this.chunkForSend(events)) {
await this.attemptSend({
events: chunk,
batchId: this.deriveBatchId(this.getState().installId, chunk),
attempt: 1,
nextAttemptAt: 0,
});
}
}
/**
* Partitions a drained event list into wire-compliant chunks: first by
* `maxEventsPerBatch` (count), then recursively byte-splits any chunk whose
* serialized envelope still exceeds `maxBodyBytes` (halving). A single event
* that alone exceeds `maxBodyBytes` is dropped-and-logged (fail loudly) rather
* than sent over-limit.
*/
private chunkForSend(events: TelemetryEvent[]): TelemetryEvent[][] {
const maxCount = Math.max(1, this.caps.maxEventsPerBatch);
const out: TelemetryEvent[][] = [];
for (let i = 0; i < events.length; i += maxCount) {
this.splitByBytes(events.slice(i, i + maxCount), out);
}
return out;
}
private splitByBytes(chunk: TelemetryEvent[], out: TelemetryEvent[][]): void {
if (chunk.length === 0) return;
const bytes = this.serializedBytes(this.buildEnvelope(chunk));
if (bytes <= this.caps.maxBodyBytes) {
out.push(chunk);
return;
}
if (chunk.length === 1) {
this.warn(
Number.isFinite(bytes)
? `dropping 1 event whose serialized envelope exceeds maxBodyBytes (${this.caps.maxBodyBytes} bytes); event="${chunk[0]?.name}"`
: `dropping 1 event with a non-serializable dimension (circular reference?); event="${chunk[0]?.name}"`,
);
return;
}
const mid = Math.ceil(chunk.length / 2);
this.splitByBytes(chunk.slice(0, mid), out);
this.splitByBytes(chunk.slice(mid), out);
}
private buildEnvelope(events: TelemetryEvent[], batchId?: string): TelemetryEventEnvelope {
const state = this.getState();
return {
app: this.config.app ?? "paperclip",
schemaVersion: this.config.schemaVersion ?? "1",
installId: state.installId,
version: this.version,
events,
batchId: batchId ?? this.deriveBatchId(state.installId, events),
};
}
/**
* Deterministic, salt-free content-hash idempotency key. Hashes
* `{installId, events}` (scopes the key per install, matching the
* server's `contentSha256` scope so cross-install batches never collide on the
* server ledger key) and truncates to 128 bits. Identical input always
* yields the same id, so a retried batch replays idempotently (202) instead of
* double-counting; different events/install yield a different id.
*/
private deriveBatchId(installId: string, events: TelemetryEvent[]): string {
try {
return createHash("sha256")
.update(stableStringify({ installId, events }))
.digest("hex")
.slice(0, BATCH_ID_HEX_LENGTH);
} catch {
// A plugin can pass a circular (or otherwise non-serializable) `dimensions`
// object via `trackDynamic`; `stableStringify` would then recurse until it
// throws a `RangeError`. Fall back to a count-based hash so this call never
// crashes the flush. Such an event can't be JSON-serialized for the wire
// either, so it is dropped-and-logged in `splitByBytes` — this fallback id
// is only ever attached to a batch that is about to be dropped, so its
// weakened idempotency is moot. The single operator-facing signal is the
// drop-and-log warning, not a (recursively-repeated) warning here.
return createHash("sha256")
.update(JSON.stringify({ installId, count: events.length }))
.digest("hex")
.slice(0, BATCH_ID_HEX_LENGTH);
}
}
private serializedBytes(envelope: TelemetryEventEnvelope): number {
try {
return Buffer.byteLength(JSON.stringify(envelope));
} catch {
// A non-serializable event (e.g. a circular `dimensions` object from a
// plugin) can neither be byte-measured nor sent over the wire. Report it
// as effectively unbounded so `splitByBytes` routes it to the existing
// over-limit drop-and-log path instead of throwing out of the flush.
return Number.POSITIVE_INFINITY;
}
}
/**
* Sends one batch (its current `attempt`). On a retryable failure the EXACT
* same events + `batchId` are re-queued with capped, jittered backoff; on a
* terminal failure or after `maxAttempts` the batch is dropped-and-logged.
*/
private async attemptSend(batch: PendingBatch): Promise<void> {
const outcome = this.classifyOutcome(await this.deliver(batch));
if (outcome.kind === "ok") return;
if (outcome.kind === "terminal") {
this.warn(
`dropping batch ${batch.batchId} on terminal response (HTTP ${outcome.status}); ${batch.events.length} event(s) lost`,
);
return;
}
// Retryable (429/502/503/504 or network/timeout).
if (batch.attempt >= this.caps.backoff.maxAttempts) {
this.warn(
`dropping batch ${batch.batchId} after ${batch.attempt} attempt(s); ${batch.events.length} event(s) lost`,
);
return;
}
// Cap the delay at maxDelayMs. `computeBackoffMs` is already capped, but a
// server `Retry-After` hint is not — an out-of-range value could otherwise
// overflow the runtime timer range and be clamped by Node to a near-immediate
// timeout, causing rapid retries. The cap keeps every retry within the
// configured backoff ceiling.
const delayMs = Math.min(
outcome.retryAfterMs ?? this.computeBackoffMs(batch.attempt),
this.caps.backoff.maxDelayMs,
);
this.enqueuePending({
events: batch.events,
batchId: batch.batchId,
attempt: batch.attempt + 1,
nextAttemptAt: Date.now() + delayMs,
});
}
/** Sends one batch through exactly one sender: the transport, or else the collector. */
private async deliver(batch: PendingBatch): Promise<TelemetrySendResult> {
const envelope = this.buildEnvelope(batch.events, batch.batchId);
if (!this.transport) return this.postEnvelope(JSON.stringify(envelope));
try {
return await this.transport.send({
app: envelope.app,
schemaVersion: envelope.schemaVersion,
installId: envelope.installId,
version: envelope.version,
batchId: envelope.batchId,
events: batch.events.map((event) => ({ event, context: this.eventContexts.get(event) })),
});
} catch {
return { kind: "network" };
}
}
private classifyOutcome(
result: TelemetrySendResult,
): { kind: "ok" } | { kind: "retry"; retryAfterMs?: number } | { kind: "terminal"; status: number } {
if (result.kind === "ok") return { kind: "ok" };
if (result.kind === "network") return { kind: "retry" };
if (this.isRetryableStatus(result.status)) {
return { kind: "retry", retryAfterMs: result.retryAfterMs };
}
return { kind: "terminal", status: result.status };
}
private isRetryableStatus(status: number): boolean {
return status === 429 || status === 502 || status === 503 || status === 504;
}
/**
* Capped exponential backoff with symmetric jitter:
* `min(maxDelayMs, baseDelayMs * 2^(attempt-1)) * (1 ± jitterRatio)`, using the
* injected RNG. `attempt` is the failed attempt (1-based).
*/
private computeBackoffMs(attempt: number): number {
const { baseDelayMs, maxDelayMs, jitterRatio } = this.caps.backoff;
const base = Math.min(maxDelayMs, baseDelayMs * 2 ** (attempt - 1));
const jitter = base * jitterRatio * (this.random() * 2 - 1);
return Math.max(0, Math.min(maxDelayMs, Math.round(base + jitter)));
}
/**
* Pushes a batch onto the pending store (bounded), then schedules its retry
* wake-up. On overflow the OLDEST batches (front) are evicted first — newest
* prioritized — and each eviction is logged so no batch is lost silently.
*/
private enqueuePending(batch: PendingBatch): void {
this.pending.push(batch);
const bound = Math.max(0, this.caps.maxPendingRetryBatches);
while (this.pending.length > bound) {
const evicted = this.pending.shift();
// Cancel the evicted batch's scheduled retry so its timer doesn't keep the
// wake-up alive for a batch that is no longer in the store. Without this a
// flush that overflows the bound would strand one live timer per evicted
// batch even though `pending` itself stays bounded.
if (evicted) this.cancelRetryTimer(evicted);
this.warn(
`pending-retry store full (bound=${bound}); evicted oldest batch ${evicted?.batchId}; ${evicted?.events.length ?? 0} event(s) lost`,
);
}
// Only schedule a wake-up if this batch actually survived eviction. When the
// batch is immediately evicted by the bound (e.g. a large failed flush, or
// `maxPendingRetryBatches: 0`) it has no pending work, so scheduling a timer
// for it would strand thousands of no-op timers behind a small bound.
if (this.pending.includes(batch)) {
this.scheduleDrain(batch);
}
}
/** Cancels a pending batch's scheduled retry wake-up, if it has one. */
private cancelRetryTimer(batch: PendingBatch): void {
if (batch.timerId === undefined) return;
clearTimeout(batch.timerId);
this.retryTimers.delete(batch.timerId);
batch.timerId = undefined;
}
private scheduleDrain(batch: PendingBatch): void {
const delay = Math.max(0, batch.nextAttemptAt - Date.now());
const timer = setTimeout(() => {
this.retryTimers.delete(timer);
batch.timerId = undefined;
void this.drainPending();
}, delay);
// Don't keep the process alive for a best-effort retry (CLI exits promptly).
if (typeof timer === "object" && timer !== null && "unref" in timer) {
(timer as { unref(): void }).unref();
}
this.retryTimers.add(timer);
batch.timerId = timer;
}
/** Re-sends every pending batch whose `nextAttemptAt` is due. */
private async drainPending(): Promise<void> {
if (this.pending.length === 0) return;
const now = Date.now();
const due: PendingBatch[] = [];
const waiting: PendingBatch[] = [];
for (const batch of this.pending) {
(batch.nextAttemptAt <= now ? due : waiting).push(batch);
}
this.pending = waiting;
for (const batch of due) {
await this.attemptSend(batch);
}
}
/**
* POSTs a serialized envelope, trying each endpoint in order. Returns the
* definitive outcome: `ok` on a 2xx; the HTTP `status` on a definitive non-2xx
* response (4xx incl. 429, which the shared API gateway fronts for every
* endpoint, so it is authoritative — stop here); or `network` when every
* endpoint threw. A transient upstream 5xx (502/503/504) does NOT stop the
* loop: a sibling endpoint may be healthy, so we fall through to it and only
* surface the last transient status if every endpoint returns one — matching
* the pre-retry loop's endpoint-fallback behavior.
*/
private async postEnvelope(body: string): Promise<TelemetrySendResult> {
const endpoints = this.resolveEndpoints();
let lastTransient: { kind: "status"; status: number; retryAfterMs?: number } | undefined;
for (const endpoint of endpoints) {
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), SEND_TIMEOUT_MS);
try {
const response = await fetch(endpoint, {
method: "POST",
headers: { "Content-Type": "application/json" },
body,
signal: controller.signal,
});
if (response.ok) return { kind: "ok" };
const status = { kind: "status" as const, status: response.status, retryAfterMs: parseRetryAfterMs(response) };
// Transient upstream 5xx: remember it and try the next endpoint.
if (this.isTransientServerStatus(response.status)) {
lastTransient = status;
continue;
}
return status;
} catch {
// Network/timeout on this endpoint — try the next built-in endpoint.
} finally {
clearTimeout(timer);
}
}
// Every endpoint failed: surface the last transient status (still retryable)
// if we saw one, otherwise report a pure network failure.
return lastTransient ?? { kind: "network" };
}
/** Upstream 5xx that a healthy sibling endpoint may still be able to serve. */
private isTransientServerStatus(status: number): boolean {
return status === 502 || status === 503 || status === 504;
}
private warn(message: string): void {
console.warn(`[telemetry] ${message}`);
}
startPeriodicFlush(intervalMs: number = 60_000): void {
if (this.flushInterval) return;
this.flushInterval = setInterval(() => {
void this.flush();
}, intervalMs);
// Allow the process to exit even if the interval is still active
if (typeof this.flushInterval === "object" && "unref" in this.flushInterval) {
this.flushInterval.unref();
}
}
stop(): void {
if (this.flushInterval) {
clearInterval(this.flushInterval);
this.flushInterval = null;
}
for (const timer of this.retryTimers) {
clearTimeout(timer);
}
this.retryTimers.clear();
}
hashPrivateRef(value: string): string {
const state = this.getState();
return createHash("sha256")
.update(state.salt + value)
.digest("hex")
.slice(0, 16);
}
/**
* Whether typed helpers hash private references on this client. Always true
* on the default collector path; only a transport can turn it off.
*/
get privateRefsHashed(): boolean {
return this.transport?.hashesPrivateRefs ?? true;
}
private getState(): TelemetryState {
if (!this.state) {
this.state = this.stateFactory();
}
return this.state;
}
private resolveEndpoints(): readonly string[] {
const configured = this.config.endpoint?.trim();
return configured ? [configured] : DEFAULT_ENDPOINTS;
}
}