From 27a825da2f960d4042f2fd91bab85112b0d99acf Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Tue, 3 Nov 2026 19:00:47 -0800 Subject: [PATCH] feat(telemetry): track the server OS with the registered contract (#18897) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its first-party telemetry uses a generated event contract and typed helpers. > - The existing events do not identify the server's OS family. > - A proposed event is dropped until the generated contract registers it. > - This PR syncs the registered contract and emits the OS family when an enabled server telemetry client starts. > - The benefit is a way to compare Windows, Linux, and macOS installations while keeping machine details out of the event. ## Linked Issues or Issue Description **Problem or motivation** We cannot compare Paperclip installations by host OS. Counting task or install events does not provide this dimension. **Proposed solution** Send one `server.started` event with Node's raw `process.platform` as `os_platform` each time the server initializes its enabled telemetry client. Compare distinct install identifiers per platform, rather than restart event totals. **Alternatives considered** A proposal-only emitter would send no data. A separate later contract-sync PR would leave the feature incomplete. This PR includes the canonical generated SDK and promotes the emitter together. CPU architecture and OS versions are outside this feature's scope. **Roadmap alignment** This is a narrow telemetry addition to existing server startup, with no new user flow. The full contract copy also adopts the three existing connector proposals described in #13578; their schemas are already registered. The generated file stays an exact copy, so their helpers and tests must be promoted in the same change. ## What Changed - Export `trackServerStarted` and call it once when server telemetry initializes. Repeated initialization returns the same client and emits no duplicate event. - Copy the canonical generated telemetry SDK byte for byte from schema source commit `5369e6b`. Remove the startup proposal marker. Normal opt-out and CI suppression apply before initialization. - Promote the three connector events included in the canonical SDK. Derive enum types from the generated map and preserve raw values. Update their real-client transport and disabled-state tests. - Assign startup to the existing `operational_enum_count` retention class. Document OS privacy, container/WSL interpretation, and distinct-install counting. - Add real-client startup transport tests and server initialization tests, including file opt-out, environment opt-out, and CI suppression. - Keep startup running when telemetry state storage fails. Publish the client and start periodic flushing only after the startup event succeeds. On failure, clean up the client, disable telemetry for the process, and log a generic warning without the storage path or underlying error. ## Verification - `pnpm --filter @paperclipai/shared typecheck`: passed. - `pnpm --filter @paperclipai/shared build`: passed. - `pnpm --filter @paperclipai/shared exec vitest run`: 98 files, 1,055 tests passed. - `pnpm --filter @paperclipai/shared exec vitest run src/telemetry`: 6 files, 39 tests passed. - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/telemetry-startup.test.ts src/__tests__/shared-telemetry-events.test.ts`: 2 files, 19 tests passed. - Exact generated artifact comparison, `git diff --check`, and local secret/PII review passed. - Proposal inventory contains no startup or connector proposals. The repository extractor fails with TypeScript 7's missing compiler API; rerunning it with the installed TypeScript 5.9 compiler API passed. No extractor source change is included. - Initial local server typecheck attempts found missing workspace dependencies and a missing plugin SDK link in this fresh checkout. Dependencies and the local SDK link were repaired; `pnpm --filter @paperclipai/server typecheck` then passed. Full repository typecheck, tests, and build are covered by the PR CI rather than claimed as local passes. Greptile found that a telemetry state-file error could abort startup. New regression tests reproduced this on the previous head with both an injected permission error and the real state store blocked by a file. Both now pass, including no published client, no periodic flushing, no send, and no retry after failure. Shared telemetry tests and server typecheck were rerun after the fix. Full PR CI passed on `a41540b7899bc8a0aedbd78d72f8b69ad1a4b318`, including typecheck, build, all selected test shards, and the final verify gate. [Successful workflow attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41752039595/attempts/2). Attempt 1 was interrupted when the server (1/12) runner received a shutdown signal; its incomplete result was retained in the PR discussion. The same-commit shard retry and dependent gate passed. Greptile reviewed this exact head at 5/5 with all threads resolved. ## Risks This change starts sending one OS family value per server startup when telemetry is enabled. It sends no OS version, architecture, hostname, path, prompt, or machine identifier beyond the existing envelope. Containers and WSL report the runtime's OS. Event totals reflect restart frequency, so use distinct install identifiers for OS adoption analysis. The complete generated SDK also enables the previously registered connector proposals. Those events keep their existing approved payloads; no connection names, credentials, tool arguments, or results are added. Privacy review is requested for both newly active paths. Deploy the registered schema before merging this app change. Live production delivery has not been tested. No schema migration is needed. Revert this PR to stop the new startup emitter and restore the prior contract. ## Model Used - Original implementation: Anthropic Claude Opus 5.5, as recorded in the original commit. - Completion: OpenAI `gpt-6.1-sol`, xhigh reasoning, through Codex in T3 Code; shell, Git, and automated tests. Context window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the feature issue template - [x] The final change and description contain no private issue links or proposal markers - [x] The existing PR branch is preserved to retain its review history - [x] I have run the targeted tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Claude Opus 5.5 Co-authored-by: Paperclip --- packages/shared/src/telemetry/README.md | 18 +++ ...vents.test.ts => connector-events.test.ts} | 33 +++-- packages/shared/src/telemetry/events.ts | 87 ++++++----- .../generated/paperclip-telemetry.ts | 140 +++++++++++++++++- packages/shared/src/telemetry/index.ts | 1 + packages/shared/src/telemetry/retention.ts | 1 + .../telemetry/server-started-event.test.ts | 69 +++++++++ .../src/__tests__/telemetry-startup.test.ts | 113 ++++++++++++++ server/src/telemetry.ts | 30 ++-- 9 files changed, 424 insertions(+), 68 deletions(-) rename packages/shared/src/telemetry/{proposed-connector-events.test.ts => connector-events.test.ts} (75%) create mode 100644 packages/shared/src/telemetry/server-started-event.test.ts create mode 100644 server/src/__tests__/telemetry-startup.test.ts diff --git a/packages/shared/src/telemetry/README.md b/packages/shared/src/telemetry/README.md index da0403c260b..e792827b9f1 100644 --- a/packages/shared/src/telemetry/README.md +++ b/packages/shared/src/telemetry/README.md @@ -65,6 +65,24 @@ provider-supplied error name that may contain a credential. This is a privacy boundary, not enum canonicalization. Test the value reaching the telemetry helper as well as the separate crash-reporting and local logging sinks. +## Server Startup Privacy + +The startup helper emits the raw Node.js operating system family once when an +enabled server telemetry client initializes. It omits OS versions, CPU +architecture, hostnames, and other machine details. The usual telemetry opt-out +and CI suppression apply before client initialization. If state storage fails +at startup, the server disables telemetry for that process and continues to run. +The warning contains no storage path or underlying error. Container and WSL servers +report their runtime platform; this does not identify the underlying physical +host's OS. Count distinct installation identifiers by platform to compare +installations, since restart event counts also reflect restart frequency. + +Contract syncs may also register existing proposals. Remove each adopted +proposal marker and update its helper and transport tests in the same change. +The connector helpers preserve raw enum values and send only catalog connector +keys and the approved lifecycle and invocation dimensions. Connection names, +credentials, tool arguments, and tool results stay out of those events. + ## Interaction Resolver Events `interaction.created` records the interaction kind and whether the create diff --git a/packages/shared/src/telemetry/proposed-connector-events.test.ts b/packages/shared/src/telemetry/connector-events.test.ts similarity index 75% rename from packages/shared/src/telemetry/proposed-connector-events.test.ts rename to packages/shared/src/telemetry/connector-events.test.ts index a8a6826e9de..c060017c7a6 100644 --- a/packages/shared/src/telemetry/proposed-connector-events.test.ts +++ b/packages/shared/src/telemetry/connector-events.test.ts @@ -28,7 +28,7 @@ function makeClient(config?: { enabled?: boolean }) { }; } -function trackAllProposedConnectorEvents(client: TelemetryClient) { +function trackAllConnectorEvents(client: TelemetryClient) { trackConnectionCreated(client, { connector_key: "github", transport: "mcp_remote", @@ -56,28 +56,33 @@ function trackAllProposedConnectorEvents(client: TelemetryClient) { }); } -describe("proposed connector events against the real TelemetryClient", () => { +describe("registered connector events against the real TelemetryClient", () => { afterEach(() => { vi.restoreAllMocks(); vi.unstubAllEnvs(); + vi.unstubAllGlobals(); }); - it("cannot queue or send: state stays untouched and nothing hits the wire", async () => { + it("queues and sends the three registered connector events", async () => { vi.stubGlobal("fetch", vi.fn().mockResolvedValue({ ok: true })); const { client, stateFactory } = makeClient(); - trackAllProposedConnectorEvents(client); + trackAllConnectorEvents(client); await client.flush(); - expect(stateFactory).not.toHaveBeenCalled(); - expect(fetch).not.toHaveBeenCalled(); + expect(stateFactory).toHaveBeenCalledTimes(1); + expect(fetch).toHaveBeenCalledTimes(1); + const body = JSON.parse(String(vi.mocked(fetch).mock.calls[0]?.[1]?.body)); + expect(body.events.map((event: { name: string }) => event.name)).toEqual([ + "connection.created", "connection.updated", "connection.invoked", + ]); }); - it("differential control: a registered event through the same client does send", async () => { + it("batches connector events with other registered events", async () => { vi.stubGlobal("fetch", vi.fn().mockResolvedValue({ ok: true })); const { client } = makeClient(); - trackAllProposedConnectorEvents(client); + trackAllConnectorEvents(client); client.track("project.created", {}); await client.flush(); @@ -86,15 +91,18 @@ describe("proposed connector events against the real TelemetryClient", () => { String((vi.mocked(fetch).mock.calls[0]?.[1] as RequestInit).body), ); expect(body.events).toEqual([ + expect.objectContaining({ name: "connection.created" }), + expect.objectContaining({ name: "connection.updated" }), + expect.objectContaining({ name: "connection.invoked" }), expect.objectContaining({ name: "project.created" }), ]); }); - it("reports the three connector event names as unregistered", () => { + it("reports the three connector event names as registered", () => { const { client } = makeClient(); - expect(client.isRegisteredEventName("connection.created")).toBe(false); - expect(client.isRegisteredEventName("connection.updated")).toBe(false); - expect(client.isRegisteredEventName("connection.invoked")).toBe(false); + expect(client.isRegisteredEventName("connection.created")).toBe(true); + expect(client.isRegisteredEventName("connection.updated")).toBe(true); + expect(client.isRegisteredEventName("connection.invoked")).toBe(true); expect(client.isRegisteredEventName("project.created")).toBe(true); }); @@ -102,6 +110,7 @@ describe("proposed connector events against the real TelemetryClient", () => { vi.stubGlobal("fetch", vi.fn().mockResolvedValue({ ok: true })); const { client, stateFactory } = makeClient({ enabled: false }); + trackAllConnectorEvents(client); client.track("project.created", {}); await client.flush(); diff --git a/packages/shared/src/telemetry/events.ts b/packages/shared/src/telemetry/events.ts index 71f78bb2f3f..78f2031313f 100644 --- a/packages/shared/src/telemetry/events.ts +++ b/packages/shared/src/telemetry/events.ts @@ -156,70 +156,69 @@ export function trackAgentTaskRun( export function trackConnectionCreated( client: TelemetryClient, dims: { - connector_key: RawDimension<"custom">; - transport: RawDimension<"mcp_remote" | "rest_api" | "local_stdio">; - auth_kind: RawDimension<"oauth" | "api_key" | "none">; - setup_flow: RawDimension<"gallery" | "api" | "example">; - status: RawDimension<"draft" | "active" | "disabled" | "archived">; + connector_key: string; + transport: RawDimension; + auth_kind: RawDimension; + setup_flow: RawDimension; + status: RawDimension; enabled: boolean; }, ): void { - client.track( - // @ts-expect-error -- proposed-telemetry(https://github.com/paperclipai/paperclip/issues/13578): measure which catalog connectors installations create connections for - "connection.created", - dims, - ); + client.track("connection.created", { + ...dims, + transport: asEventDimension(dims.transport), + auth_kind: asEventDimension(dims.auth_kind), + setup_flow: asEventDimension(dims.setup_flow), + status: asEventDimension(dims.status), + }); } export function trackConnectionUpdated( client: TelemetryClient, dims: { - connector_key: RawDimension<"custom">; - transport: RawDimension<"mcp_remote" | "rest_api" | "local_stdio">; - auth_kind: RawDimension<"oauth" | "api_key" | "none">; - change_source: RawDimension< - | "api" - | "gallery" - | "oauth_callback" - | "credential_refresh" - | "archive" - | "example" - >; - previous_status: RawDimension<"draft" | "active" | "disabled" | "archived">; - status: RawDimension<"draft" | "active" | "disabled" | "archived">; + connector_key: string; + transport: RawDimension; + auth_kind: RawDimension; + change_source: RawDimension; + previous_status: RawDimension; + status: RawDimension; previous_enabled: boolean; enabled: boolean; }, ): void { - client.track( - // @ts-expect-error -- proposed-telemetry(https://github.com/paperclipai/paperclip/issues/13578): measure connector lifecycle transitions (configured, paused, archived) after creation - "connection.updated", - dims, - ); + client.track("connection.updated", { + ...dims, + transport: asEventDimension(dims.transport), + auth_kind: asEventDimension(dims.auth_kind), + change_source: asEventDimension(dims.change_source), + previous_status: asEventDimension(dims.previous_status), + status: asEventDimension(dims.status), + }); } export function trackConnectionInvoked( client: TelemetryClient, dims: { - connector_key: RawDimension<"custom">; - transport: RawDimension<"mcp_remote" | "rest_api" | "local_stdio">; - status: RawDimension< - | "succeeded" - | "failed" - | "denied" - | "cancelled" - | "timed_out" - | "rate_limited" - >; - origin: RawDimension<"setup_test" | "agent" | "user" | "system" | "plugin">; + connector_key: string; + transport: RawDimension; + status: RawDimension; + origin: RawDimension; duration_seconds?: number; }, ): void { - client.track( - // @ts-expect-error -- proposed-telemetry(https://github.com/paperclipai/paperclip/issues/13578): measure whether connected connectors are successfully used and where invocations fail; records completed invocation attempts and their terminal status, never invocation starts - "connection.invoked", - dims, - ); + client.track("connection.invoked", { + ...dims, + transport: asEventDimension(dims.transport), + status: asEventDimension(dims.status), + origin: asEventDimension(dims.origin), + }); +} + +export function trackServerStarted( + client: TelemetryClient, + dims: { os_platform: RawDimension }, +): void { + client.track("server.started", { os_platform: asEventDimension(dims.os_platform) }); } export function trackErrorHandlerCrash( diff --git a/packages/shared/src/telemetry/generated/paperclip-telemetry.ts b/packages/shared/src/telemetry/generated/paperclip-telemetry.ts index 0502b6e2a1c..b44d37d3428 100644 --- a/packages/shared/src/telemetry/generated/paperclip-telemetry.ts +++ b/packages/shared/src/telemetry/generated/paperclip-telemetry.ts @@ -37,6 +37,34 @@ source_ref?: string source_ref_hashed?: boolean } +export interface PaperclipConnectionCreatedDimensions { +auth_kind: ("oauth" | "api_key" | "none") +connector_key: string +enabled: boolean +setup_flow: ("gallery" | "api" | "example") +status: ("draft" | "active" | "disabled" | "archived") +transport: ("mcp_remote" | "rest_api" | "local_stdio") +} + +export interface PaperclipConnectionInvokedDimensions { +connector_key: string +duration_seconds?: number +origin: ("setup_test" | "agent" | "user" | "system" | "plugin") +status: ("succeeded" | "failed" | "denied" | "cancelled" | "timed_out" | "rate_limited") +transport: ("mcp_remote" | "rest_api" | "local_stdio") +} + +export interface PaperclipConnectionUpdatedDimensions { +auth_kind: ("oauth" | "api_key" | "none") +change_source: ("api" | "gallery" | "oauth_callback" | "credential_refresh" | "archive" | "example") +connector_key: string +enabled: boolean +previous_enabled: boolean +previous_status: ("draft" | "active" | "disabled" | "archived") +status: ("draft" | "active" | "disabled" | "archived") +transport: ("mcp_remote" | "rest_api" | "local_stdio") +} + export interface PaperclipErrorHandlerCrashDimensions { error_code: string } @@ -58,7 +86,6 @@ interaction_kind: ("suggest_tasks" | "ask_user_questions" | "request_confirmatio used_deprecated_resolver_policy_alias: boolean } -/** Emit only after the complete resolution transaction commits, including response provenance. */ export interface PaperclipInteractionResolvedDimensions { interaction_kind: ("suggest_tasks" | "ask_user_questions" | "request_confirmation" | "request_checkbox_confirmation" | "request_item_verdicts" | "other") status: ("accepted" | "rejected" | "answered" | "cancelled" | "expired" | "failed" | "other") @@ -95,9 +122,11 @@ source: ("schedule" | "manual" | "api" | "webhook" | "other") status: ("received" | "coalesced" | "skipped" | "issue_created" | "completed" | "failed" | "other") } +export interface PaperclipServerStartedDimensions { +os_platform: ("aix" | "android" | "darwin" | "freebsd" | "haiku" | "linux" | "openbsd" | "sunos" | "win32" | "cygwin" | "netbsd" | "other") +} + export interface PaperclipSkillImportedDimensions { -/** skill_ref may identify a known public catalog source only. Omit it for - * GitHub-synced sources, which may be private; never emit repository paths or names. */ source_type: ("local_path" | "github" | "url" | "catalog" | "skills_sh" | "unknown") skill_ref?: string } @@ -108,6 +137,9 @@ export type PaperclipEventName = | "agent.task_completed" | "agent.task_run" | "company.imported" + | "connection.created" + | "connection.invoked" + | "connection.updated" | "error.handler_crash" | "goal.created" | "install.completed" @@ -117,6 +149,7 @@ export type PaperclipEventName = | "project.created" | "routine.created" | "routine.run" + | "server.started" | "skill.imported"; export interface EventDimensionsMap { @@ -125,6 +158,9 @@ export interface EventDimensionsMap { "agent.task_completed": PaperclipAgentTaskCompletedDimensions; "agent.task_run": PaperclipAgentTaskRunDimensions; "company.imported": PaperclipCompanyImportedDimensions; + "connection.created": PaperclipConnectionCreatedDimensions; + "connection.invoked": PaperclipConnectionInvokedDimensions; + "connection.updated": PaperclipConnectionUpdatedDimensions; "error.handler_crash": PaperclipErrorHandlerCrashDimensions; "goal.created": PaperclipGoalCreatedDimensions; "install.completed": PaperclipInstallCompletedDimensions; @@ -134,6 +170,7 @@ export interface EventDimensionsMap { "project.created": PaperclipProjectCreatedDimensions; "routine.created": PaperclipRoutineCreatedDimensions; "routine.run": PaperclipRoutineRunDimensions; + "server.started": PaperclipServerStartedDimensions; "skill.imported": PaperclipSkillImportedDimensions; } @@ -143,6 +180,9 @@ export const PAPERCLIP_EVENTS = { "agent.task_completed": "agent.task_completed", "agent.task_run": "agent.task_run", "company.imported": "company.imported", + "connection.created": "connection.created", + "connection.invoked": "connection.invoked", + "connection.updated": "connection.updated", "error.handler_crash": "error.handler_crash", "goal.created": "goal.created", "install.completed": "install.completed", @@ -152,6 +192,7 @@ export const PAPERCLIP_EVENTS = { "project.created": "project.created", "routine.created": "routine.created", "routine.run": "routine.run", + "server.started": "server.started", "skill.imported": "skill.imported", } as const; @@ -277,6 +318,83 @@ export const PAPERCLIP_ENUM_DESCRIPTIONS = { "unknown": "Source type could not be classified." } }, + "connection.created": { + "auth_kind": { + "oauth": "Connection authenticates with an OAuth flow.", + "api_key": "Connection authenticates with a stored API key.", + "none": "Connection requires no authentication." + }, + "setup_flow": { + "gallery": "Connection was created through the app-gallery setup flow.", + "api": "Connection was created through the connections API.", + "example": "Connection was created or upserted as a bundled example connection." + }, + "status": { + "draft": "Connection row exists but setup has not produced a usable configuration yet.", + "active": "Connection is fully configured and available for use.", + "disabled": "Connection is disabled and excluded from tool use.", + "archived": "Connection is archived and excluded from tool use." + }, + "transport": { + "mcp_remote": "Connection talks to a remote MCP server over HTTP.", + "rest_api": "Connection calls the provider's REST API directly.", + "local_stdio": "Connection runs a local MCP server over stdio." + } + }, + "connection.invoked": { + "origin": { + "setup_test": "A human ran the connection's setup test, identified by durable invocation columns: user actor with no run, issue, or gateway reference and a connection id.", + "agent": "An agent run invoked the tool.", + "user": "A human invoked the tool outside the setup-test shape.", + "system": "A system-initiated process invoked the tool.", + "plugin": "A plugin invoked the tool." + }, + "status": { + "succeeded": "The tool invocation completed successfully.", + "failed": "The tool invocation reached a terminal failure.", + "denied": "The tool invocation was denied by policy or by a human approval decision.", + "cancelled": "The tool invocation was cancelled before completion. Declared in the application status enum; no current application code path writes this value.", + "timed_out": "The tool invocation exceeded its time limit.", + "rate_limited": "The tool invocation was rejected by rate limiting." + }, + "transport": { + "mcp_remote": "Connection talks to a remote MCP server over HTTP.", + "rest_api": "Connection calls the provider's REST API directly.", + "local_stdio": "Connection runs a local MCP server over stdio." + } + }, + "connection.updated": { + "auth_kind": { + "oauth": "Connection authenticates with an OAuth flow.", + "api_key": "Connection authenticates with a stored API key.", + "none": "Connection requires no authentication." + }, + "change_source": { + "api": "A connections API update changed the persisted lifecycle state.", + "gallery": "An app-gallery setup step (finish, OAuth access finalization, or reconnect; Apps gallery and inline task cards) changed the persisted lifecycle state.", + "oauth_callback": "An OAuth callback completion (provider OAuth, Paperclip Cloud connector, or Vercel Connect) changed the persisted lifecycle state.", + "credential_refresh": "A credential refresh changed the persisted lifecycle state.", + "archive": "An archive operation changed the persisted lifecycle state.", + "example": "An example-connection upsert changed the persisted lifecycle state." + }, + "previous_status": { + "draft": "Connection row exists but setup has not produced a usable configuration yet.", + "active": "Connection is fully configured and available for use.", + "disabled": "Connection is disabled and excluded from tool use.", + "archived": "Connection is archived and excluded from tool use." + }, + "status": { + "draft": "Connection row exists but setup has not produced a usable configuration yet.", + "active": "Connection is fully configured and available for use.", + "disabled": "Connection is disabled and excluded from tool use.", + "archived": "Connection is archived and excluded from tool use." + }, + "transport": { + "mcp_remote": "Connection talks to a remote MCP server over HTTP.", + "rest_api": "Connection calls the provider's REST API directly.", + "local_stdio": "Connection runs a local MCP server over stdio." + } + }, "goal.created": { "goal_level": { "company": "Goal applies at company scope.", @@ -402,6 +520,22 @@ export const PAPERCLIP_ENUM_DESCRIPTIONS = { "other": "Fallback when the status is unknown or not represented by the tracked enum." } }, + "server.started": { + "os_platform": { + "aix": "Server host runs IBM AIX.", + "android": "Server host runs Android.", + "darwin": "Server host runs macOS.", + "freebsd": "Server host runs FreeBSD.", + "haiku": "Server host runs Haiku.", + "linux": "Server host runs Linux, including WSL and container hosts.", + "openbsd": "Server host runs OpenBSD.", + "sunos": "Server host runs SunOS, Solaris, or illumos.", + "win32": "Server host runs Windows, both 32-bit and 64-bit.", + "cygwin": "Server host runs under Cygwin on Windows.", + "netbsd": "Server host runs NetBSD.", + "other": "Fallback when the platform is unknown or not represented by the tracked enum." + } + }, "skill.imported": { "source_type": { "local_path": "Import source came from a filesystem path on the operator's machine.", diff --git a/packages/shared/src/telemetry/index.ts b/packages/shared/src/telemetry/index.ts index 61f945ad834..d6436ffd5ba 100644 --- a/packages/shared/src/telemetry/index.ts +++ b/packages/shared/src/telemetry/index.ts @@ -21,6 +21,7 @@ export { trackConnectionCreated, trackConnectionUpdated, trackConnectionInvoked, + trackServerStarted, } from "./events.js"; export type { TelemetryConfig, diff --git a/packages/shared/src/telemetry/retention.ts b/packages/shared/src/telemetry/retention.ts index 543e1085298..023004d4e49 100644 --- a/packages/shared/src/telemetry/retention.ts +++ b/packages/shared/src/telemetry/retention.ts @@ -47,4 +47,5 @@ export const EVENT_RETENTION_CLASS: Partial> = { "codex.credential_health": "operational_enum_count", "interaction.created": "operational_enum_count", "interaction.resolved": "operational_enum_count", + "server.started": "operational_enum_count", }; diff --git a/packages/shared/src/telemetry/server-started-event.test.ts b/packages/shared/src/telemetry/server-started-event.test.ts new file mode 100644 index 00000000000..3d0e0e2289b --- /dev/null +++ b/packages/shared/src/telemetry/server-started-event.test.ts @@ -0,0 +1,69 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; +import { TelemetryClient } from "./client.js"; +import { trackServerStarted } from "./events.js"; +import type { TelemetryState } from "./types.js"; + +const TEST_STATE: TelemetryState = { + installId: "test-install", + salt: "test-salt", + createdAt: "2026-01-01T00:00:00Z", + firstSeenVersion: "0.0.0", +}; + +function makeClient(enabled = true) { + const stateFactory = vi.fn(() => TEST_STATE); + return { + client: new TelemetryClient( + { enabled, endpoint: "http://localhost:9999/ingest" }, + stateFactory, + "0.0.0-test", + () => 0.5, + ), + stateFactory, + }; +} + +describe("registered server.started event", () => { + afterEach(() => { + vi.restoreAllMocks(); + vi.unstubAllGlobals(); + }); + + it("passes the raw process.platform value through without normalization", () => { + const { client } = makeClient(); + const track = vi.spyOn(client, "track"); + + trackServerStarted(client, { os_platform: "win32" }); + trackServerStarted(client, { os_platform: "some-future-platform" }); + + expect(track.mock.calls).toEqual([ + ["server.started", { os_platform: "win32" }], + ["server.started", { os_platform: "some-future-platform" }], + ]); + }); + + it("sends the registered event with only the raw OS family", async () => { + vi.stubGlobal("fetch", vi.fn().mockResolvedValue({ ok: true })); + const { client, stateFactory } = makeClient(); + + expect(client.isRegisteredEventName("server.started")).toBe(true); + trackServerStarted(client, { os_platform: process.platform }); + await client.flush(); + + expect(stateFactory).toHaveBeenCalledTimes(1); + expect(fetch).toHaveBeenCalledTimes(1); + const body = JSON.parse(String(vi.mocked(fetch).mock.calls[0]?.[1]?.body)); + expect(body.events).toEqual([ + { name: "server.started", occurredAt: expect.any(String), dimensions: { os_platform: process.platform } }, + ]); + }); + + it("leaves state and network untouched when disabled", async () => { + vi.stubGlobal("fetch", vi.fn()); + const { client, stateFactory } = makeClient(false); + trackServerStarted(client, { os_platform: process.platform }); + await client.flush(); + expect(stateFactory).not.toHaveBeenCalled(); + expect(fetch).not.toHaveBeenCalled(); + }); +}); diff --git a/server/src/__tests__/telemetry-startup.test.ts b/server/src/__tests__/telemetry-startup.test.ts new file mode 100644 index 00000000000..b132405f681 --- /dev/null +++ b/server/src/__tests__/telemetry-startup.test.ts @@ -0,0 +1,113 @@ +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +const stateFactory = vi.hoisted(() => vi.fn(() => ({ + installId: "test-install", + salt: "test-salt", + createdAt: "2026-01-01T00:00:00Z", + firstSeenVersion: "0.0.0", +}))); + +vi.mock("@paperclipai/shared/telemetry", async (importOriginal) => ({ + ...await importOriginal(), + loadOrCreateState: stateFactory, +})); +vi.mock("../home-paths.js", () => ({ resolvePaperclipInstanceRoot: () => "/unused-test-instance" })); +vi.mock("../version.js", () => ({ serverVersion: "0.0.0-test" })); + +describe("server telemetry startup", () => { + const fixtureDirs: string[] = []; + beforeEach(() => { + vi.resetModules(); + stateFactory.mockClear(); + for (const key of ["CI", "CONTINUOUS_INTEGRATION", "BUILD_NUMBER", "GITHUB_ACTIONS", "GITLAB_CI", "PAPERCLIP_TELEMETRY_DISABLED", "DO_NOT_TRACK"]) { + vi.stubEnv(key, ""); + } + vi.stubEnv("PAPERCLIP_TELEMETRY_ENDPOINT", "http://localhost:9999/ingest"); + vi.stubGlobal("fetch", vi.fn().mockResolvedValue({ ok: true })); + }); + + afterEach(async () => { + const { getTelemetryClient } = await import("../telemetry.js"); + getTelemetryClient()?.stop(); + vi.unstubAllEnvs(); + vi.unstubAllGlobals(); + vi.restoreAllMocks(); + for (const dir of fixtureDirs.splice(0)) rmSync(dir, { recursive: true, force: true }); + }); + + it("emits the OS family exactly once even when initialization is repeated", async () => { + const { initTelemetry, getTelemetryClient } = await import("../telemetry.js"); + const client = initTelemetry({ enabled: true }); + expect(client).not.toBeNull(); + expect(initTelemetry({ enabled: true })).toBe(client); + expect(getTelemetryClient()).toBe(client); + await client!.flush(); + + expect(stateFactory).toHaveBeenCalledTimes(1); + expect(fetch).toHaveBeenCalledTimes(1); + const body = JSON.parse(String(vi.mocked(fetch).mock.calls[0]?.[1]?.body)); + expect(body.events).toEqual([ + { name: "server.started", occurredAt: expect.any(String), dimensions: { os_platform: process.platform } }, + ]); + }); + + it("disables telemetry after a state failure without aborting startup or retrying", async () => { + stateFactory.mockImplementationOnce(() => { + throw Object.assign(new Error("private telemetry path cannot be written"), { code: "EACCES" }); + }); + const warning = vi.spyOn(console, "warn").mockImplementation(() => {}); + const { TelemetryClient } = await import("@paperclipai/shared/telemetry"); + const periodicFlush = vi.spyOn(TelemetryClient.prototype, "startPeriodicFlush"); + const { initTelemetry, getTelemetryClient } = await import("../telemetry.js"); + + expect(initTelemetry({ enabled: true })).toBeNull(); + expect(getTelemetryClient()).toBeNull(); + expect(initTelemetry({ enabled: true })).toBeNull(); + expect(stateFactory).toHaveBeenCalledTimes(1); + expect(periodicFlush).not.toHaveBeenCalled(); + expect(fetch).not.toHaveBeenCalled(); + expect(warning.mock.calls).toEqual([ + ["[telemetry] Initialization failed; telemetry is disabled for this process."], + ]); + }); + + it("continues startup when the real state store cannot create its directory", async () => { + const fixtureDir = mkdtempSync(path.join(tmpdir(), "paperclip-telemetry-startup-")); + fixtureDirs.push(fixtureDir); + const blockedStateDir = path.join(fixtureDir, "telemetry"); + writeFileSync(blockedStateDir, "this file prevents directory creation"); + const { loadOrCreateState } = await vi.importActual( + "@paperclipai/shared/telemetry", + ); + stateFactory.mockImplementationOnce(() => loadOrCreateState(blockedStateDir, "0.0.0-test")); + vi.spyOn(console, "warn").mockImplementation(() => {}); + const { initTelemetry, getTelemetryClient } = await import("../telemetry.js"); + + expect(initTelemetry({ enabled: true })).toBeNull(); + expect(getTelemetryClient()).toBeNull(); + expect(fetch).not.toHaveBeenCalled(); + }); + + it("does not initialize state or send when disabled in file configuration", async () => { + const { initTelemetry, getTelemetryClient } = await import("../telemetry.js"); + expect(initTelemetry({ enabled: false })).toBeNull(); + expect(getTelemetryClient()).toBeNull(); + expect(stateFactory).not.toHaveBeenCalled(); + expect(fetch).not.toHaveBeenCalled(); + }); + + it.each(["PAPERCLIP_TELEMETRY_DISABLED", "DO_NOT_TRACK", "CI", "GITHUB_ACTIONS"])( + "respects %s before creating the client or state", + async (key) => { + vi.stubEnv(key, "1"); + const { initTelemetry, getTelemetryClient } = await import("../telemetry.js"); + expect(initTelemetry({ enabled: true })).toBeNull(); + expect(getTelemetryClient()).toBeNull(); + expect(stateFactory).not.toHaveBeenCalled(); + expect(fetch).not.toHaveBeenCalled(); + }, + ); +}); diff --git a/server/src/telemetry.ts b/server/src/telemetry.ts index eb2b28749e6..bfa7c1930bb 100644 --- a/server/src/telemetry.ts +++ b/server/src/telemetry.ts @@ -3,26 +3,38 @@ import { TelemetryClient, resolveTelemetryConfig, loadOrCreateState, + trackServerStarted, } from "@paperclipai/shared/telemetry"; import { resolvePaperclipInstanceRoot } from "./home-paths.js"; import { serverVersion } from "./version.js"; let client: TelemetryClient | null = null; +let initializationFailed = false; export function initTelemetry(fileConfig?: { enabled?: boolean }): TelemetryClient | null { - if (client) return client; + if (client || initializationFailed) return client; const config = resolveTelemetryConfig(fileConfig); if (!config.enabled) return null; - const stateDir = path.join(resolvePaperclipInstanceRoot(), "telemetry"); - client = new TelemetryClient( - config, - () => loadOrCreateState(stateDir, serverVersion), - serverVersion, - ); - client.startPeriodicFlush(60_000); - return client; + let pendingClient: TelemetryClient | null = null; + try { + const stateDir = path.join(resolvePaperclipInstanceRoot(), "telemetry"); + pendingClient = new TelemetryClient( + config, + () => loadOrCreateState(stateDir, serverVersion), + serverVersion, + ); + trackServerStarted(pendingClient, { os_platform: process.platform }); + pendingClient.startPeriodicFlush(60_000); + client = pendingClient; + return client; + } catch { + pendingClient?.stop(); + initializationFailed = true; + console.warn("[telemetry] Initialization failed; telemetry is disabled for this process."); + return null; + } } export function getTelemetryClient(): TelemetryClient | null {