From b7177db6aada0f4317b078e35503418eead0e44e Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 19 Nov 2026 11:20:01 -0600 Subject: [PATCH] fix(telemetry): report a Paperclip runner agent's underlying harness as adapter_type (#22324) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server sends product telemetry. Two events describe agent work: `agent.task_run` and `agent.task_completed`. > - Each event has an `adapter_type` dimension. The generated telemetry contract gives this dimension a fixed enum. > - A Paperclip runner agent stores `paperclip_runner` as its adapter type. The enum does not have this value. > - The collector and Cloud direct delivery change a value that is not in the enum to `other`. > - Thus, as more agents use the runner, the adapter breakdown in analytics becomes mostly `other`. The data is not lost, but it is in one bucket. > - This pull request makes a runner agent report the harness that it runs, for example `claude_local` or `codex_local`. > - The benefit is that the adapter breakdown shows the real harness again. ## Linked Issues or Issue Description Refs #8818 (this PR added the generated telemetry contract and its `adapter_type` enum). I did not find an existing issue. This is the bug description: **What happened?** `agent.task_run` and `agent.task_completed` send the stored `agents.adapter_type`. For a Paperclip runner agent, this value is `paperclip_runner`. The `adapter_type` enum in `packages/shared/src/telemetry/generated/paperclip-telemetry.ts` does not include `paperclip_runner`. The telemetry normalizer changes the value to `other`. In analytics, almost all runner agent events show `adapter_type = other`. **Expected behavior** A runner agent reports the harness that it runs. The server already resolves this harness with `agentHarnessType()` from `@paperclipai/shared`. For example, a runner agent with the Claude ACP configuration must report `claude_local`. **Steps to reproduce** 1. Create a Paperclip runner agent (`adapterType: "paperclip_runner"`). 2. Let the agent complete a heartbeat run, or let it set a task to `done`. 3. Look at the `agent.task_run` or `agent.task_completed` event. `adapter_type` is `paperclip_runner` before delivery, and `other` after normalization. **Paperclip version or commit** `master` at 1355f93546. **Agent adapter(s) involved** Paperclip runner (all harnesses). ## What Changed - `server/src/services/agent-task-run-telemetry.ts`: new `telemetryAdapterType(agent)` helper. It returns `agentHarnessType(agent.adapterType, adapterConfig)`. `agent.task_run` sends this value as `adapterType`. - `server/src/routes/issues.ts`: `agent.task_completed` calls the same helper. Thus, the two events cannot report different adapter types for the same agent. - A legacy agent (for example `claude_local`) sends the same value as before. `agentHarnessType()` returns a legacy adapter type without a change. - `server/src/__tests__/agent-task-run-telemetry.test.ts`: - New tests for the default, Claude, and OpenCode runner configurations. They expect `codex_local`, `claude_local`, and `opencode_local`. - New contract test. It resolves every runner provider and every ACPX agent through `telemetryAdapterType()`, and compares the result with the generated `adapter_type` enum of both events. The test fails when a runner harness is missing from the contract and is not in the known-gap list. - `server/src/__tests__/issue-telemetry-routes.test.ts`: new test that a runner agent with the Claude ACP configuration sends `claude_local` on `agent.task_completed`. - `packages/shared/src/telemetry/README.md`: - The `agent.task_run` section tells what `adapter_type` means for a runner agent, and lists the runner harnesses that the contract does not have yet. - The `Emission Boundary` section tells why runner harness resolution is not an alias map. The receiving layer does not get the agent configuration, so it cannot derive the harness from `paperclip_runner`. The emitter sends the resolved value raw and does not fall back to `other`. - `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pin for the changed README. I ran `node tests/runner-e2e/source-sdk-public-inputs.mjs --write`. ## Verification - Red-green: before the fix, the new emitter tests failed with `paperclip_runner`. After the fix, they pass. - `cd server && npx vitest run src/__tests__/agent-task-run-telemetry.test.ts src/__tests__/issue-telemetry-routes.test.ts src/__tests__/posthog-direct-event-shape.test.ts` and the three native tests that mock the telemetry service: 182 tests pass. - The contract test fails when I remove one known gap (`muse`) from the list. Then it passes again when I restore the list. - `cd server && pnpm typecheck`: passes. - `node tests/runner-e2e/source-sdk-public-inputs.mjs --check`: `changed: false`. - Before the rebase, I sent the old and new values through the real PostHog normalizer. The default, Claude, and OpenCode runner configurations changed from `other` to `codex_local`, `claude_local`, and `opencode_local`. - I did not run the full local suite. `pnpm verify` asks for broader coverage because the pin registry is a global CI input. CI supplies that coverage. ## Risks - Low risk. The change affects only the telemetry dimension value. It does not change agent behavior, the database, or an API. - Eight of the fifteen runner harnesses map to enum values: `codex_local`, `opencode_local`, `claude_local`, `cursor`, `cursor_cloud`, `grok_local`, `pi_local`, and `kimi_local`. - Seven runner harnesses continue to show `other`: `acpx:copilot`, `antigravity_runtime`, `aws_agentcore`, `claude_managed`, `grok_bot`, `muse`, and `openai_dot`. The fix for these is a change to the contract in the telemetry backend. - After this change, `adapter_type` does not show the difference between a runner agent and a legacy agent with the same harness. A separate dimension in the telemetry contract can show this difference later. - Events from before this change keep `other`. ## Privacy Review Request `rules/data-paths.mdc` requires a privacy review for a telemetry change. I request a privacy review from a maintainer. This is my analysis: - The change does not add an event, a dimension, or an enum value. Thus, the generated contract does not change. The contract comes from the telemetry backend, and this repository must not edit it. - The new value comes from `agentHarnessType()`. On each validated write path (create, hire, PATCH, config rollback, and company import), the server accepts only a closed set of runner `provider` and `acpxAgent` values. Thus, the emitted value is a fixed harness identifier, not user text. - An agent row that does not pass these checks (for example, a plugin-managed hire or a direct database write) can hold a different `provider` string. Before this change, the emitter already sent the raw `agents.adapter_type` string, which has the same exposure. The receiving layer changes any value that is not in the enum to `other` before analytics. - The value never contains a prompt, a credential, a path, a model output, or a person's name. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) from Anthropic, in Claude Code. It used tool use (shell, file read and edit, `gh`) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- packages/shared/src/telemetry/README.md | 18 +++++++++ .../agent-task-run-telemetry.test.ts | 39 ++++++++++++++++++- .../__tests__/issue-telemetry-routes.test.ts | 28 +++++++++++++ server/src/routes/issues.ts | 4 +- .../src/services/agent-task-run-telemetry.ts | 8 +++- .../runner-e2e/source-sdk-public-inputs.json | 2 +- 6 files changed, 93 insertions(+), 6 deletions(-) diff --git a/packages/shared/src/telemetry/README.md b/packages/shared/src/telemetry/README.md index 36b4a57d187..41b9c614005 100644 --- a/packages/shared/src/telemetry/README.md +++ b/packages/shared/src/telemetry/README.md @@ -53,6 +53,13 @@ to a stable reporting shape. Do not add client-side lowercasing, alias mapping, or fallback mapping unless the generated telemetry contract specifically requires that emitted value. +The `adapter_type` dimension on `agent.task_run` and `agent.task_completed` is +not an alias map. For a Paperclip runner agent, the server sends the harness +that `agentHarnessType()` resolves from the agent configuration. The receiving +layer does not get that configuration, so it cannot derive the harness from +`paperclip_runner`. The emitter sends the resolved value raw. It does not fall +back to `other`. + If a dimension is privacy-protected before emission, emit only the protected value and its matching public marker as defined by the typed helper or generated contract. Do not emit private source material in telemetry dimensions. @@ -130,6 +137,17 @@ category decides the code only for the generic `adapter_failed` and run failure code sets in `server/src/services/continuation-cause.ts`, and a test fails when a code in those sets has no telemetry code. +The `adapter_type` dimension on `agent.task_run` and `agent.task_completed` +reports the harness the agent runs, as `agentHarnessType()` resolves it. A +Paperclip runner agent therefore reports its underlying harness (for example +`claude_local` or `codex_local`), not `paperclip_runner`. Some runner harnesses +are not in the generated `adapter_type` enum yet: `acpx:copilot`, +`antigravity_runtime`, `aws_agentcore`, `claude_managed`, `grok_bot`, `muse`, +and `openai_dot`. The receiving layer reports them as `other` until the +contract lists them. A test in +`server/src/__tests__/agent-task-run-telemetry.test.ts` fails when a runner +harness that is not in this list is missing from the contract. + ## Product Lifecycle Events `model.connected`, `task.created`, `task.completed`, `task.blocked`, diff --git a/server/src/__tests__/agent-task-run-telemetry.test.ts b/server/src/__tests__/agent-task-run-telemetry.test.ts index cbce97e41f8..fa762527e9e 100644 --- a/server/src/__tests__/agent-task-run-telemetry.test.ts +++ b/server/src/__tests__/agent-task-run-telemetry.test.ts @@ -8,6 +8,8 @@ import { } from "./helpers/embedded-postgres.js"; import { scopedTelemetryCompanyId, scopedTelemetryPerson } from "../telemetry-company-scope.js"; import { EXECUTION_RECONCILIATION_CAUSES } from "@paperclipai/shared"; +import { PAPERCLIP_ENUM_DESCRIPTIONS } from "@paperclipai/shared/telemetry"; +import { PAPERCLIP_RUNNER_ACPX_PROFILES, PAPERCLIP_RUNNER_PERMISSION_CAPABILITIES } from "@paperclipai/adapter-utils"; import { PERMANENT_FAILURE_CODES, PERSON_OWNED_FAILURE_CODES, RESTART_CLASS_FAILURE_CODES, SETUP_FAILURE_CODES } from "../services/continuation-cause.ts"; const mockTelemetryClient = vi.hoisted(() => ({ track: vi.fn() })); @@ -27,7 +29,7 @@ vi.mock("@paperclipai/shared/telemetry", async () => { }; }); -import { agentRunFailureCause, emitAgentTaskRun, emitAgentTaskRunById } from "../services/agent-task-run-telemetry.ts"; +import { agentRunFailureCause, emitAgentTaskRun, emitAgentTaskRunById, telemetryAdapterType } from "../services/agent-task-run-telemetry.ts"; const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; @@ -376,6 +378,19 @@ describeEmbeddedPostgres("emitAgentTaskRun", () => { await expect(emitAgentTaskRun(db, run)).resolves.toBeUndefined(); }); + it.each([ + { adapterConfig: {}, adapterType: "codex_local" }, + { adapterConfig: { provider: "acpx", acpxAgent: "claude" }, adapterType: "claude_local" }, + { adapterConfig: { provider: "opencode" }, adapterType: "opencode_local" }, + ])("reports a Paperclip runner agent's underlying harness $adapterType as adapterType", async ({ adapterConfig, adapterType }) => { + await seedCompanyAndAgent({ adapterType: "paperclip_runner", adapterConfig }); + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "succeeded" }).returning(); + + await emitAgentTaskRun(db, run); + + expect(mockTrackAgentTaskRun.mock.calls[0][1].adapterType).toBe(adapterType); + }); + it("omits adapterType and agentRole when the agent row is absent", async () => { await seedCompanyAndAgent(); const run = { @@ -454,3 +469,25 @@ describe("agentRunFailureCause", () => { expect(unclassified).toEqual([]); }); }); + +describe("Paperclip runner adapter_type coverage", () => { + it.each(["agent.task_run", "agent.task_completed"] as const)("lists every runner harness in the %s adapter_type contract except the known gaps", (event) => { + const contract = Object.keys(PAPERCLIP_ENUM_DESCRIPTIONS[event].adapter_type); + const harnesses = [ + ...Object.keys(PAPERCLIP_RUNNER_PERMISSION_CAPABILITIES) + .filter((provider) => provider !== "acpx") + .map((provider) => telemetryAdapterType({ adapterType: "paperclip_runner", adapterConfig: { provider } })), + ...PAPERCLIP_RUNNER_ACPX_PROFILES.map(({ value }) => telemetryAdapterType({ adapterType: "paperclip_runner", adapterConfig: { provider: "acpx", acpxAgent: value } })), + ]; + + expect(harnesses.filter((harness) => !contract.includes(harness)).sort()).toEqual([ + "acpx:copilot", + "antigravity_runtime", + "aws_agentcore", + "claude_managed", + "grok_bot", + "muse", + "openai_dot", + ]); + }); +}); diff --git a/server/src/__tests__/issue-telemetry-routes.test.ts b/server/src/__tests__/issue-telemetry-routes.test.ts index 04e75132724..a8888f54054 100644 --- a/server/src/__tests__/issue-telemetry-routes.test.ts +++ b/server/src/__tests__/issue-telemetry-routes.test.ts @@ -253,6 +253,34 @@ describe("issue telemetry routes", () => { await vi.waitFor(() => expect(people).toEqual([expected])); }, 10_000); + it("emits a Paperclip runner agent's underlying harness as the adapter type", async () => { + mockAgentService.getById.mockResolvedValue({ + id: "agent-1", + companyId: "company-1", + role: "engineer", + adapterType: "paperclip_runner", + adapterConfig: { provider: "acpx", acpxAgent: "claude", model: "claude-sonnet-4-6" }, + }); + + const app = await createApp({ + type: "agent", + agentId: "agent-1", + companyId: "company-1", + runId: "22222222-2222-4222-8222-222222222222", + }); + const res = await request(app) + .patch("/api/issues/11111111-1111-4111-8111-111111111111") + .send({ status: "done" }); + + expect(res.status).toBe(200); + await vi.waitFor(() => { + expect(mockTrackAgentTaskCompleted).toHaveBeenCalledWith( + expect.anything(), + expect.objectContaining({ adapterType: "claude_local", model: "claude-sonnet-4-6" }), + ); + }); + }, 10_000); + it("does not emit agent task-completed telemetry for board-driven completions", async () => { const app = await createApp({ type: "board", diff --git a/server/src/routes/issues.ts b/server/src/routes/issues.ts index 8f1092dacc0..0da962ebb8e 100644 --- a/server/src/routes/issues.ts +++ b/server/src/routes/issues.ts @@ -249,7 +249,7 @@ import { ContextCompactionError, contextCompactionService } from "../services/co import { issueContextUsage } from "../services/issue-context-usage.js"; import { queueLiveRunnerPrpCommand } from "../realtime/runner-prp-ws.js"; import { questionResponseDeliveryService } from "../services/question-response-delivery.js"; -import { emitAgentTaskRunById } from "../services/agent-task-run-telemetry.js"; +import { emitAgentTaskRunById, telemetryAdapterType } from "../services/agent-task-run-telemetry.js"; import { createAutomaticGoalCommentSteering, createQueuedCommentQueue, @@ -16328,7 +16328,7 @@ export function issueRoutes( withTelemetryActor({ companyId: issue.companyId, person }, () => trackAgentTaskCompleted(tc, { agentRole: actorAgent.role, agentId: actorAgent.id, - adapterType: actorAgent.adapterType, + adapterType: telemetryAdapterType(actorAgent), model, taskId: issue.id, })); diff --git a/server/src/services/agent-task-run-telemetry.ts b/server/src/services/agent-task-run-telemetry.ts index c05b7dd1111..ceab7157111 100644 --- a/server/src/services/agent-task-run-telemetry.ts +++ b/server/src/services/agent-task-run-telemetry.ts @@ -1,6 +1,6 @@ import { and, eq, sql } from "drizzle-orm"; import { agents, heartbeatRuns, issues, type Db } from "@paperclipai/db"; -import { requiresExecutionReconciliation } from "@paperclipai/shared"; +import { agentHarnessType, requiresExecutionReconciliation } from "@paperclipai/shared"; import { PERMANENT_FAILURE_CODES, RESTART_CLASS_FAILURE_CODES, SETUP_FAILURE_CODES } from "./continuation-cause.js"; import { trackAgentTaskRun } from "@paperclipai/shared/telemetry"; import { parseObject } from "../adapters/utils.js"; @@ -143,6 +143,10 @@ export function agentRunFailureCause(input: { return PLATFORM_FAILURE_CODES.has(code) ? "platform_error" : "unknown"; } +export function telemetryAdapterType(agent: { adapterType: string; adapterConfig: unknown }): string { + return agentHarnessType(agent.adapterType, parseObject(agent.adapterConfig)); +} + function providerFailureCategory(resultJson: unknown): string | null { const providerFailure = parseObject(parseObject(resultJson).providerFailure); return typeof providerFailure.category === "string" ? providerFailure.category : null; @@ -243,7 +247,7 @@ export async function emitAgentTaskRun(db: Db, run: TelemetryRun): Promise agentId: run.agentId, state: run.status, ...(agent - ? { adapterType: agent.adapterType, agentRole: agent.role } + ? { adapterType: telemetryAdapterType(agent), agentRole: agent.role } : {}), ...(model ? { model } : {}), ...(() => { diff --git a/tests/runner-e2e/source-sdk-public-inputs.json b/tests/runner-e2e/source-sdk-public-inputs.json index ba5e145aece..51c7cdf0aea 100644 --- a/tests/runner-e2e/source-sdk-public-inputs.json +++ b/tests/runner-e2e/source-sdk-public-inputs.json @@ -371,7 +371,7 @@ "packages/shared/src/standing-checks.ts": "7ab71e4b6984e41e1c1dfd1bddf0116041d5abf1", "packages/shared/src/subscriptions.ts": "ef70e63395ca967bbc30776b71560e96eb354a75", "packages/shared/src/summary-slot.test.ts": "5f8e222dea425f0d5bff0adb5c683087bd7cf7fb", - "packages/shared/src/telemetry/README.md": "36b4a57d187218ec141256d7f093c96c793ff901", + "packages/shared/src/telemetry/README.md": "41b9c614005d53b74b6a4c9c9a55c715e9144653", "packages/shared/src/telemetry/client-types.test.ts": "3b630edda4972b929ba4decd7457f226eb9cfc5b", "packages/shared/src/telemetry/client.test.ts": "5a95f2305a393c5b2e7750b10e0ed65da0f30cf8", "packages/shared/src/telemetry/client.ts": "d6534ce8aaa6f69fb2dd03e12487db024e250a96",