mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 09:50:48 +01:00
fix(telemetry): report a Paperclip runner agent's underlying harness as adapter_type (#22324)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server sends product telemetry. Two events describe agent work:
`agent.task_run` and `agent.task_completed`.
> - Each event has an `adapter_type` dimension. The generated telemetry
contract gives this dimension a fixed enum.
> - A Paperclip runner agent stores `paperclip_runner` as its adapter
type. The enum does not have this value.
> - The collector and Cloud direct delivery change a value that is not
in the enum to `other`.
> - Thus, as more agents use the runner, the adapter breakdown in
analytics becomes mostly `other`. The data is not lost, but it is in one
bucket.
> - This pull request makes a runner agent report the harness that it
runs, for example `claude_local` or `codex_local`.
> - The benefit is that the adapter breakdown shows the real harness
again.
## Linked Issues or Issue Description
Refs #8818 (this PR added the generated telemetry contract and its
`adapter_type` enum).
I did not find an existing issue. This is the bug description:
**What happened?**
`agent.task_run` and `agent.task_completed` send the stored
`agents.adapter_type`. For a Paperclip runner agent, this value is
`paperclip_runner`. The `adapter_type` enum in
`packages/shared/src/telemetry/generated/paperclip-telemetry.ts` does
not include `paperclip_runner`. The telemetry normalizer changes the
value to `other`. In analytics, almost all runner agent events show
`adapter_type = other`.
**Expected behavior**
A runner agent reports the harness that it runs. The server already
resolves this harness with `agentHarnessType()` from
`@paperclipai/shared`. For example, a runner agent with the Claude ACP
configuration must report `claude_local`.
**Steps to reproduce**
1. Create a Paperclip runner agent (`adapterType: "paperclip_runner"`).
2. Let the agent complete a heartbeat run, or let it set a task to
`done`.
3. Look at the `agent.task_run` or `agent.task_completed` event.
`adapter_type` is `paperclip_runner` before delivery, and `other` after
normalization.
**Paperclip version or commit**
`master` at 1355f93546.
**Agent adapter(s) involved**
Paperclip runner (all harnesses).
## What Changed
- `server/src/services/agent-task-run-telemetry.ts`: new
`telemetryAdapterType(agent)` helper. It returns
`agentHarnessType(agent.adapterType, adapterConfig)`. `agent.task_run`
sends this value as `adapterType`.
- `server/src/routes/issues.ts`: `agent.task_completed` calls the same
helper. Thus, the two events cannot report different adapter types for
the same agent.
- A legacy agent (for example `claude_local`) sends the same value as
before. `agentHarnessType()` returns a legacy adapter type without a
change.
- `server/src/__tests__/agent-task-run-telemetry.test.ts`:
- New tests for the default, Claude, and OpenCode runner configurations.
They expect `codex_local`, `claude_local`, and `opencode_local`.
- New contract test. It resolves every runner provider and every ACPX
agent through `telemetryAdapterType()`, and compares the result with the
generated `adapter_type` enum of both events. The test fails when a
runner harness is missing from the contract and is not in the known-gap
list.
- `server/src/__tests__/issue-telemetry-routes.test.ts`: new test that a
runner agent with the Claude ACP configuration sends `claude_local` on
`agent.task_completed`.
- `packages/shared/src/telemetry/README.md`:
- The `agent.task_run` section tells what `adapter_type` means for a
runner agent, and lists the runner harnesses that the contract does not
have yet.
- The `Emission Boundary` section tells why runner harness resolution is
not an alias map. The receiving layer does not get the agent
configuration, so it cannot derive the harness from `paperclip_runner`.
The emitter sends the resolved value raw and does not fall back to
`other`.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pin for
the changed README. I ran `node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`.
## Verification
- Red-green: before the fix, the new emitter tests failed with
`paperclip_runner`. After the fix, they pass.
- `cd server && npx vitest run
src/__tests__/agent-task-run-telemetry.test.ts
src/__tests__/issue-telemetry-routes.test.ts
src/__tests__/posthog-direct-event-shape.test.ts` and the three native
tests that mock the telemetry service: 182 tests pass.
- The contract test fails when I remove one known gap (`muse`) from the
list. Then it passes again when I restore the list.
- `cd server && pnpm typecheck`: passes.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
`changed: false`.
- Before the rebase, I sent the old and new values through the real
PostHog normalizer. The default, Claude, and OpenCode runner
configurations changed from `other` to `codex_local`, `claude_local`,
and `opencode_local`.
- I did not run the full local suite. `pnpm verify` asks for broader
coverage because the pin registry is a global CI input. CI supplies that
coverage.
## Risks
- Low risk. The change affects only the telemetry dimension value. It
does not change agent behavior, the database, or an API.
- Eight of the fifteen runner harnesses map to enum values:
`codex_local`, `opencode_local`, `claude_local`, `cursor`,
`cursor_cloud`, `grok_local`, `pi_local`, and `kimi_local`.
- Seven runner harnesses continue to show `other`: `acpx:copilot`,
`antigravity_runtime`, `aws_agentcore`, `claude_managed`, `grok_bot`,
`muse`, and `openai_dot`. The fix for these is a change to the contract
in the telemetry backend.
- After this change, `adapter_type` does not show the difference between
a runner agent and a legacy agent with the same harness. A separate
dimension in the telemetry contract can show this difference later.
- Events from before this change keep `other`.
## Privacy Review Request
`rules/data-paths.mdc` requires a privacy review for a telemetry change.
I request a privacy review from a maintainer. This is my analysis:
- The change does not add an event, a dimension, or an enum value. Thus,
the generated contract does not change. The contract comes from the
telemetry backend, and this repository must not edit it.
- The new value comes from `agentHarnessType()`. On each validated write
path (create, hire, PATCH, config rollback, and company import), the
server accepts only a closed set of runner `provider` and `acpxAgent`
values. Thus, the emitted value is a fixed harness identifier, not user
text.
- An agent row that does not pass these checks (for example, a
plugin-managed hire or a direct database write) can hold a different
`provider` string. Before this change, the emitter already sent the raw
`agents.adapter_type` string, which has the same exposure. The receiving
layer changes any value that is not in the enum to `other` before
analytics.
- The value never contains a prompt, a credential, a path, a model
output, or a person's name.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) from Anthropic, in Claude Code. It
used tool use (shell, file read and edit, `gh`) and extended reasoning.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
deacaf727c
commit
b7177db6aa
6 files changed
+93
-6
No files matched your search
@@ -53,6 +53,13 @@ to a stable reporting shape.
|
||||
Do not add client-side lowercasing, alias mapping, or fallback mapping unless
|
||||
the generated telemetry contract specifically requires that emitted value.
|
||||
|
||||
The `adapter_type` dimension on `agent.task_run` and `agent.task_completed` is
|
||||
not an alias map. For a Paperclip runner agent, the server sends the harness
|
||||
that `agentHarnessType()` resolves from the agent configuration. The receiving
|
||||
layer does not get that configuration, so it cannot derive the harness from
|
||||
`paperclip_runner`. The emitter sends the resolved value raw. It does not fall
|
||||
back to `other`.
|
||||
|
||||
If a dimension is privacy-protected before emission, emit only the protected
|
||||
value and its matching public marker as defined by the typed helper or generated
|
||||
contract. Do not emit private source material in telemetry dimensions.
|
||||
@@ -130,6 +137,17 @@ category decides the code only for the generic `adapter_failed` and
|
||||
run failure code sets in `server/src/services/continuation-cause.ts`, and a
|
||||
test fails when a code in those sets has no telemetry code.
|
||||
|
||||
The `adapter_type` dimension on `agent.task_run` and `agent.task_completed`
|
||||
reports the harness the agent runs, as `agentHarnessType()` resolves it. A
|
||||
Paperclip runner agent therefore reports its underlying harness (for example
|
||||
`claude_local` or `codex_local`), not `paperclip_runner`. Some runner harnesses
|
||||
are not in the generated `adapter_type` enum yet: `acpx:copilot`,
|
||||
`antigravity_runtime`, `aws_agentcore`, `claude_managed`, `grok_bot`, `muse`,
|
||||
and `openai_dot`. The receiving layer reports them as `other` until the
|
||||
contract lists them. A test in
|
||||
`server/src/__tests__/agent-task-run-telemetry.test.ts` fails when a runner
|
||||
harness that is not in this list is missing from the contract.
|
||||
|
||||
## Product Lifecycle Events
|
||||
|
||||
`model.connected`, `task.created`, `task.completed`, `task.blocked`,
|
||||
|
||||
@@ -8,6 +8,8 @@ import {
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
import { scopedTelemetryCompanyId, scopedTelemetryPerson } from "../telemetry-company-scope.js";
|
||||
import { EXECUTION_RECONCILIATION_CAUSES } from "@paperclipai/shared";
|
||||
import { PAPERCLIP_ENUM_DESCRIPTIONS } from "@paperclipai/shared/telemetry";
|
||||
import { PAPERCLIP_RUNNER_ACPX_PROFILES, PAPERCLIP_RUNNER_PERMISSION_CAPABILITIES } from "@paperclipai/adapter-utils";
|
||||
import { PERMANENT_FAILURE_CODES, PERSON_OWNED_FAILURE_CODES, RESTART_CLASS_FAILURE_CODES, SETUP_FAILURE_CODES } from "../services/continuation-cause.ts";
|
||||
|
||||
const mockTelemetryClient = vi.hoisted(() => ({ track: vi.fn() }));
|
||||
@@ -27,7 +29,7 @@ vi.mock("@paperclipai/shared/telemetry", async () => {
|
||||
};
|
||||
});
|
||||
|
||||
import { agentRunFailureCause, emitAgentTaskRun, emitAgentTaskRunById } from "../services/agent-task-run-telemetry.ts";
|
||||
import { agentRunFailureCause, emitAgentTaskRun, emitAgentTaskRunById, telemetryAdapterType } from "../services/agent-task-run-telemetry.ts";
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
@@ -376,6 +378,19 @@ describeEmbeddedPostgres("emitAgentTaskRun", () => {
|
||||
await expect(emitAgentTaskRun(db, run)).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ adapterConfig: {}, adapterType: "codex_local" },
|
||||
{ adapterConfig: { provider: "acpx", acpxAgent: "claude" }, adapterType: "claude_local" },
|
||||
{ adapterConfig: { provider: "opencode" }, adapterType: "opencode_local" },
|
||||
])("reports a Paperclip runner agent's underlying harness $adapterType as adapterType", async ({ adapterConfig, adapterType }) => {
|
||||
await seedCompanyAndAgent({ adapterType: "paperclip_runner", adapterConfig });
|
||||
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "succeeded" }).returning();
|
||||
|
||||
await emitAgentTaskRun(db, run);
|
||||
|
||||
expect(mockTrackAgentTaskRun.mock.calls[0][1].adapterType).toBe(adapterType);
|
||||
});
|
||||
|
||||
it("omits adapterType and agentRole when the agent row is absent", async () => {
|
||||
await seedCompanyAndAgent();
|
||||
const run = {
|
||||
@@ -454,3 +469,25 @@ describe("agentRunFailureCause", () => {
|
||||
expect(unclassified).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("Paperclip runner adapter_type coverage", () => {
|
||||
it.each(["agent.task_run", "agent.task_completed"] as const)("lists every runner harness in the %s adapter_type contract except the known gaps", (event) => {
|
||||
const contract = Object.keys(PAPERCLIP_ENUM_DESCRIPTIONS[event].adapter_type);
|
||||
const harnesses = [
|
||||
...Object.keys(PAPERCLIP_RUNNER_PERMISSION_CAPABILITIES)
|
||||
.filter((provider) => provider !== "acpx")
|
||||
.map((provider) => telemetryAdapterType({ adapterType: "paperclip_runner", adapterConfig: { provider } })),
|
||||
...PAPERCLIP_RUNNER_ACPX_PROFILES.map(({ value }) => telemetryAdapterType({ adapterType: "paperclip_runner", adapterConfig: { provider: "acpx", acpxAgent: value } })),
|
||||
];
|
||||
|
||||
expect(harnesses.filter((harness) => !contract.includes(harness)).sort()).toEqual([
|
||||
"acpx:copilot",
|
||||
"antigravity_runtime",
|
||||
"aws_agentcore",
|
||||
"claude_managed",
|
||||
"grok_bot",
|
||||
"muse",
|
||||
"openai_dot",
|
||||
]);
|
||||
});
|
||||
});
|
||||
@@ -253,6 +253,34 @@ describe("issue telemetry routes", () => {
|
||||
await vi.waitFor(() => expect(people).toEqual([expected]));
|
||||
}, 10_000);
|
||||
|
||||
it("emits a Paperclip runner agent's underlying harness as the adapter type", async () => {
|
||||
mockAgentService.getById.mockResolvedValue({
|
||||
id: "agent-1",
|
||||
companyId: "company-1",
|
||||
role: "engineer",
|
||||
adapterType: "paperclip_runner",
|
||||
adapterConfig: { provider: "acpx", acpxAgent: "claude", model: "claude-sonnet-4-6" },
|
||||
});
|
||||
|
||||
const app = await createApp({
|
||||
type: "agent",
|
||||
agentId: "agent-1",
|
||||
companyId: "company-1",
|
||||
runId: "22222222-2222-4222-8222-222222222222",
|
||||
});
|
||||
const res = await request(app)
|
||||
.patch("/api/issues/11111111-1111-4111-8111-111111111111")
|
||||
.send({ status: "done" });
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
await vi.waitFor(() => {
|
||||
expect(mockTrackAgentTaskCompleted).toHaveBeenCalledWith(
|
||||
expect.anything(),
|
||||
expect.objectContaining({ adapterType: "claude_local", model: "claude-sonnet-4-6" }),
|
||||
);
|
||||
});
|
||||
}, 10_000);
|
||||
|
||||
it("does not emit agent task-completed telemetry for board-driven completions", async () => {
|
||||
const app = await createApp({
|
||||
type: "board",
|
||||
|
||||
@@ -249,7 +249,7 @@ import { ContextCompactionError, contextCompactionService } from "../services/co
|
||||
import { issueContextUsage } from "../services/issue-context-usage.js";
|
||||
import { queueLiveRunnerPrpCommand } from "../realtime/runner-prp-ws.js";
|
||||
import { questionResponseDeliveryService } from "../services/question-response-delivery.js";
|
||||
import { emitAgentTaskRunById } from "../services/agent-task-run-telemetry.js";
|
||||
import { emitAgentTaskRunById, telemetryAdapterType } from "../services/agent-task-run-telemetry.js";
|
||||
import {
|
||||
createAutomaticGoalCommentSteering,
|
||||
createQueuedCommentQueue,
|
||||
@@ -16328,7 +16328,7 @@ export function issueRoutes(
|
||||
withTelemetryActor({ companyId: issue.companyId, person }, () => trackAgentTaskCompleted(tc, {
|
||||
agentRole: actorAgent.role,
|
||||
agentId: actorAgent.id,
|
||||
adapterType: actorAgent.adapterType,
|
||||
adapterType: telemetryAdapterType(actorAgent),
|
||||
model,
|
||||
taskId: issue.id,
|
||||
}));
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { agents, heartbeatRuns, issues, type Db } from "@paperclipai/db";
|
||||
import { requiresExecutionReconciliation } from "@paperclipai/shared";
|
||||
import { agentHarnessType, requiresExecutionReconciliation } from "@paperclipai/shared";
|
||||
import { PERMANENT_FAILURE_CODES, RESTART_CLASS_FAILURE_CODES, SETUP_FAILURE_CODES } from "./continuation-cause.js";
|
||||
import { trackAgentTaskRun } from "@paperclipai/shared/telemetry";
|
||||
import { parseObject } from "../adapters/utils.js";
|
||||
@@ -143,6 +143,10 @@ export function agentRunFailureCause(input: {
|
||||
return PLATFORM_FAILURE_CODES.has(code) ? "platform_error" : "unknown";
|
||||
}
|
||||
|
||||
export function telemetryAdapterType(agent: { adapterType: string; adapterConfig: unknown }): string {
|
||||
return agentHarnessType(agent.adapterType, parseObject(agent.adapterConfig));
|
||||
}
|
||||
|
||||
function providerFailureCategory(resultJson: unknown): string | null {
|
||||
const providerFailure = parseObject(parseObject(resultJson).providerFailure);
|
||||
return typeof providerFailure.category === "string" ? providerFailure.category : null;
|
||||
@@ -243,7 +247,7 @@ export async function emitAgentTaskRun(db: Db, run: TelemetryRun): Promise<void>
|
||||
agentId: run.agentId,
|
||||
state: run.status,
|
||||
...(agent
|
||||
? { adapterType: agent.adapterType, agentRole: agent.role }
|
||||
? { adapterType: telemetryAdapterType(agent), agentRole: agent.role }
|
||||
: {}),
|
||||
...(model ? { model } : {}),
|
||||
...(() => {
|
||||
|
||||
@@ -371,7 +371,7 @@
|
||||
"packages/shared/src/standing-checks.ts": "7ab71e4b6984e41e1c1dfd1bddf0116041d5abf1",
|
||||
"packages/shared/src/subscriptions.ts": "ef70e63395ca967bbc30776b71560e96eb354a75",
|
||||
"packages/shared/src/summary-slot.test.ts": "5f8e222dea425f0d5bff0adb5c683087bd7cf7fb",
|
||||
"packages/shared/src/telemetry/README.md": "36b4a57d187218ec141256d7f093c96c793ff901",
|
||||
"packages/shared/src/telemetry/README.md": "41b9c614005d53b74b6a4c9c9a55c715e9144653",
|
||||
"packages/shared/src/telemetry/client-types.test.ts": "3b630edda4972b929ba4decd7457f226eb9cfc5b",
|
||||
"packages/shared/src/telemetry/client.test.ts": "5a95f2305a393c5b2e7750b10e0ed65da0f30cf8",
|
||||
"packages/shared/src/telemetry/client.ts": "d6534ce8aaa6f69fb2dd03e12487db024e250a96",
|
||||
|
||||
Reference in new issue
Block a user