Files
PaperClipAI/server/src/services/agent-task-run-telemetry.ts
T
DottaandClaude Opus 5.5 b7177db6aa fix(telemetry): report a Paperclip runner agent's underlying harness as adapter_type (#22324)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server sends product telemetry. Two events describe agent work:
`agent.task_run` and `agent.task_completed`.
> - Each event has an `adapter_type` dimension. The generated telemetry
contract gives this dimension a fixed enum.
> - A Paperclip runner agent stores `paperclip_runner` as its adapter
type. The enum does not have this value.
> - The collector and Cloud direct delivery change a value that is not
in the enum to `other`.
> - Thus, as more agents use the runner, the adapter breakdown in
analytics becomes mostly `other`. The data is not lost, but it is in one
bucket.
> - This pull request makes a runner agent report the harness that it
runs, for example `claude_local` or `codex_local`.
> - The benefit is that the adapter breakdown shows the real harness
again.

## Linked Issues or Issue Description

Refs #8818 (this PR added the generated telemetry contract and its
`adapter_type` enum).

I did not find an existing issue. This is the bug description:

**What happened?**

`agent.task_run` and `agent.task_completed` send the stored
`agents.adapter_type`. For a Paperclip runner agent, this value is
`paperclip_runner`. The `adapter_type` enum in
`packages/shared/src/telemetry/generated/paperclip-telemetry.ts` does
not include `paperclip_runner`. The telemetry normalizer changes the
value to `other`. In analytics, almost all runner agent events show
`adapter_type = other`.

**Expected behavior**

A runner agent reports the harness that it runs. The server already
resolves this harness with `agentHarnessType()` from
`@paperclipai/shared`. For example, a runner agent with the Claude ACP
configuration must report `claude_local`.

**Steps to reproduce**

1. Create a Paperclip runner agent (`adapterType: "paperclip_runner"`).
2. Let the agent complete a heartbeat run, or let it set a task to
`done`.
3. Look at the `agent.task_run` or `agent.task_completed` event.
`adapter_type` is `paperclip_runner` before delivery, and `other` after
normalization.

**Paperclip version or commit**

`master` at 1355f93546.

**Agent adapter(s) involved**

Paperclip runner (all harnesses).

## What Changed

- `server/src/services/agent-task-run-telemetry.ts`: new
`telemetryAdapterType(agent)` helper. It returns
`agentHarnessType(agent.adapterType, adapterConfig)`. `agent.task_run`
sends this value as `adapterType`.
- `server/src/routes/issues.ts`: `agent.task_completed` calls the same
helper. Thus, the two events cannot report different adapter types for
the same agent.
- A legacy agent (for example `claude_local`) sends the same value as
before. `agentHarnessType()` returns a legacy adapter type without a
change.
- `server/src/__tests__/agent-task-run-telemetry.test.ts`:
- New tests for the default, Claude, and OpenCode runner configurations.
They expect `codex_local`, `claude_local`, and `opencode_local`.
- New contract test. It resolves every runner provider and every ACPX
agent through `telemetryAdapterType()`, and compares the result with the
generated `adapter_type` enum of both events. The test fails when a
runner harness is missing from the contract and is not in the known-gap
list.
- `server/src/__tests__/issue-telemetry-routes.test.ts`: new test that a
runner agent with the Claude ACP configuration sends `claude_local` on
`agent.task_completed`.
- `packages/shared/src/telemetry/README.md`:
- The `agent.task_run` section tells what `adapter_type` means for a
runner agent, and lists the runner harnesses that the contract does not
have yet.
- The `Emission Boundary` section tells why runner harness resolution is
not an alias map. The receiving layer does not get the agent
configuration, so it cannot derive the harness from `paperclip_runner`.
The emitter sends the resolved value raw and does not fall back to
`other`.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pin for
the changed README. I ran `node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`.

## Verification

- Red-green: before the fix, the new emitter tests failed with
`paperclip_runner`. After the fix, they pass.
- `cd server && npx vitest run
src/__tests__/agent-task-run-telemetry.test.ts
src/__tests__/issue-telemetry-routes.test.ts
src/__tests__/posthog-direct-event-shape.test.ts` and the three native
tests that mock the telemetry service: 182 tests pass.
- The contract test fails when I remove one known gap (`muse`) from the
list. Then it passes again when I restore the list.
- `cd server && pnpm typecheck`: passes.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
`changed: false`.
- Before the rebase, I sent the old and new values through the real
PostHog normalizer. The default, Claude, and OpenCode runner
configurations changed from `other` to `codex_local`, `claude_local`,
and `opencode_local`.
- I did not run the full local suite. `pnpm verify` asks for broader
coverage because the pin registry is a global CI input. CI supplies that
coverage.

## Risks

- Low risk. The change affects only the telemetry dimension value. It
does not change agent behavior, the database, or an API.
- Eight of the fifteen runner harnesses map to enum values:
`codex_local`, `opencode_local`, `claude_local`, `cursor`,
`cursor_cloud`, `grok_local`, `pi_local`, and `kimi_local`.
- Seven runner harnesses continue to show `other`: `acpx:copilot`,
`antigravity_runtime`, `aws_agentcore`, `claude_managed`, `grok_bot`,
`muse`, and `openai_dot`. The fix for these is a change to the contract
in the telemetry backend.
- After this change, `adapter_type` does not show the difference between
a runner agent and a legacy agent with the same harness. A separate
dimension in the telemetry contract can show this difference later.
- Events from before this change keep `other`.

## Privacy Review Request

`rules/data-paths.mdc` requires a privacy review for a telemetry change.
I request a privacy review from a maintainer. This is my analysis:

- The change does not add an event, a dimension, or an enum value. Thus,
the generated contract does not change. The contract comes from the
telemetry backend, and this repository must not edit it.
- The new value comes from `agentHarnessType()`. On each validated write
path (create, hire, PATCH, config rollback, and company import), the
server accepts only a closed set of runner `provider` and `acpxAgent`
values. Thus, the emitted value is a fixed harness identifier, not user
text.
- An agent row that does not pass these checks (for example, a
plugin-managed hire or a direct database write) can hold a different
`provider` string. Before this change, the emitter already sent the raw
`agents.adapter_type` string, which has the same exposure. The receiving
layer changes any value that is not in the enum to `other` before
analytics.
- The value never contains a prompt, a credential, a path, a model
output, or a person's name.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) from Anthropic, in Claude Code. It
used tool use (shell, file read and edit, `gh`) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 11:20:01 -06:00

321 lines
12 KiB
TypeScript

import { and, eq, sql } from "drizzle-orm";
import { agents, heartbeatRuns, issues, type Db } from "@paperclipai/db";
import { agentHarnessType, requiresExecutionReconciliation } from "@paperclipai/shared";
import { PERMANENT_FAILURE_CODES, RESTART_CLASS_FAILURE_CODES, SETUP_FAILURE_CODES } from "./continuation-cause.js";
import { trackAgentTaskRun } from "@paperclipai/shared/telemetry";
import { parseObject } from "../adapters/utils.js";
import { logger } from "../middleware/logger.js";
import { getTelemetryClient } from "../telemetry.js";
import { resolveTelemetryPerson, withTelemetryActor } from "../telemetry-company-scope.js";
type HeartbeatRun = typeof heartbeatRuns.$inferSelect;
type TelemetryRun = Pick<HeartbeatRun,
"id" | "companyId" | "agentId" | "status" | "startedAt" | "finishedAt" | "usageJson" | "contextSnapshot"
| "errorCode" | "resultJson">
& Partial<Pick<HeartbeatRun, "responsibleUserId">>;
const UUID_RE = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
export type AgentRunFailureCause =
| "missing_access"
| "provider_limit"
| "provider_error"
| "waiting_for_person"
| "task_changed"
| "platform_error"
| "agent_error"
| "unknown";
const FAILURE_STATES = new Set(["failed", "timed_out", "interrupted"]);
const PLATFORM_FAILURE_CODES = new Set<string>([
...RESTART_CLASS_FAILURE_CODES,
...SETUP_FAILURE_CODES,
...PERMANENT_FAILURE_CODES,
"workspace_validation_failed",
"workspace_restore_failed",
"remote_workspace_restore_failed",
"workspace_busy",
"runner_revision_mismatch",
"storage_unavailable",
"native_execution_ownership_unverified",
"lease_released_before_terminal",
"plugin_worker_unavailable",
"native_workspace_sync_out_retry_exhausted",
"queued_run_claim_rejected",
]);
const FAILURE_CAUSE_BY_ERROR_CODE: Record<string, AgentRunFailureCause> = {
configuration_incomplete: "missing_access",
slack_configuration_incomplete: "missing_access",
signing_secret_unconfigured: "missing_access",
user_secret_missing: "missing_access",
user_secret_definition_missing: "missing_access",
user_secret_definition_inactive: "missing_access",
github_access_denied: "missing_access",
github_repo_not_accessible: "missing_access",
github_auth_required: "missing_access",
project_access_denied: "missing_access",
permission_denied: "missing_access",
responsible_user_unauthorized: "missing_access",
oauth_access_projection_disabled: "missing_access",
acpx_auth_required: "missing_access",
codex_auth_required: "missing_access",
claude_auth_required: "missing_access",
provider_auth_failed: "missing_access",
hermes_gateway_auth_failed: "missing_access",
connection_unavailable: "missing_access",
provider_quota: "provider_limit",
provider_limit_wait: "provider_limit",
native_provider_usage_limit: "provider_limit",
native_provider_overloaded: "provider_limit",
github_rate_limited: "provider_limit",
ai_connection_busy: "provider_limit",
native_provider_model_rejected: "provider_error",
provider_tool_definition_invalid: "provider_error",
provider_transport_failed: "provider_error",
codex_transient_upstream: "provider_error",
provider_failed: "provider_error",
issue_waiting_for_response: "waiting_for_person",
issue_continuation_waiting_on_review: "waiting_for_person",
approval_required: "waiting_for_person",
formal_approval_unavailable: "waiting_for_person",
budget_blocked: "waiting_for_person",
budget_hard_stop: "waiting_for_person",
budget_exceeded: "waiting_for_person",
cancelled: "task_changed",
operator_cancelled: "task_changed",
operator_interrupted: "task_changed",
interrupted: "task_changed",
issue_cancelled: "task_changed",
issue_paused: "task_changed",
agent_paused: "task_changed",
issue_reassigned: "task_changed",
issue_assignee_changed: "task_changed",
issue_terminal_status: "task_changed",
issue_not_in_progress: "task_changed",
issue_execution_lock_changed: "task_changed",
issue_dependencies_blocked: "task_changed",
issue_blocked: "task_changed",
issue_not_found: "task_changed",
issue_review_participant_changed: "task_changed",
queued_comment_discarded: "task_changed",
invalid_output: "agent_error",
invalid_response: "agent_error",
tool_error: "agent_error",
acpx_turn_failed: "agent_error",
test_run_failed: "agent_error",
prompt_injection_blocked: "agent_error",
tool_execution_outcome_unknown: "agent_error",
turn_limit_exhausted: "agent_error",
timeout: "agent_error",
adapter_failed: "agent_error",
};
const FAILURE_CAUSE_BY_PROVIDER_CATEGORY: Record<string, AgentRunFailureCause> = {
authentication: "missing_access",
permission: "missing_access",
rate_limit: "provider_limit",
quota_exhausted: "provider_limit",
overloaded: "provider_limit",
model_unavailable: "provider_error",
invalid_request: "provider_error",
timeout: "provider_error",
unknown: "provider_error",
};
const PROVIDER_CATEGORY_CODES = new Set(["adapter_failed", "provider_failed"]);
export function agentRunFailureCause(input: {
errorCode: string | null | undefined;
providerFailureCategory: string | null | undefined;
}): AgentRunFailureCause {
const code = input.errorCode ?? null;
if (code && (code === "execution_reconciliation_required" || requiresExecutionReconciliation(code))) {
return "waiting_for_person";
}
const providerCause = input.providerFailureCategory
? FAILURE_CAUSE_BY_PROVIDER_CATEGORY[input.providerFailureCategory]
: undefined;
if (providerCause && (!code || PROVIDER_CATEGORY_CODES.has(code))) return providerCause;
if (!code) return "unknown";
if (FAILURE_CAUSE_BY_ERROR_CODE[code]) return FAILURE_CAUSE_BY_ERROR_CODE[code];
return PLATFORM_FAILURE_CODES.has(code) ? "platform_error" : "unknown";
}
export function telemetryAdapterType(agent: { adapterType: string; adapterConfig: unknown }): string {
return agentHarnessType(agent.adapterType, parseObject(agent.adapterConfig));
}
function providerFailureCategory(resultJson: unknown): string | null {
const providerFailure = parseObject(parseObject(resultJson).providerFailure);
return typeof providerFailure.category === "string" ? providerFailure.category : null;
}
function resolveFailureCause(run: TelemetryRun): AgentRunFailureCause | undefined {
if (!FAILURE_STATES.has(run.status)) return undefined;
return agentRunFailureCause({ errorCode: run.errorCode, providerFailureCategory: providerFailureCategory(run.resultJson) });
}
function readNonEmptyString(value: unknown): string | null {
return typeof value === "string" && value.trim().length > 0
? value.trim()
: null;
}
function readUsageToken(
source: Record<string, unknown>,
...keys: string[]
): number {
for (const key of keys) {
const value = source[key];
if (typeof value === "number" && Number.isFinite(value)) {
return Math.max(0, Math.floor(value));
}
}
return 0;
}
function resolveDurationSeconds(
run: Pick<TelemetryRun, "startedAt" | "finishedAt">,
now: Date,
): number | undefined {
if (!run.startedAt) return undefined;
const startedAtMs = new Date(run.startedAt).getTime();
const endMs = run.finishedAt ? new Date(run.finishedAt).getTime() : now.getTime();
return Math.max(0, Math.round((endMs - startedAtMs) / 1000));
}
/**
* Emits one `agent.task_run` telemetry event for a run that a caller already
* wrote to a terminal status. The native runtime and the legacy run engine
* both call this function beside each terminal status write. Call this
* function after the run commits.
*
* This function never throws. A telemetry failure must never fail a run.
* Telemetry enrichment does an awaited database lookup, so treat this call
* as best-effort background work: do not let a caller's own required
* lifecycle work (publishing a live event, cancelling a wake, clearing an
* issue lock, returning a response) wait behind it. Fire this call with
* `void emitAgentTaskRun(...)` and never await it, even alongside other
* required work with `Promise.all` — an awaited `Promise.all` still blocks
* the caller's return on the slower of the two promises, so it does not
* remove the delay.
*/
export async function emitAgentTaskRun(db: Db, run: TelemetryRun): Promise<void> {
try {
const client = getTelemetryClient();
if (!client) return;
const taskId =
readNonEmptyString(parseObject(run.contextSnapshot).issueId) ??
undefined;
const taskRowId = taskId && UUID_RE.test(taskId) ? taskId : null;
const agent = await db
.select({
role: agents.role,
adapterType: agents.adapterType,
adapterConfig: agents.adapterConfig,
taskCreatedByUserId: issues.createdByUserId,
taskResponsibleUserId: issues.responsibleUserId,
})
.from(agents)
.leftJoin(
issues,
taskRowId
? and(eq(issues.id, taskRowId), eq(issues.companyId, run.companyId))
: sql`false`,
)
.where(eq(agents.id, run.agentId))
.then((rows) => rows[0] ?? null);
const usage = parseObject(run.usageJson);
const model =
readNonEmptyString(usage.model) ??
readNonEmptyString(parseObject(agent?.adapterConfig).model) ??
undefined;
const person = resolveTelemetryPerson({
taskCreatedByUserId: agent?.taskCreatedByUserId,
taskAssignedByUserId: agent?.taskResponsibleUserId,
runOwnerUserId: run.responsibleUserId,
});
const failureCause = resolveFailureCause(run);
withTelemetryActor({ companyId: run.companyId, person }, () => trackAgentTaskRun(client, {
agentId: run.agentId,
state: run.status,
...(agent
? { adapterType: telemetryAdapterType(agent), agentRole: agent.role }
: {}),
...(model ? { model } : {}),
...(() => {
const durationSeconds = resolveDurationSeconds(run, new Date());
return durationSeconds === undefined ? {} : { durationSeconds };
})(),
...(run.usageJson != null
? {
inputTokens: readUsageToken(
usage,
"inputTokens",
"input_tokens",
"rawInputTokens",
"raw_input_tokens",
),
outputTokens: readUsageToken(
usage,
"outputTokens",
"output_tokens",
"rawOutputTokens",
"raw_output_tokens",
),
cachedTokens: readUsageToken(
usage,
"cachedInputTokens",
"cached_input_tokens",
"cacheReadInputTokens",
"cache_read_input_tokens",
),
}
: {}),
...(taskId ? { taskId } : {}),
...(failureCause ? { failureCause } : {}),
}));
} catch (err) {
logger.warn(
{ err, runId: run.id },
"failed to emit agent.task_run telemetry",
);
}
}
/** Loads a committed run before emitting telemetry, keeping database rows out of application contracts. */
export async function emitAgentTaskRunById(
db: Db,
input: { runId: string; companyId: string },
): Promise<void> {
try {
const run = await db
.select({
id: heartbeatRuns.id,
companyId: heartbeatRuns.companyId,
agentId: heartbeatRuns.agentId,
status: heartbeatRuns.status,
startedAt: heartbeatRuns.startedAt,
finishedAt: heartbeatRuns.finishedAt,
usageJson: heartbeatRuns.usageJson,
responsibleUserId: heartbeatRuns.responsibleUserId,
contextSnapshot: sql<Record<string, unknown>>`jsonb_build_object('issueId', ${heartbeatRuns.contextSnapshot}->'issueId')`,
errorCode: heartbeatRuns.errorCode,
resultJson: sql<Record<string, unknown> | null>`jsonb_build_object('providerFailure', jsonb_build_object('category', ${heartbeatRuns.resultJson}->'providerFailure'->'category'))`,
})
.from(heartbeatRuns)
.where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId)))
.then((rows) => rows[0] ?? null);
if (run) await emitAgentTaskRun(db, run);
} catch (err) {
logger.warn({ err, runId: input.runId }, "failed to load run for agent.task_run telemetry");
}
}