mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server sends product telemetry. Two events describe agent work:
`agent.task_run` and `agent.task_completed`.
> - Each event has an `adapter_type` dimension. The generated telemetry
contract gives this dimension a fixed enum.
> - A Paperclip runner agent stores `paperclip_runner` as its adapter
type. The enum does not have this value.
> - The collector and Cloud direct delivery change a value that is not
in the enum to `other`.
> - Thus, as more agents use the runner, the adapter breakdown in
analytics becomes mostly `other`. The data is not lost, but it is in one
bucket.
> - This pull request makes a runner agent report the harness that it
runs, for example `claude_local` or `codex_local`.
> - The benefit is that the adapter breakdown shows the real harness
again.
## Linked Issues or Issue Description
Refs #8818 (this PR added the generated telemetry contract and its
`adapter_type` enum).
I did not find an existing issue. This is the bug description:
**What happened?**
`agent.task_run` and `agent.task_completed` send the stored
`agents.adapter_type`. For a Paperclip runner agent, this value is
`paperclip_runner`. The `adapter_type` enum in
`packages/shared/src/telemetry/generated/paperclip-telemetry.ts` does
not include `paperclip_runner`. The telemetry normalizer changes the
value to `other`. In analytics, almost all runner agent events show
`adapter_type = other`.
**Expected behavior**
A runner agent reports the harness that it runs. The server already
resolves this harness with `agentHarnessType()` from
`@paperclipai/shared`. For example, a runner agent with the Claude ACP
configuration must report `claude_local`.
**Steps to reproduce**
1. Create a Paperclip runner agent (`adapterType: "paperclip_runner"`).
2. Let the agent complete a heartbeat run, or let it set a task to
`done`.
3. Look at the `agent.task_run` or `agent.task_completed` event.
`adapter_type` is `paperclip_runner` before delivery, and `other` after
normalization.
**Paperclip version or commit**
`master` at 1355f93546.
**Agent adapter(s) involved**
Paperclip runner (all harnesses).
## What Changed
- `server/src/services/agent-task-run-telemetry.ts`: new
`telemetryAdapterType(agent)` helper. It returns
`agentHarnessType(agent.adapterType, adapterConfig)`. `agent.task_run`
sends this value as `adapterType`.
- `server/src/routes/issues.ts`: `agent.task_completed` calls the same
helper. Thus, the two events cannot report different adapter types for
the same agent.
- A legacy agent (for example `claude_local`) sends the same value as
before. `agentHarnessType()` returns a legacy adapter type without a
change.
- `server/src/__tests__/agent-task-run-telemetry.test.ts`:
- New tests for the default, Claude, and OpenCode runner configurations.
They expect `codex_local`, `claude_local`, and `opencode_local`.
- New contract test. It resolves every runner provider and every ACPX
agent through `telemetryAdapterType()`, and compares the result with the
generated `adapter_type` enum of both events. The test fails when a
runner harness is missing from the contract and is not in the known-gap
list.
- `server/src/__tests__/issue-telemetry-routes.test.ts`: new test that a
runner agent with the Claude ACP configuration sends `claude_local` on
`agent.task_completed`.
- `packages/shared/src/telemetry/README.md`:
- The `agent.task_run` section tells what `adapter_type` means for a
runner agent, and lists the runner harnesses that the contract does not
have yet.
- The `Emission Boundary` section tells why runner harness resolution is
not an alias map. The receiving layer does not get the agent
configuration, so it cannot derive the harness from `paperclip_runner`.
The emitter sends the resolved value raw and does not fall back to
`other`.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pin for
the changed README. I ran `node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`.
## Verification
- Red-green: before the fix, the new emitter tests failed with
`paperclip_runner`. After the fix, they pass.
- `cd server && npx vitest run
src/__tests__/agent-task-run-telemetry.test.ts
src/__tests__/issue-telemetry-routes.test.ts
src/__tests__/posthog-direct-event-shape.test.ts` and the three native
tests that mock the telemetry service: 182 tests pass.
- The contract test fails when I remove one known gap (`muse`) from the
list. Then it passes again when I restore the list.
- `cd server && pnpm typecheck`: passes.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
`changed: false`.
- Before the rebase, I sent the old and new values through the real
PostHog normalizer. The default, Claude, and OpenCode runner
configurations changed from `other` to `codex_local`, `claude_local`,
and `opencode_local`.
- I did not run the full local suite. `pnpm verify` asks for broader
coverage because the pin registry is a global CI input. CI supplies that
coverage.
## Risks
- Low risk. The change affects only the telemetry dimension value. It
does not change agent behavior, the database, or an API.
- Eight of the fifteen runner harnesses map to enum values:
`codex_local`, `opencode_local`, `claude_local`, `cursor`,
`cursor_cloud`, `grok_local`, `pi_local`, and `kimi_local`.
- Seven runner harnesses continue to show `other`: `acpx:copilot`,
`antigravity_runtime`, `aws_agentcore`, `claude_managed`, `grok_bot`,
`muse`, and `openai_dot`. The fix for these is a change to the contract
in the telemetry backend.
- After this change, `adapter_type` does not show the difference between
a runner agent and a legacy agent with the same harness. A separate
dimension in the telemetry contract can show this difference later.
- Events from before this change keep `other`.
## Privacy Review Request
`rules/data-paths.mdc` requires a privacy review for a telemetry change.
I request a privacy review from a maintainer. This is my analysis:
- The change does not add an event, a dimension, or an enum value. Thus,
the generated contract does not change. The contract comes from the
telemetry backend, and this repository must not edit it.
- The new value comes from `agentHarnessType()`. On each validated write
path (create, hire, PATCH, config rollback, and company import), the
server accepts only a closed set of runner `provider` and `acpxAgent`
values. Thus, the emitted value is a fixed harness identifier, not user
text.
- An agent row that does not pass these checks (for example, a
plugin-managed hire or a direct database write) can hold a different
`provider` string. Before this change, the emitter already sent the raw
`agents.adapter_type` string, which has the same exposure. The receiving
layer changes any value that is not in the enum to `other` before
analytics.
- The value never contains a prompt, a credential, a path, a model
output, or a person's name.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) from Anthropic, in Claude Code. It
used tool use (shell, file read and edit, `gh`) and extended reasoning.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
321 lines
12 KiB
TypeScript
321 lines
12 KiB
TypeScript
import { and, eq, sql } from "drizzle-orm";
|
|
import { agents, heartbeatRuns, issues, type Db } from "@paperclipai/db";
|
|
import { agentHarnessType, requiresExecutionReconciliation } from "@paperclipai/shared";
|
|
import { PERMANENT_FAILURE_CODES, RESTART_CLASS_FAILURE_CODES, SETUP_FAILURE_CODES } from "./continuation-cause.js";
|
|
import { trackAgentTaskRun } from "@paperclipai/shared/telemetry";
|
|
import { parseObject } from "../adapters/utils.js";
|
|
import { logger } from "../middleware/logger.js";
|
|
import { getTelemetryClient } from "../telemetry.js";
|
|
import { resolveTelemetryPerson, withTelemetryActor } from "../telemetry-company-scope.js";
|
|
|
|
type HeartbeatRun = typeof heartbeatRuns.$inferSelect;
|
|
type TelemetryRun = Pick<HeartbeatRun,
|
|
"id" | "companyId" | "agentId" | "status" | "startedAt" | "finishedAt" | "usageJson" | "contextSnapshot"
|
|
| "errorCode" | "resultJson">
|
|
& Partial<Pick<HeartbeatRun, "responsibleUserId">>;
|
|
|
|
const UUID_RE = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
|
|
|
|
export type AgentRunFailureCause =
|
|
| "missing_access"
|
|
| "provider_limit"
|
|
| "provider_error"
|
|
| "waiting_for_person"
|
|
| "task_changed"
|
|
| "platform_error"
|
|
| "agent_error"
|
|
| "unknown";
|
|
|
|
const FAILURE_STATES = new Set(["failed", "timed_out", "interrupted"]);
|
|
|
|
const PLATFORM_FAILURE_CODES = new Set<string>([
|
|
...RESTART_CLASS_FAILURE_CODES,
|
|
...SETUP_FAILURE_CODES,
|
|
...PERMANENT_FAILURE_CODES,
|
|
"workspace_validation_failed",
|
|
"workspace_restore_failed",
|
|
"remote_workspace_restore_failed",
|
|
"workspace_busy",
|
|
"runner_revision_mismatch",
|
|
"storage_unavailable",
|
|
"native_execution_ownership_unverified",
|
|
"lease_released_before_terminal",
|
|
"plugin_worker_unavailable",
|
|
"native_workspace_sync_out_retry_exhausted",
|
|
"queued_run_claim_rejected",
|
|
]);
|
|
|
|
const FAILURE_CAUSE_BY_ERROR_CODE: Record<string, AgentRunFailureCause> = {
|
|
configuration_incomplete: "missing_access",
|
|
slack_configuration_incomplete: "missing_access",
|
|
signing_secret_unconfigured: "missing_access",
|
|
user_secret_missing: "missing_access",
|
|
user_secret_definition_missing: "missing_access",
|
|
user_secret_definition_inactive: "missing_access",
|
|
github_access_denied: "missing_access",
|
|
github_repo_not_accessible: "missing_access",
|
|
github_auth_required: "missing_access",
|
|
project_access_denied: "missing_access",
|
|
permission_denied: "missing_access",
|
|
responsible_user_unauthorized: "missing_access",
|
|
oauth_access_projection_disabled: "missing_access",
|
|
acpx_auth_required: "missing_access",
|
|
codex_auth_required: "missing_access",
|
|
claude_auth_required: "missing_access",
|
|
provider_auth_failed: "missing_access",
|
|
hermes_gateway_auth_failed: "missing_access",
|
|
connection_unavailable: "missing_access",
|
|
provider_quota: "provider_limit",
|
|
provider_limit_wait: "provider_limit",
|
|
native_provider_usage_limit: "provider_limit",
|
|
native_provider_overloaded: "provider_limit",
|
|
github_rate_limited: "provider_limit",
|
|
ai_connection_busy: "provider_limit",
|
|
native_provider_model_rejected: "provider_error",
|
|
provider_tool_definition_invalid: "provider_error",
|
|
provider_transport_failed: "provider_error",
|
|
codex_transient_upstream: "provider_error",
|
|
provider_failed: "provider_error",
|
|
issue_waiting_for_response: "waiting_for_person",
|
|
issue_continuation_waiting_on_review: "waiting_for_person",
|
|
approval_required: "waiting_for_person",
|
|
formal_approval_unavailable: "waiting_for_person",
|
|
budget_blocked: "waiting_for_person",
|
|
budget_hard_stop: "waiting_for_person",
|
|
budget_exceeded: "waiting_for_person",
|
|
cancelled: "task_changed",
|
|
operator_cancelled: "task_changed",
|
|
operator_interrupted: "task_changed",
|
|
interrupted: "task_changed",
|
|
issue_cancelled: "task_changed",
|
|
issue_paused: "task_changed",
|
|
agent_paused: "task_changed",
|
|
issue_reassigned: "task_changed",
|
|
issue_assignee_changed: "task_changed",
|
|
issue_terminal_status: "task_changed",
|
|
issue_not_in_progress: "task_changed",
|
|
issue_execution_lock_changed: "task_changed",
|
|
issue_dependencies_blocked: "task_changed",
|
|
issue_blocked: "task_changed",
|
|
issue_not_found: "task_changed",
|
|
issue_review_participant_changed: "task_changed",
|
|
queued_comment_discarded: "task_changed",
|
|
invalid_output: "agent_error",
|
|
invalid_response: "agent_error",
|
|
tool_error: "agent_error",
|
|
acpx_turn_failed: "agent_error",
|
|
test_run_failed: "agent_error",
|
|
prompt_injection_blocked: "agent_error",
|
|
tool_execution_outcome_unknown: "agent_error",
|
|
turn_limit_exhausted: "agent_error",
|
|
timeout: "agent_error",
|
|
adapter_failed: "agent_error",
|
|
};
|
|
|
|
const FAILURE_CAUSE_BY_PROVIDER_CATEGORY: Record<string, AgentRunFailureCause> = {
|
|
authentication: "missing_access",
|
|
permission: "missing_access",
|
|
rate_limit: "provider_limit",
|
|
quota_exhausted: "provider_limit",
|
|
overloaded: "provider_limit",
|
|
model_unavailable: "provider_error",
|
|
invalid_request: "provider_error",
|
|
timeout: "provider_error",
|
|
unknown: "provider_error",
|
|
};
|
|
|
|
const PROVIDER_CATEGORY_CODES = new Set(["adapter_failed", "provider_failed"]);
|
|
|
|
export function agentRunFailureCause(input: {
|
|
errorCode: string | null | undefined;
|
|
providerFailureCategory: string | null | undefined;
|
|
}): AgentRunFailureCause {
|
|
const code = input.errorCode ?? null;
|
|
if (code && (code === "execution_reconciliation_required" || requiresExecutionReconciliation(code))) {
|
|
return "waiting_for_person";
|
|
}
|
|
const providerCause = input.providerFailureCategory
|
|
? FAILURE_CAUSE_BY_PROVIDER_CATEGORY[input.providerFailureCategory]
|
|
: undefined;
|
|
if (providerCause && (!code || PROVIDER_CATEGORY_CODES.has(code))) return providerCause;
|
|
if (!code) return "unknown";
|
|
if (FAILURE_CAUSE_BY_ERROR_CODE[code]) return FAILURE_CAUSE_BY_ERROR_CODE[code];
|
|
return PLATFORM_FAILURE_CODES.has(code) ? "platform_error" : "unknown";
|
|
}
|
|
|
|
export function telemetryAdapterType(agent: { adapterType: string; adapterConfig: unknown }): string {
|
|
return agentHarnessType(agent.adapterType, parseObject(agent.adapterConfig));
|
|
}
|
|
|
|
function providerFailureCategory(resultJson: unknown): string | null {
|
|
const providerFailure = parseObject(parseObject(resultJson).providerFailure);
|
|
return typeof providerFailure.category === "string" ? providerFailure.category : null;
|
|
}
|
|
|
|
function resolveFailureCause(run: TelemetryRun): AgentRunFailureCause | undefined {
|
|
if (!FAILURE_STATES.has(run.status)) return undefined;
|
|
return agentRunFailureCause({ errorCode: run.errorCode, providerFailureCategory: providerFailureCategory(run.resultJson) });
|
|
}
|
|
|
|
function readNonEmptyString(value: unknown): string | null {
|
|
return typeof value === "string" && value.trim().length > 0
|
|
? value.trim()
|
|
: null;
|
|
}
|
|
|
|
function readUsageToken(
|
|
source: Record<string, unknown>,
|
|
...keys: string[]
|
|
): number {
|
|
for (const key of keys) {
|
|
const value = source[key];
|
|
if (typeof value === "number" && Number.isFinite(value)) {
|
|
return Math.max(0, Math.floor(value));
|
|
}
|
|
}
|
|
return 0;
|
|
}
|
|
|
|
function resolveDurationSeconds(
|
|
run: Pick<TelemetryRun, "startedAt" | "finishedAt">,
|
|
now: Date,
|
|
): number | undefined {
|
|
if (!run.startedAt) return undefined;
|
|
const startedAtMs = new Date(run.startedAt).getTime();
|
|
const endMs = run.finishedAt ? new Date(run.finishedAt).getTime() : now.getTime();
|
|
return Math.max(0, Math.round((endMs - startedAtMs) / 1000));
|
|
}
|
|
|
|
/**
|
|
* Emits one `agent.task_run` telemetry event for a run that a caller already
|
|
* wrote to a terminal status. The native runtime and the legacy run engine
|
|
* both call this function beside each terminal status write. Call this
|
|
* function after the run commits.
|
|
*
|
|
* This function never throws. A telemetry failure must never fail a run.
|
|
* Telemetry enrichment does an awaited database lookup, so treat this call
|
|
* as best-effort background work: do not let a caller's own required
|
|
* lifecycle work (publishing a live event, cancelling a wake, clearing an
|
|
* issue lock, returning a response) wait behind it. Fire this call with
|
|
* `void emitAgentTaskRun(...)` and never await it, even alongside other
|
|
* required work with `Promise.all` — an awaited `Promise.all` still blocks
|
|
* the caller's return on the slower of the two promises, so it does not
|
|
* remove the delay.
|
|
*/
|
|
export async function emitAgentTaskRun(db: Db, run: TelemetryRun): Promise<void> {
|
|
try {
|
|
const client = getTelemetryClient();
|
|
if (!client) return;
|
|
|
|
const taskId =
|
|
readNonEmptyString(parseObject(run.contextSnapshot).issueId) ??
|
|
undefined;
|
|
const taskRowId = taskId && UUID_RE.test(taskId) ? taskId : null;
|
|
|
|
const agent = await db
|
|
.select({
|
|
role: agents.role,
|
|
adapterType: agents.adapterType,
|
|
adapterConfig: agents.adapterConfig,
|
|
taskCreatedByUserId: issues.createdByUserId,
|
|
taskResponsibleUserId: issues.responsibleUserId,
|
|
})
|
|
.from(agents)
|
|
.leftJoin(
|
|
issues,
|
|
taskRowId
|
|
? and(eq(issues.id, taskRowId), eq(issues.companyId, run.companyId))
|
|
: sql`false`,
|
|
)
|
|
.where(eq(agents.id, run.agentId))
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
const usage = parseObject(run.usageJson);
|
|
const model =
|
|
readNonEmptyString(usage.model) ??
|
|
readNonEmptyString(parseObject(agent?.adapterConfig).model) ??
|
|
undefined;
|
|
|
|
const person = resolveTelemetryPerson({
|
|
taskCreatedByUserId: agent?.taskCreatedByUserId,
|
|
taskAssignedByUserId: agent?.taskResponsibleUserId,
|
|
runOwnerUserId: run.responsibleUserId,
|
|
});
|
|
const failureCause = resolveFailureCause(run);
|
|
|
|
withTelemetryActor({ companyId: run.companyId, person }, () => trackAgentTaskRun(client, {
|
|
agentId: run.agentId,
|
|
state: run.status,
|
|
...(agent
|
|
? { adapterType: telemetryAdapterType(agent), agentRole: agent.role }
|
|
: {}),
|
|
...(model ? { model } : {}),
|
|
...(() => {
|
|
const durationSeconds = resolveDurationSeconds(run, new Date());
|
|
return durationSeconds === undefined ? {} : { durationSeconds };
|
|
})(),
|
|
...(run.usageJson != null
|
|
? {
|
|
inputTokens: readUsageToken(
|
|
usage,
|
|
"inputTokens",
|
|
"input_tokens",
|
|
"rawInputTokens",
|
|
"raw_input_tokens",
|
|
),
|
|
outputTokens: readUsageToken(
|
|
usage,
|
|
"outputTokens",
|
|
"output_tokens",
|
|
"rawOutputTokens",
|
|
"raw_output_tokens",
|
|
),
|
|
cachedTokens: readUsageToken(
|
|
usage,
|
|
"cachedInputTokens",
|
|
"cached_input_tokens",
|
|
"cacheReadInputTokens",
|
|
"cache_read_input_tokens",
|
|
),
|
|
}
|
|
: {}),
|
|
...(taskId ? { taskId } : {}),
|
|
...(failureCause ? { failureCause } : {}),
|
|
}));
|
|
} catch (err) {
|
|
logger.warn(
|
|
{ err, runId: run.id },
|
|
"failed to emit agent.task_run telemetry",
|
|
);
|
|
}
|
|
}
|
|
|
|
/** Loads a committed run before emitting telemetry, keeping database rows out of application contracts. */
|
|
export async function emitAgentTaskRunById(
|
|
db: Db,
|
|
input: { runId: string; companyId: string },
|
|
): Promise<void> {
|
|
try {
|
|
const run = await db
|
|
.select({
|
|
id: heartbeatRuns.id,
|
|
companyId: heartbeatRuns.companyId,
|
|
agentId: heartbeatRuns.agentId,
|
|
status: heartbeatRuns.status,
|
|
startedAt: heartbeatRuns.startedAt,
|
|
finishedAt: heartbeatRuns.finishedAt,
|
|
usageJson: heartbeatRuns.usageJson,
|
|
responsibleUserId: heartbeatRuns.responsibleUserId,
|
|
contextSnapshot: sql<Record<string, unknown>>`jsonb_build_object('issueId', ${heartbeatRuns.contextSnapshot}->'issueId')`,
|
|
errorCode: heartbeatRuns.errorCode,
|
|
resultJson: sql<Record<string, unknown> | null>`jsonb_build_object('providerFailure', jsonb_build_object('category', ${heartbeatRuns.resultJson}->'providerFailure'->'category'))`,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId)))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (run) await emitAgentTaskRun(db, run);
|
|
} catch (err) {
|
|
logger.warn({ err, runId: input.runId }, "failed to load run for agent.task_run telemetry");
|
|
}
|
|
}
|