Files
PaperClipAI/server/src/__tests__/issue-thread-interactions-telemetry.test.ts
T
DottaandClaude Opus 5.5 2a43fe3e34 feat(telemetry): add company_id to Cloud direct events from background work (#22183)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Cloud sends server telemetry straight to PostHog
(`server/src/services/posthog-direct/`)
> - Request events get `company_id` from the request context. Background
events have no request.
> - So `agent.task_run`, `routine.run`, `connection.invoked`, and other
background events arrive with `stack_id` but without `company_id`. On
staging, 66 of 66 `agent.task_run` events had a null `company_id`.
> - Without `company_id`, product analytics cannot split agent and
routine activity by company.
> - This pull request lets each background emit site name the company of
the entity that it reports on. Only the direct PostHog transport reads
that company.
> - The benefit is that Cloud background events carry `company_id`, and
the self-hosted collector output does not change.

## Linked Issues or Issue Description

Refs #22006

**Subsystem affected**
Server telemetry: Cloud direct delivery to PostHog
(`server/src/services/posthog-direct/`) and the server emit sites for
background events.

**Problem or motivation**
Cloud direct events from background work have no `company_id`. Heartbeat
runs, scheduled routines, connection invocations, and interaction
updates do not run inside an HTTP request, so the request context gives
no company. On paperclip.staging, all 66 sampled `agent.task_run` events
had `company_id` null.

**Proposed solution**
Add a small AsyncLocalStorage scope, `withTelemetryCompany(companyId,
fn)`, in `server/src/telemetry-company-scope.ts`. Each background emit
site wraps its typed `track*` call with the company id of the row that
it already has. The direct transport reads the scope at track time,
through the existing `captureEventContext()` seam. A live request
company still wins. The collector path never reads the scope.

**Alternatives considered**
1. An optional context argument on the shared `track*` helpers. This
changes the public helpers in `@paperclipai/shared` for a Cloud-only
value, so I did not use it.
2. A scope around each whole work unit (heartbeat run, routine run,
invocation). The emit calls are fire-and-forget and are spread across
many modules, so it is easy to miss one or to leak a company into other
work. A scope at the emit site is narrow and explicit.
3. Let the transport look up the company in the database. The transport
must not query the database.

**Roadmap alignment**
This completes the Cloud direct PostHog work from #22006. It does not
change the telemetry contract.

## What Changed

- Add `server/src/telemetry-company-scope.ts`. It has
`withTelemetryCompany(companyId, fn)` and `scopedTelemetryCompanyId()`.
With no company id, it calls `fn` directly.
- `capturePostHogEventContext()` uses the scoped company when no live
request gives a company. The request company, user, and session behave
as before.
- Wrap these emit sites with the company of their entity:
- `agent.task_run` (`emitAgentTaskRun`, `emitAgentTaskRunById`;
`companyId` is now part of the run that the function reads). This covers
heartbeat, native runtime, recovery, run-dispatch, and watchdog callers.
- `routine.run` (`routineService` dispatch, which covers scheduler,
webhook, and manual runs)
  - `agent.first_heartbeat` (heartbeat agent status)
  - `connection.created`, `connection.updated`, `connection.invoked`
  - `interaction.created`, `interaction.resolved`
- `agent.task_completed` (issue route; the issue company is the fallback
when the request gives no company)
- Request-only events (`project.created`, `goal.created`,
`agent.created`, `routine.created`, `skill.imported`) keep the request
context. `server.started` and `error_handler.crash` are not
company-scoped. Dynamic plugin and `question_response.delivery` events
are not in the contract, so the direct path drops them.
- Update the "Cloud Direct Delivery" section of
`packages/shared/src/telemetry/README.md`.

Design note on module boundaries: the scope lives beside
`server/src/telemetry.ts`, not inside `posthog-direct/`. Emit sites do
not import the transport, and the transport does not know which services
exist. The shared telemetry package does not change.

## Verification

- Red/green: with the source changes removed, these 4 new tests fail.
With the changes, they pass.
- `agent-task-run-telemetry.test.ts`: `emitAgentTaskRun` and
`emitAgentTaskRunById` track in the run company scope.
- `routine-run-telemetry.test.ts`: a routine run tracks in the routine
company scope.
- `posthog-direct-telemetry.test.ts`: on Cloud, background
`agent.task_run` and `routine.run` (also after an `await`) carry
`company_id`, and an unscoped event does not.
- `posthog-direct-telemetry.test.ts`: a live request company wins over
the scope, and the scope fills the gap when the request has no company.
- Red/green per emit family: the tests for `connection.created`,
`connection.updated`, `connection.invoked`, `interaction.created`,
`interaction.resolved`, `agent.first_heartbeat`, and
`agent.task_completed` assert the company scope at track time. With the
emit-site wrappers removed, all 7 assertions fail.
- New guard test: on self-hosted, a scoped `agent.task_run` and
`routine.run` produce a byte-identical collector envelope with no
`company` text.
- `cd server && npx vitest run` on 16 telemetry, PostHog, routine,
heartbeat status, interaction, and connector test files: 408 passed.
- `pnpm --filter @paperclipai/server typecheck`: passed.
- Local check with a real `posthog-node` client sent to a local capture
server (never real PostHog), embedded Postgres, and a real
`routineService.tickScheduledTriggers()` tick plus `emitAgentTaskRun`:
- Cloud mode: the captured `routine.run` (`source: schedule`) and
`agent.task_run` both had the seeded `company_id`, plus `stack_id` and
`environment`. `server.started` had no `company_id`.
- Self-hosted mode: the collector body from this branch and from
`origin/master` are identical after I normalize random ids and
timestamps.
- I did not boot a full Cloud-managed server, because it needs S3 and an
external Postgres. The check above uses the same server modules and the
real PostHog client.

## Risks

- Privacy review: this is a Telemetry change under
`rules/data-paths.mdc`, so I request a privacy review. Background Cloud
events now send the real Paperclip `company_id` to PostHog. #22006
already approved `company_id` as a Cloud direct property for request
events. This PR only fills it for background events. It adds no new
property, dimension, or event, so the generated contract does not
change. `packages/shared/src/telemetry/README.md` is updated in this PR.
- Low runtime risk. The self-hosted collector path does not read the
scope, and a test proves its output does not change.
- On Cloud, more events now have `company_id`. No event loses a field.
- If a request with a company tracks an event for an entity in a
different company, the request company wins. This is the same as before
this change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, and test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:34:39 -06:00

610 lines
18 KiB
TypeScript

import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import {
agents,
companies,
createDb,
documentRevisions,
documents,
goals,
heartbeatRuns,
issueComments,
issueDocuments,
issueThreadInteractions,
issues,
} from "@paperclipai/db";
import {
getEmbeddedPostgresTestSupport,
startEmbeddedPostgresTestDatabase,
} from "./helpers/embedded-postgres.js";
import { issueThreadInteractionService } from "../services/issue-thread-interactions.js";
import { scopedTelemetryCompanyId } from "../telemetry-company-scope.js";
const telemetryMocks = vi.hoisted(() => ({
track: vi.fn(),
getTelemetryClient: vi.fn(),
}));
vi.mock("../telemetry.js", () => ({
getTelemetryClient: telemetryMocks.getTelemetryClient,
}));
const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_INTERACTION_TELEMETRY_TEST_DATABASE_URL");
const embeddedPostgresSupport = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
describeEmbeddedPostgres("issueThreadInteractionService telemetry", () => {
let db!: ReturnType<typeof createDb>;
let interactionsSvc!: ReturnType<typeof issueThreadInteractionService>;
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
beforeAll(async () => {
if (!externalUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-issue-interaction-telemetry-");
db = createDb(externalUrl ?? tempDb!.connectionString);
interactionsSvc = issueThreadInteractionService(db);
}, 20_000);
beforeEach(() => {
telemetryMocks.track.mockClear();
telemetryMocks.track.mockImplementation(() => scopedTelemetryCompanyId());
telemetryMocks.getTelemetryClient.mockReturnValue({
track: telemetryMocks.track,
hashPrivateRef: vi.fn((value: string) => `hashed:${value}`),
});
});
afterEach(async () => {
await db.delete(issueThreadInteractions);
await db.delete(issueComments);
await db.delete(issueDocuments);
await db.delete(documentRevisions);
await db.delete(documents);
await db.delete(issues);
await db.delete(goals);
await db.delete(heartbeatRuns);
await db.delete(agents);
await db.delete(companies);
});
afterAll(async () => {
await db?.$client.end({ timeout: 1 });
await tempDb?.cleanup();
});
async function seedIssue(title = "Interaction telemetry") {
const companyId = randomUUID();
const goalId = randomUUID();
const issueId = randomUUID();
await db.insert(companies).values({
id: companyId,
name: "Paperclip",
issuePrefix: `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`,
requireBoardApprovalForNewAgents: false,
});
await db.insert(goals).values({
id: goalId,
companyId,
title,
level: "task",
status: "active",
});
await db.insert(issues).values({
id: issueId,
companyId,
goalId,
title,
status: "in_progress",
priority: "medium",
});
return { companyId, goalId, issueId };
}
async function seedAgent(companyId: string, role: string) {
const agentId = randomUUID();
await db.insert(agents).values({
id: agentId,
companyId,
name: role,
role,
status: "active",
adapterType: "codex_local",
adapterConfig: {},
runtimeConfig: {},
permissions: {},
});
return agentId;
}
function expectNoRawInteractionIds(dimensions: Record<string, unknown>) {
expect(dimensions).not.toHaveProperty("interaction_id");
expect(dimensions).not.toHaveProperty("created_by_agent_id");
expect(dimensions).not.toHaveProperty("source_run_id");
}
function lastInteractionResolvedDimensions() {
expect(telemetryMocks.track).toHaveBeenCalledWith("interaction.resolved", expect.any(Object));
const calls = telemetryMocks.track.mock.calls.filter((call) => call[0] === "interaction.resolved");
return calls.at(-1)?.[1] as Record<string, unknown>;
}
function lastInteractionCreatedDimensions() {
expect(telemetryMocks.track).toHaveBeenCalledWith("interaction.created", expect.any(Object));
const calls = telemetryMocks.track.mock.calls.filter((call) => call[0] === "interaction.created");
return calls.at(-1)?.[1] as Record<string, unknown>;
}
function scopesOf(name: string) {
return telemetryMocks.track.mock.calls.flatMap(([eventName], index) =>
eventName === name ? [telemetryMocks.track.mock.results[index]?.value] : []);
}
it("emits deprecated resolver alias use at creation without interaction content", async () => {
const { companyId, issueId } = await seedIssue("Deprecated alias telemetry issue title");
await interactionsSvc.create({
id: issueId,
companyId,
}, {
kind: "request_confirmation",
resolverPolicy: "board_only",
title: "Private confirmation title",
summary: "Private confirmation summary",
payload: {
version: 1,
prompt: "Private confirmation prompt",
detailsMarkdown: "Private confirmation details",
acceptLabel: "Private accept label",
rejectLabel: "Private reject label",
},
}, {
userId: "local-board",
});
expect(lastInteractionCreatedDimensions()).toEqual({
interaction_kind: "request_confirmation",
used_deprecated_resolver_policy_alias: true,
});
expect(scopesOf("interaction.created")).toEqual([companyId]);
});
it("emits skipped resolution telemetry for composer Skip", async () => {
const { companyId, issueId } = await seedIssue("Skipped interaction telemetry");
const created = await interactionsSvc.create({
id: issueId,
companyId,
}, {
kind: "request_confirmation",
continuationPolicy: "none",
payload: {
version: 1,
prompt: "Continue with the proposed change?",
},
}, {
userId: "local-board",
});
await interactionsSvc.skipInteraction({
id: issueId,
companyId,
}, created.id, {}, {
userId: "local-board",
});
const dimensions = lastInteractionResolvedDimensions();
expect(dimensions).toMatchObject({
interaction_kind: "request_confirmation",
status: "cancelled",
resolved_by_kind: "user",
resolution_reason: "skipped",
created_by_kind: "user",
continuation_policy: "none",
target_type: "none",
});
expectNoRawInteractionIds(dimensions);
expect(scopesOf("interaction.resolved")).toEqual([companyId]);
});
it("emits accepted suggested-task telemetry with created and skipped task counts", async () => {
const { companyId, goalId, issueId } = await seedIssue("Accept suggested tasks telemetry");
const created = await interactionsSvc.create({
id: issueId,
companyId,
}, {
kind: "suggest_tasks",
continuationPolicy: "wake_assignee",
payload: {
version: 1,
tasks: [
{
clientKey: "root",
title: "Create the root follow-up",
},
{
clientKey: "child",
parentClientKey: "root",
title: "Create the nested follow-up",
},
{
clientKey: "sibling",
title: "Create the sibling follow-up",
},
],
},
}, {
userId: "local-board",
});
await interactionsSvc.acceptInteraction({
id: issueId,
companyId,
goalId,
projectId: null,
}, created.id, {
selectedClientKeys: ["root"],
}, {
userId: "local-board",
});
const dimensions = lastInteractionResolvedDimensions();
expect(dimensions).toMatchObject({
interaction_kind: "suggest_tasks",
status: "accepted",
resolved_by_kind: "user",
resolution_reason: "accepted",
created_by_kind: "user",
continuation_policy: "wake_assignee",
target_type: "none",
created_task_count: 1,
skipped_task_count: 2,
});
expectNoRawInteractionIds(dimensions);
});
it("emits accepted checkbox telemetry with raw role, target, and counts", async () => {
const { companyId, goalId, issueId } = await seedIssue("Accept checkbox telemetry");
const creatorAgentId = await seedAgent(companyId, "Backend Engineer");
const created = await interactionsSvc.create({
id: issueId,
companyId,
}, {
kind: "request_checkbox_confirmation",
continuationPolicy: "wake_assignee",
payload: {
version: 1,
prompt: "Which files should be deleted?",
options: [
{ id: "file-a", label: "a.txt" },
{ id: "file-b", label: "b.txt" },
],
target: {
type: "custom",
key: "cleanup-plan",
},
},
}, {
agentId: creatorAgentId,
});
await interactionsSvc.acceptInteraction({
id: issueId,
companyId,
goalId,
projectId: null,
}, created.id, {
selectedOptionIds: ["file-b"],
}, {
userId: "local-board",
});
const dimensions = lastInteractionResolvedDimensions();
expect(dimensions).toMatchObject({
interaction_kind: "request_checkbox_confirmation",
status: "accepted",
resolved_by_kind: "user",
resolution_reason: "accepted",
created_by_kind: "agent",
creator_agent_role: "Backend Engineer",
continuation_policy: "wake_assignee",
target_type: "custom",
option_count: 2,
selected_option_count: 1,
});
expectNoRawInteractionIds(dimensions);
});
it("emits rejected confirmation telemetry and omits creator_agent_role for user-created interactions", async () => {
const { companyId, issueId } = await seedIssue("Reject confirmation telemetry");
const resolverAgentId = await seedAgent(companyId, "SecurityEngineer");
const resolverRunId = randomUUID();
await db.insert(heartbeatRuns).values({
id: resolverRunId,
companyId,
agentId: resolverAgentId,
status: "running",
});
const created = await interactionsSvc.create({
id: issueId,
companyId,
}, {
kind: "request_confirmation",
resolverPolicy: "board_or_agents",
payload: {
version: 1,
prompt: "Approve this?",
},
}, {
userId: "local-board",
});
await interactionsSvc.rejectInteraction({
id: issueId,
companyId,
}, created.id, {
reason: "Needs edits before approval.",
}, {
agentId: resolverAgentId,
runId: resolverRunId,
});
const dimensions = lastInteractionResolvedDimensions();
expect(dimensions).toMatchObject({
interaction_kind: "request_confirmation",
status: "rejected",
resolved_by_kind: "agent",
resolution_reason: "rejected",
created_by_kind: "user",
continuation_policy: "none",
target_type: "none",
});
expect(dimensions).not.toHaveProperty("creator_agent_role");
expect(dimensions).not.toHaveProperty("reason");
expectNoRawInteractionIds(dimensions);
});
it("emits answered question telemetry with system resolver and legacy restriction signal", async () => {
const { companyId, issueId } = await seedIssue("Answer question telemetry");
const creatorAgentId = await seedAgent(companyId, "Wizard");
const created = await interactionsSvc.create({
id: issueId,
companyId,
}, {
kind: "ask_user_questions",
continuationPolicy: "wake_assignee",
payload: {
version: 1,
questions: [
{
id: "scope",
prompt: "Choose the scope",
selectionMode: "single",
required: true,
options: [{ id: "phase-1", label: "Phase 1" }],
},
{
id: "extras",
prompt: "Optional extras",
selectionMode: "multi",
options: [{ id: "docs", label: "Docs" }],
},
],
},
}, {
agentId: creatorAgentId,
});
await db
.update(issueThreadInteractions)
.set({
continuationPolicy: "",
resolverPolicyProvenance: "legacy_inherited_restriction",
})
.where(eq(issueThreadInteractions.id, created.id));
await interactionsSvc.answerQuestions({
id: issueId,
companyId,
}, created.id, {
answers: [{ questionId: "scope", optionIds: ["phase-1"] }],
summaryMarkdown: "Do not emit this free text.",
}, {
systemId: "system:pr-merged",
});
const dimensions = lastInteractionResolvedDimensions();
expect(dimensions).toMatchObject({
interaction_kind: "ask_user_questions",
status: "answered",
resolved_by_kind: "system",
created_by_kind: "agent",
creator_agent_role: "Wizard",
target_type: "none",
question_count: 2,
answered_question_count: 1,
legacy_inherited_restriction: true,
});
expect(dimensions).not.toHaveProperty("continuation_policy");
expect(dimensions).not.toHaveProperty("summaryMarkdown");
expect(dimensions).not.toHaveProperty("answers");
expect(JSON.stringify(dimensions)).not.toContain("Choose the scope");
expect(JSON.stringify(dimensions)).not.toContain("Phase 1");
expect(JSON.stringify(dimensions)).not.toContain("Do not emit this free text.");
expectNoRawInteractionIds(dimensions);
});
it("emits expired question telemetry with zero answered question count", async () => {
const { companyId, issueId } = await seedIssue("Expired question telemetry");
const commentId = randomUUID();
const created = await interactionsSvc.create({
id: issueId,
companyId,
}, {
kind: "ask_user_questions",
continuationPolicy: "wake_assignee",
payload: {
version: 1,
supersedeOnUserComment: true,
questions: [
{
id: "scope",
prompt: "Choose the scope",
selectionMode: "single",
required: true,
options: [{ id: "phase-1", label: "Phase 1" }],
},
],
},
}, {
userId: "local-board",
});
const expired = await interactionsSvc.expireRequestConfirmationsSupersededByComment({
id: issueId,
companyId,
}, {
id: commentId,
createdAt: new Date(new Date(created.createdAt).getTime() + 1_000),
authorUserId: "local-board",
}, {
userId: "local-board",
});
expect(expired).toHaveLength(1);
const dimensions = lastInteractionResolvedDimensions();
expect(dimensions).toMatchObject({
interaction_kind: "ask_user_questions",
status: "expired",
resolved_by_kind: "user",
resolution_reason: "superseded_by_comment",
created_by_kind: "user",
continuation_policy: "wake_assignee",
target_type: "none",
question_count: 1,
answered_question_count: 0,
});
expectNoRawInteractionIds(dimensions);
});
it("emits expired stale-target telemetry without stale target identifiers", async () => {
const { companyId, issueId } = await seedIssue("Stale target telemetry");
const documentId = randomUUID();
const revisionId = randomUUID();
await db.insert(documents).values({
id: documentId,
companyId,
title: "Plan",
format: "markdown",
latestBody: "v1",
latestRevisionId: revisionId,
latestRevisionNumber: 1,
});
await db.insert(issueDocuments).values({
companyId,
issueId,
documentId,
key: "plan",
});
await db.insert(documentRevisions).values({
id: revisionId,
companyId,
documentId,
revisionNumber: 1,
title: "Plan",
format: "markdown",
body: "v1",
});
await interactionsSvc.create({
id: issueId,
companyId,
}, {
kind: "request_confirmation",
payload: {
version: 1,
prompt: "Approve the plan?",
target: {
type: "issue_document",
issueId,
documentId,
key: "plan",
revisionId,
revisionNumber: 1,
},
},
}, {
userId: "local-board",
});
const expired = await interactionsSvc.expireStaleRequestConfirmationsForIssueDocument({
id: issueId,
companyId,
}, null, {});
expect(expired).toHaveLength(1);
const dimensions = lastInteractionResolvedDimensions();
expect(dimensions).toMatchObject({
interaction_kind: "request_confirmation",
status: "expired",
resolved_by_kind: "system",
resolution_reason: "stale_target",
created_by_kind: "user",
continuation_policy: "none",
target_type: "issue_document",
});
expect(dimensions).not.toHaveProperty("staleTarget");
expect(dimensions).not.toHaveProperty("revisionId");
expectNoRawInteractionIds(dimensions);
});
it("emits superseded expiration telemetry without comment identifiers", async () => {
const { companyId, issueId } = await seedIssue("Superseded telemetry");
const commentId = randomUUID();
const created = await interactionsSvc.create({
id: issueId,
companyId,
}, {
kind: "request_confirmation",
payload: {
version: 1,
prompt: "Approve this plan?",
supersedeOnUserComment: true,
},
}, {
userId: "local-board",
});
const expired = await interactionsSvc.expireRequestConfirmationsSupersededByComment({
id: issueId,
companyId,
}, {
id: commentId,
createdAt: new Date(new Date(created.createdAt).getTime() + 1_000),
authorUserId: "local-board",
}, {
userId: "local-board",
});
expect(expired).toHaveLength(1);
const dimensions = lastInteractionResolvedDimensions();
expect(dimensions).toMatchObject({
interaction_kind: "request_confirmation",
status: "expired",
resolved_by_kind: "user",
resolution_reason: "superseded_by_comment",
created_by_kind: "user",
target_type: "none",
});
expect(dimensions).not.toHaveProperty("commentId");
expectNoRawInteractionIds(dimensions);
});
});