Files
PaperClipAI/server/src/__tests__/posthog-direct-telemetry.test.ts
T
DottaandClaude Opus 5.5 067f7a0815 feat(telemetry): send product lifecycle events and agent run failure cause (#22250)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - First-party telemetry tells the team which product steps people
finish
> - Telemetry did not record when a model is connected, when a task is
created, completed, or blocked, when a plan is created or approved, or
when an artifact is created
> - Telemetry also did not tell why an agent run failed, so failures
could not be grouped by cause
> - This pull request syncs the new telemetry contract and sends these
events from the server once per real change
> - The benefit is a product funnel and a failure breakdown that use
only enums, booleans, and counts

## Linked Issues or Issue Description

**Subsystem affected**
Server telemetry (`packages/shared/src/telemetry`,
`server/src/services/product-telemetry.ts`) and the emit sites in the
issue, document, work product, interaction, and AI connection services.

**Problem or motivation**
The team cannot measure the main product path. There is no event for a
new model connection, a new task, a completed task, a blocked task, a
new plan, an approved plan, or a new artifact. `agent.task_run` tells
that a run failed, but not why. The signup funnel also must know when
the onboarding wizard created the first task.

**Proposed solution**
Register the events in the telemetry backend. Sync the generated
contract here. Add typed helpers. Send each event once, after the
transaction that makes the change commits. Use fixed enums for every
kind, reason, and cause.

**Alternatives considered**
A post-commit action queue on the issue service only. The native status
path and many other callers use an external transaction without that
queue, so most agent completions would not send an event. A generic
after-commit hook on the database client covers every caller.

**Roadmap alignment**
This is telemetry for existing features. It adds no new product feature.

**Additional context**
This change depends on paperclipai/telemetry-backend#150, which is
merged. Deploy that backend change before this change ships. Until then,
the collector rejects the new event names.

## What Changed

- Sync `generated/paperclip-telemetry.ts` from the backend branch as a
direct copy. Regenerate `posthog-direct/enum-fallbacks.generated.ts`
with `scripts/generate-posthog-direct-fixtures.mjs`. The loader golden
file did not change.
- Add `afterTransactionCommit()` to `@paperclipai/db`. It runs a
callback after the outermost transaction commits. It drops the callback
on rollback. Outside a transaction it runs the callback at once. A
callback that throws never fails the committed work.
- Add helpers `trackModelConnected`, `trackTaskCreated`,
`trackTaskCompleted`, `trackTaskBlocked`, `trackPlanCreated`,
`trackPlanApproved`, and `trackArtifactCreated`. Add `failureCause` to
`trackAgentTaskRun`. Assign the new events to `operational_enum_count`
retention.
- Add `server/src/services/product-telemetry.ts`. It sends each event
after commit inside `withTelemetryCompany()`, so Cloud direct delivery
gets `company_id` for background writes.
- Emit sites:
- `model.connected`: `aiConnectionService.save`, for a new connection
only. A reconnect sends nothing. `first_for_company` is true when the
company had no earlier model connection.
- `task.created`: `issueService.create`, after the insert. A
deduplicated create, a company import, and a chat conversation send
nothing. `created_by_kind` is `routine` for routine runs, `onboarding`
for the onboarding wizard's first task, and otherwise `user`, `agent`,
or `system`.
- `task.completed` and `task.blocked`: `issueService.update`, when the
status changes to `done` or `blocked`. The same status again sends
nothing.
- `plan.created`: `documentService.upsertIssueDocument`, for the first
revision of the `plan` document.
- `plan.approved`: the accept path of a `request_confirmation` that
targets the task's `plan` document.
- `artifact.created`: `workProductService.createForIssue` (`direct`),
the promotion route (`promotion`), and a run file deliverable
(`run_deliverable`).
- `agent.task_run.failure_cause`: `emitAgentTaskRun`, for `failed`,
`timed_out`, and `interrupted` runs only.
- `blocked_reason` mapping: unfinished blocking tasks give
`waiting_on_task`. Otherwise an unblock owner of a person or the board
gives `waiting_for_person`, and an agent owner gives
`waiting_for_agent`. A system write with neither gives `needs_recovery`.
A person or agent write with neither gives `unspecified`.
- `failure_cause` mapping (stored `errorCode`, with
`resultJson.providerFailure.category` for generic provider and adapter
codes):
- `missing_access`: auth and permission codes (`acpx_auth_required`,
`provider_auth_failed`, `user_secret_missing`, `github_access_denied`,
`github_repo_not_accessible`, `permission_denied`,
`project_access_denied`, `configuration_incomplete`,
`connection_unavailable`, `signing_secret_unconfigured`), and provider
category `authentication` or `permission`.
- `provider_limit`: `provider_quota`, `native_provider_usage_limit`,
`native_provider_overloaded`, `github_rate_limited`,
`ai_connection_busy`, and provider category `rate_limit`,
`quota_exhausted`, or `overloaded`.
- `provider_error`: `native_provider_model_rejected`,
`provider_transport_failed`, `codex_transient_upstream`,
`provider_failed`, `provider_tool_definition_invalid`, and the other
provider categories.
- `waiting_for_person`: `execution_reconciliation_required`, every code
in `EXECUTION_RECONCILIATION_CAUSES`, `issue_waiting_for_response`, and
`issue_continuation_waiting_on_review`.
- `task_changed`: the task was reassigned, paused, closed, blocked, or
stopped by a person (`operator_interrupted`, `cancelled`, `interrupted`,
`issue_*` lifecycle codes, `agent_paused`, `queued_comment_discarded`).
- `platform_error`: process, server, workspace, sandbox, runner, and
storage codes (`process_lost`, `server_shutdown_interrupted`,
`setup_failed`, `workspace_*`, `environment_provider_unavailable`,
`controller_lease_lost`, and others).
- `agent_error`: `adapter_failed` without a provider failure,
`invalid_output`, `invalid_response`, `tool_error`, `acpx_turn_failed`,
`test_run_failed`, `prompt_injection_blocked`,
`tool_execution_outcome_unknown`, `timeout`.
  - `unknown`: any other code, or no code.
- On Cloud direct delivery, every new emit site names the responsible
person with `withTelemetryActor()` and `resolveTelemetryPerson()`, as
`agent.task_run` does: the task creator, else the task's responsible
user, else the run owner. `task.created` uses the creating user, else
the agent's on-behalf user. `model.connected` uses the user who owns the
connection. A live request still wins. The self-hosted collector
envelope does not change.
- `failure_cause` builds on the run failure code sets in
`continuation-cause.ts`. A test fails when a code in those sets has no
telemetry code. The provider failure category decides only for
`adapter_failed`, `provider_failed`, or no code.
- Work product providers outside `paperclip`, `github`, `vercel`, and
`s3` are sent as `custom`.
- The dispatch workspace check and execution recovery write `blocked`
directly. They call the same after-commit helper, with the task's real
blockers and unblock owner. The blocker read runs in a savepoint in the
issue dependency layer, so a failed read never aborts the block write.
- Update the telemetry README with the emit rules, the after-commit
rule, and the `failure_cause` source.
- Refresh `tests/runner-e2e/source-sdk-public-inputs.json`, because its
check reported a change.

## Privacy Review

Privacy review requested. Every new dimension is a fixed enum, a
boolean, or a count. No dimension carries a title, a body, a plan step,
a connection or account name, an email, a URL, a repository name, a
prompt, an error message, or an endpoint address. `failure_cause` is a
code that the server picks. The run's error text is never sent. The
tests check that a provider message and a connection name do not reach
the event.

## Verification

- `pnpm vitest run packages/shared/src/telemetry/`: 7 files pass,
including the new `product-events.test.ts` (helper payloads, omitted
optional values, collector envelope with only registered dimensions,
retention class).
- `packages/db/src/after-transaction-commit.test.ts`: 4 pass (after
commit, rollback drop, savepoint, outside a transaction).
- `server/src/__tests__/product-telemetry.test.ts`: 14 pass. They check
that each event fires once with the right kind and company scope, that a
rollback sends nothing, that a repeated status sends nothing, that a
later plan revision sends nothing, and that a reconnect sends nothing.
- `server/src/__tests__/agent-task-run-telemetry.test.ts`: 35 pass,
including the `failure_cause` mapping table and the emit path.
- `server/src/__tests__/issue-thread-interactions-service.test.ts -t
Plan`: `plan.approved` fires for a plan confirmation and not for an
unrelated confirmation.
- `server/src/__tests__/posthog-direct-telemetry.test.ts`: the Cloud
direct path sends `company_id`, `user_id`, and `attributed_via` for
background lifecycle events, an event with no person stays on the
workspace, and `failure_cause` passes through. A self-hosted test checks
that a person scope leaves the collector events unchanged.
- Existing telemetry tests (connector, interaction, run finalizer,
heartbeat, AI connection): 299 pass.
- `pnpm --filter @paperclipai/shared typecheck`, `pnpm --filter
@paperclipai/db typecheck`, and `pnpm --filter @paperclipai/server
typecheck`: pass.
- `pnpm verify` returned exit 2 (global CI input changed). Required CI
supplies the broad coverage.

## Risks

- Merge order: merge and deploy paperclipai/telemetry-backend#150 first.
If this change ships first, the collector quarantines the new events.
The product behavior does not change.
- `afterTransactionCommit` uses the existing open-transaction marker in
the database client. A callback that a savepoint registers runs when the
outer transaction commits, even if that savepoint rolled back.
- `task.blocked` adds one small indexed read inside the update
transaction, only when a task enters `blocked`.
- `failure_cause` adds one small read for a failed run when the caller
did not pass the error code. It runs off the run's critical path, as
before.
- Low risk otherwise. No schema migration. No change to existing event
dimensions.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code,
with tool use (shell, Git, tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 19:32:27 -06:00

603 lines
28 KiB
TypeScript

import { createHash } from "node:crypto";
import { mkdtempSync, readFileSync, rmSync } from "node:fs";
import os from "node:os";
import path from "node:path";
import express from "express";
import request from "supertest";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { trackAgentTaskRun, trackCompanyImported, trackRoutineCreated, trackRoutineRun, trackPlanApproved, trackTaskBlocked, trackTaskCompleted, trackTaskCreated } from "@paperclipai/shared/telemetry";
const hoisted = vi.hoisted(() => ({ instanceRoot: "", posthogNodeLoads: 0 }));
vi.mock("../home-paths.js", () => ({
resolvePaperclipInstanceRoot: () => hoisted.instanceRoot,
}));
// Counts real loads of the package. Self-hosted servers must never load it.
vi.mock("posthog-node", () => {
hoisted.posthogNodeLoads += 1;
return { PostHog: class {} };
});
import {
getTelemetryClient,
initTelemetry,
isPostHogDirectTelemetryActive,
resetTelemetryForTests,
shutdownTelemetry,
} from "../telemetry.js";
import type { PostHogNodeClient } from "../services/posthog-direct/index.js";
import {
capturePostHogEventContext,
postHogRequestContextMiddlewareForEnv,
} from "../services/posthog-direct/request-context.js";
import { resolveTelemetryPerson, withTelemetryActor, withTelemetryCompany } from "../telemetry-company-scope.js";
import { serverVersion } from "../version.js";
const COLLECTOR = "http://collector.test/ingest";
const CLOUD_ENV = {
PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN: "tenant-token",
PAPERCLIP_CLOUD_STACK_ID: "stack-123",
PAPERCLIP_STACK_SLUG: "acme",
PAPERCLIP_PRIMARY_HOST: "acme.paperclip.app",
PAPERCLIP_POSTHOG_PROJECT_TOKEN: "phc_test",
PAPERCLIP_POSTHOG_ENVIRONMENT: "staging",
};
function fakePostHog() {
const captured: Array<Parameters<PostHogNodeClient["capture"]>[0]> = [];
const groups: Array<Parameters<PostHogNodeClient["groupIdentify"]>[0]> = [];
const client: PostHogNodeClient & { shutdownCalls: number } = {
shutdownCalls: 0,
capture: (message) => void captured.push(message),
groupIdentify: (message) => void groups.push(message),
flush: async () => {},
shutdown: async () => {
client.shutdownCalls += 1;
},
};
const factory = vi.fn(async () => client);
return { client, captured, groups, factory };
}
function readState() {
return JSON.parse(readFileSync(path.join(hoisted.instanceRoot, "telemetry", "state.json"), "utf-8")) as {
installId: string;
salt: string;
};
}
describe("server telemetry sender selection", () => {
let fetchMock: ReturnType<typeof vi.fn>;
beforeEach(() => {
hoisted.instanceRoot = mkdtempSync(path.join(os.tmpdir(), "paperclip-posthog-direct-"));
hoisted.posthogNodeLoads = 0;
for (const name of ["CI", "CONTINUOUS_INTEGRATION", "BUILD_NUMBER", "GITHUB_ACTIONS", "GITLAB_CI", "DO_NOT_TRACK", "PAPERCLIP_TELEMETRY_DISABLED"]) {
vi.stubEnv(name, "");
}
vi.stubEnv("PAPERCLIP_TELEMETRY_ENDPOINT", COLLECTOR);
fetchMock = vi.fn().mockResolvedValue({ ok: true });
vi.stubGlobal("fetch", fetchMock);
});
afterEach(() => {
resetTelemetryForTests();
vi.unstubAllEnvs();
vi.unstubAllGlobals();
rmSync(hoisted.instanceRoot, { recursive: true, force: true });
});
it("self-hosted: posts the unchanged hashed envelope to the collector and never loads posthog-node", async () => {
const client = initTelemetry({ enabled: true }, { env: {} })!;
expect(isPostHogDirectTelemetryActive()).toBe(false);
trackAgentTaskRun(client, { agentId: "agent-1", state: "succeeded", taskId: "task-raw-id" });
await shutdownTelemetry();
expect(hoisted.posthogNodeLoads).toBe(0);
expect(fetchMock).toHaveBeenCalledTimes(1);
expect(fetchMock.mock.calls[0]![0]).toBe(COLLECTOR);
const body = String((fetchMock.mock.calls[0]![1] as RequestInit).body);
const parsed = JSON.parse(body);
const state = readState();
const hashedTaskId = createHash("sha256").update(state.salt + "task-raw-id").digest("hex").slice(0, 16);
// Byte-identical to the envelope the collector path has always sent.
expect(body).toBe(
JSON.stringify({
app: "paperclip",
schemaVersion: "1",
installId: state.installId,
version: serverVersion,
events: [
{ name: "server.started", occurredAt: parsed.events[0].occurredAt, dimensions: { os_platform: process.platform } },
{
name: "agent.task_run",
occurredAt: parsed.events[1].occurredAt,
dimensions: { agent_id: "agent-1", state: "succeeded", task_id: hashedTaskId },
},
],
batchId: parsed.batchId,
}),
);
});
it("self-hosted: a background company scope leaves the collector envelope byte-identical", async () => {
const client = initTelemetry({ enabled: true }, { env: {} })!;
await withTelemetryCompany("company-bg", async () => {
await Promise.resolve();
trackAgentTaskRun(client, { agentId: "agent-1", state: "succeeded" });
trackRoutineRun(client, { source: "schedule", status: "issue_created" });
});
await shutdownTelemetry();
expect(hoisted.posthogNodeLoads).toBe(0);
expect(fetchMock).toHaveBeenCalledTimes(1);
const body = String((fetchMock.mock.calls[0]![1] as RequestInit).body);
const parsed = JSON.parse(body);
expect(body).toBe(
JSON.stringify({
app: "paperclip",
schemaVersion: "1",
installId: readState().installId,
version: serverVersion,
events: [
{ name: "server.started", occurredAt: parsed.events[0].occurredAt, dimensions: { os_platform: process.platform } },
{
name: "agent.task_run",
occurredAt: parsed.events[1].occurredAt,
dimensions: { agent_id: "agent-1", state: "succeeded" },
},
{
name: "routine.run",
occurredAt: parsed.events[2].occurredAt,
dimensions: { source: "schedule", status: "issue_created" },
},
],
batchId: parsed.batchId,
}),
);
expect(body).not.toContain("company");
});
it("self-hosted: a background person scope leaves the collector envelope byte-identical", async () => {
const client = initTelemetry({ enabled: true }, { env: {} })!;
const person = resolveTelemetryPerson({ taskCreatedByUserId: "user-creator" });
await withTelemetryActor({ companyId: "company-bg", person }, async () => {
await Promise.resolve();
trackAgentTaskRun(client, { agentId: "agent-1", state: "succeeded" });
});
await shutdownTelemetry();
expect(hoisted.posthogNodeLoads).toBe(0);
expect(fetchMock).toHaveBeenCalledTimes(1);
const body = String((fetchMock.mock.calls[0]![1] as RequestInit).body);
const parsed = JSON.parse(body);
expect(body).toBe(
JSON.stringify({
app: "paperclip",
schemaVersion: "1",
installId: readState().installId,
version: serverVersion,
events: [
{ name: "server.started", occurredAt: parsed.events[0].occurredAt, dimensions: { os_platform: process.platform } },
{
name: "agent.task_run",
occurredAt: parsed.events[1].occurredAt,
dimensions: { agent_id: "agent-1", state: "succeeded" },
},
],
batchId: parsed.batchId,
}),
);
expect(body).not.toContain("user-creator");
expect(body).not.toContain("attributed_via");
});
it("self-hosted: a person scope leaves product lifecycle collector events unchanged", async () => {
const client = initTelemetry({ enabled: true }, { env: {} })!;
const person = resolveTelemetryPerson({ taskCreatedByUserId: "user-creator" });
withTelemetryActor({ companyId: "company-bg", person }, () => {
trackTaskCompleted(client, { completedByKind: "agent", isSubtask: false });
trackTaskCreated(client, { createdByKind: "onboarding", isSubtask: false });
});
await shutdownTelemetry();
const body = String((fetchMock.mock.calls[0]![1] as RequestInit).body);
const events = JSON.parse(body).events.filter((event: { name: string }) => event.name !== "server.started");
expect(events.map((event: { name: string; dimensions: unknown }) => [event.name, event.dimensions])).toEqual([
["task.completed", { completed_by_kind: "agent", is_subtask: false }],
["task.created", { created_by_kind: "onboarding", is_subtask: false }],
]);
expect(body).not.toContain("user-creator");
expect(body).not.toContain("attributed_via");
expect(body).not.toContain("company-bg");
});
it("self-hosted: ignores posthog-js tracing headers", async () => {
expect(postHogRequestContextMiddlewareForEnv({})).toBeNull();
expect(postHogRequestContextMiddlewareForEnv({ PAPERCLIP_POSTHOG_PROJECT_TOKEN: "phc_test" })).toBeNull();
const app = express();
let context: unknown = "unset";
app.get("/probe", (_req, res) => {
context = capturePostHogEventContext();
res.json({});
});
await request(app).get("/probe").set("X-POSTHOG-SESSION-ID", "sess-1").expect(200);
expect(context).toBeUndefined();
});
it("Cloud: sends each batch only to PostHog with real ids, and identifies the workspace group", async () => {
const posthog = fakePostHog();
const client = initTelemetry({ enabled: true }, { env: CLOUD_ENV, postHogFactory: posthog.factory })!;
expect(isPostHogDirectTelemetryActive()).toBe(true);
trackAgentTaskRun(client, { agentId: "agent-1", state: "succeeded", taskId: "task-raw-id" });
trackCompanyImported(client, { sourceType: "github", sourceRef: "acme/private-repo", isPrivate: true });
await shutdownTelemetry();
expect(fetchMock).not.toHaveBeenCalled();
expect(posthog.captured.map((message) => message.event).sort()).toEqual([
"agent.task_run",
"cloud.workspace_started",
"company.imported",
"server.started",
]);
// A private source ref is customer content: it stays hashed on Cloud too.
const imported = posthog.captured.find((message) => message.event === "company.imported")!;
const salt = readState().salt;
expect(imported.properties).toMatchObject({
source_ref: createHash("sha256").update(salt + "acme/private-repo").digest("hex").slice(0, 16),
source_ref_hashed: true,
});
expect(JSON.stringify(imported)).not.toContain("acme/private-repo");
const run = posthog.captured.find((message) => message.event === "agent.task_run")!;
expect(run.distinctId).toBe("workspace:stack-123");
expect(run.groups).toEqual({ workspace: "stack-123" });
expect(run.disableGeoip).toBe(true);
expect(run.properties).toMatchObject({
agent_id: "agent-1",
task_id: "task-raw-id",
stack_id: "stack-123",
environment: "staging",
deployment_source: "cloud",
$groups: { workspace: "stack-123" },
$process_person_profile: false,
});
expect(run.properties).not.toHaveProperty("install_hash");
const installHash = createHash("sha256").update(readState().installId).digest("hex");
const started = posthog.captured.find((message) => message.event === "cloud.workspace_started")!;
expect(started.distinctId).toBe("workspace:stack-123");
expect(started.groups).toEqual({ workspace: "stack-123" });
expect(started.properties).toMatchObject({
app: "paperclip",
schema_version: "1",
source: "client",
deployment_source: "cloud",
version: serverVersion,
stack_id: "stack-123",
install_hash: installHash,
attributed_via: "none",
environment: "staging",
distinct_id: "workspace:stack-123",
$process_person_profile: false,
});
for (const message of posthog.captured) {
expect(message.properties?.environment, message.event).toBe("staging");
if (message.event !== "$groupidentify") expect(message.properties?.attributed_via, message.event).toBe("none");
if (message.event !== "cloud.workspace_started") expect(message.properties).not.toHaveProperty("install_hash");
}
expect(posthog.groups).toEqual([
{
groupType: "workspace",
groupKey: "stack-123",
disableGeoip: true,
properties: { stack_slug: "acme", primary_host: "acme.paperclip.app", environment: "staging" },
},
]);
expect(posthog.client.shutdownCalls).toBe(1);
});
it("Cloud: stays on the collector when PAPERCLIP_POSTHOG_ENVIRONMENT is unset", async () => {
const posthog = fakePostHog();
const { PAPERCLIP_POSTHOG_ENVIRONMENT: _omit, ...noEnvironment } = CLOUD_ENV;
const client = initTelemetry({ enabled: true }, { env: noEnvironment, postHogFactory: posthog.factory })!;
expect(isPostHogDirectTelemetryActive()).toBe(false);
trackRoutineCreated(client);
await shutdownTelemetry();
expect(posthog.factory).not.toHaveBeenCalled();
expect(fetchMock).toHaveBeenCalledTimes(1);
expect(fetchMock.mock.calls[0]![0]).toBe(COLLECTOR);
expect(postHogRequestContextMiddlewareForEnv(noEnvironment)).toBeNull();
});
it("Cloud: the kill switch falls back to the collector", async () => {
const posthog = fakePostHog();
const client = initTelemetry(
{ enabled: true },
{ env: { ...CLOUD_ENV, PAPERCLIP_POSTHOG_DIRECT: "0" }, postHogFactory: posthog.factory },
)!;
expect(isPostHogDirectTelemetryActive()).toBe(false);
trackRoutineCreated(client);
await shutdownTelemetry();
expect(posthog.factory).not.toHaveBeenCalled();
expect(fetchMock).toHaveBeenCalledTimes(1);
expect(postHogRequestContextMiddlewareForEnv({ ...CLOUD_ENV, PAPERCLIP_POSTHOG_DIRECT: "0" })).toBeNull();
});
it("Cloud: links request events to the acting user and browser session", async () => {
const posthog = fakePostHog();
initTelemetry({ enabled: true }, { env: CLOUD_ENV, postHogFactory: posthog.factory });
const middleware = postHogRequestContextMiddlewareForEnv(CLOUD_ENV);
expect(middleware).not.toBeNull();
const app = express();
app.use(middleware!);
app.use((req, _res, next) => {
const as = req.header("x-test-actor");
req.actor =
as === "user"
? { type: "board", userId: "user-1", source: "session" }
: as === "agent"
? { type: "agent", agentId: "agent-1", companyId: "company-1", source: "agent_key" }
: { type: "none", source: "none" };
next();
});
app.post("/companies/:companyId/routines", (_req, res) => {
trackRoutineCreated(getTelemetryClient()!);
res.json({});
});
await request(app)
.post("/companies/company-9/routines")
.set("x-test-actor", "user")
.set("X-POSTHOG-SESSION-ID", "0190f6b2-7c1d-7a3e-9f00-123456789abc")
.set("X-POSTHOG-WINDOW-ID", "0190f6b2-7c1d-7a3e-9f00-abcdefabcdef")
// Never used for identity.
.set("X-POSTHOG-DISTINCT-ID", "spoofed-person")
.expect(200);
await request(app).post("/companies/company-9/routines").set("x-test-actor", "agent").expect(200);
await request(app)
.post("/companies/company-9/routines")
.set("X-POSTHOG-SESSION-ID", "bad id with spaces")
// An anonymous request cannot claim a person through the header either.
.set("X-POSTHOG-DISTINCT-ID", "spoofed-person")
.expect(200);
await shutdownTelemetry();
const routines = posthog.captured.filter((message) => message.event === "routine.created");
expect(routines).toHaveLength(3);
expect(routines[0]!.distinctId).toBe("user-1");
expect(routines[0]!.properties).toMatchObject({
user_id: "user-1",
company_id: "company-9",
$session_id: "0190f6b2-7c1d-7a3e-9f00-123456789abc",
$window_id: "0190f6b2-7c1d-7a3e-9f00-abcdefabcdef",
});
expect(routines[0]!.properties).not.toHaveProperty("$process_person_profile");
expect(JSON.stringify(routines)).not.toContain("spoofed-person");
expect(routines[1]!.distinctId).toBe("workspace:stack-123");
expect(routines[1]!.properties).toMatchObject({ company_id: "company-1", $process_person_profile: false });
expect(routines[1]!.properties).not.toHaveProperty("user_id");
expect(routines[2]!.distinctId).toBe("workspace:stack-123");
expect(routines[2]!.properties).not.toHaveProperty("$session_id");
expect(routines[2]!.properties).not.toHaveProperty("user_id");
expect(routines[2]!.properties).toMatchObject({ distinct_id: "workspace:stack-123", $process_person_profile: false });
});
it("Cloud: uses a board user's only company on routes without a company id", async () => {
const posthog = fakePostHog();
initTelemetry({ enabled: true }, { env: CLOUD_ENV, postHogFactory: posthog.factory });
const app = express();
app.use(postHogRequestContextMiddlewareForEnv(CLOUD_ENV)!);
app.use((req, _res, next) => {
const companyIds = req.header("x-test-companies")!.split(",");
req.actor = { type: "board", userId: "user-1", companyIds, source: "cloud_tenant" };
next();
});
app.post("/issues/:id/interactions/:interactionId/reject", (_req, res) => {
trackRoutineCreated(getTelemetryClient()!);
res.json({});
});
await request(app).post("/issues/i-1/interactions/x-1/reject").set("x-test-companies", "company-1").expect(200);
await request(app)
.post("/issues/i-1/interactions/x-1/reject")
.set("x-test-companies", "company-1,company-2")
.expect(200);
await shutdownTelemetry();
const routines = posthog.captured.filter((message) => message.event === "routine.created");
expect(routines[0]!.properties).toMatchObject({ company_id: "company-1", user_id: "user-1" });
expect(routines[1]!.properties).not.toHaveProperty("company_id");
});
it("Cloud: work that outlives its request does not inherit the request's user", async () => {
const posthog = fakePostHog();
initTelemetry({ enabled: true }, { env: CLOUD_ENV, postHogFactory: posthog.factory });
const app = express();
app.use(postHogRequestContextMiddlewareForEnv(CLOUD_ENV)!);
let later: Promise<void> | null = null;
app.post("/start", (req, res) => {
req.actor = { type: "board", userId: "user-1", source: "session" };
later = new Promise((resolve) => {
res.on("finish", () =>
setTimeout(() => {
trackRoutineCreated(getTelemetryClient()!);
resolve();
}, 5),
);
});
res.json({});
});
await request(app).post("/start").set("X-POSTHOG-SESSION-ID", "sess-1").expect(200);
await later;
await shutdownTelemetry();
const routine = posthog.captured.find((message) => message.event === "routine.created")!;
expect(routine.distinctId).toBe("workspace:stack-123");
expect(routine.properties).not.toHaveProperty("$session_id");
});
it("Cloud: background events carry the company in scope", async () => {
const posthog = fakePostHog();
const client = initTelemetry({ enabled: true }, { env: CLOUD_ENV, postHogFactory: posthog.factory })!;
await withTelemetryCompany("company-bg", async () => {
await new Promise((resolve) => setTimeout(resolve, 1));
trackAgentTaskRun(client, { agentId: "agent-1", state: "succeeded" });
});
withTelemetryCompany("company-bg", () => trackRoutineRun(client, { source: "schedule", status: "issue_created" }));
trackRoutineCreated(client);
await shutdownTelemetry();
const run = posthog.captured.find((message) => message.event === "agent.task_run")!;
expect(run.distinctId).toBe("workspace:stack-123");
expect(run.properties).toMatchObject({ company_id: "company-bg", stack_id: "stack-123", $process_person_profile: false });
expect(run.properties).not.toHaveProperty("user_id");
const routineRun = posthog.captured.find((message) => message.event === "routine.run")!;
expect(routineRun.properties).toMatchObject({ company_id: "company-bg", stack_id: "stack-123" });
const unscoped = posthog.captured.find((message) => message.event === "routine.created")!;
expect(unscoped.properties).not.toHaveProperty("company_id");
});
it("Cloud: background product lifecycle events carry the company in scope", async () => {
const posthog = fakePostHog();
const client = initTelemetry({ enabled: true }, { env: CLOUD_ENV, postHogFactory: posthog.factory })!;
const person = resolveTelemetryPerson({ taskCreatedByUserId: "user-creator" });
withTelemetryActor({ companyId: "company-bg", person }, () => {
trackTaskCreated(client, { createdByKind: "routine", isSubtask: false });
trackTaskCompleted(client, { completedByKind: "agent", isSubtask: true });
trackTaskBlocked(client, { blockedReason: "waiting_on_task", blockedByKind: "system" });
trackAgentTaskRun(client, { agentId: "agent-1", state: "failed", failureCause: "provider_limit" });
});
withTelemetryCompany("company-bg", () => trackPlanApproved(client, { approvedByKind: "system", startsExecution: false }));
await shutdownTelemetry();
for (const event of ["task.created", "task.completed", "task.blocked", "agent.task_run"]) {
const message = posthog.captured.find((captured) => captured.event === event)!;
expect(message.distinctId).toBe("user-creator");
expect(message.properties).toMatchObject({ company_id: "company-bg", stack_id: "stack-123",
user_id: "user-creator", attributed_via: "task_creator" });
}
const unattributed = posthog.captured.find((captured) => captured.event === "plan.approved")!;
expect(unattributed.distinctId).toBe("workspace:stack-123");
expect(unattributed.properties).toMatchObject({ company_id: "company-bg", attributed_via: "none", $process_person_profile: false });
expect(posthog.captured.find((captured) => captured.event === "agent.task_run")!.properties)
.toMatchObject({ failure_cause: "provider_limit", normalized_properties: [] });
});
it("Cloud: a live request's company wins over a background company scope", async () => {
const posthog = fakePostHog();
initTelemetry({ enabled: true }, { env: CLOUD_ENV, postHogFactory: posthog.factory });
const app = express();
app.use(postHogRequestContextMiddlewareForEnv(CLOUD_ENV)!);
app.use((req, _res, next) => {
req.actor = { type: "board", userId: "user-1", source: "session" };
next();
});
app.post("/companies/:companyId/routines", (_req, res) => {
withTelemetryCompany("company-bg", () => trackRoutineCreated(getTelemetryClient()!));
res.json({});
});
app.post("/runs", (_req, res) => {
withTelemetryCompany("company-bg", () => trackRoutineCreated(getTelemetryClient()!));
res.json({});
});
await request(app).post("/companies/company-9/routines").expect(200);
await request(app).post("/runs").expect(200);
await shutdownTelemetry();
const routines = posthog.captured.filter((message) => message.event === "routine.created");
expect(routines).toHaveLength(2);
expect(routines[0]!.properties).toMatchObject({ company_id: "company-9", user_id: "user-1" });
expect(routines[1]!.properties).toMatchObject({ company_id: "company-bg", user_id: "user-1" });
});
it("Cloud: resolves the person responsible for agent work in priority order", () => {
expect(resolveTelemetryPerson({
taskCreatedByUserId: "user-creator",
taskAssignedByUserId: "user-assigner",
runOwnerUserId: "user-owner",
})).toEqual({ userId: "user-creator", attributedVia: "task_creator" });
expect(resolveTelemetryPerson({ taskAssignedByUserId: "user-assigner", runOwnerUserId: "user-owner" }))
.toEqual({ userId: "user-assigner", attributedVia: "task_assigner" });
expect(resolveTelemetryPerson({ taskCreatedByUserId: " ", runOwnerUserId: "user-owner" }))
.toEqual({ userId: "user-owner", attributedVia: "run_owner" });
expect(resolveTelemetryPerson({ taskCreatedByUserId: "local-board", runOwnerUserId: null })).toBeNull();
expect(resolveTelemetryPerson({})).toBeNull();
});
it.each([
{ name: "task creator", input: { taskCreatedByUserId: "paperclip-id:creator", taskAssignedByUserId: "paperclip-id:assigner" }, distinctId: "paperclip-id:creator", via: "task_creator" },
{ name: "task assigner", input: { taskAssignedByUserId: "paperclip-id:assigner", runOwnerUserId: "paperclip-id:owner" }, distinctId: "paperclip-id:assigner", via: "task_assigner" },
{ name: "run owner", input: { runOwnerUserId: "paperclip-id:owner" }, distinctId: "paperclip-id:owner", via: "run_owner" },
])("Cloud: background agent work is attributed to the $name", async ({ input, distinctId, via }) => {
const posthog = fakePostHog();
const client = initTelemetry({ enabled: true }, { env: CLOUD_ENV, postHogFactory: posthog.factory })!;
await withTelemetryActor({ companyId: "company-bg", person: resolveTelemetryPerson(input) }, async () => {
await new Promise((resolve) => setTimeout(resolve, 1));
trackAgentTaskRun(client, { agentId: "agent-1", state: "succeeded" });
});
await shutdownTelemetry();
const run = posthog.captured.find((message) => message.event === "agent.task_run")!;
expect(run.distinctId).toBe(distinctId);
expect(run.groups).toEqual({ workspace: "stack-123" });
expect(run.properties).toMatchObject({
company_id: "company-bg",
user_id: distinctId,
attributed_via: via,
distinct_id: distinctId,
});
expect(run.properties).not.toHaveProperty("$process_person_profile");
});
it("Cloud: background agent work with no person stays on the workspace", async () => {
const posthog = fakePostHog();
const client = initTelemetry({ enabled: true }, { env: CLOUD_ENV, postHogFactory: posthog.factory })!;
withTelemetryActor({ companyId: "company-bg", person: resolveTelemetryPerson({}) }, () =>
trackAgentTaskRun(client, { agentId: "agent-1", state: "succeeded" }));
await shutdownTelemetry();
const run = posthog.captured.find((message) => message.event === "agent.task_run")!;
expect(run.distinctId).toBe("workspace:stack-123");
expect(run.properties).toMatchObject({ attributed_via: "none", $process_person_profile: false });
expect(run.properties).not.toHaveProperty("user_id");
});
it("Cloud: the signed-in request user wins over a background person scope", async () => {
const posthog = fakePostHog();
initTelemetry({ enabled: true }, { env: CLOUD_ENV, postHogFactory: posthog.factory });
const app = express();
app.use(postHogRequestContextMiddlewareForEnv(CLOUD_ENV)!);
app.use((req, _res, next) => {
if (req.path === "/agent") req.actor = { type: "agent", agentId: "agent-1", companyId: "company-1", source: "agent_jwt" };
else req.actor = { type: "board", userId: "paperclip-id:request-user", source: "session" };
next();
});
const person = resolveTelemetryPerson({ taskCreatedByUserId: "paperclip-id:creator" });
app.post("/board", (_req, res) => {
withTelemetryActor({ companyId: "company-1", person }, () =>
trackAgentTaskRun(getTelemetryClient()!, { agentId: "agent-1", state: "cancelled" }));
res.json({});
});
app.post("/agent", (_req, res) => {
withTelemetryActor({ companyId: "company-1", person }, () =>
trackAgentTaskRun(getTelemetryClient()!, { agentId: "agent-1", state: "succeeded" }));
res.json({});
});
await request(app).post("/board").expect(200);
await request(app).post("/agent").expect(200);
await shutdownTelemetry();
const runs = posthog.captured.filter((message) => message.event === "agent.task_run");
const byState = Object.fromEntries(runs.map((message) => [message.properties?.state, message]));
expect(byState.cancelled!.distinctId).toBe("paperclip-id:request-user");
expect(byState.cancelled!.properties).toMatchObject({ attributed_via: "request" });
expect(byState.succeeded!.distinctId).toBe("paperclip-id:creator");
expect(byState.succeeded!.properties).toMatchObject({ attributed_via: "task_creator", company_id: "company-1" });
});
});