From 3293700448a3b5badd5e5b05ffbbf4f330914c49 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 17 Nov 2026 08:38:49 -0600 Subject: [PATCH] docs(plans): Cloud sends analytics directly to PostHog (#22003) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Cloud is the hosted version, and it shares the product telemetry path with self-hosted installs > - Self-hosted telemetry goes through the Paperclip collector, which anonymizes every identifier > - Cloud needs real identifiers, page analytics, and session replay in PostHog to find and fix bugs for specific users > - This pull request adds the plan that moves Cloud to direct PostHog delivery and keeps the self-hosted path unchanged > - The benefit is one reviewed plan that the follow-up pull requests implement ## Linked Issues or Issue Description **What is the problem?** The PostHog Product analytics and Web analytics pages show no data for Paperclip Cloud. Only server events arrive, and they carry anonymized install identifiers. **What do you propose?** Cloud sends events directly to PostHog with real identifiers. Self-hosted installs keep the collector path. Existing Cloud stacks keep their PostHog history through a precomputed legacy identifier. **Why does it matter?** The team can trace a bug from a specific user or workspace to events and session replays. ## What Changed - Add `doc/plans/2026-11-17-cloud-direct-posthog.md`. ## Verification - Documentation only. No code changes. ## Risks - Low risk. Documentation only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge ๐Ÿค– Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/plans/2026-11-17-cloud-direct-posthog.md | 329 +++++++++++++++++++ 1 file changed, 329 insertions(+) create mode 100644 doc/plans/2026-11-17-cloud-direct-posthog.md diff --git a/doc/plans/2026-11-17-cloud-direct-posthog.md b/doc/plans/2026-11-17-cloud-direct-posthog.md new file mode 100644 index 00000000000..ac5f4a73fc2 --- /dev/null +++ b/doc/plans/2026-11-17-cloud-direct-posthog.md @@ -0,0 +1,329 @@ +# Paperclip Cloud sends analytics directly to PostHog + +Status: proposed, 2026-11-17 (revised: Cloud uses real IDs) +PostHog project: 483530 (US), kept as the only project so history is preserved. + +## Delivery state (coordinator keeps this current) + +Finish line: Phases 1โ€“3 merged with green checks and Greptile 5/5, deployed +to staging, and verified there in PostHog 483530: real IDs, server events +inside session replays, no duplicates, self-hosted output unchanged. +Production enablement is gated on the Phase 0 compliance checklist owner. + +| Item | Owner | State | Evidence / link | +| --- | --- | --- | --- | +| Phase 0 A: install ID stability + Cloud ingest URL source | subagent | done. Stacks run on Railway with a `/paperclip` volume (state.json persists), except `single-ephemeral` stacks (new ID each boot). Legacy three-service stacks had ephemeral disks, and 126 of 166 live staging stacks are unclaimed warm-pool stacks: this explains the ID churn. No supported way to read state.json per stack. Env vars are set by fleet `env_config` campaigns (drain + restart; fleet keys override per-stack). `PAPERCLIP_TELEMETRY_ENDPOINT` lives in the fleet managed-env store. Staging vs production: separate fleets, and `isCloudStagingInstance()` by host. Production admin returned 401 | paperclip-cloud `src/providers/railway.ts:515-521`, `src/fleet/campaigns.ts:5545-5558`, `src/registry/model.ts:1300-1310` | +| Phase 0 B: compliance checklist status | subagent | done. Blocker for ANY production token: Terms ยง8 says telemetry "does not identify individual users" and Privacy ยง2 says "anonymized". No subprocessor list, no consent banner (website already runs GA4 + X pixel without consent), no SOC 2 program, DPA status unknown. Workspace delete (`paperclip-cloud` `SleepController.deleteStackInner`) makes no PostHog call; no self-serve account deletion | paperclip-website `web/app/terms/page.tsx:146-165`, `web/app/privacy/page.tsx:59-61` | +| Phase 0 C: PostHog entitlements | coordinator | done: session replay available pay-as-you-go ($0 now); Group analytics add-on NOT enabled (needs Dotta; not blocking: events carry `stack_id`, and the history join uses link events, not group properties); product analytics projected $1,186 this period from 40.1M events, almost all OSS | PostHog billing page, 2026-11-17 | +| Phase 1: server direct sender | subagent | PR green (61 checks, Greptile 5/5); coordinator review asked for private repo `source_ref` to stay hashed on Cloud (customer content, not an ID); rebasing | https://github.com/paperclipai/paperclip/pull/22006 | +| Phase 2: browser analytics | subagent | in progress | | +| Phase 3a: workspace-delete cleanup (paperclip-cloud) | coordinator | REVERTED by Dotta's decision (no automatic deletion): https://github.com/paperclipai/paperclip-cloud/pull/1604. It never ran. Original notes: merged b4ddeff; inert until `CLOUD_HARNESS_POSTHOG_PERSONAL_API_KEY_SECRET_REF` is set (scopes group:read, group:write, data_deletion:write; no person scopes). Limits: PostHog has no per-group delete (properties are cleared instead); event deletion uses the alpha Data Deletion API, which PostHog must enable for 483530, needs approval per request, max 5 active; no durable retry (2 in-process retries + audit record) | https://github.com/paperclipai/paperclip-cloud/pull/1603 | +| Phase 3b: legacy link backfill script (telemetry-backend) | subagent | merged 9c9aaae (operator script, not deployed); run after Phase 1 is live on staging | https://github.com/paperclipai/telemetry-backend/pull/148 | +| Phase 4: staging rollout + verification | coordinator | pending | | + +### Continuous deployment rule + +Merged PRs roll out automatically. So every PR must be inert until its +environment switch is set, and the switches must be set in a safe order: + +1. Server direct sending turns on only when `PAPERCLIP_POSTHOG_PROJECT_TOKEN` + is set on a Cloud-managed instance. Without it, the collector path is + unchanged. +2. Browser analytics turns on only when `PAPERCLIP_POSTHOG_BROWSER_ENABLED=1` + is also set. Production stays off until the compliance checklist is done. +3. Fleet env vars are set with fleet `env_config` campaigns, which drain and + restart stacks and rebuild the warm pool. Set all switches for a fleet in + one campaign where possible. +4. Order: token on staging โ†’ verify โ†’ legacy link backfill โ†’ + browser switch on staging โ†’ verify โ†’ **policy and terms updated** โ†’ token + on production โ†’ browser switch on production after consent approach is in + place. + +Production gate (Phase 0 B): the current Terms ยง8 and Privacy ยง2 promise +anonymized telemetry. Even the server-side token (real user IDs) must wait +for the updated policy and terms to be live. + +### No automatic deletion (Dotta, 2026-11-17) + +PostHog data is never deleted automatically: not on workspace deletion, not +on account deletion, and never for existing history. Deletion happens only +explicitly and separately, when someone decides to run it for a specific +request. The automatic workspace-delete cleanup (paperclip-cloud#1603) was +reverted (paperclip-cloud#1604); it never ran. + +## Goal + +1. Self-hosted (OSS) installs keep sending telemetry through the existing + collector (`telemetry.paperclip.ing/ingest` โ†’ S3 โ†’ loader โ†’ PostHog), with + all of the collector's anonymization. Nothing about the OSS path changes. +2. Paperclip Cloud sends its events straight to PostHog using **real + identifiers**: workspace (stack) ID, user ID, agent ID, task ID. Anyone + debugging can go from a PostHog event or session replay to the exact + workspace, user and task. +3. History continues across the switch for workspaces that already exist, with + no signing keys in Cloud infrastructure. +4. Direct sending lets Cloud use all of PostHog: page views, sessions, web + analytics, autocapture, heatmaps, web vitals, session replay, people and + workspace groups. Server events link to the browser session that caused + them. + +## Why the Product and Web analytics pages are empty today + +- Project 483530 receives about 9.4M events per week, all server-side, all + copied in by the telemetry backend's loader (`src/loader/sinks/posthog.mjs` in + `telemetry-backend`). +- There is no browser SDK, so there are no `$pageview`s or sessions. Both pages + are built on those. +- The loader sets `$process_person_profile: false` and turns off location + lookup, so the project has no people and no locations. +- Every event already carries `deployment_source` (`oss` / `cloud`). The backend + sets it from the request path (`V2_CLOUD_INGEST_PATH` in + `src/v2/ingest-handler.mjs`). Cloud: ~740 installs and ~29k events per week; + first Cloud event on 2026-09-11. + +## How each existing event is built today + +| PostHog field | Built by the backend as | +| --- | --- | +| `event` | Client event name, unchanged | +| `timestamp` | Client `occurredAt` | +| `distinct_id` | `HMAC-SHA256(LOADER_POSTHOG_ID_HASH_SECRET, "posthog:v1:install_id:" + sha256(rawInstallId))` | +| Dimensions | Client dimensions after enum fallback normalization. `agent_id` is HMAC-hashed by the backend; `task_id` and private source refs are salted-hashed on the client (`hashPrivateRef`) | +| Metadata | `app`, `schema_version`, `source`, `deployment_source`, `version`, `event_date`, `hour`, `ingested_at`, `normalized_properties` | +| Flags | `$process_person_profile: false`, location lookup off | + +`rawInstallId` is a random UUID that the server creates on first boot and +stores in `/telemetry/state.json`. That anonymization exists for +self-hosted users' privacy. Cloud is our own service, covered by our own terms, +and needs real IDs instead. + +## Identity model on Cloud + +### Real IDs from now on + +| What | PostHog field | Value | +| --- | --- | --- | +| Workspace | group `workspace`, key | Raw stack ID (`PAPERCLIP_CLOUD_STACK_ID`), the same key on server and browser | +| Workspace properties | `$groupidentify` | stack slug, primary host, plan, created_at | +| Person (browser) | `distinct_id` via `identify()` | Paperclip user ID | +| Person properties | `$set` | email, name (decided 2026-11-17). Set only on the person profile, never on event properties, so deleting the person removes them | +| Server events | `distinct_id` | Acting user ID when a person triggered it; otherwise `workspace:` | +| Server events | properties | `stack_id`, `environment` (from `PAPERCLIP_POSTHOG_ENVIRONMENT`, required), `user_id` (when known), raw `agent_id`, raw `task_id`, `company_id` | +| Browser to server link | `$session_id`, `$window_id` | Sent by posthog-js tracing headers on API requests and copied onto server events triggered by that request | + +Server events with no acting person keep `$process_person_profile: false`, so +`workspace:` never turns into a fake person. Events a user triggers +attach to that user's person record. With `$session_id` attached, those events +show up in the user's session replay timeline. + +### Existing workspaces: history joined through link events + +Phase 0 A showed that most Cloud install IDs already changed over time: +legacy stacks had ephemeral disks, warm-pool stacks report before anyone +claims them, and single-ephemeral stacks get a new ID on every boot. There is +also no supported way to read each stack's install ID, and every per-stack +env change restarts that stack. So the plan does not keep the old +`distinct_id` on new events. It joins history through plain events, so the +join does not depend on the paid Group analytics add-on (group properties are +not stored without it): + +- On startup the server sends one `cloud.workspace_started` event with + `stack_id` and `install_hash` = sha256(raw install ID). That value is not + secret; it is the canonical install ID the collector already computes. +- A one-time operator script (in `telemetry-backend`, which owns the hashing) + reads distinct `(stack_id, install_hash)` pairs from those events, computes + the old PostHog ID locally with the signing key from the operator's own + environment, and sends one `cloud.workspace_legacy_link` event per match + with `stack_id` and `legacy_distinct_id`. The key never leaves that machine. + No env vars, no restarts. +- HogQL joins pre-switch events (`distinct_id = legacy_distinct_id`) to the + workspace through the link events. If the Group analytics add-on is later + enabled, the script can also copy the mapping onto the workspace group. +- The script sends a link only when the computed ID matches pre-switch Cloud + events, and never sends a second link for a stack that already has a + matched one. A later boot with a new install ID therefore cannot replace a + good mapping. +- Scope of the history guarantee: workspaces whose install ID survived to the + switch (stacks on a persistent `/paperclip` volume). `single-ephemeral` + stacks, and stacks whose ID already changed before the switch (legacy + ephemeral disks, reshapes), cannot be matched. Their pre-switch events stay + in PostHog under the old IDs. The script reports them as unmatched. +- All new events use real IDs only: acting user ID, else `workspace:`. +- Known cost: a one-time step in "unique installs" counts on cutover day, + because events with no acting person switch from the old ID to + `workspace:`. + +What doesn't carry over: per-agent and per-task history before the switch, +because those were hashed. Old `agent_id` and `task_id` values stay hashed; +new ones are raw. Agent- and task-level questions are almost always about +recent work, so this plan accepts the break. + +## Architecture after the change + +``` +OSS server โ”€โ”€โ–บ collector /ingest โ”€โ”€โ–บ S3 โ”€โ”€โ–บ loader โ”€โ”€โ–บ PostHog 483530 (unchanged, anonymized) + +Cloud server โ”€โ”€โ–บ posthog-node (direct, real IDs) โ”€โ”€โ”€โ”€โ”€โ–บ PostHog 483530 +Cloud browser โ”€โ–บ posthog-js (direct, via our proxy) โ”€โ”€โ–บ PostHog 483530 + โ””โ”€ tracing headers โ”€โ”€โ–บ Cloud server (links events to sessions) +``` + +Each event name has exactly one sender. A process sends a given batch either to +the collector or directly to PostHog, never both. + +## Work plan + +### Phase 0: confirm and decide (no code) + +- [ ] Compliance checklist (owner: whoever owns Cloud compliance; not legal + advice): + - Privacy policy names PostHog as a processor and covers product analytics + and session recording. + - PostHog data processing agreement signed. The US region relies on Standard + Contractual Clauses or the EUโ€“US Data Privacy Framework for EU users. + - Consent approach for EU visitors: a banner, or cookieless mode until the + visitor consents. + - PostHog access goes through SSO, least privilege, and limited access to + replays. Retention periods set. PostHog added to the SOC 2 vendor list. +- [x] Check whether install IDs survive redeploys on Cloud. Result: mostly + yes now, but historically no (see the state table). +- [ ] Find where Cloud stacks get the Cloud ingest URL (`V2_CLOUD_INGEST_PATH`); + it is not in `paperclip-cloud` source. The direct sender replaces it. +- [ ] Check the PostHog plan for group analytics (paid add-on) and session + replay quotas. + +### Phase 1: server sends directly on Cloud (paperclip repo, one PR) + +- Add a direct sending path to `TelemetryClient`. When the server is Cloud + (`isCloudManagedInstance()`) and `PAPERCLIP_POSTHOG_PROJECT_TOKEN` is set, + batches go to PostHog through `posthog-node` (loaded with dynamic `import()`, + so self-hosted installs never load it). Otherwise behavior is unchanged. +- On the direct path, ID dimensions are raw (`agent_id`, `task_id`, user and + workspace IDs), and typed helpers add `stack_id`, `user_id` and + `company_id`. Private repo `source_ref` stays hashed on every path, + Cloud included, because it is customer content, not an ID. Event names, dimension names and + enum normalization stay identical to the collector path, so insights keep + working. +- Keep `deployment_source: "cloud"`, `app`, `version`, `schema_version` and + `source`, so filters built on today's properties keep working. +- Request context: an Express middleware reads posthog-js tracing headers + (`X-POSTHOG-SESSION-ID`, `X-POSTHOG-WINDOW-ID`). Only on Cloud, it stores + them in request-scoped context so events emitted during that request carry + `$session_id` and `$window_id`. It ignores `X-POSTHOG-DISTINCT-ID`. The + acting user always comes from the authenticated request. +- On startup, send `cloud.workspace_started` (with `stack_id`, + `install_hash`, `environment`) and the workspace `$groupidentify`. Both use + the direct posthog-node path, not `TelemetryClient.track()`, which drops + names that are not in `PAPERCLIP_EVENTS`. These Cloud-only events are never + added to `PAPERCLIP_EVENTS`. +- Flush on shutdown so deploy restarts don't drop queued events. +- Kill switch: `PAPERCLIP_POSTHOG_DIRECT=0` sends through the collector again. +- Tests: + - Self-hosted installs (no Cloud env) never import `posthog-node`, keep + posting to `/ingest`, and still hash private refs. + - On Cloud: real IDs and the workspace group on every event. + - On Cloud: a private repo `source_ref` stays hashed + (`source_ref_hashed: true`). + - `cloud.workspace_started` reaches the PostHog sender with `stack_id` and + `install_hash`. `install_hash` appears on no other event. + - No double sending: one batch goes to exactly one sender. + - Session headers are ignored on self-hosted installs. + +### Phase 2: browser analytics on Cloud (paperclip repo, one PR) + +- Server sends `{ token, apiHost }` to the UI in the existing instance or health + bootstrap response, only when the server is Cloud, the token is set, and + `PAPERCLIP_POSTHOG_BROWSER_ENABLED=1`. A token alone leaves browser + analytics off (tested). The + token is never baked in at build time, because the same UI ships in the OSS + npm package. +- The UI loads `posthog-js` with dynamic `import()` after first paint, and only + when it receives a token. This keeps cold loads in the OSS and Cloud bundles + as fast as they are today. +- `posthog.init` settings: + - pageviews on history changes, pageleave, autocapture, web vitals, heatmaps + - `person_profiles: "identified_only"` + - tracing headers on our own API host only + - every event also gets `deployment_source: "cloud"` and `environment` +- After sign-in: `identify(userId, props)` and + `group("workspace", stackId, props)`. On sign-out: `reset()`. +- Session replay is on. All inputs and all text are masked by default. Only UI + chrome (navigation, buttons, headings, labels) is unmasked, through an + explicit allowlist class. So a missed surface stays masked instead of + leaking. +- Reverse proxy: serve PostHog through our own Cloud domain so ad blockers don't + drop events. Allowlist only the PostHog ingest and static asset hosts, only + the paths and methods the SDK uses, and strip inbound `Host`, `Cookie` and + `Authorization` headers. Never a generic proxy. +- Test that the UI never calls `posthog.capture()` with an event name from + `PAPERCLIP_EVENTS`, to keep one sender per event. +- Per-workspace switch to turn off session replay, or all browser analytics, + for customers whose contracts forbid recording. +- Deletion requests: handled explicitly and separately by a person, for that + specific request. Nothing in the product deletes PostHog data + automatically. +- Masking and autocapture defaults never capture input values. Secret and + credential fields are always excluded (`ph-no-capture`). + +### Phase 3: legacy link backfill (telemetry-backend PR) + +- telemetry-backend: the legacy link backfill script described above, with a + dry-run mode and a count of workspaces matched and skipped. +- Fleet env vars are set by the coordinator during Phase 4 with fleet + `env_config` campaigns: `PAPERCLIP_POSTHOG_PROJECT_TOKEN`, + `PAPERCLIP_POSTHOG_HOST`, `PAPERCLIP_POSTHOG_ENVIRONMENT`, and later + `PAPERCLIP_POSTHOG_BROWSER_ENABLED`. No signing keys. +- Keep the Cloud ingest URL setting while the `PAPERCLIP_POSTHOG_DIRECT=0` + rollback is supported. Rollback relies on it to reach the Cloud path and + keep the `cloud` label. + +### Phase 4: rollout + +1. Staging stacks first, for 24 hours: + - Events show real IDs. + - Each stack sends `cloud.workspace_started` with `install_hash`. After + the backfill, `cloud.workspace_legacy_link` events join pre-switch + events to their stack. + - Server events triggered from the UI show up in that user's session replay. + - No duplicate uuids. + - Per-hour counts are within normal variance. +2. Production, by turning on direct sending through fleet config. No Paperclip + release is needed to switch back. +3. In PostHog, set the project's internal/test filter to exclude + `environment = staging`. +4. Optional cleanup, only after the `PAPERCLIP_POSTHOG_DIRECT=0` rollback is + retired: remove the Cloud ingest URL setting and the backend's Cloud + ingest path. + +## Success criteria + +- [ ] The Web analytics and Product analytics pages for 483530 show Cloud + visitors, sessions, pages and countries, and no self-hosted traffic. +- [ ] Given a user ID or stack ID, you can find that user's or workspace's + events, sessions and replays in PostHog, including server events from + their session. +- [ ] For workspaces that existed before the switch, pre-switch events can be + joined to the workspace through `cloud.workspace_legacy_link` events. +- [ ] No signing key is stored anywhere in Cloud infrastructure. +- [ ] No duplicate events. Server event uuids are derived from a stable + per-event key (the same formula the loader uses), so + `count() = uniq(uuid)` for Cloud server events on cutover day detects a + double send. Per-event, per-hour counts also stay within the + pre-cutover baseline. +- [ ] Self-hosted traffic is unchanged in volume, shape and anonymization. + Self-hosted builds never load `posthog-node` or `posthog-js`. +- [ ] Session replays play back with customer content masked. +- [ ] Cold load time on Cloud is not measurably worse (before/after numbers). + +## Risks and mitigations + +| Risk | Mitigation | +| --- | --- | +| Real IDs leak into the OSS path | Direct-path-only code branch; tests that self-hosted output stays hashed | +| Install IDs weren't stable on Cloud | History joins through `cloud.workspace_legacy_link` events; new events use the stack ID, which is stable | +| Customer content or pasted secrets show up in replays or autocapture | Mask all inputs and content areas by default; exclude credential fields; review sample replays before production | +| A deletion request leaves PII behind | Email and name only on person profiles; deletion requests are handled explicitly by a person | +| An enterprise customer forbids recording | Per-workspace replay and analytics switch | +| Tracing headers spoofed by a client | Used only to link to a session, never to authorize; acting user comes from the authenticated request, not the header | +| Ad blockers drop browser events | First-party reverse proxy | +| Events lost on deploy restarts | Flush on shutdown; posthog-node retries |