diff --git a/doc/plans/2026-11-17-cloud-direct-posthog.md b/doc/plans/2026-11-17-cloud-direct-posthog.md index 166a7fe9ce6..6c557623983 100644 --- a/doc/plans/2026-11-17-cloud-direct-posthog.md +++ b/doc/plans/2026-11-17-cloud-direct-posthog.md @@ -13,13 +13,15 @@ Production enablement is gated on the Phase 0 compliance checklist owner. | Item | Owner | State | Evidence / link | | --- | --- | --- | --- | | Phase 0 A: install ID stability + Cloud ingest URL source | subagent | done. Stacks run on Railway with a `/paperclip` volume (state.json persists), except `single-ephemeral` stacks (new ID each boot). Legacy three-service stacks had ephemeral disks, and 126 of 166 live staging stacks are unclaimed warm-pool stacks: this explains the ID churn. No supported way to read state.json per stack. Env vars are set by fleet `env_config` campaigns (drain + restart; fleet keys override per-stack). `PAPERCLIP_TELEMETRY_ENDPOINT` lives in the fleet managed-env store. Staging vs production: separate fleets, and `isCloudStagingInstance()` by host. Production admin returned 401 | paperclip-cloud `src/providers/railway.ts:515-521`, `src/fleet/campaigns.ts:5545-5558`, `src/registry/model.ts:1300-1310` | -| Phase 0 B: compliance checklist status | subagent | done. Blocker for ANY production token: Terms §8 says telemetry "does not identify individual users" and Privacy §2 says "anonymized". No subprocessor list, no consent banner (website already runs GA4 + X pixel without consent), no SOC 2 program, DPA status unknown. Workspace delete (`paperclip-cloud` `SleepController.deleteStackInner`) makes no PostHog call; no self-serve account deletion | paperclip-website `web/app/terms/page.tsx:146-165`, `web/app/privacy/page.tsx:59-61` | +| Phase 0 B: compliance checklist status | subagent | done. Original finding (2026-11-17): Terms §8 and Privacy §2 promised anonymized telemetry, which blocked any production token; resolved the same day by paperclip-website#200 (see the policy and terms row below). Consent: EU/EEA/UK/Swiss banner shipped in #22186. Still open: no subprocessor list, no SOC 2 program, PostHog DPA status unknown. No automatic PostHog deletion, by decision | paperclip-website `web/app/terms/page.tsx:146-165`, `web/app/privacy/page.tsx:59-61` | | Phase 0 C: PostHog entitlements | coordinator | done: session replay available pay-as-you-go ($0 now); Group analytics add-on NOT enabled (needs Dotta; not blocking: events carry `stack_id`, and the history join uses link events, not group properties); product analytics projected $1,186 this period from 40.1M events, almost all OSS | PostHog billing page, 2026-11-17 | | Phase 1: server direct sender | subagent | merged 55b692d; live on staging | https://github.com/paperclipai/paperclip/pull/22006 | | Phase 2: browser analytics | subagent | merged 8172d27; live on staging | https://github.com/paperclipai/paperclip/pull/22012 | | Phase 2b: visitor IP through the proxy | coordinator | merged fa8bc78 | https://github.com/paperclipai/paperclip/pull/22052 | | Phase 2c: replay shows text, masks marked content (Dotta's decision) | subagent | merged b56f240; live on staging | https://github.com/paperclipai/paperclip/pull/22057 | -| Phase 2d: `company_id` on background events | subagent | in progress | | +| Phase 2d: `company_id` on background events | subagent | merged 2a43fe3; verified on staging | https://github.com/paperclipai/paperclip/pull/22183 | +| Phase 2e: EU/EEA/UK/Swiss consent banner | subagent | merged 3c460f2 | https://github.com/paperclipai/paperclip/pull/22186 | +| Policy and terms update (Phase 0 B gate) | subagent | done 2026-11-17: live at https://paperclip.ing/terms/ and https://paperclip.ing/privacy/ (self-hosted telemetry scoped as anonymized; Cloud usage data with real IDs disclosed) | https://github.com/paperclipai/paperclip-website/pull/200 | | Phase 3a: workspace-delete cleanup (paperclip-cloud) | coordinator | REVERTED by Dotta's decision (no automatic deletion): https://github.com/paperclipai/paperclip-cloud/pull/1604. It never ran. Original notes: merged b4ddeff; inert until `CLOUD_HARNESS_POSTHOG_PERSONAL_API_KEY_SECRET_REF` is set (scopes group:read, group:write, data_deletion:write; no person scopes). Limits: PostHog has no per-group delete (properties are cleared instead); event deletion uses the alpha Data Deletion API, which PostHog must enable for 483530, needs approval per request, max 5 active; no durable retry (2 in-process retries + audit record) | https://github.com/paperclipai/paperclip-cloud/pull/1603 | | Phase 3b: legacy link backfill script (telemetry-backend) | subagent | merged 9c9aaae, pagination fix a2c123d; staging run: 38 of 44 stacks linked | https://github.com/paperclipai/telemetry-backend/pull/148 | | Phase 4: staging rollout + verification | coordinator | done 2026-11-18. Staging fleet has the token, `environment=staging`, browser analytics and `TRUST_PROXY=100.64.0.0/10`. 24 h: 5,326 server events from 58 stacks, 0 duplicate uuids; 4,620 browser events, 91% with geo; browser and server events join on one person; replay shows chrome text and masks content | paperclip.staging.paperclip.app | @@ -43,9 +45,10 @@ environment switch is set, and the switches must be set in a safe order: on production → browser switch on production after consent approach is in place. -Production gate (Phase 0 B): the current Terms §8 and Privacy §2 promise -anonymized telemetry. Even the server-side token (real user IDs) must wait -for the updated policy and terms to be live. +Production gate (Phase 0 B): the old Terms §8 and Privacy §2 promised +anonymized telemetry, so the server-side token (real user IDs) had to wait for +updated policy and terms. Done 2026-11-17: paperclip-website#200 is live at +https://paperclip.ing/terms/ and https://paperclip.ing/privacy/. ### No automatic deletion (Dotta, 2026-11-17) @@ -307,20 +310,112 @@ each step is verified before the next one starts. #### Prerequisites -- [ ] Production runs a release that contains #22006, #22012, #22052 and - #22057 (and the Phase 2d `company_id` PR if merged by then). Check the running app - version of the canary stack before each step. +- [ ] Production release gate: see the release gate table in "Revised plan" + below (the single list of required PRs and what each enables). - [ ] Dotta signs in to `admin.paperclip.app` in the coordinator's browser pane (the coordinator has no production admin access). -- [x] Decision: consent approach for browser analytics (Step 4). Dotta chose - option 1, the EU/UK consent banner (2026-11-18). Its PR must be merged - and in the production release before Step 4. -- [ ] PostHog: set a billing limit on Session replay before Step 4, so a spike - cannot run up the bill. +- [x] Decision: consent approach for browser analytics (Stage B). Dotta chose + option 1, the EU/UK consent banner (2026-11-18), merged in #22186. It + must be live on every production stack before Stage B is staged. +- [ ] PostHog: set a billing limit on Session replay before Stage B, so a + spike cannot run up the bill. - [ ] PostHog: set the project's internal/test account filter to exclude `environment = staging`. -#### Step 1: server direct sending +#### Revised plan: stage settings for the team's deploys (Dotta, 2026-11-18) + +Production app deploys are coordinated by the whole team, and user stacks +must not be restarted outside them. So each switch is staged ahead of time +with an `env_config` campaign that uses `"applyOnNextDeploy": true` (no drain, +no restart) and takes effect with the team's next production app deploy. This +section is the source of truth for production; it replaces the "campaign with +a canary first" mechanics of Steps 1 and 4 below. Steps 2, 3 and 5 still apply. + +Release gate (each PR and what it enables on a stack): + +| PR | Enables | +| --- | --- | +| #22006 | Server events straight to PostHog (needs the token) | +| #22183 | `company_id` on background server events | +| #22012 | Browser analytics and replay (needs the browser flag) | +| #22052 | Visitor location through the `/_ph` proxy | +| #22057 | Replay shows interface text and masks marked content | +| #22186 | EU/EEA/UK/Swiss consent banner before any browser analytics | + +Harness check (anyone can run it; `/healthz` is public, the 401 in the state +table was the admin API): + +```sh +commit=$(curl -s https://admin.paperclip.app/healthz | jq -r .commit) +git -C ~/paperclipai/paperclip-cloud fetch -q origin +git -C ~/paperclipai/paperclip-cloud merge-base --is-ancestor d1877e1 "$commit" && echo OK || echo "NOT READY: do not stage" +``` + +If the check prints anything but `OK`, or cannot run, do not create the +campaign: a harness without paperclip-cloud#1652 ignores the flag and restarts +every stack. The coordinator runs the campaigns; Dotta provides the production +admin sign-in. + +Sequence: + +1. Stage A, server events (gate: harness check OK; policy and terms updated, + done 2026-11-17 in paperclip-website#200, live on paperclip.ing). Variables: `PAPERCLIP_POSTHOG_PROJECT_TOKEN`, + `PAPERCLIP_POSTHOG_ENVIRONMENT=production`. Server events carry real IDs for + every Cloud user, in every region; this is disclosed in Terms §8 and + Privacy §2. Turns on with the first production release that contains + #22006 (#22183 adds `company_id`). A wrong token or environment only loses + analytics data; it cannot break a stack. +2. Verify Stage A on the team deploy's canary stack (it deploys first): + `cloud.workspace_started` and `server.started` with `environment=production` + and the canary's `stack_id`; `count() = uniq(uuid)` for production server + events. If it fails, stage `PAPERCLIP_POSTHOG_DIRECT=0` before the rest of + the fleet restarts. +3. Stage B, browser analytics (gate: a production app deploy that contains + #22186 has finished on every production stack, so no stack can restart onto + an older build; confirm in the deploy campaign that all targets succeeded and + that no stack is pinned to an older release). + Variable: `PAPERCLIP_POSTHOG_BROWSER_ENABLED=1`, staged with + `applyOnNextDeploy` after that deploy, so it turns on at the following + deploy. Never stage it earlier: a stack that restarts onto a build with + #22012 but without #22186 would load browser analytics and replay for EU/UK + visitors with no banner. Stacks pinned to an older release must be excluded + from the campaign with explicit targets. +4. Verify Stage B on the next deploy's canary: a sign-in outside EU time zones + produces `$pageview` with the user's `paperclip-id:` distinct id and a masked + replay; an EU time zone shows the banner and makes no `/_ph` requests until + Accept. +5. Then Step 2 (`TRUST_PROXY`, announced separately), Step 3 (legacy link + backfill) and Step 5 (one-week watch). + +Any stack that restarts before a deploy (wake, restore, another restarting +campaign) applies staged values early. For Stage A that is harmless; Stage B +waits for #22186 for exactly this reason. + +#### Note for the team's production deploy + +When the production app release that contains this work rolls out, stacks +with staged settings start sending analytics on their new deployment: + +- Stage A: server events go straight to PostHog with real IDs + (`environment=production`) instead of through the collector. Self-hosted + installs are unaffected. +- Stage B (only once staged, after a deploy containing #22186 has finished on + every production stack): browser analytics + and session replay load after the page settles. Visitors in EU, EEA, UK and + Swiss time zones see a consent banner, and browser analytics and replay do + not load until they accept. Replay shows interface text and masks customer + content and connected-account data. + +Canary checks are in steps 2 and 4 above. + +Rollback (each one is an `env_config` campaign; without `applyOnNextDeploy` +it restarts the targets, with it the change waits for the next deploy): +- Server events back to the collector: `PAPERCLIP_POSTHOG_DIRECT=0`. +- Browser analytics off: remove `PAPERCLIP_POSTHOG_BROWSER_ENABLED` (removal + cannot use `applyOnNextDeploy`). +- Replay off but keep analytics: `PAPERCLIP_POSTHOG_REPLAY_ENABLED=0`. + +#### Step 1: server direct sending (superseded by Stage A above) - Campaign: `PAPERCLIP_POSTHOG_PROJECT_TOKEN=`, `PAPERCLIP_POSTHOG_ENVIRONMENT=production`. Canary: one internal stack. @@ -346,7 +441,7 @@ each step is verified before the next one starts. then the verification join. Delete the local secret files and revoke the key. - Re-run after a week to link stacks that booted later. -#### Step 4: browser analytics and replay +#### Step 4: browser analytics and replay (superseded by Stage B above) Decided: option 1 (Dotta, 2026-11-18). The options considered: