mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-19 07:42:38 +01:00
docs(plans): stage production settings for the team's next deploy (#22224)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Cloud now sends analytics directly to PostHog, and this is live on staging > - The plan doc records the rollout state and the steps that remain > - Production needs an ordered, reversible rollout with clear decisions > - This pull request records how production settings are staged for the team's next deploy > - The benefit is one reviewed place that says what is done, what is next, and how to roll back ## Linked Issues or Issue Description **What is the problem?** Production app deploys are coordinated by the team, and the plan needs a way to prepare the settings without restarting user stacks. **What do you propose?** Record the revised production approach: stage settings now without restarting user stacks, and let the team's next production deploy turn them on. Add a note for that deploy with what turns on, canary checks and rollbacks. **Why does it matter?** Production changes must be ordered and reversible, and the open decisions must be visible. ## What Changed - Add "Revised plan" and "Note for the team's production deploy" to Phase 5 of `doc/plans/2026-11-17-cloud-direct-posthog.md`. - No code changes. ## Verification - Documentation only. ## Risks - Low risk. Documentation only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
46e38e1387
commit
4b189690b4
1 file changed
+110
-15
@@ -13,13 +13,15 @@ Production enablement is gated on the Phase 0 compliance checklist owner.
|
||||
| Item | Owner | State | Evidence / link |
|
||||
| --- | --- | --- | --- |
|
||||
| Phase 0 A: install ID stability + Cloud ingest URL source | subagent | done. Stacks run on Railway with a `/paperclip` volume (state.json persists), except `single-ephemeral` stacks (new ID each boot). Legacy three-service stacks had ephemeral disks, and 126 of 166 live staging stacks are unclaimed warm-pool stacks: this explains the ID churn. No supported way to read state.json per stack. Env vars are set by fleet `env_config` campaigns (drain + restart; fleet keys override per-stack). `PAPERCLIP_TELEMETRY_ENDPOINT` lives in the fleet managed-env store. Staging vs production: separate fleets, and `isCloudStagingInstance()` by host. Production admin returned 401 | paperclip-cloud `src/providers/railway.ts:515-521`, `src/fleet/campaigns.ts:5545-5558`, `src/registry/model.ts:1300-1310` |
|
||||
| Phase 0 B: compliance checklist status | subagent | done. Blocker for ANY production token: Terms §8 says telemetry "does not identify individual users" and Privacy §2 says "anonymized". No subprocessor list, no consent banner (website already runs GA4 + X pixel without consent), no SOC 2 program, DPA status unknown. Workspace delete (`paperclip-cloud` `SleepController.deleteStackInner`) makes no PostHog call; no self-serve account deletion | paperclip-website `web/app/terms/page.tsx:146-165`, `web/app/privacy/page.tsx:59-61` |
|
||||
| Phase 0 B: compliance checklist status | subagent | done. Original finding (2026-11-17): Terms §8 and Privacy §2 promised anonymized telemetry, which blocked any production token; resolved the same day by paperclip-website#200 (see the policy and terms row below). Consent: EU/EEA/UK/Swiss banner shipped in #22186. Still open: no subprocessor list, no SOC 2 program, PostHog DPA status unknown. No automatic PostHog deletion, by decision | paperclip-website `web/app/terms/page.tsx:146-165`, `web/app/privacy/page.tsx:59-61` |
|
||||
| Phase 0 C: PostHog entitlements | coordinator | done: session replay available pay-as-you-go ($0 now); Group analytics add-on NOT enabled (needs Dotta; not blocking: events carry `stack_id`, and the history join uses link events, not group properties); product analytics projected $1,186 this period from 40.1M events, almost all OSS | PostHog billing page, 2026-11-17 |
|
||||
| Phase 1: server direct sender | subagent | merged 55b692d; live on staging | https://github.com/paperclipai/paperclip/pull/22006 |
|
||||
| Phase 2: browser analytics | subagent | merged 8172d27; live on staging | https://github.com/paperclipai/paperclip/pull/22012 |
|
||||
| Phase 2b: visitor IP through the proxy | coordinator | merged fa8bc78 | https://github.com/paperclipai/paperclip/pull/22052 |
|
||||
| Phase 2c: replay shows text, masks marked content (Dotta's decision) | subagent | merged b56f240; live on staging | https://github.com/paperclipai/paperclip/pull/22057 |
|
||||
| Phase 2d: `company_id` on background events | subagent | in progress | |
|
||||
| Phase 2d: `company_id` on background events | subagent | merged 2a43fe3; verified on staging | https://github.com/paperclipai/paperclip/pull/22183 |
|
||||
| Phase 2e: EU/EEA/UK/Swiss consent banner | subagent | merged 3c460f2 | https://github.com/paperclipai/paperclip/pull/22186 |
|
||||
| Policy and terms update (Phase 0 B gate) | subagent | done 2026-11-17: live at https://paperclip.ing/terms/ and https://paperclip.ing/privacy/ (self-hosted telemetry scoped as anonymized; Cloud usage data with real IDs disclosed) | https://github.com/paperclipai/paperclip-website/pull/200 |
|
||||
| Phase 3a: workspace-delete cleanup (paperclip-cloud) | coordinator | REVERTED by Dotta's decision (no automatic deletion): https://github.com/paperclipai/paperclip-cloud/pull/1604. It never ran. Original notes: merged b4ddeff; inert until `CLOUD_HARNESS_POSTHOG_PERSONAL_API_KEY_SECRET_REF` is set (scopes group:read, group:write, data_deletion:write; no person scopes). Limits: PostHog has no per-group delete (properties are cleared instead); event deletion uses the alpha Data Deletion API, which PostHog must enable for 483530, needs approval per request, max 5 active; no durable retry (2 in-process retries + audit record) | https://github.com/paperclipai/paperclip-cloud/pull/1603 |
|
||||
| Phase 3b: legacy link backfill script (telemetry-backend) | subagent | merged 9c9aaae, pagination fix a2c123d; staging run: 38 of 44 stacks linked | https://github.com/paperclipai/telemetry-backend/pull/148 |
|
||||
| Phase 4: staging rollout + verification | coordinator | done 2026-11-18. Staging fleet has the token, `environment=staging`, browser analytics and `TRUST_PROXY=100.64.0.0/10`. 24 h: 5,326 server events from 58 stacks, 0 duplicate uuids; 4,620 browser events, 91% with geo; browser and server events join on one person; replay shows chrome text and masks content | paperclip.staging.paperclip.app |
|
||||
@@ -43,9 +45,10 @@ environment switch is set, and the switches must be set in a safe order:
|
||||
on production → browser switch on production after consent approach is in
|
||||
place.
|
||||
|
||||
Production gate (Phase 0 B): the current Terms §8 and Privacy §2 promise
|
||||
anonymized telemetry. Even the server-side token (real user IDs) must wait
|
||||
for the updated policy and terms to be live.
|
||||
Production gate (Phase 0 B): the old Terms §8 and Privacy §2 promised
|
||||
anonymized telemetry, so the server-side token (real user IDs) had to wait for
|
||||
updated policy and terms. Done 2026-11-17: paperclip-website#200 is live at
|
||||
https://paperclip.ing/terms/ and https://paperclip.ing/privacy/.
|
||||
|
||||
### No automatic deletion (Dotta, 2026-11-17)
|
||||
|
||||
@@ -307,20 +310,112 @@ each step is verified before the next one starts.
|
||||
|
||||
#### Prerequisites
|
||||
|
||||
- [ ] Production runs a release that contains #22006, #22012, #22052 and
|
||||
#22057 (and the Phase 2d `company_id` PR if merged by then). Check the running app
|
||||
version of the canary stack before each step.
|
||||
- [ ] Production release gate: see the release gate table in "Revised plan"
|
||||
below (the single list of required PRs and what each enables).
|
||||
- [ ] Dotta signs in to `admin.paperclip.app` in the coordinator's browser
|
||||
pane (the coordinator has no production admin access).
|
||||
- [x] Decision: consent approach for browser analytics (Step 4). Dotta chose
|
||||
option 1, the EU/UK consent banner (2026-11-18). Its PR must be merged
|
||||
and in the production release before Step 4.
|
||||
- [ ] PostHog: set a billing limit on Session replay before Step 4, so a spike
|
||||
cannot run up the bill.
|
||||
- [x] Decision: consent approach for browser analytics (Stage B). Dotta chose
|
||||
option 1, the EU/UK consent banner (2026-11-18), merged in #22186. It
|
||||
must be live on every production stack before Stage B is staged.
|
||||
- [ ] PostHog: set a billing limit on Session replay before Stage B, so a
|
||||
spike cannot run up the bill.
|
||||
- [ ] PostHog: set the project's internal/test account filter to exclude
|
||||
`environment = staging`.
|
||||
|
||||
#### Step 1: server direct sending
|
||||
#### Revised plan: stage settings for the team's deploys (Dotta, 2026-11-18)
|
||||
|
||||
Production app deploys are coordinated by the whole team, and user stacks
|
||||
must not be restarted outside them. So each switch is staged ahead of time
|
||||
with an `env_config` campaign that uses `"applyOnNextDeploy": true` (no drain,
|
||||
no restart) and takes effect with the team's next production app deploy. This
|
||||
section is the source of truth for production; it replaces the "campaign with
|
||||
a canary first" mechanics of Steps 1 and 4 below. Steps 2, 3 and 5 still apply.
|
||||
|
||||
Release gate (each PR and what it enables on a stack):
|
||||
|
||||
| PR | Enables |
|
||||
| --- | --- |
|
||||
| #22006 | Server events straight to PostHog (needs the token) |
|
||||
| #22183 | `company_id` on background server events |
|
||||
| #22012 | Browser analytics and replay (needs the browser flag) |
|
||||
| #22052 | Visitor location through the `/_ph` proxy |
|
||||
| #22057 | Replay shows interface text and masks marked content |
|
||||
| #22186 | EU/EEA/UK/Swiss consent banner before any browser analytics |
|
||||
|
||||
Harness check (anyone can run it; `/healthz` is public, the 401 in the state
|
||||
table was the admin API):
|
||||
|
||||
```sh
|
||||
commit=$(curl -s https://admin.paperclip.app/healthz | jq -r .commit)
|
||||
git -C ~/paperclipai/paperclip-cloud fetch -q origin
|
||||
git -C ~/paperclipai/paperclip-cloud merge-base --is-ancestor d1877e1 "$commit" && echo OK || echo "NOT READY: do not stage"
|
||||
```
|
||||
|
||||
If the check prints anything but `OK`, or cannot run, do not create the
|
||||
campaign: a harness without paperclip-cloud#1652 ignores the flag and restarts
|
||||
every stack. The coordinator runs the campaigns; Dotta provides the production
|
||||
admin sign-in.
|
||||
|
||||
Sequence:
|
||||
|
||||
1. Stage A, server events (gate: harness check OK; policy and terms updated,
|
||||
done 2026-11-17 in paperclip-website#200, live on paperclip.ing). Variables: `PAPERCLIP_POSTHOG_PROJECT_TOKEN`,
|
||||
`PAPERCLIP_POSTHOG_ENVIRONMENT=production`. Server events carry real IDs for
|
||||
every Cloud user, in every region; this is disclosed in Terms §8 and
|
||||
Privacy §2. Turns on with the first production release that contains
|
||||
#22006 (#22183 adds `company_id`). A wrong token or environment only loses
|
||||
analytics data; it cannot break a stack.
|
||||
2. Verify Stage A on the team deploy's canary stack (it deploys first):
|
||||
`cloud.workspace_started` and `server.started` with `environment=production`
|
||||
and the canary's `stack_id`; `count() = uniq(uuid)` for production server
|
||||
events. If it fails, stage `PAPERCLIP_POSTHOG_DIRECT=0` before the rest of
|
||||
the fleet restarts.
|
||||
3. Stage B, browser analytics (gate: a production app deploy that contains
|
||||
#22186 has finished on every production stack, so no stack can restart onto
|
||||
an older build; confirm in the deploy campaign that all targets succeeded and
|
||||
that no stack is pinned to an older release).
|
||||
Variable: `PAPERCLIP_POSTHOG_BROWSER_ENABLED=1`, staged with
|
||||
`applyOnNextDeploy` after that deploy, so it turns on at the following
|
||||
deploy. Never stage it earlier: a stack that restarts onto a build with
|
||||
#22012 but without #22186 would load browser analytics and replay for EU/UK
|
||||
visitors with no banner. Stacks pinned to an older release must be excluded
|
||||
from the campaign with explicit targets.
|
||||
4. Verify Stage B on the next deploy's canary: a sign-in outside EU time zones
|
||||
produces `$pageview` with the user's `paperclip-id:` distinct id and a masked
|
||||
replay; an EU time zone shows the banner and makes no `/_ph` requests until
|
||||
Accept.
|
||||
5. Then Step 2 (`TRUST_PROXY`, announced separately), Step 3 (legacy link
|
||||
backfill) and Step 5 (one-week watch).
|
||||
|
||||
Any stack that restarts before a deploy (wake, restore, another restarting
|
||||
campaign) applies staged values early. For Stage A that is harmless; Stage B
|
||||
waits for #22186 for exactly this reason.
|
||||
|
||||
#### Note for the team's production deploy
|
||||
|
||||
When the production app release that contains this work rolls out, stacks
|
||||
with staged settings start sending analytics on their new deployment:
|
||||
|
||||
- Stage A: server events go straight to PostHog with real IDs
|
||||
(`environment=production`) instead of through the collector. Self-hosted
|
||||
installs are unaffected.
|
||||
- Stage B (only once staged, after a deploy containing #22186 has finished on
|
||||
every production stack): browser analytics
|
||||
and session replay load after the page settles. Visitors in EU, EEA, UK and
|
||||
Swiss time zones see a consent banner, and browser analytics and replay do
|
||||
not load until they accept. Replay shows interface text and masks customer
|
||||
content and connected-account data.
|
||||
|
||||
Canary checks are in steps 2 and 4 above.
|
||||
|
||||
Rollback (each one is an `env_config` campaign; without `applyOnNextDeploy`
|
||||
it restarts the targets, with it the change waits for the next deploy):
|
||||
- Server events back to the collector: `PAPERCLIP_POSTHOG_DIRECT=0`.
|
||||
- Browser analytics off: remove `PAPERCLIP_POSTHOG_BROWSER_ENABLED` (removal
|
||||
cannot use `applyOnNextDeploy`).
|
||||
- Replay off but keep analytics: `PAPERCLIP_POSTHOG_REPLAY_ENABLED=0`.
|
||||
|
||||
#### Step 1: server direct sending (superseded by Stage A above)
|
||||
|
||||
- Campaign: `PAPERCLIP_POSTHOG_PROJECT_TOKEN=<phc token>`,
|
||||
`PAPERCLIP_POSTHOG_ENVIRONMENT=production`. Canary: one internal stack.
|
||||
@@ -346,7 +441,7 @@ each step is verified before the next one starts.
|
||||
then the verification join. Delete the local secret files and revoke the key.
|
||||
- Re-run after a week to link stacks that booted later.
|
||||
|
||||
#### Step 4: browser analytics and replay
|
||||
#### Step 4: browser analytics and replay (superseded by Stage B above)
|
||||
|
||||
Decided: option 1 (Dotta, 2026-11-18). The options considered:
|
||||
|
||||
|
||||
Reference in new issue
Block a user