docs(plans): stage production settings for the team's next deploy (#22224)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Cloud now sends analytics directly to PostHog, and this is
live on staging
> - The plan doc records the rollout state and the steps that remain
> - Production needs an ordered, reversible rollout with clear decisions
> - This pull request records how production settings are staged for the
team's next deploy
> - The benefit is one reviewed place that says what is done, what is
next, and how to roll back

## Linked Issues or Issue Description

**What is the problem?**
Production app deploys are coordinated by the team, and the plan needs a
way to prepare the settings without restarting user stacks.

**What do you propose?**
Record the revised production approach: stage settings now without
restarting user stacks, and let the team's next production deploy turn
them on. Add a note for that deploy with what turns on, canary checks
and rollbacks.

**Why does it matter?**
Production changes must be ordered and reversible, and the open
decisions must be visible.

## What Changed

- Add "Revised plan" and "Note for the team's production deploy" to
Phase 5 of `doc/plans/2026-11-17-cloud-direct-posthog.md`.
- No code changes.

## Verification

- Documentation only.

## Risks

- Low risk. Documentation only.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-18 15:43:46 -06:00
1 parent 46e38e1387
commit 4b189690b4
1 file changed
+110 -15
+110 -15
View File
@@ -13,13 +13,15 @@ Production enablement is gated on the Phase 0 compliance checklist owner.
| Item | Owner | State | Evidence / link |
| --- | --- | --- | --- |
| Phase 0 A: install ID stability + Cloud ingest URL source | subagent | done. Stacks run on Railway with a `/paperclip` volume (state.json persists), except `single-ephemeral` stacks (new ID each boot). Legacy three-service stacks had ephemeral disks, and 126 of 166 live staging stacks are unclaimed warm-pool stacks: this explains the ID churn. No supported way to read state.json per stack. Env vars are set by fleet `env_config` campaigns (drain + restart; fleet keys override per-stack). `PAPERCLIP_TELEMETRY_ENDPOINT` lives in the fleet managed-env store. Staging vs production: separate fleets, and `isCloudStagingInstance()` by host. Production admin returned 401 | paperclip-cloud `src/providers/railway.ts:515-521`, `src/fleet/campaigns.ts:5545-5558`, `src/registry/model.ts:1300-1310` |
| Phase 0 B: compliance checklist status | subagent | done. Blocker for ANY production token: Terms §8 says telemetry "does not identify individual users" and Privacy §2 says "anonymized". No subprocessor list, no consent banner (website already runs GA4 + X pixel without consent), no SOC 2 program, DPA status unknown. Workspace delete (`paperclip-cloud` `SleepController.deleteStackInner`) makes no PostHog call; no self-serve account deletion | paperclip-website `web/app/terms/page.tsx:146-165`, `web/app/privacy/page.tsx:59-61` |
| Phase 0 B: compliance checklist status | subagent | done. Original finding (2026-11-17): Terms §8 and Privacy §2 promised anonymized telemetry, which blocked any production token; resolved the same day by paperclip-website#200 (see the policy and terms row below). Consent: EU/EEA/UK/Swiss banner shipped in #22186. Still open: no subprocessor list, no SOC 2 program, PostHog DPA status unknown. No automatic PostHog deletion, by decision | paperclip-website `web/app/terms/page.tsx:146-165`, `web/app/privacy/page.tsx:59-61` |
| Phase 0 C: PostHog entitlements | coordinator | done: session replay available pay-as-you-go ($0 now); Group analytics add-on NOT enabled (needs Dotta; not blocking: events carry `stack_id`, and the history join uses link events, not group properties); product analytics projected $1,186 this period from 40.1M events, almost all OSS | PostHog billing page, 2026-11-17 |
| Phase 1: server direct sender | subagent | merged 55b692d; live on staging | https://github.com/paperclipai/paperclip/pull/22006 |
| Phase 2: browser analytics | subagent | merged 8172d27; live on staging | https://github.com/paperclipai/paperclip/pull/22012 |
| Phase 2b: visitor IP through the proxy | coordinator | merged fa8bc78 | https://github.com/paperclipai/paperclip/pull/22052 |
| Phase 2c: replay shows text, masks marked content (Dotta's decision) | subagent | merged b56f240; live on staging | https://github.com/paperclipai/paperclip/pull/22057 |
| Phase 2d: `company_id` on background events | subagent | in progress | |
| Phase 2d: `company_id` on background events | subagent | merged 2a43fe3; verified on staging | https://github.com/paperclipai/paperclip/pull/22183 |
| Phase 2e: EU/EEA/UK/Swiss consent banner | subagent | merged 3c460f2 | https://github.com/paperclipai/paperclip/pull/22186 |
| Policy and terms update (Phase 0 B gate) | subagent | done 2026-11-17: live at https://paperclip.ing/terms/ and https://paperclip.ing/privacy/ (self-hosted telemetry scoped as anonymized; Cloud usage data with real IDs disclosed) | https://github.com/paperclipai/paperclip-website/pull/200 |
| Phase 3a: workspace-delete cleanup (paperclip-cloud) | coordinator | REVERTED by Dotta's decision (no automatic deletion): https://github.com/paperclipai/paperclip-cloud/pull/1604. It never ran. Original notes: merged b4ddeff; inert until `CLOUD_HARNESS_POSTHOG_PERSONAL_API_KEY_SECRET_REF` is set (scopes group:read, group:write, data_deletion:write; no person scopes). Limits: PostHog has no per-group delete (properties are cleared instead); event deletion uses the alpha Data Deletion API, which PostHog must enable for 483530, needs approval per request, max 5 active; no durable retry (2 in-process retries + audit record) | https://github.com/paperclipai/paperclip-cloud/pull/1603 |
| Phase 3b: legacy link backfill script (telemetry-backend) | subagent | merged 9c9aaae, pagination fix a2c123d; staging run: 38 of 44 stacks linked | https://github.com/paperclipai/telemetry-backend/pull/148 |
| Phase 4: staging rollout + verification | coordinator | done 2026-11-18. Staging fleet has the token, `environment=staging`, browser analytics and `TRUST_PROXY=100.64.0.0/10`. 24 h: 5,326 server events from 58 stacks, 0 duplicate uuids; 4,620 browser events, 91% with geo; browser and server events join on one person; replay shows chrome text and masks content | paperclip.staging.paperclip.app |
@@ -43,9 +45,10 @@ environment switch is set, and the switches must be set in a safe order:
on production → browser switch on production after consent approach is in
place.
Production gate (Phase 0 B): the current Terms §8 and Privacy §2 promise
anonymized telemetry. Even the server-side token (real user IDs) must wait
for the updated policy and terms to be live.
Production gate (Phase 0 B): the old Terms §8 and Privacy §2 promised
anonymized telemetry, so the server-side token (real user IDs) had to wait for
updated policy and terms. Done 2026-11-17: paperclip-website#200 is live at
https://paperclip.ing/terms/ and https://paperclip.ing/privacy/.
### No automatic deletion (Dotta, 2026-11-17)
@@ -307,20 +310,112 @@ each step is verified before the next one starts.
#### Prerequisites
- [ ] Production runs a release that contains #22006, #22012, #22052 and
#22057 (and the Phase 2d `company_id` PR if merged by then). Check the running app
version of the canary stack before each step.
- [ ] Production release gate: see the release gate table in "Revised plan"
below (the single list of required PRs and what each enables).
- [ ] Dotta signs in to `admin.paperclip.app` in the coordinator's browser
pane (the coordinator has no production admin access).
- [x] Decision: consent approach for browser analytics (Step 4). Dotta chose
option 1, the EU/UK consent banner (2026-11-18). Its PR must be merged
and in the production release before Step 4.
- [ ] PostHog: set a billing limit on Session replay before Step 4, so a spike
cannot run up the bill.
- [x] Decision: consent approach for browser analytics (Stage B). Dotta chose
option 1, the EU/UK consent banner (2026-11-18), merged in #22186. It
must be live on every production stack before Stage B is staged.
- [ ] PostHog: set a billing limit on Session replay before Stage B, so a
spike cannot run up the bill.
- [ ] PostHog: set the project's internal/test account filter to exclude
`environment = staging`.
#### Step 1: server direct sending
#### Revised plan: stage settings for the team's deploys (Dotta, 2026-11-18)
Production app deploys are coordinated by the whole team, and user stacks
must not be restarted outside them. So each switch is staged ahead of time
with an `env_config` campaign that uses `"applyOnNextDeploy": true` (no drain,
no restart) and takes effect with the team's next production app deploy. This
section is the source of truth for production; it replaces the "campaign with
a canary first" mechanics of Steps 1 and 4 below. Steps 2, 3 and 5 still apply.
Release gate (each PR and what it enables on a stack):
| PR | Enables |
| --- | --- |
| #22006 | Server events straight to PostHog (needs the token) |
| #22183 | `company_id` on background server events |
| #22012 | Browser analytics and replay (needs the browser flag) |
| #22052 | Visitor location through the `/_ph` proxy |
| #22057 | Replay shows interface text and masks marked content |
| #22186 | EU/EEA/UK/Swiss consent banner before any browser analytics |
Harness check (anyone can run it; `/healthz` is public, the 401 in the state
table was the admin API):
```sh
commit=$(curl -s https://admin.paperclip.app/healthz | jq -r .commit)
git -C ~/paperclipai/paperclip-cloud fetch -q origin
git -C ~/paperclipai/paperclip-cloud merge-base --is-ancestor d1877e1 "$commit" && echo OK || echo "NOT READY: do not stage"
```
If the check prints anything but `OK`, or cannot run, do not create the
campaign: a harness without paperclip-cloud#1652 ignores the flag and restarts
every stack. The coordinator runs the campaigns; Dotta provides the production
admin sign-in.
Sequence:
1. Stage A, server events (gate: harness check OK; policy and terms updated,
done 2026-11-17 in paperclip-website#200, live on paperclip.ing). Variables: `PAPERCLIP_POSTHOG_PROJECT_TOKEN`,
`PAPERCLIP_POSTHOG_ENVIRONMENT=production`. Server events carry real IDs for
every Cloud user, in every region; this is disclosed in Terms §8 and
Privacy §2. Turns on with the first production release that contains
#22006 (#22183 adds `company_id`). A wrong token or environment only loses
analytics data; it cannot break a stack.
2. Verify Stage A on the team deploy's canary stack (it deploys first):
`cloud.workspace_started` and `server.started` with `environment=production`
and the canary's `stack_id`; `count() = uniq(uuid)` for production server
events. If it fails, stage `PAPERCLIP_POSTHOG_DIRECT=0` before the rest of
the fleet restarts.
3. Stage B, browser analytics (gate: a production app deploy that contains
#22186 has finished on every production stack, so no stack can restart onto
an older build; confirm in the deploy campaign that all targets succeeded and
that no stack is pinned to an older release).
Variable: `PAPERCLIP_POSTHOG_BROWSER_ENABLED=1`, staged with
`applyOnNextDeploy` after that deploy, so it turns on at the following
deploy. Never stage it earlier: a stack that restarts onto a build with
#22012 but without #22186 would load browser analytics and replay for EU/UK
visitors with no banner. Stacks pinned to an older release must be excluded
from the campaign with explicit targets.
4. Verify Stage B on the next deploy's canary: a sign-in outside EU time zones
produces `$pageview` with the user's `paperclip-id:` distinct id and a masked
replay; an EU time zone shows the banner and makes no `/_ph` requests until
Accept.
5. Then Step 2 (`TRUST_PROXY`, announced separately), Step 3 (legacy link
backfill) and Step 5 (one-week watch).
Any stack that restarts before a deploy (wake, restore, another restarting
campaign) applies staged values early. For Stage A that is harmless; Stage B
waits for #22186 for exactly this reason.
#### Note for the team's production deploy
When the production app release that contains this work rolls out, stacks
with staged settings start sending analytics on their new deployment:
- Stage A: server events go straight to PostHog with real IDs
(`environment=production`) instead of through the collector. Self-hosted
installs are unaffected.
- Stage B (only once staged, after a deploy containing #22186 has finished on
every production stack): browser analytics
and session replay load after the page settles. Visitors in EU, EEA, UK and
Swiss time zones see a consent banner, and browser analytics and replay do
not load until they accept. Replay shows interface text and masks customer
content and connected-account data.
Canary checks are in steps 2 and 4 above.
Rollback (each one is an `env_config` campaign; without `applyOnNextDeploy`
it restarts the targets, with it the change waits for the next deploy):
- Server events back to the collector: `PAPERCLIP_POSTHOG_DIRECT=0`.
- Browser analytics off: remove `PAPERCLIP_POSTHOG_BROWSER_ENABLED` (removal
cannot use `applyOnNextDeploy`).
- Replay off but keep analytics: `PAPERCLIP_POSTHOG_REPLAY_ENABLED=0`.
#### Step 1: server direct sending (superseded by Stage A above)
- Campaign: `PAPERCLIP_POSTHOG_PROJECT_TOKEN=<phc token>`,
`PAPERCLIP_POSTHOG_ENVIRONMENT=production`. Canary: one internal stack.
@@ -346,7 +441,7 @@ each step is verified before the next one starts.
then the verification join. Delete the local secret files and revoke the key.
- Re-run after a week to link stacks that booted later.
#### Step 4: browser analytics and replay
#### Step 4: browser analytics and replay (superseded by Stage B above)
Decided: option 1 (Dotta, 2026-11-18). The options considered: