mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-19 07:42:38 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On Paperclip Cloud, the board UI sends browser analytics to PostHog through a same-origin proxy at `/_ph` > - The proxy limits requests in flight: 16 per client address and 256 in total > - Each open tab runs posthog-js, and one tab can have a few requests in flight at once > - On Railway without `TRUST_PROXY`, `req.ip` is the private edge hop, so all visitors share one 16-request limit > - This pull request raises the limits, makes them configurable, and stops the per-client limit when `req.ip` is a shared private hop > - The benefit is that busy stacks with many users and many tabs do not lose analytics to 429 answers, and ops can see when a limit is hit ## Linked Issues or Issue Description **Subsystem affected** Server: the PostHog reverse proxy for Cloud browser analytics (`server/src/routes/posthog-proxy.ts`). **Problem or motivation** The proxy limits are too low for busy Cloud stacks. One user with many tabs can use 16 requests in flight. Behind the Railway edge without `TRUST_PROXY`, `req.ip` is a private `100.64.x.x` hop, so all visitors share one 16-request limit. The proxy then answers 429 and posthog-js drops or delays events and replay uploads. Ops cannot change the limits and cannot see when the proxy refuses requests. **Proposed solution** Raise the defaults to 128 per public client address and 1024 in total. Read both limits from two environment variables with a strict parse and a ceiling. When `req.ip` is not a public address, apply only the total limit. Give each request a 60 second deadline. Log a warning at most once a minute when the proxy refuses requests at a limit. **Alternatives considered** Find the visitor for the per-client limit from `X-Forwarded-For`. A client can write that header, and an earlier security review asked the proxy to keep `req.ip` as the key, so this change does not do that. Scale the per-client limit for a shared hop. All visitors share the hop, so a scaled limit is only a second total limit. The proxy applies the total limit alone instead. **Additional context** Fleet operators can set the variables with an `env_config` campaign. `doc/cloud-browser-analytics.md` tells how. Refs #22012 (the proxy) and #22052 (visitor address behind a private hop). ## What Changed - Raised `DEFAULT_POSTHOG_PROXY_LIMITS` to 128 requests in flight per client address and 1024 in total. The 64 MiB body limit did not change. - Added `PAPERCLIP_POSTHOG_PROXY_MAX_IN_FLIGHT` and `PAPERCLIP_POSTHOG_PROXY_MAX_IN_FLIGHT_PER_CLIENT`. `resolvePostHogProxyLimits` reads them where `resolvePostHogProxyMount` builds the proxy. A value must be a whole number from 1 to 65536. An invalid value logs a warning and uses the default. The server does not stop. - Added `posthogProxyClientKey`. It returns `req.ip` only when it is a public address. For a private, carrier-grade NAT, loopback, link-local or other reserved address, or no address, the proxy applies only the total limit. The key never comes from a client header. - Added `createPostHogProxyLimitReporter`. When the proxy answers 429, it logs `PostHog proxy refused requests at its in-flight limit` at most once a minute. The fields are the refusal counts per limit since the last warning, the requests in flight, the limits and the time since the last warning. The warning has no addresses. It uses no timer: it checks the time when a refusal occurs. - Added a request deadline of 60 seconds from the time the proxy accepts a request. At the deadline, the proxy stops the request to PostHog and closes the browser response, also when PostHog already finished and the browser does not read the response. The slot is then free again. Before, the 30 second upstream timeout was an idle timeout, so a slow body that sent a byte at a time could hold a slot until the server request timeout (300 seconds). - The mount object now also returns the limits in use. - Updated `doc/cloud-browser-analytics.md`: the limits, why these defaults, the variables, the shared-hop behavior, the warning fields and how to tune a fleet with an `env_config` campaign. Why these defaults: - 128 per address covers about 10 people behind one office network, each with 4 tabs and 3 requests in flight per tab, with space left over. - 1024 in total keeps memory bounded. The proxy streams bodies, so a request holds two sockets and the stream buffers between them, not the full body. In a local test on this branch, 1024 held requests added about 96 MiB of resident memory with 4 KiB bodies and about 173 MiB with 256 KiB bodies (256 held requests with 4 KiB bodies: about 40 MiB). These numbers include the test client and the test upstream in the same process, so the proxy share is smaller. Operators can lower the limit for small stacks. ## Verification - `cd server && npx vitest run src/__tests__/cloud-posthog-browser.test.ts`: 87 tests pass (5 runs in a row). - New tests: - Defaults, values from the environment, an empty value, and invalid values (`0`, `-5`, `1.5`, `1e3`, `12abc`, `abc`, `0x10`, `+8`, `65537`, `1024000`, a value larger than a safe integer). `65536` is accepted. Each invalid value keeps the default and warns once per variable. - The mount passes the configured limits to the proxy. - Per-client limit for a public address (with `trust proxy` set): a second request from the same address gets 429, a request from another address gets 200. - Shared private hop (no trusted proxy, peer is loopback): the per-client limit does not apply, and the total limit still answers 429. - A forged `X-Forwarded-For` does not select the bucket when no proxy is trusted: three held requests with the same forged public address and a per-client limit of 1 all get 200. - Request deadline: an upstream that does not answer gives 502 at the deadline and frees the slot, a body that trickles in is cut off at the deadline, and a 16 MiB response that the browser does not read is closed at the deadline. - The tests wait until the held upstream receives each request, not for a fixed time. - Total limit across public addresses, and the reported reason (`total` or `per_client`) for each 429. - The warning rate limit: one warning for the first refusal, none in the next 60 seconds, then one warning with the counts of the refusals in between, and no addresses in any warning. - Red/green: with the shared-hop rule removed (`posthogProxyClientKey` returns `req.ip`), 9 of the new tests fail. With the deadline removed, the 2 deadline tests fail. With both, all pass. - `cd server && npx tsc --noEmit -p tsconfig.json`: passes. - `pnpm verify`: exit 2, because `doc/cloud-browser-analytics.md` is a global CI input. Required CI supplies the broad coverage. - Memory numbers come from a local script that held 1024 requests through the proxy against a local upstream (Node 26, macOS). ## Risks - Higher limits let more requests use memory and sockets at once. The worst case is bounded by the total limit, and operators can lower it with `PAPERCLIP_POSTHOG_PROXY_MAX_IN_FLIGHT`. - Behind a shared private hop, one client can use all of the total limit, and other visitors then get 429. The request deadline frees each slot after at most 60 seconds, but a client that sends again can keep the limit full. Before this change, the same visitors shared a limit of 16, which one client could also fill. A separate limit for the hop would not help the visitors behind it, because one client that fills it blocks all of them. Setting `TRUST_PROXY` brings back the per-client limit. - A PostHog request that takes longer than 60 seconds now fails with 502. posthog-js bodies are small and PostHog answers in well under a second, so this does not affect normal traffic. - If an operator sets `TRUST_PROXY` to trust every hop, `req.ip` comes from the client's `X-Forwarded-For`. This was already true before this change. The total limit still applies. - The proxy is mounted only on Cloud with browser analytics on. Self-hosted installs do not change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (claude-opus-5-5), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>