Files
PaperClipAI/doc/browser-performance-tracing.md
T
DottaandClaude Opus 5.5 51c81c537b fix(browser-tracing): end hidden loads, cap timeouts, backfill hard-load API spans, record list view (#21392)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Hosted instances send real-user page-load traces to Honeycomb
(#21156), so we can set page-load SLOs
> - Staging data showed four measurement gaps: 140 of 147 hard loads
were hidden tabs that became 30 s timeouts, a few timeouts lasted about
2 hours after the computer slept, hard loads had no API child spans, and
list views on the tasks page could not be told apart
> - These gaps make every average and percentile wrong, so the data
cannot back an SLO yet
> - This pull request ends hidden loads as `hidden`, caps timeouts at
exactly 30 s and marks suspends, backfills hard-load API spans from
Resource Timing, and records the list view from an allowlist
> - The benefit is page-load numbers that describe what users see, with
no cost on the critical path

## Linked Issues or Issue Description

Refs #21156

**What is the issue?**
Browser page-load tracing records misleading data. A load in a
background tab never paints, so it always ends as a 30 s `timeout`. A
timeout that fires late (after sleep) reports hours. Hard loads have no
API spans because the SDK loads after the page settles. `?view=` is
stripped, so all task list views share one route.

**Steps to reproduce**
Open the all-tasks page in a background tab on a traced instance. The
`page.load` span is `outcome=timeout`, `was_hidden=true`, about 30.4 s
(reproduced locally on master: 30,359 ms and 30,988 ms).

**Expected behavior**
A hidden load is counted separately and ends quickly. A timeout lasts
exactly 30 s. A hard load shows its API requests. The list view is a
dimension.

## What Changed

- `page-load-recorder.ts`: new outcome `hidden`. A load that starts
hidden, or hides before it is ready, ends at the moment the tab hid
(same rule `web-vitals` uses for LCP/FCP). For a hard load, the
controller reads Chromium `visibility-state` entries, so a hide before
tracing started is caught. A redirect while still hidden renames the
hidden span instead of starting a second one.
- `page-load-recorder.ts`: any end observed after start + 30 s is a
`timeout` that ends at exactly start + 30 s.
`paperclip.page.suspended=true` marks an end more than 5 s past the
deadline or a gap of more than 5 s between frames.
- `controller.ts`: a `PerformanceObserver` (`type: resource`, `buffered:
true`) keeps only this origin's `/api/` fetch/XHR entries (max 500)
until the SDK loads. It does not depend on the 250-entry resource
buffer.
- `sdk.ts`: when the SDK loads, it creates `HTTP` child spans under each
`page.load` that started before the SDK, for requests that started in
the load window and before live fetch tracing began (no duplicates). The
name is `HTTP` because Resource Timing has no method (OpenTelemetry's
name for an unknown method). Status comes only from `responseStatus`
when the browser exposes it. The URL goes through the same sanitizer as
live fetch spans and becomes `paperclip.api.area`. Spans carry
`paperclip.api.timing_source=resource_timing`.
- `page-view.ts`: `paperclip.page.view` for `/:companyPrefix/issues`
(values from `TASK_VIEW_KEYS`, `default` when missing) and
`/:companyPrefix/decisions` (values from `ATTENTION_PERSONAL_VIEWS`,
`mine` when missing). Any other value is `other`.
- `span-sanitizer.ts`: allowlist adds `paperclip.page.suspended`,
`paperclip.page.view`, `paperclip.api.timing_source`.
- `doc/browser-performance-tracing.md`: outcome list, the meaning of
"loaded" for hidden tabs, suspend handling, page views, backfilled API
spans, attribute table, and example queries that filter `outcome =
ready`.

## Verification

- Red then green: 17 new tests failed on the old code and
`page-view.test.ts` failed to import; all 50 tests in
`ui/src/lib/browser-tracing` pass after the change.
`PageLoadTracker.test.tsx` passes. `tsc --noEmit` for `ui` passes.
- `pnpm verify` exits 2 (a doc is a global CI input), so CI supplies
broad coverage.
- Real browser (local dev server on an empty isolated instance, stand-in
OTLP collector, headless Chromium, tracing config injected into the
session response):
- Background tab (raw CDP, `Target.createTarget background:true`):
`outcome=hidden`, duration 0 ms. Master: `outcome=timeout`, 30,359 ms.
- Tab covered 400 ms into the load: `outcome=hidden`, 434 ms. Master:
`timeout`, 30,988 ms.
- Hard load of the all-tasks page: `page.load` with 13 to 38 `HTTP`
child spans (areas `companies`, `plugins`, `cli-auth`, and others,
status 200 where exposed), same trace, parent = `page.load`.
  - `?view=mine`: `paperclip.page.view=mine`.
- Leak scan of every exported body: no URL, host, company id, company
prefix, user id, or query text.
- Entry bundle: 3,822,045 → 3,823,991 B raw (+1.9 KB), 1,154,280 →
1,155,097 B gzip (+0.8 KB, 0.07%). Lazy SDK chunk: 84,322 → 85,229 B
raw, 27,304 → 27,630 B gzip.

## Risks

- Low risk. The change is limited to the operator-gated Observability
path on hosted instances; self-hosted installs never start the
controller.
- Behavior shift for dashboards: hidden tabs are now `outcome=hidden`
instead of `timeout`, so timeout rates drop. Queries should filter
`outcome = ready`.
- The observer adds one callback per batch of finished requests while
tracing is on, until the SDK loads. It keeps at most 500 entries.
- Backfilled spans do not join server traces because no `traceparent`
was sent. The docs say so.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, headless Chromium).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 09:19:25 -06:00

325 lines
18 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Browser Performance Tracing
The board UI can send real-user page-load measurements, Core Web Vitals, and
API request spans to Honeycomb (or any OTLP/HTTP collector). The data answers
questions such as "what is the P95 time to load the all-tasks page?" and can
back a Honeycomb SLO with an owner.
This is part of the operator-gated Observability path (see
[observability.md](observability.md) and `rules/data-paths.mdc`). It is not the
first-party Telemetry system and adds no Telemetry events.
## Who gets it
Browser tracing runs **only on hosted Paperclip Cloud instances** that an
operator explicitly configures. It is off by default.
The server decides, in `server/src/browser-tracing-config.ts`. It sends a
`browserTracing` object in the authenticated `GET /api/auth/get-session`
response only when both hold:
1. The instance is cloud managed (`isCloudManagedInstance`: a
`PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN` or a `PAPERCLIP_MANAGED_CONFIG`
document is present).
2. `PAPERCLIP_BROWSER_TRACES_ENDPOINT` is set and every setting is valid.
Otherwise the field is `null`. Self-hosted installs, local development, and
tests never satisfy (1). With `null`, or with no session (signed out), the
browser never downloads the tracing code: it lives in its own lazily loaded
chunk (`ui/src/lib/browser-tracing/sdk.ts`). An invalid setting fails closed:
the field is `null` and the server logs one warning naming the variable, never
its value.
## Configuration
Set these on the hosted server process:
| Variable | Required | Default | Meaning |
| --- | --- | --- | --- |
| `PAPERCLIP_BROWSER_TRACES_ENDPOINT` | yes | — | OTLP/HTTP base URL, for example `https://api.honeycomb.io` (US) or `https://api.eu1.honeycomb.io` (EU). `/v1/traces` is appended unless the path already ends with it. Must be `https` (plain `http` only for a loopback host). |
| `PAPERCLIP_BROWSER_TRACES_INGEST_KEY` | for Honeycomb | — | A Honeycomb **ingest** key (prefix `hcaik_`). Sent as `x-honeycomb-team`. Required when the endpoint is a Honeycomb host; optional for a collector you run. Any other key format is refused. |
| `PAPERCLIP_BROWSER_TRACES_SAMPLE_RATE` | no | `1` | Fraction of page sessions traced, `0`–`1`, rounded to an exact 1 in N (`0.3` becomes 1/3; `0.75` becomes 1). `0` turns tracing off. |
| `PAPERCLIP_BROWSER_TRACES_SERVICE_NAME` | no | `paperclip-web` | `service.name`; in Honeycomb this is the dataset name. |
| `PAPERCLIP_BROWSER_TRACES_ENVIRONMENT` | no | `SENTRY_ENVIRONMENT` | `deployment.environment.name`, for example `staging`. |
### Key handling
Every signed-in browser can read the key, the same way it can read the
front-end Sentry DSN. That is why only a Honeycomb **ingest** key is accepted:
it can send events but cannot read data or change the Honeycomb account. Use a
key scoped to the environment that should receive browser data, keep it in the
hosting secret store, and never commit it. To keep the key out of browsers
entirely, point the endpoint at an OTLP collector you run (it adds the key) and
leave `PAPERCLIP_BROWSER_TRACES_INGEST_KEY` unset. That collector must allow
CORS `POST` from the instance origin with the `content-type` header.
### Sampling
The sample decision is made once per page session (one tab load), before the
SDK downloads. An unsampled session downloads and sends nothing. A sampled
session sends everything, so each trace is complete. When the rate is below 1,
every span carries Honeycomb's `SampleRate` attribute (1 in N, for example `4`
for `0.25`), so Honeycomb's `COUNT` and percentiles are weighted correctly.
Volume per sampled session: one `page.load` span per navigation, up to three
`web_vital.*` spans per page lifetime, and one span per same-origin `/api/`
request made during a measured navigation or after the SDK loads.
## Load-speed cost
Nothing loads on the critical path. The main bundle carries only the
measurement code (route matching, readiness checks); on the `ui` build of this
change the initial JavaScript grew by about 6.8 KB raw / 2.6 KB gzip (0.2%). The
OpenTelemetry SDK, a small OTLP JSON exporter, fetch instrumentation, and
`web-vitals` are a separate chunk (about 84 KB raw / 27 KB gzip) fetched only when tracing is on
and the session is sampled, and only after the page settles (painted, no query
fetching, browser idle; `ui/src/lib/when-page-settles.ts`). If the page never
settles (for example a request hangs), the chunk loads 35 seconds after tracing
starts anyway, so a hard load that ended as `timeout` is still exported.
Because the SDK loads after the first page settles, it cannot trace the hard
load's API requests live. Instead the main-bundle controller registers one
`PerformanceObserver` for `resource` entries and keeps only this
origin's `/api/` `fetch`/XHR entries, at most 500. `buffered: true` delivers
the requests that finished before tracing started (it starts once the session
request returns). After that, the observer receives every entry even when the
browser's resource buffer (250 entries by default) is full; only entries that
overflowed that buffer before tracing started are missed. When the SDK loads it turns
those entries into child spans of the `page.load` they belong to (see
[Backfilled API spans](#backfilled-api-spans)). The observer stops and the
entries are dropped once no navigation that started before the SDK loaded can
still claim them: every such navigation has ended, and either a navigation
that started after the SDK loaded has begun or 60 seconds have passed (a
navigation whose history change came before the SDK can commit after it).
Measurements taken before the SDK loads are buffered and exported when it
arrives.
## What "loaded" means
Each navigation produces one `page.load` span. It ends at the first animation
frame, held for two consecutive frames, in which:
1. no full-page loading state is mounted (the `PaperclipLoading` spinner, which
is also the route `Suspense` fallback), and
2. one of:
- **Content signal** (`ready_signal = content`): when the page declared a
primary-content signal with `usePageContentReady`, every declared signal
is ready. The end time is the paint time the page reported.
- **Idle signal** (`ready_signal = idle`): otherwise, no React Query query
that has no data yet is fetching. Background refetches of data already on
screen do not count. The end time is the first such frame.
The rule lives in one place (`ui/src/lib/browser-tracing/page-load-recorder.ts`)
and the route comes from the router's own route tree (`App.tsx` passes the tree
it renders to `PageLoadTracker`), so a new page is measured without page code.
- **Hard load** (`load_type = hard`): starts at navigation start (the
document request, time 0) and includes HTML, JavaScript download and start
up, authentication, and the page's data and render. Redirects during the load
(for example `/` to `/:companyPrefix/dashboard`) keep the same span, reported
under the final route.
- **Client navigation** (`load_type = client`): starts when the URL path
changes in history (a link click, `navigate()`, or back/forward) and includes
lazy route code download, data loading, and render. Query-string and hash
changes are not navigations. A history replace while a navigation is still
open is a redirect and continues it.
Outcomes:
- `ready`: loaded, as defined above.
- `interrupted`: another navigation started first.
- `hidden`: the tab was hidden when the navigation started (a link opened in
a background tab) or became hidden before the page was ready. The span ends
at the moment the tab hid, so a load in a background tab has a duration of
about 0. Browsers do not paint hidden tabs, so such a load could never
become ready; this is the same rule `web-vitals` applies to LCP and FCP.
For a hard load, Chromium's `visibility-state` performance entries give the
time the tab first hid, even before tracing started; elsewhere a tab that is
hidden when tracing starts counts as hidden from navigation start. A
redirect while the tab is still hidden renames the hidden span rather than
starting another one. `was_hidden` is `true` on these spans.
- `timeout`: not ready 30 seconds after it started. The span always ends at
exactly start + 30 s, however late the browser ran the timer. Any end
observed after that deadline (for example a page that only became ready
after the computer woke up) is also reported as this timeout.
`paperclip.page.suspended = true` marks a span during which the computer slept
or the tab was frozen: the observed end was more than 5 seconds past the
30 s deadline, or two consecutive animation frames were more than 5 seconds
apart. Its duration does not describe the page; exclude it like `hidden`. A
suspend that ends before the next frame is not detectable, but it can never
stretch a span beyond 30 s.
A navigation open when the tab closes is not sent. For load-time percentiles
and SLOs, filter on `outcome = ready`; that excludes hidden tabs, timeouts and
interruptions, which you count separately.
Pages with an explicit content signal:
| Page | Route | Loaded when |
| --- | --- | --- |
| Task (issue) page | `/:companyPrefix/issues/:issueId` | The conversation content has painted: the same point as the `issue-detail:navigate→content-paint` User Timing measure. |
| All-tasks list (plain task list) | `/:companyPrefix/issues` | The first page of task rows has rendered (or the list shows its error state). The combined inbox and coordinator task views use the idle signal. |
An agent chat with no saved task has no content signal and uses the idle rule.
### Page views
Some pages show a different list depending on `?view=`. For those,
`paperclip.page.view` records the view (`ui/src/lib/browser-tracing/page-view.ts`),
read from the URL when the span ends, so it is the view the page resolved and
wrote into the URL. Only names from the code's view registries are sent; any
other value is `other`.
| Route | Values |
| --- | --- |
| `/:companyPrefix/issues` | The keys of `TASK_VIEW_KEYS` in `ui/src/lib/task-views.ts` (`mine`, `unread`, `blocked`, `recent`, `everything`, `needs_attention`, `all`, `active`, `backlog`, `done`), `default` when the URL has no view yet, else `other` |
| `/:companyPrefix/decisions` | `ATTENTION_PERSONAL_VIEWS` (`mine`, `unassigned`, `company`, `archived`, `all`), `mine` when the URL has no view, else `other` |
Switching views on the same page changes only the query string, so it is not a
new navigation and has no `page.load` span; the attribute describes the view a
navigation landed on.
To add one, call `usePageContentReady(ready)` in the page component. Use it
only when the idle rule is inaccurate for that page.
## Emitted spans
All spans pass through an allowlist (`ui/src/lib/browser-tracing/span-sanitizer.ts`)
before export. Any other attribute an instrumentation adds, including full
URLs, server addresses, status messages, and event attributes, is dropped.
| Span name | When |
| --- | --- |
| `page.load` | One per hard load or client navigation. |
| `web_vital.lcp`, `web_vital.inp`, `web_vital.cls` | Core Web Vitals, from Google's `web-vitals`, usually reported as the tab hides. Zero duration; the value is an attribute. |
| `GET`, `POST`, … | A `fetch` to this origin's `/api/`. Its `traceparent` header joins the server's request span to the same trace. While a `page.load` span is open it is the parent, so one trace shows the navigation, its API calls, and (when server tracing is on) their server spans. |
| `HTTP` | A request to this origin's `/api/` made before the SDK loaded, during a measured navigation (in practice the hard load), rebuilt from Resource Timing. See below. |
### Backfilled API spans
The hard load's API requests happen before the SDK exists. When the SDK loads,
it creates one `HTTP` child span under that load's `page.load` for each
same-origin `/api/` `fetch`/XHR Resource Timing entry that started inside the
load's window (and before the SDK started tracing requests live, so nothing is
counted twice). Each runs from the entry's `startTime` to its `responseEnd`.
They differ from live fetch spans in three ways:
- The name is `HTTP`, OpenTelemetry's name for a request whose method is
unknown: Resource Timing does not expose the method, so there is no
`http.request.method`.
- `http.response.status_code` is present only where the browser exposes
`responseStatus` (Chromium); it is never guessed.
- No `traceparent` was sent with these requests, so they do not join the
server's spans. Use `paperclip.api.area` and the timings; for the server
side, query server spans by time.
They carry `paperclip.api.timing_source = resource_timing`. The URL is reduced
to `paperclip.api.area` by the same sanitizer as live fetch spans.
### Attributes
Attributes (the complete allowlist):
| Attribute | On | Values |
| --- | --- | --- |
| `paperclip.page.route` | all | Route pattern, for example `/:companyPrefix/issues/:issueId`; `(unmatched)` when no route matches. Built only from route definitions, never from the URL. On fetch spans: the route current when the request started. On `web_vital.lcp`: the hard load's route. |
| `paperclip.page.load_type` | `page.load` | `hard`, `client` |
| `paperclip.page.outcome` | `page.load` | `ready`, `interrupted`, `hidden`, `timeout` |
| `paperclip.page.ready_signal` | `page.load` | `content`, `idle`, `none` |
| `paperclip.page.was_hidden` | `page.load` | boolean |
| `paperclip.page.suspended` | `page.load` | `true` when the computer slept or the tab froze during the measurement; absent otherwise |
| `paperclip.page.view` | `page.load` on pages with views | The view from `?view=`, from a fixed list per page, else `other`; absent on other pages |
| `paperclip.page.navigation_type` | hard `page.load` | `navigate`, `reload`, `back_forward`, `prerender` |
| `paperclip.page.ttfb_ms`, `paperclip.page.dom_content_loaded_ms`, `paperclip.page.load_event_ms` | hard `page.load` | Navigation Timing milestones, ms from navigation start |
| `paperclip.web_vital.name` | `web_vital.*` | `LCP`, `INP`, `CLS` |
| `paperclip.web_vital.value` | `web_vital.*` | ms for LCP and INP; unitless for CLS |
| `paperclip.web_vital.rating` | `web_vital.*` | `good`, `needs-improvement`, `poor` |
| `paperclip.api.area` | fetch | First path segment after `/api/` when it is a fixed word (`issues`, `companies`, …), else `other`. Use the joined server span's route for detail. |
| `paperclip.api.timing_source` | backfilled `HTTP` | `resource_timing` |
| `http.request.method`, `http.response.status_code`, `error.type` | fetch | OpenTelemetry HTTP semantics (backfilled spans: status only, where the browser exposes it) |
| `SampleRate` | all, when sampling | 1-in-N weight |
Resource: `service.name`, `service.version` (the UI build commit),
`deployment.environment.name`.
Not sent: URLs, paths with ids, query strings, hashes, referrers, user agent,
user, company, or task identifiers, titles, or content. Server spans carry no
company identifier, so browser spans do not either. Uncaught browser errors are
not sent here because Sentry already reports them
([observability.md](observability.md#sentry-error-monitoring)).
Trace headers go only to this origin (`propagateTraceHeaderCorsUrls` is empty),
so no third party receives a `traceparent`. Same-origin requests need no CORS
change. The server's OpenTelemetry HTTP instrumentation continues the trace
when server tracing is enabled ([observability.md](observability.md#enabling-tracing)).
## Example Honeycomb queries
Use the `paperclip-web` dataset (or your `service.name`) and the environment
that holds browser data. Load-time percentiles filter on `outcome = ready`,
which leaves out hidden tabs, timeouts, and interruptions. Also leave out
suspended spans: `paperclip.page.suspended` is only ever set to `true`, so
filter on it not existing.
**P95 load time of the all-tasks page, by view**
```text
VISUALIZE P95(duration_ms), COUNT
WHERE name = page.load
paperclip.page.route = /:companyPrefix/issues
paperclip.page.outcome = ready
paperclip.page.suspended does-not-exist
GROUP BY paperclip.page.load_type, paperclip.page.view
```
**P95 load time of an individual task page**
```text
VISUALIZE P95(duration_ms), COUNT
WHERE name = page.load
paperclip.page.route = /:companyPrefix/issues/:issueId
paperclip.page.outcome = ready
paperclip.page.suspended does-not-exist
GROUP BY paperclip.page.load_type
```
**API requests during hard loads of the all-tasks page**
```text
VISUALIZE P95(duration_ms), COUNT
WHERE paperclip.api.timing_source = resource_timing
paperclip.page.route = /:companyPrefix/issues
GROUP BY paperclip.api.area
```
Group by `service.version` to compare builds. For a timeout rate, count
`paperclip.page.outcome = timeout` against `page.load` spans for the route
whose outcome is not `hidden`.
For Web Vitals, `P75(paperclip.web_vital.value)` where
`name = web_vital.lcp` (or `inp`, `cls`), grouped by `paperclip.page.route`.
**SLO.** A Honeycomb SLO needs a boolean SLI as a derived column. For example,
"task page loads within 2 s":
```text
IF(AND(EQUALS($name, "page.load"),
EQUALS($paperclip.page.route, "/:companyPrefix/issues/:issueId"),
IN($paperclip.page.outcome, "ready", "timeout"),
NOT(EQUALS($paperclip.page.suspended, true))),
AND(EQUALS($paperclip.page.outcome, "ready"), LTE($duration_ms, 2000)))
```
The expression is null for other events (including `hidden` and
`interrupted` loads), so they do not count; a timeout counts as a failure.
## Verifying a deployment
1. Set the variables above and deploy. A cloud-managed instance logs nothing on
success; an invalid setting logs one `browser tracing is off` warning.
2. Sign in, open the browser network panel, and confirm the session response
carries `browserTracing`, a `sdk-*.js` chunk loads after the page settles,
and a `POST` to `…/v1/traces` returns 200.
3. In Honeycomb, query `name = page.load` in the browser dataset for the last
10 minutes and check `paperclip.page.route` and `service.version`.