docs: add Honeycomb tracing skill (#16453)

Document how to find, configure, and interpret Paperclip application and CI traces in Honeycomb.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

.agents/skills/honeycomb-tracing/SKILL.md

@@ -0,0 +1,169 @@
+---
+name: honeycomb-tracing
+description: >-
+  Trace Paperclip runs, requests, and full CI suites in Honeycomb. Use to find
+  traces, diagnose latency, configure opt-in OpenTelemetry, or add missing spans.
+---
+
+# Honeycomb Tracing
+
+Produce an inspectable waterfall tied to the actual run or CI attempt, explain
+which intervals dominate elapsed time, and identify the smallest useful next
+experiment. Respect diagnosis-only requests; tracing does not itself authorize
+replaying tasks, changing credentials, or deploying instrumentation.
+
+## Choose the data path
+
+| Question | Start here |
+| --- | --- |
+| Why is a task slow before, during, or after model execution? | Application workflow below; [runtime setup](references/runtime.md) if spans are absent |
+| Why does an issue page load slowly? | Request spans plus browser timing, described below |
+| Which CI jobs or test files delay the whole suite? | [CI profiling](references/ci.md) |
+| Where should a new span go? | Instrumentation guidance below and [Observability contract](../../../doc/observability.md) |
+
+Repository-relative paths and commands assume the checkout root. The maintained
+contracts are `doc/observability.md` and `doc/ci-profiling.md`; recheck them and
+the owning code when a deployed version differs.
+
+Keep the repository's three paths distinct:
+
+- **Observability:** operator-configured OpenTelemetry export to Honeycomb or
+  another OTLP destination. With no endpoint, this tracing stays off.
+- **Telemetry:** first-party events under `packages/shared/src/telemetry/`, sent
+  to Paperclip by default unless opted out. Adding these requires the generated
+  contract, Telemetry README, and privacy-review workflow in `AGENTS.md`.
+- **Run log:** local `heartbeat_run_events` and measured performance events.
+  Useful for correlation, but a local event is not proof of Honeycomb export.
+
+Observability changes within the closed span-attribute allowlist do not need the
+Telemetry privacy review. Preserve the endpoint gate and no-operation behavior.
+
+## Find the application's trace
+
+1. Resolve the instance, company, issue, agent, **run UUID**, and UTC interval.
+   Record the effective harness/model, workspace/repository, warm or fresh session,
+   and actual deployed revision. A task identifier is not a run UUID; one task
+   can have several runs, and one run can have several execution attempts.
+2. Prefer existing Honeycomb tools when exposed and authorized. Otherwise use
+   the signed-in browser. The current project link targets team `paperclip`,
+   environment `test`, dataset `paperclip`, and `service.name = paperclip`.
+   These are defaults in `ui/src/lib/honeycomb-run-link.ts`, not proof of every
+   deployment's destination. Verify the target before comparing data.
+3. Open **View in Honeycomb** on the run page when available. It requires
+   Paperclip Developer Mode and local implicit board or instance-admin access.
+   This link queries existing data. **Re-run with provider trace** is a different
+   action: it starts work and is not needed to read Honeycomb.
+4. Without the link, calculate the same correlation value locally:
+
+   ```sh
+   # Set RUN_ID to the exact saved run UUID first; no trailing newline is hashed.
+   node -e 'console.log(require("node:crypto").createHash("sha256").update(process.argv[1]).digest("hex").slice(0, 12))' "$RUN_ID"
+   ```
+
+   Filter `paperclip.task.run.run_id` to that 12-character lowercase hash and
+   `service.name` to the observed service. Start with the run's absolute time
+   window. The product link defaults to the last seven days; widen it for older
+   runs. Group `COUNT` by `trace.trace_id` to discover matching traces.
+5. The product's root filter is `name in [task.execution_attempt,
+   task.provider_session, task.run]`. Use Honeycomb's `in` operator, not an
+   invented OR group. Remove the name filter when setup failed or preparation
+   spans are missing; older/standalone preparation can have a separate trace ID.
+6. Open each relevant trace and its span details. A count-only query/export
+   cannot establish span durations, parentage, or causal attribution. Capture
+   the trace URL, time window, service/version, and relevant span names/timings.
+
+In recent investigations, `HONEYCOMB_TEST_CONFIG_API_KEY` in the operator's
+`~/.secrets` could read metadata but could not query traces. Ingest permission,
+metadata access, and query access are different. Verify current capabilities;
+when queries are unavailable, use the signed-in UI rather than creating a new
+key or upgrading permissions. Read only needed secret fields, never dump the
+file, and keep values out of commands, artifacts, repository files, and reports.
+
+## Read the timeline correctly
+
+| Interval | Meaning / next inspection |
+| --- | --- |
+| Run creation → executor start | Admission/queue delay; check drain, capacity, and dispatch evidence |
+| `task.execution_attempt` | Server executor invocation through preparation, dispatch, durable reply, and cleanup |
+| `heartbeat.prepare.*`, `native.controller.*` | Controller preparation; expand workspace, configuration, backend, and runtime-selection work |
+| `task.provider_session` | Native provider lifecycle; includes preparation and execution, ends before outer workspace persistence |
+| `task.run` | Legacy adapter root |
+| `runner.session.startup`, `environment.*`, `stage.*`, `sandbox.*` | Runner/environment acquisition, workspace transfer, and startup |
+| `agent.turn` | Agent turn time; inspect actual children before calling all of it model latency |
+| `execution.workspace_finalize`, `workspace_sync.*`, `completion.*`, `response.*` | Copyback, settlement, response persistence, and publication work |
+
+The attempt and provider lifecycle are linked roots; they need not share a trace
+ID. Neither is a guarantee of one continuous trace across queueing or restarts.
+Correlate with the run log and saved reply timestamps. The native local event
+`task.run.measured` includes terminal elapsed time and durable copyback.
+
+Use start/end times to compare serial intervals and the union of parallel work.
+Never add a parent to its children or sum parallel branches as elapsed time.
+For example, `configuration_skills` is inside `execution_configuration`, and
+`skills.prepare` overlaps the broader `heartbeat.prepare_before_environment`.
+Detail spans are capped at 16 per name per attempt; missing per-item children
+do not mean later items did no work.
+
+Distinguish first streamed text, committed reply, externally delivered message,
+and browser paint. A successful `response.transaction.commit` proves durable storage;
+`response.publish` is enqueueing and `execution.realtime` is server dispatch.
+Neither proves that the user saw the response. Choose the boundary the user asked
+to improve, then measure the whole path to that boundary.
+
+For slow issue pages, combine `issue.read.<phase>` with `Server-Timing`
+(`paperclip_issue`, `issue_<phase>`) and the browser's request waterfall and
+`issue-detail:navigate→content-paint` measure. Concurrent reads overlap. Fast
+API completion does not establish fast conversation reveal or rendering.
+
+A long parent span without children is an attribution gap. It does not prove
+SQL, network, or model time. For database hypotheses, use
+[fix-slow-queries](../fix-slow-queries/SKILL.md). For CPU hypotheses, use a bounded
+authorized profile covering the actual slow phase; a sample taken while queued
+cannot explain preparation. Low aggregate container CPU does not exclude a
+blocked Node main thread. Keep profiled diagnostics separate from unprofiled
+acceptance timings.
+
+## Add only the missing instrumentation
+
+Read applicable repository rules before editing. Reuse the existing timing seams:
+
+- `server/src/services/execution-preparation-timing.ts`:
+  `withRunExecutionTiming`, `createRunPreparationTiming`, `measureRunPhase`, and
+  `beginRunPhase`. Use the existing attempt; do not create a new root per detail.
+- `server/src/services/native-runtime/native-run-trace.ts`: native lifecycle
+  hierarchy and links to the owning attempt.
+- `server/src/services/issue-read-timing.ts`: issue-read phases and Server-Timing.
+- `packages/adapter-utils/src/acpx-engine/startup-timing.ts`: runtime parent
+  context and closed span attributes. Adapter utilities receive the host context
+  rather than constructing an independent exporter.
+
+Extend the closed phase union when a meaningful boundary is missing. Wrap the
+actual awaited operation, preserve parent context across async/provider boundaries,
+and end spans on success and failure. Keep real wall times for spans recorded
+after the operation. Use existing hashed IDs, closed enums, and allowed numeric
+counts; do not add prompts, tool arguments/results, paths, raw identifiers,
+credentials, or exception text. Do not enable all filesystem instrumentation to
+explain one slow scan.
+
+Observability must not retry the operation, replace its return/error, change
+authorization/locks, or break execution when the tracer fails. Preserve root
+sampling and the explicit endpoint gate even if another library registers an
+OTel provider. Update `doc/observability.md` when the trace contract changes.
+Run focused tests for endpoint-off behavior, parentage/concurrent-run isolation,
+failure handling, and the changed seam. Use real exported evidence to claim that
+new spans appear in Honeycomb; a unit-test tracer alone does not prove ingestion.
+
+## Compare and hand back useful evidence
+
+Choose the largest actionable serial wait or unexplained gap, then reproduce it
+at the smallest faithful layer. Compare matched runs: same harness/model,
+workspace size, session warmth, concurrency, and relevant revision/configuration.
+Report sample count and median/range when several runs are available. Preserve
+failures; a failed fast run is not a latency improvement.
+
+Return the run/attempt and trace links, observed revision, end-to-end timing,
+dominant non-overlapping phases, uncertainty, and next action or verified gain.
+Label local, synthetic, and hosted evidence precisely: a dev-host reproduction
+is not a local checkout run, and a merged fix does not prove hosted improvement.
+If traces are unavailable, state the missing access or coverage and what the
+run log/browser measurements actually establish.

.agents/skills/honeycomb-tracing/agents/openai.yaml

@@ -0,0 +1,3 @@
+interface:
+  display_name: "Honeycomb Tracing"
+  short_description: "Trace Paperclip runs, requests, and full CI waterfalls"

.agents/skills/honeycomb-tracing/references/ci.md

@@ -0,0 +1,96 @@
+# Profile the full PR suite
+
+Use [doc/ci-profiling.md](../../../../doc/ci-profiling.md) as the maintained
+operator runbook. This path exports recorded CI timing through Honeycomb's Events
+API; it does not require application OTLP settings or an application run UUID.
+
+## Select one complete attempt
+
+1. Confirm the intended PR and head. Profiling is opt-in through `profile-ci`.
+   The PR author, original workflow actor, and rerun actor must be individual
+   users authorized by trusted master's `.github/**` CODEOWNERS rule. Applying
+   the label to an outside contributor's PR does not grant profiling access.
+2. When asked to profile that PR, add the label. It starts a fresh
+   **CI profile · PR #number** run of the main `PR` workflow. The label forces
+   all main lanes even for stack-middle or release-notes-only changes. Further
+   pushes also profile while it remains present.
+3. Record run ID, attempt, tested revision, and job inventory. Keep the label
+   and head unchanged until the separate **CI profile** exporter completes.
+   Use a fresh label-triggered run for an unambiguous whole-suite baseline.
+   Rerunning only failures does not rerun the successful jobs.
+4. Open the Honeycomb waterfall from the run/exporter's Summary. The project
+   destination is team `paperclip`, environment `test`, dataset `paperclip-ci`.
+   Verify the displayed root and representative file spans; a generated URL
+   or green test workflow is not proof that export succeeded.
+5. Remove the label after a one-off profile unless ongoing profiling was requested.
+   Re-adding it starts another complete attempt. Report any remaining label so
+   future pushes do not unexpectedly keep profiling.
+
+The main suite preserves existing AWS routing (`paperclip-public-pr-x64`, with
+`paperclip-ci-routing-x64` for routing) and its authorized hosted fallback.
+Inspect actual runner assignments if AWS placement is part of acceptance. The
+separate exporter uses `ubuntu-latest` and is outside the measured suite.
+
+## Read and qualify the result
+
+The hierarchy is **workflow → job → step → test filename → tests**. Collapse
+spans at the job depth for a shared wall-clock view; expand slow jobs and search
+for a known `test.file` to prove filename-level detail arrived.
+
+- Root elapsed time runs from workflow creation to the last job completion;
+  reruns start at that attempt's start. Do not sum concurrent job durations.
+- Use `ci.finishes_last`, `ci.execution_ms`, and `ci.start_offset_ms` to find
+  late-finishing work, then inspect setup, caches, compilation, builds, and tests.
+  A long job that ends early may not delay the suite.
+- Waiting spans start at job creation. Dependency waits before job creation
+  appear as gaps; GitHub does not expose a separate ready timestamp, so those
+  gaps are not proven runner-queue delay or a computed dependency critical path.
+- Vitest, Playwright, and Node supply file/test detail with different timestamp
+  semantics. GitHub steps have second precision; Node test spans use reporter
+  receipt time. Rust and standalone scripts retain job/step coverage.
+- Scope is the main PR workflow. Separate Storybook, image, release, deployment,
+  and live-provider eval workflows are not included.
+
+Inspect `ci.profile_complete`, coverage warnings, expected versus reported
+invocations, file/test counts, `ci.partial_rerun`, and `ci.carried_job_count`.
+Failed/cancelled attempts can omit downstream work; carried jobs completed before
+the new attempt are not fresh measurements. Compare the trace inventory with
+GitHub's attempt-specific jobs and retained timing artifacts.
+
+Honeycomb's waterfall displays at most 32,000 spans, prioritizing those closest
+to the root. Missing deep test rows can be a display limit rather than missing
+ingest. Check **View events** and coverage evidence before claiming data loss.
+See the [provider's large-trace limit](https://docs.honeycomb.io/troubleshoot/common-issues/data-in-honeycomb/#your-very-large-trace-displays-only-32000-spans).
+
+## Credentials and export failures
+
+`HONEYCOMB_CI_API_KEY` is a dedicated ingest-only **environment secret** in
+GitHub environment `ci-profiling`, restricted to `master`. Only the trusted
+default-branch exporter reads it. PR tests still execute the PR branch and never
+receive the key. Do not move it to repository/org secrets, put a configuration
+key in CI, execute downloaded artifacts, or switch trusted workflow refs to a PR
+head for convenience. Ordinary CI must work with profiling disabled and no key.
+
+The exporter checks each batch response, paces uploads, and retries only
+transiently rejected events. If it fails, inspect its logs and saved artifacts
+before retrying: partial export may already have sent spans, and the Events API
+does not promise deduplication on resubmission. Timing artifacts are retained for
+14 days. Preserve incomplete-export evidence even if a later upload succeeds.
+
+For local reconstruction without uploading, use the existing exporter:
+
+```sh
+node scripts/ci-profile/export.mjs --run-id=RUN_ID --attempt=1 --historical --dry-run --output=/tmp/ci-trace.json
+```
+
+Replace `RUN_ID` and attempt with the recorded values. This needs authorized
+GitHub read access. Historical reconstruction can recover job/step timings, but
+cannot invent file/test timings that were never recorded. It does not prove the
+live label-triggered journey. A real historical upload is a separate authorized
+action using the ingest key, not a necessary part of diagnosis.
+
+When changing CI tracing, use the runbook's focused exporter, routing, reporter,
+and workflow checks. For a whole-suite tracing acceptance, return the exact
+GitHub attempt and Honeycomb links, elapsed time, coverage and export status,
+runner placement if relevant, and the late-finishing jobs/file timings that
+justify the next optimization.

.agents/skills/honeycomb-tracing/references/runtime.md

@@ -0,0 +1,90 @@
+# Runtime tracing setup and missing spans
+
+Read this when configuring an authorized runtime or diagnosing absent spans.
+The authoritative implementation is
+[instrumentation.ts](../../../../server/src/instrumentation.ts); package versions
+and the data contract live in
+[Observability](../../../../doc/observability.md#enabling-tracing).
+
+## Enable the existing bootstrap
+
+1. Identify the actual process/image and its owning package. Inspect
+   `server/package.json` for the **exact** peer versions of the OTel SDK,
+   auto-instrumentations, resources, semantic conventions, and one exporter.
+   Install them in the runtime that will resolve them. Avoid copying old version
+   pins from sessions or adding every exporter to the application manifest.
+2. Configure that process before startup. A direct US Honeycomb HTTP/protobuf
+   setup uses these values:
+
+   | Setting | Value |
+   | --- | --- |
+   | `OTEL_EXPORTER_OTLP_ENDPOINT` | `https://api.honeycomb.io` |
+   | `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` |
+   | `OTEL_EXPORTER_OTLP_HEADERS` | `x-honeycomb-team=<authorized ingest key>` |
+   | `OTEL_SERVICE_NAME` | `paperclip`, or the deliberately selected service identity |
+
+   Inject the header through the existing secret manager/process environment;
+   do not paste a key into shell history or a tracked `.env`. Confirm region and
+   environment from the operator's configuration. An existing collector may be
+   the intended endpoint instead of direct Honeycomb ingestion.
+3. For HTTP, supply the **base URL**; the exporter appends `/v1/traces`. Paperclip
+   intentionally constructs HTTP exporters without an explicit URL so they read
+   the standard environment variables. A signal-specific variable alone does
+   not activate Paperclip: the bootstrap gate checks the generic
+   `OTEL_EXPORTER_OTLP_ENDPOINT`.
+4. For gRPC, use the appropriate TLS endpoint and matching exporter; the protocol
+   defaults to gRPC when unset. Unknown protocol strings warn and fall back to
+   gRPC. Do not accidentally pair HTTP configuration with that default.
+5. Check the runtime's startup diagnostics. A missing or mismatched peer package
+   leaves the server healthy **without tracing**. `instrumentationReady` settles
+   before DB connections/server construction; shutdown awaits a bounded flush.
+   A healthy `/api/health` alone cannot validate export.
+6. Generate a small representative operation within the authorized task, allow
+   the export batch to arrive, and open its actual Honeycomb spans. Verify
+   service, revision, timestamps, parentage, and the expected operation. Record
+   a trace link as proof; a bootstrap log line only proves initialization.
+
+Honeycomb's [OTLP configuration](https://docs.honeycomb.io/send-data/opentelemetry/#using-the-honeycomb-opentelemetry-endpoint)
+documents regions, headers, and the additional dataset header required for
+Classic. Confirm current provider requirements when changing setup.
+
+## Identify the serving version
+
+Paperclip resolves `service.version` from the build stamp
+(`dist/build-info.json`), then runtime Git, then `OTEL_SERVICE_VERSION`, then
+`unknown`. The build stamp wins over a stale environment variable. Docker builds
+without `.git` use `PAPERCLIP_BUILD_COMMIT` to populate the stamp.
+
+Match the trace's version with actual runtime/deployment evidence. Record source
+revision and observation time; an opaque desired release or current local HEAD
+is not evidence that the running service includes a fix.
+
+## Troubleshoot in this order
+
+| Observation | Check |
+| --- | --- |
+| No results | Correct team/environment/dataset/service, exact run hash, absolute UTC range, retention, and query permissions |
+| Only native/provider spans | Remove root-name restriction; inspect other trace IDs for the same run hash and controller version |
+| App healthy, nothing exported | Actual process env, exact optional peers, protocol/exporter match, bootstrap diagnostic, root sampling, and authorized endpoint reachability |
+| Trace ends before saved reply | Provider root excludes outer persistence; inspect `task.execution_attempt`, settlement spans, and run-log timestamps |
+| Missing unfinished phase | Active spans may not export until they end; abrupt exit can lose them |
+| Long manual span, no SQL children | Auto-instrumentation does not guarantee children; inspect real dependency timings rather than equating the whole span with database time |
+| Query API denied, authentication works | Check query permission/plan; use the signed-in UI when available, without widening key privileges |
+| Export/CSV contains only COUNT | It is an aggregate result. Inspect span details or the authorized raw-events view for event-level analysis |
+
+The server exports traces only through this bootstrap. `fs`, `dns`, and `net`
+auto-instrumentations are disabled for volume. Local performance events can
+still exist with OTLP off; their presence is not an exporter health check.
+
+When changing the bootstrap or correlation, start with the relevant tests:
+
+```sh
+pnpm exec vitest run server/src/__tests__/instrumentation.test.ts
+pnpm exec vitest run server/src/services/execution-preparation-timing.test.ts
+pnpm exec vitest run server/src/services/native-runtime/native-run-trace-context.test.ts
+pnpm exec vitest run ui/src/lib/honeycomb-run-link.test.ts
+```
+
+Select affected checks rather than running this entire list for a read-only
+diagnosis. These establish local contracts; deployment and Honeycomb ingestion
+need their own evidence.