From 7a8b3583659150bcd819b40be5d6dc9d1df2ee88 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 15 Oct 2026 11:51:09 -0500 Subject: [PATCH] docs: add Honeycomb tracing skill (#16453) Document how to find, configure, and interpret Paperclip application and CI traces in Honeycomb. Co-Authored-By: Paperclip --- .agents/skills/honeycomb-tracing/SKILL.md | 169 ++++++++++++++++++ .../honeycomb-tracing/agents/openai.yaml | 3 + .../skills/honeycomb-tracing/references/ci.md | 96 ++++++++++ .../honeycomb-tracing/references/runtime.md | 90 ++++++++++ 4 files changed, 358 insertions(+) create mode 100644 .agents/skills/honeycomb-tracing/SKILL.md create mode 100644 .agents/skills/honeycomb-tracing/agents/openai.yaml create mode 100644 .agents/skills/honeycomb-tracing/references/ci.md create mode 100644 .agents/skills/honeycomb-tracing/references/runtime.md diff --git a/.agents/skills/honeycomb-tracing/SKILL.md b/.agents/skills/honeycomb-tracing/SKILL.md new file mode 100644 index 00000000000..1f48dcb584b --- /dev/null +++ b/.agents/skills/honeycomb-tracing/SKILL.md @@ -0,0 +1,169 @@ +--- +name: honeycomb-tracing +description: >- + Trace Paperclip runs, requests, and full CI suites in Honeycomb. Use to find + traces, diagnose latency, configure opt-in OpenTelemetry, or add missing spans. +--- + +# Honeycomb Tracing + +Produce an inspectable waterfall tied to the actual run or CI attempt, explain +which intervals dominate elapsed time, and identify the smallest useful next +experiment. Respect diagnosis-only requests; tracing does not itself authorize +replaying tasks, changing credentials, or deploying instrumentation. + +## Choose the data path + +| Question | Start here | +| --- | --- | +| Why is a task slow before, during, or after model execution? | Application workflow below; [runtime setup](references/runtime.md) if spans are absent | +| Why does an issue page load slowly? | Request spans plus browser timing, described below | +| Which CI jobs or test files delay the whole suite? | [CI profiling](references/ci.md) | +| Where should a new span go? | Instrumentation guidance below and [Observability contract](../../../doc/observability.md) | + +Repository-relative paths and commands assume the checkout root. The maintained +contracts are `doc/observability.md` and `doc/ci-profiling.md`; recheck them and +the owning code when a deployed version differs. + +Keep the repository's three paths distinct: + +- **Observability:** operator-configured OpenTelemetry export to Honeycomb or + another OTLP destination. With no endpoint, this tracing stays off. +- **Telemetry:** first-party events under `packages/shared/src/telemetry/`, sent + to Paperclip by default unless opted out. Adding these requires the generated + contract, Telemetry README, and privacy-review workflow in `AGENTS.md`. +- **Run log:** local `heartbeat_run_events` and measured performance events. + Useful for correlation, but a local event is not proof of Honeycomb export. + +Observability changes within the closed span-attribute allowlist do not need the +Telemetry privacy review. Preserve the endpoint gate and no-operation behavior. + +## Find the application's trace + +1. Resolve the instance, company, issue, agent, **run UUID**, and UTC interval. + Record the effective harness/model, workspace/repository, warm or fresh session, + and actual deployed revision. A task identifier is not a run UUID; one task + can have several runs, and one run can have several execution attempts. +2. Prefer existing Honeycomb tools when exposed and authorized. Otherwise use + the signed-in browser. The current project link targets team `paperclip`, + environment `test`, dataset `paperclip`, and `service.name = paperclip`. + These are defaults in `ui/src/lib/honeycomb-run-link.ts`, not proof of every + deployment's destination. Verify the target before comparing data. +3. Open **View in Honeycomb** on the run page when available. It requires + Paperclip Developer Mode and local implicit board or instance-admin access. + This link queries existing data. **Re-run with provider trace** is a different + action: it starts work and is not needed to read Honeycomb. +4. Without the link, calculate the same correlation value locally: + + ```sh + # Set RUN_ID to the exact saved run UUID first; no trailing newline is hashed. + node -e 'console.log(require("node:crypto").createHash("sha256").update(process.argv[1]).digest("hex").slice(0, 12))' "$RUN_ID" + ``` + + Filter `paperclip.task.run.run_id` to that 12-character lowercase hash and + `service.name` to the observed service. Start with the run's absolute time + window. The product link defaults to the last seven days; widen it for older + runs. Group `COUNT` by `trace.trace_id` to discover matching traces. +5. The product's root filter is `name in [task.execution_attempt, + task.provider_session, task.run]`. Use Honeycomb's `in` operator, not an + invented OR group. Remove the name filter when setup failed or preparation + spans are missing; older/standalone preparation can have a separate trace ID. +6. Open each relevant trace and its span details. A count-only query/export + cannot establish span durations, parentage, or causal attribution. Capture + the trace URL, time window, service/version, and relevant span names/timings. + +In recent investigations, `HONEYCOMB_TEST_CONFIG_API_KEY` in the operator's +`~/.secrets` could read metadata but could not query traces. Ingest permission, +metadata access, and query access are different. Verify current capabilities; +when queries are unavailable, use the signed-in UI rather than creating a new +key or upgrading permissions. Read only needed secret fields, never dump the +file, and keep values out of commands, artifacts, repository files, and reports. + +## Read the timeline correctly + +| Interval | Meaning / next inspection | +| --- | --- | +| Run creation → executor start | Admission/queue delay; check drain, capacity, and dispatch evidence | +| `task.execution_attempt` | Server executor invocation through preparation, dispatch, durable reply, and cleanup | +| `heartbeat.prepare.*`, `native.controller.*` | Controller preparation; expand workspace, configuration, backend, and runtime-selection work | +| `task.provider_session` | Native provider lifecycle; includes preparation and execution, ends before outer workspace persistence | +| `task.run` | Legacy adapter root | +| `runner.session.startup`, `environment.*`, `stage.*`, `sandbox.*` | Runner/environment acquisition, workspace transfer, and startup | +| `agent.turn` | Agent turn time; inspect actual children before calling all of it model latency | +| `execution.workspace_finalize`, `workspace_sync.*`, `completion.*`, `response.*` | Copyback, settlement, response persistence, and publication work | + +The attempt and provider lifecycle are linked roots; they need not share a trace +ID. Neither is a guarantee of one continuous trace across queueing or restarts. +Correlate with the run log and saved reply timestamps. The native local event +`task.run.measured` includes terminal elapsed time and durable copyback. + +Use start/end times to compare serial intervals and the union of parallel work. +Never add a parent to its children or sum parallel branches as elapsed time. +For example, `configuration_skills` is inside `execution_configuration`, and +`skills.prepare` overlaps the broader `heartbeat.prepare_before_environment`. +Detail spans are capped at 16 per name per attempt; missing per-item children +do not mean later items did no work. + +Distinguish first streamed text, committed reply, externally delivered message, +and browser paint. A successful `response.transaction.commit` proves durable storage; +`response.publish` is enqueueing and `execution.realtime` is server dispatch. +Neither proves that the user saw the response. Choose the boundary the user asked +to improve, then measure the whole path to that boundary. + +For slow issue pages, combine `issue.read.` with `Server-Timing` +(`paperclip_issue`, `issue_`) and the browser's request waterfall and +`issue-detail:navigate→content-paint` measure. Concurrent reads overlap. Fast +API completion does not establish fast conversation reveal or rendering. + +A long parent span without children is an attribution gap. It does not prove +SQL, network, or model time. For database hypotheses, use +[fix-slow-queries](../fix-slow-queries/SKILL.md). For CPU hypotheses, use a bounded +authorized profile covering the actual slow phase; a sample taken while queued +cannot explain preparation. Low aggregate container CPU does not exclude a +blocked Node main thread. Keep profiled diagnostics separate from unprofiled +acceptance timings. + +## Add only the missing instrumentation + +Read applicable repository rules before editing. Reuse the existing timing seams: + +- `server/src/services/execution-preparation-timing.ts`: + `withRunExecutionTiming`, `createRunPreparationTiming`, `measureRunPhase`, and + `beginRunPhase`. Use the existing attempt; do not create a new root per detail. +- `server/src/services/native-runtime/native-run-trace.ts`: native lifecycle + hierarchy and links to the owning attempt. +- `server/src/services/issue-read-timing.ts`: issue-read phases and Server-Timing. +- `packages/adapter-utils/src/acpx-engine/startup-timing.ts`: runtime parent + context and closed span attributes. Adapter utilities receive the host context + rather than constructing an independent exporter. + +Extend the closed phase union when a meaningful boundary is missing. Wrap the +actual awaited operation, preserve parent context across async/provider boundaries, +and end spans on success and failure. Keep real wall times for spans recorded +after the operation. Use existing hashed IDs, closed enums, and allowed numeric +counts; do not add prompts, tool arguments/results, paths, raw identifiers, +credentials, or exception text. Do not enable all filesystem instrumentation to +explain one slow scan. + +Observability must not retry the operation, replace its return/error, change +authorization/locks, or break execution when the tracer fails. Preserve root +sampling and the explicit endpoint gate even if another library registers an +OTel provider. Update `doc/observability.md` when the trace contract changes. +Run focused tests for endpoint-off behavior, parentage/concurrent-run isolation, +failure handling, and the changed seam. Use real exported evidence to claim that +new spans appear in Honeycomb; a unit-test tracer alone does not prove ingestion. + +## Compare and hand back useful evidence + +Choose the largest actionable serial wait or unexplained gap, then reproduce it +at the smallest faithful layer. Compare matched runs: same harness/model, +workspace size, session warmth, concurrency, and relevant revision/configuration. +Report sample count and median/range when several runs are available. Preserve +failures; a failed fast run is not a latency improvement. + +Return the run/attempt and trace links, observed revision, end-to-end timing, +dominant non-overlapping phases, uncertainty, and next action or verified gain. +Label local, synthetic, and hosted evidence precisely: a dev-host reproduction +is not a local checkout run, and a merged fix does not prove hosted improvement. +If traces are unavailable, state the missing access or coverage and what the +run log/browser measurements actually establish. diff --git a/.agents/skills/honeycomb-tracing/agents/openai.yaml b/.agents/skills/honeycomb-tracing/agents/openai.yaml new file mode 100644 index 00000000000..db9a552222f --- /dev/null +++ b/.agents/skills/honeycomb-tracing/agents/openai.yaml @@ -0,0 +1,3 @@ +interface: + display_name: "Honeycomb Tracing" + short_description: "Trace Paperclip runs, requests, and full CI waterfalls" diff --git a/.agents/skills/honeycomb-tracing/references/ci.md b/.agents/skills/honeycomb-tracing/references/ci.md new file mode 100644 index 00000000000..93cee4965ea --- /dev/null +++ b/.agents/skills/honeycomb-tracing/references/ci.md @@ -0,0 +1,96 @@ +# Profile the full PR suite + +Use [doc/ci-profiling.md](../../../../doc/ci-profiling.md) as the maintained +operator runbook. This path exports recorded CI timing through Honeycomb's Events +API; it does not require application OTLP settings or an application run UUID. + +## Select one complete attempt + +1. Confirm the intended PR and head. Profiling is opt-in through `profile-ci`. + The PR author, original workflow actor, and rerun actor must be individual + users authorized by trusted master's `.github/**` CODEOWNERS rule. Applying + the label to an outside contributor's PR does not grant profiling access. +2. When asked to profile that PR, add the label. It starts a fresh + **CI profile · PR #number** run of the main `PR` workflow. The label forces + all main lanes even for stack-middle or release-notes-only changes. Further + pushes also profile while it remains present. +3. Record run ID, attempt, tested revision, and job inventory. Keep the label + and head unchanged until the separate **CI profile** exporter completes. + Use a fresh label-triggered run for an unambiguous whole-suite baseline. + Rerunning only failures does not rerun the successful jobs. +4. Open the Honeycomb waterfall from the run/exporter's Summary. The project + destination is team `paperclip`, environment `test`, dataset `paperclip-ci`. + Verify the displayed root and representative file spans; a generated URL + or green test workflow is not proof that export succeeded. +5. Remove the label after a one-off profile unless ongoing profiling was requested. + Re-adding it starts another complete attempt. Report any remaining label so + future pushes do not unexpectedly keep profiling. + +The main suite preserves existing AWS routing (`paperclip-public-pr-x64`, with +`paperclip-ci-routing-x64` for routing) and its authorized hosted fallback. +Inspect actual runner assignments if AWS placement is part of acceptance. The +separate exporter uses `ubuntu-latest` and is outside the measured suite. + +## Read and qualify the result + +The hierarchy is **workflow → job → step → test filename → tests**. Collapse +spans at the job depth for a shared wall-clock view; expand slow jobs and search +for a known `test.file` to prove filename-level detail arrived. + +- Root elapsed time runs from workflow creation to the last job completion; + reruns start at that attempt's start. Do not sum concurrent job durations. +- Use `ci.finishes_last`, `ci.execution_ms`, and `ci.start_offset_ms` to find + late-finishing work, then inspect setup, caches, compilation, builds, and tests. + A long job that ends early may not delay the suite. +- Waiting spans start at job creation. Dependency waits before job creation + appear as gaps; GitHub does not expose a separate ready timestamp, so those + gaps are not proven runner-queue delay or a computed dependency critical path. +- Vitest, Playwright, and Node supply file/test detail with different timestamp + semantics. GitHub steps have second precision; Node test spans use reporter + receipt time. Rust and standalone scripts retain job/step coverage. +- Scope is the main PR workflow. Separate Storybook, image, release, deployment, + and live-provider eval workflows are not included. + +Inspect `ci.profile_complete`, coverage warnings, expected versus reported +invocations, file/test counts, `ci.partial_rerun`, and `ci.carried_job_count`. +Failed/cancelled attempts can omit downstream work; carried jobs completed before +the new attempt are not fresh measurements. Compare the trace inventory with +GitHub's attempt-specific jobs and retained timing artifacts. + +Honeycomb's waterfall displays at most 32,000 spans, prioritizing those closest +to the root. Missing deep test rows can be a display limit rather than missing +ingest. Check **View events** and coverage evidence before claiming data loss. +See the [provider's large-trace limit](https://docs.honeycomb.io/troubleshoot/common-issues/data-in-honeycomb/#your-very-large-trace-displays-only-32000-spans). + +## Credentials and export failures + +`HONEYCOMB_CI_API_KEY` is a dedicated ingest-only **environment secret** in +GitHub environment `ci-profiling`, restricted to `master`. Only the trusted +default-branch exporter reads it. PR tests still execute the PR branch and never +receive the key. Do not move it to repository/org secrets, put a configuration +key in CI, execute downloaded artifacts, or switch trusted workflow refs to a PR +head for convenience. Ordinary CI must work with profiling disabled and no key. + +The exporter checks each batch response, paces uploads, and retries only +transiently rejected events. If it fails, inspect its logs and saved artifacts +before retrying: partial export may already have sent spans, and the Events API +does not promise deduplication on resubmission. Timing artifacts are retained for +14 days. Preserve incomplete-export evidence even if a later upload succeeds. + +For local reconstruction without uploading, use the existing exporter: + +```sh +node scripts/ci-profile/export.mjs --run-id=RUN_ID --attempt=1 --historical --dry-run --output=/tmp/ci-trace.json +``` + +Replace `RUN_ID` and attempt with the recorded values. This needs authorized +GitHub read access. Historical reconstruction can recover job/step timings, but +cannot invent file/test timings that were never recorded. It does not prove the +live label-triggered journey. A real historical upload is a separate authorized +action using the ingest key, not a necessary part of diagnosis. + +When changing CI tracing, use the runbook's focused exporter, routing, reporter, +and workflow checks. For a whole-suite tracing acceptance, return the exact +GitHub attempt and Honeycomb links, elapsed time, coverage and export status, +runner placement if relevant, and the late-finishing jobs/file timings that +justify the next optimization. diff --git a/.agents/skills/honeycomb-tracing/references/runtime.md b/.agents/skills/honeycomb-tracing/references/runtime.md new file mode 100644 index 00000000000..a39fa5866be --- /dev/null +++ b/.agents/skills/honeycomb-tracing/references/runtime.md @@ -0,0 +1,90 @@ +# Runtime tracing setup and missing spans + +Read this when configuring an authorized runtime or diagnosing absent spans. +The authoritative implementation is +[instrumentation.ts](../../../../server/src/instrumentation.ts); package versions +and the data contract live in +[Observability](../../../../doc/observability.md#enabling-tracing). + +## Enable the existing bootstrap + +1. Identify the actual process/image and its owning package. Inspect + `server/package.json` for the **exact** peer versions of the OTel SDK, + auto-instrumentations, resources, semantic conventions, and one exporter. + Install them in the runtime that will resolve them. Avoid copying old version + pins from sessions or adding every exporter to the application manifest. +2. Configure that process before startup. A direct US Honeycomb HTTP/protobuf + setup uses these values: + + | Setting | Value | + | --- | --- | + | `OTEL_EXPORTER_OTLP_ENDPOINT` | `https://api.honeycomb.io` | + | `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` | + | `OTEL_EXPORTER_OTLP_HEADERS` | `x-honeycomb-team=` | + | `OTEL_SERVICE_NAME` | `paperclip`, or the deliberately selected service identity | + + Inject the header through the existing secret manager/process environment; + do not paste a key into shell history or a tracked `.env`. Confirm region and + environment from the operator's configuration. An existing collector may be + the intended endpoint instead of direct Honeycomb ingestion. +3. For HTTP, supply the **base URL**; the exporter appends `/v1/traces`. Paperclip + intentionally constructs HTTP exporters without an explicit URL so they read + the standard environment variables. A signal-specific variable alone does + not activate Paperclip: the bootstrap gate checks the generic + `OTEL_EXPORTER_OTLP_ENDPOINT`. +4. For gRPC, use the appropriate TLS endpoint and matching exporter; the protocol + defaults to gRPC when unset. Unknown protocol strings warn and fall back to + gRPC. Do not accidentally pair HTTP configuration with that default. +5. Check the runtime's startup diagnostics. A missing or mismatched peer package + leaves the server healthy **without tracing**. `instrumentationReady` settles + before DB connections/server construction; shutdown awaits a bounded flush. + A healthy `/api/health` alone cannot validate export. +6. Generate a small representative operation within the authorized task, allow + the export batch to arrive, and open its actual Honeycomb spans. Verify + service, revision, timestamps, parentage, and the expected operation. Record + a trace link as proof; a bootstrap log line only proves initialization. + +Honeycomb's [OTLP configuration](https://docs.honeycomb.io/send-data/opentelemetry/#using-the-honeycomb-opentelemetry-endpoint) +documents regions, headers, and the additional dataset header required for +Classic. Confirm current provider requirements when changing setup. + +## Identify the serving version + +Paperclip resolves `service.version` from the build stamp +(`dist/build-info.json`), then runtime Git, then `OTEL_SERVICE_VERSION`, then +`unknown`. The build stamp wins over a stale environment variable. Docker builds +without `.git` use `PAPERCLIP_BUILD_COMMIT` to populate the stamp. + +Match the trace's version with actual runtime/deployment evidence. Record source +revision and observation time; an opaque desired release or current local HEAD +is not evidence that the running service includes a fix. + +## Troubleshoot in this order + +| Observation | Check | +| --- | --- | +| No results | Correct team/environment/dataset/service, exact run hash, absolute UTC range, retention, and query permissions | +| Only native/provider spans | Remove root-name restriction; inspect other trace IDs for the same run hash and controller version | +| App healthy, nothing exported | Actual process env, exact optional peers, protocol/exporter match, bootstrap diagnostic, root sampling, and authorized endpoint reachability | +| Trace ends before saved reply | Provider root excludes outer persistence; inspect `task.execution_attempt`, settlement spans, and run-log timestamps | +| Missing unfinished phase | Active spans may not export until they end; abrupt exit can lose them | +| Long manual span, no SQL children | Auto-instrumentation does not guarantee children; inspect real dependency timings rather than equating the whole span with database time | +| Query API denied, authentication works | Check query permission/plan; use the signed-in UI when available, without widening key privileges | +| Export/CSV contains only COUNT | It is an aggregate result. Inspect span details or the authorized raw-events view for event-level analysis | + +The server exports traces only through this bootstrap. `fs`, `dns`, and `net` +auto-instrumentations are disabled for volume. Local performance events can +still exist with OTLP off; their presence is not an exporter health check. + +When changing the bootstrap or correlation, start with the relevant tests: + +```sh +pnpm exec vitest run server/src/__tests__/instrumentation.test.ts +pnpm exec vitest run server/src/services/execution-preparation-timing.test.ts +pnpm exec vitest run server/src/services/native-runtime/native-run-trace-context.test.ts +pnpm exec vitest run ui/src/lib/honeycomb-run-link.test.ts +``` + +Select affected checks rather than running this entire list for a read-only +diagnosis. These establish local contracts; deployment and Honeycomb ingestion +need their own evidence.