docs: teach readable trace waterfall design (#16798)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Operators use traces to find slow work and explain task execution.
> - The Honeycomb skill explains how to find traces and add spans.
> - It gives little guidance on how a trace should read when its
branches collapse.
> - This pull request adds semantic grouping and lifecycle ownership
guidance.
> - The benefit is a clear overview that preserves useful detail for
diagnosis.

## Linked Issues or Issue Description

Related work: Refs #16453, which added this skill. Refs #16419, which
added execution preparation and response spans.

**Issue type**
Missing documentation.

**Where is the issue?**
The Honeycomb tracing skill under `.agents/skills/honeycomb-tracing/`.

**What's wrong?**
A trace can contain accurate timings but remain hard to read when most
spans share one parent. Local tests can also miss incorrect parentage in
exported traces, especially on failure.

**Suggested fix**
Explain how to group operations by purpose, preserve real parent
contexts, and inspect both collapsed and expanded exported traces. Keep
the current Observability contract as the source for exact span names
and root boundaries.

## What Changed

- Add trace design to the skill description and task routing.
- Define readable phases, meaningful folds, and real parent contexts.
- Add a reference with staging examples, failure and concurrency
pitfalls, and checks for exported traces.
- Preserve useful measurements and documented links between separate
lifecycles.
- Update the skill's display description.

## Verification

- Passed the skill creator's `quick_validate.py` check.
- Checked local Markdown references and skill metadata.
- Passed `git diff --check` after rebase onto current master.
- All 55 current-head GitHub checks passed. This includes application
build, TypeScript and Rust checks, general and serialized tests, runner
checks, browser E2E shards, and canary packaging.
- Greptile scored commit `714560594b69fb08afe7eb9fff42afef4a38e49a` at
5/5. It found no actionable issues. There are no unresolved review
threads.
- Local skill validation, reference checks, metadata checks, and
whitespace checks passed. The full local application checks did not
finish: the cold native runner build was stopped after the full CI suite
passed. Full application verification was completed in CI.

## Risks

Low risk. This change updates skill instructions only. An agent could
apply an example tree too rigidly. The reference states that examples do
not replace the current Observability contract. It preserves documented
linked roots and scope limits for verification.

## Model Used

OpenAI `gpt-6.1-sol`, with `xhigh` reasoning. Codex used local tools to
inspect session evidence, edit Markdown and YAML, and run validation.
The host does not expose the context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (skill validation,
references, and metadata; application checks passed in CI)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>

.agents/skills/honeycomb-tracing/SKILL.md

@@ -2,7 +2,8 @@
 name: honeycomb-tracing
 description: >-
   Trace Paperclip runs, requests, and full CI suites in Honeycomb. Use to find
-  traces, diagnose latency, configure opt-in OpenTelemetry, or add missing spans.
+  traces, diagnose latency, design readable and foldable waterfalls, configure
+  opt-in OpenTelemetry, or add missing spans.
 ---
 
 # Honeycomb Tracing
@@ -19,6 +20,7 @@ replaying tasks, changing credentials, or deploying instrumentation.
 | Why is a task slow before, during, or after model execution? | Application workflow below; [runtime setup](references/runtime.md) if spans are absent |
 | Why does an issue page load slowly? | Request spans plus browser timing, described below |
 | Which CI jobs or test files delay the whole suite? | [CI profiling](references/ci.md) |
+| How should a trace be organized so a person can understand it? | Human-readable waterfall guidance below and [trace design](references/trace-design.md) |
 | Where should a new span go? | Instrumentation guidance below and [Observability contract](../../../doc/observability.md) |
 
 Repository-relative paths and commands assume the checkout root. The maintained
@@ -123,6 +125,32 @@ cannot explain preparation. Low aggregate container CPU does not exclude a
 blocked Node main thread. Keep profiled diagnostics separate from unprofiled
 acceptance timings.
 
+## Design a human-readable waterfall
+
+A collapsed trace should tell a person what the system is doing; expanding a
+phase should explain where its time went. Before adding or reorganizing spans,
+sketch the meaningful phases and compare them with the existing exported tree.
+For application runs, make preparation, runner startup, agent work, and settlement
+easy to distinguish. For CI, retain workflow/job/step/file/test ownership.
+
+- Group by purpose: file staging belongs under `stage.sync`, with workspace and
+  asset branches such as `stage.asset.home`; provider operations belong beneath
+  the work that caused them. Reuse the owning path's existing semantic groups.
+- Use real OpenTelemetry parent contexts. A dotted name or a `parentName`
+  attribute alone does not make a foldable hierarchy. A flat trace can need
+  better parentage even when it already has enough measurements.
+- Preserve useful detail beneath the groups. Improve folding before deleting
+  spans; account for any measurement moved to an attribute, event, or run log.
+- Make the hierarchy truthful on failure and during concurrent work. Cleanup
+  belongs to settlement even when startup fails; long-lived processes need an
+  owner that covers their lifetime. Preserve documented links between separate
+  attempts and provider lifecycles rather than forcing everything into one root.
+
+Read [trace design](references/trace-design.md) when designing or reviewing a
+hierarchy. It includes a staging example, lifecycle pitfalls, and acceptance
+checks for the actual exported waterfall. The current Observability contract
+owns exact span names and boundaries; historical examples are not a new schema.
+
 ## Add only the missing instrumentation
 
 Read applicable repository rules before editing. Reuse the existing timing seams:
@@ -152,6 +180,8 @@ OTel provider. Update `doc/observability.md` when the trace contract changes.
 Run focused tests for endpoint-off behavior, parentage/concurrent-run isolation,
 failure handling, and the changed seam. Use real exported evidence to claim that
 new spans appear in Honeycomb; a unit-test tracer alone does not prove ingestion.
+For hierarchy changes, also inspect collapsed and expanded exported views using
+the [trace-design acceptance checks](references/trace-design.md#verify-the-exported-waterfall).
 
 ## Compare and hand back useful evidence
 

.agents/skills/honeycomb-tracing/agents/openai.yaml

@@ -1,3 +1,3 @@
 interface:
   display_name: "Honeycomb Tracing"
-  short_description: "Trace Paperclip runs, requests, and full CI waterfalls"
+  short_description: "Design and inspect readable Paperclip trace waterfalls"

.agents/skills/honeycomb-tracing/references/trace-design.md

@@ -0,0 +1,112 @@
+# Human-readable trace design
+
+Read this when adding, reorganizing, or reviewing a waterfall's hierarchy.
+The goal is to understand the work at a glance, then expand the slow phase to
+find its cause. Follow the current
+[Observability contract](../../../../doc/observability.md) for exact names,
+roots, lifecycle boundaries, sampling, and allowed attributes.
+
+## Organize by the work being done
+
+Sketch a few meaningful phase groups before choosing detailed spans. Each fold
+should answer a useful question: preparing what, connecting to what, executing
+what, or persisting what? Prefer recognizable operations over a mirror of the
+source-file or helper-function hierarchy. More nesting is useful only when it
+adds meaning; do not target a fixed number of root children or depth levels.
+
+For example, sandbox staging has a natural structure:
+
+```text
+stage.sync
+├── stage.workspace
+│   ├── pack
+│   └── sandbox.daytona.transfer
+└── stage.asset.home
+    └── sandbox.daytona.transfer
+```
+
+This is an illustrative subtree, not a complete native-run schema. Both transfer
+spans keep the same operation name; their actual parents explain their purpose.
+Keep runner artifact preparation, sandbox acquisition, file syncing, daemon
+startup, preview resolution, WebSocket activation, agent work, and settlement
+under the closest meaningful phase that actually owns them. Retain useful leaf
+timings when they explain distinct waits or operations.
+
+Use stable operation names and the existing bounded attributes for dimensions.
+Do not put raw IDs, paths, commands, or attempt numbers into new span names.
+Avoid duplicate full-duration wrappers that add no distinct meaning. If a
+measurement is redundant, state where its information remains available before
+removing it. An instantaneous milestone can be an allowed event on its semantic
+parent; it need not become another zero-duration lane. Preserve existing event
+and attribute contracts when changing representation.
+
+## Make parentage match execution
+
+- Open lifecycle parents when work begins, before their children are emitted.
+  End them after the work they own has settled. For delayed span recording,
+  retain the real operation timestamps and captured parent context; do not move
+  timings to make the drawing look tidy.
+- Carry actual span context through awaited operations and asynchronous
+  callbacks using the existing trusted runtime/provider seams. A name-based
+  parent hint cannot distinguish repeated or concurrent operations. Never use
+  a process-global current parent that lets one run adopt another run's spans.
+- Capture the owning run and phase when dispatching deferred work. Warm-session
+  reuse must not attach new tools or callbacks to a previous turn. Use the
+  documented fallback when no matching live scope remains.
+- Exercise error and cancellation exits. If startup fails, open the appropriate
+  cleanup/settlement scope before checkpoint persistence or teardown. Those
+  operations must not inherit the last startup scope merely because the normal
+  terminal event never arrived.
+- Give a long-lived daemon or streamed process an owner that covers its actual
+  lifetime. A process launched during startup may overlap the agent turn; its
+  launch step is not necessarily its lifetime parent. Keep real overlap visible.
+- Preserve the current distinction between server execution attempts, native
+  provider lifecycles, and trigger requests. Linked roots can be intentional
+  because of queueing, retries, process lifetimes, or root sampling. Do not adopt
+  an HTTP request as the run parent by accident, backdate a later parent to cover
+  historical work, or restore the old single-`task.run` design from a session.
+
+## Verify the exported waterfall
+
+For an instrumentation change, use representative real operations within the
+authorized scope and open their actual Honeycomb traces. Include success and
+the relevant failure/cleanup path; cover warm resume, concurrency, or retry when
+the changed ownership crosses those boundaries. This is not a requirement to
+run every harness and deployment for every span edit. If live evidence is
+unavailable, report that gap and the narrower evidence collected.
+
+Inspect both views:
+
+- **Collapsed:** the phases explain what happened and where elapsed time went
+  without reading dozens of sibling leaves. Separate linked traces are identified
+  and correlated; unexplained gaps remain visible.
+- **Expanded:** a slow phase leads to useful operations. Check actual parent IDs,
+  representative cross-process children, repeated/concurrent operations, error
+  status, and start/end times. Cleanup belongs to the right phase. Investigate
+  missing parents or children outside an owner's lifetime, including clock
+  skew, before treating the tree as correct.
+
+Compare the baseline and changed trace's phase grouping, root children, depth,
+and retained measurements. These are diagnostic clues, not numeric targets.
+Do not add parent and child durations or sum overlapping work. Do not relabel an
+unattributed parent interval as database, network, or model time without evidence.
+
+Return the run/attempt, deployed revision, trace links, what now folds correctly,
+and any remaining attribution gaps. Report export/structure quality separately
+from task outcome: a failed task can yield a useful trace, but successful export
+does not make that task successful. Focused tests establish local contracts;
+only inspected exported evidence supports a claim about the Honeycomb waterfall.
+
+## What the August 26 session taught us
+
+The user's request was to retain the valuable detail while improving semantic
+steps, parentage, logical grouping, and folding. The old native trace already
+had many measurements; attaching them all to its root made it hard to read.
+
+The implementation hand-off reported 75 focused tests passing and said the next
+run should produce the new waterfall. The user then had to request real runs.
+Those runs revealed an unintended HTTP parent and failure-time checkpoint
+persistence under startup. Export succeeded even though the Daytona task failed.
+Carry forward the design preferences and the need to inspect real failure
+parentage. The session's schema version, root name, counts, and particular task
+failure are historical details, not permanent requirements.