DottaandPaperclip
6acec4266e
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its PR workflow runs dozens of parallel build and test jobs. > - Individual job durations do not show the elapsed cost of the whole suite. > - Maintainers need one shared timeline to find slow work and repeated setup. > - This pull request adds full-suite profiles for selected CI CODEOWNER PRs. > - Honeycomb shows each job, step, test file, and supported test on that timeline. > - The benefit is evidence for changes that reduce total CI time. ## Linked Issues or Issue Description **What existing behavior does this improve?** The main PR workflow has about 48 jobs. Maintainers need to compare their overlapping work and find which jobs delay completion. **Current behavior** GitHub shows job and step durations separately. Test files and tests are not joined to a full workflow trace. **Proposed behavior** A CI CODEOWNER adds `profile-ci` to a PR. The tested code stays on that PR branch. All main PR lanes run, including lanes normally skipped for stack position or release notes. A trusted post-run exporter sends one wall-clock trace to Honeycomb. **Reason and benefit** The trace supports decisions about which CI work to speed up or remove. Ordinary PRs do not export traces. The roadmap and related-PR search found no duplicate CI profiling work. ## What Changed - Add label selection and a full-suite override. Verify the PR author, original actor, and rerun actors against the trusted `.github/**` CODEOWNERS rule. - Record Vitest, Playwright, and Node test file/test timings in local artifacts. Keep ordinary reporter behavior and failing exit codes. - Add a trusted `workflow_run` exporter. Use actual GitHub timestamps for all jobs, setup, steps, waiting, and teardown. Mark missing test data and partial reruns. - Keep the ingest key out of PR jobs. Store it only in the `ci-profiling` environment, restricted to `master`. The exporter reads bounded JSON and never executes artifact code. - Record expected test invocations outside reporters so a missing invocation cannot appear complete. Preserve `node --test` on older PR checkouts. Flush large shard inventories before exit. - Add security, timing, routing, and real-reporter checks. Document operation and activation. Protect the exporter with CODEOWNERS. ## Verification - 509 focused workflow/reporter/shard checks passed. They include outside authors, outside rerun actors, bots, stale revisions, malicious ZIP paths, incomplete reports, and partial ingestion failures. - Final revision `3bf5d9017d` adds multi-file qualification and fixes short-file step parenting and duplicate Node file-wrapper events. Greptile reviewed this exact revision at 5/5 with no unresolved threads. [Main PR CI run](https://github.com/paperclipai/paperclip/actions/runs/38728040302): the initial attempt had 54 successful checks, 3 intentional skips, and one serialized-shard failure from an unchanged workspace-authorization test exceeding its 15-second timeout. The single failed-shard retry passed: the final result is 55 successful checks and 3 intentional skips. The same timeout reproduces locally with profiling disabled; both assertions pass with a diagnostic 60-second timeout override (about 23 seconds total). No source or CI timeout was changed. - A dry run against real GitHub attempt 2 of run `38677445871` produced exactly 3 executed jobs, 40 spans, and 510 seconds, excluding 45 carried-over jobs. The partial trace remains explicitly incomplete; this validation did not upload data. - `actionlint` passed for all four changed workflows. `git diff --check` passed. - Full Node 24 `pnpm -r typecheck` and `pnpm build` passed. The local full `pnpm test:run` was stopped after it reported a setup-token timeout and a distribution-plugin-catalog fixture failure. The 30 setup-token tests passed on focused rerun. The plugin fixture fails at `rmSync` of a directory symlink on this macOS/Node 24.11.0 host; the same failure reproduces with the unchanged base Vitest configuration and unchanged test/service files (10 other tests pass). The local full test suite is therefore not claimed green. - A real root-config Vitest probe passed 13 tests with profiling both off and on. Multi-file Vitest, Playwright, and Node probes each verify two distinct filename rows and four correctly parented tests, including repeated test names across files. Separate passing probes exercise the full exporter hierarchy `workflow → job → step → filename → tests`. Intentional failing probes preserve failing exit status. - Verified all 48 measured jobs in latest-head run `38728040302` used AWS: one `paperclip-ci-routing-x64` gate and 47 `paperclip-public-pr-x64` jobs. Profiling preserves AWS authorization and its GitHub-hosted fallback. The separate privileged exporter runs on `ubuntu-latest` outside the measured suite. - [Live Honeycomb baseline](https://ui.honeycomb.io/paperclip/environments/test/datasets/paperclip-ci/trace/oaPMzw8zGwG): 48 jobs, 731 spans, 479 seconds. It matches [GitHub run 38671274294](https://github.com/paperclipai/paperclip/actions/runs/38671274294). This historical trace has job/step data only; it is not proof of the future label-triggered test-reporting journey. - Verified the dedicated Honeycomb key is ingest-only with dataset creation disabled. Verified the GitHub environment allows only the `master` branch. No repository-level secret of that name exists. Both credential values were scanned absent from source and branch history. After merge, add `profile-ci` to a CI CODEOWNER PR, wait for the entire PR run and exporter, then follow the run-summary link. Collapse job children to compare parallel lanes; expand a job to inspect its steps and tests. ## Risks - Activation requires merging the trusted workflow definitions to `master`. This PR continues using the current trusted reusable workflows until then. - Selected runs produce many spans and artifacts. Oversized artifacts fail export instead of truncating data. Missing or cancelled reporters are marked incomplete. - GitHub timings have second precision. Node test times use reporter receipt time. Rust retains job/step timing but lacks per-test wall timestamps. - CI CODEOWNERS are individual users in the explicit `.github/**` rule. Unsupported team rules and identity lookup failures disable profiling. - An interrupted upload can leave partial data. Retrying uses the same span IDs, but the Events API does not promise deduplication. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, repository tools, and browser verification. The runtime does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
8.8 KiB
Profile the complete PR suite
Add the profile-ci label to a PR authored by a CI CODEOWNER. Adding the label starts a fresh PR workflow
run. Further pushes while the label remains also run with profiling enabled.
The run is named CI profile · PR #number. It runs all lanes of the main PR
suite, including middle-of-stack and release-notes-only PRs that normally skip
most lanes.
After the run finishes, open its Summary and follow Open the Honeycomb waterfall. The separate CI profile workflow exports the trace after all jobs finish. Its summary has the exact time range, job/span counts, and coverage warnings. Keep the label on the PR and its head unchanged until export finishes. Remove the label to stop profiling subsequent pushes. To profile the same revision again, remove and re-add the label or rerun all jobs.
The PR author, original workflow actor, and rerun actor must all be individual
users in the trusted master branch's .github/** CODEOWNERS rule. GitHub user
IDs are resolved from that rule; PR edits to CODEOWNERS cannot grant access.
A label on an outside contributor's PR does not enable profiling, even when a
maintainer applies it. Failed identity lookups disable profiling.
Ordinary PRs do not write timing artifacts or export traces. The exporter job is skipped for ordinary run names. Adding another label does not run the CI matrix.
Profiling preserves the existing trusted AWS runner routing. Eligible PR jobs
use paperclip-public-pr-x64; the routing gate uses paperclip-ci-routing-x64.
The existing fail-closed GitHub-hosted fallback still applies if AWS routing
cannot be authorized. Only the separate post-run Honeycomb exporter uses
ubuntu-latest directly; it is outside the measured PR suite.
Read the waterfall
The workflow root measures elapsed time from run creation to the last job's
completion. For reruns, it starts at that attempt's run_started_at instead of
the original creation time. Every job is a child on the same wall clock, with
its steps and available test details beneath it. All spans use a sample rate of
one. There is no sampling within the selected attempt. For a compact Gantt view,
open a job row's menu and choose Collapse spans at this depth. Expand an
individual job to inspect its steps and test files.
The expandable hierarchy is workflow → job → step → test filename → tests.
Vitest, Playwright, and Node file rows use repository-relative filenames, also
available in test.file. Separate files retain their own children even when
individual test names repeat across files.
- Compare the jobs that finish last (
ci.finishes_last) with other overlapping work. A long job that finishes early is not necessarily delaying CI. - Expand setup, dependency installation, cache restore, Rust compilation, builds, and tests to find repeated or slow work.
ci.execution_msmeasures a job's execution.ci.start_offset_msshows when it started relative to the whole run. Do not add parallel job durations to calculate elapsed time.- Waiting spans cover job creation to runner start. Dependency waits can occur before GitHub creates a job, so they appear as gaps from the workflow start. GitHub does not expose a separate ready timestamp; do not interpret every gap as runner queue time or claim this is a computed dependency critical path.
GitHub job/step timestamps have second precision. Vitest and Playwright record
test timestamps at the runner. Node's test reporter records completion receipt
time and duration (test.clock=reporter-receipt); buffered IPC delivery can shift
those small test spans. Vitest file spans include its test/hook execution and
carry collection/setup durations as fields. Playwright file spans cover the
first through last test attempt in that file. Rust and standalone script checks
retain their full job/step timings; stable libtest does not expose per-test wall
timestamps. The trace is CI timing data, not application tracing.
Scope includes all main PR lanes: policy, typecheck/release registry, general and serialized tests, Runner checks, Docker context checks, build, canary dry run, browser E2E, and aggregate checks. Separate Storybook, image, deployment, release, and live-provider eval workflows are outside this trace.
Failure, coverage, and reruns
The exporter uses GitHub's attempt-specific job API and paginates jobs and
artifacts. It includes failed and cancelled jobs and uploads available test
timings with always(). A force-cancelled or failed setup job may leave no
artifact. Coverage warnings and ci.profile_complete=false make missing
artifacts, interrupted reporters, rejected data, and absent test details visible.
Test launchers write an expected-invocation record before starting each Vitest,
Playwright, or Node process. Each reporter must write matching start/end records,
so a missing reporter cannot hide behind another successful invocation in the
same job. Compare ci.expected_invocations with ci.reported_invocations.
Failed or cancelled workflows are conservatively incomplete because a failed
command can prevent later tests or steps from running.
Each attempt gets a different trace ID. Rerunning failed jobs alone produces an
attempt trace for those jobs; it does not replay completed jobs. Reruns are marked
ci.partial_rerun=true conservatively. Use a fresh label-triggered run for an
unambiguous full-suite baseline. GitHub may carry earlier successful jobs into
the new attempt's job inventory. Jobs completed before the new attempt started
are excluded and counted in ci.carried_job_count.
If export fails, inspect CI profile and rerun that exporter after fixing the cause. A partial HTTP export may already have sent some spans; retries use the same IDs but the Events API does not promise deduplication. Timing artifacts remain available for 14 days. Invalid/oversized artifacts stop export instead of silently truncating the suite. No logs, environment dumps, error messages, stack traces, or credentials are included in the exported field allowlist.
Operator setup and activation
Use the existing Honeycomb paperclip team, test environment, and a separate
service dataset named paperclip-ci. Create GitHub environment ci-profiling with Selected branches and tags:
allow only the branch master, with no PR-ref or tag patterns. Store environment
secret HONEYCOMB_CI_API_KEY with a dedicated ingest-only key for the Honeycomb
test environment. Do not create a repository- or organization-level secret of
that name: the environment restriction prevents in-repository PR workflows from
requesting the key. Tests still execute the PR branch; only the separate exporter
runs from master. It needs event
ingest permission; dataset creation permission is needed only if the dataset
has not been created yet. Do not store the environment configuration key in CI.
Create the repository label profile-ci if it does not exist.
ci-profile.yml runs from the trusted default-branch revision. It checks out only
the exporter, installs no dependencies, and parses PR artifacts as bounded JSON
without extracting or executing their contents. The PR jobs never receive the
Honeycomb key. The exporter independently verifies CI CODEOWNER identities,
the current label, repository, and exact tested PR revision before reading
artifacts or sending data. Its credential is sent only in an HTTPS header to
api.honeycomb.io; redirects are rejected. Exported fields are allowlisted.
CODEOWNERS also covers scripts/ci-profile/**, so changes to the privileged
exporter require infrastructure-owner review.
The new trusted workflow and routing definitions must merge to master before
the label-triggered journey is active. A PR uses pr-trusted.yml@master and
ci-routing.yml@master; do not change those trust references just to test this
feature before merge.
Local qualification
node --test .github/scripts/tests/ci-profile.test.mjs .github/scripts/tests/ci-routing.test.mjs
node --test scripts/ci-profile/reporters.test.mjs
actionlint .github/workflows/{pr,pr-trusted,ci-routing,ci-profile}.yml
node scripts/ci-profile/export.mjs --run-id=RUN_ID --attempt=1 --historical --dry-run --output=/tmp/ci-trace.json
--historical is local-only and permits a deliberate export of an existing
unlabeled run. It has complete GitHub job/step timing but cannot reconstruct
test details that were never recorded. Omit --dry-run and supply
HONEYCOMB_CI_API_KEY to send the selected baseline to Honeycomb. It is labeled
as historical/incomplete test coverage; it is not proof of the label-triggered
workflow.
References: Honeycomb trace fields, Events API, GitHub workflow_run.