mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 00:05:49 +02:00
DottaandPaperclip
6acec4266e
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its PR workflow runs dozens of parallel build and test jobs. > - Individual job durations do not show the elapsed cost of the whole suite. > - Maintainers need one shared timeline to find slow work and repeated setup. > - This pull request adds full-suite profiles for selected CI CODEOWNER PRs. > - Honeycomb shows each job, step, test file, and supported test on that timeline. > - The benefit is evidence for changes that reduce total CI time. ## Linked Issues or Issue Description **What existing behavior does this improve?** The main PR workflow has about 48 jobs. Maintainers need to compare their overlapping work and find which jobs delay completion. **Current behavior** GitHub shows job and step durations separately. Test files and tests are not joined to a full workflow trace. **Proposed behavior** A CI CODEOWNER adds `profile-ci` to a PR. The tested code stays on that PR branch. All main PR lanes run, including lanes normally skipped for stack position or release notes. A trusted post-run exporter sends one wall-clock trace to Honeycomb. **Reason and benefit** The trace supports decisions about which CI work to speed up or remove. Ordinary PRs do not export traces. The roadmap and related-PR search found no duplicate CI profiling work. ## What Changed - Add label selection and a full-suite override. Verify the PR author, original actor, and rerun actors against the trusted `.github/**` CODEOWNERS rule. - Record Vitest, Playwright, and Node test file/test timings in local artifacts. Keep ordinary reporter behavior and failing exit codes. - Add a trusted `workflow_run` exporter. Use actual GitHub timestamps for all jobs, setup, steps, waiting, and teardown. Mark missing test data and partial reruns. - Keep the ingest key out of PR jobs. Store it only in the `ci-profiling` environment, restricted to `master`. The exporter reads bounded JSON and never executes artifact code. - Record expected test invocations outside reporters so a missing invocation cannot appear complete. Preserve `node --test` on older PR checkouts. Flush large shard inventories before exit. - Add security, timing, routing, and real-reporter checks. Document operation and activation. Protect the exporter with CODEOWNERS. ## Verification - 509 focused workflow/reporter/shard checks passed. They include outside authors, outside rerun actors, bots, stale revisions, malicious ZIP paths, incomplete reports, and partial ingestion failures. - Final revision `3bf5d9017d` adds multi-file qualification and fixes short-file step parenting and duplicate Node file-wrapper events. Greptile reviewed this exact revision at 5/5 with no unresolved threads. [Main PR CI run](https://github.com/paperclipai/paperclip/actions/runs/38728040302): the initial attempt had 54 successful checks, 3 intentional skips, and one serialized-shard failure from an unchanged workspace-authorization test exceeding its 15-second timeout. The single failed-shard retry passed: the final result is 55 successful checks and 3 intentional skips. The same timeout reproduces locally with profiling disabled; both assertions pass with a diagnostic 60-second timeout override (about 23 seconds total). No source or CI timeout was changed. - A dry run against real GitHub attempt 2 of run `38677445871` produced exactly 3 executed jobs, 40 spans, and 510 seconds, excluding 45 carried-over jobs. The partial trace remains explicitly incomplete; this validation did not upload data. - `actionlint` passed for all four changed workflows. `git diff --check` passed. - Full Node 24 `pnpm -r typecheck` and `pnpm build` passed. The local full `pnpm test:run` was stopped after it reported a setup-token timeout and a distribution-plugin-catalog fixture failure. The 30 setup-token tests passed on focused rerun. The plugin fixture fails at `rmSync` of a directory symlink on this macOS/Node 24.11.0 host; the same failure reproduces with the unchanged base Vitest configuration and unchanged test/service files (10 other tests pass). The local full test suite is therefore not claimed green. - A real root-config Vitest probe passed 13 tests with profiling both off and on. Multi-file Vitest, Playwright, and Node probes each verify two distinct filename rows and four correctly parented tests, including repeated test names across files. Separate passing probes exercise the full exporter hierarchy `workflow → job → step → filename → tests`. Intentional failing probes preserve failing exit status. - Verified all 48 measured jobs in latest-head run `38728040302` used AWS: one `paperclip-ci-routing-x64` gate and 47 `paperclip-public-pr-x64` jobs. Profiling preserves AWS authorization and its GitHub-hosted fallback. The separate privileged exporter runs on `ubuntu-latest` outside the measured suite. - [Live Honeycomb baseline](https://ui.honeycomb.io/paperclip/environments/test/datasets/paperclip-ci/trace/oaPMzw8zGwG): 48 jobs, 731 spans, 479 seconds. It matches [GitHub run 38671274294](https://github.com/paperclipai/paperclip/actions/runs/38671274294). This historical trace has job/step data only; it is not proof of the future label-triggered test-reporting journey. - Verified the dedicated Honeycomb key is ingest-only with dataset creation disabled. Verified the GitHub environment allows only the `master` branch. No repository-level secret of that name exists. Both credential values were scanned absent from source and branch history. After merge, add `profile-ci` to a CI CODEOWNER PR, wait for the entire PR run and exporter, then follow the run-summary link. Collapse job children to compare parallel lanes; expand a job to inspect its steps and tests. ## Risks - Activation requires merging the trusted workflow definitions to `master`. This PR continues using the current trusted reusable workflows until then. - Selected runs produce many spans and artifacts. Oversized artifacts fail export instead of truncating data. Missing or cancelled reporters are marked incomplete. - GitHub timings have second precision. Node test times use reporter receipt time. Rust retains job/step timing but lacks per-test wall timestamps. - CI CODEOWNERS are individual users in the explicit `.github/**` rule. Unsupported team rules and identity lookup failures disable profiling. - An interrupted upload can leave partial data. Retrying uses the same span IDs, but the Events API does not promise deduplication. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, repository tools, and browser verification. The runtime does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
148 lines
8.8 KiB
Markdown
148 lines
8.8 KiB
Markdown
# Profile the complete PR suite
|
|
|
|
Add the `profile-ci` label to a PR authored by a CI CODEOWNER. Adding the label starts a fresh `PR` workflow
|
|
run. Further pushes while the label remains also run with profiling enabled.
|
|
The run is named **CI profile · PR #number**. It runs all lanes of the main PR
|
|
suite, including middle-of-stack and release-notes-only PRs that normally skip
|
|
most lanes.
|
|
|
|
After the run finishes, open its Summary and follow **Open the Honeycomb
|
|
waterfall**. The separate **CI profile** workflow exports the trace after all
|
|
jobs finish. Its summary has the exact time range, job/span counts, and coverage
|
|
warnings. Keep the label on the PR and its head unchanged until export finishes.
|
|
Remove the label to stop profiling subsequent pushes. To profile the same
|
|
revision again, remove and re-add the label or rerun all jobs.
|
|
|
|
The PR author, original workflow actor, and rerun actor must all be individual
|
|
users in the trusted `master` branch's `.github/**` CODEOWNERS rule. GitHub user
|
|
IDs are resolved from that rule; PR edits to CODEOWNERS cannot grant access.
|
|
A label on an outside contributor's PR does not enable profiling, even when a
|
|
maintainer applies it. Failed identity lookups disable profiling.
|
|
|
|
Ordinary PRs do not write timing artifacts or export traces. The exporter job is
|
|
skipped for ordinary run names. Adding another label does not run the CI matrix.
|
|
|
|
Profiling preserves the existing trusted AWS runner routing. Eligible PR jobs
|
|
use `paperclip-public-pr-x64`; the routing gate uses `paperclip-ci-routing-x64`.
|
|
The existing fail-closed GitHub-hosted fallback still applies if AWS routing
|
|
cannot be authorized. Only the separate post-run Honeycomb exporter uses
|
|
`ubuntu-latest` directly; it is outside the measured PR suite.
|
|
|
|
## Read the waterfall
|
|
|
|
The workflow root measures elapsed time from run creation to the last job's
|
|
completion. For reruns, it starts at that attempt's `run_started_at` instead of
|
|
the original creation time. Every job is a child on the same wall clock, with
|
|
its steps and available test details beneath it. All spans use a sample rate of
|
|
one. There is no sampling within the selected attempt. For a compact Gantt view,
|
|
open a job row's menu and choose **Collapse spans at this depth**. Expand an
|
|
individual job to inspect its steps and test files.
|
|
|
|
The expandable hierarchy is **workflow → job → step → test filename → tests**.
|
|
Vitest, Playwright, and Node file rows use repository-relative filenames, also
|
|
available in `test.file`. Separate files retain their own children even when
|
|
individual test names repeat across files.
|
|
|
|
- Compare the jobs that finish last (`ci.finishes_last`) with other overlapping
|
|
work. A long job that finishes early is not necessarily delaying CI.
|
|
- Expand setup, dependency installation, cache restore, Rust compilation, builds,
|
|
and tests to find repeated or slow work.
|
|
- `ci.execution_ms` measures a job's execution. `ci.start_offset_ms` shows when
|
|
it started relative to the whole run. Do not add parallel job durations to
|
|
calculate elapsed time.
|
|
- Waiting spans cover job creation to runner start. Dependency waits can occur
|
|
before GitHub creates a job, so they appear as gaps from the workflow start.
|
|
GitHub does not expose a separate ready timestamp; do not interpret every gap
|
|
as runner queue time or claim this is a computed dependency critical path.
|
|
|
|
GitHub job/step timestamps have second precision. Vitest and Playwright record
|
|
test timestamps at the runner. Node's test reporter records completion receipt
|
|
time and duration (`test.clock=reporter-receipt`); buffered IPC delivery can shift
|
|
those small test spans. Vitest file spans include its test/hook execution and
|
|
carry collection/setup durations as fields. Playwright file spans cover the
|
|
first through last test attempt in that file. Rust and standalone script checks
|
|
retain their full job/step timings; stable libtest does not expose per-test wall
|
|
timestamps. The trace is CI timing data, not application tracing.
|
|
|
|
Scope includes all main PR lanes: policy, typecheck/release registry, general and
|
|
serialized tests, Runner checks, Docker context checks, build, canary dry run,
|
|
browser E2E, and aggregate checks. Separate Storybook, image, deployment, release,
|
|
and live-provider eval workflows are outside this trace.
|
|
|
|
## Failure, coverage, and reruns
|
|
|
|
The exporter uses GitHub's attempt-specific job API and paginates jobs and
|
|
artifacts. It includes failed and cancelled jobs and uploads available test
|
|
timings with `always()`. A force-cancelled or failed setup job may leave no
|
|
artifact. Coverage warnings and `ci.profile_complete=false` make missing
|
|
artifacts, interrupted reporters, rejected data, and absent test details visible.
|
|
Test launchers write an expected-invocation record before starting each Vitest,
|
|
Playwright, or Node process. Each reporter must write matching start/end records,
|
|
so a missing reporter cannot hide behind another successful invocation in the
|
|
same job. Compare `ci.expected_invocations` with `ci.reported_invocations`.
|
|
Failed or cancelled workflows are conservatively incomplete because a failed
|
|
command can prevent later tests or steps from running.
|
|
|
|
Each attempt gets a different trace ID. Rerunning failed jobs alone produces an
|
|
attempt trace for those jobs; it does not replay completed jobs. Reruns are marked
|
|
`ci.partial_rerun=true` conservatively. Use a fresh label-triggered run for an
|
|
unambiguous full-suite baseline. GitHub may carry earlier successful jobs into
|
|
the new attempt's job inventory. Jobs completed before the new attempt started
|
|
are excluded and counted in `ci.carried_job_count`.
|
|
|
|
If export fails, inspect **CI profile** and rerun that exporter after fixing the
|
|
cause. A partial HTTP export may already have sent some spans; retries use the
|
|
same IDs but the Events API does not promise deduplication. Timing artifacts
|
|
remain available for 14 days. Invalid/oversized artifacts stop export instead of
|
|
silently truncating the suite. No logs, environment dumps, error messages, stack
|
|
traces, or credentials are included in the exported field allowlist.
|
|
|
|
## Operator setup and activation
|
|
|
|
Use the existing Honeycomb `paperclip` team, `test` environment, and a separate
|
|
service dataset named `paperclip-ci`. Create GitHub environment `ci-profiling` with **Selected branches and tags**:
|
|
allow only the branch `master`, with no PR-ref or tag patterns. Store environment
|
|
secret `HONEYCOMB_CI_API_KEY` with a dedicated ingest-only key for the Honeycomb
|
|
`test` environment. Do not create a repository- or organization-level secret of
|
|
that name: the environment restriction prevents in-repository PR workflows from
|
|
requesting the key. Tests still execute the PR branch; only the separate exporter
|
|
runs from `master`. It needs event
|
|
ingest permission; dataset creation permission is needed only if the dataset
|
|
has not been created yet. Do not store the environment configuration key in CI.
|
|
Create the repository label `profile-ci` if it does not exist.
|
|
|
|
`ci-profile.yml` runs from the trusted default-branch revision. It checks out only
|
|
the exporter, installs no dependencies, and parses PR artifacts as bounded JSON
|
|
without extracting or executing their contents. The PR jobs never receive the
|
|
Honeycomb key. The exporter independently verifies CI CODEOWNER identities,
|
|
the current label, repository, and exact tested PR revision before reading
|
|
artifacts or sending data. Its credential is sent only in an HTTPS header to
|
|
`api.honeycomb.io`; redirects are rejected. Exported fields are allowlisted.
|
|
`CODEOWNERS` also covers `scripts/ci-profile/**`, so changes to the privileged
|
|
exporter require infrastructure-owner review.
|
|
|
|
The new trusted workflow and routing definitions must merge to master before
|
|
the label-triggered journey is active. A PR uses `pr-trusted.yml@master` and
|
|
`ci-routing.yml@master`; do not change those trust references just to test this
|
|
feature before merge.
|
|
|
|
## Local qualification
|
|
|
|
```sh
|
|
node --test .github/scripts/tests/ci-profile.test.mjs .github/scripts/tests/ci-routing.test.mjs
|
|
node --test scripts/ci-profile/reporters.test.mjs
|
|
actionlint .github/workflows/{pr,pr-trusted,ci-routing,ci-profile}.yml
|
|
node scripts/ci-profile/export.mjs --run-id=RUN_ID --attempt=1 --historical --dry-run --output=/tmp/ci-trace.json
|
|
```
|
|
|
|
`--historical` is local-only and permits a deliberate export of an existing
|
|
unlabeled run. It has complete GitHub job/step timing but cannot reconstruct
|
|
test details that were never recorded. Omit `--dry-run` and supply
|
|
`HONEYCOMB_CI_API_KEY` to send the selected baseline to Honeycomb. It is labeled
|
|
as historical/incomplete test coverage; it is not proof of the label-triggered
|
|
workflow.
|
|
|
|
References: [Honeycomb trace fields](https://docs.honeycomb.io/get-started/best-practices/organizing-data/),
|
|
[Events API](https://docs.honeycomb.io/api/events/create-events),
|
|
[GitHub workflow_run](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#workflow_run).
|