Files
PaperClipAI/scripts/ci-profile/node-reporter.mjs
T
DottaandPaperclip 6acec4266e feat(ci): profile full CODEOWNER PR suites in Honeycomb (#16378)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its PR workflow runs dozens of parallel build and test jobs.
> - Individual job durations do not show the elapsed cost of the whole
suite.
> - Maintainers need one shared timeline to find slow work and repeated
setup.
> - This pull request adds full-suite profiles for selected CI CODEOWNER
PRs.
> - Honeycomb shows each job, step, test file, and supported test on
that timeline.
> - The benefit is evidence for changes that reduce total CI time.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The main PR workflow has about 48 jobs. Maintainers need to compare
their overlapping work and find which jobs delay completion.

**Current behavior**

GitHub shows job and step durations separately. Test files and tests are
not joined to a full workflow trace.

**Proposed behavior**

A CI CODEOWNER adds `profile-ci` to a PR. The tested code stays on that
PR branch. All main PR lanes run, including lanes normally skipped for
stack position or release notes. A trusted post-run exporter sends one
wall-clock trace to Honeycomb.

**Reason and benefit**

The trace supports decisions about which CI work to speed up or remove.
Ordinary PRs do not export traces. The roadmap and related-PR search
found no duplicate CI profiling work.

## What Changed

- Add label selection and a full-suite override. Verify the PR author,
original actor, and rerun actors against the trusted `.github/**`
CODEOWNERS rule.
- Record Vitest, Playwright, and Node test file/test timings in local
artifacts. Keep ordinary reporter behavior and failing exit codes.
- Add a trusted `workflow_run` exporter. Use actual GitHub timestamps
for all jobs, setup, steps, waiting, and teardown. Mark missing test
data and partial reruns.
- Keep the ingest key out of PR jobs. Store it only in the
`ci-profiling` environment, restricted to `master`. The exporter reads
bounded JSON and never executes artifact code.
- Record expected test invocations outside reporters so a missing
invocation cannot appear complete. Preserve `node --test` on older PR
checkouts. Flush large shard inventories before exit.
- Add security, timing, routing, and real-reporter checks. Document
operation and activation. Protect the exporter with CODEOWNERS.

## Verification

- 509 focused workflow/reporter/shard checks passed. They include
outside authors, outside rerun actors, bots, stale revisions, malicious
ZIP paths, incomplete reports, and partial ingestion failures.
- Final revision `3bf5d9017d` adds multi-file qualification and fixes
short-file step parenting and duplicate Node file-wrapper events.
Greptile reviewed this exact revision at 5/5 with no unresolved threads.
[Main PR CI
run](https://github.com/paperclipai/paperclip/actions/runs/38728040302):
the initial attempt had 54 successful checks, 3 intentional skips, and
one serialized-shard failure from an unchanged workspace-authorization
test exceeding its 15-second timeout. The single failed-shard retry
passed: the final result is 55 successful checks and 3 intentional
skips. The same timeout reproduces locally with profiling disabled; both
assertions pass with a diagnostic 60-second timeout override (about 23
seconds total). No source or CI timeout was changed.
- A dry run against real GitHub attempt 2 of run `38677445871` produced
exactly 3 executed jobs, 40 spans, and 510 seconds, excluding 45
carried-over jobs. The partial trace remains explicitly incomplete; this
validation did not upload data.
- `actionlint` passed for all four changed workflows. `git diff --check`
passed.
- Full Node 24 `pnpm -r typecheck` and `pnpm build` passed. The local
full `pnpm test:run` was stopped after it reported a setup-token timeout
and a distribution-plugin-catalog fixture failure. The 30 setup-token
tests passed on focused rerun. The plugin fixture fails at `rmSync` of a
directory symlink on this macOS/Node 24.11.0 host; the same failure
reproduces with the unchanged base Vitest configuration and unchanged
test/service files (10 other tests pass). The local full test suite is
therefore not claimed green.
- A real root-config Vitest probe passed 13 tests with profiling both
off and on. Multi-file Vitest, Playwright, and Node probes each verify
two distinct filename rows and four correctly parented tests, including
repeated test names across files. Separate passing probes exercise the
full exporter hierarchy `workflow → job → step → filename → tests`.
Intentional failing probes preserve failing exit status.
- Verified all 48 measured jobs in latest-head run `38728040302` used
AWS: one `paperclip-ci-routing-x64` gate and 47
`paperclip-public-pr-x64` jobs. Profiling preserves AWS authorization
and its GitHub-hosted fallback. The separate privileged exporter runs on
`ubuntu-latest` outside the measured suite.
- [Live Honeycomb
baseline](https://ui.honeycomb.io/paperclip/environments/test/datasets/paperclip-ci/trace/oaPMzw8zGwG):
48 jobs, 731 spans, 479 seconds. It matches [GitHub run
38671274294](https://github.com/paperclipai/paperclip/actions/runs/38671274294).
This historical trace has job/step data only; it is not proof of the
future label-triggered test-reporting journey.
- Verified the dedicated Honeycomb key is ingest-only with dataset
creation disabled. Verified the GitHub environment allows only the
`master` branch. No repository-level secret of that name exists. Both
credential values were scanned absent from source and branch history.

After merge, add `profile-ci` to a CI CODEOWNER PR, wait for the entire
PR run and exporter, then follow the run-summary link. Collapse job
children to compare parallel lanes; expand a job to inspect its steps
and tests.

## Risks

- Activation requires merging the trusted workflow definitions to
`master`. This PR continues using the current trusted reusable workflows
until then.
- Selected runs produce many spans and artifacts. Oversized artifacts
fail export instead of truncating data. Missing or cancelled reporters
are marked incomplete.
- GitHub timings have second precision. Node test times use reporter
receipt time. Rust retains job/step timing but lacks per-test wall
timestamps.
- CI CODEOWNERS are individual users in the explicit `.github/**` rule.
Unsupported team rules and identity lookup failures disable profiling.
- An interrupted upload can leave partial data. Retrying uses the same
span IDs, but the Events API does not promise deduplication.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, code execution, repository
tools, and browser verification. The runtime does not expose the exact
serving model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-15 06:59:50 -05:00

42 lines
1.7 KiB
JavaScript

import path from "node:path";
import { recorder, relativeFile } from "./record.mjs";
export default async function* reporter(source) {
const record = recorder("node");
record({ kind: "start" });
let index = 0;
const files = new Map();
for await (const event of source) {
// complete is emitted as execution finishes; pass/fail are buffered in
// declaration order and cannot be used to place parallel tests in time.
if (event.type !== "test:complete") continue;
const data = event.data;
const duration = data.details?.duration_ms;
if (!data.file || !Number.isFinite(duration)) continue;
const file = relativeFile(data.file);
const end = Date.now();
const start = end - duration;
const previous = files.get(file);
files.set(file, {
start: Math.min(previous?.start ?? start, start),
end: Math.max(previous?.end ?? end, end),
failed: previous?.failed || (!data.skip && !data.details.passed),
});
// File completion events may use the CLI's relative argument as their
// name while data.file is absolute. They are containers, not test cases.
if (path.resolve(data.name) === path.resolve(data.file) || data.details.type === "suite") continue;
record({
kind: "test", parent: file,
id: String(index++), name: data.name, file, start, duration,
status: data.skip ? "skipped" : data.details.passed ? "passed" : "failed",
clock: "reporter-receipt",
});
}
for (const [file, timing] of files) {
record({ kind: "file", id: file, name: file, file,
start: timing.start, duration: timing.end - timing.start,
status: timing.failed ? "failed" : "passed", clock: "reporter-receipt" });
}
record({ kind: "end" });
}