feat(ci): send one in ten CODEOWNER PR runs and their outside checks to Honeycomb (#21192)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Contributors need fast, predictable CI feedback.
> - PR #16378 added a Honeycomb trace of the main PR workflow, but only
for PRs with the `profile-ci` label.
> - Ordinary CI time was therefore never recorded. All of the last 300
`CI profile` runs were skipped.
> - Reviews and scans outside the `PR` workflow, such as Greptile, were
not in the trace.
> - This pull request exports one CODEOWNER PR run in ten without a
label, and adds outside checks to every trace.
> - The benefit is a continuous record of total CI time and of which job
or check was slowest. The Honeycomb key stays out of reach of outside
contributors.

## Linked Issues or Issue Description

Refs #16378

**What existing behavior does this improve?**

The Honeycomb CI trace that the `CI profile` workflow exports.

**Current behavior**

- The exporter runs only for a PR with the `profile-ci` label. The label
also forces the full suite.
- The trace has the jobs of the `PR` workflow only. Greptile, security
scans, commit statuses and other workflows are absent.
- The only total is the `PR` workflow time.

**Proposed behavior**

- One `PR` run in ten is exported without a label. The run number must
end in 0. The trace has every job and step. Each span has a sample rate
of 10.
- Each trace has one span for each outside check and each other workflow
on the head commit.
- The root span has `ci.workflow_ms`, `ci.all_checks_ms`,
`ci.last_to_finish` and `ci.slowest_check`.

**Reason and benefit**

Maintainers can see how long CI takes and what is slow, for example a
long Greptile review, without a label on each PR. Sampling keeps the
volume near 1.7 million events each month. Export of every run is near
17 million.

## What Changed

- `scripts/ci-profile/authorize.mjs`: add `exportMode` (label or
one-in-ten sample) and `associatedOwnerPr` (the CODEOWNER checks without
the label).
- `scripts/ci-profile/trace.mjs`: add `normalizeChecks` and spans for
outside checks. Add the sample rate and the new root fields. Sampled
runs do not warn about absent test artifacts.
- `scripts/ci-profile/export.mjs`: select the mode. Read check runs,
commit statuses and workflow runs for the head commit. Wait a maximum of
15 minutes for outside apps that are not complete. Sampled runs read no
artifacts.
- `.github/workflows/ci-profile.yml`: no new job. The job condition now
requires a branch of this repository, an actor and rerun actor on the
`AWS_CI_ROUTING_USER_IDS` roster, and a labelled or sampled run. Add
`checks: read` and `statuses: read`. Increase the timeout to 30 minutes
for the bounded wait.
- `doc/ci-profiling.md` and `doc/DEVELOPING.md`: describe sampled runs,
outside checks, the totals and the job condition.

## Verification

- `node --test .github/scripts/tests/ci-profile.test.mjs
.github/scripts/tests/ci-routing.test.mjs`: 163 pass, 0 fail.
- `actionlint .github/workflows/ci-profile.yml`: no findings.
- Dry runs against real runs with the real CODEOWNERS check (`node
scripts/ci-profile/export.mjs --run-id=ID --attempt=1 --dry-run`):
- A sampled CODEOWNER run (run number 64780): 1,144 spans, sample rate
10, 5 outside checks and 1 other workflow.
- A sampled run from an outside contributor (run number 64800): "No
CODEOWNERS-authorized PR at this revision; no data exported."
- A CODEOWNER run that is not in the sample: "This attempt was not
selected for profiling; no data exported."
- New tests cover: the sample rule, fork and outsider PRs, a bot rerun
actor, a stale head, outside-check spans and totals, the bounded wait,
and the job condition.
- Not run locally: `scripts/ci-profile/reporters.test.mjs`. It needs
installed dependencies, and this change does not touch the reporters. CI
runs it.
- No data was sent to Honeycomb before merge. The workflow runs from
`master`, so the first real export occurs after merge.

## Risks

- The label path now also needs the `AWS_CI_ROUTING_USER_IDS` roster. If
that variable is empty, nothing is exported. All four CI CODEOWNERS are
on the roster today.
- The roster is only a first filter. The exporter still checks the PR
author and both actors against CODEOWNERS on `master` before it reads or
sends data. The key stays in the `ci-profiling` environment, which
permits only `master`.
- The export job can now run for a maximum of 30 minutes on a
GitHub-hosted runner while it waits for an outside check.
- GitHub gives no completion time for a workflow run. The span for
another workflow ends at the last update of that run.
- Honeycomb volume increases by about 1.7 million events each month.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code, with
tool use, code execution and repository inspection.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Fable 5.2 authored and GitHub committed 2026-11-12 16:57:23 -06:00
1 parent 0904a65d82
commit 9a9cd20d91
7 files changed
+415 -38

No files matched your search

+20 -3
View File
@@ -41,12 +41,29 @@ export function authorizedActors(run, ownerIds) {
&& allowed(run.actor) && allowed(run.triggering_actor);
}
export function associatedProfilePr(run, pulls, ownerIds) {
// The PR at this exact revision, when its author, the run's actor and the
// rerun actor are all CI CODEOWNERS.
export function associatedOwnerPr(run, pulls, ownerIds, accept = () => true) {
if (!authorizedActors(run, ownerIds)) return undefined;
return pulls.find((pr) => pr.base.repo.id === repositoryId && pr.base.repo.full_name === repository
&& pr.head.repo?.id === run.head_repository?.id && pr.head.sha === run.head_sha
&& pr.user?.type === "User" && ownerIds.includes(pr.user.id)
&& pr.labels.some((label) => label.name === "profile-ci"));
&& pr.user?.type === "User" && ownerIds.includes(pr.user.id) && accept(pr));
}
export function associatedProfilePr(run, pulls, ownerIds) {
return associatedOwnerPr(run, pulls, ownerIds, (pr) => pr.labels.some((label) => label.name === "profile-ci"));
}
// Always-on export keeps one run in this many, chosen by the workflow's
// sequential run number so that ci-profile.yml can skip the others before
// starting a runner.
export const sampleRate = 10;
// "label": a profile-ci run with test detail, always exported.
// "sampled": an ordinary run chosen by its number, job and step timings only.
export function exportMode(run) {
if (typeof run.display_title === "string" && run.display_title.startsWith("CI profile · PR #")) return "label";
return Number.isSafeInteger(run.run_number) && run.run_number > 0 && run.run_number % sampleRate === 0 ? "sampled" : undefined;
}
async function main() {
+46 -15
View File
@@ -3,8 +3,8 @@ import { appendFileSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from
import { tmpdir } from "node:os";
import path from "node:path";
import { fileURLToPath } from "node:url";
import { buildTrace, traceUrl } from "./trace.mjs";
import { associatedProfilePr, authorizedActors, trustedOwnerIds } from "./authorize.mjs";
import { buildTrace, normalizeChecks, pendingChecks, traceUrl } from "./trace.mjs";
import { associatedOwnerPr, associatedProfilePr, authorizedActors, exportMode, sampleRate, trustedOwnerIds } from "./authorize.mjs";
const repository = "paperclipai/paperclip";
const repoPath = `/repos/${repository}`;
@@ -46,6 +46,21 @@ export async function sendEvents(events, apiKey, dataset, fetcher = fetch, pause
}
}
// Reviews and scans from outside apps can outlast the workflow. Wait a bounded
// time so the trace records how long they took; whatever is still running
// after that is reported as pending instead of holding the export open.
export async function settledChecks(run, load, { waitMs = 15 * 60_000, intervalMs = 30_000,
pause = (ms) => new Promise((resolve) => setTimeout(resolve, ms)), now = Date.now } = {}) {
// The deadline is on the wall clock, so slow GitHub responses count against it.
const deadline = now() + waitMs;
let checks = load();
while (now() < deadline && pendingChecks(run, checks).length) {
await pause(Math.min(intervalMs, deadline - now()));
checks = load();
}
return checks;
}
async function main() {
const args = process.argv.slice(2);
const arg = (name) => args.find((value) => value.startsWith(`${name}=`))?.slice(name.length + 1);
@@ -76,20 +91,31 @@ async function main() {
return;
}
}
const pr = historical ? undefined : associatedProfilePr(run, pulls, ownerIds);
if (!pr && !historical) {
console.log("No CODEOWNERS-authorized profile-ci PR at this revision; no data exported.");
return;
}
const jobs = pages(`${repoPath}/actions/runs/${runId}/attempts/${attempt}/jobs?per_page=100`, "jobs");
const artifacts = pages(`${repoPath}/actions/runs/${runId}/artifacts?per_page=100`, "artifacts")
.filter((artifact) => artifact.name.startsWith(`ci-profile-${attempt}-`));
// The run name snapshots selection at dispatch. A label added while an
// ordinary run finishes must not retroactively select that earlier run.
if (!historical && !run.display_title.startsWith("CI profile · PR #")) {
// The run name snapshots label selection at dispatch. A label added while
// an ordinary run finishes must not retroactively select that earlier run;
// such a run is exported only if its run number is in the sample.
const mode = historical ? "historical" : exportMode(run);
if (!mode) {
console.log("This attempt was not selected for profiling; no data exported.");
return;
}
const pr = mode === "label" ? associatedProfilePr(run, pulls, ownerIds)
: mode === "sampled" ? associatedOwnerPr(run, pulls, ownerIds) : undefined;
if (!pr && !historical) {
console.log("No CODEOWNERS-authorized PR at this revision; no data exported.");
return;
}
const jobs = pages(`${repoPath}/actions/runs/${runId}/attempts/${attempt}/jobs?per_page=100`, "jobs");
// Sampled runs wrote no timing artifacts, so none are read.
const artifacts = mode === "sampled" ? [] : pages(`${repoPath}/actions/runs/${runId}/artifacts?per_page=100`, "artifacts")
.filter((artifact) => artifact.name.startsWith(`ci-profile-${attempt}-`));
if (!/^[a-f\d]{40}$/.test(run.head_sha)) throw new Error("Invalid head commit");
const loadChecks = () => normalizeChecks(run, {
checkRuns: pages(`${repoPath}/commits/${run.head_sha}/check-runs?per_page=100`, "check_runs"),
statuses: pages(`${repoPath}/commits/${run.head_sha}/statuses?per_page=100`),
workflowRuns: pages(`${repoPath}/actions/runs?head_sha=${run.head_sha}&per_page=100`, "workflow_runs"),
});
const checks = await settledChecks(run, loadChecks, args.includes("--dry-run") ? { waitMs: 0 } : {});
const directory = mkdtempSync(path.join(tmpdir(), "paperclip-ci-export-"));
const records = [];
let bytes = 0;
@@ -108,8 +134,11 @@ async function main() {
} finally {
rmSync(directory, { recursive: true, force: true });
}
const trace = buildTrace(run, jobs, records);
trace.events[0].data["ci.profile_source"] = historical ? "historical" : "label";
const trace = buildTrace(run, jobs, records, {
sampleRate: mode === "sampled" ? sampleRate : 1, checks, testDetail: mode !== "sampled",
});
trace.events[0].data["ci.profile_source"] = mode;
if (pr) trace.events[0].data["ci.pr_number"] = pr.number;
const dataset = process.env.HONEYCOMB_CI_DATASET || "paperclip-ci";
const url = traceUrl(process.env.HONEYCOMB_TEAM || "paperclip", process.env.HONEYCOMB_ENVIRONMENT || "test", dataset, trace);
if (arg("--output")) writeFileSync(arg("--output"), JSON.stringify(trace, null, 2));
@@ -123,6 +152,8 @@ async function main() {
`### CI profile${pr ? ` for PR #${pr.number}` : " (historical job/step baseline)"}`,
"", `[Open the full CI waterfall in Honeycomb](${url})`, "",
`${trace.jobCount} jobs · ${(trace.wallMs / 1000).toFixed(1)} seconds elapsed · ${trace.events.length} spans`,
"", `${trace.checkCount} outside checks · ${(trace.windowMs / 1000).toFixed(1)} seconds until the last one finished`,
...(mode === "sampled" ? ["", `Sampled run (1 in ${sampleRate}): job and step timings, without per-test detail.`] : []),
"", `GitHub run: ${run.html_url}/attempts/${attempt}`, "",
...trace.warnings.map((warning) => `- ${warning}`),
"", "The workflow root measures elapsed time. Parallel job durations must not be summed as elapsed time.", "",
+113 -7
View File
@@ -2,10 +2,73 @@ import { createHash } from "node:crypto";
const millis = (value) => Date.parse(value);
const id = (value, size = 16) => createHash("sha256").update(value).digest("hex").slice(0, size);
const failed = (status) => ["failure", "failed", "timed_out", "timedOut", "cancelled", "interrupted"].includes(status);
// "error" is a commit status that could not complete.
const failed = (status) => ["failure", "failed", "error", "timed_out", "timedOut", "cancelled", "interrupted"].includes(status);
const text = (value) => typeof value === "string" && value.length <= 4096;
export function buildTrace(run, jobs, records = []) {
// Outside checks belong to the push that started this run. A check that began
// well before the run (for example, before a later label event) is not part of
// this run's elapsed time.
const checkSlackMs = 60_000;
const label = (value) => typeof value === "string" ? value.slice(0, 256) : "";
// Reduce GitHub's check-run, commit-status and workflow-run listings for the
// run's head commit to the allowlisted fields the trace needs.
export function normalizeChecks(run, { checkRuns = [], statuses = [], workflowRuns = [] } = {}) {
const checks = [];
for (const check of checkRuns) {
// GitHub Actions check runs are jobs. This run's jobs are already spans,
// and other workflows are summarized per workflow below.
if (!check?.app?.slug || check.app.slug === "github-actions") continue;
checks.push({
kind: "external_check", app: label(check.app.slug), name: label(check.name),
status: label(check.conclusion || check.status), startedAt: check.started_at,
completedAt: check.status === "completed" ? check.completed_at : null,
});
}
const contexts = new Map();
for (const status of statuses) {
if (typeof status?.context !== "string") continue;
contexts.set(status.context, [...(contexts.get(status.context) || []), status]);
}
for (const [context, entries] of contexts) {
entries.sort((a, b) => millis(a.created_at) - millis(b.created_at));
const last = entries.at(-1);
checks.push({
kind: "external_check", app: "commit-status", name: label(context), status: label(last.state),
startedAt: entries[0].created_at, completedAt: last.state === "pending" ? null : last.created_at,
});
}
const workflows = new Map();
for (const other of workflowRuns) {
if (other?.id === run.id || other?.path === run.path || other?.head_sha !== run.head_sha
|| !["pull_request", "pull_request_target"].includes(other?.event)) continue;
const known = workflows.get(other.path);
if (!known || millis(other.created_at) > millis(known.created_at)) workflows.set(other.path, other);
}
for (const other of workflows.values()) {
checks.push({
kind: "workflow_check", app: "github-actions", name: label(other.name),
status: label(other.conclusion || other.status),
startedAt: other.run_attempt > 1 ? other.run_started_at : other.created_at,
// GitHub exposes no completion time for a run; its last update is the closest record.
completedAt: other.status === "completed" ? other.updated_at : null,
});
}
return checks;
}
const runStart = (run) => millis(run.run_attempt === 1 ? run.created_at : run.run_started_at);
// A rerun repeats jobs, not the outside checks, so only a first attempt owns them.
const ownChecks = (run, checks) => run.run_attempt === 1
? checks.filter((check) => !(millis(check.startedAt) < runStart(run) - checkSlackMs)) : [];
export function pendingChecks(run, checks, kind = "external_check") {
return ownChecks(run, checks).filter((check) => check.kind === kind && !check.completedAt);
}
export function buildTrace(run, jobs, records = [], { sampleRate = 1, checks = [], testDetail = true } = {}) {
if (!Number.isSafeInteger(sampleRate) || sampleRate < 1) throw new Error("Invalid sample rate");
if (run.status !== "completed" || !jobs.length || jobs.some((job) => job.status !== "completed")) {
throw new Error("A complete workflow attempt and its full job inventory are required");
}
@@ -15,7 +78,7 @@ export function buildTrace(run, jobs, records = []) {
const key = `${run.repository.full_name}:${run.id}:${run.run_attempt}`;
const traceId = id(key, 32);
const rootId = id(`${key}:root`);
const start = millis(run.run_attempt === 1 ? run.created_at : run.run_started_at);
const start = runStart(run);
// The attempt jobs API can relabel carried-over successful jobs with the
// new attempt number. Their completion clocks still precede this attempt.
const carriedJobs = run.run_attempt > 1 ? jobs.filter((job) => millis(job.completed_at) < start) : [];
@@ -39,7 +102,9 @@ export function buildTrace(run, jobs, records = []) {
const span = (spanId, parent, name, from, to, fields) => {
if (!Number.isFinite(from) || !Number.isFinite(to) || to < from) return;
events.push({
time: new Date(from).toISOString(), samplerate: 1,
// Sampling selects whole runs. Every span of a selected run is kept, and
// this weight lets Honeycomb scale counts back to all runs.
time: new Date(from).toISOString(), samplerate: sampleRate,
data: { ...shared, "trace.span_id": spanId, ...(parent ? { "trace.parent_id": parent } : {}),
name, duration_ms: to - from, ...fields },
});
@@ -87,6 +152,32 @@ export function buildTrace(run, jobs, records = []) {
}
}
// Reviews, scans and other workflows on the same commit run beside this
// workflow. They are siblings of the jobs on the same wall clock.
const own = ownChecks(run, checks);
const stale = run.run_attempt === 1 ? checks.length - own.length : 0;
const pending = own.filter((check) => !check.completedAt);
let lastEnd = end;
let lastName = "PR workflow";
let slowest;
let exported = 0;
for (const check of own) {
const from = millis(check.startedAt);
const to = millis(check.completedAt);
// Skipped checks report a completion before their start; they did no work.
if (!check.completedAt || check.status === "skipped" || !Number.isFinite(from) || !Number.isFinite(to) || to < from) continue;
span(id(`${key}:check:${check.kind}:${check.app}:${check.name}`), rootId, check.name, from, to, {
"ci.kind": check.kind, "ci.check": check.name, "ci.check_app": check.app,
"ci.status": check.status, error: failed(check.status),
"ci.start_offset_ms": from - start,
"ci.finishes_after_workflow": to > end,
});
exported++;
if (to > lastEnd) { lastEnd = to; lastName = check.name; }
if (!slowest || to - from > slowest.ms) slowest = { name: check.name, ms: to - from };
}
if (pending.length) warnings.push(`${pending.length} outside checks were still running at export`);
const receipts = new Set();
const invocations = new Map();
const valid = [];
@@ -138,7 +229,9 @@ export function buildTrace(run, jobs, records = []) {
...(Number.isFinite(record.setupMs) ? { "test.setup_ms": record.setupMs } : {}),
});
}
const expectedReceipts = jobs.filter((job) => job.steps?.some((step) => step.name === "Initialize CI profile"));
// Ordinary runs skip the collection steps, so they owe no timing artifacts.
const expectedReceipts = testDetail
? jobs.filter((job) => job.steps?.some((step) => step.name === "Initialize CI profile")) : [];
const missing = expectedReceipts.filter((job) => !receipts.has(job.id));
if (missing.length) warnings.push(`Missing timing artifacts for ${missing.length} jobs`);
// Some package scripts invoke a reporter without using the launcher. Its
@@ -160,20 +253,33 @@ export function buildTrace(run, jobs, records = []) {
return framework && !valid.some((entry) => entry.job.id === job.id && entry.record.framework === framework);
});
if (expectedReceipts.length && missingTests.length) warnings.push(`Missing test detail for ${missingTests.length} test jobs (including jobs that failed before testing)`);
root["ci.sample_rate"] = sampleRate;
root["ci.workflow_ms"] = end - start;
// Push to the last finished check. Checks still running at export are excluded and counted.
root["ci.all_checks_ms"] = lastEnd - start;
root["ci.last_to_finish"] = lastName;
root["ci.check_count"] = exported;
root["ci.pending_check_count"] = pending.length;
root["ci.pending_checks"] = pending.map((check) => check.name).join(", ").slice(0, 1024);
root["ci.earlier_check_count"] = stale;
if (slowest) {
root["ci.slowest_check"] = slowest.name;
root["ci.slowest_check_ms"] = slowest.ms;
}
root["ci.test_count"] = valid.filter(({ record }) => record.kind === "test").length;
root["ci.file_count"] = valid.filter(({ record }) => record.kind === "file").length;
root["ci.expected_invocations"] = invocations.size;
root["ci.reported_invocations"] = [...invocations.values()].filter((value) => value.start && value.end).length;
root["ci.profile_complete"] = warnings.length === 0 && run.run_attempt === 1;
root["ci.coverage_warnings"] = warnings.join("; ");
return { traceId, events, warnings, wallMs: end - start, jobCount: jobs.length };
return { traceId, events, warnings, wallMs: end - start, windowMs: lastEnd - start, jobCount: jobs.length, checkCount: exported };
}
export function traceUrl(team, environment, dataset, trace) {
const params = new URLSearchParams({
trace_id: trace.traceId,
trace_start_ts: String(Math.floor(Date.parse(trace.events[0].time) / 1000)),
trace_end_ts: String(Math.ceil((Date.parse(trace.events[0].time) + trace.wallMs) / 1000)),
trace_end_ts: String(Math.ceil((Date.parse(trace.events[0].time) + (trace.windowMs ?? trace.wallMs)) / 1000)),
});
return `https://ui.honeycomb.io/${encodeURIComponent(team)}/environments/${encodeURIComponent(environment)}/datasets/${encodeURIComponent(dataset)}/trace?${params}`;
}