feat(ci): send one in ten CODEOWNER PR runs and their outside checks to Honeycomb (#21192)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Contributors need fast, predictable CI feedback.
> - PR #16378 added a Honeycomb trace of the main PR workflow, but only
for PRs with the `profile-ci` label.
> - Ordinary CI time was therefore never recorded. All of the last 300
`CI profile` runs were skipped.
> - Reviews and scans outside the `PR` workflow, such as Greptile, were
not in the trace.
> - This pull request exports one CODEOWNER PR run in ten without a
label, and adds outside checks to every trace.
> - The benefit is a continuous record of total CI time and of which job
or check was slowest. The Honeycomb key stays out of reach of outside
contributors.

## Linked Issues or Issue Description

Refs #16378

**What existing behavior does this improve?**

The Honeycomb CI trace that the `CI profile` workflow exports.

**Current behavior**

- The exporter runs only for a PR with the `profile-ci` label. The label
also forces the full suite.
- The trace has the jobs of the `PR` workflow only. Greptile, security
scans, commit statuses and other workflows are absent.
- The only total is the `PR` workflow time.

**Proposed behavior**

- One `PR` run in ten is exported without a label. The run number must
end in 0. The trace has every job and step. Each span has a sample rate
of 10.
- Each trace has one span for each outside check and each other workflow
on the head commit.
- The root span has `ci.workflow_ms`, `ci.all_checks_ms`,
`ci.last_to_finish` and `ci.slowest_check`.

**Reason and benefit**

Maintainers can see how long CI takes and what is slow, for example a
long Greptile review, without a label on each PR. Sampling keeps the
volume near 1.7 million events each month. Export of every run is near
17 million.

## What Changed

- `scripts/ci-profile/authorize.mjs`: add `exportMode` (label or
one-in-ten sample) and `associatedOwnerPr` (the CODEOWNER checks without
the label).
- `scripts/ci-profile/trace.mjs`: add `normalizeChecks` and spans for
outside checks. Add the sample rate and the new root fields. Sampled
runs do not warn about absent test artifacts.
- `scripts/ci-profile/export.mjs`: select the mode. Read check runs,
commit statuses and workflow runs for the head commit. Wait a maximum of
15 minutes for outside apps that are not complete. Sampled runs read no
artifacts.
- `.github/workflows/ci-profile.yml`: no new job. The job condition now
requires a branch of this repository, an actor and rerun actor on the
`AWS_CI_ROUTING_USER_IDS` roster, and a labelled or sampled run. Add
`checks: read` and `statuses: read`. Increase the timeout to 30 minutes
for the bounded wait.
- `doc/ci-profiling.md` and `doc/DEVELOPING.md`: describe sampled runs,
outside checks, the totals and the job condition.

## Verification

- `node --test .github/scripts/tests/ci-profile.test.mjs
.github/scripts/tests/ci-routing.test.mjs`: 163 pass, 0 fail.
- `actionlint .github/workflows/ci-profile.yml`: no findings.
- Dry runs against real runs with the real CODEOWNERS check (`node
scripts/ci-profile/export.mjs --run-id=ID --attempt=1 --dry-run`):
- A sampled CODEOWNER run (run number 64780): 1,144 spans, sample rate
10, 5 outside checks and 1 other workflow.
- A sampled run from an outside contributor (run number 64800): "No
CODEOWNERS-authorized PR at this revision; no data exported."
- A CODEOWNER run that is not in the sample: "This attempt was not
selected for profiling; no data exported."
- New tests cover: the sample rule, fork and outsider PRs, a bot rerun
actor, a stale head, outside-check spans and totals, the bounded wait,
and the job condition.
- Not run locally: `scripts/ci-profile/reporters.test.mjs`. It needs
installed dependencies, and this change does not touch the reporters. CI
runs it.
- No data was sent to Honeycomb before merge. The workflow runs from
`master`, so the first real export occurs after merge.

## Risks

- The label path now also needs the `AWS_CI_ROUTING_USER_IDS` roster. If
that variable is empty, nothing is exported. All four CI CODEOWNERS are
on the roster today.
- The roster is only a first filter. The exporter still checks the PR
author and both actors against CODEOWNERS on `master` before it reads or
sends data. The key stays in the `ci-profiling` environment, which
permits only `master`.
- The export job can now run for a maximum of 30 minutes on a
GitHub-hosted runner while it waits for an outside check.
- GitHub gives no completion time for a workflow run. The span for
another workflow ends at the last update of that run.
- Honeycomb volume increases by about 1.7 million events each month.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code, with
tool use, code execution and repository inspection.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Fable 5.2 authored and GitHub committed 2026-11-12 16:57:23 -06:00
1 parent 0904a65d82
commit 9a9cd20d91
7 files changed
+415 -38

No files matched your search

+148 -3
View File
@@ -5,9 +5,9 @@ import { execFileSync, spawnSync } from "node:child_process";
import { tmpdir } from "node:os";
import path from "node:path";
import { fileURLToPath } from "node:url";
import { buildTrace } from "../../../scripts/ci-profile/trace.mjs";
import { sendEvents } from "../../../scripts/ci-profile/export.mjs";
import { associatedProfilePr, ciOwners, trustedOwnerIds } from "../../../scripts/ci-profile/authorize.mjs";
import { buildTrace, normalizeChecks, pendingChecks, traceUrl } from "../../../scripts/ci-profile/trace.mjs";
import { sendEvents, settledChecks } from "../../../scripts/ci-profile/export.mjs";
import { associatedOwnerPr, associatedProfilePr, ciOwners, exportMode, sampleRate, trustedOwnerIds } from "../../../scripts/ci-profile/authorize.mjs";
const at = (seconds) => new Date(1_800_000_000_000 + seconds * 1000).toISOString();
const run = {
@@ -186,6 +186,136 @@ test("label alone cannot authorize outsiders, bots, stale revisions or outsider
assert.equal(associatedProfilePr(run, [pr], []), undefined);
});
test("always-on export samples whole CODEOWNER runs and never needs the label", () => {
const pr = { user: { id: 123, type: "User" }, base: { repo: run.repository }, head: { repo: { id: 99 }, sha: "abc" }, labels: [] };
assert.equal(sampleRate, 10);
assert.equal(exportMode({ ...run, run_number: 64800, display_title: "fix: something" }), "sampled");
for (const number of [64801, 64805, 64809, 0, undefined, "64800"]) {
assert.equal(exportMode({ ...run, run_number: number, display_title: "fix: something" }), undefined);
}
assert.equal(exportMode({ ...run, run_number: 64801, display_title: "CI profile · PR #5" }), "label");
assert.equal(associatedOwnerPr(run, [pr], [123]), pr);
assert.equal(associatedProfilePr(run, [pr], [123]), undefined);
for (const changed of [
{ ...pr, user: { id: 999, type: "User" } },
{ ...pr, user: { id: 123, type: "Bot" } },
{ ...pr, head: { ...pr.head, sha: "next" } },
{ ...pr, head: { ...pr.head, repo: { id: 100 } } },
{ ...pr, head: { ...pr.head, repo: null } },
{ ...pr, base: { repo: { ...run.repository, id: 100 } } },
]) assert.equal(associatedOwnerPr(run, [changed], [123]), undefined);
assert.equal(associatedOwnerPr({ ...run, actor: { id: 999, type: "User" } }, [pr], [123]), undefined);
assert.equal(associatedOwnerPr({ ...run, triggering_actor: { id: 41898282, type: "Bot" } }, [pr], [123]), undefined);
assert.equal(associatedOwnerPr(run, [pr], []), undefined);
const trace = buildTrace({ ...run, conclusion: "success" }, jobs, [], { sampleRate, testDetail: false });
assert.deepEqual([...new Set(trace.events.map((event) => event.samplerate))], [10]);
assert.equal(trace.events[0].data["ci.sample_rate"], 10);
assert.deepEqual(trace.warnings, ["Job/step timings only; no test reporter collection steps were present"]);
assert.equal(buildTrace(run, jobs).events[0].samplerate, 1);
for (const invalid of [0, 1.5, -1, "10"]) assert.throws(() => buildTrace(run, jobs, [], { sampleRate: invalid }), /sample rate/);
});
test("outside checks are spans on the run's clock, with elapsed time to the last one", () => {
const head = { ...run, conclusion: "success", path: ".github/workflows/pr.yml" };
const checks = normalizeChecks(head, {
checkRuns: [
{ app: { slug: "greptile-apps" }, name: "Greptile Review", status: "completed", conclusion: "success", started_at: at(1), completed_at: at(90), output: { text: "must not leave GitHub" } },
{ app: { slug: "github-actions" }, name: "ci / first", status: "completed", conclusion: "success", started_at: at(2), completed_at: at(12) },
{ app: { slug: "scanner" }, name: "Scan", status: "in_progress", conclusion: null, started_at: at(1), completed_at: null },
{ app: { slug: "scanner" }, name: "Skipped", status: "completed", conclusion: "skipped", started_at: at(9), completed_at: at(1) },
{ app: { slug: "old" }, name: "Before this run", status: "completed", conclusion: "success", started_at: at(-600), completed_at: at(-500) },
{ name: "no app", status: "completed", started_at: at(1), completed_at: at(2) },
],
statuses: [
{ context: "security/scan", state: "success", created_at: at(15) },
{ context: "security/scan", state: "pending", created_at: at(0) },
{ context: "slow/status", state: "pending", created_at: at(0) },
{ context: "broken/status", state: "error", created_at: at(4) },
{ context: "broken/status", state: "pending", created_at: at(1) },
],
workflowRuns: [
{ id: 42, path: ".github/workflows/pr.yml", head_sha: "abc", event: "pull_request", name: "PR", status: "completed", created_at: at(0), updated_at: at(20) },
{ id: 50, path: ".github/workflows/review.yml", head_sha: "abc", event: "pull_request_target", name: "Review", status: "completed", conclusion: "success", run_attempt: 1, created_at: at(0), updated_at: at(40) },
{ id: 51, path: ".github/workflows/review.yml", head_sha: "abc", event: "pull_request_target", name: "Review", status: "completed", conclusion: "failure", run_attempt: 1, created_at: at(5), updated_at: at(60) },
{ id: 52, path: ".github/workflows/other.yml", head_sha: "other", event: "pull_request", name: "Other commit", status: "completed", created_at: at(0), updated_at: at(9) },
{ id: 53, path: ".github/workflows/ci-profile.yml", head_sha: "abc", event: "workflow_run", name: "CI profile", status: "completed", created_at: at(0), updated_at: at(9) },
],
});
assert.deepEqual(checks.map((check) => check.name).sort(),
["Before this run", "Greptile Review", "Review", "Scan", "Skipped", "broken/status", "security/scan", "slow/status"]);
assert.deepEqual(pendingChecks(head, checks).map((check) => check.name), ["Scan", "slow/status"]);
const trace = buildTrace(head, jobs, [], { checks });
const spans = trace.events.filter((event) => /_check$/.test(event.data["ci.kind"]));
assert.deepEqual(spans.map((event) => [event.data.name, event.data["ci.kind"], event.data.duration_ms, event.data["ci.status"]]), [
["Greptile Review", "external_check", 89000, "success"],
["security/scan", "external_check", 15000, "success"],
["broken/status", "external_check", 3000, "error"],
["Review", "workflow_check", 55000, "failure"],
]);
assert.deepEqual(spans.map((event) => event.data.error), [false, false, true, true]);
const root = trace.events[0].data;
assert.ok(spans.every((event) => event.data["trace.parent_id"] === root["trace.span_id"]));
assert.equal(spans[0].data["ci.finishes_after_workflow"], true);
assert.equal(spans[1].data["ci.finishes_after_workflow"], false);
assert.equal(root.duration_ms, 20000);
assert.equal(root["ci.workflow_ms"], 20000);
assert.equal(root["ci.all_checks_ms"], 90000);
assert.equal(root["ci.last_to_finish"], "Greptile Review");
assert.equal(root["ci.slowest_check"], "Greptile Review");
assert.equal(root["ci.slowest_check_ms"], 89000);
assert.equal(root["ci.check_count"], 4);
assert.equal(root["ci.pending_check_count"], 2);
assert.equal(root["ci.pending_checks"], "Scan, slow/status");
assert.equal(root["ci.earlier_check_count"], 1);
assert.equal(root["ci.profile_complete"], false);
assert.match(trace.warnings.join(), /2 outside checks were still running/);
assert.doesNotMatch(JSON.stringify(trace), /must not leave GitHub/);
assert.match(traceUrl("team", "test", "paperclip-ci", trace), /trace_end_ts=1800000090$/);
const alone = buildTrace(head, jobs).events[0].data;
assert.equal(alone["ci.all_checks_ms"], 20000);
assert.equal(alone["ci.last_to_finish"], "PR workflow");
assert.equal("ci.slowest_check" in alone, false);
// A rerun repeats jobs only; the push's outside checks are not its elapsed time.
const rerun = buildTrace({ ...head, run_attempt: 2, run_started_at: at(0) }, jobs.map((entry) => ({ ...entry, run_attempt: 2 })), [], { checks });
assert.equal(rerun.events[0].data["ci.check_count"], 0);
assert.equal(rerun.events[0].data["ci.all_checks_ms"], 20000);
});
test("export waits a bounded time for outside checks, never for other workflows", async () => {
const head = { ...run, path: ".github/workflows/pr.yml" };
const check = (completed) => ({ kind: "external_check", app: "greptile-apps", name: "Greptile Review", startedAt: at(1), completedAt: completed ? at(90) : null });
const pauses = [];
let clock = 0;
const now = () => clock;
const pause = async (ms) => { pauses.push(ms); clock += ms; };
let loads = 0;
const settled = await settledChecks(head, () => [check(++loads >= 3)], { pause, now });
assert.equal(settled[0].completedAt, at(90));
assert.deepEqual(pauses, [30_000, 30_000]);
pauses.length = 0;
const stuck = await settledChecks(head, () => [check(false)], { pause, now, waitMs: 90_000 });
assert.equal(stuck[0].completedAt, null);
assert.deepEqual(pauses, [30_000, 30_000, 30_000]);
// Slow GitHub responses count against the same deadline as the pauses.
pauses.length = 0;
loads = 0;
await settledChecks(head, () => { clock += 20_000; loads += 1; return [check(false)]; }, { pause, now, waitMs: 90_000 });
assert.deepEqual(pauses, [30_000, 20_000]);
assert.equal(loads, 3);
pauses.length = 0;
await settledChecks(head, () => [{ ...check(false), kind: "workflow_check" }], { pause, now });
await settledChecks(head, () => [{ ...check(false), startedAt: at(-600) }], { pause, now });
await settledChecks({ ...head, run_attempt: 2, run_started_at: at(0) }, () => [check(false)], { pause, now });
await settledChecks(head, () => [check(false)], { pause, now, waitMs: 0 });
assert.deepEqual(pauses, []);
});
test("profiling uses only infrastructure CODEOWNERS resolved at trusted master", () => {
const source = '.github/** @owner @second\ndoc/** @outsider\n';
assert.deepEqual(ciOwners(source), ["owner", "second"]);
@@ -295,6 +425,21 @@ test("workflow exporter runs trusted code and keeps keys out of the PR jobs", ()
const workflow = readFileSync(new URL("../../workflows/ci-profile.yml", import.meta.url), "utf8");
assert.match(workflow, /ref: \$\{\{ github.sha \}\}/);
assert.match(workflow, /startsWith\(github.event.workflow_run.display_title/);
// The job condition keeps outside contributors' runs from starting a runner
// or entering the environment, and matches the exporter's sample rule.
const condition = workflow.split(" if: >-\n")[1].split("\n runs-on:")[0];
for (const required of [
"github.repository_id == '1170821064'",
"github.event.workflow_run.event == 'pull_request'",
"github.event.workflow_run.path == '.github/workflows/pr.yml'",
"github.event.workflow_run.head_repository.full_name == 'paperclipai/paperclip'",
"contains(vars.AWS_CI_ROUTING_USER_IDS, format('|{0}|', github.event.workflow_run.actor.id))",
"contains(vars.AWS_CI_ROUTING_USER_IDS, format('|{0}|', github.event.workflow_run.triggering_actor.id))",
"endsWith(format('{0}', github.event.workflow_run.run_number), '0')",
]) assert.ok(condition.includes(required), required);
assert.equal(condition.match(/\|\|/g).length, 1, "only label-or-sample may be an alternative");
assert.equal(workflow.match(/^ [a-z_]+:$/gm).filter((line) => ![" workflow_run:", " export:"].includes(line)).length, 0);
assert.equal(workflow.match(/environment:/g).length, 1);
assert.match(workflow, /environment: ci-profiling/);
assert.doesNotMatch(workflow, /head_sha|secrets: inherit|contents: write|pull-requests: write|pnpm install/);
const pr = readFileSync(new URL("../../workflows/pr-trusted.yml", import.meta.url), "utf8");
+21 -2
View File
@@ -7,17 +7,36 @@ on:
permissions:
actions: read
# Check runs and commit statuses time the outside reviews and scans.
checks: read
contents: read
pull-requests: read
statuses: read
jobs:
export:
# No runner starts, and the environment is never entered, unless all of
# these hold:
# - The run is the main PR workflow on a branch of this repository.
# - Its actor and rerun actor are on the maintainer roster that CI routing
# already uses. The exporter then verifies them and the PR author against
# master's CODEOWNERS before it reads or sends anything.
# - The run carries the profile-ci label, or its run number ends in 0. That
# is the one-in-ten sample; scripts/ci-profile/authorize.mjs applies the
# same rule.
if: >-
github.repository == 'paperclipai/paperclip' &&
github.repository_id == '1170821064' &&
github.event.workflow_run.event == 'pull_request' &&
startsWith(github.event.workflow_run.display_title, 'CI profile · PR #')
github.event.workflow_run.path == '.github/workflows/pr.yml' &&
github.event.workflow_run.head_repository.full_name == 'paperclipai/paperclip' &&
contains(vars.AWS_CI_ROUTING_USER_IDS, format('|{0}|', github.event.workflow_run.actor.id)) &&
contains(vars.AWS_CI_ROUTING_USER_IDS, format('|{0}|', github.event.workflow_run.triggering_actor.id)) &&
(startsWith(github.event.workflow_run.display_title, 'CI profile · PR #') ||
endsWith(format('{0}', github.event.workflow_run.run_number), '0'))
runs-on: ubuntu-latest
timeout-minutes: 10
# Covers the bounded wait for outside checks that outlast the workflow.
timeout-minutes: 30
# This environment permits only the master branch. Do not use a repository
# secret: an in-repository PR must not be able to request this credential.
environment: ci-profiling