Files
PaperClipAI/scripts/ci-profile/export.mjs
T
DottaandClaude Fable 5.2 9a9cd20d91 feat(ci): send one in ten CODEOWNER PR runs and their outside checks to Honeycomb (#21192)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Contributors need fast, predictable CI feedback.
> - PR #16378 added a Honeycomb trace of the main PR workflow, but only
for PRs with the `profile-ci` label.
> - Ordinary CI time was therefore never recorded. All of the last 300
`CI profile` runs were skipped.
> - Reviews and scans outside the `PR` workflow, such as Greptile, were
not in the trace.
> - This pull request exports one CODEOWNER PR run in ten without a
label, and adds outside checks to every trace.
> - The benefit is a continuous record of total CI time and of which job
or check was slowest. The Honeycomb key stays out of reach of outside
contributors.

## Linked Issues or Issue Description

Refs #16378

**What existing behavior does this improve?**

The Honeycomb CI trace that the `CI profile` workflow exports.

**Current behavior**

- The exporter runs only for a PR with the `profile-ci` label. The label
also forces the full suite.
- The trace has the jobs of the `PR` workflow only. Greptile, security
scans, commit statuses and other workflows are absent.
- The only total is the `PR` workflow time.

**Proposed behavior**

- One `PR` run in ten is exported without a label. The run number must
end in 0. The trace has every job and step. Each span has a sample rate
of 10.
- Each trace has one span for each outside check and each other workflow
on the head commit.
- The root span has `ci.workflow_ms`, `ci.all_checks_ms`,
`ci.last_to_finish` and `ci.slowest_check`.

**Reason and benefit**

Maintainers can see how long CI takes and what is slow, for example a
long Greptile review, without a label on each PR. Sampling keeps the
volume near 1.7 million events each month. Export of every run is near
17 million.

## What Changed

- `scripts/ci-profile/authorize.mjs`: add `exportMode` (label or
one-in-ten sample) and `associatedOwnerPr` (the CODEOWNER checks without
the label).
- `scripts/ci-profile/trace.mjs`: add `normalizeChecks` and spans for
outside checks. Add the sample rate and the new root fields. Sampled
runs do not warn about absent test artifacts.
- `scripts/ci-profile/export.mjs`: select the mode. Read check runs,
commit statuses and workflow runs for the head commit. Wait a maximum of
15 minutes for outside apps that are not complete. Sampled runs read no
artifacts.
- `.github/workflows/ci-profile.yml`: no new job. The job condition now
requires a branch of this repository, an actor and rerun actor on the
`AWS_CI_ROUTING_USER_IDS` roster, and a labelled or sampled run. Add
`checks: read` and `statuses: read`. Increase the timeout to 30 minutes
for the bounded wait.
- `doc/ci-profiling.md` and `doc/DEVELOPING.md`: describe sampled runs,
outside checks, the totals and the job condition.

## Verification

- `node --test .github/scripts/tests/ci-profile.test.mjs
.github/scripts/tests/ci-routing.test.mjs`: 163 pass, 0 fail.
- `actionlint .github/workflows/ci-profile.yml`: no findings.
- Dry runs against real runs with the real CODEOWNERS check (`node
scripts/ci-profile/export.mjs --run-id=ID --attempt=1 --dry-run`):
- A sampled CODEOWNER run (run number 64780): 1,144 spans, sample rate
10, 5 outside checks and 1 other workflow.
- A sampled run from an outside contributor (run number 64800): "No
CODEOWNERS-authorized PR at this revision; no data exported."
- A CODEOWNER run that is not in the sample: "This attempt was not
selected for profiling; no data exported."
- New tests cover: the sample rule, fork and outsider PRs, a bot rerun
actor, a stale head, outside-check spans and totals, the bounded wait,
and the job condition.
- Not run locally: `scripts/ci-profile/reporters.test.mjs`. It needs
installed dependencies, and this change does not touch the reporters. CI
runs it.
- No data was sent to Honeycomb before merge. The workflow runs from
`master`, so the first real export occurs after merge.

## Risks

- The label path now also needs the `AWS_CI_ROUTING_USER_IDS` roster. If
that variable is empty, nothing is exported. All four CI CODEOWNERS are
on the roster today.
- The roster is only a first filter. The exporter still checks the PR
author and both actors against CODEOWNERS on `master` before it reads or
sends data. The key stays in the `ci-profiling` environment, which
permits only `master`.
- The export job can now run for a maximum of 30 minutes on a
GitHub-hosted runner while it waits for an outside check.
- GitHub gives no completion time for a workflow run. The span for
another workflow ends at the last update of that run.
- Honeycomb volume increases by about 1.7 million events each month.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code, with
tool use, code execution and repository inspection.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 16:57:23 -06:00

168 lines
9.8 KiB
JavaScript

import { execFileSync } from "node:child_process";
import { appendFileSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { fileURLToPath } from "node:url";
import { buildTrace, normalizeChecks, pendingChecks, traceUrl } from "./trace.mjs";
import { associatedOwnerPr, associatedProfilePr, authorizedActors, exportMode, sampleRate, trustedOwnerIds } from "./authorize.mjs";
const repository = "paperclipai/paperclip";
const repoPath = `/repos/${repository}`;
const maxBytes = 32 * 1024 * 1024;
const gh = (args) => execFileSync("gh", ["api", ...args], { maxBuffer: maxBytes, timeout: 60_000 });
const get = (endpoint) => JSON.parse(gh([endpoint]).toString());
const pages = (endpoint, field) => JSON.parse(gh(["--paginate", "--slurp", endpoint]).toString())
.flatMap((page) => field ? page[field] : page);
export async function sendEvents(events, apiKey, dataset, fetcher = fetch, pause = (ms) => new Promise((resolve) => setTimeout(resolve, ms))) {
if (!apiKey) throw new Error("HONEYCOMB_CI_API_KEY is not configured");
const transient = new Set([429, 500, 502, 503, 504]);
for (let offset = 0; offset < events.length; offset += 250) {
let pending = events.slice(offset, offset + 250);
for (let attempt = 0; pending.length; attempt += 1) {
if (attempt >= 8) throw new Error(`Honeycomb did not accept ${pending.length} spans after retries`);
const response = await fetcher(`https://api.honeycomb.io/1/batch/${encodeURIComponent(dataset)}`, {
method: "POST", headers: { "X-Honeycomb-Team": apiKey, "Content-Type": "application/json" },
body: JSON.stringify(pending), redirect: "error", signal: AbortSignal.timeout(30_000),
});
if (!response.ok) {
if (!transient.has(response.status)) throw new Error(`Honeycomb export failed (HTTP ${response.status})`);
await pause(Math.min(30_000, 1000 * 2 ** attempt));
continue;
}
const results = await response.json();
if (!Array.isArray(results) || results.length !== pending.length) {
throw new Error("Honeycomb returned an unexpected batch response");
}
const rejected = results.flatMap((result, index) => result.status === 202 ? [] : [{ event: pending[index], status: result.status }]);
const permanent = rejected.find(({ status }) => !transient.has(status));
if (permanent) throw new Error(`Honeycomb rejected ${rejected.length} spans (status ${permanent.status})`);
pending = rejected.map(({ event }) => event);
if (pending.length) await pause(Math.min(30_000, 1000 * 2 ** attempt));
}
// The full suite can exceed Honeycomb's burst allowance. Leave capacity
// for the team's other telemetry rather than exhausting it in one export.
if (offset + 250 < events.length) await pause(1000);
}
}
// Reviews and scans from outside apps can outlast the workflow. Wait a bounded
// time so the trace records how long they took; whatever is still running
// after that is reported as pending instead of holding the export open.
export async function settledChecks(run, load, { waitMs = 15 * 60_000, intervalMs = 30_000,
pause = (ms) => new Promise((resolve) => setTimeout(resolve, ms)), now = Date.now } = {}) {
// The deadline is on the wall clock, so slow GitHub responses count against it.
const deadline = now() + waitMs;
let checks = load();
while (now() < deadline && pendingChecks(run, checks).length) {
await pause(Math.min(intervalMs, deadline - now()));
checks = load();
}
return checks;
}
async function main() {
const args = process.argv.slice(2);
const arg = (name) => args.find((value) => value.startsWith(`${name}=`))?.slice(name.length + 1);
const event = process.env.GITHUB_EVENT_PATH ? JSON.parse(readFileSync(process.env.GITHUB_EVENT_PATH, "utf8")) : null;
const runId = event?.workflow_run?.id ?? arg("--run-id");
const attempt = event?.workflow_run?.run_attempt ?? arg("--attempt");
if (!/^\d+$/.test(String(runId)) || !/^[1-9]\d*$/.test(String(attempt))) {
throw new Error("Provide --run-id and --attempt, or a workflow_run event");
}
const run = get(`${repoPath}/actions/runs/${runId}/attempts/${attempt}`);
if (run.repository?.full_name !== repository || run.event !== "pull_request"
|| run.path !== ".github/workflows/pr.yml" || run.status !== "completed") {
throw new Error("Only completed main PR workflow attempts can be profiled");
}
// GitHub often leaves workflow_run.pull_requests empty for fork PRs. Resolve
// association through the head commit, then validate repo and exact head.
const pulls = pages(`${repoPath}/commits/${run.head_sha}/pulls?per_page=100`);
const historical = args.includes("--historical");
if (historical && process.env.GITHUB_ACTIONS) throw new Error("Historical export is a local-only operation");
const ownerIds = historical ? [] : trustedOwnerIds(get);
if (process.env.GITHUB_ACTIONS) {
const exporterId = process.env.GITHUB_RUN_ID;
const exporterAttempt = process.env.GITHUB_RUN_ATTEMPT;
if (!/^\d+$/.test(exporterId) || !/^[1-9]\d*$/.test(exporterAttempt)) throw new Error("Invalid exporter run identity");
const exporter = get(`${repoPath}/actions/runs/${exporterId}/attempts/${exporterAttempt}`);
if (!authorizedActors(exporter, ownerIds)) {
console.log("Exporter actor is not a CI CODEOWNER; no data exported.");
return;
}
}
// The run name snapshots label selection at dispatch. A label added while
// an ordinary run finishes must not retroactively select that earlier run;
// such a run is exported only if its run number is in the sample.
const mode = historical ? "historical" : exportMode(run);
if (!mode) {
console.log("This attempt was not selected for profiling; no data exported.");
return;
}
const pr = mode === "label" ? associatedProfilePr(run, pulls, ownerIds)
: mode === "sampled" ? associatedOwnerPr(run, pulls, ownerIds) : undefined;
if (!pr && !historical) {
console.log("No CODEOWNERS-authorized PR at this revision; no data exported.");
return;
}
const jobs = pages(`${repoPath}/actions/runs/${runId}/attempts/${attempt}/jobs?per_page=100`, "jobs");
// Sampled runs wrote no timing artifacts, so none are read.
const artifacts = mode === "sampled" ? [] : pages(`${repoPath}/actions/runs/${runId}/artifacts?per_page=100`, "artifacts")
.filter((artifact) => artifact.name.startsWith(`ci-profile-${attempt}-`));
if (!/^[a-f\d]{40}$/.test(run.head_sha)) throw new Error("Invalid head commit");
const loadChecks = () => normalizeChecks(run, {
checkRuns: pages(`${repoPath}/commits/${run.head_sha}/check-runs?per_page=100`, "check_runs"),
statuses: pages(`${repoPath}/commits/${run.head_sha}/statuses?per_page=100`),
workflowRuns: pages(`${repoPath}/actions/runs?head_sha=${run.head_sha}&per_page=100`, "workflow_runs"),
});
const checks = await settledChecks(run, loadChecks, args.includes("--dry-run") ? { waitMs: 0 } : {});
const directory = mkdtempSync(path.join(tmpdir(), "paperclip-ci-export-"));
const records = [];
let bytes = 0;
try {
for (const artifact of artifacts) {
if (artifact.expired || artifact.size_in_bytes > maxBytes) throw new Error("Timing artifact is expired or too large");
const archive = path.join(directory, `${artifact.id}.zip`);
writeFileSync(archive, gh([`${repoPath}/actions/artifacts/${artifact.id}/zip`]));
const decoded = execFileSync("python3", [fileURLToPath(new URL("./read-artifact.py", import.meta.url)), archive], {
encoding: "utf8", maxBuffer: maxBytes, timeout: 30_000,
});
bytes += Buffer.byteLength(decoded);
if (bytes > 128 * 1024 * 1024) throw new Error("Run timing artifacts exceed 128 MiB; refusing to truncate the trace");
for (const line of decoded.trim().split("\n").filter(Boolean)) records.push(JSON.parse(line));
}
} finally {
rmSync(directory, { recursive: true, force: true });
}
const trace = buildTrace(run, jobs, records, {
sampleRate: mode === "sampled" ? sampleRate : 1, checks, testDetail: mode !== "sampled",
});
trace.events[0].data["ci.profile_source"] = mode;
if (pr) trace.events[0].data["ci.pr_number"] = pr.number;
const dataset = process.env.HONEYCOMB_CI_DATASET || "paperclip-ci";
const url = traceUrl(process.env.HONEYCOMB_TEAM || "paperclip", process.env.HONEYCOMB_ENVIRONMENT || "test", dataset, trace);
if (arg("--output")) writeFileSync(arg("--output"), JSON.stringify(trace, null, 2));
if (!args.includes("--dry-run")) {
// Publish the completed root only after every child has been accepted.
// A failed partial export must not advertise complete coverage.
await sendEvents(trace.events.slice(1), process.env.HONEYCOMB_CI_API_KEY, dataset);
await sendEvents([trace.events[0]], process.env.HONEYCOMB_CI_API_KEY, dataset);
}
const summary = [
`### CI profile${pr ? ` for PR #${pr.number}` : " (historical job/step baseline)"}`,
"", `[Open the full CI waterfall in Honeycomb](${url})`, "",
`${trace.jobCount} jobs · ${(trace.wallMs / 1000).toFixed(1)} seconds elapsed · ${trace.events.length} spans`,
"", `${trace.checkCount} outside checks · ${(trace.windowMs / 1000).toFixed(1)} seconds until the last one finished`,
...(mode === "sampled" ? ["", `Sampled run (1 in ${sampleRate}): job and step timings, without per-test detail.`] : []),
"", `GitHub run: ${run.html_url}/attempts/${attempt}`, "",
...trace.warnings.map((warning) => `- ${warning}`),
"", "The workflow root measures elapsed time. Parallel job durations must not be summed as elapsed time.", "",
].join("\n");
console.log(summary);
if (process.env.GITHUB_STEP_SUMMARY) appendFileSync(process.env.GITHUB_STEP_SUMMARY, summary);
}
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
main().catch((error) => { console.error(error.message); process.exitCode = 1; });
}