Files
PaperClipAI/scripts/ci-profile/trace.mjs
T
DottaandClaude Fable 5.2 9a9cd20d91 feat(ci): send one in ten CODEOWNER PR runs and their outside checks to Honeycomb (#21192)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Contributors need fast, predictable CI feedback.
> - PR #16378 added a Honeycomb trace of the main PR workflow, but only
for PRs with the `profile-ci` label.
> - Ordinary CI time was therefore never recorded. All of the last 300
`CI profile` runs were skipped.
> - Reviews and scans outside the `PR` workflow, such as Greptile, were
not in the trace.
> - This pull request exports one CODEOWNER PR run in ten without a
label, and adds outside checks to every trace.
> - The benefit is a continuous record of total CI time and of which job
or check was slowest. The Honeycomb key stays out of reach of outside
contributors.

## Linked Issues or Issue Description

Refs #16378

**What existing behavior does this improve?**

The Honeycomb CI trace that the `CI profile` workflow exports.

**Current behavior**

- The exporter runs only for a PR with the `profile-ci` label. The label
also forces the full suite.
- The trace has the jobs of the `PR` workflow only. Greptile, security
scans, commit statuses and other workflows are absent.
- The only total is the `PR` workflow time.

**Proposed behavior**

- One `PR` run in ten is exported without a label. The run number must
end in 0. The trace has every job and step. Each span has a sample rate
of 10.
- Each trace has one span for each outside check and each other workflow
on the head commit.
- The root span has `ci.workflow_ms`, `ci.all_checks_ms`,
`ci.last_to_finish` and `ci.slowest_check`.

**Reason and benefit**

Maintainers can see how long CI takes and what is slow, for example a
long Greptile review, without a label on each PR. Sampling keeps the
volume near 1.7 million events each month. Export of every run is near
17 million.

## What Changed

- `scripts/ci-profile/authorize.mjs`: add `exportMode` (label or
one-in-ten sample) and `associatedOwnerPr` (the CODEOWNER checks without
the label).
- `scripts/ci-profile/trace.mjs`: add `normalizeChecks` and spans for
outside checks. Add the sample rate and the new root fields. Sampled
runs do not warn about absent test artifacts.
- `scripts/ci-profile/export.mjs`: select the mode. Read check runs,
commit statuses and workflow runs for the head commit. Wait a maximum of
15 minutes for outside apps that are not complete. Sampled runs read no
artifacts.
- `.github/workflows/ci-profile.yml`: no new job. The job condition now
requires a branch of this repository, an actor and rerun actor on the
`AWS_CI_ROUTING_USER_IDS` roster, and a labelled or sampled run. Add
`checks: read` and `statuses: read`. Increase the timeout to 30 minutes
for the bounded wait.
- `doc/ci-profiling.md` and `doc/DEVELOPING.md`: describe sampled runs,
outside checks, the totals and the job condition.

## Verification

- `node --test .github/scripts/tests/ci-profile.test.mjs
.github/scripts/tests/ci-routing.test.mjs`: 163 pass, 0 fail.
- `actionlint .github/workflows/ci-profile.yml`: no findings.
- Dry runs against real runs with the real CODEOWNERS check (`node
scripts/ci-profile/export.mjs --run-id=ID --attempt=1 --dry-run`):
- A sampled CODEOWNER run (run number 64780): 1,144 spans, sample rate
10, 5 outside checks and 1 other workflow.
- A sampled run from an outside contributor (run number 64800): "No
CODEOWNERS-authorized PR at this revision; no data exported."
- A CODEOWNER run that is not in the sample: "This attempt was not
selected for profiling; no data exported."
- New tests cover: the sample rule, fork and outsider PRs, a bot rerun
actor, a stale head, outside-check spans and totals, the bounded wait,
and the job condition.
- Not run locally: `scripts/ci-profile/reporters.test.mjs`. It needs
installed dependencies, and this change does not touch the reporters. CI
runs it.
- No data was sent to Honeycomb before merge. The workflow runs from
`master`, so the first real export occurs after merge.

## Risks

- The label path now also needs the `AWS_CI_ROUTING_USER_IDS` roster. If
that variable is empty, nothing is exported. All four CI CODEOWNERS are
on the roster today.
- The roster is only a first filter. The exporter still checks the PR
author and both actors against CODEOWNERS on `master` before it reads or
sends data. The key stays in the `ci-profiling` environment, which
permits only `master`.
- The export job can now run for a maximum of 30 minutes on a
GitHub-hosted runner while it waits for an outside check.
- GitHub gives no completion time for a workflow run. The span for
another workflow ends at the last update of that run.
- Honeycomb volume increases by about 1.7 million events each month.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code, with
tool use, code execution and repository inspection.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 16:57:23 -06:00

286 lines
16 KiB
JavaScript

import { createHash } from "node:crypto";
const millis = (value) => Date.parse(value);
const id = (value, size = 16) => createHash("sha256").update(value).digest("hex").slice(0, size);
// "error" is a commit status that could not complete.
const failed = (status) => ["failure", "failed", "error", "timed_out", "timedOut", "cancelled", "interrupted"].includes(status);
const text = (value) => typeof value === "string" && value.length <= 4096;
// Outside checks belong to the push that started this run. A check that began
// well before the run (for example, before a later label event) is not part of
// this run's elapsed time.
const checkSlackMs = 60_000;
const label = (value) => typeof value === "string" ? value.slice(0, 256) : "";
// Reduce GitHub's check-run, commit-status and workflow-run listings for the
// run's head commit to the allowlisted fields the trace needs.
export function normalizeChecks(run, { checkRuns = [], statuses = [], workflowRuns = [] } = {}) {
const checks = [];
for (const check of checkRuns) {
// GitHub Actions check runs are jobs. This run's jobs are already spans,
// and other workflows are summarized per workflow below.
if (!check?.app?.slug || check.app.slug === "github-actions") continue;
checks.push({
kind: "external_check", app: label(check.app.slug), name: label(check.name),
status: label(check.conclusion || check.status), startedAt: check.started_at,
completedAt: check.status === "completed" ? check.completed_at : null,
});
}
const contexts = new Map();
for (const status of statuses) {
if (typeof status?.context !== "string") continue;
contexts.set(status.context, [...(contexts.get(status.context) || []), status]);
}
for (const [context, entries] of contexts) {
entries.sort((a, b) => millis(a.created_at) - millis(b.created_at));
const last = entries.at(-1);
checks.push({
kind: "external_check", app: "commit-status", name: label(context), status: label(last.state),
startedAt: entries[0].created_at, completedAt: last.state === "pending" ? null : last.created_at,
});
}
const workflows = new Map();
for (const other of workflowRuns) {
if (other?.id === run.id || other?.path === run.path || other?.head_sha !== run.head_sha
|| !["pull_request", "pull_request_target"].includes(other?.event)) continue;
const known = workflows.get(other.path);
if (!known || millis(other.created_at) > millis(known.created_at)) workflows.set(other.path, other);
}
for (const other of workflows.values()) {
checks.push({
kind: "workflow_check", app: "github-actions", name: label(other.name),
status: label(other.conclusion || other.status),
startedAt: other.run_attempt > 1 ? other.run_started_at : other.created_at,
// GitHub exposes no completion time for a run; its last update is the closest record.
completedAt: other.status === "completed" ? other.updated_at : null,
});
}
return checks;
}
const runStart = (run) => millis(run.run_attempt === 1 ? run.created_at : run.run_started_at);
// A rerun repeats jobs, not the outside checks, so only a first attempt owns them.
const ownChecks = (run, checks) => run.run_attempt === 1
? checks.filter((check) => !(millis(check.startedAt) < runStart(run) - checkSlackMs)) : [];
export function pendingChecks(run, checks, kind = "external_check") {
return ownChecks(run, checks).filter((check) => check.kind === kind && !check.completedAt);
}
export function buildTrace(run, jobs, records = [], { sampleRate = 1, checks = [], testDetail = true } = {}) {
if (!Number.isSafeInteger(sampleRate) || sampleRate < 1) throw new Error("Invalid sample rate");
if (run.status !== "completed" || !jobs.length || jobs.some((job) => job.status !== "completed")) {
throw new Error("A complete workflow attempt and its full job inventory are required");
}
if (jobs.some((job) => job.run_id !== run.id || job.run_attempt !== run.run_attempt)) {
throw new Error("Job inventory contains another run or attempt");
}
const key = `${run.repository.full_name}:${run.id}:${run.run_attempt}`;
const traceId = id(key, 32);
const rootId = id(`${key}:root`);
const start = runStart(run);
// The attempt jobs API can relabel carried-over successful jobs with the
// new attempt number. Their completion clocks still precede this attempt.
const carriedJobs = run.run_attempt > 1 ? jobs.filter((job) => millis(job.completed_at) < start) : [];
jobs = jobs.filter((job) => !carriedJobs.includes(job));
if (!jobs.length) throw new Error("No jobs executed in this workflow attempt");
const end = Math.max(...jobs.map((job) => millis(job.completed_at)).filter(Number.isFinite));
if (!Number.isFinite(start) || !Number.isFinite(end) || end < start) throw new Error("Invalid run timing");
const events = [];
const warnings = [];
if (carriedJobs.length) warnings.push(`${carriedJobs.length} completed jobs carried over from earlier attempts were excluded`);
const shared = {
"service.name": "paperclip-ci",
"trace.trace_id": traceId,
"ci.repository": run.repository.full_name,
"ci.run_id": String(run.id),
"ci.attempt": run.run_attempt,
"ci.commit": run.head_sha,
"ci.branch": run.head_branch,
"ci.url": run.html_url,
};
const span = (spanId, parent, name, from, to, fields) => {
if (!Number.isFinite(from) || !Number.isFinite(to) || to < from) return;
events.push({
// Sampling selects whole runs. Every span of a selected run is kept, and
// this weight lets Honeycomb scale counts back to all runs.
time: new Date(from).toISOString(), samplerate: sampleRate,
data: { ...shared, "trace.span_id": spanId, ...(parent ? { "trace.parent_id": parent } : {}),
name, duration_ms: to - from, ...fields },
});
};
span(rootId, null, `PR CI #${run.run_number || run.id} · attempt ${run.run_attempt}`, start, end, {
"ci.kind": "workflow", "ci.status": run.conclusion, error: failed(run.conclusion),
"ci.job_count": jobs.length,
"ci.carried_job_count": carriedJobs.length,
"ci.wall_ms": end - start,
// A partial rerun intentionally does not re-execute successful jobs.
"ci.partial_rerun": run.run_attempt > 1,
});
const roots = new Map();
const byName = new Map();
for (const job of jobs) {
const jobId = id(`${key}:job:${job.id}`);
roots.set(job.id, jobId);
byName.set(job.name.replace(/^ci \/ /, ""), job);
const created = Math.max(start, millis(job.created_at) || start);
const started = Math.max(created, millis(job.started_at) || created);
const completed = Math.max(started, millis(job.completed_at) || started);
// Include allocation wait in the job container, then distinguish it from
// execution. Dependency wait may precede job creation; it stays visible as
// a gap from the workflow start. GitHub does not expose a separate readyAt.
span(jobId, rootId, job.name, Math.min(created, started), completed, {
"ci.kind": "job", "ci.job": job.name, "ci.job_id": String(job.id),
"ci.status": job.conclusion, error: failed(job.conclusion),
"ci.runner": job.runner_name, "ci.runner_labels": (job.labels || []).join(","),
"ci.start_offset_ms": started - start,
"ci.execution_ms": Math.max(0, completed - started),
"ci.finishes_last": completed === end,
"ci.job_url": job.html_url,
});
if (started > created) span(id(`${jobId}:wait`), jobId, "Waiting for runner / scheduling", created, started, {
"ci.kind": "wait", "ci.job": job.name,
});
for (const step of job.steps || []) {
const from = millis(step.started_at);
const to = millis(step.completed_at);
if (from < start || to > end) continue; // skipped steps may use epoch timestamps
span(id(`${jobId}:step:${step.number}`), jobId, step.name, from, to, {
"ci.kind": "step", "ci.job": job.name, "ci.step": step.name,
"ci.status": step.conclusion, error: failed(step.conclusion),
});
}
}
// Reviews, scans and other workflows on the same commit run beside this
// workflow. They are siblings of the jobs on the same wall clock.
const own = ownChecks(run, checks);
const stale = run.run_attempt === 1 ? checks.length - own.length : 0;
const pending = own.filter((check) => !check.completedAt);
let lastEnd = end;
let lastName = "PR workflow";
let slowest;
let exported = 0;
for (const check of own) {
const from = millis(check.startedAt);
const to = millis(check.completedAt);
// Skipped checks report a completion before their start; they did no work.
if (!check.completedAt || check.status === "skipped" || !Number.isFinite(from) || !Number.isFinite(to) || to < from) continue;
span(id(`${key}:check:${check.kind}:${check.app}:${check.name}`), rootId, check.name, from, to, {
"ci.kind": check.kind, "ci.check": check.name, "ci.check_app": check.app,
"ci.status": check.status, error: failed(check.status),
"ci.start_offset_ms": from - start,
"ci.finishes_after_workflow": to > end,
});
exported++;
if (to > lastEnd) { lastEnd = to; lastName = check.name; }
if (!slowest || to - from > slowest.ms) slowest = { name: check.name, ms: to - from };
}
if (pending.length) warnings.push(`${pending.length} outside checks were still running at export`);
const receipts = new Set();
const invocations = new Map();
const valid = [];
let rejected = 0;
for (const record of records) {
const job = byName.get(record?.job);
if (!job || record.version !== 1 || String(record.runId) !== String(run.id)
|| String(record.attempt) !== String(run.run_attempt) || !text(record.invocation)) {
rejected++; continue;
}
if (record.kind === "job") { receipts.add(job.id); continue; }
const invocation = `${job.id}:${record.framework}:${record.invocation}`;
if (["expected", "start", "end"].includes(record.kind)) {
const state = invocations.get(invocation) || {};
state[record.kind] = true;
invocations.set(invocation, state);
continue;
}
if (!["file", "test"].includes(record.kind) || !text(record.id)
|| (record.kind === "file" ? !text(record.name) : typeof record.name !== "string")
|| !text(record.file) || !["vitest", "playwright", "node"].includes(record.framework)
|| !Number.isFinite(record.start) || !Number.isFinite(record.duration) || record.duration < 0
|| record.start < millis(job.started_at) - 1000
|| record.start + record.duration > millis(job.completed_at) + 1000) {
rejected++; continue;
}
valid.push({ record, job, invocation });
}
const recordId = ({ record, invocation }) => id(`${key}:${invocation}:${record.kind}:${record.id}`);
const files = new Map(valid.filter(({ record }) => record.kind === "file")
.map((entry) => [`${entry.invocation}:${entry.record.id}`, recordId(entry)]));
for (const entry of valid) {
const { record, job, invocation } = entry;
const jobId = roots.get(job.id);
// GitHub steps are ordered and have second-precision timestamps. Prefer
// the latest containing step: rounding may also make a preceding step fit
// a short file/test. Preserve the runner's own clock.
const step = job.steps?.findLast((candidate) => record.start >= millis(candidate.started_at)
&& record.start + record.duration <= millis(candidate.completed_at) + 1000
&& candidate.conclusion !== "skipped");
const parent = files.get(`${invocation}:${record.parent}`)
|| (step ? id(`${jobId}:step:${step.number}`) : jobId);
span(recordId(entry), parent, record.kind === "test" ? record.name.slice(0, 256) : record.name, record.start, record.start + record.duration, {
"ci.kind": record.kind, "ci.job": job.name, "test.framework": record.framework,
"test.file": record.file, "test.status": typeof record.status === "string" ? record.status.slice(0, 32) : "unknown", error: failed(record.status),
"test.retries": Number.isFinite(record.retries) ? record.retries : 0,
"test.clock": record.clock === "reporter-receipt" ? "reporter-receipt" : "runner",
...(Number.isFinite(record.collectMs) ? { "test.collect_ms": record.collectMs } : {}),
...(Number.isFinite(record.setupMs) ? { "test.setup_ms": record.setupMs } : {}),
});
}
// Ordinary runs skip the collection steps, so they owe no timing artifacts.
const expectedReceipts = testDetail
? jobs.filter((job) => job.steps?.some((step) => step.name === "Initialize CI profile")) : [];
const missing = expectedReceipts.filter((job) => !receipts.has(job.id));
if (missing.length) warnings.push(`Missing timing artifacts for ${missing.length} jobs`);
// Some package scripts invoke a reporter without using the launcher. Its
// start/end pair proves collection even when no intent record was written.
const incomplete = [...invocations.values()].filter((value) => !value.start || !value.end).length;
if (incomplete) warnings.push(`${incomplete} test invocations did not finish reporting`);
if (rejected) warnings.push(`${rejected} invalid timing records ignored`);
if (run.conclusion !== "success") warnings.push("Workflow did not succeed; later tests or steps may not have run");
if (!expectedReceipts.length) warnings.push("Job/step timings only; no test reporter collection steps were present");
const root = events[0].data;
const expectedFramework = (name) => {
if (/General tests|Verify serialized server suites|Verify Paperclip Runner \(vitest/.test(name)) return "vitest";
if (/e2e shard/.test(name)) return "playwright";
if (/\/ policy$|Typecheck \+ Release Registry|Verify Paperclip Runner \(static checks/.test(name)) return "node";
return null;
};
const missingTests = jobs.filter((job) => {
const framework = expectedFramework(job.name);
return framework && !valid.some((entry) => entry.job.id === job.id && entry.record.framework === framework);
});
if (expectedReceipts.length && missingTests.length) warnings.push(`Missing test detail for ${missingTests.length} test jobs (including jobs that failed before testing)`);
root["ci.sample_rate"] = sampleRate;
root["ci.workflow_ms"] = end - start;
// Push to the last finished check. Checks still running at export are excluded and counted.
root["ci.all_checks_ms"] = lastEnd - start;
root["ci.last_to_finish"] = lastName;
root["ci.check_count"] = exported;
root["ci.pending_check_count"] = pending.length;
root["ci.pending_checks"] = pending.map((check) => check.name).join(", ").slice(0, 1024);
root["ci.earlier_check_count"] = stale;
if (slowest) {
root["ci.slowest_check"] = slowest.name;
root["ci.slowest_check_ms"] = slowest.ms;
}
root["ci.test_count"] = valid.filter(({ record }) => record.kind === "test").length;
root["ci.file_count"] = valid.filter(({ record }) => record.kind === "file").length;
root["ci.expected_invocations"] = invocations.size;
root["ci.reported_invocations"] = [...invocations.values()].filter((value) => value.start && value.end).length;
root["ci.profile_complete"] = warnings.length === 0 && run.run_attempt === 1;
root["ci.coverage_warnings"] = warnings.join("; ");
return { traceId, events, warnings, wallMs: end - start, windowMs: lastEnd - start, jobCount: jobs.length, checkCount: exported };
}
export function traceUrl(team, environment, dataset, trace) {
const params = new URLSearchParams({
trace_id: trace.traceId,
trace_start_ts: String(Math.floor(Date.parse(trace.events[0].time) / 1000)),
trace_end_ts: String(Math.ceil((Date.parse(trace.events[0].time) + (trace.windowMs ?? trace.wallMs)) / 1000)),
});
return `https://ui.honeycomb.io/${encodeURIComponent(team)}/environments/${encodeURIComponent(environment)}/datasets/${encodeURIComponent(dataset)}/trace?${params}`;
}