mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Contributors need fast, predictable CI feedback. > - PR #16378 added a Honeycomb trace of the main PR workflow, but only for PRs with the `profile-ci` label. > - Ordinary CI time was therefore never recorded. All of the last 300 `CI profile` runs were skipped. > - Reviews and scans outside the `PR` workflow, such as Greptile, were not in the trace. > - This pull request exports one CODEOWNER PR run in ten without a label, and adds outside checks to every trace. > - The benefit is a continuous record of total CI time and of which job or check was slowest. The Honeycomb key stays out of reach of outside contributors. ## Linked Issues or Issue Description Refs #16378 **What existing behavior does this improve?** The Honeycomb CI trace that the `CI profile` workflow exports. **Current behavior** - The exporter runs only for a PR with the `profile-ci` label. The label also forces the full suite. - The trace has the jobs of the `PR` workflow only. Greptile, security scans, commit statuses and other workflows are absent. - The only total is the `PR` workflow time. **Proposed behavior** - One `PR` run in ten is exported without a label. The run number must end in 0. The trace has every job and step. Each span has a sample rate of 10. - Each trace has one span for each outside check and each other workflow on the head commit. - The root span has `ci.workflow_ms`, `ci.all_checks_ms`, `ci.last_to_finish` and `ci.slowest_check`. **Reason and benefit** Maintainers can see how long CI takes and what is slow, for example a long Greptile review, without a label on each PR. Sampling keeps the volume near 1.7 million events each month. Export of every run is near 17 million. ## What Changed - `scripts/ci-profile/authorize.mjs`: add `exportMode` (label or one-in-ten sample) and `associatedOwnerPr` (the CODEOWNER checks without the label). - `scripts/ci-profile/trace.mjs`: add `normalizeChecks` and spans for outside checks. Add the sample rate and the new root fields. Sampled runs do not warn about absent test artifacts. - `scripts/ci-profile/export.mjs`: select the mode. Read check runs, commit statuses and workflow runs for the head commit. Wait a maximum of 15 minutes for outside apps that are not complete. Sampled runs read no artifacts. - `.github/workflows/ci-profile.yml`: no new job. The job condition now requires a branch of this repository, an actor and rerun actor on the `AWS_CI_ROUTING_USER_IDS` roster, and a labelled or sampled run. Add `checks: read` and `statuses: read`. Increase the timeout to 30 minutes for the bounded wait. - `doc/ci-profiling.md` and `doc/DEVELOPING.md`: describe sampled runs, outside checks, the totals and the job condition. ## Verification - `node --test .github/scripts/tests/ci-profile.test.mjs .github/scripts/tests/ci-routing.test.mjs`: 163 pass, 0 fail. - `actionlint .github/workflows/ci-profile.yml`: no findings. - Dry runs against real runs with the real CODEOWNERS check (`node scripts/ci-profile/export.mjs --run-id=ID --attempt=1 --dry-run`): - A sampled CODEOWNER run (run number 64780): 1,144 spans, sample rate 10, 5 outside checks and 1 other workflow. - A sampled run from an outside contributor (run number 64800): "No CODEOWNERS-authorized PR at this revision; no data exported." - A CODEOWNER run that is not in the sample: "This attempt was not selected for profiling; no data exported." - New tests cover: the sample rule, fork and outsider PRs, a bot rerun actor, a stale head, outside-check spans and totals, the bounded wait, and the job condition. - Not run locally: `scripts/ci-profile/reporters.test.mjs`. It needs installed dependencies, and this change does not touch the reporters. CI runs it. - No data was sent to Honeycomb before merge. The workflow runs from `master`, so the first real export occurs after merge. ## Risks - The label path now also needs the `AWS_CI_ROUTING_USER_IDS` roster. If that variable is empty, nothing is exported. All four CI CODEOWNERS are on the roster today. - The roster is only a first filter. The exporter still checks the PR author and both actors against CODEOWNERS on `master` before it reads or sends data. The key stays in the `ci-profiling` environment, which permits only `master`. - The export job can now run for a maximum of 30 minutes on a GitHub-hosted runner while it waits for an outside check. - GitHub gives no completion time for a workflow run. The span for another workflow ends at the last update of that run. - Honeycomb volume increases by about 1.7 million events each month. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code, with tool use, code execution and repository inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
92 lines
4.6 KiB
JavaScript
92 lines
4.6 KiB
JavaScript
import { execFileSync } from "node:child_process";
|
|
import { appendFileSync, readFileSync } from "node:fs";
|
|
import path from "node:path";
|
|
import { fileURLToPath } from "node:url";
|
|
|
|
const repository = "paperclipai/paperclip";
|
|
const repositoryId = 1170821064;
|
|
export const getGithub = (endpoint) => JSON.parse(execFileSync("gh", ["api", endpoint], {
|
|
encoding: "utf8", maxBuffer: 4 * 1024 * 1024, timeout: 30_000,
|
|
}));
|
|
|
|
// This is deliberately the CI infrastructure owner list, not the union of
|
|
// unrelated documentation owners. Unsupported teams/ambiguous rules deny access.
|
|
export function ciOwners(contents) {
|
|
const rules = contents.split(/\r?\n/).map((line) => line.split("#")[0].trim().split(/\s+/))
|
|
.filter(([pattern]) => pattern === ".github/**" || pattern === "/.github/**");
|
|
if (rules.length !== 1 || rules[0].length < 2
|
|
|| rules[0].slice(1).some((owner) => !/^@[a-z\d](?:[a-z\d-]*[a-z\d])?$/i.test(owner))) {
|
|
throw new Error("CI CODEOWNERS must have one explicit .github/** rule with individual GitHub users");
|
|
}
|
|
return rules[0].slice(1).map((owner) => owner.slice(1));
|
|
}
|
|
|
|
export function trustedOwnerIds(get = getGithub, trustedSha) {
|
|
const sha = trustedSha || get(`/repos/${repository}/git/ref/heads/master`).object.sha;
|
|
if (!/^[a-f\d]{40}$/.test(sha)) throw new Error("Invalid trusted CODEOWNERS revision");
|
|
const file = get(`/repos/${repository}/contents/.github/CODEOWNERS?ref=${sha}`);
|
|
if (file.encoding !== "base64") throw new Error("Cannot read trusted CODEOWNERS");
|
|
return ciOwners(Buffer.from(file.content, "base64").toString("utf8")).map((login) => {
|
|
const user = get(`/users/${login}`);
|
|
if (user.type !== "User" || !Number.isSafeInteger(user.id) || user.id <= 0) {
|
|
throw new Error("Cannot resolve a CI CODEOWNER identity");
|
|
}
|
|
return user.id;
|
|
});
|
|
}
|
|
|
|
export function authorizedActors(run, ownerIds) {
|
|
const allowed = (user) => user?.type === "User" && ownerIds.includes(user.id);
|
|
return run.repository?.id === repositoryId && run.repository?.full_name === repository
|
|
&& allowed(run.actor) && allowed(run.triggering_actor);
|
|
}
|
|
|
|
// The PR at this exact revision, when its author, the run's actor and the
|
|
// rerun actor are all CI CODEOWNERS.
|
|
export function associatedOwnerPr(run, pulls, ownerIds, accept = () => true) {
|
|
if (!authorizedActors(run, ownerIds)) return undefined;
|
|
return pulls.find((pr) => pr.base.repo.id === repositoryId && pr.base.repo.full_name === repository
|
|
&& pr.head.repo?.id === run.head_repository?.id && pr.head.sha === run.head_sha
|
|
&& pr.user?.type === "User" && ownerIds.includes(pr.user.id) && accept(pr));
|
|
}
|
|
|
|
export function associatedProfilePr(run, pulls, ownerIds) {
|
|
return associatedOwnerPr(run, pulls, ownerIds, (pr) => pr.labels.some((label) => label.name === "profile-ci"));
|
|
}
|
|
|
|
// Always-on export keeps one run in this many, chosen by the workflow's
|
|
// sequential run number so that ci-profile.yml can skip the others before
|
|
// starting a runner.
|
|
export const sampleRate = 10;
|
|
|
|
// "label": a profile-ci run with test detail, always exported.
|
|
// "sampled": an ordinary run chosen by its number, job and step timings only.
|
|
export function exportMode(run) {
|
|
if (typeof run.display_title === "string" && run.display_title.startsWith("CI profile · PR #")) return "label";
|
|
return Number.isSafeInteger(run.run_number) && run.run_number > 0 && run.run_number % sampleRate === 0 ? "sampled" : undefined;
|
|
}
|
|
|
|
async function main() {
|
|
let enabled = false;
|
|
try {
|
|
const event = JSON.parse(readFileSync(process.env.GITHUB_EVENT_PATH, "utf8"));
|
|
if (event.pull_request?.labels?.some((label) => label.name === "profile-ci")) {
|
|
const runId = process.env.GITHUB_RUN_ID;
|
|
const attempt = process.env.GITHUB_RUN_ATTEMPT;
|
|
if (!/^\d+$/.test(runId) || !/^[1-9]\d*$/.test(attempt)) throw new Error("Invalid CI run identity");
|
|
const run = getGithub(`/repos/${repository}/actions/runs/${runId}/attempts/${attempt}`);
|
|
const pr = getGithub(`/repos/${repository}/pulls/${event.pull_request.number}`);
|
|
enabled = run.event === "pull_request" && run.path === ".github/workflows/pr.yml"
|
|
&& Boolean(associatedProfilePr(run, [pr], trustedOwnerIds(getGithub, process.env.TRUSTED_PROFILE_SHA)));
|
|
}
|
|
} catch {
|
|
console.log("::notice::CI profiling authorization could not be verified; profiling disabled.");
|
|
}
|
|
appendFileSync(process.env.GITHUB_OUTPUT, `profile_ci=${enabled}\n`);
|
|
console.log(`::notice::CI CODEOWNERS profiling authorization: ${enabled}`);
|
|
}
|
|
|
|
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
|
main().catch(() => { console.error("CI profiling authorization failed"); process.exitCode = 1; });
|
|
}
|