mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Contributors need fast, predictable CI feedback. > - PR #16378 added a Honeycomb trace of the main PR workflow, but only for PRs with the `profile-ci` label. > - Ordinary CI time was therefore never recorded. All of the last 300 `CI profile` runs were skipped. > - Reviews and scans outside the `PR` workflow, such as Greptile, were not in the trace. > - This pull request exports one CODEOWNER PR run in ten without a label, and adds outside checks to every trace. > - The benefit is a continuous record of total CI time and of which job or check was slowest. The Honeycomb key stays out of reach of outside contributors. ## Linked Issues or Issue Description Refs #16378 **What existing behavior does this improve?** The Honeycomb CI trace that the `CI profile` workflow exports. **Current behavior** - The exporter runs only for a PR with the `profile-ci` label. The label also forces the full suite. - The trace has the jobs of the `PR` workflow only. Greptile, security scans, commit statuses and other workflows are absent. - The only total is the `PR` workflow time. **Proposed behavior** - One `PR` run in ten is exported without a label. The run number must end in 0. The trace has every job and step. Each span has a sample rate of 10. - Each trace has one span for each outside check and each other workflow on the head commit. - The root span has `ci.workflow_ms`, `ci.all_checks_ms`, `ci.last_to_finish` and `ci.slowest_check`. **Reason and benefit** Maintainers can see how long CI takes and what is slow, for example a long Greptile review, without a label on each PR. Sampling keeps the volume near 1.7 million events each month. Export of every run is near 17 million. ## What Changed - `scripts/ci-profile/authorize.mjs`: add `exportMode` (label or one-in-ten sample) and `associatedOwnerPr` (the CODEOWNER checks without the label). - `scripts/ci-profile/trace.mjs`: add `normalizeChecks` and spans for outside checks. Add the sample rate and the new root fields. Sampled runs do not warn about absent test artifacts. - `scripts/ci-profile/export.mjs`: select the mode. Read check runs, commit statuses and workflow runs for the head commit. Wait a maximum of 15 minutes for outside apps that are not complete. Sampled runs read no artifacts. - `.github/workflows/ci-profile.yml`: no new job. The job condition now requires a branch of this repository, an actor and rerun actor on the `AWS_CI_ROUTING_USER_IDS` roster, and a labelled or sampled run. Add `checks: read` and `statuses: read`. Increase the timeout to 30 minutes for the bounded wait. - `doc/ci-profiling.md` and `doc/DEVELOPING.md`: describe sampled runs, outside checks, the totals and the job condition. ## Verification - `node --test .github/scripts/tests/ci-profile.test.mjs .github/scripts/tests/ci-routing.test.mjs`: 163 pass, 0 fail. - `actionlint .github/workflows/ci-profile.yml`: no findings. - Dry runs against real runs with the real CODEOWNERS check (`node scripts/ci-profile/export.mjs --run-id=ID --attempt=1 --dry-run`): - A sampled CODEOWNER run (run number 64780): 1,144 spans, sample rate 10, 5 outside checks and 1 other workflow. - A sampled run from an outside contributor (run number 64800): "No CODEOWNERS-authorized PR at this revision; no data exported." - A CODEOWNER run that is not in the sample: "This attempt was not selected for profiling; no data exported." - New tests cover: the sample rule, fork and outsider PRs, a bot rerun actor, a stale head, outside-check spans and totals, the bounded wait, and the job condition. - Not run locally: `scripts/ci-profile/reporters.test.mjs`. It needs installed dependencies, and this change does not touch the reporters. CI runs it. - No data was sent to Honeycomb before merge. The workflow runs from `master`, so the first real export occurs after merge. ## Risks - The label path now also needs the `AWS_CI_ROUTING_USER_IDS` roster. If that variable is empty, nothing is exported. All four CI CODEOWNERS are on the roster today. - The roster is only a first filter. The exporter still checks the PR author and both actors against CODEOWNERS on `master` before it reads or sends data. The key stays in the `ci-profiling` environment, which permits only `master`. - The export job can now run for a maximum of 30 minutes on a GitHub-hosted runner while it waits for an outside check. - GitHub gives no completion time for a workflow run. The span for another workflow ends at the last update of that run. - Honeycomb volume increases by about 1.7 million events each month. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code, with tool use, code execution and repository inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
168 lines
9.8 KiB
JavaScript
168 lines
9.8 KiB
JavaScript
import { execFileSync } from "node:child_process";
|
|
import { appendFileSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
|
import { tmpdir } from "node:os";
|
|
import path from "node:path";
|
|
import { fileURLToPath } from "node:url";
|
|
import { buildTrace, normalizeChecks, pendingChecks, traceUrl } from "./trace.mjs";
|
|
import { associatedOwnerPr, associatedProfilePr, authorizedActors, exportMode, sampleRate, trustedOwnerIds } from "./authorize.mjs";
|
|
|
|
const repository = "paperclipai/paperclip";
|
|
const repoPath = `/repos/${repository}`;
|
|
const maxBytes = 32 * 1024 * 1024;
|
|
const gh = (args) => execFileSync("gh", ["api", ...args], { maxBuffer: maxBytes, timeout: 60_000 });
|
|
const get = (endpoint) => JSON.parse(gh([endpoint]).toString());
|
|
const pages = (endpoint, field) => JSON.parse(gh(["--paginate", "--slurp", endpoint]).toString())
|
|
.flatMap((page) => field ? page[field] : page);
|
|
|
|
export async function sendEvents(events, apiKey, dataset, fetcher = fetch, pause = (ms) => new Promise((resolve) => setTimeout(resolve, ms))) {
|
|
if (!apiKey) throw new Error("HONEYCOMB_CI_API_KEY is not configured");
|
|
const transient = new Set([429, 500, 502, 503, 504]);
|
|
for (let offset = 0; offset < events.length; offset += 250) {
|
|
let pending = events.slice(offset, offset + 250);
|
|
for (let attempt = 0; pending.length; attempt += 1) {
|
|
if (attempt >= 8) throw new Error(`Honeycomb did not accept ${pending.length} spans after retries`);
|
|
const response = await fetcher(`https://api.honeycomb.io/1/batch/${encodeURIComponent(dataset)}`, {
|
|
method: "POST", headers: { "X-Honeycomb-Team": apiKey, "Content-Type": "application/json" },
|
|
body: JSON.stringify(pending), redirect: "error", signal: AbortSignal.timeout(30_000),
|
|
});
|
|
if (!response.ok) {
|
|
if (!transient.has(response.status)) throw new Error(`Honeycomb export failed (HTTP ${response.status})`);
|
|
await pause(Math.min(30_000, 1000 * 2 ** attempt));
|
|
continue;
|
|
}
|
|
const results = await response.json();
|
|
if (!Array.isArray(results) || results.length !== pending.length) {
|
|
throw new Error("Honeycomb returned an unexpected batch response");
|
|
}
|
|
const rejected = results.flatMap((result, index) => result.status === 202 ? [] : [{ event: pending[index], status: result.status }]);
|
|
const permanent = rejected.find(({ status }) => !transient.has(status));
|
|
if (permanent) throw new Error(`Honeycomb rejected ${rejected.length} spans (status ${permanent.status})`);
|
|
pending = rejected.map(({ event }) => event);
|
|
if (pending.length) await pause(Math.min(30_000, 1000 * 2 ** attempt));
|
|
}
|
|
// The full suite can exceed Honeycomb's burst allowance. Leave capacity
|
|
// for the team's other telemetry rather than exhausting it in one export.
|
|
if (offset + 250 < events.length) await pause(1000);
|
|
}
|
|
}
|
|
|
|
// Reviews and scans from outside apps can outlast the workflow. Wait a bounded
|
|
// time so the trace records how long they took; whatever is still running
|
|
// after that is reported as pending instead of holding the export open.
|
|
export async function settledChecks(run, load, { waitMs = 15 * 60_000, intervalMs = 30_000,
|
|
pause = (ms) => new Promise((resolve) => setTimeout(resolve, ms)), now = Date.now } = {}) {
|
|
// The deadline is on the wall clock, so slow GitHub responses count against it.
|
|
const deadline = now() + waitMs;
|
|
let checks = load();
|
|
while (now() < deadline && pendingChecks(run, checks).length) {
|
|
await pause(Math.min(intervalMs, deadline - now()));
|
|
checks = load();
|
|
}
|
|
return checks;
|
|
}
|
|
|
|
async function main() {
|
|
const args = process.argv.slice(2);
|
|
const arg = (name) => args.find((value) => value.startsWith(`${name}=`))?.slice(name.length + 1);
|
|
const event = process.env.GITHUB_EVENT_PATH ? JSON.parse(readFileSync(process.env.GITHUB_EVENT_PATH, "utf8")) : null;
|
|
const runId = event?.workflow_run?.id ?? arg("--run-id");
|
|
const attempt = event?.workflow_run?.run_attempt ?? arg("--attempt");
|
|
if (!/^\d+$/.test(String(runId)) || !/^[1-9]\d*$/.test(String(attempt))) {
|
|
throw new Error("Provide --run-id and --attempt, or a workflow_run event");
|
|
}
|
|
const run = get(`${repoPath}/actions/runs/${runId}/attempts/${attempt}`);
|
|
if (run.repository?.full_name !== repository || run.event !== "pull_request"
|
|
|| run.path !== ".github/workflows/pr.yml" || run.status !== "completed") {
|
|
throw new Error("Only completed main PR workflow attempts can be profiled");
|
|
}
|
|
// GitHub often leaves workflow_run.pull_requests empty for fork PRs. Resolve
|
|
// association through the head commit, then validate repo and exact head.
|
|
const pulls = pages(`${repoPath}/commits/${run.head_sha}/pulls?per_page=100`);
|
|
const historical = args.includes("--historical");
|
|
if (historical && process.env.GITHUB_ACTIONS) throw new Error("Historical export is a local-only operation");
|
|
const ownerIds = historical ? [] : trustedOwnerIds(get);
|
|
if (process.env.GITHUB_ACTIONS) {
|
|
const exporterId = process.env.GITHUB_RUN_ID;
|
|
const exporterAttempt = process.env.GITHUB_RUN_ATTEMPT;
|
|
if (!/^\d+$/.test(exporterId) || !/^[1-9]\d*$/.test(exporterAttempt)) throw new Error("Invalid exporter run identity");
|
|
const exporter = get(`${repoPath}/actions/runs/${exporterId}/attempts/${exporterAttempt}`);
|
|
if (!authorizedActors(exporter, ownerIds)) {
|
|
console.log("Exporter actor is not a CI CODEOWNER; no data exported.");
|
|
return;
|
|
}
|
|
}
|
|
// The run name snapshots label selection at dispatch. A label added while
|
|
// an ordinary run finishes must not retroactively select that earlier run;
|
|
// such a run is exported only if its run number is in the sample.
|
|
const mode = historical ? "historical" : exportMode(run);
|
|
if (!mode) {
|
|
console.log("This attempt was not selected for profiling; no data exported.");
|
|
return;
|
|
}
|
|
const pr = mode === "label" ? associatedProfilePr(run, pulls, ownerIds)
|
|
: mode === "sampled" ? associatedOwnerPr(run, pulls, ownerIds) : undefined;
|
|
if (!pr && !historical) {
|
|
console.log("No CODEOWNERS-authorized PR at this revision; no data exported.");
|
|
return;
|
|
}
|
|
const jobs = pages(`${repoPath}/actions/runs/${runId}/attempts/${attempt}/jobs?per_page=100`, "jobs");
|
|
// Sampled runs wrote no timing artifacts, so none are read.
|
|
const artifacts = mode === "sampled" ? [] : pages(`${repoPath}/actions/runs/${runId}/artifacts?per_page=100`, "artifacts")
|
|
.filter((artifact) => artifact.name.startsWith(`ci-profile-${attempt}-`));
|
|
if (!/^[a-f\d]{40}$/.test(run.head_sha)) throw new Error("Invalid head commit");
|
|
const loadChecks = () => normalizeChecks(run, {
|
|
checkRuns: pages(`${repoPath}/commits/${run.head_sha}/check-runs?per_page=100`, "check_runs"),
|
|
statuses: pages(`${repoPath}/commits/${run.head_sha}/statuses?per_page=100`),
|
|
workflowRuns: pages(`${repoPath}/actions/runs?head_sha=${run.head_sha}&per_page=100`, "workflow_runs"),
|
|
});
|
|
const checks = await settledChecks(run, loadChecks, args.includes("--dry-run") ? { waitMs: 0 } : {});
|
|
const directory = mkdtempSync(path.join(tmpdir(), "paperclip-ci-export-"));
|
|
const records = [];
|
|
let bytes = 0;
|
|
try {
|
|
for (const artifact of artifacts) {
|
|
if (artifact.expired || artifact.size_in_bytes > maxBytes) throw new Error("Timing artifact is expired or too large");
|
|
const archive = path.join(directory, `${artifact.id}.zip`);
|
|
writeFileSync(archive, gh([`${repoPath}/actions/artifacts/${artifact.id}/zip`]));
|
|
const decoded = execFileSync("python3", [fileURLToPath(new URL("./read-artifact.py", import.meta.url)), archive], {
|
|
encoding: "utf8", maxBuffer: maxBytes, timeout: 30_000,
|
|
});
|
|
bytes += Buffer.byteLength(decoded);
|
|
if (bytes > 128 * 1024 * 1024) throw new Error("Run timing artifacts exceed 128 MiB; refusing to truncate the trace");
|
|
for (const line of decoded.trim().split("\n").filter(Boolean)) records.push(JSON.parse(line));
|
|
}
|
|
} finally {
|
|
rmSync(directory, { recursive: true, force: true });
|
|
}
|
|
const trace = buildTrace(run, jobs, records, {
|
|
sampleRate: mode === "sampled" ? sampleRate : 1, checks, testDetail: mode !== "sampled",
|
|
});
|
|
trace.events[0].data["ci.profile_source"] = mode;
|
|
if (pr) trace.events[0].data["ci.pr_number"] = pr.number;
|
|
const dataset = process.env.HONEYCOMB_CI_DATASET || "paperclip-ci";
|
|
const url = traceUrl(process.env.HONEYCOMB_TEAM || "paperclip", process.env.HONEYCOMB_ENVIRONMENT || "test", dataset, trace);
|
|
if (arg("--output")) writeFileSync(arg("--output"), JSON.stringify(trace, null, 2));
|
|
if (!args.includes("--dry-run")) {
|
|
// Publish the completed root only after every child has been accepted.
|
|
// A failed partial export must not advertise complete coverage.
|
|
await sendEvents(trace.events.slice(1), process.env.HONEYCOMB_CI_API_KEY, dataset);
|
|
await sendEvents([trace.events[0]], process.env.HONEYCOMB_CI_API_KEY, dataset);
|
|
}
|
|
const summary = [
|
|
`### CI profile${pr ? ` for PR #${pr.number}` : " (historical job/step baseline)"}`,
|
|
"", `[Open the full CI waterfall in Honeycomb](${url})`, "",
|
|
`${trace.jobCount} jobs · ${(trace.wallMs / 1000).toFixed(1)} seconds elapsed · ${trace.events.length} spans`,
|
|
"", `${trace.checkCount} outside checks · ${(trace.windowMs / 1000).toFixed(1)} seconds until the last one finished`,
|
|
...(mode === "sampled" ? ["", `Sampled run (1 in ${sampleRate}): job and step timings, without per-test detail.`] : []),
|
|
"", `GitHub run: ${run.html_url}/attempts/${attempt}`, "",
|
|
...trace.warnings.map((warning) => `- ${warning}`),
|
|
"", "The workflow root measures elapsed time. Parallel job durations must not be summed as elapsed time.", "",
|
|
].join("\n");
|
|
console.log(summary);
|
|
if (process.env.GITHUB_STEP_SUMMARY) appendFileSync(process.env.GITHUB_STEP_SUMMARY, summary);
|
|
}
|
|
|
|
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
|
main().catch((error) => { console.error(error.message); process.exitCode = 1; });
|
|
}
|