mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Contributors need fast, predictable CI feedback. > - PR #16378 added a Honeycomb trace of the main PR workflow, but only for PRs with the `profile-ci` label. > - Ordinary CI time was therefore never recorded. All of the last 300 `CI profile` runs were skipped. > - Reviews and scans outside the `PR` workflow, such as Greptile, were not in the trace. > - This pull request exports one CODEOWNER PR run in ten without a label, and adds outside checks to every trace. > - The benefit is a continuous record of total CI time and of which job or check was slowest. The Honeycomb key stays out of reach of outside contributors. ## Linked Issues or Issue Description Refs #16378 **What existing behavior does this improve?** The Honeycomb CI trace that the `CI profile` workflow exports. **Current behavior** - The exporter runs only for a PR with the `profile-ci` label. The label also forces the full suite. - The trace has the jobs of the `PR` workflow only. Greptile, security scans, commit statuses and other workflows are absent. - The only total is the `PR` workflow time. **Proposed behavior** - One `PR` run in ten is exported without a label. The run number must end in 0. The trace has every job and step. Each span has a sample rate of 10. - Each trace has one span for each outside check and each other workflow on the head commit. - The root span has `ci.workflow_ms`, `ci.all_checks_ms`, `ci.last_to_finish` and `ci.slowest_check`. **Reason and benefit** Maintainers can see how long CI takes and what is slow, for example a long Greptile review, without a label on each PR. Sampling keeps the volume near 1.7 million events each month. Export of every run is near 17 million. ## What Changed - `scripts/ci-profile/authorize.mjs`: add `exportMode` (label or one-in-ten sample) and `associatedOwnerPr` (the CODEOWNER checks without the label). - `scripts/ci-profile/trace.mjs`: add `normalizeChecks` and spans for outside checks. Add the sample rate and the new root fields. Sampled runs do not warn about absent test artifacts. - `scripts/ci-profile/export.mjs`: select the mode. Read check runs, commit statuses and workflow runs for the head commit. Wait a maximum of 15 minutes for outside apps that are not complete. Sampled runs read no artifacts. - `.github/workflows/ci-profile.yml`: no new job. The job condition now requires a branch of this repository, an actor and rerun actor on the `AWS_CI_ROUTING_USER_IDS` roster, and a labelled or sampled run. Add `checks: read` and `statuses: read`. Increase the timeout to 30 minutes for the bounded wait. - `doc/ci-profiling.md` and `doc/DEVELOPING.md`: describe sampled runs, outside checks, the totals and the job condition. ## Verification - `node --test .github/scripts/tests/ci-profile.test.mjs .github/scripts/tests/ci-routing.test.mjs`: 163 pass, 0 fail. - `actionlint .github/workflows/ci-profile.yml`: no findings. - Dry runs against real runs with the real CODEOWNERS check (`node scripts/ci-profile/export.mjs --run-id=ID --attempt=1 --dry-run`): - A sampled CODEOWNER run (run number 64780): 1,144 spans, sample rate 10, 5 outside checks and 1 other workflow. - A sampled run from an outside contributor (run number 64800): "No CODEOWNERS-authorized PR at this revision; no data exported." - A CODEOWNER run that is not in the sample: "This attempt was not selected for profiling; no data exported." - New tests cover: the sample rule, fork and outsider PRs, a bot rerun actor, a stale head, outside-check spans and totals, the bounded wait, and the job condition. - Not run locally: `scripts/ci-profile/reporters.test.mjs`. It needs installed dependencies, and this change does not touch the reporters. CI runs it. - No data was sent to Honeycomb before merge. The workflow runs from `master`, so the first real export occurs after merge. ## Risks - The label path now also needs the `AWS_CI_ROUTING_USER_IDS` roster. If that variable is empty, nothing is exported. All four CI CODEOWNERS are on the roster today. - The roster is only a first filter. The exporter still checks the PR author and both actors against CODEOWNERS on `master` before it reads or sends data. The key stays in the `ci-profiling` environment, which permits only `master`. - The export job can now run for a maximum of 30 minutes on a GitHub-hosted runner while it waits for an outside check. - GitHub gives no completion time for a workflow run. The span for another workflow ends at the last update of that run. - Honeycomb volume increases by about 1.7 million events each month. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code, with tool use, code execution and repository inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
501 lines
30 KiB
JavaScript
501 lines
30 KiB
JavaScript
import assert from "node:assert/strict";
|
|
import test from "node:test";
|
|
import { mkdtempSync, mkdirSync, readFileSync, existsSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
|
|
import { execFileSync, spawnSync } from "node:child_process";
|
|
import { tmpdir } from "node:os";
|
|
import path from "node:path";
|
|
import { fileURLToPath } from "node:url";
|
|
import { buildTrace, normalizeChecks, pendingChecks, traceUrl } from "../../../scripts/ci-profile/trace.mjs";
|
|
import { sendEvents, settledChecks } from "../../../scripts/ci-profile/export.mjs";
|
|
import { associatedOwnerPr, associatedProfilePr, ciOwners, exportMode, sampleRate, trustedOwnerIds } from "../../../scripts/ci-profile/authorize.mjs";
|
|
|
|
const at = (seconds) => new Date(1_800_000_000_000 + seconds * 1000).toISOString();
|
|
const run = {
|
|
id: 42, run_attempt: 1, run_number: 7, status: "completed", conclusion: "failure",
|
|
repository: { id: 1170821064, full_name: "paperclipai/paperclip" }, head_repository: { id: 99 },
|
|
actor: { id: 123, type: "User" }, triggering_actor: { id: 123, type: "User" },
|
|
head_sha: "abc", head_branch: "branch", created_at: at(0), run_started_at: at(0),
|
|
html_url: "https://github.com/paperclipai/paperclip/actions/runs/42",
|
|
};
|
|
const job = (id, name, start, end) => ({
|
|
id, run_id: 42, run_attempt: 1, name: `ci / ${name}`, status: "completed",
|
|
conclusion: "success", created_at: at(1), started_at: at(start), completed_at: at(end),
|
|
labels: ["ubuntu-latest"], steps: [
|
|
{ number: 1, name: "Initialize CI profile", started_at: at(start), completed_at: at(start + 1), conclusion: "success" },
|
|
{ number: 2, name: "Tests", started_at: at(start + 1), completed_at: at(end), conclusion: "success" },
|
|
],
|
|
});
|
|
const jobs = [job(1, "first", 2, 12), job(2, "second", 3, 20)];
|
|
const record = (fields) => ({ version: 1, runId: "42", attempt: "1", invocation: "invocation", job: "first", framework: "vitest", ...fields });
|
|
|
|
test("one wall-clock trace includes parallel jobs, wait, and every step", () => {
|
|
const trace = buildTrace(run, jobs);
|
|
assert.equal(trace.wallMs, 20000);
|
|
assert.equal(trace.events[0].data.duration_ms, 20000);
|
|
assert.equal(trace.events.filter((event) => event.data["ci.kind"] === "job").length, 2);
|
|
assert.equal(trace.events.filter((event) => event.data["ci.kind"] === "step").length, 4);
|
|
assert.equal(trace.events.filter((event) => event.data["ci.kind"] === "wait").length, 2);
|
|
assert.equal(new Set(trace.events.map((event) => event.data["trace.trace_id"])).size, 1);
|
|
assert.equal(trace.events.find((event) => event.data.name === "ci / second").data["ci.finishes_last"], true);
|
|
assert.equal(trace.events[0].data.error, true);
|
|
});
|
|
|
|
test("file/test hierarchy keeps runner timestamps and rejects unrelated artifact data", () => {
|
|
const records = [
|
|
record({ kind: "job" }), record({ kind: "job", job: "second" }),
|
|
record({ kind: "start" }), record({ kind: "end" }),
|
|
record({ kind: "file", id: "file", name: "a.test.ts", file: "a.test.ts", start: Date.parse(at(4)), duration: 6000, status: "passed" }),
|
|
record({ kind: "test", id: "case", parent: "file", name: "runs", file: "a.test.ts", start: Date.parse(at(5)), duration: 100, status: "passed", secret: "must not leave artifact" }),
|
|
record({ kind: "test", id: "foreign", runId: "41", name: "wrong run" }),
|
|
];
|
|
const trace = buildTrace(run, jobs, records);
|
|
const file = trace.events.find((event) => event.data.name === "a.test.ts");
|
|
const entry = trace.events.find((event) => event.data.name === "runs");
|
|
assert.equal(entry.time, at(5));
|
|
assert.equal(entry.data["trace.parent_id"], file.data["trace.span_id"]);
|
|
assert.equal(file.data["trace.parent_id"], trace.events.find((event) => event.data.name === "Tests").data["trace.span_id"]);
|
|
assert.equal(trace.events.some((event) => event.data.name === "wrong run"), false);
|
|
assert.doesNotMatch(JSON.stringify(trace), /must not leave artifact/);
|
|
assert.equal(trace.events[0].data["ci.profile_complete"], false);
|
|
assert.match(trace.warnings.join(), /1 invalid/);
|
|
});
|
|
|
|
test("short files belong to the latest containing step despite second-precision boundaries", () => {
|
|
const records = [record({ kind: "file", id: "file", name: "short.test.ts", file: "short.test.ts",
|
|
start: Date.parse(at(3)) + 100, duration: 50, status: "passed" })];
|
|
const trace = buildTrace(run, jobs, records);
|
|
const file = trace.events.find((event) => event.data.name === "short.test.ts");
|
|
const step = trace.events.find((event) => event.data.name === "Tests");
|
|
assert.equal(file.data["trace.parent_id"], step.data["trace.span_id"]);
|
|
});
|
|
|
|
test("cancelled reporters and missing shards remain explicit", () => {
|
|
const trace = buildTrace({ ...run, conclusion: "cancelled" }, jobs, [record({ kind: "job" }), record({ kind: "start" })]);
|
|
assert.equal(trace.events[0].data["ci.profile_complete"], false);
|
|
assert.match(trace.warnings.join(), /Missing timing artifacts for 1 jobs/);
|
|
assert.match(trace.warnings.join(), /1 test invocations/);
|
|
});
|
|
|
|
test("a completely absent reporter invocation cannot hide behind a successful sibling", () => {
|
|
const records = [
|
|
record({ kind: "job" }), record({ kind: "job", job: "second" }),
|
|
record({ kind: "expected" }), record({ kind: "start" }), record({ kind: "end" }),
|
|
record({ kind: "file", id: "file", name: "a.test.ts", file: "a.test.ts", start: Date.parse(at(4)), duration: 100, status: "passed" }),
|
|
record({ kind: "expected", invocation: "reporter-never-loaded" }),
|
|
];
|
|
const trace = buildTrace(run, jobs, records);
|
|
assert.equal(trace.events[0].data["ci.profile_complete"], false);
|
|
assert.equal(trace.events[0].data["ci.expected_invocations"], 2);
|
|
assert.equal(trace.events[0].data["ci.reported_invocations"], 1);
|
|
assert.match(trace.warnings.join(), /1 test invocations/);
|
|
});
|
|
|
|
test("coverage is conservative after failures and does not merge framework invocation IDs", () => {
|
|
const records = [
|
|
record({ kind: "job" }), record({ kind: "job", job: "second" }),
|
|
record({ kind: "expected" }), record({ kind: "start" }), record({ kind: "end" }),
|
|
];
|
|
assert.equal(buildTrace({ ...run, conclusion: "success" }, jobs, records).events[0].data["ci.profile_complete"], true);
|
|
assert.equal(buildTrace(run, jobs, records).events[0].data["ci.profile_complete"], false);
|
|
const trace = buildTrace({ ...run, conclusion: "success" }, jobs, [
|
|
...records, record({ framework: "node", kind: "expected" }),
|
|
]);
|
|
assert.equal(trace.events[0].data["ci.profile_complete"], false);
|
|
assert.equal(trace.events[0].data["ci.expected_invocations"], 2);
|
|
assert.equal(trace.events[0].data["ci.reported_invocations"], 1);
|
|
});
|
|
|
|
test("complete reporter pairs count without launcher intent, and long test names remain timed", () => {
|
|
const records = [
|
|
record({ kind: "job" }), record({ kind: "job", job: "second" }),
|
|
record({ framework: "node", kind: "start" }), record({ framework: "node", kind: "end" }),
|
|
record({ kind: "test", id: "long", parent: "file", name: "x".repeat(5000),
|
|
file: "a.test.ts", start: Date.parse(at(4)), duration: 100, status: "passed" }),
|
|
record({ kind: "file", id: "file", name: "a.test.ts", file: "a.test.ts",
|
|
start: Date.parse(at(4)), duration: 100, status: "passed" }),
|
|
];
|
|
const trace = buildTrace({ ...run, conclusion: "success" }, jobs, records);
|
|
assert.equal(trace.events[0].data["ci.profile_complete"], true);
|
|
assert.equal(trace.events[0].data["ci.test_count"], 1);
|
|
assert.equal(trace.events.find((event) => event.data["ci.kind"] === "test").data.name.length, 256);
|
|
});
|
|
|
|
test("trusted policy commands work with both current and pre-profile checkouts", () => {
|
|
const source = readFileSync(new URL("../../workflows/pr-trusted.yml", import.meta.url), "utf8");
|
|
const block = source.split(" - name: Select compatible Node test command\n")[1].split("\n - name:")[0];
|
|
const script = block.split(" run: |\n")[1].split("\n").map((line) => line.replace(/^ {10}/, "")).join("\n");
|
|
const directory = mkdtempSync(path.join(tmpdir(), "ci-profile-legacy-"));
|
|
try {
|
|
writeFileSync(path.join(directory, "probe.test.cjs"), 'require("node:test")("old checkout works", () => {});');
|
|
for (const current of [false, true]) {
|
|
if (current) {
|
|
mkdirSync(path.join(directory, "scripts"));
|
|
symlinkSync(fileURLToPath(new URL("../../../scripts/ci-profile", import.meta.url)), path.join(directory, "scripts/ci-profile"));
|
|
}
|
|
const env = { ...process.env, GITHUB_ENV: path.join(directory, "env"), PAPERCLIP_CI_PROFILE: "false" };
|
|
delete env.NODE_TEST_CONTEXT;
|
|
const result = spawnSync("bash", ["-c", `${script}\nsource "$GITHUB_ENV"\nnode "$CI_NODE_TEST" probe.test.cjs`], {
|
|
cwd: directory, env, encoding: "utf8",
|
|
});
|
|
assert.equal(result.status, 0, result.stdout + result.stderr);
|
|
assert.match(readFileSync(env.GITHUB_ENV, "utf8"), current ? /CI_NODE_TEST=scripts\/ci-profile\/node-test.mjs/ : /CI_NODE_TEST=--test/);
|
|
}
|
|
} finally { rmSync(directory, { recursive: true, force: true }); }
|
|
});
|
|
|
|
test("reruns use separate IDs and clocks and cannot mix job attempts", () => {
|
|
const rerun = { ...run, run_attempt: 2, run_started_at: at(2) };
|
|
assert.throws(() => buildTrace(rerun, jobs), /another run or attempt/);
|
|
const trace = buildTrace(rerun, jobs.map((entry) => ({ ...entry, run_attempt: 2 })));
|
|
assert.equal(trace.wallMs, 18000);
|
|
assert.notEqual(trace.traceId, buildTrace(run, jobs).traceId);
|
|
assert.equal(trace.events[0].data["ci.partial_rerun"], true);
|
|
assert.throws(() => buildTrace({ ...run, status: "in_progress" }, jobs), /complete/);
|
|
});
|
|
|
|
test("partial reruns exclude old jobs that GitHub relabels with the new attempt number", () => {
|
|
const rerun = { ...run, run_attempt: 2, run_started_at: at(30) };
|
|
const inventory = [
|
|
{ ...job(1, "carried build", 2, 12), run_attempt: 2 },
|
|
{ ...job(3, "retried tests", 32, 50), run_attempt: 2, created_at: at(31) },
|
|
];
|
|
const trace = buildTrace(rerun, inventory);
|
|
assert.equal(trace.wallMs, 20000);
|
|
assert.equal(trace.jobCount, 1);
|
|
assert.equal(trace.events[0].data["ci.carried_job_count"], 1);
|
|
assert.equal(trace.events.some((event) => event.data.name === "ci / carried build"), false);
|
|
assert.equal(trace.events[0].data["ci.profile_complete"], false);
|
|
assert.match(trace.warnings.join(), /1 completed jobs carried over/);
|
|
assert.throws(() => buildTrace(rerun, inventory.slice(0, 1)), /No jobs executed/);
|
|
});
|
|
|
|
test("label alone cannot authorize outsiders, bots, stale revisions or outsider reruns", () => {
|
|
const pr = { user: { id: 123, type: "User" }, base: { repo: run.repository }, head: { repo: { id: 99 }, sha: "abc" }, labels: [{ name: "profile-ci" }] };
|
|
assert.equal(associatedProfilePr(run, [pr], [123]), pr);
|
|
for (const changed of [
|
|
{ ...pr, labels: [] },
|
|
{ ...pr, head: { ...pr.head, sha: "next" } },
|
|
{ ...pr, head: { ...pr.head, repo: { id: 100 } } },
|
|
{ ...pr, user: { id: 999, type: "User" } },
|
|
{ ...pr, user: { id: 123, type: "Bot" } },
|
|
{ ...pr, base: { repo: { ...run.repository, id: 100 } } },
|
|
]) assert.equal(associatedProfilePr(run, [changed], [123]), undefined);
|
|
assert.equal(associatedProfilePr({ ...run, actor: { id: 999, type: "User" } }, [pr], [123]), undefined);
|
|
assert.equal(associatedProfilePr({ ...run, triggering_actor: { id: 999, type: "User" } }, [pr], [123]), undefined);
|
|
assert.equal(associatedProfilePr({ ...run, repository: { ...run.repository, id: 100 } }, [pr], [123]), undefined);
|
|
assert.equal(associatedProfilePr(run, [pr], []), undefined);
|
|
});
|
|
|
|
test("always-on export samples whole CODEOWNER runs and never needs the label", () => {
|
|
const pr = { user: { id: 123, type: "User" }, base: { repo: run.repository }, head: { repo: { id: 99 }, sha: "abc" }, labels: [] };
|
|
assert.equal(sampleRate, 10);
|
|
assert.equal(exportMode({ ...run, run_number: 64800, display_title: "fix: something" }), "sampled");
|
|
for (const number of [64801, 64805, 64809, 0, undefined, "64800"]) {
|
|
assert.equal(exportMode({ ...run, run_number: number, display_title: "fix: something" }), undefined);
|
|
}
|
|
assert.equal(exportMode({ ...run, run_number: 64801, display_title: "CI profile · PR #5" }), "label");
|
|
assert.equal(associatedOwnerPr(run, [pr], [123]), pr);
|
|
assert.equal(associatedProfilePr(run, [pr], [123]), undefined);
|
|
for (const changed of [
|
|
{ ...pr, user: { id: 999, type: "User" } },
|
|
{ ...pr, user: { id: 123, type: "Bot" } },
|
|
{ ...pr, head: { ...pr.head, sha: "next" } },
|
|
{ ...pr, head: { ...pr.head, repo: { id: 100 } } },
|
|
{ ...pr, head: { ...pr.head, repo: null } },
|
|
{ ...pr, base: { repo: { ...run.repository, id: 100 } } },
|
|
]) assert.equal(associatedOwnerPr(run, [changed], [123]), undefined);
|
|
assert.equal(associatedOwnerPr({ ...run, actor: { id: 999, type: "User" } }, [pr], [123]), undefined);
|
|
assert.equal(associatedOwnerPr({ ...run, triggering_actor: { id: 41898282, type: "Bot" } }, [pr], [123]), undefined);
|
|
assert.equal(associatedOwnerPr(run, [pr], []), undefined);
|
|
|
|
const trace = buildTrace({ ...run, conclusion: "success" }, jobs, [], { sampleRate, testDetail: false });
|
|
assert.deepEqual([...new Set(trace.events.map((event) => event.samplerate))], [10]);
|
|
assert.equal(trace.events[0].data["ci.sample_rate"], 10);
|
|
assert.deepEqual(trace.warnings, ["Job/step timings only; no test reporter collection steps were present"]);
|
|
assert.equal(buildTrace(run, jobs).events[0].samplerate, 1);
|
|
for (const invalid of [0, 1.5, -1, "10"]) assert.throws(() => buildTrace(run, jobs, [], { sampleRate: invalid }), /sample rate/);
|
|
});
|
|
|
|
test("outside checks are spans on the run's clock, with elapsed time to the last one", () => {
|
|
const head = { ...run, conclusion: "success", path: ".github/workflows/pr.yml" };
|
|
const checks = normalizeChecks(head, {
|
|
checkRuns: [
|
|
{ app: { slug: "greptile-apps" }, name: "Greptile Review", status: "completed", conclusion: "success", started_at: at(1), completed_at: at(90), output: { text: "must not leave GitHub" } },
|
|
{ app: { slug: "github-actions" }, name: "ci / first", status: "completed", conclusion: "success", started_at: at(2), completed_at: at(12) },
|
|
{ app: { slug: "scanner" }, name: "Scan", status: "in_progress", conclusion: null, started_at: at(1), completed_at: null },
|
|
{ app: { slug: "scanner" }, name: "Skipped", status: "completed", conclusion: "skipped", started_at: at(9), completed_at: at(1) },
|
|
{ app: { slug: "old" }, name: "Before this run", status: "completed", conclusion: "success", started_at: at(-600), completed_at: at(-500) },
|
|
{ name: "no app", status: "completed", started_at: at(1), completed_at: at(2) },
|
|
],
|
|
statuses: [
|
|
{ context: "security/scan", state: "success", created_at: at(15) },
|
|
{ context: "security/scan", state: "pending", created_at: at(0) },
|
|
{ context: "slow/status", state: "pending", created_at: at(0) },
|
|
{ context: "broken/status", state: "error", created_at: at(4) },
|
|
{ context: "broken/status", state: "pending", created_at: at(1) },
|
|
],
|
|
workflowRuns: [
|
|
{ id: 42, path: ".github/workflows/pr.yml", head_sha: "abc", event: "pull_request", name: "PR", status: "completed", created_at: at(0), updated_at: at(20) },
|
|
{ id: 50, path: ".github/workflows/review.yml", head_sha: "abc", event: "pull_request_target", name: "Review", status: "completed", conclusion: "success", run_attempt: 1, created_at: at(0), updated_at: at(40) },
|
|
{ id: 51, path: ".github/workflows/review.yml", head_sha: "abc", event: "pull_request_target", name: "Review", status: "completed", conclusion: "failure", run_attempt: 1, created_at: at(5), updated_at: at(60) },
|
|
{ id: 52, path: ".github/workflows/other.yml", head_sha: "other", event: "pull_request", name: "Other commit", status: "completed", created_at: at(0), updated_at: at(9) },
|
|
{ id: 53, path: ".github/workflows/ci-profile.yml", head_sha: "abc", event: "workflow_run", name: "CI profile", status: "completed", created_at: at(0), updated_at: at(9) },
|
|
],
|
|
});
|
|
assert.deepEqual(checks.map((check) => check.name).sort(),
|
|
["Before this run", "Greptile Review", "Review", "Scan", "Skipped", "broken/status", "security/scan", "slow/status"]);
|
|
assert.deepEqual(pendingChecks(head, checks).map((check) => check.name), ["Scan", "slow/status"]);
|
|
|
|
const trace = buildTrace(head, jobs, [], { checks });
|
|
const spans = trace.events.filter((event) => /_check$/.test(event.data["ci.kind"]));
|
|
assert.deepEqual(spans.map((event) => [event.data.name, event.data["ci.kind"], event.data.duration_ms, event.data["ci.status"]]), [
|
|
["Greptile Review", "external_check", 89000, "success"],
|
|
["security/scan", "external_check", 15000, "success"],
|
|
["broken/status", "external_check", 3000, "error"],
|
|
["Review", "workflow_check", 55000, "failure"],
|
|
]);
|
|
assert.deepEqual(spans.map((event) => event.data.error), [false, false, true, true]);
|
|
const root = trace.events[0].data;
|
|
assert.ok(spans.every((event) => event.data["trace.parent_id"] === root["trace.span_id"]));
|
|
assert.equal(spans[0].data["ci.finishes_after_workflow"], true);
|
|
assert.equal(spans[1].data["ci.finishes_after_workflow"], false);
|
|
assert.equal(root.duration_ms, 20000);
|
|
assert.equal(root["ci.workflow_ms"], 20000);
|
|
assert.equal(root["ci.all_checks_ms"], 90000);
|
|
assert.equal(root["ci.last_to_finish"], "Greptile Review");
|
|
assert.equal(root["ci.slowest_check"], "Greptile Review");
|
|
assert.equal(root["ci.slowest_check_ms"], 89000);
|
|
assert.equal(root["ci.check_count"], 4);
|
|
assert.equal(root["ci.pending_check_count"], 2);
|
|
assert.equal(root["ci.pending_checks"], "Scan, slow/status");
|
|
assert.equal(root["ci.earlier_check_count"], 1);
|
|
assert.equal(root["ci.profile_complete"], false);
|
|
assert.match(trace.warnings.join(), /2 outside checks were still running/);
|
|
assert.doesNotMatch(JSON.stringify(trace), /must not leave GitHub/);
|
|
assert.match(traceUrl("team", "test", "paperclip-ci", trace), /trace_end_ts=1800000090$/);
|
|
|
|
const alone = buildTrace(head, jobs).events[0].data;
|
|
assert.equal(alone["ci.all_checks_ms"], 20000);
|
|
assert.equal(alone["ci.last_to_finish"], "PR workflow");
|
|
assert.equal("ci.slowest_check" in alone, false);
|
|
// A rerun repeats jobs only; the push's outside checks are not its elapsed time.
|
|
const rerun = buildTrace({ ...head, run_attempt: 2, run_started_at: at(0) }, jobs.map((entry) => ({ ...entry, run_attempt: 2 })), [], { checks });
|
|
assert.equal(rerun.events[0].data["ci.check_count"], 0);
|
|
assert.equal(rerun.events[0].data["ci.all_checks_ms"], 20000);
|
|
});
|
|
|
|
test("export waits a bounded time for outside checks, never for other workflows", async () => {
|
|
const head = { ...run, path: ".github/workflows/pr.yml" };
|
|
const check = (completed) => ({ kind: "external_check", app: "greptile-apps", name: "Greptile Review", startedAt: at(1), completedAt: completed ? at(90) : null });
|
|
const pauses = [];
|
|
let clock = 0;
|
|
const now = () => clock;
|
|
const pause = async (ms) => { pauses.push(ms); clock += ms; };
|
|
let loads = 0;
|
|
const settled = await settledChecks(head, () => [check(++loads >= 3)], { pause, now });
|
|
assert.equal(settled[0].completedAt, at(90));
|
|
assert.deepEqual(pauses, [30_000, 30_000]);
|
|
|
|
pauses.length = 0;
|
|
const stuck = await settledChecks(head, () => [check(false)], { pause, now, waitMs: 90_000 });
|
|
assert.equal(stuck[0].completedAt, null);
|
|
assert.deepEqual(pauses, [30_000, 30_000, 30_000]);
|
|
|
|
// Slow GitHub responses count against the same deadline as the pauses.
|
|
pauses.length = 0;
|
|
loads = 0;
|
|
await settledChecks(head, () => { clock += 20_000; loads += 1; return [check(false)]; }, { pause, now, waitMs: 90_000 });
|
|
assert.deepEqual(pauses, [30_000, 20_000]);
|
|
assert.equal(loads, 3);
|
|
|
|
pauses.length = 0;
|
|
await settledChecks(head, () => [{ ...check(false), kind: "workflow_check" }], { pause, now });
|
|
await settledChecks(head, () => [{ ...check(false), startedAt: at(-600) }], { pause, now });
|
|
await settledChecks({ ...head, run_attempt: 2, run_started_at: at(0) }, () => [check(false)], { pause, now });
|
|
await settledChecks(head, () => [check(false)], { pause, now, waitMs: 0 });
|
|
assert.deepEqual(pauses, []);
|
|
});
|
|
|
|
test("profiling uses only infrastructure CODEOWNERS resolved at trusted master", () => {
|
|
const source = '.github/** @owner @second\ndoc/** @outsider\n';
|
|
assert.deepEqual(ciOwners(source), ["owner", "second"]);
|
|
for (const invalid of ['', '.github/**', '.github/** @org/team', '.github/** @owner\n/.github/** @other']) {
|
|
assert.throws(() => ciOwners(invalid), /CODEOWNERS/);
|
|
}
|
|
const endpoints = [];
|
|
const sha = "a".repeat(40);
|
|
assert.deepEqual(trustedOwnerIds((endpoint) => {
|
|
endpoints.push(endpoint);
|
|
if (endpoint.endsWith('git/ref/heads/master')) return { object: { sha } };
|
|
if (endpoint.endsWith(`CODEOWNERS?ref=${sha}`)) return { encoding: 'base64', content: Buffer.from(source).toString('base64') };
|
|
return { id: endpoint === '/users/owner' ? 123 : 456, type: 'User' };
|
|
}), [123, 456]);
|
|
assert.deepEqual(endpoints, [
|
|
'/repos/paperclipai/paperclip/git/ref/heads/master',
|
|
`/repos/paperclipai/paperclip/contents/.github/CODEOWNERS?ref=${sha}`, '/users/owner', '/users/second',
|
|
]);
|
|
});
|
|
|
|
test("export paces batches and retries only rejected transient events", async () => {
|
|
const events = Array.from({ length: 501 }, () => ({ data: {} }));
|
|
const sizes = [];
|
|
const pauses = [];
|
|
await sendEvents(events, "test-only", "paperclip-ci", async (url, options) => {
|
|
assert.equal(url, "https://api.honeycomb.io/1/batch/paperclip-ci");
|
|
assert.equal(options.redirect, "error");
|
|
const size = JSON.parse(options.body).length;
|
|
sizes.push(size);
|
|
return { ok: true, json: async () => Array.from({ length: size }, () => ({ status: 202 })) };
|
|
}, async (ms) => { pauses.push(ms); });
|
|
assert.deepEqual(sizes, [250, 250, 1]);
|
|
assert.deepEqual(pauses, [1000, 1000]);
|
|
let calls = 0;
|
|
const retried = [];
|
|
await sendEvents([{ data: { id: 1 } }, { data: { id: 2 } }], "test-only", "paperclip-ci", async (_url, options) => {
|
|
const batch = JSON.parse(options.body);
|
|
retried.push(batch.map((event) => event.data.id));
|
|
calls += 1;
|
|
return { ok: true, json: async () => calls === 1 ? [{ status: 202 }, { status: 429 }] : [{ status: 202 }] };
|
|
}, async (ms) => { pauses.push(ms); });
|
|
assert.deepEqual(retried, [[1, 2], [2]]);
|
|
assert.equal(pauses.at(-1), 1000);
|
|
await assert.rejects(sendEvents([events[0]], "test-only", "paperclip-ci", async () => ({ ok: true, json: async () => [{ status: 400 }] })), /status 400/);
|
|
});
|
|
|
|
function secretReference(text, includeQuoted = false) {
|
|
for (const line of text.split(/[\r\n\u2028\u2029]/)) {
|
|
const start = line.indexOf("HONEYCOMB");
|
|
if (start >= 0 && line.indexOf("KEY", start + "HONEYCOMB".length) >= 0) return true;
|
|
}
|
|
let offset = 0;
|
|
while (offset < text.length) {
|
|
const start = text.indexOf("${{", offset);
|
|
if (start < 0) return false;
|
|
offset = start + 3;
|
|
let quoted = false;
|
|
let expression = "";
|
|
while (offset < text.length) {
|
|
const char = text[offset];
|
|
if (char === "'") {
|
|
if (quoted && text[offset + 1] === "'") {
|
|
offset += 2;
|
|
continue;
|
|
}
|
|
quoted = !quoted;
|
|
expression += includeQuoted ? char : " ";
|
|
} else if (!quoted) {
|
|
if (char === "}" && text[offset + 1] === "}") {
|
|
offset += 2;
|
|
break;
|
|
}
|
|
expression += char;
|
|
} else if (includeQuoted) {
|
|
expression += char;
|
|
}
|
|
offset += 1;
|
|
}
|
|
if (/\bsecrets\b/.test(expression)) return true;
|
|
}
|
|
return false;
|
|
}
|
|
|
|
function assertNoPrWorkflowCredentials(source) {
|
|
assert.equal(secretReference(source, true), false);
|
|
}
|
|
|
|
test("PR credential guard permits filenames but rejects secret expressions", () => {
|
|
assertNoPrWorkflowCredentials("run: node --test scripts/docker-compose-secrets.test.mjs");
|
|
assertNoPrWorkflowCredentials("if: ${{ always() }}\nrun: node --test scripts/docker-compose-secrets.test.mjs");
|
|
assertNoPrWorkflowCredentials("if: ${{ startsWith(github.ref, 'ref') }}\nrun: 'node --test scripts/docker-compose-secrets.test.mjs'");
|
|
for (const source of ["env: ${{ secrets.EXAMPLE }}", "env: ${{ secrets['EXAMPLE'] }}", "env: ${{ toJSON(secrets) }}", "env: ${{ format('}}', secrets.EXAMPLE) }}",
|
|
"# Expressions start with '${{'.\nTOKEN: ${{ secrets.PR_TOKEN }}",
|
|
"name: Example # Expressions start with '${{'.\nTOKEN: ${{ secrets.PR_TOKEN }}"]) {
|
|
assert.throws(() => assertNoPrWorkflowCredentials(source));
|
|
}
|
|
for (const source of [
|
|
"${{secrets.EXAMPLE_TOKEN}}", '${{ secrets["EXAMPLE_TOKEN"] }}',
|
|
"${{ failure() && secrets.EXAMPLE_TOKEN }}", "${{\n secrets.EXAMPLE_TOKEN\n}}",
|
|
"${{ format('{0}', secrets.EXAMPLE_TOKEN) }}", "HONEYCOMB_CI_API_KEY",
|
|
]) assert.throws(() => assertNoPrWorkflowCredentials(source));
|
|
assert.throws(() => assertNoPrWorkflowCredentials("HONEYCOMB_API_KEY: example"));
|
|
assert.throws(() => assertNoPrWorkflowCredentials("${{ format('secrets.EXAMPLE', 'literal') }}"));
|
|
});
|
|
|
|
test("workflow exporter runs trusted code and keeps keys out of the PR jobs", () => {
|
|
const workflow = readFileSync(new URL("../../workflows/ci-profile.yml", import.meta.url), "utf8");
|
|
assert.match(workflow, /ref: \$\{\{ github.sha \}\}/);
|
|
assert.match(workflow, /startsWith\(github.event.workflow_run.display_title/);
|
|
// The job condition keeps outside contributors' runs from starting a runner
|
|
// or entering the environment, and matches the exporter's sample rule.
|
|
const condition = workflow.split(" if: >-\n")[1].split("\n runs-on:")[0];
|
|
for (const required of [
|
|
"github.repository_id == '1170821064'",
|
|
"github.event.workflow_run.event == 'pull_request'",
|
|
"github.event.workflow_run.path == '.github/workflows/pr.yml'",
|
|
"github.event.workflow_run.head_repository.full_name == 'paperclipai/paperclip'",
|
|
"contains(vars.AWS_CI_ROUTING_USER_IDS, format('|{0}|', github.event.workflow_run.actor.id))",
|
|
"contains(vars.AWS_CI_ROUTING_USER_IDS, format('|{0}|', github.event.workflow_run.triggering_actor.id))",
|
|
"endsWith(format('{0}', github.event.workflow_run.run_number), '0')",
|
|
]) assert.ok(condition.includes(required), required);
|
|
assert.equal(condition.match(/\|\|/g).length, 1, "only label-or-sample may be an alternative");
|
|
assert.equal(workflow.match(/^ [a-z_]+:$/gm).filter((line) => ![" workflow_run:", " export:"].includes(line)).length, 0);
|
|
assert.equal(workflow.match(/environment:/g).length, 1);
|
|
assert.match(workflow, /environment: ci-profiling/);
|
|
assert.doesNotMatch(workflow, /head_sha|secrets: inherit|contents: write|pull-requests: write|pnpm install/);
|
|
const pr = readFileSync(new URL("../../workflows/pr-trusted.yml", import.meta.url), "utf8");
|
|
assert.equal(secretReference("run: node --test scripts/docker-compose-secrets.test.mjs"), false);
|
|
for (const lookup of ["${{ secrets.PR_TOKEN }}", "${{ secrets['PR_TOKEN'] }}", "${{\n secrets.PR_TOKEN\n }}", "${{ toJSON(secrets) }}", "${{ secrets.HONEYCOMB_API_KEY }}"]) {
|
|
assert.equal(secretReference(lookup), true);
|
|
}
|
|
for (const lookup of [
|
|
"${{ format('Bearer {0}', secrets.PR_TOKEN) }}",
|
|
"${{ format('{0}', toJSON(secrets)) }}",
|
|
"${{ format('{{Bearer}} {0}', secrets['PR_TOKEN']) }}",
|
|
"${{ format('it''s {0}',\n secrets.PR_TOKEN) }}",
|
|
]) {
|
|
assert.equal(secretReference(lookup), true);
|
|
}
|
|
for (const nonSecret of [
|
|
"${{ format('secrets.PR_TOKEN', 'literal') }}",
|
|
"${{ github.sha }}\nrun: node --test scripts/docker-compose-secrets.test.mjs",
|
|
]) {
|
|
assert.equal(secretReference(nonSecret), false);
|
|
}
|
|
const escapedExpressions = [28, 10_000].flatMap(count => {
|
|
const expression = "${{ format('" + "it''s ".repeat(count) + "{0}', github.sha) }}";
|
|
return [expression, expression.replace("github.sha", "secrets.PR_TOKEN"), expression.replace("github.sha", "toJSON(secrets)")];
|
|
});
|
|
const boundedScan = spawnSync(process.execPath, ["-e", `const fs=require('node:fs');const scan=${secretReference.toString()};process.stdout.write(JSON.stringify(JSON.parse(fs.readFileSync(0,'utf8')).map(value=>[scan(value),scan(value,true)])));`], {
|
|
input: JSON.stringify(escapedExpressions),
|
|
encoding: "utf8", timeout: 2_000,
|
|
});
|
|
assert.equal(boundedScan.status, 0, boundedScan.error?.message);
|
|
assert.deepEqual(JSON.parse(boundedScan.stdout), [[false, false], [true, true], [true, true], [false, false], [true, true], [true, true]]);
|
|
assert.equal(secretReference(pr), false);
|
|
assertNoPrWorkflowCredentials(pr);
|
|
assert.match(pr, /if: always\(\) && env.PAPERCLIP_CI_PROFILE == 'true'/);
|
|
assert.doesNotMatch(pr, /PAPERCLIP_CI_PROFILE:.*labels/);
|
|
assert.equal(pr.match(/PAPERCLIP_CI_PROFILE: \$\{\{ needs.gate.outputs.profile_ci \}\}/g).length, 11);
|
|
// upload-artifact excludes files inside dot-directories by default. Keep
|
|
// this opt-in scoped to timing NDJSON, never the workspace or other secrets.
|
|
assert.equal(pr.match(/path: \.ci-profile\/\*\.ndjson\n {10}include-hidden-files: true/g).length, 11);
|
|
const routing = readFileSync(new URL("../../workflows/ci-routing.yml", import.meta.url), "utf8");
|
|
assert.match(routing, /PROFILE_CI: \$\{\{ steps.profile.outputs.profile_ci \}\}/);
|
|
assert.match(routing, /\/git\/ref\/heads\/master/);
|
|
assert.match(routing, /authorize\.mjs\?ref=\$TRUSTED_PROFILE_SHA/);
|
|
});
|
|
|
|
test("artifact parsing never extracts paths or executes files and rejects oversized records", () => {
|
|
const directory = mkdtempSync(path.join(tmpdir(), "ci-artifact-test-"));
|
|
const archive = path.join(directory, "artifact.zip");
|
|
const reader = fileURLToPath(new URL("../../../scripts/ci-profile/read-artifact.py", import.meta.url));
|
|
try {
|
|
execFileSync("python3", ["-c", "import zipfile,sys; z=zipfile.ZipFile(sys.argv[1],'w'); z.writestr('../escape.ndjson', '{\"kind\":\"test\"}\\n'); z.close()", archive]);
|
|
const result = execFileSync("python3", [reader, archive], { encoding: "utf8" });
|
|
assert.deepEqual(JSON.parse(result), { kind: "test" });
|
|
assert.equal(existsSync(path.join(directory, "escape.ndjson")), false);
|
|
execFileSync("python3", ["-c", "import zipfile,sys; z=zipfile.ZipFile(sys.argv[1],'w'); z.writestr('large.ndjson', 'x'*33000); z.close()", archive]);
|
|
assert.notEqual(spawnSync("python3", [reader, archive]).status, 0);
|
|
} finally { rmSync(directory, { recursive: true, force: true }); }
|
|
});
|