mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
feat(ci): send one in ten CODEOWNER PR runs and their outside checks to Honeycomb (#21192)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Contributors need fast, predictable CI feedback. > - PR #16378 added a Honeycomb trace of the main PR workflow, but only for PRs with the `profile-ci` label. > - Ordinary CI time was therefore never recorded. All of the last 300 `CI profile` runs were skipped. > - Reviews and scans outside the `PR` workflow, such as Greptile, were not in the trace. > - This pull request exports one CODEOWNER PR run in ten without a label, and adds outside checks to every trace. > - The benefit is a continuous record of total CI time and of which job or check was slowest. The Honeycomb key stays out of reach of outside contributors. ## Linked Issues or Issue Description Refs #16378 **What existing behavior does this improve?** The Honeycomb CI trace that the `CI profile` workflow exports. **Current behavior** - The exporter runs only for a PR with the `profile-ci` label. The label also forces the full suite. - The trace has the jobs of the `PR` workflow only. Greptile, security scans, commit statuses and other workflows are absent. - The only total is the `PR` workflow time. **Proposed behavior** - One `PR` run in ten is exported without a label. The run number must end in 0. The trace has every job and step. Each span has a sample rate of 10. - Each trace has one span for each outside check and each other workflow on the head commit. - The root span has `ci.workflow_ms`, `ci.all_checks_ms`, `ci.last_to_finish` and `ci.slowest_check`. **Reason and benefit** Maintainers can see how long CI takes and what is slow, for example a long Greptile review, without a label on each PR. Sampling keeps the volume near 1.7 million events each month. Export of every run is near 17 million. ## What Changed - `scripts/ci-profile/authorize.mjs`: add `exportMode` (label or one-in-ten sample) and `associatedOwnerPr` (the CODEOWNER checks without the label). - `scripts/ci-profile/trace.mjs`: add `normalizeChecks` and spans for outside checks. Add the sample rate and the new root fields. Sampled runs do not warn about absent test artifacts. - `scripts/ci-profile/export.mjs`: select the mode. Read check runs, commit statuses and workflow runs for the head commit. Wait a maximum of 15 minutes for outside apps that are not complete. Sampled runs read no artifacts. - `.github/workflows/ci-profile.yml`: no new job. The job condition now requires a branch of this repository, an actor and rerun actor on the `AWS_CI_ROUTING_USER_IDS` roster, and a labelled or sampled run. Add `checks: read` and `statuses: read`. Increase the timeout to 30 minutes for the bounded wait. - `doc/ci-profiling.md` and `doc/DEVELOPING.md`: describe sampled runs, outside checks, the totals and the job condition. ## Verification - `node --test .github/scripts/tests/ci-profile.test.mjs .github/scripts/tests/ci-routing.test.mjs`: 163 pass, 0 fail. - `actionlint .github/workflows/ci-profile.yml`: no findings. - Dry runs against real runs with the real CODEOWNERS check (`node scripts/ci-profile/export.mjs --run-id=ID --attempt=1 --dry-run`): - A sampled CODEOWNER run (run number 64780): 1,144 spans, sample rate 10, 5 outside checks and 1 other workflow. - A sampled run from an outside contributor (run number 64800): "No CODEOWNERS-authorized PR at this revision; no data exported." - A CODEOWNER run that is not in the sample: "This attempt was not selected for profiling; no data exported." - New tests cover: the sample rule, fork and outsider PRs, a bot rerun actor, a stale head, outside-check spans and totals, the bounded wait, and the job condition. - Not run locally: `scripts/ci-profile/reporters.test.mjs`. It needs installed dependencies, and this change does not touch the reporters. CI runs it. - No data was sent to Honeycomb before merge. The workflow runs from `master`, so the first real export occurs after merge. ## Risks - The label path now also needs the `AWS_CI_ROUTING_USER_IDS` roster. If that variable is empty, nothing is exported. All four CI CODEOWNERS are on the roster today. - The roster is only a first filter. The exporter still checks the PR author and both actors against CODEOWNERS on `master` before it reads or sends data. The key stays in the `ci-profiling` environment, which permits only `master`. - The export job can now run for a maximum of 30 minutes on a GitHub-hosted runner while it waits for an outside check. - GitHub gives no completion time for a workflow run. The span for another workflow ends at the last update of that run. - Honeycomb volume increases by about 1.7 million events each month. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code, with tool use, code execution and repository inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
1 parent
0904a65d82
commit
9a9cd20d91
7 files changed
+415
-38
No files matched your search
@@ -5,9 +5,9 @@ import { execFileSync, spawnSync } from "node:child_process";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { buildTrace } from "../../../scripts/ci-profile/trace.mjs";
|
||||
import { sendEvents } from "../../../scripts/ci-profile/export.mjs";
|
||||
import { associatedProfilePr, ciOwners, trustedOwnerIds } from "../../../scripts/ci-profile/authorize.mjs";
|
||||
import { buildTrace, normalizeChecks, pendingChecks, traceUrl } from "../../../scripts/ci-profile/trace.mjs";
|
||||
import { sendEvents, settledChecks } from "../../../scripts/ci-profile/export.mjs";
|
||||
import { associatedOwnerPr, associatedProfilePr, ciOwners, exportMode, sampleRate, trustedOwnerIds } from "../../../scripts/ci-profile/authorize.mjs";
|
||||
|
||||
const at = (seconds) => new Date(1_800_000_000_000 + seconds * 1000).toISOString();
|
||||
const run = {
|
||||
@@ -186,6 +186,136 @@ test("label alone cannot authorize outsiders, bots, stale revisions or outsider
|
||||
assert.equal(associatedProfilePr(run, [pr], []), undefined);
|
||||
});
|
||||
|
||||
test("always-on export samples whole CODEOWNER runs and never needs the label", () => {
|
||||
const pr = { user: { id: 123, type: "User" }, base: { repo: run.repository }, head: { repo: { id: 99 }, sha: "abc" }, labels: [] };
|
||||
assert.equal(sampleRate, 10);
|
||||
assert.equal(exportMode({ ...run, run_number: 64800, display_title: "fix: something" }), "sampled");
|
||||
for (const number of [64801, 64805, 64809, 0, undefined, "64800"]) {
|
||||
assert.equal(exportMode({ ...run, run_number: number, display_title: "fix: something" }), undefined);
|
||||
}
|
||||
assert.equal(exportMode({ ...run, run_number: 64801, display_title: "CI profile · PR #5" }), "label");
|
||||
assert.equal(associatedOwnerPr(run, [pr], [123]), pr);
|
||||
assert.equal(associatedProfilePr(run, [pr], [123]), undefined);
|
||||
for (const changed of [
|
||||
{ ...pr, user: { id: 999, type: "User" } },
|
||||
{ ...pr, user: { id: 123, type: "Bot" } },
|
||||
{ ...pr, head: { ...pr.head, sha: "next" } },
|
||||
{ ...pr, head: { ...pr.head, repo: { id: 100 } } },
|
||||
{ ...pr, head: { ...pr.head, repo: null } },
|
||||
{ ...pr, base: { repo: { ...run.repository, id: 100 } } },
|
||||
]) assert.equal(associatedOwnerPr(run, [changed], [123]), undefined);
|
||||
assert.equal(associatedOwnerPr({ ...run, actor: { id: 999, type: "User" } }, [pr], [123]), undefined);
|
||||
assert.equal(associatedOwnerPr({ ...run, triggering_actor: { id: 41898282, type: "Bot" } }, [pr], [123]), undefined);
|
||||
assert.equal(associatedOwnerPr(run, [pr], []), undefined);
|
||||
|
||||
const trace = buildTrace({ ...run, conclusion: "success" }, jobs, [], { sampleRate, testDetail: false });
|
||||
assert.deepEqual([...new Set(trace.events.map((event) => event.samplerate))], [10]);
|
||||
assert.equal(trace.events[0].data["ci.sample_rate"], 10);
|
||||
assert.deepEqual(trace.warnings, ["Job/step timings only; no test reporter collection steps were present"]);
|
||||
assert.equal(buildTrace(run, jobs).events[0].samplerate, 1);
|
||||
for (const invalid of [0, 1.5, -1, "10"]) assert.throws(() => buildTrace(run, jobs, [], { sampleRate: invalid }), /sample rate/);
|
||||
});
|
||||
|
||||
test("outside checks are spans on the run's clock, with elapsed time to the last one", () => {
|
||||
const head = { ...run, conclusion: "success", path: ".github/workflows/pr.yml" };
|
||||
const checks = normalizeChecks(head, {
|
||||
checkRuns: [
|
||||
{ app: { slug: "greptile-apps" }, name: "Greptile Review", status: "completed", conclusion: "success", started_at: at(1), completed_at: at(90), output: { text: "must not leave GitHub" } },
|
||||
{ app: { slug: "github-actions" }, name: "ci / first", status: "completed", conclusion: "success", started_at: at(2), completed_at: at(12) },
|
||||
{ app: { slug: "scanner" }, name: "Scan", status: "in_progress", conclusion: null, started_at: at(1), completed_at: null },
|
||||
{ app: { slug: "scanner" }, name: "Skipped", status: "completed", conclusion: "skipped", started_at: at(9), completed_at: at(1) },
|
||||
{ app: { slug: "old" }, name: "Before this run", status: "completed", conclusion: "success", started_at: at(-600), completed_at: at(-500) },
|
||||
{ name: "no app", status: "completed", started_at: at(1), completed_at: at(2) },
|
||||
],
|
||||
statuses: [
|
||||
{ context: "security/scan", state: "success", created_at: at(15) },
|
||||
{ context: "security/scan", state: "pending", created_at: at(0) },
|
||||
{ context: "slow/status", state: "pending", created_at: at(0) },
|
||||
{ context: "broken/status", state: "error", created_at: at(4) },
|
||||
{ context: "broken/status", state: "pending", created_at: at(1) },
|
||||
],
|
||||
workflowRuns: [
|
||||
{ id: 42, path: ".github/workflows/pr.yml", head_sha: "abc", event: "pull_request", name: "PR", status: "completed", created_at: at(0), updated_at: at(20) },
|
||||
{ id: 50, path: ".github/workflows/review.yml", head_sha: "abc", event: "pull_request_target", name: "Review", status: "completed", conclusion: "success", run_attempt: 1, created_at: at(0), updated_at: at(40) },
|
||||
{ id: 51, path: ".github/workflows/review.yml", head_sha: "abc", event: "pull_request_target", name: "Review", status: "completed", conclusion: "failure", run_attempt: 1, created_at: at(5), updated_at: at(60) },
|
||||
{ id: 52, path: ".github/workflows/other.yml", head_sha: "other", event: "pull_request", name: "Other commit", status: "completed", created_at: at(0), updated_at: at(9) },
|
||||
{ id: 53, path: ".github/workflows/ci-profile.yml", head_sha: "abc", event: "workflow_run", name: "CI profile", status: "completed", created_at: at(0), updated_at: at(9) },
|
||||
],
|
||||
});
|
||||
assert.deepEqual(checks.map((check) => check.name).sort(),
|
||||
["Before this run", "Greptile Review", "Review", "Scan", "Skipped", "broken/status", "security/scan", "slow/status"]);
|
||||
assert.deepEqual(pendingChecks(head, checks).map((check) => check.name), ["Scan", "slow/status"]);
|
||||
|
||||
const trace = buildTrace(head, jobs, [], { checks });
|
||||
const spans = trace.events.filter((event) => /_check$/.test(event.data["ci.kind"]));
|
||||
assert.deepEqual(spans.map((event) => [event.data.name, event.data["ci.kind"], event.data.duration_ms, event.data["ci.status"]]), [
|
||||
["Greptile Review", "external_check", 89000, "success"],
|
||||
["security/scan", "external_check", 15000, "success"],
|
||||
["broken/status", "external_check", 3000, "error"],
|
||||
["Review", "workflow_check", 55000, "failure"],
|
||||
]);
|
||||
assert.deepEqual(spans.map((event) => event.data.error), [false, false, true, true]);
|
||||
const root = trace.events[0].data;
|
||||
assert.ok(spans.every((event) => event.data["trace.parent_id"] === root["trace.span_id"]));
|
||||
assert.equal(spans[0].data["ci.finishes_after_workflow"], true);
|
||||
assert.equal(spans[1].data["ci.finishes_after_workflow"], false);
|
||||
assert.equal(root.duration_ms, 20000);
|
||||
assert.equal(root["ci.workflow_ms"], 20000);
|
||||
assert.equal(root["ci.all_checks_ms"], 90000);
|
||||
assert.equal(root["ci.last_to_finish"], "Greptile Review");
|
||||
assert.equal(root["ci.slowest_check"], "Greptile Review");
|
||||
assert.equal(root["ci.slowest_check_ms"], 89000);
|
||||
assert.equal(root["ci.check_count"], 4);
|
||||
assert.equal(root["ci.pending_check_count"], 2);
|
||||
assert.equal(root["ci.pending_checks"], "Scan, slow/status");
|
||||
assert.equal(root["ci.earlier_check_count"], 1);
|
||||
assert.equal(root["ci.profile_complete"], false);
|
||||
assert.match(trace.warnings.join(), /2 outside checks were still running/);
|
||||
assert.doesNotMatch(JSON.stringify(trace), /must not leave GitHub/);
|
||||
assert.match(traceUrl("team", "test", "paperclip-ci", trace), /trace_end_ts=1800000090$/);
|
||||
|
||||
const alone = buildTrace(head, jobs).events[0].data;
|
||||
assert.equal(alone["ci.all_checks_ms"], 20000);
|
||||
assert.equal(alone["ci.last_to_finish"], "PR workflow");
|
||||
assert.equal("ci.slowest_check" in alone, false);
|
||||
// A rerun repeats jobs only; the push's outside checks are not its elapsed time.
|
||||
const rerun = buildTrace({ ...head, run_attempt: 2, run_started_at: at(0) }, jobs.map((entry) => ({ ...entry, run_attempt: 2 })), [], { checks });
|
||||
assert.equal(rerun.events[0].data["ci.check_count"], 0);
|
||||
assert.equal(rerun.events[0].data["ci.all_checks_ms"], 20000);
|
||||
});
|
||||
|
||||
test("export waits a bounded time for outside checks, never for other workflows", async () => {
|
||||
const head = { ...run, path: ".github/workflows/pr.yml" };
|
||||
const check = (completed) => ({ kind: "external_check", app: "greptile-apps", name: "Greptile Review", startedAt: at(1), completedAt: completed ? at(90) : null });
|
||||
const pauses = [];
|
||||
let clock = 0;
|
||||
const now = () => clock;
|
||||
const pause = async (ms) => { pauses.push(ms); clock += ms; };
|
||||
let loads = 0;
|
||||
const settled = await settledChecks(head, () => [check(++loads >= 3)], { pause, now });
|
||||
assert.equal(settled[0].completedAt, at(90));
|
||||
assert.deepEqual(pauses, [30_000, 30_000]);
|
||||
|
||||
pauses.length = 0;
|
||||
const stuck = await settledChecks(head, () => [check(false)], { pause, now, waitMs: 90_000 });
|
||||
assert.equal(stuck[0].completedAt, null);
|
||||
assert.deepEqual(pauses, [30_000, 30_000, 30_000]);
|
||||
|
||||
// Slow GitHub responses count against the same deadline as the pauses.
|
||||
pauses.length = 0;
|
||||
loads = 0;
|
||||
await settledChecks(head, () => { clock += 20_000; loads += 1; return [check(false)]; }, { pause, now, waitMs: 90_000 });
|
||||
assert.deepEqual(pauses, [30_000, 20_000]);
|
||||
assert.equal(loads, 3);
|
||||
|
||||
pauses.length = 0;
|
||||
await settledChecks(head, () => [{ ...check(false), kind: "workflow_check" }], { pause, now });
|
||||
await settledChecks(head, () => [{ ...check(false), startedAt: at(-600) }], { pause, now });
|
||||
await settledChecks({ ...head, run_attempt: 2, run_started_at: at(0) }, () => [check(false)], { pause, now });
|
||||
await settledChecks(head, () => [check(false)], { pause, now, waitMs: 0 });
|
||||
assert.deepEqual(pauses, []);
|
||||
});
|
||||
|
||||
test("profiling uses only infrastructure CODEOWNERS resolved at trusted master", () => {
|
||||
const source = '.github/** @owner @second\ndoc/** @outsider\n';
|
||||
assert.deepEqual(ciOwners(source), ["owner", "second"]);
|
||||
@@ -295,6 +425,21 @@ test("workflow exporter runs trusted code and keeps keys out of the PR jobs", ()
|
||||
const workflow = readFileSync(new URL("../../workflows/ci-profile.yml", import.meta.url), "utf8");
|
||||
assert.match(workflow, /ref: \$\{\{ github.sha \}\}/);
|
||||
assert.match(workflow, /startsWith\(github.event.workflow_run.display_title/);
|
||||
// The job condition keeps outside contributors' runs from starting a runner
|
||||
// or entering the environment, and matches the exporter's sample rule.
|
||||
const condition = workflow.split(" if: >-\n")[1].split("\n runs-on:")[0];
|
||||
for (const required of [
|
||||
"github.repository_id == '1170821064'",
|
||||
"github.event.workflow_run.event == 'pull_request'",
|
||||
"github.event.workflow_run.path == '.github/workflows/pr.yml'",
|
||||
"github.event.workflow_run.head_repository.full_name == 'paperclipai/paperclip'",
|
||||
"contains(vars.AWS_CI_ROUTING_USER_IDS, format('|{0}|', github.event.workflow_run.actor.id))",
|
||||
"contains(vars.AWS_CI_ROUTING_USER_IDS, format('|{0}|', github.event.workflow_run.triggering_actor.id))",
|
||||
"endsWith(format('{0}', github.event.workflow_run.run_number), '0')",
|
||||
]) assert.ok(condition.includes(required), required);
|
||||
assert.equal(condition.match(/\|\|/g).length, 1, "only label-or-sample may be an alternative");
|
||||
assert.equal(workflow.match(/^ [a-z_]+:$/gm).filter((line) => ![" workflow_run:", " export:"].includes(line)).length, 0);
|
||||
assert.equal(workflow.match(/environment:/g).length, 1);
|
||||
assert.match(workflow, /environment: ci-profiling/);
|
||||
assert.doesNotMatch(workflow, /head_sha|secrets: inherit|contents: write|pull-requests: write|pnpm install/);
|
||||
const pr = readFileSync(new URL("../../workflows/pr-trusted.yml", import.meta.url), "utf8");
|
||||
|
||||
@@ -7,17 +7,36 @@ on:
|
||||
|
||||
permissions:
|
||||
actions: read
|
||||
# Check runs and commit statuses time the outside reviews and scans.
|
||||
checks: read
|
||||
contents: read
|
||||
pull-requests: read
|
||||
statuses: read
|
||||
|
||||
jobs:
|
||||
export:
|
||||
# No runner starts, and the environment is never entered, unless all of
|
||||
# these hold:
|
||||
# - The run is the main PR workflow on a branch of this repository.
|
||||
# - Its actor and rerun actor are on the maintainer roster that CI routing
|
||||
# already uses. The exporter then verifies them and the PR author against
|
||||
# master's CODEOWNERS before it reads or sends anything.
|
||||
# - The run carries the profile-ci label, or its run number ends in 0. That
|
||||
# is the one-in-ten sample; scripts/ci-profile/authorize.mjs applies the
|
||||
# same rule.
|
||||
if: >-
|
||||
github.repository == 'paperclipai/paperclip' &&
|
||||
github.repository_id == '1170821064' &&
|
||||
github.event.workflow_run.event == 'pull_request' &&
|
||||
startsWith(github.event.workflow_run.display_title, 'CI profile · PR #')
|
||||
github.event.workflow_run.path == '.github/workflows/pr.yml' &&
|
||||
github.event.workflow_run.head_repository.full_name == 'paperclipai/paperclip' &&
|
||||
contains(vars.AWS_CI_ROUTING_USER_IDS, format('|{0}|', github.event.workflow_run.actor.id)) &&
|
||||
contains(vars.AWS_CI_ROUTING_USER_IDS, format('|{0}|', github.event.workflow_run.triggering_actor.id)) &&
|
||||
(startsWith(github.event.workflow_run.display_title, 'CI profile · PR #') ||
|
||||
endsWith(format('{0}', github.event.workflow_run.run_number), '0'))
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
# Covers the bounded wait for outside checks that outlast the workflow.
|
||||
timeout-minutes: 30
|
||||
# This environment permits only the master branch. Do not use a repository
|
||||
# secret: an in-repository PR must not be able to request this credential.
|
||||
environment: ci-profiling
|
||||
|
||||
+2
-1
@@ -462,7 +462,8 @@ Changing that file selects its consumers. Generated/untracked files and unknown
|
||||
query loaders retain their consumers conservatively. Other filesystem, process
|
||||
and network effects keep their existing fallback.
|
||||
|
||||
For a full wall-clock profile of all main PR CI jobs, add `profile-ci` to the PR.
|
||||
One in ten CI CODEOWNER PR runs sends its job, step and outside-check timings to
|
||||
Honeycomb. For a full wall-clock profile of all main PR CI jobs, add `profile-ci` to the PR.
|
||||
See [CI profiling](ci-profiling.md) for the Honeycomb waterfall, coverage,
|
||||
operator setup, and activation requirements.
|
||||
|
||||
|
||||
+65
-7
@@ -1,4 +1,53 @@
|
||||
# Profile the complete PR suite
|
||||
# CI timing in Honeycomb
|
||||
|
||||
CI CODEOWNER PRs send their CI timing to Honeycomb in two ways:
|
||||
|
||||
| | Sampled (always on) | `profile-ci` label |
|
||||
| --- | --- | --- |
|
||||
| Which runs | One `PR` run in ten, by run number | Every run while the label is on |
|
||||
| CI scope | Whatever the PR normally runs | All lanes of the main PR suite |
|
||||
| Trace detail | Workflow, jobs, steps, outside checks | The same, plus test files and tests |
|
||||
| Sample rate on spans | 10 | 1 |
|
||||
|
||||
Both are exported by the **CI profile** workflow after the `PR` run finishes,
|
||||
to the `paperclip-ci` dataset. Outside contributors' PRs are never exported.
|
||||
|
||||
## Sampled runs
|
||||
|
||||
A `PR` run is sampled when its run number ends in 0. Sampling keeps or drops a
|
||||
whole run: a sampled trace has every job and step. Each span carries a sample
|
||||
rate of 10, so Honeycomb's counts and sums scale back to all runs, and duration
|
||||
percentiles need no correction. `ci.profile_source` is `sampled`, and
|
||||
`ci.pr_number` identifies the PR. Sampled runs change nothing in the PR jobs
|
||||
and write no timing artifacts; job and step times come from the GitHub API.
|
||||
|
||||
A particular PR's run is in Honeycomb only if it was sampled. To inspect one
|
||||
PR, add the label.
|
||||
|
||||
## Outside checks and total time
|
||||
|
||||
Every trace also has one span for each check on the PR's head commit that is
|
||||
not a job of the `PR` workflow:
|
||||
|
||||
- `ci.kind=external_check`: reviews and scans from other apps, such as
|
||||
Greptile, and commit statuses (`ci.check_app=commit-status`).
|
||||
- `ci.kind=workflow_check`: other workflows the PR started, one span each.
|
||||
|
||||
The root span has two totals. `ci.workflow_ms` is the `PR` workflow, from run
|
||||
creation to its last job. `ci.all_checks_ms` runs from the same start to the
|
||||
last check of any kind, and `ci.last_to_finish` names it. `ci.slowest_check`
|
||||
and `ci.slowest_check_ms` name the longest outside check.
|
||||
|
||||
If an outside app's check is still running when the `PR` workflow finishes,
|
||||
the exporter waits up to 15 minutes for it. It does not wait for other
|
||||
workflows. Checks still running at export have no span; they are counted in
|
||||
`ci.pending_check_count` and listed in `ci.pending_checks`, and
|
||||
`ci.all_checks_ms` excludes them. Checks that started more than a minute
|
||||
before the run, such as those from a push before a later label event, are
|
||||
counted in `ci.earlier_check_count` and left out. Reruns carry no outside
|
||||
checks. A workflow's completion time is its last update in GitHub.
|
||||
|
||||
## Profile the complete PR suite
|
||||
|
||||
Add the `profile-ci` label to a PR authored by a CI CODEOWNER. Adding the label starts a fresh `PR` workflow
|
||||
run. Further pushes while the label remains also run with profiling enabled.
|
||||
@@ -20,8 +69,9 @@ A label on an outside contributor's PR does not enable profiling, even when a
|
||||
maintainer applies it. The PR still runs normal CI and needs no Honeycomb
|
||||
access. Failed identity lookups disable profiling.
|
||||
|
||||
Ordinary PRs do not write timing artifacts or export traces. The exporter job is
|
||||
skipped for ordinary run names. Adding another label does not run the CI matrix.
|
||||
Ordinary PRs do not write timing artifacts. The exporter job is skipped for
|
||||
ordinary run names unless the run is sampled. Adding another label does not run
|
||||
the CI matrix.
|
||||
|
||||
Profiling preserves the existing trusted AWS runner routing. Eligible PR jobs
|
||||
use `paperclip-public-pr-x64`. The synchronized maintainer admission roster
|
||||
@@ -40,8 +90,8 @@ cannot be authorized. The separate post-run Honeycomb exporter also uses
|
||||
The workflow root measures elapsed time from run creation to the last job's
|
||||
completion. For reruns, it starts at that attempt's `run_started_at` instead of
|
||||
the original creation time. Every job is a child on the same wall clock, with
|
||||
its steps and available test details beneath it. All spans use a sample rate of
|
||||
one. There is no sampling within the selected attempt. For a compact Gantt view,
|
||||
its steps and available test details beneath it. Labelled runs use a sample
|
||||
rate of one. There is no sampling within the selected attempt. For a compact Gantt view,
|
||||
open a job row's menu and choose **Collapse spans at this depth**. Expand an
|
||||
individual job to inspect its steps and test files.
|
||||
|
||||
@@ -129,8 +179,16 @@ Create the repository label `profile-ci` if it does not exist.
|
||||
the exporter, installs no dependencies, and parses PR artifacts as bounded JSON
|
||||
without extracting or executing their contents. The PR jobs never receive the
|
||||
Honeycomb key. The exporter independently verifies CI CODEOWNER identities,
|
||||
the current label, repository, and exact tested PR revision before reading
|
||||
artifacts or sending data. Its credential is sent only in an HTTPS header to
|
||||
the repository, the exact tested PR revision and, for labelled runs, the
|
||||
current label before reading artifacts or sending data.
|
||||
|
||||
The export job's condition is a first filter, evaluated before a runner starts
|
||||
or the environment is entered. It requires the main `PR` workflow on a branch
|
||||
of this repository, an actor and rerun actor on the `AWS_CI_ROUTING_USER_IDS`
|
||||
maintainer roster that CI routing uses, and a labelled or sampled run. A fork
|
||||
PR, or a run started or rerun by anyone off the roster, skips the job. The
|
||||
roster only filters: the CODEOWNERS check in the exporter decides. If the
|
||||
roster variable is empty, nothing is exported. Its credential is sent only in an HTTPS header to
|
||||
`api.honeycomb.io`; redirects are rejected. Exported fields are allowlisted.
|
||||
`CODEOWNERS` also covers `scripts/ci-profile/**`, so changes to the privileged
|
||||
exporter require infrastructure-owner review.
|
||||
|
||||
@@ -41,12 +41,29 @@ export function authorizedActors(run, ownerIds) {
|
||||
&& allowed(run.actor) && allowed(run.triggering_actor);
|
||||
}
|
||||
|
||||
export function associatedProfilePr(run, pulls, ownerIds) {
|
||||
// The PR at this exact revision, when its author, the run's actor and the
|
||||
// rerun actor are all CI CODEOWNERS.
|
||||
export function associatedOwnerPr(run, pulls, ownerIds, accept = () => true) {
|
||||
if (!authorizedActors(run, ownerIds)) return undefined;
|
||||
return pulls.find((pr) => pr.base.repo.id === repositoryId && pr.base.repo.full_name === repository
|
||||
&& pr.head.repo?.id === run.head_repository?.id && pr.head.sha === run.head_sha
|
||||
&& pr.user?.type === "User" && ownerIds.includes(pr.user.id)
|
||||
&& pr.labels.some((label) => label.name === "profile-ci"));
|
||||
&& pr.user?.type === "User" && ownerIds.includes(pr.user.id) && accept(pr));
|
||||
}
|
||||
|
||||
export function associatedProfilePr(run, pulls, ownerIds) {
|
||||
return associatedOwnerPr(run, pulls, ownerIds, (pr) => pr.labels.some((label) => label.name === "profile-ci"));
|
||||
}
|
||||
|
||||
// Always-on export keeps one run in this many, chosen by the workflow's
|
||||
// sequential run number so that ci-profile.yml can skip the others before
|
||||
// starting a runner.
|
||||
export const sampleRate = 10;
|
||||
|
||||
// "label": a profile-ci run with test detail, always exported.
|
||||
// "sampled": an ordinary run chosen by its number, job and step timings only.
|
||||
export function exportMode(run) {
|
||||
if (typeof run.display_title === "string" && run.display_title.startsWith("CI profile · PR #")) return "label";
|
||||
return Number.isSafeInteger(run.run_number) && run.run_number > 0 && run.run_number % sampleRate === 0 ? "sampled" : undefined;
|
||||
}
|
||||
|
||||
async function main() {
|
||||
|
||||
@@ -3,8 +3,8 @@ import { appendFileSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { buildTrace, traceUrl } from "./trace.mjs";
|
||||
import { associatedProfilePr, authorizedActors, trustedOwnerIds } from "./authorize.mjs";
|
||||
import { buildTrace, normalizeChecks, pendingChecks, traceUrl } from "./trace.mjs";
|
||||
import { associatedOwnerPr, associatedProfilePr, authorizedActors, exportMode, sampleRate, trustedOwnerIds } from "./authorize.mjs";
|
||||
|
||||
const repository = "paperclipai/paperclip";
|
||||
const repoPath = `/repos/${repository}`;
|
||||
@@ -46,6 +46,21 @@ export async function sendEvents(events, apiKey, dataset, fetcher = fetch, pause
|
||||
}
|
||||
}
|
||||
|
||||
// Reviews and scans from outside apps can outlast the workflow. Wait a bounded
|
||||
// time so the trace records how long they took; whatever is still running
|
||||
// after that is reported as pending instead of holding the export open.
|
||||
export async function settledChecks(run, load, { waitMs = 15 * 60_000, intervalMs = 30_000,
|
||||
pause = (ms) => new Promise((resolve) => setTimeout(resolve, ms)), now = Date.now } = {}) {
|
||||
// The deadline is on the wall clock, so slow GitHub responses count against it.
|
||||
const deadline = now() + waitMs;
|
||||
let checks = load();
|
||||
while (now() < deadline && pendingChecks(run, checks).length) {
|
||||
await pause(Math.min(intervalMs, deadline - now()));
|
||||
checks = load();
|
||||
}
|
||||
return checks;
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const args = process.argv.slice(2);
|
||||
const arg = (name) => args.find((value) => value.startsWith(`${name}=`))?.slice(name.length + 1);
|
||||
@@ -76,20 +91,31 @@ async function main() {
|
||||
return;
|
||||
}
|
||||
}
|
||||
const pr = historical ? undefined : associatedProfilePr(run, pulls, ownerIds);
|
||||
if (!pr && !historical) {
|
||||
console.log("No CODEOWNERS-authorized profile-ci PR at this revision; no data exported.");
|
||||
return;
|
||||
}
|
||||
const jobs = pages(`${repoPath}/actions/runs/${runId}/attempts/${attempt}/jobs?per_page=100`, "jobs");
|
||||
const artifacts = pages(`${repoPath}/actions/runs/${runId}/artifacts?per_page=100`, "artifacts")
|
||||
.filter((artifact) => artifact.name.startsWith(`ci-profile-${attempt}-`));
|
||||
// The run name snapshots selection at dispatch. A label added while an
|
||||
// ordinary run finishes must not retroactively select that earlier run.
|
||||
if (!historical && !run.display_title.startsWith("CI profile · PR #")) {
|
||||
// The run name snapshots label selection at dispatch. A label added while
|
||||
// an ordinary run finishes must not retroactively select that earlier run;
|
||||
// such a run is exported only if its run number is in the sample.
|
||||
const mode = historical ? "historical" : exportMode(run);
|
||||
if (!mode) {
|
||||
console.log("This attempt was not selected for profiling; no data exported.");
|
||||
return;
|
||||
}
|
||||
const pr = mode === "label" ? associatedProfilePr(run, pulls, ownerIds)
|
||||
: mode === "sampled" ? associatedOwnerPr(run, pulls, ownerIds) : undefined;
|
||||
if (!pr && !historical) {
|
||||
console.log("No CODEOWNERS-authorized PR at this revision; no data exported.");
|
||||
return;
|
||||
}
|
||||
const jobs = pages(`${repoPath}/actions/runs/${runId}/attempts/${attempt}/jobs?per_page=100`, "jobs");
|
||||
// Sampled runs wrote no timing artifacts, so none are read.
|
||||
const artifacts = mode === "sampled" ? [] : pages(`${repoPath}/actions/runs/${runId}/artifacts?per_page=100`, "artifacts")
|
||||
.filter((artifact) => artifact.name.startsWith(`ci-profile-${attempt}-`));
|
||||
if (!/^[a-f\d]{40}$/.test(run.head_sha)) throw new Error("Invalid head commit");
|
||||
const loadChecks = () => normalizeChecks(run, {
|
||||
checkRuns: pages(`${repoPath}/commits/${run.head_sha}/check-runs?per_page=100`, "check_runs"),
|
||||
statuses: pages(`${repoPath}/commits/${run.head_sha}/statuses?per_page=100`),
|
||||
workflowRuns: pages(`${repoPath}/actions/runs?head_sha=${run.head_sha}&per_page=100`, "workflow_runs"),
|
||||
});
|
||||
const checks = await settledChecks(run, loadChecks, args.includes("--dry-run") ? { waitMs: 0 } : {});
|
||||
const directory = mkdtempSync(path.join(tmpdir(), "paperclip-ci-export-"));
|
||||
const records = [];
|
||||
let bytes = 0;
|
||||
@@ -108,8 +134,11 @@ async function main() {
|
||||
} finally {
|
||||
rmSync(directory, { recursive: true, force: true });
|
||||
}
|
||||
const trace = buildTrace(run, jobs, records);
|
||||
trace.events[0].data["ci.profile_source"] = historical ? "historical" : "label";
|
||||
const trace = buildTrace(run, jobs, records, {
|
||||
sampleRate: mode === "sampled" ? sampleRate : 1, checks, testDetail: mode !== "sampled",
|
||||
});
|
||||
trace.events[0].data["ci.profile_source"] = mode;
|
||||
if (pr) trace.events[0].data["ci.pr_number"] = pr.number;
|
||||
const dataset = process.env.HONEYCOMB_CI_DATASET || "paperclip-ci";
|
||||
const url = traceUrl(process.env.HONEYCOMB_TEAM || "paperclip", process.env.HONEYCOMB_ENVIRONMENT || "test", dataset, trace);
|
||||
if (arg("--output")) writeFileSync(arg("--output"), JSON.stringify(trace, null, 2));
|
||||
@@ -123,6 +152,8 @@ async function main() {
|
||||
`### CI profile${pr ? ` for PR #${pr.number}` : " (historical job/step baseline)"}`,
|
||||
"", `[Open the full CI waterfall in Honeycomb](${url})`, "",
|
||||
`${trace.jobCount} jobs · ${(trace.wallMs / 1000).toFixed(1)} seconds elapsed · ${trace.events.length} spans`,
|
||||
"", `${trace.checkCount} outside checks · ${(trace.windowMs / 1000).toFixed(1)} seconds until the last one finished`,
|
||||
...(mode === "sampled" ? ["", `Sampled run (1 in ${sampleRate}): job and step timings, without per-test detail.`] : []),
|
||||
"", `GitHub run: ${run.html_url}/attempts/${attempt}`, "",
|
||||
...trace.warnings.map((warning) => `- ${warning}`),
|
||||
"", "The workflow root measures elapsed time. Parallel job durations must not be summed as elapsed time.", "",
|
||||
|
||||
@@ -2,10 +2,73 @@ import { createHash } from "node:crypto";
|
||||
|
||||
const millis = (value) => Date.parse(value);
|
||||
const id = (value, size = 16) => createHash("sha256").update(value).digest("hex").slice(0, size);
|
||||
const failed = (status) => ["failure", "failed", "timed_out", "timedOut", "cancelled", "interrupted"].includes(status);
|
||||
// "error" is a commit status that could not complete.
|
||||
const failed = (status) => ["failure", "failed", "error", "timed_out", "timedOut", "cancelled", "interrupted"].includes(status);
|
||||
const text = (value) => typeof value === "string" && value.length <= 4096;
|
||||
|
||||
export function buildTrace(run, jobs, records = []) {
|
||||
// Outside checks belong to the push that started this run. A check that began
|
||||
// well before the run (for example, before a later label event) is not part of
|
||||
// this run's elapsed time.
|
||||
const checkSlackMs = 60_000;
|
||||
const label = (value) => typeof value === "string" ? value.slice(0, 256) : "";
|
||||
|
||||
// Reduce GitHub's check-run, commit-status and workflow-run listings for the
|
||||
// run's head commit to the allowlisted fields the trace needs.
|
||||
export function normalizeChecks(run, { checkRuns = [], statuses = [], workflowRuns = [] } = {}) {
|
||||
const checks = [];
|
||||
for (const check of checkRuns) {
|
||||
// GitHub Actions check runs are jobs. This run's jobs are already spans,
|
||||
// and other workflows are summarized per workflow below.
|
||||
if (!check?.app?.slug || check.app.slug === "github-actions") continue;
|
||||
checks.push({
|
||||
kind: "external_check", app: label(check.app.slug), name: label(check.name),
|
||||
status: label(check.conclusion || check.status), startedAt: check.started_at,
|
||||
completedAt: check.status === "completed" ? check.completed_at : null,
|
||||
});
|
||||
}
|
||||
const contexts = new Map();
|
||||
for (const status of statuses) {
|
||||
if (typeof status?.context !== "string") continue;
|
||||
contexts.set(status.context, [...(contexts.get(status.context) || []), status]);
|
||||
}
|
||||
for (const [context, entries] of contexts) {
|
||||
entries.sort((a, b) => millis(a.created_at) - millis(b.created_at));
|
||||
const last = entries.at(-1);
|
||||
checks.push({
|
||||
kind: "external_check", app: "commit-status", name: label(context), status: label(last.state),
|
||||
startedAt: entries[0].created_at, completedAt: last.state === "pending" ? null : last.created_at,
|
||||
});
|
||||
}
|
||||
const workflows = new Map();
|
||||
for (const other of workflowRuns) {
|
||||
if (other?.id === run.id || other?.path === run.path || other?.head_sha !== run.head_sha
|
||||
|| !["pull_request", "pull_request_target"].includes(other?.event)) continue;
|
||||
const known = workflows.get(other.path);
|
||||
if (!known || millis(other.created_at) > millis(known.created_at)) workflows.set(other.path, other);
|
||||
}
|
||||
for (const other of workflows.values()) {
|
||||
checks.push({
|
||||
kind: "workflow_check", app: "github-actions", name: label(other.name),
|
||||
status: label(other.conclusion || other.status),
|
||||
startedAt: other.run_attempt > 1 ? other.run_started_at : other.created_at,
|
||||
// GitHub exposes no completion time for a run; its last update is the closest record.
|
||||
completedAt: other.status === "completed" ? other.updated_at : null,
|
||||
});
|
||||
}
|
||||
return checks;
|
||||
}
|
||||
|
||||
const runStart = (run) => millis(run.run_attempt === 1 ? run.created_at : run.run_started_at);
|
||||
// A rerun repeats jobs, not the outside checks, so only a first attempt owns them.
|
||||
const ownChecks = (run, checks) => run.run_attempt === 1
|
||||
? checks.filter((check) => !(millis(check.startedAt) < runStart(run) - checkSlackMs)) : [];
|
||||
|
||||
export function pendingChecks(run, checks, kind = "external_check") {
|
||||
return ownChecks(run, checks).filter((check) => check.kind === kind && !check.completedAt);
|
||||
}
|
||||
|
||||
export function buildTrace(run, jobs, records = [], { sampleRate = 1, checks = [], testDetail = true } = {}) {
|
||||
if (!Number.isSafeInteger(sampleRate) || sampleRate < 1) throw new Error("Invalid sample rate");
|
||||
if (run.status !== "completed" || !jobs.length || jobs.some((job) => job.status !== "completed")) {
|
||||
throw new Error("A complete workflow attempt and its full job inventory are required");
|
||||
}
|
||||
@@ -15,7 +78,7 @@ export function buildTrace(run, jobs, records = []) {
|
||||
const key = `${run.repository.full_name}:${run.id}:${run.run_attempt}`;
|
||||
const traceId = id(key, 32);
|
||||
const rootId = id(`${key}:root`);
|
||||
const start = millis(run.run_attempt === 1 ? run.created_at : run.run_started_at);
|
||||
const start = runStart(run);
|
||||
// The attempt jobs API can relabel carried-over successful jobs with the
|
||||
// new attempt number. Their completion clocks still precede this attempt.
|
||||
const carriedJobs = run.run_attempt > 1 ? jobs.filter((job) => millis(job.completed_at) < start) : [];
|
||||
@@ -39,7 +102,9 @@ export function buildTrace(run, jobs, records = []) {
|
||||
const span = (spanId, parent, name, from, to, fields) => {
|
||||
if (!Number.isFinite(from) || !Number.isFinite(to) || to < from) return;
|
||||
events.push({
|
||||
time: new Date(from).toISOString(), samplerate: 1,
|
||||
// Sampling selects whole runs. Every span of a selected run is kept, and
|
||||
// this weight lets Honeycomb scale counts back to all runs.
|
||||
time: new Date(from).toISOString(), samplerate: sampleRate,
|
||||
data: { ...shared, "trace.span_id": spanId, ...(parent ? { "trace.parent_id": parent } : {}),
|
||||
name, duration_ms: to - from, ...fields },
|
||||
});
|
||||
@@ -87,6 +152,32 @@ export function buildTrace(run, jobs, records = []) {
|
||||
}
|
||||
}
|
||||
|
||||
// Reviews, scans and other workflows on the same commit run beside this
|
||||
// workflow. They are siblings of the jobs on the same wall clock.
|
||||
const own = ownChecks(run, checks);
|
||||
const stale = run.run_attempt === 1 ? checks.length - own.length : 0;
|
||||
const pending = own.filter((check) => !check.completedAt);
|
||||
let lastEnd = end;
|
||||
let lastName = "PR workflow";
|
||||
let slowest;
|
||||
let exported = 0;
|
||||
for (const check of own) {
|
||||
const from = millis(check.startedAt);
|
||||
const to = millis(check.completedAt);
|
||||
// Skipped checks report a completion before their start; they did no work.
|
||||
if (!check.completedAt || check.status === "skipped" || !Number.isFinite(from) || !Number.isFinite(to) || to < from) continue;
|
||||
span(id(`${key}:check:${check.kind}:${check.app}:${check.name}`), rootId, check.name, from, to, {
|
||||
"ci.kind": check.kind, "ci.check": check.name, "ci.check_app": check.app,
|
||||
"ci.status": check.status, error: failed(check.status),
|
||||
"ci.start_offset_ms": from - start,
|
||||
"ci.finishes_after_workflow": to > end,
|
||||
});
|
||||
exported++;
|
||||
if (to > lastEnd) { lastEnd = to; lastName = check.name; }
|
||||
if (!slowest || to - from > slowest.ms) slowest = { name: check.name, ms: to - from };
|
||||
}
|
||||
if (pending.length) warnings.push(`${pending.length} outside checks were still running at export`);
|
||||
|
||||
const receipts = new Set();
|
||||
const invocations = new Map();
|
||||
const valid = [];
|
||||
@@ -138,7 +229,9 @@ export function buildTrace(run, jobs, records = []) {
|
||||
...(Number.isFinite(record.setupMs) ? { "test.setup_ms": record.setupMs } : {}),
|
||||
});
|
||||
}
|
||||
const expectedReceipts = jobs.filter((job) => job.steps?.some((step) => step.name === "Initialize CI profile"));
|
||||
// Ordinary runs skip the collection steps, so they owe no timing artifacts.
|
||||
const expectedReceipts = testDetail
|
||||
? jobs.filter((job) => job.steps?.some((step) => step.name === "Initialize CI profile")) : [];
|
||||
const missing = expectedReceipts.filter((job) => !receipts.has(job.id));
|
||||
if (missing.length) warnings.push(`Missing timing artifacts for ${missing.length} jobs`);
|
||||
// Some package scripts invoke a reporter without using the launcher. Its
|
||||
@@ -160,20 +253,33 @@ export function buildTrace(run, jobs, records = []) {
|
||||
return framework && !valid.some((entry) => entry.job.id === job.id && entry.record.framework === framework);
|
||||
});
|
||||
if (expectedReceipts.length && missingTests.length) warnings.push(`Missing test detail for ${missingTests.length} test jobs (including jobs that failed before testing)`);
|
||||
root["ci.sample_rate"] = sampleRate;
|
||||
root["ci.workflow_ms"] = end - start;
|
||||
// Push to the last finished check. Checks still running at export are excluded and counted.
|
||||
root["ci.all_checks_ms"] = lastEnd - start;
|
||||
root["ci.last_to_finish"] = lastName;
|
||||
root["ci.check_count"] = exported;
|
||||
root["ci.pending_check_count"] = pending.length;
|
||||
root["ci.pending_checks"] = pending.map((check) => check.name).join(", ").slice(0, 1024);
|
||||
root["ci.earlier_check_count"] = stale;
|
||||
if (slowest) {
|
||||
root["ci.slowest_check"] = slowest.name;
|
||||
root["ci.slowest_check_ms"] = slowest.ms;
|
||||
}
|
||||
root["ci.test_count"] = valid.filter(({ record }) => record.kind === "test").length;
|
||||
root["ci.file_count"] = valid.filter(({ record }) => record.kind === "file").length;
|
||||
root["ci.expected_invocations"] = invocations.size;
|
||||
root["ci.reported_invocations"] = [...invocations.values()].filter((value) => value.start && value.end).length;
|
||||
root["ci.profile_complete"] = warnings.length === 0 && run.run_attempt === 1;
|
||||
root["ci.coverage_warnings"] = warnings.join("; ");
|
||||
return { traceId, events, warnings, wallMs: end - start, jobCount: jobs.length };
|
||||
return { traceId, events, warnings, wallMs: end - start, windowMs: lastEnd - start, jobCount: jobs.length, checkCount: exported };
|
||||
}
|
||||
|
||||
export function traceUrl(team, environment, dataset, trace) {
|
||||
const params = new URLSearchParams({
|
||||
trace_id: trace.traceId,
|
||||
trace_start_ts: String(Math.floor(Date.parse(trace.events[0].time) / 1000)),
|
||||
trace_end_ts: String(Math.ceil((Date.parse(trace.events[0].time) + trace.wallMs) / 1000)),
|
||||
trace_end_ts: String(Math.ceil((Date.parse(trace.events[0].time) + (trace.windowMs ?? trace.wallMs)) / 1000)),
|
||||
});
|
||||
return `https://ui.honeycomb.io/${encodeURIComponent(team)}/environments/${encodeURIComponent(environment)}/datasets/${encodeURIComponent(dataset)}/trace?${params}`;
|
||||
}
|
||||
Reference in new issue
Block a user