mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - PR CI runs about 50 jobs per run on AWS spot instances. AWS can reclaim a machine mid-job, and the job dies. > - `ci-runner-loss-rerun.yml` re-runs such a run once, automatically, when every failed job lost its runner. > - In a 17-hour sample, 42 runs lost a runner and were still the PR's current head. The automation rescued only 14. > - Two gaps caused most of the misses. A spot reclaim on RunsOn v3.2.3 usually ends the job, and the run, as `cancelled`; the workflow and script only look at `failure`. Also, the Runner support suite always carries `gateway_indexing_incomplete` error annotations from a passing test, so a lost runner there reads as a real failure. > - On-demand fallback for the fleet was turned off on 2026-11-13, so reclaims will happen more often. About 20% of runs are expected to lose at least one job. > - This pull request makes the re-run act on cancelled runs and jobs, but only with positive runner-loss evidence, and lets that evidence win over other error annotations. > - The benefit is fewer manual "Re-run failed jobs" clicks: an estimated 34 of 42 instead of 14 of 42 in the sample. ## Linked Issues or Issue Description No public issue exists. **What happened?** Run [43285064435](https://github.com/paperclipai/paperclip/actions/runs/43285064435) lost the runner of `ci / General tests (server (8/12))`. The job concluded `cancelled` (its log has `##[error]The runner has received a shutdown signal`). The run concluded `cancelled`, so the re-run workflow's `if:` skipped it. Run [43325855069](https://github.com/paperclipai/paperclip/actions/runs/43325855069) lost the runners of `ci / Runner support suite` and `ci / General tests (server (2/12))`. Both logs have the shutdown signal. The re-run bot ([decision log](https://github.com/paperclipai/paperclip/actions/runs/43327873358)) said "1 of 1 failed jobs did not lose their runner", with evidence `gateway_indexing_incomplete`. It ignored the cancelled job. It read the annotation as a real failure. **Expected behavior** Both runs get one automatic re-run of their failed jobs. **Steps to reproduce** The new tests replay trimmed snapshots of both runs (`.github/scripts/tests/fixtures/runner-loss-rerun/run-43285064435.json` and `run-43325855069.json`) through the script. In a 17-hour sample on 2026-11-12, 42 runs lost a runner and were still the PR's current head. Only 14 were re-run automatically. 16 stayed red because the run concluded `cancelled`. 4 stayed red because an error annotation in the lost job was read as a test failure. **Additional context** The `gateway_indexing_incomplete` annotation does not come from the shutdown. `scheduled-gateway-indexing.test.ts` prints `::error` lines while it passes, 84 seconds before the shutdown signal in this run. Attempt 2 of the same run passed with the same two failure annotations. So, before this change, a lost runner in the Runner support suite could never be re-run. ## What Changed - `ci-runner-loss-rerun.yml`: the job also runs when the PR run concluded `cancelled`. The trigger type, `permissions`, the repository, path, event and attempt-1 conditions do not change. - `runner-loss-rerun.mjs`, `eligibleRun`: accepts `completed/cancelled` as well as `completed/failure`. - `runner-loss-rerun.mjs`, `decide()`: classifies jobs that concluded `cancelled` as well as `failure`. It still skips the `ci / verify` and `ci / e2e` aggregates, and a failed or cancelled routing gate still blocks the re-run. - `decide()` now runs `staleReason` (open PR, current head, no newer run) before it reads any annotation or log, and again just before the re-run, as on master. A run cancelled by a new push has about 40 cancelled jobs; it now costs two API listings and no log reads. The final check closes the window where a push lands during the log reads: a re-run of the old run would join the PR's `cancel-in-progress` concurrency group (`pr-<number>` in `pr-trusted.yml`) and cancel the new commit's CI. - Following CONTRIBUTING.md ("Do not submit a PR that includes code comments"), this PR adds no new code comments. The reasons for each rule are in this description and in `doc/DEVELOPING.md`. - `classifyJob`: runner-loss evidence wins. A lost-communication annotation means lost. A cancellation annotation plus a shutdown signal in the log means lost, even if the job also has other error annotations. A cancelled job without this evidence (for example, a person cancelled the run) counts as a real failure and blocks the re-run. - Fixtures: two new trimmed snapshots from the example runs (jobs with `id`, `name`, `status`, `conclusion`; failure annotations; `##[error]` log lines only). The two older fixtures get the real annotations and log error lines of their cancelled jobs. Those jobs were spot-reclaimed too (each log has the shutdown signal), but the old code ignored them. - `doc/DEVELOPING.md`: the runner-loss paragraph describes the new rules. ## Verification - Red first. Against the unchanged script, the new tests fail for the reasons above: 43285064435 gives `run is completed/cancelled, not completed/failure`; 43325855069 gives `1 of 1 failed jobs did not lose their runner` with evidence `gateway_indexing_incomplete`, the same decision the bot logged. - Green: `node --test .github/scripts/tests/runner-loss-rerun.test.mjs`: 25 of 25 pass. - Push during log reads: the PR head moves after the first log read. With the final freshness check removed, this test fails (the script re-runs); with it, there is no re-run and the reason is `no open pull request still has this run's head commit`. - Guard cases, all with no re-run and no POST: - Superseded run: 52 jobs cancelled, the PR head moved. The test asserts zero annotation and log reads. - Person cancelled the run: cancelled jobs with only "The operation was canceled." in the annotation and log, and some never-started jobs with no annotation. - Mixed: one cancelled lost job and one real test failure (`1 of 2 failed jobs did not lose their runner`). - Once only: a cancelled run on attempt 2 is refused after one API read. The workflow condition test still requires `run_attempt == 1`. - Cancelled routing gate: left for a person. - Policy job command: `node --test '.github/scripts/tests/*.test.mjs'`: 806 of 806 pass. - `pnpm verify`: exit 2 ("Global CI input changed: .github/scripts/runner-loss-rerun.mjs"). It selects no local Vitest files and asks for required CI coverage. I did not run `pnpm verify --full`; required CI on this PR supplies that coverage. - This workflow runs on `workflow_run` from the default branch. This PR's own CI cannot exercise the new behavior. The fixture tests are the evidence. I did not dry-run the script against live runs. ## Risks - A test that really fails, and then a spot reclaim of the same job a minute later, now gets one wasted re-run. Attempt 2 then fails for real. It cannot loop: the workflow `if:` and `eligibleRun` both accept attempt 1 only. - A run a person cancels on the current head now starts the re-run job. The job reads the annotations, and the log of each job with a cancellation annotation, and then does not re-run. This costs some API calls on a GitHub-hosted runner. - A concurrency cancel by a label event that does not request CI (the newer run is `skipped`) passes the head check. Its cancelled jobs have no runner-loss evidence, so it is not re-run. - Trust: no change to `permissions`, the trigger, the accepted repository, event, workflow path, attempt, or the routing-gate rule. A cancelled run reuses its attempt-1 gate output, the same as a failed run does today. - Estimated effect, from a 17-hour sample: 14 of 42 rescued before; 16 cancelled runs and 4 runs with error-annotation noise become eligible (about 34 of 42). The other 8 stay skipped on purpose (a real test failure in the same run, a person re-ran first, or the PR closed). > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), Claude Code agent. Model ID claude-opus-5-5, with tool use (shell, GitHub CLI) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
460 lines
23 KiB
JavaScript
460 lines
23 KiB
JavaScript
import assert from "node:assert/strict";
|
|
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
|
import { tmpdir } from "node:os";
|
|
import path from "node:path";
|
|
import test from "node:test";
|
|
import {
|
|
classifyJob, createGithubClient, decide, logShowsRunnerLoss, main, renderSummary,
|
|
} from "../runner-loss-rerun.mjs";
|
|
|
|
// Trimmed attempt-1 snapshots of real PR runs: jobs, check-run annotations for
|
|
// each failed job, and the error lines of job logs whose only annotation is
|
|
// "The operation was canceled.".
|
|
const fixture = (runId) => JSON.parse(readFileSync(
|
|
new URL(`./fixtures/runner-loss-rerun/run-${runId}.json`, import.meta.url), "utf8",
|
|
));
|
|
const MIXED = 42271918832; // six lost runners, four shutdowns in the log, one real test timeout
|
|
const REAL = 42264785415; // a real assertion failure
|
|
const ALL_LOST = 42272083862; // one lost runner
|
|
const LOG_ONLY = 42277969418; // shutdown signals visible only in the job logs
|
|
const CANCELLED_RUN = 43285064435;
|
|
const NOISY_LOST = 43325855069;
|
|
const CANCELLED_MESSAGE = "The operation was canceled.";
|
|
const workflow = readFileSync(new URL("../../workflows/ci-runner-loss-rerun.yml", import.meta.url), "utf8");
|
|
|
|
const json = (body, headers = {}) => new Response(JSON.stringify(body), {
|
|
status: 200, headers: { "content-type": "application/json", ...headers },
|
|
});
|
|
|
|
/**
|
|
* Serves a fixture as the GitHub REST API. By default the run is the latest
|
|
* CI for an open PR's current head.
|
|
*/
|
|
function fakeGithub(data, {
|
|
jobsPageSize = 100, pulls, prRuns, mutateRun = (run) => run,
|
|
} = {}) {
|
|
const calls = [];
|
|
const run = mutateRun(structuredClone(data.run));
|
|
const openPr = {
|
|
number: 1, state: "open",
|
|
base: { repo: { id: data.run.repository.id } },
|
|
head: { sha: data.run.head_sha, repo: { id: data.run.head_repository.id } },
|
|
};
|
|
const fetcher = async (href, init) => {
|
|
const url = new URL(href);
|
|
calls.push({ method: init.method, path: url.pathname, query: url.search, auth: init.headers.Authorization });
|
|
const base = `/repos/paperclipai/paperclip`;
|
|
const route = url.pathname.slice(base.length);
|
|
if (init.method === "POST" && route === `/actions/runs/${run.id}/rerun-failed-jobs`) {
|
|
return new Response(null, { status: 201 });
|
|
}
|
|
if (route === `/actions/runs/${run.id}`) return json(run);
|
|
if (route === `/actions/runs/${run.id}/attempts/1/jobs`) {
|
|
const page = Number(url.searchParams.get("page") ?? 1);
|
|
const jobs = data.jobs.slice((page - 1) * jobsPageSize, page * jobsPageSize);
|
|
const last = page * jobsPageSize >= data.jobs.length;
|
|
const next = new URL(url);
|
|
next.searchParams.set("page", String(page + 1));
|
|
return json({ total_count: data.total_count, jobs }, last ? {} : { link: `<${next.href}>; rel="next"` });
|
|
}
|
|
let match = route.match(/^\/check-runs\/(\d+)\/annotations$/);
|
|
if (match) return json(data.annotations[match[1]] ?? []);
|
|
match = route.match(/^\/actions\/jobs\/(\d+)\/logs$/);
|
|
if (match) return new Response(data.logs[match[1]] ?? "", { status: 200 });
|
|
if (route === `/commits/${run.head_sha}/pulls`) return json((typeof pulls === "function" ? pulls(calls) : pulls) ?? [openPr]);
|
|
if (route === "/actions/workflows/pr.yml/runs") {
|
|
const runs = prRuns ?? [{ id: run.id, conclusion: "failure" }];
|
|
return json({ total_count: runs.length, workflow_runs: runs });
|
|
}
|
|
return new Response("not found", { status: 404 });
|
|
};
|
|
return { calls, fetcher, run, client: createGithubClient({ token: "test-token", fetcher }) };
|
|
}
|
|
|
|
const posted = (calls) => calls.filter((call) => call.method === "POST");
|
|
|
|
async function runMain(data, options = {}, eventRun = {}) {
|
|
const github = fakeGithub(data, options);
|
|
const directory = mkdtempSync(path.join(tmpdir(), "runner-loss-rerun-"));
|
|
try {
|
|
const eventPath = path.join(directory, "event.json");
|
|
const summaryPath = path.join(directory, "summary.md");
|
|
writeFileSync(eventPath, JSON.stringify({ workflow_run: { id: data.run.id, run_attempt: 1, ...eventRun } }));
|
|
writeFileSync(summaryPath, "");
|
|
const decision = await main({
|
|
env: { GITHUB_TOKEN: "test-token", GITHUB_EVENT_PATH: eventPath, GITHUB_STEP_SUMMARY: summaryPath },
|
|
argv: [], fetcher: github.fetcher, log: () => {},
|
|
});
|
|
return { decision, calls: github.calls, summary: readFileSync(summaryPath, "utf8") };
|
|
} finally {
|
|
rmSync(directory, { recursive: true, force: true });
|
|
}
|
|
}
|
|
|
|
test("all failed jobs lost their runner: re-runs failed jobs once", async () => {
|
|
const { decision, calls, summary } = await runMain(fixture(ALL_LOST));
|
|
assert.equal(decision.rerun, true);
|
|
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [["ci / General tests (server (3/12))", true]]);
|
|
assert.deepEqual(posted(calls).map((call) => call.path),
|
|
[`/repos/paperclipai/paperclip/actions/runs/${ALL_LOST}/rerun-failed-jobs`]);
|
|
assert.ok(calls.every((call) => call.auth === "Bearer test-token"));
|
|
// `ci / verify` failed too, but only because its dependency failed.
|
|
assert.equal(calls.some((call) => call.path.endsWith("/check-runs/128088995071/annotations")), false);
|
|
assert.match(summary, /\*\*Re-ran failed jobs\*\*/);
|
|
assert.match(summary, /\| ci \/ General tests \(server \(3\/12\)\) \| yes \| The self-hosted runner lost communication/);
|
|
});
|
|
|
|
test("a shutdown signal found only in the job log counts as runner loss", async () => {
|
|
const data = fixture(LOG_ONLY);
|
|
assert.deepEqual(data.annotations["128101016102"].map((annotation) => annotation.message), ["The operation was canceled."]);
|
|
const { decision, calls } = await runMain(data);
|
|
assert.equal(decision.rerun, true);
|
|
assert.equal(decision.jobs.length, 11);
|
|
assert.ok(decision.jobs.every((job) => job.lost && /job log/.test(job.evidence)));
|
|
assert.ok(calls.some((call) => call.path.endsWith("/actions/jobs/128101016102/logs")));
|
|
assert.equal(posted(calls).length, 1);
|
|
});
|
|
|
|
test("a cancelled run whose cancelled job lost its runner is re-run once", async () => {
|
|
const data = fixture(CANCELLED_RUN);
|
|
assert.equal(data.run.conclusion, "cancelled");
|
|
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
|
|
assert.equal(decision.rerun, true);
|
|
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [["ci / General tests (server (8/12))", true]]);
|
|
assert.match(decision.jobs[0].evidence, /job log/);
|
|
assert.deepEqual(posted(calls).map((call) => call.path),
|
|
[`/repos/paperclipai/paperclip/actions/runs/${CANCELLED_RUN}/rerun-failed-jobs`]);
|
|
});
|
|
|
|
test("runner loss in the log wins over error annotations the workload printed", async () => {
|
|
const data = fixture(NOISY_LOST);
|
|
const suite = data.jobs.find((job) => job.name === "ci / Runner support suite");
|
|
assert.ok(data.annotations[suite.id].some((annotation) => annotation.message === "gateway_indexing_incomplete"));
|
|
const { decision, calls } = await runMain(data);
|
|
assert.equal(decision.rerun, true);
|
|
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [
|
|
["ci / Runner support suite", true],
|
|
["ci / General tests (server (2/12))", true],
|
|
]);
|
|
assert.equal(posted(calls).length, 1);
|
|
});
|
|
|
|
function cancelEverything(data, { log = `2026-11-12T19:03:32.0000000Z ##[error]${CANCELLED_MESSAGE}\n` } = {}) {
|
|
data.annotations = {};
|
|
data.logs = {};
|
|
for (const [index, job] of data.jobs.entries()) {
|
|
if (job.name.startsWith("ci / gate / ")) continue;
|
|
job.conclusion = "cancelled";
|
|
if (index % 4 === 3) continue;
|
|
data.annotations[job.id] = [{ annotation_level: "failure", message: CANCELLED_MESSAGE }];
|
|
data.logs[job.id] = log;
|
|
}
|
|
return data;
|
|
}
|
|
|
|
test("a run superseded by a new push is not re-run, and no job is classified", async () => {
|
|
const data = cancelEverything(fixture(CANCELLED_RUN));
|
|
assert.ok(data.jobs.filter((job) => job.conclusion === "cancelled").length >= 40);
|
|
const { decision, calls } = await runMain(data, {
|
|
pulls: [{ state: "open", base: { repo: { id: 1170821064 } }, head: { sha: "0".repeat(40), repo: { id: 1170821064 } } }],
|
|
}, { conclusion: "cancelled" });
|
|
assert.equal(decision.rerun, false);
|
|
assert.match(decision.reason, /no open pull request/);
|
|
assert.equal(calls.some((call) => call.path.includes("/check-runs/") || call.path.endsWith("/logs")), false);
|
|
assert.equal(posted(calls).length, 0);
|
|
});
|
|
|
|
test("a push while job logs are read stops the re-run", async () => {
|
|
const data = fixture(CANCELLED_RUN);
|
|
const moved = [{ state: "open", base: { repo: { id: 1170821064 } }, head: { sha: "0".repeat(40), repo: { id: 1170821064 } } }];
|
|
const { decision, calls } = await runMain(data, {
|
|
pulls: (seen) => (seen.some((call) => call.path.endsWith("/logs")) ? moved : undefined),
|
|
}, { conclusion: "cancelled" });
|
|
assert.equal(decision.rerun, false);
|
|
assert.match(decision.reason, /no open pull request/);
|
|
assert.equal(decision.jobs[0].lost, true);
|
|
assert.equal(calls.filter((call) => call.path.endsWith("/pulls")).length, 2);
|
|
assert.equal(posted(calls).length, 0);
|
|
});
|
|
|
|
test("a run a person cancelled is not re-run", async () => {
|
|
const data = cancelEverything(fixture(CANCELLED_RUN));
|
|
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
|
|
assert.equal(decision.rerun, false);
|
|
assert.ok(decision.jobs.length >= 40);
|
|
assert.equal(decision.jobs.some((job) => job.lost), false);
|
|
assert.match(decision.reason, /failed jobs did not lose their runner/);
|
|
assert.equal(posted(calls).length, 0);
|
|
});
|
|
|
|
test("a cancelled run with one lost job and one real test failure is not re-run", async () => {
|
|
const data = fixture(CANCELLED_RUN);
|
|
const failing = data.jobs.find((job) => job.name === "ci / General tests (server (1/12))");
|
|
failing.conclusion = "failure";
|
|
data.annotations[failing.id] = [
|
|
{ annotation_level: "failure", message: "AssertionError: expected 'queued' to be 'running'" },
|
|
{ annotation_level: "failure", message: "Process completed with exit code 1." },
|
|
];
|
|
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
|
|
assert.equal(decision.rerun, false);
|
|
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [
|
|
["ci / General tests (server (1/12))", false],
|
|
["ci / General tests (server (8/12))", true],
|
|
]);
|
|
assert.match(decision.reason, /1 of 2 failed jobs did not lose their runner/);
|
|
assert.equal(posted(calls).length, 0);
|
|
});
|
|
|
|
test("a cancelled run is re-run at most once", async () => {
|
|
const { decision, calls } = await runMain(fixture(CANCELLED_RUN), {
|
|
mutateRun: (run) => ({ ...run, run_attempt: 2 }),
|
|
}, { conclusion: "cancelled" });
|
|
assert.equal(decision.rerun, false);
|
|
assert.match(decision.reason, /attempt 2/);
|
|
assert.deepEqual(calls.map((call) => call.path), [`/repos/paperclipai/paperclip/actions/runs/${CANCELLED_RUN}`]);
|
|
});
|
|
|
|
test("a cancelled routing gate is left for a human re-run", async () => {
|
|
const data = fixture(CANCELLED_RUN);
|
|
const gate = data.jobs.find((job) => job.name === "ci / gate / Select trusted runner");
|
|
gate.conclusion = "cancelled";
|
|
data.annotations[gate.id] = [{ annotation_level: "failure", message: CANCELLED_MESSAGE }];
|
|
data.logs[gate.id] = data.logs[Object.keys(data.logs)[0]];
|
|
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
|
|
assert.equal(decision.rerun, false);
|
|
assert.match(decision.jobs.find((job) => job.id === gate.id).evidence, /routing gate/);
|
|
assert.equal(posted(calls).length, 0);
|
|
});
|
|
|
|
test("mixed runner loss and a real test failure: no re-run", async () => {
|
|
const { decision, calls, summary } = await runMain(fixture(MIXED));
|
|
assert.equal(decision.rerun, false);
|
|
assert.equal(decision.jobs.length, 11);
|
|
assert.deepEqual(decision.jobs.filter((job) => !job.lost).map((job) => job.name), ["ci / General tests (workspaces-a (1/2))"]);
|
|
assert.equal(decision.jobs.find((job) => job.name === "ci / Verify Paperclip Runner (vitest package)").lost, true);
|
|
assert.equal(posted(calls).length, 0);
|
|
assert.match(decision.reason, /1 of 11 failed jobs did not lose their runner/);
|
|
assert.match(summary, /\*\*No automatic re-run\*\*/);
|
|
});
|
|
|
|
test("real test failures: no re-run", async () => {
|
|
const { decision, calls } = await runMain(fixture(REAL));
|
|
assert.equal(decision.rerun, false);
|
|
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [["ci / General tests (server (4/12))", false]]);
|
|
assert.equal(posted(calls).length, 0);
|
|
});
|
|
|
|
test("only the verify and e2e aggregates failed: no re-run", async () => {
|
|
const data = fixture(ALL_LOST);
|
|
data.jobs = data.jobs.map((job) => job.name.startsWith("ci / General tests") ? { ...job, conclusion: "success" } : job);
|
|
const { decision, calls } = await runMain(data);
|
|
assert.equal(decision.rerun, false);
|
|
assert.match(decision.reason, /no failed jobs other than the verify\/e2e aggregates/);
|
|
assert.equal(calls.some((call) => call.path.includes("/check-runs/")), false);
|
|
assert.equal(posted(calls).length, 0);
|
|
});
|
|
|
|
test("attempt 2 or later is never re-run automatically", async () => {
|
|
// The event was for attempt 1, but someone re-ran the run before this job started.
|
|
const { decision, calls } = await runMain(fixture(ALL_LOST), { mutateRun: (run) => ({ ...run, run_attempt: 2 }) });
|
|
assert.equal(decision.rerun, false);
|
|
assert.match(decision.reason, /attempt 2/);
|
|
assert.deepEqual(calls.map((call) => call.path), [`/repos/paperclipai/paperclip/actions/runs/${ALL_LOST}`]);
|
|
});
|
|
|
|
test("a failed routing gate is left for a human re-run", async () => {
|
|
const data = fixture(ALL_LOST);
|
|
const gate = data.jobs.find((job) => job.name === "ci / gate / Select trusted runner");
|
|
gate.conclusion = "failure";
|
|
data.annotations[gate.id] = [{ annotation_level: "failure", message: "The self-hosted runner lost communication with the server." }];
|
|
const { decision, calls } = await runMain(data);
|
|
assert.equal(decision.rerun, false);
|
|
assert.match(decision.jobs.find((job) => job.id === gate.id).evidence, /routing gate/);
|
|
assert.equal(posted(calls).length, 0);
|
|
});
|
|
|
|
test("runs that are no longer the PR's latest CI are not re-run", async () => {
|
|
const data = fixture(ALL_LOST);
|
|
const moved = await decide(fakeGithub(data, {
|
|
pulls: [{ state: "open", base: { repo: { id: 1170821064 } }, head: { sha: "0".repeat(40), repo: { id: 1170821064 } } }],
|
|
}).client, data.run);
|
|
assert.equal(moved.rerun, false);
|
|
assert.match(moved.reason, /no open pull request/);
|
|
|
|
const merged = await decide(fakeGithub(data, {
|
|
pulls: [{ state: "closed", base: { repo: { id: 1170821064 } }, head: { sha: data.run.head_sha, repo: { id: 1170821064 } } }],
|
|
}).client, data.run);
|
|
assert.equal(merged.rerun, false);
|
|
|
|
// A later full-ci label run on the same head shares the PR concurrency group;
|
|
// re-running the older run would cancel it.
|
|
const newer = await decide(fakeGithub(data, {
|
|
prRuns: [{ id: data.run.id + 1, conclusion: null }, { id: data.run.id, conclusion: "failure" }],
|
|
}).client, data.run);
|
|
assert.equal(newer.rerun, false);
|
|
assert.match(newer.reason, /newer PR run/);
|
|
|
|
// A later label event that did not request CI skips its only job; it is not CI.
|
|
const skipped = await decide(fakeGithub(data, {
|
|
prRuns: [{ id: data.run.id + 1, conclusion: "skipped" }, { id: data.run.id, conclusion: "failure" }],
|
|
}).client, data.run);
|
|
assert.equal(skipped.rerun, true);
|
|
});
|
|
|
|
test("ineligible runs are rejected before any job is read", async () => {
|
|
const data = fixture(ALL_LOST);
|
|
for (const [change, reason] of [
|
|
[{ event: "push" }, /event is push/],
|
|
[{ path: ".github/workflows/other.yml" }, /workflow is/],
|
|
[{ conclusion: "success" }, /completed\/success/],
|
|
[{ repository: { id: 1, full_name: "someone/paperclip" } }, /canonical repository/],
|
|
]) {
|
|
const github = fakeGithub(data);
|
|
const decision = await decide(github.client, { ...data.run, ...change });
|
|
assert.equal(decision.rerun, false);
|
|
assert.match(decision.reason, reason);
|
|
assert.equal(github.calls.length, 0);
|
|
}
|
|
});
|
|
|
|
test("job listing follows Link pagination across pages", async () => {
|
|
const data = fixture(MIXED);
|
|
const github = fakeGithub(data, { jobsPageSize: 20 });
|
|
const decision = await decide(github.client, data.run);
|
|
const pages = github.calls.filter((call) => call.path.endsWith("/attempts/1/jobs"));
|
|
assert.deepEqual(pages.map((call) => new URLSearchParams(call.query).get("page")), [null, "2", "3"]);
|
|
// Failed jobs on pages 2 and 3 are all classified.
|
|
assert.equal(decision.jobs.length, 11);
|
|
assert.equal(decision.rerun, false);
|
|
});
|
|
|
|
test("an incomplete job listing fails instead of deciding on partial data", async () => {
|
|
const data = fixture(ALL_LOST);
|
|
data.total_count += 1;
|
|
const github = fakeGithub(data);
|
|
await assert.rejects(decide(github.client, data.run), /Incomplete listing/);
|
|
assert.equal(posted(github.calls).length, 0);
|
|
});
|
|
|
|
test("the client refuses to follow pagination to another host", async () => {
|
|
const client = createGithubClient({
|
|
token: "test-token",
|
|
fetcher: async () => json([], { link: '<https://example.com/steal?page=2>; rel="next"' }),
|
|
});
|
|
await assert.rejects(client.paginate("/repos/paperclipai/paperclip/commits/x/pulls"), /Refusing to call https:\/\/example.com/);
|
|
});
|
|
|
|
test("classification is conservative about cancelled jobs", async () => {
|
|
const cancelled = [{ annotation_level: "failure", message: "The operation was canceled." }];
|
|
const noSignal = await classifyJob({ id: 1 }, { annotations: cancelled, readLog: async () => "##[error]The operation was canceled.\n" });
|
|
assert.equal(noSignal.lost, false);
|
|
// A job timeout also cancels the step, but adds its own failure annotation.
|
|
const timedOut = await classifyJob({ id: 1 }, {
|
|
annotations: [{ annotation_level: "failure", message: "The job has exceeded the maximum execution time of 30m0s" }, ...cancelled],
|
|
readLog: async () => "##[error]The job has exceeded the maximum execution time of 30m0s\n##[error]The operation was canceled.\n",
|
|
});
|
|
assert.equal(timedOut.lost, false);
|
|
assert.match(timedOut.evidence, /maximum execution time/);
|
|
const notStarted = await classifyJob({ id: 1, conclusion: "cancelled" }, {
|
|
annotations: [],
|
|
readLog: async () => { throw new Error("log must not be read"); },
|
|
});
|
|
assert.equal(notStarted.lost, false);
|
|
const exitOnly = await classifyJob({ id: 1 }, {
|
|
annotations: [{ annotation_level: "failure", message: "Process completed with exit code 1." }],
|
|
readLog: async () => { throw new Error("log must not be read"); },
|
|
});
|
|
assert.equal(exitOnly.lost, false);
|
|
// Notices and warnings are not failure evidence.
|
|
const warning = await classifyJob({ id: 1 }, {
|
|
annotations: [{ annotation_level: "warning", message: "The self-hosted runner lost communication with the server." }],
|
|
readLog: async () => "",
|
|
});
|
|
assert.equal(warning.lost, false);
|
|
// Test output that merely prints the phrase is not a runner error line.
|
|
assert.equal(logShowsRunnerLoss("2026-11-06T00:00:00Z expected 'received a shutdown signal'\n"), false);
|
|
assert.equal(logShowsRunnerLoss(fixture(MIXED).logs["128078674151"]), true);
|
|
});
|
|
|
|
test("runner-loss evidence wins over other failure annotations in the same job", async () => {
|
|
const lostMessage = "The self-hosted runner lost communication with the server. Verify the machine is running.";
|
|
const noLog = async () => { throw new Error("log must not be read"); };
|
|
// A test fails, then the runner is lost during an always() upload step.
|
|
const testThenLost = await classifyJob({ id: 1 }, {
|
|
annotations: [
|
|
{ annotation_level: "failure", message: "Error: Test timed out in 15000ms." },
|
|
{ annotation_level: "failure", message: "Process completed with exit code 1." },
|
|
{ annotation_level: "failure", message: lostMessage },
|
|
],
|
|
readLog: noLog,
|
|
});
|
|
assert.equal(testThenLost.lost, true);
|
|
assert.match(testThenLost.evidence, /lost communication/);
|
|
// The order of the annotations does not matter.
|
|
const lostThenExit = await classifyJob({ id: 1 }, {
|
|
annotations: [
|
|
{ annotation_level: "failure", message: lostMessage },
|
|
{ annotation_level: "failure", message: "Process completed with exit code 1." },
|
|
],
|
|
readLog: noLog,
|
|
});
|
|
assert.equal(lostThenExit.lost, true);
|
|
const noisy = [
|
|
{ annotation_level: "failure", message: "gateway_indexing_incomplete" },
|
|
{ annotation_level: "failure", message: CANCELLED_MESSAGE },
|
|
];
|
|
const noisyLost = await classifyJob({ id: 1 }, {
|
|
annotations: noisy,
|
|
readLog: async () => "##[error]gateway_indexing_incomplete\n##[error]The runner has received a shutdown signal.\n",
|
|
});
|
|
assert.equal(noisyLost.lost, true);
|
|
const noisyCancelled = await classifyJob({ id: 1 }, {
|
|
annotations: noisy,
|
|
readLog: async () => "##[error]gateway_indexing_incomplete\n##[error]The operation was canceled.\n",
|
|
});
|
|
assert.equal(noisyCancelled.lost, false);
|
|
assert.match(noisyCancelled.evidence, /gateway_indexing_incomplete/);
|
|
// Runner loss with a cancellation only is still runner loss.
|
|
const lostAndCancelled = await classifyJob({ id: 1 }, {
|
|
annotations: [
|
|
{ annotation_level: "failure", message: lostMessage },
|
|
{ annotation_level: "failure", message: "The operation was canceled." },
|
|
],
|
|
readLog: noLog,
|
|
});
|
|
assert.equal(lostAndCancelled.lost, true);
|
|
});
|
|
|
|
test("summary cells cannot break the table", () => {
|
|
const summary = renderSummary({
|
|
runId: 1, attempt: 1, rerun: false, reason: "test",
|
|
jobs: [{ name: "ci / a|b", lost: false, evidence: "line one\nline | two" }],
|
|
});
|
|
assert.match(summary, /\| ci \/ a\\\|b \| no \| line one line \\\| two \|/);
|
|
});
|
|
|
|
test("workflow acts only on failed or cancelled first attempts of the canonical PR workflow", () => {
|
|
const condition = workflow.match(/^ {4}if: >-\n((?: {6}.+\n)+)/m)?.[1].replace(/\s+/g, " ").trim();
|
|
assert.equal(condition, [
|
|
"github.repository == 'paperclipai/paperclip' &&",
|
|
"github.repository_id == '1170821064' &&",
|
|
"github.event.workflow_run.event == 'pull_request' &&",
|
|
"github.event.workflow_run.path == '.github/workflows/pr.yml' &&",
|
|
"(github.event.workflow_run.conclusion == 'failure' ||",
|
|
"github.event.workflow_run.conclusion == 'cancelled') &&",
|
|
"github.event.workflow_run.run_attempt == 1",
|
|
].join(" "));
|
|
assert.match(workflow, /^on:\n {2}workflow_run:\n {4}workflows: \[PR\]\n {4}types: \[completed\]$/m);
|
|
assert.match(workflow, /^ {2}group: ci-runner-loss-rerun-\$\{\{ github\.event\.workflow_run\.id \}\}$/m);
|
|
const permissions = workflow.match(/^permissions:\n((?: {2}.+\n)+)/m)[1]
|
|
.split("\n").filter((line) => /^ {2}[a-z-]+:/.test(line)).map((line) => line.trim());
|
|
assert.deepEqual(permissions, ["actions: write", "checks: read", "contents: read", "pull-requests: read"]);
|
|
});
|
|
|
|
test("workflow runs only trusted default-branch code", () => {
|
|
assert.match(workflow, /ref: \$\{\{ github\.sha \}\}\n {10}persist-credentials: false\n {10}sparse-checkout: \.github\/scripts\n/);
|
|
assert.doesNotMatch(workflow, /head_sha|head_branch|download-artifact|secrets\.|pull_request_target/);
|
|
assert.equal([...workflow.matchAll(/uses: /g)].length, 1);
|
|
assert.match(workflow, /runs-on: ubuntu-latest/);
|
|
});
|