mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - PR CI runs on AWS spot runners, and `ci-runner-loss-rerun.yml` re-runs a PR's first attempt once when every failed job lost its runner (#21291) > - On 2026-11-13 a watcher was added to the runner image that stops the GitHub runner about 10 seconds after AWS announces a spot reclaim, so the lost job ends in about 28 seconds instead of up to 10 minutes > - When the runner is stopped this way, the cancelled step's check annotation is usually "The operation was canceled.", but a step that was still starting up gets "A task was canceled." instead > - The classifier only recognised the first text, so it counted such a job as a real failure and refused the re-run, even though the job log has the shutdown-signal line > - This pull request recognises both cancellation texts, still only together with shutdown evidence in the log > - The benefit is that lost jobs stopped early in a step are re-run automatically too ## Linked Issues or Issue Description Refs #21291 **What happened?** In the first hour after the spot-notice watcher went live, a `ci / Build` job was stopped 13 s after it started. Its job log has `##[error]The runner has received a shutdown signal...`, but its only failure annotation was `A task was canceled.`. `classifyJob` treated that annotation as a real failure, so the whole run was not re-run. **Expected behavior** A job whose log shows the shutdown signal is counted as runner loss, whichever cancellation text the step was given. **Actual behavior** It is counted as a real failure, and the run stays red until a person re-runs it. ## What Changed - `.github/scripts/runner-loss-rerun.mjs`: the cancellation annotation check accepts "The operation was canceled." and "A task was canceled.". The log check still decides. A cancellation with no shutdown or lost-communication line in the log is still a real failure, and its evidence now quotes the actual annotation text. - `.github/scripts/tests/runner-loss-rerun.test.mjs`: a test for "A task was canceled." with and without shutdown evidence. - `doc/DEVELOPING.md`: the runner-loss paragraph names both texts. ## Verification - Red: the new test fails against the unchanged script (25 pass, 1 fail). - Green: `node --test .github/scripts/tests/runner-loss-rerun.test.mjs`: 26 pass. - `node --test '.github/scripts/tests/*.test.mjs'` (the `policy` job's command): 808 pass. - The workflow runs from `master`, so this PR's own CI cannot exercise the new behaviour; the tests are the evidence. ## Risks - A job cancelled as "A task was canceled." for another reason is still treated as a real failure unless its log has the shutdown or lost-communication line, so a person cancelling a run still blocks the automatic re-run. - No change to triggers, permissions, the once-only rule or the stale-head checks. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
472 lines
23 KiB
JavaScript
472 lines
23 KiB
JavaScript
import assert from "node:assert/strict";
|
|
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
|
import { tmpdir } from "node:os";
|
|
import path from "node:path";
|
|
import test from "node:test";
|
|
import {
|
|
classifyJob, createGithubClient, decide, logShowsRunnerLoss, main, renderSummary,
|
|
} from "../runner-loss-rerun.mjs";
|
|
|
|
// Trimmed attempt-1 snapshots of real PR runs: jobs, check-run annotations for
|
|
// each failed job, and the error lines of job logs whose only annotation is
|
|
// "The operation was canceled.".
|
|
const fixture = (runId) => JSON.parse(readFileSync(
|
|
new URL(`./fixtures/runner-loss-rerun/run-${runId}.json`, import.meta.url), "utf8",
|
|
));
|
|
const MIXED = 42271918832; // six lost runners, four shutdowns in the log, one real test timeout
|
|
const REAL = 42264785415; // a real assertion failure
|
|
const ALL_LOST = 42272083862; // one lost runner
|
|
const LOG_ONLY = 42277969418; // shutdown signals visible only in the job logs
|
|
const CANCELLED_RUN = 43285064435;
|
|
const NOISY_LOST = 43325855069;
|
|
const CANCELLED_MESSAGE = "The operation was canceled.";
|
|
const workflow = readFileSync(new URL("../../workflows/ci-runner-loss-rerun.yml", import.meta.url), "utf8");
|
|
|
|
const json = (body, headers = {}) => new Response(JSON.stringify(body), {
|
|
status: 200, headers: { "content-type": "application/json", ...headers },
|
|
});
|
|
|
|
/**
|
|
* Serves a fixture as the GitHub REST API. By default the run is the latest
|
|
* CI for an open PR's current head.
|
|
*/
|
|
function fakeGithub(data, {
|
|
jobsPageSize = 100, pulls, prRuns, mutateRun = (run) => run,
|
|
} = {}) {
|
|
const calls = [];
|
|
const run = mutateRun(structuredClone(data.run));
|
|
const openPr = {
|
|
number: 1, state: "open",
|
|
base: { repo: { id: data.run.repository.id } },
|
|
head: { sha: data.run.head_sha, repo: { id: data.run.head_repository.id } },
|
|
};
|
|
const fetcher = async (href, init) => {
|
|
const url = new URL(href);
|
|
calls.push({ method: init.method, path: url.pathname, query: url.search, auth: init.headers.Authorization });
|
|
const base = `/repos/paperclipai/paperclip`;
|
|
const route = url.pathname.slice(base.length);
|
|
if (init.method === "POST" && route === `/actions/runs/${run.id}/rerun-failed-jobs`) {
|
|
return new Response(null, { status: 201 });
|
|
}
|
|
if (route === `/actions/runs/${run.id}`) return json(run);
|
|
if (route === `/actions/runs/${run.id}/attempts/1/jobs`) {
|
|
const page = Number(url.searchParams.get("page") ?? 1);
|
|
const jobs = data.jobs.slice((page - 1) * jobsPageSize, page * jobsPageSize);
|
|
const last = page * jobsPageSize >= data.jobs.length;
|
|
const next = new URL(url);
|
|
next.searchParams.set("page", String(page + 1));
|
|
return json({ total_count: data.total_count, jobs }, last ? {} : { link: `<${next.href}>; rel="next"` });
|
|
}
|
|
let match = route.match(/^\/check-runs\/(\d+)\/annotations$/);
|
|
if (match) return json(data.annotations[match[1]] ?? []);
|
|
match = route.match(/^\/actions\/jobs\/(\d+)\/logs$/);
|
|
if (match) return new Response(data.logs[match[1]] ?? "", { status: 200 });
|
|
if (route === `/commits/${run.head_sha}/pulls`) return json((typeof pulls === "function" ? pulls(calls) : pulls) ?? [openPr]);
|
|
if (route === "/actions/workflows/pr.yml/runs") {
|
|
const runs = prRuns ?? [{ id: run.id, conclusion: "failure" }];
|
|
return json({ total_count: runs.length, workflow_runs: runs });
|
|
}
|
|
return new Response("not found", { status: 404 });
|
|
};
|
|
return { calls, fetcher, run, client: createGithubClient({ token: "test-token", fetcher }) };
|
|
}
|
|
|
|
const posted = (calls) => calls.filter((call) => call.method === "POST");
|
|
|
|
async function runMain(data, options = {}, eventRun = {}) {
|
|
const github = fakeGithub(data, options);
|
|
const directory = mkdtempSync(path.join(tmpdir(), "runner-loss-rerun-"));
|
|
try {
|
|
const eventPath = path.join(directory, "event.json");
|
|
const summaryPath = path.join(directory, "summary.md");
|
|
writeFileSync(eventPath, JSON.stringify({ workflow_run: { id: data.run.id, run_attempt: 1, ...eventRun } }));
|
|
writeFileSync(summaryPath, "");
|
|
const decision = await main({
|
|
env: { GITHUB_TOKEN: "test-token", GITHUB_EVENT_PATH: eventPath, GITHUB_STEP_SUMMARY: summaryPath },
|
|
argv: [], fetcher: github.fetcher, log: () => {},
|
|
});
|
|
return { decision, calls: github.calls, summary: readFileSync(summaryPath, "utf8") };
|
|
} finally {
|
|
rmSync(directory, { recursive: true, force: true });
|
|
}
|
|
}
|
|
|
|
test("all failed jobs lost their runner: re-runs failed jobs once", async () => {
|
|
const { decision, calls, summary } = await runMain(fixture(ALL_LOST));
|
|
assert.equal(decision.rerun, true);
|
|
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [["ci / General tests (server (3/12))", true]]);
|
|
assert.deepEqual(posted(calls).map((call) => call.path),
|
|
[`/repos/paperclipai/paperclip/actions/runs/${ALL_LOST}/rerun-failed-jobs`]);
|
|
assert.ok(calls.every((call) => call.auth === "Bearer test-token"));
|
|
// `ci / verify` failed too, but only because its dependency failed.
|
|
assert.equal(calls.some((call) => call.path.endsWith("/check-runs/128088995071/annotations")), false);
|
|
assert.match(summary, /\*\*Re-ran failed jobs\*\*/);
|
|
assert.match(summary, /\| ci \/ General tests \(server \(3\/12\)\) \| yes \| The self-hosted runner lost communication/);
|
|
});
|
|
|
|
test("a shutdown signal found only in the job log counts as runner loss", async () => {
|
|
const data = fixture(LOG_ONLY);
|
|
assert.deepEqual(data.annotations["128101016102"].map((annotation) => annotation.message), ["The operation was canceled."]);
|
|
const { decision, calls } = await runMain(data);
|
|
assert.equal(decision.rerun, true);
|
|
assert.equal(decision.jobs.length, 11);
|
|
assert.ok(decision.jobs.every((job) => job.lost && /job log/.test(job.evidence)));
|
|
assert.ok(calls.some((call) => call.path.endsWith("/actions/jobs/128101016102/logs")));
|
|
assert.equal(posted(calls).length, 1);
|
|
});
|
|
|
|
test("a cancelled run whose cancelled job lost its runner is re-run once", async () => {
|
|
const data = fixture(CANCELLED_RUN);
|
|
assert.equal(data.run.conclusion, "cancelled");
|
|
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
|
|
assert.equal(decision.rerun, true);
|
|
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [["ci / General tests (server (8/12))", true]]);
|
|
assert.match(decision.jobs[0].evidence, /job log/);
|
|
assert.deepEqual(posted(calls).map((call) => call.path),
|
|
[`/repos/paperclipai/paperclip/actions/runs/${CANCELLED_RUN}/rerun-failed-jobs`]);
|
|
});
|
|
|
|
test("runner loss in the log wins over error annotations the workload printed", async () => {
|
|
const data = fixture(NOISY_LOST);
|
|
const suite = data.jobs.find((job) => job.name === "ci / Runner support suite");
|
|
assert.ok(data.annotations[suite.id].some((annotation) => annotation.message === "gateway_indexing_incomplete"));
|
|
const { decision, calls } = await runMain(data);
|
|
assert.equal(decision.rerun, true);
|
|
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [
|
|
["ci / Runner support suite", true],
|
|
["ci / General tests (server (2/12))", true],
|
|
]);
|
|
assert.equal(posted(calls).length, 1);
|
|
});
|
|
|
|
function cancelEverything(data, { log = `2026-11-12T19:03:32.0000000Z ##[error]${CANCELLED_MESSAGE}\n` } = {}) {
|
|
data.annotations = {};
|
|
data.logs = {};
|
|
for (const [index, job] of data.jobs.entries()) {
|
|
if (job.name.startsWith("ci / gate / ")) continue;
|
|
job.conclusion = "cancelled";
|
|
if (index % 4 === 3) continue;
|
|
data.annotations[job.id] = [{ annotation_level: "failure", message: CANCELLED_MESSAGE }];
|
|
data.logs[job.id] = log;
|
|
}
|
|
return data;
|
|
}
|
|
|
|
test("a run superseded by a new push is not re-run, and no job is classified", async () => {
|
|
const data = cancelEverything(fixture(CANCELLED_RUN));
|
|
assert.ok(data.jobs.filter((job) => job.conclusion === "cancelled").length >= 40);
|
|
const { decision, calls } = await runMain(data, {
|
|
pulls: [{ state: "open", base: { repo: { id: 1170821064 } }, head: { sha: "0".repeat(40), repo: { id: 1170821064 } } }],
|
|
}, { conclusion: "cancelled" });
|
|
assert.equal(decision.rerun, false);
|
|
assert.match(decision.reason, /no open pull request/);
|
|
assert.equal(calls.some((call) => call.path.includes("/check-runs/") || call.path.endsWith("/logs")), false);
|
|
assert.equal(posted(calls).length, 0);
|
|
});
|
|
|
|
test("a push while job logs are read stops the re-run", async () => {
|
|
const data = fixture(CANCELLED_RUN);
|
|
const moved = [{ state: "open", base: { repo: { id: 1170821064 } }, head: { sha: "0".repeat(40), repo: { id: 1170821064 } } }];
|
|
const { decision, calls } = await runMain(data, {
|
|
pulls: (seen) => (seen.some((call) => call.path.endsWith("/logs")) ? moved : undefined),
|
|
}, { conclusion: "cancelled" });
|
|
assert.equal(decision.rerun, false);
|
|
assert.match(decision.reason, /no open pull request/);
|
|
assert.equal(decision.jobs[0].lost, true);
|
|
assert.equal(calls.filter((call) => call.path.endsWith("/pulls")).length, 2);
|
|
assert.equal(posted(calls).length, 0);
|
|
});
|
|
|
|
test("a run a person cancelled is not re-run", async () => {
|
|
const data = cancelEverything(fixture(CANCELLED_RUN));
|
|
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
|
|
assert.equal(decision.rerun, false);
|
|
assert.ok(decision.jobs.length >= 40);
|
|
assert.equal(decision.jobs.some((job) => job.lost), false);
|
|
assert.match(decision.reason, /failed jobs did not lose their runner/);
|
|
assert.equal(posted(calls).length, 0);
|
|
});
|
|
|
|
test("a cancelled run with one lost job and one real test failure is not re-run", async () => {
|
|
const data = fixture(CANCELLED_RUN);
|
|
const failing = data.jobs.find((job) => job.name === "ci / General tests (server (1/12))");
|
|
failing.conclusion = "failure";
|
|
data.annotations[failing.id] = [
|
|
{ annotation_level: "failure", message: "AssertionError: expected 'queued' to be 'running'" },
|
|
{ annotation_level: "failure", message: "Process completed with exit code 1." },
|
|
];
|
|
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
|
|
assert.equal(decision.rerun, false);
|
|
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [
|
|
["ci / General tests (server (1/12))", false],
|
|
["ci / General tests (server (8/12))", true],
|
|
]);
|
|
assert.match(decision.reason, /1 of 2 failed jobs did not lose their runner/);
|
|
assert.equal(posted(calls).length, 0);
|
|
});
|
|
|
|
test("a cancelled run is re-run at most once", async () => {
|
|
const { decision, calls } = await runMain(fixture(CANCELLED_RUN), {
|
|
mutateRun: (run) => ({ ...run, run_attempt: 2 }),
|
|
}, { conclusion: "cancelled" });
|
|
assert.equal(decision.rerun, false);
|
|
assert.match(decision.reason, /attempt 2/);
|
|
assert.deepEqual(calls.map((call) => call.path), [`/repos/paperclipai/paperclip/actions/runs/${CANCELLED_RUN}`]);
|
|
});
|
|
|
|
test("a cancelled routing gate is left for a human re-run", async () => {
|
|
const data = fixture(CANCELLED_RUN);
|
|
const gate = data.jobs.find((job) => job.name === "ci / gate / Select trusted runner");
|
|
gate.conclusion = "cancelled";
|
|
data.annotations[gate.id] = [{ annotation_level: "failure", message: CANCELLED_MESSAGE }];
|
|
data.logs[gate.id] = data.logs[Object.keys(data.logs)[0]];
|
|
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
|
|
assert.equal(decision.rerun, false);
|
|
assert.match(decision.jobs.find((job) => job.id === gate.id).evidence, /routing gate/);
|
|
assert.equal(posted(calls).length, 0);
|
|
});
|
|
|
|
test("mixed runner loss and a real test failure: no re-run", async () => {
|
|
const { decision, calls, summary } = await runMain(fixture(MIXED));
|
|
assert.equal(decision.rerun, false);
|
|
assert.equal(decision.jobs.length, 11);
|
|
assert.deepEqual(decision.jobs.filter((job) => !job.lost).map((job) => job.name), ["ci / General tests (workspaces-a (1/2))"]);
|
|
assert.equal(decision.jobs.find((job) => job.name === "ci / Verify Paperclip Runner (vitest package)").lost, true);
|
|
assert.equal(posted(calls).length, 0);
|
|
assert.match(decision.reason, /1 of 11 failed jobs did not lose their runner/);
|
|
assert.match(summary, /\*\*No automatic re-run\*\*/);
|
|
});
|
|
|
|
test("real test failures: no re-run", async () => {
|
|
const { decision, calls } = await runMain(fixture(REAL));
|
|
assert.equal(decision.rerun, false);
|
|
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [["ci / General tests (server (4/12))", false]]);
|
|
assert.equal(posted(calls).length, 0);
|
|
});
|
|
|
|
test("only the verify and e2e aggregates failed: no re-run", async () => {
|
|
const data = fixture(ALL_LOST);
|
|
data.jobs = data.jobs.map((job) => job.name.startsWith("ci / General tests") ? { ...job, conclusion: "success" } : job);
|
|
const { decision, calls } = await runMain(data);
|
|
assert.equal(decision.rerun, false);
|
|
assert.match(decision.reason, /no failed jobs other than the verify\/e2e aggregates/);
|
|
assert.equal(calls.some((call) => call.path.includes("/check-runs/")), false);
|
|
assert.equal(posted(calls).length, 0);
|
|
});
|
|
|
|
test("attempt 2 or later is never re-run automatically", async () => {
|
|
// The event was for attempt 1, but someone re-ran the run before this job started.
|
|
const { decision, calls } = await runMain(fixture(ALL_LOST), { mutateRun: (run) => ({ ...run, run_attempt: 2 }) });
|
|
assert.equal(decision.rerun, false);
|
|
assert.match(decision.reason, /attempt 2/);
|
|
assert.deepEqual(calls.map((call) => call.path), [`/repos/paperclipai/paperclip/actions/runs/${ALL_LOST}`]);
|
|
});
|
|
|
|
test("a failed routing gate is left for a human re-run", async () => {
|
|
const data = fixture(ALL_LOST);
|
|
const gate = data.jobs.find((job) => job.name === "ci / gate / Select trusted runner");
|
|
gate.conclusion = "failure";
|
|
data.annotations[gate.id] = [{ annotation_level: "failure", message: "The self-hosted runner lost communication with the server." }];
|
|
const { decision, calls } = await runMain(data);
|
|
assert.equal(decision.rerun, false);
|
|
assert.match(decision.jobs.find((job) => job.id === gate.id).evidence, /routing gate/);
|
|
assert.equal(posted(calls).length, 0);
|
|
});
|
|
|
|
test("runs that are no longer the PR's latest CI are not re-run", async () => {
|
|
const data = fixture(ALL_LOST);
|
|
const moved = await decide(fakeGithub(data, {
|
|
pulls: [{ state: "open", base: { repo: { id: 1170821064 } }, head: { sha: "0".repeat(40), repo: { id: 1170821064 } } }],
|
|
}).client, data.run);
|
|
assert.equal(moved.rerun, false);
|
|
assert.match(moved.reason, /no open pull request/);
|
|
|
|
const merged = await decide(fakeGithub(data, {
|
|
pulls: [{ state: "closed", base: { repo: { id: 1170821064 } }, head: { sha: data.run.head_sha, repo: { id: 1170821064 } } }],
|
|
}).client, data.run);
|
|
assert.equal(merged.rerun, false);
|
|
|
|
// A later full-ci label run on the same head shares the PR concurrency group;
|
|
// re-running the older run would cancel it.
|
|
const newer = await decide(fakeGithub(data, {
|
|
prRuns: [{ id: data.run.id + 1, conclusion: null }, { id: data.run.id, conclusion: "failure" }],
|
|
}).client, data.run);
|
|
assert.equal(newer.rerun, false);
|
|
assert.match(newer.reason, /newer PR run/);
|
|
|
|
// A later label event that did not request CI skips its only job; it is not CI.
|
|
const skipped = await decide(fakeGithub(data, {
|
|
prRuns: [{ id: data.run.id + 1, conclusion: "skipped" }, { id: data.run.id, conclusion: "failure" }],
|
|
}).client, data.run);
|
|
assert.equal(skipped.rerun, true);
|
|
});
|
|
|
|
test("ineligible runs are rejected before any job is read", async () => {
|
|
const data = fixture(ALL_LOST);
|
|
for (const [change, reason] of [
|
|
[{ event: "push" }, /event is push/],
|
|
[{ path: ".github/workflows/other.yml" }, /workflow is/],
|
|
[{ conclusion: "success" }, /completed\/success/],
|
|
[{ repository: { id: 1, full_name: "someone/paperclip" } }, /canonical repository/],
|
|
]) {
|
|
const github = fakeGithub(data);
|
|
const decision = await decide(github.client, { ...data.run, ...change });
|
|
assert.equal(decision.rerun, false);
|
|
assert.match(decision.reason, reason);
|
|
assert.equal(github.calls.length, 0);
|
|
}
|
|
});
|
|
|
|
test("job listing follows Link pagination across pages", async () => {
|
|
const data = fixture(MIXED);
|
|
const github = fakeGithub(data, { jobsPageSize: 20 });
|
|
const decision = await decide(github.client, data.run);
|
|
const pages = github.calls.filter((call) => call.path.endsWith("/attempts/1/jobs"));
|
|
assert.deepEqual(pages.map((call) => new URLSearchParams(call.query).get("page")), [null, "2", "3"]);
|
|
// Failed jobs on pages 2 and 3 are all classified.
|
|
assert.equal(decision.jobs.length, 11);
|
|
assert.equal(decision.rerun, false);
|
|
});
|
|
|
|
test("an incomplete job listing fails instead of deciding on partial data", async () => {
|
|
const data = fixture(ALL_LOST);
|
|
data.total_count += 1;
|
|
const github = fakeGithub(data);
|
|
await assert.rejects(decide(github.client, data.run), /Incomplete listing/);
|
|
assert.equal(posted(github.calls).length, 0);
|
|
});
|
|
|
|
test("the client refuses to follow pagination to another host", async () => {
|
|
const client = createGithubClient({
|
|
token: "test-token",
|
|
fetcher: async () => json([], { link: '<https://example.com/steal?page=2>; rel="next"' }),
|
|
});
|
|
await assert.rejects(client.paginate("/repos/paperclipai/paperclip/commits/x/pulls"), /Refusing to call https:\/\/example.com/);
|
|
});
|
|
|
|
test("a step cancelled as \"A task was canceled.\" counts as lost only with shutdown evidence", async () => {
|
|
const annotations = [{ annotation_level: "failure", message: "A task was canceled." }];
|
|
const lost = await classifyJob({ id: 1 }, {
|
|
annotations,
|
|
readLog: async () => "##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n##[error]A task was canceled.\n",
|
|
});
|
|
assert.equal(lost.lost, true);
|
|
const noSignal = await classifyJob({ id: 1 }, { annotations, readLog: async () => "##[error]A task was canceled.\n" });
|
|
assert.equal(noSignal.lost, false);
|
|
assert.match(noSignal.evidence, /A task was canceled\./);
|
|
});
|
|
|
|
test("classification is conservative about cancelled jobs", async () => {
|
|
const cancelled = [{ annotation_level: "failure", message: "The operation was canceled." }];
|
|
const noSignal = await classifyJob({ id: 1 }, { annotations: cancelled, readLog: async () => "##[error]The operation was canceled.\n" });
|
|
assert.equal(noSignal.lost, false);
|
|
// A job timeout also cancels the step, but adds its own failure annotation.
|
|
const timedOut = await classifyJob({ id: 1 }, {
|
|
annotations: [{ annotation_level: "failure", message: "The job has exceeded the maximum execution time of 30m0s" }, ...cancelled],
|
|
readLog: async () => "##[error]The job has exceeded the maximum execution time of 30m0s\n##[error]The operation was canceled.\n",
|
|
});
|
|
assert.equal(timedOut.lost, false);
|
|
assert.match(timedOut.evidence, /maximum execution time/);
|
|
const notStarted = await classifyJob({ id: 1, conclusion: "cancelled" }, {
|
|
annotations: [],
|
|
readLog: async () => { throw new Error("log must not be read"); },
|
|
});
|
|
assert.equal(notStarted.lost, false);
|
|
const exitOnly = await classifyJob({ id: 1 }, {
|
|
annotations: [{ annotation_level: "failure", message: "Process completed with exit code 1." }],
|
|
readLog: async () => { throw new Error("log must not be read"); },
|
|
});
|
|
assert.equal(exitOnly.lost, false);
|
|
// Notices and warnings are not failure evidence.
|
|
const warning = await classifyJob({ id: 1 }, {
|
|
annotations: [{ annotation_level: "warning", message: "The self-hosted runner lost communication with the server." }],
|
|
readLog: async () => "",
|
|
});
|
|
assert.equal(warning.lost, false);
|
|
// Test output that merely prints the phrase is not a runner error line.
|
|
assert.equal(logShowsRunnerLoss("2026-11-06T00:00:00Z expected 'received a shutdown signal'\n"), false);
|
|
assert.equal(logShowsRunnerLoss(fixture(MIXED).logs["128078674151"]), true);
|
|
});
|
|
|
|
test("runner-loss evidence wins over other failure annotations in the same job", async () => {
|
|
const lostMessage = "The self-hosted runner lost communication with the server. Verify the machine is running.";
|
|
const noLog = async () => { throw new Error("log must not be read"); };
|
|
// A test fails, then the runner is lost during an always() upload step.
|
|
const testThenLost = await classifyJob({ id: 1 }, {
|
|
annotations: [
|
|
{ annotation_level: "failure", message: "Error: Test timed out in 15000ms." },
|
|
{ annotation_level: "failure", message: "Process completed with exit code 1." },
|
|
{ annotation_level: "failure", message: lostMessage },
|
|
],
|
|
readLog: noLog,
|
|
});
|
|
assert.equal(testThenLost.lost, true);
|
|
assert.match(testThenLost.evidence, /lost communication/);
|
|
// The order of the annotations does not matter.
|
|
const lostThenExit = await classifyJob({ id: 1 }, {
|
|
annotations: [
|
|
{ annotation_level: "failure", message: lostMessage },
|
|
{ annotation_level: "failure", message: "Process completed with exit code 1." },
|
|
],
|
|
readLog: noLog,
|
|
});
|
|
assert.equal(lostThenExit.lost, true);
|
|
const noisy = [
|
|
{ annotation_level: "failure", message: "gateway_indexing_incomplete" },
|
|
{ annotation_level: "failure", message: CANCELLED_MESSAGE },
|
|
];
|
|
const noisyLost = await classifyJob({ id: 1 }, {
|
|
annotations: noisy,
|
|
readLog: async () => "##[error]gateway_indexing_incomplete\n##[error]The runner has received a shutdown signal.\n",
|
|
});
|
|
assert.equal(noisyLost.lost, true);
|
|
const noisyCancelled = await classifyJob({ id: 1 }, {
|
|
annotations: noisy,
|
|
readLog: async () => "##[error]gateway_indexing_incomplete\n##[error]The operation was canceled.\n",
|
|
});
|
|
assert.equal(noisyCancelled.lost, false);
|
|
assert.match(noisyCancelled.evidence, /gateway_indexing_incomplete/);
|
|
// Runner loss with a cancellation only is still runner loss.
|
|
const lostAndCancelled = await classifyJob({ id: 1 }, {
|
|
annotations: [
|
|
{ annotation_level: "failure", message: lostMessage },
|
|
{ annotation_level: "failure", message: "The operation was canceled." },
|
|
],
|
|
readLog: noLog,
|
|
});
|
|
assert.equal(lostAndCancelled.lost, true);
|
|
});
|
|
|
|
test("summary cells cannot break the table", () => {
|
|
const summary = renderSummary({
|
|
runId: 1, attempt: 1, rerun: false, reason: "test",
|
|
jobs: [{ name: "ci / a|b", lost: false, evidence: "line one\nline | two" }],
|
|
});
|
|
assert.match(summary, /\| ci \/ a\\\|b \| no \| line one line \\\| two \|/);
|
|
});
|
|
|
|
test("workflow acts only on failed or cancelled first attempts of the canonical PR workflow", () => {
|
|
const condition = workflow.match(/^ {4}if: >-\n((?: {6}.+\n)+)/m)?.[1].replace(/\s+/g, " ").trim();
|
|
assert.equal(condition, [
|
|
"github.repository == 'paperclipai/paperclip' &&",
|
|
"github.repository_id == '1170821064' &&",
|
|
"github.event.workflow_run.event == 'pull_request' &&",
|
|
"github.event.workflow_run.path == '.github/workflows/pr.yml' &&",
|
|
"(github.event.workflow_run.conclusion == 'failure' ||",
|
|
"github.event.workflow_run.conclusion == 'cancelled') &&",
|
|
"github.event.workflow_run.run_attempt == 1",
|
|
].join(" "));
|
|
assert.match(workflow, /^on:\n {2}workflow_run:\n {4}workflows: \[PR\]\n {4}types: \[completed\]$/m);
|
|
assert.match(workflow, /^ {2}group: ci-runner-loss-rerun-\$\{\{ github\.event\.workflow_run\.id \}\}$/m);
|
|
const permissions = workflow.match(/^permissions:\n((?: {2}.+\n)+)/m)[1]
|
|
.split("\n").filter((line) => /^ {2}[a-z-]+:/.test(line)).map((line) => line.trim());
|
|
assert.deepEqual(permissions, ["actions: write", "checks: read", "contents: read", "pull-requests: read"]);
|
|
});
|
|
|
|
test("workflow runs only trusted default-branch code", () => {
|
|
assert.match(workflow, /ref: \$\{\{ github\.sha \}\}\n {10}persist-credentials: false\n {10}sparse-checkout: \.github\/scripts\n/);
|
|
assert.doesNotMatch(workflow, /head_sha|head_branch|download-artifact|secrets\.|pull_request_target/);
|
|
assert.equal([...workflow.matchAll(/uses: /g)].length, 1);
|
|
assert.match(workflow, /runs-on: ubuntu-latest/);
|
|
});
|