Files
PaperClipAI/.github/scripts/tests/runner-loss-rerun.test.mjs
T
DottaandClaude Opus 5.5 e6df84a992 fix(ci): count a step cancelled as "A task was canceled." as runner loss when the log shows the shutdown (#21514)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - PR CI runs on AWS spot runners, and `ci-runner-loss-rerun.yml`
re-runs a PR's first attempt once when every failed job lost its runner
(#21291)
> - On 2026-11-13 a watcher was added to the runner image that stops the
GitHub runner about 10 seconds after AWS announces a spot reclaim, so
the lost job ends in about 28 seconds instead of up to 10 minutes
> - When the runner is stopped this way, the cancelled step's check
annotation is usually "The operation was canceled.", but a step that was
still starting up gets "A task was canceled." instead
> - The classifier only recognised the first text, so it counted such a
job as a real failure and refused the re-run, even though the job log
has the shutdown-signal line
> - This pull request recognises both cancellation texts, still only
together with shutdown evidence in the log
> - The benefit is that lost jobs stopped early in a step are re-run
automatically too

## Linked Issues or Issue Description

Refs #21291

**What happened?**
In the first hour after the spot-notice watcher went live, a `ci /
Build` job was stopped 13 s after it started. Its job log has
`##[error]The runner has received a shutdown signal...`, but its only
failure annotation was `A task was canceled.`. `classifyJob` treated
that annotation as a real failure, so the whole run was not re-run.

**Expected behavior**
A job whose log shows the shutdown signal is counted as runner loss,
whichever cancellation text the step was given.

**Actual behavior**
It is counted as a real failure, and the run stays red until a person
re-runs it.

## What Changed

- `.github/scripts/runner-loss-rerun.mjs`: the cancellation annotation
check accepts "The operation was canceled." and "A task was canceled.".
The log check still decides. A cancellation with no shutdown or
lost-communication line in the log is still a real failure, and its
evidence now quotes the actual annotation text.
- `.github/scripts/tests/runner-loss-rerun.test.mjs`: a test for "A task
was canceled." with and without shutdown evidence.
- `doc/DEVELOPING.md`: the runner-loss paragraph names both texts.

## Verification

- Red: the new test fails against the unchanged script (25 pass, 1
fail).
- Green: `node --test .github/scripts/tests/runner-loss-rerun.test.mjs`:
26 pass.
- `node --test '.github/scripts/tests/*.test.mjs'` (the `policy` job's
command): 808 pass.
- The workflow runs from `master`, so this PR's own CI cannot exercise
the new behaviour; the tests are the evidence.

## Risks

- A job cancelled as "A task was canceled." for another reason is still
treated as a real failure unless its log has the shutdown or
lost-communication line, so a person cancelling a run still blocks the
automatic re-run.
- No change to triggers, permissions, the once-only rule or the
stale-head checks.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:41:10 -06:00

472 lines
23 KiB
JavaScript

import assert from "node:assert/strict";
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import test from "node:test";
import {
classifyJob, createGithubClient, decide, logShowsRunnerLoss, main, renderSummary,
} from "../runner-loss-rerun.mjs";
// Trimmed attempt-1 snapshots of real PR runs: jobs, check-run annotations for
// each failed job, and the error lines of job logs whose only annotation is
// "The operation was canceled.".
const fixture = (runId) => JSON.parse(readFileSync(
new URL(`./fixtures/runner-loss-rerun/run-${runId}.json`, import.meta.url), "utf8",
));
const MIXED = 42271918832; // six lost runners, four shutdowns in the log, one real test timeout
const REAL = 42264785415; // a real assertion failure
const ALL_LOST = 42272083862; // one lost runner
const LOG_ONLY = 42277969418; // shutdown signals visible only in the job logs
const CANCELLED_RUN = 43285064435;
const NOISY_LOST = 43325855069;
const CANCELLED_MESSAGE = "The operation was canceled.";
const workflow = readFileSync(new URL("../../workflows/ci-runner-loss-rerun.yml", import.meta.url), "utf8");
const json = (body, headers = {}) => new Response(JSON.stringify(body), {
status: 200, headers: { "content-type": "application/json", ...headers },
});
/**
* Serves a fixture as the GitHub REST API. By default the run is the latest
* CI for an open PR's current head.
*/
function fakeGithub(data, {
jobsPageSize = 100, pulls, prRuns, mutateRun = (run) => run,
} = {}) {
const calls = [];
const run = mutateRun(structuredClone(data.run));
const openPr = {
number: 1, state: "open",
base: { repo: { id: data.run.repository.id } },
head: { sha: data.run.head_sha, repo: { id: data.run.head_repository.id } },
};
const fetcher = async (href, init) => {
const url = new URL(href);
calls.push({ method: init.method, path: url.pathname, query: url.search, auth: init.headers.Authorization });
const base = `/repos/paperclipai/paperclip`;
const route = url.pathname.slice(base.length);
if (init.method === "POST" && route === `/actions/runs/${run.id}/rerun-failed-jobs`) {
return new Response(null, { status: 201 });
}
if (route === `/actions/runs/${run.id}`) return json(run);
if (route === `/actions/runs/${run.id}/attempts/1/jobs`) {
const page = Number(url.searchParams.get("page") ?? 1);
const jobs = data.jobs.slice((page - 1) * jobsPageSize, page * jobsPageSize);
const last = page * jobsPageSize >= data.jobs.length;
const next = new URL(url);
next.searchParams.set("page", String(page + 1));
return json({ total_count: data.total_count, jobs }, last ? {} : { link: `<${next.href}>; rel="next"` });
}
let match = route.match(/^\/check-runs\/(\d+)\/annotations$/);
if (match) return json(data.annotations[match[1]] ?? []);
match = route.match(/^\/actions\/jobs\/(\d+)\/logs$/);
if (match) return new Response(data.logs[match[1]] ?? "", { status: 200 });
if (route === `/commits/${run.head_sha}/pulls`) return json((typeof pulls === "function" ? pulls(calls) : pulls) ?? [openPr]);
if (route === "/actions/workflows/pr.yml/runs") {
const runs = prRuns ?? [{ id: run.id, conclusion: "failure" }];
return json({ total_count: runs.length, workflow_runs: runs });
}
return new Response("not found", { status: 404 });
};
return { calls, fetcher, run, client: createGithubClient({ token: "test-token", fetcher }) };
}
const posted = (calls) => calls.filter((call) => call.method === "POST");
async function runMain(data, options = {}, eventRun = {}) {
const github = fakeGithub(data, options);
const directory = mkdtempSync(path.join(tmpdir(), "runner-loss-rerun-"));
try {
const eventPath = path.join(directory, "event.json");
const summaryPath = path.join(directory, "summary.md");
writeFileSync(eventPath, JSON.stringify({ workflow_run: { id: data.run.id, run_attempt: 1, ...eventRun } }));
writeFileSync(summaryPath, "");
const decision = await main({
env: { GITHUB_TOKEN: "test-token", GITHUB_EVENT_PATH: eventPath, GITHUB_STEP_SUMMARY: summaryPath },
argv: [], fetcher: github.fetcher, log: () => {},
});
return { decision, calls: github.calls, summary: readFileSync(summaryPath, "utf8") };
} finally {
rmSync(directory, { recursive: true, force: true });
}
}
test("all failed jobs lost their runner: re-runs failed jobs once", async () => {
const { decision, calls, summary } = await runMain(fixture(ALL_LOST));
assert.equal(decision.rerun, true);
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [["ci / General tests (server (3/12))", true]]);
assert.deepEqual(posted(calls).map((call) => call.path),
[`/repos/paperclipai/paperclip/actions/runs/${ALL_LOST}/rerun-failed-jobs`]);
assert.ok(calls.every((call) => call.auth === "Bearer test-token"));
// `ci / verify` failed too, but only because its dependency failed.
assert.equal(calls.some((call) => call.path.endsWith("/check-runs/128088995071/annotations")), false);
assert.match(summary, /\*\*Re-ran failed jobs\*\*/);
assert.match(summary, /\| ci \/ General tests \(server \(3\/12\)\) \| yes \| The self-hosted runner lost communication/);
});
test("a shutdown signal found only in the job log counts as runner loss", async () => {
const data = fixture(LOG_ONLY);
assert.deepEqual(data.annotations["128101016102"].map((annotation) => annotation.message), ["The operation was canceled."]);
const { decision, calls } = await runMain(data);
assert.equal(decision.rerun, true);
assert.equal(decision.jobs.length, 11);
assert.ok(decision.jobs.every((job) => job.lost && /job log/.test(job.evidence)));
assert.ok(calls.some((call) => call.path.endsWith("/actions/jobs/128101016102/logs")));
assert.equal(posted(calls).length, 1);
});
test("a cancelled run whose cancelled job lost its runner is re-run once", async () => {
const data = fixture(CANCELLED_RUN);
assert.equal(data.run.conclusion, "cancelled");
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
assert.equal(decision.rerun, true);
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [["ci / General tests (server (8/12))", true]]);
assert.match(decision.jobs[0].evidence, /job log/);
assert.deepEqual(posted(calls).map((call) => call.path),
[`/repos/paperclipai/paperclip/actions/runs/${CANCELLED_RUN}/rerun-failed-jobs`]);
});
test("runner loss in the log wins over error annotations the workload printed", async () => {
const data = fixture(NOISY_LOST);
const suite = data.jobs.find((job) => job.name === "ci / Runner support suite");
assert.ok(data.annotations[suite.id].some((annotation) => annotation.message === "gateway_indexing_incomplete"));
const { decision, calls } = await runMain(data);
assert.equal(decision.rerun, true);
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [
["ci / Runner support suite", true],
["ci / General tests (server (2/12))", true],
]);
assert.equal(posted(calls).length, 1);
});
function cancelEverything(data, { log = `2026-11-12T19:03:32.0000000Z ##[error]${CANCELLED_MESSAGE}\n` } = {}) {
data.annotations = {};
data.logs = {};
for (const [index, job] of data.jobs.entries()) {
if (job.name.startsWith("ci / gate / ")) continue;
job.conclusion = "cancelled";
if (index % 4 === 3) continue;
data.annotations[job.id] = [{ annotation_level: "failure", message: CANCELLED_MESSAGE }];
data.logs[job.id] = log;
}
return data;
}
test("a run superseded by a new push is not re-run, and no job is classified", async () => {
const data = cancelEverything(fixture(CANCELLED_RUN));
assert.ok(data.jobs.filter((job) => job.conclusion === "cancelled").length >= 40);
const { decision, calls } = await runMain(data, {
pulls: [{ state: "open", base: { repo: { id: 1170821064 } }, head: { sha: "0".repeat(40), repo: { id: 1170821064 } } }],
}, { conclusion: "cancelled" });
assert.equal(decision.rerun, false);
assert.match(decision.reason, /no open pull request/);
assert.equal(calls.some((call) => call.path.includes("/check-runs/") || call.path.endsWith("/logs")), false);
assert.equal(posted(calls).length, 0);
});
test("a push while job logs are read stops the re-run", async () => {
const data = fixture(CANCELLED_RUN);
const moved = [{ state: "open", base: { repo: { id: 1170821064 } }, head: { sha: "0".repeat(40), repo: { id: 1170821064 } } }];
const { decision, calls } = await runMain(data, {
pulls: (seen) => (seen.some((call) => call.path.endsWith("/logs")) ? moved : undefined),
}, { conclusion: "cancelled" });
assert.equal(decision.rerun, false);
assert.match(decision.reason, /no open pull request/);
assert.equal(decision.jobs[0].lost, true);
assert.equal(calls.filter((call) => call.path.endsWith("/pulls")).length, 2);
assert.equal(posted(calls).length, 0);
});
test("a run a person cancelled is not re-run", async () => {
const data = cancelEverything(fixture(CANCELLED_RUN));
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
assert.equal(decision.rerun, false);
assert.ok(decision.jobs.length >= 40);
assert.equal(decision.jobs.some((job) => job.lost), false);
assert.match(decision.reason, /failed jobs did not lose their runner/);
assert.equal(posted(calls).length, 0);
});
test("a cancelled run with one lost job and one real test failure is not re-run", async () => {
const data = fixture(CANCELLED_RUN);
const failing = data.jobs.find((job) => job.name === "ci / General tests (server (1/12))");
failing.conclusion = "failure";
data.annotations[failing.id] = [
{ annotation_level: "failure", message: "AssertionError: expected 'queued' to be 'running'" },
{ annotation_level: "failure", message: "Process completed with exit code 1." },
];
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
assert.equal(decision.rerun, false);
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [
["ci / General tests (server (1/12))", false],
["ci / General tests (server (8/12))", true],
]);
assert.match(decision.reason, /1 of 2 failed jobs did not lose their runner/);
assert.equal(posted(calls).length, 0);
});
test("a cancelled run is re-run at most once", async () => {
const { decision, calls } = await runMain(fixture(CANCELLED_RUN), {
mutateRun: (run) => ({ ...run, run_attempt: 2 }),
}, { conclusion: "cancelled" });
assert.equal(decision.rerun, false);
assert.match(decision.reason, /attempt 2/);
assert.deepEqual(calls.map((call) => call.path), [`/repos/paperclipai/paperclip/actions/runs/${CANCELLED_RUN}`]);
});
test("a cancelled routing gate is left for a human re-run", async () => {
const data = fixture(CANCELLED_RUN);
const gate = data.jobs.find((job) => job.name === "ci / gate / Select trusted runner");
gate.conclusion = "cancelled";
data.annotations[gate.id] = [{ annotation_level: "failure", message: CANCELLED_MESSAGE }];
data.logs[gate.id] = data.logs[Object.keys(data.logs)[0]];
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
assert.equal(decision.rerun, false);
assert.match(decision.jobs.find((job) => job.id === gate.id).evidence, /routing gate/);
assert.equal(posted(calls).length, 0);
});
test("mixed runner loss and a real test failure: no re-run", async () => {
const { decision, calls, summary } = await runMain(fixture(MIXED));
assert.equal(decision.rerun, false);
assert.equal(decision.jobs.length, 11);
assert.deepEqual(decision.jobs.filter((job) => !job.lost).map((job) => job.name), ["ci / General tests (workspaces-a (1/2))"]);
assert.equal(decision.jobs.find((job) => job.name === "ci / Verify Paperclip Runner (vitest package)").lost, true);
assert.equal(posted(calls).length, 0);
assert.match(decision.reason, /1 of 11 failed jobs did not lose their runner/);
assert.match(summary, /\*\*No automatic re-run\*\*/);
});
test("real test failures: no re-run", async () => {
const { decision, calls } = await runMain(fixture(REAL));
assert.equal(decision.rerun, false);
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [["ci / General tests (server (4/12))", false]]);
assert.equal(posted(calls).length, 0);
});
test("only the verify and e2e aggregates failed: no re-run", async () => {
const data = fixture(ALL_LOST);
data.jobs = data.jobs.map((job) => job.name.startsWith("ci / General tests") ? { ...job, conclusion: "success" } : job);
const { decision, calls } = await runMain(data);
assert.equal(decision.rerun, false);
assert.match(decision.reason, /no failed jobs other than the verify\/e2e aggregates/);
assert.equal(calls.some((call) => call.path.includes("/check-runs/")), false);
assert.equal(posted(calls).length, 0);
});
test("attempt 2 or later is never re-run automatically", async () => {
// The event was for attempt 1, but someone re-ran the run before this job started.
const { decision, calls } = await runMain(fixture(ALL_LOST), { mutateRun: (run) => ({ ...run, run_attempt: 2 }) });
assert.equal(decision.rerun, false);
assert.match(decision.reason, /attempt 2/);
assert.deepEqual(calls.map((call) => call.path), [`/repos/paperclipai/paperclip/actions/runs/${ALL_LOST}`]);
});
test("a failed routing gate is left for a human re-run", async () => {
const data = fixture(ALL_LOST);
const gate = data.jobs.find((job) => job.name === "ci / gate / Select trusted runner");
gate.conclusion = "failure";
data.annotations[gate.id] = [{ annotation_level: "failure", message: "The self-hosted runner lost communication with the server." }];
const { decision, calls } = await runMain(data);
assert.equal(decision.rerun, false);
assert.match(decision.jobs.find((job) => job.id === gate.id).evidence, /routing gate/);
assert.equal(posted(calls).length, 0);
});
test("runs that are no longer the PR's latest CI are not re-run", async () => {
const data = fixture(ALL_LOST);
const moved = await decide(fakeGithub(data, {
pulls: [{ state: "open", base: { repo: { id: 1170821064 } }, head: { sha: "0".repeat(40), repo: { id: 1170821064 } } }],
}).client, data.run);
assert.equal(moved.rerun, false);
assert.match(moved.reason, /no open pull request/);
const merged = await decide(fakeGithub(data, {
pulls: [{ state: "closed", base: { repo: { id: 1170821064 } }, head: { sha: data.run.head_sha, repo: { id: 1170821064 } } }],
}).client, data.run);
assert.equal(merged.rerun, false);
// A later full-ci label run on the same head shares the PR concurrency group;
// re-running the older run would cancel it.
const newer = await decide(fakeGithub(data, {
prRuns: [{ id: data.run.id + 1, conclusion: null }, { id: data.run.id, conclusion: "failure" }],
}).client, data.run);
assert.equal(newer.rerun, false);
assert.match(newer.reason, /newer PR run/);
// A later label event that did not request CI skips its only job; it is not CI.
const skipped = await decide(fakeGithub(data, {
prRuns: [{ id: data.run.id + 1, conclusion: "skipped" }, { id: data.run.id, conclusion: "failure" }],
}).client, data.run);
assert.equal(skipped.rerun, true);
});
test("ineligible runs are rejected before any job is read", async () => {
const data = fixture(ALL_LOST);
for (const [change, reason] of [
[{ event: "push" }, /event is push/],
[{ path: ".github/workflows/other.yml" }, /workflow is/],
[{ conclusion: "success" }, /completed\/success/],
[{ repository: { id: 1, full_name: "someone/paperclip" } }, /canonical repository/],
]) {
const github = fakeGithub(data);
const decision = await decide(github.client, { ...data.run, ...change });
assert.equal(decision.rerun, false);
assert.match(decision.reason, reason);
assert.equal(github.calls.length, 0);
}
});
test("job listing follows Link pagination across pages", async () => {
const data = fixture(MIXED);
const github = fakeGithub(data, { jobsPageSize: 20 });
const decision = await decide(github.client, data.run);
const pages = github.calls.filter((call) => call.path.endsWith("/attempts/1/jobs"));
assert.deepEqual(pages.map((call) => new URLSearchParams(call.query).get("page")), [null, "2", "3"]);
// Failed jobs on pages 2 and 3 are all classified.
assert.equal(decision.jobs.length, 11);
assert.equal(decision.rerun, false);
});
test("an incomplete job listing fails instead of deciding on partial data", async () => {
const data = fixture(ALL_LOST);
data.total_count += 1;
const github = fakeGithub(data);
await assert.rejects(decide(github.client, data.run), /Incomplete listing/);
assert.equal(posted(github.calls).length, 0);
});
test("the client refuses to follow pagination to another host", async () => {
const client = createGithubClient({
token: "test-token",
fetcher: async () => json([], { link: '<https://example.com/steal?page=2>; rel="next"' }),
});
await assert.rejects(client.paginate("/repos/paperclipai/paperclip/commits/x/pulls"), /Refusing to call https:\/\/example.com/);
});
test("a step cancelled as \"A task was canceled.\" counts as lost only with shutdown evidence", async () => {
const annotations = [{ annotation_level: "failure", message: "A task was canceled." }];
const lost = await classifyJob({ id: 1 }, {
annotations,
readLog: async () => "##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n##[error]A task was canceled.\n",
});
assert.equal(lost.lost, true);
const noSignal = await classifyJob({ id: 1 }, { annotations, readLog: async () => "##[error]A task was canceled.\n" });
assert.equal(noSignal.lost, false);
assert.match(noSignal.evidence, /A task was canceled\./);
});
test("classification is conservative about cancelled jobs", async () => {
const cancelled = [{ annotation_level: "failure", message: "The operation was canceled." }];
const noSignal = await classifyJob({ id: 1 }, { annotations: cancelled, readLog: async () => "##[error]The operation was canceled.\n" });
assert.equal(noSignal.lost, false);
// A job timeout also cancels the step, but adds its own failure annotation.
const timedOut = await classifyJob({ id: 1 }, {
annotations: [{ annotation_level: "failure", message: "The job has exceeded the maximum execution time of 30m0s" }, ...cancelled],
readLog: async () => "##[error]The job has exceeded the maximum execution time of 30m0s\n##[error]The operation was canceled.\n",
});
assert.equal(timedOut.lost, false);
assert.match(timedOut.evidence, /maximum execution time/);
const notStarted = await classifyJob({ id: 1, conclusion: "cancelled" }, {
annotations: [],
readLog: async () => { throw new Error("log must not be read"); },
});
assert.equal(notStarted.lost, false);
const exitOnly = await classifyJob({ id: 1 }, {
annotations: [{ annotation_level: "failure", message: "Process completed with exit code 1." }],
readLog: async () => { throw new Error("log must not be read"); },
});
assert.equal(exitOnly.lost, false);
// Notices and warnings are not failure evidence.
const warning = await classifyJob({ id: 1 }, {
annotations: [{ annotation_level: "warning", message: "The self-hosted runner lost communication with the server." }],
readLog: async () => "",
});
assert.equal(warning.lost, false);
// Test output that merely prints the phrase is not a runner error line.
assert.equal(logShowsRunnerLoss("2026-11-06T00:00:00Z expected 'received a shutdown signal'\n"), false);
assert.equal(logShowsRunnerLoss(fixture(MIXED).logs["128078674151"]), true);
});
test("runner-loss evidence wins over other failure annotations in the same job", async () => {
const lostMessage = "The self-hosted runner lost communication with the server. Verify the machine is running.";
const noLog = async () => { throw new Error("log must not be read"); };
// A test fails, then the runner is lost during an always() upload step.
const testThenLost = await classifyJob({ id: 1 }, {
annotations: [
{ annotation_level: "failure", message: "Error: Test timed out in 15000ms." },
{ annotation_level: "failure", message: "Process completed with exit code 1." },
{ annotation_level: "failure", message: lostMessage },
],
readLog: noLog,
});
assert.equal(testThenLost.lost, true);
assert.match(testThenLost.evidence, /lost communication/);
// The order of the annotations does not matter.
const lostThenExit = await classifyJob({ id: 1 }, {
annotations: [
{ annotation_level: "failure", message: lostMessage },
{ annotation_level: "failure", message: "Process completed with exit code 1." },
],
readLog: noLog,
});
assert.equal(lostThenExit.lost, true);
const noisy = [
{ annotation_level: "failure", message: "gateway_indexing_incomplete" },
{ annotation_level: "failure", message: CANCELLED_MESSAGE },
];
const noisyLost = await classifyJob({ id: 1 }, {
annotations: noisy,
readLog: async () => "##[error]gateway_indexing_incomplete\n##[error]The runner has received a shutdown signal.\n",
});
assert.equal(noisyLost.lost, true);
const noisyCancelled = await classifyJob({ id: 1 }, {
annotations: noisy,
readLog: async () => "##[error]gateway_indexing_incomplete\n##[error]The operation was canceled.\n",
});
assert.equal(noisyCancelled.lost, false);
assert.match(noisyCancelled.evidence, /gateway_indexing_incomplete/);
// Runner loss with a cancellation only is still runner loss.
const lostAndCancelled = await classifyJob({ id: 1 }, {
annotations: [
{ annotation_level: "failure", message: lostMessage },
{ annotation_level: "failure", message: "The operation was canceled." },
],
readLog: noLog,
});
assert.equal(lostAndCancelled.lost, true);
});
test("summary cells cannot break the table", () => {
const summary = renderSummary({
runId: 1, attempt: 1, rerun: false, reason: "test",
jobs: [{ name: "ci / a|b", lost: false, evidence: "line one\nline | two" }],
});
assert.match(summary, /\| ci \/ a\\\|b \| no \| line one line \\\| two \|/);
});
test("workflow acts only on failed or cancelled first attempts of the canonical PR workflow", () => {
const condition = workflow.match(/^ {4}if: >-\n((?: {6}.+\n)+)/m)?.[1].replace(/\s+/g, " ").trim();
assert.equal(condition, [
"github.repository == 'paperclipai/paperclip' &&",
"github.repository_id == '1170821064' &&",
"github.event.workflow_run.event == 'pull_request' &&",
"github.event.workflow_run.path == '.github/workflows/pr.yml' &&",
"(github.event.workflow_run.conclusion == 'failure' ||",
"github.event.workflow_run.conclusion == 'cancelled') &&",
"github.event.workflow_run.run_attempt == 1",
].join(" "));
assert.match(workflow, /^on:\n {2}workflow_run:\n {4}workflows: \[PR\]\n {4}types: \[completed\]$/m);
assert.match(workflow, /^ {2}group: ci-runner-loss-rerun-\$\{\{ github\.event\.workflow_run\.id \}\}$/m);
const permissions = workflow.match(/^permissions:\n((?: {2}.+\n)+)/m)[1]
.split("\n").filter((line) => /^ {2}[a-z-]+:/.test(line)).map((line) => line.trim());
assert.deepEqual(permissions, ["actions: write", "checks: read", "contents: read", "pull-requests: read"]);
});
test("workflow runs only trusted default-branch code", () => {
assert.match(workflow, /ref: \$\{\{ github\.sha \}\}\n {10}persist-credentials: false\n {10}sparse-checkout: \.github\/scripts\n/);
assert.doesNotMatch(workflow, /head_sha|head_branch|download-artifact|secrets\.|pull_request_target/);
assert.equal([...workflow.matchAll(/uses: /g)].length, 1);
assert.match(workflow, /runs-on: ubuntu-latest/);
});