Files
PaperClipAI/.github/scripts/tests/runner-loss-rerun.test.mjs
T
DottaandClaude Fable 5.2 c4f639ed1b fix(ci): re-run cancelled runs that lost a runner, and let runner-loss evidence win (#21291)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - PR CI runs about 50 jobs per run on AWS spot instances. AWS can
reclaim a machine mid-job, and the job dies.
> - `ci-runner-loss-rerun.yml` re-runs such a run once, automatically,
when every failed job lost its runner.
> - In a 17-hour sample, 42 runs lost a runner and were still the PR's
current head. The automation rescued only 14.
> - Two gaps caused most of the misses. A spot reclaim on RunsOn v3.2.3
usually ends the job, and the run, as `cancelled`; the workflow and
script only look at `failure`. Also, the Runner support suite always
carries `gateway_indexing_incomplete` error annotations from a passing
test, so a lost runner there reads as a real failure.
> - On-demand fallback for the fleet was turned off on 2026-11-13, so
reclaims will happen more often. About 20% of runs are expected to lose
at least one job.
> - This pull request makes the re-run act on cancelled runs and jobs,
but only with positive runner-loss evidence, and lets that evidence win
over other error annotations.
> - The benefit is fewer manual "Re-run failed jobs" clicks: an
estimated 34 of 42 instead of 14 of 42 in the sample.

## Linked Issues or Issue Description

No public issue exists.

**What happened?**
Run
[43285064435](https://github.com/paperclipai/paperclip/actions/runs/43285064435)
lost the runner of `ci / General tests (server (8/12))`. The job
concluded `cancelled` (its log has `##[error]The runner has received a
shutdown signal`). The run concluded `cancelled`, so the re-run
workflow's `if:` skipped it.

Run
[43325855069](https://github.com/paperclipai/paperclip/actions/runs/43325855069)
lost the runners of `ci / Runner support suite` and `ci / General tests
(server (2/12))`. Both logs have the shutdown signal. The re-run bot
([decision
log](https://github.com/paperclipai/paperclip/actions/runs/43327873358))
said "1 of 1 failed jobs did not lose their runner", with evidence
`gateway_indexing_incomplete`. It ignored the cancelled job. It read the
annotation as a real failure.

**Expected behavior**
Both runs get one automatic re-run of their failed jobs.

**Steps to reproduce**
The new tests replay trimmed snapshots of both runs
(`.github/scripts/tests/fixtures/runner-loss-rerun/run-43285064435.json`
and `run-43325855069.json`) through the script.

In a 17-hour sample on 2026-11-12, 42 runs lost a runner and were still
the PR's current head. Only 14 were re-run automatically. 16 stayed red
because the run concluded `cancelled`. 4 stayed red because an error
annotation in the lost job was read as a test failure.

**Additional context**
The `gateway_indexing_incomplete` annotation does not come from the
shutdown. `scheduled-gateway-indexing.test.ts` prints `::error` lines
while it passes, 84 seconds before the shutdown signal in this run.
Attempt 2 of the same run passed with the same two failure annotations.
So, before this change, a lost runner in the Runner support suite could
never be re-run.

## What Changed

- `ci-runner-loss-rerun.yml`: the job also runs when the PR run
concluded `cancelled`. The trigger type, `permissions`, the repository,
path, event and attempt-1 conditions do not change.
- `runner-loss-rerun.mjs`, `eligibleRun`: accepts `completed/cancelled`
as well as `completed/failure`.
- `runner-loss-rerun.mjs`, `decide()`: classifies jobs that concluded
`cancelled` as well as `failure`. It still skips the `ci / verify` and
`ci / e2e` aggregates, and a failed or cancelled routing gate still
blocks the re-run.
- `decide()` now runs `staleReason` (open PR, current head, no newer
run) before it reads any annotation or log, and again just before the
re-run, as on master. A run cancelled by a new push has about 40
cancelled jobs; it now costs two API listings and no log reads. The
final check closes the window where a push lands during the log reads: a
re-run of the old run would join the PR's `cancel-in-progress`
concurrency group (`pr-<number>` in `pr-trusted.yml`) and cancel the new
commit's CI.
- Following CONTRIBUTING.md ("Do not submit a PR that includes code
comments"), this PR adds no new code comments. The reasons for each rule
are in this description and in `doc/DEVELOPING.md`.
- `classifyJob`: runner-loss evidence wins. A lost-communication
annotation means lost. A cancellation annotation plus a shutdown signal
in the log means lost, even if the job also has other error annotations.
A cancelled job without this evidence (for example, a person cancelled
the run) counts as a real failure and blocks the re-run.
- Fixtures: two new trimmed snapshots from the example runs (jobs with
`id`, `name`, `status`, `conclusion`; failure annotations; `##[error]`
log lines only). The two older fixtures get the real annotations and log
error lines of their cancelled jobs. Those jobs were spot-reclaimed too
(each log has the shutdown signal), but the old code ignored them.
- `doc/DEVELOPING.md`: the runner-loss paragraph describes the new
rules.

## Verification

- Red first. Against the unchanged script, the new tests fail for the
reasons above: 43285064435 gives `run is completed/cancelled, not
completed/failure`; 43325855069 gives `1 of 1 failed jobs did not lose
their runner` with evidence `gateway_indexing_incomplete`, the same
decision the bot logged.
- Green: `node --test .github/scripts/tests/runner-loss-rerun.test.mjs`:
25 of 25 pass.
- Push during log reads: the PR head moves after the first log read.
With the final freshness check removed, this test fails (the script
re-runs); with it, there is no re-run and the reason is `no open pull
request still has this run's head commit`.
- Guard cases, all with no re-run and no POST:
- Superseded run: 52 jobs cancelled, the PR head moved. The test asserts
zero annotation and log reads.
- Person cancelled the run: cancelled jobs with only "The operation was
canceled." in the annotation and log, and some never-started jobs with
no annotation.
- Mixed: one cancelled lost job and one real test failure (`1 of 2
failed jobs did not lose their runner`).
- Once only: a cancelled run on attempt 2 is refused after one API read.
The workflow condition test still requires `run_attempt == 1`.
  - Cancelled routing gate: left for a person.
- Policy job command: `node --test '.github/scripts/tests/*.test.mjs'`:
806 of 806 pass.
- `pnpm verify`: exit 2 ("Global CI input changed:
.github/scripts/runner-loss-rerun.mjs"). It selects no local Vitest
files and asks for required CI coverage. I did not run `pnpm verify
--full`; required CI on this PR supplies that coverage.
- This workflow runs on `workflow_run` from the default branch. This
PR's own CI cannot exercise the new behavior. The fixture tests are the
evidence. I did not dry-run the script against live runs.

## Risks

- A test that really fails, and then a spot reclaim of the same job a
minute later, now gets one wasted re-run. Attempt 2 then fails for real.
It cannot loop: the workflow `if:` and `eligibleRun` both accept attempt
1 only.
- A run a person cancels on the current head now starts the re-run job.
The job reads the annotations, and the log of each job with a
cancellation annotation, and then does not re-run. This costs some API
calls on a GitHub-hosted runner.
- A concurrency cancel by a label event that does not request CI (the
newer run is `skipped`) passes the head check. Its cancelled jobs have
no runner-loss evidence, so it is not re-run.
- Trust: no change to `permissions`, the trigger, the accepted
repository, event, workflow path, attempt, or the routing-gate rule. A
cancelled run reuses its attempt-1 gate output, the same as a failed run
does today.
- Estimated effect, from a 17-hour sample: 14 of 42 rescued before; 16
cancelled runs and 4 runs with error-annotation noise become eligible
(about 34 of 42). The other 8 stay skipped on purpose (a real test
failure in the same run, a person re-ran first, or the PR closed).

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), Claude Code agent. Model ID claude-opus-5-5, with
tool use (shell, GitHub CLI) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:31:50 -06:00

460 lines
23 KiB
JavaScript

import assert from "node:assert/strict";
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import test from "node:test";
import {
classifyJob, createGithubClient, decide, logShowsRunnerLoss, main, renderSummary,
} from "../runner-loss-rerun.mjs";
// Trimmed attempt-1 snapshots of real PR runs: jobs, check-run annotations for
// each failed job, and the error lines of job logs whose only annotation is
// "The operation was canceled.".
const fixture = (runId) => JSON.parse(readFileSync(
new URL(`./fixtures/runner-loss-rerun/run-${runId}.json`, import.meta.url), "utf8",
));
const MIXED = 42271918832; // six lost runners, four shutdowns in the log, one real test timeout
const REAL = 42264785415; // a real assertion failure
const ALL_LOST = 42272083862; // one lost runner
const LOG_ONLY = 42277969418; // shutdown signals visible only in the job logs
const CANCELLED_RUN = 43285064435;
const NOISY_LOST = 43325855069;
const CANCELLED_MESSAGE = "The operation was canceled.";
const workflow = readFileSync(new URL("../../workflows/ci-runner-loss-rerun.yml", import.meta.url), "utf8");
const json = (body, headers = {}) => new Response(JSON.stringify(body), {
status: 200, headers: { "content-type": "application/json", ...headers },
});
/**
* Serves a fixture as the GitHub REST API. By default the run is the latest
* CI for an open PR's current head.
*/
function fakeGithub(data, {
jobsPageSize = 100, pulls, prRuns, mutateRun = (run) => run,
} = {}) {
const calls = [];
const run = mutateRun(structuredClone(data.run));
const openPr = {
number: 1, state: "open",
base: { repo: { id: data.run.repository.id } },
head: { sha: data.run.head_sha, repo: { id: data.run.head_repository.id } },
};
const fetcher = async (href, init) => {
const url = new URL(href);
calls.push({ method: init.method, path: url.pathname, query: url.search, auth: init.headers.Authorization });
const base = `/repos/paperclipai/paperclip`;
const route = url.pathname.slice(base.length);
if (init.method === "POST" && route === `/actions/runs/${run.id}/rerun-failed-jobs`) {
return new Response(null, { status: 201 });
}
if (route === `/actions/runs/${run.id}`) return json(run);
if (route === `/actions/runs/${run.id}/attempts/1/jobs`) {
const page = Number(url.searchParams.get("page") ?? 1);
const jobs = data.jobs.slice((page - 1) * jobsPageSize, page * jobsPageSize);
const last = page * jobsPageSize >= data.jobs.length;
const next = new URL(url);
next.searchParams.set("page", String(page + 1));
return json({ total_count: data.total_count, jobs }, last ? {} : { link: `<${next.href}>; rel="next"` });
}
let match = route.match(/^\/check-runs\/(\d+)\/annotations$/);
if (match) return json(data.annotations[match[1]] ?? []);
match = route.match(/^\/actions\/jobs\/(\d+)\/logs$/);
if (match) return new Response(data.logs[match[1]] ?? "", { status: 200 });
if (route === `/commits/${run.head_sha}/pulls`) return json((typeof pulls === "function" ? pulls(calls) : pulls) ?? [openPr]);
if (route === "/actions/workflows/pr.yml/runs") {
const runs = prRuns ?? [{ id: run.id, conclusion: "failure" }];
return json({ total_count: runs.length, workflow_runs: runs });
}
return new Response("not found", { status: 404 });
};
return { calls, fetcher, run, client: createGithubClient({ token: "test-token", fetcher }) };
}
const posted = (calls) => calls.filter((call) => call.method === "POST");
async function runMain(data, options = {}, eventRun = {}) {
const github = fakeGithub(data, options);
const directory = mkdtempSync(path.join(tmpdir(), "runner-loss-rerun-"));
try {
const eventPath = path.join(directory, "event.json");
const summaryPath = path.join(directory, "summary.md");
writeFileSync(eventPath, JSON.stringify({ workflow_run: { id: data.run.id, run_attempt: 1, ...eventRun } }));
writeFileSync(summaryPath, "");
const decision = await main({
env: { GITHUB_TOKEN: "test-token", GITHUB_EVENT_PATH: eventPath, GITHUB_STEP_SUMMARY: summaryPath },
argv: [], fetcher: github.fetcher, log: () => {},
});
return { decision, calls: github.calls, summary: readFileSync(summaryPath, "utf8") };
} finally {
rmSync(directory, { recursive: true, force: true });
}
}
test("all failed jobs lost their runner: re-runs failed jobs once", async () => {
const { decision, calls, summary } = await runMain(fixture(ALL_LOST));
assert.equal(decision.rerun, true);
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [["ci / General tests (server (3/12))", true]]);
assert.deepEqual(posted(calls).map((call) => call.path),
[`/repos/paperclipai/paperclip/actions/runs/${ALL_LOST}/rerun-failed-jobs`]);
assert.ok(calls.every((call) => call.auth === "Bearer test-token"));
// `ci / verify` failed too, but only because its dependency failed.
assert.equal(calls.some((call) => call.path.endsWith("/check-runs/128088995071/annotations")), false);
assert.match(summary, /\*\*Re-ran failed jobs\*\*/);
assert.match(summary, /\| ci \/ General tests \(server \(3\/12\)\) \| yes \| The self-hosted runner lost communication/);
});
test("a shutdown signal found only in the job log counts as runner loss", async () => {
const data = fixture(LOG_ONLY);
assert.deepEqual(data.annotations["128101016102"].map((annotation) => annotation.message), ["The operation was canceled."]);
const { decision, calls } = await runMain(data);
assert.equal(decision.rerun, true);
assert.equal(decision.jobs.length, 11);
assert.ok(decision.jobs.every((job) => job.lost && /job log/.test(job.evidence)));
assert.ok(calls.some((call) => call.path.endsWith("/actions/jobs/128101016102/logs")));
assert.equal(posted(calls).length, 1);
});
test("a cancelled run whose cancelled job lost its runner is re-run once", async () => {
const data = fixture(CANCELLED_RUN);
assert.equal(data.run.conclusion, "cancelled");
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
assert.equal(decision.rerun, true);
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [["ci / General tests (server (8/12))", true]]);
assert.match(decision.jobs[0].evidence, /job log/);
assert.deepEqual(posted(calls).map((call) => call.path),
[`/repos/paperclipai/paperclip/actions/runs/${CANCELLED_RUN}/rerun-failed-jobs`]);
});
test("runner loss in the log wins over error annotations the workload printed", async () => {
const data = fixture(NOISY_LOST);
const suite = data.jobs.find((job) => job.name === "ci / Runner support suite");
assert.ok(data.annotations[suite.id].some((annotation) => annotation.message === "gateway_indexing_incomplete"));
const { decision, calls } = await runMain(data);
assert.equal(decision.rerun, true);
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [
["ci / Runner support suite", true],
["ci / General tests (server (2/12))", true],
]);
assert.equal(posted(calls).length, 1);
});
function cancelEverything(data, { log = `2026-11-12T19:03:32.0000000Z ##[error]${CANCELLED_MESSAGE}\n` } = {}) {
data.annotations = {};
data.logs = {};
for (const [index, job] of data.jobs.entries()) {
if (job.name.startsWith("ci / gate / ")) continue;
job.conclusion = "cancelled";
if (index % 4 === 3) continue;
data.annotations[job.id] = [{ annotation_level: "failure", message: CANCELLED_MESSAGE }];
data.logs[job.id] = log;
}
return data;
}
test("a run superseded by a new push is not re-run, and no job is classified", async () => {
const data = cancelEverything(fixture(CANCELLED_RUN));
assert.ok(data.jobs.filter((job) => job.conclusion === "cancelled").length >= 40);
const { decision, calls } = await runMain(data, {
pulls: [{ state: "open", base: { repo: { id: 1170821064 } }, head: { sha: "0".repeat(40), repo: { id: 1170821064 } } }],
}, { conclusion: "cancelled" });
assert.equal(decision.rerun, false);
assert.match(decision.reason, /no open pull request/);
assert.equal(calls.some((call) => call.path.includes("/check-runs/") || call.path.endsWith("/logs")), false);
assert.equal(posted(calls).length, 0);
});
test("a push while job logs are read stops the re-run", async () => {
const data = fixture(CANCELLED_RUN);
const moved = [{ state: "open", base: { repo: { id: 1170821064 } }, head: { sha: "0".repeat(40), repo: { id: 1170821064 } } }];
const { decision, calls } = await runMain(data, {
pulls: (seen) => (seen.some((call) => call.path.endsWith("/logs")) ? moved : undefined),
}, { conclusion: "cancelled" });
assert.equal(decision.rerun, false);
assert.match(decision.reason, /no open pull request/);
assert.equal(decision.jobs[0].lost, true);
assert.equal(calls.filter((call) => call.path.endsWith("/pulls")).length, 2);
assert.equal(posted(calls).length, 0);
});
test("a run a person cancelled is not re-run", async () => {
const data = cancelEverything(fixture(CANCELLED_RUN));
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
assert.equal(decision.rerun, false);
assert.ok(decision.jobs.length >= 40);
assert.equal(decision.jobs.some((job) => job.lost), false);
assert.match(decision.reason, /failed jobs did not lose their runner/);
assert.equal(posted(calls).length, 0);
});
test("a cancelled run with one lost job and one real test failure is not re-run", async () => {
const data = fixture(CANCELLED_RUN);
const failing = data.jobs.find((job) => job.name === "ci / General tests (server (1/12))");
failing.conclusion = "failure";
data.annotations[failing.id] = [
{ annotation_level: "failure", message: "AssertionError: expected 'queued' to be 'running'" },
{ annotation_level: "failure", message: "Process completed with exit code 1." },
];
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
assert.equal(decision.rerun, false);
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [
["ci / General tests (server (1/12))", false],
["ci / General tests (server (8/12))", true],
]);
assert.match(decision.reason, /1 of 2 failed jobs did not lose their runner/);
assert.equal(posted(calls).length, 0);
});
test("a cancelled run is re-run at most once", async () => {
const { decision, calls } = await runMain(fixture(CANCELLED_RUN), {
mutateRun: (run) => ({ ...run, run_attempt: 2 }),
}, { conclusion: "cancelled" });
assert.equal(decision.rerun, false);
assert.match(decision.reason, /attempt 2/);
assert.deepEqual(calls.map((call) => call.path), [`/repos/paperclipai/paperclip/actions/runs/${CANCELLED_RUN}`]);
});
test("a cancelled routing gate is left for a human re-run", async () => {
const data = fixture(CANCELLED_RUN);
const gate = data.jobs.find((job) => job.name === "ci / gate / Select trusted runner");
gate.conclusion = "cancelled";
data.annotations[gate.id] = [{ annotation_level: "failure", message: CANCELLED_MESSAGE }];
data.logs[gate.id] = data.logs[Object.keys(data.logs)[0]];
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
assert.equal(decision.rerun, false);
assert.match(decision.jobs.find((job) => job.id === gate.id).evidence, /routing gate/);
assert.equal(posted(calls).length, 0);
});
test("mixed runner loss and a real test failure: no re-run", async () => {
const { decision, calls, summary } = await runMain(fixture(MIXED));
assert.equal(decision.rerun, false);
assert.equal(decision.jobs.length, 11);
assert.deepEqual(decision.jobs.filter((job) => !job.lost).map((job) => job.name), ["ci / General tests (workspaces-a (1/2))"]);
assert.equal(decision.jobs.find((job) => job.name === "ci / Verify Paperclip Runner (vitest package)").lost, true);
assert.equal(posted(calls).length, 0);
assert.match(decision.reason, /1 of 11 failed jobs did not lose their runner/);
assert.match(summary, /\*\*No automatic re-run\*\*/);
});
test("real test failures: no re-run", async () => {
const { decision, calls } = await runMain(fixture(REAL));
assert.equal(decision.rerun, false);
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [["ci / General tests (server (4/12))", false]]);
assert.equal(posted(calls).length, 0);
});
test("only the verify and e2e aggregates failed: no re-run", async () => {
const data = fixture(ALL_LOST);
data.jobs = data.jobs.map((job) => job.name.startsWith("ci / General tests") ? { ...job, conclusion: "success" } : job);
const { decision, calls } = await runMain(data);
assert.equal(decision.rerun, false);
assert.match(decision.reason, /no failed jobs other than the verify\/e2e aggregates/);
assert.equal(calls.some((call) => call.path.includes("/check-runs/")), false);
assert.equal(posted(calls).length, 0);
});
test("attempt 2 or later is never re-run automatically", async () => {
// The event was for attempt 1, but someone re-ran the run before this job started.
const { decision, calls } = await runMain(fixture(ALL_LOST), { mutateRun: (run) => ({ ...run, run_attempt: 2 }) });
assert.equal(decision.rerun, false);
assert.match(decision.reason, /attempt 2/);
assert.deepEqual(calls.map((call) => call.path), [`/repos/paperclipai/paperclip/actions/runs/${ALL_LOST}`]);
});
test("a failed routing gate is left for a human re-run", async () => {
const data = fixture(ALL_LOST);
const gate = data.jobs.find((job) => job.name === "ci / gate / Select trusted runner");
gate.conclusion = "failure";
data.annotations[gate.id] = [{ annotation_level: "failure", message: "The self-hosted runner lost communication with the server." }];
const { decision, calls } = await runMain(data);
assert.equal(decision.rerun, false);
assert.match(decision.jobs.find((job) => job.id === gate.id).evidence, /routing gate/);
assert.equal(posted(calls).length, 0);
});
test("runs that are no longer the PR's latest CI are not re-run", async () => {
const data = fixture(ALL_LOST);
const moved = await decide(fakeGithub(data, {
pulls: [{ state: "open", base: { repo: { id: 1170821064 } }, head: { sha: "0".repeat(40), repo: { id: 1170821064 } } }],
}).client, data.run);
assert.equal(moved.rerun, false);
assert.match(moved.reason, /no open pull request/);
const merged = await decide(fakeGithub(data, {
pulls: [{ state: "closed", base: { repo: { id: 1170821064 } }, head: { sha: data.run.head_sha, repo: { id: 1170821064 } } }],
}).client, data.run);
assert.equal(merged.rerun, false);
// A later full-ci label run on the same head shares the PR concurrency group;
// re-running the older run would cancel it.
const newer = await decide(fakeGithub(data, {
prRuns: [{ id: data.run.id + 1, conclusion: null }, { id: data.run.id, conclusion: "failure" }],
}).client, data.run);
assert.equal(newer.rerun, false);
assert.match(newer.reason, /newer PR run/);
// A later label event that did not request CI skips its only job; it is not CI.
const skipped = await decide(fakeGithub(data, {
prRuns: [{ id: data.run.id + 1, conclusion: "skipped" }, { id: data.run.id, conclusion: "failure" }],
}).client, data.run);
assert.equal(skipped.rerun, true);
});
test("ineligible runs are rejected before any job is read", async () => {
const data = fixture(ALL_LOST);
for (const [change, reason] of [
[{ event: "push" }, /event is push/],
[{ path: ".github/workflows/other.yml" }, /workflow is/],
[{ conclusion: "success" }, /completed\/success/],
[{ repository: { id: 1, full_name: "someone/paperclip" } }, /canonical repository/],
]) {
const github = fakeGithub(data);
const decision = await decide(github.client, { ...data.run, ...change });
assert.equal(decision.rerun, false);
assert.match(decision.reason, reason);
assert.equal(github.calls.length, 0);
}
});
test("job listing follows Link pagination across pages", async () => {
const data = fixture(MIXED);
const github = fakeGithub(data, { jobsPageSize: 20 });
const decision = await decide(github.client, data.run);
const pages = github.calls.filter((call) => call.path.endsWith("/attempts/1/jobs"));
assert.deepEqual(pages.map((call) => new URLSearchParams(call.query).get("page")), [null, "2", "3"]);
// Failed jobs on pages 2 and 3 are all classified.
assert.equal(decision.jobs.length, 11);
assert.equal(decision.rerun, false);
});
test("an incomplete job listing fails instead of deciding on partial data", async () => {
const data = fixture(ALL_LOST);
data.total_count += 1;
const github = fakeGithub(data);
await assert.rejects(decide(github.client, data.run), /Incomplete listing/);
assert.equal(posted(github.calls).length, 0);
});
test("the client refuses to follow pagination to another host", async () => {
const client = createGithubClient({
token: "test-token",
fetcher: async () => json([], { link: '<https://example.com/steal?page=2>; rel="next"' }),
});
await assert.rejects(client.paginate("/repos/paperclipai/paperclip/commits/x/pulls"), /Refusing to call https:\/\/example.com/);
});
test("classification is conservative about cancelled jobs", async () => {
const cancelled = [{ annotation_level: "failure", message: "The operation was canceled." }];
const noSignal = await classifyJob({ id: 1 }, { annotations: cancelled, readLog: async () => "##[error]The operation was canceled.\n" });
assert.equal(noSignal.lost, false);
// A job timeout also cancels the step, but adds its own failure annotation.
const timedOut = await classifyJob({ id: 1 }, {
annotations: [{ annotation_level: "failure", message: "The job has exceeded the maximum execution time of 30m0s" }, ...cancelled],
readLog: async () => "##[error]The job has exceeded the maximum execution time of 30m0s\n##[error]The operation was canceled.\n",
});
assert.equal(timedOut.lost, false);
assert.match(timedOut.evidence, /maximum execution time/);
const notStarted = await classifyJob({ id: 1, conclusion: "cancelled" }, {
annotations: [],
readLog: async () => { throw new Error("log must not be read"); },
});
assert.equal(notStarted.lost, false);
const exitOnly = await classifyJob({ id: 1 }, {
annotations: [{ annotation_level: "failure", message: "Process completed with exit code 1." }],
readLog: async () => { throw new Error("log must not be read"); },
});
assert.equal(exitOnly.lost, false);
// Notices and warnings are not failure evidence.
const warning = await classifyJob({ id: 1 }, {
annotations: [{ annotation_level: "warning", message: "The self-hosted runner lost communication with the server." }],
readLog: async () => "",
});
assert.equal(warning.lost, false);
// Test output that merely prints the phrase is not a runner error line.
assert.equal(logShowsRunnerLoss("2026-11-06T00:00:00Z expected 'received a shutdown signal'\n"), false);
assert.equal(logShowsRunnerLoss(fixture(MIXED).logs["128078674151"]), true);
});
test("runner-loss evidence wins over other failure annotations in the same job", async () => {
const lostMessage = "The self-hosted runner lost communication with the server. Verify the machine is running.";
const noLog = async () => { throw new Error("log must not be read"); };
// A test fails, then the runner is lost during an always() upload step.
const testThenLost = await classifyJob({ id: 1 }, {
annotations: [
{ annotation_level: "failure", message: "Error: Test timed out in 15000ms." },
{ annotation_level: "failure", message: "Process completed with exit code 1." },
{ annotation_level: "failure", message: lostMessage },
],
readLog: noLog,
});
assert.equal(testThenLost.lost, true);
assert.match(testThenLost.evidence, /lost communication/);
// The order of the annotations does not matter.
const lostThenExit = await classifyJob({ id: 1 }, {
annotations: [
{ annotation_level: "failure", message: lostMessage },
{ annotation_level: "failure", message: "Process completed with exit code 1." },
],
readLog: noLog,
});
assert.equal(lostThenExit.lost, true);
const noisy = [
{ annotation_level: "failure", message: "gateway_indexing_incomplete" },
{ annotation_level: "failure", message: CANCELLED_MESSAGE },
];
const noisyLost = await classifyJob({ id: 1 }, {
annotations: noisy,
readLog: async () => "##[error]gateway_indexing_incomplete\n##[error]The runner has received a shutdown signal.\n",
});
assert.equal(noisyLost.lost, true);
const noisyCancelled = await classifyJob({ id: 1 }, {
annotations: noisy,
readLog: async () => "##[error]gateway_indexing_incomplete\n##[error]The operation was canceled.\n",
});
assert.equal(noisyCancelled.lost, false);
assert.match(noisyCancelled.evidence, /gateway_indexing_incomplete/);
// Runner loss with a cancellation only is still runner loss.
const lostAndCancelled = await classifyJob({ id: 1 }, {
annotations: [
{ annotation_level: "failure", message: lostMessage },
{ annotation_level: "failure", message: "The operation was canceled." },
],
readLog: noLog,
});
assert.equal(lostAndCancelled.lost, true);
});
test("summary cells cannot break the table", () => {
const summary = renderSummary({
runId: 1, attempt: 1, rerun: false, reason: "test",
jobs: [{ name: "ci / a|b", lost: false, evidence: "line one\nline | two" }],
});
assert.match(summary, /\| ci \/ a\\\|b \| no \| line one line \\\| two \|/);
});
test("workflow acts only on failed or cancelled first attempts of the canonical PR workflow", () => {
const condition = workflow.match(/^ {4}if: >-\n((?: {6}.+\n)+)/m)?.[1].replace(/\s+/g, " ").trim();
assert.equal(condition, [
"github.repository == 'paperclipai/paperclip' &&",
"github.repository_id == '1170821064' &&",
"github.event.workflow_run.event == 'pull_request' &&",
"github.event.workflow_run.path == '.github/workflows/pr.yml' &&",
"(github.event.workflow_run.conclusion == 'failure' ||",
"github.event.workflow_run.conclusion == 'cancelled') &&",
"github.event.workflow_run.run_attempt == 1",
].join(" "));
assert.match(workflow, /^on:\n {2}workflow_run:\n {4}workflows: \[PR\]\n {4}types: \[completed\]$/m);
assert.match(workflow, /^ {2}group: ci-runner-loss-rerun-\$\{\{ github\.event\.workflow_run\.id \}\}$/m);
const permissions = workflow.match(/^permissions:\n((?: {2}.+\n)+)/m)[1]
.split("\n").filter((line) => /^ {2}[a-z-]+:/.test(line)).map((line) => line.trim());
assert.deepEqual(permissions, ["actions: write", "checks: read", "contents: read", "pull-requests: read"]);
});
test("workflow runs only trusted default-branch code", () => {
assert.match(workflow, /ref: \$\{\{ github\.sha \}\}\n {10}persist-credentials: false\n {10}sparse-checkout: \.github\/scripts\n/);
assert.doesNotMatch(workflow, /head_sha|head_branch|download-artifact|secrets\.|pull_request_target/);
assert.equal([...workflow.matchAll(/uses: /g)].length, 1);
assert.match(workflow, /runs-on: ubuntu-latest/);
});