mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - PR CI runs about 50 jobs per run on AWS spot instances. AWS can reclaim a machine mid-job, and the job dies. > - `ci-runner-loss-rerun.yml` re-runs such a run once, automatically, when every failed job lost its runner. > - In a 17-hour sample, 42 runs lost a runner and were still the PR's current head. The automation rescued only 14. > - Two gaps caused most of the misses. A spot reclaim on RunsOn v3.2.3 usually ends the job, and the run, as `cancelled`; the workflow and script only look at `failure`. Also, the Runner support suite always carries `gateway_indexing_incomplete` error annotations from a passing test, so a lost runner there reads as a real failure. > - On-demand fallback for the fleet was turned off on 2026-11-13, so reclaims will happen more often. About 20% of runs are expected to lose at least one job. > - This pull request makes the re-run act on cancelled runs and jobs, but only with positive runner-loss evidence, and lets that evidence win over other error annotations. > - The benefit is fewer manual "Re-run failed jobs" clicks: an estimated 34 of 42 instead of 14 of 42 in the sample. ## Linked Issues or Issue Description No public issue exists. **What happened?** Run [43285064435](https://github.com/paperclipai/paperclip/actions/runs/43285064435) lost the runner of `ci / General tests (server (8/12))`. The job concluded `cancelled` (its log has `##[error]The runner has received a shutdown signal`). The run concluded `cancelled`, so the re-run workflow's `if:` skipped it. Run [43325855069](https://github.com/paperclipai/paperclip/actions/runs/43325855069) lost the runners of `ci / Runner support suite` and `ci / General tests (server (2/12))`. Both logs have the shutdown signal. The re-run bot ([decision log](https://github.com/paperclipai/paperclip/actions/runs/43327873358)) said "1 of 1 failed jobs did not lose their runner", with evidence `gateway_indexing_incomplete`. It ignored the cancelled job. It read the annotation as a real failure. **Expected behavior** Both runs get one automatic re-run of their failed jobs. **Steps to reproduce** The new tests replay trimmed snapshots of both runs (`.github/scripts/tests/fixtures/runner-loss-rerun/run-43285064435.json` and `run-43325855069.json`) through the script. In a 17-hour sample on 2026-11-12, 42 runs lost a runner and were still the PR's current head. Only 14 were re-run automatically. 16 stayed red because the run concluded `cancelled`. 4 stayed red because an error annotation in the lost job was read as a test failure. **Additional context** The `gateway_indexing_incomplete` annotation does not come from the shutdown. `scheduled-gateway-indexing.test.ts` prints `::error` lines while it passes, 84 seconds before the shutdown signal in this run. Attempt 2 of the same run passed with the same two failure annotations. So, before this change, a lost runner in the Runner support suite could never be re-run. ## What Changed - `ci-runner-loss-rerun.yml`: the job also runs when the PR run concluded `cancelled`. The trigger type, `permissions`, the repository, path, event and attempt-1 conditions do not change. - `runner-loss-rerun.mjs`, `eligibleRun`: accepts `completed/cancelled` as well as `completed/failure`. - `runner-loss-rerun.mjs`, `decide()`: classifies jobs that concluded `cancelled` as well as `failure`. It still skips the `ci / verify` and `ci / e2e` aggregates, and a failed or cancelled routing gate still blocks the re-run. - `decide()` now runs `staleReason` (open PR, current head, no newer run) before it reads any annotation or log, and again just before the re-run, as on master. A run cancelled by a new push has about 40 cancelled jobs; it now costs two API listings and no log reads. The final check closes the window where a push lands during the log reads: a re-run of the old run would join the PR's `cancel-in-progress` concurrency group (`pr-<number>` in `pr-trusted.yml`) and cancel the new commit's CI. - Following CONTRIBUTING.md ("Do not submit a PR that includes code comments"), this PR adds no new code comments. The reasons for each rule are in this description and in `doc/DEVELOPING.md`. - `classifyJob`: runner-loss evidence wins. A lost-communication annotation means lost. A cancellation annotation plus a shutdown signal in the log means lost, even if the job also has other error annotations. A cancelled job without this evidence (for example, a person cancelled the run) counts as a real failure and blocks the re-run. - Fixtures: two new trimmed snapshots from the example runs (jobs with `id`, `name`, `status`, `conclusion`; failure annotations; `##[error]` log lines only). The two older fixtures get the real annotations and log error lines of their cancelled jobs. Those jobs were spot-reclaimed too (each log has the shutdown signal), but the old code ignored them. - `doc/DEVELOPING.md`: the runner-loss paragraph describes the new rules. ## Verification - Red first. Against the unchanged script, the new tests fail for the reasons above: 43285064435 gives `run is completed/cancelled, not completed/failure`; 43325855069 gives `1 of 1 failed jobs did not lose their runner` with evidence `gateway_indexing_incomplete`, the same decision the bot logged. - Green: `node --test .github/scripts/tests/runner-loss-rerun.test.mjs`: 25 of 25 pass. - Push during log reads: the PR head moves after the first log read. With the final freshness check removed, this test fails (the script re-runs); with it, there is no re-run and the reason is `no open pull request still has this run's head commit`. - Guard cases, all with no re-run and no POST: - Superseded run: 52 jobs cancelled, the PR head moved. The test asserts zero annotation and log reads. - Person cancelled the run: cancelled jobs with only "The operation was canceled." in the annotation and log, and some never-started jobs with no annotation. - Mixed: one cancelled lost job and one real test failure (`1 of 2 failed jobs did not lose their runner`). - Once only: a cancelled run on attempt 2 is refused after one API read. The workflow condition test still requires `run_attempt == 1`. - Cancelled routing gate: left for a person. - Policy job command: `node --test '.github/scripts/tests/*.test.mjs'`: 806 of 806 pass. - `pnpm verify`: exit 2 ("Global CI input changed: .github/scripts/runner-loss-rerun.mjs"). It selects no local Vitest files and asks for required CI coverage. I did not run `pnpm verify --full`; required CI on this PR supplies that coverage. - This workflow runs on `workflow_run` from the default branch. This PR's own CI cannot exercise the new behavior. The fixture tests are the evidence. I did not dry-run the script against live runs. ## Risks - A test that really fails, and then a spot reclaim of the same job a minute later, now gets one wasted re-run. Attempt 2 then fails for real. It cannot loop: the workflow `if:` and `eligibleRun` both accept attempt 1 only. - A run a person cancels on the current head now starts the re-run job. The job reads the annotations, and the log of each job with a cancellation annotation, and then does not re-run. This costs some API calls on a GitHub-hosted runner. - A concurrency cancel by a label event that does not request CI (the newer run is `skipped`) passes the head check. Its cancelled jobs have no runner-loss evidence, so it is not re-run. - Trust: no change to `permissions`, the trigger, the accepted repository, event, workflow path, attempt, or the routing-gate rule. A cancelled run reuses its attempt-1 gate output, the same as a failed run does today. - Estimated effect, from a 17-hour sample: 14 of 42 rescued before; 16 cancelled runs and 4 runs with error-annotation noise become eligible (about 34 of 42). The other 8 stay skipped on purpose (a real test failure in the same run, a person re-ran first, or the PR closed). > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), Claude Code agent. Model ID claude-opus-5-5, with tool use (shell, GitHub CLI) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
206 lines
10 KiB
JavaScript
206 lines
10 KiB
JavaScript
#!/usr/bin/env node
|
|
/**
|
|
* runner-loss-rerun.mjs
|
|
*
|
|
* Re-runs the failed jobs of a first PR CI attempt once, but only when every
|
|
* failed or cancelled job ended because its runner was lost. Runs from
|
|
* .github/workflows/ci-runner-loss-rerun.yml on workflow_run. It reads GitHub
|
|
* API metadata only and never touches the PR's code or artifacts.
|
|
*
|
|
* Local dry run against a real run (no rerun is requested):
|
|
* GITHUB_TOKEN=$(gh auth token) node .github/scripts/runner-loss-rerun.mjs --run-id <id> --dry-run
|
|
*/
|
|
import { appendFileSync, readFileSync } from "node:fs";
|
|
import path from "node:path";
|
|
import { fileURLToPath } from "node:url";
|
|
|
|
export const REPOSITORY = "paperclipai/paperclip";
|
|
export const REPOSITORY_ID = 1170821064;
|
|
export const PR_WORKFLOW_PATH = ".github/workflows/pr.yml";
|
|
|
|
// The runner's own messages when an AWS instance disappears mid-job.
|
|
export const RUNNER_LOSS_PATTERN = /lost communication with the server|received a shutdown signal/;
|
|
// `verify` and `e2e` in pr-trusted.yml only assert their `needs` results. They
|
|
// fail whenever a dependency failed and re-run with it.
|
|
export const AGGREGATE_JOBS = new Set(["ci / verify", "ci / e2e"]);
|
|
// ci-routing.yml checks the attempt's triggering actor. A bot-triggered attempt
|
|
// would move the whole run to GitHub-hosted runners, so leave it to a human.
|
|
export const GATE_JOB_PREFIX = "ci / gate / ";
|
|
export const RERUN_CONCLUSIONS = new Set(["failure", "cancelled"]);
|
|
// A shutdown signal often reaches the check run only as this annotation; the
|
|
// signal itself is logged as an error line in the job log.
|
|
const CANCELLED_MESSAGE = "The operation was canceled.";
|
|
|
|
export function createGithubClient({ token, fetcher = fetch, apiUrl = "https://api.github.com" }) {
|
|
if (!token) throw new Error("GITHUB_TOKEN is required");
|
|
const origin = new URL(apiUrl).origin;
|
|
const request = async (target, { method = "GET" } = {}) => {
|
|
const url = new URL(target, apiUrl);
|
|
if (url.origin !== origin) throw new Error(`Refusing to call ${url.origin}`);
|
|
const response = await fetcher(url.href, {
|
|
method,
|
|
headers: { Accept: "application/vnd.github+json", Authorization: `Bearer ${token}`, "X-GitHub-Api-Version": "2022-11-28" },
|
|
signal: AbortSignal.timeout(60_000),
|
|
});
|
|
if (!response.ok) throw new Error(`GitHub API ${method} ${url.pathname} failed (HTTP ${response.status})`);
|
|
return response;
|
|
};
|
|
return {
|
|
json: async (target) => (await request(target)).json(),
|
|
// Follows Link rel="next" until the last page. `field` selects the array
|
|
// in wrapped responses such as { total_count, jobs }.
|
|
paginate: async (target, field) => {
|
|
const items = [];
|
|
let totalCount;
|
|
for (let next = target, pages = 0; next; pages += 1) {
|
|
if (pages >= 100) throw new Error(`Too many pages for ${target}`);
|
|
const response = await request(next);
|
|
const body = await response.json();
|
|
const page = field ? body[field] : body;
|
|
if (!Array.isArray(page)) throw new Error(`Unexpected page shape for ${target}`);
|
|
items.push(...page);
|
|
if (field && Number.isInteger(body.total_count)) totalCount = body.total_count;
|
|
next = /<([^>]+)>;\s*rel="next"/.exec(response.headers.get("link") ?? "")?.[1];
|
|
}
|
|
if (totalCount !== undefined && items.length !== totalCount) {
|
|
throw new Error(`Incomplete listing for ${target}: ${items.length} of ${totalCount}`);
|
|
}
|
|
return items;
|
|
},
|
|
// Job logs answer with a redirect to a signed download URL.
|
|
text: async (target) => (await request(target)).text(),
|
|
post: async (target) => { await request(target, { method: "POST" }); },
|
|
};
|
|
}
|
|
|
|
export function logShowsRunnerLoss(log) {
|
|
return log.split("\n").some((line) => line.includes("##[error]") && RUNNER_LOSS_PATTERN.test(line));
|
|
}
|
|
|
|
/** Classifies one failed job. Logs are read only when annotations are inconclusive. */
|
|
export async function classifyJob(job, { annotations, readLog }) {
|
|
const failures = annotations.filter((annotation) => annotation.annotation_level === "failure");
|
|
const isRunnerLoss = (annotation) => RUNNER_LOSS_PATTERN.test(annotation.message ?? "");
|
|
const match = failures.find(isRunnerLoss);
|
|
if (match) return { lost: true, evidence: firstLine(match.message) };
|
|
if (failures.some((annotation) => annotation.message === CANCELLED_MESSAGE) && logShowsRunnerLoss(await readLog(job))) {
|
|
return { lost: true, evidence: "job log: runner received a shutdown signal or lost communication" };
|
|
}
|
|
const real = failures.find((annotation) => annotation.message !== CANCELLED_MESSAGE);
|
|
if (real) return { lost: false, evidence: firstLine(real.message) };
|
|
if (failures.length > 0) return { lost: false, evidence: `${CANCELLED_MESSAGE} (no runner-loss error in the job log)` };
|
|
return { lost: false, evidence: "no failure annotation" };
|
|
}
|
|
|
|
function firstLine(message) {
|
|
const line = String(message).split("\n")[0];
|
|
return line.length > 160 ? `${line.slice(0, 157)}...` : line;
|
|
}
|
|
|
|
export function eligibleRun(run) {
|
|
if (run.repository?.full_name !== REPOSITORY || run.repository?.id !== REPOSITORY_ID) return "run is not in the canonical repository";
|
|
if (run.path !== PR_WORKFLOW_PATH) return `workflow is ${run.path}, not ${PR_WORKFLOW_PATH}`;
|
|
if (run.event !== "pull_request") return `event is ${run.event}, not pull_request`;
|
|
if (run.status !== "completed" || !RERUN_CONCLUSIONS.has(run.conclusion)) {
|
|
return `run is ${run.status}/${run.conclusion}, not completed/failure or completed/cancelled`;
|
|
}
|
|
// One automatic re-run only. A higher attempt also means someone re-ran it.
|
|
if (run.run_attempt !== 1) return `run is on attempt ${run.run_attempt}; only attempt 1 is re-run automatically`;
|
|
return undefined;
|
|
}
|
|
|
|
/**
|
|
* Re-running an older run joins the PR's `cancel-in-progress` concurrency
|
|
* group and would cancel newer CI. Require the run to be the latest CI for the
|
|
* PR's current head.
|
|
*/
|
|
export async function staleReason(client, run) {
|
|
const pulls = await client.paginate(`/repos/${REPOSITORY}/commits/${run.head_sha}/pulls?per_page=100`);
|
|
const open = pulls.filter((pr) => pr.state === "open" && pr.base?.repo?.id === REPOSITORY_ID
|
|
&& pr.head?.repo?.id === run.head_repository?.id && pr.head?.sha === run.head_sha);
|
|
if (open.length === 0) return "no open pull request still has this run's head commit";
|
|
const runs = await client.paginate(
|
|
`/repos/${REPOSITORY}/actions/workflows/pr.yml/runs?event=pull_request&head_sha=${run.head_sha}&per_page=100`,
|
|
"workflow_runs",
|
|
);
|
|
const newer = runs.find((candidate) => candidate.id > run.id && candidate.conclusion !== "skipped");
|
|
if (newer) return `a newer PR run (${newer.id}) exists for this head commit`;
|
|
return undefined;
|
|
}
|
|
|
|
export async function decide(client, run) {
|
|
const decision = { runId: run.id, attempt: run.run_attempt, rerun: false, jobs: [] };
|
|
const ineligible = eligibleRun(run);
|
|
if (ineligible) return { ...decision, reason: ineligible };
|
|
|
|
const jobs = await client.paginate(`/repos/${REPOSITORY}/actions/runs/${run.id}/attempts/1/jobs?per_page=100`, "jobs");
|
|
const failed = jobs.filter((job) => RERUN_CONCLUSIONS.has(job.conclusion) && !AGGREGATE_JOBS.has(job.name));
|
|
if (failed.length === 0) return { ...decision, reason: "no failed jobs other than the verify/e2e aggregates" };
|
|
|
|
const staleBefore = await staleReason(client, run);
|
|
if (staleBefore) return { ...decision, reason: staleBefore };
|
|
|
|
for (const job of failed) {
|
|
if (job.name.startsWith(GATE_JOB_PREFIX)) {
|
|
decision.jobs.push({ id: job.id, name: job.name, lost: false, evidence: "routing gate failed; a bot re-run would lose AWS routing" });
|
|
continue;
|
|
}
|
|
const annotations = await client.paginate(`/repos/${REPOSITORY}/check-runs/${job.id}/annotations?per_page=100`);
|
|
const readLog = (entry) => client.text(`/repos/${REPOSITORY}/actions/jobs/${entry.id}/logs`);
|
|
decision.jobs.push({ id: job.id, name: job.name, ...(await classifyJob(job, { annotations, readLog })) });
|
|
}
|
|
|
|
const real = decision.jobs.filter((job) => !job.lost);
|
|
if (real.length) return { ...decision, reason: `${real.length} of ${failed.length} failed jobs did not lose their runner` };
|
|
|
|
const stale = await staleReason(client, run);
|
|
if (stale) return { ...decision, reason: stale };
|
|
return { ...decision, rerun: true, reason: `all ${failed.length} failed jobs lost their runner` };
|
|
}
|
|
|
|
export function renderSummary(decision, { dryRun = false } = {}) {
|
|
const action = decision.rerun ? (dryRun ? "Would re-run failed jobs (dry run)" : "Re-ran failed jobs") : "No automatic re-run";
|
|
const lines = [
|
|
"### CI runner-loss re-run", "",
|
|
`**${action}** for run ${decision.runId} attempt ${decision.attempt}: ${decision.reason}.`, "",
|
|
];
|
|
if (decision.jobs.length) {
|
|
lines.push("| Job | Runner lost | Evidence |", "| --- | --- | --- |");
|
|
for (const job of decision.jobs) {
|
|
lines.push(`| ${escapeCell(job.name)} | ${job.lost ? "yes" : "no"} | ${escapeCell(job.evidence)} |`);
|
|
}
|
|
lines.push("");
|
|
}
|
|
return `${lines.join("\n")}\n`;
|
|
}
|
|
|
|
const escapeCell = (value) => String(value).replace(/\|/g, "\\|").replace(/[\r\n]+/g, " ");
|
|
|
|
export async function main({ env = process.env, argv = process.argv.slice(2), fetcher = fetch, log = console.log } = {}) {
|
|
const dryRun = argv.includes("--dry-run");
|
|
const argRunId = argv[argv.indexOf("--run-id") + 1];
|
|
const event = env.GITHUB_EVENT_PATH ? JSON.parse(readFileSync(env.GITHUB_EVENT_PATH, "utf8")) : undefined;
|
|
const runId = argv.includes("--run-id") ? argRunId : event?.workflow_run?.id;
|
|
if (!/^[1-9]\d*$/.test(String(runId))) throw new Error("Provide --run-id or a workflow_run event");
|
|
const client = createGithubClient({ token: env.GITHUB_TOKEN, fetcher });
|
|
|
|
// Read the run's current state rather than the event snapshot: its attempt
|
|
// number is above 1 if anyone re-ran it after this event was queued.
|
|
const run = await client.json(`/repos/${REPOSITORY}/actions/runs/${runId}`);
|
|
const decision = await decide(client, run);
|
|
if (decision.rerun && !dryRun) {
|
|
await client.post(`/repos/${REPOSITORY}/actions/runs/${run.id}/rerun-failed-jobs`);
|
|
}
|
|
const summary = renderSummary(decision, { dryRun });
|
|
log(summary);
|
|
if (env.GITHUB_STEP_SUMMARY) appendFileSync(env.GITHUB_STEP_SUMMARY, summary);
|
|
return decision;
|
|
}
|
|
|
|
if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
|
main().catch((error) => {
|
|
console.error(error.message);
|
|
process.exitCode = 1;
|
|
});
|
|
}
|