1 Commits

Include renames

Author	SHA1	Message	Date
Nicky LeachandClaude Opus 5.5	f2299cb00f	feat(ci): re-run failed PR jobs once when every failure was a lost runner (#19640) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors wait for PR CI before merge, so CI wall-clock time sets the pace of work > - PR CI runs on self-hosted AWS runners, and these runners are sometimes lost mid-job > - On 2026-11-06, 21 of 51 failed jobs on commits that later passed were runner loss. The losses came in fleet-wide bursts, for example 10 jobs on 10 instances in 51 seconds > - After each loss, a person must notice the failure and click "Re-run failed jobs" > - This pull request adds a workflow that re-runs the failed jobs once, only when every failure was a lost runner > - The benefit is fewer manual re-runs and a shorter wait for CI to pass ## Linked Issues or Issue Description **What existing behavior does this improve?** The recovery of PR CI runs that failed only because a self-hosted runner was lost. **Subsystem affected** CI: the trusted PR workflow (`pr.yml` → `pr-trusted.yml`). **Current behavior** When an AWS runner is lost, the job fails with "The self-hosted runner lost communication with the server" or "The runner has received a shutdown signal". The PR stays red until a person re-runs the failed jobs. **Proposed behavior** When the `PR` workflow completes with a failure on attempt 1, a new workflow examines the failed jobs. If every failed job lost its runner, it re-runs the failed jobs one time. If any job failed for another reason, it does nothing. **Reason and benefit** A runner loss is not a code problem. The re-run starts without waiting for a person to notice the failure. **Breaking changes** None. ## What Changed - Add `.github/workflows/ci-runner-loss-rerun.yml`. It triggers on `workflow_run` for `PR`, and it acts only on attempt 1 failures of `pull_request` runs in this repository. It runs on `ubuntu-latest`, so it does not depend on the AWS fleet. - Add `.github/scripts/runner-loss-rerun.mjs`. The script: - lists the attempt-1 jobs and ignores the aggregate jobs `ci / verify` and `ci / e2e`; - finds runner loss in the check-run annotations. If the only annotation is "The operation was canceled.", it reads the `##[error]` lines in the job log; - calls `rerun-failed-jobs` only if every remaining failed job lost its runner; - does not re-run when the gate job failed, the PR head moved, the PR is closed, or a newer run exists for the same commit; - writes the decision and a job table to the step summary. A `--dry-run` mode never posts. - Add 16 tests, with trimmed fixtures from four real runs. - Add a paragraph about the workflow to the "Trusted PR Workflow" section of `doc/DEVELOPING.md`. ## Verification - `node --test .github/scripts/tests/runner-loss-rerun.test.mjs`: 16/16 pass. The tests cover: - re-run when all jobs were lost, including the case where the signal is only in the log; - no re-run for mixed failures, real failures, only aggregate failures, attempt 2, a failed gate, or a stale run; - pagination, incomplete listings, a cross-host link that is refused, and the workflow guards. - `node --test '.github/scripts/tests/*.test.mjs'`: 760/760 pass. - A read-only dry run against the real GitHub API for six recent runs classified all six correctly. - The workflow can run on GitHub only after it is on master. Reviewers can make sure that the first runner-loss failure after the merge triggers one re-run. ## Risks - **Trust:** a re-run of failed jobs uses the outputs of the gate job that succeeded on attempt 1, including its AWS runner admission. A person who clicks "Re-run failed jobs" gets the same result. The only difference is that `github-actions[bot]` triggers attempt 2. The workflow never checks out or runs PR code. - **Failed gate:** if the gate job itself failed, the script does nothing. A re-run by the bot would put all jobs on GitHub-hosted runners. - **Profiling:** `scripts/ci-profile/export.mjs` exports only attempts triggered by authorized users. The bot-triggered attempt 2 is not exported to Honeycomb. Attempt 1 is still exported. - **Events:** GitHub may not create `workflow_run` events for attempts that `GITHUB_TOKEN` started. Then other `workflow_run` consumers do not react to attempt 2. Required checks still update. - **Permissions:** `actions: write`, `checks: read`, `contents: read` and `pull-requests: read`. The last one lets the script confirm that the run still belongs to the open PR's current head. - **Cancelled jobs:** a runner loss where the lost jobs end as `cancelled` and only the aggregate jobs show `failure` is not re-run. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. A Claude sub-agent wrote the change, and the parent session reviewed it. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 15:05:42 -08:00