From c4f639ed1b01985593f1c70bb31f9cd1da441ecb Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 12 Nov 2026 22:31:50 -0600 Subject: [PATCH] fix(ci): re-run cancelled runs that lost a runner, and let runner-loss evidence win (#21291) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - PR CI runs about 50 jobs per run on AWS spot instances. AWS can reclaim a machine mid-job, and the job dies. > - `ci-runner-loss-rerun.yml` re-runs such a run once, automatically, when every failed job lost its runner. > - In a 17-hour sample, 42 runs lost a runner and were still the PR's current head. The automation rescued only 14. > - Two gaps caused most of the misses. A spot reclaim on RunsOn v3.2.3 usually ends the job, and the run, as `cancelled`; the workflow and script only look at `failure`. Also, the Runner support suite always carries `gateway_indexing_incomplete` error annotations from a passing test, so a lost runner there reads as a real failure. > - On-demand fallback for the fleet was turned off on 2026-11-13, so reclaims will happen more often. About 20% of runs are expected to lose at least one job. > - This pull request makes the re-run act on cancelled runs and jobs, but only with positive runner-loss evidence, and lets that evidence win over other error annotations. > - The benefit is fewer manual "Re-run failed jobs" clicks: an estimated 34 of 42 instead of 14 of 42 in the sample. ## Linked Issues or Issue Description No public issue exists. **What happened?** Run [43285064435](https://github.com/paperclipai/paperclip/actions/runs/43285064435) lost the runner of `ci / General tests (server (8/12))`. The job concluded `cancelled` (its log has `##[error]The runner has received a shutdown signal`). The run concluded `cancelled`, so the re-run workflow's `if:` skipped it. Run [43325855069](https://github.com/paperclipai/paperclip/actions/runs/43325855069) lost the runners of `ci / Runner support suite` and `ci / General tests (server (2/12))`. Both logs have the shutdown signal. The re-run bot ([decision log](https://github.com/paperclipai/paperclip/actions/runs/43327873358)) said "1 of 1 failed jobs did not lose their runner", with evidence `gateway_indexing_incomplete`. It ignored the cancelled job. It read the annotation as a real failure. **Expected behavior** Both runs get one automatic re-run of their failed jobs. **Steps to reproduce** The new tests replay trimmed snapshots of both runs (`.github/scripts/tests/fixtures/runner-loss-rerun/run-43285064435.json` and `run-43325855069.json`) through the script. In a 17-hour sample on 2026-11-12, 42 runs lost a runner and were still the PR's current head. Only 14 were re-run automatically. 16 stayed red because the run concluded `cancelled`. 4 stayed red because an error annotation in the lost job was read as a test failure. **Additional context** The `gateway_indexing_incomplete` annotation does not come from the shutdown. `scheduled-gateway-indexing.test.ts` prints `::error` lines while it passes, 84 seconds before the shutdown signal in this run. Attempt 2 of the same run passed with the same two failure annotations. So, before this change, a lost runner in the Runner support suite could never be re-run. ## What Changed - `ci-runner-loss-rerun.yml`: the job also runs when the PR run concluded `cancelled`. The trigger type, `permissions`, the repository, path, event and attempt-1 conditions do not change. - `runner-loss-rerun.mjs`, `eligibleRun`: accepts `completed/cancelled` as well as `completed/failure`. - `runner-loss-rerun.mjs`, `decide()`: classifies jobs that concluded `cancelled` as well as `failure`. It still skips the `ci / verify` and `ci / e2e` aggregates, and a failed or cancelled routing gate still blocks the re-run. - `decide()` now runs `staleReason` (open PR, current head, no newer run) before it reads any annotation or log, and again just before the re-run, as on master. A run cancelled by a new push has about 40 cancelled jobs; it now costs two API listings and no log reads. The final check closes the window where a push lands during the log reads: a re-run of the old run would join the PR's `cancel-in-progress` concurrency group (`pr-` in `pr-trusted.yml`) and cancel the new commit's CI. - Following CONTRIBUTING.md ("Do not submit a PR that includes code comments"), this PR adds no new code comments. The reasons for each rule are in this description and in `doc/DEVELOPING.md`. - `classifyJob`: runner-loss evidence wins. A lost-communication annotation means lost. A cancellation annotation plus a shutdown signal in the log means lost, even if the job also has other error annotations. A cancelled job without this evidence (for example, a person cancelled the run) counts as a real failure and blocks the re-run. - Fixtures: two new trimmed snapshots from the example runs (jobs with `id`, `name`, `status`, `conclusion`; failure annotations; `##[error]` log lines only). The two older fixtures get the real annotations and log error lines of their cancelled jobs. Those jobs were spot-reclaimed too (each log has the shutdown signal), but the old code ignored them. - `doc/DEVELOPING.md`: the runner-loss paragraph describes the new rules. ## Verification - Red first. Against the unchanged script, the new tests fail for the reasons above: 43285064435 gives `run is completed/cancelled, not completed/failure`; 43325855069 gives `1 of 1 failed jobs did not lose their runner` with evidence `gateway_indexing_incomplete`, the same decision the bot logged. - Green: `node --test .github/scripts/tests/runner-loss-rerun.test.mjs`: 25 of 25 pass. - Push during log reads: the PR head moves after the first log read. With the final freshness check removed, this test fails (the script re-runs); with it, there is no re-run and the reason is `no open pull request still has this run's head commit`. - Guard cases, all with no re-run and no POST: - Superseded run: 52 jobs cancelled, the PR head moved. The test asserts zero annotation and log reads. - Person cancelled the run: cancelled jobs with only "The operation was canceled." in the annotation and log, and some never-started jobs with no annotation. - Mixed: one cancelled lost job and one real test failure (`1 of 2 failed jobs did not lose their runner`). - Once only: a cancelled run on attempt 2 is refused after one API read. The workflow condition test still requires `run_attempt == 1`. - Cancelled routing gate: left for a person. - Policy job command: `node --test '.github/scripts/tests/*.test.mjs'`: 806 of 806 pass. - `pnpm verify`: exit 2 ("Global CI input changed: .github/scripts/runner-loss-rerun.mjs"). It selects no local Vitest files and asks for required CI coverage. I did not run `pnpm verify --full`; required CI on this PR supplies that coverage. - This workflow runs on `workflow_run` from the default branch. This PR's own CI cannot exercise the new behavior. The fixture tests are the evidence. I did not dry-run the script against live runs. ## Risks - A test that really fails, and then a spot reclaim of the same job a minute later, now gets one wasted re-run. Attempt 2 then fails for real. It cannot loop: the workflow `if:` and `eligibleRun` both accept attempt 1 only. - A run a person cancels on the current head now starts the re-run job. The job reads the annotations, and the log of each job with a cancellation annotation, and then does not re-run. This costs some API calls on a GitHub-hosted runner. - A concurrency cancel by a label event that does not request CI (the newer run is `skipped`) passes the head check. Its cancelled jobs have no runner-loss evidence, so it is not re-run. - Trust: no change to `permissions`, the trigger, the accepted repository, event, workflow path, attempt, or the routing-gate rule. A cancelled run reuses its attempt-1 gate output, the same as a failed run does today. - Estimated effect, from a 17-hour sample: 14 of 42 rescued before; 16 cancelled runs and 4 runs with error-annotation noise become eligible (about 34 of 42). The other 8 stay skipped on purpose (a real test failure in the same run, a person re-ran first, or the PR closed). > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), Claude Code agent. Model ID claude-opus-5-5, with tool use (shell, GitHub CLI) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge šŸ¤– Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 --- .github/scripts/runner-loss-rerun.mjs | 28 +- .../runner-loss-rerun/run-42271918832.json | 42 +- .../runner-loss-rerun/run-42277969418.json | 122 +++++- .../runner-loss-rerun/run-43285064435.json | 360 +++++++++++++++++ .../runner-loss-rerun/run-43325855069.json | 363 ++++++++++++++++++ .../scripts/tests/runner-loss-rerun.test.mjs | 170 +++++++- .github/workflows/ci-runner-loss-rerun.yml | 8 +- doc/DEVELOPING.md | 23 +- 8 files changed, 1073 insertions(+), 43 deletions(-) create mode 100644 .github/scripts/tests/fixtures/runner-loss-rerun/run-43285064435.json create mode 100644 .github/scripts/tests/fixtures/runner-loss-rerun/run-43325855069.json diff --git a/.github/scripts/runner-loss-rerun.mjs b/.github/scripts/runner-loss-rerun.mjs index 8a94ece7ddd..f581cdbbca5 100644 --- a/.github/scripts/runner-loss-rerun.mjs +++ b/.github/scripts/runner-loss-rerun.mjs @@ -3,7 +3,7 @@ * runner-loss-rerun.mjs * * Re-runs the failed jobs of a first PR CI attempt once, but only when every - * failed job failed because its runner was lost. Runs from + * failed or cancelled job ended because its runner was lost. Runs from * .github/workflows/ci-runner-loss-rerun.yml on workflow_run. It reads GitHub * API metadata only and never touches the PR's code or artifacts. * @@ -26,6 +26,7 @@ export const AGGREGATE_JOBS = new Set(["ci / verify", "ci / e2e"]); // ci-routing.yml checks the attempt's triggering actor. A bot-triggered attempt // would move the whole run to GitHub-hosted runners, so leave it to a human. export const GATE_JOB_PREFIX = "ci / gate / "; +export const RERUN_CONCLUSIONS = new Set(["failure", "cancelled"]); // A shutdown signal often reaches the check run only as this annotation; the // signal itself is logged as an error line in the job log. const CANCELLED_MESSAGE = "The operation was canceled."; @@ -80,20 +81,14 @@ export function logShowsRunnerLoss(log) { export async function classifyJob(job, { annotations, readLog }) { const failures = annotations.filter((annotation) => annotation.annotation_level === "failure"); const isRunnerLoss = (annotation) => RUNNER_LOSS_PATTERN.test(annotation.message ?? ""); - // A step can fail for a real reason before the runner is lost (for example a - // test failure, then a loss during an always() upload step). Any failure - // annotation other than runner loss or cancellation marks a real failure. - const real = failures.find((annotation) => !isRunnerLoss(annotation) && annotation.message !== CANCELLED_MESSAGE); - if (real) return { lost: false, evidence: firstLine(real.message) }; const match = failures.find(isRunnerLoss); if (match) return { lost: true, evidence: firstLine(match.message) }; - // Only cancellations are left: the shutdown signal, if any, is in the log. - if (failures.length > 0) { - if (logShowsRunnerLoss(await readLog(job))) { - return { lost: true, evidence: "job log: runner received a shutdown signal or lost communication" }; - } - return { lost: false, evidence: `${CANCELLED_MESSAGE} (no runner-loss error in the job log)` }; + if (failures.some((annotation) => annotation.message === CANCELLED_MESSAGE) && logShowsRunnerLoss(await readLog(job))) { + return { lost: true, evidence: "job log: runner received a shutdown signal or lost communication" }; } + const real = failures.find((annotation) => annotation.message !== CANCELLED_MESSAGE); + if (real) return { lost: false, evidence: firstLine(real.message) }; + if (failures.length > 0) return { lost: false, evidence: `${CANCELLED_MESSAGE} (no runner-loss error in the job log)` }; return { lost: false, evidence: "no failure annotation" }; } @@ -106,7 +101,9 @@ export function eligibleRun(run) { if (run.repository?.full_name !== REPOSITORY || run.repository?.id !== REPOSITORY_ID) return "run is not in the canonical repository"; if (run.path !== PR_WORKFLOW_PATH) return `workflow is ${run.path}, not ${PR_WORKFLOW_PATH}`; if (run.event !== "pull_request") return `event is ${run.event}, not pull_request`; - if (run.status !== "completed" || run.conclusion !== "failure") return `run is ${run.status}/${run.conclusion}, not completed/failure`; + if (run.status !== "completed" || !RERUN_CONCLUSIONS.has(run.conclusion)) { + return `run is ${run.status}/${run.conclusion}, not completed/failure or completed/cancelled`; + } // One automatic re-run only. A higher attempt also means someone re-ran it. if (run.run_attempt !== 1) return `run is on attempt ${run.run_attempt}; only attempt 1 is re-run automatically`; return undefined; @@ -137,9 +134,12 @@ export async function decide(client, run) { if (ineligible) return { ...decision, reason: ineligible }; const jobs = await client.paginate(`/repos/${REPOSITORY}/actions/runs/${run.id}/attempts/1/jobs?per_page=100`, "jobs"); - const failed = jobs.filter((job) => job.conclusion === "failure" && !AGGREGATE_JOBS.has(job.name)); + const failed = jobs.filter((job) => RERUN_CONCLUSIONS.has(job.conclusion) && !AGGREGATE_JOBS.has(job.name)); if (failed.length === 0) return { ...decision, reason: "no failed jobs other than the verify/e2e aggregates" }; + const staleBefore = await staleReason(client, run); + if (staleBefore) return { ...decision, reason: staleBefore }; + for (const job of failed) { if (job.name.startsWith(GATE_JOB_PREFIX)) { decision.jobs.push({ id: job.id, name: job.name, lost: false, evidence: "routing gate failed; a bot re-run would lose AWS routing" }); diff --git a/.github/scripts/tests/fixtures/runner-loss-rerun/run-42271918832.json b/.github/scripts/tests/fixtures/runner-loss-rerun/run-42271918832.json index 2dca8cc8caa..9be00e01728 100644 --- a/.github/scripts/tests/fixtures/runner-loss-rerun/run-42271918832.json +++ b/.github/scripts/tests/fixtures/runner-loss-rerun/run-42271918832.json @@ -1042,7 +1042,7 @@ "end_line": 123, "annotation_level": "failure", "title": "[paperclipai] src/__tests__/company-import-transfer.test.ts > resolveChunkedImportZip > takes the chunked path for a small zip whose entries inflate past the threshold", - "message": "Error: Test timed out in 5000ms.\nIf this is a long-running test, pass a timeout value as the last argument or configure it globally with \"testTimeout\".\n āÆ src/__tests__/company-import-transfer.test.ts:123:3\n\n", + "message": "Error: Test timed out in 5000ms.\nIf this is a long-running test, pass a timeout value as the last argument or configure it globally with \"testTimeout\".\n \u276f src/__tests__/company-import-transfer.test.ts:123:3\n\n", "raw_details": "" }, { @@ -1051,7 +1051,7 @@ "end_line": 280, "annotation_level": "failure", "title": "[paperclipai] src/__tests__/company-import-export-e2e.test.ts > paperclipai company import/export e2e", - "message": "Error: Timed out waiting for the child server recorded in /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/paperclip-home/instances/company-cli-e2e/runtime-info.json.\nstdout:\nUpdate available: 2026.1023.0 — run `paperclipai update`\nā”Œ paperclipai run \n│\n│ Home: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/paperclip-home\n│\n│ Instance: company-cli-e2e\n│\n│ Config: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/config.json\n│\nā—‡ Running doctor checks...\nUpdate available: 2026.1023.0 — run `paperclipai update`\n\nā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•— ā–ˆā–ˆā–ˆā–ˆā–ˆā•— ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•— ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•—ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•— ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•—ā–ˆā–ˆā•— ā–ˆā–ˆā•—ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•— \nā–ˆā–ˆā•”ā•ā•ā–ˆā–ˆā•—ā–ˆā–ˆā•”ā•ā•ā–ˆā–ˆā•—ā–ˆā–ˆā•”ā•ā•ā–ˆā–ˆā•—ā–ˆā–ˆā•”ā•ā•ā•ā•ā•ā–ˆā–ˆā•”ā•ā•ā–ˆā–ˆā•—ā–ˆā–ˆā•”ā•ā•ā•ā•ā•ā–ˆā–ˆā•‘ ā–ˆā–ˆā•‘ā–ˆā–ˆā•”ā•ā•ā–ˆā–ˆā•—\nā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•”ā•ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•‘ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•”ā•ā–ˆā–ˆā–ˆā–ˆā–ˆā•— ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•”ā•ā–ˆā–ˆā•‘ ā–ˆā–ˆā•‘ ā–ˆā–ˆā•‘ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•”ā•\nā–ˆā–ˆā•”ā•ā•ā•ā• ā–ˆā–ˆā•”ā•ā•ā–ˆā–ˆā•‘ā–ˆā–ˆā•”ā•ā•ā•ā• ā–ˆā–ˆā•”ā•ā•ā• ā–ˆā–ˆā•”ā•ā•ā–ˆā–ˆā•—ā–ˆā–ˆā•‘ ā–ˆā–ˆā•‘ ā–ˆā–ˆā•‘ā–ˆā–ˆā•”ā•ā•ā•ā• \nā–ˆā–ˆā•‘ ā–ˆā–ˆā•‘ ā–ˆā–ˆā•‘ā–ˆā–ˆā•‘ ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•—ā–ˆā–ˆā•‘ ā–ˆā–ˆā•‘ā•šā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•—ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā•—ā–ˆā–ˆā•‘ā–ˆā–ˆā•‘ \nā•šā•ā• ā•šā•ā• ā•šā•ā•ā•šā•ā• ā•šā•ā•ā•ā•ā•ā•ā•ā•šā•ā• ā•šā•ā• ā•šā•ā•ā•ā•ā•ā•ā•šā•ā•ā•ā•ā•ā•ā•ā•šā•ā•ā•šā•ā• \n ───────────────────────────────────────────────────────\n The app people use to manage AI agents for work\n\nā”Œ paperclip doctor \n│\n│ āœ“ Config file: Valid config at /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/config.json\n│\n│ āœ“ Deployment/auth mode: local_trusted mode is configured for loopback-only access\n│\n│ āœ— Agent JWT secret: PAPERCLIP_AGENT_JWT_SECRET missing from environment and /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/.env\n│\n│ Run with --repair to create /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/.env containing PAPERCLIP_AGENT_JWT_SECRET\n│\nā—† Repaired: Agent JWT secret\n│\n│ āœ“ Agent JWT secret: PAPERCLIP_AGENT_JWT_SECRET is set in environment\n│\n│ ! Secrets adapter: Secrets key file does not exist yet: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/secrets/master.key; strict secret mode is disabled for postgres deployment\n│\n│ Run with --repair to create a local encrypted secrets key file. Consider enabling secrets.strictMode\n│\nā—† Repaired: Secrets adapter\n│\n│ ! Secrets adapter: Local encrypted provider configured with key file /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/secrets/master.key; strict secret mode is disabled for postgres deployment\n│\n│ Consider enabling secrets.strictMode\n│\n│ āœ“ Storage: Local disk storage is writable: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/storage\n│\n│ āœ“ Database: PostgreSQL connection successful\n│\n│ āœ“ LLM provider: No LLM provider configured (optional)\n│\n│ āœ“ Log directory: Log directory is writable: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/logs\n│\n│ ! Server port: Port 37925 is already in use\n│\n│ Check what's using port 37925 with: lsof -i :37925\n│\n│ āœ“ Node.js runtime: Node.js 24.22.2\n│\n│ āœ“ Managed install: Not present (optional for npx, global npm, and source-checkout usage)\n│\n│ āœ“ Background service: No usable systemd user manager was detected (common in containers and WSL1). Use paperclipai run instead.\n│\nā—‡ Summary ───────────────╮\n│ │\n│ 10 passed, 2 warnings │\n│ │\nā”œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā•Æ\n│\nā”” All critical checks passed with some warnings.\n\n│\nā—‡ Start", + "message": "Error: Timed out waiting for the child server recorded in /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/paperclip-home/instances/company-cli-e2e/runtime-info.json.\nstdout:\nUpdate available: 2026.1023.0 \u2014 run `paperclipai update`\n\u250c paperclipai run \n\u2502\n\u2502 Home: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/paperclip-home\n\u2502\n\u2502 Instance: company-cli-e2e\n\u2502\n\u2502 Config: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/config.json\n\u2502\n\u25c7 Running doctor checks...\nUpdate available: 2026.1023.0 \u2014 run `paperclipai update`\n\n\u2588\u2588\u2588\u2588\u2588\u2588\u2557 \u2588\u2588\u2588\u2588\u2588\u2557 \u2588\u2588\u2588\u2588\u2588\u2588\u2557 \u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2557\u2588\u2588\u2588\u2588\u2588\u2588\u2557 \u2588\u2588\u2588\u2588\u2588\u2588\u2557\u2588\u2588\u2557 \u2588\u2588\u2557\u2588\u2588\u2588\u2588\u2588\u2588\u2557 \n\u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2557\u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2557\u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2557\u2588\u2588\u2554\u2550\u2550\u2550\u2550\u255d\u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2557\u2588\u2588\u2554\u2550\u2550\u2550\u2550\u255d\u2588\u2588\u2551 \u2588\u2588\u2551\u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2557\n\u2588\u2588\u2588\u2588\u2588\u2588\u2554\u255d\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2551\u2588\u2588\u2588\u2588\u2588\u2588\u2554\u255d\u2588\u2588\u2588\u2588\u2588\u2557 \u2588\u2588\u2588\u2588\u2588\u2588\u2554\u255d\u2588\u2588\u2551 \u2588\u2588\u2551 \u2588\u2588\u2551\u2588\u2588\u2588\u2588\u2588\u2588\u2554\u255d\n\u2588\u2588\u2554\u2550\u2550\u2550\u255d \u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2551\u2588\u2588\u2554\u2550\u2550\u2550\u255d \u2588\u2588\u2554\u2550\u2550\u255d \u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2557\u2588\u2588\u2551 \u2588\u2588\u2551 \u2588\u2588\u2551\u2588\u2588\u2554\u2550\u2550\u2550\u255d \n\u2588\u2588\u2551 \u2588\u2588\u2551 \u2588\u2588\u2551\u2588\u2588\u2551 \u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2557\u2588\u2588\u2551 \u2588\u2588\u2551\u255a\u2588\u2588\u2588\u2588\u2588\u2588\u2557\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2557\u2588\u2588\u2551\u2588\u2588\u2551 \n\u255a\u2550\u255d \u255a\u2550\u255d \u255a\u2550\u255d\u255a\u2550\u255d \u255a\u2550\u2550\u2550\u2550\u2550\u2550\u255d\u255a\u2550\u255d \u255a\u2550\u255d \u255a\u2550\u2550\u2550\u2550\u2550\u255d\u255a\u2550\u2550\u2550\u2550\u2550\u2550\u255d\u255a\u2550\u255d\u255a\u2550\u255d \n \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n The app people use to manage AI agents for work\n\n\u250c paperclip doctor \n\u2502\n\u2502 \u2713 Config file: Valid config at /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/config.json\n\u2502\n\u2502 \u2713 Deployment/auth mode: local_trusted mode is configured for loopback-only access\n\u2502\n\u2502 \u2717 Agent JWT secret: PAPERCLIP_AGENT_JWT_SECRET missing from environment and /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/.env\n\u2502\n\u2502 Run with --repair to create /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/.env containing PAPERCLIP_AGENT_JWT_SECRET\n\u2502\n\u25c6 Repaired: Agent JWT secret\n\u2502\n\u2502 \u2713 Agent JWT secret: PAPERCLIP_AGENT_JWT_SECRET is set in environment\n\u2502\n\u2502 ! Secrets adapter: Secrets key file does not exist yet: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/secrets/master.key; strict secret mode is disabled for postgres deployment\n\u2502\n\u2502 Run with --repair to create a local encrypted secrets key file. Consider enabling secrets.strictMode\n\u2502\n\u25c6 Repaired: Secrets adapter\n\u2502\n\u2502 ! Secrets adapter: Local encrypted provider configured with key file /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/secrets/master.key; strict secret mode is disabled for postgres deployment\n\u2502\n\u2502 Consider enabling secrets.strictMode\n\u2502\n\u2502 \u2713 Storage: Local disk storage is writable: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/storage\n\u2502\n\u2502 \u2713 Database: PostgreSQL connection successful\n\u2502\n\u2502 \u2713 LLM provider: No LLM provider configured (optional)\n\u2502\n\u2502 \u2713 Log directory: Log directory is writable: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/logs\n\u2502\n\u2502 ! Server port: Port 37925 is already in use\n\u2502\n\u2502 Check what's using port 37925 with: lsof -i :37925\n\u2502\n\u2502 \u2713 Node.js runtime: Node.js 24.22.2\n\u2502\n\u2502 \u2713 Managed install: Not present (optional for npx, global npm, and source-checkout usage)\n\u2502\n\u2502 \u2713 Background service: No usable systemd user manager was detected (common in containers and WSL1). Use paperclipai run instead.\n\u2502\n\u25c7 Summary \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 \u2502\n\u2502 10 passed, 2 warnings \u2502\n\u2502 \u2502\n\u251c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n\u2502\n\u2514 All critical checks passed with some warnings.\n\n\u2502\n\u25c7 Start", "raw_details": "" }, { @@ -1085,10 +1085,46 @@ "message": "Process completed with exit code 1.", "raw_details": "" } + ], + "128079212951": [ + { + "path": ".github", + "start_line": 130, + "end_line": 130, + "annotation_level": "failure", + "title": "", + "message": "The operation was canceled.", + "raw_details": "" + } + ], + "128079213109": [ + { + "path": ".github", + "start_line": 130, + "end_line": 130, + "annotation_level": "failure", + "title": "", + "message": "The operation was canceled.", + "raw_details": "" + } + ], + "128079213211": [ + { + "path": ".github", + "start_line": 133, + "end_line": 133, + "annotation_level": "failure", + "title": "", + "message": "The operation was canceled.", + "raw_details": "" + } ] }, "logs": { - "128078674151": "2026-11-06T18:42:31.0413375Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T18:42:43.5529901Z ##[error]The operation was canceled.\n2026-11-06T18:42:43.5656287Z Cleaning up orphan processes\n" + "128078674151": "2026-11-06T18:42:31.0413375Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T18:42:43.5529901Z ##[error]The operation was canceled.\n2026-11-06T18:42:43.5656287Z Cleaning up orphan processes\n", + "128079212951": "2026-11-06T18:42:41.1643719Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T18:42:53.6869870Z ##[error]The operation was canceled.\n", + "128079213109": "2026-11-06T18:42:35.0486149Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T18:42:47.5624672Z ##[error]The operation was canceled.\n", + "128079213211": "2026-11-06T18:42:38.0453932Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T18:42:50.5615051Z ##[error]The operation was canceled.\n" }, "total_count": 48 } diff --git a/.github/scripts/tests/fixtures/runner-loss-rerun/run-42277969418.json b/.github/scripts/tests/fixtures/runner-loss-rerun/run-42277969418.json index 77461a61e12..ee38d22d965 100644 --- a/.github/scripts/tests/fixtures/runner-loss-rerun/run-42277969418.json +++ b/.github/scripts/tests/fixtures/runner-loss-rerun/run-42277969418.json @@ -970,10 +970,130 @@ "message": "Process completed with exit code 1.", "raw_details": "" } + ], + "128101015916": [ + { + "path": ".github", + "start_line": 130, + "end_line": 130, + "annotation_level": "failure", + "title": "", + "message": "The operation was canceled.", + "raw_details": "" + } + ], + "128101015978": [ + { + "path": ".github", + "start_line": 130, + "end_line": 130, + "annotation_level": "failure", + "title": "", + "message": "The operation was canceled.", + "raw_details": "" + } + ], + "128101016153": [ + { + "path": ".github", + "start_line": 133, + "end_line": 133, + "annotation_level": "failure", + "title": "", + "message": "The operation was canceled.", + "raw_details": "" + } + ], + "128101016200": [ + { + "path": ".github", + "start_line": 133, + "end_line": 133, + "annotation_level": "failure", + "title": "", + "message": "The operation was canceled.", + "raw_details": "" + } + ], + "128101016304": [ + { + "path": ".github", + "start_line": 133, + "end_line": 133, + "annotation_level": "failure", + "title": "", + "message": "The operation was canceled.", + "raw_details": "" + } + ], + "128101016340": [ + { + "path": ".github", + "start_line": 133, + "end_line": 133, + "annotation_level": "failure", + "title": "", + "message": "The operation was canceled.", + "raw_details": "" + } + ], + "128101016402": [ + { + "path": ".github", + "start_line": 133, + "end_line": 133, + "annotation_level": "failure", + "title": "", + "message": "The operation was canceled.", + "raw_details": "" + } + ], + "128101016432": [ + { + "path": ".github", + "start_line": 133, + "end_line": 133, + "annotation_level": "failure", + "title": "", + "message": "The operation was canceled.", + "raw_details": "" + } + ], + "128101016453": [ + { + "path": ".github", + "start_line": 133, + "end_line": 133, + "annotation_level": "failure", + "title": "", + "message": "The operation was canceled.", + "raw_details": "" + } + ], + "128101016461": [ + { + "path": ".github", + "start_line": 133, + "end_line": 133, + "annotation_level": "failure", + "title": "", + "message": "The operation was canceled.", + "raw_details": "" + } ] }, "logs": { - "128101016102": "2026-11-06T19:24:29.1450769Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:29.1898916Z ##[error]The operation was canceled.\n2026-11-06T19:24:29.1977091Z Cleaning up orphan processes\n" + "128101016102": "2026-11-06T19:24:29.1450769Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:29.1898916Z ##[error]The operation was canceled.\n2026-11-06T19:24:29.1977091Z Cleaning up orphan processes\n", + "128101015916": "2026-11-06T19:24:29.1391073Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.6585509Z ##[error]The operation was canceled.\n", + "128101015978": "2026-11-06T19:24:29.1982327Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.7170843Z ##[error]The operation was canceled.\n", + "128101016153": "2026-11-06T19:24:29.1786229Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.6972420Z ##[error]The operation was canceled.\n", + "128101016200": "2026-11-06T19:24:29.1721198Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.6975527Z ##[error]The operation was canceled.\n", + "128101016304": "2026-11-06T19:24:29.1377822Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.6673266Z ##[error]The operation was canceled.\n", + "128101016340": "2026-11-06T19:24:29.1341317Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.6500655Z ##[error]The operation was canceled.\n", + "128101016402": "2026-11-06T19:24:29.1167345Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.6423716Z ##[error]The operation was canceled.\n", + "128101016432": "2026-11-06T19:24:29.0692803Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.5851488Z ##[error]The operation was canceled.\n", + "128101016453": "2026-11-06T19:24:29.2339015Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.7586244Z ##[error]The operation was canceled.\n", + "128101016461": "2026-11-06T19:24:29.1057916Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.6269036Z ##[error]The operation was canceled.\n" }, "total_count": 48 } diff --git a/.github/scripts/tests/fixtures/runner-loss-rerun/run-43285064435.json b/.github/scripts/tests/fixtures/runner-loss-rerun/run-43285064435.json new file mode 100644 index 00000000000..c0b9b6f7d69 --- /dev/null +++ b/.github/scripts/tests/fixtures/runner-loss-rerun/run-43285064435.json @@ -0,0 +1,360 @@ +{ + "source": "GET /repos/paperclipai/paperclip/actions/runs/43285064435/attempts/1 (+ jobs, annotations and job-log error lines), trimmed", + "run": { + "id": 43285064435, + "name": "PR", + "head_sha": "cd287efba2ff0c557f0c63fc78d5ad1d48f192e8", + "path": ".github/workflows/pr.yml", + "event": "pull_request", + "status": "completed", + "conclusion": "cancelled", + "html_url": "https://github.com/paperclipai/paperclip/actions/runs/43285064435", + "run_attempt": 1, + "repository": { + "id": 1170821064, + "full_name": "paperclipai/paperclip" + }, + "head_repository": { + "id": 1170821064, + "full_name": "paperclipai/paperclip" + } + }, + "jobs": [ + { + "id": 131416164450, + "name": "ci / gate / Select trusted runner", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416492147, + "name": "ci / policy", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416492206, + "name": "ci / Typecheck + Release Registry", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416492248, + "name": "ci / Verify Paperclip Runner (static checks)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416492288, + "name": "ci / Runner support suite", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416492337, + "name": "ci / Verify Paperclip Runner (rust)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416492390, + "name": "ci / Build", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416492402, + "name": "ci / Canary Dry Run", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416492420, + "name": "ci / Verify Paperclip Runner (vitest package (3/3))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416492421, + "name": "ci / Sandbox provider tests", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416492424, + "name": "ci / Docker context integrity", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416492430, + "name": "ci / Verify Paperclip Runner (vitest native server)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416492442, + "name": "ci / Verify Paperclip Runner (vitest package (2/3))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416492489, + "name": "ci / Verify Paperclip Runner (vitest package (1/3))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416492538, + "name": "ci / e2e shard (5/8)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416492565, + "name": "ci / e2e shard (1/8)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416492568, + "name": "ci / e2e shard (2/8)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416492582, + "name": "ci / e2e shard (6/8)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416492588, + "name": "ci / e2e shard (7/8)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416492592, + "name": "ci / e2e shard (4/8)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416492708, + "name": "ci / e2e shard (3/8)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416493348, + "name": "ci / e2e shard (8/8)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416493582, + "name": "ci / Verify serialized server suites (3/9)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416493601, + "name": "ci / Verify serialized server suites (4/9)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416493630, + "name": "ci / Verify serialized server suites (2/9)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416493633, + "name": "ci / Verify serialized server suites (1/9)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416493669, + "name": "ci / Verify serialized server suites (5/9)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416493759, + "name": "ci / Verify serialized server suites (7/9)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416493766, + "name": "ci / Verify serialized server suites (9/9)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416493829, + "name": "ci / Verify serialized server suites (8/9)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416493947, + "name": "ci / General tests (server (5/12))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416494052, + "name": "ci / General tests (server (1/12))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416494077, + "name": "ci / General tests (server (3/12))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416494102, + "name": "ci / General tests (server (7/12))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416494186, + "name": "ci / General tests (server (4/12))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416494194, + "name": "ci / General tests (workspaces-c)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416494201, + "name": "ci / General tests (chat (3/3))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416494251, + "name": "ci / General tests (server (12/12))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416494281, + "name": "ci / General tests (server (2/12))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416494286, + "name": "ci / General tests (chat (1/3))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416494333, + "name": "ci / General tests (workspaces-a (1/2))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416494339, + "name": "ci / General tests (server (10/12))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416494340, + "name": "ci / General tests (server (9/12))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416494376, + "name": "ci / General tests (chat (2/3))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416494446, + "name": "ci / General tests (server (6/12))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416494459, + "name": "ci / General tests (server (8/12))", + "status": "completed", + "conclusion": "cancelled" + }, + { + "id": 131416494489, + "name": "ci / General tests (workspaces-b)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416494538, + "name": "ci / Verify serialized server suites (6/9)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416494546, + "name": "ci / General tests (server (11/12))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416494552, + "name": "ci / General tests (workspaces-a (2/2))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131416494939, + "name": "ci / Select Vitest files", + "status": "completed", + "conclusion": "skipped" + }, + { + "id": 131421415244, + "name": "ci / e2e", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131424041299, + "name": "ci / verify", + "status": "completed", + "conclusion": "failure" + } + ], + "annotations": { + "131416494459": [ + { + "annotation_level": "failure", + "message": "The operation was canceled." + } + ], + "131424041299": [ + { + "annotation_level": "failure", + "message": "Process completed with exit code 1." + } + ] + }, + "logs": { + "131416494459": "2026-11-12T19:03:20.1001059Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-12T19:03:32.6527460Z ##[error]The operation was canceled.\n" + }, + "total_count": 53 +} diff --git a/.github/scripts/tests/fixtures/runner-loss-rerun/run-43325855069.json b/.github/scripts/tests/fixtures/runner-loss-rerun/run-43325855069.json new file mode 100644 index 00000000000..314dc2cd82d --- /dev/null +++ b/.github/scripts/tests/fixtures/runner-loss-rerun/run-43325855069.json @@ -0,0 +1,363 @@ +{ + "source": "GET /repos/paperclipai/paperclip/actions/runs/43325855069/attempts/1 (+ jobs, annotations and job-log error lines), trimmed", + "run": { + "id": 43325855069, + "name": "PR", + "head_sha": "4b4f6b2ad0fac4ede9f4e20a56e3c93fac5577d5", + "path": ".github/workflows/pr.yml", + "event": "pull_request", + "status": "completed", + "conclusion": "failure", + "html_url": "https://github.com/paperclipai/paperclip/actions/runs/43325855069", + "run_attempt": 1, + "repository": { + "id": 1170821064, + "full_name": "paperclipai/paperclip" + }, + "head_repository": { + "id": 1170821064, + "full_name": "paperclipai/paperclip" + } + }, + "jobs": [ + { + "id": 131557987038, + "name": "ci / gate / Select trusted runner", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558292300, + "name": "ci / Docker context integrity", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558292349, + "name": "ci / Canary Dry Run", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558292381, + "name": "ci / Runner support suite", + "status": "completed", + "conclusion": "failure" + }, + { + "id": 131558292405, + "name": "ci / Typecheck + Release Registry", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558292466, + "name": "ci / policy", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558292573, + "name": "ci / Build", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558292627, + "name": "ci / Verify Paperclip Runner (vitest package (1/3))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558292664, + "name": "ci / Verify Paperclip Runner (static checks)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558292669, + "name": "ci / Verify Paperclip Runner (vitest package (2/3))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558292679, + "name": "ci / e2e shard (8/8)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558292686, + "name": "ci / Verify Paperclip Runner (vitest native server)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558292707, + "name": "ci / Verify Paperclip Runner (vitest package (3/3))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558292731, + "name": "ci / e2e shard (5/8)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558292738, + "name": "ci / e2e shard (2/8)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558292751, + "name": "ci / e2e shard (1/8)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558292779, + "name": "ci / e2e shard (4/8)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558292841, + "name": "ci / e2e shard (3/8)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558292887, + "name": "ci / e2e shard (7/8)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293043, + "name": "ci / e2e shard (6/8)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293328, + "name": "ci / Verify serialized server suites (4/9)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293330, + "name": "ci / Verify serialized server suites (2/9)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293380, + "name": "ci / Verify serialized server suites (7/9)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293389, + "name": "ci / Verify serialized server suites (3/9)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293399, + "name": "ci / Verify serialized server suites (8/9)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293420, + "name": "ci / Verify serialized server suites (1/9)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293433, + "name": "ci / Verify serialized server suites (5/9)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293450, + "name": "ci / Verify serialized server suites (6/9)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293683, + "name": "ci / Verify serialized server suites (9/9)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293690, + "name": "ci / General tests (server (12/12))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293701, + "name": "ci / General tests (server (11/12))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293710, + "name": "ci / General tests (server (8/12))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293784, + "name": "ci / General tests (server (3/12))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293818, + "name": "ci / General tests (server (2/12))", + "status": "completed", + "conclusion": "cancelled" + }, + { + "id": 131558293819, + "name": "ci / General tests (workspaces-a (1/2))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293825, + "name": "ci / General tests (server (4/12))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293835, + "name": "ci / General tests (server (5/12))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293844, + "name": "ci / General tests (server (9/12))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293850, + "name": "ci / General tests (workspaces-a (2/2))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293852, + "name": "ci / General tests (server (1/12))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293858, + "name": "ci / General tests (chat (2/3))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293865, + "name": "ci / General tests (server (10/12))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293875, + "name": "ci / General tests (workspaces-b)", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293940, + "name": "ci / General tests (server (6/12))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558293992, + "name": "ci / General tests (chat (3/3))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558294007, + "name": "ci / General tests (chat (1/3))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558294057, + "name": "ci / General tests (server (7/12))", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131558294396, + "name": "ci / Sandbox provider tests", + "status": "completed", + "conclusion": "skipped" + }, + { + "id": 131558294604, + "name": "ci / Select Vitest files", + "status": "completed", + "conclusion": "skipped" + }, + { + "id": 131562309925, + "name": "ci / e2e", + "status": "completed", + "conclusion": "success" + }, + { + "id": 131564602271, + "name": "ci / verify", + "status": "completed", + "conclusion": "failure" + } + ], + "annotations": { + "131558292381": [ + { + "annotation_level": "failure", + "message": "The operation was canceled." + }, + { + "annotation_level": "failure", + "message": "gateway_indexing_incomplete" + }, + { + "annotation_level": "failure", + "message": "gateway_indexing_incomplete" + } + ], + "131558293818": [ + { + "annotation_level": "failure", + "message": "The operation was canceled." + } + ], + "131564602271": [ + { + "annotation_level": "failure", + "message": "Process completed with exit code 1." + } + ] + }, + "logs": { + "131558292381": "2026-11-12T23:13:37.4339910Z ##[error]gateway_indexing_incomplete\n2026-11-12T23:13:37.4368921Z ##[error]gateway_indexing_incomplete\n2026-11-12T23:15:01.1805404Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-12T23:15:13.7401251Z ##[error]The operation was canceled.\n", + "131558293818": "2026-11-12T23:15:36.1981336Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-12T23:15:48.7226826Z ##[error]The operation was canceled.\n" + }, + "total_count": 51 +} diff --git a/.github/scripts/tests/runner-loss-rerun.test.mjs b/.github/scripts/tests/runner-loss-rerun.test.mjs index bdc89f1d986..819f160881e 100644 --- a/.github/scripts/tests/runner-loss-rerun.test.mjs +++ b/.github/scripts/tests/runner-loss-rerun.test.mjs @@ -13,10 +13,13 @@ import { const fixture = (runId) => JSON.parse(readFileSync( new URL(`./fixtures/runner-loss-rerun/run-${runId}.json`, import.meta.url), "utf8", )); -const MIXED = 42271918832; // six lost runners, one shutdown in the log, one real test timeout +const MIXED = 42271918832; // six lost runners, four shutdowns in the log, one real test timeout const REAL = 42264785415; // a real assertion failure const ALL_LOST = 42272083862; // one lost runner -const LOG_ONLY = 42277969418; // shutdown signal visible only in the job log +const LOG_ONLY = 42277969418; // shutdown signals visible only in the job logs +const CANCELLED_RUN = 43285064435; +const NOISY_LOST = 43325855069; +const CANCELLED_MESSAGE = "The operation was canceled."; const workflow = readFileSync(new URL("../../workflows/ci-runner-loss-rerun.yml", import.meta.url), "utf8"); const json = (body, headers = {}) => new Response(JSON.stringify(body), { @@ -58,7 +61,7 @@ function fakeGithub(data, { if (match) return json(data.annotations[match[1]] ?? []); match = route.match(/^\/actions\/jobs\/(\d+)\/logs$/); if (match) return new Response(data.logs[match[1]] ?? "", { status: 200 }); - if (route === `/commits/${run.head_sha}/pulls`) return json(pulls ?? [openPr]); + if (route === `/commits/${run.head_sha}/pulls`) return json((typeof pulls === "function" ? pulls(calls) : pulls) ?? [openPr]); if (route === "/actions/workflows/pr.yml/runs") { const runs = prRuns ?? [{ id: run.id, conclusion: "failure" }]; return json({ total_count: runs.length, workflow_runs: runs }); @@ -106,19 +109,131 @@ test("a shutdown signal found only in the job log counts as runner loss", async assert.deepEqual(data.annotations["128101016102"].map((annotation) => annotation.message), ["The operation was canceled."]); const { decision, calls } = await runMain(data); assert.equal(decision.rerun, true); - assert.match(decision.jobs[0].evidence, /job log/); + assert.equal(decision.jobs.length, 11); + assert.ok(decision.jobs.every((job) => job.lost && /job log/.test(job.evidence))); assert.ok(calls.some((call) => call.path.endsWith("/actions/jobs/128101016102/logs"))); assert.equal(posted(calls).length, 1); }); +test("a cancelled run whose cancelled job lost its runner is re-run once", async () => { + const data = fixture(CANCELLED_RUN); + assert.equal(data.run.conclusion, "cancelled"); + const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" }); + assert.equal(decision.rerun, true); + assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [["ci / General tests (server (8/12))", true]]); + assert.match(decision.jobs[0].evidence, /job log/); + assert.deepEqual(posted(calls).map((call) => call.path), + [`/repos/paperclipai/paperclip/actions/runs/${CANCELLED_RUN}/rerun-failed-jobs`]); +}); + +test("runner loss in the log wins over error annotations the workload printed", async () => { + const data = fixture(NOISY_LOST); + const suite = data.jobs.find((job) => job.name === "ci / Runner support suite"); + assert.ok(data.annotations[suite.id].some((annotation) => annotation.message === "gateway_indexing_incomplete")); + const { decision, calls } = await runMain(data); + assert.equal(decision.rerun, true); + assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [ + ["ci / Runner support suite", true], + ["ci / General tests (server (2/12))", true], + ]); + assert.equal(posted(calls).length, 1); +}); + +function cancelEverything(data, { log = `2026-11-12T19:03:32.0000000Z ##[error]${CANCELLED_MESSAGE}\n` } = {}) { + data.annotations = {}; + data.logs = {}; + for (const [index, job] of data.jobs.entries()) { + if (job.name.startsWith("ci / gate / ")) continue; + job.conclusion = "cancelled"; + if (index % 4 === 3) continue; + data.annotations[job.id] = [{ annotation_level: "failure", message: CANCELLED_MESSAGE }]; + data.logs[job.id] = log; + } + return data; +} + +test("a run superseded by a new push is not re-run, and no job is classified", async () => { + const data = cancelEverything(fixture(CANCELLED_RUN)); + assert.ok(data.jobs.filter((job) => job.conclusion === "cancelled").length >= 40); + const { decision, calls } = await runMain(data, { + pulls: [{ state: "open", base: { repo: { id: 1170821064 } }, head: { sha: "0".repeat(40), repo: { id: 1170821064 } } }], + }, { conclusion: "cancelled" }); + assert.equal(decision.rerun, false); + assert.match(decision.reason, /no open pull request/); + assert.equal(calls.some((call) => call.path.includes("/check-runs/") || call.path.endsWith("/logs")), false); + assert.equal(posted(calls).length, 0); +}); + +test("a push while job logs are read stops the re-run", async () => { + const data = fixture(CANCELLED_RUN); + const moved = [{ state: "open", base: { repo: { id: 1170821064 } }, head: { sha: "0".repeat(40), repo: { id: 1170821064 } } }]; + const { decision, calls } = await runMain(data, { + pulls: (seen) => (seen.some((call) => call.path.endsWith("/logs")) ? moved : undefined), + }, { conclusion: "cancelled" }); + assert.equal(decision.rerun, false); + assert.match(decision.reason, /no open pull request/); + assert.equal(decision.jobs[0].lost, true); + assert.equal(calls.filter((call) => call.path.endsWith("/pulls")).length, 2); + assert.equal(posted(calls).length, 0); +}); + +test("a run a person cancelled is not re-run", async () => { + const data = cancelEverything(fixture(CANCELLED_RUN)); + const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" }); + assert.equal(decision.rerun, false); + assert.ok(decision.jobs.length >= 40); + assert.equal(decision.jobs.some((job) => job.lost), false); + assert.match(decision.reason, /failed jobs did not lose their runner/); + assert.equal(posted(calls).length, 0); +}); + +test("a cancelled run with one lost job and one real test failure is not re-run", async () => { + const data = fixture(CANCELLED_RUN); + const failing = data.jobs.find((job) => job.name === "ci / General tests (server (1/12))"); + failing.conclusion = "failure"; + data.annotations[failing.id] = [ + { annotation_level: "failure", message: "AssertionError: expected 'queued' to be 'running'" }, + { annotation_level: "failure", message: "Process completed with exit code 1." }, + ]; + const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" }); + assert.equal(decision.rerun, false); + assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [ + ["ci / General tests (server (1/12))", false], + ["ci / General tests (server (8/12))", true], + ]); + assert.match(decision.reason, /1 of 2 failed jobs did not lose their runner/); + assert.equal(posted(calls).length, 0); +}); + +test("a cancelled run is re-run at most once", async () => { + const { decision, calls } = await runMain(fixture(CANCELLED_RUN), { + mutateRun: (run) => ({ ...run, run_attempt: 2 }), + }, { conclusion: "cancelled" }); + assert.equal(decision.rerun, false); + assert.match(decision.reason, /attempt 2/); + assert.deepEqual(calls.map((call) => call.path), [`/repos/paperclipai/paperclip/actions/runs/${CANCELLED_RUN}`]); +}); + +test("a cancelled routing gate is left for a human re-run", async () => { + const data = fixture(CANCELLED_RUN); + const gate = data.jobs.find((job) => job.name === "ci / gate / Select trusted runner"); + gate.conclusion = "cancelled"; + data.annotations[gate.id] = [{ annotation_level: "failure", message: CANCELLED_MESSAGE }]; + data.logs[gate.id] = data.logs[Object.keys(data.logs)[0]]; + const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" }); + assert.equal(decision.rerun, false); + assert.match(decision.jobs.find((job) => job.id === gate.id).evidence, /routing gate/); + assert.equal(posted(calls).length, 0); +}); + test("mixed runner loss and a real test failure: no re-run", async () => { const { decision, calls, summary } = await runMain(fixture(MIXED)); assert.equal(decision.rerun, false); - assert.equal(decision.jobs.length, 8); + assert.equal(decision.jobs.length, 11); assert.deepEqual(decision.jobs.filter((job) => !job.lost).map((job) => job.name), ["ci / General tests (workspaces-a (1/2))"]); assert.equal(decision.jobs.find((job) => job.name === "ci / Verify Paperclip Runner (vitest package)").lost, true); assert.equal(posted(calls).length, 0); - assert.match(decision.reason, /1 of 8 failed jobs did not lose their runner/); + assert.match(decision.reason, /1 of 11 failed jobs did not lose their runner/); assert.match(summary, /\*\*No automatic re-run\*\*/); }); @@ -209,7 +324,7 @@ test("job listing follows Link pagination across pages", async () => { const pages = github.calls.filter((call) => call.path.endsWith("/attempts/1/jobs")); assert.deepEqual(pages.map((call) => new URLSearchParams(call.query).get("page")), [null, "2", "3"]); // Failed jobs on pages 2 and 3 are all classified. - assert.equal(decision.jobs.length, 8); + assert.equal(decision.jobs.length, 11); assert.equal(decision.rerun, false); }); @@ -236,9 +351,20 @@ test("classification is conservative about cancelled jobs", async () => { // A job timeout also cancels the step, but adds its own failure annotation. const timedOut = await classifyJob({ id: 1 }, { annotations: [{ annotation_level: "failure", message: "The job has exceeded the maximum execution time of 30m0s" }, ...cancelled], - readLog: async () => { throw new Error("log must not be read"); }, + readLog: async () => "##[error]The job has exceeded the maximum execution time of 30m0s\n##[error]The operation was canceled.\n", }); assert.equal(timedOut.lost, false); + assert.match(timedOut.evidence, /maximum execution time/); + const notStarted = await classifyJob({ id: 1, conclusion: "cancelled" }, { + annotations: [], + readLog: async () => { throw new Error("log must not be read"); }, + }); + assert.equal(notStarted.lost, false); + const exitOnly = await classifyJob({ id: 1 }, { + annotations: [{ annotation_level: "failure", message: "Process completed with exit code 1." }], + readLog: async () => { throw new Error("log must not be read"); }, + }); + assert.equal(exitOnly.lost, false); // Notices and warnings are not failure evidence. const warning = await classifyJob({ id: 1 }, { annotations: [{ annotation_level: "warning", message: "The self-hosted runner lost communication with the server." }], @@ -250,7 +376,7 @@ test("classification is conservative about cancelled jobs", async () => { assert.equal(logShowsRunnerLoss(fixture(MIXED).logs["128078674151"]), true); }); -test("a real failure before the runner was lost is not counted as runner loss", async () => { +test("runner-loss evidence wins over other failure annotations in the same job", async () => { const lostMessage = "The self-hosted runner lost communication with the server. Verify the machine is running."; const noLog = async () => { throw new Error("log must not be read"); }; // A test fails, then the runner is lost during an always() upload step. @@ -262,8 +388,8 @@ test("a real failure before the runner was lost is not counted as runner loss", ], readLog: noLog, }); - assert.equal(testThenLost.lost, false); - assert.match(testThenLost.evidence, /Test timed out/); + assert.equal(testThenLost.lost, true); + assert.match(testThenLost.evidence, /lost communication/); // The order of the annotations does not matter. const lostThenExit = await classifyJob({ id: 1 }, { annotations: [ @@ -272,7 +398,22 @@ test("a real failure before the runner was lost is not counted as runner loss", ], readLog: noLog, }); - assert.equal(lostThenExit.lost, false); + assert.equal(lostThenExit.lost, true); + const noisy = [ + { annotation_level: "failure", message: "gateway_indexing_incomplete" }, + { annotation_level: "failure", message: CANCELLED_MESSAGE }, + ]; + const noisyLost = await classifyJob({ id: 1 }, { + annotations: noisy, + readLog: async () => "##[error]gateway_indexing_incomplete\n##[error]The runner has received a shutdown signal.\n", + }); + assert.equal(noisyLost.lost, true); + const noisyCancelled = await classifyJob({ id: 1 }, { + annotations: noisy, + readLog: async () => "##[error]gateway_indexing_incomplete\n##[error]The operation was canceled.\n", + }); + assert.equal(noisyCancelled.lost, false); + assert.match(noisyCancelled.evidence, /gateway_indexing_incomplete/); // Runner loss with a cancellation only is still runner loss. const lostAndCancelled = await classifyJob({ id: 1 }, { annotations: [ @@ -292,14 +433,15 @@ test("summary cells cannot break the table", () => { assert.match(summary, /\| ci \/ a\\\|b \| no \| line one line \\\| two \|/); }); -test("workflow acts only on failed first attempts of the canonical PR workflow", () => { +test("workflow acts only on failed or cancelled first attempts of the canonical PR workflow", () => { const condition = workflow.match(/^ {4}if: >-\n((?: {6}.+\n)+)/m)?.[1].replace(/\s+/g, " ").trim(); assert.equal(condition, [ "github.repository == 'paperclipai/paperclip' &&", "github.repository_id == '1170821064' &&", "github.event.workflow_run.event == 'pull_request' &&", "github.event.workflow_run.path == '.github/workflows/pr.yml' &&", - "github.event.workflow_run.conclusion == 'failure' &&", + "(github.event.workflow_run.conclusion == 'failure' ||", + "github.event.workflow_run.conclusion == 'cancelled') &&", "github.event.workflow_run.run_attempt == 1", ].join(" ")); assert.match(workflow, /^on:\n {2}workflow_run:\n {4}workflows: \[PR\]\n {4}types: \[completed\]$/m); diff --git a/.github/workflows/ci-runner-loss-rerun.yml b/.github/workflows/ci-runner-loss-rerun.yml index bc718fdb95b..4278198db61 100644 --- a/.github/workflows/ci-runner-loss-rerun.yml +++ b/.github/workflows/ci-runner-loss-rerun.yml @@ -1,7 +1,8 @@ name: CI runner-loss rerun -# Self-hosted AWS runners are sometimes lost mid-job. When every failed job of -# a first PR CI attempt lost its runner, re-run the failed jobs once. This is +# Self-hosted AWS runners are sometimes lost mid-job. When every failed or +# cancelled job of a first PR CI attempt lost its runner, re-run the failed +# jobs once. A spot reclaim often concludes the run `cancelled`. This is # what a maintainer's "Re-run failed jobs" click does; github-actions[bot] # becomes attempt 2's triggering actor. See "Trusted PR Workflow" in # doc/DEVELOPING.md. @@ -31,7 +32,8 @@ jobs: github.repository_id == '1170821064' && github.event.workflow_run.event == 'pull_request' && github.event.workflow_run.path == '.github/workflows/pr.yml' && - github.event.workflow_run.conclusion == 'failure' && + (github.event.workflow_run.conclusion == 'failure' || + github.event.workflow_run.conclusion == 'cancelled') && github.event.workflow_run.run_attempt == 1 # GitHub-hosted on purpose: this job must not depend on the fleet it repairs. runs-on: ubuntu-latest diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 95a99fa0132..7bb129325df 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -187,14 +187,21 @@ AWS fleet. A stale admission roster can allocate only the short metadata gate; the fresh CODEOWNERS check still denies PR execution for a removed owner. AWS runners are sometimes lost mid-job. `ci-runner-loss-rerun.yml` re-runs the -failed jobs of a PR's first CI attempt once when every failed job lost its -runner. A lost runner is recognized by the check-run annotation "lost -communication with the server" or "received a shutdown signal". When the only -annotation is "The operation was canceled.", the job log is checked for the -same messages. The `verify` and `e2e` aggregates are not counted. Any other -failure, a failed routing gate, or a PR whose head has moved prevents the -automatic re-run. The workflow job summary lists each failed job and its -evidence. Like a maintainer's "Re-run failed jobs", the re-run reuses the +failed jobs of a PR's first CI attempt once when every failed or cancelled job +lost its runner. A spot reclaim often ends the job, and the run, as +`cancelled`. A lost runner is recognized by the check-run annotation "lost +communication with the server" or "received a shutdown signal". When a job has +the annotation "The operation was canceled.", the job log is checked for the +same messages. This evidence counts even if the job also has other error +annotations, so a test that really failed just before a reclaim gets one +wasted re-run. A cancelled job without this evidence, such as a job a person +cancelled, counts as a real failure. The `verify` and `e2e` aggregates are not +counted. Any other failure, a failed or cancelled routing gate, or a PR whose +head has moved prevents the automatic re-run. The head check runs before any +job is classified, so a run cancelled by a newer push costs no log reads. It +runs again just before the re-run, because a re-run of an old run joins the +PR's `cancel-in-progress` concurrency group and would cancel the newer CI. The +workflow job summary lists each failed or cancelled job and its evidence. Like a maintainer's "Re-run failed jobs", the re-run reuses the gate's successful outputs, including its AWS admission. Attempt 2's triggering actor is `github-actions[bot]`, so `profile-ci` does not export that attempt. Later attempts are never re-run automatically.