mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(ci): re-run cancelled runs that lost a runner, and let runner-loss evidence win (#21291)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - PR CI runs about 50 jobs per run on AWS spot instances. AWS can reclaim a machine mid-job, and the job dies. > - `ci-runner-loss-rerun.yml` re-runs such a run once, automatically, when every failed job lost its runner. > - In a 17-hour sample, 42 runs lost a runner and were still the PR's current head. The automation rescued only 14. > - Two gaps caused most of the misses. A spot reclaim on RunsOn v3.2.3 usually ends the job, and the run, as `cancelled`; the workflow and script only look at `failure`. Also, the Runner support suite always carries `gateway_indexing_incomplete` error annotations from a passing test, so a lost runner there reads as a real failure. > - On-demand fallback for the fleet was turned off on 2026-11-13, so reclaims will happen more often. About 20% of runs are expected to lose at least one job. > - This pull request makes the re-run act on cancelled runs and jobs, but only with positive runner-loss evidence, and lets that evidence win over other error annotations. > - The benefit is fewer manual "Re-run failed jobs" clicks: an estimated 34 of 42 instead of 14 of 42 in the sample. ## Linked Issues or Issue Description No public issue exists. **What happened?** Run [43285064435](https://github.com/paperclipai/paperclip/actions/runs/43285064435) lost the runner of `ci / General tests (server (8/12))`. The job concluded `cancelled` (its log has `##[error]The runner has received a shutdown signal`). The run concluded `cancelled`, so the re-run workflow's `if:` skipped it. Run [43325855069](https://github.com/paperclipai/paperclip/actions/runs/43325855069) lost the runners of `ci / Runner support suite` and `ci / General tests (server (2/12))`. Both logs have the shutdown signal. The re-run bot ([decision log](https://github.com/paperclipai/paperclip/actions/runs/43327873358)) said "1 of 1 failed jobs did not lose their runner", with evidence `gateway_indexing_incomplete`. It ignored the cancelled job. It read the annotation as a real failure. **Expected behavior** Both runs get one automatic re-run of their failed jobs. **Steps to reproduce** The new tests replay trimmed snapshots of both runs (`.github/scripts/tests/fixtures/runner-loss-rerun/run-43285064435.json` and `run-43325855069.json`) through the script. In a 17-hour sample on 2026-11-12, 42 runs lost a runner and were still the PR's current head. Only 14 were re-run automatically. 16 stayed red because the run concluded `cancelled`. 4 stayed red because an error annotation in the lost job was read as a test failure. **Additional context** The `gateway_indexing_incomplete` annotation does not come from the shutdown. `scheduled-gateway-indexing.test.ts` prints `::error` lines while it passes, 84 seconds before the shutdown signal in this run. Attempt 2 of the same run passed with the same two failure annotations. So, before this change, a lost runner in the Runner support suite could never be re-run. ## What Changed - `ci-runner-loss-rerun.yml`: the job also runs when the PR run concluded `cancelled`. The trigger type, `permissions`, the repository, path, event and attempt-1 conditions do not change. - `runner-loss-rerun.mjs`, `eligibleRun`: accepts `completed/cancelled` as well as `completed/failure`. - `runner-loss-rerun.mjs`, `decide()`: classifies jobs that concluded `cancelled` as well as `failure`. It still skips the `ci / verify` and `ci / e2e` aggregates, and a failed or cancelled routing gate still blocks the re-run. - `decide()` now runs `staleReason` (open PR, current head, no newer run) before it reads any annotation or log, and again just before the re-run, as on master. A run cancelled by a new push has about 40 cancelled jobs; it now costs two API listings and no log reads. The final check closes the window where a push lands during the log reads: a re-run of the old run would join the PR's `cancel-in-progress` concurrency group (`pr-<number>` in `pr-trusted.yml`) and cancel the new commit's CI. - Following CONTRIBUTING.md ("Do not submit a PR that includes code comments"), this PR adds no new code comments. The reasons for each rule are in this description and in `doc/DEVELOPING.md`. - `classifyJob`: runner-loss evidence wins. A lost-communication annotation means lost. A cancellation annotation plus a shutdown signal in the log means lost, even if the job also has other error annotations. A cancelled job without this evidence (for example, a person cancelled the run) counts as a real failure and blocks the re-run. - Fixtures: two new trimmed snapshots from the example runs (jobs with `id`, `name`, `status`, `conclusion`; failure annotations; `##[error]` log lines only). The two older fixtures get the real annotations and log error lines of their cancelled jobs. Those jobs were spot-reclaimed too (each log has the shutdown signal), but the old code ignored them. - `doc/DEVELOPING.md`: the runner-loss paragraph describes the new rules. ## Verification - Red first. Against the unchanged script, the new tests fail for the reasons above: 43285064435 gives `run is completed/cancelled, not completed/failure`; 43325855069 gives `1 of 1 failed jobs did not lose their runner` with evidence `gateway_indexing_incomplete`, the same decision the bot logged. - Green: `node --test .github/scripts/tests/runner-loss-rerun.test.mjs`: 25 of 25 pass. - Push during log reads: the PR head moves after the first log read. With the final freshness check removed, this test fails (the script re-runs); with it, there is no re-run and the reason is `no open pull request still has this run's head commit`. - Guard cases, all with no re-run and no POST: - Superseded run: 52 jobs cancelled, the PR head moved. The test asserts zero annotation and log reads. - Person cancelled the run: cancelled jobs with only "The operation was canceled." in the annotation and log, and some never-started jobs with no annotation. - Mixed: one cancelled lost job and one real test failure (`1 of 2 failed jobs did not lose their runner`). - Once only: a cancelled run on attempt 2 is refused after one API read. The workflow condition test still requires `run_attempt == 1`. - Cancelled routing gate: left for a person. - Policy job command: `node --test '.github/scripts/tests/*.test.mjs'`: 806 of 806 pass. - `pnpm verify`: exit 2 ("Global CI input changed: .github/scripts/runner-loss-rerun.mjs"). It selects no local Vitest files and asks for required CI coverage. I did not run `pnpm verify --full`; required CI on this PR supplies that coverage. - This workflow runs on `workflow_run` from the default branch. This PR's own CI cannot exercise the new behavior. The fixture tests are the evidence. I did not dry-run the script against live runs. ## Risks - A test that really fails, and then a spot reclaim of the same job a minute later, now gets one wasted re-run. Attempt 2 then fails for real. It cannot loop: the workflow `if:` and `eligibleRun` both accept attempt 1 only. - A run a person cancels on the current head now starts the re-run job. The job reads the annotations, and the log of each job with a cancellation annotation, and then does not re-run. This costs some API calls on a GitHub-hosted runner. - A concurrency cancel by a label event that does not request CI (the newer run is `skipped`) passes the head check. Its cancelled jobs have no runner-loss evidence, so it is not re-run. - Trust: no change to `permissions`, the trigger, the accepted repository, event, workflow path, attempt, or the routing-gate rule. A cancelled run reuses its attempt-1 gate output, the same as a failed run does today. - Estimated effect, from a 17-hour sample: 14 of 42 rescued before; 16 cancelled runs and 4 runs with error-annotation noise become eligible (about 34 of 42). The other 8 stay skipped on purpose (a real test failure in the same run, a person re-ran first, or the PR closed). > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), Claude Code agent. Model ID claude-opus-5-5, with tool use (shell, GitHub CLI) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
1 parent
af1c685f5f
commit
c4f639ed1b
8 files changed
+1073
-43
No files matched your search
@@ -3,7 +3,7 @@
|
||||
* runner-loss-rerun.mjs
|
||||
*
|
||||
* Re-runs the failed jobs of a first PR CI attempt once, but only when every
|
||||
* failed job failed because its runner was lost. Runs from
|
||||
* failed or cancelled job ended because its runner was lost. Runs from
|
||||
* .github/workflows/ci-runner-loss-rerun.yml on workflow_run. It reads GitHub
|
||||
* API metadata only and never touches the PR's code or artifacts.
|
||||
*
|
||||
@@ -26,6 +26,7 @@ export const AGGREGATE_JOBS = new Set(["ci / verify", "ci / e2e"]);
|
||||
// ci-routing.yml checks the attempt's triggering actor. A bot-triggered attempt
|
||||
// would move the whole run to GitHub-hosted runners, so leave it to a human.
|
||||
export const GATE_JOB_PREFIX = "ci / gate / ";
|
||||
export const RERUN_CONCLUSIONS = new Set(["failure", "cancelled"]);
|
||||
// A shutdown signal often reaches the check run only as this annotation; the
|
||||
// signal itself is logged as an error line in the job log.
|
||||
const CANCELLED_MESSAGE = "The operation was canceled.";
|
||||
@@ -80,20 +81,14 @@ export function logShowsRunnerLoss(log) {
|
||||
export async function classifyJob(job, { annotations, readLog }) {
|
||||
const failures = annotations.filter((annotation) => annotation.annotation_level === "failure");
|
||||
const isRunnerLoss = (annotation) => RUNNER_LOSS_PATTERN.test(annotation.message ?? "");
|
||||
// A step can fail for a real reason before the runner is lost (for example a
|
||||
// test failure, then a loss during an always() upload step). Any failure
|
||||
// annotation other than runner loss or cancellation marks a real failure.
|
||||
const real = failures.find((annotation) => !isRunnerLoss(annotation) && annotation.message !== CANCELLED_MESSAGE);
|
||||
if (real) return { lost: false, evidence: firstLine(real.message) };
|
||||
const match = failures.find(isRunnerLoss);
|
||||
if (match) return { lost: true, evidence: firstLine(match.message) };
|
||||
// Only cancellations are left: the shutdown signal, if any, is in the log.
|
||||
if (failures.length > 0) {
|
||||
if (logShowsRunnerLoss(await readLog(job))) {
|
||||
return { lost: true, evidence: "job log: runner received a shutdown signal or lost communication" };
|
||||
}
|
||||
return { lost: false, evidence: `${CANCELLED_MESSAGE} (no runner-loss error in the job log)` };
|
||||
if (failures.some((annotation) => annotation.message === CANCELLED_MESSAGE) && logShowsRunnerLoss(await readLog(job))) {
|
||||
return { lost: true, evidence: "job log: runner received a shutdown signal or lost communication" };
|
||||
}
|
||||
const real = failures.find((annotation) => annotation.message !== CANCELLED_MESSAGE);
|
||||
if (real) return { lost: false, evidence: firstLine(real.message) };
|
||||
if (failures.length > 0) return { lost: false, evidence: `${CANCELLED_MESSAGE} (no runner-loss error in the job log)` };
|
||||
return { lost: false, evidence: "no failure annotation" };
|
||||
}
|
||||
|
||||
@@ -106,7 +101,9 @@ export function eligibleRun(run) {
|
||||
if (run.repository?.full_name !== REPOSITORY || run.repository?.id !== REPOSITORY_ID) return "run is not in the canonical repository";
|
||||
if (run.path !== PR_WORKFLOW_PATH) return `workflow is ${run.path}, not ${PR_WORKFLOW_PATH}`;
|
||||
if (run.event !== "pull_request") return `event is ${run.event}, not pull_request`;
|
||||
if (run.status !== "completed" || run.conclusion !== "failure") return `run is ${run.status}/${run.conclusion}, not completed/failure`;
|
||||
if (run.status !== "completed" || !RERUN_CONCLUSIONS.has(run.conclusion)) {
|
||||
return `run is ${run.status}/${run.conclusion}, not completed/failure or completed/cancelled`;
|
||||
}
|
||||
// One automatic re-run only. A higher attempt also means someone re-ran it.
|
||||
if (run.run_attempt !== 1) return `run is on attempt ${run.run_attempt}; only attempt 1 is re-run automatically`;
|
||||
return undefined;
|
||||
@@ -137,9 +134,12 @@ export async function decide(client, run) {
|
||||
if (ineligible) return { ...decision, reason: ineligible };
|
||||
|
||||
const jobs = await client.paginate(`/repos/${REPOSITORY}/actions/runs/${run.id}/attempts/1/jobs?per_page=100`, "jobs");
|
||||
const failed = jobs.filter((job) => job.conclusion === "failure" && !AGGREGATE_JOBS.has(job.name));
|
||||
const failed = jobs.filter((job) => RERUN_CONCLUSIONS.has(job.conclusion) && !AGGREGATE_JOBS.has(job.name));
|
||||
if (failed.length === 0) return { ...decision, reason: "no failed jobs other than the verify/e2e aggregates" };
|
||||
|
||||
const staleBefore = await staleReason(client, run);
|
||||
if (staleBefore) return { ...decision, reason: staleBefore };
|
||||
|
||||
for (const job of failed) {
|
||||
if (job.name.startsWith(GATE_JOB_PREFIX)) {
|
||||
decision.jobs.push({ id: job.id, name: job.name, lost: false, evidence: "routing gate failed; a bot re-run would lose AWS routing" });
|
||||
|
||||
@@ -1042,7 +1042,7 @@
|
||||
"end_line": 123,
|
||||
"annotation_level": "failure",
|
||||
"title": "[paperclipai] src/__tests__/company-import-transfer.test.ts > resolveChunkedImportZip > takes the chunked path for a small zip whose entries inflate past the threshold",
|
||||
"message": "Error: Test timed out in 5000ms.\nIf this is a long-running test, pass a timeout value as the last argument or configure it globally with \"testTimeout\".\n ❯ src/__tests__/company-import-transfer.test.ts:123:3\n\n",
|
||||
"message": "Error: Test timed out in 5000ms.\nIf this is a long-running test, pass a timeout value as the last argument or configure it globally with \"testTimeout\".\n \u276f src/__tests__/company-import-transfer.test.ts:123:3\n\n",
|
||||
"raw_details": ""
|
||||
},
|
||||
{
|
||||
@@ -1051,7 +1051,7 @@
|
||||
"end_line": 280,
|
||||
"annotation_level": "failure",
|
||||
"title": "[paperclipai] src/__tests__/company-import-export-e2e.test.ts > paperclipai company import/export e2e",
|
||||
"message": "Error: Timed out waiting for the child server recorded in /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/paperclip-home/instances/company-cli-e2e/runtime-info.json.\nstdout:\nUpdate available: 2026.1023.0 — run `paperclipai update`\n┌ paperclipai run \n│\n│ Home: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/paperclip-home\n│\n│ Instance: company-cli-e2e\n│\n│ Config: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/config.json\n│\n◇ Running doctor checks...\nUpdate available: 2026.1023.0 — run `paperclipai update`\n\n██████╗ █████╗ ██████╗ ███████╗██████╗ ██████╗██╗ ██╗██████╗ \n██╔══██╗██╔══██╗██╔══██╗██╔════╝██╔══██╗██╔════╝██║ ██║██╔══██╗\n██████╔╝███████║██████╔╝█████╗ ██████╔╝██║ ██║ ██║██████╔╝\n██╔═══╝ ██╔══██║██╔═══╝ ██╔══╝ ██╔══██╗██║ ██║ ██║██╔═══╝ \n██║ ██║ ██║██║ ███████╗██║ ██║╚██████╗███████╗██║██║ \n╚═╝ ╚═╝ ╚═╝╚═╝ ╚══════╝╚═╝ ╚═╝ ╚═════╝╚══════╝╚═╝╚═╝ \n ───────────────────────────────────────────────────────\n The app people use to manage AI agents for work\n\n┌ paperclip doctor \n│\n│ ✓ Config file: Valid config at /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/config.json\n│\n│ ✓ Deployment/auth mode: local_trusted mode is configured for loopback-only access\n│\n│ ✗ Agent JWT secret: PAPERCLIP_AGENT_JWT_SECRET missing from environment and /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/.env\n│\n│ Run with --repair to create /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/.env containing PAPERCLIP_AGENT_JWT_SECRET\n│\n◆ Repaired: Agent JWT secret\n│\n│ ✓ Agent JWT secret: PAPERCLIP_AGENT_JWT_SECRET is set in environment\n│\n│ ! Secrets adapter: Secrets key file does not exist yet: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/secrets/master.key; strict secret mode is disabled for postgres deployment\n│\n│ Run with --repair to create a local encrypted secrets key file. Consider enabling secrets.strictMode\n│\n◆ Repaired: Secrets adapter\n│\n│ ! Secrets adapter: Local encrypted provider configured with key file /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/secrets/master.key; strict secret mode is disabled for postgres deployment\n│\n│ Consider enabling secrets.strictMode\n│\n│ ✓ Storage: Local disk storage is writable: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/storage\n│\n│ ✓ Database: PostgreSQL connection successful\n│\n│ ✓ LLM provider: No LLM provider configured (optional)\n│\n│ ✓ Log directory: Log directory is writable: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/logs\n│\n│ ! Server port: Port 37925 is already in use\n│\n│ Check what's using port 37925 with: lsof -i :37925\n│\n│ ✓ Node.js runtime: Node.js 24.22.2\n│\n│ ✓ Managed install: Not present (optional for npx, global npm, and source-checkout usage)\n│\n│ ✓ Background service: No usable systemd user manager was detected (common in containers and WSL1). Use paperclipai run instead.\n│\n◇ Summary ───────────────╮\n│ │\n│ 10 passed, 2 warnings │\n│ │\n├─────────────────────────╯\n│\n└ All critical checks passed with some warnings.\n\n│\n◇ Start",
|
||||
"message": "Error: Timed out waiting for the child server recorded in /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/paperclip-home/instances/company-cli-e2e/runtime-info.json.\nstdout:\nUpdate available: 2026.1023.0 \u2014 run `paperclipai update`\n\u250c paperclipai run \n\u2502\n\u2502 Home: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/paperclip-home\n\u2502\n\u2502 Instance: company-cli-e2e\n\u2502\n\u2502 Config: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/config.json\n\u2502\n\u25c7 Running doctor checks...\nUpdate available: 2026.1023.0 \u2014 run `paperclipai update`\n\n\u2588\u2588\u2588\u2588\u2588\u2588\u2557 \u2588\u2588\u2588\u2588\u2588\u2557 \u2588\u2588\u2588\u2588\u2588\u2588\u2557 \u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2557\u2588\u2588\u2588\u2588\u2588\u2588\u2557 \u2588\u2588\u2588\u2588\u2588\u2588\u2557\u2588\u2588\u2557 \u2588\u2588\u2557\u2588\u2588\u2588\u2588\u2588\u2588\u2557 \n\u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2557\u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2557\u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2557\u2588\u2588\u2554\u2550\u2550\u2550\u2550\u255d\u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2557\u2588\u2588\u2554\u2550\u2550\u2550\u2550\u255d\u2588\u2588\u2551 \u2588\u2588\u2551\u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2557\n\u2588\u2588\u2588\u2588\u2588\u2588\u2554\u255d\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2551\u2588\u2588\u2588\u2588\u2588\u2588\u2554\u255d\u2588\u2588\u2588\u2588\u2588\u2557 \u2588\u2588\u2588\u2588\u2588\u2588\u2554\u255d\u2588\u2588\u2551 \u2588\u2588\u2551 \u2588\u2588\u2551\u2588\u2588\u2588\u2588\u2588\u2588\u2554\u255d\n\u2588\u2588\u2554\u2550\u2550\u2550\u255d \u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2551\u2588\u2588\u2554\u2550\u2550\u2550\u255d \u2588\u2588\u2554\u2550\u2550\u255d \u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2557\u2588\u2588\u2551 \u2588\u2588\u2551 \u2588\u2588\u2551\u2588\u2588\u2554\u2550\u2550\u2550\u255d \n\u2588\u2588\u2551 \u2588\u2588\u2551 \u2588\u2588\u2551\u2588\u2588\u2551 \u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2557\u2588\u2588\u2551 \u2588\u2588\u2551\u255a\u2588\u2588\u2588\u2588\u2588\u2588\u2557\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2557\u2588\u2588\u2551\u2588\u2588\u2551 \n\u255a\u2550\u255d \u255a\u2550\u255d \u255a\u2550\u255d\u255a\u2550\u255d \u255a\u2550\u2550\u2550\u2550\u2550\u2550\u255d\u255a\u2550\u255d \u255a\u2550\u255d \u255a\u2550\u2550\u2550\u2550\u2550\u255d\u255a\u2550\u2550\u2550\u2550\u2550\u2550\u255d\u255a\u2550\u255d\u255a\u2550\u255d \n \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n The app people use to manage AI agents for work\n\n\u250c paperclip doctor \n\u2502\n\u2502 \u2713 Config file: Valid config at /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/config.json\n\u2502\n\u2502 \u2713 Deployment/auth mode: local_trusted mode is configured for loopback-only access\n\u2502\n\u2502 \u2717 Agent JWT secret: PAPERCLIP_AGENT_JWT_SECRET missing from environment and /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/.env\n\u2502\n\u2502 Run with --repair to create /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/.env containing PAPERCLIP_AGENT_JWT_SECRET\n\u2502\n\u25c6 Repaired: Agent JWT secret\n\u2502\n\u2502 \u2713 Agent JWT secret: PAPERCLIP_AGENT_JWT_SECRET is set in environment\n\u2502\n\u2502 ! Secrets adapter: Secrets key file does not exist yet: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/secrets/master.key; strict secret mode is disabled for postgres deployment\n\u2502\n\u2502 Run with --repair to create a local encrypted secrets key file. Consider enabling secrets.strictMode\n\u2502\n\u25c6 Repaired: Secrets adapter\n\u2502\n\u2502 ! Secrets adapter: Local encrypted provider configured with key file /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/secrets/master.key; strict secret mode is disabled for postgres deployment\n\u2502\n\u2502 Consider enabling secrets.strictMode\n\u2502\n\u2502 \u2713 Storage: Local disk storage is writable: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/storage\n\u2502\n\u2502 \u2713 Database: PostgreSQL connection successful\n\u2502\n\u2502 \u2713 LLM provider: No LLM provider configured (optional)\n\u2502\n\u2502 \u2713 Log directory: Log directory is writable: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/logs\n\u2502\n\u2502 ! Server port: Port 37925 is already in use\n\u2502\n\u2502 Check what's using port 37925 with: lsof -i :37925\n\u2502\n\u2502 \u2713 Node.js runtime: Node.js 24.22.2\n\u2502\n\u2502 \u2713 Managed install: Not present (optional for npx, global npm, aLine truncated
|
||||
"raw_details": ""
|
||||
},
|
||||
{
|
||||
@@ -1085,10 +1085,46 @@
|
||||
"message": "Process completed with exit code 1.",
|
||||
"raw_details": ""
|
||||
}
|
||||
],
|
||||
"128079212951": [
|
||||
{
|
||||
"path": ".github",
|
||||
"start_line": 130,
|
||||
"end_line": 130,
|
||||
"annotation_level": "failure",
|
||||
"title": "",
|
||||
"message": "The operation was canceled.",
|
||||
"raw_details": ""
|
||||
}
|
||||
],
|
||||
"128079213109": [
|
||||
{
|
||||
"path": ".github",
|
||||
"start_line": 130,
|
||||
"end_line": 130,
|
||||
"annotation_level": "failure",
|
||||
"title": "",
|
||||
"message": "The operation was canceled.",
|
||||
"raw_details": ""
|
||||
}
|
||||
],
|
||||
"128079213211": [
|
||||
{
|
||||
"path": ".github",
|
||||
"start_line": 133,
|
||||
"end_line": 133,
|
||||
"annotation_level": "failure",
|
||||
"title": "",
|
||||
"message": "The operation was canceled.",
|
||||
"raw_details": ""
|
||||
}
|
||||
]
|
||||
},
|
||||
"logs": {
|
||||
"128078674151": "2026-11-06T18:42:31.0413375Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T18:42:43.5529901Z ##[error]The operation was canceled.\n2026-11-06T18:42:43.5656287Z Cleaning up orphan processes\n"
|
||||
"128078674151": "2026-11-06T18:42:31.0413375Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T18:42:43.5529901Z ##[error]The operation was canceled.\n2026-11-06T18:42:43.5656287Z Cleaning up orphan processes\n",
|
||||
"128079212951": "2026-11-06T18:42:41.1643719Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T18:42:53.6869870Z ##[error]The operation was canceled.\n",
|
||||
"128079213109": "2026-11-06T18:42:35.0486149Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T18:42:47.5624672Z ##[error]The operation was canceled.\n",
|
||||
"128079213211": "2026-11-06T18:42:38.0453932Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T18:42:50.5615051Z ##[error]The operation was canceled.\n"
|
||||
},
|
||||
"total_count": 48
|
||||
}
|
||||
@@ -970,10 +970,130 @@
|
||||
"message": "Process completed with exit code 1.",
|
||||
"raw_details": ""
|
||||
}
|
||||
],
|
||||
"128101015916": [
|
||||
{
|
||||
"path": ".github",
|
||||
"start_line": 130,
|
||||
"end_line": 130,
|
||||
"annotation_level": "failure",
|
||||
"title": "",
|
||||
"message": "The operation was canceled.",
|
||||
"raw_details": ""
|
||||
}
|
||||
],
|
||||
"128101015978": [
|
||||
{
|
||||
"path": ".github",
|
||||
"start_line": 130,
|
||||
"end_line": 130,
|
||||
"annotation_level": "failure",
|
||||
"title": "",
|
||||
"message": "The operation was canceled.",
|
||||
"raw_details": ""
|
||||
}
|
||||
],
|
||||
"128101016153": [
|
||||
{
|
||||
"path": ".github",
|
||||
"start_line": 133,
|
||||
"end_line": 133,
|
||||
"annotation_level": "failure",
|
||||
"title": "",
|
||||
"message": "The operation was canceled.",
|
||||
"raw_details": ""
|
||||
}
|
||||
],
|
||||
"128101016200": [
|
||||
{
|
||||
"path": ".github",
|
||||
"start_line": 133,
|
||||
"end_line": 133,
|
||||
"annotation_level": "failure",
|
||||
"title": "",
|
||||
"message": "The operation was canceled.",
|
||||
"raw_details": ""
|
||||
}
|
||||
],
|
||||
"128101016304": [
|
||||
{
|
||||
"path": ".github",
|
||||
"start_line": 133,
|
||||
"end_line": 133,
|
||||
"annotation_level": "failure",
|
||||
"title": "",
|
||||
"message": "The operation was canceled.",
|
||||
"raw_details": ""
|
||||
}
|
||||
],
|
||||
"128101016340": [
|
||||
{
|
||||
"path": ".github",
|
||||
"start_line": 133,
|
||||
"end_line": 133,
|
||||
"annotation_level": "failure",
|
||||
"title": "",
|
||||
"message": "The operation was canceled.",
|
||||
"raw_details": ""
|
||||
}
|
||||
],
|
||||
"128101016402": [
|
||||
{
|
||||
"path": ".github",
|
||||
"start_line": 133,
|
||||
"end_line": 133,
|
||||
"annotation_level": "failure",
|
||||
"title": "",
|
||||
"message": "The operation was canceled.",
|
||||
"raw_details": ""
|
||||
}
|
||||
],
|
||||
"128101016432": [
|
||||
{
|
||||
"path": ".github",
|
||||
"start_line": 133,
|
||||
"end_line": 133,
|
||||
"annotation_level": "failure",
|
||||
"title": "",
|
||||
"message": "The operation was canceled.",
|
||||
"raw_details": ""
|
||||
}
|
||||
],
|
||||
"128101016453": [
|
||||
{
|
||||
"path": ".github",
|
||||
"start_line": 133,
|
||||
"end_line": 133,
|
||||
"annotation_level": "failure",
|
||||
"title": "",
|
||||
"message": "The operation was canceled.",
|
||||
"raw_details": ""
|
||||
}
|
||||
],
|
||||
"128101016461": [
|
||||
{
|
||||
"path": ".github",
|
||||
"start_line": 133,
|
||||
"end_line": 133,
|
||||
"annotation_level": "failure",
|
||||
"title": "",
|
||||
"message": "The operation was canceled.",
|
||||
"raw_details": ""
|
||||
}
|
||||
]
|
||||
},
|
||||
"logs": {
|
||||
"128101016102": "2026-11-06T19:24:29.1450769Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:29.1898916Z ##[error]The operation was canceled.\n2026-11-06T19:24:29.1977091Z Cleaning up orphan processes\n"
|
||||
"128101016102": "2026-11-06T19:24:29.1450769Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:29.1898916Z ##[error]The operation was canceled.\n2026-11-06T19:24:29.1977091Z Cleaning up orphan processes\n",
|
||||
"128101015916": "2026-11-06T19:24:29.1391073Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.6585509Z ##[error]The operation was canceled.\n",
|
||||
"128101015978": "2026-11-06T19:24:29.1982327Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.7170843Z ##[error]The operation was canceled.\n",
|
||||
"128101016153": "2026-11-06T19:24:29.1786229Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.6972420Z ##[error]The operation was canceled.\n",
|
||||
"128101016200": "2026-11-06T19:24:29.1721198Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.6975527Z ##[error]The operation was canceled.\n",
|
||||
"128101016304": "2026-11-06T19:24:29.1377822Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.6673266Z ##[error]The operation was canceled.\n",
|
||||
"128101016340": "2026-11-06T19:24:29.1341317Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.6500655Z ##[error]The operation was canceled.\n",
|
||||
"128101016402": "2026-11-06T19:24:29.1167345Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.6423716Z ##[error]The operation was canceled.\n",
|
||||
"128101016432": "2026-11-06T19:24:29.0692803Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.5851488Z ##[error]The operation was canceled.\n",
|
||||
"128101016453": "2026-11-06T19:24:29.2339015Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.7586244Z ##[error]The operation was canceled.\n",
|
||||
"128101016461": "2026-11-06T19:24:29.1057916Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.6269036Z ##[error]The operation was canceled.\n"
|
||||
},
|
||||
"total_count": 48
|
||||
}
|
||||
@@ -0,0 +1,360 @@
|
||||
{
|
||||
"source": "GET /repos/paperclipai/paperclip/actions/runs/43285064435/attempts/1 (+ jobs, annotations and job-log error lines), trimmed",
|
||||
"run": {
|
||||
"id": 43285064435,
|
||||
"name": "PR",
|
||||
"head_sha": "cd287efba2ff0c557f0c63fc78d5ad1d48f192e8",
|
||||
"path": ".github/workflows/pr.yml",
|
||||
"event": "pull_request",
|
||||
"status": "completed",
|
||||
"conclusion": "cancelled",
|
||||
"html_url": "https://github.com/paperclipai/paperclip/actions/runs/43285064435",
|
||||
"run_attempt": 1,
|
||||
"repository": {
|
||||
"id": 1170821064,
|
||||
"full_name": "paperclipai/paperclip"
|
||||
},
|
||||
"head_repository": {
|
||||
"id": 1170821064,
|
||||
"full_name": "paperclipai/paperclip"
|
||||
}
|
||||
},
|
||||
"jobs": [
|
||||
{
|
||||
"id": 131416164450,
|
||||
"name": "ci / gate / Select trusted runner",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416492147,
|
||||
"name": "ci / policy",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416492206,
|
||||
"name": "ci / Typecheck + Release Registry",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416492248,
|
||||
"name": "ci / Verify Paperclip Runner (static checks)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416492288,
|
||||
"name": "ci / Runner support suite",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416492337,
|
||||
"name": "ci / Verify Paperclip Runner (rust)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416492390,
|
||||
"name": "ci / Build",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416492402,
|
||||
"name": "ci / Canary Dry Run",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416492420,
|
||||
"name": "ci / Verify Paperclip Runner (vitest package (3/3))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416492421,
|
||||
"name": "ci / Sandbox provider tests",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416492424,
|
||||
"name": "ci / Docker context integrity",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416492430,
|
||||
"name": "ci / Verify Paperclip Runner (vitest native server)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416492442,
|
||||
"name": "ci / Verify Paperclip Runner (vitest package (2/3))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416492489,
|
||||
"name": "ci / Verify Paperclip Runner (vitest package (1/3))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416492538,
|
||||
"name": "ci / e2e shard (5/8)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416492565,
|
||||
"name": "ci / e2e shard (1/8)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416492568,
|
||||
"name": "ci / e2e shard (2/8)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416492582,
|
||||
"name": "ci / e2e shard (6/8)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416492588,
|
||||
"name": "ci / e2e shard (7/8)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416492592,
|
||||
"name": "ci / e2e shard (4/8)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416492708,
|
||||
"name": "ci / e2e shard (3/8)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416493348,
|
||||
"name": "ci / e2e shard (8/8)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416493582,
|
||||
"name": "ci / Verify serialized server suites (3/9)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416493601,
|
||||
"name": "ci / Verify serialized server suites (4/9)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416493630,
|
||||
"name": "ci / Verify serialized server suites (2/9)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416493633,
|
||||
"name": "ci / Verify serialized server suites (1/9)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416493669,
|
||||
"name": "ci / Verify serialized server suites (5/9)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416493759,
|
||||
"name": "ci / Verify serialized server suites (7/9)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416493766,
|
||||
"name": "ci / Verify serialized server suites (9/9)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416493829,
|
||||
"name": "ci / Verify serialized server suites (8/9)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416493947,
|
||||
"name": "ci / General tests (server (5/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416494052,
|
||||
"name": "ci / General tests (server (1/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416494077,
|
||||
"name": "ci / General tests (server (3/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416494102,
|
||||
"name": "ci / General tests (server (7/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416494186,
|
||||
"name": "ci / General tests (server (4/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416494194,
|
||||
"name": "ci / General tests (workspaces-c)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416494201,
|
||||
"name": "ci / General tests (chat (3/3))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416494251,
|
||||
"name": "ci / General tests (server (12/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416494281,
|
||||
"name": "ci / General tests (server (2/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416494286,
|
||||
"name": "ci / General tests (chat (1/3))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416494333,
|
||||
"name": "ci / General tests (workspaces-a (1/2))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416494339,
|
||||
"name": "ci / General tests (server (10/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416494340,
|
||||
"name": "ci / General tests (server (9/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416494376,
|
||||
"name": "ci / General tests (chat (2/3))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416494446,
|
||||
"name": "ci / General tests (server (6/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416494459,
|
||||
"name": "ci / General tests (server (8/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "cancelled"
|
||||
},
|
||||
{
|
||||
"id": 131416494489,
|
||||
"name": "ci / General tests (workspaces-b)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416494538,
|
||||
"name": "ci / Verify serialized server suites (6/9)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416494546,
|
||||
"name": "ci / General tests (server (11/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416494552,
|
||||
"name": "ci / General tests (workspaces-a (2/2))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131416494939,
|
||||
"name": "ci / Select Vitest files",
|
||||
"status": "completed",
|
||||
"conclusion": "skipped"
|
||||
},
|
||||
{
|
||||
"id": 131421415244,
|
||||
"name": "ci / e2e",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131424041299,
|
||||
"name": "ci / verify",
|
||||
"status": "completed",
|
||||
"conclusion": "failure"
|
||||
}
|
||||
],
|
||||
"annotations": {
|
||||
"131416494459": [
|
||||
{
|
||||
"annotation_level": "failure",
|
||||
"message": "The operation was canceled."
|
||||
}
|
||||
],
|
||||
"131424041299": [
|
||||
{
|
||||
"annotation_level": "failure",
|
||||
"message": "Process completed with exit code 1."
|
||||
}
|
||||
]
|
||||
},
|
||||
"logs": {
|
||||
"131416494459": "2026-11-12T19:03:20.1001059Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-12T19:03:32.6527460Z ##[error]The operation was canceled.\n"
|
||||
},
|
||||
"total_count": 53
|
||||
}
|
||||
@@ -0,0 +1,363 @@
|
||||
{
|
||||
"source": "GET /repos/paperclipai/paperclip/actions/runs/43325855069/attempts/1 (+ jobs, annotations and job-log error lines), trimmed",
|
||||
"run": {
|
||||
"id": 43325855069,
|
||||
"name": "PR",
|
||||
"head_sha": "4b4f6b2ad0fac4ede9f4e20a56e3c93fac5577d5",
|
||||
"path": ".github/workflows/pr.yml",
|
||||
"event": "pull_request",
|
||||
"status": "completed",
|
||||
"conclusion": "failure",
|
||||
"html_url": "https://github.com/paperclipai/paperclip/actions/runs/43325855069",
|
||||
"run_attempt": 1,
|
||||
"repository": {
|
||||
"id": 1170821064,
|
||||
"full_name": "paperclipai/paperclip"
|
||||
},
|
||||
"head_repository": {
|
||||
"id": 1170821064,
|
||||
"full_name": "paperclipai/paperclip"
|
||||
}
|
||||
},
|
||||
"jobs": [
|
||||
{
|
||||
"id": 131557987038,
|
||||
"name": "ci / gate / Select trusted runner",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558292300,
|
||||
"name": "ci / Docker context integrity",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558292349,
|
||||
"name": "ci / Canary Dry Run",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558292381,
|
||||
"name": "ci / Runner support suite",
|
||||
"status": "completed",
|
||||
"conclusion": "failure"
|
||||
},
|
||||
{
|
||||
"id": 131558292405,
|
||||
"name": "ci / Typecheck + Release Registry",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558292466,
|
||||
"name": "ci / policy",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558292573,
|
||||
"name": "ci / Build",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558292627,
|
||||
"name": "ci / Verify Paperclip Runner (vitest package (1/3))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558292664,
|
||||
"name": "ci / Verify Paperclip Runner (static checks)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558292669,
|
||||
"name": "ci / Verify Paperclip Runner (vitest package (2/3))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558292679,
|
||||
"name": "ci / e2e shard (8/8)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558292686,
|
||||
"name": "ci / Verify Paperclip Runner (vitest native server)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558292707,
|
||||
"name": "ci / Verify Paperclip Runner (vitest package (3/3))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558292731,
|
||||
"name": "ci / e2e shard (5/8)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558292738,
|
||||
"name": "ci / e2e shard (2/8)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558292751,
|
||||
"name": "ci / e2e shard (1/8)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558292779,
|
||||
"name": "ci / e2e shard (4/8)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558292841,
|
||||
"name": "ci / e2e shard (3/8)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558292887,
|
||||
"name": "ci / e2e shard (7/8)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293043,
|
||||
"name": "ci / e2e shard (6/8)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293328,
|
||||
"name": "ci / Verify serialized server suites (4/9)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293330,
|
||||
"name": "ci / Verify serialized server suites (2/9)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293380,
|
||||
"name": "ci / Verify serialized server suites (7/9)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293389,
|
||||
"name": "ci / Verify serialized server suites (3/9)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293399,
|
||||
"name": "ci / Verify serialized server suites (8/9)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293420,
|
||||
"name": "ci / Verify serialized server suites (1/9)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293433,
|
||||
"name": "ci / Verify serialized server suites (5/9)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293450,
|
||||
"name": "ci / Verify serialized server suites (6/9)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293683,
|
||||
"name": "ci / Verify serialized server suites (9/9)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293690,
|
||||
"name": "ci / General tests (server (12/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293701,
|
||||
"name": "ci / General tests (server (11/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293710,
|
||||
"name": "ci / General tests (server (8/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293784,
|
||||
"name": "ci / General tests (server (3/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293818,
|
||||
"name": "ci / General tests (server (2/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "cancelled"
|
||||
},
|
||||
{
|
||||
"id": 131558293819,
|
||||
"name": "ci / General tests (workspaces-a (1/2))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293825,
|
||||
"name": "ci / General tests (server (4/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293835,
|
||||
"name": "ci / General tests (server (5/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293844,
|
||||
"name": "ci / General tests (server (9/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293850,
|
||||
"name": "ci / General tests (workspaces-a (2/2))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293852,
|
||||
"name": "ci / General tests (server (1/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293858,
|
||||
"name": "ci / General tests (chat (2/3))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293865,
|
||||
"name": "ci / General tests (server (10/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293875,
|
||||
"name": "ci / General tests (workspaces-b)",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293940,
|
||||
"name": "ci / General tests (server (6/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558293992,
|
||||
"name": "ci / General tests (chat (3/3))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558294007,
|
||||
"name": "ci / General tests (chat (1/3))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558294057,
|
||||
"name": "ci / General tests (server (7/12))",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131558294396,
|
||||
"name": "ci / Sandbox provider tests",
|
||||
"status": "completed",
|
||||
"conclusion": "skipped"
|
||||
},
|
||||
{
|
||||
"id": 131558294604,
|
||||
"name": "ci / Select Vitest files",
|
||||
"status": "completed",
|
||||
"conclusion": "skipped"
|
||||
},
|
||||
{
|
||||
"id": 131562309925,
|
||||
"name": "ci / e2e",
|
||||
"status": "completed",
|
||||
"conclusion": "success"
|
||||
},
|
||||
{
|
||||
"id": 131564602271,
|
||||
"name": "ci / verify",
|
||||
"status": "completed",
|
||||
"conclusion": "failure"
|
||||
}
|
||||
],
|
||||
"annotations": {
|
||||
"131558292381": [
|
||||
{
|
||||
"annotation_level": "failure",
|
||||
"message": "The operation was canceled."
|
||||
},
|
||||
{
|
||||
"annotation_level": "failure",
|
||||
"message": "gateway_indexing_incomplete"
|
||||
},
|
||||
{
|
||||
"annotation_level": "failure",
|
||||
"message": "gateway_indexing_incomplete"
|
||||
}
|
||||
],
|
||||
"131558293818": [
|
||||
{
|
||||
"annotation_level": "failure",
|
||||
"message": "The operation was canceled."
|
||||
}
|
||||
],
|
||||
"131564602271": [
|
||||
{
|
||||
"annotation_level": "failure",
|
||||
"message": "Process completed with exit code 1."
|
||||
}
|
||||
]
|
||||
},
|
||||
"logs": {
|
||||
"131558292381": "2026-11-12T23:13:37.4339910Z ##[error]gateway_indexing_incomplete\n2026-11-12T23:13:37.4368921Z ##[error]gateway_indexing_incomplete\n2026-11-12T23:15:01.1805404Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-12T23:15:13.7401251Z ##[error]The operation was canceled.\n",
|
||||
"131558293818": "2026-11-12T23:15:36.1981336Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-12T23:15:48.7226826Z ##[error]The operation was canceled.\n"
|
||||
},
|
||||
"total_count": 51
|
||||
}
|
||||
@@ -13,10 +13,13 @@ import {
|
||||
const fixture = (runId) => JSON.parse(readFileSync(
|
||||
new URL(`./fixtures/runner-loss-rerun/run-${runId}.json`, import.meta.url), "utf8",
|
||||
));
|
||||
const MIXED = 42271918832; // six lost runners, one shutdown in the log, one real test timeout
|
||||
const MIXED = 42271918832; // six lost runners, four shutdowns in the log, one real test timeout
|
||||
const REAL = 42264785415; // a real assertion failure
|
||||
const ALL_LOST = 42272083862; // one lost runner
|
||||
const LOG_ONLY = 42277969418; // shutdown signal visible only in the job log
|
||||
const LOG_ONLY = 42277969418; // shutdown signals visible only in the job logs
|
||||
const CANCELLED_RUN = 43285064435;
|
||||
const NOISY_LOST = 43325855069;
|
||||
const CANCELLED_MESSAGE = "The operation was canceled.";
|
||||
const workflow = readFileSync(new URL("../../workflows/ci-runner-loss-rerun.yml", import.meta.url), "utf8");
|
||||
|
||||
const json = (body, headers = {}) => new Response(JSON.stringify(body), {
|
||||
@@ -58,7 +61,7 @@ function fakeGithub(data, {
|
||||
if (match) return json(data.annotations[match[1]] ?? []);
|
||||
match = route.match(/^\/actions\/jobs\/(\d+)\/logs$/);
|
||||
if (match) return new Response(data.logs[match[1]] ?? "", { status: 200 });
|
||||
if (route === `/commits/${run.head_sha}/pulls`) return json(pulls ?? [openPr]);
|
||||
if (route === `/commits/${run.head_sha}/pulls`) return json((typeof pulls === "function" ? pulls(calls) : pulls) ?? [openPr]);
|
||||
if (route === "/actions/workflows/pr.yml/runs") {
|
||||
const runs = prRuns ?? [{ id: run.id, conclusion: "failure" }];
|
||||
return json({ total_count: runs.length, workflow_runs: runs });
|
||||
@@ -106,19 +109,131 @@ test("a shutdown signal found only in the job log counts as runner loss", async
|
||||
assert.deepEqual(data.annotations["128101016102"].map((annotation) => annotation.message), ["The operation was canceled."]);
|
||||
const { decision, calls } = await runMain(data);
|
||||
assert.equal(decision.rerun, true);
|
||||
assert.match(decision.jobs[0].evidence, /job log/);
|
||||
assert.equal(decision.jobs.length, 11);
|
||||
assert.ok(decision.jobs.every((job) => job.lost && /job log/.test(job.evidence)));
|
||||
assert.ok(calls.some((call) => call.path.endsWith("/actions/jobs/128101016102/logs")));
|
||||
assert.equal(posted(calls).length, 1);
|
||||
});
|
||||
|
||||
test("a cancelled run whose cancelled job lost its runner is re-run once", async () => {
|
||||
const data = fixture(CANCELLED_RUN);
|
||||
assert.equal(data.run.conclusion, "cancelled");
|
||||
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
|
||||
assert.equal(decision.rerun, true);
|
||||
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [["ci / General tests (server (8/12))", true]]);
|
||||
assert.match(decision.jobs[0].evidence, /job log/);
|
||||
assert.deepEqual(posted(calls).map((call) => call.path),
|
||||
[`/repos/paperclipai/paperclip/actions/runs/${CANCELLED_RUN}/rerun-failed-jobs`]);
|
||||
});
|
||||
|
||||
test("runner loss in the log wins over error annotations the workload printed", async () => {
|
||||
const data = fixture(NOISY_LOST);
|
||||
const suite = data.jobs.find((job) => job.name === "ci / Runner support suite");
|
||||
assert.ok(data.annotations[suite.id].some((annotation) => annotation.message === "gateway_indexing_incomplete"));
|
||||
const { decision, calls } = await runMain(data);
|
||||
assert.equal(decision.rerun, true);
|
||||
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [
|
||||
["ci / Runner support suite", true],
|
||||
["ci / General tests (server (2/12))", true],
|
||||
]);
|
||||
assert.equal(posted(calls).length, 1);
|
||||
});
|
||||
|
||||
function cancelEverything(data, { log = `2026-11-12T19:03:32.0000000Z ##[error]${CANCELLED_MESSAGE}\n` } = {}) {
|
||||
data.annotations = {};
|
||||
data.logs = {};
|
||||
for (const [index, job] of data.jobs.entries()) {
|
||||
if (job.name.startsWith("ci / gate / ")) continue;
|
||||
job.conclusion = "cancelled";
|
||||
if (index % 4 === 3) continue;
|
||||
data.annotations[job.id] = [{ annotation_level: "failure", message: CANCELLED_MESSAGE }];
|
||||
data.logs[job.id] = log;
|
||||
}
|
||||
return data;
|
||||
}
|
||||
|
||||
test("a run superseded by a new push is not re-run, and no job is classified", async () => {
|
||||
const data = cancelEverything(fixture(CANCELLED_RUN));
|
||||
assert.ok(data.jobs.filter((job) => job.conclusion === "cancelled").length >= 40);
|
||||
const { decision, calls } = await runMain(data, {
|
||||
pulls: [{ state: "open", base: { repo: { id: 1170821064 } }, head: { sha: "0".repeat(40), repo: { id: 1170821064 } } }],
|
||||
}, { conclusion: "cancelled" });
|
||||
assert.equal(decision.rerun, false);
|
||||
assert.match(decision.reason, /no open pull request/);
|
||||
assert.equal(calls.some((call) => call.path.includes("/check-runs/") || call.path.endsWith("/logs")), false);
|
||||
assert.equal(posted(calls).length, 0);
|
||||
});
|
||||
|
||||
test("a push while job logs are read stops the re-run", async () => {
|
||||
const data = fixture(CANCELLED_RUN);
|
||||
const moved = [{ state: "open", base: { repo: { id: 1170821064 } }, head: { sha: "0".repeat(40), repo: { id: 1170821064 } } }];
|
||||
const { decision, calls } = await runMain(data, {
|
||||
pulls: (seen) => (seen.some((call) => call.path.endsWith("/logs")) ? moved : undefined),
|
||||
}, { conclusion: "cancelled" });
|
||||
assert.equal(decision.rerun, false);
|
||||
assert.match(decision.reason, /no open pull request/);
|
||||
assert.equal(decision.jobs[0].lost, true);
|
||||
assert.equal(calls.filter((call) => call.path.endsWith("/pulls")).length, 2);
|
||||
assert.equal(posted(calls).length, 0);
|
||||
});
|
||||
|
||||
test("a run a person cancelled is not re-run", async () => {
|
||||
const data = cancelEverything(fixture(CANCELLED_RUN));
|
||||
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
|
||||
assert.equal(decision.rerun, false);
|
||||
assert.ok(decision.jobs.length >= 40);
|
||||
assert.equal(decision.jobs.some((job) => job.lost), false);
|
||||
assert.match(decision.reason, /failed jobs did not lose their runner/);
|
||||
assert.equal(posted(calls).length, 0);
|
||||
});
|
||||
|
||||
test("a cancelled run with one lost job and one real test failure is not re-run", async () => {
|
||||
const data = fixture(CANCELLED_RUN);
|
||||
const failing = data.jobs.find((job) => job.name === "ci / General tests (server (1/12))");
|
||||
failing.conclusion = "failure";
|
||||
data.annotations[failing.id] = [
|
||||
{ annotation_level: "failure", message: "AssertionError: expected 'queued' to be 'running'" },
|
||||
{ annotation_level: "failure", message: "Process completed with exit code 1." },
|
||||
];
|
||||
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
|
||||
assert.equal(decision.rerun, false);
|
||||
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [
|
||||
["ci / General tests (server (1/12))", false],
|
||||
["ci / General tests (server (8/12))", true],
|
||||
]);
|
||||
assert.match(decision.reason, /1 of 2 failed jobs did not lose their runner/);
|
||||
assert.equal(posted(calls).length, 0);
|
||||
});
|
||||
|
||||
test("a cancelled run is re-run at most once", async () => {
|
||||
const { decision, calls } = await runMain(fixture(CANCELLED_RUN), {
|
||||
mutateRun: (run) => ({ ...run, run_attempt: 2 }),
|
||||
}, { conclusion: "cancelled" });
|
||||
assert.equal(decision.rerun, false);
|
||||
assert.match(decision.reason, /attempt 2/);
|
||||
assert.deepEqual(calls.map((call) => call.path), [`/repos/paperclipai/paperclip/actions/runs/${CANCELLED_RUN}`]);
|
||||
});
|
||||
|
||||
test("a cancelled routing gate is left for a human re-run", async () => {
|
||||
const data = fixture(CANCELLED_RUN);
|
||||
const gate = data.jobs.find((job) => job.name === "ci / gate / Select trusted runner");
|
||||
gate.conclusion = "cancelled";
|
||||
data.annotations[gate.id] = [{ annotation_level: "failure", message: CANCELLED_MESSAGE }];
|
||||
data.logs[gate.id] = data.logs[Object.keys(data.logs)[0]];
|
||||
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
|
||||
assert.equal(decision.rerun, false);
|
||||
assert.match(decision.jobs.find((job) => job.id === gate.id).evidence, /routing gate/);
|
||||
assert.equal(posted(calls).length, 0);
|
||||
});
|
||||
|
||||
test("mixed runner loss and a real test failure: no re-run", async () => {
|
||||
const { decision, calls, summary } = await runMain(fixture(MIXED));
|
||||
assert.equal(decision.rerun, false);
|
||||
assert.equal(decision.jobs.length, 8);
|
||||
assert.equal(decision.jobs.length, 11);
|
||||
assert.deepEqual(decision.jobs.filter((job) => !job.lost).map((job) => job.name), ["ci / General tests (workspaces-a (1/2))"]);
|
||||
assert.equal(decision.jobs.find((job) => job.name === "ci / Verify Paperclip Runner (vitest package)").lost, true);
|
||||
assert.equal(posted(calls).length, 0);
|
||||
assert.match(decision.reason, /1 of 8 failed jobs did not lose their runner/);
|
||||
assert.match(decision.reason, /1 of 11 failed jobs did not lose their runner/);
|
||||
assert.match(summary, /\*\*No automatic re-run\*\*/);
|
||||
});
|
||||
|
||||
@@ -209,7 +324,7 @@ test("job listing follows Link pagination across pages", async () => {
|
||||
const pages = github.calls.filter((call) => call.path.endsWith("/attempts/1/jobs"));
|
||||
assert.deepEqual(pages.map((call) => new URLSearchParams(call.query).get("page")), [null, "2", "3"]);
|
||||
// Failed jobs on pages 2 and 3 are all classified.
|
||||
assert.equal(decision.jobs.length, 8);
|
||||
assert.equal(decision.jobs.length, 11);
|
||||
assert.equal(decision.rerun, false);
|
||||
});
|
||||
|
||||
@@ -236,9 +351,20 @@ test("classification is conservative about cancelled jobs", async () => {
|
||||
// A job timeout also cancels the step, but adds its own failure annotation.
|
||||
const timedOut = await classifyJob({ id: 1 }, {
|
||||
annotations: [{ annotation_level: "failure", message: "The job has exceeded the maximum execution time of 30m0s" }, ...cancelled],
|
||||
readLog: async () => { throw new Error("log must not be read"); },
|
||||
readLog: async () => "##[error]The job has exceeded the maximum execution time of 30m0s\n##[error]The operation was canceled.\n",
|
||||
});
|
||||
assert.equal(timedOut.lost, false);
|
||||
assert.match(timedOut.evidence, /maximum execution time/);
|
||||
const notStarted = await classifyJob({ id: 1, conclusion: "cancelled" }, {
|
||||
annotations: [],
|
||||
readLog: async () => { throw new Error("log must not be read"); },
|
||||
});
|
||||
assert.equal(notStarted.lost, false);
|
||||
const exitOnly = await classifyJob({ id: 1 }, {
|
||||
annotations: [{ annotation_level: "failure", message: "Process completed with exit code 1." }],
|
||||
readLog: async () => { throw new Error("log must not be read"); },
|
||||
});
|
||||
assert.equal(exitOnly.lost, false);
|
||||
// Notices and warnings are not failure evidence.
|
||||
const warning = await classifyJob({ id: 1 }, {
|
||||
annotations: [{ annotation_level: "warning", message: "The self-hosted runner lost communication with the server." }],
|
||||
@@ -250,7 +376,7 @@ test("classification is conservative about cancelled jobs", async () => {
|
||||
assert.equal(logShowsRunnerLoss(fixture(MIXED).logs["128078674151"]), true);
|
||||
});
|
||||
|
||||
test("a real failure before the runner was lost is not counted as runner loss", async () => {
|
||||
test("runner-loss evidence wins over other failure annotations in the same job", async () => {
|
||||
const lostMessage = "The self-hosted runner lost communication with the server. Verify the machine is running.";
|
||||
const noLog = async () => { throw new Error("log must not be read"); };
|
||||
// A test fails, then the runner is lost during an always() upload step.
|
||||
@@ -262,8 +388,8 @@ test("a real failure before the runner was lost is not counted as runner loss",
|
||||
],
|
||||
readLog: noLog,
|
||||
});
|
||||
assert.equal(testThenLost.lost, false);
|
||||
assert.match(testThenLost.evidence, /Test timed out/);
|
||||
assert.equal(testThenLost.lost, true);
|
||||
assert.match(testThenLost.evidence, /lost communication/);
|
||||
// The order of the annotations does not matter.
|
||||
const lostThenExit = await classifyJob({ id: 1 }, {
|
||||
annotations: [
|
||||
@@ -272,7 +398,22 @@ test("a real failure before the runner was lost is not counted as runner loss",
|
||||
],
|
||||
readLog: noLog,
|
||||
});
|
||||
assert.equal(lostThenExit.lost, false);
|
||||
assert.equal(lostThenExit.lost, true);
|
||||
const noisy = [
|
||||
{ annotation_level: "failure", message: "gateway_indexing_incomplete" },
|
||||
{ annotation_level: "failure", message: CANCELLED_MESSAGE },
|
||||
];
|
||||
const noisyLost = await classifyJob({ id: 1 }, {
|
||||
annotations: noisy,
|
||||
readLog: async () => "##[error]gateway_indexing_incomplete\n##[error]The runner has received a shutdown signal.\n",
|
||||
});
|
||||
assert.equal(noisyLost.lost, true);
|
||||
const noisyCancelled = await classifyJob({ id: 1 }, {
|
||||
annotations: noisy,
|
||||
readLog: async () => "##[error]gateway_indexing_incomplete\n##[error]The operation was canceled.\n",
|
||||
});
|
||||
assert.equal(noisyCancelled.lost, false);
|
||||
assert.match(noisyCancelled.evidence, /gateway_indexing_incomplete/);
|
||||
// Runner loss with a cancellation only is still runner loss.
|
||||
const lostAndCancelled = await classifyJob({ id: 1 }, {
|
||||
annotations: [
|
||||
@@ -292,14 +433,15 @@ test("summary cells cannot break the table", () => {
|
||||
assert.match(summary, /\| ci \/ a\\\|b \| no \| line one line \\\| two \|/);
|
||||
});
|
||||
|
||||
test("workflow acts only on failed first attempts of the canonical PR workflow", () => {
|
||||
test("workflow acts only on failed or cancelled first attempts of the canonical PR workflow", () => {
|
||||
const condition = workflow.match(/^ {4}if: >-\n((?: {6}.+\n)+)/m)?.[1].replace(/\s+/g, " ").trim();
|
||||
assert.equal(condition, [
|
||||
"github.repository == 'paperclipai/paperclip' &&",
|
||||
"github.repository_id == '1170821064' &&",
|
||||
"github.event.workflow_run.event == 'pull_request' &&",
|
||||
"github.event.workflow_run.path == '.github/workflows/pr.yml' &&",
|
||||
"github.event.workflow_run.conclusion == 'failure' &&",
|
||||
"(github.event.workflow_run.conclusion == 'failure' ||",
|
||||
"github.event.workflow_run.conclusion == 'cancelled') &&",
|
||||
"github.event.workflow_run.run_attempt == 1",
|
||||
].join(" "));
|
||||
assert.match(workflow, /^on:\n {2}workflow_run:\n {4}workflows: \[PR\]\n {4}types: \[completed\]$/m);
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
name: CI runner-loss rerun
|
||||
|
||||
# Self-hosted AWS runners are sometimes lost mid-job. When every failed job of
|
||||
# a first PR CI attempt lost its runner, re-run the failed jobs once. This is
|
||||
# Self-hosted AWS runners are sometimes lost mid-job. When every failed or
|
||||
# cancelled job of a first PR CI attempt lost its runner, re-run the failed
|
||||
# jobs once. A spot reclaim often concludes the run `cancelled`. This is
|
||||
# what a maintainer's "Re-run failed jobs" click does; github-actions[bot]
|
||||
# becomes attempt 2's triggering actor. See "Trusted PR Workflow" in
|
||||
# doc/DEVELOPING.md.
|
||||
@@ -31,7 +32,8 @@ jobs:
|
||||
github.repository_id == '1170821064' &&
|
||||
github.event.workflow_run.event == 'pull_request' &&
|
||||
github.event.workflow_run.path == '.github/workflows/pr.yml' &&
|
||||
github.event.workflow_run.conclusion == 'failure' &&
|
||||
(github.event.workflow_run.conclusion == 'failure' ||
|
||||
github.event.workflow_run.conclusion == 'cancelled') &&
|
||||
github.event.workflow_run.run_attempt == 1
|
||||
# GitHub-hosted on purpose: this job must not depend on the fleet it repairs.
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
+15
-8
@@ -187,14 +187,21 @@ AWS fleet. A stale admission roster can allocate only the short metadata gate;
|
||||
the fresh CODEOWNERS check still denies PR execution for a removed owner.
|
||||
|
||||
AWS runners are sometimes lost mid-job. `ci-runner-loss-rerun.yml` re-runs the
|
||||
failed jobs of a PR's first CI attempt once when every failed job lost its
|
||||
runner. A lost runner is recognized by the check-run annotation "lost
|
||||
communication with the server" or "received a shutdown signal". When the only
|
||||
annotation is "The operation was canceled.", the job log is checked for the
|
||||
same messages. The `verify` and `e2e` aggregates are not counted. Any other
|
||||
failure, a failed routing gate, or a PR whose head has moved prevents the
|
||||
automatic re-run. The workflow job summary lists each failed job and its
|
||||
evidence. Like a maintainer's "Re-run failed jobs", the re-run reuses the
|
||||
failed jobs of a PR's first CI attempt once when every failed or cancelled job
|
||||
lost its runner. A spot reclaim often ends the job, and the run, as
|
||||
`cancelled`. A lost runner is recognized by the check-run annotation "lost
|
||||
communication with the server" or "received a shutdown signal". When a job has
|
||||
the annotation "The operation was canceled.", the job log is checked for the
|
||||
same messages. This evidence counts even if the job also has other error
|
||||
annotations, so a test that really failed just before a reclaim gets one
|
||||
wasted re-run. A cancelled job without this evidence, such as a job a person
|
||||
cancelled, counts as a real failure. The `verify` and `e2e` aggregates are not
|
||||
counted. Any other failure, a failed or cancelled routing gate, or a PR whose
|
||||
head has moved prevents the automatic re-run. The head check runs before any
|
||||
job is classified, so a run cancelled by a newer push costs no log reads. It
|
||||
runs again just before the re-run, because a re-run of an old run joins the
|
||||
PR's `cancel-in-progress` concurrency group and would cancel the newer CI. The
|
||||
workflow job summary lists each failed or cancelled job and its evidence. Like a maintainer's "Re-run failed jobs", the re-run reuses the
|
||||
gate's successful outputs, including its AWS admission. Attempt 2's
|
||||
triggering actor is `github-actions[bot]`, so `profile-ci` does not export
|
||||
that attempt. Later attempts are never re-run automatically.
|
||||
|
||||
Reference in new issue
Block a user