fix(ci): re-run cancelled runs that lost a runner, and let runner-loss evidence win (#21291)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - PR CI runs about 50 jobs per run on AWS spot instances. AWS can
reclaim a machine mid-job, and the job dies.
> - `ci-runner-loss-rerun.yml` re-runs such a run once, automatically,
when every failed job lost its runner.
> - In a 17-hour sample, 42 runs lost a runner and were still the PR's
current head. The automation rescued only 14.
> - Two gaps caused most of the misses. A spot reclaim on RunsOn v3.2.3
usually ends the job, and the run, as `cancelled`; the workflow and
script only look at `failure`. Also, the Runner support suite always
carries `gateway_indexing_incomplete` error annotations from a passing
test, so a lost runner there reads as a real failure.
> - On-demand fallback for the fleet was turned off on 2026-11-13, so
reclaims will happen more often. About 20% of runs are expected to lose
at least one job.
> - This pull request makes the re-run act on cancelled runs and jobs,
but only with positive runner-loss evidence, and lets that evidence win
over other error annotations.
> - The benefit is fewer manual "Re-run failed jobs" clicks: an
estimated 34 of 42 instead of 14 of 42 in the sample.

## Linked Issues or Issue Description

No public issue exists.

**What happened?**
Run
[43285064435](https://github.com/paperclipai/paperclip/actions/runs/43285064435)
lost the runner of `ci / General tests (server (8/12))`. The job
concluded `cancelled` (its log has `##[error]The runner has received a
shutdown signal`). The run concluded `cancelled`, so the re-run
workflow's `if:` skipped it.

Run
[43325855069](https://github.com/paperclipai/paperclip/actions/runs/43325855069)
lost the runners of `ci / Runner support suite` and `ci / General tests
(server (2/12))`. Both logs have the shutdown signal. The re-run bot
([decision
log](https://github.com/paperclipai/paperclip/actions/runs/43327873358))
said "1 of 1 failed jobs did not lose their runner", with evidence
`gateway_indexing_incomplete`. It ignored the cancelled job. It read the
annotation as a real failure.

**Expected behavior**
Both runs get one automatic re-run of their failed jobs.

**Steps to reproduce**
The new tests replay trimmed snapshots of both runs
(`.github/scripts/tests/fixtures/runner-loss-rerun/run-43285064435.json`
and `run-43325855069.json`) through the script.

In a 17-hour sample on 2026-11-12, 42 runs lost a runner and were still
the PR's current head. Only 14 were re-run automatically. 16 stayed red
because the run concluded `cancelled`. 4 stayed red because an error
annotation in the lost job was read as a test failure.

**Additional context**
The `gateway_indexing_incomplete` annotation does not come from the
shutdown. `scheduled-gateway-indexing.test.ts` prints `::error` lines
while it passes, 84 seconds before the shutdown signal in this run.
Attempt 2 of the same run passed with the same two failure annotations.
So, before this change, a lost runner in the Runner support suite could
never be re-run.

## What Changed

- `ci-runner-loss-rerun.yml`: the job also runs when the PR run
concluded `cancelled`. The trigger type, `permissions`, the repository,
path, event and attempt-1 conditions do not change.
- `runner-loss-rerun.mjs`, `eligibleRun`: accepts `completed/cancelled`
as well as `completed/failure`.
- `runner-loss-rerun.mjs`, `decide()`: classifies jobs that concluded
`cancelled` as well as `failure`. It still skips the `ci / verify` and
`ci / e2e` aggregates, and a failed or cancelled routing gate still
blocks the re-run.
- `decide()` now runs `staleReason` (open PR, current head, no newer
run) before it reads any annotation or log, and again just before the
re-run, as on master. A run cancelled by a new push has about 40
cancelled jobs; it now costs two API listings and no log reads. The
final check closes the window where a push lands during the log reads: a
re-run of the old run would join the PR's `cancel-in-progress`
concurrency group (`pr-<number>` in `pr-trusted.yml`) and cancel the new
commit's CI.
- Following CONTRIBUTING.md ("Do not submit a PR that includes code
comments"), this PR adds no new code comments. The reasons for each rule
are in this description and in `doc/DEVELOPING.md`.
- `classifyJob`: runner-loss evidence wins. A lost-communication
annotation means lost. A cancellation annotation plus a shutdown signal
in the log means lost, even if the job also has other error annotations.
A cancelled job without this evidence (for example, a person cancelled
the run) counts as a real failure and blocks the re-run.
- Fixtures: two new trimmed snapshots from the example runs (jobs with
`id`, `name`, `status`, `conclusion`; failure annotations; `##[error]`
log lines only). The two older fixtures get the real annotations and log
error lines of their cancelled jobs. Those jobs were spot-reclaimed too
(each log has the shutdown signal), but the old code ignored them.
- `doc/DEVELOPING.md`: the runner-loss paragraph describes the new
rules.

## Verification

- Red first. Against the unchanged script, the new tests fail for the
reasons above: 43285064435 gives `run is completed/cancelled, not
completed/failure`; 43325855069 gives `1 of 1 failed jobs did not lose
their runner` with evidence `gateway_indexing_incomplete`, the same
decision the bot logged.
- Green: `node --test .github/scripts/tests/runner-loss-rerun.test.mjs`:
25 of 25 pass.
- Push during log reads: the PR head moves after the first log read.
With the final freshness check removed, this test fails (the script
re-runs); with it, there is no re-run and the reason is `no open pull
request still has this run's head commit`.
- Guard cases, all with no re-run and no POST:
- Superseded run: 52 jobs cancelled, the PR head moved. The test asserts
zero annotation and log reads.
- Person cancelled the run: cancelled jobs with only "The operation was
canceled." in the annotation and log, and some never-started jobs with
no annotation.
- Mixed: one cancelled lost job and one real test failure (`1 of 2
failed jobs did not lose their runner`).
- Once only: a cancelled run on attempt 2 is refused after one API read.
The workflow condition test still requires `run_attempt == 1`.
  - Cancelled routing gate: left for a person.
- Policy job command: `node --test '.github/scripts/tests/*.test.mjs'`:
806 of 806 pass.
- `pnpm verify`: exit 2 ("Global CI input changed:
.github/scripts/runner-loss-rerun.mjs"). It selects no local Vitest
files and asks for required CI coverage. I did not run `pnpm verify
--full`; required CI on this PR supplies that coverage.
- This workflow runs on `workflow_run` from the default branch. This
PR's own CI cannot exercise the new behavior. The fixture tests are the
evidence. I did not dry-run the script against live runs.

## Risks

- A test that really fails, and then a spot reclaim of the same job a
minute later, now gets one wasted re-run. Attempt 2 then fails for real.
It cannot loop: the workflow `if:` and `eligibleRun` both accept attempt
1 only.
- A run a person cancels on the current head now starts the re-run job.
The job reads the annotations, and the log of each job with a
cancellation annotation, and then does not re-run. This costs some API
calls on a GitHub-hosted runner.
- A concurrency cancel by a label event that does not request CI (the
newer run is `skipped`) passes the head check. Its cancelled jobs have
no runner-loss evidence, so it is not re-run.
- Trust: no change to `permissions`, the trigger, the accepted
repository, event, workflow path, attempt, or the routing-gate rule. A
cancelled run reuses its attempt-1 gate output, the same as a failed run
does today.
- Estimated effect, from a 17-hour sample: 14 of 42 rescued before; 16
cancelled runs and 4 runs with error-annotation noise become eligible
(about 34 of 42). The other 8 stay skipped on purpose (a real test
failure in the same run, a person re-ran first, or the PR closed).

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), Claude Code agent. Model ID claude-opus-5-5, with
tool use (shell, GitHub CLI) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Fable 5.2 authored and GitHub committed 2026-11-12 22:31:50 -06:00
1 parent af1c685f5f
commit c4f639ed1b
8 files changed
+1073 -43

No files matched your search

+14 -14
View File
@@ -3,7 +3,7 @@
* runner-loss-rerun.mjs
*
* Re-runs the failed jobs of a first PR CI attempt once, but only when every
* failed job failed because its runner was lost. Runs from
* failed or cancelled job ended because its runner was lost. Runs from
* .github/workflows/ci-runner-loss-rerun.yml on workflow_run. It reads GitHub
* API metadata only and never touches the PR's code or artifacts.
*
@@ -26,6 +26,7 @@ export const AGGREGATE_JOBS = new Set(["ci / verify", "ci / e2e"]);
// ci-routing.yml checks the attempt's triggering actor. A bot-triggered attempt
// would move the whole run to GitHub-hosted runners, so leave it to a human.
export const GATE_JOB_PREFIX = "ci / gate / ";
export const RERUN_CONCLUSIONS = new Set(["failure", "cancelled"]);
// A shutdown signal often reaches the check run only as this annotation; the
// signal itself is logged as an error line in the job log.
const CANCELLED_MESSAGE = "The operation was canceled.";
@@ -80,20 +81,14 @@ export function logShowsRunnerLoss(log) {
export async function classifyJob(job, { annotations, readLog }) {
const failures = annotations.filter((annotation) => annotation.annotation_level === "failure");
const isRunnerLoss = (annotation) => RUNNER_LOSS_PATTERN.test(annotation.message ?? "");
// A step can fail for a real reason before the runner is lost (for example a
// test failure, then a loss during an always() upload step). Any failure
// annotation other than runner loss or cancellation marks a real failure.
const real = failures.find((annotation) => !isRunnerLoss(annotation) && annotation.message !== CANCELLED_MESSAGE);
if (real) return { lost: false, evidence: firstLine(real.message) };
const match = failures.find(isRunnerLoss);
if (match) return { lost: true, evidence: firstLine(match.message) };
// Only cancellations are left: the shutdown signal, if any, is in the log.
if (failures.length > 0) {
if (logShowsRunnerLoss(await readLog(job))) {
return { lost: true, evidence: "job log: runner received a shutdown signal or lost communication" };
}
return { lost: false, evidence: `${CANCELLED_MESSAGE} (no runner-loss error in the job log)` };
if (failures.some((annotation) => annotation.message === CANCELLED_MESSAGE) && logShowsRunnerLoss(await readLog(job))) {
return { lost: true, evidence: "job log: runner received a shutdown signal or lost communication" };
}
const real = failures.find((annotation) => annotation.message !== CANCELLED_MESSAGE);
if (real) return { lost: false, evidence: firstLine(real.message) };
if (failures.length > 0) return { lost: false, evidence: `${CANCELLED_MESSAGE} (no runner-loss error in the job log)` };
return { lost: false, evidence: "no failure annotation" };
}
@@ -106,7 +101,9 @@ export function eligibleRun(run) {
if (run.repository?.full_name !== REPOSITORY || run.repository?.id !== REPOSITORY_ID) return "run is not in the canonical repository";
if (run.path !== PR_WORKFLOW_PATH) return `workflow is ${run.path}, not ${PR_WORKFLOW_PATH}`;
if (run.event !== "pull_request") return `event is ${run.event}, not pull_request`;
if (run.status !== "completed" || run.conclusion !== "failure") return `run is ${run.status}/${run.conclusion}, not completed/failure`;
if (run.status !== "completed" || !RERUN_CONCLUSIONS.has(run.conclusion)) {
return `run is ${run.status}/${run.conclusion}, not completed/failure or completed/cancelled`;
}
// One automatic re-run only. A higher attempt also means someone re-ran it.
if (run.run_attempt !== 1) return `run is on attempt ${run.run_attempt}; only attempt 1 is re-run automatically`;
return undefined;
@@ -137,9 +134,12 @@ export async function decide(client, run) {
if (ineligible) return { ...decision, reason: ineligible };
const jobs = await client.paginate(`/repos/${REPOSITORY}/actions/runs/${run.id}/attempts/1/jobs?per_page=100`, "jobs");
const failed = jobs.filter((job) => job.conclusion === "failure" && !AGGREGATE_JOBS.has(job.name));
const failed = jobs.filter((job) => RERUN_CONCLUSIONS.has(job.conclusion) && !AGGREGATE_JOBS.has(job.name));
if (failed.length === 0) return { ...decision, reason: "no failed jobs other than the verify/e2e aggregates" };
const staleBefore = await staleReason(client, run);
if (staleBefore) return { ...decision, reason: staleBefore };
for (const job of failed) {
if (job.name.startsWith(GATE_JOB_PREFIX)) {
decision.jobs.push({ id: job.id, name: job.name, lost: false, evidence: "routing gate failed; a bot re-run would lose AWS routing" });
@@ -1042,7 +1042,7 @@
"end_line": 123,
"annotation_level": "failure",
"title": "[paperclipai] src/__tests__/company-import-transfer.test.ts > resolveChunkedImportZip > takes the chunked path for a small zip whose entries inflate past the threshold",
"message": "Error: Test timed out in 5000ms.\nIf this is a long-running test, pass a timeout value as the last argument or configure it globally with \"testTimeout\".\n ❯ src/__tests__/company-import-transfer.test.ts:123:3\n\n",
"message": "Error: Test timed out in 5000ms.\nIf this is a long-running test, pass a timeout value as the last argument or configure it globally with \"testTimeout\".\n \u276f src/__tests__/company-import-transfer.test.ts:123:3\n\n",
"raw_details": ""
},
{
@@ -1051,7 +1051,7 @@
"end_line": 280,
"annotation_level": "failure",
"title": "[paperclipai] src/__tests__/company-import-export-e2e.test.ts > paperclipai company import/export e2e",
"message": "Error: Timed out waiting for the child server recorded in /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/paperclip-home/instances/company-cli-e2e/runtime-info.json.\nstdout:\nUpdate available: 2026.1023.0 — run `paperclipai update`\n┌ paperclipai run \n│\n│ Home: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/paperclip-home\n│\n│ Instance: company-cli-e2e\n│\n│ Config: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/config.json\n│\n◇ Running doctor checks...\nUpdate available: 2026.1023.0 — run `paperclipai update`\n\n██████╗ █████╗ ██████╗ ███████╗██████╗ ██████╗██╗ ██╗██████╗ \n██╔══██╗██╔══██╗██╔══██╗██╔════╝██╔══██╗██╔════╝██║ ██║██╔══██╗\n██████╔╝███████║██████╔╝█████╗ ██████╔╝██║ ██║ ██║██████╔╝\n██╔═══╝ ██╔══██║██╔═══╝ ██╔══╝ ██╔══██╗██║ ██║ ██║██╔═══╝ \n██║ ██║ ██║██║ ███████╗██║ ██║╚██████╗███████╗██║██║ \n╚═╝ ╚═╝ ╚═╝╚═╝ ╚══════╝╚═╝ ╚═╝ ╚═════╝╚══════╝╚═╝╚═╝ \n ───────────────────────────────────────────────────────\n The app people use to manage AI agents for work\n\n┌ paperclip doctor \n│\n│ ✓ Config file: Valid config at /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/config.json\n│\n│ ✓ Deployment/auth mode: local_trusted mode is configured for loopback-only access\n│\n│ ✗ Agent JWT secret: PAPERCLIP_AGENT_JWT_SECRET missing from environment and /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/.env\n│\n│ Run with --repair to create /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/.env containing PAPERCLIP_AGENT_JWT_SECRET\n│\n◆ Repaired: Agent JWT secret\n│\n│ ✓ Agent JWT secret: PAPERCLIP_AGENT_JWT_SECRET is set in environment\n│\n│ ! Secrets adapter: Secrets key file does not exist yet: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/secrets/master.key; strict secret mode is disabled for postgres deployment\n│\n│ Run with --repair to create a local encrypted secrets key file. Consider enabling secrets.strictMode\n│\n◆ Repaired: Secrets adapter\n│\n│ ! Secrets adapter: Local encrypted provider configured with key file /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/secrets/master.key; strict secret mode is disabled for postgres deployment\n│\n│ Consider enabling secrets.strictMode\n│\n│ ✓ Storage: Local disk storage is writable: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/storage\n│\n│ ✓ Database: PostgreSQL connection successful\n│\n│ ✓ LLM provider: No LLM provider configured (optional)\n│\n│ ✓ Log directory: Log directory is writable: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/logs\n│\n│ ! Server port: Port 37925 is already in use\n│\n│ Check what's using port 37925 with: lsof -i :37925\n│\n│ ✓ Node.js runtime: Node.js 24.22.2\n│\n│ ✓ Managed install: Not present (optional for npx, global npm, and source-checkout usage)\n│\n│ ✓ Background service: No usable systemd user manager was detected (common in containers and WSL1). Use paperclipai run instead.\n│\n◇ Summary ───────────────╮\n│ │\n│ 10 passed, 2 warnings │\n│ │\n├─────────────────────────╯\n│\n└ All critical checks passed with some warnings.\n\n│\n◇ Start",
"message": "Error: Timed out waiting for the child server recorded in /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/paperclip-home/instances/company-cli-e2e/runtime-info.json.\nstdout:\nUpdate available: 2026.1023.0 \u2014 run `paperclipai update`\n\u250c paperclipai run \n\u2502\n\u2502 Home: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/paperclip-home\n\u2502\n\u2502 Instance: company-cli-e2e\n\u2502\n\u2502 Config: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/config.json\n\u2502\n\u25c7 Running doctor checks...\nUpdate available: 2026.1023.0 \u2014 run `paperclipai update`\n\n\u2588\u2588\u2588\u2588\u2588\u2588\u2557 \u2588\u2588\u2588\u2588\u2588\u2557 \u2588\u2588\u2588\u2588\u2588\u2588\u2557 \u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2557\u2588\u2588\u2588\u2588\u2588\u2588\u2557 \u2588\u2588\u2588\u2588\u2588\u2588\u2557\u2588\u2588\u2557 \u2588\u2588\u2557\u2588\u2588\u2588\u2588\u2588\u2588\u2557 \n\u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2557\u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2557\u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2557\u2588\u2588\u2554\u2550\u2550\u2550\u2550\u255d\u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2557\u2588\u2588\u2554\u2550\u2550\u2550\u2550\u255d\u2588\u2588\u2551 \u2588\u2588\u2551\u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2557\n\u2588\u2588\u2588\u2588\u2588\u2588\u2554\u255d\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2551\u2588\u2588\u2588\u2588\u2588\u2588\u2554\u255d\u2588\u2588\u2588\u2588\u2588\u2557 \u2588\u2588\u2588\u2588\u2588\u2588\u2554\u255d\u2588\u2588\u2551 \u2588\u2588\u2551 \u2588\u2588\u2551\u2588\u2588\u2588\u2588\u2588\u2588\u2554\u255d\n\u2588\u2588\u2554\u2550\u2550\u2550\u255d \u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2551\u2588\u2588\u2554\u2550\u2550\u2550\u255d \u2588\u2588\u2554\u2550\u2550\u255d \u2588\u2588\u2554\u2550\u2550\u2588\u2588\u2557\u2588\u2588\u2551 \u2588\u2588\u2551 \u2588\u2588\u2551\u2588\u2588\u2554\u2550\u2550\u2550\u255d \n\u2588\u2588\u2551 \u2588\u2588\u2551 \u2588\u2588\u2551\u2588\u2588\u2551 \u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2557\u2588\u2588\u2551 \u2588\u2588\u2551\u255a\u2588\u2588\u2588\u2588\u2588\u2588\u2557\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2557\u2588\u2588\u2551\u2588\u2588\u2551 \n\u255a\u2550\u255d \u255a\u2550\u255d \u255a\u2550\u255d\u255a\u2550\u255d \u255a\u2550\u2550\u2550\u2550\u2550\u2550\u255d\u255a\u2550\u255d \u255a\u2550\u255d \u255a\u2550\u2550\u2550\u2550\u2550\u255d\u255a\u2550\u2550\u2550\u2550\u2550\u2550\u255d\u255a\u2550\u255d\u255a\u2550\u255d \n \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n The app people use to manage AI agents for work\n\n\u250c paperclip doctor \n\u2502\n\u2502 \u2713 Config file: Valid config at /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/config.json\n\u2502\n\u2502 \u2713 Deployment/auth mode: local_trusted mode is configured for loopback-only access\n\u2502\n\u2502 \u2717 Agent JWT secret: PAPERCLIP_AGENT_JWT_SECRET missing from environment and /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/.env\n\u2502\n\u2502 Run with --repair to create /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/config/.env containing PAPERCLIP_AGENT_JWT_SECRET\n\u2502\n\u25c6 Repaired: Agent JWT secret\n\u2502\n\u2502 \u2713 Agent JWT secret: PAPERCLIP_AGENT_JWT_SECRET is set in environment\n\u2502\n\u2502 ! Secrets adapter: Secrets key file does not exist yet: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/secrets/master.key; strict secret mode is disabled for postgres deployment\n\u2502\n\u2502 Run with --repair to create a local encrypted secrets key file. Consider enabling secrets.strictMode\n\u2502\n\u25c6 Repaired: Secrets adapter\n\u2502\n\u2502 ! Secrets adapter: Local encrypted provider configured with key file /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/secrets/master.key; strict secret mode is disabled for postgres deployment\n\u2502\n\u2502 Consider enabling secrets.strictMode\n\u2502\n\u2502 \u2713 Storage: Local disk storage is writable: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/storage\n\u2502\n\u2502 \u2713 Database: PostgreSQL connection successful\n\u2502\n\u2502 \u2713 LLM provider: No LLM provider configured (optional)\n\u2502\n\u2502 \u2713 Log directory: Log directory is writable: /tmp/pv-KMcoQP/t/paperclip-company-cli-e2e-8VX8vL/logs\n\u2502\n\u2502 ! Server port: Port 37925 is already in use\n\u2502\n\u2502 Check what's using port 37925 with: lsof -i :37925\n\u2502\n\u2502 \u2713 Node.js runtime: Node.js 24.22.2\n\u2502\n\u2502 \u2713 Managed install: Not present (optional for npx, global npm, aLine truncated
"raw_details": ""
},
{
@@ -1085,10 +1085,46 @@
"message": "Process completed with exit code 1.",
"raw_details": ""
}
],
"128079212951": [
{
"path": ".github",
"start_line": 130,
"end_line": 130,
"annotation_level": "failure",
"title": "",
"message": "The operation was canceled.",
"raw_details": ""
}
],
"128079213109": [
{
"path": ".github",
"start_line": 130,
"end_line": 130,
"annotation_level": "failure",
"title": "",
"message": "The operation was canceled.",
"raw_details": ""
}
],
"128079213211": [
{
"path": ".github",
"start_line": 133,
"end_line": 133,
"annotation_level": "failure",
"title": "",
"message": "The operation was canceled.",
"raw_details": ""
}
]
},
"logs": {
"128078674151": "2026-11-06T18:42:31.0413375Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T18:42:43.5529901Z ##[error]The operation was canceled.\n2026-11-06T18:42:43.5656287Z Cleaning up orphan processes\n"
"128078674151": "2026-11-06T18:42:31.0413375Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T18:42:43.5529901Z ##[error]The operation was canceled.\n2026-11-06T18:42:43.5656287Z Cleaning up orphan processes\n",
"128079212951": "2026-11-06T18:42:41.1643719Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T18:42:53.6869870Z ##[error]The operation was canceled.\n",
"128079213109": "2026-11-06T18:42:35.0486149Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T18:42:47.5624672Z ##[error]The operation was canceled.\n",
"128079213211": "2026-11-06T18:42:38.0453932Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T18:42:50.5615051Z ##[error]The operation was canceled.\n"
},
"total_count": 48
}
@@ -970,10 +970,130 @@
"message": "Process completed with exit code 1.",
"raw_details": ""
}
],
"128101015916": [
{
"path": ".github",
"start_line": 130,
"end_line": 130,
"annotation_level": "failure",
"title": "",
"message": "The operation was canceled.",
"raw_details": ""
}
],
"128101015978": [
{
"path": ".github",
"start_line": 130,
"end_line": 130,
"annotation_level": "failure",
"title": "",
"message": "The operation was canceled.",
"raw_details": ""
}
],
"128101016153": [
{
"path": ".github",
"start_line": 133,
"end_line": 133,
"annotation_level": "failure",
"title": "",
"message": "The operation was canceled.",
"raw_details": ""
}
],
"128101016200": [
{
"path": ".github",
"start_line": 133,
"end_line": 133,
"annotation_level": "failure",
"title": "",
"message": "The operation was canceled.",
"raw_details": ""
}
],
"128101016304": [
{
"path": ".github",
"start_line": 133,
"end_line": 133,
"annotation_level": "failure",
"title": "",
"message": "The operation was canceled.",
"raw_details": ""
}
],
"128101016340": [
{
"path": ".github",
"start_line": 133,
"end_line": 133,
"annotation_level": "failure",
"title": "",
"message": "The operation was canceled.",
"raw_details": ""
}
],
"128101016402": [
{
"path": ".github",
"start_line": 133,
"end_line": 133,
"annotation_level": "failure",
"title": "",
"message": "The operation was canceled.",
"raw_details": ""
}
],
"128101016432": [
{
"path": ".github",
"start_line": 133,
"end_line": 133,
"annotation_level": "failure",
"title": "",
"message": "The operation was canceled.",
"raw_details": ""
}
],
"128101016453": [
{
"path": ".github",
"start_line": 133,
"end_line": 133,
"annotation_level": "failure",
"title": "",
"message": "The operation was canceled.",
"raw_details": ""
}
],
"128101016461": [
{
"path": ".github",
"start_line": 133,
"end_line": 133,
"annotation_level": "failure",
"title": "",
"message": "The operation was canceled.",
"raw_details": ""
}
]
},
"logs": {
"128101016102": "2026-11-06T19:24:29.1450769Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:29.1898916Z ##[error]The operation was canceled.\n2026-11-06T19:24:29.1977091Z Cleaning up orphan processes\n"
"128101016102": "2026-11-06T19:24:29.1450769Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:29.1898916Z ##[error]The operation was canceled.\n2026-11-06T19:24:29.1977091Z Cleaning up orphan processes\n",
"128101015916": "2026-11-06T19:24:29.1391073Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.6585509Z ##[error]The operation was canceled.\n",
"128101015978": "2026-11-06T19:24:29.1982327Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.7170843Z ##[error]The operation was canceled.\n",
"128101016153": "2026-11-06T19:24:29.1786229Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.6972420Z ##[error]The operation was canceled.\n",
"128101016200": "2026-11-06T19:24:29.1721198Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.6975527Z ##[error]The operation was canceled.\n",
"128101016304": "2026-11-06T19:24:29.1377822Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.6673266Z ##[error]The operation was canceled.\n",
"128101016340": "2026-11-06T19:24:29.1341317Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.6500655Z ##[error]The operation was canceled.\n",
"128101016402": "2026-11-06T19:24:29.1167345Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.6423716Z ##[error]The operation was canceled.\n",
"128101016432": "2026-11-06T19:24:29.0692803Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.5851488Z ##[error]The operation was canceled.\n",
"128101016453": "2026-11-06T19:24:29.2339015Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.7586244Z ##[error]The operation was canceled.\n",
"128101016461": "2026-11-06T19:24:29.1057916Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-06T19:24:41.6269036Z ##[error]The operation was canceled.\n"
},
"total_count": 48
}
@@ -0,0 +1,360 @@
{
"source": "GET /repos/paperclipai/paperclip/actions/runs/43285064435/attempts/1 (+ jobs, annotations and job-log error lines), trimmed",
"run": {
"id": 43285064435,
"name": "PR",
"head_sha": "cd287efba2ff0c557f0c63fc78d5ad1d48f192e8",
"path": ".github/workflows/pr.yml",
"event": "pull_request",
"status": "completed",
"conclusion": "cancelled",
"html_url": "https://github.com/paperclipai/paperclip/actions/runs/43285064435",
"run_attempt": 1,
"repository": {
"id": 1170821064,
"full_name": "paperclipai/paperclip"
},
"head_repository": {
"id": 1170821064,
"full_name": "paperclipai/paperclip"
}
},
"jobs": [
{
"id": 131416164450,
"name": "ci / gate / Select trusted runner",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416492147,
"name": "ci / policy",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416492206,
"name": "ci / Typecheck + Release Registry",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416492248,
"name": "ci / Verify Paperclip Runner (static checks)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416492288,
"name": "ci / Runner support suite",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416492337,
"name": "ci / Verify Paperclip Runner (rust)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416492390,
"name": "ci / Build",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416492402,
"name": "ci / Canary Dry Run",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416492420,
"name": "ci / Verify Paperclip Runner (vitest package (3/3))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416492421,
"name": "ci / Sandbox provider tests",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416492424,
"name": "ci / Docker context integrity",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416492430,
"name": "ci / Verify Paperclip Runner (vitest native server)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416492442,
"name": "ci / Verify Paperclip Runner (vitest package (2/3))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416492489,
"name": "ci / Verify Paperclip Runner (vitest package (1/3))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416492538,
"name": "ci / e2e shard (5/8)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416492565,
"name": "ci / e2e shard (1/8)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416492568,
"name": "ci / e2e shard (2/8)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416492582,
"name": "ci / e2e shard (6/8)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416492588,
"name": "ci / e2e shard (7/8)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416492592,
"name": "ci / e2e shard (4/8)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416492708,
"name": "ci / e2e shard (3/8)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416493348,
"name": "ci / e2e shard (8/8)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416493582,
"name": "ci / Verify serialized server suites (3/9)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416493601,
"name": "ci / Verify serialized server suites (4/9)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416493630,
"name": "ci / Verify serialized server suites (2/9)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416493633,
"name": "ci / Verify serialized server suites (1/9)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416493669,
"name": "ci / Verify serialized server suites (5/9)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416493759,
"name": "ci / Verify serialized server suites (7/9)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416493766,
"name": "ci / Verify serialized server suites (9/9)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416493829,
"name": "ci / Verify serialized server suites (8/9)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416493947,
"name": "ci / General tests (server (5/12))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416494052,
"name": "ci / General tests (server (1/12))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416494077,
"name": "ci / General tests (server (3/12))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416494102,
"name": "ci / General tests (server (7/12))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416494186,
"name": "ci / General tests (server (4/12))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416494194,
"name": "ci / General tests (workspaces-c)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416494201,
"name": "ci / General tests (chat (3/3))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416494251,
"name": "ci / General tests (server (12/12))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416494281,
"name": "ci / General tests (server (2/12))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416494286,
"name": "ci / General tests (chat (1/3))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416494333,
"name": "ci / General tests (workspaces-a (1/2))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416494339,
"name": "ci / General tests (server (10/12))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416494340,
"name": "ci / General tests (server (9/12))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416494376,
"name": "ci / General tests (chat (2/3))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416494446,
"name": "ci / General tests (server (6/12))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416494459,
"name": "ci / General tests (server (8/12))",
"status": "completed",
"conclusion": "cancelled"
},
{
"id": 131416494489,
"name": "ci / General tests (workspaces-b)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416494538,
"name": "ci / Verify serialized server suites (6/9)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416494546,
"name": "ci / General tests (server (11/12))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416494552,
"name": "ci / General tests (workspaces-a (2/2))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131416494939,
"name": "ci / Select Vitest files",
"status": "completed",
"conclusion": "skipped"
},
{
"id": 131421415244,
"name": "ci / e2e",
"status": "completed",
"conclusion": "success"
},
{
"id": 131424041299,
"name": "ci / verify",
"status": "completed",
"conclusion": "failure"
}
],
"annotations": {
"131416494459": [
{
"annotation_level": "failure",
"message": "The operation was canceled."
}
],
"131424041299": [
{
"annotation_level": "failure",
"message": "Process completed with exit code 1."
}
]
},
"logs": {
"131416494459": "2026-11-12T19:03:20.1001059Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-12T19:03:32.6527460Z ##[error]The operation was canceled.\n"
},
"total_count": 53
}
@@ -0,0 +1,363 @@
{
"source": "GET /repos/paperclipai/paperclip/actions/runs/43325855069/attempts/1 (+ jobs, annotations and job-log error lines), trimmed",
"run": {
"id": 43325855069,
"name": "PR",
"head_sha": "4b4f6b2ad0fac4ede9f4e20a56e3c93fac5577d5",
"path": ".github/workflows/pr.yml",
"event": "pull_request",
"status": "completed",
"conclusion": "failure",
"html_url": "https://github.com/paperclipai/paperclip/actions/runs/43325855069",
"run_attempt": 1,
"repository": {
"id": 1170821064,
"full_name": "paperclipai/paperclip"
},
"head_repository": {
"id": 1170821064,
"full_name": "paperclipai/paperclip"
}
},
"jobs": [
{
"id": 131557987038,
"name": "ci / gate / Select trusted runner",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558292300,
"name": "ci / Docker context integrity",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558292349,
"name": "ci / Canary Dry Run",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558292381,
"name": "ci / Runner support suite",
"status": "completed",
"conclusion": "failure"
},
{
"id": 131558292405,
"name": "ci / Typecheck + Release Registry",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558292466,
"name": "ci / policy",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558292573,
"name": "ci / Build",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558292627,
"name": "ci / Verify Paperclip Runner (vitest package (1/3))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558292664,
"name": "ci / Verify Paperclip Runner (static checks)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558292669,
"name": "ci / Verify Paperclip Runner (vitest package (2/3))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558292679,
"name": "ci / e2e shard (8/8)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558292686,
"name": "ci / Verify Paperclip Runner (vitest native server)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558292707,
"name": "ci / Verify Paperclip Runner (vitest package (3/3))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558292731,
"name": "ci / e2e shard (5/8)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558292738,
"name": "ci / e2e shard (2/8)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558292751,
"name": "ci / e2e shard (1/8)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558292779,
"name": "ci / e2e shard (4/8)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558292841,
"name": "ci / e2e shard (3/8)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558292887,
"name": "ci / e2e shard (7/8)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293043,
"name": "ci / e2e shard (6/8)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293328,
"name": "ci / Verify serialized server suites (4/9)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293330,
"name": "ci / Verify serialized server suites (2/9)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293380,
"name": "ci / Verify serialized server suites (7/9)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293389,
"name": "ci / Verify serialized server suites (3/9)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293399,
"name": "ci / Verify serialized server suites (8/9)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293420,
"name": "ci / Verify serialized server suites (1/9)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293433,
"name": "ci / Verify serialized server suites (5/9)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293450,
"name": "ci / Verify serialized server suites (6/9)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293683,
"name": "ci / Verify serialized server suites (9/9)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293690,
"name": "ci / General tests (server (12/12))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293701,
"name": "ci / General tests (server (11/12))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293710,
"name": "ci / General tests (server (8/12))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293784,
"name": "ci / General tests (server (3/12))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293818,
"name": "ci / General tests (server (2/12))",
"status": "completed",
"conclusion": "cancelled"
},
{
"id": 131558293819,
"name": "ci / General tests (workspaces-a (1/2))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293825,
"name": "ci / General tests (server (4/12))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293835,
"name": "ci / General tests (server (5/12))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293844,
"name": "ci / General tests (server (9/12))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293850,
"name": "ci / General tests (workspaces-a (2/2))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293852,
"name": "ci / General tests (server (1/12))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293858,
"name": "ci / General tests (chat (2/3))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293865,
"name": "ci / General tests (server (10/12))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293875,
"name": "ci / General tests (workspaces-b)",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293940,
"name": "ci / General tests (server (6/12))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558293992,
"name": "ci / General tests (chat (3/3))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558294007,
"name": "ci / General tests (chat (1/3))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558294057,
"name": "ci / General tests (server (7/12))",
"status": "completed",
"conclusion": "success"
},
{
"id": 131558294396,
"name": "ci / Sandbox provider tests",
"status": "completed",
"conclusion": "skipped"
},
{
"id": 131558294604,
"name": "ci / Select Vitest files",
"status": "completed",
"conclusion": "skipped"
},
{
"id": 131562309925,
"name": "ci / e2e",
"status": "completed",
"conclusion": "success"
},
{
"id": 131564602271,
"name": "ci / verify",
"status": "completed",
"conclusion": "failure"
}
],
"annotations": {
"131558292381": [
{
"annotation_level": "failure",
"message": "The operation was canceled."
},
{
"annotation_level": "failure",
"message": "gateway_indexing_incomplete"
},
{
"annotation_level": "failure",
"message": "gateway_indexing_incomplete"
}
],
"131558293818": [
{
"annotation_level": "failure",
"message": "The operation was canceled."
}
],
"131564602271": [
{
"annotation_level": "failure",
"message": "Process completed with exit code 1."
}
]
},
"logs": {
"131558292381": "2026-11-12T23:13:37.4339910Z ##[error]gateway_indexing_incomplete\n2026-11-12T23:13:37.4368921Z ##[error]gateway_indexing_incomplete\n2026-11-12T23:15:01.1805404Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-12T23:15:13.7401251Z ##[error]The operation was canceled.\n",
"131558293818": "2026-11-12T23:15:36.1981336Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n2026-11-12T23:15:48.7226826Z ##[error]The operation was canceled.\n"
},
"total_count": 51
}
+156 -14
View File
@@ -13,10 +13,13 @@ import {
const fixture = (runId) => JSON.parse(readFileSync(
new URL(`./fixtures/runner-loss-rerun/run-${runId}.json`, import.meta.url), "utf8",
));
const MIXED = 42271918832; // six lost runners, one shutdown in the log, one real test timeout
const MIXED = 42271918832; // six lost runners, four shutdowns in the log, one real test timeout
const REAL = 42264785415; // a real assertion failure
const ALL_LOST = 42272083862; // one lost runner
const LOG_ONLY = 42277969418; // shutdown signal visible only in the job log
const LOG_ONLY = 42277969418; // shutdown signals visible only in the job logs
const CANCELLED_RUN = 43285064435;
const NOISY_LOST = 43325855069;
const CANCELLED_MESSAGE = "The operation was canceled.";
const workflow = readFileSync(new URL("../../workflows/ci-runner-loss-rerun.yml", import.meta.url), "utf8");
const json = (body, headers = {}) => new Response(JSON.stringify(body), {
@@ -58,7 +61,7 @@ function fakeGithub(data, {
if (match) return json(data.annotations[match[1]] ?? []);
match = route.match(/^\/actions\/jobs\/(\d+)\/logs$/);
if (match) return new Response(data.logs[match[1]] ?? "", { status: 200 });
if (route === `/commits/${run.head_sha}/pulls`) return json(pulls ?? [openPr]);
if (route === `/commits/${run.head_sha}/pulls`) return json((typeof pulls === "function" ? pulls(calls) : pulls) ?? [openPr]);
if (route === "/actions/workflows/pr.yml/runs") {
const runs = prRuns ?? [{ id: run.id, conclusion: "failure" }];
return json({ total_count: runs.length, workflow_runs: runs });
@@ -106,19 +109,131 @@ test("a shutdown signal found only in the job log counts as runner loss", async
assert.deepEqual(data.annotations["128101016102"].map((annotation) => annotation.message), ["The operation was canceled."]);
const { decision, calls } = await runMain(data);
assert.equal(decision.rerun, true);
assert.match(decision.jobs[0].evidence, /job log/);
assert.equal(decision.jobs.length, 11);
assert.ok(decision.jobs.every((job) => job.lost && /job log/.test(job.evidence)));
assert.ok(calls.some((call) => call.path.endsWith("/actions/jobs/128101016102/logs")));
assert.equal(posted(calls).length, 1);
});
test("a cancelled run whose cancelled job lost its runner is re-run once", async () => {
const data = fixture(CANCELLED_RUN);
assert.equal(data.run.conclusion, "cancelled");
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
assert.equal(decision.rerun, true);
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [["ci / General tests (server (8/12))", true]]);
assert.match(decision.jobs[0].evidence, /job log/);
assert.deepEqual(posted(calls).map((call) => call.path),
[`/repos/paperclipai/paperclip/actions/runs/${CANCELLED_RUN}/rerun-failed-jobs`]);
});
test("runner loss in the log wins over error annotations the workload printed", async () => {
const data = fixture(NOISY_LOST);
const suite = data.jobs.find((job) => job.name === "ci / Runner support suite");
assert.ok(data.annotations[suite.id].some((annotation) => annotation.message === "gateway_indexing_incomplete"));
const { decision, calls } = await runMain(data);
assert.equal(decision.rerun, true);
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [
["ci / Runner support suite", true],
["ci / General tests (server (2/12))", true],
]);
assert.equal(posted(calls).length, 1);
});
function cancelEverything(data, { log = `2026-11-12T19:03:32.0000000Z ##[error]${CANCELLED_MESSAGE}\n` } = {}) {
data.annotations = {};
data.logs = {};
for (const [index, job] of data.jobs.entries()) {
if (job.name.startsWith("ci / gate / ")) continue;
job.conclusion = "cancelled";
if (index % 4 === 3) continue;
data.annotations[job.id] = [{ annotation_level: "failure", message: CANCELLED_MESSAGE }];
data.logs[job.id] = log;
}
return data;
}
test("a run superseded by a new push is not re-run, and no job is classified", async () => {
const data = cancelEverything(fixture(CANCELLED_RUN));
assert.ok(data.jobs.filter((job) => job.conclusion === "cancelled").length >= 40);
const { decision, calls } = await runMain(data, {
pulls: [{ state: "open", base: { repo: { id: 1170821064 } }, head: { sha: "0".repeat(40), repo: { id: 1170821064 } } }],
}, { conclusion: "cancelled" });
assert.equal(decision.rerun, false);
assert.match(decision.reason, /no open pull request/);
assert.equal(calls.some((call) => call.path.includes("/check-runs/") || call.path.endsWith("/logs")), false);
assert.equal(posted(calls).length, 0);
});
test("a push while job logs are read stops the re-run", async () => {
const data = fixture(CANCELLED_RUN);
const moved = [{ state: "open", base: { repo: { id: 1170821064 } }, head: { sha: "0".repeat(40), repo: { id: 1170821064 } } }];
const { decision, calls } = await runMain(data, {
pulls: (seen) => (seen.some((call) => call.path.endsWith("/logs")) ? moved : undefined),
}, { conclusion: "cancelled" });
assert.equal(decision.rerun, false);
assert.match(decision.reason, /no open pull request/);
assert.equal(decision.jobs[0].lost, true);
assert.equal(calls.filter((call) => call.path.endsWith("/pulls")).length, 2);
assert.equal(posted(calls).length, 0);
});
test("a run a person cancelled is not re-run", async () => {
const data = cancelEverything(fixture(CANCELLED_RUN));
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
assert.equal(decision.rerun, false);
assert.ok(decision.jobs.length >= 40);
assert.equal(decision.jobs.some((job) => job.lost), false);
assert.match(decision.reason, /failed jobs did not lose their runner/);
assert.equal(posted(calls).length, 0);
});
test("a cancelled run with one lost job and one real test failure is not re-run", async () => {
const data = fixture(CANCELLED_RUN);
const failing = data.jobs.find((job) => job.name === "ci / General tests (server (1/12))");
failing.conclusion = "failure";
data.annotations[failing.id] = [
{ annotation_level: "failure", message: "AssertionError: expected 'queued' to be 'running'" },
{ annotation_level: "failure", message: "Process completed with exit code 1." },
];
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
assert.equal(decision.rerun, false);
assert.deepEqual(decision.jobs.map((job) => [job.name, job.lost]), [
["ci / General tests (server (1/12))", false],
["ci / General tests (server (8/12))", true],
]);
assert.match(decision.reason, /1 of 2 failed jobs did not lose their runner/);
assert.equal(posted(calls).length, 0);
});
test("a cancelled run is re-run at most once", async () => {
const { decision, calls } = await runMain(fixture(CANCELLED_RUN), {
mutateRun: (run) => ({ ...run, run_attempt: 2 }),
}, { conclusion: "cancelled" });
assert.equal(decision.rerun, false);
assert.match(decision.reason, /attempt 2/);
assert.deepEqual(calls.map((call) => call.path), [`/repos/paperclipai/paperclip/actions/runs/${CANCELLED_RUN}`]);
});
test("a cancelled routing gate is left for a human re-run", async () => {
const data = fixture(CANCELLED_RUN);
const gate = data.jobs.find((job) => job.name === "ci / gate / Select trusted runner");
gate.conclusion = "cancelled";
data.annotations[gate.id] = [{ annotation_level: "failure", message: CANCELLED_MESSAGE }];
data.logs[gate.id] = data.logs[Object.keys(data.logs)[0]];
const { decision, calls } = await runMain(data, {}, { conclusion: "cancelled" });
assert.equal(decision.rerun, false);
assert.match(decision.jobs.find((job) => job.id === gate.id).evidence, /routing gate/);
assert.equal(posted(calls).length, 0);
});
test("mixed runner loss and a real test failure: no re-run", async () => {
const { decision, calls, summary } = await runMain(fixture(MIXED));
assert.equal(decision.rerun, false);
assert.equal(decision.jobs.length, 8);
assert.equal(decision.jobs.length, 11);
assert.deepEqual(decision.jobs.filter((job) => !job.lost).map((job) => job.name), ["ci / General tests (workspaces-a (1/2))"]);
assert.equal(decision.jobs.find((job) => job.name === "ci / Verify Paperclip Runner (vitest package)").lost, true);
assert.equal(posted(calls).length, 0);
assert.match(decision.reason, /1 of 8 failed jobs did not lose their runner/);
assert.match(decision.reason, /1 of 11 failed jobs did not lose their runner/);
assert.match(summary, /\*\*No automatic re-run\*\*/);
});
@@ -209,7 +324,7 @@ test("job listing follows Link pagination across pages", async () => {
const pages = github.calls.filter((call) => call.path.endsWith("/attempts/1/jobs"));
assert.deepEqual(pages.map((call) => new URLSearchParams(call.query).get("page")), [null, "2", "3"]);
// Failed jobs on pages 2 and 3 are all classified.
assert.equal(decision.jobs.length, 8);
assert.equal(decision.jobs.length, 11);
assert.equal(decision.rerun, false);
});
@@ -236,9 +351,20 @@ test("classification is conservative about cancelled jobs", async () => {
// A job timeout also cancels the step, but adds its own failure annotation.
const timedOut = await classifyJob({ id: 1 }, {
annotations: [{ annotation_level: "failure", message: "The job has exceeded the maximum execution time of 30m0s" }, ...cancelled],
readLog: async () => { throw new Error("log must not be read"); },
readLog: async () => "##[error]The job has exceeded the maximum execution time of 30m0s\n##[error]The operation was canceled.\n",
});
assert.equal(timedOut.lost, false);
assert.match(timedOut.evidence, /maximum execution time/);
const notStarted = await classifyJob({ id: 1, conclusion: "cancelled" }, {
annotations: [],
readLog: async () => { throw new Error("log must not be read"); },
});
assert.equal(notStarted.lost, false);
const exitOnly = await classifyJob({ id: 1 }, {
annotations: [{ annotation_level: "failure", message: "Process completed with exit code 1." }],
readLog: async () => { throw new Error("log must not be read"); },
});
assert.equal(exitOnly.lost, false);
// Notices and warnings are not failure evidence.
const warning = await classifyJob({ id: 1 }, {
annotations: [{ annotation_level: "warning", message: "The self-hosted runner lost communication with the server." }],
@@ -250,7 +376,7 @@ test("classification is conservative about cancelled jobs", async () => {
assert.equal(logShowsRunnerLoss(fixture(MIXED).logs["128078674151"]), true);
});
test("a real failure before the runner was lost is not counted as runner loss", async () => {
test("runner-loss evidence wins over other failure annotations in the same job", async () => {
const lostMessage = "The self-hosted runner lost communication with the server. Verify the machine is running.";
const noLog = async () => { throw new Error("log must not be read"); };
// A test fails, then the runner is lost during an always() upload step.
@@ -262,8 +388,8 @@ test("a real failure before the runner was lost is not counted as runner loss",
],
readLog: noLog,
});
assert.equal(testThenLost.lost, false);
assert.match(testThenLost.evidence, /Test timed out/);
assert.equal(testThenLost.lost, true);
assert.match(testThenLost.evidence, /lost communication/);
// The order of the annotations does not matter.
const lostThenExit = await classifyJob({ id: 1 }, {
annotations: [
@@ -272,7 +398,22 @@ test("a real failure before the runner was lost is not counted as runner loss",
],
readLog: noLog,
});
assert.equal(lostThenExit.lost, false);
assert.equal(lostThenExit.lost, true);
const noisy = [
{ annotation_level: "failure", message: "gateway_indexing_incomplete" },
{ annotation_level: "failure", message: CANCELLED_MESSAGE },
];
const noisyLost = await classifyJob({ id: 1 }, {
annotations: noisy,
readLog: async () => "##[error]gateway_indexing_incomplete\n##[error]The runner has received a shutdown signal.\n",
});
assert.equal(noisyLost.lost, true);
const noisyCancelled = await classifyJob({ id: 1 }, {
annotations: noisy,
readLog: async () => "##[error]gateway_indexing_incomplete\n##[error]The operation was canceled.\n",
});
assert.equal(noisyCancelled.lost, false);
assert.match(noisyCancelled.evidence, /gateway_indexing_incomplete/);
// Runner loss with a cancellation only is still runner loss.
const lostAndCancelled = await classifyJob({ id: 1 }, {
annotations: [
@@ -292,14 +433,15 @@ test("summary cells cannot break the table", () => {
assert.match(summary, /\| ci \/ a\\\|b \| no \| line one line \\\| two \|/);
});
test("workflow acts only on failed first attempts of the canonical PR workflow", () => {
test("workflow acts only on failed or cancelled first attempts of the canonical PR workflow", () => {
const condition = workflow.match(/^ {4}if: >-\n((?: {6}.+\n)+)/m)?.[1].replace(/\s+/g, " ").trim();
assert.equal(condition, [
"github.repository == 'paperclipai/paperclip' &&",
"github.repository_id == '1170821064' &&",
"github.event.workflow_run.event == 'pull_request' &&",
"github.event.workflow_run.path == '.github/workflows/pr.yml' &&",
"github.event.workflow_run.conclusion == 'failure' &&",
"(github.event.workflow_run.conclusion == 'failure' ||",
"github.event.workflow_run.conclusion == 'cancelled') &&",
"github.event.workflow_run.run_attempt == 1",
].join(" "));
assert.match(workflow, /^on:\n {2}workflow_run:\n {4}workflows: \[PR\]\n {4}types: \[completed\]$/m);
+5 -3
View File
@@ -1,7 +1,8 @@
name: CI runner-loss rerun
# Self-hosted AWS runners are sometimes lost mid-job. When every failed job of
# a first PR CI attempt lost its runner, re-run the failed jobs once. This is
# Self-hosted AWS runners are sometimes lost mid-job. When every failed or
# cancelled job of a first PR CI attempt lost its runner, re-run the failed
# jobs once. A spot reclaim often concludes the run `cancelled`. This is
# what a maintainer's "Re-run failed jobs" click does; github-actions[bot]
# becomes attempt 2's triggering actor. See "Trusted PR Workflow" in
# doc/DEVELOPING.md.
@@ -31,7 +32,8 @@ jobs:
github.repository_id == '1170821064' &&
github.event.workflow_run.event == 'pull_request' &&
github.event.workflow_run.path == '.github/workflows/pr.yml' &&
github.event.workflow_run.conclusion == 'failure' &&
(github.event.workflow_run.conclusion == 'failure' ||
github.event.workflow_run.conclusion == 'cancelled') &&
github.event.workflow_run.run_attempt == 1
# GitHub-hosted on purpose: this job must not depend on the fleet it repairs.
runs-on: ubuntu-latest
+15 -8
View File
@@ -187,14 +187,21 @@ AWS fleet. A stale admission roster can allocate only the short metadata gate;
the fresh CODEOWNERS check still denies PR execution for a removed owner.
AWS runners are sometimes lost mid-job. `ci-runner-loss-rerun.yml` re-runs the
failed jobs of a PR's first CI attempt once when every failed job lost its
runner. A lost runner is recognized by the check-run annotation "lost
communication with the server" or "received a shutdown signal". When the only
annotation is "The operation was canceled.", the job log is checked for the
same messages. The `verify` and `e2e` aggregates are not counted. Any other
failure, a failed routing gate, or a PR whose head has moved prevents the
automatic re-run. The workflow job summary lists each failed job and its
evidence. Like a maintainer's "Re-run failed jobs", the re-run reuses the
failed jobs of a PR's first CI attempt once when every failed or cancelled job
lost its runner. A spot reclaim often ends the job, and the run, as
`cancelled`. A lost runner is recognized by the check-run annotation "lost
communication with the server" or "received a shutdown signal". When a job has
the annotation "The operation was canceled.", the job log is checked for the
same messages. This evidence counts even if the job also has other error
annotations, so a test that really failed just before a reclaim gets one
wasted re-run. A cancelled job without this evidence, such as a job a person
cancelled, counts as a real failure. The `verify` and `e2e` aggregates are not
counted. Any other failure, a failed or cancelled routing gate, or a PR whose
head has moved prevents the automatic re-run. The head check runs before any
job is classified, so a run cancelled by a newer push costs no log reads. It
runs again just before the re-run, because a re-run of an old run joins the
PR's `cancel-in-progress` concurrency group and would cancel the newer CI. The
workflow job summary lists each failed or cancelled job and its evidence. Like a maintainer's "Re-run failed jobs", the re-run reuses the
gate's successful outputs, including its AWS admission. Attempt 2's
triggering actor is `github-actions[bot]`, so `profile-ci` does not export
that attempt. Later attempts are never re-run automatically.