fix(release): retry Cloud readiness jobs that lost their Spot runner (#21399)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every push to `master` publishes a canary, and the canary's image
reaches staging
> - Each canary waits for `Cloud source verified v1` from the Cloud
readiness run of its exact SHA
> - Cloud readiness runs on AWS Spot workers. When a worker is
reclaimed, the gate failed and the image watcher cancelled the readiness
run
> - Nothing retried that attempt. On a busy master the newest commit's
release failed, and merged work did not reach staging until a person
re-ran the run
> - This pull request re-runs the failed jobs of a readiness attempt
whose every failure is a lost runner. A real check failure still fails
closed
> - The benefit is that a Spot reclaim no longer holds a release

## Linked Issues or Issue Description

**What happened?**
On 2026-11-13, #21266 merged as `8af889f612`. No canary that contained
it published for more than an hour. The release workflow keeps one
running and one pending canary. A newer push replaces the pending one,
so the newest commit always runs next. That part works as designed. The
failures came from the readiness gate:

| Release (SHA) | Readiness run / attempt | Gate that blocked it | Cause
|
| --- | --- | --- | --- |
| `11840f9904` | 43435593630 / 2 | `Reuse exact-source verification` |
Spot workers reclaimed at 13:49:56Z. The run was then cancelled by
`github-actions[bot]` (the image watcher). Dotta re-ran attempt 3 by
hand |
| `8af889f612` | 43434587048 / 2 | Release superseded while pending;
readiness failed in the same reclaim | Spot reclaim |
| `9fbfd93c9b` | 43423674511 / 2 | `Reuse exact-source verification` |
Spot reclaim |
| `6917d0568d` | 43439942063 / 1 | `Reuse exact-source verification` |
Flaky test `decision-models.test.ts` (concurrent budget admission) |
| `18fb6ee571` | 43422042261 / 2 | `Reuse exact-source verification` |
Flaky test `fresh-continuation-fallback.test.ts` (truncate) |
| `5ede64f247` | 43421480302 / 1 | `Reuse exact-source verification` |
Both flaky tests |

Over the last 200 master pushes, release runs ended 14 success, 33
failure, and 150 cancelled (superseded while pending, which is
expected).

**Expected behavior**
A lost runner is not a source failure. The gate retries the lost jobs
and publishes the newest commit that passes.

**Steps to reproduce**
See `scripts/cloud-readiness-runner-loss.test.mjs`, "busy-master
simulation". It replays the 13:49:56Z reclaim through both waiters.

## What Changed

- `scripts/cloud-readiness-runner-loss.mjs` (new): classifies a failed
readiness attempt with the existing PR classifier
(`.github/scripts/runner-loss-rerun.mjs`). It is a runner loss only when
every causing job (`failure`, `timed_out`, `startup_failure`) lost its
runner. Cancelled siblings are fallout. A run that is still going is
cancelled first, then `rerun-failed-jobs` runs on the next poll. The
ceiling is readiness attempt 4. Writes go only to `cancel` or
`rerun-failed-jobs` of a canonical run, after an identity check.
- `scripts/cloud-source-verification.mjs` (canary gate in
`release.yml`): on `SourceVerificationFailed`, ask the retry first. The
source wait goes from 60 to 90 minutes.
- `scripts/watch-release-source.mjs` (master image watcher in
`docker.yml`): on a runner loss, keep the image attempt alive and wait
for the retried proof. Do not cancel it.
- `.github/workflows/release.yml`: `verify_canary` timeout goes from 70
to 100 minutes.
- `.github/workflows/pr-trusted.yml`: run the new test file.
- `doc/RELEASING.md`: new "Busy master and lost runners" section.

## Verification

- `node --test scripts/cloud-readiness-runner-loss.test.mjs
scripts/cloud-source-verification.test.mjs
.github/scripts/tests/*.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs`: all pass.
`release-verify-workflow.test.mjs` passes 51 of 51 but does not exit
locally. This also happens on `origin/master` without this change.
- Red/green: with the two waiter files reverted, 3 of the 6 simulation
tests fail (the canary gate, two reclaims in a row, and the image
watcher). With the change, all 20 pass.
- Dry run against real runs (read-only, writes captured): 43435593630/2,
43434587048/2, and 43423674511/2 give `rerun-failed-jobs`.
43439942063/1, 43421480302/1, and 43422042261/2 (real test failures)
give `failed` with no write.
- `pnpm verify` exits 2 because a workflow file changed. Required CI
supplies broad coverage.

## Risks

- A real failure classified as a runner loss would be retried. The
classifier needs the runner's own shutdown or lost-communication
`##[error]` line in the job log. It is the same classifier PR CI uses,
and unreadable evidence fails closed.
- On a reclaim, the waiter cancels the readiness run to retry sooner.
Jobs that were still healthy run again. This costs runner time, not
correctness.
- Flaky tests still fail a canary. This change does not retry them. They
need their own fixes.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 09:20:30 -06:00
1 parent 51c81c537b
commit 2659ff1d22
9 files changed
+470 -10

No files matched your search

+2
View File
@@ -85,6 +85,7 @@ jobs:
permissions:
contents: read
actions: write
checks: read # runner-loss evidence (annotations) for readiness retries
outputs:
verified: ${{ steps.watch.outputs.verified }}
steps:
@@ -494,6 +495,7 @@ jobs:
permissions:
contents: read
actions: write # restart only exact-source producers deferred before admission
checks: read # runner-loss evidence (annotations) for readiness retries
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
with:
+1 -1
View File
@@ -249,7 +249,7 @@ jobs:
run: node "$CI_NODE_TEST" ./scripts/__tests__/e2e-shard.test.mjs
- name: Test release verify workflow wiring
run: node "$CI_NODE_TEST" ./scripts/__tests__/release-verify-workflow.test.mjs ./scripts/__tests__/eval-schedule-monitor.test.mjs ./scripts/__tests__/prepare-runner-release-image.test.mjs ./scripts/cloud-source-verification.test.mjs ./scripts/standard-image-contract.test.mjs ./scripts/reuse-canary-image.test.mjs
run: node "$CI_NODE_TEST" ./scripts/__tests__/release-verify-workflow.test.mjs ./scripts/__tests__/eval-schedule-monitor.test.mjs ./scripts/__tests__/prepare-runner-release-image.test.mjs ./scripts/cloud-source-verification.test.mjs ./scripts/cloud-readiness-runner-loss.test.mjs ./scripts/standard-image-contract.test.mjs ./scripts/reuse-canary-image.test.mjs
- name: Test standalone package build concurrency
run: node "$CI_NODE_TEST" ./scripts/__tests__/build-standalone-concurrency.test.mjs
+5 -1
View File
@@ -322,10 +322,14 @@ jobs:
name: Reuse exact-source verification
if: github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.event_name == 'push' && github.ref == 'refs/heads/master'
runs-on: ubuntu-latest
timeout-minutes: 70
# The source wait allows 90 minutes: a deferred full readiness run plus two
# retries of jobs whose Spot runner was reclaimed.
timeout-minutes: 100
permissions:
contents: read
# Also re-runs readiness jobs whose runner was lost, never a failed check.
actions: write # restart only an exact-source push deferred before admission
checks: read # runner-loss evidence (annotations) for readiness retries
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
+24 -1
View File
@@ -108,6 +108,27 @@ It:
- fails by default if npm leaves `latest` pointing at a canary; use `--allow-canary-latest` only when that state is intentional
- creates a git tag `canary/vYYYY.MDD.P-canary.N`
#### Busy master and lost runners
Master often receives a commit every few minutes, faster than one canary
publishes (about 60-90 minutes). The release workflow keeps one running canary
and one pending canary. A newer push replaces the pending one, so the canary
that runs next is always the newest master commit. Commits in between are not
published on their own; they ship inside the next canary.
Each canary waits for `Cloud source verified v1` from the Cloud readiness run
of its exact SHA (up to 90 minutes). Cloud readiness runs on AWS Spot workers.
When an attempt fails only because its workers were reclaimed, the canary gate
and the master image watcher re-run that attempt's failed jobs instead of
failing the release
([`scripts/cloud-readiness-runner-loss.mjs`](../scripts/cloud-readiness-runner-loss.mjs)).
A job counts as lost when its log shows the runner's shutdown or lost
communication error. Retries continue up to attempt 4 of the readiness run. A
real check failure, including a flaky test, is never retried: that canary fails
and the next push publishes. If a canary is missing after a quiet period, look
at the failed `Reuse exact-source verification` job. It names the readiness run
and lists each failed job as a lost runner or a failure.
After publication, the onboarding smoke checks out the same source SHA and
installs its test dependencies from the publisher's resolved lockfile with
`--frozen-lockfile`. The same-run artifact is captured before tracked files are
@@ -316,7 +337,9 @@ latest attempt. A green admission-only run with skipped build and publish jobs
is pending work. The source observer retries only producers that were strictly
deferred, continues watching after build admission, and waits for publication.
The final publication step rechecks both proofs before it creates image tags.
Failed or cancelled producers require inspection and are not retried automatically.
Failed or cancelled producers require inspection and are not retried automatically,
with one exception: a Cloud readiness attempt that failed only because Spot
runners were reclaimed is retried (see "Busy master and lost runners" below).
[`docker.yml`](../.github/workflows/docker.yml) publishes the standard image:
@@ -123,6 +123,8 @@ test("canary can admit deferred exact-source checks while stable keeps full veri
assert.match(canary, /ref: \$\{\{ github\.sha \}\}/);
assert.match(canary, /SOURCE_SHA: \$\{\{ github\.sha \}\}/);
assert.match(canary, /run: node scripts\/cloud-source-verification\.mjs "\$SOURCE_SHA"/);
// Two runner-loss retries of a deferred full readiness run fit inside the job.
assert.match(canary, /timeout-minutes: 100\n/);
assert.doesNotMatch(canary, /release-verify\.yml|continue-on-error|always\(\)/);
assert.match(releaseWorkflow, /publish_canary:\n\s+if: .*github\.event_name == 'push'.*\n\s+needs: \[verify_canary, release_runner_source, release_runner_binaries\]/);
// The stable lane is gated on the stable channel since the nightly lane
+130
View File
@@ -0,0 +1,130 @@
// A Cloud readiness attempt that failed only because AWS Spot workers were
// reclaimed proves nothing about the source. Before this module, the first
// lost runner failed the release's source gate, the image watcher cancelled
// the readiness run, and nothing retried it: on 2026-11-13 a 13:49:56Z fleet
// reclaim held every release containing #21266 until a person re-ran the run.
//
// Both waiters (the npm canary gate in release.yml and the image watcher in
// docker.yml) call this when the source proof fails. A failure whose every
// causing job lost its runner re-runs the failed jobs instead, up to a fixed
// attempt ceiling. A real check failure still fails closed: it is never
// retried here.
import { classifyJob, createGithubClient } from "../.github/scripts/runner-loss-rerun.mjs";
const repository = "paperclipai/paperclip";
const workflowPath = ".github/workflows/cloud-readiness.yml";
// Attempt 1 is often the deferred admission-only pass and attempt 2 the
// admitted full run, so 4 allows at least two runner-loss retries. The ceiling
// is stateless: every waiter of every release attempt agrees on it.
export const maxRunnerLossAttempt = 4;
// A causing job ended the attempt on its own. Cancelled jobs are fallout from
// that failure (the image watcher or a reusable workflow cancelled them).
const causeConclusions = new Set(["failure", "timed_out", "startup_failure"]);
const nonGating = name => name.startsWith("Runner chaos evals (not a publication gate) / ");
function trusted(run, sha, runId) {
return run?.id === runId && run.path === workflowPath && run.head_sha === sha && run.head_branch === "master" &&
run.event === "push" && run.repository?.full_name === repository && run.head_repository?.full_name === repository &&
Number.isSafeInteger(run.run_attempt) && run.run_attempt > 0;
}
async function attemptJobs(api, runId, attempt) {
const jobs = [];
for (let page = 1; page <= 10; page += 1) {
const batch = await api(`/repos/${repository}/actions/runs/${runId}/attempts/${attempt}/jobs?per_page=100&page=${page}`);
if (!Array.isArray(batch.jobs)) throw new Error("Cloud readiness job listing is malformed.");
jobs.push(...batch.jobs);
if (batch.jobs.length < 100) return jobs;
}
throw new Error("Cloud readiness job listing is incomplete.");
}
/** Which failed jobs lost their runner. `lost` is true only when at least one
* job lost its runner and no job failed for any other reason. */
export async function classifyAttempt(jobs, { readAnnotations, readLog }) {
const ended = jobs.filter(job => job.status === "completed" && !nonGating(job.name ?? ""));
const causes = ended.filter(job => causeConclusions.has(job.conclusion));
// A reclaimed worker sometimes concludes its job `cancelled`. With no other
// cause, a cancelled job counts only when its own log shows the loss.
const candidates = causes.length ? causes : ended.filter(job => job.conclusion === "cancelled");
const results = [];
for (const job of candidates) {
try {
results.push({ id: job.id, name: job.name, conclusion: job.conclusion,
...(await classifyJob(job, { annotations: await readAnnotations(job), readLog })) });
} catch (error) {
results.push({ id: job.id, name: job.name, conclusion: job.conclusion, lost: false, evidence: `could not read job evidence: ${error.message}` });
}
}
const lost = causes.length ? results.length > 0 && results.every(job => job.lost) : results.some(job => job.lost);
return { lost, jobs: results };
}
/**
* Called after a source proof read failed for `failure.runId`/`failure.attempt`.
* Returns "retrying" when the waiter should keep waiting (a runner-loss retry
* was requested, is pending, or a newer attempt already exists) and "failed"
* when the failure stands.
*/
export async function retryRunnerLoss({ sha, failure, api, readAnnotations, readLog, cancel, rerunFailed, log = console.log }) {
const run = await api(`/repos/${repository}/actions/runs/${failure.runId}`);
if (!trusted(run, sha, failure.runId)) throw new Error("Cloud readiness identity changed before a runner-loss retry.");
// Someone (another waiter or a person) already started the next attempt.
if (run.run_attempt > failure.attempt) return "retrying";
if (run.run_attempt !== failure.attempt) return "failed";
const jobs = (await attemptJobs(api, run.id, run.run_attempt))
.filter(job => job.run_id === run.id && job.run_attempt === run.run_attempt && job.head_sha === sha);
const verdict = await classifyAttempt(jobs, { readAnnotations, readLog });
for (const job of verdict.jobs) log(`Cloud readiness ${run.id} attempt ${run.run_attempt}: ${job.name} ${job.lost ? "lost its runner" : "failed"} (${job.evidence}).`);
if (!verdict.lost) return "failed";
if (run.run_attempt >= maxRunnerLossAttempt) {
log(`Cloud readiness ${run.id} lost runners on attempt ${run.run_attempt}, the last automatic attempt.`);
return "failed";
}
// Re-running failed jobs needs a completed run. Healthy jobs still running
// would be repeated anyway once they are cancelled, so end the attempt now.
if (run.status !== "completed") {
// Job evidence reads take time. Another waiter may have ended this attempt
// and started the next one meanwhile: never cancel a newer attempt.
const fresh = await api(`/repos/${repository}/actions/runs/${run.id}`);
if (!trusted(fresh, sha, run.id)) throw new Error("Cloud readiness identity changed before a runner-loss retry.");
if (fresh.run_attempt !== run.run_attempt || fresh.status === "completed") return "retrying";
log(`Cloud readiness ${run.id} attempt ${run.run_attempt} lost its runner; ending the attempt to retry its failed jobs.`);
try {
await cancel(`/repos/${repository}/actions/runs/${run.id}/cancel`);
} catch (error) {
// The attempt finished or was cancelled by the other waiter first.
const latest = await api(`/repos/${repository}/actions/runs/${run.id}`);
if (!trusted(latest, sha, run.id)) throw error;
if (latest.run_attempt > run.run_attempt || latest.status === "completed" || [409, 422].includes(error.status)) return "retrying";
throw error;
}
return "retrying";
}
try {
await rerunFailed(`/repos/${repository}/actions/runs/${run.id}/rerun-failed-jobs`);
log(`Cloud readiness ${run.id} attempt ${run.run_attempt} lost its runner; re-ran its failed jobs.`);
} catch (error) {
// The other waiter may have won the same write, or GitHub has not
// finished cancelling the attempt. Both resolve on the next poll.
const latest = await api(`/repos/${repository}/actions/runs/${run.id}`);
if (!trusted(latest, sha, run.id)) throw error;
if (latest.run_attempt > run.run_attempt || latest.status !== "completed" || [409, 422].includes(error.status)) return "retrying";
throw error;
}
return "retrying";
}
/** The canonical writes and evidence reads `retryRunnerLoss` needs. */
export function createRunnerLossRetry({ token, fetcher = fetch, log = console.log }) {
const client = createGithubClient({ token, fetcher });
const write = async path => {
if (!/^\/repos\/paperclipai\/paperclip\/actions\/runs\/[1-9]\d*\/(cancel|rerun-failed-jobs)$/.test(path)) throw new Error("Invalid runner-loss retry path.");
const response = await fetcher(`https://api.github.com${path}`, { method: "POST", redirect: "error",
headers: { Authorization: `Bearer ${token}`, Accept: "application/vnd.github+json" }, signal: AbortSignal.timeout(30_000) });
if (!response.ok) throw Object.assign(new Error(`Could not retry a lost readiness runner: HTTP ${response.status}.`), { status: response.status });
};
return ({ sha, failure, api }) => retryRunnerLoss({ sha, failure, api, log, cancel: write, rerunFailed: write,
readAnnotations: job => client.paginate(`/repos/${repository}/check-runs/${job.id}/annotations?per_page=100`),
readLog: job => client.text(`/repos/${repository}/actions/jobs/${job.id}/logs`) });
}
@@ -0,0 +1,284 @@
import assert from "node:assert/strict";
import test, { describe } from "node:test";
import { classifyAttempt, createRunnerLossRetry, maxRunnerLossAttempt, retryRunnerLoss } from "./cloud-readiness-runner-loss.mjs";
import { waitForSourceVerification } from "./cloud-source-verification.mjs";
import { watchReleaseSource } from "./watch-release-source.mjs";
const sha = "a".repeat(40);
const repository = { id: 1170821064, full_name: "paperclipai/paperclip" };
const lostLog = "2026-11-13T13:49:56.0521573Z ##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.\n";
const testLog = "2026-11-13T14:19:18.2551007Z ##[error]AssertionError: expected \"vi.fn()\" to be called 2 times, but got 1 times\n";
const cancelledNote = { annotation_level: "failure", message: "The operation was canceled." };
const assertionNote = { annotation_level: "failure", message: "AssertionError: expected \"vi.fn()\" to be called 2 times, but got 1 times" };
/**
* A stateful fake of the GitHub Actions API for one Cloud readiness run. Each
* attempt is a list of jobs; `rerun-failed-jobs` starts the next attempt from
* `script`, and `cancel` completes the current one. Job evidence (annotations
* and logs) is keyed by job id, as GitHub serves it.
*/
function github({ attempts, script = [], startAttempt = 1, status = "completed" }) {
const state = { attempt: startAttempt, status, attempts: { ...attempts }, writes: [], evidence: {}, script };
const run = () => ({ id: 99, workflow_id: 123, path: ".github/workflows/cloud-readiness.yml", run_attempt: state.attempt,
repository, head_repository: repository, head_sha: sha, head_branch: "master", event: "push",
status: state.status, conclusion: state.status === "completed" ? "failure" : null });
state.api = async path => {
if (path.endsWith("/workflows/cloud-readiness.yml")) return { id: 123, path: ".github/workflows/cloud-readiness.yml" };
if (path.includes("/workflows/123/runs?")) return { total_count: 1, workflow_runs: [run()] };
const jobs = path.match(/\/runs\/99\/attempts\/(\d+)\/jobs\?/);
if (jobs) return { total_count: (state.attempts[jobs[1]] ?? []).length, jobs: state.attempts[jobs[1]] ?? [] };
if (path.endsWith("/runs/99")) return run();
// The image watcher's own run and the migrator lookup.
if (path.includes("/runs/50/attempts/1/jobs?")) return { total_count: 0, jobs: [] };
if (path.endsWith("/runs/50")) return { id: 50, run_attempt: 1, head_sha: sha, head_branch: "master", event: "push",
repository, head_repository: repository, path: ".github/workflows/docker.yml", status: "in_progress" };
if (path.endsWith("/workflows/cloud-migrator-artifacts.yml")) return { id: 124, path: ".github/workflows/cloud-migrator-artifacts.yml" };
if (path.includes("/workflows/124/runs?")) return { total_count: 0, workflow_runs: [] };
throw new Error(`Unexpected API path: ${path}`);
};
const write = async path => {
state.writes.push(path);
if (path.endsWith("/cancel")) state.status = "completed";
if (path.endsWith("/rerun-failed-jobs")) {
if (state.status !== "completed") throw Object.assign(new Error("HTTP 409"), { status: 409 });
state.attempt += 1;
state.attempts[state.attempt] = state.script.shift()?.(state.attempt) ?? [];
state.status = "in_progress";
}
};
state.cancel = write;
state.rerunFailed = write;
state.readAnnotations = async job => state.evidence[job.id]?.annotations ?? [];
state.readLog = async job => state.evidence[job.id]?.log ?? "";
state.retry = ({ failure, api }) => retryRunnerLoss({ sha, failure, api, log: () => {}, cancel: path => state.cancel(path),
rerunFailed: path => state.rerunFailed(path), readAnnotations: job => state.readAnnotations(job), readLog: job => state.readLog(job) });
state.job = (attempt, id, name, conclusion, evidence) => {
if (evidence) state.evidence[id] = evidence;
return { id, name, run_id: 99, run_attempt: attempt, head_sha: sha, status: conclusion ? "completed" : "in_progress", conclusion: conclusion ?? null };
};
return state;
}
const proofName = "Cloud source verified v1";
// The 2026-11-13 13:49:56Z reclaim of readiness run 43435593630 attempt 2:
// several jobs lost their Spot runners at once, siblings were cancelled, and
// the source proof was cancelled.
function reclaimedAttempt(g, attempt) {
return [
g.job(attempt, attempt * 100 + 1, "verify / Serialized tests (1/5)", "failure", { annotations: [cancelledNote], log: lostLog }),
g.job(attempt, attempt * 100 + 2, "verify / Verify Paperclip Runner (rust)", "failure", { annotations: [cancelledNote], log: lostLog }),
g.job(attempt, attempt * 100 + 3, "verify / General tests (server (1/10))", "cancelled", { annotations: [cancelledNote], log: "" }),
g.job(attempt, attempt * 100 + 4, "verify / Build", "success"),
g.job(attempt, attempt * 100 + 5, proofName, "cancelled"),
];
}
const greenAttempt = g => attempt => [g.job(attempt, attempt * 100 + 4, "verify / Build", "success"), g.job(attempt, attempt * 100 + 5, proofName, "success")];
// A genuine test failure, as in readiness run 43439942063 (decision-models).
const flakyAttempt = g => attempt => [
g.job(attempt, attempt * 100 + 1, "verify / General tests (server (4/10))", "failure", { annotations: [assertionNote], log: testLog }),
g.job(attempt, attempt * 100 + 3, "verify / General tests (server (1/10))", "cancelled", { annotations: [cancelledNote] }),
g.job(attempt, attempt * 100 + 5, proofName, "cancelled"),
];
const clock = () => { let time = 0; return { now: () => time, sleep: async ms => { time += ms; } }; };
describe("runner-loss classification", () => {
test("an attempt whose every failed job lost its runner is a runner loss; cancelled siblings are fallout", async () => {
const g = github({ attempts: {} });
const verdict = await classifyAttempt(reclaimedAttempt(g, 2), g);
assert.equal(verdict.lost, true);
assert.deepEqual(verdict.jobs.map(job => job.conclusion), ["failure", "failure"]);
});
test("one real failure beside lost runners is not a runner loss", async () => {
const g = github({ attempts: {} });
const jobs = [...reclaimedAttempt(g, 2), ...flakyAttempt(g)(2).slice(0, 1)];
assert.equal((await classifyAttempt(jobs, g)).lost, false);
});
test("a failing test is not a runner loss", async () => {
const g = github({ attempts: {} });
assert.equal((await classifyAttempt(flakyAttempt(g)(2), g)).lost, false);
});
test("a reclaimed job that concluded cancelled counts only with a runner-loss log", async () => {
const g = github({ attempts: {} });
const lost = [g.job(2, 1, "verify / Build", "cancelled", { annotations: [cancelledNote], log: lostLog }), g.job(2, 2, proofName, "cancelled")];
assert.equal((await classifyAttempt(lost, g)).lost, true);
// A person cancelling the run is not a runner loss.
const manual = [g.job(2, 3, "verify / Build", "cancelled", { annotations: [cancelledNote, { annotation_level: "failure", message: "The run was canceled by @someone." }] }), g.job(2, 4, proofName, "cancelled")];
assert.equal((await classifyAttempt(manual, g)).lost, false);
});
test("non-gating Runner chaos evals neither cause nor block a retry", async () => {
const g = github({ attempts: {} });
const chaos = g.job(2, 9, "Runner chaos evals (not a publication gate) / Restart, replay, trace, and recovery faults", "failure", { annotations: [assertionNote], log: testLog });
assert.equal((await classifyAttempt([...reclaimedAttempt(g, 2), chaos], g)).lost, true);
});
test("unreadable job evidence fails closed", async () => {
const g = github({ attempts: {} });
const verdict = await classifyAttempt(reclaimedAttempt(g, 2), { ...g, readLog: async () => { throw new Error("HTTP 404"); } });
assert.equal(verdict.lost, false);
});
});
describe("runner-loss retry", () => {
test("a completed reclaimed attempt re-runs only its failed jobs", async () => {
const g = github({ attempts: {}, startAttempt: 2 });
g.attempts[2] = reclaimedAttempt(g, 2);
assert.equal(await g.retry({ failure: { runId: 99, attempt: 2 }, api: g.api }), "retrying");
assert.deepEqual(g.writes, ["/repos/paperclipai/paperclip/actions/runs/99/rerun-failed-jobs"]);
});
test("a reclaimed attempt still running is ended first, then re-run on the next poll", async () => {
const g = github({ attempts: {}, startAttempt: 2, status: "in_progress" });
g.attempts[2] = reclaimedAttempt(g, 2);
assert.equal(await g.retry({ failure: { runId: 99, attempt: 2 }, api: g.api }), "retrying");
assert.deepEqual(g.writes, ["/repos/paperclipai/paperclip/actions/runs/99/cancel"]);
assert.equal(await g.retry({ failure: { runId: 99, attempt: 2 }, api: g.api }), "retrying");
assert.equal(g.writes.at(-1), "/repos/paperclipai/paperclip/actions/runs/99/rerun-failed-jobs");
});
test("a newer attempt started while evidence was read is never cancelled", async () => {
const g = github({ attempts: {}, startAttempt: 2, status: "in_progress" });
g.attempts[2] = reclaimedAttempt(g, 2);
const readLog = g.readLog;
// The other waiter ends attempt 2 and starts attempt 3 during our log read.
g.readLog = async job => { g.attempt = 3; g.status = "in_progress"; return readLog(job); };
assert.equal(await g.retry({ failure: { runId: 99, attempt: 2 }, api: g.api }), "retrying");
assert.deepEqual(g.writes, []);
});
test("a cancellation conflict after the attempt completed keeps waiting", async () => {
const g = github({ attempts: {}, startAttempt: 2, status: "in_progress" });
g.attempts[2] = reclaimedAttempt(g, 2);
g.cancel = async () => { g.status = "completed"; throw Object.assign(new Error("HTTP 409"), { status: 409 }); };
assert.equal(await g.retry({ failure: { runId: 99, attempt: 2 }, api: g.api }), "retrying");
});
test("an unexpected cancellation error with the attempt still running is surfaced", async () => {
const g = github({ attempts: {}, startAttempt: 2, status: "in_progress" });
g.attempts[2] = reclaimedAttempt(g, 2);
g.cancel = async () => { throw Object.assign(new Error("HTTP 500"), { status: 500 }); };
await assert.rejects(g.retry({ failure: { runId: 99, attempt: 2 }, api: g.api }), /HTTP 500/);
});
test("a real failure is never retried", async () => {
const g = github({ attempts: {}, startAttempt: 2 });
g.attempts[2] = flakyAttempt(g)(2);
assert.equal(await g.retry({ failure: { runId: 99, attempt: 2 }, api: g.api }), "failed");
assert.deepEqual(g.writes, []);
});
test("retries stop at the attempt ceiling", async () => {
const g = github({ attempts: {}, startAttempt: maxRunnerLossAttempt });
g.attempts[maxRunnerLossAttempt] = reclaimedAttempt(g, maxRunnerLossAttempt);
assert.equal(await g.retry({ failure: { runId: 99, attempt: maxRunnerLossAttempt }, api: g.api }), "failed");
assert.deepEqual(g.writes, []);
});
test("a newer attempt started by the other waiter or a person is waited on, not re-run", async () => {
const g = github({ attempts: {}, startAttempt: 3, status: "in_progress" });
assert.equal(await g.retry({ failure: { runId: 99, attempt: 2 }, api: g.api }), "retrying");
assert.deepEqual(g.writes, []);
});
test("the other waiter winning the same re-run is not a failure", async () => {
const g = github({ attempts: {}, startAttempt: 2 });
g.attempts[2] = reclaimedAttempt(g, 2);
g.rerunFailed = async () => { g.attempt += 1; g.status = "in_progress"; throw Object.assign(new Error("HTTP 403"), { status: 403 }); };
assert.equal(await g.retry({ failure: { runId: 99, attempt: 2 }, api: g.api }), "retrying");
});
test("a changed run identity refuses to write", async () => {
const g = github({ attempts: {}, startAttempt: 2 });
g.attempts[2] = reclaimedAttempt(g, 2);
await assert.rejects(retryRunnerLoss({ sha: "b".repeat(40), failure: { runId: 99, attempt: 2 }, api: g.api, log: () => {},
cancel: g.cancel, rerunFailed: g.rerunFailed, readAnnotations: g.readAnnotations, readLog: g.readLog }), /identity changed/);
assert.deepEqual(g.writes, []);
});
test("every waiter job that may retry can read runner-loss evidence", async () => {
const { readFileSync } = await import("node:fs");
const read = file => readFileSync(new URL(`../.github/workflows/${file}`, import.meta.url), "utf8");
const job = (text, name) => text.match(new RegExp(`\\n ${name}:\\n[\\s\\S]*?(?=\\n [a-z_-]+:\\n)`))[0];
for (const [file, name] of [["release.yml", "verify_canary"], ["docker.yml", "source_verification"], ["docker.yml", "publication_verification"]]) {
const text = job(read(file), name);
assert.match(text, /\n checks: read\b/, `${file} ${name} needs checks: read for annotations`);
assert.match(text, /\n actions: write\b/, `${file} ${name} needs actions: write for logs and retries`);
}
});
test("the production writer only posts cancel or rerun-failed-jobs to a canonical run", async () => {
const posts = [];
const fetcher = async (url, init) => { posts.push([url, init.method]); return new Response(null, { status: 201 }); };
const g = github({ attempts: {}, startAttempt: 2 });
g.attempts[2] = reclaimedAttempt(g, 2);
const evidence = async url => url.includes("/annotations") ? Response.json([cancelledNote]) : new Response(lostLog);
const retry = createRunnerLossRetry({ token: "t", log: () => {}, fetcher: async (url, init = {}) => init.method === "POST" ? fetcher(url, init) : evidence(url) });
assert.equal(await retry({ sha, failure: { runId: 99, attempt: 2 }, api: g.api }), "retrying");
assert.deepEqual(posts, [["https://api.github.com/repos/paperclipai/paperclip/actions/runs/99/rerun-failed-jobs", "POST"]]);
});
});
// Simulations of the 2026-11-13 hold: readiness attempt 2 loses its runners.
// Before this change both waiters failed the release on that attempt.
describe("busy-master simulation", () => {
test("the npm canary gate retries a reclaimed attempt and publishes on the retried proof", async () => {
const g = github({ attempts: {}, startAttempt: 2, script: [] });
g.attempts[2] = reclaimedAttempt(g, 2);
g.script.push(greenAttempt(g));
const proof = await waitForSourceVerification(sha, { api: g.api, ...clock(), log: () => {}, retryRunnerLoss: g.retry });
assert.deepEqual({ runId: proof.runId, attempt: proof.attempt }, { runId: 99, attempt: 3 });
assert.deepEqual(g.writes, ["/repos/paperclipai/paperclip/actions/runs/99/rerun-failed-jobs"]);
});
test("without the retry the same reclaim fails the release (the 2026-11-13 behavior)", async () => {
const g = github({ attempts: {}, startAttempt: 2 });
g.attempts[2] = reclaimedAttempt(g, 2);
await assert.rejects(waitForSourceVerification(sha, { api: g.api, ...clock(), log: () => {} }), /did not pass/);
});
test("two reclaims in a row still publish; a third stops at the ceiling", async () => {
const g = github({ attempts: {}, startAttempt: 2 });
g.attempts[2] = reclaimedAttempt(g, 2);
g.script.push(attempt => reclaimedAttempt(g, attempt), greenAttempt(g));
const proof = await waitForSourceVerification(sha, { api: g.api, ...clock(), log: () => {}, retryRunnerLoss: g.retry });
assert.equal(proof.attempt, 4);
const h = github({ attempts: {}, startAttempt: 2 });
h.attempts[2] = reclaimedAttempt(h, 2);
h.script.push(attempt => reclaimedAttempt(h, attempt), attempt => reclaimedAttempt(h, attempt));
await assert.rejects(waitForSourceVerification(sha, { api: h.api, ...clock(), log: () => {}, retryRunnerLoss: h.retry }), /attempt 4/);
});
test("a real test failure still fails the release without a retry", async () => {
const g = github({ attempts: {}, startAttempt: 2 });
g.attempts[2] = flakyAttempt(g)(2);
await assert.rejects(waitForSourceVerification(sha, { api: g.api, ...clock(), log: () => {}, retryRunnerLoss: g.retry }), /did not pass/);
assert.deepEqual(g.writes, []);
});
test("the image watcher keeps its image attempt alive through a reclaim instead of cancelling it", async () => {
const g = github({ attempts: {}, startAttempt: 2, status: "in_progress" });
g.attempts[2] = reclaimedAttempt(g, 2);
g.script.push(greenAttempt(g));
const imageCancels = [];
const proof = await watchReleaseSource({ sha, runId: 50, attempt: 1, api: g.api, ...clock(), log: () => {},
cancel: async path => imageCancels.push(path), retryRunnerLoss: g.retry });
assert.equal(proof.attempt, 3);
assert.deepEqual(imageCancels, []);
assert.deepEqual(g.writes, ["/repos/paperclipai/paperclip/actions/runs/99/cancel", "/repos/paperclipai/paperclip/actions/runs/99/rerun-failed-jobs"]);
});
test("the image watcher still cancels image work on a real failure", async () => {
const g = github({ attempts: {}, startAttempt: 2, status: "in_progress" });
g.attempts[2] = flakyAttempt(g)(2);
const imageCancels = [];
await assert.rejects(watchReleaseSource({ sha, runId: 50, attempt: 1, api: g.api, ...clock(), log: () => {},
cancel: async path => imageCancels.push(path), retryRunnerLoss: g.retry }), /General tests/);
assert.deepEqual(imageCancels, ["/repos/paperclipai/paperclip/actions/runs/99/cancel", "/repos/paperclipai/paperclip/actions/runs/50/cancel"]);
});
});
+15 -6
View File
@@ -2,6 +2,7 @@ import { appendFile } from "node:fs/promises";
import { pathToFileURL } from "node:url";
import { createActionsReader } from "./github-actions-reader.mjs";
import { deferredProducer, requestDeferredProducer, sourceAdmissionJob, createReleaseRerunner } from "./cloud-release-admission.mjs";
import { createRunnerLossRetry } from "./cloud-readiness-runner-loss.mjs";
const repository = "paperclipai/paperclip";
const workflowPath = ".github/workflows/cloud-readiness.yml";
@@ -96,7 +97,7 @@ export { createActionsReader };
export async function waitForSourceVerification(sha, {
api, now = Date.now, sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)),
timeoutMs = 60 * 60_000, intervalMs = 30_000, log = console.log, rerun,
timeoutMs = 90 * 60_000, intervalMs = 30_000, log = console.log, rerun, retryRunnerLoss,
} = {}) {
assertSha(sha);
const deadline = now() + timeoutMs;
@@ -107,8 +108,13 @@ export async function waitForSourceVerification(sha, {
const result = await readSourceVerification(sha, api);
if (result) return result;
} catch (error) {
if (!(error instanceof SourceVerificationDeferred) || !rerun) throw error;
await requestDeferredProducer({ file: "cloud-readiness.yml", sha, admissionJob: sourceAdmissionJob, api, rerun, requested });
if (error instanceof SourceVerificationFailed && retryRunnerLoss) {
// A reclaimed Spot worker is not a source failure: retry its jobs.
if (await retryRunnerLoss({ sha, failure: error, api }) !== "retrying") throw error;
} else {
if (!(error instanceof SourceVerificationDeferred) || !rerun) throw error;
await requestDeferredProducer({ file: "cloud-readiness.yml", sha, admissionJob: sourceAdmissionJob, api, rerun, requested });
}
}
const remaining = deadline - now();
if (remaining > 0) await sleep(Math.min(intervalMs, remaining));
@@ -121,11 +127,14 @@ if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href)
if (!process.env.GITHUB_TOKEN) throw new Error("GITHUB_TOKEN with Actions read access is required.");
// One deadline for both layers: the reader stops retrying when the poll it
// serves is out of time, instead of extending the wait past its timeout.
const timeoutMs = 60 * 60_000;
// Long enough for a deferred full run plus two runner-loss retries.
const timeoutMs = 90 * 60_000;
const deadline = Date.now() + timeoutMs;
const api = createActionsReader({ token: process.env.GITHUB_TOKEN, deadlineAt: () => deadline });
const rerun = process.env.GITHUB_REPOSITORY === repository && process.env.GITHUB_REF === "refs/heads/master" ? createReleaseRerunner(process.env.GITHUB_TOKEN) : undefined;
const proof = await waitForSourceVerification(process.argv[2], { api, timeoutMs, rerun });
const canonical = process.env.GITHUB_REPOSITORY === repository && process.env.GITHUB_REF === "refs/heads/master";
const rerun = canonical ? createReleaseRerunner(process.env.GITHUB_TOKEN) : undefined;
const retryRunnerLoss = canonical ? createRunnerLossRetry({ token: process.env.GITHUB_TOKEN }) : undefined;
const proof = await waitForSourceVerification(process.argv[2], { api, timeoutMs, rerun, retryRunnerLoss });
const message = `Source verification passed for ${proof.sha}: https://github.com/${repository}/actions/runs/${proof.runId}/attempts/${proof.attempt} (job ${proof.jobId}).`;
console.log(message);
if (process.env.GITHUB_STEP_SUMMARY) await appendFile(process.env.GITHUB_STEP_SUMMARY, `${message}\n`);
+7 -1
View File
@@ -2,6 +2,7 @@ import { appendFile } from "node:fs/promises";
import { pathToFileURL } from "node:url";
import { createActionsReader, readSourceVerification, SourceVerificationFailed, SourceVerificationDeferred } from "./cloud-source-verification.mjs";
import { requestDeferredProducer, readMigratorPublication, sourceAdmissionJob, migratorAdmissionJob, createReleaseRerunner } from "./cloud-release-admission.mjs";
import { createRunnerLossRetry } from "./cloud-readiness-runner-loss.mjs";
const repository = "paperclipai/paperclip";
@@ -35,7 +36,7 @@ async function cancelFailedVerification({ api, cancel, sha }, failure) {
if (run.status !== "completed") await cancel(`/repos/${repository}/actions/runs/${failure.runId}/cancel`);
}
export async function watchReleaseSource({ sha, runId, attempt, api, cancel, rerun, now = Date.now,
export async function watchReleaseSource({ sha, runId, attempt, api, cancel, rerun, retryRunnerLoss, now = Date.now,
sleep = ms => new Promise(resolve => setTimeout(resolve, ms)), log = console.log,
timeoutMs = 40 * 60_000, intervalMs = 30_000, requireProof = false, summary = async () => {} }) {
if (!/^[a-f0-9]{40}$/.test(sha ?? "") || !Number.isSafeInteger(runId) || runId < 1 ||
@@ -68,6 +69,10 @@ export async function watchReleaseSource({ sha, runId, attempt, api, cancel, rer
} catch (error) {
if (error instanceof SourceVerificationDeferred && rerun) {
await requestDeferredProducer({ file: "cloud-readiness.yml", sha, admissionJob: sourceAdmissionJob, api, rerun, requested });
} else if (error instanceof SourceVerificationFailed && retryRunnerLoss &&
await retryRunnerLoss({ sha, failure: error, api }) === "retrying") {
// A reclaimed Spot worker is retried, not treated as a failed source:
// keep this image attempt alive for the retried proof.
} else {
if (error instanceof SourceVerificationFailed) {
await cancelFailedVerification({ api, cancel, sha }, error);
@@ -95,6 +100,7 @@ if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href)
const proof = checkProof ? await readSourceVerification(process.env.GITHUB_SHA, api, { failFast: true }) : await watchReleaseSource({ requireProof, sha: process.env.GITHUB_SHA, runId: Number(process.env.GITHUB_RUN_ID), attempt: Number(process.env.GITHUB_RUN_ATTEMPT),
api,
rerun: createReleaseRerunner(token),
retryRunnerLoss: createRunnerLossRetry({ token }),
cancel: async path => {
const response = await fetch(`https://api.github.com${path}`, { method: "POST", redirect: "error",
headers: { Authorization: `Bearer ${token}`, Accept: "application/vnd.github+json" }, signal: AbortSignal.timeout(30_000) });