mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every push to `master` publishes a canary, and the canary's image reaches staging > - Each canary waits for `Cloud source verified v1` from the Cloud readiness run of its exact SHA > - Cloud readiness runs on AWS Spot workers. When a worker is reclaimed, the gate failed and the image watcher cancelled the readiness run > - Nothing retried that attempt. On a busy master the newest commit's release failed, and merged work did not reach staging until a person re-ran the run > - This pull request re-runs the failed jobs of a readiness attempt whose every failure is a lost runner. A real check failure still fails closed > - The benefit is that a Spot reclaim no longer holds a release ## Linked Issues or Issue Description **What happened?** On 2026-11-13, #21266 merged as `8af889f612`. No canary that contained it published for more than an hour. The release workflow keeps one running and one pending canary. A newer push replaces the pending one, so the newest commit always runs next. That part works as designed. The failures came from the readiness gate: | Release (SHA) | Readiness run / attempt | Gate that blocked it | Cause | | --- | --- | --- | --- | | `11840f9904` | 43435593630 / 2 | `Reuse exact-source verification` | Spot workers reclaimed at 13:49:56Z. The run was then cancelled by `github-actions[bot]` (the image watcher). Dotta re-ran attempt 3 by hand | | `8af889f612` | 43434587048 / 2 | Release superseded while pending; readiness failed in the same reclaim | Spot reclaim | | `9fbfd93c9b` | 43423674511 / 2 | `Reuse exact-source verification` | Spot reclaim | | `6917d0568d` | 43439942063 / 1 | `Reuse exact-source verification` | Flaky test `decision-models.test.ts` (concurrent budget admission) | | `18fb6ee571` | 43422042261 / 2 | `Reuse exact-source verification` | Flaky test `fresh-continuation-fallback.test.ts` (truncate) | | `5ede64f247` | 43421480302 / 1 | `Reuse exact-source verification` | Both flaky tests | Over the last 200 master pushes, release runs ended 14 success, 33 failure, and 150 cancelled (superseded while pending, which is expected). **Expected behavior** A lost runner is not a source failure. The gate retries the lost jobs and publishes the newest commit that passes. **Steps to reproduce** See `scripts/cloud-readiness-runner-loss.test.mjs`, "busy-master simulation". It replays the 13:49:56Z reclaim through both waiters. ## What Changed - `scripts/cloud-readiness-runner-loss.mjs` (new): classifies a failed readiness attempt with the existing PR classifier (`.github/scripts/runner-loss-rerun.mjs`). It is a runner loss only when every causing job (`failure`, `timed_out`, `startup_failure`) lost its runner. Cancelled siblings are fallout. A run that is still going is cancelled first, then `rerun-failed-jobs` runs on the next poll. The ceiling is readiness attempt 4. Writes go only to `cancel` or `rerun-failed-jobs` of a canonical run, after an identity check. - `scripts/cloud-source-verification.mjs` (canary gate in `release.yml`): on `SourceVerificationFailed`, ask the retry first. The source wait goes from 60 to 90 minutes. - `scripts/watch-release-source.mjs` (master image watcher in `docker.yml`): on a runner loss, keep the image attempt alive and wait for the retried proof. Do not cancel it. - `.github/workflows/release.yml`: `verify_canary` timeout goes from 70 to 100 minutes. - `.github/workflows/pr-trusted.yml`: run the new test file. - `doc/RELEASING.md`: new "Busy master and lost runners" section. ## Verification - `node --test scripts/cloud-readiness-runner-loss.test.mjs scripts/cloud-source-verification.test.mjs .github/scripts/tests/*.test.mjs scripts/__tests__/release-verify-workflow.test.mjs`: all pass. `release-verify-workflow.test.mjs` passes 51 of 51 but does not exit locally. This also happens on `origin/master` without this change. - Red/green: with the two waiter files reverted, 3 of the 6 simulation tests fail (the canary gate, two reclaims in a row, and the image watcher). With the change, all 20 pass. - Dry run against real runs (read-only, writes captured): 43435593630/2, 43434587048/2, and 43423674511/2 give `rerun-failed-jobs`. 43439942063/1, 43421480302/1, and 43422042261/2 (real test failures) give `failed` with no write. - `pnpm verify` exits 2 because a workflow file changed. Required CI supplies broad coverage. ## Risks - A real failure classified as a runner loss would be retried. The classifier needs the runner's own shutdown or lost-communication `##[error]` line in the job log. It is the same classifier PR CI uses, and unreadable evidence fails closed. - On a reclaim, the waiter cancels the readiness run to retry sooner. Jobs that were still healthy run again. This costs runner time, not correctness. - Flaky tests still fail a canary. This change does not retry them. They need their own fixes. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
131 lines
7.9 KiB
JavaScript
131 lines
7.9 KiB
JavaScript
// A Cloud readiness attempt that failed only because AWS Spot workers were
|
|
// reclaimed proves nothing about the source. Before this module, the first
|
|
// lost runner failed the release's source gate, the image watcher cancelled
|
|
// the readiness run, and nothing retried it: on 2026-11-13 a 13:49:56Z fleet
|
|
// reclaim held every release containing #21266 until a person re-ran the run.
|
|
//
|
|
// Both waiters (the npm canary gate in release.yml and the image watcher in
|
|
// docker.yml) call this when the source proof fails. A failure whose every
|
|
// causing job lost its runner re-runs the failed jobs instead, up to a fixed
|
|
// attempt ceiling. A real check failure still fails closed: it is never
|
|
// retried here.
|
|
import { classifyJob, createGithubClient } from "../.github/scripts/runner-loss-rerun.mjs";
|
|
|
|
const repository = "paperclipai/paperclip";
|
|
const workflowPath = ".github/workflows/cloud-readiness.yml";
|
|
// Attempt 1 is often the deferred admission-only pass and attempt 2 the
|
|
// admitted full run, so 4 allows at least two runner-loss retries. The ceiling
|
|
// is stateless: every waiter of every release attempt agrees on it.
|
|
export const maxRunnerLossAttempt = 4;
|
|
// A causing job ended the attempt on its own. Cancelled jobs are fallout from
|
|
// that failure (the image watcher or a reusable workflow cancelled them).
|
|
const causeConclusions = new Set(["failure", "timed_out", "startup_failure"]);
|
|
const nonGating = name => name.startsWith("Runner chaos evals (not a publication gate) / ");
|
|
|
|
function trusted(run, sha, runId) {
|
|
return run?.id === runId && run.path === workflowPath && run.head_sha === sha && run.head_branch === "master" &&
|
|
run.event === "push" && run.repository?.full_name === repository && run.head_repository?.full_name === repository &&
|
|
Number.isSafeInteger(run.run_attempt) && run.run_attempt > 0;
|
|
}
|
|
|
|
async function attemptJobs(api, runId, attempt) {
|
|
const jobs = [];
|
|
for (let page = 1; page <= 10; page += 1) {
|
|
const batch = await api(`/repos/${repository}/actions/runs/${runId}/attempts/${attempt}/jobs?per_page=100&page=${page}`);
|
|
if (!Array.isArray(batch.jobs)) throw new Error("Cloud readiness job listing is malformed.");
|
|
jobs.push(...batch.jobs);
|
|
if (batch.jobs.length < 100) return jobs;
|
|
}
|
|
throw new Error("Cloud readiness job listing is incomplete.");
|
|
}
|
|
|
|
/** Which failed jobs lost their runner. `lost` is true only when at least one
|
|
* job lost its runner and no job failed for any other reason. */
|
|
export async function classifyAttempt(jobs, { readAnnotations, readLog }) {
|
|
const ended = jobs.filter(job => job.status === "completed" && !nonGating(job.name ?? ""));
|
|
const causes = ended.filter(job => causeConclusions.has(job.conclusion));
|
|
// A reclaimed worker sometimes concludes its job `cancelled`. With no other
|
|
// cause, a cancelled job counts only when its own log shows the loss.
|
|
const candidates = causes.length ? causes : ended.filter(job => job.conclusion === "cancelled");
|
|
const results = [];
|
|
for (const job of candidates) {
|
|
try {
|
|
results.push({ id: job.id, name: job.name, conclusion: job.conclusion,
|
|
...(await classifyJob(job, { annotations: await readAnnotations(job), readLog })) });
|
|
} catch (error) {
|
|
results.push({ id: job.id, name: job.name, conclusion: job.conclusion, lost: false, evidence: `could not read job evidence: ${error.message}` });
|
|
}
|
|
}
|
|
const lost = causes.length ? results.length > 0 && results.every(job => job.lost) : results.some(job => job.lost);
|
|
return { lost, jobs: results };
|
|
}
|
|
|
|
/**
|
|
* Called after a source proof read failed for `failure.runId`/`failure.attempt`.
|
|
* Returns "retrying" when the waiter should keep waiting (a runner-loss retry
|
|
* was requested, is pending, or a newer attempt already exists) and "failed"
|
|
* when the failure stands.
|
|
*/
|
|
export async function retryRunnerLoss({ sha, failure, api, readAnnotations, readLog, cancel, rerunFailed, log = console.log }) {
|
|
const run = await api(`/repos/${repository}/actions/runs/${failure.runId}`);
|
|
if (!trusted(run, sha, failure.runId)) throw new Error("Cloud readiness identity changed before a runner-loss retry.");
|
|
// Someone (another waiter or a person) already started the next attempt.
|
|
if (run.run_attempt > failure.attempt) return "retrying";
|
|
if (run.run_attempt !== failure.attempt) return "failed";
|
|
const jobs = (await attemptJobs(api, run.id, run.run_attempt))
|
|
.filter(job => job.run_id === run.id && job.run_attempt === run.run_attempt && job.head_sha === sha);
|
|
const verdict = await classifyAttempt(jobs, { readAnnotations, readLog });
|
|
for (const job of verdict.jobs) log(`Cloud readiness ${run.id} attempt ${run.run_attempt}: ${job.name} ${job.lost ? "lost its runner" : "failed"} (${job.evidence}).`);
|
|
if (!verdict.lost) return "failed";
|
|
if (run.run_attempt >= maxRunnerLossAttempt) {
|
|
log(`Cloud readiness ${run.id} lost runners on attempt ${run.run_attempt}, the last automatic attempt.`);
|
|
return "failed";
|
|
}
|
|
// Re-running failed jobs needs a completed run. Healthy jobs still running
|
|
// would be repeated anyway once they are cancelled, so end the attempt now.
|
|
if (run.status !== "completed") {
|
|
// Job evidence reads take time. Another waiter may have ended this attempt
|
|
// and started the next one meanwhile: never cancel a newer attempt.
|
|
const fresh = await api(`/repos/${repository}/actions/runs/${run.id}`);
|
|
if (!trusted(fresh, sha, run.id)) throw new Error("Cloud readiness identity changed before a runner-loss retry.");
|
|
if (fresh.run_attempt !== run.run_attempt || fresh.status === "completed") return "retrying";
|
|
log(`Cloud readiness ${run.id} attempt ${run.run_attempt} lost its runner; ending the attempt to retry its failed jobs.`);
|
|
try {
|
|
await cancel(`/repos/${repository}/actions/runs/${run.id}/cancel`);
|
|
} catch (error) {
|
|
// The attempt finished or was cancelled by the other waiter first.
|
|
const latest = await api(`/repos/${repository}/actions/runs/${run.id}`);
|
|
if (!trusted(latest, sha, run.id)) throw error;
|
|
if (latest.run_attempt > run.run_attempt || latest.status === "completed" || [409, 422].includes(error.status)) return "retrying";
|
|
throw error;
|
|
}
|
|
return "retrying";
|
|
}
|
|
try {
|
|
await rerunFailed(`/repos/${repository}/actions/runs/${run.id}/rerun-failed-jobs`);
|
|
log(`Cloud readiness ${run.id} attempt ${run.run_attempt} lost its runner; re-ran its failed jobs.`);
|
|
} catch (error) {
|
|
// The other waiter may have won the same write, or GitHub has not
|
|
// finished cancelling the attempt. Both resolve on the next poll.
|
|
const latest = await api(`/repos/${repository}/actions/runs/${run.id}`);
|
|
if (!trusted(latest, sha, run.id)) throw error;
|
|
if (latest.run_attempt > run.run_attempt || latest.status !== "completed" || [409, 422].includes(error.status)) return "retrying";
|
|
throw error;
|
|
}
|
|
return "retrying";
|
|
}
|
|
|
|
/** The canonical writes and evidence reads `retryRunnerLoss` needs. */
|
|
export function createRunnerLossRetry({ token, fetcher = fetch, log = console.log }) {
|
|
const client = createGithubClient({ token, fetcher });
|
|
const write = async path => {
|
|
if (!/^\/repos\/paperclipai\/paperclip\/actions\/runs\/[1-9]\d*\/(cancel|rerun-failed-jobs)$/.test(path)) throw new Error("Invalid runner-loss retry path.");
|
|
const response = await fetcher(`https://api.github.com${path}`, { method: "POST", redirect: "error",
|
|
headers: { Authorization: `Bearer ${token}`, Accept: "application/vnd.github+json" }, signal: AbortSignal.timeout(30_000) });
|
|
if (!response.ok) throw Object.assign(new Error(`Could not retry a lost readiness runner: HTTP ${response.status}.`), { status: response.status });
|
|
};
|
|
return ({ sha, failure, api }) => retryRunnerLoss({ sha, failure, api, log, cancel: write, rerunFailed: write,
|
|
readAnnotations: job => client.paginate(`/repos/${repository}/check-runs/${job.id}/annotations?per_page=100`),
|
|
readLog: job => client.text(`/repos/${repository}/actions/jobs/${job.id}/logs`) });
|
|
}
|