mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every push to `master` publishes a canary, and the canary's image reaches staging > - Each canary waits for `Cloud source verified v1` from the Cloud readiness run of its exact SHA > - Cloud readiness runs on AWS Spot workers. When a worker is reclaimed, the gate failed and the image watcher cancelled the readiness run > - Nothing retried that attempt. On a busy master the newest commit's release failed, and merged work did not reach staging until a person re-ran the run > - This pull request re-runs the failed jobs of a readiness attempt whose every failure is a lost runner. A real check failure still fails closed > - The benefit is that a Spot reclaim no longer holds a release ## Linked Issues or Issue Description **What happened?** On 2026-11-13, #21266 merged as `8af889f612`. No canary that contained it published for more than an hour. The release workflow keeps one running and one pending canary. A newer push replaces the pending one, so the newest commit always runs next. That part works as designed. The failures came from the readiness gate: | Release (SHA) | Readiness run / attempt | Gate that blocked it | Cause | | --- | --- | --- | --- | | `11840f9904` | 43435593630 / 2 | `Reuse exact-source verification` | Spot workers reclaimed at 13:49:56Z. The run was then cancelled by `github-actions[bot]` (the image watcher). Dotta re-ran attempt 3 by hand | | `8af889f612` | 43434587048 / 2 | Release superseded while pending; readiness failed in the same reclaim | Spot reclaim | | `9fbfd93c9b` | 43423674511 / 2 | `Reuse exact-source verification` | Spot reclaim | | `6917d0568d` | 43439942063 / 1 | `Reuse exact-source verification` | Flaky test `decision-models.test.ts` (concurrent budget admission) | | `18fb6ee571` | 43422042261 / 2 | `Reuse exact-source verification` | Flaky test `fresh-continuation-fallback.test.ts` (truncate) | | `5ede64f247` | 43421480302 / 1 | `Reuse exact-source verification` | Both flaky tests | Over the last 200 master pushes, release runs ended 14 success, 33 failure, and 150 cancelled (superseded while pending, which is expected). **Expected behavior** A lost runner is not a source failure. The gate retries the lost jobs and publishes the newest commit that passes. **Steps to reproduce** See `scripts/cloud-readiness-runner-loss.test.mjs`, "busy-master simulation". It replays the 13:49:56Z reclaim through both waiters. ## What Changed - `scripts/cloud-readiness-runner-loss.mjs` (new): classifies a failed readiness attempt with the existing PR classifier (`.github/scripts/runner-loss-rerun.mjs`). It is a runner loss only when every causing job (`failure`, `timed_out`, `startup_failure`) lost its runner. Cancelled siblings are fallout. A run that is still going is cancelled first, then `rerun-failed-jobs` runs on the next poll. The ceiling is readiness attempt 4. Writes go only to `cancel` or `rerun-failed-jobs` of a canonical run, after an identity check. - `scripts/cloud-source-verification.mjs` (canary gate in `release.yml`): on `SourceVerificationFailed`, ask the retry first. The source wait goes from 60 to 90 minutes. - `scripts/watch-release-source.mjs` (master image watcher in `docker.yml`): on a runner loss, keep the image attempt alive and wait for the retried proof. Do not cancel it. - `.github/workflows/release.yml`: `verify_canary` timeout goes from 70 to 100 minutes. - `.github/workflows/pr-trusted.yml`: run the new test file. - `doc/RELEASING.md`: new "Busy master and lost runners" section. ## Verification - `node --test scripts/cloud-readiness-runner-loss.test.mjs scripts/cloud-source-verification.test.mjs .github/scripts/tests/*.test.mjs scripts/__tests__/release-verify-workflow.test.mjs`: all pass. `release-verify-workflow.test.mjs` passes 51 of 51 but does not exit locally. This also happens on `origin/master` without this change. - Red/green: with the two waiter files reverted, 3 of the 6 simulation tests fail (the canary gate, two reclaims in a row, and the image watcher). With the change, all 20 pass. - Dry run against real runs (read-only, writes captured): 43435593630/2, 43434587048/2, and 43423674511/2 give `rerun-failed-jobs`. 43439942063/1, 43421480302/1, and 43422042261/2 (real test failures) give `failed` with no write. - `pnpm verify` exits 2 because a workflow file changed. Required CI supplies broad coverage. ## Risks - A real failure classified as a runner loss would be retried. The classifier needs the runner's own shutdown or lost-communication `##[error]` line in the job log. It is the same classifier PR CI uses, and unreadable evidence fails closed. - On a reclaim, the waiter cancels the readiness run to retry sooner. Jobs that were still healthy run again. This costs runner time, not correctness. - Flaky tests still fail a canary. This change does not retry them. They need their own fixes. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
146 lines
8.4 KiB
JavaScript
146 lines
8.4 KiB
JavaScript
import { appendFile } from "node:fs/promises";
|
|
import { pathToFileURL } from "node:url";
|
|
import { createActionsReader } from "./github-actions-reader.mjs";
|
|
import { deferredProducer, requestDeferredProducer, sourceAdmissionJob, createReleaseRerunner } from "./cloud-release-admission.mjs";
|
|
import { createRunnerLossRetry } from "./cloud-readiness-runner-loss.mjs";
|
|
|
|
const repository = "paperclipai/paperclip";
|
|
const workflowPath = ".github/workflows/cloud-readiness.yml";
|
|
export const sourceVerificationJob = "Cloud source verified v1";
|
|
// Owner decision (2026-11-12): the Runner chaos evals run in the same Cloud
|
|
// readiness run but are not a publication prerequisite. Their reusable-workflow
|
|
// jobs are named "<this> / <inner job>". Their failure still fails their own
|
|
// check and the run; it must not fail or cancel the source proof or the image.
|
|
export const runnerChaosEvalsJob = "Runner chaos evals (not a publication gate)";
|
|
const nonGating = job => typeof job.name === "string" && job.name.startsWith(`${runnerChaosEvalsJob} / `);
|
|
export class SourceVerificationFailed extends Error {
|
|
constructor(message, run) {
|
|
super(message);
|
|
this.runId = run.id;
|
|
this.attempt = run.run_attempt;
|
|
}
|
|
}
|
|
export class SourceVerificationDeferred extends Error {}
|
|
|
|
function assertSha(sha) {
|
|
if (!/^[a-f0-9]{40}$/.test(sha ?? "")) throw new Error("A full lowercase source SHA is required.");
|
|
}
|
|
|
|
function trustedRun(run, sha, workflowId) {
|
|
return run.workflow_id === workflowId && run.path === workflowPath &&
|
|
run.repository?.full_name === repository && run.head_repository?.full_name === repository &&
|
|
run.head_sha === sha && run.head_branch === "master" && run.event === "push" &&
|
|
Number.isSafeInteger(run.id) && run.id > 0 &&
|
|
Number.isSafeInteger(run.run_attempt) && run.run_attempt > 0;
|
|
}
|
|
|
|
// Consume one versioned job, independent of image/migrator availability. A
|
|
// failed image build must not invalidate source checks that already passed.
|
|
export async function readSourceVerification(sha, api, { failFast = false, onJobs } = {}) {
|
|
assertSha(sha);
|
|
const workflow = await api(`/repos/${repository}/actions/workflows/cloud-readiness.yml`);
|
|
if (workflow.path !== workflowPath || !Number.isSafeInteger(workflow.id) || workflow.id < 1) {
|
|
throw new Error("Cloud readiness workflow identity does not match.");
|
|
}
|
|
const listing = await api(`/repos/${repository}/actions/workflows/${workflow.id}/runs?head_sha=${sha}&event=push&branch=master&per_page=100`);
|
|
if (!Array.isArray(listing.workflow_runs) || !Number.isSafeInteger(listing.total_count) ||
|
|
listing.total_count < 0 || listing.total_count > 100 || listing.workflow_runs.length !== listing.total_count) {
|
|
throw new Error("Cloud readiness run listing is incomplete.");
|
|
}
|
|
const run = listing.workflow_runs.filter((candidate) => trustedRun(candidate, sha, workflow.id))
|
|
.sort((a, b) => b.id - a.id)[0];
|
|
if (!run) return undefined;
|
|
|
|
// Attempt-specific jobs prevent an earlier successful attempt from blessing
|
|
// a later rerun. Keep pagination even though today's matrix fits one page.
|
|
const jobs = [];
|
|
for (let page = 1; page <= 10; page += 1) {
|
|
const batch = await api(`/repos/${repository}/actions/runs/${run.id}/attempts/${run.run_attempt}/jobs?per_page=100&page=${page}`);
|
|
if (!Array.isArray(batch.jobs)) throw new Error("Cloud readiness job listing is malformed.");
|
|
jobs.push(...batch.jobs);
|
|
if (batch.jobs.length < 100) break;
|
|
if (page === 10) throw new Error("Cloud readiness job listing is incomplete.");
|
|
}
|
|
const matches = jobs.filter((job) => job.name === sourceVerificationJob);
|
|
if (matches.length > 1) throw new Error("Cloud source verification job is ambiguous.");
|
|
await onJobs?.(jobs.filter(job => job.head_sha === sha && job.run_id === run.id && job.run_attempt === run.run_attempt));
|
|
if (failFast) {
|
|
const failure = jobs.find((job) => !nonGating(job) && job.status === "completed" &&
|
|
["failure", "cancelled", "timed_out", "action_required", "startup_failure", "stale"].includes(job.conclusion) &&
|
|
job.head_sha === sha && job.run_id === run.id && job.run_attempt === run.run_attempt);
|
|
if (failure) {
|
|
const current = await api(`/repos/${repository}/actions/runs/${run.id}`);
|
|
if (!trustedRun(current, sha, workflow.id) || current.run_attempt !== run.run_attempt) return undefined;
|
|
throw new SourceVerificationFailed(`Cloud source verification failed: ${failure.name} (${failure.conclusion}).`, run);
|
|
}
|
|
}
|
|
const job = matches[0];
|
|
if (job?.status === "completed" && job.conclusion === "success" &&
|
|
job.head_sha === sha && job.run_id === run.id && job.run_attempt === run.run_attempt) {
|
|
// Re-read the run after its jobs: a rerun that started during polling must
|
|
// not let the previous attempt through. Changes are retried next poll.
|
|
const current = await api(`/repos/${repository}/actions/runs/${run.id}`);
|
|
if (!trustedRun(current, sha, workflow.id) || current.run_attempt !== run.run_attempt) return undefined;
|
|
return { sha, runId: run.id, attempt: run.run_attempt, jobId: job.id };
|
|
}
|
|
if (job?.status === "completed" || run.status === "completed") {
|
|
const current = await api(`/repos/${repository}/actions/runs/${run.id}`);
|
|
if (!trustedRun(current, sha, workflow.id) || current.run_attempt !== run.run_attempt) return undefined;
|
|
if (job?.conclusion === "skipped" && current.status !== "completed") return undefined;
|
|
if (deferredProducer(jobs, current, sourceAdmissionJob)) throw new SourceVerificationDeferred("Source checks are deferred until release admission.");
|
|
throw new SourceVerificationFailed(`Cloud source verification did not pass for ${sha} (run ${run.id}, attempt ${run.run_attempt}). Rerun Cloud readiness before retrying the release.`, run);
|
|
}
|
|
return undefined;
|
|
}
|
|
|
|
export { createActionsReader };
|
|
|
|
export async function waitForSourceVerification(sha, {
|
|
api, now = Date.now, sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)),
|
|
timeoutMs = 90 * 60_000, intervalMs = 30_000, log = console.log, rerun, retryRunnerLoss,
|
|
} = {}) {
|
|
assertSha(sha);
|
|
const deadline = now() + timeoutMs;
|
|
const requested = new Set();
|
|
log(`Waiting for ${sourceVerificationJob} for ${sha}.`);
|
|
while (now() < deadline) {
|
|
try {
|
|
const result = await readSourceVerification(sha, api);
|
|
if (result) return result;
|
|
} catch (error) {
|
|
if (error instanceof SourceVerificationFailed && retryRunnerLoss) {
|
|
// A reclaimed Spot worker is not a source failure: retry its jobs.
|
|
if (await retryRunnerLoss({ sha, failure: error, api }) !== "retrying") throw error;
|
|
} else {
|
|
if (!(error instanceof SourceVerificationDeferred) || !rerun) throw error;
|
|
await requestDeferredProducer({ file: "cloud-readiness.yml", sha, admissionJob: sourceAdmissionJob, api, rerun, requested });
|
|
}
|
|
}
|
|
const remaining = deadline - now();
|
|
if (remaining > 0) await sleep(Math.min(intervalMs, remaining));
|
|
}
|
|
throw new Error(`Cloud source verification timed out for ${sha}. Rerun Cloud readiness before retrying the release.`);
|
|
}
|
|
|
|
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
|
try {
|
|
if (!process.env.GITHUB_TOKEN) throw new Error("GITHUB_TOKEN with Actions read access is required.");
|
|
// One deadline for both layers: the reader stops retrying when the poll it
|
|
// serves is out of time, instead of extending the wait past its timeout.
|
|
// Long enough for a deferred full run plus two runner-loss retries.
|
|
const timeoutMs = 90 * 60_000;
|
|
const deadline = Date.now() + timeoutMs;
|
|
const api = createActionsReader({ token: process.env.GITHUB_TOKEN, deadlineAt: () => deadline });
|
|
const canonical = process.env.GITHUB_REPOSITORY === repository && process.env.GITHUB_REF === "refs/heads/master";
|
|
const rerun = canonical ? createReleaseRerunner(process.env.GITHUB_TOKEN) : undefined;
|
|
const retryRunnerLoss = canonical ? createRunnerLossRetry({ token: process.env.GITHUB_TOKEN }) : undefined;
|
|
const proof = await waitForSourceVerification(process.argv[2], { api, timeoutMs, rerun, retryRunnerLoss });
|
|
const message = `Source verification passed for ${proof.sha}: https://github.com/${repository}/actions/runs/${proof.runId}/attempts/${proof.attempt} (job ${proof.jobId}).`;
|
|
console.log(message);
|
|
if (process.env.GITHUB_STEP_SUMMARY) await appendFile(process.env.GITHUB_STEP_SUMMARY, `${message}\n`);
|
|
} catch (error) {
|
|
console.error(error.message);
|
|
process.exitCode = 1;
|
|
}
|
|
}
|