Files
PaperClipAI/scripts/watch-release-source.mjs
T
DottaandClaude Opus 5.5 2659ff1d22 fix(release): retry Cloud readiness jobs that lost their Spot runner (#21399)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every push to `master` publishes a canary, and the canary's image
reaches staging
> - Each canary waits for `Cloud source verified v1` from the Cloud
readiness run of its exact SHA
> - Cloud readiness runs on AWS Spot workers. When a worker is
reclaimed, the gate failed and the image watcher cancelled the readiness
run
> - Nothing retried that attempt. On a busy master the newest commit's
release failed, and merged work did not reach staging until a person
re-ran the run
> - This pull request re-runs the failed jobs of a readiness attempt
whose every failure is a lost runner. A real check failure still fails
closed
> - The benefit is that a Spot reclaim no longer holds a release

## Linked Issues or Issue Description

**What happened?**
On 2026-11-13, #21266 merged as `8af889f612`. No canary that contained
it published for more than an hour. The release workflow keeps one
running and one pending canary. A newer push replaces the pending one,
so the newest commit always runs next. That part works as designed. The
failures came from the readiness gate:

| Release (SHA) | Readiness run / attempt | Gate that blocked it | Cause
|
| --- | --- | --- | --- |
| `11840f9904` | 43435593630 / 2 | `Reuse exact-source verification` |
Spot workers reclaimed at 13:49:56Z. The run was then cancelled by
`github-actions[bot]` (the image watcher). Dotta re-ran attempt 3 by
hand |
| `8af889f612` | 43434587048 / 2 | Release superseded while pending;
readiness failed in the same reclaim | Spot reclaim |
| `9fbfd93c9b` | 43423674511 / 2 | `Reuse exact-source verification` |
Spot reclaim |
| `6917d0568d` | 43439942063 / 1 | `Reuse exact-source verification` |
Flaky test `decision-models.test.ts` (concurrent budget admission) |
| `18fb6ee571` | 43422042261 / 2 | `Reuse exact-source verification` |
Flaky test `fresh-continuation-fallback.test.ts` (truncate) |
| `5ede64f247` | 43421480302 / 1 | `Reuse exact-source verification` |
Both flaky tests |

Over the last 200 master pushes, release runs ended 14 success, 33
failure, and 150 cancelled (superseded while pending, which is
expected).

**Expected behavior**
A lost runner is not a source failure. The gate retries the lost jobs
and publishes the newest commit that passes.

**Steps to reproduce**
See `scripts/cloud-readiness-runner-loss.test.mjs`, "busy-master
simulation". It replays the 13:49:56Z reclaim through both waiters.

## What Changed

- `scripts/cloud-readiness-runner-loss.mjs` (new): classifies a failed
readiness attempt with the existing PR classifier
(`.github/scripts/runner-loss-rerun.mjs`). It is a runner loss only when
every causing job (`failure`, `timed_out`, `startup_failure`) lost its
runner. Cancelled siblings are fallout. A run that is still going is
cancelled first, then `rerun-failed-jobs` runs on the next poll. The
ceiling is readiness attempt 4. Writes go only to `cancel` or
`rerun-failed-jobs` of a canonical run, after an identity check.
- `scripts/cloud-source-verification.mjs` (canary gate in
`release.yml`): on `SourceVerificationFailed`, ask the retry first. The
source wait goes from 60 to 90 minutes.
- `scripts/watch-release-source.mjs` (master image watcher in
`docker.yml`): on a runner loss, keep the image attempt alive and wait
for the retried proof. Do not cancel it.
- `.github/workflows/release.yml`: `verify_canary` timeout goes from 70
to 100 minutes.
- `.github/workflows/pr-trusted.yml`: run the new test file.
- `doc/RELEASING.md`: new "Busy master and lost runners" section.

## Verification

- `node --test scripts/cloud-readiness-runner-loss.test.mjs
scripts/cloud-source-verification.test.mjs
.github/scripts/tests/*.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs`: all pass.
`release-verify-workflow.test.mjs` passes 51 of 51 but does not exit
locally. This also happens on `origin/master` without this change.
- Red/green: with the two waiter files reverted, 3 of the 6 simulation
tests fail (the canary gate, two reclaims in a row, and the image
watcher). With the change, all 20 pass.
- Dry run against real runs (read-only, writes captured): 43435593630/2,
43434587048/2, and 43423674511/2 give `rerun-failed-jobs`.
43439942063/1, 43421480302/1, and 43422042261/2 (real test failures)
give `failed` with no write.
- `pnpm verify` exits 2 because a workflow file changed. Required CI
supplies broad coverage.

## Risks

- A real failure classified as a runner loss would be retried. The
classifier needs the runner's own shutdown or lost-communication
`##[error]` line in the job log. It is the same classifier PR CI uses,
and unreadable evidence fails closed.
- On a reclaim, the waiter cancels the readiness run to retry sooner.
Jobs that were still healthy run again. This costs runner time, not
correctness.
- Flaky tests still fail a canary. This change does not retry them. They
need their own fixes.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 09:20:30 -06:00

115 lines
7.3 KiB
JavaScript

import { appendFile } from "node:fs/promises";
import { pathToFileURL } from "node:url";
import { createActionsReader, readSourceVerification, SourceVerificationFailed, SourceVerificationDeferred } from "./cloud-source-verification.mjs";
import { requestDeferredProducer, readMigratorPublication, sourceAdmissionJob, migratorAdmissionJob, createReleaseRerunner } from "./cloud-release-admission.mjs";
import { createRunnerLossRetry } from "./cloud-readiness-runner-loss.mjs";
const repository = "paperclipai/paperclip";
export function runnerWaits(jobs, now = Date.now()) {
return jobs.filter(job => ["queued", "in_progress", "completed"].includes(job.status) &&
job.conclusion !== "skipped" && job.labels?.length > 0).map(job => ({
job: job.name, id: job.id,
seconds: Math.max(0, Math.floor(((job.runner_id > 0 && job.started_at ? Date.parse(job.started_at) : now) - Date.parse(job.created_at)) / 1000)),
})).filter(job => Number.isFinite(job.seconds));
}
// The write is limited to this canonical master Docker run. Never cancel
// another workflow, a tag release, or a run selected from untrusted input.
export async function cancelImageRun({ api, cancel, runId, attempt, sha }) {
const run = await api(`/repos/${repository}/actions/runs/${runId}`);
if (run.id !== runId || run.run_attempt !== attempt || run.head_sha !== sha || run.head_branch !== "master" ||
run.repository?.id !== 1170821064 || run.head_repository?.id !== 1170821064 ||
run.path !== ".github/workflows/docker.yml" || !["push", "workflow_dispatch"].includes(run.event)) {
throw new Error("Refusing to cancel a different image workflow identity.");
}
if (run.status !== "completed") await cancel(`/repos/${repository}/actions/runs/${runId}/cancel`);
}
async function cancelFailedVerification({ api, cancel, sha }, failure) {
const run = await api(`/repos/${repository}/actions/runs/${failure.runId}`);
// A concurrent retry owns the new attempt; a late observer leaves it alone.
if (run.run_attempt !== failure.attempt) return;
if (run.id !== failure.runId || run.head_sha !== sha || run.head_branch !== "master" || run.event !== "push" ||
run.repository?.full_name !== repository || run.head_repository?.full_name !== repository ||
run.path !== ".github/workflows/cloud-readiness.yml") throw new Error("Failed verification identity changed.");
if (run.status !== "completed") await cancel(`/repos/${repository}/actions/runs/${failure.runId}/cancel`);
}
export async function watchReleaseSource({ sha, runId, attempt, api, cancel, rerun, retryRunnerLoss, now = Date.now,
sleep = ms => new Promise(resolve => setTimeout(resolve, ms)), log = console.log,
timeoutMs = 40 * 60_000, intervalMs = 30_000, requireProof = false, summary = async () => {} }) {
if (!/^[a-f0-9]{40}$/.test(sha ?? "") || !Number.isSafeInteger(runId) || runId < 1 ||
!Number.isSafeInteger(attempt) || attempt < 1) throw new Error("Invalid release identity.");
const deadline = now() + timeoutMs;
const warned = new Set();
const requested = new Set();
const reportWaits = async jobs => {
for (const wait of runnerWaits(jobs, now())) {
if (wait.seconds > 60 && !warned.has(wait.id)) {
warned.add(wait.id);
log(`::warning title=Release runner waited over 60 seconds::Job ${wait.id} waited ${wait.seconds}s for a runner.`);
await summary(`Release runner wait: ${wait.job} — ${wait.seconds}s (job ${wait.id}).\n`);
}
}
};
while (now() < deadline) {
// Keep observing until publication, even after a build appeared admitted.
// GitHub may transiently label a job in progress before skipping it.
if (rerun) await requestDeferredProducer({ file: "cloud-migrator-artifacts.yml", sha, admissionJob: migratorAdmissionJob, api, rerun, requested });
const batch = await api(`/repos/${repository}/actions/runs/${runId}/attempts/${attempt}/jobs?per_page=100`);
if (!Array.isArray(batch.jobs) || batch.total_count > 100) throw new Error("Incomplete image job listing.");
await reportWaits(batch.jobs);
try {
const proof = await readSourceVerification(sha, api, { failFast: true, onJobs: reportWaits });
if (proof && (!rerun || await readMigratorPublication(sha, api))) {
await summary(`Exact source verified: ${sha}; readiness run ${proof.runId}, attempt ${proof.attempt}.\n`);
return proof;
}
} catch (error) {
if (error instanceof SourceVerificationDeferred && rerun) {
await requestDeferredProducer({ file: "cloud-readiness.yml", sha, admissionJob: sourceAdmissionJob, api, rerun, requested });
} else if (error instanceof SourceVerificationFailed && retryRunnerLoss &&
await retryRunnerLoss({ sha, failure: error, api }) === "retrying") {
// A reclaimed Spot worker is retried, not treated as a failed source:
// keep this image attempt alive for the retried proof.
} else {
if (error instanceof SourceVerificationFailed) {
await cancelFailedVerification({ api, cancel, sha }, error);
await cancelImageRun({ api, cancel, runId, attempt, sha });
}
throw error;
}
}
await sleep(Math.min(intervalMs, Math.max(0, deadline - now())));
}
// This early failure observer is not the publication deadline. Healthy
// checks can outlive a Spot worker; a separate exact-source gate still
// waits for proof before publishing any image.
if (requireProof) throw new Error("Exact-source verification and migrator publication did not become ready before the publication deadline.");
await summary("Source monitoring window ended; final publication still requires exact-source verification and migrator publication.\n");
return undefined;
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
const token = process.env.GITHUB_TOKEN;
if (!token || process.env.GITHUB_REPOSITORY !== repository || process.env.GITHUB_REF !== "refs/heads/master") throw new Error("Canonical master token required.");
const requireProof = process.argv.includes("--require-proof");
const checkProof = process.argv.includes("--check-proof");
const api = createActionsReader({ token });
const proof = checkProof ? await readSourceVerification(process.env.GITHUB_SHA, api, { failFast: true }) : await watchReleaseSource({ requireProof, sha: process.env.GITHUB_SHA, runId: Number(process.env.GITHUB_RUN_ID), attempt: Number(process.env.GITHUB_RUN_ATTEMPT),
api,
rerun: createReleaseRerunner(token),
retryRunnerLoss: createRunnerLossRetry({ token }),
cancel: async path => {
const response = await fetch(`https://api.github.com${path}`, { method: "POST", redirect: "error",
headers: { Authorization: `Bearer ${token}`, Accept: "application/vnd.github+json" }, signal: AbortSignal.timeout(30_000) });
if (!response.ok) throw new Error(`Could not cancel failed-source image work: HTTP ${response.status}.`);
},
summary: text => process.env.GITHUB_STEP_SUMMARY ? appendFile(process.env.GITHUB_STEP_SUMMARY, text) : Promise.resolve(),
});
if (checkProof && !proof) throw new Error("Exact-source proof changed before publication.");
if (checkProof && !await readMigratorPublication(process.env.GITHUB_SHA, api)) throw new Error("Exact-source migrator has not published.");
if (process.env.GITHUB_OUTPUT) await appendFile(process.env.GITHUB_OUTPUT, `verified=${Boolean(proof)}\n`);
}