mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(release): keep the Runner chaos evals out of the Core image publication gate (#21088)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A Core image is published only after "Cloud source verified v1"
passes for the same commit in `cloud-readiness.yml`
> - That job needs `verify`, which is the whole `release-verify.yml`
call, and that call included the Runner chaos evals
> - The chaos eval job ("Restart, replay, trace, and recovery faults")
took 10.2-11.9 min, with a 7-9 min server finalization step. It is one
of the two slowest jobs on the gate
> - The owner decided on 2026-11-12 that the Runner chaos evals must not
be a publication prerequisite
> - This pull request runs the chaos evals as a separate job in the same
Cloud readiness run. They still run and report, but the source proof
does not wait for them
> - The benefit is an earlier image publication on every master commit.
A chaos failure stays visible as a failed check
## Linked Issues or Issue Description
**What happened?**
The Core image gate waited for the Runner chaos evals. In master runs
43133035381, 43134251278 and 43145252393 the chaos job took 10.2, 11.9
and 11.0 min. The gate opened at +13.5, +13.2 and +13.9 min.
**Proposed change**
Owner decision (2026-11-12): keep the chaos evals running on every
admitted master commit, but remove them from the set of jobs that the
image and Cloud discovery wait for.
**Why**
Every minute on the gate is a minute on every staging release.
## What Changed
- `.github/workflows/release-verify.yml`: new boolean input
`runner_chaos_evals`, default `true`. The `runner_chaos_evals` job runs
only when it is true. Release callers (`release.yml`,
`release-smoke.yml`) do not set it, so they stay gated on the evals.
- `.github/workflows/cloud-readiness.yml`:
- `verify` passes `runner_chaos_evals: false`.
- New job `runner_chaos_evals`, named "Runner chaos evals (not a
publication gate)". It needs `admission`, has the same admission
condition as `verify`, and calls `runner-chaos-evals.yml` with `ref:
github.sha`. It has no `continue-on-error`.
- `source_verified` ("Cloud source verified v1") still has `needs:
[verify]`. A comment names the owner decision and says not to add the
chaos job.
- `scripts/cloud-source-verification.mjs`: new `runnerChaosEvalsJob`
constant. The fail-fast check ignores jobs named `"<that name> / ..."`.
A chaos failure therefore does not raise `SourceVerificationFailed`. So
`watch-release-source.mjs` no longer cancels the readiness run and the
image run when only the chaos evals fail. Every other failed job still
fails fast and cancels as before.
- Tests:
- `scripts/cloud-source-verification.test.mjs`: a chaos failure does not
cancel the image or block the proof; a chaos failure while the proof is
pending keeps waiting; a required job whose name only resembles the
chaos job still fails fast.
- `.github/scripts/tests/cloud-readiness.test.mjs`: pins the job set
(`admission`, `verify`, `runner_chaos_evals`, `source_verified`),
`source_verified` `needs: [verify]` exactly, `verify` passes
`runner_chaos_evals: false`, the chaos job calls the eval workflow, and
release callers keep the default. Putting the chaos job back into
`source_verified`'s `needs` fails this test (checked locally: 2
failures).
Publication gate, before: `Cloud source verified v1` waited for
typecheck, build, general tests (17 jobs), serialized tests (5), Runner
workflow evals, Verify Paperclip Runner (3 lanes), and the Runner chaos
evals. After: the same list without the Runner chaos evals.
How a chaos failure stays visible:
- Its own check fails on the master commit.
- The Cloud readiness run concludes `failure`.
- The chaos eval bundle is still uploaded (`if: always()`).
- Nothing reports the chaos evals as passed. They are just not a
prerequisite.
Consumers:
- `docker.yml` reads the job "Cloud source verified v1" through
`scripts/watch-release-source.mjs`. No change to `docker.yml` is needed.
- Paperclip Cloud discovery (`src/fleet/cloud-readiness.ts`) runs with
`sourceOnly: true` in `cloud-distribution-policy.ts` and
`release-resolver.ts`. In that mode it needs the workflow path
`.github/workflows/cloud-readiness.yml` and one successful job named
"Cloud source verified v1" in the latest attempt. It does not need the
run conclusion. Both are unchanged, so Cloud needs no change.
- The non-`sourceOnly` mode reads the run conclusion and a job "Cloud
deployable v1". That job no longer exists in this workflow.
## Verification
- `node --test scripts/cloud-source-verification.test.mjs
.github/scripts/tests/cloud-readiness.test.mjs
.github/scripts/tests/post-merge-runner-routing.test.mjs`: 395/395 pass.
- Red/green:
- Before the fail-fast change, the two new chaos tests failed: the
watcher cancelled the image.
- With the chaos job added back to `needs`, the workflow test failed.
- `actionlint .github/workflows/cloud-readiness.yml
.github/workflows/release-verify.yml`: clean.
- Not run: a real Cloud readiness run of this change. GitHub shows the
result after merge.
- Expected gate: the slowest remaining gating job sets it. After #21016
that is a general-server shard at about 10.5 min, then chat (8.8-10.5
min) and serialized (9.2-9.5 min). Expected gate open is about +11.5 to
+12 min, down from +13.2 to +13.9.
## Risks
- A chaos-eval regression no longer blocks the staging image. This is
the owner decision. The failure stays visible as a red check and a
failed Cloud readiness run.
- npm canary publication reuses the same source proof, so it is also no
longer gated on the chaos evals. Stable and preview releases
(`release.yml`) still run them as a gate.
- The non-gating rule depends on the job name prefix. A test pins the
name to the constant in `cloud-source-verification.mjs`.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, GitHub CLI). Standard reasoning mode.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
Related: #21016 (server shard rebalance), #21013 (image publish time;
owns `docker.yml`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
b1330c98e8
commit
ca62ca9e2a
6 files changed
+89
-4
No files matched your search
@@ -18,6 +18,36 @@ test("retirement preserves exact-source verification without issuing legacy depl
|
||||
assert.equal(existsSync(new URL("../../workflows/docker-cloud.yml", import.meta.url)), false);
|
||||
});
|
||||
|
||||
// Owner decision (2026-11-12): the Runner chaos evals run on every admitted
|
||||
// master commit and report, but are not a prerequisite of the source proof
|
||||
// that image publication and Cloud discovery wait for.
|
||||
test("Runner chaos evals still run but stay out of the publication gate", async () => {
|
||||
const { runnerChaosEvalsJob } = await import("../../../scripts/cloud-source-verification.mjs");
|
||||
const jobs = Object.fromEntries([...workflow.split(/^jobs:\n/m)[1].matchAll(/^ ([a-z_]+):\n([\s\S]*?)(?=^ [a-z_]+:\n|(?![\s\S]))/gm)]
|
||||
.map(([, name, body]) => [name, body]));
|
||||
assert.deepEqual(Object.keys(jobs), ["admission", "verify", "runner_chaos_evals", "source_verified"]);
|
||||
|
||||
const chaos = jobs.runner_chaos_evals;
|
||||
assert.match(chaos, new RegExp(`^ name: ${runnerChaosEvalsJob.replace(/[()]/g, "\\$&")}$`, "m"));
|
||||
assert.match(chaos, /^ needs: admission$/m);
|
||||
assert.match(chaos, /needs\.admission\.outputs\.admitted == 'true'/);
|
||||
assert.match(chaos, /uses: \.\/\.github\/workflows\/runner-chaos-evals.yml\s+with:\s+ref: \$\{\{ github.sha \}\}/);
|
||||
assert.doesNotMatch(chaos, /continue-on-error/);
|
||||
|
||||
// The proof needs exactly the source checks; re-adding chaos must fail here.
|
||||
assert.deepEqual(jobs.source_verified.match(/^ needs: (.+)$/m)?.[1], "[verify]");
|
||||
assert.match(jobs.verify, /runner_chaos_evals: false/);
|
||||
|
||||
// Release callers keep the evals as a gate through the default.
|
||||
const releaseVerify = readFileSync(new URL("../../workflows/release-verify.yml", import.meta.url), "utf8");
|
||||
assert.match(releaseVerify, /runner_chaos_evals:\n\s+description: [^\n]+\n\s+required: false\n\s+default: true\n\s+type: boolean/);
|
||||
assert.match(releaseVerify, /^ runner_chaos_evals:\n(?: #[^\n]*\n)* if: \$\{\{ !inputs.runner_binaries_only && inputs.runner_chaos_evals \}\}$/m);
|
||||
for (const file of ["release.yml", "release-smoke.yml"]) {
|
||||
const caller = readFileSync(new URL(`../../workflows/${file}`, import.meta.url), "utf8");
|
||||
assert.doesNotMatch(caller, /runner_chaos_evals:\s*false/, file);
|
||||
}
|
||||
});
|
||||
|
||||
test("standard image provenance and independent exact-source migrators remain available", () => {
|
||||
const docker = readFileSync(new URL("../../workflows/docker.yml", import.meta.url), "utf8");
|
||||
assert.match(docker, /target: production/);
|
||||
|
||||
@@ -45,6 +45,24 @@ jobs:
|
||||
uses: ./.github/workflows/release-verify.yml
|
||||
with:
|
||||
ref: ${{ github.sha }}
|
||||
runner_chaos_evals: false
|
||||
|
||||
# Owner decision (2026-11-12): the Runner chaos evals are NOT a publication
|
||||
# prerequisite. They still run on every admitted master commit and still
|
||||
# fail their own check (and this run) when they fail, but source_verified
|
||||
# must not need this job, and scripts/cloud-source-verification.mjs must not
|
||||
# treat its failure as a failed source verification. They took 10-12 min and
|
||||
# set the image gate. Do not add this job to source_verified's needs.
|
||||
# .github/scripts/tests/cloud-readiness.test.mjs fails if it is re-added.
|
||||
runner_chaos_evals:
|
||||
name: Runner chaos evals (not a publication gate)
|
||||
needs: admission
|
||||
if: github.repository == 'paperclipai/paperclip' && github.ref == 'refs/heads/master' && needs.admission.outputs.admitted == 'true'
|
||||
permissions:
|
||||
contents: read
|
||||
uses: ./.github/workflows/runner-chaos-evals.yml
|
||||
with:
|
||||
ref: ${{ github.sha }}
|
||||
|
||||
source_verified:
|
||||
# npm canary publication reuses this exact-source verification proof.
|
||||
|
||||
@@ -12,6 +12,11 @@ on:
|
||||
required: false
|
||||
default: false
|
||||
type: boolean
|
||||
runner_chaos_evals:
|
||||
description: Run the Runner chaos evals as part of this verification. Cloud readiness passes false and runs them as a separate, non-gating job.
|
||||
required: false
|
||||
default: true
|
||||
type: boolean
|
||||
outputs:
|
||||
runner_artifact_prefix:
|
||||
value: ${{ jobs.runner_binaries.outputs.artifact_prefix }}
|
||||
@@ -24,8 +29,10 @@ on:
|
||||
# the caller runs on canonical master and verifies that event's exact SHA.
|
||||
# The organization group also restricts these workflow files to master.
|
||||
jobs:
|
||||
# Release callers keep the default and stay gated on these evals. Cloud
|
||||
# readiness turns them off here and runs them beside its source proof.
|
||||
runner_chaos_evals:
|
||||
if: ${{ !inputs.runner_binaries_only }}
|
||||
if: ${{ !inputs.runner_binaries_only && inputs.runner_chaos_evals }}
|
||||
name: Pre-release Runner chaos evals
|
||||
uses: ./.github/workflows/runner-chaos-evals.yml
|
||||
with:
|
||||
|
||||
@@ -923,9 +923,12 @@ test("release daemon production uses exact-source native hosts and no publisher
|
||||
assert.match(runner.scripts['build:binary'], /cargo build --release --manifest-path runner\/Cargo.toml --locked/);
|
||||
assert.match(runner.scripts['build:binary'], /&& node scripts\/stage-runner-binary.mjs$/);
|
||||
assert.match(producer, /stage-release-runner-binaries.mjs record/);
|
||||
for (const job of ["runner_chaos_evals", "typecheck", "general_tests", "serialized_tests", "runner_workflow_evals", "verify_paperclip_runner", "build"]) {
|
||||
for (const job of ["typecheck", "general_tests", "serialized_tests", "runner_workflow_evals", "verify_paperclip_runner", "build"]) {
|
||||
assert.ok(workflow.includes(`${job}:\n if: \${{ !inputs.runner_binaries_only }}`));
|
||||
}
|
||||
// Binary-only calls still skip the chaos evals; Cloud readiness also turns
|
||||
// them off here and runs them as its own non-gating job (2026-11-12).
|
||||
assert.ok(workflow.includes("runner_chaos_evals:\n if: ${{ !inputs.runner_binaries_only && inputs.runner_chaos_evals }}"));
|
||||
const release = readWorkflow("release.yml");
|
||||
const source = release.split(" release_runner_source:\n")[1].split(/\n [a-z_]+:\n/)[0];
|
||||
const binaries = release.split(" release_runner_binaries:\n")[1].split(/\n [a-z_]+:\n/)[0];
|
||||
|
||||
@@ -6,6 +6,12 @@ import { deferredProducer, requestDeferredProducer, sourceAdmissionJob, createRe
|
||||
const repository = "paperclipai/paperclip";
|
||||
const workflowPath = ".github/workflows/cloud-readiness.yml";
|
||||
export const sourceVerificationJob = "Cloud source verified v1";
|
||||
// Owner decision (2026-11-12): the Runner chaos evals run in the same Cloud
|
||||
// readiness run but are not a publication prerequisite. Their reusable-workflow
|
||||
// jobs are named "<this> / <inner job>". Their failure still fails their own
|
||||
// check and the run; it must not fail or cancel the source proof or the image.
|
||||
export const runnerChaosEvalsJob = "Runner chaos evals (not a publication gate)";
|
||||
const nonGating = job => typeof job.name === "string" && job.name.startsWith(`${runnerChaosEvalsJob} / `);
|
||||
export class SourceVerificationFailed extends Error {
|
||||
constructor(message, run) {
|
||||
super(message);
|
||||
@@ -58,7 +64,7 @@ export async function readSourceVerification(sha, api, { failFast = false, onJob
|
||||
if (matches.length > 1) throw new Error("Cloud source verification job is ambiguous.");
|
||||
await onJobs?.(jobs.filter(job => job.head_sha === sha && job.run_id === run.id && job.run_attempt === run.run_attempt));
|
||||
if (failFast) {
|
||||
const failure = jobs.find((job) => job.status === "completed" &&
|
||||
const failure = jobs.find((job) => !nonGating(job) && job.status === "completed" &&
|
||||
["failure", "cancelled", "timed_out", "action_required", "startup_failure", "stale"].includes(job.conclusion) &&
|
||||
job.head_sha === sha && job.run_id === run.id && job.run_attempt === run.run_attempt);
|
||||
if (failure) {
|
||||
|
||||
@@ -5,7 +5,7 @@ import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { createActionsReader, readSourceVerification, sourceVerificationJob, waitForSourceVerification, SourceVerificationDeferred } from "./cloud-source-verification.mjs";
|
||||
import { createActionsReader, readSourceVerification, runnerChaosEvalsJob, sourceVerificationJob, waitForSourceVerification, SourceVerificationDeferred } from "./cloud-source-verification.mjs";
|
||||
|
||||
import { releaseAdmitted, deferredProducer, requestDeferredProducer, readMigratorPublication, sourceAdmissionJob, migratorAdmissionJob, createReleaseRerunner } from "./cloud-release-admission.mjs";
|
||||
|
||||
@@ -378,6 +378,27 @@ describe("image build source monitor", () => {
|
||||
assert.deepEqual(f.cancellations, ["/repos/paperclipai/paperclip/actions/runs/99/cancel", `/repos/paperclipai/paperclip/actions/runs/${runId}/cancel`]);
|
||||
});
|
||||
|
||||
// Owner decision 2026-11-12: Runner chaos evals run and report, but do not
|
||||
// gate image publication or the exact-source proof.
|
||||
const chaosJob = { ...proof, id: 112, name: `${runnerChaosEvalsJob} / Restart, replay, trace, and recovery faults` };
|
||||
|
||||
test("a failed Runner chaos eval neither cancels image work nor blocks the source proof", async () => {
|
||||
const f = fixture({ jobs: [{ ...chaosJob, conclusion: "failure" }, proof] });
|
||||
assert.equal((await watchReleaseSource(f)).jobId, proof.id);
|
||||
assert.deepEqual(f.cancellations, []);
|
||||
});
|
||||
|
||||
test("a failed Runner chaos eval while the proof is pending keeps waiting instead of failing", async () => {
|
||||
const f = fixture({ jobs: [{ ...chaosJob, conclusion: "failure" }, { ...proof, status: "in_progress", conclusion: null }] });
|
||||
assert.equal(await readSourceVerification(sha, f.api, { failFast: true }), undefined);
|
||||
assert.deepEqual(f.cancellations, []);
|
||||
});
|
||||
|
||||
test("a required check that only resembles the chaos job name still fails fast", async () => {
|
||||
const f = fixture({ jobs: [{ ...proof, name: "Runner chaos evals / Restart", conclusion: "failure" }] });
|
||||
await assert.rejects(watchReleaseSource(f), /Runner chaos evals \/ Restart/);
|
||||
});
|
||||
|
||||
test("a superseded readiness attempt cancels image work with no deployable future", async () => {
|
||||
const f = fixture({ jobs: [], run: { ...sourceRun, status: "completed", conclusion: "cancelled" } });
|
||||
await assert.rejects(watchReleaseSource(f), /did not pass/);
|
||||
|
||||
Reference in new issue
Block a user