mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:48:15 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A Core image is published only after "Cloud source verified v1"
passes for the same commit in `cloud-readiness.yml`
> - That job needs `verify`, which is the whole `release-verify.yml`
call, and that call included the Runner chaos evals
> - The chaos eval job ("Restart, replay, trace, and recovery faults")
took 10.2-11.9 min, with a 7-9 min server finalization step. It is one
of the two slowest jobs on the gate
> - The owner decided on 2026-11-12 that the Runner chaos evals must not
be a publication prerequisite
> - This pull request runs the chaos evals as a separate job in the same
Cloud readiness run. They still run and report, but the source proof
does not wait for them
> - The benefit is an earlier image publication on every master commit.
A chaos failure stays visible as a failed check
## Linked Issues or Issue Description
**What happened?**
The Core image gate waited for the Runner chaos evals. In master runs
43133035381, 43134251278 and 43145252393 the chaos job took 10.2, 11.9
and 11.0 min. The gate opened at +13.5, +13.2 and +13.9 min.
**Proposed change**
Owner decision (2026-11-12): keep the chaos evals running on every
admitted master commit, but remove them from the set of jobs that the
image and Cloud discovery wait for.
**Why**
Every minute on the gate is a minute on every staging release.
## What Changed
- `.github/workflows/release-verify.yml`: new boolean input
`runner_chaos_evals`, default `true`. The `runner_chaos_evals` job runs
only when it is true. Release callers (`release.yml`,
`release-smoke.yml`) do not set it, so they stay gated on the evals.
- `.github/workflows/cloud-readiness.yml`:
- `verify` passes `runner_chaos_evals: false`.
- New job `runner_chaos_evals`, named "Runner chaos evals (not a
publication gate)". It needs `admission`, has the same admission
condition as `verify`, and calls `runner-chaos-evals.yml` with `ref:
github.sha`. It has no `continue-on-error`.
- `source_verified` ("Cloud source verified v1") still has `needs:
[verify]`. A comment names the owner decision and says not to add the
chaos job.
- `scripts/cloud-source-verification.mjs`: new `runnerChaosEvalsJob`
constant. The fail-fast check ignores jobs named `"<that name> / ..."`.
A chaos failure therefore does not raise `SourceVerificationFailed`. So
`watch-release-source.mjs` no longer cancels the readiness run and the
image run when only the chaos evals fail. Every other failed job still
fails fast and cancels as before.
- Tests:
- `scripts/cloud-source-verification.test.mjs`: a chaos failure does not
cancel the image or block the proof; a chaos failure while the proof is
pending keeps waiting; a required job whose name only resembles the
chaos job still fails fast.
- `.github/scripts/tests/cloud-readiness.test.mjs`: pins the job set
(`admission`, `verify`, `runner_chaos_evals`, `source_verified`),
`source_verified` `needs: [verify]` exactly, `verify` passes
`runner_chaos_evals: false`, the chaos job calls the eval workflow, and
release callers keep the default. Putting the chaos job back into
`source_verified`'s `needs` fails this test (checked locally: 2
failures).
Publication gate, before: `Cloud source verified v1` waited for
typecheck, build, general tests (17 jobs), serialized tests (5), Runner
workflow evals, Verify Paperclip Runner (3 lanes), and the Runner chaos
evals. After: the same list without the Runner chaos evals.
How a chaos failure stays visible:
- Its own check fails on the master commit.
- The Cloud readiness run concludes `failure`.
- The chaos eval bundle is still uploaded (`if: always()`).
- Nothing reports the chaos evals as passed. They are just not a
prerequisite.
Consumers:
- `docker.yml` reads the job "Cloud source verified v1" through
`scripts/watch-release-source.mjs`. No change to `docker.yml` is needed.
- Paperclip Cloud discovery (`src/fleet/cloud-readiness.ts`) runs with
`sourceOnly: true` in `cloud-distribution-policy.ts` and
`release-resolver.ts`. In that mode it needs the workflow path
`.github/workflows/cloud-readiness.yml` and one successful job named
"Cloud source verified v1" in the latest attempt. It does not need the
run conclusion. Both are unchanged, so Cloud needs no change.
- The non-`sourceOnly` mode reads the run conclusion and a job "Cloud
deployable v1". That job no longer exists in this workflow.
## Verification
- `node --test scripts/cloud-source-verification.test.mjs
.github/scripts/tests/cloud-readiness.test.mjs
.github/scripts/tests/post-merge-runner-routing.test.mjs`: 395/395 pass.
- Red/green:
- Before the fail-fast change, the two new chaos tests failed: the
watcher cancelled the image.
- With the chaos job added back to `needs`, the workflow test failed.
- `actionlint .github/workflows/cloud-readiness.yml
.github/workflows/release-verify.yml`: clean.
- Not run: a real Cloud readiness run of this change. GitHub shows the
result after merge.
- Expected gate: the slowest remaining gating job sets it. After #21016
that is a general-server shard at about 10.5 min, then chat (8.8-10.5
min) and serialized (9.2-9.5 min). Expected gate open is about +11.5 to
+12 min, down from +13.2 to +13.9.
## Risks
- A chaos-eval regression no longer blocks the staging image. This is
the owner decision. The failure stays visible as a red check and a
failed Cloud readiness run.
- npm canary publication reuses the same source proof, so it is also no
longer gated on the chaos evals. Stable and preview releases
(`release.yml`) still run them as a gate.
- The non-gating rule depends on the job name prefix. A test pins the
name to the constant in `cloud-source-verification.mjs`.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, GitHub CLI). Standard reasoning mode.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
Related: #21016 (server shard rebalance), #21013 (image publish time;
owns `docker.yml`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
137 lines
7.8 KiB
JavaScript
137 lines
7.8 KiB
JavaScript
import { appendFile } from "node:fs/promises";
|
|
import { pathToFileURL } from "node:url";
|
|
import { createActionsReader } from "./github-actions-reader.mjs";
|
|
import { deferredProducer, requestDeferredProducer, sourceAdmissionJob, createReleaseRerunner } from "./cloud-release-admission.mjs";
|
|
|
|
const repository = "paperclipai/paperclip";
|
|
const workflowPath = ".github/workflows/cloud-readiness.yml";
|
|
export const sourceVerificationJob = "Cloud source verified v1";
|
|
// Owner decision (2026-11-12): the Runner chaos evals run in the same Cloud
|
|
// readiness run but are not a publication prerequisite. Their reusable-workflow
|
|
// jobs are named "<this> / <inner job>". Their failure still fails their own
|
|
// check and the run; it must not fail or cancel the source proof or the image.
|
|
export const runnerChaosEvalsJob = "Runner chaos evals (not a publication gate)";
|
|
const nonGating = job => typeof job.name === "string" && job.name.startsWith(`${runnerChaosEvalsJob} / `);
|
|
export class SourceVerificationFailed extends Error {
|
|
constructor(message, run) {
|
|
super(message);
|
|
this.runId = run.id;
|
|
this.attempt = run.run_attempt;
|
|
}
|
|
}
|
|
export class SourceVerificationDeferred extends Error {}
|
|
|
|
function assertSha(sha) {
|
|
if (!/^[a-f0-9]{40}$/.test(sha ?? "")) throw new Error("A full lowercase source SHA is required.");
|
|
}
|
|
|
|
function trustedRun(run, sha, workflowId) {
|
|
return run.workflow_id === workflowId && run.path === workflowPath &&
|
|
run.repository?.full_name === repository && run.head_repository?.full_name === repository &&
|
|
run.head_sha === sha && run.head_branch === "master" && run.event === "push" &&
|
|
Number.isSafeInteger(run.id) && run.id > 0 &&
|
|
Number.isSafeInteger(run.run_attempt) && run.run_attempt > 0;
|
|
}
|
|
|
|
// Consume one versioned job, independent of image/migrator availability. A
|
|
// failed image build must not invalidate source checks that already passed.
|
|
export async function readSourceVerification(sha, api, { failFast = false, onJobs } = {}) {
|
|
assertSha(sha);
|
|
const workflow = await api(`/repos/${repository}/actions/workflows/cloud-readiness.yml`);
|
|
if (workflow.path !== workflowPath || !Number.isSafeInteger(workflow.id) || workflow.id < 1) {
|
|
throw new Error("Cloud readiness workflow identity does not match.");
|
|
}
|
|
const listing = await api(`/repos/${repository}/actions/workflows/${workflow.id}/runs?head_sha=${sha}&event=push&branch=master&per_page=100`);
|
|
if (!Array.isArray(listing.workflow_runs) || !Number.isSafeInteger(listing.total_count) ||
|
|
listing.total_count < 0 || listing.total_count > 100 || listing.workflow_runs.length !== listing.total_count) {
|
|
throw new Error("Cloud readiness run listing is incomplete.");
|
|
}
|
|
const run = listing.workflow_runs.filter((candidate) => trustedRun(candidate, sha, workflow.id))
|
|
.sort((a, b) => b.id - a.id)[0];
|
|
if (!run) return undefined;
|
|
|
|
// Attempt-specific jobs prevent an earlier successful attempt from blessing
|
|
// a later rerun. Keep pagination even though today's matrix fits one page.
|
|
const jobs = [];
|
|
for (let page = 1; page <= 10; page += 1) {
|
|
const batch = await api(`/repos/${repository}/actions/runs/${run.id}/attempts/${run.run_attempt}/jobs?per_page=100&page=${page}`);
|
|
if (!Array.isArray(batch.jobs)) throw new Error("Cloud readiness job listing is malformed.");
|
|
jobs.push(...batch.jobs);
|
|
if (batch.jobs.length < 100) break;
|
|
if (page === 10) throw new Error("Cloud readiness job listing is incomplete.");
|
|
}
|
|
const matches = jobs.filter((job) => job.name === sourceVerificationJob);
|
|
if (matches.length > 1) throw new Error("Cloud source verification job is ambiguous.");
|
|
await onJobs?.(jobs.filter(job => job.head_sha === sha && job.run_id === run.id && job.run_attempt === run.run_attempt));
|
|
if (failFast) {
|
|
const failure = jobs.find((job) => !nonGating(job) && job.status === "completed" &&
|
|
["failure", "cancelled", "timed_out", "action_required", "startup_failure", "stale"].includes(job.conclusion) &&
|
|
job.head_sha === sha && job.run_id === run.id && job.run_attempt === run.run_attempt);
|
|
if (failure) {
|
|
const current = await api(`/repos/${repository}/actions/runs/${run.id}`);
|
|
if (!trustedRun(current, sha, workflow.id) || current.run_attempt !== run.run_attempt) return undefined;
|
|
throw new SourceVerificationFailed(`Cloud source verification failed: ${failure.name} (${failure.conclusion}).`, run);
|
|
}
|
|
}
|
|
const job = matches[0];
|
|
if (job?.status === "completed" && job.conclusion === "success" &&
|
|
job.head_sha === sha && job.run_id === run.id && job.run_attempt === run.run_attempt) {
|
|
// Re-read the run after its jobs: a rerun that started during polling must
|
|
// not let the previous attempt through. Changes are retried next poll.
|
|
const current = await api(`/repos/${repository}/actions/runs/${run.id}`);
|
|
if (!trustedRun(current, sha, workflow.id) || current.run_attempt !== run.run_attempt) return undefined;
|
|
return { sha, runId: run.id, attempt: run.run_attempt, jobId: job.id };
|
|
}
|
|
if (job?.status === "completed" || run.status === "completed") {
|
|
const current = await api(`/repos/${repository}/actions/runs/${run.id}`);
|
|
if (!trustedRun(current, sha, workflow.id) || current.run_attempt !== run.run_attempt) return undefined;
|
|
if (job?.conclusion === "skipped" && current.status !== "completed") return undefined;
|
|
if (deferredProducer(jobs, current, sourceAdmissionJob)) throw new SourceVerificationDeferred("Source checks are deferred until release admission.");
|
|
throw new SourceVerificationFailed(`Cloud source verification did not pass for ${sha} (run ${run.id}, attempt ${run.run_attempt}). Rerun Cloud readiness before retrying the release.`, run);
|
|
}
|
|
return undefined;
|
|
}
|
|
|
|
export { createActionsReader };
|
|
|
|
export async function waitForSourceVerification(sha, {
|
|
api, now = Date.now, sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)),
|
|
timeoutMs = 60 * 60_000, intervalMs = 30_000, log = console.log, rerun,
|
|
} = {}) {
|
|
assertSha(sha);
|
|
const deadline = now() + timeoutMs;
|
|
const requested = new Set();
|
|
log(`Waiting for ${sourceVerificationJob} for ${sha}.`);
|
|
while (now() < deadline) {
|
|
try {
|
|
const result = await readSourceVerification(sha, api);
|
|
if (result) return result;
|
|
} catch (error) {
|
|
if (!(error instanceof SourceVerificationDeferred) || !rerun) throw error;
|
|
await requestDeferredProducer({ file: "cloud-readiness.yml", sha, admissionJob: sourceAdmissionJob, api, rerun, requested });
|
|
}
|
|
const remaining = deadline - now();
|
|
if (remaining > 0) await sleep(Math.min(intervalMs, remaining));
|
|
}
|
|
throw new Error(`Cloud source verification timed out for ${sha}. Rerun Cloud readiness before retrying the release.`);
|
|
}
|
|
|
|
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
|
try {
|
|
if (!process.env.GITHUB_TOKEN) throw new Error("GITHUB_TOKEN with Actions read access is required.");
|
|
// One deadline for both layers: the reader stops retrying when the poll it
|
|
// serves is out of time, instead of extending the wait past its timeout.
|
|
const timeoutMs = 60 * 60_000;
|
|
const deadline = Date.now() + timeoutMs;
|
|
const api = createActionsReader({ token: process.env.GITHUB_TOKEN, deadlineAt: () => deadline });
|
|
const rerun = process.env.GITHUB_REPOSITORY === repository && process.env.GITHUB_REF === "refs/heads/master" ? createReleaseRerunner(process.env.GITHUB_TOKEN) : undefined;
|
|
const proof = await waitForSourceVerification(process.argv[2], { api, timeoutMs, rerun });
|
|
const message = `Source verification passed for ${proof.sha}: https://github.com/${repository}/actions/runs/${proof.runId}/attempts/${proof.attempt} (job ${proof.jobId}).`;
|
|
console.log(message);
|
|
if (process.env.GITHUB_STEP_SUMMARY) await appendFile(process.env.GITHUB_STEP_SUMMARY, `${message}\n`);
|
|
} catch (error) {
|
|
console.error(error.message);
|
|
process.exitCode = 1;
|
|
}
|
|
}
|