Commit Graph
6 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 2659ff1d22 fix(release): retry Cloud readiness jobs that lost their Spot runner (#21399)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every push to `master` publishes a canary, and the canary's image
reaches staging
> - Each canary waits for `Cloud source verified v1` from the Cloud
readiness run of its exact SHA
> - Cloud readiness runs on AWS Spot workers. When a worker is
reclaimed, the gate failed and the image watcher cancelled the readiness
run
> - Nothing retried that attempt. On a busy master the newest commit's
release failed, and merged work did not reach staging until a person
re-ran the run
> - This pull request re-runs the failed jobs of a readiness attempt
whose every failure is a lost runner. A real check failure still fails
closed
> - The benefit is that a Spot reclaim no longer holds a release

## Linked Issues or Issue Description

**What happened?**
On 2026-11-13, #21266 merged as `8af889f612`. No canary that contained
it published for more than an hour. The release workflow keeps one
running and one pending canary. A newer push replaces the pending one,
so the newest commit always runs next. That part works as designed. The
failures came from the readiness gate:

| Release (SHA) | Readiness run / attempt | Gate that blocked it | Cause
|
| --- | --- | --- | --- |
| `11840f9904` | 43435593630 / 2 | `Reuse exact-source verification` |
Spot workers reclaimed at 13:49:56Z. The run was then cancelled by
`github-actions[bot]` (the image watcher). Dotta re-ran attempt 3 by
hand |
| `8af889f612` | 43434587048 / 2 | Release superseded while pending;
readiness failed in the same reclaim | Spot reclaim |
| `9fbfd93c9b` | 43423674511 / 2 | `Reuse exact-source verification` |
Spot reclaim |
| `6917d0568d` | 43439942063 / 1 | `Reuse exact-source verification` |
Flaky test `decision-models.test.ts` (concurrent budget admission) |
| `18fb6ee571` | 43422042261 / 2 | `Reuse exact-source verification` |
Flaky test `fresh-continuation-fallback.test.ts` (truncate) |
| `5ede64f247` | 43421480302 / 1 | `Reuse exact-source verification` |
Both flaky tests |

Over the last 200 master pushes, release runs ended 14 success, 33
failure, and 150 cancelled (superseded while pending, which is
expected).

**Expected behavior**
A lost runner is not a source failure. The gate retries the lost jobs
and publishes the newest commit that passes.

**Steps to reproduce**
See `scripts/cloud-readiness-runner-loss.test.mjs`, "busy-master
simulation". It replays the 13:49:56Z reclaim through both waiters.

## What Changed

- `scripts/cloud-readiness-runner-loss.mjs` (new): classifies a failed
readiness attempt with the existing PR classifier
(`.github/scripts/runner-loss-rerun.mjs`). It is a runner loss only when
every causing job (`failure`, `timed_out`, `startup_failure`) lost its
runner. Cancelled siblings are fallout. A run that is still going is
cancelled first, then `rerun-failed-jobs` runs on the next poll. The
ceiling is readiness attempt 4. Writes go only to `cancel` or
`rerun-failed-jobs` of a canonical run, after an identity check.
- `scripts/cloud-source-verification.mjs` (canary gate in
`release.yml`): on `SourceVerificationFailed`, ask the retry first. The
source wait goes from 60 to 90 minutes.
- `scripts/watch-release-source.mjs` (master image watcher in
`docker.yml`): on a runner loss, keep the image attempt alive and wait
for the retried proof. Do not cancel it.
- `.github/workflows/release.yml`: `verify_canary` timeout goes from 70
to 100 minutes.
- `.github/workflows/pr-trusted.yml`: run the new test file.
- `doc/RELEASING.md`: new "Busy master and lost runners" section.

## Verification

- `node --test scripts/cloud-readiness-runner-loss.test.mjs
scripts/cloud-source-verification.test.mjs
.github/scripts/tests/*.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs`: all pass.
`release-verify-workflow.test.mjs` passes 51 of 51 but does not exit
locally. This also happens on `origin/master` without this change.
- Red/green: with the two waiter files reverted, 3 of the 6 simulation
tests fail (the canary gate, two reclaims in a row, and the image
watcher). With the change, all 20 pass.
- Dry run against real runs (read-only, writes captured): 43435593630/2,
43434587048/2, and 43423674511/2 give `rerun-failed-jobs`.
43439942063/1, 43421480302/1, and 43422042261/2 (real test failures)
give `failed` with no write.
- `pnpm verify` exits 2 because a workflow file changed. Required CI
supplies broad coverage.

## Risks

- A real failure classified as a runner loss would be retried. The
classifier needs the runner's own shutdown or lost-communication
`##[error]` line in the job log. It is the same classifier PR CI uses,
and unreadable evidence fails closed.
- On a reclaim, the waiter cancels the readiness run to retry sooner.
Jobs that were still healthy run again. This costs runner time, not
correctness.
- Flaky tests still fail a canary. This change does not retry them. They
need their own fixes.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 09:20:30 -06:00
DottaandClaude Opus 5.5 ca62ca9e2a fix(release): keep the Runner chaos evals out of the Core image publication gate (#21088)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A Core image is published only after "Cloud source verified v1"
passes for the same commit in `cloud-readiness.yml`
> - That job needs `verify`, which is the whole `release-verify.yml`
call, and that call included the Runner chaos evals
> - The chaos eval job ("Restart, replay, trace, and recovery faults")
took 10.2-11.9 min, with a 7-9 min server finalization step. It is one
of the two slowest jobs on the gate
> - The owner decided on 2026-11-12 that the Runner chaos evals must not
be a publication prerequisite
> - This pull request runs the chaos evals as a separate job in the same
Cloud readiness run. They still run and report, but the source proof
does not wait for them
> - The benefit is an earlier image publication on every master commit.
A chaos failure stays visible as a failed check

## Linked Issues or Issue Description

**What happened?**
The Core image gate waited for the Runner chaos evals. In master runs
43133035381, 43134251278 and 43145252393 the chaos job took 10.2, 11.9
and 11.0 min. The gate opened at +13.5, +13.2 and +13.9 min.

**Proposed change**
Owner decision (2026-11-12): keep the chaos evals running on every
admitted master commit, but remove them from the set of jobs that the
image and Cloud discovery wait for.

**Why**
Every minute on the gate is a minute on every staging release.

## What Changed

- `.github/workflows/release-verify.yml`: new boolean input
`runner_chaos_evals`, default `true`. The `runner_chaos_evals` job runs
only when it is true. Release callers (`release.yml`,
`release-smoke.yml`) do not set it, so they stay gated on the evals.
- `.github/workflows/cloud-readiness.yml`:
  - `verify` passes `runner_chaos_evals: false`.
- New job `runner_chaos_evals`, named "Runner chaos evals (not a
publication gate)". It needs `admission`, has the same admission
condition as `verify`, and calls `runner-chaos-evals.yml` with `ref:
github.sha`. It has no `continue-on-error`.
- `source_verified` ("Cloud source verified v1") still has `needs:
[verify]`. A comment names the owner decision and says not to add the
chaos job.
- `scripts/cloud-source-verification.mjs`: new `runnerChaosEvalsJob`
constant. The fail-fast check ignores jobs named `"<that name> / ..."`.
A chaos failure therefore does not raise `SourceVerificationFailed`. So
`watch-release-source.mjs` no longer cancels the readiness run and the
image run when only the chaos evals fail. Every other failed job still
fails fast and cancels as before.
- Tests:
- `scripts/cloud-source-verification.test.mjs`: a chaos failure does not
cancel the image or block the proof; a chaos failure while the proof is
pending keeps waiting; a required job whose name only resembles the
chaos job still fails fast.
- `.github/scripts/tests/cloud-readiness.test.mjs`: pins the job set
(`admission`, `verify`, `runner_chaos_evals`, `source_verified`),
`source_verified` `needs: [verify]` exactly, `verify` passes
`runner_chaos_evals: false`, the chaos job calls the eval workflow, and
release callers keep the default. Putting the chaos job back into
`source_verified`'s `needs` fails this test (checked locally: 2
failures).

Publication gate, before: `Cloud source verified v1` waited for
typecheck, build, general tests (17 jobs), serialized tests (5), Runner
workflow evals, Verify Paperclip Runner (3 lanes), and the Runner chaos
evals. After: the same list without the Runner chaos evals.

How a chaos failure stays visible:
- Its own check fails on the master commit.
- The Cloud readiness run concludes `failure`.
- The chaos eval bundle is still uploaded (`if: always()`).
- Nothing reports the chaos evals as passed. They are just not a
prerequisite.

Consumers:
- `docker.yml` reads the job "Cloud source verified v1" through
`scripts/watch-release-source.mjs`. No change to `docker.yml` is needed.
- Paperclip Cloud discovery (`src/fleet/cloud-readiness.ts`) runs with
`sourceOnly: true` in `cloud-distribution-policy.ts` and
`release-resolver.ts`. In that mode it needs the workflow path
`.github/workflows/cloud-readiness.yml` and one successful job named
"Cloud source verified v1" in the latest attempt. It does not need the
run conclusion. Both are unchanged, so Cloud needs no change.
- The non-`sourceOnly` mode reads the run conclusion and a job "Cloud
deployable v1". That job no longer exists in this workflow.

## Verification

- `node --test scripts/cloud-source-verification.test.mjs
.github/scripts/tests/cloud-readiness.test.mjs
.github/scripts/tests/post-merge-runner-routing.test.mjs`: 395/395 pass.
- Red/green:
- Before the fail-fast change, the two new chaos tests failed: the
watcher cancelled the image.
  - With the chaos job added back to `needs`, the workflow test failed.
- `actionlint .github/workflows/cloud-readiness.yml
.github/workflows/release-verify.yml`: clean.
- Not run: a real Cloud readiness run of this change. GitHub shows the
result after merge.
- Expected gate: the slowest remaining gating job sets it. After #21016
that is a general-server shard at about 10.5 min, then chat (8.8-10.5
min) and serialized (9.2-9.5 min). Expected gate open is about +11.5 to
+12 min, down from +13.2 to +13.9.

## Risks

- A chaos-eval regression no longer blocks the staging image. This is
the owner decision. The failure stays visible as a red check and a
failed Cloud readiness run.
- npm canary publication reuses the same source proof, so it is also no
longer gated on the chaos evals. Stable and preview releases
(`release.yml`) still run them as a gate.
- The non-gating rule depends on the job name prefix. A test pins the
name to the constant in `cloud-source-verification.mjs`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, GitHub CLI). Standard reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Related: #21016 (server shard rebalance), #21013 (image publish time;
owns `docker.yml`).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:08:07 -06:00
DottaandPaperclip 44a4ca1c61 fix(ci): remove the admission CLI import cycle (#18889)
Move the unchanged Actions read transport into an independent module so the admission CLI can evaluate consumers without awaiting a circular import of source verification. Preserve the existing source verification export and all verification and placement contracts.

Five child-process cases reproduce the original exit-13 failure before the repair and pass afterward. All 59 source/admission/image checks and 665 workflow contracts pass. Both CLI modes also pass read-only smoke checks against real GitHub metadata. Complete latest-head CI passed on its first attempt, with Greptile 5/5 and no unresolved comments.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-03 18:21:55 -06:00
DottaandPaperclip f774992832 perf(ci): admit release candidates and isolate native image capacity (#18869)
Admit full source verification only for exact-source Docker or npm release work, while superseded pending sources complete metadata-only admission. Restart only strictly deferred runs; failed or cancelled checks require an explicit request.

Route canonical master native images to dedicated 32-core Spot profiles and keep source checks on protected verification capacity. Use the four-slot GitHub-hosted publication pool for registry work and signing, with proof waits outside that pool and a fresh source-proof check before tagging. Trim historical checkout blobs and prepared-runner disk cleanup while preserving tag and fork publication behavior.

Validation: release/source and source/artifact regressions, 665 workflow contracts, actionlint, complete latest-head PR CI, and Greptile 5/5 with all comments resolved. Existing CLI and large-file repository timing tests also passed independently; their CI shards received one scoped retry.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-03 18:01:07 -06:00
Devin Foley 08adcc70d5 fix(ci): retry transient GitHub reads while waiting for source verification (#13429)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every commit on master is published as an npm canary, and a canary
is the only thing a nightly, a beta and then a stable can be promoted
from
> - A canary only publishes after the release run confirms that Cloud
readiness passed for that exact commit, which it does by polling the
GitHub Actions API for up to 45 minutes
> - That poll used a read that threw on any non-OK response, so one
gateway error ended the wait immediately
> - The release run then failed and the commit published no canary,
although readiness itself had passed
> - A commit with no canary can never be promoted, so this silently
removes commits from the release path
> - This pull request retries the reads that mean "ask again" and leaves
every real failure fast
> - The benefit is that a transient API error costs a few seconds
instead of a release

## Linked Issues or Issue Description

No existing issue. The problem, in the bug report format:

**What happened**
The `Reuse exact-source verification` job failed 12 seconds into a
45-minute wait:

```
Waiting for Cloud source verified v1 for 5054c9ef9b.
GitHub Actions read failed (HTTP 502).
##[error]Process completed with exit code 1.
```

`publish_canary` was skipped, so the commit published no canary. Cloud
readiness for that same commit had already completed successfully.

**Expected behavior**
A gateway error from the GitHub API is a reason to ask again, not a
verdict on the commit. The wait should continue and the canary should
publish.

**Steps to reproduce**
1. Push to master and let Cloud readiness pass for that commit.
2. Have the GitHub Actions API return 502 for any single read the
verification poll makes.
3. The release run fails and no canary is published for that commit.

**Paperclip version or commit**
Present on master. Observed on 2026-09-14 in a release run for
`5054c9ef9`.

## What Changed

- `scripts/cloud-source-verification.mjs` gains `createActionsReader`,
the transport the CLI entry point now uses. It retries transient
transport failures — network errors and HTTP 408, 425, 429, 500, 502,
503 and 504 — with four bounded attempts and a growing backoff.
- Every other non-OK status still throws on the first response. 401, 403
and 404 mean the token or the target is wrong, and waiting those out
would only delay the failure.
- The verification logic itself is untouched. A readiness run that
genuinely failed still stops the release immediately, and an ambiguous
or mismatched run still throws.

## Verification

```
node --test scripts/cloud-source-verification.test.mjs   # 16 passed
node scripts/cloud-source-verification.mjs               # still exits cleanly with the token message
```

New tests cover a retried gateway error, each transient status with its
growing backoff, a retried network failure and the message that survives
exhaustion, no retry for authorization failures, and the attempt budget.
The pre-existing verification tests are unchanged and still pass.

## Risks

Low risk, and limited to one CI script.

- A genuinely unreachable API now takes four attempts before failing,
which adds a few seconds to a run that was going to fail anyway.
- The retry cannot mask a failed readiness run: that path throws a
different error, from the verification logic rather than the transport.
- No product code and no workflow changes.

## Model Used

- Claude Fable 5 (`claude-fable-5`), 1M context, extended thinking, run
through Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes — no
documented behavior changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-09-14 23:20:15 -07:00
Devin FoleyandPaperclip 4fde92107e fix(ci): reuse cloud source verification for npm canaries (#13233)
Reuse the exact master source-verification result before npm canary publication, removing a duplicate verification matrix while preserving fail-closed release checks.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-11 09:26:26 -07:00