Commit Graph
3 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 2659ff1d22 fix(release): retry Cloud readiness jobs that lost their Spot runner (#21399)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every push to `master` publishes a canary, and the canary's image
reaches staging
> - Each canary waits for `Cloud source verified v1` from the Cloud
readiness run of its exact SHA
> - Cloud readiness runs on AWS Spot workers. When a worker is
reclaimed, the gate failed and the image watcher cancelled the readiness
run
> - Nothing retried that attempt. On a busy master the newest commit's
release failed, and merged work did not reach staging until a person
re-ran the run
> - This pull request re-runs the failed jobs of a readiness attempt
whose every failure is a lost runner. A real check failure still fails
closed
> - The benefit is that a Spot reclaim no longer holds a release

## Linked Issues or Issue Description

**What happened?**
On 2026-11-13, #21266 merged as `8af889f612`. No canary that contained
it published for more than an hour. The release workflow keeps one
running and one pending canary. A newer push replaces the pending one,
so the newest commit always runs next. That part works as designed. The
failures came from the readiness gate:

| Release (SHA) | Readiness run / attempt | Gate that blocked it | Cause
|
| --- | --- | --- | --- |
| `11840f9904` | 43435593630 / 2 | `Reuse exact-source verification` |
Spot workers reclaimed at 13:49:56Z. The run was then cancelled by
`github-actions[bot]` (the image watcher). Dotta re-ran attempt 3 by
hand |
| `8af889f612` | 43434587048 / 2 | Release superseded while pending;
readiness failed in the same reclaim | Spot reclaim |
| `9fbfd93c9b` | 43423674511 / 2 | `Reuse exact-source verification` |
Spot reclaim |
| `6917d0568d` | 43439942063 / 1 | `Reuse exact-source verification` |
Flaky test `decision-models.test.ts` (concurrent budget admission) |
| `18fb6ee571` | 43422042261 / 2 | `Reuse exact-source verification` |
Flaky test `fresh-continuation-fallback.test.ts` (truncate) |
| `5ede64f247` | 43421480302 / 1 | `Reuse exact-source verification` |
Both flaky tests |

Over the last 200 master pushes, release runs ended 14 success, 33
failure, and 150 cancelled (superseded while pending, which is
expected).

**Expected behavior**
A lost runner is not a source failure. The gate retries the lost jobs
and publishes the newest commit that passes.

**Steps to reproduce**
See `scripts/cloud-readiness-runner-loss.test.mjs`, "busy-master
simulation". It replays the 13:49:56Z reclaim through both waiters.

## What Changed

- `scripts/cloud-readiness-runner-loss.mjs` (new): classifies a failed
readiness attempt with the existing PR classifier
(`.github/scripts/runner-loss-rerun.mjs`). It is a runner loss only when
every causing job (`failure`, `timed_out`, `startup_failure`) lost its
runner. Cancelled siblings are fallout. A run that is still going is
cancelled first, then `rerun-failed-jobs` runs on the next poll. The
ceiling is readiness attempt 4. Writes go only to `cancel` or
`rerun-failed-jobs` of a canonical run, after an identity check.
- `scripts/cloud-source-verification.mjs` (canary gate in
`release.yml`): on `SourceVerificationFailed`, ask the retry first. The
source wait goes from 60 to 90 minutes.
- `scripts/watch-release-source.mjs` (master image watcher in
`docker.yml`): on a runner loss, keep the image attempt alive and wait
for the retried proof. Do not cancel it.
- `.github/workflows/release.yml`: `verify_canary` timeout goes from 70
to 100 minutes.
- `.github/workflows/pr-trusted.yml`: run the new test file.
- `doc/RELEASING.md`: new "Busy master and lost runners" section.

## Verification

- `node --test scripts/cloud-readiness-runner-loss.test.mjs
scripts/cloud-source-verification.test.mjs
.github/scripts/tests/*.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs`: all pass.
`release-verify-workflow.test.mjs` passes 51 of 51 but does not exit
locally. This also happens on `origin/master` without this change.
- Red/green: with the two waiter files reverted, 3 of the 6 simulation
tests fail (the canary gate, two reclaims in a row, and the image
watcher). With the change, all 20 pass.
- Dry run against real runs (read-only, writes captured): 43435593630/2,
43434587048/2, and 43423674511/2 give `rerun-failed-jobs`.
43439942063/1, 43421480302/1, and 43422042261/2 (real test failures)
give `failed` with no write.
- `pnpm verify` exits 2 because a workflow file changed. Required CI
supplies broad coverage.

## Risks

- A real failure classified as a runner loss would be retried. The
classifier needs the runner's own shutdown or lost-communication
`##[error]` line in the job log. It is the same classifier PR CI uses,
and unreadable evidence fails closed.
- On a reclaim, the waiter cancels the readiness run to retry sooner.
Jobs that were still healthy run again. This costs runner time, not
correctness.
- Flaky tests still fail a canary. This change does not retry them. They
need their own fixes.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 09:20:30 -06:00
Devin FoleyandPaperclip e65c68726b Require published migrators before publishing Core images (#19023)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its release pipeline publishes Core images and exact-source database
migrators.
> - A migrator run can pass admission but skip all build and publication
work.
> - The image observer could stop checking after provisional build
admission.
> - The final image gate checked source proof without checking migrator
publication.
> - This pull request requires the exact migrator publisher to finish
before image publication.
> - Consumers can then use the published image with its matching
migration artifact.

## Linked Issues or Issue Description

**What happened?**

The [migrator run for
e968131](https://github.com/paperclipai/paperclip/actions/runs/41868122685)
passed its admission job but skipped build and publish. Its [Docker
run](https://github.com/paperclipai/paperclip/actions/runs/41868122627)
still published an image. The exact-source migrator artifact was absent.

**Expected behavior**

An image consumer restarts a strictly deferred producer. It waits for
the exact migrator publication before it publishes the image. A
successful admission job alone does not prove publication.

**Steps to reproduce**

1. Let a migrator run defer work while the image consumer starts.
2. Make the producer appear admitted before its build later becomes
skipped.
3. Complete the source verification job.
4. Observe that the old image gate can accept source proof without a
successful migrator publisher.

The regression fixtures model this state. The real producer records
establish the missing publication; the exact transient API response seen
by the original observer is not retained.

**Paperclip version or commit**

The observed source was `e968131efa7e63d321f3bcca8d47a62513c2efb4`. This
patch is based on current master.

No duplicate fix was found in open migrator PRs. PR #18406 concerns npm
canary scheduling and does not provide this image publication gate.

## What Changed

- Require the successful canonical migrator `publish` job for the exact
source and latest attempt. Check complete job pages and re-read the run
after the jobs.
- Continue observing deferred producers until publication, even after a
build appeared admitted. Only a physical runner proves build admission.
- Check migrator publication again in the final read-only
`--check-proof` image gate. Failed producers still require inspection
and are not retried automatically.
- Add regression coverage for deferred recovery, skipped and stale
publication, attempt races, and the final CLI gate. Update release
documentation.

## Verification

- All 57 current-head check contexts are successful or expected skips on
`daed1c756a9fc02b8c15128b96af4da86d447e3f`; both required checks (`ci /
verify`, `ci / e2e`) passed. Greptile is 5/5 with no review threads, and
the branch has no merge conflicts. GitHub master rules require CODEOWNER
review before merging.
- The first CI attempt failed serialized server shard 2/9:
`heartbeat-process-recovery.test.ts` timed out after three seconds
waiting for an adapter execution to start (1 failed, 411 passed).
[Failed
job](https://github.com/paperclipai/paperclip/actions/runs/41894941958/job/126802368555).
This PR changes no server or package files. The exact timeout cause is
not established; this failure is retained and is not described as
infrastructure noise.
- The same first attempt also failed general server shard 6/12:
`heartbeat-stale-queue-invalidation.test.ts` waited five seconds for an
adapter execution after native-owner completion; the run remained
running with no error (1 failed, 2,627 passed, 3 skipped). [Failed
job](https://github.com/paperclipai/paperclip/actions/runs/41894941958/job/126802369329).
Its cause is also not established. The first-attempt failures remain
part of this record.
- The single controlled failed-jobs rerun completed successfully on
attempt 2 at the identical commit. Both previously failed heartbeat
shards and the aggregate verifier passed. This demonstrates recovery on
rerun; the underlying first-attempt failure causes are not established.
[Successful
rerun](https://github.com/paperclipai/paperclip/actions/runs/41894941958/attempts/2).
- Node 24: `node --test scripts/cloud-source-verification.test.mjs
scripts/standard-image-contract.test.mjs` passed: 62 tests, 0 failures,
after rebasing on current master.
- An explicitly gated read-only live probe rejected the real skipped
producer for `e968131` and accepted the published producer for
`afbc5a885b52e96df10c1c6e643bea31b51caac8`.
- `git diff --check` and Gitleaks passed before push.
- A broader run that also included
`scripts/__tests__/release-verify-workflow.test.mjs` failed when local
fixture creation and package installation hit `ENOSPC`. Those failures
remain unverified locally; the focused source and image tests passed
separately, and CI Typecheck + Release Registry subsequently passed.
- Full workspace typecheck, Vitest and app builds were not run locally.
Disk capacity does not permit installing the full workspace in this
separate checkout. No application TypeScript or package manifest
changed. The complete CI run subsequently passed, including typecheck,
build, general and serialized tests, release checks and e2e.
- No live Slack posts, workflow dispatches, producer reruns, deployments
or provider mutations.

## Risks

- Image publication now waits for the exact migrator publisher. A failed
publisher blocks the image and reports a publication failure. Existing
source failure cancellation still applies.
- This adds bounded Actions reads to the existing source observer. The
observer keeps its existing deadline. The final publication runner only
reads evidence and cannot restart producers.
- The publisher job name is part of this gate. A workflow rename must
update the gate.
- No database schema, migration content, image contents, credentials or
application behavior change. Revert this PR to restore the prior
publication gate.

## Model Used

- OpenAI `gpt-6.1-sol`, xhigh reasoning effort, through Codex in T3
Code. Source inspection, tool use and code execution. Context window
size is not exposed by the harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — focused tests pass;
broader local ENOSPC failures are recorded above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 11:51:42 -08:00
DottaandPaperclip f774992832 perf(ci): admit release candidates and isolate native image capacity (#18869)
Admit full source verification only for exact-source Docker or npm release work, while superseded pending sources complete metadata-only admission. Restart only strictly deferred runs; failed or cancelled checks require an explicit request.

Route canonical master native images to dedicated 32-core Spot profiles and keep source checks on protected verification capacity. Use the four-slot GitHub-hosted publication pool for registry work and signing, with proof waits outside that pool and a fresh source-proof check before tagging. Trim historical checkout blobs and prepared-runner disk cleanup while preserving tag and fork publication behavior.

Validation: release/source and source/artifact regressions, 665 workflow contracts, actionlint, complete latest-head PR CI, and Greptile 5/5 with all comments resolved. Existing CLI and large-file repository timing tests also passed independently; their CI shards received one scoped retry.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-03 18:01:07 -06:00