## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its release pipeline publishes Core images and exact-source database
migrators.
> - A migrator run can pass admission but skip all build and publication
work.
> - The image observer could stop checking after provisional build
admission.
> - The final image gate checked source proof without checking migrator
publication.
> - This pull request requires the exact migrator publisher to finish
before image publication.
> - Consumers can then use the published image with its matching
migration artifact.
## Linked Issues or Issue Description
**What happened?**
The [migrator run for
e968131](https://github.com/paperclipai/paperclip/actions/runs/41868122685)
passed its admission job but skipped build and publish. Its [Docker
run](https://github.com/paperclipai/paperclip/actions/runs/41868122627)
still published an image. The exact-source migrator artifact was absent.
**Expected behavior**
An image consumer restarts a strictly deferred producer. It waits for
the exact migrator publication before it publishes the image. A
successful admission job alone does not prove publication.
**Steps to reproduce**
1. Let a migrator run defer work while the image consumer starts.
2. Make the producer appear admitted before its build later becomes
skipped.
3. Complete the source verification job.
4. Observe that the old image gate can accept source proof without a
successful migrator publisher.
The regression fixtures model this state. The real producer records
establish the missing publication; the exact transient API response seen
by the original observer is not retained.
**Paperclip version or commit**
The observed source was `e968131efa7e63d321f3bcca8d47a62513c2efb4`. This
patch is based on current master.
No duplicate fix was found in open migrator PRs. PR #18406 concerns npm
canary scheduling and does not provide this image publication gate.
## What Changed
- Require the successful canonical migrator `publish` job for the exact
source and latest attempt. Check complete job pages and re-read the run
after the jobs.
- Continue observing deferred producers until publication, even after a
build appeared admitted. Only a physical runner proves build admission.
- Check migrator publication again in the final read-only
`--check-proof` image gate. Failed producers still require inspection
and are not retried automatically.
- Add regression coverage for deferred recovery, skipped and stale
publication, attempt races, and the final CLI gate. Update release
documentation.
## Verification
- All 57 current-head check contexts are successful or expected skips on
`daed1c756a9fc02b8c15128b96af4da86d447e3f`; both required checks (`ci /
verify`, `ci / e2e`) passed. Greptile is 5/5 with no review threads, and
the branch has no merge conflicts. GitHub master rules require CODEOWNER
review before merging.
- The first CI attempt failed serialized server shard 2/9:
`heartbeat-process-recovery.test.ts` timed out after three seconds
waiting for an adapter execution to start (1 failed, 411 passed).
[Failed
job](https://github.com/paperclipai/paperclip/actions/runs/41894941958/job/126802368555).
This PR changes no server or package files. The exact timeout cause is
not established; this failure is retained and is not described as
infrastructure noise.
- The same first attempt also failed general server shard 6/12:
`heartbeat-stale-queue-invalidation.test.ts` waited five seconds for an
adapter execution after native-owner completion; the run remained
running with no error (1 failed, 2,627 passed, 3 skipped). [Failed
job](https://github.com/paperclipai/paperclip/actions/runs/41894941958/job/126802369329).
Its cause is also not established. The first-attempt failures remain
part of this record.
- The single controlled failed-jobs rerun completed successfully on
attempt 2 at the identical commit. Both previously failed heartbeat
shards and the aggregate verifier passed. This demonstrates recovery on
rerun; the underlying first-attempt failure causes are not established.
[Successful
rerun](https://github.com/paperclipai/paperclip/actions/runs/41894941958/attempts/2).
- Node 24: `node --test scripts/cloud-source-verification.test.mjs
scripts/standard-image-contract.test.mjs` passed: 62 tests, 0 failures,
after rebasing on current master.
- An explicitly gated read-only live probe rejected the real skipped
producer for `e968131` and accepted the published producer for
`afbc5a885b52e96df10c1c6e643bea31b51caac8`.
- `git diff --check` and Gitleaks passed before push.
- A broader run that also included
`scripts/__tests__/release-verify-workflow.test.mjs` failed when local
fixture creation and package installation hit `ENOSPC`. Those failures
remain unverified locally; the focused source and image tests passed
separately, and CI Typecheck + Release Registry subsequently passed.
- Full workspace typecheck, Vitest and app builds were not run locally.
Disk capacity does not permit installing the full workspace in this
separate checkout. No application TypeScript or package manifest
changed. The complete CI run subsequently passed, including typecheck,
build, general and serialized tests, release checks and e2e.
- No live Slack posts, workflow dispatches, producer reruns, deployments
or provider mutations.
## Risks
- Image publication now waits for the exact migrator publisher. A failed
publisher blocks the image and reports a publication failure. Existing
source failure cancellation still applies.
- This adds bounded Actions reads to the existing source observer. The
observer keeps its existing deadline. The final publication runner only
reads evidence and cannot restart producers.
- The publisher job name is part of this gate. A workflow rename must
update the gate.
- No database schema, migration content, image contents, credentials or
application behavior change. Revert this PR to restore the prior
publication gate.
## Model Used
- OpenAI `gpt-6.1-sol`, xhigh reasoning effort, through Codex in T3
Code. Source inspection, tool use and code execution. Context window
size is not exposed by the harness.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — focused tests pass;
broader local ENOSPC failures are recorded above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
Move the unchanged Actions read transport into an independent module so the admission CLI can evaluate consumers without awaiting a circular import of source verification. Preserve the existing source verification export and all verification and placement contracts.
Five child-process cases reproduce the original exit-13 failure before the repair and pass afterward. All 59 source/admission/image checks and 665 workflow contracts pass. Both CLI modes also pass read-only smoke checks against real GitHub metadata. Complete latest-head CI passed on its first attempt, with Greptile 5/5 and no unresolved comments.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Admit full source verification only for exact-source Docker or npm release work, while superseded pending sources complete metadata-only admission. Restart only strictly deferred runs; failed or cancelled checks require an explicit request.
Route canonical master native images to dedicated 32-core Spot profiles and keep source checks on protected verification capacity. Use the four-slot GitHub-hosted publication pool for registry work and signing, with proof waits outside that pool and a fresh source-proof check before tagging. Trim historical checkout blobs and prepared-runner disk cleanup while preserving tag and fork publication behavior.
Validation: release/source and source/artifact regressions, 665 workflow contracts, actionlint, complete latest-head PR CI, and Greptile 5/5 with all comments resolved. Existing CLI and large-file repository timing tests also passed independently; their CI shards received one scoped retry.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Every commit on master is published as an npm canary, and a canary
is the only thing a nightly, a beta and then a stable can be promoted
from
> - A canary only publishes after the release run confirms that Cloud
readiness passed for that exact commit, which it does by polling the
GitHub Actions API for up to 45 minutes
> - That poll used a read that threw on any non-OK response, so one
gateway error ended the wait immediately
> - The release run then failed and the commit published no canary,
although readiness itself had passed
> - A commit with no canary can never be promoted, so this silently
removes commits from the release path
> - This pull request retries the reads that mean "ask again" and leaves
every real failure fast
> - The benefit is that a transient API error costs a few seconds
instead of a release
## Linked Issues or Issue Description
No existing issue. The problem, in the bug report format:
**What happened**
The `Reuse exact-source verification` job failed 12 seconds into a
45-minute wait:
```
Waiting for Cloud source verified v1 for 5054c9ef9b.
GitHub Actions read failed (HTTP 502).
##[error]Process completed with exit code 1.
```
`publish_canary` was skipped, so the commit published no canary. Cloud
readiness for that same commit had already completed successfully.
**Expected behavior**
A gateway error from the GitHub API is a reason to ask again, not a
verdict on the commit. The wait should continue and the canary should
publish.
**Steps to reproduce**
1. Push to master and let Cloud readiness pass for that commit.
2. Have the GitHub Actions API return 502 for any single read the
verification poll makes.
3. The release run fails and no canary is published for that commit.
**Paperclip version or commit**
Present on master. Observed on 2026-09-14 in a release run for
`5054c9ef9`.
## What Changed
- `scripts/cloud-source-verification.mjs` gains `createActionsReader`,
the transport the CLI entry point now uses. It retries transient
transport failures — network errors and HTTP 408, 425, 429, 500, 502,
503 and 504 — with four bounded attempts and a growing backoff.
- Every other non-OK status still throws on the first response. 401, 403
and 404 mean the token or the target is wrong, and waiting those out
would only delay the failure.
- The verification logic itself is untouched. A readiness run that
genuinely failed still stops the release immediately, and an ambiguous
or mismatched run still throws.
## Verification
```
node --test scripts/cloud-source-verification.test.mjs # 16 passed
node scripts/cloud-source-verification.mjs # still exits cleanly with the token message
```
New tests cover a retried gateway error, each transient status with its
growing backoff, a retried network failure and the message that survives
exhaustion, no retry for authorization failures, and the attempt budget.
The pre-existing verification tests are unchanged and still pass.
## Risks
Low risk, and limited to one CI script.
- A genuinely unreachable API now takes four attempts before failing,
which adds a few seconds to a run that was going to fail anyway.
- The retry cannot mask a failed readiness run: that path throws a
different error, from the verification logic rather than the transport.
- No product code and no workflow changes.
## Model Used
- Claude Fable 5 (`claude-fable-5`), 1M context, extended thinking, run
through Claude Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes — no
documented behavior changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Reuse the exact master source-verification result before npm canary publication, removing a duplicate verification matrix while preserving fail-closed release checks.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
