feat(ci): publish immutable cloud migrator artifacts (#13455)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Cloud deploys images and a matching database migrator.
> - New migrator versions must currently become available on npm before
cloud can use them.
> - npm can serve package metadata while the named archive still returns
404.
> - This pull request publishes immutable migrator archives and a
complete dependency lockfile through the existing artifact store.
> - Cloud can install these exact packages without waiting for their new
npm versions.
> - This producer change prepares a separate cloud consumer and
readiness cutover.

## Linked Issues or Issue Description

Refs: #13454

**What happened?**
A recent master run built both packages by 06:22:41 UTC on 2026-09-15.
Both archives became downloadable from npm at 06:31:56 UTC. Fresh
metadata requests did not remove the delay.

**What did you expect to happen?**
Cloud should be able to install the verified migrator as soon as its
package build and artifact upload finish.

**Steps to reproduce**
Compare package build completion, npm publication, version metadata
availability, and tarball download availability for a fresh full commit
SHA.

**Version**
Master commit `08adcc70d5ec45b7ced9619a3dc10c1d1bec397d`.

## What Changed

- Add a master-only workflow that builds the DB and shared archives
without publication credentials.
- Resolve the dependency lockfile from local archives, then pin those
archives to content-addressed URLs.
- Publish the complete bundle to a separate prefix in the existing
S3/CloudFront artifact store. Write the commit manifest last and verify
public downloads.
- Add a dedicated OIDC role policy. Only canonical master can assume it.
Writes require `If-None-Match: *`; the role cannot overwrite or delete
objects.
- Add source, integrity, lockfile, publication, and real npm install
tests. Document the format and staged rollout.
- Attest the validated manifest with GitHub/Sigstore before S3
publication. The signature binds every package and lockfile hash to the
exact master workflow and source commit.

## Verification

- `node --test scripts/cloud-migrator-artifacts.test.mjs`: 7 tests pass,
including real `npm ci` with an empty cache and no new-version metadata
lookup.
- `pnpm test:release-registry`: 136 tests pass.
- `actionlint .github/workflows/cloud-migrator-artifacts.yml` and `git
diff --check`: pass.
- Ran the workflow's filtered install and package build against the
exact master source. Built and validated the dependency lockfile from
those real archives.
- Latest-head application tests passed, including reruns of two failures
in unchanged application tests. The final CI aggregate passed. The
application source is unchanged. Common-source local typecheck and build
passed; the full local suite has the same documented macOS
read-only-directory rename limitation as #13454 (13 failures in two
unchanged suites).
- The dedicated role and additive bucket read permission are configured.
IAM simulation allows only conditional writes in the intended prefix;
overwrite without the condition, other prefixes, and deletion are
denied.
- Published the verified master 08adcc70d5
bundle with the operator session and verified all public downloads.
GitHub OIDC publication is still pending the master workflow run.
- Cloud resolved the real bundle and checked all 278 SQL migrations in
2.9 seconds with zero npm metadata requests or npm processes. The
existing migration runner applied it to a disposable local PostgreSQL
database and succeeded again on repeat.
- The producer now requires an empty-cache smoke install of the actual
package archives and their full dependency graph before upload. That
check and imports of both installed packages passed locally.

## Risks

- This is an additive producer rollout. It does not yet change the cloud
resolver or the deployable marker.
- The dedicated role and bucket read statement must be installed before
the workflow can publish. Existing bucket policy statements and
public-access blocks must be preserved.
- Referenced artifacts must be retained for rollback. No expiry rule
applies to this prefix.
- Existing external dependencies still download from npm, with SHA-512
pins. New DB and shared versions do not require npm metadata.
- The workflow uses GitHub-hosted runners and has no PR trigger. It adds
no AWS compute routing or PR access.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused release and
real-artifact tests; full-suite host limitation documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin FoleyandPaperclip authored and GitHub committed 2026-09-15 00:46:05 -07:00
1 parent 058c55bf8f
commit da77a0c28c
8 files changed
+559 -1

No files matched your search

+59
View File
@@ -122,3 +122,62 @@ node scripts/preview-artifacts.mjs pack /path/to/source /path/to/output FULL_SHA
This executes source build scripts. Keep output outside the repository and use an
environment without publishing or cloud-admin credentials.
## Direct cloud migrator artifacts
`cloud-migrator-artifacts.yml` builds the DB and shared preview packages on each
canonical `master` push. A manual run also requires `master` and uses its exact
commit. This workflow runs on GitHub-hosted runners. It has no PR trigger.
The build resolves a complete npm lockfile from the two local archives. It then
pins their download URLs to immutable, content-addressed objects. The cloud
migration runner can use `npm ci` with this lockfile before either new package
version is available on npm. Existing external dependencies still come from npm
and carry SHA-512 integrity pins. Package lifecycle scripts remain disabled.
Before upload, the build job smoke-installs the real archives and their complete
external and bundled dependency graph with an empty npm cache. It imports both
installed packages. This check uses local archive URLs because public objects
do not exist yet; all versions and integrity pins remain unchanged.
Artifacts use the existing runner-history S3 bucket and CloudFront distribution,
under the separate `cloud-migrators/v1/` prefix. The manifest at
`https://d1p6rlowie26tp.cloudfront.net/cloud-migrators/v1/<full-sha>/manifest.json`
records the full source SHA, exact preview version, and the size, URL, and SHA-512
hash of each archive and the lockfile. Blob URLs include the content hash.
The publisher validates the complete bundle before any write, writes all blobs
before the manifest, and verifies downloads through the public endpoint.
A retry reuses a complete existing manifest after verification. The publisher
also creates a GitHub/Sigstore build-provenance attestation for the manifest
before upload. This independently binds all package and lockfile content hashes
to the canonical workflow, master ref, repository identity, and source commit.
Cloud must verify this signature and its certificate claims before accepting
the executable archives; hashes served by the artifact store alone are not
sufficient provenance.
The build job has no AWS credential. The publish job downloads only the four
fixed files, validates them, and uploads them without executing their code.
The dedicated `paperclip-cloud-migrator-github` OIDC role trusts only
`repo:paperclipai/paperclip:ref:refs/heads/master`. Its policy permits prefix
listing and conditional `PutObject` calls in this one prefix. It permits no
object deletion or overwrite. PRs, including allowlisted PRs, cannot assume it.
The deploy policies are checked in under `.github/cloud-migrator-deploy/`:
- `trust-policy.json`: the role trust policy.
- `upload-policy.json`: the role's inline permission policy.
- `cloudfront-read-statement.json`: append this statement to the existing bucket
policy, preserving its other statements and public-access blocks.
There is no lifecycle expiry on this prefix. Keep referenced artifacts for
rollback; deleting them can prevent a fresh migrator install for an old release.
This producer rollout is additive. npm preview publication and the current
cloud readiness gate remain active until the cloud consumer supports the new
manifest. A later cutover must preserve source verification, image identity,
migration compatibility, and the cloud runner's integrity checks.
Local verification:
```sh
node --test scripts/cloud-migrator-artifacts.test.mjs
node scripts/cloud-migrator-artifacts.mjs verify <full-sha>
```