mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-06 10:48:12 +02:00
feat(ci): publish immutable cloud migrator artifacts (#13455)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Cloud deploys images and a matching database migrator.
> - New migrator versions must currently become available on npm before
cloud can use them.
> - npm can serve package metadata while the named archive still returns
404.
> - This pull request publishes immutable migrator archives and a
complete dependency lockfile through the existing artifact store.
> - Cloud can install these exact packages without waiting for their new
npm versions.
> - This producer change prepares a separate cloud consumer and
readiness cutover.
## Linked Issues or Issue Description
Refs: #13454
**What happened?**
A recent master run built both packages by 06:22:41 UTC on 2026-09-15.
Both archives became downloadable from npm at 06:31:56 UTC. Fresh
metadata requests did not remove the delay.
**What did you expect to happen?**
Cloud should be able to install the verified migrator as soon as its
package build and artifact upload finish.
**Steps to reproduce**
Compare package build completion, npm publication, version metadata
availability, and tarball download availability for a fresh full commit
SHA.
**Version**
Master commit `08adcc70d5ec45b7ced9619a3dc10c1d1bec397d`.
## What Changed
- Add a master-only workflow that builds the DB and shared archives
without publication credentials.
- Resolve the dependency lockfile from local archives, then pin those
archives to content-addressed URLs.
- Publish the complete bundle to a separate prefix in the existing
S3/CloudFront artifact store. Write the commit manifest last and verify
public downloads.
- Add a dedicated OIDC role policy. Only canonical master can assume it.
Writes require `If-None-Match: *`; the role cannot overwrite or delete
objects.
- Add source, integrity, lockfile, publication, and real npm install
tests. Document the format and staged rollout.
- Attest the validated manifest with GitHub/Sigstore before S3
publication. The signature binds every package and lockfile hash to the
exact master workflow and source commit.
## Verification
- `node --test scripts/cloud-migrator-artifacts.test.mjs`: 7 tests pass,
including real `npm ci` with an empty cache and no new-version metadata
lookup.
- `pnpm test:release-registry`: 136 tests pass.
- `actionlint .github/workflows/cloud-migrator-artifacts.yml` and `git
diff --check`: pass.
- Ran the workflow's filtered install and package build against the
exact master source. Built and validated the dependency lockfile from
those real archives.
- Latest-head application tests passed, including reruns of two failures
in unchanged application tests. The final CI aggregate passed. The
application source is unchanged. Common-source local typecheck and build
passed; the full local suite has the same documented macOS
read-only-directory rename limitation as #13454 (13 failures in two
unchanged suites).
- The dedicated role and additive bucket read permission are configured.
IAM simulation allows only conditional writes in the intended prefix;
overwrite without the condition, other prefixes, and deletion are
denied.
- Published the verified master 08adcc70d5
bundle with the operator session and verified all public downloads.
GitHub OIDC publication is still pending the master workflow run.
- Cloud resolved the real bundle and checked all 278 SQL migrations in
2.9 seconds with zero npm metadata requests or npm processes. The
existing migration runner applied it to a disposable local PostgreSQL
database and succeeded again on repeat.
- The producer now requires an empty-cache smoke install of the actual
package archives and their full dependency graph before upload. That
check and imports of both installed packages passed locally.
## Risks
- This is an additive producer rollout. It does not yet change the cloud
resolver or the deployable marker.
- The dedicated role and bucket read statement must be installed before
the workflow can publish. Existing bucket policy statements and
public-access blocks must be preserved.
- Referenced artifacts must be retained for rollback. No expiry rule
applies to this prefix.
- Existing external dependencies still download from npm, with SHA-512
pins. New DB and shared versions do not require npm metadata.
- The workflow uses GitHub-hosted runners and has no PR trigger. It adds
no AWS compute routing or PR access.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused release and
real-artifact tests; full-suite host limitation documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
058c55bf8f
commit
da77a0c28c
8 files changed
+559
-1
No files matched your search
@@ -122,3 +122,62 @@ node scripts/preview-artifacts.mjs pack /path/to/source /path/to/output FULL_SHA
|
||||
|
||||
This executes source build scripts. Keep output outside the repository and use an
|
||||
environment without publishing or cloud-admin credentials.
|
||||
|
||||
## Direct cloud migrator artifacts
|
||||
|
||||
`cloud-migrator-artifacts.yml` builds the DB and shared preview packages on each
|
||||
canonical `master` push. A manual run also requires `master` and uses its exact
|
||||
commit. This workflow runs on GitHub-hosted runners. It has no PR trigger.
|
||||
|
||||
The build resolves a complete npm lockfile from the two local archives. It then
|
||||
pins their download URLs to immutable, content-addressed objects. The cloud
|
||||
migration runner can use `npm ci` with this lockfile before either new package
|
||||
version is available on npm. Existing external dependencies still come from npm
|
||||
and carry SHA-512 integrity pins. Package lifecycle scripts remain disabled.
|
||||
Before upload, the build job smoke-installs the real archives and their complete
|
||||
external and bundled dependency graph with an empty npm cache. It imports both
|
||||
installed packages. This check uses local archive URLs because public objects
|
||||
do not exist yet; all versions and integrity pins remain unchanged.
|
||||
|
||||
Artifacts use the existing runner-history S3 bucket and CloudFront distribution,
|
||||
under the separate `cloud-migrators/v1/` prefix. The manifest at
|
||||
`https://d1p6rlowie26tp.cloudfront.net/cloud-migrators/v1/<full-sha>/manifest.json`
|
||||
records the full source SHA, exact preview version, and the size, URL, and SHA-512
|
||||
hash of each archive and the lockfile. Blob URLs include the content hash.
|
||||
The publisher validates the complete bundle before any write, writes all blobs
|
||||
before the manifest, and verifies downloads through the public endpoint.
|
||||
A retry reuses a complete existing manifest after verification. The publisher
|
||||
also creates a GitHub/Sigstore build-provenance attestation for the manifest
|
||||
before upload. This independently binds all package and lockfile content hashes
|
||||
to the canonical workflow, master ref, repository identity, and source commit.
|
||||
Cloud must verify this signature and its certificate claims before accepting
|
||||
the executable archives; hashes served by the artifact store alone are not
|
||||
sufficient provenance.
|
||||
|
||||
The build job has no AWS credential. The publish job downloads only the four
|
||||
fixed files, validates them, and uploads them without executing their code.
|
||||
The dedicated `paperclip-cloud-migrator-github` OIDC role trusts only
|
||||
`repo:paperclipai/paperclip:ref:refs/heads/master`. Its policy permits prefix
|
||||
listing and conditional `PutObject` calls in this one prefix. It permits no
|
||||
object deletion or overwrite. PRs, including allowlisted PRs, cannot assume it.
|
||||
|
||||
The deploy policies are checked in under `.github/cloud-migrator-deploy/`:
|
||||
|
||||
- `trust-policy.json`: the role trust policy.
|
||||
- `upload-policy.json`: the role's inline permission policy.
|
||||
- `cloudfront-read-statement.json`: append this statement to the existing bucket
|
||||
policy, preserving its other statements and public-access blocks.
|
||||
|
||||
There is no lifecycle expiry on this prefix. Keep referenced artifacts for
|
||||
rollback; deleting them can prevent a fresh migrator install for an old release.
|
||||
This producer rollout is additive. npm preview publication and the current
|
||||
cloud readiness gate remain active until the cloud consumer supports the new
|
||||
manifest. A later cutover must preserve source verification, image identity,
|
||||
migration compatibility, and the cloud runner's integrity checks.
|
||||
|
||||
Local verification:
|
||||
|
||||
```sh
|
||||
node --test scripts/cloud-migrator-artifacts.test.mjs
|
||||
node scripts/cloud-migrator-artifacts.mjs verify <full-sha>
|
||||
```
|
||||
Reference in new issue
Block a user