Reuse signed master images for npm canaries (#18404)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Release workflows publish npm packages and multi-platform images.
> - A master push can finish its image before the npm canary dispatch
starts.
> - The dispatch currently builds that same source again on two runners.
> - This PR verifies and reuses the signed master image by digest.
> - Missing images retain the native build path. Channel promotion still
follows npm.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The Docker workflow dispatched after npm canary publication.

**Current behavior**

Every canary dispatch builds both architectures, even when master has
already published the same source. This consumes runner capacity and
delays the image channel update.

**Proposed behavior**

Resolve the full-SHA master image. Verify its platform index, digest and
signed source identity. Retag that immutable digest. Build normally when
the image does not exist.

**Reason and benefit**

Remove duplicate image builds while keeping missing-image recovery and
the current npm channel check. A recent duplicate used about 25
runner-minutes. The saving depends on whether the master image exists.

## What Changed

- Add a canary-only reuse job before the native matrix.
- Require signed provenance from the canonical Docker workflow on master
at the exact source and signer commit. Read the proof from GHCR.
- Reuse the validated digest or build both platforms when the full-SHA
image is absent. Stop on registry and verification errors.
- Keep the serialized npm dist-tag promotion after either path. Keep
other build lanes independent.
- Add failure tests, register them in CI and update release
documentation.

## Verification

- Passed 603 tests: `node --test '.github/scripts/tests/*.test.mjs'
scripts/standard-image-contract.test.mjs
scripts/reuse-canary-image.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs
scripts/cloud-source-verification.test.mjs`.
- Passed `actionlint -shellcheck='' .github/workflows/docker.yml
.github/workflows/pr-trusted.yml` and `git diff --check`.
- Full actionlint reports the same six existing shellcheck findings as
the unchanged master workflow. No new findings.
- A published master image passed the exact provenance command. This was
a read-only check. No images were retagged.
- Gitleaks and manual diff inspection found no secrets or private
references.
- Attempted `pnpm -r typecheck`, `pnpm test:run` and `pnpm build`. They
cannot start because this isolated checkout has no installed
dependencies (`tsc` and `tsx` are absent). Repository CI passed on
`3c46f0152d4db94a7f393bc79f466aa6c2c745e2` after the documented
browser-test rerun. All checks are complete. The original failure and
prior baseline occurrence remain recorded in the PR comments; its root
cause remains unknown.

## Risks

- The image can appear just before its attestation. Verification then
fails closed. Rerun after the master producer completes.
- Reuse preserves the first build's resolved dependencies and tool
versions. It intentionally does not rebuild the same source to refresh
floating inputs.
- A skipped or unfinished master image still needs a full build. No
speedup is claimed for that path.
- End-to-end reuse and runner savings require observation after merge.
No live publication was performed.

## Model Used

OpenAI `gpt-6.1-sol`, through the Codex harness in T3 Code, with xhigh
reasoning effort and shell/code tools. The context window size is not
exposed by this runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>

.github/scripts/tests/docker-canary-promotion.test.mjs

@@ -11,7 +11,7 @@ const script = job.split("        run: |\n")[1].split("\n").map(line => line.rep
 const sha = "a".repeat(40);
 
 test("canary promotion waits for the standard manifest and keeps its serialized channel", () => {
-  assert.match(job, /needs: \[merge-and-push\]/);
+  assert.match(job, /needs: \[reuse_canary_image, merge-and-push\]/);
   assert.match(job, /group: docker-canary-channel-promotion/);
   assert.match(job, /cancel-in-progress: false/);
 });

.github/workflows/docker.yml

@@ -33,6 +33,35 @@ concurrency:
   cancel-in-progress: false
 
 jobs:
+  reuse_canary_image:
+    if: github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && startsWith(github.ref, 'refs/tags/canary/v')
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+      packages: write
+      attestations: read
+    outputs:
+      reused: ${{ steps.reuse.outputs.reused }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
+      - name: Setup Node.js
+        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020
+        with:
+          node-version: 24
+      - name: Login to GitHub Container Registry
+        uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f
+        with:
+          registry: ghcr.io
+          username: ${{ github.repository_owner }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      - name: Reuse the verified master image when present
+        id: reuse
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: node scripts/reuse-canary-image.mjs
+
   # Multi-arch by native runner, not QEMU.
   #
   # This was one job building linux/amd64,linux/arm64 together on an x86
@@ -56,6 +85,8 @@ jobs:
   # digest; `merge` assembles the manifest list. Both architectures remain
   # available to self-hosted installations and downstream image composers.
   build-and-push:
+    needs: [reuse_canary_image]
+    if: ${{ !cancelled() && (needs.reuse_canary_image.result == 'skipped' || (needs.reuse_canary_image.result == 'success' && needs.reuse_canary_image.outputs.reused == 'false')) }}
     strategy:
       # Independent legs: one architecture failing should still publish
       # nothing, but it must not also hide the other's logs behind a
@@ -277,6 +308,7 @@ jobs:
   # tags. Nothing is publicly tagged until this runs, so a half-published
   # multi-arch image is not a state anything can pull.
   merge-and-push:
+    if: ${{ !cancelled() && needs.build-and-push.result == 'success' }}
     needs: build-and-push
     runs-on: ubuntu-latest
     timeout-minutes: 20
@@ -407,11 +439,8 @@ jobs:
   # promotion is gated on its own completed pushes). Every interleaving
   # converges the Docker channel onto the npm channel.
   promote_canary_channel:
-    if: startsWith(github.ref, 'refs/tags/canary/v')
-    # merge-and-push, not build-and-push: the per-arch legs push untagged
-    # digests, and the production `sha-*` tags this promotion retags only
-    # exist once the manifest merge has named them.
-    needs: [merge-and-push]
+    if: ${{ !cancelled() && startsWith(github.ref, 'refs/tags/canary/v') && (needs.merge-and-push.result == 'success' || (needs.reuse_canary_image.result == 'success' && needs.reuse_canary_image.outputs.reused == 'true')) }}
+    needs: [reuse_canary_image, merge-and-push]
     runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:

.github/workflows/pr-trusted.yml

@@ -202,7 +202,7 @@ jobs:
         run: node "$CI_NODE_TEST" ./scripts/__tests__/e2e-shard.test.mjs
 
       - name: Test release verify workflow wiring
-        run: node "$CI_NODE_TEST" ./scripts/__tests__/release-verify-workflow.test.mjs ./scripts/__tests__/eval-schedule-monitor.test.mjs ./scripts/__tests__/prepare-runner-release-image.test.mjs ./scripts/cloud-source-verification.test.mjs ./scripts/standard-image-contract.test.mjs
+        run: node "$CI_NODE_TEST" ./scripts/__tests__/release-verify-workflow.test.mjs ./scripts/__tests__/eval-schedule-monitor.test.mjs ./scripts/__tests__/prepare-runner-release-image.test.mjs ./scripts/cloud-source-verification.test.mjs ./scripts/standard-image-contract.test.mjs ./scripts/reuse-canary-image.test.mjs
 
       - name: Test standalone package build concurrency
         run: node "$CI_NODE_TEST" ./scripts/__tests__/build-standalone-concurrency.test.mjs

doc/RELEASING.md

@@ -86,7 +86,7 @@ internal traceability tag.
   moves them to the versioned name
 - canaries, nightlies, and betas never create GitHub Releases
 - canaries, nightlies, and betas never require changelog generation
-- Docker `:latest` moves only on stable releases; master builds publish
+- Docker `:latest` moves only on stable releases; canary dispatches promote
   `:canary`, nightly builds `:nightly`, and beta builds `:beta`
 
 ## TL;DR
@@ -310,12 +310,12 @@ The workflow:
 
 ## Docker Image Tags
 
-[`docker.yml`](../.github/workflows/docker.yml) publishes both the self-hosted
-image and the `-cloud` variant with the same lane mapping:
+[`docker.yml`](../.github/workflows/docker.yml) publishes the standard image:
 
 | Build ref | Tags |
 | --- | --- |
-| `master` push | `:canary`, `:sha-<short>` |
+| `master` push | `:sha-<short>`, `:sha-<full>` |
+| `canary/v*` dispatch | `:sha-<short>`, then `:canary` for the current npm canary |
 | `nightly/v*` tag | `:nightly`, `:sha-<short>` |
 | `beta/v*` tag | `:beta`, `:sha-<short>` |
 | `v*` tag (stable) | `:latest`, `:YYYY.MDD.P`, `:YYYY.MDD`, `:sha-<short>` |
@@ -325,6 +325,21 @@ suppresses push-triggered workflow runs for those pushes. The release jobs
 therefore dispatch `docker.yml` explicitly at the new tag ref; the tag
 mapping keys off `github.ref` either way.
 
+Canary dispatches first look up the full-SHA master image. Reuse requires a
+valid multi-platform index with both `linux/amd64` and `linux/arm64`, a matching
+registry digest, and signed provenance from this repository's `docker.yml`
+on `master` at the exact source commit. The image is retagged by immutable
+digest. This preserves the original build inputs and skips both native builds
+and the manifest merge. A missing full-SHA image runs the normal build instead;
+registry errors or invalid provenance fail the run. Rerun after the master
+image's attestation finishes if the image became visible before its proof.
+
+Both paths retain the serialized promotion that resolves the current npm
+`canary` dist-tag before moving Docker `:canary`. Nightly, beta, stable, and
+master builds retain their existing build paths. Downstream image builds can
+continue to consume the signed full-SHA master image independently of npm
+canary publication.
+
 The Docker build matrix and the release preview image builder configure
 BuildKit to check `mirror.gcr.io` for Docker Hub images first. This covers
 both the Dockerfile frontend and base images. BuildKit keeps Docker Hub as

scripts/reuse-canary-image.mjs

@@ -0,0 +1,34 @@
+import assert from "node:assert/strict";
+import { execFileSync } from "node:child_process";
+import { appendFile } from "node:fs/promises";
+import { pathToFileURL } from "node:url";
+import { resolveStandardImageDigest } from "./standard-image-contract.mjs";
+
+export async function reuseCanaryImage(sha, {
+  resolve = source => resolveStandardImageDigest(source, fetch, { missing: "return-null" }),
+  run = (command, args) => execFileSync(command, args, { stdio: "inherit" }),
+} = {}) {
+  assert.match(sha ?? "", /^[a-f0-9]{40}$/);
+  const digest = await resolve(sha);
+  if (digest === null) return false;
+  assert.match(digest, /^sha256:[a-f0-9]{64}$/);
+  const image = `ghcr.io/paperclipai/paperclip@${digest}`;
+  await run("gh", ["attestation", "verify", `oci://${image}`,
+    "--repo", "paperclipai/paperclip", "--bundle-from-oci",
+    "--signer-workflow", "paperclipai/paperclip/.github/workflows/docker.yml",
+    "--source-ref", "refs/heads/master", "--source-digest", sha,
+    "--signer-digest", sha, "--deny-self-hosted-runners"]);
+  await run("docker", ["buildx", "imagetools", "create", "-t",
+    `ghcr.io/paperclipai/paperclip:sha-${sha.slice(0, 7)}`, image]);
+  return true;
+}
+
+if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
+  const reused = await reuseCanaryImage(process.env.GITHUB_SHA);
+  if (process.env.GITHUB_OUTPUT) await appendFile(process.env.GITHUB_OUTPUT, `reused=${reused}\n`);
+  const summary = reused
+    ? "Reused the signed multi-platform master image. Skipped both image builds and the manifest merge."
+    : "No full-SHA master image exists. Building both platforms for this canary.";
+  console.log(summary);
+  if (process.env.GITHUB_STEP_SUMMARY) await appendFile(process.env.GITHUB_STEP_SUMMARY, `${summary}\n`);
+}

scripts/reuse-canary-image.test.mjs

@@ -0,0 +1,75 @@
+import assert from "node:assert/strict";
+import { readFileSync } from "node:fs";
+import test from "node:test";
+import { reuseCanaryImage } from "./reuse-canary-image.mjs";
+
+const sha = "a".repeat(40);
+const digest = `sha256:${"b".repeat(64)}`;
+
+test("reuses only an exact-source, master-signed multi-platform image by digest", async () => {
+  const calls = [];
+  const reused = await reuseCanaryImage(sha, {
+    resolve: async source => { assert.equal(source, sha); return digest; },
+    run: async (...args) => calls.push(args),
+  });
+  assert.equal(reused, true);
+  assert.deepEqual(calls, [
+    ["gh", ["attestation", "verify", `oci://ghcr.io/paperclipai/paperclip@${digest}`,
+      "--repo", "paperclipai/paperclip", "--bundle-from-oci",
+      "--signer-workflow", "paperclipai/paperclip/.github/workflows/docker.yml",
+      "--source-ref", "refs/heads/master", "--source-digest", sha,
+      "--signer-digest", sha, "--deny-self-hosted-runners"]],
+    ["docker", ["buildx", "imagetools", "create", "-t",
+      "ghcr.io/paperclipai/paperclip:sha-aaaaaaa", `ghcr.io/paperclipai/paperclip@${digest}`]],
+  ]);
+});
+
+test("a missing master image enables the fallback without retagging", async () => {
+  assert.equal(await reuseCanaryImage(sha, {
+    resolve: async () => null,
+    run: () => assert.fail("A missing image must not be promoted"),
+  }), false);
+});
+
+test("registry and provenance failures stop promotion rather than bypass verification", async () => {
+  await assert.rejects(reuseCanaryImage(sha, {
+    resolve: async () => { throw new Error("HTTP 403"); },
+    run: () => assert.fail("A registry failure must not be promoted"),
+  }), /HTTP 403/);
+  let commands = 0;
+  await assert.rejects(reuseCanaryImage(sha, {
+    resolve: async () => digest,
+    run: command => {
+      assert.equal(command, "gh");
+      commands++;
+      throw new Error("Untrusted source or signature");
+    },
+  }), /Untrusted source/);
+  assert.equal(commands, 1);
+});
+
+test("invalid source refs and digests cannot reach external commands", async () => {
+  for (const invalid of ["master", "a".repeat(7), "a".repeat(40) + "\n"]) {
+    await assert.rejects(reuseCanaryImage(invalid, {
+      resolve: () => assert.fail("Invalid source accepted"),
+    }));
+  }
+  await assert.rejects(reuseCanaryImage(sha, {
+    resolve: async () => "mutable:tag",
+    run: () => assert.fail("Invalid digest accepted"),
+  }));
+});
+
+test("canary reuse gates both native builds and still runs convergent promotion", () => {
+  const workflow = readFileSync(new URL("../.github/workflows/docker.yml", import.meta.url), "utf8");
+  const reuse = workflow.split("  reuse_canary_image:\n")[1].split("  build-and-push:\n")[0];
+  assert.match(reuse, /github.repository_id == '1170821064'.*startsWith\(github.ref, 'refs\/tags\/canary\/v'\)/);
+  assert.match(reuse, /node scripts\/reuse-canary-image.mjs/);
+  const build = workflow.split("  build-and-push:\n")[1].split("  merge-and-push:\n")[0];
+  assert.match(build, /needs: \[reuse_canary_image\]/);
+  assert.match(build, /!cancelled\(\).*result == 'skipped'.*result == 'success'.*outputs.reused == 'false'/);
+  assert.match(build, /platform: linux\/amd64/);
+  assert.match(build, /platform: linux\/arm64/);
+  const promotion = workflow.split("  promote_canary_channel:\n")[1];
+  assert.match(promotion, /!cancelled\(\).*needs.merge-and-push.result == 'success'.*outputs.reused == 'true'/);
+});

scripts/standard-image-contract.mjs

@@ -21,6 +21,7 @@ export function standardImageDigest(bytes) {
 /** Read registry bytes directly; CLI stdout formatting is not artifact content. */
 export async function resolveStandardImageDigest(sha, fetchImpl = fetch, {
   sleep = ms => new Promise(resolve => setTimeout(resolve, ms)),
+  missing = "error",
 } = {}) {
   assert.match(sha ?? "", /^[a-f0-9]{40}$/);
   const request = async (url, headers, consume, allowNotFound = false) => {
@@ -35,6 +36,7 @@ export async function resolveStandardImageDigest(sha, fetchImpl = fetch, {
       }
       if (!response.ok) {
         await response.body?.cancel();
+        if (allowNotFound && response.status === 404 && missing === "return-null") return null;
         const transient = [408, 425, 429].includes(response.status) || response.status >= 500 || (allowNotFound && response.status === 404);
         assert.ok(transient && attempt < 5, `Registry lookup failed: HTTP ${response.status}`);
         const retryAfter = Number(response.headers.get("retry-after"));
@@ -53,7 +55,7 @@ export async function resolveStandardImageDigest(sha, fetchImpl = fetch, {
   };
   const { token } = await request("https://ghcr.io/token?service=ghcr.io&scope=repository:paperclipai/paperclip:pull", {}, response => response.json());
   assert.ok(typeof token === "string" && token, "Missing public pull token");
-  const { raw, expected } = await request(`https://ghcr.io/v2/paperclipai/paperclip/manifests/sha-${sha}`, {
+  const manifest = await request(`https://ghcr.io/v2/paperclipai/paperclip/manifests/sha-${sha}`, {
     authorization: `Bearer ${token}`, accept: "application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json",
   }, async response => {
     assert.ok(response.body, "Standard image index has no body");
@@ -71,6 +73,8 @@ export async function resolveStandardImageDigest(sha, fetchImpl = fetch, {
     } finally { await reader.cancel(); }
     return { raw: Buffer.concat(chunks), expected: response.headers.get("docker-content-digest") };
   }, true);
+  if (manifest === null) return null;
+  const { raw, expected } = manifest;
   const digest = standardImageDigest(raw);
   assert.equal(digest, expected, "Registry digest does not match the index bytes");
   return digest;

scripts/standard-image-contract.test.mjs

@@ -77,3 +77,17 @@ test("permanent registry errors fail immediately and transient errors exhaust a
     assert.equal(calls, expectedCalls);
   }
 });
+
+test("optional lookup falls back only for a missing manifest", async () => {
+  for (const status of [401, 403, 404, 503]) {
+    let calls = 0;
+    const lookup = resolveStandardImageDigest("a".repeat(40), async url => {
+      if (url.includes("/token?")) return Response.json({ token: "public-fixture" });
+      calls++;
+      return new Response(null, { status });
+    }, { missing: "return-null", sleep: async () => {} });
+    if (status === 404) assert.equal(await lookup, null);
+    else await assert.rejects(lookup, /Registry lookup failed/);
+    assert.equal(calls, status === 503 ? 5 : 1);
+  }
+});