Require published migrators before publishing Core images (#19023)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its release pipeline publishes Core images and exact-source database
migrators.
> - A migrator run can pass admission but skip all build and publication
work.
> - The image observer could stop checking after provisional build
admission.
> - The final image gate checked source proof without checking migrator
publication.
> - This pull request requires the exact migrator publisher to finish
before image publication.
> - Consumers can then use the published image with its matching
migration artifact.

## Linked Issues or Issue Description

**What happened?**

The [migrator run for
e968131](https://github.com/paperclipai/paperclip/actions/runs/41868122685)
passed its admission job but skipped build and publish. Its [Docker
run](https://github.com/paperclipai/paperclip/actions/runs/41868122627)
still published an image. The exact-source migrator artifact was absent.

**Expected behavior**

An image consumer restarts a strictly deferred producer. It waits for
the exact migrator publication before it publishes the image. A
successful admission job alone does not prove publication.

**Steps to reproduce**

1. Let a migrator run defer work while the image consumer starts.
2. Make the producer appear admitted before its build later becomes
skipped.
3. Complete the source verification job.
4. Observe that the old image gate can accept source proof without a
successful migrator publisher.

The regression fixtures model this state. The real producer records
establish the missing publication; the exact transient API response seen
by the original observer is not retained.

**Paperclip version or commit**

The observed source was `e968131efa7e63d321f3bcca8d47a62513c2efb4`. This
patch is based on current master.

No duplicate fix was found in open migrator PRs. PR #18406 concerns npm
canary scheduling and does not provide this image publication gate.

## What Changed

- Require the successful canonical migrator `publish` job for the exact
source and latest attempt. Check complete job pages and re-read the run
after the jobs.
- Continue observing deferred producers until publication, even after a
build appeared admitted. Only a physical runner proves build admission.
- Check migrator publication again in the final read-only
`--check-proof` image gate. Failed producers still require inspection
and are not retried automatically.
- Add regression coverage for deferred recovery, skipped and stale
publication, attempt races, and the final CLI gate. Update release
documentation.

## Verification

- All 57 current-head check contexts are successful or expected skips on
`daed1c756a9fc02b8c15128b96af4da86d447e3f`; both required checks (`ci /
verify`, `ci / e2e`) passed. Greptile is 5/5 with no review threads, and
the branch has no merge conflicts. GitHub master rules require CODEOWNER
review before merging.
- The first CI attempt failed serialized server shard 2/9:
`heartbeat-process-recovery.test.ts` timed out after three seconds
waiting for an adapter execution to start (1 failed, 411 passed).
[Failed
job](https://github.com/paperclipai/paperclip/actions/runs/41894941958/job/126802368555).
This PR changes no server or package files. The exact timeout cause is
not established; this failure is retained and is not described as
infrastructure noise.
- The same first attempt also failed general server shard 6/12:
`heartbeat-stale-queue-invalidation.test.ts` waited five seconds for an
adapter execution after native-owner completion; the run remained
running with no error (1 failed, 2,627 passed, 3 skipped). [Failed
job](https://github.com/paperclipai/paperclip/actions/runs/41894941958/job/126802369329).
Its cause is also not established. The first-attempt failures remain
part of this record.
- The single controlled failed-jobs rerun completed successfully on
attempt 2 at the identical commit. Both previously failed heartbeat
shards and the aggregate verifier passed. This demonstrates recovery on
rerun; the underlying first-attempt failure causes are not established.
[Successful
rerun](https://github.com/paperclipai/paperclip/actions/runs/41894941958/attempts/2).
- Node 24: `node --test scripts/cloud-source-verification.test.mjs
scripts/standard-image-contract.test.mjs` passed: 62 tests, 0 failures,
after rebasing on current master.
- An explicitly gated read-only live probe rejected the real skipped
producer for `e968131` and accepted the published producer for
`afbc5a885b52e96df10c1c6e643bea31b51caac8`.
- `git diff --check` and Gitleaks passed before push.
- A broader run that also included
`scripts/__tests__/release-verify-workflow.test.mjs` failed when local
fixture creation and package installation hit `ENOSPC`. Those failures
remain unverified locally; the focused source and image tests passed
separately, and CI Typecheck + Release Registry subsequently passed.
- Full workspace typecheck, Vitest and app builds were not run locally.
Disk capacity does not permit installing the full workspace in this
separate checkout. No application TypeScript or package manifest
changed. The complete CI run subsequently passed, including typecheck,
build, general and serialized tests, release checks and e2e.
- No live Slack posts, workflow dispatches, producer reruns, deployments
or provider mutations.

## Risks

- Image publication now waits for the exact migrator publisher. A failed
publisher blocks the image and reports a publication failure. Existing
source failure cancellation still applies.
- This adds bounded Actions reads to the existing source observer. The
observer keeps its existing deadline. The final publication runner only
reads evidence and cannot restart producers.
- The publisher job name is part of this gate. A workflow rename must
update the gate.
- No database schema, migration content, image contents, credentials or
application behavior change. Revert this PR to restore the prior
publication gate.

## Model Used

- OpenAI `gpt-6.1-sol`, xhigh reasoning effort, through Codex in T3
Code. Source inspection, tool use and code execution. Context window
size is not exposed by the harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — focused tests pass;
broader local ENOSPC failures are recorded above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>

doc/RELEASING.md

@@ -310,6 +310,14 @@ The workflow:
 
 ## Docker Image Tags
 
+Master image publication requires both exact-source verification and a successful
+`publish` job from the canonical migrator workflow at the same source SHA and
+latest attempt. A green admission-only run with skipped build and publish jobs
+is pending work. The source observer retries only producers that were strictly
+deferred, continues watching after build admission, and waits for publication.
+The final publication step rechecks both proofs before it creates image tags.
+Failed or cancelled producers require inspection and are not retried automatically.
+
 [`docker.yml`](../.github/workflows/docker.yml) publishes the standard image:
 
 | Build ref | Tags |

scripts/cloud-release-admission.mjs

@@ -56,14 +56,7 @@ export function deferredProducer(jobs, run, admissionJob) {
     jobs.every(job => job.run_id === run.id && job.run_attempt === run.run_attempt && job.head_sha === run.head_sha);
 }
 
-// Only skipped admission is restartable here. A failed check or build never
-// becomes an automatic retry, and every write revalidates canonical identity.
-export async function requestDeferredProducer({ file, sha, admissionJob, api, rerun, requested = new Set() }) {
-  if (({ "cloud-readiness.yml": sourceAdmissionJob, "cloud-migrator-artifacts.yml": migratorAdmissionJob })[file] !== admissionJob) throw new Error("Unknown deferred producer contract.");
-  const { workflow, run } = await producer(file, sha, api);
-  if (!run) return "pending";
-  const key = `${run.id}/${run.run_attempt}`;
-  if (requested.has(key)) return "pending";
+async function producerJobs(run, api) {
   const jobs = [];
   let total;
   for (let page = 1; page <= 10; page++) {
@@ -73,14 +66,43 @@ export async function requestDeferredProducer({ file, sha, admissionJob, api, re
         batch.jobs.length !== Math.min(100, Math.max(0, batch.total_count - (page - 1) * 100))) throw new Error("Incomplete release producer job list.");
     total = batch.total_count;
     jobs.push(...batch.jobs);
-    if (page * 100 >= total) break;
-    if (page === 10) throw new Error("Incomplete release producer job list.");
+    if (page * 100 >= total) return jobs;
   }
+  throw new Error("Incomplete release producer job list.");
+}
+
+/** Admission is not publication. The final image gate requires the exact
+ * canonical migrator publisher, including a fresh read after its job evidence. */
+export async function readMigratorPublication(sha, api) {
+  const { workflow, run } = await producer("cloud-migrator-artifacts.yml", sha, api);
+  if (!run) return false;
+  const current = await api(`/repos/${repository}/actions/runs/${run.id}`);
+  if (current.id !== run.id || !trusted(current, sha, workflow) || current.run_attempt !== run.run_attempt || current.status !== "completed") return false;
+  if (current.conclusion !== "success") throw new Error("Exact-source migrator publication failed; inspect its workflow before retrying.");
+  const jobs = await producerJobs(current, api);
+  if (!jobs.every(job => job.run_id === run.id && job.run_attempt === current.run_attempt && job.head_sha === sha)) return false;
+  const publish = jobs.filter(job => job.name === "publish");
+  if (publish.length !== 1 || publish[0].status !== "completed" || publish[0].conclusion !== "success") return false;
+  const confirmed = await api(`/repos/${repository}/actions/runs/${run.id}`);
+  return confirmed.id === run.id && trusted(confirmed, sha, workflow) && confirmed.run_attempt === current.run_attempt &&
+    confirmed.status === "completed" && confirmed.conclusion === "success";
+}
+
+// Only skipped admission is restartable here. A failed check or build never
+// becomes an automatic retry, and every write revalidates canonical identity.
+export async function requestDeferredProducer({ file, sha, admissionJob, api, rerun, requested = new Set() }) {
+  if (({ "cloud-readiness.yml": sourceAdmissionJob, "cloud-migrator-artifacts.yml": migratorAdmissionJob })[file] !== admissionJob) throw new Error("Unknown deferred producer contract.");
+  const { workflow, run } = await producer(file, sha, api);
+  if (!run) return "pending";
+  const key = `${run.id}/${run.run_attempt}`;
+  if (requested.has(key)) return "pending";
+  const jobs = await producerJobs(run, api);
   if (run.status !== "completed") {
     const current = jobs.filter(job => job.run_id === run.id && job.run_attempt === run.run_attempt && job.head_sha === sha);
     // Admission itself succeeds for both outcomes. Only an actual build is
     // evidence that its output admitted work; queued jobs may still be skipped.
-    return current.some(job => job.name !== admissionJob && job.labels?.length > 0 && ["in_progress", "completed"].includes(job.status) && job.conclusion !== "skipped")
+    return current.some(job => job.name !== admissionJob && Number.isSafeInteger(job.runner_id) && job.runner_id > 0 &&
+      ["in_progress", "completed"].includes(job.status) && job.conclusion !== "skipped")
       ? "admitted" : "pending";
   }
   if (!deferredProducer(jobs, run, admissionJob)) return "finished";

scripts/cloud-source-verification.test.mjs

@@ -7,7 +7,7 @@ import { join } from "node:path";
 import { fileURLToPath } from "node:url";
 import { createActionsReader, readSourceVerification, sourceVerificationJob, waitForSourceVerification, SourceVerificationDeferred } from "./cloud-source-verification.mjs";
 
-import { releaseAdmitted, deferredProducer, requestDeferredProducer, sourceAdmissionJob, migratorAdmissionJob, createReleaseRerunner } from "./cloud-release-admission.mjs";
+import { releaseAdmitted, deferredProducer, requestDeferredProducer, readMigratorPublication, sourceAdmissionJob, migratorAdmissionJob, createReleaseRerunner } from "./cloud-release-admission.mjs";
 
 import { cancelImageRun, runnerWaits, watchReleaseSource } from "./watch-release-source.mjs";
 
@@ -63,6 +63,48 @@ describe("admission CLI startup", () => {
     assert.equal(result.output, "");
   });
 });
+
+test("the final image publication CLI rejects a green workflow that skipped its migrator", () => {
+  const root = mkdtempSync(join(tmpdir(), "migrator-publication-cli-"));
+  try {
+    const preload = join(root, "fixture.mjs");
+    const installFixture = () => {
+      const repository = { id: 1170821064, full_name: "paperclipai/paperclip" };
+      const run = (id, file) => ({ id, workflow_id: id, run_attempt: 1, path: `.github/workflows/${file}`, event: "push",
+        head_branch: "master", head_sha: "a".repeat(40), repository, head_repository: repository, status: "completed", conclusion: "success" });
+      const source = run(1, "cloud-readiness.yml"), migrator = run(2, "cloud-migrator-artifacts.yml");
+      globalThis.fetch = async (input, init) => {
+        if (init?.method && init.method !== "GET") throw new Error("Final publication must be read-only");
+        const url = new URL(input), path = url.pathname;
+        if (url.origin !== "https://api.github.com") throw new Error("Unexpected origin");
+        if (path.endsWith("/cloud-readiness.yml")) return Response.json({ id: 1, path: source.path });
+        if (path.endsWith("/cloud-migrator-artifacts.yml")) return Response.json({ id: 2, path: migrator.path });
+        if (path.endsWith("/workflows/1/runs")) return Response.json({ total_count: 1, workflow_runs: [source] });
+        if (path.endsWith("/workflows/2/runs")) return Response.json({ total_count: 1, workflow_runs: [migrator] });
+        if (path.endsWith("/runs/1")) return Response.json(source);
+        if (path.endsWith("/runs/2")) return Response.json(migrator);
+        if (path.endsWith("/jobs")) {
+          const id = path.includes("/runs/1/") ? 1 : 2;
+          return Response.json({ total_count: 1, jobs: [{ id: 3, name: id === 1 ? "Cloud source verified v1" : "publish",
+            run_id: id, run_attempt: 1, head_sha: "a".repeat(40), status: "completed",
+            conclusion: id === 1 || process.env.FIXTURE_PUBLISHED === "1" ? "success" : "skipped" }] });
+        }
+        throw new Error(`Unexpected path ${path}`);
+      };
+    };
+    writeFileSync(preload, "(" + installFixture.toString() + ")();\n");
+    for (const published of [false, true]) {
+      const output = join(root, `output-${published}`);
+      const result = spawnSync(process.execPath, ["--import", preload, fileURLToPath(new URL("./watch-release-source.mjs", import.meta.url)), "--check-proof"], {
+        encoding: "utf8", timeout: 5_000, env: { ...process.env, GITHUB_REPOSITORY: "paperclipai/paperclip", GITHUB_REF: "refs/heads/master",
+          GITHUB_SHA: sha, GITHUB_TOKEN: "fixture", GITHUB_OUTPUT: output, FIXTURE_PUBLISHED: published ? "1" : "0" },
+      });
+      assert.equal(result.status, published ? 0 : 1, result.stderr || result.error?.message);
+      if (published) assert.equal(readFileSync(output, "utf8"), "verified=true\n");
+      else assert.match(result.stderr, /Exact-source migrator has not published/);
+    }
+  } finally { rmSync(root, { recursive: true, force: true }); }
+});
 const workflow = { id: 123, path: ".github/workflows/cloud-readiness.yml" };
 const baseRun = {
   id: 456, workflow_id: workflow.id, path: workflow.path, run_attempt: 2,
@@ -583,6 +625,8 @@ describe("release candidate admission", () => {
     f.jobs[producer.id] = [job(producer, migratorAdmissionJob, "success"), build];
     assert.equal(await requestDeferredProducer(f), "pending");
     build.status = "in_progress";
+    assert.equal(await requestDeferredProducer(f), "pending", "labels alone do not prove a worker started");
+    build.runner_id = 123;
     assert.equal(await requestDeferredProducer(f), "admitted");
     build.run_attempt = 2;
     assert.equal(await requestDeferredProducer(f), "pending", "a stale build cannot admit work");
@@ -615,7 +659,7 @@ describe("release candidate admission", () => {
     assert.equal(f.writes.length, 1);
   });
 
-  test("the image monitor admits a late migrator even when source proof is ready", async () => {
+  test("the image monitor waits for actual migrator publication after deferred admission", async () => {
     const f = fixture("cloud-migrator-artifacts.yml");
     const source = run("cloud-readiness.yml");
     f.runs["cloud-readiness.yml"] = [source];
@@ -627,15 +671,45 @@ describe("release candidate admission", () => {
     const proof = await watchReleaseSource({ ...f, runId: image.id, attempt: 1, cancel: () => assert.fail("must not cancel"),
       now: () => time, intervalMs: 10, timeoutMs: 30, log: () => {}, summary: async () => {}, sleep: async ms => {
         time += ms;
-        const selected = run(f.file, { run_attempt: 2 });
+        const selected = run(f.file, { run_attempt: 2, ...(time >= 20 ? { status: "completed", conclusion: "success" } : {}) });
         f.runs[f.file] = [selected];
-        f.jobs[selected.id] = [{ ...job(selected, "Build migrator", null), status: "in_progress", labels: ["cloud-release-32"] }];
+        f.jobs[selected.id] = [time >= 20 ? job(selected, "publish", "success") :
+          { ...job(selected, "Build migrator", null), status: "in_progress", labels: ["cloud-release-32"], runner_id: 123 }];
       } });
     assert.equal(proof.sha, sha);
-    assert.equal(time, 10);
+    assert.equal(time, 20);
     assert.equal(f.writes.length, 1);
   });
 
+  test("migrator publication rejects skipped, stale, failed and duplicate publisher evidence", async () => {
+    const f = fixture("cloud-migrator-artifacts.yml");
+    const selected = f.runs[f.file][0];
+    assert.equal(await readMigratorPublication(sha, f.api), false);
+    f.jobs[selected.id] = [job(selected, "publish", "success")];
+    assert.equal(await readMigratorPublication(sha, f.api), true);
+    for (const change of [{ conclusion: "skipped" }, { run_attempt: 2 }, { run_id: 1 }, { head_sha: "b".repeat(40) }, { status: "queued" }]) {
+      f.jobs[selected.id] = [{ ...job(selected, "publish", "success"), ...change }];
+      assert.equal(await readMigratorPublication(sha, f.api), false);
+    }
+    f.jobs[selected.id] = [job(selected, "publish", "success"), job(selected, "publish", "success")];
+    assert.equal(await readMigratorPublication(sha, f.api), false);
+    f.current[selected.id] = { ...selected, conclusion: "failure" };
+    await assert.rejects(readMigratorPublication(sha, f.api), /migrator publication failed/);
+    assert.deepEqual(f.writes, [], "a read never retries a failed producer");
+  });
+
+  test("a publication retry racing job evidence invalidates the old success", async () => {
+    const f = fixture("cloud-migrator-artifacts.yml");
+    const selected = f.runs[f.file][0];
+    f.jobs[selected.id] = [job(selected, "publish", "success")];
+    let reads = 0;
+    const api = async path => {
+      if (path.endsWith(`/runs/${selected.id}`) && ++reads === 2) return { ...selected, run_attempt: 2 };
+      return f.api(path);
+    };
+    assert.equal(await readMigratorPublication(sha, api), false);
+  });
+
   test("the retry transport is a bounded POST to a canonical run", async () => {
     const calls = [];
     const rerun = createReleaseRerunner("fixture", async (url, options) => { calls.push({ url, options }); return { ok: true }; });

scripts/watch-release-source.mjs

@@ -1,7 +1,7 @@
 import { appendFile } from "node:fs/promises";
 import { pathToFileURL } from "node:url";
 import { createActionsReader, readSourceVerification, SourceVerificationFailed, SourceVerificationDeferred } from "./cloud-source-verification.mjs";
-import { requestDeferredProducer, sourceAdmissionJob, migratorAdmissionJob, createReleaseRerunner } from "./cloud-release-admission.mjs";
+import { requestDeferredProducer, readMigratorPublication, sourceAdmissionJob, migratorAdmissionJob, createReleaseRerunner } from "./cloud-release-admission.mjs";
 
 const repository = "paperclipai/paperclip";
 
@@ -43,7 +43,6 @@ export async function watchReleaseSource({ sha, runId, attempt, api, cancel, rer
   const deadline = now() + timeoutMs;
   const warned = new Set();
   const requested = new Set();
-  let migratorAdmitted = !rerun;
   const reportWaits = async jobs => {
     for (const wait of runnerWaits(jobs, now())) {
       if (wait.seconds > 60 && !warned.has(wait.id)) {
@@ -54,15 +53,15 @@ export async function watchReleaseSource({ sha, runId, attempt, api, cancel, rer
     }
   };
   while (now() < deadline) {
-    if (!migratorAdmitted) {
-      migratorAdmitted = await requestDeferredProducer({ file: "cloud-migrator-artifacts.yml", sha, admissionJob: migratorAdmissionJob, api, rerun, requested }) !== "pending";
-    }
+    // Keep observing until publication, even after a build appeared admitted.
+    // GitHub may transiently label a job in progress before skipping it.
+    if (rerun) await requestDeferredProducer({ file: "cloud-migrator-artifacts.yml", sha, admissionJob: migratorAdmissionJob, api, rerun, requested });
     const batch = await api(`/repos/${repository}/actions/runs/${runId}/attempts/${attempt}/jobs?per_page=100`);
     if (!Array.isArray(batch.jobs) || batch.total_count > 100) throw new Error("Incomplete image job listing.");
     await reportWaits(batch.jobs);
     try {
       const proof = await readSourceVerification(sha, api, { failFast: true, onJobs: reportWaits });
-      if (proof && migratorAdmitted) {
+      if (proof && (!rerun || await readMigratorPublication(sha, api))) {
         await summary(`Exact source verified: ${sha}; readiness run ${proof.runId}, attempt ${proof.attempt}.\n`);
         return proof;
       }
@@ -82,8 +81,8 @@ export async function watchReleaseSource({ sha, runId, attempt, api, cancel, rer
   // This early failure observer is not the publication deadline. Healthy
   // checks can outlive a Spot worker; a separate exact-source gate still
   // waits for proof before publishing any image.
-  if (requireProof) throw new Error("Exact-source verification did not become ready before the publication deadline.");
-  await summary("Source monitoring window ended; final publication still requires exact-source verification.\n");
+  if (requireProof) throw new Error("Exact-source verification and migrator publication did not become ready before the publication deadline.");
+  await summary("Source monitoring window ended; final publication still requires exact-source verification and migrator publication.\n");
   return undefined;
 }
 
@@ -104,5 +103,6 @@ if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href)
     summary: text => process.env.GITHUB_STEP_SUMMARY ? appendFile(process.env.GITHUB_STEP_SUMMARY, text) : Promise.resolve(),
   });
   if (checkProof && !proof) throw new Error("Exact-source proof changed before publication.");
+  if (checkProof && !await readMigratorPublication(process.env.GITHUB_SHA, api)) throw new Error("Exact-source migrator has not published.");
   if (process.env.GITHUB_OUTPUT) await appendFile(process.env.GITHUB_OUTPUT, `verified=${Boolean(proof)}\n`);
 }