mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 10:40:33 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its release pipeline publishes Core images and exact-source database
migrators.
> - A migrator run can pass admission but skip all build and publication
work.
> - The image observer could stop checking after provisional build
admission.
> - The final image gate checked source proof without checking migrator
publication.
> - This pull request requires the exact migrator publisher to finish
before image publication.
> - Consumers can then use the published image with its matching
migration artifact.
## Linked Issues or Issue Description
**What happened?**
The [migrator run for
e968131](https://github.com/paperclipai/paperclip/actions/runs/41868122685)
passed its admission job but skipped build and publish. Its [Docker
run](https://github.com/paperclipai/paperclip/actions/runs/41868122627)
still published an image. The exact-source migrator artifact was absent.
**Expected behavior**
An image consumer restarts a strictly deferred producer. It waits for
the exact migrator publication before it publishes the image. A
successful admission job alone does not prove publication.
**Steps to reproduce**
1. Let a migrator run defer work while the image consumer starts.
2. Make the producer appear admitted before its build later becomes
skipped.
3. Complete the source verification job.
4. Observe that the old image gate can accept source proof without a
successful migrator publisher.
The regression fixtures model this state. The real producer records
establish the missing publication; the exact transient API response seen
by the original observer is not retained.
**Paperclip version or commit**
The observed source was `e968131efa7e63d321f3bcca8d47a62513c2efb4`. This
patch is based on current master.
No duplicate fix was found in open migrator PRs. PR #18406 concerns npm
canary scheduling and does not provide this image publication gate.
## What Changed
- Require the successful canonical migrator `publish` job for the exact
source and latest attempt. Check complete job pages and re-read the run
after the jobs.
- Continue observing deferred producers until publication, even after a
build appeared admitted. Only a physical runner proves build admission.
- Check migrator publication again in the final read-only
`--check-proof` image gate. Failed producers still require inspection
and are not retried automatically.
- Add regression coverage for deferred recovery, skipped and stale
publication, attempt races, and the final CLI gate. Update release
documentation.
## Verification
- All 57 current-head check contexts are successful or expected skips on
`daed1c756a9fc02b8c15128b96af4da86d447e3f`; both required checks (`ci /
verify`, `ci / e2e`) passed. Greptile is 5/5 with no review threads, and
the branch has no merge conflicts. GitHub master rules require CODEOWNER
review before merging.
- The first CI attempt failed serialized server shard 2/9:
`heartbeat-process-recovery.test.ts` timed out after three seconds
waiting for an adapter execution to start (1 failed, 411 passed).
[Failed
job](https://github.com/paperclipai/paperclip/actions/runs/41894941958/job/126802368555).
This PR changes no server or package files. The exact timeout cause is
not established; this failure is retained and is not described as
infrastructure noise.
- The same first attempt also failed general server shard 6/12:
`heartbeat-stale-queue-invalidation.test.ts` waited five seconds for an
adapter execution after native-owner completion; the run remained
running with no error (1 failed, 2,627 passed, 3 skipped). [Failed
job](https://github.com/paperclipai/paperclip/actions/runs/41894941958/job/126802369329).
Its cause is also not established. The first-attempt failures remain
part of this record.
- The single controlled failed-jobs rerun completed successfully on
attempt 2 at the identical commit. Both previously failed heartbeat
shards and the aggregate verifier passed. This demonstrates recovery on
rerun; the underlying first-attempt failure causes are not established.
[Successful
rerun](https://github.com/paperclipai/paperclip/actions/runs/41894941958/attempts/2).
- Node 24: `node --test scripts/cloud-source-verification.test.mjs
scripts/standard-image-contract.test.mjs` passed: 62 tests, 0 failures,
after rebasing on current master.
- An explicitly gated read-only live probe rejected the real skipped
producer for `e968131` and accepted the published producer for
`afbc5a885b52e96df10c1c6e643bea31b51caac8`.
- `git diff --check` and Gitleaks passed before push.
- A broader run that also included
`scripts/__tests__/release-verify-workflow.test.mjs` failed when local
fixture creation and package installation hit `ENOSPC`. Those failures
remain unverified locally; the focused source and image tests passed
separately, and CI Typecheck + Release Registry subsequently passed.
- Full workspace typecheck, Vitest and app builds were not run locally.
Disk capacity does not permit installing the full workspace in this
separate checkout. No application TypeScript or package manifest
changed. The complete CI run subsequently passed, including typecheck,
build, general and serialized tests, release checks and e2e.
- No live Slack posts, workflow dispatches, producer reruns, deployments
or provider mutations.
## Risks
- Image publication now waits for the exact migrator publisher. A failed
publisher blocks the image and reports a publication failure. Existing
source failure cancellation still applies.
- This adds bounded Actions reads to the existing source observer. The
observer keeps its existing deadline. The final publication runner only
reads evidence and cannot restart producers.
- The publisher job name is part of this gate. A workflow rename must
update the gate.
- No database schema, migration content, image contents, credentials or
application behavior change. Revert this PR to restore the prior
publication gate.
## Model Used
- OpenAI `gpt-6.1-sol`, xhigh reasoning effort, through Codex in T3
Code. Source inspection, tool use and code execution. Context window
size is not exposed by the harness.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — focused tests pass;
broader local ENOSPC failures are recorded above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
146 lines
9.6 KiB
JavaScript
146 lines
9.6 KiB
JavaScript
import { appendFile } from "node:fs/promises";
|
|
import { pathToFileURL } from "node:url";
|
|
import { createActionsReader } from "./github-actions-reader.mjs";
|
|
|
|
const repository = "paperclipai/paperclip";
|
|
const repositoryId = 1170821064;
|
|
const workflows = new Set(["docker.yml", "release.yml", "cloud-readiness.yml", "cloud-migrator-artifacts.yml"]);
|
|
export const sourceAdmissionJob = "Cloud candidate admission v1";
|
|
export const migratorAdmissionJob = "Cloud image candidate admission v1";
|
|
|
|
function assertSha(sha) {
|
|
if (!/^[a-f0-9]{40}$/.test(sha ?? "")) throw new Error("Admission requires a full source SHA.");
|
|
}
|
|
|
|
function trusted(run, sha, workflow) {
|
|
return run.workflow_id === workflow.id && run.path === workflow.path && run.head_sha === sha &&
|
|
run.repository?.id === repositoryId && run.head_repository?.id === repositoryId &&
|
|
run.repository?.full_name === repository && run.head_repository?.full_name === repository &&
|
|
run.head_branch === "master" && run.event === "push" &&
|
|
Number.isSafeInteger(run.id) && run.id > 0 && Number.isSafeInteger(run.run_attempt) && run.run_attempt > 0;
|
|
}
|
|
|
|
async function producer(file, sha, api, status) {
|
|
assertSha(sha);
|
|
if (!workflows.has(file)) throw new Error("Unknown release workflow.");
|
|
const workflow = await api(`/repos/${repository}/actions/workflows/${file}`);
|
|
if (workflow.path !== `.github/workflows/${file}` || !Number.isSafeInteger(workflow.id) || workflow.id < 1) throw new Error("Release workflow identity mismatch.");
|
|
const list = await api(`/repos/${repository}/actions/workflows/${workflow.id}/runs?branch=master&event=push&head_sha=${sha}&per_page=100${status ? `&status=${status}` : ""}`);
|
|
if (!Array.isArray(list.workflow_runs) || !Number.isSafeInteger(list.total_count) || list.total_count < 0 ||
|
|
list.total_count > 100 || list.workflow_runs.length !== list.total_count) throw new Error("Incomplete release workflow listing.");
|
|
const run = list.workflow_runs.filter(run => trusted(run, sha, workflow) && (!status || run.status === status)).sort((a, b) => b.id - a.id)[0];
|
|
return { workflow, run };
|
|
}
|
|
|
|
// A retry is an explicit request for this exact immutable source. Normal push
|
|
// work starts only for a candidate admitted by an image build or npm canary.
|
|
export async function releaseAdmitted(sha, { api, event = "push", attempt = 1, imageOnly = false }) {
|
|
assertSha(sha);
|
|
if (!Number.isSafeInteger(attempt) || attempt < 1) throw new Error("Invalid release attempt.");
|
|
if (!["push", "workflow_dispatch"].includes(event)) return false;
|
|
if (event === "workflow_dispatch" || (Number.isSafeInteger(attempt) && attempt > 1)) return true;
|
|
for (const file of imageOnly ? ["docker.yml"] : ["docker.yml", "release.yml"]) {
|
|
const { workflow, run } = await producer(file, sha, api, "in_progress");
|
|
if (!run) continue;
|
|
const current = await api(`/repos/${repository}/actions/runs/${run.id}`);
|
|
if (current.id === run.id && trusted(current, sha, workflow) && current.run_attempt === run.run_attempt && current.status === "in_progress") return true;
|
|
}
|
|
return false;
|
|
}
|
|
|
|
export function deferredProducer(jobs, run, admissionJob) {
|
|
const admission = jobs.filter(job => job.name === admissionJob);
|
|
const remaining = jobs.filter(job => job.name !== admissionJob);
|
|
return run.status === "completed" && run.conclusion === "success" && admission.length === 1 && admission[0].status === "completed" && admission[0].conclusion === "success" && remaining.length > 0 &&
|
|
remaining.every(job => job.status === "completed" && job.conclusion === "skipped") &&
|
|
jobs.every(job => job.run_id === run.id && job.run_attempt === run.run_attempt && job.head_sha === run.head_sha);
|
|
}
|
|
|
|
async function producerJobs(run, api) {
|
|
const jobs = [];
|
|
let total;
|
|
for (let page = 1; page <= 10; page++) {
|
|
const batch = await api(`/repos/${repository}/actions/runs/${run.id}/attempts/${run.run_attempt}/jobs?per_page=100&page=${page}`);
|
|
if (!Array.isArray(batch.jobs) || !Number.isSafeInteger(batch.total_count) || batch.total_count < 0) throw new Error("Invalid release producer job list.");
|
|
if ((total !== undefined && total !== batch.total_count) ||
|
|
batch.jobs.length !== Math.min(100, Math.max(0, batch.total_count - (page - 1) * 100))) throw new Error("Incomplete release producer job list.");
|
|
total = batch.total_count;
|
|
jobs.push(...batch.jobs);
|
|
if (page * 100 >= total) return jobs;
|
|
}
|
|
throw new Error("Incomplete release producer job list.");
|
|
}
|
|
|
|
/** Admission is not publication. The final image gate requires the exact
|
|
* canonical migrator publisher, including a fresh read after its job evidence. */
|
|
export async function readMigratorPublication(sha, api) {
|
|
const { workflow, run } = await producer("cloud-migrator-artifacts.yml", sha, api);
|
|
if (!run) return false;
|
|
const current = await api(`/repos/${repository}/actions/runs/${run.id}`);
|
|
if (current.id !== run.id || !trusted(current, sha, workflow) || current.run_attempt !== run.run_attempt || current.status !== "completed") return false;
|
|
if (current.conclusion !== "success") throw new Error("Exact-source migrator publication failed; inspect its workflow before retrying.");
|
|
const jobs = await producerJobs(current, api);
|
|
if (!jobs.every(job => job.run_id === run.id && job.run_attempt === current.run_attempt && job.head_sha === sha)) return false;
|
|
const publish = jobs.filter(job => job.name === "publish");
|
|
if (publish.length !== 1 || publish[0].status !== "completed" || publish[0].conclusion !== "success") return false;
|
|
const confirmed = await api(`/repos/${repository}/actions/runs/${run.id}`);
|
|
return confirmed.id === run.id && trusted(confirmed, sha, workflow) && confirmed.run_attempt === current.run_attempt &&
|
|
confirmed.status === "completed" && confirmed.conclusion === "success";
|
|
}
|
|
|
|
// Only skipped admission is restartable here. A failed check or build never
|
|
// becomes an automatic retry, and every write revalidates canonical identity.
|
|
export async function requestDeferredProducer({ file, sha, admissionJob, api, rerun, requested = new Set() }) {
|
|
if (({ "cloud-readiness.yml": sourceAdmissionJob, "cloud-migrator-artifacts.yml": migratorAdmissionJob })[file] !== admissionJob) throw new Error("Unknown deferred producer contract.");
|
|
const { workflow, run } = await producer(file, sha, api);
|
|
if (!run) return "pending";
|
|
const key = `${run.id}/${run.run_attempt}`;
|
|
if (requested.has(key)) return "pending";
|
|
const jobs = await producerJobs(run, api);
|
|
if (run.status !== "completed") {
|
|
const current = jobs.filter(job => job.run_id === run.id && job.run_attempt === run.run_attempt && job.head_sha === sha);
|
|
// Admission itself succeeds for both outcomes. Only an actual build is
|
|
// evidence that its output admitted work; queued jobs may still be skipped.
|
|
return current.some(job => job.name !== admissionJob && Number.isSafeInteger(job.runner_id) && job.runner_id > 0 &&
|
|
["in_progress", "completed"].includes(job.status) && job.conclusion !== "skipped")
|
|
? "admitted" : "pending";
|
|
}
|
|
if (!deferredProducer(jobs, run, admissionJob)) return "finished";
|
|
const current = await api(`/repos/${repository}/actions/runs/${run.id}`);
|
|
if (current.id !== run.id || !trusted(current, sha, workflow) || current.run_attempt !== run.run_attempt || current.status !== "completed" || current.conclusion !== "success") return "pending";
|
|
try {
|
|
await rerun(`/repos/${repository}/actions/runs/${run.id}/rerun`);
|
|
} catch (error) {
|
|
// Docker and npm may select the same source at the same time. A rejected
|
|
// duplicate write is harmless only when a fresh read proves a retry began.
|
|
const latest = await api(`/repos/${repository}/actions/runs/${run.id}`);
|
|
if (latest.id !== run.id || !trusted(latest, sha, workflow)) throw error;
|
|
if (!(latest.run_attempt > run.run_attempt || (latest.run_attempt === run.run_attempt &&
|
|
["queued", "in_progress", "requested", "pending", "waiting"].includes(latest.status)))) {
|
|
// GitHub may reject a duplicate before its read API shows the retry.
|
|
// Leave it pending for the next poll; this is not proof of admission.
|
|
if ([409, 422].includes(error.status) && latest.run_attempt === run.run_attempt && latest.status === "completed" && latest.conclusion === "success") return "pending";
|
|
throw error;
|
|
}
|
|
}
|
|
requested.add(key);
|
|
return "pending";
|
|
}
|
|
|
|
export function createReleaseRerunner(token, fetchImpl = fetch) {
|
|
return async path => {
|
|
if (!/^\/repos\/paperclipai\/paperclip\/actions\/runs\/[1-9]\d*\/rerun$/.test(path)) throw new Error("Invalid release retry path.");
|
|
const response = await fetchImpl(`https://api.github.com${path}`, { method: "POST", redirect: "error",
|
|
headers: { Authorization: `Bearer ${token}`, Accept: "application/vnd.github+json" }, signal: AbortSignal.timeout(30_000) });
|
|
if (!response.ok) throw Object.assign(new Error(`Could not admit deferred release work: HTTP ${response.status}.`), { status: response.status });
|
|
};
|
|
}
|
|
|
|
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
|
if (process.env.GITHUB_REPOSITORY !== repository || process.env.GITHUB_REF !== "refs/heads/master" || !process.env.GITHUB_TOKEN) throw new Error("Canonical master admission context required.");
|
|
const admitted = await releaseAdmitted(process.env.GITHUB_SHA, { api: createActionsReader({ token: process.env.GITHUB_TOKEN }),
|
|
event: process.env.GITHUB_EVENT_NAME, attempt: Number(process.env.GITHUB_RUN_ATTEMPT), imageOnly: process.argv[2] === "image-only" });
|
|
console.log(admitted ? "Release candidate admitted." : "Full build deferred until a release consumer admits this source.");
|
|
if (process.env.GITHUB_OUTPUT) await appendFile(process.env.GITHUB_OUTPUT, `admitted=${admitted}\n`);
|
|
}
|