From efc0b359da0b8076b9201f9a01903a257da39df1 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 22:29:11 -0600 Subject: [PATCH] fix(ci): rebuild and sign an unsigned reused Daytona e2e image (#19677) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paid runner E2E campaigns run in `runner-full-stack-e2e.yml`. Daytona cells need a signed, content-addressed runner image > - The "Publish verified Daytona image" job reuses `e2e-content-` when the tag exists, and signs only after a fresh build > - A run that stops between the push and `cosign sign` leaves an unsigned tag. Every later run reuses it and fails `cosign verify` with "no signatures found" > - That failure skips the whole test matrix, including local cells. The first Reliability TestOps launch (run 42298071794) ran 0/7 cells for this reason > - This pull request reuses a tag only when its signature verifies. Otherwise it rebuilds, pushes and signs the tag > - The benefit is that an interrupted run cannot block every later Daytona campaign ## Linked Issues or Issue Description Refs #19549 **What happened?** Run https://github.com/paperclipai/paperclip/actions/runs/42298071794 (TestOps launch of the Reliability plan) failed in "Reuse or publish immutable image" with `Error: no signatures found` (exit code 10). No test cell ran. The report shows 7 missing attempts. **Expected behavior** A campaign with Daytona cells always produces a signed, verified image, or it fails for a real build or signing error. A stale unsigned tag must not block it. **Steps to reproduce** 1. A run pushes `ghcr.io/paperclipai/paperclip-daytona-runner:e2e-content-` and stops before `cosign sign` (for example, a runner shutdown). 2. A later campaign with the same content ID finds the tag and skips the build. 3. `cosign verify` fails with "no signatures found", and the matrix is skipped. **Paperclip version or commit** master 02a2299173. ## What Changed - `.github/workflows/runner-full-stack-e2e.yml`: add a `verify_signed` helper. Reuse an existing content tag only when its digest has this workflow's signature. If verification fails, warn, rebuild, push and sign. The final `cosign verify`, logout and config checks are unchanged. - `tests/runner-e2e/daytona-image.test.ts`: assert that reuse is gated on signature verification and that the build-and-sign path follows it. ## Verification - `npx vitest run daytona-image workflow-security` in `tests/runner-e2e`: 38/38 pass. - Red before the fix: the new assertion fails on the unchanged workflow (1 failed, 18 passed in `daytona-image`). - The workflow YAML parses. - Live check after merge: relaunch the Reliability TestOps plan. The image job should warn about the unsigned tag, rebuild it, sign it and verify it, and then the cells run. ## Risks - Low risk. The rebuild uses the same trusted inputs and cache, and it only happens when verification fails. The new digest replaces the unsigned tag. The identity and issuer checks are unchanged. - Rebuilding an unsigned tag takes extra time on that one run. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code with tool use (shell, browser read of the GitHub job log), standard context. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .github/workflows/runner-full-stack-e2e.yml | 21 +++++++++++++++++++-- tests/runner-e2e/daytona-image.test.ts | 8 ++++++++ 2 files changed, 27 insertions(+), 2 deletions(-) diff --git a/.github/workflows/runner-full-stack-e2e.yml b/.github/workflows/runner-full-stack-e2e.yml index 55a08b6e689..6d216fc600b 100644 --- a/.github/workflows/runner-full-stack-e2e.yml +++ b/.github/workflows/runner-full-stack-e2e.yml @@ -647,9 +647,26 @@ jobs: [[ "$IMAGE_CONTENT_ID" =~ ^[0-9a-f]{64}$ ]] [[ "$TARGET_LOCK_SHA256" =~ ^[0-9a-f]{64}$ ]] identity="^https://github.com/${GITHUB_REPOSITORY}/.github/workflows/runner-full-stack-e2e.yml@" + verify_signed() { + cosign verify \ + --certificate-identity-regexp "$identity" \ + --certificate-oidc-issuer https://token.actions.githubusercontent.com \ + "$IMAGE_TAG@$1" >/dev/null + } + # Reuse an existing content tag only when its digest carries this + # workflow's signature. A run interrupted between push and sign + # leaves an unsigned tag; rebuilding, pushing and signing replaces it + # instead of failing every later campaign with "no signatures found". + digest="" if docker buildx imagetools inspect "$IMAGE_TAG" >/dev/null 2>&1; then - digest="$(docker buildx imagetools inspect "$IMAGE_TAG" --format '{{json .Manifest.Digest}}' | tr -d '"')" - else + existing="$(docker buildx imagetools inspect "$IMAGE_TAG" --format '{{json .Manifest.Digest}}' | tr -d '"')" + if verify_signed "$existing" 2>/dev/null; then + digest="$existing" + else + echo "::warning title=Daytona image::Existing $IMAGE_TAG ($existing) has no verifiable workflow signature; rebuilding and signing it" + fi + fi + if [ -z "$digest" ]; then cache_args=( --cache-from "type=registry,ref=${IMAGE_CACHE}" ) diff --git a/tests/runner-e2e/daytona-image.test.ts b/tests/runner-e2e/daytona-image.test.ts index cd6a7019b88..cd0eb22f67a 100644 --- a/tests/runner-e2e/daytona-image.test.ts +++ b/tests/runner-e2e/daytona-image.test.ts @@ -282,6 +282,14 @@ describe("runner E2E Daytona image contract", () => { ); expect(workflow).not.toContain("e2e-git-${{ github.sha }}"); expect(workflow).toContain("cosign sign --yes"); + // An existing content tag is reused only when its signature verifies; an + // unsigned tag left by an interrupted run is rebuilt, pushed and signed. + const imageJob = daytonaImageJob ?? ""; + const reuse = imageJob.indexOf('if verify_signed "$existing"'); + expect(reuse).toBeGreaterThan(-1); + expect(imageJob.indexOf('if [ -z "$digest" ]; then')).toBeGreaterThan(reuse); + expect(imageJob.indexOf("cosign sign --yes")).toBeGreaterThan(reuse); + expect(imageJob).not.toMatch(/imagetools inspect "\$IMAGE_TAG" >\/dev\/null 2>&1; then\n\s*digest=/); expect(workflow).toContain("docker logout ghcr.io"); expect(workflow).toContain(`docker buildx imagetools inspect "$immutable"`); expect(workflow).toContain(`--format '{{json .Image}}'`);