From 83b5f7d486e70e484acd9a41bc957054f4e10175 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 3 Nov 2026 12:47:16 -0600 Subject: [PATCH] fix(ci): keep CODEOWNER PRs on AWS as master advances (#18821) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its PR checks use reserved AWS capacity for authorized maintainers. > - The routing gate waited for GitHub capacity and compared an old test merge with a newer base branch. > - Normal merges into master then sent trusted PR jobs to the GitHub queue. > - This change restores AWS admission and validates the immutable merge selected for the run. > - Current CODEOWNER and rerun identity checks still control PR execution. ## Linked Issues or Issue Description Related: #18376 moved the gate to GitHub-hosted capacity. #16042 introduced reserved AWS admission. **What happened?** In PR #18811, GitHub created test merge `608d862ff1650db108a3832a2c67f8795924f0e7` at 17:17 UTC. Master advanced at 17:25. The gate started at 17:30 and rejected the older merge parent because it did not contain the newer master tip. The authorized PR then selected GitHub-hosted runners. The gate itself also needed a GitHub-hosted slot. **Expected behavior** Authorized maintainers use reserved AWS admission and PR workers. Normal base advancement does not change runner placement. Current owner checks, PR head checks, run-attempt checks, and base-history validation remain required. **Steps to reproduce** 1. Start a PR run whose author and sender are current CI CODEOWNERS. 2. Advance master while its routing job waits. 3. Run the gate against the original event merge and the current base tip. 4. Before this change, the gate selects `ubuntu-latest`. After this change, it selects the public PR AWS fleet. **Paperclip version or commit** Reproduced on master `367f29478d14003cc5c0f161b7ce8f3164993634`. **Deployment mode** GitHub Actions with RunsOn Fleet on AWS. ## What Changed - Restore the reserved routing fleet for the synchronized maintainer admission roster. - Check the immutable event merge and its PR head. Accept base snapshots between the event base and the current base tip. - Allow GitHub to regenerate its live merge ref while the original run waits. - Retain stacked-base validation and all current CODEOWNER, repository, author, sender, and rerun-attempt checks. - Add advancement, regenerated-merge, unrelated-history, and admission-placement regressions, including a stacked parent that descends from the event base but diverges from the live base. - Align the developing and CI profiling guides with reserved AWS admission and current CODEOWNER checks before PR execution. ## Verification - `node --test .github/scripts/tests/ci-routing.test.mjs scripts/__tests__/e2e-shard.test.mjs`: 103 passed. - `node --test .github/scripts/tests/ci-profile.test.mjs .github/scripts/tests/ci-plan.test.mjs`: 31 passed. - `actionlint .github/workflows/ci-routing.yml`: passed. - `git diff --check`: passed. - Mutation check: removing the snapshot-to-live-base comparison in a temporary workflow copy makes the new side-branch regression fail, proving it protects that ancestry guard. - [Greptile: 5/5](https://github.com/paperclipai/paperclip/pull/18821#issuecomment-6612170970) with a successful check on `abd6a8ce347228219d8e5127f34f57151788830e`; both review threads are resolved. - A regression reproduction on the original workflow selected GitHub instead of AWS. The patched workflow passes that scenario. - A read-only execution of the patched routing step against the actual event for run [41682867664](https://github.com/paperclipai/paperclip/actions/runs/41682867664), using live GitHub metadata, selected `runs-on/fleet=paperclip-public-pr-x64/env=public-ci`. This did not allocate a runner or rerun CI. - Full repository typecheck, Vitest, and build were not run in this worktree, which has no installed dependencies. The change is limited to workflow routing, its tests, and documentation. Broader CI checks remain queued. ## Risks - A stale admission roster can allocate the five-minute metadata gate. The gate still checks current CODEOWNERS before it authorizes any PR code on AWS. - The routing runner group must authorize `ci-routing.yml@refs/heads/master`. The fleet already exists; live placement still needs confirmation after merge. - Rewritten base history, changed PR heads, and unavailable identity or ancestry evidence continue to deny AWS PR execution. - Existing runs retain their previously resolved workflow. New runs use this change after it reaches master. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository inspection, code execution, and live GitHub API tools. The runtime does not expose the exact serving model variant or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .github/scripts/tests/ci-routing.test.mjs | 88 +++++++++++++++++++++-- .github/workflows/ci-routing.yml | 60 ++++++++-------- doc/DEVELOPING.md | 33 ++++++--- doc/ci-profiling.md | 9 ++- scripts/__tests__/e2e-shard.test.mjs | 4 +- 5 files changed, 145 insertions(+), 49 deletions(-) diff --git a/.github/scripts/tests/ci-routing.test.mjs b/.github/scripts/tests/ci-routing.test.mjs index ae43a65791f..3b590332789 100644 --- a/.github/scripts/tests/ci-routing.test.mjs +++ b/.github/scripts/tests/ci-routing.test.mjs @@ -4,15 +4,17 @@ import { spawnSync } from "node:child_process"; import { tmpdir } from "node:os"; import path from "node:path"; import test from "node:test"; +import vm from "node:vm"; const workflow = readFileSync(new URL("../../workflows/ci-routing.yml", import.meta.url), "utf8"); const jobs = workflow.split(/^jobs:\n/m)[1]; -test("authorization uses hosted capacity and runs no PR code or actions", () => { +test("authorization uses reserved capacity and runs no PR code or actions", () => { assert.deepEqual([...jobs.matchAll(/^ {2}([a-z_]+):$/gm)].map((match) => match[1]), ["gate"]); const placement = jobs.match(/^ {4}runs-on: (.+)$/m)?.[1]; assert.ok(placement, "routing placement must be defined"); - assert.equal(placement, "ubuntu-latest", "verify current owners before allocating any AWS runner"); + assert.match(placement, /runs-on\/fleet=paperclip-ci-routing-x64\/env=public-ci/); + assert.doesNotMatch(placement, /fromJSON/); assert.doesNotMatch(jobs, /uses:|actions\/checkout|secrets\./); assert.doesNotMatch(workflow, /^ {2}(?:actions|contents|pull-requests): write$/m); }); @@ -24,7 +26,7 @@ test("routing exports both placement and stack scope to callers", () => { assert.match(jobs, /^ {6}full_ci: \$\{\{ steps\.scope\.outputs\.full_ci \}\}$/m); }); -const githubStub = "#!/usr/bin/env bash\nset -Eeuo pipefail\n\nendpoint=\"\"\nquery=\"\"\nfor arg in \"$@\"; do\n case \"$arg\" in /repos/*|/users/*) endpoint=\"$arg\" ;; esac\ndone\nif [[ \"$*\" == *\"--jq\"* ]]; then query=\"${*: -1}\"; fi\ncase \"$endpoint\" in\n */contents/scripts/ci-profile/authorize.mjs?ref=*)\n [[ -z \"${TEST_OWNER_HELPER_FAIL:-}\" ]] || exit 1\n [[ \"$endpoint\" == *\"ref=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\" ]] || exit 1\n base64 < \"$TEST_OWNER_HELPER\"\n ;;\n */contents/.github/CODEOWNERS?ref=*)\n [[ -z \"${TEST_CODEOWNERS_FAIL:-}\" ]] || exit 1\n [[ \"$endpoint\" == *\"ref=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\" ]] || exit 1\n jq -cn --arg content \"$TEST_CODEOWNERS_BASE64\" '{encoding: \"base64\", content: $content}'\n ;;\n /users/ci-owner)\n [[ -z \"${TEST_OWNER_LOOKUP_FAIL:-}\" ]] || exit 1\n jq -cn --argjson id \"${TEST_CI_OWNER_ID:-34892728}\" --arg type \"${TEST_CI_OWNER_TYPE:-User}\" '{id: $id, type: $type}'\n ;;\n */pulls/*)\n jq -cn \\\n --argjson author_id \"${TEST_PR_AUTHOR_ID:-34892728}\" \\\n --argjson repository_id \"${TEST_REPOSITORY_ID:-1170821064}\" \\\n --arg base_ref \"${TEST_BASE_REF:-master}\" \\\n --arg base_sha \"${TEST_PR_BASE_SHA:-$EVENT_BASE_SHA}\" \\\n --arg head_sha \"${TEST_PR_HEAD_SHA:-1111111111111111111111111111111111111111}\" \\\n --arg merge_sha \"${TEST_PR_MERGE_SHA:-4444444444444444444444444444444444444444}\" '{\n state: \"open\",\n user: {id: $author_id},\n base: {repo: {id: $repository_id}, ref: $base_ref, sha: $base_sha},\n head: {sha: $head_sha},\n merge_commit_sha: $merge_sha\n }'\n ;;\n */git/ref/heads/*)\n if [[ \"$query\" == \".object.sha\" ]]; then\n [[ -z \"${TEST_MASTER_FAIL:-}\" ]] || exit 1\n echo \"${TEST_TRUSTED_MASTER_SHA:-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa}\"\n exit 0\n fi\n jq -cn \\\n --arg sha \"${TEST_LIVE_BASE_REF_SHA:-$EVENT_BASE_SHA}\" '{object: {sha: $sha}}'\n ;;\n */compare/*)\n [[ -z \"${TEST_COMPARE_FAIL:-}\" ]] || exit 1\n default_files='[{\"filename\":\"server/src/index.ts\"}]'\n jq -cn \\\n --arg merge_base_sha \"${TEST_BASE_COMPARISON_MERGE_BASE_SHA:-$EVENT_BASE_SHA}\" \\\n --arg status \"${TEST_BASE_COMPARISON_STATUS:-ahead}\" \\\n --argjson files \"${TEST_COMPARE_FILES:-$default_files}\" '{\n merge_base_commit: {sha: $merge_base_sha},\n status: $status,\n files: $files\n }'\n ;;\n */git/commits/*)\n requested_sha=\"${endpoint##*/}\"\n live_merge_sha=\"${TEST_PR_MERGE_SHA:-4444444444444444444444444444444444444444}\"\n synthetic_base_merge_sha=\"${TEST_SYNTHETIC_BASE_MERGE_SHA:-}\"\n if [[ \"$requested_sha\" == \"$EVENT_MERGE_SHA\" ]]; then\n tree_sha=\"${TEST_EVENT_MERGE_TREE_SHA:-5555555555555555555555555555555555555555}\"\n base_sha=\"${TEST_EVENT_MERGE_BASE_SHA:-$EVENT_BASE_SHA}\"\n head_sha=\"${TEST_EVENT_MERGE_HEAD_SHA:-$EVENT_HEAD_SHA}\"\n elif [[ \"$requested_sha\" == \"$live_merge_sha\" ]]; then\n tree_sha=\"${TEST_LIVE_MERGE_TREE_SHA:-5555555555555555555555555555555555555555}\"\n base_sha=\"${TEST_LIVE_MERGE_BASE_SHA:-$EVENT_BASE_SHA}\"\n head_sha=\"${TEST_LIVE_MERGE_HEAD_SHA:-$EVENT_HEAD_SHA}\"\n elif [[ -n \"$synthetic_base_merge_sha\" && \"$requested_sha\" == \"$synthetic_base_merge_sha\" ]]; then\n tree_sha=\"${TEST_SYNTHETIC_BASE_MERGE_TREE_SHA:-8888888888888888888888888888888888888888}\"\n base_sha=\"${TEST_SYNTHETIC_MASTER_PARENT_SHA:-9999999999999999999999999999999999999999}\"\n head_sha=\"${TEST_SYNTHETIC_BASE_REF_PARENT_SHA:-${TEST_LIVE_BASE_REF_SHA:-$EVENT_BASE_SHA}}\"\n else\n echo \"unexpected merge commit: $requested_sha\" >&2\n exit 1\n fi\n jq -cn \\\n --arg sha \"$requested_sha\" \\\n --arg tree_sha \"$tree_sha\" \\\n --arg base_sha \"$base_sha\" \\\n --arg head_sha \"$head_sha\" '{\n sha: $sha,\n tree: {sha: $tree_sha},\n parents: [{sha: $base_sha}, {sha: $head_sha}]\n }'\n ;;\n */actions/runs/*/attempts/*)\n jq -cn \\\n --argjson actor_id \"${TEST_TRIGGERING_ACTOR_ID:-34892728}\" \\\n --argjson repository_id \"${TEST_REPOSITORY_ID:-1170821064}\" \\\n --argjson run_attempt \"${TEST_RUN_ATTEMPT:-1}\" \\\n --arg event_name \"${TEST_EVENT_NAME:-pull_request}\" '{\n run_attempt: $run_attempt,\n triggering_actor: {id: $actor_id},\n repository: {id: $repository_id},\n event: $event_name\n }'\n ;;\n *)\n echo \"unexpected gh endpoint: $endpoint\" >&2\n exit 1\n ;;\nesac"; +const githubStub = "#!/usr/bin/env bash\nset -Eeuo pipefail\n\nendpoint=\"\"\nquery=\"\"\nfor arg in \"$@\"; do\n case \"$arg\" in /repos/*|/users/*) endpoint=\"$arg\" ;; esac\ndone\nif [[ \"$*\" == *\"--jq\"* ]]; then query=\"${*: -1}\"; fi\ncase \"$endpoint\" in\n */contents/scripts/ci-profile/authorize.mjs?ref=*)\n [[ -z \"${TEST_OWNER_HELPER_FAIL:-}\" ]] || exit 1\n [[ \"$endpoint\" == *\"ref=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\" ]] || exit 1\n base64 < \"$TEST_OWNER_HELPER\"\n ;;\n */contents/.github/CODEOWNERS?ref=*)\n [[ -z \"${TEST_CODEOWNERS_FAIL:-}\" ]] || exit 1\n [[ \"$endpoint\" == *\"ref=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\" ]] || exit 1\n jq -cn --arg content \"$TEST_CODEOWNERS_BASE64\" '{encoding: \"base64\", content: $content}'\n ;;\n /users/ci-owner)\n [[ -z \"${TEST_OWNER_LOOKUP_FAIL:-}\" ]] || exit 1\n jq -cn --argjson id \"${TEST_CI_OWNER_ID:-34892728}\" --arg type \"${TEST_CI_OWNER_TYPE:-User}\" '{id: $id, type: $type}'\n ;;\n */pulls/*)\n jq -cn \\\n --argjson author_id \"${TEST_PR_AUTHOR_ID:-34892728}\" \\\n --argjson repository_id \"${TEST_REPOSITORY_ID:-1170821064}\" \\\n --arg base_ref \"${TEST_BASE_REF:-master}\" \\\n --arg base_sha \"${TEST_PR_BASE_SHA:-$EVENT_BASE_SHA}\" \\\n --arg head_sha \"${TEST_PR_HEAD_SHA:-1111111111111111111111111111111111111111}\" \\\n --arg merge_sha \"${TEST_PR_MERGE_SHA:-4444444444444444444444444444444444444444}\" '{\n state: \"open\",\n user: {id: $author_id},\n base: {repo: {id: $repository_id}, ref: $base_ref, sha: $base_sha},\n head: {sha: $head_sha},\n merge_commit_sha: $merge_sha\n }'\n ;;\n */git/ref/heads/*)\n if [[ \"$query\" == \".object.sha\" ]]; then\n [[ -z \"${TEST_MASTER_FAIL:-}\" ]] || exit 1\n echo \"${TEST_TRUSTED_MASTER_SHA:-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa}\"\n exit 0\n fi\n jq -cn \\\n --arg sha \"${TEST_LIVE_BASE_REF_SHA:-$EVENT_BASE_SHA}\" '{object: {sha: $sha}}'\n ;;\n */compare/*)\n [[ -z \"${TEST_COMPARE_FAIL:-}\" ]] || exit 1\n comparison=\"${endpoint##*/}\"\n comparison_base=\"${comparison%%...*}\"\n comparison_head=\"${comparison##*...}\"\n default_files='[{\"filename\":\"server/src/index.ts\"}]'\n jq -cn \\\n --arg base \"$comparison_base\" \\\n --arg head \"$comparison_head\" \\\n --arg event_base \"$EVENT_BASE_SHA\" \\\n --arg intermediate \"${TEST_INTERMEDIATE_BASE_SHA:-}\" \\\n --arg current \"${TEST_LIVE_BASE_REF_SHA:-$EVENT_BASE_SHA}\" \\\n --arg side_branch \"${TEST_SIDE_BRANCH_SHA:-}\" \\\n --arg merge_base_override \"${TEST_BASE_COMPARISON_MERGE_BASE_SHA:-}\" \\\n --arg status_override \"${TEST_BASE_COMPARISON_STATUS:-}\" \\\n --argjson files \"${TEST_COMPARE_FILES:-$default_files}\" '\n [$event_base, $intermediate, $current] | map(select(length > 0)) as $history |\n ($history | index($base)) as $from |\n ($history | index($head)) as $to |\n (($from != null and $to != null and $from <= $to) or\n ($side_branch != \"\" and $base == $event_base and $head == $side_branch)) as $ancestor |\n {\n merge_base_commit: {sha: (if $merge_base_override != \"\" then $merge_base_override\n elif $ancestor then $base\n elif $base == $side_branch and $to != null then $event_base\n else \"0000000000000000000000000000000000000000\" end)},\n status: (if $status_override != \"\" then $status_override\n elif $base == $head then \"identical\" elif $ancestor then \"ahead\" else \"diverged\" end),\n files: $files\n }'\n ;;\n */git/commits/*)\n requested_sha=\"${endpoint##*/}\"\n live_merge_sha=\"${TEST_PR_MERGE_SHA:-4444444444444444444444444444444444444444}\"\n synthetic_base_merge_sha=\"${TEST_SYNTHETIC_BASE_MERGE_SHA:-}\"\n if [[ \"$requested_sha\" == \"$EVENT_MERGE_SHA\" ]]; then\n tree_sha=\"${TEST_EVENT_MERGE_TREE_SHA:-5555555555555555555555555555555555555555}\"\n base_sha=\"${TEST_EVENT_MERGE_BASE_SHA:-$EVENT_BASE_SHA}\"\n head_sha=\"${TEST_EVENT_MERGE_HEAD_SHA:-$EVENT_HEAD_SHA}\"\n elif [[ \"$requested_sha\" == \"$live_merge_sha\" ]]; then\n tree_sha=\"${TEST_LIVE_MERGE_TREE_SHA:-5555555555555555555555555555555555555555}\"\n base_sha=\"${TEST_LIVE_MERGE_BASE_SHA:-$EVENT_BASE_SHA}\"\n head_sha=\"${TEST_LIVE_MERGE_HEAD_SHA:-$EVENT_HEAD_SHA}\"\n elif [[ -n \"$synthetic_base_merge_sha\" && \"$requested_sha\" == \"$synthetic_base_merge_sha\" ]]; then\n tree_sha=\"${TEST_SYNTHETIC_BASE_MERGE_TREE_SHA:-8888888888888888888888888888888888888888}\"\n base_sha=\"${TEST_SYNTHETIC_MASTER_PARENT_SHA:-9999999999999999999999999999999999999999}\"\n head_sha=\"${TEST_SYNTHETIC_BASE_REF_PARENT_SHA:-${TEST_LIVE_BASE_REF_SHA:-$EVENT_BASE_SHA}}\"\n else\n echo \"unexpected merge commit: $requested_sha\" >&2\n exit 1\n fi\n jq -cn \\\n --arg sha \"$requested_sha\" \\\n --arg tree_sha \"$tree_sha\" \\\n --arg base_sha \"$base_sha\" \\\n --arg head_sha \"$head_sha\" '{\n sha: $sha,\n tree: {sha: $tree_sha},\n parents: [{sha: $base_sha}, {sha: $head_sha}]\n }'\n ;;\n */actions/runs/*/attempts/*)\n jq -cn \\\n --argjson actor_id \"${TEST_TRIGGERING_ACTOR_ID:-34892728}\" \\\n --argjson repository_id \"${TEST_REPOSITORY_ID:-1170821064}\" \\\n --argjson run_attempt \"${TEST_RUN_ATTEMPT:-1}\" \\\n --arg event_name \"${TEST_EVENT_NAME:-pull_request}\" '{\n run_attempt: $run_attempt,\n triggering_actor: {id: $actor_id},\n repository: {id: $repository_id},\n event: $event_name\n }'\n ;;\n *)\n echo \"unexpected gh endpoint: $endpoint\" >&2\n exit 1\n ;;\nesac"; function stepScript(name) { const start = workflow.indexOf(` - name: ${name}\n`); @@ -100,6 +102,31 @@ const routeCases = [ ["unrelated label", { EVENT_ACTION: "labeled", EVENT_LABEL_NAME: "documentation" }], ["validated stacked base", stackedBase, awsRunner], ["lagging PR API base snapshot", { TEST_PR_BASE_SHA: "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" }, awsRunner], + ["base advances while the routing job waits", { TEST_LIVE_BASE_REF_SHA: "7777777777777777777777777777777777777777" }, awsRunner], + ["merge uses a newer base snapshot than the event payload", { + TEST_LIVE_BASE_REF_SHA: "7777777777777777777777777777777777777777", + TEST_INTERMEDIATE_BASE_SHA: "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + TEST_EVENT_MERGE_BASE_SHA: "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + }, awsRunner], + ["GitHub regenerates the live merge after base advancement", { + TEST_LIVE_BASE_REF_SHA: "7777777777777777777777777777777777777777", + TEST_LIVE_MERGE_BASE_SHA: "7777777777777777777777777777777777777777", + TEST_LIVE_MERGE_TREE_SHA: "6666666666666666666666666666666666666666", + }, awsRunner], + ["live merge is temporarily unavailable", { TEST_PR_MERGE_SHA: "unavailable" }, awsRunner], + ["stacked base advances while the routing job waits", { + ...stackedBase, + TEST_SYNTHETIC_BASE_REF_PARENT_SHA: "7777777777777777777777777777777777777777", + TEST_INTERMEDIATE_BASE_SHA: "7777777777777777777777777777777777777777", + TEST_LIVE_BASE_REF_SHA: "cccccccccccccccccccccccccccccccccccccccc", + }, awsRunner], + ["merge parent outside the base branch history", { TEST_EVENT_MERGE_BASE_SHA: "cccccccccccccccccccccccccccccccccccccccc" }], + ["stacked base parent descends from the event base but diverges from the live base", { + ...stackedBase, + TEST_SIDE_BRANCH_SHA: "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + TEST_SYNTHETIC_BASE_REF_PARENT_SHA: "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + }], + ["base history lookup unavailable", { TEST_COMPARE_FAIL: "1" }], ["disabled AWS", { AWS_CI_ENABLED: "false" }], ["missing AWS variables", { AWS_CI_ENABLED: "", TRUSTED_USER_IDS: "" }], ["unlisted author", { EVENT_PR_AUTHOR_ID: "99999999" }], @@ -125,9 +152,9 @@ const routeCases = [ ["wrong base repository", { EVENT_BASE_REPOSITORY_ID: "999999999" }], ["replaced base ancestry", { TEST_BASE_COMPARISON_MERGE_BASE_SHA: "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" }], ["stale head", { TEST_PR_HEAD_SHA: "3333333333333333333333333333333333333333" }], - ["changed merge head", { TEST_LIVE_MERGE_HEAD_SHA: "3333333333333333333333333333333333333333" }], - ["stacked base with wrong parent", { ...stackedBase, TEST_SYNTHETIC_BASE_REF_PARENT_SHA: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" }], - ["changed merge tree", { TEST_LIVE_MERGE_TREE_SHA: "6666666666666666666666666666666666666666" }], + ["event merge has the wrong head", { TEST_EVENT_MERGE_HEAD_SHA: "3333333333333333333333333333333333333333" }], + ["stacked base with wrong parent", { ...stackedBase, TEST_SYNTHETIC_BASE_REF_PARENT_SHA: "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" }], + ["event merge has an invalid tree", { TEST_EVENT_MERGE_TREE_SHA: "invalid" }], ["wrong repository", { EVENT_REPOSITORY_ID: "999999999" }], ["malformed allowlist", { TRUSTED_USER_IDS: '["34892728"]' }], ["manual dispatch", { EVENT_NAME: "workflow_dispatch" }], @@ -241,3 +268,52 @@ test("path scope skips full CI for a single-PR stack that only touches release n TEST_COMPARE_FILES: files("releases/v2026.1015.0.md"), }).full_ci, "false"); }); + +function routingPlacement(vars = {}, github = {}) { + const source = "[34892728]"; + const expression = jobs.match(/^ {4}runs-on: \$\{\{ (.+) \}\}$/m)?.[1]; + assert.ok(expression, "routing expression is missing"); + return vm.runInNewContext(expression, { + vars: { + AWS_CI_ENABLED: "true", + AWS_CI_TRUSTED_USER_IDS: source, + AWS_CI_ROUTING_USER_IDS_SOURCE: source, + AWS_CI_ROUTING_USER_IDS: "|34892728|", + ...vars, + }, + github: { + repository: "paperclipai/paperclip", + repository_id: "1170821064", + event_name: "pull_request", + ref: "refs/heads/master", + actor_id: "34892728", + event: { pull_request: { user: { id: 34892728 } }, sender: { id: 34892728 } }, + ...github, + }, + contains: (search, item) => String(search).toLowerCase().includes(String(item).toLowerCase()), + format: (template, ...args) => template.replace(/\{(\d+)\}/g, (_, index) => String(args[Number(index)])), + }, { timeout: 1000 }); +} + +const placementCases = [ + ["approved PR", {}, {}, "runs-on/fleet=paperclip-ci-routing-x64/env=public-ci"], + ["unapproved author", {}, { event: { pull_request: { user: { id: 999 } }, sender: { id: 34892728 } } }], + ["unapproved sender", {}, { event: { pull_request: { user: { id: 34892728 } }, sender: { id: 999 } } }], + ["partial ID cannot match", {}, { event: { pull_request: { user: { id: 4892728 } }, sender: { id: 34892728 } } }], + ["invalid JSON keeps hosted fallback", { AWS_CI_TRUSTED_USER_IDS: "[34892728,]" }, {}], + ["stale snapshot", { AWS_CI_TRUSTED_USER_IDS: "[34892728,999]" }, {}], + ["missing snapshot", { AWS_CI_ROUTING_USER_IDS_SOURCE: "" }, {}], + ["missing admission members", { AWS_CI_ROUTING_USER_IDS: "" }, {}], + ["disabled routing", { AWS_CI_ENABLED: "false" }, {}], + ["different repository", {}, { repository: "other/repo" }], + ["different repository ID", {}, { repository_id: "999" }], + ["approved manual master canary", {}, { event_name: "workflow_dispatch" }, "runs-on/fleet=paperclip-ci-routing-x64/env=public-ci"], + ["unapproved manual actor", {}, { event_name: "workflow_dispatch", actor_id: "999" }], + ["manual branch canary", {}, { event_name: "workflow_dispatch", ref: "refs/heads/other" }], +]; + +for (const [name, vars, github, expected = "ubuntu-latest"] of placementCases) { + test(`placement before allocation: ${name}`, () => { + assert.equal(routingPlacement(vars, github), expected); + }); +} diff --git a/.github/workflows/ci-routing.yml b/.github/workflows/ci-routing.yml index 8cfaebf342c..7fc7416f369 100644 --- a/.github/workflows/ci-routing.yml +++ b/.github/workflows/ci-routing.yml @@ -21,9 +21,7 @@ permissions: jobs: gate: name: Select trusted runner - # No AWS capacity is allocated until current CI CODEOWNERS are verified. - # A cached numeric allowlist alone can outlive removal from CODEOWNERS. - runs-on: ubuntu-latest + runs-on: ${{ vars.AWS_CI_ENABLED == 'true' && vars.AWS_CI_ROUTING_USER_IDS_SOURCE != '' && vars.AWS_CI_TRUSTED_USER_IDS == vars.AWS_CI_ROUTING_USER_IDS_SOURCE && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && ((github.event_name == 'pull_request' && contains(vars.AWS_CI_ROUTING_USER_IDS, format('|{0}|', github.event.pull_request.user.id)) && contains(vars.AWS_CI_ROUTING_USER_IDS, format('|{0}|', github.event.sender.id))) || (github.event_name == 'workflow_dispatch' && github.ref == 'refs/heads/master' && contains(vars.AWS_CI_ROUTING_USER_IDS, format('|{0}|', github.actor_id)))) && 'runs-on/fleet=paperclip-ci-routing-x64/env=public-ci' || 'ubuntu-latest' }} timeout-minutes: 5 outputs: plan: ${{ steps.plan.outputs.plan }} @@ -145,9 +143,6 @@ jobs: ' <<< "$pr_json" >/dev/null 2>&1 \ || fail_closed 'current pull request state does not match the triggering event' - live_merge_sha="$(jq -r '.merge_commit_sha // empty' <<< "$pr_json")" - is_commit_sha "$live_merge_sha" || fail_closed 'current pull request has no valid merge SHA' - base_ref_json="$(gh api \ -H 'Accept: application/vnd.github+json' \ -H 'X-GitHub-Api-Version: 2022-11-28' \ @@ -174,12 +169,6 @@ jobs: "/repos/paperclipai/paperclip/git/commits/$EVENT_MERGE_SHA" 2>/dev/null)" \ || fail_closed 'could not inspect the event merge commit' - live_merge_json="$(gh api \ - -H 'Accept: application/vnd.github+json' \ - -H 'X-GitHub-Api-Version: 2022-11-28' \ - "/repos/paperclipai/paperclip/git/commits/$live_merge_sha" 2>/dev/null)" \ - || fail_closed 'could not inspect the current merge commit' - validate_merge_commit() { local merge_json="$1" local expected_sha="$2" @@ -195,37 +184,50 @@ jobs: } validate_merge_commit "$event_merge_json" "$EVENT_MERGE_SHA" \ - || fail_closed 'event merge commit does not match the current base and head' - validate_merge_commit "$live_merge_json" "$live_merge_sha" \ - || fail_closed 'current merge commit does not match the triggering base and head' + || fail_closed 'event merge commit does not match the triggering head' + + is_base_snapshot() { + local snapshot_sha="$1" + local comparison + [[ "$snapshot_sha" == "$EVENT_BASE_SHA" || "$snapshot_sha" == "$live_base_ref_sha" ]] && return 0 + comparison="$(gh api "/repos/paperclipai/paperclip/compare/$EVENT_BASE_SHA...$snapshot_sha" 2>/dev/null)" || return 1 + jq -e --arg base "$EVENT_BASE_SHA" ' + .merge_base_commit.sha == $base and + (.status == "ahead" or .status == "identical") + ' <<< "$comparison" >/dev/null 2>&1 || return 1 + comparison="$(gh api "/repos/paperclipai/paperclip/compare/$snapshot_sha...$live_base_ref_sha" 2>/dev/null)" || return 1 + jq -e --arg base "$snapshot_sha" ' + .merge_base_commit.sha == $base and + (.status == "ahead" or .status == "identical") + ' <<< "$comparison" >/dev/null 2>&1 + } event_merge_parent="$(jq -r '.parents[0].sha' <<< "$event_merge_json")" - live_merge_parent="$(jq -r '.parents[0].sha' <<< "$live_merge_json")" - [[ "$event_merge_parent" == "$live_merge_parent" ]] \ - || fail_closed 'current merge commit uses a different base merge parent' - - if [[ "$event_merge_parent" != "$live_base_ref_sha" ]]; then + if ! is_base_snapshot "$event_merge_parent"; then base_merge_json="$(gh api \ -H 'Accept: application/vnd.github+json' \ -H 'X-GitHub-Api-Version: 2022-11-28' \ "/repos/paperclipai/paperclip/git/commits/$event_merge_parent" 2>/dev/null)" \ || fail_closed 'could not inspect the stacked base merge commit' jq -e \ - --arg expected_sha "$event_merge_parent" \ - --arg live_base_ref_sha "$live_base_ref_sha" ' + --arg expected_sha "$event_merge_parent" ' .sha == $expected_sha and (.parents | length) == 2 and - any(.parents[]; .sha == $live_base_ref_sha) and + all(.parents[]; .sha | test("^[0-9a-f]{40}$")) and (.tree.sha | test("^[0-9a-f]{40}$")) ' <<< "$base_merge_json" >/dev/null 2>&1 \ - || fail_closed 'stacked base merge commit does not contain the current base branch' + || fail_closed 'invalid stacked base merge commit' + base_snapshot_found=false + while read -r parent_sha; do + if is_base_snapshot "$parent_sha"; then + base_snapshot_found=true + break + fi + done < <(jq -r '.parents[].sha' <<< "$base_merge_json") + [[ "$base_snapshot_found" == 'true' ]] \ + || fail_closed 'stacked base merge commit does not contain a validated base snapshot' fi - event_merge_tree="$(jq -r '.tree.sha' <<< "$event_merge_json")" - live_merge_tree="$(jq -r '.tree.sha' <<< "$live_merge_json")" - [[ "$event_merge_tree" == "$live_merge_tree" ]] \ - || fail_closed 'current merge tree differs from the triggering merge tree' - is_positive_integer "$RUN_ID" || fail_closed 'invalid workflow run ID' is_positive_integer "$RUN_ATTEMPT" || fail_closed 'invalid workflow run attempt' run_json="$(gh api \ diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 3cf16c21fab..3812107da15 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -147,19 +147,26 @@ New workflow versions merged into master then receive runner access without a separate SHA allowlist update. Dependabot leaves this first-party reference on master. -Runner selection calls `ci-routing.yml@master` on a GitHub-hosted runner. It -verifies authorization before allocating any AWS capacity and never checks out -or executes PR code. The author, event sender and rerun actor must each be both +Runner selection calls `ci-routing.yml@master`. Allowlisted maintainers use the +reserved `paperclip-ci-routing-x64` AWS fleet for this bounded metadata job; +other callers use GitHub-hosted capacity. The gate never checks out or executes +PR code. The author, event sender and rerun actor must each be both in `AWS_CI_TRUSTED_USER_IDS` and a current owner of `.github/**` in CODEOWNERS. The resolver and CODEOWNERS are read from the same protected master revision; PR edits cannot change the trusted owner list. Documentation-only owners, unsupported owner rules and failed identity lookups do not authorize AWS. -Repository, head and merge validation also apply. Missing or malformed settings +Repository, head and merge validation also apply. The gate validates this run's +immutable event merge, including its PR head and a base snapshot between the +event's base and the current branch tip. Normal base advancement and regeneration +of GitHub's live merge ref do not move an authorized run to GitHub-hosted CI. +Stacked synthetic bases must contain a validated base snapshot. Rewritten base +history, changed PR heads and unverifiable ancestry still deny AWS execution. +Missing or malformed settings and unverifiable authorization use GitHub-hosted runners. A maintainer rerunning an outside contributor's PR does not grant it AWS access. Authorized test jobs and the final `verify` and `e2e` aggregates use the selected -AWS fleet. The metadata authorization job remains hosted even for maintainers, -so stale admission snapshots cannot allocate an AWS runner before verification. +AWS fleet. A stale admission roster can allocate only the short metadata gate; +the fresh CODEOWNERS check still denies PR execution for a removed owner. The routing workflow also selects the CI scope. Middle PRs in a stack and PRs whose diff changes only release notes (`releases/v*.md` or @@ -339,10 +346,16 @@ explicit. When changing shard layout, update `.github/ci-task-matrix.json` and the full fallback matrices in `pr-trusted.yml`; the contract tests require them to agree. Add new cache-affecting environment settings to `globalEnv`. -The former reserved routing Fleet is no longer selected by this workflow. -Keep its existing authorization until older assigned jobs finish. A manual -dispatch of `ci-routing.yml` on master exercises the hosted metadata job without -launching the test matrix; its non-PR event returns the hosted test-runner output. +The `paperclip-ci-routing` runner group must allow +`paperclipai/paperclip/.github/workflows/ci-routing.yml@refs/heads/master`. +Keep `AWS_CI_ROUTING_USER_IDS_SOURCE` equal to the exact JSON value of +`AWS_CI_TRUSTED_USER_IDS` and `AWS_CI_ROUTING_USER_IDS` as its delimiter-wrapped +numeric IDs (for example, `|34892728|139239|`). Missing or mismatched admission +settings use GitHub-hosted capacity for the gate. Current CODEOWNERS remain +authoritative for PR execution regardless of these admission settings. +A maintainer's manual dispatch of `ci-routing.yml` on master exercises the AWS +metadata gate without launching the test matrix; its non-PR event returns the +hosted test-runner output. Keep the `.github/**` rule in `.github/CODEOWNERS` and the active master ruleset's code-owner review requirement enabled. This covers the caller, the trusted diff --git a/doc/ci-profiling.md b/doc/ci-profiling.md index 6ed462b77a2..9975cff7d3b 100644 --- a/doc/ci-profiling.md +++ b/doc/ci-profiling.md @@ -24,8 +24,13 @@ Ordinary PRs do not write timing artifacts or export traces. The exporter job is skipped for ordinary run names. Adding another label does not run the CI matrix. Profiling preserves the existing trusted AWS runner routing. Eligible PR jobs -use `paperclip-public-pr-x64`; the routing gate uses `ubuntu-latest` to verify -current CI CODEOWNERS before allocating any AWS capacity. +use `paperclip-public-pr-x64`. The synchronized maintainer admission roster +places the metadata gate on the reserved `paperclip-ci-routing-x64` AWS fleet. +The gate checks current CI CODEOWNERS before authorizing any PR code on AWS; +PR edits cannot grant access. A stale roster can allocate only the bounded +metadata gate. Normal base-branch advancement does not change an authorized +run's placement. See [Trusted PR Workflow](DEVELOPING.md#trusted-pr-workflow) +for the admission settings and immutable merge validation. The existing fail-closed GitHub-hosted fallback still applies if AWS routing cannot be authorized. The separate post-run Honeycomb exporter also uses `ubuntu-latest`; it is outside the measured PR suite. diff --git a/scripts/__tests__/e2e-shard.test.mjs b/scripts/__tests__/e2e-shard.test.mjs index 6b2cc7a6642..8a08d5427cd 100644 --- a/scripts/__tests__/e2e-shard.test.mjs +++ b/scripts/__tests__/e2e-shard.test.mjs @@ -216,11 +216,11 @@ test("the trusted PR workflow keeps a stable aggregate check named e2e over the } }); -test("runner authorization uses hosted capacity and executes metadata checks only", () => { +test("runner authorization uses reserved capacity and executes metadata checks only", () => { const workflow = readFileSync(routingWorkflow, "utf8"); const jobs = readWorkflowJobs(workflow.split(/^jobs:\n/m)[1]); assert.deepEqual([...jobs.keys()], ["gate"]); - assert.match(jobs.get("gate"), /^ {4}runs-on: ubuntu-latest$/m); + assert.match(jobs.get("gate"), /runs-on\/fleet=paperclip-ci-routing-x64\/env=public-ci/); assert.doesNotMatch(jobs.get("gate"), /uses:|actions\/checkout|secrets\./); assert.match(workflow, /^ {6}runner:\n {8}value: \$\{\{ jobs\.gate\.outputs\.runner \}\}$/m); assert.match(workflow, /^ {6}full_ci:\n {8}value: \$\{\{ jobs\.gate\.outputs\.full_ci \}\}$/m);