ci: route PR admission through reserved AWS capacity (#16042)

## Thinking Path

> - Paperclip uses CI to verify changes before release.
> - Trusted PR tests can run on isolated AWS workers.
> - Their runner-selection job currently waits for GitHub-hosted
capacity.
> - A large hosted test queue therefore blocks admission to AWS.
> - A dedicated routing workflow lets us reserve capacity for metadata
validation.
> - This PR moves the routing job to reserved AWS capacity after the
bootstrap workflow and pool are available.

## Linked Issues or Issue Description

**Issue type**

CI reliability enhancement.

**Where is the issue?**

The Select trusted runner job in PR CI.

**What's wrong?**

The routing job competes with a large hosted test queue. Its sampled
median wait was about 20 minutes. AWS tests cannot begin before this job
finishes.

**Suggested fix**

Add a protected reusable routing workflow. Deploy its dedicated AWS
pool, then switch the PR caller in a follow-up PR. No overlapping open
PR or issue was found.

## What Changed

- Call ci-routing.yml on protected master from pr-trusted.yml.
- Check the canonical repository and approved PR author/sender before
allocating reserved AWS metadata capacity. Unapproved PRs continue to
use hosted routing.
- Permit approved manual routing canaries on protected master.
- Use a validated roster snapshot for placement. Invalid JSON, missing
settings, stale snapshots, and unapproved identities select hosted
routing without a placement-time JSON parse.
- Preserve the full in-job identity, rerun actor, repository, head,
merge, and stack checks.
- Update stack-scope and placement regression tests and document the
pool and rollback.

## Verification

- The routing scripts and placement expression tests passed (44 tests),
routing-plus-shard tests passed (56 tests), and the existing
.github/scripts suite passed (381 tests).
- Existing routing allow/deny scenarios passed with the extracted script
(16 cases).
- actionlint passed with the dedicated custom runner label declared in
its local configuration.
- git diff --check passed.
- Full pnpm typecheck/test/build cannot run in this worktree because
dependencies are not installed. The change is limited to workflow YAML
and Node tests.
- The dedicated pool is deployed with two warm On-Demand workers and a
four-worker ceiling. The routing qualification passed: it started about
five seconds after workflow creation, selected AWS, and finished while
the hosted gate in the same PR remained queued.
- Bootstrap https://github.com/paperclipai/paperclip/pull/16039 is
merged. The deployed group now authorizes only ci-routing.yml on
protected master, and its master canary passed.

## Risks

- The maintainer explicitly approved a queued-CI admin bypass after
current-head Greptile 5/5 and focused verification.

- Activation requires the deployed routing Fleet and its group to
authorize the routing workflow on protected master. Keep the bootstrap
dependency explicit.
- The group must authorize only ci-routing.yml on protected master and
canonical repository ID 1170821064.
- The routing workflow runs no checkout or PR code and has read-only
permissions. The test admission checks are unchanged.

## Model Used

- OpenAI GPT-6 via Codex, with reasoning, shell tools, and file editing.
The exact runtime model ID and context window are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused routing and
workflow tests; full pnpm commands are blocked by missing worktree
dependencies)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] Required CI status checked; the maintainer explicitly approved
admin bypass for queued CI after clean current-head review and focused
verification
- [x] Greptile is 5/5 on 993fbe695a with
no open findings or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-12 16:47:57 -05:00
1 parent 2b8da4ab64
commit f5346c05b9
5 files changed
+99 -241

No files matched your search

+58 -1
View File
@@ -4,13 +4,20 @@ import { spawnSync } from "node:child_process";
import { tmpdir } from "node:os";
import path from "node:path";
import test from "node:test";
import vm from "node:vm";
const workflow = readFileSync(new URL("../../workflows/ci-routing.yml", import.meta.url), "utf8");
const jobs = workflow.split(/^jobs:\n/m)[1];
test("routing uses reserved capacity and runs no PR code or actions", () => {
assert.deepEqual([...jobs.matchAll(/^ {2}([a-z_]+):$/gm)].map((match) => match[1]), ["gate"]);
assert.match(jobs, /^ {4}runs-on: runs-on\/fleet=paperclip-ci-routing-x64\/env=public-ci$/m);
const placement = jobs.match(/^ {4}runs-on: (.+)$/m)?.[1];
assert.ok(placement, "routing placement must be defined");
assert.doesNotMatch(placement, /fromJSON/);
assert.ok(placement.includes("vars.AWS_CI_TRUSTED_USER_IDS == vars.AWS_CI_ROUTING_USER_IDS_SOURCE"));
assert.ok(placement.includes("github.event.pull_request.user.id"));
assert.ok(placement.includes("github.event.sender.id"));
assert.ok(placement.endsWith("&& 'runs-on/fleet=paperclip-ci-routing-x64/env=public-ci' || 'ubuntu-latest' }}"));
assert.doesNotMatch(jobs, /uses:|actions\/checkout|secrets\./);
assert.doesNotMatch(workflow, /^ {2}(?:actions|contents|pull-requests): write$/m);
});
@@ -135,3 +142,53 @@ for (const [name, stack, base, expected] of scopeCases) {
}).full_ci, expected);
});
}
function routingPlacement(vars = {}, github = {}) {
const source = "[34892728]";
const expression = jobs.match(/^ {4}runs-on: \$\{\{ (.+) \}\}$/m)?.[1];
assert.ok(expression, "routing expression is missing");
return vm.runInNewContext(expression, {
vars: {
AWS_CI_ENABLED: "true",
AWS_CI_TRUSTED_USER_IDS: source,
AWS_CI_ROUTING_USER_IDS_SOURCE: source,
AWS_CI_ROUTING_USER_IDS: "|34892728|",
...vars,
},
github: {
repository: "paperclipai/paperclip",
repository_id: "1170821064",
event_name: "pull_request",
ref: "refs/heads/master",
actor_id: "34892728",
event: { pull_request: { user: { id: 34892728 } }, sender: { id: 34892728 } },
...github,
},
contains: (search, item) => String(search).toLowerCase().includes(String(item).toLowerCase()),
format: (template, ...args) => template.replace(/\{(\d+)\}/g, (_, index) => String(args[Number(index)])),
}, { timeout: 1000 });
}
const placementCases = [
["approved PR", {}, {}, "runs-on/fleet=paperclip-ci-routing-x64/env=public-ci"],
["unapproved author", {}, { event: { pull_request: { user: { id: 999 } }, sender: { id: 34892728 } } }],
["unapproved sender", {}, { event: { pull_request: { user: { id: 34892728 } }, sender: { id: 999 } } }],
["partial ID cannot match", {}, { event: { pull_request: { user: { id: 4892728 } }, sender: { id: 34892728 } } }],
["invalid JSON keeps hosted fallback", { AWS_CI_TRUSTED_USER_IDS: "[34892728,]" }, {}],
["stale snapshot", { AWS_CI_TRUSTED_USER_IDS: "[34892728,999]" }, {}],
["missing snapshot", { AWS_CI_ROUTING_USER_IDS_SOURCE: "" }, {}],
["missing admission members", { AWS_CI_ROUTING_USER_IDS: "" }, {}],
["disabled routing", { AWS_CI_ENABLED: "false" }, {}],
["different repository", {}, { repository: "other/repo" }],
["different repository ID", {}, { repository_id: "999" }],
["approved manual master canary", {}, { event_name: "workflow_dispatch" }, "runs-on/fleet=paperclip-ci-routing-x64/env=public-ci"],
["unapproved manual actor", {}, { event_name: "workflow_dispatch", actor_id: "999" }],
["manual branch canary", {}, { event_name: "workflow_dispatch", ref: "refs/heads/other" }],
];
for (const [name, vars, github, expected = "ubuntu-latest"] of placementCases) {
test(`placement before allocation: ${name}`, () => {
assert.equal(routingPlacement(vars, github), expected);
});
}