ci(evals): route trusted catalog authorization to AWS (#19301)

## Thinking Path

> - Paperclip helps people manage AI agents at work.
> - The chat catalog workflow checks each manual dispatch before
protected work.
> - Its credential-free authorize job waits for GitHub-hosted capacity.
> - A recent run waited about 16 minutes before that short gate could
start.
> - The existing public-CI runner group already allows this catalog
workflow on master.
> - This change places only the approved first manual master attempt on
that fleet.
> - Every other context retains hosted placement and the actual dispatch
guard.

## Linked Issues or Issue Description

Refs: #16042, #18067. The latter changes preparation after authorization
in a different workflow. This change covers only the chat catalog
authorize job.

**What happened?**

The authorize job requests ubuntu-latest even for the approved
main-branch operator. Hosted runner queue time delays all later work.
The current run remains untouched.

**Expected behavior**

Use existing AWS public-CI capacity only for the exact approved first
manual master context. Retain the hosted route for all other contexts.
The existing guard must still verify the actual API identity and both
actors after checkout.

**Steps to reproduce**

1. Dispatch Chat catalog TestOps from master as the approved operator.
2. Inspect authorize placement before a runner starts.
3. Compare the branch, actor, attempt and workflow rejection controls.

## What Changed

- Add repository name and ID, exact workflow ref, master ref, event,
actor ID, attempt and triggering-actor placement fences.
- Reuse the existing placement evaluator for 15 allocation cases.
- Keep the actual dispatch guard and exact-SHA checkout. Keep select and
all reporting jobs unchanged.

## Verification

- Local focused controls: 16 passed. The parent workflow failed all 15
new placement cases because it had no conditional route.
- Existing full routing and workflow suites: 192 passed on the final
patch.
- A temporary select-to-AWS mutation failed the bounded select control;
restored placement passed.
- actionlint for chat-catalog-testops.yml and git diff --check: passed.
- The first full local run hit sandbox EPERM on existing loopback HTTP
controls. That result is retained. The same unchanged suites were then
run with listener permission.
- No application build, SDK build, model call, eval dispatch, workflow
retry or fleet change was performed.
- Latest-head checks: all 55 succeeded. Required `ci / verify` and `ci /
e2e` passed; genuine Greptile review is 5/5 on this exact head with no
unresolved threads.

## Risks

This reuses the existing trusted-main fleet boundary. It is not a
privilege-free worker. The current instance profile has IMDS enabled
with required tokens and includes SSM GetParameter/GetParameters
capability. No IAM, runner-group or network policy changes are included.
The group already permits this exact catalog workflow on master.

Placement checks run before allocation. The actual API and both-actor
guard still runs after checkout. An unauthorized context selects hosted
capacity and must pass the unchanged guard before protected work. Select
still waits for hosted capacity and keeps its protected token boundary.
AWS capacity can also queue. This change does not promise a fixed
latency gain or move a queued job.

## Model Used

OpenAI gpt-6.1-sol with reasoning, code editing and tool use. The
runtime does not report its context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-11-05 12:32:22 -06:00
1 parent 088f85c5fe
commit cd3832d235
2 files changed
+46 -3

No files matched your search

+45 -2
View File
@@ -282,9 +282,9 @@ test("path scope skips full CI for a single-PR stack that only touches release n
}).full_ci, "false");
});
function routingPlacement(vars = {}, github = {}) {
function routingPlacement(vars = {}, github = {}, placementJobs = jobs) {
const source = "[34892728]";
const expression = jobs.match(/^ {4}runs-on: \$\{\{ (.+) \}\}$/m)?.[1];
const expression = placementJobs.match(/^ {4}runs-on: \$\{\{ (.+) \}\}$/m)?.[1];
assert.ok(expression, "routing expression is missing");
return vm.runInNewContext(expression, {
vars: {
@@ -330,3 +330,46 @@ for (const [name, vars, github, expected = "ubuntu-latest"] of placementCases) {
assert.equal(routingPlacement(vars, github), expected);
});
}
const catalogWorkflow = readFileSync(new URL("../../workflows/chat-catalog-testops.yml", import.meta.url), "utf8");
const catalogAuthorize = catalogWorkflow.split(" authorize:\n")[1].split("\n select:\n")[0];
const catalogContext = {
event_name: "workflow_dispatch",
workflow_ref: "paperclipai/paperclip/.github/workflows/chat-catalog-testops.yml@refs/heads/master",
actor: "dotta",
triggering_actor: "dotta",
run_attempt: "1",
};
const catalogPlacementCases = [
["approved first manual master attempt", {}, awsRunner],
["foreign repository", { repository: "other/paperclip" }],
["foreign repository ID", { repository_id: "999" }],
["untrusted branch", { ref: "refs/heads/other" }],
["tag", { ref: "refs/tags/master" }],
["pull request", { event_name: "pull_request" }],
["push", { event_name: "push" }],
["different workflow", { workflow_ref: "paperclipai/paperclip/.github/workflows/ci-routing.yml@refs/heads/master" }],
["branch workflow", { workflow_ref: "paperclipai/paperclip/.github/workflows/chat-catalog-testops.yml@refs/heads/other" }],
["unapproved actor", { actor_id: "999" }],
["partial actor ID", { actor_id: "4892728" }],
["different triggering actor", { triggering_actor: "other" }],
["missing triggering actor", { triggering_actor: "" }],
["rerun", { run_attempt: "2" }],
["missing attempt", { run_attempt: "" }],
];
for (const [name, overrides, expected = "ubuntu-latest"] of catalogPlacementCases) {
test(`catalog authorize placement before allocation: ${name}`, () => {
assert.equal(routingPlacement({}, { ...catalogContext, ...overrides }, catalogAuthorize), expected);
});
}
test("catalog placement retains downstream jobs and actual dispatch guard", () => {
assert.match(catalogWorkflow.split(" select:\n")[1].split("\n build:\n")[0], /^ {4}runs-on: ubuntu-latest$/m);
assert.match(catalogAuthorize, /ref: \$\{\{ github\.sha \}\}/);
assert.match(catalogAuthorize, /persist-credentials: false/);
assert.match(catalogAuthorize, /TESTOPS_ALLOWED_ACTOR_IDS: \$\{\{ vars\.RUNNER_E2E_ALLOWED_ACTOR_IDS \}\}/);
assert.match(catalogAuthorize, /run: node tests\/runner-e2e\/chat-completion-testops-cli\.mjs guard/);
assert.doesNotMatch(catalogAuthorize, /secrets\.|environment:/);
});
+1 -1
View File
@@ -33,7 +33,7 @@ env:
jobs:
authorize:
runs-on: ubuntu-latest
runs-on: ${{ github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.workflow_ref == 'paperclipai/paperclip/.github/workflows/chat-catalog-testops.yml@refs/heads/master' && github.ref == 'refs/heads/master' && github.event_name == 'workflow_dispatch' && github.actor_id == '34892728' && github.run_attempt == '1' && github.triggering_actor == github.actor && 'runs-on/fleet=paperclip-public-pr-x64/env=public-ci' || 'ubuntu-latest' }}
timeout-minutes: 5
outputs:
operation: ${{ steps.guard.outputs.operation }}