mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 12:33:05 +01:00
ci(evals): route trusted catalog authorization to AWS (#19301)
## Thinking Path > - Paperclip helps people manage AI agents at work. > - The chat catalog workflow checks each manual dispatch before protected work. > - Its credential-free authorize job waits for GitHub-hosted capacity. > - A recent run waited about 16 minutes before that short gate could start. > - The existing public-CI runner group already allows this catalog workflow on master. > - This change places only the approved first manual master attempt on that fleet. > - Every other context retains hosted placement and the actual dispatch guard. ## Linked Issues or Issue Description Refs: #16042, #18067. The latter changes preparation after authorization in a different workflow. This change covers only the chat catalog authorize job. **What happened?** The authorize job requests ubuntu-latest even for the approved main-branch operator. Hosted runner queue time delays all later work. The current run remains untouched. **Expected behavior** Use existing AWS public-CI capacity only for the exact approved first manual master context. Retain the hosted route for all other contexts. The existing guard must still verify the actual API identity and both actors after checkout. **Steps to reproduce** 1. Dispatch Chat catalog TestOps from master as the approved operator. 2. Inspect authorize placement before a runner starts. 3. Compare the branch, actor, attempt and workflow rejection controls. ## What Changed - Add repository name and ID, exact workflow ref, master ref, event, actor ID, attempt and triggering-actor placement fences. - Reuse the existing placement evaluator for 15 allocation cases. - Keep the actual dispatch guard and exact-SHA checkout. Keep select and all reporting jobs unchanged. ## Verification - Local focused controls: 16 passed. The parent workflow failed all 15 new placement cases because it had no conditional route. - Existing full routing and workflow suites: 192 passed on the final patch. - A temporary select-to-AWS mutation failed the bounded select control; restored placement passed. - actionlint for chat-catalog-testops.yml and git diff --check: passed. - The first full local run hit sandbox EPERM on existing loopback HTTP controls. That result is retained. The same unchanged suites were then run with listener permission. - No application build, SDK build, model call, eval dispatch, workflow retry or fleet change was performed. - Latest-head checks: all 55 succeeded. Required `ci / verify` and `ci / e2e` passed; genuine Greptile review is 5/5 on this exact head with no unresolved threads. ## Risks This reuses the existing trusted-main fleet boundary. It is not a privilege-free worker. The current instance profile has IMDS enabled with required tokens and includes SSM GetParameter/GetParameters capability. No IAM, runner-group or network policy changes are included. The group already permits this exact catalog workflow on master. Placement checks run before allocation. The actual API and both-actor guard still runs after checkout. An unauthorized context selects hosted capacity and must pass the unchanged guard before protected work. Select still waits for hosted capacity and keeps its protected token boundary. AWS capacity can also queue. This change does not promise a fixed latency gain or move a queued job. ## Model Used OpenAI gpt-6.1-sol with reasoning, code editing and tool use. The runtime does not report its context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
088f85c5fe
commit
cd3832d235
2 files changed
+46
-3
No files matched your search
@@ -282,9 +282,9 @@ test("path scope skips full CI for a single-PR stack that only touches release n
|
||||
}).full_ci, "false");
|
||||
});
|
||||
|
||||
function routingPlacement(vars = {}, github = {}) {
|
||||
function routingPlacement(vars = {}, github = {}, placementJobs = jobs) {
|
||||
const source = "[34892728]";
|
||||
const expression = jobs.match(/^ {4}runs-on: \$\{\{ (.+) \}\}$/m)?.[1];
|
||||
const expression = placementJobs.match(/^ {4}runs-on: \$\{\{ (.+) \}\}$/m)?.[1];
|
||||
assert.ok(expression, "routing expression is missing");
|
||||
return vm.runInNewContext(expression, {
|
||||
vars: {
|
||||
@@ -330,3 +330,46 @@ for (const [name, vars, github, expected = "ubuntu-latest"] of placementCases) {
|
||||
assert.equal(routingPlacement(vars, github), expected);
|
||||
});
|
||||
}
|
||||
|
||||
const catalogWorkflow = readFileSync(new URL("../../workflows/chat-catalog-testops.yml", import.meta.url), "utf8");
|
||||
const catalogAuthorize = catalogWorkflow.split(" authorize:\n")[1].split("\n select:\n")[0];
|
||||
const catalogContext = {
|
||||
event_name: "workflow_dispatch",
|
||||
workflow_ref: "paperclipai/paperclip/.github/workflows/chat-catalog-testops.yml@refs/heads/master",
|
||||
actor: "dotta",
|
||||
triggering_actor: "dotta",
|
||||
run_attempt: "1",
|
||||
};
|
||||
|
||||
const catalogPlacementCases = [
|
||||
["approved first manual master attempt", {}, awsRunner],
|
||||
["foreign repository", { repository: "other/paperclip" }],
|
||||
["foreign repository ID", { repository_id: "999" }],
|
||||
["untrusted branch", { ref: "refs/heads/other" }],
|
||||
["tag", { ref: "refs/tags/master" }],
|
||||
["pull request", { event_name: "pull_request" }],
|
||||
["push", { event_name: "push" }],
|
||||
["different workflow", { workflow_ref: "paperclipai/paperclip/.github/workflows/ci-routing.yml@refs/heads/master" }],
|
||||
["branch workflow", { workflow_ref: "paperclipai/paperclip/.github/workflows/chat-catalog-testops.yml@refs/heads/other" }],
|
||||
["unapproved actor", { actor_id: "999" }],
|
||||
["partial actor ID", { actor_id: "4892728" }],
|
||||
["different triggering actor", { triggering_actor: "other" }],
|
||||
["missing triggering actor", { triggering_actor: "" }],
|
||||
["rerun", { run_attempt: "2" }],
|
||||
["missing attempt", { run_attempt: "" }],
|
||||
];
|
||||
|
||||
for (const [name, overrides, expected = "ubuntu-latest"] of catalogPlacementCases) {
|
||||
test(`catalog authorize placement before allocation: ${name}`, () => {
|
||||
assert.equal(routingPlacement({}, { ...catalogContext, ...overrides }, catalogAuthorize), expected);
|
||||
});
|
||||
}
|
||||
|
||||
test("catalog placement retains downstream jobs and actual dispatch guard", () => {
|
||||
assert.match(catalogWorkflow.split(" select:\n")[1].split("\n build:\n")[0], /^ {4}runs-on: ubuntu-latest$/m);
|
||||
assert.match(catalogAuthorize, /ref: \$\{\{ github\.sha \}\}/);
|
||||
assert.match(catalogAuthorize, /persist-credentials: false/);
|
||||
assert.match(catalogAuthorize, /TESTOPS_ALLOWED_ACTOR_IDS: \$\{\{ vars\.RUNNER_E2E_ALLOWED_ACTOR_IDS \}\}/);
|
||||
assert.match(catalogAuthorize, /run: node tests\/runner-e2e\/chat-completion-testops-cli\.mjs guard/);
|
||||
assert.doesNotMatch(catalogAuthorize, /secrets\.|environment:/);
|
||||
});
|
||||
Reference in new issue
Block a user