mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-20 06:02:18 +02:00
ci: add protected routing workflow for reserved AWS capacity (#16039)
## Thinking Path > - Paperclip uses CI to verify changes before release. > - Trusted PR tests can run on isolated AWS workers. > - Their runner-selection job currently waits for GitHub-hosted capacity. > - A large hosted test queue therefore blocks admission to AWS. > - A dedicated routing workflow lets us reserve capacity for metadata validation. > - This PR adds that workflow without changing the active PR caller. ## Linked Issues or Issue Description **Issue type** CI reliability enhancement. **Where is the issue?** The Select trusted runner job in PR CI. **What's wrong?** The routing job competes with a large hosted test queue. Its sampled median wait was about 20 minutes. AWS tests cannot begin before this job finishes. **Suggested fix** Add a protected reusable routing workflow. Deploy its dedicated AWS pool, then switch the PR caller in a follow-up PR. No overlapping open PR or issue was found. ## What Changed - Add ci-routing.yml with the existing author, sender, rerun actor, repository, head, merge, and stack checks. - Target the dedicated paperclip-ci-routing-x64 Fleet. - Export runner and full_ci decisions to reusable workflow callers. - Add tests for metadata-only execution, reserved placement, and output forwarding. - Keep the active PR caller unchanged until the pool is deployed. ## Verification - The routing tests passed (30 tests), including 20 executed runner-selection cases and 8 executed stack-scope cases. - Existing routing allow/deny scenarios also passed with the extracted script (16 cases). - A live qualification job selected AWS and passed on the reserved routing pool in https://github.com/paperclipai/paperclip/actions/runs/38308312153/job/115007621128 while the hosted gate remained queued. - actionlint passed with the dedicated custom runner label declared in its local configuration. - git diff --check passed. - Full pnpm typecheck/test/build cannot run in this worktree because dependencies are not installed. The change is limited to workflow YAML and Node tests. - After merge, configure the restricted runner group, deploy the pool, and dispatch ci-routing.yml on master. The unused workflow does not affect current PR placement. ## Risks - The maintainer explicitly approved merging this rollout after current-head Greptile 5/5 and focused verification, while the hosted CI checks remain queued. - The dedicated Fleet must exist before the workflow can receive a runner. This PR introduces the workflow without switching active PR CI. - The group must authorize only ci-routing.yml on protected master and canonical repository ID 1170821064. - The routing workflow runs no checkout or PR code and has read-only permissions. The test admission checks are unchanged. ## Model Used - OpenAI GPT-6 via Codex, with reasoning, shell tools, and file editing. The exact runtime model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused routing and workflow tests; full pnpm commands are blocked by missing worktree dependencies) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] Required CI was checked; the maintainer explicitly approved an admin bypass for queued checks after clean review - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
e364b92c82
commit
8e7b5db822
2 files changed
+389
No files matched your search
@@ -0,0 +1,137 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { chmodSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import test from "node:test";
|
||||
|
||||
const workflow = readFileSync(new URL("../../workflows/ci-routing.yml", import.meta.url), "utf8");
|
||||
const jobs = workflow.split(/^jobs:\n/m)[1];
|
||||
|
||||
test("routing uses reserved capacity and runs no PR code or actions", () => {
|
||||
assert.deepEqual([...jobs.matchAll(/^ {2}([a-z_]+):$/gm)].map((match) => match[1]), ["gate"]);
|
||||
assert.match(jobs, /^ {4}runs-on: runs-on\/fleet=paperclip-ci-routing-x64\/env=public-ci$/m);
|
||||
assert.doesNotMatch(jobs, /uses:|actions\/checkout|secrets\./);
|
||||
assert.doesNotMatch(workflow, /^ {2}(?:actions|contents|pull-requests): write$/m);
|
||||
});
|
||||
|
||||
test("routing exports both placement and stack scope to callers", () => {
|
||||
assert.match(workflow, /^ {6}runner:\n {8}value: \$\{\{ jobs\.gate\.outputs\.runner \}\}$/m);
|
||||
assert.match(workflow, /^ {6}full_ci:\n {8}value: \$\{\{ jobs\.gate\.outputs\.full_ci \}\}$/m);
|
||||
assert.match(jobs, /^ {6}runner: \$\{\{ steps\.route\.outputs\.runner \}\}$/m);
|
||||
assert.match(jobs, /^ {6}full_ci: \$\{\{ steps\.scope\.outputs\.full_ci \}\}$/m);
|
||||
});
|
||||
|
||||
const githubStub = "#!/usr/bin/env bash\nset -Eeuo pipefail\n\nendpoint=\"${*: -1}\"\ncase \"$endpoint\" in\n */pulls/*)\n jq -cn \\\n --argjson author_id \"${TEST_PR_AUTHOR_ID:-34892728}\" \\\n --argjson repository_id \"${TEST_REPOSITORY_ID:-1170821064}\" \\\n --arg base_ref \"${TEST_BASE_REF:-master}\" \\\n --arg base_sha \"${TEST_PR_BASE_SHA:-$EVENT_BASE_SHA}\" \\\n --arg head_sha \"${TEST_PR_HEAD_SHA:-1111111111111111111111111111111111111111}\" \\\n --arg merge_sha \"${TEST_PR_MERGE_SHA:-4444444444444444444444444444444444444444}\" '{\n state: \"open\",\n user: {id: $author_id},\n base: {repo: {id: $repository_id}, ref: $base_ref, sha: $base_sha},\n head: {sha: $head_sha},\n merge_commit_sha: $merge_sha\n }'\n ;;\n */git/ref/heads/*)\n jq -cn \\\n --arg sha \"${TEST_LIVE_BASE_REF_SHA:-$EVENT_BASE_SHA}\" '{object: {sha: $sha}}'\n ;;\n */compare/*)\n jq -cn \\\n --arg merge_base_sha \"${TEST_BASE_COMPARISON_MERGE_BASE_SHA:-$EVENT_BASE_SHA}\" \\\n --arg status \"${TEST_BASE_COMPARISON_STATUS:-ahead}\" '{\n merge_base_commit: {sha: $merge_base_sha},\n status: $status\n }'\n ;;\n */git/commits/*)\n requested_sha=\"${endpoint##*/}\"\n live_merge_sha=\"${TEST_PR_MERGE_SHA:-4444444444444444444444444444444444444444}\"\n synthetic_base_merge_sha=\"${TEST_SYNTHETIC_BASE_MERGE_SHA:-}\"\n if [[ \"$requested_sha\" == \"$EVENT_MERGE_SHA\" ]]; then\n tree_sha=\"${TEST_EVENT_MERGE_TREE_SHA:-5555555555555555555555555555555555555555}\"\n base_sha=\"${TEST_EVENT_MERGE_BASE_SHA:-$EVENT_BASE_SHA}\"\n head_sha=\"${TEST_EVENT_MERGE_HEAD_SHA:-$EVENT_HEAD_SHA}\"\n elif [[ \"$requested_sha\" == \"$live_merge_sha\" ]]; then\n tree_sha=\"${TEST_LIVE_MERGE_TREE_SHA:-5555555555555555555555555555555555555555}\"\n base_sha=\"${TEST_LIVE_MERGE_BASE_SHA:-$EVENT_BASE_SHA}\"\n head_sha=\"${TEST_LIVE_MERGE_HEAD_SHA:-$EVENT_HEAD_SHA}\"\n elif [[ -n \"$synthetic_base_merge_sha\" && \"$requested_sha\" == \"$synthetic_base_merge_sha\" ]]; then\n tree_sha=\"${TEST_SYNTHETIC_BASE_MERGE_TREE_SHA:-8888888888888888888888888888888888888888}\"\n base_sha=\"${TEST_SYNTHETIC_MASTER_PARENT_SHA:-9999999999999999999999999999999999999999}\"\n head_sha=\"${TEST_SYNTHETIC_BASE_REF_PARENT_SHA:-${TEST_LIVE_BASE_REF_SHA:-$EVENT_BASE_SHA}}\"\n else\n echo \"unexpected merge commit: $requested_sha\" >&2\n exit 1\n fi\n jq -cn \\\n --arg sha \"$requested_sha\" \\\n --arg tree_sha \"$tree_sha\" \\\n --arg base_sha \"$base_sha\" \\\n --arg head_sha \"$head_sha\" '{\n sha: $sha,\n tree: {sha: $tree_sha},\n parents: [{sha: $base_sha}, {sha: $head_sha}]\n }'\n ;;\n */actions/runs/*)\n jq -cn \\\n --argjson actor_id \"${TEST_TRIGGERING_ACTOR_ID:-34892728}\" \\\n --argjson repository_id \"${TEST_REPOSITORY_ID:-1170821064}\" \\\n --arg event_name \"${TEST_EVENT_NAME:-pull_request}\" '{\n triggering_actor: {id: $actor_id},\n repository: {id: $repository_id},\n event: $event_name\n }'\n ;;\n *)\n echo \"unexpected gh endpoint: $endpoint\" >&2\n exit 1\n ;;\nesac";
|
||||
|
||||
function stepScript(name) {
|
||||
const start = workflow.indexOf(` - name: ${name}\n`);
|
||||
assert.ok(start >= 0, `missing step: ${name}`);
|
||||
const body = workflow.slice(start);
|
||||
const runStart = body.indexOf(" run: |\n");
|
||||
assert.ok(runStart >= 0, `missing script: ${name}`);
|
||||
const script = body.slice(runStart + " run: |\n".length).split("\n - name:")[0];
|
||||
return script.split("\n").map((line) => line.replace(/^ {10}/, "")).join("\n");
|
||||
}
|
||||
|
||||
function executeStep(name, overrides = {}) {
|
||||
const directory = mkdtempSync(path.join(tmpdir(), "paperclip-routing-test-"));
|
||||
const output = path.join(directory, "output");
|
||||
try {
|
||||
const gh = path.join(directory, "gh");
|
||||
writeFileSync(gh, githubStub);
|
||||
chmodSync(gh, 0o700);
|
||||
const result = spawnSync("bash", ["-c", stepScript(name)], {
|
||||
encoding: "utf8",
|
||||
env: {
|
||||
...process.env,
|
||||
PATH: `${directory}:${process.env.PATH}`,
|
||||
GITHUB_OUTPUT: output,
|
||||
AWS_CI_ENABLED: "true",
|
||||
TRUSTED_USER_IDS: "[34892728]",
|
||||
EVENT_NAME: "pull_request",
|
||||
EVENT_REPOSITORY: "paperclipai/paperclip",
|
||||
EVENT_REPOSITORY_ID: "1170821064",
|
||||
EVENT_ACTION: "synchronize",
|
||||
EVENT_PR_NUMBER: "12350",
|
||||
EVENT_PR_AUTHOR_ID: "34892728",
|
||||
EVENT_SENDER_ID: "34892728",
|
||||
EVENT_BASE_REPOSITORY_ID: "1170821064",
|
||||
EVENT_BASE_REF: "master",
|
||||
EVENT_BASE_SHA: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
|
||||
EVENT_HEAD_SHA: "1111111111111111111111111111111111111111",
|
||||
EVENT_MERGE_SHA: "2222222222222222222222222222222222222222",
|
||||
RUN_ID: "123456789",
|
||||
...overrides,
|
||||
},
|
||||
});
|
||||
assert.equal(result.status, 0, result.stderr);
|
||||
const lines = readFileSync(output, "utf8").trim().split("\n");
|
||||
assert.equal(lines.length, 1, "each step must emit exactly one decision");
|
||||
return Object.fromEntries(lines.map((line) => {
|
||||
const separator = line.indexOf("=");
|
||||
return [line.slice(0, separator), line.slice(separator + 1)];
|
||||
}));
|
||||
} finally {
|
||||
rmSync(directory, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
const awsRunner = "runs-on/fleet=paperclip-public-pr-x64/env=public-ci";
|
||||
const stackedBase = {
|
||||
EVENT_BASE_REF: "trusted-stack-base",
|
||||
TEST_BASE_REF: "trusted-stack-base",
|
||||
TEST_PR_BASE_SHA: "7777777777777777777777777777777777777777",
|
||||
TEST_LIVE_BASE_REF_SHA: "7777777777777777777777777777777777777777",
|
||||
TEST_SYNTHETIC_BASE_MERGE_SHA: "8888888888888888888888888888888888888888",
|
||||
TEST_EVENT_MERGE_BASE_SHA: "8888888888888888888888888888888888888888",
|
||||
TEST_LIVE_MERGE_BASE_SHA: "8888888888888888888888888888888888888888",
|
||||
};
|
||||
|
||||
const routeCases = [
|
||||
["allowlisted PR", {}, awsRunner],
|
||||
["validated stacked base", stackedBase, awsRunner],
|
||||
["lagging PR API base snapshot", { TEST_PR_BASE_SHA: "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" }, awsRunner],
|
||||
["disabled AWS", { AWS_CI_ENABLED: "false" }],
|
||||
["missing AWS variables", { AWS_CI_ENABLED: "", TRUSTED_USER_IDS: "" }],
|
||||
["unlisted author", { EVENT_PR_AUTHOR_ID: "99999999" }],
|
||||
["unlisted sender", { EVENT_SENDER_ID: "99999999" }],
|
||||
["unlisted rerun actor", { TEST_TRIGGERING_ACTOR_ID: "99999999" }],
|
||||
["missing base", { EVENT_BASE_REF: "" }],
|
||||
["wrong base repository", { EVENT_BASE_REPOSITORY_ID: "999999999" }],
|
||||
["replaced base ancestry", { TEST_BASE_COMPARISON_MERGE_BASE_SHA: "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" }],
|
||||
["stale head", { TEST_PR_HEAD_SHA: "3333333333333333333333333333333333333333" }],
|
||||
["changed merge head", { TEST_LIVE_MERGE_HEAD_SHA: "3333333333333333333333333333333333333333" }],
|
||||
["stacked base with wrong parent", { ...stackedBase, TEST_SYNTHETIC_BASE_REF_PARENT_SHA: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" }],
|
||||
["changed merge tree", { TEST_LIVE_MERGE_TREE_SHA: "6666666666666666666666666666666666666666" }],
|
||||
["wrong repository", { EVENT_REPOSITORY_ID: "999999999" }],
|
||||
["malformed allowlist", { TRUSTED_USER_IDS: '["34892728"]' }],
|
||||
["manual dispatch", { EVENT_NAME: "workflow_dispatch" }],
|
||||
["unsupported PR action", { EVENT_ACTION: "closed" }],
|
||||
["malformed commit SHA", { EVENT_HEAD_SHA: "invalid" }],
|
||||
];
|
||||
|
||||
for (const [name, env, expected = "ubuntu-latest"] of routeCases) {
|
||||
test(`runner decision: ${name}`, () => {
|
||||
assert.equal(executeStep("Validate PR identity and select runner", env).runner, expected);
|
||||
});
|
||||
}
|
||||
|
||||
const scopeCases = [
|
||||
["ordinary PR", null, "master", "true"],
|
||||
["top stack PR", { position: 11, size: 11, base: { ref: "master" } }, "stack-10", "true"],
|
||||
["lowest unmerged PR", { position: 1, size: 11, base: { ref: "master" } }, "master", "true"],
|
||||
["middle stack PR", { position: 6, size: 11, base: { ref: "master" } }, "stack-5", "false"],
|
||||
["malformed position", { position: "invalid", size: 11, base: { ref: "master" } }, "stack-5", "true"],
|
||||
["out-of-range position", { position: 12, size: 11, base: { ref: "master" } }, "stack-5", "true"],
|
||||
["missing stack base", { position: 6, size: 11 }, "stack-5", "true"],
|
||||
["empty metadata", {}, "master", "true"],
|
||||
];
|
||||
|
||||
for (const [name, stack, base, expected] of scopeCases) {
|
||||
test(`scope decision: ${name}`, () => {
|
||||
assert.equal(executeStep("Select stacked PR CI scope", {
|
||||
STACK_JSON: JSON.stringify(stack), PR_BASE_REF: base,
|
||||
}).full_ci, expected);
|
||||
});
|
||||
}
|
||||
Reference in new issue
Block a user