Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-31 06:04:11 +01:00
Code Issues Packages Projects Releases Wiki Activity

perf(ci): select affected suites and cache task results (#18214)

Browse Source 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Pull request CI checks the server, UI, packages, Runner, and release
paths.
> - An ordinary PR starts 48 jobs even when its diff affects only a
small part of the repository.
> - Repeated pushes also repeat successful tests with unchanged inputs.
> - This PR selects affected suites before runner allocation and caches
deterministic task results with Turbo.
> - Required checks validate the plan. Unknown inputs and failed
comparisons keep full verification.
> - The benefit is fewer runner jobs and less repeated work, with full
post-merge verification retained.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The GitHub Actions PR workflow repeats unrelated suites for each push.

**Subsystem affected**

Cross-cutting CI routing, test execution, and required-check
aggregation.

**Current behavior**

An ordinary UI component change starts the complete 48-job PR workflow.
Existing dependency and compiler caches do not reuse successful test
results.

**Proposed behavior**

A trusted planner reads the tested merge diff and task inputs. A UI
component change such as `ui/src/components/Sidebar.tsx` selects 18
jobs, including build and browser coverage. Successful general,
serialized, and standalone Rust tasks can reuse unchanged results across
pushes in the same PR.

**Reason and benefit**

Reduce unrelated jobs and repeated test execution. The 48-to-18
comparison measures planned job count. Remote wall-clock savings have
not yet been measured.

**Breaking changes**

No application API changes. Required `verify` and `e2e` names remain
stable. `verify` also checks serialized suites and the canary dry run.

## What Changed

- Added explicit task inputs and a pinned Turbo version. An execution
adapter calls existing package scripts without adding script aliases.
- Added trusted diff planning and dynamic matrices with complete
fallback matrices.
- Added PR-scoped task result caches. External-state and
revision-stamped checks remain uncached.
- Added and created the `full-ci` override label, plus repository modes
`enabled`, `shadow`, and `full`. Forced and shadow runs bypass result
caches.
- Added candidate-workflow qualification, planner and aggregate tests,
and a real Turbo cache fixture.
- Kept the repository-wide CLI guidance guard in the uncached typecheck
lane so UI-only diffs retain this security check.
- Documented rollout controls, input maintenance, and local inspection
commands.

## Verification

- Passed 537 focused CI policy, routing, matrix, cache, and shard tests,
plus 146 existing release workflow tests.
- Passed actionlint for all four changed or added workflows and `git
diff --check`.
- Passed `pnpm -r typecheck` and `pnpm build` on the implementation
base.
- Checked real repository task plans for existing general-test and
native Rust commands. A real server partition passed, then restored from
cache: Turbo task time was 20.022 seconds for execution and 151 ms for
the cache hit, excluding wrapper/setup overhead.
- The real Turbo fixture exercises the execution adapter and proves
source, argument, toolchain-scope, and environment invalidation, result
restoration, and retry of failed tasks.
- Stopped the serial local `pnpm test:run` after over 30 minutes in
favor of the complete parallel GitHub suite. A local parallel
database-lane attempt had PostgreSQL startup errors and timeouts. The
direct migration fixture passed, then all 273 database tests passed with
one local worker. This recovery does not establish the exact cause of
the earlier startup failures. The full GitHub PR suite and candidate
qualification passed on commit
`0bf07ae071c9fe7903edc604d44b5a175727fed0`. Greptile is 5/5 with no
unresolved review threads.
- All 40 CLI guidance safety tests passed with the exact uncached
workflow command.
- Verified that master rulesets require only `ci / verify` and `ci /
e2e`; both names remain unchanged.
- Trusted workflows load from master. The added GitHub-hosted
qualification workflow exercises this candidate planner before merge.

## Risks

- Changes to global inputs, including task definitions and execution
helpers, force execution and disable cache restore/save for the full PR
diff. Missing a cross-package filesystem read could omit a suite. Inputs
include known server reads of UI files and shared tests of UI assets.
Unknown paths, malformed or truncated diffs, and definition download
failures select all suites.
- Task result reuse requires complete inputs. Keys include Git source
files, scripts, resolved dependencies, environment, toolchain, checkout
path, and unclassified source files. Generated build artifacts are
excluded from input hashes. Registry smoke, browser, and
revision-stamped tasks remain uncached.
- Maintainers must update task inputs when suites start reading new
paths. `shadow` reports selection while running all work. `full` and
`full-ci` disable selection and result reuse.
- The implementation becomes active after merge. Full post-merge
verification remains unchanged.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection, tool
use, and code execution. The session does not expose a verifiable exact
runtime model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-29 16:56:28 -05:00
1 parent 61bd4ac691
commit f5dcc505c0
20 files changed
+1400 -284
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/actions/run-ci-task/action.yml
+38
View File
@@ -0,0 +1,38 @@
name: Run incremental CI task
description: Restore PR-scoped Turbo results and run a task with its complete input hash.
inputs:
task:
required: true
description: Root task from turbo.json.
args:
default: '[]'
description: JSON array of command arguments.
partition:
default: '0'
description: Matrix partition for independent cache uploads.
force:
default: 'false'
description: Execute tasks even when cached, including prerequisite builds.
runs:
using: composite
steps:
- name: Restore task results
if: inputs.force != 'true'
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
with:
path: .turbo/cache
key: turbo-v1-${{ runner.os }}-${{ runner.arch }}-${{ github.job }}-${{ inputs.partition }}-${{ github.run_id }}-${{ github.run_attempt }}
restore-keys: turbo-v1-${{ runner.os }}-${{ runner.arch }}-${{ github.job }}-${{ inputs.partition }}-
- name: Run task
shell: bash
env:
CI_TASK: ${{ inputs.task }}
CI_TASK_ARGS: ${{ inputs.args }}
CI_FORCE_TASKS: ${{ inputs.force }}
run: node scripts/ci/run-task.mjs
- name: Save successful task results
if: success() && inputs.force != 'true'
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
with:
path: .turbo/cache
key: turbo-v1-${{ runner.os }}-${{ runner.arch }}-${{ github.job }}-${{ inputs.partition }}-${{ github.run_id }}-${{ github.run_attempt }}
.github/ci-task-matrix.json
+283
View File
@@ -0,0 +1,283 @@
{
"typecheck_release_registry": [
{
"task": "ci:typecheck"
}
],
"general_tests": [
{
"group": "general-server-without-chat",
"group_label": "server (1/12)",
"shard_index": 0,
"shard_count": 12,
"task": "ci:server"
},
{
"group": "general-server-without-chat",
"group_label": "server (2/12)",
"shard_index": 1,
"shard_count": 12,
"task": "ci:server"
},
{
"group": "general-server-without-chat",
"group_label": "server (3/12)",
"shard_index": 2,
"shard_count": 12,
"task": "ci:server"
},
{
"group": "general-server-without-chat",
"group_label": "server (4/12)",
"shard_index": 3,
"shard_count": 12,
"task": "ci:server"
},
{
"group": "general-server-without-chat",
"group_label": "server (5/12)",
"shard_index": 4,
"shard_count": 12,
"task": "ci:server"
},
{
"group": "general-server-without-chat",
"group_label": "server (6/12)",
"shard_index": 5,
"shard_count": 12,
"task": "ci:server"
},
{
"group": "general-server-without-chat",
"group_label": "server (7/12)",
"shard_index": 6,
"shard_count": 12,
"task": "ci:server"
},
{
"group": "general-server-without-chat",
"group_label": "server (8/12)",
"shard_index": 7,
"shard_count": 12,
"task": "ci:server"
},
{
"group": "general-server-without-chat",
"group_label": "server (9/12)",
"shard_index": 8,
"shard_count": 12,
"task": "ci:server"
},
{
"group": "general-server-without-chat",
"group_label": "server (10/12)",
"shard_index": 9,
"shard_count": 12,
"task": "ci:server"
},
{
"group": "general-server-without-chat",
"group_label": "server (11/12)",
"shard_index": 10,
"shard_count": 12,
"task": "ci:server"
},
{
"group": "general-server-without-chat",
"group_label": "server (12/12)",
"shard_index": 11,
"shard_count": 12,
"task": "ci:server"
},
{
"group": "general-chat",
"group_label": "chat (1/3)",
"shard_index": 0,
"shard_count": 3,
"task": "ci:chat"
},
{
"group": "general-chat",
"group_label": "chat (2/3)",
"shard_index": 1,
"shard_count": 3,
"task": "ci:chat"
},
{
"group": "general-chat",
"group_label": "chat (3/3)",
"shard_index": 2,
"shard_count": 3,
"task": "ci:chat"
},
{
"group": "general-workspaces-a",
"group_label": "workspaces-a (1/2)",
"shard_index": 0,
"shard_count": 2,
"task": "ci:workspaces-a"
},
{
"group": "general-workspaces-a",
"group_label": "workspaces-a (2/2)",
"shard_index": 1,
"shard_count": 2,
"task": "ci:workspaces-a"
},
{
"group": "general-workspaces-b",
"group_label": "workspaces-b",
"task": "ci:workspaces-b"
},
{
"group": "general-workspaces-c",
"group_label": "workspaces-c",
"task": "ci:db"
}
],
"docker_context_integrity": [
{
"task": "ci:docker"
}
],
"verify_paperclip_runner": [
{
"lane_label": "static checks",
"command": "check:static",
"task": "ci:runner-static"
},
{
"lane_label": "rust",
"command": "check:runner",
"release_crate": true,
"task": "ci:runner-rust"
},
{
"lane_label": "vitest 1/2",
"command": "test:typescript:vitest --shard=1/2",
"task": "ci:runner-vitest"
},
{
"lane_label": "vitest 2/2",
"command": "test:typescript:vitest --shard=2/2",
"release_crate": true,
"task": "ci:runner-vitest"
}
],
"build": [
{
"task": "ci:build"
}
],
"verify_serialized_server": [
{
"shard_index": 0,
"shard_count": 9,
"shard_label": "1/9",
"task": "ci:serialized"
},
{
"shard_index": 1,
"shard_count": 9,
"shard_label": "2/9",
"task": "ci:serialized"
},
{
"shard_index": 2,
"shard_count": 9,
"shard_label": "3/9",
"task": "ci:serialized"
},
{
"shard_index": 3,
"shard_count": 9,
"shard_label": "4/9",
"task": "ci:serialized"
},
{
"shard_index": 4,
"shard_count": 9,
"shard_label": "5/9",
"task": "ci:serialized"
},
{
"shard_index": 5,
"shard_count": 9,
"shard_label": "6/9",
"task": "ci:serialized"
},
{
"shard_index": 6,
"shard_count": 9,
"shard_label": "7/9",
"task": "ci:serialized"
},
{
"shard_index": 7,
"shard_count": 9,
"shard_label": "8/9",
"task": "ci:serialized"
},
{
"shard_index": 8,
"shard_count": 9,
"shard_label": "9/9",
"task": "ci:serialized"
}
],
"canary_dry_run": [
{
"task": "ci:canary"
}
],
"e2e_shards": [
{
"shard_index": 0,
"shard_count": 8,
"shard_label": "1/8",
"task": "ci:e2e"
},
{
"shard_index": 1,
"shard_count": 8,
"shard_label": "2/8",
"task": "ci:e2e"
},
{
"shard_index": 2,
"shard_count": 8,
"shard_label": "3/8",
"task": "ci:e2e"
},
{
"shard_index": 3,
"shard_count": 8,
"shard_label": "4/8",
"task": "ci:e2e"
},
{
"shard_index": 4,
"shard_count": 8,
"shard_label": "5/8",
"task": "ci:e2e"
},
{
"shard_index": 5,
"shard_count": 8,
"shard_label": "6/8",
"task": "ci:e2e"
},
{
"shard_index": 6,
"shard_count": 8,
"shard_label": "7/8",
"task": "ci:e2e"
},
{
"shard_index": 7,
"shard_count": 8,
"shard_label": "8/8",
"task": "ci:e2e"
}
]
}
.github/scripts/ci-plan.mjs
+106
View File
@@ -0,0 +1,106 @@
import { appendFileSync, readFileSync, realpathSync } from "node:fs";
import { execFileSync } from "node:child_process";
import { matchesGlob } from "node:path";
import { fileURLToPath } from "node:url";
const releaseNote = /^releases\/(beta\/)?v[^/]+\.md$/;
const validPath = (file) => typeof file === "string" && file.length > 0
&& !file.startsWith("/") && !file.split("/").includes("..") && !file.includes("\0");
function validateGlobs(globs) {
if (!Array.isArray(globs) || globs.some((glob) => !validPath(glob) || /[!{}[\]?$\\]/.test(glob))) {
throw new Error("Unsupported task input pattern");
}
}
export function comparisonPaths(comparison) {
if (!Array.isArray(comparison?.files) || comparison.files.length === 0 || comparison.files.length >= 300) {
throw new Error("Missing, empty, or possibly truncated comparison");
}
return comparison.files.flatMap((file) => {
if (!validPath(file.filename) || (file.status === "renamed" && !validPath(file.previous_filename))
|| (file.previous_filename !== undefined && !validPath(file.previous_filename))) {
throw new Error("Malformed changed file");
}
return file.previous_filename ? [file.previous_filename, file.filename] : [file.filename];
});
}
export function planTasks({ config, lanes, paths, mode = "enabled", forceReason = null }) {
if (!["enabled", "shadow", "full"].includes(mode)) throw new Error("Invalid CI mode");
validateGlobs(config.globalDependencies);
const tasks = config.tasks;
const inputs = new Map();
function taskInputs(name, parents = []) {
if (inputs.has(name)) return inputs.get(name);
if (parents.includes(name)) throw new Error("Cyclic task graph");
const task = tasks[`//#${name}`];
if (!task) throw new Error(`Missing task: ${name}`);
validateGlobs(task.inputs);
const all = [...task.inputs];
for (const dependency of task.dependsOn ?? []) {
if (!dependency.startsWith("//#")) throw new Error("Unsupported task dependency");
all.push(...taskInputs(dependency.slice(3), [...parents, name]));
}
inputs.set(name, [...new Set(all)]);
return inputs.get(name);
}
for (const entries of Object.values(lanes)) {
if (!Array.isArray(entries) || !entries.length) throw new Error("Empty CI lane");
for (const entry of entries) taskInputs(entry.task);
}
const changed = [...new Set(paths)];
if (!changed.length || changed.some((file) => !validPath(file))) forceReason ??= "Missing or invalid diff";
const matches = (file, globs) => globs.some((glob) => matchesGlob(file, glob));
const globalChanges = changed.filter((file) => matches(file, config.globalDependencies));
if (globalChanges.length) forceReason ??= `Global CI input changed: ${globalChanges[0]}`;
const unknown = changed.filter((file) => !releaseNote.test(file)
&& !matches(file, config.globalDependencies) && ![...inputs.values()].some((globs) => matches(file, globs)));
if (unknown.length) forceReason ??= `Unclassified path: ${unknown[0]}`;
if (mode === "full") forceReason ??= "Full CI requested";
const reasons = Object.fromEntries([...inputs].map(([task, globs]) => [task,
forceReason ? [forceReason] : [...new Set([...globalChanges, ...changed.filter((file) => matches(file, globs))])],
]));
const jobs = Object.fromEntries(Object.entries(lanes).map(([job, entries]) => {
const affected = entries.filter((entry) => reasons[entry.task].length > 0);
const selected = mode === "shadow" ? entries : affected;
return [job, { run: selected.length > 0, matrix: { include: selected.length ? selected : entries },
selected: selected.length, proposed: affected.length, total: entries.length }];
}));
return { version: 1, mode, forceReason, changed, reasons, jobs };
}
export function planSummary(plan) {
const rows = Object.entries(plan.jobs).map(([job, lane]) =>
`| ${job} | ${lane.proposed}/${lane.total} | ${lane.selected}/${lane.total} |`);
return ["### Incremental CI", "", `Mode: **${plan.mode}**${plan.forceReason ? ` — ${plan.forceReason}` : ""}`,
"", "| Suite | Affected jobs | Scheduled jobs |", "| --- | ---: | ---: |", ...rows,
"", "<details><summary>Selection reasons</summary>", "", "```json", JSON.stringify(plan.reasons, null, 2),
"```", "</details>", ""].join("\n");
}
if (process.argv[1] && realpathSync(process.argv[1]) === fileURLToPath(import.meta.url)) {
try {
const config = JSON.parse(readFileSync(process.env.CI_TASK_CONFIG ?? "turbo.json", "utf8"));
const lanes = JSON.parse(readFileSync(process.env.CI_TASK_MATRIX ?? ".github/ci-task-matrix.json", "utf8"));
let paths = [], forceReason = process.env.CI_FORCE_REASON || null;
try {
if (process.env.CI_COMPARISON) {
paths = comparisonPaths(JSON.parse(readFileSync(process.env.CI_COMPARISON, "utf8")));
} else {
const [base, head = "HEAD"] = process.argv.slice(2);
if (!base) throw new Error("Usage: ci-plan.mjs <base> [head]");
paths = execFileSync("git", ["diff", "--no-renames", "--name-only", "-z", `${base}...${head}`, "--"],
{ encoding: "utf8" }).split("\0").filter(Boolean);
}
} catch (error) { forceReason ??= `Diff unavailable: ${error.message}`; }
const plan = planTasks({ config, lanes, paths, mode: process.env.CI_INCREMENTAL_MODE || "enabled", forceReason });
if (process.env.GITHUB_OUTPUT) appendFileSync(process.env.GITHUB_OUTPUT, `plan=${JSON.stringify(plan)}\n`);
else console.log(JSON.stringify(plan, null, 2));
if (process.env.GITHUB_STEP_SUMMARY) appendFileSync(process.env.GITHUB_STEP_SUMMARY, planSummary(plan));
} catch (error) {
console.error(`Incremental CI unavailable; selecting full CI: ${error.message}`);
if (process.env.GITHUB_OUTPUT) appendFileSync(process.env.GITHUB_OUTPUT, "plan=\n");
else process.exitCode = 1;
}
}
.github/scripts/tests/ci-plan.test.mjs
+221
View File
@@ -0,0 +1,221 @@
import assert from "node:assert/strict";
import { chmodSync, mkdtempSync, mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { execFileSync, spawnSync } from "node:child_process";
import { tmpdir } from "node:os";
import path from "node:path";
import test from "node:test";
import { comparisonPaths, planTasks } from "../ci-plan.mjs";
const root = new URL("../../../", import.meta.url);
const config = JSON.parse(readFileSync(new URL("turbo.json", root), "utf8"));
const lanes = JSON.parse(readFileSync(new URL(".github/ci-task-matrix.json", root), "utf8"));
const plan = (paths, options = {}) => planTasks({ config, lanes, paths, ...options });
const selected = (result, job) => result.jobs[job].run;
const selectedTasks = (result) => new Set(Object.values(result.jobs).filter((job) => job.run)
.flatMap((job) => job.matrix.include.map((entry) => entry.task)));
test("trusted workflow fallbacks preserve every declared partition and command", () => {
const workflow = readFileSync(new URL(".github/workflows/pr-trusted.yml", root), "utf8");
for (const [job, entries] of Object.entries(lanes)) {
const block = workflow.split(`\n ${job}:\n`)[1].split(/\n [a-z0-9_]+:\n/)[0];
assert.ok(block.includes(`fromJSON(needs.gate.outputs.plan).jobs.${job}.run`), job);
if (!block.includes(" matrix:")) { assert.equal(entries.length, 1); continue; }
const fallback = block.match(/matrix:.*\|\| '(.*)'\) \}\}/)?.[1];
assert.ok(fallback, `${job}: missing full fallback`);
assert.deepEqual(JSON.parse(fallback), { include: entries }, job);
}
});
test("a UI component retains browser and build coverage without unrelated server/Rust jobs", () => {
const result = plan(["ui/src/components/Sidebar.tsx"]);
assert.equal(selected(result, "build"), true);
assert.equal(selected(result, "e2e_shards"), true);
assert.equal(selected(result, "verify_serialized_server"), false);
assert.deepEqual([...selectedTasks(result)].filter((task) => /server|chat|runner/.test(task)), []);
assert.equal(result.jobs.general_tests.selected, 2);
});
test("source inspected across package boundaries selects its consumers", () => {
for (const file of ["ui/src/lib/issue-execution-policy.ts", "ui/src/index.css", "ui/src/pages/CompanyExport.tsx",
"ui/src/components/MarkdownBody.tsx"]) {
const tasks = selectedTasks(plan([file]));
assert.ok(tasks.has("ci:server"), file);
assert.ok(tasks.has("ci:runner-vitest"), file);
}
assert.ok(selectedTasks(plan(["ui/public/app-logos/example.svg"])).has("ci:workspaces-b"));
});
test("repository-wide guidance guards run uncached for every scanned UI file type", () => {
const source = readFileSync(new URL("server/src/__tests__/cli-invocation-safety.test.ts", root), "utf8");
const extensions = [...source.split("const SCAN_EXTENSIONS = new Set([")[1].split("]);")[0].matchAll(/"(\.[a-z]+)"/g)]
.map((match) => match[1]);
assert.ok(extensions.length > 0);
for (const extension of extensions) {
for (const folder of ["ui/src/components", "ui/src/pages", "ui/src/hooks", "ui/public"]) {
assert.ok(plan([`${folder}/guidance${extension}`]).jobs.typecheck_release_registry.run);
}
}
assert.equal(config.tasks["//#ci:typecheck"].cache, false);
const workflow = readFileSync(new URL(".github/workflows/pr-trusted.yml", root), "utf8");
const job = workflow.split("\n typecheck_release_registry:\n")[1].split("\n general_tests:\n")[0];
assert.match(job, /- name: Verify repository-wide CLI guidance safety\n run: pnpm exec vitest run --project @paperclipai\/server server\/src\/__tests__\/cli-invocation-safety.test.ts/);
assert.ok(!job.includes("uses: ./.github/actions/run-ci-task"));
});
test("shared contracts, runtime prompts, protocol inputs and fixtures reach consumers", () => {
const shared = selectedTasks(plan(["packages/shared/src/types/issue.ts"]));
for (const task of ["ci:server", "ci:db", "ci:workspaces-a", "ci:workspaces-b", "ci:serialized", "ci:e2e"]) {
assert.ok(shared.has(task), task);
}
assert.ok(selectedTasks(plan(["server/src/onboarding-assets/default/AGENTS.md"])).has("ci:runner-vitest"));
assert.ok(selectedTasks(plan(["packages/paperclip-runner/protocol/schema.json"])).has("ci:runner-rust"));
for (const file of ["tests/runner-e2e/fixtures/prompt.md", "scripts/run-vitest-stable.mjs", "pnpm-lock.yaml"]) {
assert.ok(Object.values(plan([file]).jobs).every((job) => job.selected === job.total), file);
}
});
test("renames out of source and deletions cannot masquerade as release notes", () => {
const paths = comparisonPaths({ files: [
{ filename: "releases/v2026.1029.0.md", previous_filename: "packages/shared/src/constants.ts", status: "renamed" },
{ filename: "server/src/deleted.ts", status: "removed" },
] });
assert.ok(selectedTasks(plan(paths)).has("ci:server"));
assert.ok(selectedTasks(plan(paths)).has("ci:db"));
});
test("unclassified paths and uncertain comparisons select full verification", () => {
for (const paths of [[], ["new-root/file.ts"], ["../escaped"]]) {
const result = plan(paths);
assert.ok(result.forceReason);
assert.ok(Object.values(result.jobs).every((job) => job.selected === job.total));
}
for (const files of [null, [], [{ status: "modified" }], [{ filename: "a", status: "renamed" }],
Array.from({ length: 300 }, () => ({ filename: "ui/src/test.ts" }))]) {
assert.throws(() => comparisonPaths({ files }));
}
});
test("shadow mode explains savings but runs every task; force overrides selection", () => {
const shadow = plan(["ui/src/components/Sidebar.tsx"], { mode: "shadow" });
assert.equal(shadow.jobs.verify_serialized_server.proposed, 0);
assert.equal(shadow.jobs.verify_serialized_server.selected, 9);
for (const options of [{ mode: "full" }, { forceReason: "Stack" }]) {
assert.ok(Object.values(plan(["ui/src/components/Sidebar.tsx"], options).jobs)
.every((job) => job.selected === job.total));
}
});
test("cache-affecting definition changes force execution across successive PR pushes", () => {
for (const file of ["turbo.json", "package.json", "packages/db/package.json", "scripts/ci/run-task.mjs",
".github/actions/run-ci-task/action.yml"]) {
for (const paths of [[file], [file, "packages/db/src/schema/issues.ts"]]) {
const result = plan(paths);
assert.match(result.forceReason, /Global CI input changed/);
assert.ok(Object.values(result.jobs).every((job) => job.selected === job.total));
}
}
const workflow = readFileSync(new URL(".github/workflows/pr-trusted.yml", root), "utf8");
assert.equal((workflow.match(/force:.*fromJSON\(needs.gate.outputs.plan\).forceReason != null/g) ?? []).length, 3);
const action = readFileSync(new URL(".github/actions/run-ci-task/action.yml", root), "utf8");
assert.match(action, /CI_FORCE_TASKS: \$\{\{ inputs.force \}\}/);
assert.equal((action.match(/if:.*inputs.force != 'true'/g) ?? []).length, 2);
});
test("task dependencies propagate input changes and unsupported graph syntax fails closed", () => {
const fixture = { globalDependencies: [], tasks: {
"//#build": { inputs: ["lib/**"] }, "//#test": { inputs: ["app/**"], dependsOn: ["//#build"] },
} };
assert.equal(planTasks({ config: fixture, lanes: { tests: [{ task: "test" }] }, paths: ["lib/index.ts"] }).jobs.tests.run, true);
fixture.tasks["//#build"].dependsOn = ["//#test"];
assert.throws(() => planTasks({ config: fixture, lanes: { tests: [{ task: "test" }] }, paths: ["app/index.ts"] }), /Cyclic/);
assert.throws(() => plan(["ui/src/a.ts"], { mode: "typo" }), /Invalid CI mode/);
});
test("local planner reads a real Git rename and handles a missing revision conservatively", () => {
const directory = mkdtempSync(path.join(tmpdir(), "ci-plan-"));
const git = (...args) => execFileSync("git", args, { cwd: directory, encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] });
try {
git("init", "-q"); git("config", "user.name", "CI Test"); git("config", "user.email", "ci@example.invalid");
mkdirSync(path.join(directory, "packages/shared/src"), { recursive: true });
writeFileSync(path.join(directory, "packages/shared/src/input.ts"), "original\n");
git("add", "."); git("commit", "-qm", "base");
const base = git("rev-parse", "HEAD").trim();
mkdirSync(path.join(directory, "releases"));
git("mv", "packages/shared/src/input.ts", "releases/v1.md"); git("commit", "-qm", "rename");
for (const revision of [base, "missing-revision"]) {
const result = spawnSync(process.execPath, [new URL("../ci-plan.mjs", import.meta.url).pathname, revision], {
cwd: directory, encoding: "utf8", env: { ...process.env, GITHUB_OUTPUT: "", GITHUB_STEP_SUMMARY: "",
CI_TASK_CONFIG: new URL("turbo.json", root).pathname, CI_TASK_MATRIX: new URL(".github/ci-task-matrix.json", root).pathname },
});
assert.equal(result.status, 0, result.stderr);
const resultPlan = JSON.parse(result.stdout);
assert.ok(selectedTasks(resultPlan).has("ci:server"));
if (revision !== base) assert.match(resultPlan.forceReason, /Diff unavailable/);
}
} finally { rmSync(directory, { recursive: true, force: true }); }
});
test("required aggregates reject omitted, failed and cancelled selected work", () => {
const workflow = readFileSync(new URL(".github/workflows/pr-trusted.yml", root), "utf8");
for (const id of ["verify", "e2e"]) {
const block = workflow.split(`\n ${id}:\n`)[1].split(/\n [a-z0-9_]+:\n/)[0];
const code = block.split(" node --input-type=module <<'NODE'\n")[1].split("\n NODE")[0]
.split("\n").map((line) => line.slice(10)).join("\n");
const names = /needs: \[([^\]]+)\]/.exec(block)[1].split(", ");
const resultPlan = plan(["ui/src/components/Sidebar.tsx"]);
const results = Object.fromEntries(names.map((name) => [name, { result:
name === "gate" || name === "policy" || resultPlan.jobs[name]?.run ? "success" : "skipped" }]));
const run = () => spawnSync(process.execPath, ["--input-type=module", "-e", code], { encoding: "utf8",
env: { ...process.env, FULL_CI: "true", CI_PLAN: JSON.stringify(resultPlan), CI_RESULTS: JSON.stringify(results) } });
assert.equal(run().status, 0);
for (const name of names) {
const previous = results[name].result;
for (const bad of previous === "success" ? ["failure", "cancelled", "skipped"] : ["success"]) {
results[name].result = bad;
assert.notEqual(run().status, 0, `${id}: ${name} must reject ${bad}`);
}
results[name].result = previous;
}
}
});
test("the real routing step uses pinned trusted definitions and defaults to full on fetch failure", () => {
const workflow = readFileSync(new URL(".github/workflows/ci-routing.yml", root), "utf8");
const block = workflow.split(" - name: Plan affected CI tasks from trusted definitions\n")[1];
const shell = block.split(" run: |\n")[1].split("\n").map((line) => line.slice(10)).join("\n");
const directory = mkdtempSync(path.join(tmpdir(), "ci-routing-plan-"));
try {
const gh = path.join(directory, "gh");
writeFileSync(gh, `#!/usr/bin/env node
const fs = require('node:fs'), path = require('node:path');
const endpoint = process.argv.find(arg => arg.startsWith('/repos/'));
if (endpoint?.endsWith('/git/ref/heads/master')) console.log('a'.repeat(40));
else if (endpoint?.includes('/contents/')) {
if (process.env.TEST_FETCH_FAILURE === 'true') process.exit(1);
const [file, query] = endpoint.split('/contents/')[1].split('?');
if (query !== 'ref=' + 'a'.repeat(40)) process.exit(2);
process.stdout.write(fs.readFileSync(path.join(process.env.TEST_TRUSTED_ROOT, file)));
} else if (endpoint?.includes('/compare/')) console.log(JSON.stringify({ files: [{filename:'ui/src/components/Sidebar.tsx'}] }));
else process.exit(3);
`);
chmodSync(gh, 0o755);
for (const overrides of [{}, { TEST_FETCH_FAILURE: "true" }, { PROFILE_CI: "true" }, { STACK_JSON: "{}" }, { FULL_CI: "false" }]) {
const output = path.join(directory, "output");
writeFileSync(output, "");
const result = spawnSync("bash", ["-c", shell], { encoding: "utf8", env: {
...process.env, PATH: `${directory}:${process.env.PATH}`, TEST_TRUSTED_ROOT: root.pathname,
RUNNER_TEMP: directory, FULL_CI: "true", PROFILE_CI: "false", FORCE_FULL_CI: "false",
CI_INCREMENTAL_MODE: "enabled", STACK_JSON: "null", EVENT_BASE_SHA: "b".repeat(40), EVENT_MERGE_SHA: "c".repeat(40),
GITHUB_OUTPUT: output, GITHUB_STEP_SUMMARY: "", ...overrides,
} });
assert.equal(result.status, 0, result.stderr);
const value = readFileSync(output, "utf8").trim();
if (overrides.TEST_FETCH_FAILURE || overrides.FULL_CI === "false") assert.equal(value, "plan=");
else {
assert.notEqual(value, "plan=", `${JSON.stringify(overrides)}: ${result.stderr}`);
const resultPlan = JSON.parse(value.slice("plan=".length));
assert.equal(resultPlan.jobs.general_tests.selected, overrides.PROFILE_CI || overrides.STACK_JSON ? 19 : 2);
}
}
} finally { rmSync(directory, { recursive: true, force: true }); }
});
.github/scripts/tests/ci-routing.test.mjs
+4 -2
View File
@@ -147,13 +147,15 @@ for (const [name, stack, base, expected] of scopeCases) {
const files = (...paths) => JSON.stringify(paths.map((filename) => ({ filename })));
test("profile-ci overrides both middle-stack and release-notes-only skips", () => {
test("full-suite requests override both middle-stack and release-notes-only skips", () => {
for (const overrides of [
{ STACK_JSON: JSON.stringify({ position: 2, size: 3, base: { ref: "master" } }), PR_BASE_REF: "stack-1" },
{ STACK_JSON: "null", PR_BASE_REF: "master", TEST_COMPARE_FILES: files("releases/v2026.1015.0.md") },
]) {
assert.equal(executeStep("Select PR CI scope", { ...overrides, PROFILE_CI: "false" }).full_ci, "false");
assert.equal(executeStep("Select PR CI scope", { ...overrides, PROFILE_CI: "true" }).full_ci, "true");
for (const request of [{ PROFILE_CI: "true" }, { FORCE_FULL_CI: "true" }, { CI_INCREMENTAL_MODE: "full" }]) {
assert.equal(executeStep("Select PR CI scope", { ...overrides, ...request }).full_ci, "true");
}
}
});
const manyReleaseNotes = JSON.stringify(Array.from({ length: 300 }, (_, index) => ({
.github/workflows/ci-incremental-qualification.yml
+69
View File
@@ -0,0 +1,69 @@
name: Incremental CI qualification
on:
pull_request:
paths:
- package.json
- turbo.json
- .github/ci-task-matrix.json
- .github/scripts/ci-plan.mjs
- .github/scripts/tests/ci-plan.test.mjs
- .github/actions/run-ci-task/**
- .github/workflows/ci-routing.yml
- .github/workflows/pr-trusted.yml
- .github/workflows/ci-incremental-qualification.yml
- scripts/ci/**
permissions:
contents: read
concurrency:
group: incremental-ci-proof-${{ github.event.pull_request.number }}
cancel-in-progress: true
jobs:
qualify:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
with:
persist-credentials: false
fetch-depth: 0
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020
with:
node-version: 24
package-manager-cache: false
- name: Install isolated qualification tools
run: npm install --prefix "$RUNNER_TEMP/ci-tools" --no-audit --no-fund turbo@2.11.10
- name: Prove planning and required checks
run: node --test .github/scripts/tests/ci-plan.test.mjs .github/scripts/tests/ci-routing.test.mjs
- name: Prove real cache reuse and invalidation
env:
PAPERCLIP_CI_TEST_TURBO: ${{ runner.temp }}/ci-tools/node_modules/.bin/turbo
run: node --test scripts/ci/cache.integration.test.mjs
- name: Check the real repository task graph
env:
CI_TURBO_BIN: ${{ runner.temp }}/ci-tools/node_modules/.bin/turbo
run: |
node --input-type=module <<'NODE'
import { readFileSync, writeFileSync } from 'node:fs';
import { execFileSync } from 'node:child_process';
import assert from 'node:assert/strict';
import { executionPlan } from './scripts/ci/execution-plan.mjs';
const config = JSON.parse(readFileSync('turbo.json', 'utf8'));
for (const task of ['ci:server', 'ci:db', 'ci:runner-rust']) {
const plan = executionPlan(config, task, []);
const file = process.env.RUNNER_TEMP + '/ci-task-config.json';
writeFileSync(file, JSON.stringify(plan.config));
const result = execFileSync(process.env.CI_TURBO_BIN, ['run', plan.target, '--root-turbo-json', file,
'--dry=json', '--cache-dir=.turbo/cache', '--', ...plan.args], { encoding: 'utf8', maxBuffer: 16 * 1024 * 1024 });
assert.ok(JSON.parse(result).tasks.some((task) => task.taskId === plan.target));
}
NODE
- name: Explain this pull request's plan
env:
CI_INCREMENTAL_MODE: shadow
BASE_SHA: ${{ github.event.pull_request.base.sha }}
MERGE_SHA: ${{ github.sha }}
run: node .github/scripts/ci-plan.mjs "$BASE_SHA" "$MERGE_SHA"
.github/workflows/ci-routing.yml
+45 -2
View File
@@ -8,6 +8,8 @@ on:
value: ${{ jobs.gate.outputs.runner }}
full_ci:
value: ${{ jobs.gate.outputs.full_ci }}
plan:
value: ${{ jobs.gate.outputs.plan }}
profile_ci:
value: ${{ jobs.gate.outputs.profile_ci }}
@@ -22,6 +24,7 @@ jobs:
runs-on: ${{ vars.AWS_CI_ENABLED == 'true' && vars.AWS_CI_ROUTING_USER_IDS_SOURCE != '' && vars.AWS_CI_TRUSTED_USER_IDS == vars.AWS_CI_ROUTING_USER_IDS_SOURCE && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && ((github.event_name == 'pull_request' && contains(vars.AWS_CI_ROUTING_USER_IDS, format('|{0}|', github.event.pull_request.user.id)) && contains(vars.AWS_CI_ROUTING_USER_IDS, format('|{0}|', github.event.sender.id))) || (github.event_name == 'workflow_dispatch' && github.ref == 'refs/heads/master' && contains(vars.AWS_CI_ROUTING_USER_IDS, format('|{0}|', github.actor_id)))) && 'runs-on/fleet=paperclip-ci-routing-x64/env=public-ci' || 'ubuntu-latest' }}
timeout-minutes: 5
outputs:
plan: ${{ steps.plan.outputs.plan }}
runner: ${{ steps.route.outputs.runner }}
full_ci: ${{ steps.scope.outputs.full_ci }}
profile_ci: ${{ steps.profile.outputs.profile_ci }}
@@ -256,6 +259,8 @@ jobs:
EVENT_BASE_SHA: ${{ github.event.pull_request.base.sha }}
EVENT_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
PROFILE_CI: ${{ steps.profile.outputs.profile_ci }}
FORCE_FULL_CI: ${{ contains(github.event.pull_request.labels.*.name, 'full-ci') }}
CI_INCREMENTAL_MODE: ${{ vars.CI_INCREMENTAL_MODE || 'enabled' }}
STACK_JSON: ${{ toJSON(github.event.pull_request.stack) }}
PR_BASE_REF: ${{ github.event.pull_request.base.ref }}
run: |
@@ -326,11 +331,49 @@ jobs:
reason='release notes only'
fi
if [[ "${PROFILE_CI:-false}" == 'true' ]]; then
if [[ "${PROFILE_CI:-false}" == 'true' || "${FORCE_FULL_CI:-false}" == 'true' || "${CI_INCREMENTAL_MODE:-enabled}" == 'full' ]]; then
full_ci='true'
reason='profile-ci requests the complete PR suite'
reason='profile-ci, full-ci or full mode requests the complete PR suite'
fi
echo "full_ci=$full_ci" >> "$GITHUB_OUTPUT"
echo "::notice title=PR CI scope::$reason; full_ci=$full_ci"
- name: Plan affected CI tasks from trusted definitions
id: plan
shell: bash
env:
GH_TOKEN: ${{ github.token }}
FULL_CI: ${{ steps.scope.outputs.full_ci }}
PROFILE_CI: ${{ steps.profile.outputs.profile_ci }}
FORCE_FULL_CI: ${{ contains(github.event.pull_request.labels.*.name, 'full-ci') }}
CI_INCREMENTAL_MODE: ${{ vars.CI_INCREMENTAL_MODE || 'enabled' }}
STACK_JSON: ${{ toJSON(github.event.pull_request.stack) }}
EVENT_BASE_SHA: ${{ github.event.pull_request.base.sha }}
EVENT_MERGE_SHA: ${{ github.sha }}
run: |
set -euo pipefail
fallback() { echo 'plan=' >> "$GITHUB_OUTPUT"; exit 0; }
[[ "$FULL_CI" == 'true' ]] || fallback
[[ "$EVENT_BASE_SHA" =~ ^[0-9a-f]{40}$ && "$EVENT_MERGE_SHA" =~ ^[0-9a-f]{40}$ ]] || fallback
trusted="$(gh api /repos/paperclipai/paperclip/git/ref/heads/master --jq .object.sha)" || fallback
[[ "$trusted" =~ ^[0-9a-f]{40}$ ]] || fallback
directory="$(mktemp -d)"
trap 'rm -rf "$directory"' EXIT
for file in .github/scripts/ci-plan.mjs .github/ci-task-matrix.json turbo.json; do
gh api -H 'Accept: application/vnd.github.raw+json' \
"/repos/paperclipai/paperclip/contents/$file?ref=$trusted" \
> "$directory/${file##*/}" || fallback
done
gh api "/repos/paperclipai/paperclip/compare/$EVENT_BASE_SHA...$EVENT_MERGE_SHA" \
> "$directory/comparison.json" || fallback
if [[ "$PROFILE_CI" == 'true' || "$FORCE_FULL_CI" == 'true' ]]; then
export CI_FORCE_REASON='Full suite requested by label'
elif [[ "$STACK_JSON" != 'null' && -n "$STACK_JSON" ]]; then
export CI_FORCE_REASON='Stack verification includes lower pull requests'
fi
export CI_TASK_CONFIG="$directory/turbo.json"
export CI_TASK_MATRIX="$directory/ci-task-matrix.json"
export CI_COMPARISON="$directory/comparison.json"
node "$directory/ci-plan.mjs" || fallback
.github/workflows/pr-trusted.yml
+80 -240
View File
@@ -159,7 +159,7 @@ jobs:
typecheck_release_registry:
name: Typecheck + Release Registry
needs: [gate]
if: ${{ needs.gate.outputs.full_ci == 'true' }}
if: ${{ needs.gate.outputs.full_ci == 'true' && (needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).jobs.typecheck_release_registry.run) }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 20
@@ -339,6 +339,9 @@ jobs:
- name: Typecheck workspaces whose build scripts skip TypeScript
run: pnpm run typecheck:build-gaps
- name: Verify repository-wide CLI guidance safety
run: pnpm exec vitest run --project @paperclipai/server server/src/__tests__/cli-invocation-safety.test.ts
- name: Verify release registry test coverage
run: pnpm run test:release-registry
@@ -356,100 +359,12 @@ jobs:
general_tests:
name: General tests (${{ matrix.group_label }})
needs: [gate]
if: ${{ needs.gate.outputs.full_ci == 'true' }}
if: ${{ needs.gate.outputs.full_ci == 'true' && (needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).jobs.general_tests.run) }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 20
strategy:
fail-fast: false
matrix:
include:
# Keep twelve duration-balanced server shards. Each shard overlaps
# at most two isolated fork workers; cases within a file stay serial.
# The chat fixture keeps three independent database-backed shards.
- group: general-server-without-chat
group_label: server (1/12)
shard_index: 0
shard_count: 12
- group: general-server-without-chat
group_label: server (2/12)
shard_index: 1
shard_count: 12
- group: general-server-without-chat
group_label: server (3/12)
shard_index: 2
shard_count: 12
- group: general-server-without-chat
group_label: server (4/12)
shard_index: 3
shard_count: 12
- group: general-server-without-chat
group_label: server (5/12)
shard_index: 4
shard_count: 12
- group: general-server-without-chat
group_label: server (6/12)
shard_index: 5
shard_count: 12
- group: general-server-without-chat
group_label: server (7/12)
shard_index: 6
shard_count: 12
- group: general-server-without-chat
group_label: server (8/12)
shard_index: 7
shard_count: 12
- group: general-server-without-chat
group_label: server (9/12)
shard_index: 8
shard_count: 12
- group: general-server-without-chat
group_label: server (10/12)
shard_index: 9
shard_count: 12
- group: general-server-without-chat
group_label: server (11/12)
shard_index: 10
shard_count: 12
- group: general-server-without-chat
group_label: server (12/12)
shard_index: 11
shard_count: 12
- group: general-chat
group_label: chat (1/3)
shard_index: 0
shard_count: 3
- group: general-chat
group_label: chat (2/3)
shard_index: 1
shard_count: 3
- group: general-chat
group_label: chat (3/3)
shard_index: 2
shard_count: 3
# workspaces-a was the slowest check in the fully-green PR run
# 31371439296 (2026-08-10) at 319s, with the ui project's single
# vitest invocation accounting for ~224s and the paperclipai CLI
# ~37s. Two shards use Vitest's native --shard on each project's
# file list (ui: 439 files, cli: 54), bringing each job to roughly
# half the suite time (~130s + setup) without a duration manifest.
- group: general-workspaces-a
group_label: workspaces-a (1/2)
shard_index: 0
shard_count: 2
- group: general-workspaces-a
group_label: workspaces-a (2/2)
shard_index: 1
shard_count: 2
# workspaces-b became the slowest check (499-593s) once the server
# shards were rebalanced: the db project alone was 168-250s of its
# 436-513s of vitest time (PR runs 38084171761, 38083084276 and
# 38083020834, 2026-10-10). The db project now runs as its own
# workspaces-c lane; the rest of workspaces-b is ~265-275s of
# vitest time, so both jobs land near 300-350s with setup.
- group: general-workspaces-b
group_label: workspaces-b
- group: general-workspaces-c
group_label: workspaces-c
matrix: ${{ fromJSON(needs.gate.outputs.plan != '' && toJSON(fromJSON(needs.gate.outputs.plan).jobs.general_tests.matrix) || '{"include":[{"group":"general-server-without-chat","group_label":"server (1/12)","shard_index":0,"shard_count":12,"task":"ci:server"},{"group":"general-server-without-chat","group_label":"server (2/12)","shard_index":1,"shard_count":12,"task":"ci:server"},{"group":"general-server-without-chat","group_label":"server (3/12)","shard_index":2,"shard_count":12,"task":"ci:server"},{"group":"general-server-without-chat","group_label":"server (4/12)","shard_index":3,"shard_count":12,"task":"ci:server"},{"group":"general-server-without-chat","group_label":"server (5/12)","shard_index":4,"shard_count":12,"task":"ci:server"},{"group":"general-server-without-chat","group_label":"server (6/12)","shard_index":5,"shard_count":12,"task":"ci:server"},{"group":"general-server-without-chat","group_label":"server (7/12)","shard_index":6,"shard_count":12,"task":"ci:server"},{"group":"general-server-without-chat","group_label":"server (8/12)","shard_index":7,"shard_count":12,"task":"ci:server"},{"group":"general-server-without-chat","group_label":"server (9/12)","shard_index":8,"shard_count":12,"task":"ci:server"},{"group":"general-server-without-chat","group_label":"server (10/12)","shard_index":9,"shard_count":12,"task":"ci:server"},{"group":"general-server-without-chat","group_label":"server (11/12)","shard_index":10,"shard_count":12,"task":"ci:server"},{"group":"general-server-without-chat","group_label":"server (12/12)","shard_index":11,"shard_count":12,"task":"ci:server"},{"group":"general-chat","group_label":"chat (1/3)","shard_index":0,"shard_count":3,"task":"ci:chat"},{"group":"general-chat","group_label":"chat (2/3)","shard_index":1,"shard_count":3,"task":"ci:chat"},{"group":"general-chat","group_label":"chat (3/3)","shard_index":2,"shard_count":3,"task":"ci:chat"},{"group":"general-workspaces-a","group_label":"workspaces-a (1/2)","shard_index":0,"shard_count":2,"task":"ci:workspaces-a"},{"group":"general-workspaces-a","group_label":"workspaces-a (2/2)","shard_index":1,"shard_count":2,"task":"ci:workspaces-a"},{"group":"general-workspaces-b","group_label":"workspaces-b","task":"ci:workspaces-b"},{"group":"general-workspaces-c","group_label":"workspaces-c","task":"ci:db"}]}') }}
env:
PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }}
@@ -516,7 +431,17 @@ jobs:
pnpm install --frozen-lockfile
fi
- name: Run grouped general test suites through Turbo
if: hashFiles('turbo.json', '.github/actions/run-ci-task/action.yml') != ''
uses: ./.github/actions/run-ci-task
with:
task: ${{ matrix.task }}
args: ${{ matrix.shard_count && format('["--shard-index","{0}","--shard-count","{1}"]', matrix.shard_index, matrix.shard_count) || '[]' }}
partition: ${{ strategy.job-index || 0 }}
force: ${{ needs.gate.outputs.profile_ci == 'true' || needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).mode != 'enabled' || fromJSON(needs.gate.outputs.plan).forceReason != null }}
- name: Run grouped general test suites
if: hashFiles('turbo.json') == ''
run: |
if [ -n "${{ matrix.shard_count }}" ]; then
pnpm test:run:general -- --group '${{ matrix.group }}' \
@@ -539,7 +464,7 @@ jobs:
docker_context_integrity:
name: Docker context integrity
needs: gate
if: ${{ needs.gate.outputs.full_ci == 'true' }}
if: ${{ needs.gate.outputs.full_ci == 'true' && (needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).jobs.docker_context_integrity.run) }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 15
@@ -594,77 +519,42 @@ jobs:
retention-days: 14
verify:
# Preserve the legacy required-check name while the underlying work runs in parallel.
name: verify
if: ${{ always() }}
needs: [gate, policy, typecheck_release_registry, general_tests, verify_paperclip_runner, build, docker_context_integrity]
needs: [gate, policy, typecheck_release_registry, general_tests, verify_paperclip_runner, build, docker_context_integrity, verify_serialized_server, canary_dry_run]
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 5
steps:
- name: Fail if any split verify lane failed
- name: Validate every planned suite result
env:
FULL_CI: ${{ needs.gate.outputs.full_ci }}
POLICY_RESULT: ${{ needs.policy.result }}
TYPECHECK_RELEASE_REGISTRY_RESULT: ${{ needs.typecheck_release_registry.result }}
GENERAL_TESTS_RESULT: ${{ needs.general_tests.result }}
RUNNER_VERIFICATION_RESULT: ${{ needs.verify_paperclip_runner.result }}
BUILD_RESULT: ${{ needs.build.result }}
DOCKER_CONTEXT_INTEGRITY_RESULT: ${{ needs.docker_context_integrity.result }}
CI_PLAN: ${{ needs.gate.outputs.plan }}
CI_RESULTS: ${{ toJSON(needs) }}
run: |
test "$POLICY_RESULT" = "success"
case "$FULL_CI" in
true)
test "$TYPECHECK_RELEASE_REGISTRY_RESULT" = "success"
test "$GENERAL_TESTS_RESULT" = "success"
test "$RUNNER_VERIFICATION_RESULT" = "success"
test "$BUILD_RESULT" = "success"
test "$DOCKER_CONTEXT_INTEGRITY_RESULT" = "success"
;;
false)
test "$TYPECHECK_RELEASE_REGISTRY_RESULT" = "skipped"
test "$GENERAL_TESTS_RESULT" = "skipped"
test "$RUNNER_VERIFICATION_RESULT" = "skipped"
test "$BUILD_RESULT" = "skipped"
test "$DOCKER_CONTEXT_INTEGRITY_RESULT" = "skipped"
;;
*)
echo "Invalid full_ci decision: $FULL_CI" >&2
exit 1
;;
esac
node --input-type=module <<'NODE'
import assert from 'node:assert/strict';
const results = JSON.parse(process.env.CI_RESULTS);
assert.equal(results.gate.result, 'success', 'CI routing must succeed');
assert.equal(results.policy.result, 'success', 'Policy must succeed');
assert.ok(['true', 'false'].includes(process.env.FULL_CI), 'Valid scope required');
const plan = process.env.CI_PLAN ? JSON.parse(process.env.CI_PLAN) : null;
if (plan) assert.equal(plan.version, 1, 'Supported task plan required');
for (const job of ["typecheck_release_registry","general_tests","verify_paperclip_runner","build","docker_context_integrity","verify_serialized_server","canary_dry_run"]) {
if (plan) assert.equal(typeof plan.jobs[job]?.run, 'boolean', 'Missing suite decision: ' + job);
const selected = process.env.FULL_CI === 'true' && (!plan || plan.jobs[job].run);
assert.equal(results[job].result, selected ? 'success' : 'skipped', job);
}
NODE
verify_paperclip_runner:
name: Verify Paperclip Runner (${{ matrix.lane_label }})
needs: [gate]
if: ${{ needs.gate.outputs.full_ci == 'true' }}
if: ${{ needs.gate.outputs.full_ci == 'true' && (needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).jobs.verify_paperclip_runner.run) }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 20
strategy:
fail-fast: false
matrix:
include:
# check:all spent ~380s of its ~600s cache-warm step inside the
# runner package's vitest suite (run 35036001734, 2026-09-15); the
# Rust tests took ~160s and every remaining check ~60s combined,
# which made this single job the PR critical path once the test
# lanes were rebalanced. Split the vitest suite across two runners
# with Vitest's native --shard, the ~160s Rust tests into their own
# lane, and the remaining static checks into a fourth. The four
# lanes union to exactly check:all: check:static covers
# check:eval-kernel, check:protocol-without-vitest, and
# check:api-authority; check:runner covers the Rust half; and the
# two vitest shards cover the package vitest file list.
- lane_label: static checks
command: check:static
- lane_label: rust
command: check:runner
release_crate: true
- lane_label: vitest 1/2
command: test:typescript:vitest --shard=1/2
- lane_label: vitest 2/2
command: test:typescript:vitest --shard=2/2
release_crate: true
matrix: ${{ fromJSON(needs.gate.outputs.plan != '' && toJSON(fromJSON(needs.gate.outputs.plan).jobs.verify_paperclip_runner.matrix) || '{"include":[{"lane_label":"static checks","command":"check:static","task":"ci:runner-static"},{"lane_label":"rust","command":"check:runner","release_crate":true,"task":"ci:runner-rust"},{"lane_label":"vitest 1/2","command":"test:typescript:vitest --shard=1/2","task":"ci:runner-vitest"},{"lane_label":"vitest 2/2","command":"test:typescript:vitest --shard=2/2","release_crate":true,"task":"ci:runner-vitest"}]}') }}
env:
PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }}
@@ -849,7 +739,17 @@ jobs:
if: ${{ steps.runner_release_crate.outputs.cache-hit == 'true' }}
run: node scripts/runner-release-crate-cache.mjs mark-current
- name: Verify Runner Rust through Turbo
if: matrix.command == 'check:runner' && hashFiles('turbo.json') != ''
uses: ./.github/actions/run-ci-task
with:
task: ci:runner-rust
args: '[]'
partition: ${{ strategy.job-index || 0 }}
force: ${{ needs.gate.outputs.profile_ci == 'true' || needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).mode != 'enabled' || fromJSON(needs.gate.outputs.plan).forceReason != null }}
- name: Verify Paperclip Runner
if: matrix.command != 'check:runner' || hashFiles('turbo.json') == ''
# pnpm appends trailing args to the end of the script's shell chain,
# so the vitest lanes' --shard lands on `vitest run`. Do not add a
# `--` separator: pnpm forwards it literally and vitest would then
@@ -870,7 +770,7 @@ jobs:
build:
name: Build
needs: [gate]
if: ${{ needs.gate.outputs.full_ci == 'true' }}
if: ${{ needs.gate.outputs.full_ci == 'true' && (needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).jobs.build.run) }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 20
@@ -1122,48 +1022,12 @@ jobs:
verify_serialized_server:
name: Verify serialized server suites (${{ matrix.shard_label }})
needs: [gate]
if: ${{ needs.gate.outputs.full_ci == 'true' }}
if: ${{ needs.gate.outputs.full_ci == 'true' && (needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).jobs.verify_serialized_server.run) }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 20
strategy:
fail-fast: false
matrix:
include:
# Shards are balanced by recorded duration
# (scripts/serialized-shard-durations.json, refreshed 2026-09-12);
# round-robin used to cluster the heavy suites on one runner (291s
# vs 170-201s in run 32012408876, 2026-08-17). The suite total has
# since grown to ~1926s recorded: five shards ran ~390s of tests
# each while the rebalanced general/e2e lanes run ~210-260s, so
# nine shards level this lane to ~214s and keep it off the
# critical path.
- shard_index: 0
shard_count: 9
shard_label: 1/9
- shard_index: 1
shard_count: 9
shard_label: 2/9
- shard_index: 2
shard_count: 9
shard_label: 3/9
- shard_index: 3
shard_count: 9
shard_label: 4/9
- shard_index: 4
shard_count: 9
shard_label: 5/9
- shard_index: 5
shard_count: 9
shard_label: 6/9
- shard_index: 6
shard_count: 9
shard_label: 7/9
- shard_index: 7
shard_count: 9
shard_label: 8/9
- shard_index: 8
shard_count: 9
shard_label: 9/9
matrix: ${{ fromJSON(needs.gate.outputs.plan != '' && toJSON(fromJSON(needs.gate.outputs.plan).jobs.verify_serialized_server.matrix) || '{"include":[{"shard_index":0,"shard_count":9,"shard_label":"1/9","task":"ci:serialized"},{"shard_index":1,"shard_count":9,"shard_label":"2/9","task":"ci:serialized"},{"shard_index":2,"shard_count":9,"shard_label":"3/9","task":"ci:serialized"},{"shard_index":3,"shard_count":9,"shard_label":"4/9","task":"ci:serialized"},{"shard_index":4,"shard_count":9,"shard_label":"5/9","task":"ci:serialized"},{"shard_index":5,"shard_count":9,"shard_label":"6/9","task":"ci:serialized"},{"shard_index":6,"shard_count":9,"shard_label":"7/9","task":"ci:serialized"},{"shard_index":7,"shard_count":9,"shard_label":"8/9","task":"ci:serialized"},{"shard_index":8,"shard_count":9,"shard_label":"9/9","task":"ci:serialized"}]}') }}
env:
PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }}
@@ -1230,7 +1094,17 @@ jobs:
pnpm install --frozen-lockfile
fi
- name: Run serialized server test shard through Turbo
if: hashFiles('turbo.json', '.github/actions/run-ci-task/action.yml') != ''
uses: ./.github/actions/run-ci-task
with:
task: ci:serialized
args: ${{ format('["--shard-index","{0}","--shard-count","{1}"]', matrix.shard_index, matrix.shard_count) }}
partition: ${{ strategy.job-index || 0 }}
force: ${{ needs.gate.outputs.profile_ci == 'true' || needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).mode != 'enabled' || fromJSON(needs.gate.outputs.plan).forceReason != null }}
- name: Run serialized server test shard
if: hashFiles('turbo.json') == ''
run: pnpm test:run:serialized -- --shard-index ${{ matrix.shard_index }} --shard-count ${{ matrix.shard_count }}
- name: Upload CI profile timings
@@ -1247,7 +1121,7 @@ jobs:
canary_dry_run:
name: Canary Dry Run
needs: [gate]
if: ${{ needs.gate.outputs.full_ci == 'true' }}
if: ${{ needs.gate.outputs.full_ci == 'true' && (needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).jobs.canary_dry_run.run) }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 20
@@ -1462,48 +1336,12 @@ jobs:
e2e_shards:
name: e2e shard (${{ matrix.shard_label }})
needs: [gate]
if: ${{ needs.gate.outputs.full_ci == 'true' }}
if: ${{ needs.gate.outputs.full_ci == 'true' && (needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).jobs.e2e_shards.run) }}
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
include:
# The Playwright lane is pinned to workers=1 (tests/e2e/playwright.config.ts)
# because every spec shares one throwaway server and some toggle
# instance-level flags, so it can only be parallelized across runners.
# Each shard boots its own server, which keeps that isolation intact.
# The catalog has grown to ~1694s of serial spec time (mean of runs
# 35036001734 and 35024948947, 2026-09-15): with three shards and
# stale durations the worst shard ran 745s of specs while the best
# ran 277s. The former floors — chat-adapters-ui at ~442s and
# agent-chat at ~310s — are each split into two specs, so eight
# shards with refreshed durations level to ~212s each; the heaviest
# remaining single spec is chat-adapters-ui-messaging at ~242s.
- shard_index: 0
shard_count: 8
shard_label: 1/8
- shard_index: 1
shard_count: 8
shard_label: 2/8
- shard_index: 2
shard_count: 8
shard_label: 3/8
- shard_index: 3
shard_count: 8
shard_label: 4/8
- shard_index: 4
shard_count: 8
shard_label: 5/8
- shard_index: 5
shard_count: 8
shard_label: 6/8
- shard_index: 6
shard_count: 8
shard_label: 7/8
- shard_index: 7
shard_count: 8
shard_label: 8/8
matrix: ${{ fromJSON(needs.gate.outputs.plan != '' && toJSON(fromJSON(needs.gate.outputs.plan).jobs.e2e_shards.matrix) || '{"include":[{"shard_index":0,"shard_count":8,"shard_label":"1/8","task":"ci:e2e"},{"shard_index":1,"shard_count":8,"shard_label":"2/8","task":"ci:e2e"},{"shard_index":2,"shard_count":8,"shard_label":"3/8","task":"ci:e2e"},{"shard_index":3,"shard_count":8,"shard_label":"4/8","task":"ci:e2e"},{"shard_index":4,"shard_count":8,"shard_label":"5/8","task":"ci:e2e"},{"shard_index":5,"shard_count":8,"shard_label":"6/8","task":"ci:e2e"},{"shard_index":6,"shard_count":8,"shard_label":"7/8","task":"ci:e2e"},{"shard_index":7,"shard_count":8,"shard_label":"8/8","task":"ci:e2e"}]}') }}
env:
PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }}
@@ -1628,27 +1466,29 @@ jobs:
retention-days: 14
e2e:
# Preserve the legacy required-check name while the specs run sharded
# across the matrix above (same pattern as the `verify` aggregate).
name: e2e
if: ${{ always() }}
needs: [gate, policy, e2e_shards]
runs-on: ${{ needs.gate.outputs.runner }}
timeout-minutes: 5
steps:
- name: Fail if any e2e shard failed
- name: Validate every planned suite result
env:
FULL_CI: ${{ needs.gate.outputs.full_ci }}
POLICY_RESULT: ${{ needs.policy.result }}
E2E_SHARDS_RESULT: ${{ needs.e2e_shards.result }}
CI_PLAN: ${{ needs.gate.outputs.plan }}
CI_RESULTS: ${{ toJSON(needs) }}
run: |
test "$POLICY_RESULT" = "success"
case "$FULL_CI" in
true) test "$E2E_SHARDS_RESULT" = "success" ;;
false) test "$E2E_SHARDS_RESULT" = "skipped" ;;
*)
echo "Invalid full_ci decision: $FULL_CI" >&2
exit 1
;;
esac
node --input-type=module <<'NODE'
import assert from 'node:assert/strict';
const results = JSON.parse(process.env.CI_RESULTS);
assert.equal(results.gate.result, 'success', 'CI routing must succeed');
assert.equal(results.policy.result, 'success', 'Policy must succeed');
assert.ok(['true', 'false'].includes(process.env.FULL_CI), 'Valid scope required');
const plan = process.env.CI_PLAN ? JSON.parse(process.env.CI_PLAN) : null;
if (plan) assert.equal(plan.version, 1, 'Supported task plan required');
for (const job of ["e2e_shards"]) {
if (plan) assert.equal(typeof plan.jobs[job]?.run, 'boolean', 'Missing suite decision: ' + job);
const selected = process.env.FULL_CI === 'true' && (!plan || plan.jobs[job].run);
assert.equal(results[job].result, selected ? 'success' : 'skipped', job);
}
NODE
.github/workflows/pr.yml
+1 -1
View File
@@ -13,7 +13,7 @@ permissions:
jobs:
ci:
if: github.event.action != 'labeled' || github.event.label.name == 'profile-ci'
if: github.event.action != 'labeled' || github.event.label.name == 'profile-ci' || github.event.label.name == 'full-ci'
# Master requires CODEOWNERS review for .github/**, including this workflow.
# The AWS runner group permits pr-trusted.yml@refs/heads/master.
uses: paperclipai/paperclip/.github/workflows/pr-trusted.yml@master