mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 16:27:18 +02:00
ea4aa185b0f3bc30be1c5fc160cdfd0eb433fe41
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
6acec4266e |
feat(ci): profile full CODEOWNER PR suites in Honeycomb (#16378)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its PR workflow runs dozens of parallel build and test jobs. > - Individual job durations do not show the elapsed cost of the whole suite. > - Maintainers need one shared timeline to find slow work and repeated setup. > - This pull request adds full-suite profiles for selected CI CODEOWNER PRs. > - Honeycomb shows each job, step, test file, and supported test on that timeline. > - The benefit is evidence for changes that reduce total CI time. ## Linked Issues or Issue Description **What existing behavior does this improve?** The main PR workflow has about 48 jobs. Maintainers need to compare their overlapping work and find which jobs delay completion. **Current behavior** GitHub shows job and step durations separately. Test files and tests are not joined to a full workflow trace. **Proposed behavior** A CI CODEOWNER adds `profile-ci` to a PR. The tested code stays on that PR branch. All main PR lanes run, including lanes normally skipped for stack position or release notes. A trusted post-run exporter sends one wall-clock trace to Honeycomb. **Reason and benefit** The trace supports decisions about which CI work to speed up or remove. Ordinary PRs do not export traces. The roadmap and related-PR search found no duplicate CI profiling work. ## What Changed - Add label selection and a full-suite override. Verify the PR author, original actor, and rerun actors against the trusted `.github/**` CODEOWNERS rule. - Record Vitest, Playwright, and Node test file/test timings in local artifacts. Keep ordinary reporter behavior and failing exit codes. - Add a trusted `workflow_run` exporter. Use actual GitHub timestamps for all jobs, setup, steps, waiting, and teardown. Mark missing test data and partial reruns. - Keep the ingest key out of PR jobs. Store it only in the `ci-profiling` environment, restricted to `master`. The exporter reads bounded JSON and never executes artifact code. - Record expected test invocations outside reporters so a missing invocation cannot appear complete. Preserve `node --test` on older PR checkouts. Flush large shard inventories before exit. - Add security, timing, routing, and real-reporter checks. Document operation and activation. Protect the exporter with CODEOWNERS. ## Verification - 509 focused workflow/reporter/shard checks passed. They include outside authors, outside rerun actors, bots, stale revisions, malicious ZIP paths, incomplete reports, and partial ingestion failures. - Final revision `3bf5d9017d` adds multi-file qualification and fixes short-file step parenting and duplicate Node file-wrapper events. Greptile reviewed this exact revision at 5/5 with no unresolved threads. [Main PR CI run](https://github.com/paperclipai/paperclip/actions/runs/38728040302): the initial attempt had 54 successful checks, 3 intentional skips, and one serialized-shard failure from an unchanged workspace-authorization test exceeding its 15-second timeout. The single failed-shard retry passed: the final result is 55 successful checks and 3 intentional skips. The same timeout reproduces locally with profiling disabled; both assertions pass with a diagnostic 60-second timeout override (about 23 seconds total). No source or CI timeout was changed. - A dry run against real GitHub attempt 2 of run `38677445871` produced exactly 3 executed jobs, 40 spans, and 510 seconds, excluding 45 carried-over jobs. The partial trace remains explicitly incomplete; this validation did not upload data. - `actionlint` passed for all four changed workflows. `git diff --check` passed. - Full Node 24 `pnpm -r typecheck` and `pnpm build` passed. The local full `pnpm test:run` was stopped after it reported a setup-token timeout and a distribution-plugin-catalog fixture failure. The 30 setup-token tests passed on focused rerun. The plugin fixture fails at `rmSync` of a directory symlink on this macOS/Node 24.11.0 host; the same failure reproduces with the unchanged base Vitest configuration and unchanged test/service files (10 other tests pass). The local full test suite is therefore not claimed green. - A real root-config Vitest probe passed 13 tests with profiling both off and on. Multi-file Vitest, Playwright, and Node probes each verify two distinct filename rows and four correctly parented tests, including repeated test names across files. Separate passing probes exercise the full exporter hierarchy `workflow → job → step → filename → tests`. Intentional failing probes preserve failing exit status. - Verified all 48 measured jobs in latest-head run `38728040302` used AWS: one `paperclip-ci-routing-x64` gate and 47 `paperclip-public-pr-x64` jobs. Profiling preserves AWS authorization and its GitHub-hosted fallback. The separate privileged exporter runs on `ubuntu-latest` outside the measured suite. - [Live Honeycomb baseline](https://ui.honeycomb.io/paperclip/environments/test/datasets/paperclip-ci/trace/oaPMzw8zGwG): 48 jobs, 731 spans, 479 seconds. It matches [GitHub run 38671274294](https://github.com/paperclipai/paperclip/actions/runs/38671274294). This historical trace has job/step data only; it is not proof of the future label-triggered test-reporting journey. - Verified the dedicated Honeycomb key is ingest-only with dataset creation disabled. Verified the GitHub environment allows only the `master` branch. No repository-level secret of that name exists. Both credential values were scanned absent from source and branch history. After merge, add `profile-ci` to a CI CODEOWNER PR, wait for the entire PR run and exporter, then follow the run-summary link. Collapse job children to compare parallel lanes; expand a job to inspect its steps and tests. ## Risks - Activation requires merging the trusted workflow definitions to `master`. This PR continues using the current trusted reusable workflows until then. - Selected runs produce many spans and artifacts. Oversized artifacts fail export instead of truncating data. Missing or cancelled reporters are marked incomplete. - GitHub timings have second precision. Node test times use reporter receipt time. Rust retains job/step timing but lacks per-test wall timestamps. - CI CODEOWNERS are individual users in the explicit `.github/**` rule. Unsupported team rules and identity lookup failures disable profiling. - An interrupted upload can leave partial data. Retrying uses the same span IDs, but the Events API does not promise deduplication. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, repository tools, and browser verification. The runtime does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
b226336762 |
ci: skip full CI for release-notes-only pull requests (#16207)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each beta release ends with a pull request that adds one release notes file under `releases/` > - PR CI runs the full build, test, and e2e matrix for that pull request > - No build, test, or e2e lane reads `releases/v*.md` or `releases/beta/v*.md` > - The full matrix takes about 30 minutes, and lost runners or flaky e2e specs can make it run again > - This pull request lets the existing CI scope gate skip the heavy lanes when a pull request changes only release notes > - The benefit is that release notes pull requests get a green result in minutes, and code pull requests keep full CI ## Linked Issues or Issue Description **Issue type** CI efficiency enhancement. **Where is the issue?** The `Select stacked PR CI scope` step in `.github/workflows/ci-routing.yml`. This step sets the `full_ci` output for PR CI. **What's wrong?** The step sets `full_ci=false` only for middle pull requests in a stack. A pull request that changes only one release notes markdown file runs the full matrix. For example, #16053 changed one file in `releases/beta/`. Its CI ran two times: on the first run, 8 shards lost their runners, and on the second run, one Playwright spec flaked and needed a rerun. **Suggested fix** Read the pull request diff from the GitHub compare API in the scope step. When every changed path is release notes, set `full_ci=false`. The callers already support this value. I found no open pull request or issue for this change. ## What Changed - `ci-routing.yml`: the scope step (renamed to `Select PR CI scope`) also sets `full_ci=false` when every path in the diff matches `releases/v*.md` or `releases/beta/v*.md`. - The step reads the diff from `compare/<event base SHA>...<event head SHA>`. The SHAs come from the event, so a later push cannot change the decision for this run. - For a renamed file, the step checks the old path and the new path. - The step keeps full CI when the event is not `pull_request`, a SHA is malformed, the API call fails, the diff is empty, or the file list has 300 entries. The compare API can truncate a list at 300 entries. - The step does not change the stacked pull request rules. It checks the paths only when the stack rules select full CI. - `ci-routing.test.mjs`: the GitHub stub now returns `files` for compare calls. 17 new cases cover release notes paths, mixed diffs, renames, non-markdown files, nested folders, empty and truncated lists, API failure, non-PR events, malformed SHAs, and stack interaction. - `e2e-shard.test.mjs`: the test now uses the new step name. - `doc/DEVELOPING.md`: added a short description of the CI scope rules. ## Verification - `node --test .github/scripts/tests/ci-routing.test.mjs scripts/__tests__/e2e-shard.test.mjs`: 73 pass, 0 fail. - `node --test '.github/scripts/tests/*.test.mjs'`: all pass. - I ran the jq filter on real compare API responses. #16053 (one file in `releases/beta/`) returns `true`. #16082 (10 files) returns `false`. - After merge: open a pull request that changes only a `releases/beta/v*.md` file. The `Select trusted runner` job shows the notice `release notes only; full_ci=false`. The heavy lanes show as skipped. The `policy`, `verify`, and `e2e` checks pass. Note: `pr.yml` calls `pr-trusted.yml@master`, and that workflow calls `ci-routing.yml@master`. This pull request does not change its own CI run. The change applies after merge. ## Risks - Low risk. The skip uses the existing `full_ci=false` path. The `verify` and `e2e` aggregate checks already accept that value, and the `policy` job always runs. - A future job that reads release notes files would not run for notes-only pull requests. Today, only the release workflows read these files, and those workflows do not run on pull requests. - The step now uses `GH_TOKEN` to call one more read-only API endpoint. The workflow still does not check out or run pull request code, and its permissions do not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic coding with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
f5346c05b9 |
ci: route PR admission through reserved AWS capacity (#16042)
## Thinking Path
> - Paperclip uses CI to verify changes before release.
> - Trusted PR tests can run on isolated AWS workers.
> - Their runner-selection job currently waits for GitHub-hosted
capacity.
> - A large hosted test queue therefore blocks admission to AWS.
> - A dedicated routing workflow lets us reserve capacity for metadata
validation.
> - This PR moves the routing job to reserved AWS capacity after the
bootstrap workflow and pool are available.
## Linked Issues or Issue Description
**Issue type**
CI reliability enhancement.
**Where is the issue?**
The Select trusted runner job in PR CI.
**What's wrong?**
The routing job competes with a large hosted test queue. Its sampled
median wait was about 20 minutes. AWS tests cannot begin before this job
finishes.
**Suggested fix**
Add a protected reusable routing workflow. Deploy its dedicated AWS
pool, then switch the PR caller in a follow-up PR. No overlapping open
PR or issue was found.
## What Changed
- Call ci-routing.yml on protected master from pr-trusted.yml.
- Check the canonical repository and approved PR author/sender before
allocating reserved AWS metadata capacity. Unapproved PRs continue to
use hosted routing.
- Permit approved manual routing canaries on protected master.
- Use a validated roster snapshot for placement. Invalid JSON, missing
settings, stale snapshots, and unapproved identities select hosted
routing without a placement-time JSON parse.
- Preserve the full in-job identity, rerun actor, repository, head,
merge, and stack checks.
- Update stack-scope and placement regression tests and document the
pool and rollback.
## Verification
- The routing scripts and placement expression tests passed (44 tests),
routing-plus-shard tests passed (56 tests), and the existing
.github/scripts suite passed (381 tests).
- Existing routing allow/deny scenarios passed with the extracted script
(16 cases).
- actionlint passed with the dedicated custom runner label declared in
its local configuration.
- git diff --check passed.
- Full pnpm typecheck/test/build cannot run in this worktree because
dependencies are not installed. The change is limited to workflow YAML
and Node tests.
- The dedicated pool is deployed with two warm On-Demand workers and a
four-worker ceiling. The routing qualification passed: it started about
five seconds after workflow creation, selected AWS, and finished while
the hosted gate in the same PR remained queued.
- Bootstrap https://github.com/paperclipai/paperclip/pull/16039 is
merged. The deployed group now authorizes only ci-routing.yml on
protected master, and its master canary passed.
## Risks
- The maintainer explicitly approved a queued-CI admin bypass after
current-head Greptile 5/5 and focused verification.
- Activation requires the deployed routing Fleet and its group to
authorize the routing workflow on protected master. Keep the bootstrap
dependency explicit.
- The group must authorize only ci-routing.yml on protected master and
canonical repository ID 1170821064.
- The routing workflow runs no checkout or PR code and has read-only
permissions. The test admission checks are unchanged.
## Model Used
- OpenAI GPT-6 via Codex, with reasoning, shell tools, and file editing.
The exact runtime model ID and context window are not exposed in this
session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused routing and
workflow tests; full pnpm commands are blocked by missing worktree
dependencies)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] Required CI status checked; the maintainer explicitly approved
admin bypass for queued CI after clean current-head review and focused
verification
- [x] Greptile is 5/5 on
|
||
|
DottaandPaperclip
|
8e7b5db822 |
ci: add protected routing workflow for reserved AWS capacity (#16039)
## Thinking Path > - Paperclip uses CI to verify changes before release. > - Trusted PR tests can run on isolated AWS workers. > - Their runner-selection job currently waits for GitHub-hosted capacity. > - A large hosted test queue therefore blocks admission to AWS. > - A dedicated routing workflow lets us reserve capacity for metadata validation. > - This PR adds that workflow without changing the active PR caller. ## Linked Issues or Issue Description **Issue type** CI reliability enhancement. **Where is the issue?** The Select trusted runner job in PR CI. **What's wrong?** The routing job competes with a large hosted test queue. Its sampled median wait was about 20 minutes. AWS tests cannot begin before this job finishes. **Suggested fix** Add a protected reusable routing workflow. Deploy its dedicated AWS pool, then switch the PR caller in a follow-up PR. No overlapping open PR or issue was found. ## What Changed - Add ci-routing.yml with the existing author, sender, rerun actor, repository, head, merge, and stack checks. - Target the dedicated paperclip-ci-routing-x64 Fleet. - Export runner and full_ci decisions to reusable workflow callers. - Add tests for metadata-only execution, reserved placement, and output forwarding. - Keep the active PR caller unchanged until the pool is deployed. ## Verification - The routing tests passed (30 tests), including 20 executed runner-selection cases and 8 executed stack-scope cases. - Existing routing allow/deny scenarios also passed with the extracted script (16 cases). - A live qualification job selected AWS and passed on the reserved routing pool in https://github.com/paperclipai/paperclip/actions/runs/38308312153/job/115007621128 while the hosted gate remained queued. - actionlint passed with the dedicated custom runner label declared in its local configuration. - git diff --check passed. - Full pnpm typecheck/test/build cannot run in this worktree because dependencies are not installed. The change is limited to workflow YAML and Node tests. - After merge, configure the restricted runner group, deploy the pool, and dispatch ci-routing.yml on master. The unused workflow does not affect current PR placement. ## Risks - The maintainer explicitly approved merging this rollout after current-head Greptile 5/5 and focused verification, while the hosted CI checks remain queued. - The dedicated Fleet must exist before the workflow can receive a runner. This PR introduces the workflow without switching active PR CI. - The group must authorize only ci-routing.yml on protected master and canonical repository ID 1170821064. - The routing workflow runs no checkout or PR code and has read-only permissions. The test admission checks are unchanged. ## Model Used - OpenAI GPT-6 via Codex, with reasoning, shell tools, and file editing. The exact runtime model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused routing and workflow tests; full pnpm commands are blocked by missing worktree dependencies) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] Required CI was checked; the maintainer explicitly approved an admin bypass for queued checks after clean review - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |