mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `AGENTS.md` tells every contributor and agent to read `doc/DEVELOPING.md` for setup, commands, testing, and worktrees > - That file grew to 5,495 lines. Most of it was notes on how one subsystem behaves, added one pull request at a time > - Readers could not find the commands. Agents also read behavior notes as rules > - This pull request cuts `doc/DEVELOPING.md` to a 660-line setup and workflow guide. It moves the subsystem notes to focused docs > - The benefit is a guide that a person or an agent can read in full, and a clear place for each kind of note ## Linked Issues or Issue Description **Issue type** Outdated or confusing documentation. **Where is the issue?** `doc/DEVELOPING.md` **What's wrong?** The file was 5,495 lines. It opened with 112 lines about case handoffs before it reached Prerequisites. Unrelated sections sat below the "OpenClaw Docker UI One-Command Script" heading. Statements about current behavior read like policy. **Suggested fix** Keep setup, commands, and workflow in `doc/DEVELOPING.md`. Move each subsystem note to a doc for that subsystem. Add an index. ## What Changed - `doc/DEVELOPING.md` is now 660 lines: prerequisites, start dev, health checks, local run, test drive, Docker, instance layout, database, storage, secrets, worktrees, test commands, Storybook, CLI, and an index of the other docs. - 15 new docs hold the moved text without wording changes: `ci-workflows`, `test-infrastructure`, `worktrees`, `hot-restart-deploys`, `heartbeat-internals`, `runner-internals`, `workspace-repositories`, `workspace-storage`, `skills-and-teams-catalog`, `ui-internals`, `case-handoffs`, `chat-connector-development`, `instance-configuration`, `integration-api-notes`, `database-backups`. - "Publish a branch Storybook" moved to `doc/STORYBOOK-DEPLOYMENT.md`. The privacy rollout notes moved to `doc/ISSUE-PRIVACY.md`. - Deleted about 280 lines: the story-by-story Storybook notes, "Agent Artifact Uploads" (a copy of `doc/AGENT-ARTIFACTS.md`), two pointer-only sections, "Workspace feature checks (2026-10-09)", and "Chat and task navigation caching". The test-drive build-lock paragraph is now four lines. - Repointed every inbound link and every script or workflow comment that named a moved section. - `AGENTS.md` now says where a behavior note goes, so the guide does not grow again. ## Verification - A script split the file by line range. It asserts that each of the 5,495 source lines is kept, moved, or dropped exactly one time. - A link check on every changed Markdown file found 0 missing files and 0 missing anchors. - `node scripts/check-maps.mjs` passes. - `node --test .github/scripts/tests/*.test.mjs`: 808 pass, 0 fail. - `node --check` passes on the three scripts with changed comments. - `pnpm verify` was not run. This worktree has no installed dependencies, and the change has no runtime code. ## Risks - Low risk. The change is documentation and comments only. - A link to a moved section from outside this repository will land on the top of `doc/DEVELOPING.md`. - The new docs are not in the audited prose list in `.github/scripts/ci-inputs.mjs`. An edit to one of them runs full CI. An edit to `doc/DEVELOPING.md` does not. This PR does not change that list. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Claude Code desktop, with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [ ] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
434 lines
27 KiB
Markdown
434 lines
27 KiB
Markdown
# CI workflows
|
|
|
|
How pull-request CI is routed and trusted, who owns the lockfiles, and how the typecheck and retry lanes work.
|
|
|
|
## Dependency Lockfile Policy
|
|
|
|
GitHub Actions owns every managed pnpm lockfile:
|
|
|
|
- `pnpm-lock.yaml` at the repository root
|
|
- `scripts/typecheck-sandbox-providers/pnpm-lock.yaml`, for the sandbox providers
|
|
- `packages/plugins/sandbox-providers/daytona/pnpm-lock.yaml`
|
|
|
|
`LOCKFILES` in `.github/scripts/lockfile-trust.mjs` lists them.
|
|
|
|
- Do not commit any of them in pull requests. Only the lockfile refresh workflow
|
|
and verified Dependabot commits may change them.
|
|
- Pull request CI validates dependency resolution when manifests change. A stale
|
|
lockfile is resolved in place for that run.
|
|
- Pushes to `master` run `.github/workflows/refresh-lockfile.yml`. It regenerates
|
|
each lockfile with `pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile`
|
|
and opens one `chore/refresh-lockfile` pull request for all of them. See
|
|
`doc/RELEASE-AUTOMATION-SETUP.md` for when the bot merges it.
|
|
- A local `docker build` installs the Runner with a frozen lockfile. When the
|
|
root `pnpm-lock.yaml` is behind the Runner's manifests, run that same command
|
|
in the checkout before the build, and do not commit the result. See "Runner
|
|
image" in `doc/DOCKER.md`.
|
|
|
|
The Pi and Kimi distribution lockfiles under `packages/paperclip-runner/scripts/`
|
|
are not on this list. They hold attested provider pins, so change them by hand
|
|
(see `rules/acpx-profile-identity.mdc`).
|
|
|
|
## Trusted PR Workflow
|
|
|
|
The PR caller uses `paperclipai/paperclip/.github/workflows/pr-trusted.yml@master`.
|
|
The AWS runner group `paperclip-public-pr` must allow
|
|
`paperclipai/paperclip/.github/workflows/pr-trusted.yml@refs/heads/master`.
|
|
New workflow versions merged into master then receive runner access without a
|
|
separate SHA allowlist update. Dependabot leaves this first-party reference on
|
|
master.
|
|
|
|
Runner selection calls `ci-routing.yml@master`. Allowlisted maintainers use the
|
|
reserved `paperclip-ci-routing-x64` AWS fleet for this bounded metadata job;
|
|
other callers use GitHub-hosted capacity. The gate never checks out or executes
|
|
PR code. The author, event sender and rerun actor must each be both
|
|
in `AWS_CI_TRUSTED_USER_IDS` and a current owner of `.github/**` in CODEOWNERS.
|
|
The resolver and CODEOWNERS are read from the same protected master revision;
|
|
PR edits cannot change the trusted owner list. Documentation-only owners,
|
|
unsupported owner rules and failed identity lookups do not authorize AWS.
|
|
Repository, head and merge validation also apply. The gate validates this run's
|
|
immutable event merge, including its PR head and a base snapshot between the
|
|
event's base and the current branch tip. Normal base advancement and regeneration
|
|
of GitHub's live merge ref do not move an authorized run to GitHub-hosted CI.
|
|
Stacked synthetic bases must contain a validated base snapshot. Rewritten base
|
|
history, changed PR heads and unverifiable ancestry still deny AWS execution.
|
|
Missing or malformed settings
|
|
and unverifiable authorization use GitHub-hosted runners. A maintainer rerunning
|
|
an outside contributor's PR does not grant it AWS access.
|
|
Both `full-ci` and `profile-ci` label events use this same authorization path.
|
|
The labels request CI scope; they do not grant AWS access or move an authorized
|
|
CODEOWNER run to GitHub-hosted capacity. Other label events do not request CI.
|
|
Authorized test jobs and the final `verify` and `e2e` aggregates use the selected
|
|
AWS fleet. A stale admission roster can allocate only the short metadata gate;
|
|
the fresh CODEOWNERS check still denies PR execution for a removed owner.
|
|
|
|
AWS runners are sometimes lost mid-job. `ci-runner-loss-rerun.yml` re-runs the
|
|
failed jobs of a PR's first CI attempt once when every failed or cancelled job
|
|
lost its runner. A spot reclaim often ends the job, and the run, as
|
|
`cancelled`. A lost runner is recognized by the check-run annotation "lost
|
|
communication with the server" or "received a shutdown signal". When a job has
|
|
the annotation "The operation was canceled." or "A task was canceled.", the job
|
|
log is checked for the same messages. This evidence counts even if the job also has other error
|
|
annotations, so a test that really failed just before a reclaim gets one
|
|
wasted re-run. A cancelled job without this evidence, such as a job a person
|
|
cancelled, counts as a real failure. The `verify` and `e2e` aggregates are not
|
|
counted. Any other failure, a failed or cancelled routing gate, or a PR whose
|
|
head has moved prevents the automatic re-run. The head check runs before any
|
|
job is classified, so a run cancelled by a newer push costs no log reads. It
|
|
runs again just before the re-run, because a re-run of an old run joins the
|
|
PR's `cancel-in-progress` concurrency group and would cancel the newer CI. The
|
|
workflow job summary lists each failed or cancelled job and its evidence. Like a maintainer's "Re-run failed jobs", the re-run reuses the
|
|
gate's successful outputs, including its AWS admission. Attempt 2's
|
|
triggering actor is `github-actions[bot]`, so `profile-ci` does not export
|
|
that attempt. Later attempts are never re-run automatically.
|
|
|
|
The routing workflow also selects the CI scope. Middle PRs in a stack and PRs
|
|
whose diff changes only release notes (`releases/v*.md` or
|
|
`releases/beta/v*.md`) skip the build, test, and e2e lanes. The `policy` job
|
|
and the `verify` and `e2e` aggregate checks still run. The release-notes check
|
|
reads the diff between the event's base and head commits. If that lookup
|
|
fails, or the diff is empty or may be truncated, the PR runs full CI. A
|
|
notes-only PR stacked above other PRs also runs full CI, because its own diff
|
|
does not include the lower PRs' code.
|
|
|
|
For ordinary full-scope PRs, the routing job further selects affected suites
|
|
using `turbo.json` and `.github/ci-task-matrix.json`. It downloads the planner
|
|
and its helpers and definitions from one canonical master revision, without executing PR
|
|
code. It fetches the exact tested merge commit and its parents into a temporary
|
|
bare Git repository, validates the PR head, and compares the merge tree against
|
|
its first parent. The event's base SHA can be stale. Git tree comparisons have
|
|
no GitHub compare-API 300-file cap. Renames count as deletion plus addition;
|
|
failed, empty, or malformed comparisons select the full suite. Unclassified files also select full CI. Stack verification retains the
|
|
existing scope policy and runs all suites whenever that policy requires them.
|
|
Changes to global inputs, including the task graph, package scripts, workflows,
|
|
and execution helpers, force full execution and disable result-cache restore and
|
|
save for the entire PR diff. Later pushes cannot reuse results created under
|
|
changed cache definitions.
|
|
|
|
The repository variable `CI_INCREMENTAL_MODE` accepts `enabled` (the default),
|
|
`shadow` (report selections but execute everything), or `full`. Add the `full-ci`
|
|
label to force all checks on one PR. Authorized `profile-ci` runs also execute
|
|
everything. Forced and shadow runs disable local and remote task-result cache
|
|
reads and writes. Every routing run
|
|
with a valid plan publishes affected and scheduled job counts with input reasons
|
|
in its job summary. Required aggregates reject failed, cancelled, and unexpectedly
|
|
skipped selected suites; the verify aggregate includes serialized tests and the
|
|
canary dry run.
|
|
The uncached typecheck job also runs the repository-wide CLI guidance safety
|
|
guard. It still scans UI changes when unrelated server shards are omitted.
|
|
|
|
The shared policy in `.github/scripts/ci-inputs.mjs` exempts only audited prose:
|
|
`doc/DEVELOPING.md`, `doc/DATABASE.md`, and Markdown under `doc/plans/`.
|
|
The first two still select the uncached typecheck/guidance lane; historical plans
|
|
need only the always-running policy checks. These inputs do not widen product
|
|
test selection or enter product task cache hashes. All other docs, bundled
|
|
skills, fixtures and unknown paths keep their existing conservative coverage.
|
|
A source change plus exempt prose remains graph eligible. When adding a reader
|
|
of exempt prose, update this policy as well as the task inputs.
|
|
|
|
For ordinary source PRs (including the audited prose inputs above), a separate `Select Vitest files` job refines the
|
|
general and serialized test lanes before their runners are allocated. It runs
|
|
on the gate-approved test fleet, loads the real Vitest projects, and follows
|
|
imports across workspace packages. Every project in the coarse plan is assessed,
|
|
including unchanged consumers. Projects without affected tests drop out of the
|
|
matrix; an unchanged directory alone does not prove independence. Filesystem, subprocess, network, CommonJS,
|
|
dynamic-loader, unresolved, and unapproved external-package consumers are
|
|
retained conservatively, including indirect consumers through local helpers.
|
|
Module-relative filesystem reads add explicit dependency edges when the path and
|
|
read API can be resolved without running code. This includes directory listings
|
|
and transitive helper reads. Dynamic paths, writes, escaped filesystem APIs,
|
|
mutable or shadowed bindings, generated inputs and paths outside the checkout
|
|
remain opaque. Setup-file reads invalidate the whole project when their input
|
|
changes. Object-valued constants, including URL objects, remain opaque even without
|
|
an observed mutation; inline URL construction and immutable string constants can
|
|
be resolved. Symlink reads track both the checked-in link and its physical target.
|
|
Missing inputs retain coverage.
|
|
The selector explicitly tracks setup/configuration dependencies as well as test
|
|
imports. Ordinary source names such as `session-config-metadata.test.ts` or
|
|
`setup-state.ts` remain eligible; the words `config` and `setup` alone do not
|
|
force full CI. Conventional Vite/Vitest config and setup files remain broad
|
|
inputs. Other resolved setup/config dependencies invalidate their entire project.
|
|
The small pure-dependency allowlist lives in
|
|
`scripts/ci/vitest-selection-graph.mjs`; new external packages remain opaque
|
|
until reviewed. This is deliberately broader than `vitest related` alone.
|
|
The reviewed Drizzle SQL/schema entry points have no ambient repository inputs.
|
|
Their full import closures are content-hashed in
|
|
`scripts/ci/reviewed-test-dependencies.mjs`; changed installed contents revoke the
|
|
exception. CommonJS and dynamic loaders reject a fingerprint rather than leave
|
|
unhashed children outside the closure. Database drivers and migration readers remain opaque. OS information,
|
|
in-memory streams, timers and compression do not read repository files themselves;
|
|
filesystem or process operations using their outputs retain their normal guards.
|
|
The syntax audit distinguishes data keys and erased types from runtime globals.
|
|
It examines the called member, not string arguments earlier in a chained call.
|
|
For conservative retention, explicit complete local mock factories cut the
|
|
mocked implementation's I/O edges. Partial mocks, spies, automocks, mock bypasses,
|
|
and dynamic or escaped mocking APIs do not. Edits to a mocked module still use
|
|
Vitest's unmodified related-file graph. Non-isolated projects run in full.
|
|
Literal Vite globs use Vite's expanded dependency edges, including newly added
|
|
members. Consumers of generated or untracked modules remain in the conservative
|
|
set because the import graph does not describe their build inputs.
|
|
The server's temporary-home and Supertest setup has a reviewed, content-hashed
|
|
exception. It still follows local helper imports. Any change to the reviewed
|
|
contents disables the exception until reviewed again; setup dependency changes
|
|
invalidate the whole project. Unreviewed setup I/O uses the full task plan.
|
|
|
|
Selected files are repartitioned within their existing owners using the current
|
|
per-runner budgets and server timing weights. Small selections allocate fewer
|
|
jobs; empty lanes allocate none. Serialized files remain serialized, and chat's
|
|
line shards, native Runner lanes, Rust, builds, typechecking and E2E retain their
|
|
existing selection/execution rules. The job summary reports files and jobs
|
|
selected per lane, and the `vitest-selection` artifact records the manifest and
|
|
dependency audit, including file-read edges and the first opaque dependency that
|
|
retained each test. Selected jobs execute exact Vitest test specifications, so
|
|
similarly named `.test.ts` and `.test.tsx` files cannot overlap between shards.
|
|
The wrapper consumes and validates the selection before launching Vitest, then
|
|
removes selection environment variables from the test process. Nested runner
|
|
fixtures therefore use their own checkout and test inventory.
|
|
|
|
Missing/deleted/renamed source files, non-exempt non-source inputs, unrecognized graph inputs,
|
|
setup/config changes, failed dependency installs, graph errors and timeouts use
|
|
the complete original task plan. Source mutations during dependency setup also
|
|
disable narrowing. A published manifest must match the checked-out
|
|
merge SHA and parent diff, its plan hash and the expected shard/project ownership. Missing or
|
|
invalid manifests fail selected jobs and the required aggregate; they cannot
|
|
turn reduced matrices green. Failed-job retries download the original artifact
|
|
ID saved with the plan, even when GitHub increments the run attempt. Turbo hashes the selected file assignment through
|
|
`PAPERCLIP_CI_SELECTION_HASH`, so a passing subset cannot satisfy a full run or a
|
|
different selection.
|
|
|
|
Graph discovery runs at most two isolated Node processes concurrently. Each
|
|
project retains its own Vite configuration, aliases and module graph. Any worker
|
|
error or timeout rejects the whole refinement and retains the original task plan.
|
|
The selector checkout needs only the merge commit and its two parents, not the
|
|
repository's complete history.
|
|
|
|
Server test costs are recorded in CI with tests, hooks, collection, setup and
|
|
environment preparation included. The weekly **Refresh CI test durations**
|
|
workflow opens a timing-only PR. It reads the general server weights from the
|
|
`General tests (server)` jobs of the two newest successful `PR` runs of merged
|
|
PRs, pinned to each PR's final head. Each run must have the full shard count in
|
|
`.github/ci-task-matrix.json` and cover at least 90% of the lane, so a run that
|
|
selected a few files does not count. These jobs run files three at a time, and
|
|
their contention changes relative file costs. Measured against two other PR
|
|
runs, Cloud readiness weights placed the slowest of 12 shards at 1,748-1,770 s of
|
|
summed file time, against a 1,595 s mean; PR-run weights placed it at 1,684-1,702 s. The
|
|
serialized weights still come from two successful protected-master Cloud
|
|
readiness runs. It never executes code from logs, changes test selection rules,
|
|
or grants AWS access. GitHub may require a maintainer to approve the bot PR's
|
|
workflow runs; the refresh never merges itself. An outstanding refresh PR pauses
|
|
new refreshes. Both timing manifests include source runs and measurement coverage.
|
|
The refresh requires at least 90% coverage of each current lane and uses median
|
|
measurements; missing files retain their old weights. Historical logs can bootstrap
|
|
test/hook costs until structured loading/setup measurements are available.
|
|
|
|
To reproduce a refresh from a full checkout with GitHub Actions read access:
|
|
|
|
```sh
|
|
node scripts/ci/refresh-test-durations.mjs [master-run-id ...]
|
|
node --test scripts/ci/test-selection-timing*.test.mjs
|
|
```
|
|
|
|
Set the repository variable `CI_VITEST_SELECTION=off` to disable file selection
|
|
while keeping the existing task-level optimization. Empty or `enabled` enables
|
|
it; other values disable it. `full-ci`, profiling, and task-level `full`/`shadow`
|
|
modes also bypass file selection. Qualification uses real Vitest graphs and real
|
|
Turbo cache entries, including a filesystem contract omitted by the raw graph:
|
|
|
|
```sh
|
|
node --test scripts/ci/test-selection*.test.mjs scripts/ci/cache.integration.test.mjs
|
|
```
|
|
|
|
General and serialized test suites and the standalone Rust suite use Turbo's
|
|
content-addressed cache. An execution adapter maps selected lanes to the existing
|
|
package scripts and their declared inputs. It selects tracked and non-ignored
|
|
source files, so generated build outputs do not invalidate later task results.
|
|
The test scripts keep responsibility
|
|
for their build prerequisites. GitHub Actions stores
|
|
these results in the PR's own cache scope, so a later push can reuse unchanged
|
|
tasks without giving PRs write access to master caches. Input hashes include
|
|
the relevant source trees, fixtures, scripts, manifests, resolved lockfile,
|
|
environment, toolchain, and checkout path. Unclassified tracked file contents
|
|
also enter the execution key, including changes inherited from a newer base.
|
|
Revision-stamped builds, registry smoke tests, browser suites, and Runner lanes
|
|
that perform integration checks retain their original uncached commands. The
|
|
post-merge release verification workflow continues to execute the full suite.
|
|
|
|
Inspect a proposed selection locally with `node .github/scripts/ci-plan.mjs origin/master HEAD`.
|
|
Run the planner and workflow-contract tests with
|
|
`node --test .github/scripts/tests/ci-plan.test.mjs`, and the real cache proof
|
|
with `node --test scripts/ci/cache.integration.test.mjs` after installing dependencies.
|
|
The separate Incremental CI qualification workflow exercises candidate changes
|
|
on GitHub-hosted runners before the trusted master workflows adopt them.
|
|
|
|
When a suite starts reading another package, asset, or runtime prompt, update
|
|
its inputs in `turbo.json`. This includes filesystem reads as well as imports.
|
|
Server tests inspect selected UI files, shared tests inspect UI logos, and Runner
|
|
static/Vitest lanes include server integration tests. Keep those dependencies
|
|
explicit. When changing shard layout, update `.github/ci-task-matrix.json` and
|
|
the full fallback matrices in `pr-trusted.yml`; the contract tests require them
|
|
to agree. Add new cache-affecting environment settings to `globalEnv`.
|
|
|
|
The `paperclip-ci-routing` runner group must allow
|
|
`paperclipai/paperclip/.github/workflows/ci-routing.yml@refs/heads/master`.
|
|
Keep `AWS_CI_ROUTING_USER_IDS_SOURCE` equal to the exact JSON value of
|
|
`AWS_CI_TRUSTED_USER_IDS` and `AWS_CI_ROUTING_USER_IDS` as its delimiter-wrapped
|
|
numeric IDs (for example, `|34892728|139239|`). Missing or mismatched admission
|
|
settings use GitHub-hosted capacity for the gate. Current CODEOWNERS remain
|
|
authoritative for PR execution regardless of these admission settings.
|
|
A maintainer's manual dispatch of `ci-routing.yml` on master exercises the AWS
|
|
metadata gate without launching the test matrix; its non-PR event returns the
|
|
hosted test-runner output.
|
|
|
|
GitHub starts a limited number of hosted jobs per hour for this repository, and
|
|
every hosted job waits in one first-come queue. Two settings keep community PRs
|
|
from delaying CODEOWNERS work in that queue:
|
|
|
|
- A PR run on GitHub-hosted runners queues at most four jobs per test matrix
|
|
(`max-parallel` in `pr-trusted.yml`). It runs the same jobs; the rest wait
|
|
inside the run instead of in the shared queue. AWS runs are not limited.
|
|
- With `AWS_CI_SHORT_JOBS_ENABLED` set to `true`, the `commitperclip PR Review`
|
|
job and the `PR merge review notification` job for a CODEOWNER's own
|
|
same-repository PR use the reserved `paperclip-ci-routing-x64` fleet. Both
|
|
run only master's scripts, never PR code, and use the same admission
|
|
variables as the gate. Check-run and commit-status notifications carry no
|
|
verifiable author and stay GitHub-hosted, as does every community PR.
|
|
Before setting the variable, allow
|
|
`paperclipai/paperclip/.github/workflows/commitperclip-review.yml@refs/heads/master`
|
|
and
|
|
`paperclipai/paperclip/.github/workflows/pr-merge-review.yml@refs/heads/master`
|
|
in the `paperclip-ci-routing` runner group. A job routed to a group that
|
|
does not allow its workflow stays queued.
|
|
|
|
Keep the `.github/**` rule in `.github/CODEOWNERS` and the active master ruleset's
|
|
code-owner review requirement enabled. This covers the caller, the trusted
|
|
workflow, and CODEOWNERS itself. Existing administrator pull-request bypasses
|
|
remain governed by the repository ruleset.
|
|
|
|
When changing the workflow path or branch, authorize the new reference before
|
|
updating the caller. Retain older authorized SHA references while queued runs or
|
|
supported reruns still use them.
|
|
|
|
General server shards run at most three test files at a time in separate Vitest
|
|
forks. Each file retains its own home, config and process state, and its test
|
|
cases run serially. An explicit `PAPERCLIP_*_TEST_DATABASE_URL`, including
|
|
`PAPERCLIP_TEST_DATABASE_URL`, keeps those shards serial because the database
|
|
is shared. Native Runner suites use four isolated file workers. Their long
|
|
lease, preparation, disk-loss and checkpoint tests have separate files so real
|
|
observation windows can overlap. Issue recovery suites split by producer and
|
|
recovery behavior for the same reason. Unsharded local tests and route/authz
|
|
lanes retain their existing file concurrency. Maintenance server cases overlap
|
|
two owned server/database fixtures within their file; an explicit shared
|
|
maintenance database keeps them serial. Sharded chat jobs run
|
|
three independent partitions, each with its own embedded PostgreSQL, home and
|
|
config; an explicit shared test database retains one partition per job. The
|
|
partitions keep each source line whole and balance by the recorded time in
|
|
`scripts/chat-shard-durations.json`; entries without a record take the median.
|
|
Collected-case counts were a poor proxy because one `it.each` line can run many
|
|
cases: one of six count-balanced partitions recorded 532 s of test time against
|
|
a 377 s mean. To refresh the weights, add the `profile-ci` label to a PR, download
|
|
the `ci-profile-*` artifacts of its three chat jobs into one directory, and run
|
|
`node scripts/ci/refresh-chat-shard-durations.mjs --source=<run URL> <directory>`.
|
|
Stale weights change only the balance, never which tests run. A chat job
|
|
collects the suite once, then validates its three partitions' filters at the
|
|
same time; each partition still checks that its filters select exactly its
|
|
assigned tests.
|
|
The Runner transport lease-expiry and cold-admission regressions live in their
|
|
own file so those real-clock waits overlap other files in the existing pool.
|
|
|
|
Exact-source release verification uses the post-merge AWS fleet on the first
|
|
attempt. Reruns of its verification and chaos jobs use GitHub-hosted runners.
|
|
Exact-source cloud-migrator planning and package builds use the same policy.
|
|
This gives interrupted jobs an independent recovery path. It keeps the same
|
|
source, commands, assertions and required gates. A failed test still needs a fix;
|
|
a successful rerun does not explain an earlier failure.
|
|
|
|
The PR Runner Vitest jobs separate package tests from native server tests.
|
|
Three `--package-lane=K/3` jobs each run one duration-balanced partition of the
|
|
package inventory, and one `--native-server` job runs the complete native server
|
|
group. `run-pr-vitest-lane.mjs` asks `vitest list` for the inventory, so it
|
|
follows the `vitest.config.ts` includes. It partitions the files with the general
|
|
server partitioner, using the weights in
|
|
`packages/paperclip-runner/scripts/vitest-lane-durations.json`. Files missing
|
|
from that manifest get the median weight. The legacy `--shard=1/2` and
|
|
`--shard=2/2` commands keep their earlier PR behavior: the first runs the whole
|
|
package inventory and the second runs the native group. Other workflow names and
|
|
shard counts keep their existing coverage. Every lane still prepares debug
|
|
binaries. A separate exact-key debug cache holds
|
|
the workspace library and every Cargo binary target. Its key includes source,
|
|
protocol and fixture content, Cargo configuration, compiler flags, toolchain
|
|
and checkout path. Only the Rust lane of a verified master push saves it.
|
|
PR jobs only restore it; a missing or incomplete entry rebuilds normally.
|
|
|
|
Affected-test selection treats Vite's exact `?raw` imports as file inputs.
|
|
Changing that file selects its consumers. Generated/untracked files and unknown
|
|
query loaders retain their consumers conservatively. Other filesystem, process
|
|
and network effects keep their existing fallback.
|
|
|
|
One in ten CI CODEOWNER PR runs sends its job, step and outside-check timings to
|
|
Honeycomb. For a full wall-clock profile of all main PR CI jobs, add `profile-ci` to the PR.
|
|
See [CI profiling](ci-profiling.md) for the Honeycomb waterfall, coverage,
|
|
operator setup, and activation requirements.
|
|
|
|
## PR Vitest retry
|
|
|
|
PR CI retries a failing Vitest case once, so a single contention flake does not
|
|
force a re-run of the whole workflow. `scripts/ci/vitest-retry.mjs` owns the gate.
|
|
The default is one retry when `GITHUB_ACTIONS=true` and `GITHUB_WORKFLOW=PR`.
|
|
Reusable workflows inherit the caller's name, so `pr-trusted.yml` gets the retry
|
|
through `pr.yml`. Release, release verify, cloud readiness, master and local runs
|
|
keep zero retries. `PAPERCLIP_VITEST_RETRY` overrides the default in any context.
|
|
For example, set the `CI_VITEST_RETRY` repository variable to `0` to turn the retry
|
|
off, or set `PAPERCLIP_VITEST_RETRY=1` locally to reproduce the CI behavior.
|
|
|
|
`scripts/run-vitest-stable.mjs` passes `--retry` to every `vitest run`. This
|
|
covers the general, workspace, chat, native Runner and serialized lanes and the
|
|
selected-file path. The Runner package config sets `retry` itself, because its
|
|
lane runs Vitest directly. A test that passes only on a retry is still a flake.
|
|
`scripts/ci/retry-reporter.mjs` adds a `Flaky test passed on retry` warning
|
|
annotation for each such test, and a row to the step summary with the file, test,
|
|
retry count and job. Fix a test that this reporter reports. Do not treat the
|
|
retry as the fix. Playwright retries stay at zero.
|
|
|
|
## Typecheck lane in CI
|
|
|
|
The required `Typecheck + Release Registry` job runs one command:
|
|
|
|
```sh
|
|
node scripts/ci-typecheck-lane.mjs [--concurrency <n>]
|
|
```
|
|
|
|
It first runs the steps that write shared outputs, one at a time:
|
|
|
|
1. Link the workspace packages and build the plugin SDK.
|
|
2. Build the server with `PAPERCLIP_TSC_EMIT_ONLY=1`, then check that the
|
|
server runtime assets are in `dist`. The server build no longer checks types
|
|
here: the server layers check the same files, and the `Build` job runs the
|
|
full `pnpm build`. The full program peaked at about 13.5 GiB, which left this
|
|
16 GiB runner swapping.
|
|
3. Install the sandbox providers for the test programs
|
|
(`typecheck-test-programs.mjs --prepare-only`).
|
|
4. Typecheck the Runner. Its typecheck rebuilds the eval kernel, which other
|
|
checks read, so it cannot run beside them.
|
|
5. Run the CLI guidance test. It runs here, before the parallel checks, because
|
|
one of its cases took 17 seconds beside them against a 15 second timeout.
|
|
|
|
Then it runs the remaining checks in parallel, as many as the machine has CPUs
|
|
(default) or `--concurrency`. Each `tsc` uses one thread. The server
|
|
declaration layers start first, because the test layers and the test projects
|
|
read them. The test programs and the other build-gap workspaces run at a lower
|
|
CPU priority, so they use only the CPU that the server layers and the release
|
|
registry tests leave free.
|
|
|
|
A passing check prints its output in a collapsed log group. A failing check
|
|
prints its output in full, between `=== FAILED ...` and `=== end of ...`
|
|
lines, with an error annotation. When the server declarations fail, the test
|
|
layers and test projects are reported as skipped. The lane ends with a table
|
|
of each check's start time, duration and result. If a preparation step fails,
|
|
nothing else runs.
|
|
|
|
To add a check to the job, add it to `planLane` in the script.
|
|
|
|
PR runs take the workflow from `master` but check out the PR's merge commit. A
|
|
stacked PR, whose base is another branch, can have a merge commit without this
|
|
script. The job then runs the six earlier commands one after another instead.
|