28 Commits
Author SHA1 Message Date
1d23cb6962 ci: pin a checkout-independent Rust cache path so PR lanes hit master's cache (#14394)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip ships a native Runner binary, written in Rust, and seven
CI lanes build it on every pull request
> - `Canary Dry Run` is the slowest check on every green PR run, and
most of its time is `cargo build --release` on third-party crates
> - Master saves a Rust dependency cache for these lanes, but every PR
lane logs `No cache found` and compiles every crate from zero
> - The cache key matches, but GitHub also compares a hash of the
absolute cache paths, and the master writer (RunsOn fleet,
`/home/runner/_work/...`) and the PR readers (GitHub-hosted,
`/home/runner/work/...`) hash different paths
> - This pull request gives both sides a checkout-independent workspace
path, so the hashes match and the PR lanes restore master's cache
> - The benefit is about 2.5 minutes less wall clock per PR run and
about 18 fewer runner-minutes per run

## Linked Issues or Issue Description

No public issue exists for this problem. The description below follows
the enhancement template.

Related prior PRs on the same cache: Refs #13194, Refs #13259, Refs
#13457, Refs #13459, Refs #13500, Refs #13586. None of them pins the
workspace path, so none of them fixes this miss.

**What existing behavior does this improve?**

The `Swatinem/rust-cache` restore step in the PR workflow lanes that
build the Runner: `Canary Dry Run`, `Build`, `Typecheck + Release
Registry`, and the four `Verify Paperclip Runner` lanes.

**Subsystem affected**

CI workflows under `.github/workflows/`, their guard tests under
`.github/scripts/tests/`, and `doc/RELEASE-AUTOMATION-SETUP.md`.

**Current behavior**

Every PR lane logs `No cache found` although master holds an entry with
the exact key. Run 36424309181 computed
`v0-rust-release-runner-v1-Linux-x64-c3a3ca66-a95b0328`, and master
holds a 678 MB entry with that key. GitHub matches a cache entry on the
key and on a version hash of the absolute paths in the cache. The master
writer runs on the RunsOn fleet, where the checkout is
`/home/runner/_work/paperclip/paperclip`. The PR readers run on
GitHub-hosted `ubuntu-latest`, where the checkout is
`/home/runner/work/paperclip/paperclip`. The stored version `5c40870d…`
is the sha256 of the `_work` paths plus `zstd-without-long|1.0`. The
`work` paths hash to `1656e9ee…`. The key can never match, so each lane
compiles every third-party crate again.

**Proposed behavior**

The writer and the readers pass the same checkout-independent path to
`rust-cache`. Both runner layouts then produce the same version hash,
and the PR lanes restore master's cache.

**Reason and benefit**

`Canary Dry Run` takes 533s on a green run. 251s of that is dependency
compilation that a warm cache removes. Seven lanes pay this cost in
every PR run.

**Breaking changes**

None. This change affects CI only.

## What Changed

- Add a `Pin the Runner Rust workspace path` step before `rust-cache` in
the master writer (`release-verify.yml`, typecheck and runner lanes) and
in all four PR readers (`pr-trusted.yml`). The step creates the symlink
`$HOME/paperclip-runner-rust` →
`$GITHUB_WORKSPACE/packages/paperclip-runner/runner` and passes that
path to `rust-cache` as `workspaces: <path> -> target`. `rust-cache`
resolves the input with `path.resolve`, which does not follow symlinks,
so both runner layouts now produce the same cache paths and the same
version hash. `$HOME` is `/home/runner` on both images, which is why the
`~/.cargo` paths already agreed.
- Bump the shared keys `release-runner-v1` → `release-runner-v2` and
`release-typecheck-v1` → `release-typecheck-v2`. The old, unreachable
entries are then visibly orphaned instead of sharing a key with the new
ones.
- Extend the guard tests `pr-runner-rust-cache`, `release-runner-cache`,
and `typecheck-rust-cache`. They now require the pin step in both
workflows with identical text, placed before the cache step, and they
reject a `workspaces:` value that resolves under the checkout. The
`pr-runner-rust-cache` test checks all four PR reader jobs and fails if
a `rust-cache` step appears in a PR job that is not in its reader list.
- Update `doc/RELEASE-AUTOMATION-SETUP.md` to name the
`release-runner-v2` key and to explain the pinned workspace path.

### Expected savings once merged

Measured from run 36424309181. "Removed" is the dependency-compile time
that a warm restore removes, minus about 18s to restore the 680 MB
entry. The fleet writer's own restore shows this cost.

| Lane | Today | Removed | Expected |
|---|---|---|---|
| Canary Dry Run | 533s | ~150s | ~380s |
| Typecheck + Release Registry | 462s | ~155s | ~305s |
| Verify Paperclip Runner (vitest 2/2) | 453s | ~245s | ~210s |
| Verify Paperclip Runner (rust) | 400s | ~175s | ~225s |
| Build | 348s | ~130s | ~220s |
| Verify Paperclip Runner (static checks) | 321s | ~170s | ~150s |
| Verify Paperclip Runner (vitest 1/2) | 346s | ~70s | ~275s |

- Wall clock per PR run: about 533s → about 385s. That is about 2.5
minutes faster to a green check set. `Canary Dry Run` stays the longest
check. The rest is the non-cargo work in `release.sh` (standalone
package builds ~30s, publish-payload preview ~73s).
- Runner time: about 18 runner-minutes saved per PR run across the seven
lanes.
- The first master push after merge compiles from zero once in the fleet
writer (about 4 extra minutes on that one run) and saves the v2 entry.
Later PRs hit it. When a PR changes `Cargo.lock`, the prefix restore key
still gives a partial hit, as before.

## Verification

- Run the guard tests for the three cache lanes:
`node --test .github/scripts/tests/pr-runner-rust-cache.test.mjs
.github/scripts/tests/release-runner-cache.test.mjs
.github/scripts/tests/typecheck-rust-cache.test.mjs`
  Result: 21 pass, 0 fail.
- Run the full guard suite: `node --test
'.github/scripts/tests/*.test.mjs'`. Result: 376 pass, 3 fail. The 3
failures are in `docker-canary-promotion.test.mjs`. They hit a sandbox
temp-file ENOENT and fail the same way on the unmodified branch.
- Run `node --test scripts/__tests__/release-verify-workflow.test.mjs`.
Result: 14 pass.
- Local archive test: create a tar from the `_work` layout through the
symlink (relative `../../../paperclip-runner-rust/target` entries, `tar
-P -C $GITHUB_WORKSPACE`, the same way `@actions/cache` does). Extract
it on the `work` layout. The files land in the real target directory and
the symlink stays intact.
- After merge, open any GitHub-hosted PR run and confirm that the seven
Rust lanes log `Restored from cache key ...release-runner-v2...` in
place of `No cache found`.

## Risks

- Low risk. The change touches CI workflows, their tests, and one doc
page. No product code changes.
- If the pin step fails, `rust-cache` reports a miss and the lane
compiles from zero, as it does today. The build does not break.
- Both runner layouts sit four levels under `/home/runner`, so the
relative `../../../` archive entries line up. The existing
`~/.cargo/registry` and `~/.cargo/git` cache paths already rely on this
property. A future runner image with a different `$HOME` depth would
miss the cache but would not fail the job.
- `rm -rf "$pinned"` acts on the symlink itself (no trailing slash),
never on the checkout behind it. It only matters on a reused runner.
- Squash-merge note: the branch carries commits by `Bender (Fable)`. Add
`Co-Authored-By: Bender (Fable) <bender-fable@paperclip.local>` to the
squash body to keep that authorship.

## Model Used

- Anthropic Claude Fable 5.1 (`claude-fable-5-1`), run through Claude
Code inside a Paperclip agent heartbeat. Extended thinking was on. Tool
use: shell, GitHub CLI, and the GitHub REST API for workflow logs, cache
listings, and PR operations.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [ ] My branch name describes the change and contains no internal
ticket id. The agent execution workspace fixed this branch name, so I
cannot rename it. Squash-merge drops the branch name.
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Bender (Fable) <bender-fable@paperclip.local>
2026-09-30 16:04:23 -07:00
DottaandPaperclip 18e8c121d9 fix(runner): include Grok support in public installs with sandbox prerequisites (#14024)
## Thinking Path

> - Paperclip manages agents through a shared native runner.
> - Built-in harness support should ship with Paperclip's public
distribution.
> - Grok already speaks ACP; it does not require a new public bridge
package.
> - Sandbox provisioning owns the native executable and its pinned
version.
> - The runner must verify that prerequisite without downloading it
during npm installation.
> - This change separates built-in launcher identity from external
runtime identity.
> - Clean npm installation and live staging checks verify the
distribution boundary.

## Linked Issues or Issue Description

Refs #13882, #13973, #13977, #13979.

This follow-up now targets master after #13882 was squash-merged. It
replaces the private `@paperclipai/grok-acp` workspace package with
runner-owned assets. Current master is included so the branch also
contains the merged scheduler, complete-event capture, and durable
cleanup fixes.

## What Changed

- Ship Grok launcher and qualification metadata inside the runner's
compiled output and the public server's vendored runner tree.
- Remove the separate Grok npm package and all package-manager install
hooks for this runtime.
- Require the checksum-verified Grok Build 1.0.13 binary at
`/opt/paperclip/providers/grok/1.0.13/grok` in the selected execution
environment. Provision it explicitly in the Daytona image and CI setup.
- Keep native binaries outside the provider pack. Bind the built-in
launcher into the pack manifest.
- Preserve executable leases, descriptor-backed startup, credential
fences, permissions, and exact ACP model admission.
- Use `builtin:grok-acp` and `native:grok` as profile identities.
Historical package-profile sessions fail closed on resume rather than
being silently reinterpreted.
- Resolve built-in assets from the authenticated sidecar location,
including public server npm layouts. Keep the controller path out of
provider environments.
- Add clean npm tarball installation verification to the existing
trusted canary CI job and the admitted manual EC2 verification path. It
stages a unified release version and runs npm lifecycle scripts, then
verifies missing-prerequisite rejection and admission after separate
provisioning without credentials or inference.
- Include the controller-owned provider pack in stamped Cloud images.
Unstamped local images omit the pack and remain usable; remote ACPX
requires full source provenance.
- Correct CLI approval-page metadata for an already authenticated Cloud
board user; approval authorization remains unchanged.
- Honor explicit native-runner enablement in the Cloud agent picker and
direct setup page, keeping the flag disabled by default.
- Allow selecting the execution environment before connecting
credentials. Include Grok in the existing authenticated hello-probe
flow, targeting its pinned native prerequisite for runner setup.
- Recover an existing subscription sign-in conflict through an explicit
cancel-and-retry action, serialized after cancellation succeeds.
- Preserve the selected ACPX harness before normalizing config fields,
so new Grok agents use the Grok default model.
- Keep the credential-free Cloud provider pack root-owned and readable
after runtime UID remapping; verify manifest and referenced asset access
under an unrelated unprivileged UID during image builds.
- Archive prior failover backups alongside explicitly replaced harness
state, preserving evidence while preventing stale backups from blocking
a fresh replacement.
- Update Daytona image content inputs and contract tests for the
built-in assets and explicit provisioner.
- Document and regression-test the shared `approve-all` default for Grok
setup, saved configuration, and native execution. Explicitly saved
restrictions remain unchanged.

## Verification

Current merge-repair head `df09eb3e1a619430ad8419a0ee9aedd486689b05`
incorporates master `f1a394bd30cb56fb9e479f98b9f50176fe921858` after the
base PR was squash-merged. All 12 conflicts came from incoming files
identical to the tested pre-squash base. The final tree exactly matches
a three-way merge using that original base, preserving built-in Grok
distribution and removal of the obsolete private package. All 252
focused runner/UI tests, six npm-isolation tests, and token gates pass.
Fresh exact-head Greptile review is 5/5 with no outstanding findings;
security scans and EC2 native compilation pass. All current-head CI is
green: 56 successful checks/statuses and four intentional skips ([run
36468768035](https://github.com/paperclipai/paperclip/actions/runs/36468768035)).
The repository owner explicitly authorized bypassing code-owner approval
after all checks passed; no CI checks or repository protection settings
are bypassed or changed. The only remaining PR was removed from the
completed stack metadata to permit native auto-merge.

Earlier integration head `78cb306ecc41b5c96577c26c1d89153b0ef865a1`
includes master `3447609d2247e75e55d91493dda91a608364f672` (2026-09-28).
Two master advances during verification overlapped the eval catalog; the
final merge preserves Grok qualification, completion updates, and
bounded API-response reading in all 348 cells. All 77 focused
catalog/eval/workflow tests pass. Both native stack layers (#14397) are
mergeable, and both exact-head Greptile reviews are 5/5 with successful
security scans and no unresolved review threads. All current-head CI is
green: 56 successful checks/statuses and four intentional skips ([CI
attempts](https://github.com/paperclipai/paperclip/actions/runs/36447124691)).
The initial attempt lost two EC2 runners to shutdown signals and stalled
a third shard during dependency preparation; all three passed the
same-commit failed-job-only retry. Trunk code-owner requirements remain
enforced. The review summary’s non-blocking saved-asset offset
classification note concerns code already merged in #14301; those
runtime files are identical to master and outside this stack’s diff.
Historical live evidence below retains its original source revisions.
[Final public npm
verification](https://github.com/paperclipai/paperclip/actions/runs/36445542764)
passed on `76ea70cd4d13786a042af9df82f0fd7a8c85ae30`: 17 public
packages, an executed offline lifecycle sentinel, unchanged consumer
lock, built-in launcher, missing-prerequisite rejection, and verified
separately provisioned binary/command lease. Provisioning and cleanup
require no host privilege elevation; only the positive probe mounts the
temporary native binary read-only. The verifier is unchanged by the
final master merge. All six isolation tests and an offline npm smoke
test pass. The prior head had 56 green CI checks and a 5/5 review after
two unchanged tests timed out and passed a failed-job-only retry ([CI
attempts](https://github.com/paperclipai/paperclip/actions/runs/36444597313)).
All 56 recovery-display/lineage tests pass; re-review cleared the
already-covered missed-retry concern. Earlier EC2 failures remain
retained: [npm lockfile
rejection](https://github.com/paperclipai/paperclip/actions/runs/36436311203),
[missing compiler in the slim
image](https://github.com/paperclipai/paperclip/actions/runs/36440210984),
and the aggregate 15-minute test timeouts in those broad runs. Both
broad attempts passed typecheck, token gates, Product E2E type/unit
checks and build. The focused EC2 lane preserves the existing
trusted-actor and immutable-source gates.


Earlier documentation/test checkpoint
`ff244c4fd78a7ede5a3e00efe09f475f133ef33e` leaves runtime behavior
unchanged. 154 focused tests pass across configuration building, native
provider resolution, permission policy, credentials, UI configuration,
and new-agent setup (including both Grok auth modes); token gates pass.
All fresh CI is green for this head: 56 successful checks/statuses and
two intentional skips ([run
36367065119](https://github.com/paperclipai/paperclip/actions/runs/36367065119)).
Greptile is 5/5 with no new findings. Grok already inherits the shared
`approve-all` default, so unattended setup requires no manual permission
change.

Runtime head `bb5a9307991f1ac567b781970ef11b39d518e19b` fixes a final
staging continuation failure before provider startup: explicit
replacement archived the old harness but left its failover backups
active, which caused `runner_harness_state_mismatch`. The regression
fails before the fix and passes after it; all eight adjacent
recovery-safety cases also pass. Old backups remain inspectable inside
the continuity archive. All fresh CI is green at this head ([run
36360839248](https://github.com/paperclipai/paperclip/actions/runs/36360839248)),
with a 5/5 review. One unrelated Cursor test timed out in the initial
server shard; the same-commit failed-job rerun passed, and both attempts
are retained. Staging deployment is confirmed healthy on this revision.
The controller image is
`ghcr.io/paperclipai/paperclip@sha256:6ad91c487910ccd2596ff7aed0a3a3ea5233d12b51b83cd6e1402237749b9673`.
The final browser-created staging task passed on this exact revision
with API authentication: context read → structured human question →
controller restart → answer submission → same native provider session
resumed → document saved → task Done. The two turns took approximately
119s and 77s. The actual write receipt was applied, and the saved
document has exactly one revision containing the selected answer and
requested marker. Usage and cost were not reported. [Controller image
build](https://github.com/paperclipai/paperclip/actions/runs/36360889243).

- Previous integration head `a44f7dbb6b6f77cd9ed893756ca453307f281e5f`:
all CI green (53 successful checks/statuses, two intentional skips),
including repository typecheck/build/tests, native Runner tests, browser
shards, and canary installation checks. [CI run
36358672529](https://github.com/paperclipai/paperclip/actions/runs/36358672529).
Greptile is 5/5 with no unresolved findings.
- Focused checks cover Grok credentials, executable admission, launcher
assets, provider-pack paths/permissions, workflow contracts, setup
defaults, CLI authorization, and subscription conflict recovery. All 39
protocol definitions validate. Final integration checks pass 124
catalog/evidence/cache tests and nine project-form tests; token gates
pass. Some local dependency checks could not load the stale installed
dependency tree; the corresponding fresh EC2 checks pass.
- Clean public npm installation passed on EC2 at
`8b172ebcf8e02e30662d830c00f3961e3bd459ec` ([run
36164964900](https://github.com/paperclipai/paperclip/actions/runs/36164964900)):
17 unified-version packages, lifecycle scripts enabled, built-in
launcher present, no separate Grok package or npm-downloaded binary,
missing prerequisite rejected, separately provisioned native executable
and command lease verified. No credentials or inference were used.
Subsequent changes preserve this npm asset layout.
- The immutable Daytona prerequisite image is
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:98957d5be0ac774d086b6402b5849e8e6356fec70fb8c09fca6eb4ed6de918e0`,
built from `5a2db471f3ddabe77f9f80e76ed27f996cb97fba`. The previous
Cloud controller image was
`ghcr.io/paperclipai/paperclip@sha256:fd914e1ab1e45f741e8e078ff452d16f082d7ac05f9b4b3506d3a3c64150d204`,
built from `a44f7dbb6b6f77cd9ed893756ca453307f281e5f`; it is superseded
by the latest image above. Its EC2 build verified provider-pack access
under an unrelated unprivileged UID.
- Browser staging at `40f898bc4cba73c1dff4e6344a3983ba0fb247ef` passed
full Grok onboarding with the correct `grok-4.7` model, saved credential
delivery, and pinned Daytona execution. A browser-created task read
context and asked the structured human question. After a controller
restart, answering the persisted question resumed the same native
provider session, saved the requested document, and completed the task.
Actual tool outcomes and durable state agree: one question and one
document revision. The two successful turns took 42.7s and 63.1s; usage
and cost were not reported.
- Restricted policy returned the expected `approval_required` outcome.
Functional staging tests explicitly selected `approve-all`; controller
authorization and governed approvals remain enforced. Temporary board
CLI access was revoked and verified rejected (HTTP 401), and the
disposable onboarding agent was paused.

Failures remain retained: the pre-fix continuation failure (its task
remains blocked; the passing final task is fresh), the original Cloud
provider-pack permission failure, the expected restricted-policy denial,
the superseded npm staging failure, and an earlier monolithic CI
infrastructure timeout. Browser CI exposed a project alias/form race;
the final stack uses master's stronger draft-preservation fix and all
browser shards pass. Historical full subscription/API protocol and
Product rosters retain their original source revisions and do not
qualify this packaging revision. No local Docker or Rust build was used.

## Risks

The branch includes master’s draft-preservation fix for project URL
aliases. It keeps the same project’s edit form mounted and clears prior
data when the project or company changes.

Custom sandboxes and local execution hosts must provision the pinned
binary before Grok starts. Missing, changed, unsupported-platform, and
symlinked executables fail admission. The new builtin profile cannot
resume sessions created with the former private-package profile.
Existing Claude/Codex npm bridge profiles retain their package pins.
Grok restricted modes preserve the selected policy but cannot
automatically admit Paperclip calls: ACP permission metadata does not
independently bind tool authority, so those calls stop with
`approval_required`. New Grok configurations default to `approve-all`,
including API configurations that omit the mode. Existing explicitly
restricted configurations remain restricted; controller authorization
and governed approvals remain enforced.

## Model Used

OpenAI GPT-6 through Codex, with tool use and code execution. The exact
serving model identifier and context-window size are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-28 14:54:44 -05:00
Devin FoleyandPaperclip e3d8fb0876 feat: attest standard production images at full source commits (#13797)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Operators deploy its standard production container on several CPU
architectures.
> - Downstream image builders need to identify the exact source of their
base image.
> - A short commit tag does not provide signed source evidence.
> - This pull request adds a full commit tag and signed image digest for
canonical master pushes.
> - Consumers can verify the source and compose from the immutable
digest.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Publication of the standard multi-platform production image.

**Current behavior**

The Docker workflow publishes short commit tags and channel tags. It
does not provide a signed standard-image contract tied to the complete
master commit.

**Proposed behavior**

Canonical master pushes also publish `sha-<full-commit>` and attest the
exact index digest after platform validation and an immutable-image
orphan-reaping check. The signer certificate binds the source
repository, commit, workflow and ref. Existing tags and the separate
cloud producer remain available.

**Reason and benefit**

Downstream builders can prove the source of a standard base without
adding their dependencies or repository details to the public workflow.
No matching open issue or duplicate PR was found.

## What Changed

- Add the canonical full-SHA tag without changing existing tag mappings.
- Validate amd64 and arm64 descriptors and hash the exact registry
response bytes and require its digest header to match.
- Verify the immutable image and sign it with GitHub artifact
attestations.
- Run the contract tests in trusted PR verification and document the
consumer contract.

## Verification

- `node --test scripts/__tests__/release-verify-workflow.test.mjs
scripts/cloud-source-verification.test.mjs
scripts/standard-image-contract.test.mjs`: 37 passed.
- A read-only check against an existing published index returned its
exact expected digest.
- `actionlint -shellcheck='' .github/workflows/docker.yml
.github/workflows/pr-trusted.yml`: passed. Normal ShellCheck reports
only existing `ls` and word-splitting warnings.
- `pnpm build`: passed locally with Cargo available.
- `pnpm -r typecheck`: passed locally.
- Full local Vitest was attempted: 8,260 passed, 14 failed, with 34
failing suites. The failures were missing embedded-PostgreSQL library
aliases in this fresh install and existing macOS runtime-skill-cache
rename errors. Native aliases are now restored. Rerunning the 33
affected database suites produced 577 passes and two unrelated AgentMail
skill-root lookup failures (32 suites passed). The four directly failing
database tests also pass independently. This is not a claim that the
full local suite passed.
- All final-head CI checks pass. One unrelated routine-route mock
assertion passed on the single-shard retry; its 15 tests also pass
locally. Greptile is 5/5 on this exact head, with no unresolved threads.
- Actual signing requires a canonical master push. This draft PR does
not publish trusted provenance.

## Risks

The new attestation step requires OIDC and attestation write permissions
in the merge job. Signing failure leaves the image available but without
the new admission proof. Consumers must fail closed when proof is
missing. Existing release tags, the legacy producer, and image retention
remain unchanged. No database or application behavior changes.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository inspection, shell
execution and test tools. The session does not expose a more specific
model variant or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-22 07:33:30 -07:00
Devin Foley d08abcba15 ci: cut PR wall clock from ~16 to ~6 minutes (#13521)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every pull request runs the Trusted PR CI workflow before merge
> - The test suites roughly tripled in six weeks, and shard balance did
not keep up, so PR runs crept from ~4 to ~17 minutes
> - Slow CI delays every merge and every contributor
> - This pull request rebalances the shards from fresh measurements,
splits the largest test files, reuses the Rust build cache in three more
jobs, and takes the policy job off the critical path
> - The benefit is a PR wall clock near 6 minutes with the same coverage

## Linked Issues or Issue Description

**What existing behavior does this improve?**

PR CI wall clock. A typical green run took 16-17 minutes. Two months ago
it took about 4 minutes.

**Subsystem affected**

The Trusted PR CI workflow (`.github/workflows/pr-trusted.yml`), the
shard-duration manifests, the vitest shard runner scripts, the
`paperclip-runner` package scripts, and the dry-run branch of
`release.sh`.

**Current behavior**

The shard-duration manifests were stale. The general-server manifest had
durations for ~400 of 649 suites. The e2e manifest was missing 14 of 29
specs. Stale median weights made shard steps range 417s-806s (server)
and 277s-745s (e2e). Three jobs each paid a ~3m40s cold cargo release
build. Every test lane waited ~60s for the policy job before it could
start.

**Proposed behavior**

All lanes finish in a narrow ~200-290s band. The manifests carry fresh
measured durations for every suite. The three largest test files are
split so no single file caps a shard. The Rust cache restore runs in
every job that builds the Runner binary. Test lanes start as soon as the
gate resolves.

**Reason and benefit**

Merges stop waiting on CI. The projected wall clock is ~6 minutes for
the same test coverage.

## What Changed

- Rebuild `scripts/general-server-shard-durations.json` (646 suites) and
`scripts/e2e-shard-durations.json` (all specs) from per-suite completion
timestamps in runs 35036001734 and 35024948947.
- Move the PR server lane to the release-verify shape:
`general-server-without-chat` across twelve duration-balanced shards,
plus the chat integration suite split by collected test location across
three dedicated lanes.
- Split `tests/e2e/chat-adapters-ui.spec.ts` into `-providers` and
`-messaging` specs, and `tests/e2e/agent-chat.spec.ts` into `-sessions`
and `-projects` specs. Each pair shares fixtures through a `.shared.ts`
module. Playwright collects the same test sets (39 and 20 tests).
- Raise e2e shards to eight and serialized shards to nine.
- Run the runner package's `check:all` as four matrix lanes:
`check:static`, `check:runner`, and two native vitest `--shard` halves.
The union is exactly `check:all`.
- Add the read-only Rust cache restore (toolchain pin, `save-if: false`)
to the Canary Dry Run, Build, and Typecheck jobs.
- Make release.sh preview publish payloads concurrently in batches of
eight during `--dry-run`. The real publish path stays strictly serial.
- Drop the policy-job lockfile artifact chain. Each lane installs with
`--frozen-lockfile` and falls back to an inline `--resolution-only`
regeneration. The policy job stays a required check through the `verify`
and `e2e` aggregates.
- Update the shard-count mirrors and workflow assertions in the
partition and gate tests.

## Verification

- `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs
scripts/__tests__/e2e-shard.test.mjs` — 30 pass.
- `node --test '.github/scripts/tests/'*.test.mjs` — 410 pass.
- `node --test scripts/__tests__/release-verify-workflow.test.mjs
scripts/cloud-source-verification.test.mjs
scripts/__tests__/release-dry-run-notes.test.mjs` — 42 pass.
- `playwright test --list` collects 39 tests across the chat-adapters
split and 20 across the agent-chat split, equal to the original files.
- A local vitest collection of the chat suite partitions 995 tests into
498/497 line shards.
- Projected shard weights: server 230s x12, chat ~143s x3, e2e 207-242s
x8, serialized ~216s x9.

## Risks

- The split spec files reorder tests relative to the original files.
Every describe seeds its own company, so the specs stay independent; a
hidden cross-describe dependency would surface as a deterministic
failure in one shard.
- The inline lockfile fallback changes install behavior for
manifest-changing and stacked PRs. The policy job still validates
resolution as a required check.
- `release.sh` changes are confined to the `--dry-run` preview branch.
The publish loop is untouched. `bash -n` passes and the release dry-run
tests pass.
- One PR now schedules ~44 fleet runners. If the RunsOn fleet caps
concurrency, queueing may absorb part of the gain; watch the first runs.

## Model Used

- Claude Fable 5 (`claude-fable-5`, Anthropic), extended thinking, with
tool use (shell, file edits) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-09-16 11:45:14 -07:00
Nicky LeachandClaude Opus 5 bd51f157e9 fix(ci): make the Runner Rust cache key independent of the image toolchain (#13500)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every change goes through the pull request CI workflow, and its
`Verify Paperclip Runner` lane builds and tests the native Rust runner
> - https://github.com/paperclipai/paperclip/pull/13457 made that lane
restore master's prebuilt Rust dependency cache instead of recompiling
313 crates
> - Measured over 26 runs since it went live, the cache hits on the
RunsOn fleet and misses on every GitHub-hosted runner
> - The cause is the cache key: `rust-cache` hashes every installed
toolchain, and each runner image ships a different stable Rust next to
the pinned one
> - This pull request removes the extra toolchains before the key is
computed, in the reader and the writer
> - The benefit is that the saving the cache already delivers, 4.8
minutes per run, reaches the 73% of runs that currently miss it

## Linked Issues or Issue Description

No public GitHub issue exists for this. The problem follows the
enhancement issue template below.

- Refs https://github.com/paperclipai/paperclip/pull/13457 — added the
cache this pull request repairs
- Refs https://github.com/paperclipai/paperclip/pull/13459 — activated
it
- Refs https://github.com/paperclipai/paperclip/pull/13194 — created the
`release-runner-v1` entry on master

I searched this repository for other pull requests touching this cache
and found no duplicate and nothing in flight.

**What existing behavior does this improve?**

The Rust dependency cache added in #13457 misses on GitHub-hosted
runners, so most pull requests still recompile the whole dependency
tree.

**Subsystem affected**

Cross-cutting (multiple of the above). The change touches CI workflow
configuration only. It does not change product code.

**Current behavior**

The cache works, but only on one runner class. Across 26 successful
`Verify Paperclip Runner` jobs since #13459 merged:

| Runner | Runs | Before | After | Change | Cache |
|---|---|---|---|---|---|
| RunsOn fleet | 7 | 16.4m | 11.6m | −4.8m | 4 of 4 full hit |
| ubuntu-latest | 19 | 14.9m | 14.7m | −0.2m | 0 of 6 hit |
| All | 26 | 15.0m | 13.9m | −1.1m | |

Only 27% of runs reach the fleet, so the fleet-wide saving is 1.1
minutes rather than the 4.8 minutes the cache delivers where it lands.

The two runners compute different keys:

```
fleet:         v0-rust-release-runner-v1-Linux-x64-4bb3b8ea-a95b0328
ubuntu-latest: v0-rust-release-runner-v1-Linux-x64-9fdc73e3-a95b0328
```

The lockfile half agrees. The environment half does not. `rust-cache`
logs why, under `Environment considered`:

| Runner | Toolchains it found |
|---|---|
| fleet | 1.97.1 and **1.98.0** |
| ubuntu-latest | 1.97.1 and **1.98.1** |

`rust-cache` hashes every installed toolchain, not only the active one.
Both images carry the pinned 1.97.1. Each also ships its own stable
Rust, and those differ by a patch version. The post-merge writer runs on
a RunsOn image, so the fleet agrees with it and GitHub-hosted runners
cannot.

Pinning `RUSTUP_TOOLCHAIN` in #13457 was necessary but not sufficient.
It fixes which toolchain builds the code. It does not change which
toolchains exist on the image.

**Proposed behavior**

Remove every toolchain except the pin, before the cache step, in both
the reader and the `release-runner-v1` writer. The key then depends on
the pinned compiler and the lockfile alone, not on what the image
happens to carry.

**Reason and benefit**

The cache already proves its value where it lands: release compile drops
from 5m07s to 1m30s, debug from 1m48s to 13s, and the job from 16.4m to
11.6m. This change extends that to the other 73% of runs.

Expected fleet-wide mean: about 11.5m, against 13.9m today and 15.0m
before #13457.

It also removes a standing fragility. The fleet hits today only because
two RunsOn images happen to agree. If either image updates its stable
Rust on its own, the hit rate drops to zero with no code change.

**Breaking changes**

None. The change only affects cache key computation. A miss reproduces
the current behavior.

**Additional context**

The typecheck writer in `release-verify.yml` keeps its current step on
purpose. It restores and saves on the same post-merge image, so its key
never disagrees with itself.

## What Changed

- Added a toolchain normalization block to `Select the pinned Runner
Rust toolchain` in `.github/workflows/pr-trusted.yml`, before the cache
restore. It keeps the pinned toolchain and uninstalls the rest.
- Added the identical block to the same step in the
`verify_paperclip_runner` job of `.github/workflows/release-verify.yml`,
which writes `release-runner-v1`. The reader and the writer must agree,
or the key matches nothing.
- Made the block tolerant. If a toolchain cannot be removed it prints a
notice and continues, so a pull request loses the cache rather than the
run.
- Extended `.github/scripts/tests/pr-runner-rust-cache.test.mjs` with
two tests: the block is byte-identical in both workflows, and it runs
before the cache step in each.

## Verification

Run the workflow shape tests:

```bash
node --test '.github/scripts/tests/*.test.mjs' ./scripts/__tests__/e2e-shard.test.mjs ./scripts/__tests__/release-verify-workflow.test.mjs ./scripts/__tests__/run-vitest-stable-shard.test.mjs ./scripts/cloud-source-verification.test.mjs
```

Result: 471 pass, 0 fail.

I ran the step body against a stub `rustup` to confirm the logic, rather
than only checking syntax. Three paths, all exit 0:

| Case | Result |
|---|---|
| Pin plus an extra stable toolchain | Uninstalls only the extra,
exports `RUSTUP_TOOLCHAIN=1.97.1-x86_64-unknown-linux-gnu` |
| Pin only, already normalized | No uninstall calls, no error |
| No `rustup` on `PATH` | Prints the notice and continues |

I also mutation-tested the new parity assertion. Each mutation edits
only the writer, then the reader and writer disagree:

| Mutation to `release-verify.yml` | Result |
|---|---|
| One word changed in the shared comment | Caught |
| `uninstall` changed to `remove` | Caught |
| Trailing `rustup toolchain list` deleted | Caught |

After this merges, confirm the repair in CI. Take a `Verify Paperclip
Runner` job that ran on `ubuntu-latest` and check the restore step for
`full match: true`. Under `Environment considered`, `Rust Versions` must
list only 1.97.1. The job should finish near 11.5m rather than 14.7m.

## Risks

- **Master must republish the cache once.** This changes the key, so the
existing `release-runner-v1` entry no longer matches.
`cloud-readiness.yml` runs `release-verify.yml` on every push to master,
so the first push after this lands writes the new entry. Pull requests
merged before that miss the cache, which is exactly what most of them do
today. No run breaks.
- **The reader and the writer must stay in step.** If they diverge, no
run hits the cache. The new parity test fails on any difference in the
block, including a comment.
- **Uninstalling the image toolchain is deliberate.** Everything in this
job builds through the pinned 1.97.1, resolved from
`packages/paperclip-runner/rust-toolchain.toml` and `RUSTUP_TOOLCHAIN`.
Nothing in the job uses the image default.
- **Low blast radius.** The change only affects cache key inputs. A miss
compiles from scratch, as today.
- **Image drift is now handled.** The key no longer depends on the
image's own Rust version, so a future image update cannot silently
disable the cache.

## Model Used

Claude Opus 5, provider Anthropic, exact model ID `claude-opus-5`, 1M
context window. Adaptive thinking was on. I used tool use throughout:
the `gh` CLI and the GitHub API to pull 26 post-merge job records and 10
full job logs, log parsing in Python to isolate the per-phase timings
and the two cache keys, a stub `rustup` on `PATH` to exercise the new
step, and local `node --test` runs to verify and mutation-test the
guards. Run through Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Note on the unchecked boxes. The CI and Greptile boxes stay unchecked
until those checks finish. On documentation: no document describes the
Runner cache keys, so there is nothing to update.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-15 17:24:22 -07:00
Nicky Leach f97a3f886e perf(ci): restore master's Rust dependency cache on the PR runner lane (#13457)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every change to Paperclip goes through the pull request CI workflow
before it merges
> - One lane of that workflow, `Verify Paperclip Runner`, builds and
tests the native Rust runner
> - That lane compiles all 313 third-party crates from scratch on every
pull request, in two profiles
> - Master already builds and stores exactly those compiled crates, but
the pull request lane never reads them
> - This pull request restores that existing cache on the pull request
lane, read-only
> - The benefit is about 3.9 minutes less compute per run, or about 4.7
compute-hours each day, with no change to what CI checks

## Linked Issues or Issue Description

No public GitHub issue exists for this. The problem follows the
enhancement issue template below.

Related merged pull requests, found by searching this repository:

- Refs https://github.com/paperclipai/paperclip/pull/13194 — created the
`release-runner-v1` cache that this pull request reads
- Refs https://github.com/paperclipai/paperclip/pull/13300 — established
the restore-only dependency cache pattern that this pull request follows
- Refs https://github.com/paperclipai/paperclip/pull/13326 — split the
post-merge runner job into the `protocol` and `rust` lanes used as the
warm-cache baseline below
- Refs https://github.com/paperclipai/paperclip/pull/13259 — applied the
same caching idea to the post-merge typecheck job

I found no open pull request that duplicates this work.

**What existing behavior does this improve?**

The `Verify Paperclip Runner` job in `.github/workflows/pr-trusted.yml`
recompiles the full Rust dependency tree on every pull request.

**Subsystem affected**

Cross-cutting (multiple of the above). The change touches CI workflow
configuration only. It does not change product code.

**Current behavior**

The job has no Rust cache. Its only cache step restores the pnpm store.
Each run therefore downloads about 280 crates and compiles all 313
third-party crates twice, once in the `dev` profile and once in the
`release` profile.

Measured over 12 successful runs on 2026-09-15, the job takes 15.0
minutes on average. The range is 12.1 to 17.4 minutes.

| Phase | Mean | Share |
|---|---|---|
| `check:eval-kernel` | 0.0m | — |
| `typecheck:typescript` and protocol manifest | 0.1m | 1% |
| `build:rust` — cargo `dev` profile | 1.8m | 12% |
| TypeScript tests (`node --test` and vitest, 143 files) | 5.8m | 39% |
| `check:replay-goldens` | 0.1m | 1% |
| `typecheck:rust` — `cargo fmt` and `cargo check` | 0.9m | 6% |
| `test:rust` compile — cargo `release` profile | 4.9m | 33% |
| `test:rust` run | 0.8m | 5% |
| Parity checks | 0.0m | — |
| `check:api-authority` | 0.5m | 3% |

Cargo reports the two compiles directly. The `dev` profile takes 1m24s
to 1m53s. The `release` profile takes 3m54s to 5m36s.

**Proposed behavior**

The job restores master's existing Rust dependency cache before it runs
the checks.

Master already writes this cache. The post-merge `rust` lane in
`.github/workflows/release-verify.yml` writes `release-runner-v1`. It
also warms both profiles into that entry. The entry is 677MB and lives
on `refs/heads/master`. Pull request branches are allowed to read caches
from the default branch.

The pull request lane restores that entry read-only. Master stays the
only writer. A pull request never saves a branch-scoped copy. A pull
request never evicts the shared entry. This matches the rule the pnpm
store in the same file already follows.

**Reason and benefit**

Master runs these same checks with a warm cache. That gives a direct
measurement of the saving.

| Check set | Cold (pull request today) | Warm (master) | Change |
|---|---|---|---|
| `check:runner` and `check:api-authority` | 7.1m | 3.4m | −3.7m |
| `check:eval-kernel` and `check:protocol` | 7.8m | 7.3m | −0.5m |
| Cache restore step | — | 20–21s | +0.35m |

The net saving is about 3.9 minutes per run, or about 26%. About 73 runs
execute this job each day. The daily saving is therefore about 4.7
compute-hours.

The protocol side changes very little. Vitest dominates that side, not
compilation.

The cache should almost always hit.
`packages/paperclip-runner/runner/Cargo.lock` changed in 7 of the last
1697 commits on master. A miss costs nothing more than today's behavior.

**Breaking changes**

None. The change adds two steps to one CI job. It does not change any
check, any test, or any product code.

**Additional context**

No documentation covers pull request CI caching, so this pull request
updates no documents.

## What Changed

- Added a `Select the pinned Runner Rust toolchain` step to the
`verify_paperclip_runner` job in `.github/workflows/pr-trusted.yml`. The
step exports `RUSTUP_TOOLCHAIN`. `rust-cache` hashes `rustc -vV` into
the cache key, so the key needs the pinned compiler. This lane had no
rustup step before, so `rustc` resolved to each runner image's default
instead of the pinned 1.97.1.
- Made that step tolerate a missing `rustup`. `release-verify.yml` runs
on one post-merge fleet image. The gate in this workflow routes to
either `ubuntu-latest` or the public pull request fleet. A missing
`rustup` now costs the cache. It does not fail the pull request.
- Added a `Restore Runner Rust dependencies (read only)` step that uses
`Swatinem/rust-cache` with `save-if: false`.
- Mirrored every cache key input from the master writer in
`release-verify.yml`: the same action SHA, `workspaces`, `shared-key`,
`cache-workspace-crates`, and `cache-bin`. Neither side sets
`prefix-key`. Any drift causes a silent miss and a full recompile.
- Added `.github/scripts/tests/pr-runner-rust-cache.test.mjs`. It
asserts key-input parity across the two workflow files, the step order,
and the restore-only contract.

## Verification

Run the workflow shape tests:

```bash
node --test '.github/scripts/tests/*.test.mjs'
```

Result: 413 pass, 0 fail.

I also mutation-tested the new assertions. I applied each mutation to
the workflow, ran the new test file, then restored the file. All 7
mutations fail the suite:

| Mutation | Result |
|---|---|
| `shared-key` changed to `pr-runner-v1` | Caught |
| `save-if` changed to `true` | Caught |
| `cache-bin` changed to `true` | Caught |
| `rustup show active-toolchain` line deleted | Caught |
| `RUSTUP_TOOLCHAIN` export line deleted | Caught |
| `working-directory` line deleted | Caught |
| `command -v rustup` guard deleted | Caught |

To confirm the cache key inputs match the master writer, parse both
workflows and compare:

```bash
ruby -ryaml -e 'pr=YAML.safe_load(File.read(".github/workflows/pr-trusted.yml"), aliases: true); rv=YAML.safe_load(File.read(".github/workflows/release-verify.yml"), aliases: true); a=pr["jobs"]["verify_paperclip_runner"]["steps"].find{|s| s["uses"].to_s.include?("rust-cache")}; b=rv["jobs"]["verify_paperclip_runner"]["steps"].find{|s| s["uses"].to_s.include?("rust-cache")}; puts a["uses"]==b["uses"]; %w[workspaces shared-key cache-workspace-crates cache-bin].each{|k| puts "#{k}: #{a["with"][k]==b["with"][k]}"}'
```

Every line prints `true`.

After the pin advances (see Risks), confirm the cache works in CI. The
restore step log must show `Cache restored from key:
v0-rust-release-runner-v1-Linux-x64-...`. Cargo must stop printing
`Compiling` lines for third-party crates. The job should drop from about
15.0 minutes to about 11.0 minutes.

## Risks

Low risk overall. A cache miss produces exactly today's behavior, so the
worst case is no improvement.

- **This change does nothing until a second pull request lands.**
`.github/workflows/pr.yml` pins this reusable workflow by SHA. The file
header describes this two-step rollout. A follow-up pull request must
bump that pin. That follow-up validates itself, because GitHub uses the
pull request's own `pr.yml` for `pull_request` events.
- **A key mismatch would silently remove the benefit.** The runner
images here may ship a different default `rustc` than the post-merge
fleet. The toolchain step pins the compiler to prevent this. The new
test guards the remaining key inputs. If the first runs still miss,
compare `rustup show` output against a master run.
- **A missing `rustup` degrades quietly.** The step prints a GitHub
notice and continues. CI stays green and the run is simply uncached.
- **No cache poisoning path.** Pull requests only read. `save-if: false`
stops any write. GitHub also isolates pull request cache writes from the
default branch.
- **The cache entry can expire.** GitHub evicts unused entries after 7
days and enforces a repository size limit. Master pushes are frequent,
so the entry should stay warm. Eviction only causes a miss.

## Model Used

Claude Opus 5, provider Anthropic, exact model ID `claude-opus-5`, 1M
context window. Adaptive thinking was on. I used tool use throughout:
the `gh` CLI to read 12 job logs and step timings from recent CI runs,
the GitHub Actions cache API to read cache entry keys and sizes, `git
log` to measure `Cargo.lock` churn, and local `node --test` runs to
verify and mutation-test the change. Run through Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Note on the unchecked boxes. The branch name
`claude/paperclip-runner-conditional-81a300` carries a tool-generated
suffix, so it does not meet the branch naming rule. I can rename it if
you want. The CI and Greptile boxes stay unchecked until those checks
finish on this pull request.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-09-15 01:16:34 -07:00
Devin FoleyandPaperclip 44dde2dec4 ci: reuse dependency caches without per-PR uploads (#13300)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud releases wait for source verification before deployment.
> - That verification reuses compiled Rust dependencies to finish
sooner.
> - PR jobs save large pnpm stores under separate merge refs and
different lockfile keys.
> - Those copies compete with master build caches for the repository's
10 GB cache limit.
> - This PR makes PR dependency caches restore-only and reuses
master-compatible keys.
> - A separate pin update will activate the reviewed workflow.

## Linked Issues or Issue Description

**What happened?**
PR merge refs accumulated roughly 700 MB copies of the same pnpm store.
Master Rust caches disappeared, and Cloud readiness run
[34656098157](https://github.com/paperclipai/paperclip/actions/runs/34656098157)
rebuilt dependencies after cache misses. The repository currently has a
10 GB limit. GitHub rejected a request for 50 GB; that setting needs
separate organization/billing access.

**Expected behavior**
PR jobs should reuse downloaded packages without evicting post-merge
compilation caches through duplicate uploads.

**Steps to reproduce**
1. Run several PRs while the checked-in lockfile needs policy
regeneration.
2. Compare the setup-node keys in PR jobs and master jobs.
3. List Actions caches by ref, key, and archive size. The PR keys repeat
across merge refs.

**Paperclip version or commit**
f12b647ae, before this change.

**Deployment mode**
GitHub Actions, with GitHub-hosted and allowlisted AWS PR runners.

Refs #13267 (empty pnpm store prevention). Searched open issues and PRs
for pnpm cache duplication and found no duplicate implementation. This
change leaves the paused capacity documentation PR #13280 alone.

## What Changed

- Replace setup-node cache writes with pinned `actions/cache/restore` in
all seven PR install job definitions.
- Restore against the checked-in lockfile before downloading the
regenerated policy artifact. Keep every install frozen against that
artifact.
- Allow an OS/architecture-specific pnpm fallback and disable automatic
setup-node caching.
- Remove dependency-store caching from the resolution-only policy job.
- Add eight regression tests, update the existing stacked-lockfile cache
assertion, and document cache behavior and storage settings.

## Verification

- Passed 505 workflow, routing, cache, and source-verification tests:
`node --test '.github/scripts/tests/*.test.mjs'
scripts/__tests__/e2e-shard.test.mjs
scripts/__tests__/run-vitest-stable-shard.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs
scripts/cloud-source-verification.test.mjs`.
- Passed `actionlint .github/workflows/pr-trusted.yml` and `git diff
--check`.
- The AWS routing gate is unchanged. Author, event sender, and rerun
actor must still be allowlisted.
- This definition PR does not change the active `pr.yml` pin. After
review and merge, authorize its immutable merge SHA additively and
activate it in a separate PR. Verify a populated restore and no cache
uploads in an allowlisted PR.
- All 32 checks passed or were intentionally skipped on
`44b31eca590f61b75cae646de43c491b6c4deae7`, including full native Runner
verification, application build, server/workspace tests, and browser
shards. Current-head Greptile is 5/5 with no findings. No application
code changes in this PR.

## Risks

- New dependencies present only in a PR may download again on each run
until master saves a cache containing them. Frozen installation remains
the source of dependency resolution.
- Missing or expired stores fall back to normal package downloads.
- Existing PR copies remain until expiry or a separate targeted cleanup.
No cache entries are deleted here.
- The workflow only takes effect after the separate immutable pin
rotation. Storage billing settings are not changed by this PR.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-11 17:31:15 -07:00
Devin FoleyandPaperclip 4fde92107e fix(ci): reuse cloud source verification for npm canaries (#13233)
Reuse the exact master source-verification result before npm canary publication, removing a duplicate verification matrix while preserving fail-closed release checks.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-11 09:26:26 -07:00
Nicky LeachandPaperclip 2a05b5ed34 ci: split runner verification from build (#13142)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip uses GitHub Actions to verify changes before release.
> - The Paperclip Runner has a separate verification boundary.
> - The build job currently runs this verification before the workspace
build.
> - This pull request moves runner verification into its own parallel
job.
> - The benefit is clearer CI results and less wait time for independent
work.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The trusted PR and release verification workflows run Paperclip Runner
verification inside the Build job.

**Subsystem affected**

Cross-cutting (GitHub Actions CI workflows).

**Current behavior**

The Build job runs `pnpm --filter @paperclipai/paperclip-runner
check:all` before it builds the workspace. A runner verification failure
appears as a Build failure. The workspace build cannot run in parallel
with runner verification.

**Proposed behavior**

Each workflow has a `Verify Paperclip Runner` job with the same
checkout, dependency install, and command. The Build job only builds its
required outputs. Both jobs run after the same gate and policy jobs.

**Reason and benefit**

The runner command is an independent verification boundary. A dedicated
job gives it a clear status and allows it to run in parallel with Build.

**Breaking changes**

None. The same runner verification command still runs in both workflows.

## What Changed

- Added a dedicated `Verify Paperclip Runner` job to the trusted PR
workflow.
- Added a dedicated `Verify Paperclip Runner` job to the release
verification workflow.
- Kept the Build jobs independent and retained their existing build
commands.
- Updated the trusted-workflow policy test for the additional
dependency-install job.

## Verification

- Ran `git diff --check`.
- Ran `node --test ./scripts/__tests__/e2e-shard.test.mjs`.
- Ran `pnpm exec prettier --check .github/workflows/pr-trusted.yml
.github/workflows/release-verify.yml`.
- Confirmed both jobs retain their prior runner, dependency, and policy
prerequisites.

## Risks

Low risk. The runner verification job repeats the existing setup. It
adds one parallel GitHub Actions runner to each affected workflow.

## Model Used

OpenAI Codex, GPT-5.6, 128k context window, reasoning and tool-use
capabilities.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-10 01:18:06 -07:00
Dotta d96452db05 fix(runner): restore Vite 6 viewer compatibility (#12929)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Paperclip Runner includes an issue-thread viewer for direct
Evalbook reports.
> - The runner package uses Vite 6.4.3.
> - Dependabot changed the React plugin from version 4.7.0 to version
6.1.1.
> - React plugin 6.1.1 requires Vite 7 or Vite 8 package internals.
> - The live evaluation workflow found this mismatch when it built the
viewer on a clean Linux worker.
> - This pull request restores the compatible plugin and adds the viewer
build to pull request CI.
> - The benefit is that CI detects this class of report-viewer build
failure before a paid evaluation campaign starts.

## Linked Issues or Issue Description

**What happened?**

The direct live evaluation workflow failed before model execution. The
`build:issue-thread` command could not load `@vitejs/plugin-react@6.1.1`
with Vite 6.4.3. The plugin imported the unavailable `vite/internal`
package path.

**Expected behavior**

The Evalbook issue-thread viewer must build from a clean frozen-lockfile
installation before the live evaluation matrix starts.

**Steps to reproduce**

1. Check out commit `165ca56a22adb60e5fda56045442d9c8498116a8`.
2. Run `pnpm install --frozen-lockfile --ignore-scripts`.
3. Run `pnpm --filter @paperclipai/paperclip-runner build:issue-thread`.
4. Observe the `ERR_PACKAGE_PATH_NOT_EXPORTED` error for
`vite/internal`.

**Paperclip version or commit**

`165ca56a22adb60e5fda56045442d9c8498116a8`

**Deployment mode**

Built from source in GitHub Actions on Ubuntu.

## What Changed

- Restore `@vitejs/plugin-react` 4.7.0 in the Vite 6 runner package.
- Follow the repository policy: trusted PR CI regenerates and verifies
the lockfile artifact, and the master refresh workflow commits the
lock-only update after merge.
- Build the Runner Evalbook viewer in pull request CI.

## Verification

- `ci / policy`: regenerated the dependency lock artifact successfully
- `pnpm --filter @paperclipai/paperclip-runner build:issue-thread`
- `actionlint .github/workflows/pr-trusted.yml
.github/workflows/runner-protocol-live-evals.yml`
- `node --test
packages/paperclip-runner/scripts/runner-protocol-eval-workflow-security.test.mjs`
- `git diff --check`

## Risks

Low risk. This change restores the previous React plugin major version
for one package. The selected version declares support for Vite 6. Pull
request CI now builds the affected viewer directly.

> This bug fix does not add or change a roadmap feature.

## Model Used

- OpenAI Codex with GPT-5. Tool use and code execution were enabled. The
Codex app managed the context window.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-09-06 09:16:21 -05:00
Nicky LeachandPaperclip 3ed5b7c5c8 refactor(server): extract the active-run output watchdog into a feature module (#12853)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server recovery service monitors active runs and applies
watchdog decisions
> - The watchdog rules and database operations lived in one large
recovery service
> - This structure made the rules harder to test and made company
scoping harder to inspect
> - This pull request moves the watchdog into domain, application, and
adapter layers
> - The benefit is a smaller recovery service, pure policy tests, and
clear company-scoped ports

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The active-run output watchdog that detects silence, suppression, and
terminal evidence.

**Current behavior**

The recovery service contains the watchdog policy, use cases, database
operations, and process control in one file.

**Proposed behavior**

A feature module separates pure policy, use cases and ports, and
Postgres and process adapters. The recovery service delegates its public
watchdog methods to this module.

**Reason and benefit**

The separation makes policy decisions easy to test. Company identifiers
on every reader and writer port make tenant scope clear. Smaller service
methods reduce change risk.

**Breaking changes**

None. The recovery service keeps its public methods and call sites.

Related public watchdog work includes
[#7043](https://github.com/paperclipai/paperclip/pull/7043) and
[#7770](https://github.com/paperclipai/paperclip/pull/7770).

## What Changed

- Add the `server/src/modules/active-run-watchdog/` feature module with
domain, application, and adapter layers.
- Move watchdog policy, use cases, Postgres access, and local process
control into the module.
- Keep the recovery service public methods and delegate them to the
module.
- Add 53 pure module test cases and retain 8 Postgres integration cases.
- Add company scoping and transaction rollback coverage.

## Verification

- Run `vitest run --config vitest.config.ts src/modules` and confirm 3
files and 53 cases pass.
- Run `vitest run --config vitest.config.ts
src/__tests__/heartbeat-active-run-output-watchdog.test.ts` and confirm
1 file and 8 cases pass.
- Run the full server suite in pull request CI.
- Compare the type-check result with a fresh baseline on the same
checkout.

## Risks

The main risk is a behavior change in recovery decisions during the move
across layers. The pure policy tests cover the moved rules. The
integration tests cover database behavior, company scope, and
transaction rollback. Pull request CI runs the full server suite.

## Model Used

OpenAI Codex, GPT-5, runtime-managed context window, tool use, code
execution, and repository review support.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-05 09:15:59 -07:00
Devin Foley bb920fb859 ci: keep the Docker build context complete and guard it on every PR (#12855)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - GitHub Actions builds the Docker images that ship Paperclip, and
downstream deployments consume the `-cloud` image variant on every
master merge.
> - PR #12769 slimmed the Docker build context with a broad
`.dockerignore` block for `packages/paperclip-runner`, and the block
also removed three files the image build itself reads.
> - The image build re-runs the runner's generated-file drift checks, so
it found no committed capability contract in the context and failed on
every master commit after the merge.
> - PR CI never runs those checks against the Docker context, so the
pull request stayed green and the breakage only appeared post-merge, on
every image build.
> - This pull request restores the three files with narrow
`.dockerignore` exceptions and adds a PR CI job that runs the drift
checks against the exact Docker build context.
> - The benefit is that image publishing works again now, and the next
context-slimming regression fails the pull request instead of every
post-merge image build.

## Linked Issues or Issue Description

Refs #12769 (the context-slimming change that exposed this) and #12608
(which committed the generated contract outputs the image build checks).

**What happened?**

Every `Docker` workflow run on master failed from 2026-09-04 12:58Z
onward, in both the `build-and-push` and `build-and-push-cloud` jobs.
The failing step reported `Generated contract drift:
generated/capability/capability-contract.md` from
`check:capability-contract` inside `pnpm --filter @paperclipai/server
build`. The committed contract file is current — regeneration on a full
checkout is a no-op. The file was simply absent from the build context:
the new `packages/paperclip-runner/**/*.md` ignore rule strips the
committed drift-check outputs
(`generated/capability/capability-contract.md`,
`generated/capability/downstream-handoff.md`), and the
`packages/paperclip-runner/docs` rule also strips
`docs/capability-contract.md`, which `check:capability-inventory` reads
next in the chain. No cloud image published for eight hours, which
stalled every downstream deployment that consumes the canary images.

**Expected behavior**

The Docker build context must contain every file the image build reads,
and a change that removes one must fail the pull request that introduces
it, not every image build after the merge.

**Steps to reproduce**

1. Check out master at any commit from `af3023f1` onward.
2. Run `docker buildx build -f .github/docker-context-checks.Dockerfile
.` (the probe added by this PR), or start the real `Docker` workflow
build.
3. Observe `Generated contract drift:
generated/capability/capability-contract.md` — while `node
packages/paperclip-runner/scripts/generate-capability-contract.mjs
--check` passes on the same checkout outside Docker.

**Paperclip version or commit**

`d593463ab` (master tip at diagnosis time; first failing commit
`af3023f1`).

**Deployment mode**

GitHub Actions image builds (`docker.yml`), consumed by managed cloud
deployments.

## What Changed

- `.dockerignore`: narrow exceptions (last match wins) re-include the
committed drift-check outputs
(`!packages/paperclip-runner/generated/**`) and the inventory check's
documentation input
(`!packages/paperclip-runner/docs/capability-contract.md`). Every other
exclusion from #12769 stays: no crate declares an explicit `[[test]]`
target, so cargo builds without the `tests` directories, and the image
build chain never runs the excluded smoke scripts.
- `.github/docker-context-checks.Dockerfile` (new): a small probe that
COPYs the real build context — identical `.dockerignore` semantics — and
runs the dependency-independent drift checks inside it
(`generate-capability-contract.mjs --check`,
`check-capability-inventory.mjs`). ajv installs in an isolated directory
for schema validation only; codegen checks such as
`generate-protocol-schema-module` stay out because their emitted bytes
vary with the ajv release and would raise false drift alarms outside the
locked dependency tree.
- `.github/workflows/pr-trusted.yml`: new `docker_context_integrity` job
builds the probe on every full-CI pull request, and the existing
`verify` aggregate now requires its result, so the guard gates merges
through the same required check as the other lanes.
- Activation note: `pr.yml` pins `pr-trusted.yml` by commit SHA, so the
new job starts gating pull requests after the usual follow-up `ci:
activate ...` pin bump once this merges. The `.dockerignore` fix needs
no activation — `docker.yml` reads it directly, so image builds recover
on the first master commit after this merges.

## Verification

- `docker buildx build -f .github/docker-context-checks.Dockerfile .` on
master (before the `.dockerignore` fix): fails with the exact production
error, `Generated contract drift:
generated/capability/capability-contract.md`.
- Same command with the `.dockerignore` exceptions applied: passes,
which also proves BuildKit honors the `!` exceptions, including the file
inside the excluded `docs` directory.
- `node scripts/generate-capability-contract.mjs --check` on a full
checkout: passes both before and after, which confirms the committed
contract was never stale — only missing from the context.
- Static sweep of every script in the image build chain (`build`,
`build:typescript` and their `check:*` steps) against the ignore rules:
the three restored files are the only build inputs the #12769 block
strips.
- YAML for `pr-trusted.yml` lints clean.

## Risks

- Low. The `.dockerignore` exceptions only re-add three committed files
to the build context; image contents do not change otherwise.
- The probe job adds one context transfer and two Node scripts per
full-CI pull request run (about one to two minutes, no dependency
install beyond one isolated ajv package).
- The `verify` aggregate now also requires the new job, mirroring the
existing pattern for the other lanes; on non-full-CI runs the job skips
and `verify` asserts the skip, unchanged from how the other lanes
behave.
- The new job only takes effect for pull requests after a follow-up pin
bump in `pr.yml` (same two-step flow as every `pr-trusted.yml` change).

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Claude Fable 5 (Anthropic, model id `claude-fable-5`), extended
thinking, agentic tool use in Claude Code: GitHub Actions log forensics
to isolate the failing check, static analysis of the build-chain scripts
against the ignore rules, and local docker buildx runs to reproduce the
failure and verify the fix.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (the docker probe, both
failing-before and passing-after; the drift checks themselves on a full
checkout)
- [x] I have added or updated tests where applicable (the probe IS the
regression test for this class)
- [x] I have updated relevant documentation to reflect my changes
(inline comments in `.dockerignore` and the probe explain the invariant)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-09-04 15:35:10 -07:00
Dotta 4ef6155aae ci: harden paid runner browser and lock repair (#12829)
## Thinking Path

Paid cells now reuse the AWS image's system Chrome, but Playwright video
recording still resolves its revision-pinned FFmpeg helper from the
Playwright cache. Run 33875618534 proved Chrome qualification succeeds
and then failed before provider startup because that helper was absent.
The same run also exposed that generic lock repair can churn unrelated
package platform metadata, so the automated repair paths need
resolution-only regeneration rather than lockfile-only metadata refresh.

## What Changed

- install Playwright FFmpeg only on the AWS/system-Chrome path
- retry the small helper installation up to three times before provider
secrets are exposed
- keep the GitHub-hosted Chromium fallback unchanged
- bind static coverage to the exact FFmpeg step block and its pre-secret
ordering
- add pnpm `--resolution-only` to all four automated lock-repair paths
while retaining full transitive resolution
- require resolution-only repair in the shared workflow regression

The actual generated lockfile correction remains bot-owned by PR #12828
and is intentionally not committed here.

## Verification

- `node --test
.github/scripts/tests/lockfile-refresh-workflows.test.mjs`
- `actionlint -ignore SC2012` on all modified workflows
- focused Prettier checks
- `git diff --check`
- prior run 33875618534: system Chrome 151 qualified; missing Playwright
FFmpeg was the sole cell startup failure

## Risks

Low. The new network operation is limited to Playwright's pinned FFmpeg
payload, happens before paid credentials are exposed, and leaves the
hosted-runner path unchanged. Resolution-only is still a full
dependency-resolution pass, unlike lockfile-only, while avoiding
unrelated current-platform metadata churn.

## Model Used

GPT-5
2026-09-04 08:58:44 -05:00
Dotta a0a78ee609 ci: bootstrap Node before pnpm setup (#12808)
## Thinking Path
The trusted workflows currently invoke `pnpm/action-setup@v6` before
installing the repository Node version. On hosts whose ambient Node is
older than 22.13, the action downloads standalone `@pnpm/exe`, which has
repeatedly taken several minutes. Supplying Node 24 first lets the same
pinned pnpm action use its normal Node-backed path.

## What Changed
- install Node 24 before every trusted `pnpm/action-setup` invocation
- preserve the existing pnpm-store cache setup, pinned actions,
telemetry suppression, conditions, and secret boundaries
- enforce ordering, modern Node, condition parity, and cache counts in
the workflow security contract

## Verification
- workflow security: 7/7
- Prettier
- actionlint (excluding one pre-existing SC2129 in an untouched Daytona
shell block)
- `git diff --check`

## Risks
Low. Product code, providers, paid-runner selection, credentials, and
pnpm version are unchanged. Jobs that restore pnpm cache run
`setup-node` a second time after pnpm becomes available; the first setup
is deliberately cache-free.

## Model Used
Codex (GPT-5)
2026-09-03 21:18:08 -05:00
Dotta 0ad180b85f ci(runner): skip bootstrap registry telemetry (#12797)
## Thinking Path
Every trusted PR and paid-workflow job invokes the pinned pnpm setup
action. Its internal npm install is currently waiting four to seven
minutes on npm audit telemetry before any Paperclip or provider code
runs. Audit, funding, and update notifications are not integrity
controls for this action; its committed lockfile still verifies
installed package bytes.

## What Changed
- disable npm audit, funding, and update-notifier telemetry narrowly on
all seven pinned setup steps in each of the trusted PR and full-stack
workflows
- add a workflow security contract proving every setup invocation
remains covered and the overrides do not leak elsewhere

## Verification
- focused workflow security tests: 6/6 passed
- Prettier and git diff checks passed
- observed unhealthy setup: 4-7+ minutes; historical healthy setup:
about four seconds

## Risks
This skips npm vulnerability-report telemetry for the setup action
bootstrap only. Repository dependency checks, lockfile integrity,
provider-secret authorization, and target-lock verification remain
unchanged.

## Model Used
Codex (GPT-5)
2026-09-03 19:47:46 -05:00
fa16f88d6b chore(lockfile): refresh pnpm-lock.yaml (#12771)
Use full dependency resolution in automated lockfile repair paths, add regression coverage, and refresh the stale Rollup snapshot.

Co-Authored-By: Dotta <cryppadotta@users.noreply.github.com>
Co-Authored-By: Codex <codex@openai.com>
Co-Authored-By: lockfile-bot <lockfile-bot@users.noreply.github.com>
2026-09-03 11:08:34 -05:00
DottaandPaperclip 39b8ee2960 ci: optimize checks for stacked pull requests (#12507)
## Thinking Path

> - Paperclip uses GitHub Actions to protect changes before they enter
`master`.
> - GitHub evaluates every pull request in a native stack against the
stack base.
> - The current workflow therefore starts the complete CI matrix for
every layer in a stack.
> - A large stack can queue many copies of the same integrated
verification and delay every pull request.
> - GitHub provides stack position and base metadata so workflows can
select merge-relevant layers.
> - This pull request keeps policy and required check names on every
layer, but runs full CI only for ordinary pull requests, the top layer,
and the lowest unmerged layer.
> - The benefit is much lower CI load without weakening the
required-check contract.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The trusted pull request workflow currently runs every test, build,
canary, and E2E lane for every pull request in a native stack.

**Subsystem affected**

GitHub Actions pull request verification.

**Current behavior**

A stack with 61 pull requests can start 61 complete CI matrices after a
cascading rebase.

**Proposed behavior**

Run the always-on policy job and stable required-check aggregators for
every layer. Run the complete verification matrix only for ordinary pull
requests, the top stack layer, and the lowest unmerged stack layer.

**Reason and benefit**

The top layer verifies the integrated stack. The lowest unmerged layer
verifies the current merge candidate. Middle layers keep
branch-protection checks without consuming the complete runner matrix.

**Breaking changes**

Middle stack layers no longer run the complete CI matrix. Their `ci /
verify` and `ci / e2e` checks still require the policy job to pass and
require every expensive lane to be intentionally skipped.

## What Changed

- Add a fail-safe stack scope decision to the trusted PR runner gate.
- Run typecheck, general tests, build, serialized tests, canary, and E2E
shards only for ordinary, top, and lowest-unmerged pull requests.
- Preserve the required `ci / verify` and `ci / e2e` names on every
layer.
- Make the required aggregators distinguish valid middle-layer skips
from failures or missing scope decisions.
- Add regression coverage for ordinary, top, bottom, middle, and
malformed stack metadata.

## Verification

- `node --test scripts/__tests__/e2e-shard.test.mjs` — 11 tests passed.
- `actionlint .github/workflows/pr-trusted.yml .github/workflows/pr.yml`
— passed.
- `git diff --check origin/master...HEAD` — passed.
- The caller remains pinned to the current trusted workflow. A separate
activation change must advance the immutable SHA after this pull request
lands.

## Risks

- Incorrect stack classification could skip important jobs. Missing or
malformed stack metadata defaults to full CI.
- Middle-layer required checks depend on the policy job and verify that
all expensive jobs have the `skipped` result.
- The reusable workflow change does not become active until the
immutable caller SHA advances in a separate change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex with GPT-5. The exact deployment suffix and context
window are not exposed. The model used reasoning, repository tools, code
execution, Git, and GitHub API access.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-29 10:26:38 -05:00
Dotta 1da6b37fc5 fix(ci): regenerate stale stacked lockfiles (#12461)
## Thinking Path

> - Paperclip uses trusted GitHub Actions workflows to verify every pull
request
> - Native stacked pull requests use another pull request branch as
their base
> - A parent layer can change a package manifest without committing
`pnpm-lock.yaml`
> - A child layer can inherit that manifest change without changing a
manifest itself
> - The current policy skips lockfile regeneration for that child and
downstream frozen installs fail
> - This pull request validates the complete merge tree and shares a
regenerated lockfile only when needed
> - The benefit is reliable stacked pull request verification without
weakening the trusted workflow boundary

## Linked Issues or Issue Description

**What happened?**

A stacked child pull request inherited a package manifest change from
its parent. The child did not change a manifest itself. The policy job
skipped lockfile regeneration. Downstream jobs tried to restore an
artifact that did not exist and then failed during frozen dependency
installation.

**Expected behavior**

The policy job must validate the complete pull request merge tree. It
must upload a regenerated lockfile when the checked-in lockfile is
stale, including on a stacked child layer.

**Steps to reproduce**

1. Create a parent pull request that changes `package.json` without
committing `pnpm-lock.yaml`.
2. Create a child pull request on that branch without another manifest
change.
3. Run the trusted pull request workflow for the child.
4. Observe that frozen dependency installation fails because no
`pr-lockfile` artifact exists.

**Paperclip version or commit**

`f173ee09fa5c2ced7806bba47b54c3df853ab4df`

**Deployment mode**

GitHub Actions trusted pull request workflow.

**Agent adapter(s) involved**

Not adapter-specific. This is a core CI workflow bug.

## What Changed

- Regenerate the lockfile from every checked-out merge tree.
- Compare the generated lockfile with the checked-in copy before upload.
- Download the artifact only when the policy job reports that it
uploaded one.
- Fail closed when a reported artifact is missing.
- Add a workflow contract test for stacked lockfile handling.
- Keep the caller pinned to the last merged trusted SHA; after this
implementation merges, a separate activation PR will advance the
immutable pin to its merge commit.

## Verification

- `actionlint .github/workflows/pr-trusted.yml`
- `node --test scripts/__tests__/e2e-shard.test.mjs`

## Risks

- The policy job runs one lockfile-only install for every pull request.
This can add a small amount of CI time.
- A missing artifact now fails immediately when the policy job reports
an upload. This is intentional because it exposes workflow corruption.
- No runtime or product behavior changes.
- The implementation/activation split is intentional: unmerged
PR-authored workflow code must never execute on trusted runners.

## Model Used

OpenAI Codex with model `gpt-5`, reasoning, tool use, and code
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-08-28 15:02:12 -05:00
Dotta f9c32513b2 ci: ignore stale PR API base snapshots (#12462)
## Thinking Path

> - Paperclip uses pull request checks to protect changes.
> - The trusted CI gate selects GitHub or AWS runners.
> - GitHub can return an old pull request base SHA after the live base
advances.
> - The signed event, live Git ref, ancestry, and merge parents provide
the required proof.
> - The stale API field rejects a safe run even when those proofs pass.
> - This pull request removes that unreliable equality.
> - The benefit is correct AWS routing for trusted stacked pull
requests.

## Linked Issues or Issue Description

Refs #12339
Refs #12459

## What Changed

- Stop treating pull request base.sha as a current-state signal.
- Keep the signed event base SHA and live ref descendant check.
- Keep the live base or synthetic base merge-parent proof.
- Keep all numeric identity, repository, head SHA, and triggering actor
checks.

## Verification

- actionlint .github/workflows/pr-trusted.yml .github/workflows/pr.yml
- github-runners/tests/test-workflow.sh .github/workflows/pr-trusted.yml
.github/workflows/pr.yml
- The corrected gate selected the Fleet label with the exact live event
data from PR #12339 run 33201610330.

## Risks

- The pull request API base SHA can be stale and is no longer compared.
- Replaced ancestry, a changed head, a changed merge parent, and a
changed merge tree still fail closed.

## Model Used

- OpenAI Codex, GPT-5.6, with reasoning and terminal tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used with version and capability
details
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have linked existing public pull requests
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
ticket ID
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation in the operations repository
- [x] I have considered and documented risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open findings
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-08-28 14:02:36 -05:00
Dotta f929355fb9 ci: allow validated stacked base advances (#12459)
## Thinking Path

> - Paperclip uses pull request checks to protect changes.
> - The trusted CI workflow selects GitHub or AWS runners.
> - Stacked pull requests can advance their base branch while a gate
waits.
> - The gate already proves that the live base descends from the event
base.
> - An earlier exact base check rejects that safe state before the
ancestry check runs.
> - This pull request removes the conflicting check and verifies live
API consistency.
> - The benefit is automatic AWS routing for trusted stacked pull
requests without weaker identity checks.

## Linked Issues or Issue Description

Refs #12339
Refs #12457

## What Changed

- Allow the live pull request base SHA to advance from the signed event
base snapshot.
- Require the pull request API base SHA to match the live Git ref during
validation.
- Keep the numeric author, sender, and triggering actor checks.
- Keep descendant ancestry and synthetic merge validation.

## Verification

- actionlint .github/workflows/pr-trusted.yml .github/workflows/pr.yml
- github-runners/tests/test-workflow.sh .github/workflows/pr-trusted.yml
.github/workflows/pr.yml
- The routing suite covers a live stacked base advance and a base change
during validation.

## Risks

- A trusted stacked run can use a newer descendant base than its signed
event snapshot.
- Replaced ancestry still fails closed.
- A live base change during gate validation still fails closed.

## Model Used

- OpenAI Codex, GPT-5.6, with reasoning and terminal tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in
this pull request
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
ticket ID
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation in the operations repository
- [x] I have considered and documented risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open findings
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-08-28 13:53:09 -05:00
Dotta 7b199fcafa fix(ci): validate stacked PR merge refs (#12457)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Pull request checks protect the application and its contributors.
> - The trusted CI gate verifies signed event data against live GitHub
state.
> - GitHub gives a stacked pull request a synthetic merge commit for its
base stack.
> - The current validator expects the raw base SHA as the first merge
parent.
> - That assumption sends safe stacked pull requests to GitHub-hosted
runners.
> - This pull request validates the synthetic base merge and its
ancestry explicitly.
> - The benefit is safe automatic AWS routing for approved stacked pull
requests.

## Linked Issues or Issue Description

Refs #12455
Refs #12456

**What existing behavior does this improve?**

The trusted runner gate validates merge commits for master-based pull
requests but rejects GitHub synthetic base merges for stacked pull
requests.

**Subsystem affected**

GitHub Actions pull request identity and merge-state validation.

**Current behavior**

A trusted stacked pull request passes all numeric identity checks. The
gate fails closed because the first event merge parent is a GitHub
synthetic base merge instead of the raw base snapshot SHA.

**Proposed behavior**

The gate verifies the current base-ref tip, base-snapshot ancestry,
identical event and live merge parents, the child head parent, the
synthetic base merge parents, and identical event and live merge trees.

**Reason and benefit**

The change preserves fail-closed live-state validation while allowing
approved stacked pull requests to use the isolated AWS Fleet.

**Breaking changes**

None for untrusted contributors. Trusted stacked pull requests can
select AWS after the caller pins this workflow version.

**Additional context**

GitHub builds a stacked test merge in two steps. It first merges the
stack base into its own current base. It then uses that synthetic commit
as the first parent of the child test merge.

## What Changed

- Fetch and validate the current base branch ref.
- Require the event base snapshot to remain an ancestor of that ref.
- Validate direct and synthetic base merge parent shapes.
- Preserve the existing child-head, live-state, merge-tree, repository,
and actor checks.

## Verification

- Ran actionlint on the trusted workflow.
- Ran the external routing suite.
- Added positive coverage for the GitHub stacked merge shape.
- Added fail-closed coverage for replaced base ancestry and a synthetic
merge that omits the current base ref.
- Replayed the checks against the live merge shape for pull request
#12340.

## Risks

The gate has more GitHub API reads. Its five-minute timeout and
fail-closed behavior limit the effect of API errors. The accepted
synthetic commit must be the same in the event and live merge, must
include the current base branch as a direct parent, and must produce the
same merge tree.

> For core feature work, check ROADMAP.md first and discuss it in #dev
before opening the PR. Feature PRs that overlap with planned core work
may need to be redirected — check the roadmap first. See
CONTRIBUTING.md.

## Model Used

OpenAI Codex on GPT-5. The exact deployment ID and context-window size
are not exposed. The model used reasoning, tool use, GitHub API access,
and local code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used with version and capability
details
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have linked existing public items and described the issue in-PR
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-08-28 13:34:36 -05:00
Dotta d6b33d6c16 fix(ci): route trusted stacked PRs to AWS (#12455)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Pull request checks protect the application and its contributors.
> - The trusted CI workflow sends approved contributors to isolated AWS
runners.
> - The workflow currently limits AWS routing to pull requests that
target master.
> - Stacked pull requests target another branch and stay in the GitHub
queue.
> - This pull request keeps the identity checks and accepts a live
nonempty base branch in the Paperclip repository.
> - The benefit is that trusted stacked pull requests can use the AWS
fleet automatically.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The trusted pull request runner gate currently sends only master-based
pull requests to AWS.

**Subsystem affected**

GitHub Actions pull request routing.

**Current behavior**

A trusted contributor can pass all numeric identity checks. The gate
still selects GitHub-hosted runners when the pull request targets
another branch in a stack.

**Proposed behavior**

The gate accepts any nonempty base branch in the Paperclip base
repository. It still verifies the live base ref, base SHA, head SHA,
merge SHA, author ID, sender ID, and triggering actor ID.

**Reason and benefit**

Large pull request stacks currently add all heavy jobs to the limited
GitHub-hosted queue. This change lets approved contributors use the
isolated AWS fleet for those jobs.

**Breaking changes**

Trusted pull requests that target a non-master branch now use AWS
instead of GitHub-hosted runners. Untrusted pull requests keep the
current GitHub-hosted route.

**Additional context**

Pull request #12339 is the master-based first item in a stack. Its child
pull requests show this queue pattern.

## What Changed

- Replace the master-only route check with a nonempty base-ref check.
Keep the existing base-repository and live-state checks.

## Verification

- Ran actionlint on .github/workflows/pr-trusted.yml.
- Ran the external routing test suite. It passed the trusted
stacked-base case and all fail-closed identity cases.

## Risks

The AWS trust boundary now includes code from a non-master base branch
when the current pull request author, sender, and triggering actor are
all approved numeric GitHub IDs. An approved account can already submit
arbitrary head code. The runner remains isolated and has no production
access or repository secrets.

> For core feature work, check ROADMAP.md first and discuss it in #dev
before opening the PR. Feature PRs that overlap with planned core work
may need to be redirected — check the roadmap first. See
CONTRIBUTING.md.

## Model Used

OpenAI Codex on GPT-5. The exact deployment ID and context-window size
are not exposed. The model used reasoning, tool use, GitHub API access,
and local code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with Fixes: # / Closes #
/ Refs # OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-08-28 13:20:30 -05:00
Dotta c119c4bee6 fix(ci): validate the PR base snapshot (#12449)
## Summary

- validate the event base ref/SHA against the live PR state instead of
requiring the moving `master` branch tip to remain unchanged while a
hosted gate queues
- retain exact event/live merge parent and tree validation, plus
author/sender/rerun checks

## Canary finding

A seven-minute hosted-gate queue allowed `master` to advance. Requiring
the live branch tip to equal the event base snapshot would route
otherwise valid trusted runs back to GitHub-hosted indefinitely on a
busy repository.

## Validation

- actionlint and workflow-contract tests pass
- internal routing harness passes
- replaced PR base snapshot, stale head, changed merge parent/tree, and
untrusted actors all remain fail-closed
- AWS routing remains disabled during rotation
2026-08-28 11:35:41 -05:00
Dotta 5ac66b3fdd fix(ci): validate the workflow merge ref (#12447)
## Summary

- use `${{ github.sha }}` as the event merge commit validated by the
trusted gate
- retain exact base/head parent and identical-tree comparison against
the current live merge ref

## Canary finding

GitHub leaves `pull_request.merge_commit_sha` empty on some `opened`
payloads even though the workflow runs against a valid merge ref. The
gate safely fell back to GitHub-hosted and no EC2 instance launched.

## Validation

- `actionlint .github/workflows/pr-trusted.yml .github/workflows/pr.yml`
- `node --test ./scripts/__tests__/e2e-shard.test.mjs`
- internal routing harness passes and asserts `EVENT_MERGE_SHA` is
sourced from `${{ github.sha }}`
- AWS routing remains disabled during rotation
2026-08-28 11:33:38 -05:00
Dotta b88fadb039 fix(ci): emit one runner route (#12444)
## Summary

- emit exactly one `runner` job output from the trusted gate
- write `ubuntu-latest` only inside fail-closed paths
- write the Fleet label only after every identity, PR-state,
merge-equivalence, and rerun-actor check passes

## Canary finding

The live gate reached the trusted success notice, but GitHub retained
the first of two duplicate `runner=` outputs, so policy still requested
`ubuntu-latest`. No EC2 instance launched. Routing was disabled
immediately.

## Validation

- `actionlint .github/workflows/pr-trusted.yml .github/workflows/pr.yml`
- `node --test ./scripts/__tests__/e2e-shard.test.mjs`
- internal routing harness passes and now requires exactly one runner
output in all cases
- AWS routing remains disabled during rotation
2026-08-28 11:22:05 -05:00
Dotta 3b295b05dc fix(ci): validate equivalent PR merge refs (#12441)
## Summary

- validate the live master ref and event base SHA before AWS routing
- accept GitHub synthetic merge commits only when the event and live
commits have the exact expected base/head parents and identical tree
- preserve fail-closed routing for malformed, stale, replaced, or
untrusted events

## Canary finding

A trusted reopened PR produced two synthetic merge SHAs with different
timestamps but identical current base/head parents and tree. The former
exact-SHA comparison safely fell back to GitHub-hosted runners, but
could not route a valid event to AWS.

## Validation

- `actionlint .github/workflows/pr-trusted.yml .github/workflows/pr.yml`
- `node --test ./scripts/__tests__/e2e-shard.test.mjs`
- real-event gate simulation selects the Fleet label for equivalent
merge commits
- negative simulations keep an untrusted sender and replaced head on
`ubuntu-latest`
- AWS routing remains disabled during rotation
2026-08-28 11:16:42 -05:00
Dotta d9fc93d838 ci: synchronize trusted PR policy (#12438)
## Thinking Path

> - Paperclip uses pull request CI to protect changes before merge
> - The trusted reusable workflow will select isolated AWS capacity
> - The active workflow changed while the reusable workflow waited for
merge
> - The reusable policy must contain every current CI policy step before
activation
> - This pull request synchronizes the migration-order check and shell
validation
> - The benefit is one reviewed workflow version with verified job
parity

## Linked Issues or Issue Description

Refs #12436

**What existing behavior does this improve?**

This improves the pull request CI workflow synchronization before AWS
runner activation.

**Subsystem affected**

Cross-cutting CI automation.

**Current behavior**

The active workflow validates migration order. The new reusable workflow
does not yet contain that check.

**Proposed behavior**

Both workflow definitions contain the same heavy jobs and policy steps
before the active workflow becomes a thin caller.

**Reason and benefit**

The synchronization prevents policy drift during the two-step secure
rollout.

**Breaking changes**

None. AWS routing remains disabled.

## What Changed

- Added the current migration-order validation to the trusted workflow.
- Added the existing shellcheck intent annotation to the active
workflow.
- Verified normalized heavy-job parity between both definitions.

## Verification

- actionlint on both workflow files
- Local trusted-routing and normalized workflow-parity tests
- git diff --check

## Risks

Low risk. The migration check already runs in active CI. This change
copies it into the inactive trusted definition. AWS routing stays
disabled.

## Model Used

OpenAI Codex with GPT-5, tool use, and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked a related public PR or described the issue
with the matching template fields
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
2026-08-28 10:39:53 -05:00
Dotta 07b6816829 ci: add trusted reusable PR workflow (#12436)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Pull request checks protect the quality of the control plane
> - The current checks depend only on the shared GitHub-hosted runner
limit
> - Busy periods leave many pull request jobs queued even when external
capacity is available
> - Public pull request code must not select or directly access private
runner infrastructure
> - This pull request adds an inactive reusable workflow with a
fail-closed identity gate
> - A later pull request can pin this workflow by its full master commit
SHA
> - The benefit is automatic, controlled access to isolated runner
capacity without changing current CI during bootstrap

## Linked Issues or Issue Description

**What existing behavior does this improve?**

This improves the pull request CI workflow. It prepares the existing
checks to use an administrator-controlled runner selection.

**Subsystem affected**

Cross-cutting GitHub Actions CI configuration.

**Current behavior**

Every pull request job uses `ubuntu-latest`. Jobs wait when the
GitHub-hosted concurrency limit is full.

**Proposed behavior**

Add a reusable copy of the current PR workflow. A GitHub-hosted gate
validates durable numeric user IDs and current GitHub API state. The
gate emits one runner label. The default and every validation failure
use `ubuntu-latest`. The AWS label is possible only when an
administrator enables it and every identity check passes.

This bootstrap pull request does not change the active
`.github/workflows/pr.yml` caller. A follow-up change will call this
workflow by the full master commit SHA.

**Reason and benefit**

The split bootstrap creates an immutable trust boundary before external
runners are reachable. It also keeps CI automatic for contributors.
Contributors do not select a runner.

**Breaking changes**

None in this bootstrap pull request. The active PR workflow does not
change.

## What Changed

- Added an inactive `workflow_call` copy of the current PR checks.
- Added a GitHub-hosted routing gate that checks the repository ID, pull
request author ID, event sender ID, rerun actor ID, base branch, head
SHA, merge SHA, and current pull request state.
- Made every validation failure select `ubuntu-latest`.
- Pinned every third-party action to a full commit SHA.
- Disabled persistent checkout credentials for all jobs.
- Limited the workflow token to Actions read, contents read, and pull
request read access.

## Verification

- `actionlint .github/workflows/pr-trusted.yml`
- Ran the dedicated workflow routing test harness against
`.github/workflows/pr-trusted.yml`.
- Compared the job keys with `.github/workflows/pr.yml`. The new
workflow contains every existing job plus the gate.
- Verified each pinned action commit against its current GitHub
major-version tag.

## Risks

The gate could route a trusted pull request to the wrong runner if an
identity check is incomplete. The gate checks durable numeric IDs from
the event and current GitHub API state. It checks the rerun actor
separately. It defaults to GitHub-hosted capacity before any validation
runs.

This file is inactive in this pull request. The follow-up caller and
runner-group restriction must use the exact commit that reaches
`master`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI Codex based on GPT-5. The exact serving model ID and
context-window size are not exposed in this environment. The model used
high-reasoning, terminal, GitHub API, browser, and web-research
capabilities.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-08-28 10:34:12 -05:00