Commit Graph
9 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 64ec273845 perf(ci): run the typecheck lane's checks in parallel and drop its duplicate server type check (#21404)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every pull request runs the trusted PR workflow, and the `Typecheck
+ Release Registry` job is one of its required checks
> - That job takes about 9.2 minutes on the 4 vCPU / 16 GB fleet runner
(run 43344818387), against a target of 7 minutes or less for every job
> - The job runs six steps one after another, and the first step
type-checks the whole server program a second time, at about 13.5 GiB,
which makes the runner swap
> - Typecheck results are the check that authors most want back quickly,
so this job is on the critical path of most pull requests
> - This pull request removes the duplicate server type check and runs
the independent checks in parallel through one small script, inside the
same memory budget
> - The benefit is about 39% less wall time for the job's checks on the
same machine, with peak memory down from about 14 GiB (plus swap) to
about 8.5 GiB

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The `Typecheck + Release Registry` job in
`.github/workflows/pr-trusted.yml`. It takes about 9.2 minutes. The
target for every job in the workflow is 7 minutes or less.

**Subsystem affected**

CI (`.github/workflows/pr-trusted.yml`) and the typecheck scripts under
`scripts/` and `server/scripts/`.

**Current behavior**

After about 75 s of setup, the job runs six steps in sequence on a 4
vCPU runner, about 470 s in all (run 43344818387):
`typecheck:build-gaps` 121 s, test programs 85 s, server layers 128 s,
test projects 38 s, CLI guidance test 5 s, release registry tests 95 s.
Most of these steps use one thread. The `typecheck:build-gaps` step
builds the server with a full type check of the whole server program.
The server layer step then checks the same files again, and the `Build`
job checks them a third time. That full program peaks at about 13.5 GiB,
so the job peaks at about 14 GiB of 16 GiB and uses swap.

**Proposed behavior**

The job runs `node scripts/ci-typecheck-lane.mjs`. The script builds the
server without type checks (`PAPERCLIP_TSC_EMIT_ONLY=1`, as the Docker
image does) and still checks the runtime assets. It runs the steps that
write shared files first, one at a time. Then it runs the remaining
checks in parallel, one per CPU, with the server declaration layers
first, because the test layers and test projects read them. The checks
that are not on that path run at a lower CPU priority.

**Reason and benefit**

On the same runner, the job's checks take 61% of the time and about 60%
of the memory (see Verification). The job should take about 6 minutes on
the fleet instead of 9.2.

**Breaking changes**

None. The same files are type-checked with the same options. Local
commands do not change. `pnpm run typecheck:build-gaps` still works as
before.

## What Changed

- `scripts/ci-typecheck-lane.mjs` (new): runs the job's checks. It
writes shared outputs first, in sequence: links, plugin SDK, server
build without type checks, the runtime asset check, the sandbox provider
installs, and the Runner typecheck (it rebuilds the eval kernel, which
other checks read). Then it runs these checks in parallel: the server
declaration layers, each server test layer, each test project, the test
programs, the other build-gap workspaces, the CLI guidance test and the
release registry tests. A passing check prints its output in a collapsed
log group. A failing check prints its full output between `=== FAILED`
and `=== end of` lines, with an error annotation. The script ends with a
table of each check's start, duration and result, and exits non-zero if
any check fails.
- `server/scripts/typecheck-layers.mjs`: new `--layer <name>` option
that checks one layer against the configs and declarations of an earlier
`--declarations-only` run, and an exported `listLayers`.
- `scripts/typecheck-test-programs.mjs`: new `--prepare-only` and
`--skip-prepare` options, so the installs happen before the parallel
checks start.
- `scripts/run-typecheck-build-gaps.mjs`: new `--list-packages` option,
so the script still finds the build-gap workspaces and still fails when
one has no `typecheck` script.
- `.github/workflows/pr-trusted.yml`: the `typecheck_release_registry`
job runs the script instead of the six steps. No other job changes.
- Tests: new `scripts/__tests__/ci-typecheck-lane.test.mjs` (added to
`test:release-registry`), new cases in
`server/scripts/typecheck-layers.test.mjs` and
`scripts/typecheck-test-programs.test.mjs`, and the workflow wiring
assertions in `.github/scripts/tests/ci-plan.test.mjs` and
`scripts/__tests__/runner-release-crate-cache.test.mjs`.
- `tests/runner-e2e/source-sdk-public-inputs.json`: repinned, because
`package.json` changed.
- `doc/DEVELOPING.md`: new "Typecheck lane in CI" section, and the new
options.

## Verification

This PR's own CI ran the old job definition, because PR runs use the
workflow from master. So I measured the commands on a stand-in machine:
a temporary workflow (`.github/workflows/typecheck-lane-bench-temp.yml`,
now deleted) on GitHub-hosted `ubuntu-latest` (4 vCPU Intel Xeon
6975P-C, 15.7 GiB, no secrets, `contents: read`). It ran in runs
[43444356017](https://github.com/paperclipai/paperclip/actions/runs/43444356017),
[43445393958](https://github.com/paperclipai/paperclip/actions/runs/43445393958),
[43446786105](https://github.com/paperclipai/paperclip/actions/runs/43446786105),
[43448707837](https://github.com/paperclipai/paperclip/actions/runs/43448707837)
and
[43450786493](https://github.com/paperclipai/paperclip/actions/runs/43450786493).
Each step ran under `/usr/bin/time -v`, and a sampler read
`/proc/meminfo` every 0.25 s.

**Old steps, one at a time (3 runs, run 43445393958):**

| Step | Wall s | CPU s | Peak RSS of one process | System memory peak |
| --- | --- | --- | --- | --- |
| `typecheck:build-gaps` | 128–155 | 376–400 | 13.2–13.6 GiB | 14.2–14.7
GiB |
| ...of which the full server `tsc` | 40–51 | 146–164 | 12.8–13.8 GiB |
13.7–14.7 GiB, swap up to 738 MiB |
| ...replaced by the emit-only server build | 6–10 | 20–22 | 2.9 GiB |
3.9 GiB |
| Test programs | 67–87 | 147–170 | 1.9 GiB | 2.9 GiB |
| Server layers | 102–127 | 150–172 | 4.0–4.2 GiB | 5.4–5.6 GiB |
| Test projects | 31–37 | 49–59 | 2.1–2.2 GiB | 3.1–3.2 GiB |
| CLI guidance test | 2–4 | 2–3 | 0.2 GiB | 1.6 GiB |
| Release registry tests | 82–95 | 159–183 | 1.7 GiB | 3.0 GiB |

CPU seconds are about twice the wall seconds for the `tsc` steps even on
one checker thread. So each step uses about two of the four vCPUs.

**Old sequence and new lane on the same runner (4 runs, run
43450786493).** Each job copied one installed checkout, then ran both,
in alternating order:

| Run | Old steps, wall s | New lane, wall s | Ratio | Old memory peak |
New memory peak |
| --- | --- | --- | --- | --- | --- |
| 1 | 503 | 314 | 0.62 | 13.98 GiB | 8.33 GiB |
| 2 | 647 | 388 | 0.60 | 14.04 GiB, swap 637 MiB | 8.13 GiB |
| 3 | 627 | 385 | 0.61 | 13.98 GiB | 8.63 GiB |
| 4 | 382 | 237 | 0.62 | 14.10 GiB, swap 1.1 GiB | 7.99 GiB |

Hosted runners differ in speed by up to 1.7 times, so only the
same-runner ratio is meaningful. Both arms include a cold compile of the
Runner release crate (45–80 s) that the fleet restores from cache. The
fleet ran the old steps in 470 s; at the measured ratio, the new lane
should take about 290 s, and the job about 6 minutes with setup. The new
lane peaks at about 55% of memory, with no swap.

Local checks (on a loaded Mac, for correctness only):

- `node --test scripts/__tests__/ci-typecheck-lane.test.mjs` — 8 pass.
- `node --test .github/scripts/tests/*.test.mjs` — 807 pass.
- `node --test scripts/__tests__/e2e-shard.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs
scripts/__tests__/runner-release-crate-cache.test.mjs
scripts/__tests__/run-typecheck-build-gaps.test.mjs` — pass.
- `pnpm exec vitest run scripts/typecheck-layers.test.mjs` in `server/`
— the new cases pass.

## Risks

- This changes a required job that every PR depends on, and its own CI
cannot prove it. Please watch the first `Typecheck + Release Registry`
runs on master after merge: the wall time (expect about 6 minutes), the
summary table at the end of the step, and any `terminated by` message.
- A type error now shows in that check's `=== FAILED` block, not in a
step of its own. Passing checks are folded, so the failure is the only
expanded block.
- When the server declaration layers fail, the test layers and test
projects are reported as skipped. Before, the test layers still ran
after a source error. Fix the source errors first; the next run checks
the tests.
- The Honeycomb CI time board and any step-name queries see one step,
`Typecheck, CLI guidance safety and release registry tests`, instead of
six. The script prints the duration of each check.
- The release registry tests run beside the type checks at normal
priority. Their slowest test took 46–78 s here, against a 180 s timeout,
and 114 s on the slowest stand-in runner.
- Low risk for memory: the new lane peaked at 8.0–8.6 GiB of 15.7 GiB,
far below the old 14 GiB.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool
use (shell, GitHub CLI, file editing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:06:01 -06:00
Nicky LeachandClaude Fable 5.2 84a4d345b3 refactor(docker): take the application image's Runner from the runner-artifacts stage (#21332)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work. `master` publishes
it alone as the image `ghcr.io/paperclipai/paperclip-runner`, built by
the `runner-artifacts` stages of the `Dockerfile`
> - The application image does not use those stages. It has five stages
of its own that compile the daemon, it builds the Runner again inside
the server build, and it builds a second provider pack
> - So one `Dockerfile` holds two recipes for the Runner, and the two
results differ: another base image and another daemon binary
> - This pull request removes the second recipe. The application image
copies the Runner out of the `runner-artifacts` stage, which the same
build runs once
> - The benefit is one Runner recipe, and an application image whose
Runner is the Runner the contract describes. A later pull request can
replace the stage with the published image

## Linked Issues or Issue Description

This is the second part of a chain. The first part is #21331, which
removes the cache check that cannot run on this `Dockerfile`. Refs
#21204, #21229, #21230 (the Runner image), #21306 (the no-rebuild mode
of the vendor step), #21307 (the pull-request check of the application
image) and #21312. No issue exists for this change.

**Problem or motivation**

The stages `rust-toolchain`, `rust-chef`, `runner-plan`, `runner-deps`
and `runner-build` compile the daemon on Debian trixie. The server build
then runs the full build of the Runner package. The
`runner-provider-pack` stage builds the provider pack. The
`runner-artifacts` stages do all of that again for the Runner image, on
Debian bookworm, with a stripped daemon.

**Proposed solution**

Read only the `runner-artifacts` stage from the application stages.
Build it inline in every build. Do not pass a published image yet.

**Alternatives considered**

- Pass the published Runner image to the build in this pull request.
That needs the resolver, labels and a rule for commits without an image.
It is the next step, and this one must be safe alone.
- Keep the old stages as a fallback. Two recipes is the problem.

## What Changed

- `Dockerfile`: remove the five stages. `build` starts from `base`. It
copies `/runner-package/dist` and the build stamp from
`runner-artifacts`, sets `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` and
runs the vendor check before the first application build.
- `Dockerfile`: `runner-provider-pack` starts from `base`. One `RUN`
with a bind mount of `runner-artifacts` copies the pack, the receipt and
the archives, and moves the large files to their own tree. `build` and
`runner-provider-pack` now come after the stages they read.
- `Dockerfile`: the three `COPY` instructions of `production` that take
the pack are byte-identical. A test pins all three.
- `Dockerfile`: `build` declares the build argument
`PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST`, empty by default, and one step
runs the new `scripts/check-runner-image-expectation.mjs` on
`/runner-image.json` of `runner-artifacts`. **This pull request declares
the argument. No build sets it. The next pull request, which gives a
published Runner image to the build, is its first user.** With the
argument set, the build fails unless `runner-artifacts` is the published
image of exactly those inputs. `docker-app-check.yml` also runs when the
script changes.
- Comments: the `Dockerfile` has 220 comment lines, down from 247. The
comments of the two moved stages moved with them, unchanged, so the diff
shows them as added lines. Eight comments that named a removed stage or
a step that moved get the smallest correction. Two comments in
`prepare-runner-vendor.mjs` and `build-inputs.mjs` said that the
`Dockerfile` relies on the standalone rebuild; that clause is removed.
- Tests: replace the two cargo-chef tests of
`docker-build-stamp.test.ts` with three tests of the new shape. Update
the pack test, `docker-image-layer-reuse.test.ts`,
`vite-build-commit.test.ts`, the stage rules of
`runner-image-identity.test.ts`, and the include-layout test of
`runner-release-crate-cache.test.mjs`, which now reads
`runner-artifacts-runnerd`.
- Docs: `doc/DOCKER.md`, `doc/DEVELOPING.md`, and a new section "The
application image" in `doc/runner-image-contract.md`.

## Verification

All builds: `linux/arm64`, local, `--build-arg
PAPERCLIP_BUILD_COMMIT=<sha>` with one commit for every build. The
builds ran before the rebase onto `master`, on the head of #21310 as it
was before it merged. `master` changed neither the `Dockerfile` nor an
input of the Runner in between.

- `docker buildx build --target production`, cold, before and after: 506
s and 410 s. Compressed size: 2,971 MB and 2,900 MB.
- The image config and the text of all 33 history rows are identical.
The row `COPY /provider-pack-transport/ /opt/paperclip-runner/ #
buildkit` matches the pattern of `retained-provider-pack-oci.ts` exactly
once.
- A comparison of every file of the two images (path, type, size, mode,
link target, and SHA-256 under `/app` and `/opt`). The differences are
in "Risks". All other files are byte-identical, the root `node_modules`
included.
- From the image after: the four `/app/packages/paperclip-runner/...`
paths and the contents of `/opt/paperclip-runner/` exist.
`paperclip-runnerd --build-metadata` runs. `provider-pack.json` has
`runnerSourceRevision` equal to the build commit and no
`runnerInputsDigest`. The entry points `index.js`, `live/index.js` and
`testing.js` of the Runner import.
- The vendor step logged `reusing the current Runner build` in both of
its runs. Every `Compiling` line of the log is in
`runner-artifacts-runnerd`.
- `--target build`, `--target runner-provider-pack` and `--target cloud`
built. The `build` stage has no `gcc`, `cc`, `cargo`, `rustc` or
`pkg-config`.
- `--target runner-artifacts` and `runner-artifacts-slim`, before and
after: `runner-image-layout.mjs check` and `runner-image-oci.mjs
check-layers` pass on both, in `node:24.22.3-bookworm`. The daemon,
`provider-pack.json` and the two stable layers have the same digests
before and after.
- Without a build commit: `build` and `runner-provider-pack` built, and
the pack was skipped with the same message. This run was on the next
branch of this chain, after the rebase. The last steps of `production`
were not run for this case.
- The expectation argument, on this `Dockerfile`, `--target build
--output type=cacheonly`, after the rebase:
- No argument, inline Runner: the build passes (233 s). The step passes.
- Argument, inline Runner: the build fails at the step. The message says
that the `runner-artifacts` of the build are not a published Runner
image.
- Argument `d4df566f…` and the published image of those inputs as
`--build-context
runner-artifacts=docker-image://ghcr.io/paperclipai/paperclip-runner@sha256:0cd8250b…`:
the step passes, and the build runs no Runner stage. The vendor step
then fails with `inputs changed since the recorded build`, because this
tree differs from the tree of that image by one comment line in
`build-inputs.mjs`. With that line put back, the same build completes,
and the vendor step reuses the published build.
- Another digest with the same context: the build fails at the step. The
message names the inputs the image has.
- A tree whose lockfile is behind the manifests of the Runner fails at
the frozen install of the Runner stages, after 169 s, with the message
that stage already had.
- `docker buildx build --check`: no warning, before and after.
`scripts/check-docker-deps-stage.mjs` and
`scripts/check-node-version-policy.mjs` pass.
- Tests, on macOS and in `node:24.22.3-bookworm`, after the rebase: the
changed server test files with the other tests that read the
`Dockerfile` or the image workflows (231 pass in 10 files on Linux),
`vite-build-commit.test.ts` (10 pass),
`runner-release-crate-cache.test.mjs` (11 pass),
`docker-deps-postinstall.test.mjs`. The changed server test files
typecheck.
- Not run: the full test suite, any `linux/amd64` build, `podman`, the
workflows and the release pipeline. Only CI can prove amd64,
`docker.yml`, `docker-merge-group.yml`, `release.yml`,
`docker-app-check.yml` and `runner-image.yml`.

## Risks

- **The digest of the Runner image changes.** Its identity covers the
whole `Dockerfile`. The next `master` push publishes a new
`inputs-<digest>`. The contents do not change: see the comparison above.
- **A build that is given a Runner of other inputs fails; it does not
ship that Runner.** With the argument, it fails at the expectation step.
Without the argument, it fails at the vendor step when the build inputs
of the Runner differ. One comment line in `build-inputs.mjs` changes
those build inputs in this pull request, so the image published before
this pull request cannot be vendored into a tree after it.
- **What a consumer of the application image sees** (measured on
`linux/arm64`):
- The daemon is the stripped binary compiled on Debian bookworm: 23.2
MB, was 36.4 MB, at
`/app/packages/paperclip-runner/dist/bin/paperclip-runnerd` and at the
vendored copy in `server/dist`. Its hash changes. It has no symbols for
a crash backtrace.
- `/app/packages/paperclip-eval-kernel/dist` is absent. Only
`dist/eval/workflow-harness.js` of the Runner imports that package. Only
`eval/index.js` exports the harness again, and no module imports that
file. The server runtime bundle does not name the package.
- The provider pack: `provider-pack.json` is byte-identical. Two
provider executables and one licence file are hard links to their other
copy in the pack, where they were separate files. One empty directory is
absent. The pack is 195 MB smaller, and the layer of its large files
gets a new digest once.
- The transport archives have other bytes, because the Runner build
gives large files a fixed timestamp. The receipt names the same files
with the same digests.
- Paths, modes, owners, the image config and the layer order do not
change.
- **What a contributor and a maintainer of the image build see:**
- **The image build does not run the drift checks of the Runner.**
Before, the server build ran the full `build` script of the Runner, so a
failed drift check failed the image build on `master`. Now the Runner
stages run only the output steps of that script. Repository CI runs the
checks on every pull request. `.github/docker-context-checks.Dockerfile`
and three exceptions in `.dockerignore` exist for those checks in the
image build; a separate follow-up removes them, and this chain does not
edit them.
- **The dependency-caching layer of the daemon is gone.** Before,
cargo-chef compiled the Rust dependencies in a layer of their own, and a
change to the Rust source compiled only the Runner's own crates. Now
such a change compiles the dependencies again (146 s here for the whole
daemon). A change to the server, the UI or the Runner's TypeScript still
reuses the compiled daemon.
- **A lockfile behind the manifests of the Runner fails the build.**
Before, `docker build` resolved it inside the image. The image workflows
refresh the lockfile first. A local build prints the command to run. The
next pull request makes this failure early and clear.
- **The build needs more disk**: about 25 GB of builder storage here,
was about 20 GB, with the image export.
- Rollback: revert this pull request. Images that exist do not change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude
Code. The model used tools: it edited files, ran the tests and ran the
Docker builds that this description reports.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 09:48:49 -08:00
DottaandClaude Opus 5.5 b5b972ceb1 fix(runner): stop provider tool processes that leave the owned process group (#19727)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The local runner (`paperclip-runnerd`) supervises each provider
process, such as the ACPX sidecar for Claude, in its own process group
> - Run teardown stops that process group, and nothing else
> - Claude Code starts each Bash tool command in a new session
(`setsid`). A command such as `nohup ./bench.sh & echo $!` leaves a job
in that session, and the wrapper exits at once. The job is then a child
of init and outside the owned group
> - In a restart chaos storm, a run failed after a restart resume. Its
tool job kept running beside the task's next run, in the same workspace,
for more than four minutes
> - This pull request makes the supervisor record every process that its
provider starts, including processes that leave the group, and stop them
all before teardown reports success
> - The benefit is that a run's tools cannot outlive the run and cause
duplicate side effects in the next run

## Linked Issues or Issue Description

**What happened?**
A restart chaos storm ran a local ACPX Claude task. After a restart
resume, the agent ran `nohup ./bench.sh <nonce> > log 2>&1 & echo
"started pid $!"` and then ended its turn without a result. The run
failed with `native_finalization_missing`. The runner tore down the
sidecar group, and the guardian killed the provider group. `bench.sh`
was in the session that Claude Code made for the tool (pgid and sid =
the exited wrapper, ppid = 1). It was never signalled. It kept forking
`sleep 1` while the task's next run started and finished in the same
workspace.

**Steps to reproduce**
1. Run a local ACPX Claude task whose agent runs a Bash tool command
such as `nohup ./bench.sh x > log 2>&1 & echo $!`.
2. End the run (any outcome, including failure after a restart resume).
3. Run `ps -axo pid,ppid,pgid,command | grep bench.sh`. The job is still
running with ppid 1.

**Agent adapter(s) involved**
Claude Code (local ACPX). The gap is in the shared supervisor, so any
provider whose tools leave the group is affected.

**Root cause**
`SupervisedProcess`
(`packages/paperclip-runner/runner/crates/runner-core/src/process_supervisor.rs`)
spawns with `process_group(0)` and stops only that group in
`terminate_group`, `terminate_group_immediately`, `wait` and `Drop`. Its
own comment said that a caller that must exclude escaped or re-parented
descendants needs a separate lifetime fence. No caller had one. The ACPX
sidecar transport (`acpx_sidecar_transport.rs`, `shutdown` and `poison`)
and the ACPX provider guardian (`installation-integrity.ts`, which kills
only its own group) therefore left every `setsid` tool job behind. The
failure path after the resume was not special: the same gap applies to
every teardown.

**Expected behavior**
When a run ends for any reason, every process that its provider or tools
started stops, and teardown proves that they are gone. Identity comes
from PID, group, session and start time, never from a command name.

## What Changed

- Add the `paperclip-process-table` crate. It reads PID, parent, process
group and start time for every process (libproc on macOS, `/proc` on
Linux). `runner-core` forbids unsafe code, so the two checked libproc
calls live in this small crate.
- Add `process_lineage.rs` to `runner-core`. A sampler thread (every 50
ms) records each descendant of the supervised process by PID and start
time. It also records each group and session that a descendant leads,
with the leader's start time. A process belongs to the lineage when its
parent belongs, or when it is in a recorded group or session and started
no earlier than its leader. Groups and sessions that the supervised
process only inherited (runnerd's session) are never recorded. A
recorded ID whose PID now names a different process is dropped.
- `SupervisedProcess` samples the lineage just before it signals, and
after the leader is reaped it stops every live member: TERM, then KILL
after the shutdown grace. It fails with an error until it proves that no
member is alive (grace plus 5 s). This applies to `terminate_group`,
`terminate_group_immediately`, `wait` and `Drop`.
- If the lineage sampler cannot start, the supervisor stops the process
it just started and reports the failure, so a provider never runs
without tool cleanup.
- Table reads and state updates happen under one lock, so an older read
cannot erase members that a newer read recorded.
- The process table reports processes that exist but cannot be read. A
recorded member that cannot be read stays unresolved, and teardown fails
until its exit is proven.
- The Runner workspace now has two members.
`scripts/runner-release-crate-cache.mjs` and
`scripts/runner-debug-crate-cache.mjs` cache and mark current the
process-table lib unit with the Runner crate (otherwise Cargo rebuilds
the Runner crate after every restore). The debug contract accepts only
this library-only member and still refuses any other workspace change.
The workflow cache path lists and the debug clean step are updated to
match.
- `terminate_group_immediately` now accepts `EPERM` from the group
signal when the leader has already exited. macOS returns `EPERM` for a
group whose only member is the unreaped leader. The lineage pass then
proves that no other member remains.

## Verification

- `cargo test -p paperclip-runner-core --lib -- process_lineage tool_job
immediate_group_fence` (macOS, local).
- New real-process tests in `process_supervisor.rs`: a supervised shell
starts a wrapper with `setsid` (Perl `POSIX::setsid`), the wrapper runs
`nohup <loop> &`, writes the job PID and exits. The test asserts that
the job is outside the owned group and re-parented, then calls
`terminate_group`, `terminate_group_immediately` or drops the process,
and asserts that the job is gone when the call returns.
- Red: with the tracker disabled, all three fail with `tool job <pid>
outlived its provider`.
  - Green: all three pass.
- Unit tests in `process_lineage.rs` cover a re-parented job, a job that
left its group but stayed in the recorded session, a session made after
the first sighting, PID reuse, and no claim through the supervisor's
inherited session.
- `node --test scripts/__tests__/runner-release-crate-cache.test.mjs
scripts/__tests__/runner-debug-crate-cache.test.mjs` (new case: the
process-table library is cached with the crate, and any other member
still needs review).
- `cargo test -p paperclip-process-table` (includes an exited process
reported as gone), and `cargo test -p paperclip-runner-core --lib` (570
passed).
`acpx_provider_backend::tests::unconfirmed_suspension_state_becomes_recoverable_only_after_cleanup_and_suspend`
fails sometimes under full-suite load. It fails at the same rate with
the tracker disabled, so it is a flake that already existed.
- Sampler cost: one table read takes about 0.4 ms for 1,200 processes on
macOS (about 0.8% of one core for each supervised process).
- The restart-chaos replay (`--replay w-20261107T023921-s7106/t67`) did
not run. The local session was not permitted to load provider
credentials. It is not proven against a live Claude run.

## Risks

- Teardown now stops all the processes that a provider started, also
intentional background jobs. This is the intended contract: nothing
outlives the run.
- A wrapper that lives less than one sample (50 ms) and orphans a job
before the sampler sees it is not recorded. Claude Code's wrapper
sources a shell snapshot and lives for seconds. On Linux, a
child-subreaper would close this gap. That is not in this change.
- When `runnerd` itself is killed with SIGKILL, this code does not run.
The provider guardian still kills its own group, but escaped tool
sessions are not stopped in that case.
- Teardown can return a new error when it cannot prove the lineage
stopped. Callers already handle `terminate_group` errors as provider
shutdown failures.
- The server does not yet gate the next run on the tool lineage of the
prior run directly. It relies on the runner finishing its teardown
before the runner exits.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 08:24:41 -06:00
Nicky LeachandClaude Opus 5.5 a2aa69bc62 perf(ci): split the Runner package Vitest lane into three duration-balanced lanes (#19646)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Contributors wait for PR CI before merge, so CI wall-clock time sets
the pace of work
> - The Runner package Vitest lane is the PR CI critical path. It
finishes a median 15.3 minutes after the push
> - That lane runs all 311 package test files in one job with 2 workers:
1,381 seconds of serial work and 727 seconds of wall time
> - Ten files account for 61% of the time. Vitest's `--shard` splits by
file count, not by duration, so it cannot balance this work
> - This pull request splits the package files into three lanes balanced
by duration, and runs the native server group in its own lane
> - The benefit is about 232 seconds of tests per package lane instead
of 727 seconds

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The time that PR CI needs to run the Runner package Vitest tests.

**Subsystem affected**

CI: the `Verify Paperclip Runner` Vitest lanes in `pr-trusted.yml`, and
`packages/paperclip-runner/scripts/run-pr-vitest-lane.mjs`.

**Current behavior**

The `vitest package` lane takes a median of 13.6 minutes. It is the last
job to finish in most PR runs. The lane keeps a 2-worker limit so that
Runner and sidecar processes can start within their deadlines. Average
CPU use is 29%.

**Proposed behavior**

Three package lanes each run about one third of the files, balanced by
measured duration. Each lane keeps the 2-worker limit. The native server
group runs in its own lane, as before.

**Reason and benefit**

The Runner lane stops being the critical path. More workers per lane
would add CPU contention and timeouts, so this change adds lanes
instead. The cost is about 3 runner-minutes per run for the extra setup.

**Breaking changes**

None. The `master` ruleset requires only `ci / verify` and `ci / e2e`,
so the new job names do not change required checks.

## What Changed

- Add `packages/paperclip-runner/scripts/vitest-package-lanes.mjs`.
- It gets the file inventory from `vitest list --filesOnly --json`, so
it follows `vitest.config.ts`.
- It partitions the files with the `partitionGeneralServerSuites`
function from `scripts/general-server-shard.mjs`.
- Add `packages/paperclip-runner/scripts/vitest-lane-durations.json`.
- It uses the same schema as
`scripts/general-server-shard-durations.json`.
- It is seeded from CI run 42290920882. It measures 310 of the 311
files.
- `run-pr-vitest-lane.mjs` accepts two new arguments:
  - `--package-lane=K/N` runs one partition.
  - `--native-server` runs only the native server group.
- The `--shard=1/2` and `--shard=2/2` arguments keep their exact
behavior for the matrix that master still sends.
- `.github/ci-task-matrix.json` and the fallback matrix in
`pr-trusted.yml` have three `vitest package (K/3)` lanes and the `vitest
native server` lane. The labels keep the `vitest` prefix because
`scripts/ci-profile/trace.mjs` matches `Verify Paperclip Runner
(vitest`.
- Tests:
- the lanes cover the inventory exactly once, checked with real `vitest
list`;
  - the old `--shard` commands plan the same commands as before;
  - the manifest schema is correct;
  - bad lane arguments are refused.
- Update the Runner Vitest lane paragraph in `doc/DEVELOPING.md`.

## Verification

- `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`:
44/44 pass.
- `node --test scripts/__tests__/runner-release-crate-cache.test.mjs
.github/scripts/tests/ci-plan.test.mjs`, together with the file above:
71/71 pass.
- Related workflow, cache, routing and profile tests: 446/446, 153/153
and 14/14 pass.
- Local end-to-end run of `GITHUB_WORKFLOW=PR node
scripts/run-pr-vitest-lane.mjs --package-lane=3/3` on macOS:
- the prep steps ran, and the lane ran 104 of 311 files in 123 seconds;
- 2 tests failed: a macOS `kill EPERM` and a 5-second timeout under full
local load without the CI worker limit;
  - both files pass when run alone (35/35 tests).
- This PR's own CI runs the matrix from master, so it uses the old
`--shard` commands. The new lanes run in CI only after merge. Reviewers
can compare the job durations of the first PR runs after the merge.

| Lane | Files | Serial estimate | Wall estimate with 2 workers |
| --- | --- | --- | --- |
| 1/3 | 103 | 464.9 s | ~232 s |
| 2/3 | 104 | 465.0 s | ~232 s |
| 3/3 | 104 | 465.0 s | ~232 s |

## Risks

- The weights come from one CI log and do not include collection or
setup time. The balance can be uneven until a refresh measures real lane
runs.
- Each lane passes its file paths to Vitest as filters, and Vitest
matches a filter as a substring of the path. A path that is a substring
of another path would run a file in two lanes. No current path does
this. If one does, the script prints a warning, and no file is dropped.
- Each package lane runs `ensure:eval-build-deps` and `build:rust`
again, which adds about 3 runner-minutes per run.
- A comment in `scripts/run-vitest-stable.mjs` (about line 110) still
says that the last Runner shard runs the native group. A follow-up will
correct it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution. A Claude sub-agent wrote the change, and the parent
session reviewed it and ran the local lane check.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 15:12:05 -08:00
DottaandPaperclip 1c9ca5829f perf(ci): overlap Runner workloads and cache debug workspace builds (#18794)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Required CI checks protect its server and Runner contracts.
> - Recent full runs waited on general server files and the final Runner
lane.
> - That Runner lane ran package tests before native server tests.
> - PR jobs also rebuilt unchanged debug workspace binaries.
> - This pull request overlaps those workloads and reuses exact trusted
build inputs.
> - The benefit is a shorter critical path with the same test coverage
and AWS authorization.

## Linked Issues or Issue Description

Refs #18544, #18773, #18774, #18798, #18799.

**What existing behavior does this improve?**

The wall time of required PR tests and the precision of affected-test
selection.

**Current behavior**

General server and native files use two workers. The second Runner
Vitest lane runs its package shard and then the native server group.
Each Runner job compiles debug workspace units again. A Vite raw import
also retains its consumer for unrelated source edits.

**Proposed behavior**

Use three isolated workers for general server files and four for native
files. Reuse the two Runner slots for a complete package suite and a
complete native server suite. Restore all debug binaries only on an
exact input key. Record raw imports as explicit file inputs.

## What Changed

- Separate the Runner package and native server workloads without adding
CI jobs. Preserve the old command arguments for master-pinned workflows,
local commands, and other shard counts.
- Increase file workers while retaining fork isolation, serial cases,
and the shared-database guard.
- Add an exact debug workspace cache. Derive binary targets from Cargo.
Include source, fixtures, generated inputs, manifests, compiler
identity, flags, Cargo configuration, and checkout path in its key.
Discard incomplete restores. Only verified master pushes can save it.
- Keep older PR checkouts working when they lack the new cache helper.
- Track Vite `?raw` imports as repository file inputs. Keep unknown
loaders, untracked files, and opaque effects conservative.
- Add cache and coverage guards and document the behavior. AWS runner
authorization is unchanged.
- Keep paired test benchmarking available when install hooks correctly
disable materialized dependency reuse. The old benchmark asserted a
cache hit in that safe fallback.
- Wait for active dispatch work to finish before asserting idle reads.
Analyze the two 16,400-row database fixtures before their queries so
they have current planner statistics. Suppress only PostgreSQL notices
inside the cleanup transaction to avoid logging every cascaded table;
warnings/errors remain visible. Row counts, real queries, assertions,
database reset, and timeouts remain unchanged.

## Verification

- 196 affected CI tests passed, including routing, coverage, selection
failure fallbacks, shared-database serialization, and existing/new cache
guards. Final worker/cache adjustments also passed their focused checks.
- `pnpm build` and `pnpm -r typecheck` passed locally. Full test
coverage is being verified by the Linux CI profile; `pnpm test:run` was
not repeated as a full serial local run. The paired CI benchmark passed.
Both paired benchmarks passed, and full CI passed again after
integrating newer master commits. The external Paperclip Review app is
the only pending status.
- Actual debug artifacts remained fresh after simulating checkout
timestamps: Cargo completed in 0.82 seconds; the content key was
unchanged. This is local reuse evidence, not a measured GitHub cache
transfer.
- Same real UI inventory for a retention-test-only change: 861 files;
retained selection fell from 413 to 401. All 12 omitted files are
raw-input consumers; edits to their inputs remain covered by regression
tests.
- The first local worker comparison is invalid. Development dependency
links disappeared during its second run. The two-worker baseline also
had two macOS path assertions in company-skills. Neither result is
counted as a successful speedup. Dependencies were restored. A local
repeat was interrupted before producing results. The completed Linux
comparison below supplies the valid paired evidence.
- [Paired Linux
benchmark](https://github.com/paperclipai/paperclip/actions/runs/41671244016):
baseline and candidate ran the same 113 general-server files and 1,777
cases on one host. Wall time fell from 381.29s to 346.89s (9.0%). Both
exited successfully and file/case inventories matched exactly. The
fixture command took 31.77s versus 28.65s. This is one sequential pair,
excluding setup, queueing and the full CI critical path.
- [Repeat paired Linux
benchmark](https://github.com/paperclipai/paperclip/actions/runs/41675496862):
the same 113 files and 1,974 cases took 485.06s in the baseline versus
368.54s in the candidate (24.0%). File and case inventories again
matched exactly. The baseline includes newer master changes than the
first pair, so these are two independent comparisons, not repeated
timings of an identical tree.
- Initial full AWS profile: package suite passed 4,891 tests (289 files,
one skipped); native suite passed 172 tests (9 files, 12 pre-existing
platform/feature skips). Native execution took 239.06 seconds; the older
sample ran 165 native tests in 391.29 seconds. The package lane now
takes 11m01 including setup and debug compilation. This is an indicative
cross-run comparison, not a paired full-CI speedup claim.
- The initial run failed two UI fixtures, one serialized inbox
settlement assertion, and one dispatch idle assertion. The two UI
failures reproduced locally. The fixes preserve authorization,
admission, and zero-idle-read assertions. All four affected files passed
locally: 48/48 tests. Token gates passed. All four also passed in the
next AWS profile. That profile passed every other workload but timed out
the 16,400-row keyed attention fixture at its existing 180s limit; this
is accounted for by the explicit fixture-statistics change. Local
focused comparisons pass both large cases before and after; their
roughly 33s test execution is unchanged locally, so this is not claimed
as an additional speed gain. A full local file run passed both large
cases (7.57s / 18.05s) but hit the unchanged 30s teardown deadline on
its first test while emitting thousands of cascade notices. Cleanup now
suppresses those notices within its transaction; the complete file then
passed 34/34 in 107.72s. The latest-head AWS run passed the previously
failing shard: the keyed bulk case took 40.25s (previously exceeded
180s), the keyless case took 8.02s, and the complete attention file took
89.61s. The 16,400-row fixtures and original timeouts remain intact.
- [Full CI profile before master
integration](https://github.com/paperclipai/paperclip/actions/runs/41675497699)
passed on commit `5858956d4ac2c99c911ad2447b63e1f8fc84e31e`, including
all tests, build, typecheck and final verification. Greptile reviewed
that exact commit at 5/5; the only actionable review thread is resolved.
The separate Paperclip Review app is still queued.
- [Final merged-head
CI](https://github.com/paperclipai/paperclip/actions/runs/41678956312)
passed on `499aa206a6e03bf8f3249692ea731367a000591d`: 48 successful jobs
and one intentionally skipped file-selection job. Backup checkpoint
qualification and legacy security status passed. Greptile reviewed this
exact commit at 5/5. The PR is conflict-free; only the separate
Paperclip Review app remains queued.
- Integrated current master after #18798 and #18799 merged the inbox and
UI fixture fixes separately. Preserved master's inbox settlement fix and
verified its entire file (6/6 tests). Those three files are no longer
part of this PR's diff. Removed the temporary `profile-ci` label after
both successful benchmark pairs; the normal planner still schedules
every job because the CI configuration changes are global inputs.
- Second AWS profile: native tests took 229.97s with the same 172 passes
/ 12 skips; package tests took 427.44s with 4,891 passes / 10 skips. The
complete package job took 10m44, including setup and 79s debug
compilation.
- The PR uses the trusted workflow from master. Candidate worker/layout
code runs now; workflow-level debug restores begin after merge and the
first successful master cache writer.

## Risks

- Extra workers can expose load-sensitive tests. The final full CI
profile passed, but one run is not a long-term flake-rate measurement.
- The cache is an optimization only. Misses rebuild, and test commands
still execute. A new Cargo workspace member requires a cache-contract
review.
- Compiler alternatives remain experiments. Required TypeScript checks
are unchanged.

Typechecker experiment: three sequential checks per tool on identical
built projects, with TypeScript 7.0.2, Bun 1.4.4, and tsrs
0.20.1-ts7.1.0-beta. Median seconds (TypeScript / Bun / tsrs): shared
0.429 / 0.083 / 0.157; UI 2.554 / 0.390 / 0.740; server 9.949 / 1.194 /
1.782. TypeScript and Bun passed all three projects. All tools caught
the same four deliberate assignment, property, nullability, and
generic-constraint errors. tsrs reported one additional server TS2345
diagnostic in the run-dispatch adapter, so it did not pass the project
comparison. These local compiler times exclude generation and build
steps; they are not full CI savings.

## Model Used

OpenAI GPT-6 Codex. Tool use and code execution assisted this change.
The exact deployment identifier and context window are not exposed in
this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 11:17:53 -06:00
DottaandPaperclip f5dcc505c0 perf(ci): select affected suites and cache task results (#18214)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Pull request CI checks the server, UI, packages, Runner, and release
paths.
> - An ordinary PR starts 48 jobs even when its diff affects only a
small part of the repository.
> - Repeated pushes also repeat successful tests with unchanged inputs.
> - This PR selects affected suites before runner allocation and caches
deterministic task results with Turbo.
> - Required checks validate the plan. Unknown inputs and failed
comparisons keep full verification.
> - The benefit is fewer runner jobs and less repeated work, with full
post-merge verification retained.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The GitHub Actions PR workflow repeats unrelated suites for each push.

**Subsystem affected**

Cross-cutting CI routing, test execution, and required-check
aggregation.

**Current behavior**

An ordinary UI component change starts the complete 48-job PR workflow.
Existing dependency and compiler caches do not reuse successful test
results.

**Proposed behavior**

A trusted planner reads the tested merge diff and task inputs. A UI
component change such as `ui/src/components/Sidebar.tsx` selects 18
jobs, including build and browser coverage. Successful general,
serialized, and standalone Rust tasks can reuse unchanged results across
pushes in the same PR.

**Reason and benefit**

Reduce unrelated jobs and repeated test execution. The 48-to-18
comparison measures planned job count. Remote wall-clock savings have
not yet been measured.

**Breaking changes**

No application API changes. Required `verify` and `e2e` names remain
stable. `verify` also checks serialized suites and the canary dry run.

## What Changed

- Added explicit task inputs and a pinned Turbo version. An execution
adapter calls existing package scripts without adding script aliases.
- Added trusted diff planning and dynamic matrices with complete
fallback matrices.
- Added PR-scoped task result caches. External-state and
revision-stamped checks remain uncached.
- Added and created the `full-ci` override label, plus repository modes
`enabled`, `shadow`, and `full`. Forced and shadow runs bypass result
caches.
- Added candidate-workflow qualification, planner and aggregate tests,
and a real Turbo cache fixture.
- Kept the repository-wide CLI guidance guard in the uncached typecheck
lane so UI-only diffs retain this security check.
- Documented rollout controls, input maintenance, and local inspection
commands.

## Verification

- Passed 537 focused CI policy, routing, matrix, cache, and shard tests,
plus 146 existing release workflow tests.
- Passed actionlint for all four changed or added workflows and `git
diff --check`.
- Passed `pnpm -r typecheck` and `pnpm build` on the implementation
base.
- Checked real repository task plans for existing general-test and
native Rust commands. A real server partition passed, then restored from
cache: Turbo task time was 20.022 seconds for execution and 151 ms for
the cache hit, excluding wrapper/setup overhead.
- The real Turbo fixture exercises the execution adapter and proves
source, argument, toolchain-scope, and environment invalidation, result
restoration, and retry of failed tasks.
- Stopped the serial local `pnpm test:run` after over 30 minutes in
favor of the complete parallel GitHub suite. A local parallel
database-lane attempt had PostgreSQL startup errors and timeouts. The
direct migration fixture passed, then all 273 database tests passed with
one local worker. This recovery does not establish the exact cause of
the earlier startup failures. The full GitHub PR suite and candidate
qualification passed on commit
`0bf07ae071c9fe7903edc604d44b5a175727fed0`. Greptile is 5/5 with no
unresolved review threads.
- All 40 CLI guidance safety tests passed with the exact uncached
workflow command.
- Verified that master rulesets require only `ci / verify` and `ci /
e2e`; both names remain unchanged.
- Trusted workflows load from master. The added GitHub-hosted
qualification workflow exercises this candidate planner before merge.

## Risks

- Changes to global inputs, including task definitions and execution
helpers, force execution and disable cache restore/save for the full PR
diff. Missing a cross-package filesystem read could omit a suite. Inputs
include known server reads of UI files and shared tests of UI assets.
Unknown paths, malformed or truncated diffs, and definition download
failures select all suites.
- Task result reuse requires complete inputs. Keys include Git source
files, scripts, resolved dependencies, environment, toolchain, checkout
path, and unclassified source files. Generated build artifacts are
excluded from input hashes. Registry smoke, browser, and
revision-stamped tasks remain uncached.
- Maintainers must update task inputs when suites start reading new
paths. `shadow` reports selection while running all work. `full` and
`full-ci` disable selection and result reuse.
- The implementation becomes active after merge. Full post-merge
verification remains unchanged.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection, tool
use, and code execution. The session does not expose a verifiable exact
runtime model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-29 16:56:28 -05:00
DottaandPaperclip 54cc522e86 feat(runner): add gated native Antigravity ACP execution (#18188)
## Thinking Path

> - Paperclip manages agents and their work.
> - Native Runner must bind each provider turn to the selected task,
account, model, and instructions.
> - Antigravity needs native ACP admission and physical process
ownership across continuation.
> - A saved conversation must not reuse an old process owner or silently
switch its effective model.
> - This pull request adds the native candidate behind exact operator
admission.
> - The default release gate remains closed while qualification is
pending.

## Linked Issues or Issue Description

Builds on merged #18187. Product account setup and the candidate
producer are in dependent #18189. The repeatable browser journey is in
#18190. Refs: #12503, #14938; those proposals use legacy CLI adapters.

**Agent or provider**

Google Antigravity ACP 1.4.3.

**Why this adapter is useful**

Users need to set up an Antigravity agent and run real work through
their managed Google subscription.

**How the agent is invoked**

Native Paperclip Runner uses the pinned Antigravity ACP runtime. Exact
operator admission is required while qualification remains pending.

## What Changed

- Add native Antigravity ACP model and instruction admission,
selected-account homes, and fenced refresh ownership.
- Carry exact candidate authority from the control plane through Runner,
Rust, the sidecar, and each current run. Clamp daemon lifetime to the
absolute permission expiry, including replacement owners.
- Preserve a rotated OAuth refresh token before later identity
verification can fail, including failed or cancelled startup admission.
Make private lease-close retries safe.
- Bound native probe retirement and retry the exact original owner; do
not delete credential homes before physical proof.
- Preserve Gemini API-key recovery when shared Google subscription
capability is added.
- Resume the same tagged native conversation with fresh per-turn process
ownership and physical native retirement.
- Include native manifests in isolated Rust image stages, release cache
keys, and dev rebuild detection. The workflow changes only extend
existing cache input lists.
- Keep current master provider behavior and append Grok 13, Pi 35, and
Copilot 84 input identities. Preserve prior identities, including saved
Copilot 82 and 83 runs.

## Verification

- Runner TypeScript main checks, profile generators, Rust formatting,
and offline workspace checks pass. The native backend integration target
passes 28 tests.
- The complete transport and cleanup group passes 387 tests without
unhandled rejections. These checks use local scripted children and
actual process and port ownership.
- Saved-profile decoding, native-execution parsing, installation, and
selected-credential controls pass 208 focused tests. All 109
pre-existing profile identity files remain byte-for-byte unchanged.
Copilot, Grok, and shared TypeScript/Rust generation checks pass.
- The final recovery fixture passes all seven cases. After the master
rebase, the combined recovery and owned-Node-runtime group passes all 10
cases. Each expiry case sets one 45-second absolute grant before
startup. It includes the deliberate six-second cold delay, four existing
eight-second command budgets, and active-turn headroom. The test proves
an active turn before expiry and then checks the daemon, owned process
group, child processes, and all three native port fences. The 60-second
test deadline does not renew provider authority.
- The consolidated model-discovery and pending-release group passes 22
tests. Google credential compatibility and auth recovery pass 25 tests,
including existing Gemini API-key recovery. The full product settings
test group remains intact in #18189.
- Earlier failures remain in their original CI runs and local records.
They exposed the Copilot 82 decoder gap, an expired five-second fixture
bootstrap ticket, and a fixture grant consumed before activation on
loaded CI. Each repair has a separate result. Production ticket limits
and admission behavior are unchanged by the final test commits.
- An unrelated decision-expiry check failed in [the earlier CI
run](https://github.com/paperclipai/paperclip/actions/runs/40941986698/job/123688668997).
Its owning code is unchanged from the base. The unchanged suite passes
20 tests on isolated PostgreSQL. A controlled database rollback
reproduces the earlier wake: clean startup schedules the 120-second
decision expiry, while rollback reconciliation leaves a 30-second
continuation-coalescing wake. The decision stays open, with no wake
delivery or error. The fresh final-head server shard passes; the failed
result is preserved.
- Final native commit `ec6bc7e4d77f5ae0cc4946dbba6c662e95a87f5b` passed
`ci / verify` and `ci / e2e`. Fresh Greptile and Paperclip reviews both
score 5/5, with no open findings. Antigravity retains the recorded
native `.5` closure and exact low-model profile. This PR does not claim
global or cross-platform qualification.

## Risks

- The new transport carries private managed credentials. Exact
ownership, account generation, model, instructions, and native
retirement are fail-closed.
- Only local macOS ARM64 has a verified complete Antigravity runtime.
Default Antigravity task execution remains disabled without current
trusted operator admission. The Google server sign-in lane and
production candidate producer ship in #18189; this native slice does not
expose the subscription catalog choice.
- Shared launch inputs change provider identities. New identities are
append-only; existing recorded identities remain available.

## Model Used

OpenAI Codex (GPT-6). The host did not expose the exact model revision
or context limit. The model used tools for code edits, tests, and
review.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-29 16:36:12 -05:00
DottaandPaperclip d855e550ad Finish Copilot native runtime on current master (#15701)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner controls provider processes, permissions and task
completion.
> - Copilot needs its pinned distribution, native message identities and
accepted semantic completion receipts.
> - The release must preserve saved-token access and task lifecycle
guarantees.
> - This pull request will keep the current completion and recovery
fixes while adding the Copilot runtime.
> - The benefit is a reviewable part of the Copilot production release.

## Linked Issues or Issue Description

Refs #14434. Supersedes the runtime work in #15560. This is the first PR
in the stack; #15703, #15705 and #15706 follow. Related alternative
adapter work: #15881. This change uses the shared Runner path.

## What Changed

- Pin Copilot CLI 1.0.88 and bind the integrated implementation as
profile45.
- Keep native instruction, permission, receipt, process ownership and
cancellation boundaries.
- Preserve current master completion continuation and remote
verification behavior. Other provider release identities stay unchanged.

## Verification

Final reviewed source
[`4f3f1af807d2ed21adabb04c2f27f4df9c4bc8d5`](https://github.com/paperclipai/paperclip/commit/4f3f1af807d2ed21adabb04c2f27f4df9c4bc8d5)
includes master `cde944ed75648adfb569e86133adc8b118b55dd6`. The combined
four-part PR tree matches it except for the CI-owned lockfile. Every
prior public PR head remains an ancestor. Released master migrations
remain unchanged; Copilot adds migration361. No unrelated provider is
enabled.

Copilot CLI1.0.88 and profile47 retain digest
`sha256:55044aa235e88b3fe36ec8a556ee1e3382590870aab907e4462ef7d64af49cea`.
The pending qualification marker is removed. The [qualification
record](https://github.com/paperclipai/paperclip/blob/4f3f1af807d2ed21adabb04c2f27f4df9c4bc8d5/doc/plans/2026-10-20-copilot-qualification-v47.json)
reports **20/20 Product cells,154 assertions;7/7 protocol cases,46
assertions** against frozen installed application
`c2616370800f5f6df45246b0890ba35a7cd2faab`. Both environments use the
exact authenticated model `gpt-5.6-luna`. Every passing Product case
confirms cleanup. Automatic behavior retries are disabled. Original
failed attempts remain recorded.

The ordinary installed remote journey passed saved-token selection,
authenticated model discovery, normal UI save/reopen, metadata-only
Test, task completion, accepted receipt and an exact19-byte browser
download. An open model catalog caused no new discovery requests
during117seconds. Actual installed runtime smokes passed on macOS ARM64,
macOS x64 under Rosetta and native Linux x64 on Daytona. Raw token
browser entry was not exercised; the encrypted connection was created
through the normal API.

The original Daytona file attempt timed out during delayed semantic
dispatch. Its precise stalled layer remains unresolved. Independent
ordinary delivery and the other19 cells passed before one instrumented
recheck passed8/8 with unchanged source,prompt,assertions and deadlines.
No causal latency repair is claimed.

After current-master reconciliation,129 connection tests,59 runner
tests,72 shared eval tests and88 recovery/workspace tests pass.
Migration generation/safety, unchanged profile identity and UI token
gates pass. Prior required CI was green. Fresh exact-head required CI
and E2E are green for all four updated heads. Each has a fresh
Greptile5/5 review and zero unresolved review threads. The [final
normal-package cloud
build](https://github.com/paperclipai/paperclip/actions/runs/39414239704)
is running. Recursive typecheck,full CI test shards,build and all8
browser shards pass. These checks cover the final source; no additional
local full-suite pass is claimed.

The [frozen image
build](https://github.com/paperclipai/paperclip/actions/runs/39334401638)
and [original normal package
build](https://github.com/paperclipai/paperclip/actions/runs/39334407471)
passed. The exact image is published and independently retrieved
anonymously. No separate full-image scan is claimed. All18 original
normal archives were scanned with zero selected credentials. Docker
builds run in the cloud. Provider USD remains unknown and
GitHub-attributed.

**Qualification is complete on the recorded frozen build. Normal
protected merge, automatic master canary publication and final
shipped-package local/Daytona canaries remain. No production deployment
is claimed.**

## Risks

Shared Runner behavior can affect active tasks. Exact profile admission
rejects stale runtime identities. The final campaign and installed
canaries remain release gates.

## Model Used

OpenAI Codex, based on GPT-6. The exact serving model ID and context
window are not exposed in this task. Used for code changes, review and
tool-assisted verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-20 06:31:47 -05:00
8227485372 perf(ci): cache the Runner's release crate artifacts keyed by their inputs (#15932)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Its trusted PR workflow builds the Runner's Rust binary in five
jobs: Typecheck + Release Registry, Build, Canary Dry Run, and the
`rust` and `vitest 2/2` lanes of Verify Paperclip Runner. The `Build`
job is the slowest check in almost every green run (461s, 489s and 515s
in runs 38180574768, 38182717052 and 38183149920 on 2026-10-11).
> - Each of those jobs restores master's `release-runner-v2` dependency
cache. That cache holds only third-party crates. `rust-cache` removes
workspace crates when it saves, and a fresh checkout invalidates cargo's
mtime fingerprints for workspace crates in any case. So cargo compiles
`paperclip-runner-core` in release mode from scratch on every run. The
job logs show one `Compiling` line and 28s to 49s for it.
> - 18 of the 239 commits on `master` in the seven days before this
change touched the crate's inputs. The other 221 compiled the same
sources to the same binary again.
> - This pull request adds a second cache that holds only that crate's
release artifacts (about 70 MB). The key is made from the compiler, the
checkout root and a hash of every input the build reads. The master Rust
lane writes it. A pull request whose merge tree has byte-identical
inputs restores it and marks the artifacts newer than the checkout, so
cargo keeps them. Every other pull request misses and compiles as
before.
> - The benefit is about 28s to 49s less on `Build`, which is the run's
critical path, and the same on Typecheck, Canary Dry Run and two Runner
lanes, for about nine of ten pull requests.

## Linked Issues or Issue Description

No GitHub issue exists for this. The description below follows the
enhancement template.

**What existing behavior does this improve?**

The time the trusted PR workflow spends to compile the Runner's own
crate on every run.

**Subsystem affected**

CI / build

**Current behavior**

Every PR job that builds the Runner binary compiles
`paperclip-runner-core` in release mode from scratch. The compile takes
28s to 49s on the `runs-on` fleet (one `Compiling paperclip-runner-core`
line, then `Finished release profile` in runs 38169436532 to 38183149920
on 2026-10-11). The shared Rust cache cannot help: it holds only
third-party crates, and a fresh checkout gives every source a new mtime,
which makes cargo discard the crate's fingerprints.

**Proposed behavior**

The master Rust lane in `release-verify.yml` saves the crate's release
artifacts under a key made from the compiler, the checkout root and a
hash of every input the release build reads. The PR jobs that build the
binary restore that entry read-only. On a hit, a script sets the mtime
of every restored artifact to now, so cargo sees every source as older
than the build and keeps the artifacts. `pnpm build` then prints
`Finished release profile` in under a second for that crate. On a miss,
cargo compiles the crate exactly as before.

**Reason and benefit**

About 28s to 49s off `Build`, the slowest check in almost every green
run, and the same off the Typecheck, Canary Dry Run and two Verify
Paperclip Runner lanes. The saving applies to every pull request whose
merge tree does not change the crate's inputs, which was 221 of 239
`master` commits in the week before this change.

**Breaking changes**

None. The checks are unchanged. A miss or an eviction costs the compile
time it costs today and nothing else.

**Additional context**

This is a workflow change. `pr.yml` calls `pr-trusted.yml@master`, so
this pull request's own run uses the current reader steps, and the
writer runs only on a master push. The saving becomes visible on the
first PR runs after the merge and after the next master push has written
the entry.

## What Changed

- `scripts/runner-release-crate-cache.mjs` (new): the one place that
defines the cached path list (`CACHED_PATHS`), the hashed inputs
(`HASHED_INPUTS`) and the files a restore must contain (`REQUIRED`). The
`scope` subcommand prints the platform, the `rustc --version --verbose`
release, commit and host, and a hash of the checkout root. The
`mark-current` subcommand sets the mtime of every restored artifact to
now and prints a `paperclip.runner-release-crate.v1` JSON line with the
file count and bytes. If the restored set lacks any of the seven files
cargo's freshness check reads or checks for (the fingerprint hash and
dep-info of each unit, the lib's rlib and rmeta, the binary under
`deps`), it removes the restored files, prints `action: discarded`
naming the missing file, and exits 0, so cargo compiles as on a miss.
- `.github/workflows/pr-trusted.yml`: after the `release-runner-v2`
restore in `typecheck_release_registry`, `build` and `canary_dry_run`,
and in the `rust` and `vitest 2/2` lanes of `verify_paperclip_runner`
(new matrix key `release_crate: true`), three steps: compute the key
scope, `actions/cache/restore` the crate under
`runner-release-crate-v1-<scope>-<hashFiles(...)>` with no
`restore-keys`, and run `mark-current` when `cache-hit == 'true'`. The
`hashFiles` list covers the toolchain pin, the manifests and lockfile,
the crate sources, the protocol tree and the three fixture trees the
test modules include (two under the Runner package and
`tests/runner-e2e/fixtures` at the repository root).
- `.github/workflows/release-verify.yml`: in the Rust lane of an
own-repository master push, after the debug warm-up: compute the key
scope, look the key up (`lookup-only: true`), and on a miss run `cargo
clean --release -p paperclip-runner-core` and `pnpm build:binary` (the
exact command the readers run, so the entry carries no test-profile
variants), then `actions/cache/save` the same path list under the same
key.
- `scripts/__tests__/runner-release-crate-cache.test.mjs` (new, 10
tests, in `test:release-registry`): the path list and hashed inputs are
the expected ones and exist; every `include_str!` and `include_bytes!`
in the crate sources resolves under a hashed input, whether its argument
is a literal or a `concat!` of literals, `env!("CARGO_MANIFEST_DIR")`
and macro metavariables, and any other form fails the test by name (this
guard found the `test/fixtures` tree during development and the
`tests/runner-e2e/fixtures` tree in review); the parser itself is tested
on a synthetic crate for each supported form and six rejected ones;
every PR reader restores after the dependency cache and marks before its
build command with the script's exact path list and key and no
`restore-keys`; the Verify lanes are gated by `release_crate`; the
writer runs after the checks with the lookup, the clean and build, and
the save under the readers' exact path list and key; the scope changes
with the compiler and the checkout root and survives a missing `rustc`;
`mark-current` makes every artifact newer than the checkout, leaves a
dependency's artifacts alone, discards a restore missing any one of the
seven required files (one case each, checking the named reason), and
still marks a restore missing any one of the seven files cargo does not
need.

## Review fixes

Greptile's two findings on the first commit were both valid and are
fixed in the second commit:

- **Incomplete caches looked usable.** `REQUIRED` checked four files,
but cargo's freshness check needs seven. Measured with cargo 1.97.1 on
the real crate (restore, touch the artifacts, delete one file, build):
without the lib's fingerprint hash, dep-info, rlib or rmeta it
recompiles the crate (24s to 27s); without the bin's fingerprint hash,
dep-info or the binary under `deps` it relinks (1.5s); without either
`.json` fingerprint record, either rustc `.d` file, the uplifted binary
or its `.d` it reports the crate fresh in 0.15s and recreates the
uplifted binary from `deps`, and the staging step then copies it.
`REQUIRED` now lists exactly the seven files, the glob for the binary
under `deps` excludes the `.d` beside it, and the test removes each of
the seven in turn (discarded, naming it) and each of the other seven in
turn (still marked).
- **Composed include paths escaped the guard.** The regex matched only
`include_str!("literal")` and skipped `include_str!(concat!(...))`,
which the crate uses four times (30 of 34 includes were checked). The
guard now parses every include argument and resolves the `concat!`
forms; one of them, in `provider_events.rs`, reads
`tests/runner-e2e/fixtures/native-completion/terminal-tool-carrier.json`
at the repository root, outside every hashed input. It is test-only, so
today's release binary does not read it, but the key is meant to cover
every include whatever its build, so `tests/runner-e2e/fixtures/**` (9
files, 80 KB) joins `HASHED_INPUTS` and the six key embeddings in the
two workflows. With the first commit's input list, the new guard fails
on exactly that include.
- `doc/RELEASE-AUTOMATION-SETUP.md`: a "Runner release crate cache"
section under the Runner verification dependency cache, with the key,
the writer, the readers and the trust boundary.
- `package.json`: the new test file in `test:release-registry`.

## Verification

- `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`:
10 tests pass, three consecutive runs.
- The second commit, with the real toolchain on the real crate (`rustc
1.97.1`, cold build 2m20s, the crate's artifacts 74 MB): every source
touched to a fresh-checkout mtime and the archived artifacts restored,
cargo compiles (`Compiling paperclip-runner-core`, 24.8s and 25.1s, the
two controls); restored and marked by `mark-current`, `Finished release
profile in 0.16s` with no `Compiling`, and the same on a second build.
Then, for each of the fourteen files cargo wrote for the two units, one
at a time: restored, marked and that file deleted, cargo recompiles
without any of the seven `REQUIRED` files and reports fresh without any
of the other seven; restored with that file deleted and run through
`mark-current`, the seven required ones print `action: discarded` naming
the file and the others print `marked`. A restore holding only the seven
required files builds fresh in 0.15s, cargo recreates the uplifted
binary, and `stage-runner-binary.mjs` copies it into `dist/bin`.
- `node --test '.github/scripts/tests/*.test.mjs'`: 378 of 381 pass. The
3 failures are in `docker-canary-promotion.test.mjs` and fail the same
way on an unmodified `master` checkout in this sandbox (the stand-in
docker writes no `calls.jsonl`).
- `node --test scripts/__tests__/release-verify-workflow.test.mjs
scripts/__tests__/e2e-shard.test.mjs
scripts/__tests__/run-vitest-stable-shard.test.mjs`: 99 of 101 pass; the
2 failures spawn `pnpm`, which is not on this sandbox's PATH, and fail
the same way on `master`.
- Both workflows parse as YAML. No job has a duplicate step name or step
id.
- The mechanism, with the real toolchain (`rustc 1.97.1`, the package's
pin) on a copy of the crate: a cold `cargo build --release -p
paperclip-runner-core --bin paperclip-runnerd` (2m23s here). Then the
artifact set was archived with `tar`, removed, every source re-copied so
each had a new mtime, and the set restored:
- restored, not marked: `Compiling paperclip-runner-core`, 24.6s (the
control; this is what CI does today);
- restored and marked by `mark-current`: `Finished release profile in
0.15s`, no `Compiling`; a second build is the same;
  - one Rust source edited after a marked restore: `Compiling`, 24.0s;
- one protocol schema edited after a marked restore: `Compiling`, 23.9s
(the protocol files enter cargo's dep-info as absolute paths and are
hashed into the key);
- the rlib removed from a restored set: `mark-current` prints `action:
discarded, reason: incomplete: the lib rlib is missing` and removes the
rest, and cargo compiles.
- Not run here: the workflow steps themselves. `pr.yml` pins
`pr-trusted.yml@master`, so this pull request's own `Build` job runs the
current reader steps, and the writer runs only on a master push. The
first PR runs after the merge and after the next master push show the
`Restore the Runner release crate` step's result and the
`paperclip.runner-release-crate.v1` line in their `Build` logs.

## Risks

- Correctness: a reader can only restore artifacts that master built
from inputs byte-identical to its own merge tree, because every input
the build reads is in the key, and the key has no prefix match. A pull
request cannot write. The trust boundary is unchanged: master writes, a
pull request restores master's entry.
- An input the build reads that is outside the hashed patterns would let
a stale artifact pass. The release dep-info lists the crate sources and
the protocol schemas and fixtures; all are hashed. The guard test
resolves every `include_str!` and `include_bytes!` in the crate, literal
or `concat!`, and fails if one lands outside the hashed inputs or uses a
form it cannot resolve, so a new include form cannot slip past it by not
matching a regex.
- GitHub-hosted runners check out under `/home/runner/work`, the
`runs-on` fleets under `/home/runner/_work`. Cargo records the included
protocol files as absolute paths, so an entry from one root is useless
under the other. The checkout root is in the key, so the other root
misses instead of restoring something cargo then discards.
- A cache that does not keep mtimes on restore changes nothing: the
artifacts are then already newer than the checkout, and `mark-current`
sets them to now in any case.
- Cost: one new entry of about 70 MB per distinct input set, written
only by the master Rust lane on a miss, plus one `cargo clean` and
`build:binary` (about 40s to 50s) on that lane when the inputs changed.
The readers pay one restore of about 70 MB (two to three seconds at the
fleet's observed restore speed) and a sub-second `mark-current`.
- An incomplete entry (a writer that saved a partial set) is discarded
by `mark-current` and costs one compile, the same as a miss.

## Model Used

- Claude (Anthropic), model id `claude-fable-5-1` (Claude Fable 5.1),
extended thinking, with tool use (shell, file editing, GitHub CLI, a
local Rust toolchain for the experiment) in a Paperclip agent session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-12 11:25:31 -07:00