Commit Graph
348 Commits
Author SHA1 Message Date
Devin FoleyandPaperclip 21bf7ebcee feat(connectors): connect Tailscale networks to SSH environments (#20617)
## Thinking Path

> - Paperclip manages work for AI agents.
> - SSH environments run tasks on computers that users own.
> - A Paperclip server needs network access before it can reach a
private computer.
> - A reusable Tailscale connection belongs in Connectors.
> - Connected machines should become choices in the existing SSH
environment form.
> - This change connects a tailnet and resolves selected machines
through a private network client.
> - Agents retain their choice of harness and AI account.

## Linked Issues or Issue Description

**Subsystem affected**

Connectors, environment settings, SSH execution, company secret storage,
and runtime packaging.

**Problem or motivation**

Entering a Tailscale address in the SSH form does not connect the
Paperclip server to that tailnet. Requiring a separate device key for
every environment also duplicates network setup.

**Proposed solution**

Add a Tailscale Connector with **Connect Tailscale Network**. Use
brokered OAuth consent to enroll one private network client per
connection. Let SSH environments select a connected network and stable
machine identity. Preserve the enrolled node state through the company
vault so a restart does not reuse the one-time token.

**Alternatives considered**

A separate Tailscale environment type would duplicate SSH behavior and
mix network access with harness selection. This change uses the existing
`ssh` driver. Related #15648 adds OAuth client credentials and a health
check; this change uses user consent and supplies outbound task
execution. Related #11743 concerns HTTPS brokering.

**Roadmap alignment**

This extends the existing Connected Apps, environment, and Personal &
Shared AI Accounts capabilities. This PR is stacked on #21248, which
provides SSH account selection independently of Tailscale. The hosted
OAuth broker is a companion control-plane change.

## What Changed

- Preserve Cloud reauthentication in the current browser tab. Explain
canceled or failed consent in the connector and offer a retry of the
same authorized saved connection.
- Record Cloud enrollment starts initiated through the Tailscale
connector in the company activity log.
- Check install-preparation descendant liveness through the kernel
process table in the Runner support test. Keep the direct child’s strict
reaping assertion and reject any non-zombie descendant.
- Align the application image stage contract with the Tailscale client
stage after the upstream Runner-image refactor. Keep the exact stage
list and check every application stage, including `tailscale-client`,
for prohibited Runner compiler commands.
- Group Tailscale consent, discovery, daemon lifecycle, pairing, and
deletion under `server/src/services/connectors/tailscale/`, with their
service tests. Update existing route, runtime, environment, vault, and
company-deletion imports. Keep the existing contracts, authorization,
audit, and transaction owners.
- Handle early stdin closure in the Kubernetes real-shell test helper.
Stop the input stream on `EPIPE`, preserve nonzero child exits, and
reject a zero exit when input was lost. Add two real-process regression
cases. Production transfer code and confinement checks are unchanged.
- Read configured Tailscale executable paths when a client starts, after
supported `.env` loading. Keep explicit command overrides available.
- Discard pending pairing results after switching to Direct SSH or
closing the form. Retain the existing company, connection, machine, and
port checks.
- Store network credentials with an immutable server-owned purpose from
their first vault row. Exclude them from generic catalogs, bindings,
environment projection, and value reads. Use a dedicated audited server
consumer for pinned connector reads. Block unclassified legacy
serialized node state at the same value boundary. Keep SSH private-key
binding separate from connector credentials.
- Gate network queries and setup on verified local-implicit or
instance-admin access. Hide Tailscale catalog rows for other members,
show an admin requirement on direct setup navigation, and avoid device
discovery before management access is known.
- Keep failed-client cleanup in the network manager. A failed request
cannot disconnect a newer retry for the same identity; explicit
connector disconnect and enrollment retirement still stop their
networks.
- Check for readable Ed25519, ECDSA, or RSA host public keys before
pairing edits `authorized_keys`. Print a known-hosts entry for every
available key, or fail before changing authorized keys if none are
available.
- Return self-hosted Cloud enrollment to the Tailscale connector. Save
the unprefixed setup path and let the shared callback add the verified
company prefix. Keep external and unrelated destinations rejected.
- Hide pairing instructions after an SSH port change. Ignore late
key-selection results for the old port. Operators prepare again for
matching known-hosts output.
- Retire connector references before deleting a company or its vault
credential. Stop company daemons after the database commit. Keep failed
secret retirement retryable and prevent late node-state writes from
restoring deleted connections. Discard retired state without blocking
behind the vault deletion lock.
- Reload a mismatched credential before replacing a ready daemon.
Recheck the active entry and manager generation after the read. An older
request cannot stop a newer healthy socket or cross shutdown.
- Pin connector credentials to the exact vault version and SHA-256
digest recorded by trusted enrollment and node-state writes. Ignore
generic secret rotations. Advance both fields with a compare-and-set
guard. Reject unpinned or tampered credentials before network startup.
- Save an explicit Direct SSH selection and clear old Tailscale fields
after config merging. Preserve the operator's entered direct host.
- Route all Tailscale SSH runtime access through the connection service
after SSH-key loading. Pin access to the environment's saved connector
identity and recheck after device discovery. Remove standalone key-based
daemon startup and reject raw enrollment keys or credential-only network
configurations. Align the shared config type and SDK pins.
- Correct two E2E assumptions exposed during connector qualification.
Keep the two-window chat fixture stale until its own mutation. Validate
the original failed run and its permitted write-revocation timestamp in
the legacy recovery browser test.
- Add the Tailscale Connector page, catalog entry, OAuth return flow,
connected-network management, device discovery, and disconnect. Document
all six routes in OpenAPI and the API reference, including their
instance-admin requirement.
- Add company-scoped connection storage in `0480_lazy_lethal_legion`,
trusted credential version/digest fields in `0481_round_pet_avengers`,
and server-owned secret purpose/backfill in
`0482_aberrant_rumiko_fujikawa`. Generate all three migrations through
the maintained command and retain the five newest snapshots.
- Add machine pairing that generates a fresh SSH key per request, stores
its private half in the company vault, and uses the generated public
half in the install command. Do not trust editable saved metadata.
Preserve existing authorized keys, including a file without a final
newline. Keep SSH username, workspace, and trusted host keys explicit.
- Use the existing SSH driver. Resolve the machine's current address
from its stable device ID. Reject disconnected, removed, offline, and
wrong-company selections.
- Hide Network when Direct is the only option. Select a sole connected
tailnet automatically and hide its selector. Remove the requested setup
helper copy. Do not restrict the harness to Claude.
- Run a private userspace Tailscale daemon with a separate socket.
Persist its node state before execution. Restore it after restart, bound
startup, serialize reconnect cleanup, and close it during server
shutdown.
- Guard OAuth completion with a durable attempt marker. Disconnect and
newer authorization invalidate older callbacks. Close replaced daemons
and remove cancelled enrollment secrets without changing newer
connection state.
- Preserve SSH host verification and tailnet-specific session identity.
Honor the configured Tailscale executable path, including spaces and
OpenSSH percent escaping.
- Package checksum-pinned Tailscale clients and their license in the
standard runtime image. Exercise the client build stage in PR CI and
include it in image smoke checks.

## Verification

- Merge status: all CI and image checks pass, and current-head Apex is
5/5 with no unresolved comments. GitHub rejected a direct merge because
the active master ruleset requires code-owner approval for
`.github/workflows/pr-trusted.yml`. The [CODEOWNERS workflow
pattern](https://github.com/paperclipai/paperclip/blob/master/.github/CODEOWNERS#L3)
applies; eligible maintainers already have review requests. Auto-merge
is enabled with the squash method for the verified head. The PR remains
open pending that required approval; no branch policy was bypassed.
- Final revision: `fdb29b2cc366f4945b7f4c3e2b2d25070329a21b`, including
master `999709b0a6dd416af335b5debded13e1de7fcb7a`. [Final CI
verification
passed](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132213508320).
All required checks pass, all applicable test/build gates are green, and
the branch is mergeable against master. SSH prerequisite #21248 and
Cloud broker paperclipai/paperclip-cloud#1314 are already merged.
- [Apex reviewed this exact revision at
5/5](https://github.com/paperclipai/paperclip/pull/20617#issuecomment-6776629437).
There are no unresolved review threads. The PR is classified as Critical
impact.
- Final image qualification passes for the [application Runner
path](https://github.com/paperclipai/paperclip/actions/runs/43518584522/job/132206784267),
[amd64 Runner
image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296198),
and [arm64 Runner
image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296156).
Typechecks, Runner support, native/Rust tests, provider tests,
serialized-server suites, browser shards, and the remaining applicable
CI checks pass. This is source/image CI qualification, not a production
rollout.
- The last master conflict affected only Docker stage declarations and
their boundary test. Both the checksum-pinned Tailscale stage and
upstream `runner-lockfile` guard are retained, including the upstream
`deps` inheritance. The Tailscale client stage is byte-identical. All
180 cases across five Docker/image contract suites pass locally, along
with server and UI typechecks. The maintained 543-input SDK check has no
drift. Runtime and lock pins are unchanged. Secret/PII and whitespace
scans pass.
- Seven OAuth regression cases failed before the fixes. The current-tab
test uses the real handoff helper, checks that the session is stored
before reauthentication navigation, mounts the shared return page,
resumes consent, and verifies handoff cleanup. Callback tests check
trusted saved-connection context, fixed safe outcomes, failure audit
metadata, denied membership and expired-state rejection, JSON API
errors, and no provider text in browser returns. UI retries require a
connection in the current company's catalog. All 80 OAuth
route/service/UI/shared-handoff cases pass; server and UI source/test
typechecks pass.
- [Server shard 4/12 on the prior
refresh](https://github.com/paperclipai/paperclip/actions/runs/43510727031/job/132180301436)
passed 2,784 cases and skipped 21, then failed a reassignment fixture's
`TRUNCATE companies CASCADE` cleanup with PostgreSQL `40P01`. Its
automatic retry found the surviving `REA` prefix. The unchanged test
passes all three cases locally. A controlled reader/cleanup overlap
reproduces the same deadlock, but the exact competing CI query was not
logged. Each case now has a distinct company prefix and identifier; the
owned database is removed at suite end. The three test bodies and
assertions are byte-identical. Eight local reassignment/lock-release
cases pass, and the [final server shard 4/12
passes](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132207159392).
Production reassignment logic, deadlines, and retries are unchanged.
- The initial master integration passed 330 cases across 17 suites.
Coverage includes connector state, device discovery, network lifecycle,
pairing, routes, SSH binding and exports, environment persistence, and
connector UI. All three unpublished migrations were regenerated after
master's `0479_short_darkstar`: `0480_lazy_lethal_legion`,
`0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa`. Each SQL
body is byte-identical to its earlier unpublished migration. Generated
snapshots preserve existing master tables and add only the intended
Tailscale table and secret-purpose column. Exactly five snapshots
remain, 0478–0482. Numbering, safety, compatibility, and snapshot-drift
checks pass. The later Docker refresh did not alter this migration
sequence.
- Earlier attempts remain part of the verification record. On `1c020a0`,
Rust cache restoration and server shard 5/12 were interrupted by
shutdown signals, Runner shard 2/3 lost communication with no
retrievable log, and an amd64 provider download failed with npm
`ECONNRESET`. Same-head retries recovered, without establishing the
cause of the communication loss. Later superseded runs also had
cancellations with incomplete logs. These interrupted attempts are not
counted as passes. The final revision has fresh successful CI.
- Local platform limits remain recorded. The Kubernetes suite on macOS
passed 191 cases, failed 12 Linux `/proc/self/fd` transfer cases, and
skipped four Linux-only cases. Unchanged source reproduced the same 12
failures. An initial run also lacked provider dependencies until the
maintained preparation command installed them. Those local attempts are
not green qualification. Final Linux provider CI passes. Local Docker
qualification was unavailable; final image checks passed in CI.
- Before PR separation, a live tailnet/SSH Claude Code task completed on
Ubuntu ARM64 without provider login for the remote user. The selected
Paperclip subscription ran it, wrote the expected file, posted
completion, and marked the task done. File contents and private runtime
cleanup were checked. This is prior live evidence; the final revision
was not live-retested. Other providers have automated coverage only.
Hosted registration for unrelated tailnets and production concurrency
remain qualification limits below.
- The Superagent security and supply-chain reports remain neutral and
inconclusive. Earlier attempts to restart the failed security scan
returned HTTP 404. Supply-chain intelligence lacked coverage for
extracted pinned dependencies and the shared workflow reference. These
reports do not establish successful security qualification. The required
CI and review gates pass; this external coverage gap remains disclosed.

## Risks

- The Superagent security and supply-chain reports are inconclusive. The
security scan re-run request returned HTTP 404, and the supply-chain
scanner lacks coverage for the extracted dependencies and shared
workflow reference. These neutral results are not successful security
qualification.
- The server-owned secret purpose is internal database metadata, not an
HTTP-writable field. The generic secret service owns enforcement so
direct API reads and runtime projection share the same restriction.
Previously copied node state remains valid outside Paperclip until the
enrolled device is revoked in Tailscale; a local disconnect cannot stop
an external copy.
- Tailscale network configurations now require a connected Connector and
selected machine. The unreleased key-only path is removed; any saved
configuration from that path needs Connector setup.
- The connector remains a pilot. The local test used an OAuth app
registered in the consenting user's tailnet. Qualify hosted app
registration for unrelated customer tailnets before general
availability.
- This implementation supports one serving process per instance.
Concurrent replicas need exclusive ownership of the persisted network
identity before production rollout. Preserve the database and company
vault master key across restarts.
- Each machine-pairing preparation creates a new vault secret. Unused
generated keys follow the existing secret deletion lifecycle. A saved
SSH key remains available as an explicit choice.
- Apply additive migrations `0480_lazy_lethal_legion`,
`0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa` before
application rollout. The new credential version and digest columns are
nullable. Existing unpinned connections require fresh OAuth consent; the
migration deliberately does not trust or backfill a generic secret
value. Regenerate or renumber migrations if master advances the sequence
before merge.
- The target needs an SSH server, its chosen harness, and allowed
tailnet access. A reachable network does not grant SSH or provider
authorization. Pairing reads standard `/etc/ssh` host-public-key files;
custom host-key locations require manual known-hosts configuration.
- Roll back the application revision to remove new entry points.
Existing connection secrets and registered Tailscale nodes retain their
separate lifecycle. Do not drop the table or delete node state as part
of an application rollback.
- Production qualification still needs denied-policy, revocation,
cancellation, and deployment-overlap checks. Local Docker and a
production deployment were not performed.

## Model Used

OpenAI Codex, gpt-6.1-sol, xhigh reasoning, with local code execution
and tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and this extends existing connector and
environment features
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR using the feature request fields
- [x] I have not referenced internal issues, local URLs, or private
machine identifiers
- [x] My branch name describes the change
- [x] I have run focused tests locally and they pass
- [x] I have added tests for new behavior and failure paths
- [x] I have updated documentation
- [x] I have documented migration and runtime risks
- [x] All Paperclip CI gates are green
- [ ] Required code-owner approval received for the workflow change
- [x] Greptile Apex is 5/5 on the final head with no unresolved comments
- [x] I will address CI and review findings before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 15:15:43 -08:00
Nicky LeachandClaude Fable 5.2 647641a99b feat(ci): let docker.yml copy the published Runner image, behind a repository variable (#21340)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work. `master` publishes
it alone as an attested image, once for each set of its inputs
> - Since #21332 the application image takes its Runner only from the
`runner-artifacts` stage, and a build can replace that stage with the
published image
> - No workflow does that yet, so every application image still builds
its own Runner, and nothing on an application image says which Runner it
has
> - A build that copies the published Runner must also build the
lockfile that Runner was built from. The workflow refreshes the lockfile
before every build, and the server build then refuses the published
Runner
> - This pull request lets `docker.yml` copy the published Runner image
and label the application image with it, behind a repository variable
that is off, and makes that build use the committed lockfile
> - The benefit is one Runner for all images of the same inputs, named
on the image, with a switch that goes back to today's builds without a
revert

## Linked Issues or Issue Description

Refs #21204, #21229, #21230 (the Runner image), #21306 (the resolver),
#21307 (the report-only job), #21308 (the server reads a pack from its
Runner image), and #21331, #21332, #21333 (the application image takes
its Runner from `runner-artifacts`). No issue exists for this change.

**Problem or motivation**

`docker.yml` builds the application image on `master` and on tags. Each
build makes its own Runner. The report-only job that runs the resolver
on `master` feeds no build. Downstream image composers cannot tell from
an application image which Runner it has.

There is a second problem, which only shows when a build copies the
published Runner. The Runner's build stamp covers the whole
`pnpm-lock.yaml`. The step that vendors the Runner into the server fails
when the stamp does not match the tree. `docker.yml` refreshes
`pnpm-lock.yaml` on the runner before every build. The refresh rewrites
the file whenever the manifest of any package is ahead of it, also a
package the Runner does not read. On the tree of this pull request it
rewrites the file even for a pristine checkout: the registry changed the
deprecation text of one package. The published Runner was built from the
committed file, so the image build would fail.

**Proposed solution**

One job resolves the Runner once for the run. Both architecture builds
need it and add its answer to their build: a build context that replaces
the `runner-artifacts` stage, two labels, and the build argument with
which the build checks the Runner. A build that copies the published
Runner does not refresh the lockfile: it builds the committed tree, the
tree the Runner was published for. The job runs only when the repository
variable `RUNNER_IMAGE_CONSUME_ENABLED` is exactly `true`. This pull
request merges with the variable unset, so no image changes when it
merges.

**Alternatives considered**

- Resolve in each architecture build. The two builds could then
disagree, and one image index would hold two Runners.
- Write the labels in the `Dockerfile`. A `LABEL` cannot be left out for
an inline build.
- For the lockfile, copy the Runner image's `/pnpm-lock.yaml` over
`/app/pnpm-lock.yaml` in the `build` stage. Nothing after that step
reads the file: the two vendor checks are its only readers in `build`,
`production` copies it and takes `node_modules` from `deps`, both
`cloud-*` stages install with `--ignore-workspace --no-lockfile`, and
the server reads only a plugin's own lockfile and the pack's own
lockfile at run time. But the image would then ship a lockfile that its
`node_modules` did not come from, the copy changes that file's layer for
every inline build, and each `Dockerfile` edit makes a new Runner image.
- For the lockfile, bind-mount the Runner image's lockfile over the file
for the two steps that run the vendor check. This also edits the
`Dockerfile`, and it makes the lockfile part of the check always true
for a published Runner.
- For the lockfile, let the resolver refresh and compare. The answer
would depend on the registry at two different moments.
- Skipping the refresh needs no `Dockerfile` edit, keeps the stamp check
as strict as it is, and builds exactly the tree the inputs digest was
computed from.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This pull request is built on `runner/app-image-consume`. Merge in this
order:

1. #21331, #21332, #21333: the application image takes its Runner from
the `runner-artifacts` stage, checks the inputs digest a build names,
and stops early on a stale lockfile.
2. `runner/app-image-consume`: the wait and the `unsupported` answer of
the resolver.
3. This pull request.

The same change for the preview cloud image build in `release.yml` is
deferred. It has no pull request yet.

## What Changed

- `.github/workflows/docker.yml`: the job `runner_image` replaces
`runner_image_report`.
- It runs only when `vars.RUNNER_IMAGE_CONSUME_ENABLED == 'true'`, in
this repository. A fork never runs it.
- Steps: `inputs` with no credential; `resolve`; then `gh attestation
verify` of the full index against the signer `runner-image.yml` on
`refs/heads/master`.
- For a `master` push, `resolve` runs with `--wait` and the job token.
For a tag or a manual run it does not wait and receives no token.
- Only the verification step writes the job's outputs. An image that
does not verify fails that step with a warning and writes no output. The
job is `continue-on-error`: the job is red, the run is not, and the
builds are inline.
- Permissions: `contents: read`, `actions: read`, `attestations: read`.
No registry login, no write.
- Limits: 5 minutes for `inputs`, 55 for the lookup, 5 for the
verification, 70 for the job. The wait of 40 minutes ends inside them.
- `build-and-push` needs `runner_image` and adds its outputs to the
build: `build-contexts`, one more line of `build-args`
(`PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST`), two more lines of `labels`.
With empty outputs these are the values the build had before.
- The step "Refresh lockfile for Docker build context" runs only when
the answer is not `published`.
- The step that looks for a merge-queue image of the commit runs only
when the answer is not `published`. The two steps after it already
depend on its result.
- Labels on an image that copied the published Runner:
`io.paperclip.runner.image=ghcr.io/paperclipai/paperclip-runner@sha256:<full
index digest>` and `io.paperclip.runner.inputs-digest=<digest>`. An
image without them made its own Runner.
- The `Dockerfile` does not change. The Runner image digest does not
move.
- Tests that pin lines this changes follow:
`scripts/merge-group-image.test.mjs` (the condition of the merge-queue
lookup), `scripts/reuse-canary-image.test.mjs` (the `needs` of the build
job), and `docker-build-stamp.test.ts` (no workflow names a build
context; `docker.yml` now names exactly one).
- `server/src/__tests__/docker-runner-workflows.test.ts`: the
report-only tests give way to tests of the new job.
- `server/scripts/prepare-runner-vendor.test.mjs`: one fixture test of
the lockfile case.
- `doc/runner-image-contract.md` and `doc/DOCKER.md`: how an application
image gets its Runner, the labels and what their absence means, the
variable, the wait, the verification, the lockfile rule, the merge-queue
rule and the rollback.

## Verification

### Real builds

`docker buildx build --target build --platform linux/arm64 --output
type=cacheonly`, with a `docker-container` builder made for this test.
The Runner image is the publication form
(`PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1`, `--target runner-artifacts`)
built from a clean export of this branch, exported as an OCI layout:
inputs `c4f9d089…34f0`, image `sha256:bef7e9cb…ffef`. The application
builds receive `--build-context
runner-artifacts=oci-layout://…@sha256:bef7e9cb…` and
`PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST=c4f9d089…`. The refresh command
is the one in the workflow, run with pnpm 9.15.4 in the export: `pnpm
install --resolution-only --ignore-scripts --no-frozen-lockfile`.

| Build context | Result |
| --- | --- |
| The committed tree | Passes in 78 s. Both vendor checks: `reusing the
current Runner build (inputs unchanged …; 1333 files and 1860 outputs
checked)` |
| `ui/package.json`: `clsx` `^2.1.1` → `^2.1.0`, then the refresh
command (what the workflow did before this change) | Fails at `RUN node
server/scripts/prepare-runner-vendor.mjs`:
`PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1 forbids the Runner rebuild
this step needs (inputs changed since the recorded build; 1333 files
checked)` |
| The committed tree, then only the refresh command | Fails the same
way. The refresh changed only the `deprecated` text of one package |
| The same `ui/package.json` change with no refresh (what the workflow
does now for a `published` answer) | Passes in 66 s. `deps` resolves the
manifest inside the image; both vendor checks reuse the Runner |
| One line added to `packages/paperclip-runner/src/index.ts` | Fails at
the vendor step, as before: a Runner of other inputs is still refused |

Lockfile behind the Runner's own manifests
(`packages/paperclip-runner/package.json`: `smol-toml` `^1.9.1` →
`^1.9.0`), no build context:

| Step | Result |
| --- | --- |
| `runner-image-resolve.mjs inputs`, then `resolve` | `lock: behind`, no
digest; `inline (lock-behind)` |
| Build without the refresh | Fails in 4 s at the `runner-lockfile`
stage: `ERROR: pnpm-lock.yaml is behind the manifests the Runner is
installed from …` |
| Build after the refresh command (what the workflow does for an
`inline` answer) | Passes in 193 s: the `runner-lockfile` stage passes,
the build makes its own Runner with its pack, and both vendor checks
reuse it |

### Tests

macOS (Node 26.12.0) and Linux (`node:24.22.3-bookworm`, linux/arm64,
with `jq` and `zstd` added), on this branch:

- `pnpm exec vitest run` on `docker-runner-workflows`,
`runner-image-resolve`, `runner-image-workflow`, `docker-build-stamp`,
`container-init-reaping`, `cloud-image-bundled-plugins`,
`docker-image-layer-reuse`, `cloud-image-sentry`,
`runner-lockfile-guards` and `runner-image-identity` in
`server/src/__tests__`, and
`server/scripts/prepare-runner-vendor.test.mjs`: 225 tests in 11 files
pass on Linux; on macOS 224 pass and 1 is skipped (it needs bash 4.4).
- `node --test .github/scripts/tests/*.test.mjs
scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs
scripts/cloud-source-verification.test.mjs
scripts/standard-image-contract.test.mjs
scripts/preview-artifacts.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs
scripts/__tests__/release-dry-run-notes.test.mjs`: 991 of 991 pass on
both.
- The changed test files typecheck with the options of
`server/src/__tests__/tsconfig.json`.

What the workflow tests hold:

- Variable off: the job's condition is the variable and the repository,
with no alternative. With empty outputs, the build step has exactly the
3 build arguments, 0 build contexts and 1 label entry it had before, and
the refresh step runs.
- Published: both architecture builds are one job that reads only the
outputs of `runner_image`, with no architecture in those lines.
- The verification step runs as real shell against a stub `gh`. When
`gh` succeeds it writes the context, the labels and the build argument.
When `gh` fails the step exits 1, writes no output and prints the
warning. The step has no `continue-on-error`; the job has. The condition
of the build job does not name the resolver job and has no `success()`.
- The lookup step runs as real shell against a stub `node`. It passes
`--wait` only when the event is a `master` push.
- Credentials: only the lookup and the verification steps have a token.
The steps that build have no `env`.
- The refresh step has the condition `needs.runner_image.outputs.mode !=
'published'`, and it is the only `pnpm install` of the build job.
- The fixture test gives the vendor step a Runner that another tree
built for the committed lockfile. The step reuses it. After a change of
one specifier of another package in the lockfile the step fails and
names the lockfile. With the committed lockfile back it reuses the
Runner again.
- The merge-queue workflow and the pull-request check do not name the
resolver, the variable, a build context or a Runner label.

Read-only checks against the real services:

- `gh attestation verify
oci://ghcr.io/paperclipai/paperclip-runner@sha256:0cd8250b…1bf7 --repo
paperclipai/paperclip --signer-workflow
paperclipai/paperclip/.github/workflows/runner-image.yml --source-ref
refs/heads/master`: exit 0 for the full index, and exit 0 for the slim
index `sha256:69ab2001…69b3`.
- The same command with `--signer-workflow …/docker.yml`: exit 1 for
both. With `--source-ref refs/heads/not-master`: exit 1. With `--repo
cli/cli`: exit 1. For a digest that does not exist: exit 1. With no
token: exit 4.
- The parsed YAML of `docker.yml` differs from its base only in the
resolver job, the `needs` of the build job, the conditions of the
refresh step and of the merge-queue lookup, and the three build inputs.

Never run:

- `docker.yml`. Only GitHub can run it, and it has no pull-request
trigger.
- A build with the variable on. The first one is the first run after a
maintainer sets the variable.
- A build that pulls the published Runner from the registry as
`docker-image://`. The real builds used the same image form from a local
OCI layout.
- An `amd64` build.
- The `production` and `cloud` targets. The real builds stopped at
`build`, the last stage that checks the Runner.

## Risks

- With the variable on, an application build on `master` starts after
the resolver job. That is 2 to 3 minutes when the image exists, and up
to 40 minutes from the push when the commit's own Runner publication is
in progress. `docker.yml` holds one concurrency slot for each ref, so a
long wait can make more `master` commits lose their image build to a
newer push.
- A build that copies the published Runner ships the committed
`pnpm-lock.yaml` in `/app`, while `deps` resolves newer manifests of
other packages inside the image. A local `docker build` has always done
this; a CI image did not. Nothing in the image reads that file.
- Any other build that gives the `Dockerfile` a published Runner and a
refreshed lockfile fails at the vendor step. That is the intended answer
to a tree the Runner was not built for, and the message names the
lockfile.
- The pack of an image that copied the published Runner names the commit
that published the Runner (`runnerSourceRevision`), which can be older
than the commit of the image. A consumer that compares the two must
accept that before the variable is set.
- A failed verification makes the resolver job red on a green run. That
is on purpose: it means a published tag whose attestation does not
verify.
- `scripts/merge-group-image.mjs` refuses to promote a merge-queue image
for a push that changes `docker.yml`. The push that lands this change is
such a push. The merge queue is not enabled.
- With the variable off there is no report of the resolver on `master`
any more: the job that replaced the report-only job does not run.
- Rollback: set the variable to another value or delete it. The next run
builds inline, with the refresh. Images that exist do not change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds, read-only
calls to the GitHub API and the container registry).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 14:47:20 -08:00
DottaandClaude Fable 5.2 c6301c92ff fix(ci): keep the typecheck job working on stacked PRs and stop its CLI guidance test timing out (#21451)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every pull request runs the required `Typecheck + Release Registry`
job from the trusted workflow on `master`
> - #21404 replaced that job's six sequential steps with one script,
`scripts/ci-typecheck-lane.mjs`, that runs independent checks in
parallel
> - PR runs take the workflow from `master` but check out the PR's merge
commit. A stacked PR's merge commit is built on its base branch, so it
can lack a script that only `master` has
> - The lane also ran the CLI guidance test beside the typechecks at a
lower CPU priority, and one of its cases is slow enough to cross its 15
second timeout there
> - This pull request makes the job fall back to the earlier commands
when the script is absent, and runs the CLI guidance test before the
parallel checks
> - The benefit is that stacked PRs typecheck again and the job stops
failing on a timeout that has nothing to do with the PR

## Linked Issues or Issue Description

Refs #21404

**What happened?**
In the first eight PR runs after #21404 merged (2026-11-13 18:06Z), the
typecheck job failed three times for reasons unrelated to the PRs:

- Twice on #20617, a PR stacked on `feat/ssh-agent-ai-accounts`: `Error:
Cannot find module
'/home/runner/_work/paperclip/paperclip/scripts/ci-typecheck-lane.mjs'`
(jobs 132076887259 and 132079570039). The job failed 0 seconds into the
step.
- Once on run 43480469363 (job 132075678943): `CLI guidance safety
failed with exit code 1`, from `Error: Test timed out in 15000ms.` in
`allows only exact-allowlist pnpm paperclipai commands on every guidance
surface`, which took 17059 ms. The same case passes in about 5 seconds
when the test runs alone.

**Expected behavior**
The typecheck job passes or fails on the PR's own code, including for
stacked PRs.

**Actual behavior**
Stacked PRs always fail the job. Other PRs fail it intermittently (1 of
6 runs that reached the lane).

**Steps to reproduce**
Open a PR whose base branch does not contain
`scripts/ci-typecheck-lane.mjs`, or run the lane on a loaded 4 vCPU
runner.

## What Changed

- `.github/workflows/pr-trusted.yml`: the lane step runs only when
`scripts/ci-typecheck-lane.mjs` exists in the checkout. A new step runs
the six earlier commands, in their earlier order, when it does not. Only
the `typecheck_release_registry` job changed.
- `scripts/ci-typecheck-lane.mjs`: the CLI guidance test moved from the
parallel checks to the end of the sequential preparation steps, at
normal priority.
- Tests: `scripts/__tests__/ci-typecheck-lane.test.mjs` and
`.github/scripts/tests/ci-plan.test.mjs` assert the two conditions, that
the earlier commands appear only in the fallback step, and that the CLI
guidance test is a preparation step.
- `doc/DEVELOPING.md`: the "Typecheck lane in CI" section describes
both.

## Verification

- `node --test scripts/__tests__/ci-typecheck-lane.test.mjs`: 8 pass.
- `node --test '.github/scripts/tests/*.test.mjs'` (the `policy` job's
command): 807 pass.
- `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`:
11 pass.
- Not exercised by this PR's own CI: PR runs use the workflow from
`master`, so the new conditions first run after merge. The script change
does run here, because the script is read from the PR's checkout.
- Not measured: the lane's wall time with the CLI guidance test moved.
The test takes 2 to 5 seconds alone, so the expected cost is that much.

## Risks

- The fallback step repeats six commands in the workflow. It is needed
only until open stacked PRs are based on a branch that has the script,
and can be removed later.
- `hashFiles` returns an empty string when no file matches, which is
what the two conditions rely on.
- If a future PR deletes or renames the script, the job silently uses
the slower commands instead of failing. The wiring tests would fail in
that PR.

## Model Used

Claude Fable 5.2 (`claude-fable-5-2`), via Claude Code, with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 13:24:52 -06:00
DottaandClaude Opus 5.5 64ec273845 perf(ci): run the typecheck lane's checks in parallel and drop its duplicate server type check (#21404)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every pull request runs the trusted PR workflow, and the `Typecheck
+ Release Registry` job is one of its required checks
> - That job takes about 9.2 minutes on the 4 vCPU / 16 GB fleet runner
(run 43344818387), against a target of 7 minutes or less for every job
> - The job runs six steps one after another, and the first step
type-checks the whole server program a second time, at about 13.5 GiB,
which makes the runner swap
> - Typecheck results are the check that authors most want back quickly,
so this job is on the critical path of most pull requests
> - This pull request removes the duplicate server type check and runs
the independent checks in parallel through one small script, inside the
same memory budget
> - The benefit is about 39% less wall time for the job's checks on the
same machine, with peak memory down from about 14 GiB (plus swap) to
about 8.5 GiB

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The `Typecheck + Release Registry` job in
`.github/workflows/pr-trusted.yml`. It takes about 9.2 minutes. The
target for every job in the workflow is 7 minutes or less.

**Subsystem affected**

CI (`.github/workflows/pr-trusted.yml`) and the typecheck scripts under
`scripts/` and `server/scripts/`.

**Current behavior**

After about 75 s of setup, the job runs six steps in sequence on a 4
vCPU runner, about 470 s in all (run 43344818387):
`typecheck:build-gaps` 121 s, test programs 85 s, server layers 128 s,
test projects 38 s, CLI guidance test 5 s, release registry tests 95 s.
Most of these steps use one thread. The `typecheck:build-gaps` step
builds the server with a full type check of the whole server program.
The server layer step then checks the same files again, and the `Build`
job checks them a third time. That full program peaks at about 13.5 GiB,
so the job peaks at about 14 GiB of 16 GiB and uses swap.

**Proposed behavior**

The job runs `node scripts/ci-typecheck-lane.mjs`. The script builds the
server without type checks (`PAPERCLIP_TSC_EMIT_ONLY=1`, as the Docker
image does) and still checks the runtime assets. It runs the steps that
write shared files first, one at a time. Then it runs the remaining
checks in parallel, one per CPU, with the server declaration layers
first, because the test layers and test projects read them. The checks
that are not on that path run at a lower CPU priority.

**Reason and benefit**

On the same runner, the job's checks take 61% of the time and about 60%
of the memory (see Verification). The job should take about 6 minutes on
the fleet instead of 9.2.

**Breaking changes**

None. The same files are type-checked with the same options. Local
commands do not change. `pnpm run typecheck:build-gaps` still works as
before.

## What Changed

- `scripts/ci-typecheck-lane.mjs` (new): runs the job's checks. It
writes shared outputs first, in sequence: links, plugin SDK, server
build without type checks, the runtime asset check, the sandbox provider
installs, and the Runner typecheck (it rebuilds the eval kernel, which
other checks read). Then it runs these checks in parallel: the server
declaration layers, each server test layer, each test project, the test
programs, the other build-gap workspaces, the CLI guidance test and the
release registry tests. A passing check prints its output in a collapsed
log group. A failing check prints its full output between `=== FAILED`
and `=== end of` lines, with an error annotation. The script ends with a
table of each check's start, duration and result, and exits non-zero if
any check fails.
- `server/scripts/typecheck-layers.mjs`: new `--layer <name>` option
that checks one layer against the configs and declarations of an earlier
`--declarations-only` run, and an exported `listLayers`.
- `scripts/typecheck-test-programs.mjs`: new `--prepare-only` and
`--skip-prepare` options, so the installs happen before the parallel
checks start.
- `scripts/run-typecheck-build-gaps.mjs`: new `--list-packages` option,
so the script still finds the build-gap workspaces and still fails when
one has no `typecheck` script.
- `.github/workflows/pr-trusted.yml`: the `typecheck_release_registry`
job runs the script instead of the six steps. No other job changes.
- Tests: new `scripts/__tests__/ci-typecheck-lane.test.mjs` (added to
`test:release-registry`), new cases in
`server/scripts/typecheck-layers.test.mjs` and
`scripts/typecheck-test-programs.test.mjs`, and the workflow wiring
assertions in `.github/scripts/tests/ci-plan.test.mjs` and
`scripts/__tests__/runner-release-crate-cache.test.mjs`.
- `tests/runner-e2e/source-sdk-public-inputs.json`: repinned, because
`package.json` changed.
- `doc/DEVELOPING.md`: new "Typecheck lane in CI" section, and the new
options.

## Verification

This PR's own CI ran the old job definition, because PR runs use the
workflow from master. So I measured the commands on a stand-in machine:
a temporary workflow (`.github/workflows/typecheck-lane-bench-temp.yml`,
now deleted) on GitHub-hosted `ubuntu-latest` (4 vCPU Intel Xeon
6975P-C, 15.7 GiB, no secrets, `contents: read`). It ran in runs
[43444356017](https://github.com/paperclipai/paperclip/actions/runs/43444356017),
[43445393958](https://github.com/paperclipai/paperclip/actions/runs/43445393958),
[43446786105](https://github.com/paperclipai/paperclip/actions/runs/43446786105),
[43448707837](https://github.com/paperclipai/paperclip/actions/runs/43448707837)
and
[43450786493](https://github.com/paperclipai/paperclip/actions/runs/43450786493).
Each step ran under `/usr/bin/time -v`, and a sampler read
`/proc/meminfo` every 0.25 s.

**Old steps, one at a time (3 runs, run 43445393958):**

| Step | Wall s | CPU s | Peak RSS of one process | System memory peak |
| --- | --- | --- | --- | --- |
| `typecheck:build-gaps` | 128–155 | 376–400 | 13.2–13.6 GiB | 14.2–14.7
GiB |
| ...of which the full server `tsc` | 40–51 | 146–164 | 12.8–13.8 GiB |
13.7–14.7 GiB, swap up to 738 MiB |
| ...replaced by the emit-only server build | 6–10 | 20–22 | 2.9 GiB |
3.9 GiB |
| Test programs | 67–87 | 147–170 | 1.9 GiB | 2.9 GiB |
| Server layers | 102–127 | 150–172 | 4.0–4.2 GiB | 5.4–5.6 GiB |
| Test projects | 31–37 | 49–59 | 2.1–2.2 GiB | 3.1–3.2 GiB |
| CLI guidance test | 2–4 | 2–3 | 0.2 GiB | 1.6 GiB |
| Release registry tests | 82–95 | 159–183 | 1.7 GiB | 3.0 GiB |

CPU seconds are about twice the wall seconds for the `tsc` steps even on
one checker thread. So each step uses about two of the four vCPUs.

**Old sequence and new lane on the same runner (4 runs, run
43450786493).** Each job copied one installed checkout, then ran both,
in alternating order:

| Run | Old steps, wall s | New lane, wall s | Ratio | Old memory peak |
New memory peak |
| --- | --- | --- | --- | --- | --- |
| 1 | 503 | 314 | 0.62 | 13.98 GiB | 8.33 GiB |
| 2 | 647 | 388 | 0.60 | 14.04 GiB, swap 637 MiB | 8.13 GiB |
| 3 | 627 | 385 | 0.61 | 13.98 GiB | 8.63 GiB |
| 4 | 382 | 237 | 0.62 | 14.10 GiB, swap 1.1 GiB | 7.99 GiB |

Hosted runners differ in speed by up to 1.7 times, so only the
same-runner ratio is meaningful. Both arms include a cold compile of the
Runner release crate (45–80 s) that the fleet restores from cache. The
fleet ran the old steps in 470 s; at the measured ratio, the new lane
should take about 290 s, and the job about 6 minutes with setup. The new
lane peaks at about 55% of memory, with no swap.

Local checks (on a loaded Mac, for correctness only):

- `node --test scripts/__tests__/ci-typecheck-lane.test.mjs` — 8 pass.
- `node --test .github/scripts/tests/*.test.mjs` — 807 pass.
- `node --test scripts/__tests__/e2e-shard.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs
scripts/__tests__/runner-release-crate-cache.test.mjs
scripts/__tests__/run-typecheck-build-gaps.test.mjs` — pass.
- `pnpm exec vitest run scripts/typecheck-layers.test.mjs` in `server/`
— the new cases pass.

## Risks

- This changes a required job that every PR depends on, and its own CI
cannot prove it. Please watch the first `Typecheck + Release Registry`
runs on master after merge: the wall time (expect about 6 minutes), the
summary table at the end of the step, and any `terminated by` message.
- A type error now shows in that check's `=== FAILED` block, not in a
step of its own. Passing checks are folded, so the failure is the only
expanded block.
- When the server declaration layers fail, the test layers and test
projects are reported as skipped. Before, the test layers still ran
after a source error. Fix the source errors first; the next run checks
the tests.
- The Honeycomb CI time board and any step-name queries see one step,
`Typecheck, CLI guidance safety and release registry tests`, instead of
six. The script prints the duration of each check.
- The release registry tests run beside the type checks at normal
priority. Their slowest test took 46–78 s here, against a 180 s timeout,
and 114 s on the slowest stand-in runner.
- Low risk for memory: the new lane peaked at 8.0–8.6 GiB of 15.7 GiB,
far below the old 14 GiB.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool
use (shell, GitHub CLI, file editing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:06:01 -06:00
Nicky LeachandClaude Fable 5.2 84a4d345b3 refactor(docker): take the application image's Runner from the runner-artifacts stage (#21332)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work. `master` publishes
it alone as the image `ghcr.io/paperclipai/paperclip-runner`, built by
the `runner-artifacts` stages of the `Dockerfile`
> - The application image does not use those stages. It has five stages
of its own that compile the daemon, it builds the Runner again inside
the server build, and it builds a second provider pack
> - So one `Dockerfile` holds two recipes for the Runner, and the two
results differ: another base image and another daemon binary
> - This pull request removes the second recipe. The application image
copies the Runner out of the `runner-artifacts` stage, which the same
build runs once
> - The benefit is one Runner recipe, and an application image whose
Runner is the Runner the contract describes. A later pull request can
replace the stage with the published image

## Linked Issues or Issue Description

This is the second part of a chain. The first part is #21331, which
removes the cache check that cannot run on this `Dockerfile`. Refs
#21204, #21229, #21230 (the Runner image), #21306 (the no-rebuild mode
of the vendor step), #21307 (the pull-request check of the application
image) and #21312. No issue exists for this change.

**Problem or motivation**

The stages `rust-toolchain`, `rust-chef`, `runner-plan`, `runner-deps`
and `runner-build` compile the daemon on Debian trixie. The server build
then runs the full build of the Runner package. The
`runner-provider-pack` stage builds the provider pack. The
`runner-artifacts` stages do all of that again for the Runner image, on
Debian bookworm, with a stripped daemon.

**Proposed solution**

Read only the `runner-artifacts` stage from the application stages.
Build it inline in every build. Do not pass a published image yet.

**Alternatives considered**

- Pass the published Runner image to the build in this pull request.
That needs the resolver, labels and a rule for commits without an image.
It is the next step, and this one must be safe alone.
- Keep the old stages as a fallback. Two recipes is the problem.

## What Changed

- `Dockerfile`: remove the five stages. `build` starts from `base`. It
copies `/runner-package/dist` and the build stamp from
`runner-artifacts`, sets `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` and
runs the vendor check before the first application build.
- `Dockerfile`: `runner-provider-pack` starts from `base`. One `RUN`
with a bind mount of `runner-artifacts` copies the pack, the receipt and
the archives, and moves the large files to their own tree. `build` and
`runner-provider-pack` now come after the stages they read.
- `Dockerfile`: the three `COPY` instructions of `production` that take
the pack are byte-identical. A test pins all three.
- `Dockerfile`: `build` declares the build argument
`PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST`, empty by default, and one step
runs the new `scripts/check-runner-image-expectation.mjs` on
`/runner-image.json` of `runner-artifacts`. **This pull request declares
the argument. No build sets it. The next pull request, which gives a
published Runner image to the build, is its first user.** With the
argument set, the build fails unless `runner-artifacts` is the published
image of exactly those inputs. `docker-app-check.yml` also runs when the
script changes.
- Comments: the `Dockerfile` has 220 comment lines, down from 247. The
comments of the two moved stages moved with them, unchanged, so the diff
shows them as added lines. Eight comments that named a removed stage or
a step that moved get the smallest correction. Two comments in
`prepare-runner-vendor.mjs` and `build-inputs.mjs` said that the
`Dockerfile` relies on the standalone rebuild; that clause is removed.
- Tests: replace the two cargo-chef tests of
`docker-build-stamp.test.ts` with three tests of the new shape. Update
the pack test, `docker-image-layer-reuse.test.ts`,
`vite-build-commit.test.ts`, the stage rules of
`runner-image-identity.test.ts`, and the include-layout test of
`runner-release-crate-cache.test.mjs`, which now reads
`runner-artifacts-runnerd`.
- Docs: `doc/DOCKER.md`, `doc/DEVELOPING.md`, and a new section "The
application image" in `doc/runner-image-contract.md`.

## Verification

All builds: `linux/arm64`, local, `--build-arg
PAPERCLIP_BUILD_COMMIT=<sha>` with one commit for every build. The
builds ran before the rebase onto `master`, on the head of #21310 as it
was before it merged. `master` changed neither the `Dockerfile` nor an
input of the Runner in between.

- `docker buildx build --target production`, cold, before and after: 506
s and 410 s. Compressed size: 2,971 MB and 2,900 MB.
- The image config and the text of all 33 history rows are identical.
The row `COPY /provider-pack-transport/ /opt/paperclip-runner/ #
buildkit` matches the pattern of `retained-provider-pack-oci.ts` exactly
once.
- A comparison of every file of the two images (path, type, size, mode,
link target, and SHA-256 under `/app` and `/opt`). The differences are
in "Risks". All other files are byte-identical, the root `node_modules`
included.
- From the image after: the four `/app/packages/paperclip-runner/...`
paths and the contents of `/opt/paperclip-runner/` exist.
`paperclip-runnerd --build-metadata` runs. `provider-pack.json` has
`runnerSourceRevision` equal to the build commit and no
`runnerInputsDigest`. The entry points `index.js`, `live/index.js` and
`testing.js` of the Runner import.
- The vendor step logged `reusing the current Runner build` in both of
its runs. Every `Compiling` line of the log is in
`runner-artifacts-runnerd`.
- `--target build`, `--target runner-provider-pack` and `--target cloud`
built. The `build` stage has no `gcc`, `cc`, `cargo`, `rustc` or
`pkg-config`.
- `--target runner-artifacts` and `runner-artifacts-slim`, before and
after: `runner-image-layout.mjs check` and `runner-image-oci.mjs
check-layers` pass on both, in `node:24.22.3-bookworm`. The daemon,
`provider-pack.json` and the two stable layers have the same digests
before and after.
- Without a build commit: `build` and `runner-provider-pack` built, and
the pack was skipped with the same message. This run was on the next
branch of this chain, after the rebase. The last steps of `production`
were not run for this case.
- The expectation argument, on this `Dockerfile`, `--target build
--output type=cacheonly`, after the rebase:
- No argument, inline Runner: the build passes (233 s). The step passes.
- Argument, inline Runner: the build fails at the step. The message says
that the `runner-artifacts` of the build are not a published Runner
image.
- Argument `d4df566f…` and the published image of those inputs as
`--build-context
runner-artifacts=docker-image://ghcr.io/paperclipai/paperclip-runner@sha256:0cd8250b…`:
the step passes, and the build runs no Runner stage. The vendor step
then fails with `inputs changed since the recorded build`, because this
tree differs from the tree of that image by one comment line in
`build-inputs.mjs`. With that line put back, the same build completes,
and the vendor step reuses the published build.
- Another digest with the same context: the build fails at the step. The
message names the inputs the image has.
- A tree whose lockfile is behind the manifests of the Runner fails at
the frozen install of the Runner stages, after 169 s, with the message
that stage already had.
- `docker buildx build --check`: no warning, before and after.
`scripts/check-docker-deps-stage.mjs` and
`scripts/check-node-version-policy.mjs` pass.
- Tests, on macOS and in `node:24.22.3-bookworm`, after the rebase: the
changed server test files with the other tests that read the
`Dockerfile` or the image workflows (231 pass in 10 files on Linux),
`vite-build-commit.test.ts` (10 pass),
`runner-release-crate-cache.test.mjs` (11 pass),
`docker-deps-postinstall.test.mjs`. The changed server test files
typecheck.
- Not run: the full test suite, any `linux/amd64` build, `podman`, the
workflows and the release pipeline. Only CI can prove amd64,
`docker.yml`, `docker-merge-group.yml`, `release.yml`,
`docker-app-check.yml` and `runner-image.yml`.

## Risks

- **The digest of the Runner image changes.** Its identity covers the
whole `Dockerfile`. The next `master` push publishes a new
`inputs-<digest>`. The contents do not change: see the comparison above.
- **A build that is given a Runner of other inputs fails; it does not
ship that Runner.** With the argument, it fails at the expectation step.
Without the argument, it fails at the vendor step when the build inputs
of the Runner differ. One comment line in `build-inputs.mjs` changes
those build inputs in this pull request, so the image published before
this pull request cannot be vendored into a tree after it.
- **What a consumer of the application image sees** (measured on
`linux/arm64`):
- The daemon is the stripped binary compiled on Debian bookworm: 23.2
MB, was 36.4 MB, at
`/app/packages/paperclip-runner/dist/bin/paperclip-runnerd` and at the
vendored copy in `server/dist`. Its hash changes. It has no symbols for
a crash backtrace.
- `/app/packages/paperclip-eval-kernel/dist` is absent. Only
`dist/eval/workflow-harness.js` of the Runner imports that package. Only
`eval/index.js` exports the harness again, and no module imports that
file. The server runtime bundle does not name the package.
- The provider pack: `provider-pack.json` is byte-identical. Two
provider executables and one licence file are hard links to their other
copy in the pack, where they were separate files. One empty directory is
absent. The pack is 195 MB smaller, and the layer of its large files
gets a new digest once.
- The transport archives have other bytes, because the Runner build
gives large files a fixed timestamp. The receipt names the same files
with the same digests.
- Paths, modes, owners, the image config and the layer order do not
change.
- **What a contributor and a maintainer of the image build see:**
- **The image build does not run the drift checks of the Runner.**
Before, the server build ran the full `build` script of the Runner, so a
failed drift check failed the image build on `master`. Now the Runner
stages run only the output steps of that script. Repository CI runs the
checks on every pull request. `.github/docker-context-checks.Dockerfile`
and three exceptions in `.dockerignore` exist for those checks in the
image build; a separate follow-up removes them, and this chain does not
edit them.
- **The dependency-caching layer of the daemon is gone.** Before,
cargo-chef compiled the Rust dependencies in a layer of their own, and a
change to the Rust source compiled only the Runner's own crates. Now
such a change compiles the dependencies again (146 s here for the whole
daemon). A change to the server, the UI or the Runner's TypeScript still
reuses the compiled daemon.
- **A lockfile behind the manifests of the Runner fails the build.**
Before, `docker build` resolved it inside the image. The image workflows
refresh the lockfile first. A local build prints the command to run. The
next pull request makes this failure early and clear.
- **The build needs more disk**: about 25 GB of builder storage here,
was about 20 GB, with the image export.
- Rollback: revert this pull request. Images that exist do not change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude
Code. The model used tools: it edited files, ran the tests and ran the
Docker builds that this description reports.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 09:48:49 -08:00
Nicky LeachandClaude Fable 5.2 bef2aac905 chore(ci): remove the Docker dependency cache check of the Runner stages (#21331)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work. The application
image contains it
> - Today the application image compiles the Runner in its own stages,
with a cargo-chef dependency layer
> - A pull-request job proves that a fresh builder reuses that layer.
The job runs a script that appends to the `Dockerfile` and names those
stages
> - The next pull request removes those stages. The application image
then takes its Runner from the `runner-artifacts` stages. The job cannot
run against that `Dockerfile`
> - This pull request removes the job and its script
> - The benefit is that the next pull request changes only the image
build, and its checks stay green

## Linked Issues or Issue Description

Refs #21204, #21229, #21230, #21306 and #21307 (the Runner image and the
preparation for this change). No issue exists for this change.

**Problem or motivation**

`.github/workflows/docker-runner-check.yml` has a pull-request job,
`Compile isolated native Runner`. It runs
`scripts/check-docker-runner-cache.sh`. The script adds stages to a copy
of the `Dockerfile` that start from `runner-build` and copy from
`runner-plan`. It reads the build log for the `runner-deps` step. The
job runs on every pull request that changes the `Dockerfile`. The pull
request that removes those stages would fail this job.

**Proposed solution**

Remove the job, its trigger and its script first, in a pull request of
its own.

**Alternatives considered**

- Remove the job in the same pull request as the stages. That pull
request is already large.
- Rewrite the script for the `runner-artifacts` stages. Those stages
have no dependency layer, so the property does not exist.
`runner-image.yml` already builds them on a pull request.

This is the first of three pull requests that move the application image
onto the `runner-artifacts` stages. It has to merge first: the removed
job runs on every `Dockerfile` change and builds stages that the second
pull request deletes. The other two follow on top of it.

## What Changed

- Delete `scripts/check-docker-runner-cache.sh`.
- Remove the `runner` job and the `pull_request` trigger from
`docker-runner-check.yml`. Only the manual jobs remain. The concurrency
group no longer reads a pull request number.
- Remove `Docker Runner check` from the workflow list of
`pr-merge-review.yml`. That notifier acts only on pull-request runs, and
the workflow has none now.
- Remove the paragraph of `doc/DOCKER.md` that describes the check.

## Verification

- `node --test .github/scripts/tests/notify-pr-merge-review.test.mjs`:
22 pass.
- `vitest run server/src/__tests__/runner-image-workflow.test.ts`: pass
(1 test is skipped on macOS, where bash is 3.2).
- `vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/workflow-security.test.ts
tests/runner-e2e/workflow-target.test.ts`: 28 pass on macOS. These files
read `docker-runner-check.yml`.
- The same commands in a `node:24.22.3-bookworm` container: all pass,
except one test of `workflow-target.test.ts`. It fails with `jq: command
not found`. It reads another workflow, and the container has no `jq`.
- Not run: the full test suite, and the workflows themselves. Only CI
can run the workflows.

## Risks

- Until the next pull request merges, a pull request that changes only
the native inputs of the Runner (`packages/paperclip-runner/runner/**`,
`protocol/**`, `rust-toolchain.toml`) does not build the `runner-build`
stage of the application image. `runner-image.yml` builds the daemon
from the same files in the `runner-artifacts` stages on such a pull
request. `docker-app-check.yml` builds `runner-build` when the
`Dockerfile` or `.dockerignore` changes.
- No check proves the cargo-chef layer reuse after this change. The next
pull request removes that layer.
- The job was not a required check.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude
Code. The model used tools: it edited files, ran the tests and ran the
Docker builds that this description reports.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 08:20:29 -08:00
DottaandClaude Opus 5.5 2659ff1d22 fix(release): retry Cloud readiness jobs that lost their Spot runner (#21399)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every push to `master` publishes a canary, and the canary's image
reaches staging
> - Each canary waits for `Cloud source verified v1` from the Cloud
readiness run of its exact SHA
> - Cloud readiness runs on AWS Spot workers. When a worker is
reclaimed, the gate failed and the image watcher cancelled the readiness
run
> - Nothing retried that attempt. On a busy master the newest commit's
release failed, and merged work did not reach staging until a person
re-ran the run
> - This pull request re-runs the failed jobs of a readiness attempt
whose every failure is a lost runner. A real check failure still fails
closed
> - The benefit is that a Spot reclaim no longer holds a release

## Linked Issues or Issue Description

**What happened?**
On 2026-11-13, #21266 merged as `8af889f612`. No canary that contained
it published for more than an hour. The release workflow keeps one
running and one pending canary. A newer push replaces the pending one,
so the newest commit always runs next. That part works as designed. The
failures came from the readiness gate:

| Release (SHA) | Readiness run / attempt | Gate that blocked it | Cause
|
| --- | --- | --- | --- |
| `11840f9904` | 43435593630 / 2 | `Reuse exact-source verification` |
Spot workers reclaimed at 13:49:56Z. The run was then cancelled by
`github-actions[bot]` (the image watcher). Dotta re-ran attempt 3 by
hand |
| `8af889f612` | 43434587048 / 2 | Release superseded while pending;
readiness failed in the same reclaim | Spot reclaim |
| `9fbfd93c9b` | 43423674511 / 2 | `Reuse exact-source verification` |
Spot reclaim |
| `6917d0568d` | 43439942063 / 1 | `Reuse exact-source verification` |
Flaky test `decision-models.test.ts` (concurrent budget admission) |
| `18fb6ee571` | 43422042261 / 2 | `Reuse exact-source verification` |
Flaky test `fresh-continuation-fallback.test.ts` (truncate) |
| `5ede64f247` | 43421480302 / 1 | `Reuse exact-source verification` |
Both flaky tests |

Over the last 200 master pushes, release runs ended 14 success, 33
failure, and 150 cancelled (superseded while pending, which is
expected).

**Expected behavior**
A lost runner is not a source failure. The gate retries the lost jobs
and publishes the newest commit that passes.

**Steps to reproduce**
See `scripts/cloud-readiness-runner-loss.test.mjs`, "busy-master
simulation". It replays the 13:49:56Z reclaim through both waiters.

## What Changed

- `scripts/cloud-readiness-runner-loss.mjs` (new): classifies a failed
readiness attempt with the existing PR classifier
(`.github/scripts/runner-loss-rerun.mjs`). It is a runner loss only when
every causing job (`failure`, `timed_out`, `startup_failure`) lost its
runner. Cancelled siblings are fallout. A run that is still going is
cancelled first, then `rerun-failed-jobs` runs on the next poll. The
ceiling is readiness attempt 4. Writes go only to `cancel` or
`rerun-failed-jobs` of a canonical run, after an identity check.
- `scripts/cloud-source-verification.mjs` (canary gate in
`release.yml`): on `SourceVerificationFailed`, ask the retry first. The
source wait goes from 60 to 90 minutes.
- `scripts/watch-release-source.mjs` (master image watcher in
`docker.yml`): on a runner loss, keep the image attempt alive and wait
for the retried proof. Do not cancel it.
- `.github/workflows/release.yml`: `verify_canary` timeout goes from 70
to 100 minutes.
- `.github/workflows/pr-trusted.yml`: run the new test file.
- `doc/RELEASING.md`: new "Busy master and lost runners" section.

## Verification

- `node --test scripts/cloud-readiness-runner-loss.test.mjs
scripts/cloud-source-verification.test.mjs
.github/scripts/tests/*.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs`: all pass.
`release-verify-workflow.test.mjs` passes 51 of 51 but does not exit
locally. This also happens on `origin/master` without this change.
- Red/green: with the two waiter files reverted, 3 of the 6 simulation
tests fail (the canary gate, two reclaims in a row, and the image
watcher). With the change, all 20 pass.
- Dry run against real runs (read-only, writes captured): 43435593630/2,
43434587048/2, and 43423674511/2 give `rerun-failed-jobs`.
43439942063/1, 43421480302/1, and 43422042261/2 (real test failures)
give `failed` with no write.
- `pnpm verify` exits 2 because a workflow file changed. Required CI
supplies broad coverage.

## Risks

- A real failure classified as a runner loss would be retried. The
classifier needs the runner's own shutdown or lost-communication
`##[error]` line in the job log. It is the same classifier PR CI uses,
and unreadable evidence fails closed.
- On a reclaim, the waiter cancels the readiness run to retry sooner.
Jobs that were still healthy run again. This costs runner time, not
correctness.
- Flaky tests still fail a canary. This change does not retry them. They
need their own fixes.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 09:20:30 -06:00
Nicky LeachandClaude Fable 5.2 503a736195 feat(ci): read a registry build cache in the Runner image build (#21310)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - `.github/workflows/runner-image.yml` builds the Runner alone and
publishes it from `master` as `ghcr.io/paperclipai/paperclip-runner`,
one time for each set of inputs
> - The build used no cache, so every run compiled the daemon again. The
first publication took 25m46s from push to tag
> - The toolchain and the daemon change only with the Rust sources. Most
commits that change the Runner's inputs do not touch them
> - This pull request lets the build legs read a registry build cache
for those stages. Only a new job can write the cache, and only for a
`master` push
> - A weekly run builds with no cache, so the cache cannot hide a broken
build for long
> - The benefit is about 3 minutes less on each build leg when the Rust
sources did not change, with no change to the published image

## Linked Issues or Issue Description

Refs #21230 (publication of the Runner image), #21229 and #21204 (the
build this change makes faster). No issue exists for this change.

**What existing behavior does this improve?**

The Runner image build in `.github/workflows/runner-image.yml`. It
builds the same image in less time.

**Current behavior**

Each build leg starts from an empty builder. It installs the Rust
toolchain and compiles the daemon in every run. In the first `master`
publication (run 43365670428) the amd64 leg spent 7m22s in the build
step. The toolchain and the cargo build took 3m26s of that.

**Proposed behavior**

The build legs read
`ghcr.io/paperclipai/paperclip-runner-buildcache:<arch>` for the stages
up to the compiled daemon. A new `cache` job writes that cache after a
publication. A scheduled run builds with no cache one time each week.

**Reason and benefit**

The stages before the last one read the Rust sources, the protocol files
and the toolchain pin. In the last 30 days, 438 `master` commits changed
an input of the image, and at most 181 of them changed a file those
stages read. For the other commits, and for each pull request that does
not touch those files, the build step is about 3 minutes shorter on each
leg.

**Breaking changes**

None. The `Dockerfile` and each file in the image identity are
unchanged. The published digest for a set of inputs does not change.

This is part 4 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path
3. #21308: reading a retained provider pack from its Runner image
4. #21310: a registry build cache for the Runner image build (this pull
request)

This pull request uses no code from parts 1 to 3. It is stacked on them
because all four parts edit `doc/runner-image-contract.md`.

## What Changed

- The build legs pass `--cache-from
type=registry,ref=ghcr.io/paperclipai/paperclip-runner-buildcache:<arch>`
to the first build command. They do not log in. Their permissions stay
`contents: read` and `packages: read`.
- A new `cache` job writes the cache with `--cache-to
type=registry,...,mode=max` and `--output type=cacheonly`. It runs only
for a `master` push in the canonical repository, for inputs that had no
image. Its permissions are `contents: read` and `packages: write`. It
has no `id-token` and no `attestations` permission.
- The `cache` job needs the `publish` job, and no job needs the `cache`
job. A cache write cannot delay or fail a publication.
- The `cache` job builds the target `runner-artifacts-runnerd` and no
later stage. The last stage takes the commit as a build argument, so no
other commit can reuse it. The install and its dependency lifecycle
scripts are in that stage, so the job with the write token does not run
them.
- The `cache` job has one concurrency group for each architecture, a job
time limit and a step time limit.
- A `schedule` trigger starts a weekly build. It passes `--no-cache` to
the first command and reads no cache. It runs all checks and publishes
nothing. A fork's schedule runs nothing.
- `server/src/__tests__/runner-image-workflow.test.ts` pins the new
security properties: which jobs hold `packages: write`, that the build
legs have no credential and cannot write the cache, the conditions of
the `cache` job, that the `publish` job does not depend on it, and that
a scheduled run cannot publish or write the cache.
- `doc/runner-image-contract.md` has a "Build cache" section.

## Verification

Tests, on macOS (Node 26.12.0) and in a `node:24.22.3-bookworm`
container (Linux arm64, Node 24.22.3):

- `pnpm exec vitest run src/__tests__/runner-image-workflow.test.ts
src/__tests__/runner-image-identity.test.ts` in `server/`: 55 of 55
tests pass on both.
- `node --test .github/scripts/tests/*.test.mjs`: 807 of 807 tests pass
on both. The container needs `jq`.
- `pnpm verify` exits 2 for this change, because a workflow file is a
global CI input. Required CI supplies the broad coverage.
- Each new assertion fails when the workflow breaks its rule. I checked
seven such edits, for example: the `publish` job needs the `cache` job,
the `cache` job gets `id-token: write`, a build leg gets `--cache-to`.

Cache mechanics, on linux/arm64 with a local registry, BuildKit v0.34.2
(the version the hosted runners use), and an empty builder for each
build:

| Build of both variants | Time | Result |
| --- | --- | --- |
| No cache, cache ref does not exist | 260 s | An importer error in the
log, exit 0 |
| One changed TypeScript file, with the cache | 149 s | Toolchain and
cargo steps cached |
| Unchanged inputs and a new commit, with the cache | 129 s | Toolchain
and cargo steps cached |
| Each stage in the cache (a whole-build cache, which this change does
not write) | 16 s | Both OCI archives complete |
| `--no-cache` on the first command | 272 s | No cached step in the
first command. The second command reuses all of it |

- The images built from the cache pass `runner-image-layout.mjs check`
and `runner-image-oci.mjs check-layers`. The daemon has the same SHA-256
as the daemon of a cold build.
- A cache ref that does not exist, a host that refuses the connection, a
name that does not resolve, and the real
`ghcr.io/paperclipai/paperclip-runner-buildcache:arm64` (which does not
exist yet) each give an importer error in the log and a complete build
with exit 0.
- The `cache` job command on an empty builder with the cache present
takes 6 s. It downloads no layer.

Not tested before merge:

- The `cache` job. It runs only on `master`.
- The scheduled run. It starts only from the default branch.
- The cache package. It does not exist. I did not test a private
package, which answers an anonymous read with an authorization error. I
expect the same importer error and a cold build.
- A fully cached build on the hosted runners. The local test used the
same BuildKit version and a different buildx version (0.37.0, and the
runners have 0.38.1).

## Risks

- **A second job can write packages.** Before this change, only the
`publish` job held `packages: write`, and it runs no build. Now the
`cache` job holds it too, and it runs BuildKit on the repository's
`Dockerfile`. GitHub cannot limit that token to one package, so it can
also write to `ghcr.io/paperclipai/paperclip-runner`. The limits are:
the job runs only for a `master` push in the canonical repository, the
token goes only to the registry login, the build steps run inside
BuildKit without it, the job does not run the install, and the job
cannot sign. An image that it wrote has no attestation.
- **Part of a published image can come from an earlier run.** The daemon
in a published image can come from a step that an earlier `master` run
compiled and cached. The attestation names the run that published, not
the run that compiled. BuildKit reuses a step only for the same
instruction and the same file content, but a party that can write the
cache package can replace what a later build reuses. Write access to the
cache package must stay as narrow as write access to the image package.
- **A cached download is not repeated.** The toolchain installer, the
toolchain and the crates are not downloaded again until the inputs of
their step change. The weekly cold build finds a download that is gone.
It does not compare its output with the cache.
- **A registry failure after the cache import fails the leg.** A cache
that cannot be read at the start is not a failure. A layer download that
fails later is a failure. Run the job again.
- **The cache package starts private.** GitHub creates it on the first
write. A maintainer must make it public one time. Until then each build
runs cold and nothing fails.
- **The first cache write waits for an input change.** The merge of this
pull request changes no input of the image, so its `master` run builds
nothing and writes no cache.
- **A cancelled cache job marks its run as cancelled.** When three cache
writes for one architecture wait at the same time, GitHub cancels the
middle one. The image of that run is published.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, through Claude Code.
Tool use and shell execution. It wrote the change, ran the local builds
and tests, and wrote this description.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 23:46:15 -08:00
Nicky LeachandClaude Fable 5.2 03f9b83e52 feat(ci): report the Runner image on master and build the app image's Runner path on pull requests (#21307)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work, and `master` now
publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner`
> - The application image will soon copy that Runner instead of
compiling its own. Part 1 of this stack adds the resolver that picks the
Runner for a build
> - Two gaps make that next change risky. The resolver has never run in
CI on `master`, so its lookup time and how often a commit has a
published Runner are unknown. And `docker.yml` has no pull-request
trigger, so a change to how the application image gets its Runner first
runs after it merged
> - This pull request adds a report-only job that runs the resolver on
every `master` push, and a pull-request check that builds the Runner
path of the application image
> - The benefit is real data from `master` before any build depends on
the resolver, and a failed check on the pull request instead of a failed
image build on `master`

## Linked Issues or Issue Description

Refs #21204, #21229 and #21230 (the Runner image). No issue exists for
this change.

**Problem or motivation**

The resolver decides whether a build copies the published Runner. On
`master`, a commit that changes the Runner's inputs is ahead of its
publication for some minutes, and nobody knows yet how often an
application build will meet that. The application image itself is built
only on `master` and on tags. The change that removes the Runner's
compile stages from that image will edit the `Dockerfile`, and today
nothing builds those stages on a pull request.

**Proposed solution**

Add one job to `docker.yml` that only reports, and one new workflow that
only checks. Neither changes what an image contains, and no existing job
reads either of them.

**Alternatives considered**

- Run the resolver inside the build legs. A failure there could fail the
image build. A job of its own, with `continue-on-error`, cannot.
- Give the lookup the job token. The Runner package is public, so an
anonymous lookup works and the job needs no credential.
- Build `production` in the pull-request check. It adds three `COPY`
instructions and a readability check to the two stages that this check
builds, at the cost of the tool layer and two more copies of several
gigabytes.
- Trigger the check on every Runner source change. `runner-image.yml`
already builds the Runner on those pull requests, and CI builds the
server and the UI.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 2 of 4 of a stack ([stack
#21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge
the parts in order:

1. #21306: the resolver and the no-rebuild mode
2. #21307: the report-only job in `docker.yml` and the pull-request
check of the application image's Runner path (this pull request)
3. #21308: reading a retained provider pack from its Runner image
4. #21310: a registry build cache for the Runner image build

## What Changed

- `.github/workflows/docker.yml`: new job `runner_image_report`.
  - It runs only for a `master` push in this repository.
- It runs `runner-image-resolve.mjs inputs`, then `resolve`, and the
result goes to the job summary with the line "Report only: no build read
this result."
- It has `continue-on-error: true`, declares no `outputs`, and no job
needs it. Its failure does not fail the run.
- It holds no credential. Its only permission is `contents: read`, the
checkout does not keep its token, and the lookup is anonymous. It does
not log in to a registry and writes to none.
- Time limits: 8 minutes for the job, 5 for the step that builds, 2 for
the lookup.
- It is the first job in the file. Existing tests cut the other jobs out
of the file by name, up to the next job or to the end of the file, so no
other place leaves those tests unchanged.
- `.github/workflows/docker-app-check.yml` (new): on a pull request,
builds the `build` and `runner-provider-pack` targets with
`PAPERCLIP_BUILD_COMMIT` set, for `linux/amd64` only.
- `permissions: {}` at the top, `contents: read` for the job, no
credential, no `pull_request_target`.
- It exports nothing (`--output type=cacheonly`) and pushes nothing. It
reads the public build cache that `docker.yml` writes and never writes
it.
- It refreshes the lockfile in the build context first, as `docker.yml`
does.
  - Time limits: 60 minutes for the job, 45 for the two builds.
  - Trigger paths, and why:

    | Path | Reason |
    | --- | --- |
    | `.github/workflows/docker-app-check.yml` | the check itself |
| `Dockerfile`, `.dockerignore` | the recipe, and the files it receives
|
| `server/scripts/prepare-runner-vendor.mjs` | the step of `build` that
reuses or rebuilds the Runner |
| `packages/paperclip-runner/scripts/build-inputs.mjs` | the build stamp
that step compares with the files the image build received |

- `server/src/__tests__/docker-runner-workflows.test.ts` (new, 90
lines): only security properties. The report job has no credential and
one read permission, nothing can read it, and its two steps have time
limits. The check has no credential, cannot publish, and exports
nothing. In both, every action is pinned by commit and no shell script
holds an expression.
- `doc/runner-image-contract.md` and `doc/DOCKER.md` name the job and
the check.

## Verification

macOS (Node 26.12.0):

- In `server/`: `docker-runner-workflows`, `runner-image-workflow`,
`docker-build-stamp`, `container-init-reaping`,
`cloud-image-bundled-plugins`, `docker-image-layer-reuse` and
`cloud-image-sentry` tests: 52 tests pass in 7 files.
- `tests/runner-e2e` workflow tests (`workflow-security`,
`private-trust-workflow-security`, `workflow-image-only`,
`paid-runner-routing`): 27 tests pass in 4 files.
- `node --test .github/scripts/tests/*.test.mjs
scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs
scripts/cloud-source-verification.test.mjs
scripts/standard-image-contract.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs`: 962 of 962 pass.
- Both workflow files parse as YAML.
- `docker buildx build --target runner-artifacts-context --cache-from
type=registry,ref=ghcr.io/paperclipai/paperclip:buildcache-amd64
--output type=cacheonly .` succeeds, so both flags are accepted and the
cache is readable.

Linux (`node:24.22.3-bookworm`, with `zstd` and `jq` added): the same
three sets pass: 52, 27 and 962 of 962.

The two commands the report job runs were run by hand, with no
credential, on clean exports of the tree. For the inputs `0e6662ad…d1ca`
the answer was `published`, with the two index digests that `master`
published. For a commit whose image was not yet published the answer was
`inline`, `absent`. Part 1 has the output.

With the job at the end of `docker.yml`, two existing tests failed. They
are the reason for its place: `docker-canary-promotion.test.mjs` reads
the last job to the end of the file, and
`release-verify-workflow.test.mjs` allows no BuildKit configuration
after `merge-and-push`.

Never run:

- Either workflow. Only GitHub can run them. This pull request starts
the check by its own path; the report job first runs on `master`.
- The two builds of the check (`--target build`, `--target
runner-provider-pack`) on a local host. Their time and disk use on a
hosted runner are not known.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The check builds the whole `build` stage, with the UI and server
builds. Its duration is not measured. It runs only when one of five
paths changes, and its first real run is on this pull request.
- `docker.yml` holds one concurrency slot for each ref until all its
jobs end. The report job normally ends long before the build legs. A
stalled job can hold the slot for at most 8 minutes when the legs end
sooner.
- `scripts/watch-release-source.mjs` warns about every job of the image
run that waited more than 60 seconds for a runner. The report job is one
more job that it can name.
- `scripts/merge-group-image.mjs` refuses to promote a merge-queue image
for a push that changes `docker.yml`. The push that lands this change is
such a push. The merge queue is not enabled.
- The lookup has 2 minutes. The registry client retries a failing
request for longer than that, so a registry outage ends the step by its
time limit, and the job then reports nothing. The run stays green.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:29:33 -08:00
DottaandClaude Fable 5.2 c4f639ed1b fix(ci): re-run cancelled runs that lost a runner, and let runner-loss evidence win (#21291)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - PR CI runs about 50 jobs per run on AWS spot instances. AWS can
reclaim a machine mid-job, and the job dies.
> - `ci-runner-loss-rerun.yml` re-runs such a run once, automatically,
when every failed job lost its runner.
> - In a 17-hour sample, 42 runs lost a runner and were still the PR's
current head. The automation rescued only 14.
> - Two gaps caused most of the misses. A spot reclaim on RunsOn v3.2.3
usually ends the job, and the run, as `cancelled`; the workflow and
script only look at `failure`. Also, the Runner support suite always
carries `gateway_indexing_incomplete` error annotations from a passing
test, so a lost runner there reads as a real failure.
> - On-demand fallback for the fleet was turned off on 2026-11-13, so
reclaims will happen more often. About 20% of runs are expected to lose
at least one job.
> - This pull request makes the re-run act on cancelled runs and jobs,
but only with positive runner-loss evidence, and lets that evidence win
over other error annotations.
> - The benefit is fewer manual "Re-run failed jobs" clicks: an
estimated 34 of 42 instead of 14 of 42 in the sample.

## Linked Issues or Issue Description

No public issue exists.

**What happened?**
Run
[43285064435](https://github.com/paperclipai/paperclip/actions/runs/43285064435)
lost the runner of `ci / General tests (server (8/12))`. The job
concluded `cancelled` (its log has `##[error]The runner has received a
shutdown signal`). The run concluded `cancelled`, so the re-run
workflow's `if:` skipped it.

Run
[43325855069](https://github.com/paperclipai/paperclip/actions/runs/43325855069)
lost the runners of `ci / Runner support suite` and `ci / General tests
(server (2/12))`. Both logs have the shutdown signal. The re-run bot
([decision
log](https://github.com/paperclipai/paperclip/actions/runs/43327873358))
said "1 of 1 failed jobs did not lose their runner", with evidence
`gateway_indexing_incomplete`. It ignored the cancelled job. It read the
annotation as a real failure.

**Expected behavior**
Both runs get one automatic re-run of their failed jobs.

**Steps to reproduce**
The new tests replay trimmed snapshots of both runs
(`.github/scripts/tests/fixtures/runner-loss-rerun/run-43285064435.json`
and `run-43325855069.json`) through the script.

In a 17-hour sample on 2026-11-12, 42 runs lost a runner and were still
the PR's current head. Only 14 were re-run automatically. 16 stayed red
because the run concluded `cancelled`. 4 stayed red because an error
annotation in the lost job was read as a test failure.

**Additional context**
The `gateway_indexing_incomplete` annotation does not come from the
shutdown. `scheduled-gateway-indexing.test.ts` prints `::error` lines
while it passes, 84 seconds before the shutdown signal in this run.
Attempt 2 of the same run passed with the same two failure annotations.
So, before this change, a lost runner in the Runner support suite could
never be re-run.

## What Changed

- `ci-runner-loss-rerun.yml`: the job also runs when the PR run
concluded `cancelled`. The trigger type, `permissions`, the repository,
path, event and attempt-1 conditions do not change.
- `runner-loss-rerun.mjs`, `eligibleRun`: accepts `completed/cancelled`
as well as `completed/failure`.
- `runner-loss-rerun.mjs`, `decide()`: classifies jobs that concluded
`cancelled` as well as `failure`. It still skips the `ci / verify` and
`ci / e2e` aggregates, and a failed or cancelled routing gate still
blocks the re-run.
- `decide()` now runs `staleReason` (open PR, current head, no newer
run) before it reads any annotation or log, and again just before the
re-run, as on master. A run cancelled by a new push has about 40
cancelled jobs; it now costs two API listings and no log reads. The
final check closes the window where a push lands during the log reads: a
re-run of the old run would join the PR's `cancel-in-progress`
concurrency group (`pr-<number>` in `pr-trusted.yml`) and cancel the new
commit's CI.
- Following CONTRIBUTING.md ("Do not submit a PR that includes code
comments"), this PR adds no new code comments. The reasons for each rule
are in this description and in `doc/DEVELOPING.md`.
- `classifyJob`: runner-loss evidence wins. A lost-communication
annotation means lost. A cancellation annotation plus a shutdown signal
in the log means lost, even if the job also has other error annotations.
A cancelled job without this evidence (for example, a person cancelled
the run) counts as a real failure and blocks the re-run.
- Fixtures: two new trimmed snapshots from the example runs (jobs with
`id`, `name`, `status`, `conclusion`; failure annotations; `##[error]`
log lines only). The two older fixtures get the real annotations and log
error lines of their cancelled jobs. Those jobs were spot-reclaimed too
(each log has the shutdown signal), but the old code ignored them.
- `doc/DEVELOPING.md`: the runner-loss paragraph describes the new
rules.

## Verification

- Red first. Against the unchanged script, the new tests fail for the
reasons above: 43285064435 gives `run is completed/cancelled, not
completed/failure`; 43325855069 gives `1 of 1 failed jobs did not lose
their runner` with evidence `gateway_indexing_incomplete`, the same
decision the bot logged.
- Green: `node --test .github/scripts/tests/runner-loss-rerun.test.mjs`:
25 of 25 pass.
- Push during log reads: the PR head moves after the first log read.
With the final freshness check removed, this test fails (the script
re-runs); with it, there is no re-run and the reason is `no open pull
request still has this run's head commit`.
- Guard cases, all with no re-run and no POST:
- Superseded run: 52 jobs cancelled, the PR head moved. The test asserts
zero annotation and log reads.
- Person cancelled the run: cancelled jobs with only "The operation was
canceled." in the annotation and log, and some never-started jobs with
no annotation.
- Mixed: one cancelled lost job and one real test failure (`1 of 2
failed jobs did not lose their runner`).
- Once only: a cancelled run on attempt 2 is refused after one API read.
The workflow condition test still requires `run_attempt == 1`.
  - Cancelled routing gate: left for a person.
- Policy job command: `node --test '.github/scripts/tests/*.test.mjs'`:
806 of 806 pass.
- `pnpm verify`: exit 2 ("Global CI input changed:
.github/scripts/runner-loss-rerun.mjs"). It selects no local Vitest
files and asks for required CI coverage. I did not run `pnpm verify
--full`; required CI on this PR supplies that coverage.
- This workflow runs on `workflow_run` from the default branch. This
PR's own CI cannot exercise the new behavior. The fixture tests are the
evidence. I did not dry-run the script against live runs.

## Risks

- A test that really fails, and then a spot reclaim of the same job a
minute later, now gets one wasted re-run. Attempt 2 then fails for real.
It cannot loop: the workflow `if:` and `eligibleRun` both accept attempt
1 only.
- A run a person cancels on the current head now starts the re-run job.
The job reads the annotations, and the log of each job with a
cancellation annotation, and then does not re-run. This costs some API
calls on a GitHub-hosted runner.
- A concurrency cancel by a label event that does not request CI (the
newer run is `skipped`) passes the head check. Its cancelled jobs have
no runner-loss evidence, so it is not re-run.
- Trust: no change to `permissions`, the trigger, the accepted
repository, event, workflow path, attempt, or the routing-gate rule. A
cancelled run reuses its attempt-1 gate output, the same as a failed run
does today.
- Estimated effect, from a 17-hour sample: 14 of 42 rescued before; 16
cancelled runs and 4 runs with error-annotation noise become eligible
(about 34 of 42). The other 8 stay skipped on purpose (a real test
failure in the same run, a person re-ran first, or the PR closed).

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), Claude Code agent. Model ID claude-opus-5-5, with
tool use (shell, GitHub CLI) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:31:50 -06:00
Nicky LeachandClaude Fable 5.2 25e9a9c2e9 feat(ci): publish both variants of the Runner image from master (#21230)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - Parts 2 and 3 of this stack build the Runner alone as an image with
two variants, and check it on pull requests. Nothing publishes it
> - Downstream sandbox and VM images can copy the Runner only from a
registry, and they must be able to verify where it came from
> - A published image must be exactly the bytes that were checked, and
its name must never point at something else later
> - This pull request publishes both variants from `master` as
`ghcr.io/paperclipai/paperclip-runner`, one time for each distinct set
of inputs, and attests them
> - Nothing consumes the image in this pull request. The application
image build does not change
> - The benefit is one verified, attested Runner that later changes can
copy by digest

## Linked Issues or Issue Description

Refs #21204 (part 2 of this stack) and #21096 (the merge-queue image
build). No issue exists for this change.

**Problem or motivation**

The Runner image exists only inside a pull-request check. A consumer
cannot pull it. There is no rule for which commit publishes an image, no
protection for its tags, and no way to verify a copied Runner.

**Proposed solution**

Publish from a `master` push in this repository, and from nothing else.
Build only when no image has the digest of the commit's inputs. Let the
build legs hold no write access. Push the bytes that the legs checked by
digest, attest each multi-platform index, and write the tags last. Write
the full tag after the slim tag, so that the full tag means "both
exist".

**Alternatives considered**

- Publish from `docker.yml`. That workflow holds one concurrency slot
for each ref, and the Runner image needs its own signer identity, so
that its attestation cannot stand in for the application image's.
- Resolve the lockfile in the workflow before a `master` build, as
`docker.yml` does. Then the published Runner depends on the state of the
npm registry at run time, and the digest cannot be computed again from a
commit.
- Hand each variant to the publish job as its own OCI archive. The two
archives hold the two shared layers two times: 1.27 GB for each
architecture in place of 0.85 GB.
- Write the full tag first. Then a consumer can see a full image whose
slim image does not exist yet.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 4 of 4. Each part is on top of the one before it:

1. #21228 (merged): an optional field in the provider pack manifest
2. #21204: build and check the Runner image, with no publication
3. #21229: layers by change rate, and the slim variant
4. #21230: publish both variants from `master` (this pull request)

## What Changed

- `.github/workflows/runner-image.yml`: adds the trigger `push` to
`master` and the job `publish`.
- `identity` checks the committed lockfile against the Runner's
manifests, computes the digest in the build context, and asks the
registry whether these inputs are published.
- `build` is the job of part 3. It also records what it built, on every
event. On a canonical `master` push it uploads that. It cannot write to
the registry on any event. The only token in the job can read packages,
and only the lookup step receives it.
- `publish` runs only for a `master` push in this repository. It builds
nothing and runs no dependency code.
- Lockfile rule. A `master` push builds from the committed
`pnpm-lock.yaml` and resolves nothing, so the digest is a function of
the commit. When the committed lockfile is behind the manifests that the
Runner build reads, the run writes a notice, publishes nothing, and
stays green. The commit that merges the lockfile refresh publishes the
image. Pull requests and manual runs resolve the lockfile first, always
build, and never publish.
- Hand-off. Each leg joins its two OCI archives into one OCI layout that
names the images `slim` and `full`. The layout stores the two shared
layers one time. `record` then reads the layout back, each image by its
name, and checks every blob against its digest. The leg's record binds
the layout to both image digests, the architecture, the inputs digest,
and the commit. Those digests are the digests of the archives that part
3 unpacks and checks, so the tree that is checked and the image that is
pushed are the same bytes. The publish job accepts only records for its
own inputs and commit, and only a layout that still has the recorded
bytes.
- `record` runs on pull requests too. Only the upload is for a `master`
push. A pull request therefore runs the hand-off on real build output.
- Publication order. The publish job pushes the four images by digest
(`skopeo copy --preserve-digests`). It makes a candidate index for the
slim variant, then one for the full variant that names the slim index in
the annotation `io.paperclip.runner.slim-digest`. It verifies that each
index names exactly the recorded images, attests both, and then writes
`inputs-<digest>-slim` and, last, `inputs-<digest>`.
- The full tag is the completion marker. Every "is it published" lookup
reads the full tag. `publication-state` reports one of:

  | State | Registry | What a run does |
  | --- | --- | --- |
  | `absent` | Neither tag | Builds and publishes |
| `incomplete` | The slim tag alone | Builds and publishes. It replaces
the slim tag with its own image, then writes the full tag |
| `published` | The full tag, and the slim tag at the digest that the
full index names | Nothing |
| broken | The full tag, and a slim tag that is missing or points at
another image | Fails, with the steps for an operator |
| `unknown` | No clear answer | Builds. The publish job fails before it
writes a final tag |

- Recovery rule. A run that stops between the two tags leaves the state
`incomplete`. It does not block those inputs. The next run never keeps
the earlier slim image: it writes its own slim tag immediately before
its full tag, verifies both against its own index digests, and then
reads the pair back as every later run will see it. The broken state
cannot be repaired by a run, because no run moves a full tag. An
operator must delete the two package versions and re-run the workflow
run of a `master` push with those inputs.
- Pairing rule for consumers, in the contract: resolve the full tag
first, take the slim digest from the annotation of the full index, and
pull the slim image by that digest.
- `verify-index` accepts `linux/arm64` with the variant `v8`, which some
BuildKit versions record, and no other variant.
- `scripts/runner-image-publish.mjs` (533 lines): the commands
`lock-state`, `record`, `verify-legs`, `publication-state` and
`verify-index`. It reads archives with the reader of
`scripts/runner-image-oci.mjs` (607 lines here), which can now read one
named image of a layout. `lock-state` refuses a checkout, because pnpm
can rewrite the lockfile of the tree that it checks. The workflow gives
it an export that no build reads.
- The workflow test pins the build arguments of the image build. Build
arguments are not part of the image identity.
- `doc/runner-image-contract.md` (197 lines): the section "Publication":
the lockfile rule, the order, the pairing rule, the recovery rule, the
create-only tags, the internal `candidate-` tags, the command that
verifies provenance, and the step to make the package public.
`doc/DOCKER.md` says where the image is published.

## Verification

Tests, and the build jobs of the workflow on GitHub. Nothing in this
pull request ran against a registry, and no publish job has run.

- Linux (the pinned `node:24.22.3-bookworm` base with GNU tar 1.34 and
zstd): `runner-image-publish.test.ts` (75),
`runner-image-workflow.test.ts` (12), the three other Runner image test
files, the provider pack archive tests and the existing Dockerfile
tests: 372 tests pass in 10 files. The same files pass on macOS.
- The tests of the leg record failed on Linux in the first runs of this
pull request: their fixtures named the archive members `./index.json`,
which bsdtar finds and GNU tar does not. The fixtures now name the
members as buildx does, and the publish script uses the one reader of
part 3.
- Leg records: both variants are accepted for this run's inputs and
commit; other inputs, another commit, a changed layout, a record without
a variant, an archive that lacks a blob, a full image that does not
start with the slim image's layers, and a build that is not a
publication build are refused.
- Publication state: all five states, and the half-finished publication
end to end. A slim tag alone lets the next run publish. A full image
that is left with another run's slim tag is reported as broken.
- Index: exactly the recorded images of one variant; the full index must
name its slim index, and the slim index must name none; `v8` is accepted
on arm64 only.
- Registry lookups: a missing tag only when the registry says so; every
refused, failed, or unreachable request is `unknown`.
- Workflow: job permissions; the publish job and the upload are behind
the canonical `master` push; the build steps receive no credential; the
full tag is written last, after a strict lookup and both attestations;
one export and a time limit for each build command; the two build
arguments; actions pinned by commit; no expression inside a shell
script.
- `record` on the two OCI archives of a real `linux/arm64` build from
part 3: the joined layout is 850,149,376 bytes, and both images read
back by name with every blob under its digest. The two archives together
are 1,273,646,592 bytes. The image digests in the record are the digests
that `unpack` reports for the trees that were checked.
- `tests/runner-e2e` workflow tests: 39 pass. `node --test
.github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image
contract scripts: 33 pass. `docker buildx build --check` reports
nothing. The image identity is the same as in part 3
(`a544fc9dde780963419ede0f34252f3a57de43a760d41d3eb8fcd4f577504f6a`),
because this pull request changes no input of the image.

GitHub, at this head:

- The `Runner image` workflow passes with these changes: `Build and
check (amd64)` in 18m30s and `Build and check (arm64)` in 7m48s. Each
leg builds both variants, checks them, and writes the leg record.
`Publish and attest` is skipped, as it must be on a pull request.

Never run:

- The publish job. It runs only for a `master` push.
- The publish path: the registry lookups with a workflow token, `skopeo
copy` from `oci-archive:<layout>:<name>` by digest, `docker buildx
imagetools create` with index annotations, and the tag writes.
- The attestations, and the `gh attestation verify` command in the
document.
- `actionlint`. It is not installed on the local host.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- **Manual step.** GitHub creates the `paperclip-runner` package private
on the first publication. A maintainer must make it public one time.
Until then, consumers cannot pull the image without credentials.
- The first `master` runs of the workflow are the first test of the
publish path. A failure before the tags publishes nothing. A failure
between the two tags leaves the `incomplete` state, and the next run
with those inputs repairs it.
- A slim tag can move while its publication is incomplete. A consumer
must resolve the full tag first and take the slim digest from the
annotation of the full index. It must never resolve
`inputs-<digest>-slim` by tag on its own. The contract says so.
- `candidate-` tags are written before the attestations and can name an
index that is not attested. They are internal.
- The broken state needs an operator, and it fails every `master` run
with those inputs until then. It can arise only when someone changes or
deletes a tag outside the workflow.
- **Known residual.** Two `master` runs that both start before a new
digest is published can each build the image. Each build leg and the
publish job look again before they work, and the publish job runs one at
a time for each digest. The cost is runner time, not correctness.
- A commit whose lockfile is behind the Runner's manifests has no
published Runner image until the lockfile refresh merges. A lockfile
that the refresh cannot fix leaves every `master` run green with only a
notice, for as long as it lasts.
- The tags are create-only because the workflow never moves a tag of a
complete publication. The registry does not enforce that. Consumers must
pin the digest and verify the attestation.
- `payload.runnerSourceRevision` in a pack from this image is the commit
that published the image. Code that reads it as the application's commit
must change before it consumes the image.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 20:27:36 -08:00
Nicky LeachandClaude Fable 5.2 9b613562b7 feat(docker): split the Runner image into stable layers and add a slim variant (#21229)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - Part 2 of this stack builds the Runner alone as the image
`runner-artifacts`, in one layer of about 850 MB
> - Most of those bytes are provider binaries and Node. They change only
with a provider version or the Node pin, but one layer means that every
Runner source change makes all 850 MB new
> - About half of the image is the pack's transport archives. A consumer
that copies the pack as files does not need them
> - This pull request splits the image into layers by how often their
content changes, and adds a slim variant without the transport archives
> - The benefit is that a new image shares about 93% of its bytes with
the one before it, and a consumer that needs no transport archives pulls
half as much

## Linked Issues or Issue Description

Refs #21204 (part 2 of this stack) and #21167 (provider pack layers). No
issue exists for this change.

**Problem or motivation**

The image from part 2 is one layer. A registry and a builder store and
move layers, not files. A change to one Runner source file therefore
makes a new 850 MB layer, although 790 MB of its content is the same as
before. The image also holds the provider pack two times: as files, and
as transport archives of those files. Some consumers need only the
files.

**Proposed solution**

Write the image as four layers: the pack's large dependency files,
everything else, the transport archives of those large files, and the
other transport content. Make the two large layers byte-identical from
build to build. Add the target `runner-artifacts-slim`, which is the
first two layers. The full image is the slim image plus the two
transport layers, as the same layer blobs. Check all of this on the
built images.

**Alternatives considered**

- Two unrelated images. Then the registry stores the shared content two
times unless the layers happen to match, and nothing ties the two
together.
- Split by path only, with all files of 1 MiB or more in the first
layer. Six of those files are the Runner's own bundles under `dist`.
They change with the Runner's source, so the first layer would change
with it too.
- Leave the transport archives as they were written. Their bytes then
depended on timing (see "What Changed"), and the archive layer never
repeated.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 3 of 4. Each part is on top of the one before it:

1. #21228 (merged): an optional field in the provider pack manifest
2. #21204: build and check the Runner image, with no publication
3. #21229: layers by change rate, and the slim variant (this pull
request)
4. #21230: publish both variants from `master`

## What Changed

- `Dockerfile`: two final stages. `runner-artifacts-slim` is two `COPY
--link` layers. `runner-artifacts` starts from it and adds two more. The
build stage sorts its output into the four layers. Every path is where
it was in part 2: consumers copy by path and see no difference.

  | Layer | Content | Changes with |
  | --- | --- | --- |
| 1 | The pack's dependency files of 1 MiB or more: provider binaries
and Node | A provider version or the Node pin |
| 2 | Everything else, with `runner-image.json` | Every Runner source
change |
| 3 (full only) | The transport archive of each file in layer 1 | The
same as layer 1 |
| 4 (full only) | The other transport archives, the overlay archive and
the receipt | The same as layer 2 |

- Stable bytes. The files of layers 1 and 3 get timestamp 0, and the
large dependency files get it before their transport archives are
written. A file under the pack's `dist` is the Runner's own output, so
it and its archive are in layers 2 and 4.
- `scripts/create-provider-pack-archive.mjs`, two separate commits.
**This script also writes the application image's archives.**
- The same file now gives the same archive every time. The script piped
tar into Node's zstd compressor. When the pipe ended while the
compressor still had its last chunk queued, Node finished the zstd
stream two times, and the archive got an empty frame of nine bytes at
its end. That depended on timing. In the pinned Linux base, 37 of 60
runs of the old code had the frame. The compressor is now ended only
after it has taken every chunk.
- Archive N holds file N of the receipt, and the receipt listed the
files in path order. `dist` sorts before `node_modules`, so one more
large file in the Runner's own output gave every dependency file another
archive name, and layer 3 became a new blob with the same contents. The
receipt now lists the files outside `dist` first. The readers
(`provider-pack-layers.ts`, `retained-provider-pack-source.ts`) pair
file N with `N.tar.zst` and do not use the path order, so they need no
change. For the application image, the archive numbers change one time,
and the bytes of each archive do not change.
- `runner-image.json` is the same file in both variants. Its new field
`fullImageOnly` names the two paths that only the full image has.
`paths` did not change.
- `scripts/runner-image-layout.mjs check` takes `--variant slim|full`.
It is required and has no default. A slim tree that holds a transport
path fails. A full tree without them fails.
- `scripts/runner-image-oci.mjs` (new, 585 lines). It is not an input of
the image, because the image build does not run it.
- `unpack <oci archive> --out <tree>` writes the tree of an image from
the layers of its archive, in order. It checks every blob against its
digest. It refuses a whiteout, an absolute path, a path with `..`, a
path that is written two times, an entry behind a link, and every entry
that is not a file, a directory or a link.
- `check-layers --slim <oci archive> --full <oci archive>` checks the
table above on the built images. It reads the receipt to decide which
archive belongs in which layer. It fails when the full image does not
start with the slim image's layer blobs in the same order, when a path
that is not a directory is in two layers, and when a shared layer holds
transport content under any name or directory.
- The script reads the archive and its layers itself. It does not use
the system's `tar`, so every platform reads them the same way. It finds
a member only under the name that buildx gives it.
- `.github/workflows/runner-image.yml`: each leg builds each variant one
time, as an OCI archive and nothing else. It unpacks both archives, runs
`check` on each tree with its variant, runs `check-layers`, and runs the
smoke matrix on the slim tree. The tree that is checked is therefore the
bytes of the image, not a second export of the same build. Every step
that builds has a time limit. The workflow still has no registry access.
- Why one export: in the first runs of this pull request, the slim build
stalled on all four legs. Each build command had two exports, a tree and
an archive. The slim command, with every stage from the cache, wrote its
archive and then made no progress on the tree for 50 minutes. A test now
holds the rule of one export and a time limit for each build command.
- The smoke matrix runs on the slim tree. The daemon is in a layer that
both variants share, so one run covers both.
- `server/src/__tests__/runner-image-identity.test.ts`: the test that
holds the rules for the Runner stages now finds a stage wherever
BuildKit starts one. It fails for `FROM x as name`, a flag on `FROM`, a
lowercase or indented `COPY`, an `ADD` or `ONBUILD`, a `COPY` with
another flag, and a `COPY` that continues on the next line. Nine samples
show each case.
- Two comments are corrected. The comment on
`PAPERCLIP_RUNNER_INPUTS_DIGEST` in `build-provider-pack.mjs` said that
no check reads the field. The `Dockerfile` comment on the publication
argument named a workflow "that builds the image for publication".
- `doc/runner-image-contract.md` (159 lines): the two variants, the
section "Layers", how the checks get their tree, and the new checks.
`doc/DOCKER.md` names the slim target.

Script sizes: `runner-image-layout.mjs` 969 lines,
`runner-image-oci.mjs` 585, `runner-image-identity.mjs` 166,
`create-provider-pack-archive.mjs` 114.

## Verification

Image identity at this commit:
`a544fc9dde780963419ede0f34252f3a57de43a760d41d3eb8fcd4f577504f6a`.

Linux (the pinned `node:24.22.3-bookworm` base with GNU tar 1.34 and
zstd, the worktree mounted read-only):

- `runner-image-identity.test.ts` (41), `runner-image-layout.test.ts`
(100), `runner-image-oci.test.ts` (50), `runner-image-workflow.test.ts`
(8), `provider-pack-archive.test.ts` (47, none skipped),
`retained-provider-pack-source.test.ts` (19),
`remote-provider-pack-verification.test.ts`,
`docker-image-layer-reuse.test.ts`, `docker-build-stamp.test.ts`: 293
tests pass in 9 files.
- The same files pass on macOS. The tests of `check-layers` failed on
Linux in the first runs of this pull request: their fixtures named the
archive members `./index.json`, which bsdtar finds and GNU tar does not.
The fixtures now name the members as buildx does.

`linux/arm64`, on a local Docker host, with the new export path:

- Each variant builds as an OCI archive only: the full target in 156
seconds without a cache, the slim target in 5 seconds from the cache.
- `unpack` writes the full image (10,519 entries that are not
directories) in 4 seconds, and the slim image in 4 seconds.
- `check <tree> --variant <variant> --expect-inputs-digest <digest>
--expect-source-revision <sha> --expect-architecture arm64` passes on
each unpacked tree. `runner-image.json` is the same file in both.
- `check-layers --slim <slim> --full <full>` passes.
- The smoke matrix passes on all five base images, from the unpacked
slim tree.

| Layer | Digest | Bytes | In |
| --- | --- | --- | --- |
| 1 | `sha256:3f21b15b3a99…70ac1c` | 379,609,589 | slim and full |
| 2 | `sha256:d5567acc977f…b4f12a` | 43,906,956 | slim and full |
| 3 | `sha256:6a340088ca2c…6ba5b5` | 410,938,259 | full |
| 4 | `sha256:789e97f5db88…3eab44` | 15,656,249 | full |

- The slim image is 423,516,545 bytes in two layers. The full image is
850,111,053 bytes in four.

Stability, with three builds that share no cache for the build step:

| Build | Source revision | Runner build output | Layer 1 | Layer 2 |
Layer 3 | Layer 4 |
| --- | --- | --- | --- | --- | --- | --- |
| A | `aaaa…` | this commit: 6 files of 1 MiB or more under `dist` |
`3f21b15b…` | `d5567acc…` | `6a340088…` | `789e97f5…` |
| B | `bbbb…` | this commit | `3f21b15b…` | `a2f8a4fe…` | `6a340088…` |
`4e6c492d…` |
| D | `dddd…` | one new source file that adds two large files under
`dist` | `3f21b15b…` | `33b0bdaf…` | `6a340088…` | `0ea322bf…` |

- Layers 1 and 3 are byte-identical in all three: 790,547,848 of
850,111,053 bytes, 93%. Build D has two more large files under `dist`,
eight in place of six, and another image identity. Its dependency files
keep the archive numbers 0 to 20. With the earlier numbering, those
numbers moved by two, and layer 3 was a new blob with the same contents.
- Layer 1 has the same digest as before this round. Layer 3 has a new
digest one time, because the archives have new numbers.

Other checks:

- `tests/runner-e2e` workflow tests: 39 pass. `node --test
.github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image
contract scripts: 33 pass. `check-docker-deps-stage.mjs`,
`check-node-version-policy.mjs` and `docker buildx build --check` report
nothing.

GitHub, at this head:

- The `Runner image` workflow passes on both architectures with the new
export path: `Build and check (amd64)` in 14m41s and `Build and check
(arm64)` in 8m41s. Each leg builds both variants, checks both layouts
and the layer rule, and runs the smoke matrix. The first runs of this
pull request stalled in the slim export before any check.

Never run:

- `native-session-executor.test.ts`. It does not load on this host, with
or without this change, because `@paperclipai/plugin-sdk` is not built
here. It imports the reader of the retained pack and no archive number.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The two changes to `create-provider-pack-archive.mjs` change the
application image's transport archives: about half of them lose nine
bytes, and all of them get new numbers one time. An archive's content
does not change. A reader pairs file N of a receipt with archive N of
the same build.
- The large dependency files in the pack, and in their transport
archives, now have the modification time 0 (1970). Before, they had the
build time. Nothing in this repository reads those times.
- `unpack` is a reader of tar and gzip that this repository owns. It
reads the layers that BuildKit, GNU tar and bsdtar write (tested with
each), and it fails for a header it does not know.
- The check on the built images, not the recipe, is what proves the
layer rule; it runs in each leg of the workflow.
- `check` now needs `--variant`. A caller that does not pass it fails
with a clear message. No caller exists outside this repository yet.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 19:52:02 -08:00
Nicky LeachandClaude Fable 5.2 5d3fa712aa feat(docker): build and check the Runner alone as the runner-artifacts target (#21204)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - Today the application image build compiles the Runner. Other images
that need the Runner build it again, or pull the whole application image
to copy it out
> - Each of those builds can produce a slightly different Runner, and
each one costs time
> - The Runner should be built one time for each change to its inputs,
and every image should copy the same bytes
> - This pull request adds the build target `runner-artifacts`, a
contract for its layout, and a workflow that builds and checks it. It
publishes nothing
> - Nothing consumes the target in this pull request. The application
image build does not change
> - The benefit is one checked Runner build that later changes can
publish and copy by digest

## Linked Issues or Issue Description

Refs #21096 (the merge-queue image build) and #21167 (provider pack
layers). Those are earlier work on the same build. No issue exists for
this change.

**Problem or motivation**

The Runner is compiled inside the application image build. The daemon
comes from the `runner-build` stage. The provider pack is built on top
of the full server and UI build. Downstream sandbox and VM images need
the same Runner. They rebuild it from source, or they pull the
application image and copy paths out of it. There is no contract for
those paths and no single identity for a Runner build.

**Proposed solution**

Build the Runner alone, from its own inputs, as a plain file tree. Give
each build an identity: a digest of its inputs. Document the layout as a
contract, and check every built tree against it. Keep every check that
proves the Runner is good in this one build.

**Alternatives considered**

- Keep the current builds and only document the paths in the application
image. This does not remove the duplicate builds, and consumers still
need the full application image.
- Build the Runner in `docker.yml`. That workflow has no pull-request
trigger, so a change to the Runner build would have no check before it
merges.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 2 of 4. Each part is on top of the one before it:

1. `runner/pack-inputs-digest`: an optional field in the provider pack
manifest
2. `runner/runner-image-build` (#21204): build and check the Runner
image, with no publication (this pull request)
3. `runner/runner-image-layers`: layers by change rate, and the slim
variant
4. `runner/runner-image-publish`: publish both variants from `master`

This pull request had one larger form before. Publication, the image
layers and the second variant are now parts 3 and 4.

## What Changed

- `Dockerfile`: eight new stages that end in the target
`runner-artifacts` (`FROM scratch`). They install with a filtered,
frozen lockfile from a context that holds only the Runner's inputs. The
base is a Node image on Debian bookworm, pinned by digest. No existing
stage changed. `cloud` is still the default target.
- The tree holds: the stripped daemon, the Runner package's complete
`dist` with the build stamp that its own build records, the provider
pack with its transport archives, the Grok provisioning inputs, the
transport contract source and a bundled transport module, the lockfile,
and `runner-image.json`.
- `scripts/runner-image-identity.mjs`: computes the image identity. It
hashes every file under the paths that the build copies from the build
context (path, executable bit, content; symbolic links by target), the
whole `Dockerfile`, and the two platforms. The build runs it on the
files that the build context supplies after `.dockerignore`.
- `scripts/runner-image-layout.mjs`: the layout contract. `write`
assembles the tree in the build. `check` verifies a built tree: every
documented path, the build stamp, every pack artifact and tree digest in
`provider-pack.json`, and every transport archive. It also extracts the
transport archives the way a sandbox does and compares the result with
the pack. No checked path may resolve outside the tree.
- Publication form. The build argument
`PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1` is off by default. Only with it
does the pack manifest carry `payload.runnerInputsDigest` (part 1), and
`runner-image.json` records `publication: true`. Another image build
that runs the stage inline builds the same Runner, but its pack names no
image. `check` requires the field in a publication build and refuses it
in any other. `--expect-inputs-digest` passes only for a publication
build.
- `docker/runner-image/Dockerfile.smoke`: runs `paperclip-runnerd
--build-metadata` on each base image that a consumer uses. Each base is
pinned by digest.
- `.github/workflows/runner-image.yml`: two jobs, on pull requests that
touch an input and on manual dispatch. `identity` resolves the lockfile
and computes the digest in the build context. `build` builds each
architecture on a native runner, checks the layout, and runs the smoke
matrix. The workflow has `contents: read` only. It holds no credential,
reads no registry, and publishes nothing. It builds the publication
form, so the check can bind the tree to the digest that the identity job
resolved. The tree stays on the runner.
- `packages/paperclip-runner/scripts/provider-pack-runtime-tree.mjs`:
the tree digest function moves here from `build-provider-pack.mjs`
without a change in behavior, so the layout check can use it. A test
pins its value.
- `.github/workflows/pr-merge-review.yml`: lists the new workflow, as
`doc/PR-MERGE-REVIEW-WEBHOOK.md` requires.
- `doc/runner-image-contract.md` (127 lines): the contract for
consumers. `doc/DOCKER.md` links to it.

Changes from the earlier form of this pull request:

- The identity hashes the whole `Dockerfile`. The script no longer
parses stages, and it no longer refuses recipes.
`scripts/runner-image-identity.mjs` went from 397 lines to 166. A short
test keeps the three rules that make the hash enough: the Runner stages
start only from pinned images, only `runner-artifacts-context` reads the
build context, and it copies only the paths that the identity hashes.
- The contract document went from 384 lines to 127.
- The workflow tests keep only the security properties: job permissions,
no credential and no registry access, triggers, actions pinned by
commit, and no expression inside a shell script. They no longer pin step
names, order, or wording.

## Verification

Verified on `linux/arm64` only, on a local Docker host:

- `docker buildx build --target runner-artifacts --platform linux/arm64
--build-arg PAPERCLIP_RUNNER_SOURCE_REVISION=<sha> --build-arg
PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1 --output type=local,dest=<tree> .`
builds in 166 seconds without a cache. The image is one layer of
850,121,093 bytes. The tree is 1,755,345,443 bytes in 9,969 files.
- `node scripts/runner-image-layout.mjs check <tree>
--expect-inputs-digest <digest> --expect-source-revision <sha>
--expect-architecture arm64` passes on the exported tree.
- `docker buildx build --file docker/runner-image/Dockerfile.smoke
--build-context runner-artifacts=<tree> docker/runner-image` passes on
all five base images.
- The digest from `--target runner-artifacts-identity` equals the digest
in the built `runner-image.json`:
`d05f51f075cb8fec687a1d27f574ba7cb41cd8ebbe9927c5cbb2cfc13b147880`.
- The same target with `PAPERCLIP_BUILD_COMMIT=<sha>` and no publication
argument builds. Its pack has `runnerSourceRevision` and no
`runnerInputsDigest`. `check` passes on it, and `check
--expect-inputs-digest` fails with "this is not a publication build".
- An edit to `ui/package.json` and to `server/src` leaves the digest
unchanged. A comment added to an application stage of the `Dockerfile`
changes it.
- The two points above ran one commit state earlier. The only difference
to this commit is one type annotation in a comment of
`scripts/runner-image-layout.mjs`.
- A named build context replaces the `runner-artifacts` stage, and no
Runner stage runs. A build with the application build's view of the
context accepts the tree's `dist` and build stamp as current
(`assessRunnerBuild`).
- The daemon has the same bytes in the five cold builds of this stack
that were compared.
- Tests: `cd server && node node_modules/vitest/vitest.mjs run` on
`runner-image-identity.test.ts`, `runner-image-layout.test.ts` and
`runner-image-workflow.test.ts` (132 tests), with
`remote-provider-pack-verification.test.ts` and the existing Dockerfile
tests: 186 tests pass. `tests/runner-e2e` workflow tests: 39 pass. `node
--test .github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the
image contract scripts and `provider-pack-runtime-tree.test.mjs`: 33
pass. `check-docker-deps-stage.mjs`, `check-node-version-policy.mjs` and
`docker buildx build --check` report nothing.

Never run:

- The `linux/amd64` build. It is the leg that downloads the Cursor and
Copilot assets for the pack.
- The workflow. No job has run on GitHub.
- The transport archive round trip with GNU tar outside the build. The
build runs it with GNU tar. The local host ran it with bsdtar.
- `actionlint` and `hadolint`. They are not installed on the local host.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- Low risk to the application image. The change adds stages and files.
No existing target reaches the new stages.
- The first run of the workflow is the first test of the amd64 build.
This pull request triggers it.
- The `Dockerfile` is one file for all images. Any edit to it, also to
an application stage, changes the Runner image identity. That is the
price of not parsing it.
- The Dockerfile frontend (`# syntax=docker/dockerfile:1.20`) is a tag,
not a digest. A new frontend release under that tag does not change the
identity.
- The image does not run the drift checks of the Runner's `build`
script. They need files outside the Runner's inputs, and repository CI
runs them. A test fails when the `build` script gains a step.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 17:30:51 -08:00
DottaandClaude Fable 5.2 9a9cd20d91 feat(ci): send one in ten CODEOWNER PR runs and their outside checks to Honeycomb (#21192)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Contributors need fast, predictable CI feedback.
> - PR #16378 added a Honeycomb trace of the main PR workflow, but only
for PRs with the `profile-ci` label.
> - Ordinary CI time was therefore never recorded. All of the last 300
`CI profile` runs were skipped.
> - Reviews and scans outside the `PR` workflow, such as Greptile, were
not in the trace.
> - This pull request exports one CODEOWNER PR run in ten without a
label, and adds outside checks to every trace.
> - The benefit is a continuous record of total CI time and of which job
or check was slowest. The Honeycomb key stays out of reach of outside
contributors.

## Linked Issues or Issue Description

Refs #16378

**What existing behavior does this improve?**

The Honeycomb CI trace that the `CI profile` workflow exports.

**Current behavior**

- The exporter runs only for a PR with the `profile-ci` label. The label
also forces the full suite.
- The trace has the jobs of the `PR` workflow only. Greptile, security
scans, commit statuses and other workflows are absent.
- The only total is the `PR` workflow time.

**Proposed behavior**

- One `PR` run in ten is exported without a label. The run number must
end in 0. The trace has every job and step. Each span has a sample rate
of 10.
- Each trace has one span for each outside check and each other workflow
on the head commit.
- The root span has `ci.workflow_ms`, `ci.all_checks_ms`,
`ci.last_to_finish` and `ci.slowest_check`.

**Reason and benefit**

Maintainers can see how long CI takes and what is slow, for example a
long Greptile review, without a label on each PR. Sampling keeps the
volume near 1.7 million events each month. Export of every run is near
17 million.

## What Changed

- `scripts/ci-profile/authorize.mjs`: add `exportMode` (label or
one-in-ten sample) and `associatedOwnerPr` (the CODEOWNER checks without
the label).
- `scripts/ci-profile/trace.mjs`: add `normalizeChecks` and spans for
outside checks. Add the sample rate and the new root fields. Sampled
runs do not warn about absent test artifacts.
- `scripts/ci-profile/export.mjs`: select the mode. Read check runs,
commit statuses and workflow runs for the head commit. Wait a maximum of
15 minutes for outside apps that are not complete. Sampled runs read no
artifacts.
- `.github/workflows/ci-profile.yml`: no new job. The job condition now
requires a branch of this repository, an actor and rerun actor on the
`AWS_CI_ROUTING_USER_IDS` roster, and a labelled or sampled run. Add
`checks: read` and `statuses: read`. Increase the timeout to 30 minutes
for the bounded wait.
- `doc/ci-profiling.md` and `doc/DEVELOPING.md`: describe sampled runs,
outside checks, the totals and the job condition.

## Verification

- `node --test .github/scripts/tests/ci-profile.test.mjs
.github/scripts/tests/ci-routing.test.mjs`: 163 pass, 0 fail.
- `actionlint .github/workflows/ci-profile.yml`: no findings.
- Dry runs against real runs with the real CODEOWNERS check (`node
scripts/ci-profile/export.mjs --run-id=ID --attempt=1 --dry-run`):
- A sampled CODEOWNER run (run number 64780): 1,144 spans, sample rate
10, 5 outside checks and 1 other workflow.
- A sampled run from an outside contributor (run number 64800): "No
CODEOWNERS-authorized PR at this revision; no data exported."
- A CODEOWNER run that is not in the sample: "This attempt was not
selected for profiling; no data exported."
- New tests cover: the sample rule, fork and outsider PRs, a bot rerun
actor, a stale head, outside-check spans and totals, the bounded wait,
and the job condition.
- Not run locally: `scripts/ci-profile/reporters.test.mjs`. It needs
installed dependencies, and this change does not touch the reporters. CI
runs it.
- No data was sent to Honeycomb before merge. The workflow runs from
`master`, so the first real export occurs after merge.

## Risks

- The label path now also needs the `AWS_CI_ROUTING_USER_IDS` roster. If
that variable is empty, nothing is exported. All four CI CODEOWNERS are
on the roster today.
- The roster is only a first filter. The exporter still checks the PR
author and both actors against CODEOWNERS on `master` before it reads or
sends data. The key stays in the `ci-profiling` environment, which
permits only `master`.
- The export job can now run for a maximum of 30 minutes on a
GitHub-hosted runner while it waits for an outside check.
- GitHub gives no completion time for a workflow run. The span for
another workflow ends at the last update of that run.
- Honeycomb volume increases by about 1.7 million events each month.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code, with
tool use, code execution and repository inspection.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 16:57:23 -06:00
DottaandClaude Fable 5.2 8990e94005 fix(ci): keep community PR runs from delaying CODEOWNERS jobs in the hosted queue (#21166)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Its pull request CI runs on two pools: an AWS fleet for CODEOWNERS,
and GitHub-hosted runners for every other contributor
> - Some short jobs for CODEOWNERS pull requests also run on
GitHub-hosted runners: the commitperclip review and the merge review
notifier
> - GitHub starts about 550 hosted jobs each hour for this repository,
and all hosted jobs wait in one first-come queue
> - A community pull request run puts 52 jobs in that queue at one time,
so the short CODEOWNERS jobs wait behind them
> - This pull request limits how many jobs one hosted run can queue, and
lets the short CODEOWNERS jobs use the reserved AWS metadata fleet
> - The benefit is that community pull requests do not delay CODEOWNERS
work

## Linked Issues or Issue Description

**What happened?**

On 2026-11-12 the repository had about 348 queued workflow runs. Jobs on
the AWS fleet started in a median of 29 seconds. Jobs on `ubuntu-latest`
did not:

| Job | Median wait | Job length |
| --- | --- | --- |
| PR merge review notification | 42 min | 9 s |
| commitperclip PR Review | 36 min | 15 s |
| Community PR test jobs | 57 min (p90 76 min) | 7 min |

Between 12:00Z and 18:00Z, hosted job starts stayed between 492 and 567
each hour while the number of running hosted jobs changed from 63 to
134. The start rate is the limit. Community PR runs supplied 3,462 of
the 3,744 hosted jobs in that window (92%).

**Expected behavior**

A short job for a CODEOWNERS pull request starts in about one minute,
whatever the community pull request load is.

**Steps to reproduce**

1. Open several pull requests from forks in a short time.
2. Open a pull request as a CODEOWNER.
3. Look at the start time of `commitperclip PR Review` and `PR merge
review notification` for the CODEOWNER pull request.

## What Changed

- `pr-trusted.yml`: each of the four test matrices has `max-parallel: 4`
when the run uses `ubuntu-latest`. The run has the same jobs. The other
jobs wait in the run, not in the shared queue. AWS runs have no limit
(256).
- `commitperclip-review.yml` and `pr-merge-review.yml`: the job uses the
reserved `paperclip-ci-routing-x64` fleet when
`AWS_CI_SHORT_JOBS_ENABLED` is `true`, the pull request branch is in
this repository, and the author and the sender (or the run actor and the
triggering actor) are in the AWS admission list. The admission variables
are the same as in `ci-routing.yml`. All other events use
`ubuntu-latest`.
- Check-run and commit-status notifications stay on `ubuntu-latest`.
Those events do not identify the author.
- `ci-routing.test.mjs`: 24 placement cases for the two jobs, and a
contract test for the matrix limit.
- `doc/DEVELOPING.md`: the two settings and the runner group step.

## Verification

- `node --test .github/scripts/tests/ci-routing.test.mjs
.github/scripts/tests/notify-pr-merge-review.test.mjs
.github/scripts/tests/ci-plan.test.mjs
.github/scripts/tests/ci-profile.test.mjs
.github/scripts/tests/run-quality-gates.test.mjs`: 200 tests pass.
- `actionlint` accepts the three workflows, including the `max-parallel`
expression.
- Not verified before merge: `pr.yml` calls `pr-trusted.yml@master`, and
the two short workflows run from master. The effect on queue time can
only be measured after merge.

## Risks

- A community pull request run takes longer when the hosted queue is
empty: about 45 minutes for the server matrix, compared with about 13
minutes. Today these runs take 150 to 200 minutes because of the queue.
- The AWS routing does nothing until a maintainer sets
`AWS_CI_SHORT_JOBS_ENABLED`. Before that, the `paperclip-ci-routing`
runner group must allow the two workflow references in
`doc/DEVELOPING.md`. If the variable is set first, the two jobs for
CODEOWNERS pull requests stay queued.
- The two jobs run only scripts from master and no pull request code.
The secrets they read do not change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code, with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 13:48:25 -06:00
DottaandClaude Opus 5.5 1d153f62b6 feat(docker): build the image in the merge queue and promote it on master (#21096)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `docker.yml` builds, probes, gates and signs the standard Core image
for every master push. Paperclip Cloud deploys a Core commit only after
that image is published.
> - Each master push rebuilds the image from scratch: about 10.6 min for
amd64 and 7.8 min for arm64.
> - The owner approved a merge queue for Core. A merge queue tests the
exact commit that master will become, before the merge.
> - This pull request builds and probes the image for that exact commit
in the queue, with no registry credentials. The later master push pushes
those exact bytes instead of rebuilding. When no matching verified build
exists, master builds from source exactly as today.
> - The benefit is that the image legs on master drop from about 8–11
min to about 3–4 min, and nothing changes until the owner turns the
queue on.

## Linked Issues or Issue Description

Refs #21013 (it measured the image path and proposed this plan).

**What existing behavior does this improve?**
Every master push rebuilds the standard image, even when the same commit
could already have been built while it waited to merge.

**Subsystem affected**
`.github/workflows/docker.yml` (`build-and-push` only), new
`.github/workflows/docker-merge-group.yml`, new
`scripts/merge-group-image.mjs`.

**Current behavior**
`build-and-push` always runs the full BuildKit build on master: 639 s
(amd64) and 468 s (arm64) in run 43133035209.

**Proposed behavior**
On a `merge_group` event, build both architectures for the merge-group
commit, probe PID 1, and upload an OCI archive plus a record of commit,
tree, architecture and digest. On the master push for the same commit,
each leg pushes that archive by digest instead of building. Everything
after the leg (probe, exact-source gate, signed-index check,
attestation) is unchanged.

**Reason and benefit**
The build moves before the merge, where it runs in parallel with the
queue's other checks. The master leg only downloads and pushes.

**Breaking changes**
None. Without `merge_group` events, `find` returns no run and the leg
builds as before.

**Additional context**
This PR does not touch the exact-source gate jobs
(`source_verification`, `publication_verification`, `merge-and-push`),
`scripts/watch-release-source.mjs` or `cloud-readiness.yml`. Another
change owns them.

## What Changed

- **`docker-merge-group.yml` (new, `on: merge_group` only).** It runs
only for this repository's master queue. Workflow permissions are `{}`
and the job has `contents: read` only. There are no secrets, no registry
login, no `cache-to`, no `push: true`, no environments, and checkout
uses `persist-credentials: false`. It runs on GitHub-hosted runners, not
the paid AWS fleets. Steps: build `target: production` with the same
build args as master (`PAPERCLIP_BUILD_COMMIT` is the merge-group SHA,
which becomes the master SHA). Read the public registry cache. Write
`type=oci` to a tar. `docker load` it and run
`scripts/assert-orphan-reaping.sh`. Only then write `record.json` and
upload `merge-group-image-<arch>` (3-day retention).
- **`scripts/merge-group-image.mjs` (new).**
- `find` (master push only): selects a completed, successful run of this
workflow, with event `merge_group`, the same `head_sha`, repository and
head repository both `1170821064`, and a `gh-readonly-queue/master/`
head branch. It needs exactly one unexpired artifact for the
architecture.
- `record` (merge group only): reads the archive's `index.json`, checks
each blob's sha256 and that the image is `linux/<arch>`, and writes
`{sha, tree, arch, digest}`.
- `verify` (master push only): re-reads the archive the same way and
requires record SHA = `GITHUB_SHA`, record tree = `git rev-parse
HEAD^{tree}` of the master checkout, same architecture, and archive
digest = recorded digest.
- **`docker.yml` `build-and-push`.**
  - The leg gets `actions: read` so it can download the artifact.
- New steps run after the registry login: find, download, then promote
(`verify`, then `skopeo copy --all --preserve-digests oci-archive:…
docker://ghcr.io/…@<digest>`). The step sets its `digest` output only
after the push succeeds.
- All three new steps are `continue-on-error`. Buildx, labels, metadata
and "Build and push by digest" run when `steps.promote.outputs.digest ==
''`, so any failure builds from source.
- "Select image digest" feeds the existing PID 1 probe and digest
export.
- **Tests (`scripts/merge-group-image.test.mjs`, 8 tests):**
- Run selection rejects another commit, another event, another workflow,
a fork head, another repository, another queue base, and failed,
cancelled or running builds.
  - Artifact selection.
- Archive digest with and without provenance, a tampered blob, the wrong
architecture, and a real tar read through the system `tar`.
  - Record checks for commit, tree, architecture and digest mismatch.
- The CLI refuses `record` outside `merge_group` and `find`/`verify`
outside a canonical master push.
- Static checks of the merge-group workflow: credentials, triggers,
permissions and runners.
- Static checks of the `docker.yml` wiring: promotion is gated and falls
back, the output is set only after the push, and every build step has
the fallback condition.

### How promotion proves it publishes the verified tree

1. `find` accepts only a run whose `head_sha` equals `github.sha` of the
push, meaning the queue landed the commit it tested (unverified: see
"Merge method" in Risks).
2. `verify` compares the record's tree with the tree of the master
checkout, and its commit with `GITHUB_SHA`.
3. The archive is content-addressed. The digest is recomputed from the
archive blobs and must equal the recorded digest. `skopeo
--preserve-digests` to that digest reference fails rather than produce
other bytes.
4. The leg then runs the PID 1 probe on that registry digest.
`merge-and-push` still signs only an index whose Linux images are the
probed digests (#21013), after the unchanged exact-source gate.
5. The pipeline itself is trusted only as master already ran it. `find`
and `verify` run the copy of `scripts/merge-group-image.mjs` from the
push's `before` commit (`git show "$BEFORE:…"`), not the copy the push
brings. `verify` refuses promotion if anything in `before..HEAD` (every
commit of a multi-PR group, not only the tip) changes
`docker-merge-group.yml`, `docker.yml`, the promotion script or the PID
1 probe. Such a push builds from source. So does the first push after
this PR merges, because its `before` has no verifier.

## Verification

- `node --test scripts/merge-group-image.test.mjs`: 10 pass. One builds
a real git history in which a two-PR push changes the pipeline in its
first commit only, and checks that the range check catches it.
- `node --test` on the workflow and contract suites (cloud-readiness,
docker-canary-promotion, post-merge-runner-routing,
lockfile-refresh-workflows, release-verify-workflow, reuse-canary-image,
standard-image-contract, cloud-source-verification, merge-group-image):
492 pass.
- `vitest run server/src/__tests__/docker-build-stamp.test.ts
server/src/__tests__/container-init-reaping.test.ts`: 22 pass.
- `actionlint` on both workflows: no new findings (the 6 remaining are
pre-existing `ls` and word-splitting notes in `docker.yml`).
- Local: `skopeo 1.24.1 copy --all --preserve-digests` between OCI
layouts kept the index digest. It accepted a `docker://…@sha256:`
destination and failed only at the network step.

Cannot be exercised before the queue exists, because no `merge_group`
event fires:
- the merge-group run itself: the hosted-runner build time, disk use
with tar plus `docker load`, `docker load` of a BuildKit OCI tar, and a
~5 GB artifact upload;
- the cross-run artifact download on master;
- whether `skopeo` exists on the runs-on AMI and accepts digest
destinations in its installed version.

Each master-side failure falls back to the full build. A merge-group
failure only blocks the queue if its check is required, which is why
that check is required last in the steps below.

## Risks

- **Inert until enabled.** No `merge_group` events fire before the owner
adds a merge queue. On master, `find` lists runs (one API call per leg,
about 1 s), finds none, and the leg builds as today. The only always-on
change is the leg's `actions: read` permission.
- **Build cache freshness.** A promoted master push writes no
`buildcache-<arch>`, and merge-group builds read but never write it.
With every push promoted, the cache ages, so dependency or weekly
tool-layer changes rebuild in each merge-group run until a fallback
build refreshes the cache. A follow-up can refresh the cache from master
without blocking publication.
- **Publication time is still gate-bound.** The exact-source gate (about
13–14 min after push) still decides when the image publishes. See
"Expected timing".
- **Merge method (unverified).** Promotion needs the master commit to be
exactly the commit the queue tested, because the image stamps the commit
SHA (`PAPERCLIP_BUILD_COMMIT`, `build-info.json`, the provider pack
revision). Matching only the tree would publish an image stamped with
another SHA, so this PR does not relax the check. I expect the queue to
fast-forward master to its tested commit. GitHub's merge-queue page does
not state this, and it cannot be tested before the queue exists. If the
queue lands a different SHA, promotion never matches and every push
builds from source. That is safe but inert. Step 4 below checks this.
- **skopeo on the AWS fleet.** The promote step checks for `skopeo`
first and logs "Merge-queue promotion unavailable" if it is missing,
then builds from source. If that notice appears, add skopeo to the fleet
image.
- **Hosted runner capacity.** amd64 merge-group builds run on 4-core
hosted runners, colder and slower than the AWS fleet. That adds queue
latency, not master latency.

### Settings the owner must enable, in order

1. **Prerequisite, a code change, not a setting.** The `master` ruleset
(id 13619726) requires `ci / verify` and `ci / e2e`. Those come from
`pr.yml`, which triggers only on `pull_request`. A merge queue waits for
required checks on the `merge_group` event too, so every entry would
stall until its timeout. First decide, and merge in a separate PR, how
`ci / verify` and `ci / e2e` report on `merge_group`: a full CI run on
the merge-group commit, or a fast pass that relies on the PR-head
result. Today the ruleset does not require up-to-date branches either.
2. Merge this PR. Nothing changes yet.
3. In Settings → Rules → Rulesets → `master` (13619726), add **Require
merge queue**:
   - merge method Squash;
- build concurrency 5, minimum group size 1, maximum group size 5, wait
0 min;
   - "Require all queue entries to pass required checks";
   - status check timeout 60 min.
Do not add the image build to the required checks yet. Promotion is then
opportunistic: if the merge-group build has not finished when master is
pushed, master builds from source.
4. After the first queued merge, confirm promotion can match. The master
commit SHA must equal the `head_sha` of its `Docker merge-queue build`
run, and the next master `Docker` run's "Promote the merge-queue image"
step must succeed. If the SHAs differ, promotion stays inert under this
merge method. Stop here and report it.
5. After about a week of green `Docker merge-queue build` runs, add
`build (linux/amd64, ubuntu-latest, amd64)` and `build (linux/arm64,
ubuntu-24.04-arm, arm64)` (app: GitHub Actions) to the ruleset's
required status checks. Then every queued commit has a verified image
ready to promote.
6. The existing bypass actors (org admins, repository admins,
integration 5534676) can still merge outside the queue. Those pushes
have no merge-group run and build from source.

### Rollback

- Remove the image checks from the required status checks (step 5), or
remove **Require merge queue** (step 3). Master pushes then find no
merge-group run and build from source. Nothing to migrate.
- Code: revert this PR. Old merge-group artifacts expire after 3 days.
Promoted images are ordinary digests under
`ghcr.io/paperclipai/paperclip` and stay valid.

### Expected timing

- Image leg on master: about 3–4 min (download about 5 GB, push the new
layers by digest, PID 1 probe) instead of 10.6 min (amd64) and 7.8 min
(arm64).
- Push to published: still bounded by the exact-source gate on master,
about 14–15 min (gate plus about 45 s for `merge-and-push`). Today it is
the same bound, but with the legs as a second 11-min path and the
serialized-run queue in front.
- If the gate later accepts proof produced in the merge queue (owned by
the gate change), push to published would be about 4–5 min (leg
promotion plus `merge-and-push`).

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, read-only GitHub API and registry reads).
Default reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 07:10:11 -06:00
DottaandClaude Opus 5.5 ca62ca9e2a fix(release): keep the Runner chaos evals out of the Core image publication gate (#21088)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A Core image is published only after "Cloud source verified v1"
passes for the same commit in `cloud-readiness.yml`
> - That job needs `verify`, which is the whole `release-verify.yml`
call, and that call included the Runner chaos evals
> - The chaos eval job ("Restart, replay, trace, and recovery faults")
took 10.2-11.9 min, with a 7-9 min server finalization step. It is one
of the two slowest jobs on the gate
> - The owner decided on 2026-11-12 that the Runner chaos evals must not
be a publication prerequisite
> - This pull request runs the chaos evals as a separate job in the same
Cloud readiness run. They still run and report, but the source proof
does not wait for them
> - The benefit is an earlier image publication on every master commit.
A chaos failure stays visible as a failed check

## Linked Issues or Issue Description

**What happened?**
The Core image gate waited for the Runner chaos evals. In master runs
43133035381, 43134251278 and 43145252393 the chaos job took 10.2, 11.9
and 11.0 min. The gate opened at +13.5, +13.2 and +13.9 min.

**Proposed change**
Owner decision (2026-11-12): keep the chaos evals running on every
admitted master commit, but remove them from the set of jobs that the
image and Cloud discovery wait for.

**Why**
Every minute on the gate is a minute on every staging release.

## What Changed

- `.github/workflows/release-verify.yml`: new boolean input
`runner_chaos_evals`, default `true`. The `runner_chaos_evals` job runs
only when it is true. Release callers (`release.yml`,
`release-smoke.yml`) do not set it, so they stay gated on the evals.
- `.github/workflows/cloud-readiness.yml`:
  - `verify` passes `runner_chaos_evals: false`.
- New job `runner_chaos_evals`, named "Runner chaos evals (not a
publication gate)". It needs `admission`, has the same admission
condition as `verify`, and calls `runner-chaos-evals.yml` with `ref:
github.sha`. It has no `continue-on-error`.
- `source_verified` ("Cloud source verified v1") still has `needs:
[verify]`. A comment names the owner decision and says not to add the
chaos job.
- `scripts/cloud-source-verification.mjs`: new `runnerChaosEvalsJob`
constant. The fail-fast check ignores jobs named `"<that name> / ..."`.
A chaos failure therefore does not raise `SourceVerificationFailed`. So
`watch-release-source.mjs` no longer cancels the readiness run and the
image run when only the chaos evals fail. Every other failed job still
fails fast and cancels as before.
- Tests:
- `scripts/cloud-source-verification.test.mjs`: a chaos failure does not
cancel the image or block the proof; a chaos failure while the proof is
pending keeps waiting; a required job whose name only resembles the
chaos job still fails fast.
- `.github/scripts/tests/cloud-readiness.test.mjs`: pins the job set
(`admission`, `verify`, `runner_chaos_evals`, `source_verified`),
`source_verified` `needs: [verify]` exactly, `verify` passes
`runner_chaos_evals: false`, the chaos job calls the eval workflow, and
release callers keep the default. Putting the chaos job back into
`source_verified`'s `needs` fails this test (checked locally: 2
failures).

Publication gate, before: `Cloud source verified v1` waited for
typecheck, build, general tests (17 jobs), serialized tests (5), Runner
workflow evals, Verify Paperclip Runner (3 lanes), and the Runner chaos
evals. After: the same list without the Runner chaos evals.

How a chaos failure stays visible:
- Its own check fails on the master commit.
- The Cloud readiness run concludes `failure`.
- The chaos eval bundle is still uploaded (`if: always()`).
- Nothing reports the chaos evals as passed. They are just not a
prerequisite.

Consumers:
- `docker.yml` reads the job "Cloud source verified v1" through
`scripts/watch-release-source.mjs`. No change to `docker.yml` is needed.
- Paperclip Cloud discovery (`src/fleet/cloud-readiness.ts`) runs with
`sourceOnly: true` in `cloud-distribution-policy.ts` and
`release-resolver.ts`. In that mode it needs the workflow path
`.github/workflows/cloud-readiness.yml` and one successful job named
"Cloud source verified v1" in the latest attempt. It does not need the
run conclusion. Both are unchanged, so Cloud needs no change.
- The non-`sourceOnly` mode reads the run conclusion and a job "Cloud
deployable v1". That job no longer exists in this workflow.

## Verification

- `node --test scripts/cloud-source-verification.test.mjs
.github/scripts/tests/cloud-readiness.test.mjs
.github/scripts/tests/post-merge-runner-routing.test.mjs`: 395/395 pass.
- Red/green:
- Before the fail-fast change, the two new chaos tests failed: the
watcher cancelled the image.
  - With the chaos job added back to `needs`, the workflow test failed.
- `actionlint .github/workflows/cloud-readiness.yml
.github/workflows/release-verify.yml`: clean.
- Not run: a real Cloud readiness run of this change. GitHub shows the
result after merge.
- Expected gate: the slowest remaining gating job sets it. After #21016
that is a general-server shard at about 10.5 min, then chat (8.8-10.5
min) and serialized (9.2-9.5 min). Expected gate open is about +11.5 to
+12 min, down from +13.2 to +13.9.

## Risks

- A chaos-eval regression no longer blocks the staging image. This is
the owner decision. The failure stays visible as a red check and a
failed Cloud readiness run.
- npm canary publication reuses the same source proof, so it is also no
longer gated on the chaos evals. Stable and preview releases
(`release.yml`) still run them as a gate.
- The non-gating rule depends on the job name prefix. A test pins the
name to the constant in `cloud-source-verification.mjs`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, GitHub CLI). Standard reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Related: #21016 (server shard rebalance), #21013 (image publish time;
owns `docker.yml`).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:08:07 -06:00
DottaandClaude Opus 5.5 bed5f70278 perf(ci): rebalance server test shards and repair the weekly weight refresh (#21016)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A Core image is published only after the release verification of the
same commit passes (`cloud-readiness.yml` calls `release-verify.yml`)
> - The image is ready about 3 minutes before that gate opens, so the
slowest verification job sets the staging release time
> - The slowest job is a general-server test shard: 11.8-12.7 min, while
other server shards take 8-9 min
> - The shards are balanced by a duration manifest, but the manifest has
not changed since 2026-10-31. The weekly refresh that updates it failed
on 2026-11-02 and 2026-11-09
> - About 40 newer suites per shard use the 1 s median fallback weight,
but several of them run for 90-200 s, so they pile up on one shard
> - This pull request repairs the weekly refresh and refreshes the
server weights from three current master verifications
> - The benefit is a shorter slowest server shard (predicted 10.3 -> 8.8
min of test time) on every master verification and every PR run

## Linked Issues or Issue Description

**What happened?**
The Core image publication waits about 13.5 min for source verification.
The critical path is one general-server shard. In three master
verifications (runs 43133035381, 43134251278, 43145252393) the slowest
server shard took 12.1, 11.0 and 12.7 min. The fastest took 8.0, 8.9 and
8.4 min.

**Root cause**
`scripts/general-server-shard-durations.json` came from one run on
2026-10-31. The weekly `ci-test-durations.yml` refresh failed twice:
- 2026-11-02 (run 41408033686): `gh api .../jobs/<id>/logs` failed. A
server shard log is 30-65 MB. One download took 59 s locally. Three
parallel downloads overran the 120 s `execFile` timeout.
- 2026-11-09 (run 42580964686): the refresh's own check failed.
Merged-PR sources for the Runner lane have a leading `pull` key, and the
test accepted only `run, attempt, sha, url`.

**Expected behavior**
The weekly refresh finishes and keeps the shards balanced as suites are
added.

## What Changed

- `scripts/ci/gh-api.mjs`: job-log downloads get a 15 min timeout and a
512 MB buffer. JSON calls keep 120 s. A failed call now gives a short
error and does not print the partial 60 MB log.
- `scripts/ci/pr-run-durations.mjs`: new shared `assertManifestSource`.
It accepts the optional leading `pull` key that `collectMergedPrSamples`
writes. The checked-in Runner manifest test and the collector test both
use it, so the collector format stays covered.
- `.github/workflows/ci-test-durations.yml`: refresh job timeout 20 ->
45 min (the server lanes alone fetch about 30 large logs).
- `scripts/general-server-shard-durations.json` and
`scripts/serialized-shard-durations.json`: refreshed with `node
scripts/ci/refresh-test-durations.mjs --lanes=server 43133035381
43134251278 43145252393` (median of three successful protected-master
Cloud readiness runs; 1757/1815 general files and 173/173 serialized
files measured).
- No test is added, removed, skipped or reassigned to a different lane.
No shard count changes. `docker.yml`, the Dockerfile and the PID-1 check
are not touched.

## Verification

Measured job timings (minutes from the first job start; three master
verifications):

| Job | 43133035381 | 43134251278 | 43145252393 |
| --- | --- | --- | --- |
| Slowest server shard (duration) | 7/10: 12.1 | 6/10: 11.0 | 8/10: 12.7
|
| Fastest server shard | 2/10: 8.0 | 2/10: 8.9 | 5/10: 8.4 |
| Runner chaos evals (restart/replay) | 10.2 | 11.9 | 11.0 |
| Verify Paperclip Runner (protocol) | 8.9 | 8.9 | 9.6 |
| Chat (slowest of 3) | 8.8 | 10.5 | 9.4 |
| Serialized (slowest of 5) | 9.3 | 9.5 | 9.2 |
| Build / Typecheck | 4.5 / 2.4 | 4.5 / 3.1 | 4.8 / 2.5 |
| Gate "Cloud source verified" ends | +13.5 | +13.2 | +13.9 |

- Queue to runner start is 0.5-1.1 min for every job. In a server shard,
setup (checkout, pnpm, Node, install) is about 50 s and the preflight
plus plugin-sdk build is about 8 s. The test step is the rest (9-11.8
min). Setup is not worth caching.
- Shard prediction, using the refreshed medians as the cost of each file
and the 3 fork workers per shard:
- general-server, 10 shards: old weights max 10.3 / min 8.1 min; new
weights 8.8 / 8.8 min.
- serialized, 5 shards: old weights max 8.2 / min 4.6 min; new weights
6.6 / 6.6 min.
- The old-weight prediction matches what CI saw: predicted 10.3 and 9.9
min for shards 7/10 and 6/10, measured test steps 11.25 and 10.1 min in
run 43133035381.
- Red/green:
- `node --test --test-name-pattern="gh api"
scripts/ci/test-selection-pr-timings.test.mjs` failed before the fix
(`log timeout 120000`) and passes after it.
- With a `pull` key in the Runner manifest sources, the old "package
lane manifest" test failed with the same assertion as run 42580964686.
The new test passes.
- The refresh script, which died before the fix, completed locally in 8
min after it.
- `node --test scripts/ci/test-selection-timings.test.mjs
scripts/ci/test-selection-pr-timings.test.mjs
scripts/__tests__/run-vitest-stable-shard.test.mjs
scripts/__tests__/e2e-shard.test.mjs`: 76/76 pass. This includes the
complete, non-overlapping shard cover checks.
- `actionlint .github/workflows/ci-test-durations.yml`: clean.
- Not run: the release-verify workflow itself, the scheduled refresh
workflow, and the Runner/e2e lane refresh. The shard times above are
predictions from logged per-file timings, not a measured CI run on this
branch.

## Risks

- Low risk. Weights only move files between shards of the same lane. The
shard tests prove every file still runs exactly once.
- The predicted gain is about 1.5 min on the slowest server shard
(12-12.7 -> about 10.5 min job time). The gate then waits for
"Pre-release Runner chaos evals / Restart, replay, trace, and recovery
faults" (10.2-11.9 min). So the expected gate saving is about 1 min
(13.5 -> about 12.5 min), not the full 1.5 min. Run-to-run noise on a
shard is about ±1 min.
- A larger log timeout can make a stuck download wait up to 15 min. The
45 min job timeout still bounds the run.

Candidates for the owner (not changed here):
- The chaos-eval job is now the bound. Its "server finalization" step
takes 7-9 min. Decide if it must gate image publication, or split it.
- Server shard logs are up to 811k lines, mostly repeated stack traces
from `server/src/services/execution-preparation-timing.ts`. That slows
log upload and this refresh.
- 9 test files run over 180 s (largest:
`chat-channels.integration.test.ts` 429 s,
`heartbeat-process-recovery.test.ts` 344 s). One file cannot be split
across shards.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, GitHub CLI read-only). Standard reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Related: #19650 (earlier repair of the weekly refresh), #21013 (Core
image publish time; separate files).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:29:27 -06:00
DottaandClaude Opus 5.5 f8e1fb0278 perf(docker): take the orphan-reaping pull and pack chmod off the image path (#21013)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `docker.yml` builds and signs the standard Core image for every
master push. Paperclip Cloud deploys a Core commit only after this image
is published and attested.
> - A master commit takes 29 min (p50) and 56 min (p90) to reach a
published image. Every minute here delays a staging fix.
> - The run logs show where the time goes. The exact-source gate (about
14 min) is the critical path today. The build legs finish about 3 min
before it. After the gate, `merge-and-push` spends about 95 s to pull
the 4.9 GB amd64 image for one probe.
> - This pull request moves that probe into the build legs, which run
before the gate. It also removes a duplicate 187 MB layer from the amd64
image.
> - The benefit is about 95–100 s less from gate to signed image on
every master commit, with the same proof. arm64 is now also probed.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The time from a commit landing on master to the signed standard image
(`ghcr.io/paperclipai/paperclip:sha-<full>` plus its attestation).

**Subsystem affected**
`.github/workflows/docker.yml`, `Dockerfile`,
`scripts/standard-image-contract.mjs`.

**Current behavior**
Measured on run 43133035209 (sha 42fd790e, total 999 s):

- Push 01:13:47. The slowest exact-source job, `General tests (server
(7/10))`, ends 01:27:08. `Cloud source verified v1` waits 20 s for a
runner and ends 01:27:44. The watcher sees it at its next 30 s poll,
01:27:58.
- `build-and-push (amd64)` ends 01:24:52 and arm64 ends 01:21:59. Both
have slack before the gate. Run 43134250912 shows the same shape: the
gate opened 01:44:37 and amd64 ended 01:41:53.
- `merge-and-push` starts 01:28:03 and pushes the manifest at 01:28:34.
Then "Verify PID 1 reaps orphaned processes" pulls the full amd64 image
(01:28:34 to 01:30:09, 95 s). "Verify the immutable standard image" runs
the same probe again on the same bytes (5 s). The attestation ends
01:30:22.

Last 60 master pushes (2026-11-11 20:17Z to 2026-11-12 03:21Z): 43
cancelled or superseded, 14 succeeded, 2 failed, 1 in progress. Push to
first published image that contains the commit: p50 29.4 min, p90 55.5
min, max 85.6 min. A successful run takes 22.5 min (p50) from push. The
median gap between pushes is 4.5 min.

Where the amd64 "Build and push by digest" step spends its 582 s:

| Time | Phase | What happens |
| --- | --- | --- |
| 0–8 s | Load | Dockerfile frontend, cache manifest import |
| 8–54 s | Pull cached layers (46 s) | `deps`, `rust-chef`,
`runner-deps`, `runner-build` and `base` are **all cache hits**. The
`build` stage starts `FROM runner-build` and copies `deps`, so BuildKit
downloads and extracts them: a 932 MB node_modules layer (13 s download,
24.5 s extract) and about 650 MB of Rust `target/` |
| 54–108 s | Per-commit build (54 s) | `COPY . .` 10 s, mtime
normalization 11 s, UI 5 s, plugin SDK 3 s, server tsc 18.5 s, runtime
bundle 5.6 s |
| 108–206 s | `runner-provider-pack` (98 s) | Rebuilt every commit: it
is `FROM build` and stamps the commit. pnpm install about 25 s, pack 9
s, **Kimi smoke 37 s (amd64 only)**, archive 28 s |
| parallel | `production` | Tool layer is a cache hit but pulls 1.11 GB
(11 s). `COPY --from=build /app` 24 s |
| 206–294 s | Pack copy and chmod | `COPY` pack 7.5 s. `chmod -R a+rX`
plus UID check **71 s** (arm64: 20.5 s) |
| 294–420 s | Image export (126 s) | 115 s compressing new layers:
`/app` 1.40 GB, pack 1.01 GB, pack archives 1.06 GB, chmod copy 0.19 GB
(compressed). Push 10 s |
| 420–582 s | Cache export (172 s) | `mode=max`: 152 s "preparing"
(compressing the per-commit layers of `build` and
`runner-provider-pack`, which no later commit can reuse), 20 s upload |

What rebuilds every commit, and why: only the `build` stage and
everything after it. `PAPERCLIP_BUILD_COMMIT` is already declared after
the dependency layers, `CLI_TOOLS_CACHE_EPOCH` is weekly, and dependency
install already copies manifests first. The commit ARG is used by the UI
build, the server stamp, and the provider pack, so those must rebuild. A
pnpm `--mount=type=cache` gives nothing here: the runners are ephemeral,
and cache mounts are not exported to the registry cache.

Why amd64 is slower than arm64: both legs run on runs-on AWS 32-core
instances (m6a.8xlarge and m6g.8xlarge), not on `ubuntu-latest`. The
difference is content, not CPU. amd64 has a larger provider pack with
the Kimi smoke test (37 s), a 187 MB chmod copy layer (arm64: 0 MB), a
4.9 GB image (arm64: 2.9 GB), and a 172 s cache export (arm64: 100 s).

**Proposed behavior**
Run the PID 1 probe in each build leg, on its own architecture, against
the digest that leg pushed. Before signing, require that every Linux
image in the standard index is one of those probed digests. Set the
provider pack modes in the stage that writes the pack.

**Reason and benefit**
The probe pull leaves the post-gate path. The leg has about 3 min of
slack before the gate. The chmod change removes about 50 s and 187 MB
from the amd64 leg, which offsets most of the probe pull that the leg
now does.

**Breaking changes**
None. Tags, digests, attestation subject, and signer identity are
unchanged.

**Additional context**
The concurrency proposal and the build-in-PR plan are below, in "Risks".
They are not implemented.

## What Changed

- `docker.yml` `build-and-push`: new step "Verify PID 1 reaps orphaned
processes". It runs `scripts/assert-orphan-reaping.sh` in
`ghcr.io/...@<pushed digest>` with `--platform` set to the leg platform.
It runs before "Export digest", so `merge-and-push` only receives probed
digests.
- `docker.yml` `merge-and-push`: removed the two `docker run` probes
(tag and digest). "Resolve standard image digest" now runs
`standard-image-contract.mjs --resolve "$GITHUB_SHA" --verified-digests
/tmp/digests`.
- `standard-image-contract.mjs`: `--verified-digests <dir>` reads each
probed leg digest from the registry, checks that its bytes hash to that
digest, and collects its Linux image digests. It fails unless every
Linux image in the standard index is in that set. Registry content is
addressed by digest, so this proves that the signed index names exactly
the probed bytes, without a second pull.
- `Dockerfile`: `chmod -R a+rX /provider-pack` now runs in the same
`RUN` that writes the pack. `COPY` keeps the modes. The production stage
keeps the readability check as UID 65534.
- Tests: new contract tests (membership, bare manifest legs, digest
mismatch, empty set, lookup by digest). New workflow test: the probe
runs between the build and "Export digest", and the publisher does not
`docker run`. Updated the provider-pack Dockerfile test.
- `doc/standard-image-contract.md`: describes the per-leg probe and the
membership check.

Expected savings, per master commit:

| Change | Saving | Confidence |
| --- | --- | --- |
| Probe moved out of `merge-and-push` | 95–100 s from gate to
attestation (measured: 95 s pull + 5 s second probe) | High |
| Pack chmod in its own stage | About 50 s and 187 MB in the amd64 leg
(71 s vs 20.5 s on arm64, where the chmod layer is empty) | Medium-high
on size, medium on seconds |
| Net amd64 leg | About +10 to +45 s (probe pull estimated at 60–95 s,
minus about 50 s). Still before the gate | Medium |

## Verification

- `node --test scripts/standard-image-contract.test.mjs`: 10 pass (2
new).
- `node --test` on
`.github/scripts/tests/{cloud-readiness,docker-canary-promotion,post-merge-runner-routing,lockfile-refresh-workflows}.test.mjs`,
`scripts/__tests__/release-verify-workflow.test.mjs`,
`scripts/reuse-canary-image.test.mjs`,
`scripts/standard-image-contract.test.mjs`,
`scripts/cloud-source-verification.test.mjs`: 484 pass.
- `vitest run server/src/__tests__/docker-build-stamp.test.ts
server/src/__tests__/container-init-reaping.test.ts`: 22 pass. Red
check: with the `origin/master` `Dockerfile` and `docker.yml`, the 2 new
or changed assertions fail.
- Real registry, read only: `node scripts/standard-image-contract.mjs
--resolve 42fd790e80 --verified-digests
<dir>` with the two leg digests from run 43133035209 returns the
published index digest. With only the arm64 leg digest, it fails:
"linux/amd64 image sha256:0f0642e6… did not pass the build-leg
orphan-reaping probe".
- `actionlint .github/workflows/docker.yml`: the same 6 shellcheck
findings as `origin/master`, none new.
- Shell syntax check (`sh -n`) of the changed pack `RUN`.

Not exercised: the workflow itself (it runs only on master pushes, and I
did not dispatch it). No local `docker build`: the Docker daemon is not
running on this machine, and a full build does not fit a few minutes.
The leg probe pull time is an estimate from the publisher runner.

## Risks

- The leg now pulls its own image (4.9 GB amd64). AWS legs have about
175 GB free. Hosted tag builds run "Free runner disk" first, and the old
publisher pulled the same image on a hosted runner for tag builds.
- If the registry returns a leg index in an unexpected form, the resolve
step fails closed and no master image is signed. The real-registry check
above used today's exact shapes (OCI index with one platform image and
one attestation manifest per leg).
- Tag builds (`v*`, `nightly/v*`, `beta/v*`) keep the probe, now per
leg. They have no standard-image signing step, and the manifest list is
made only from probed digests.

### Deliberately not changed

- **The exact-source gate.** It is a deliberate "never publish an
unverified image" rule (`watch-release-source.mjs --require-proof` and
`--check-proof`, `doc/cloud-build-readiness.md`). I kept it. The owner
levers are: the slowest `cloud-readiness` shard (`server (7/10)`, 12
min, sets the gate), the 20 s runner wait for `Cloud source verified
v1`, and the watcher's 30 s poll (15 s average delay). I did not shorten
the poll because the org API rate limit was exhausted today.
- **Cache export (172 s amd64, 100 s arm64).** `mode=max` compresses
about 2 GB of per-commit pack layers every build. A fix: a separate
cache-only build of a small aggregate stage (`deps`, `runner-build`, the
production tool layer) with `mode=max`, and the main build with
`cache-from` only. Not done here: the legs are not on the master
critical path today, and I cannot measure it without a master run. It is
worth doing for tag builds or if the gate gets faster.
- **Cached layer pull (46 s).** The `build` stage starts `FROM
runner-build`, so it downloads about 650 MB of Rust `target/` that it
deletes at the end. The server build's `prepare:runner-vendor` uses that
target, so this needs a separate change.
- **Provider pack per commit (98 s amd64).** It rebuilds because it is
`FROM build` and stamps the commit. Its 1.06 GB transport archives also
duplicate the 1.01 GB pack inside the image. Keying the pack on its own
inputs is a product-contract change.

### Concurrency proposal (not implemented)

Today one run holds `docker-${{ github.ref }}` for 17–23 min. Most of
that is waiting for the gate, not building. A newer commit waits for
that run before its build starts (run 43134250912 waited 6.4 min).
Proposal:

1. Remove the workflow-level group. Add job-level groups:
`build-and-push` uses `docker-build-${{ github.ref }}-${{ matrix.arch
}}` with `cancel-in-progress: false`. At most one build per architecture
runs and the newest commit replaces the pending one. So not every commit
is built.
2. `merge-and-push` uses `docker-publish-${{ github.ref }}` with
`cancel-in-progress: false`. It skips publication when a newer master
commit already has a signed image, so the image order stays monotonic.
3. A superseded run must not keep its source watcher alive, or the API
load grows with each push. Fold the early watcher into the gate job, or
end it when its build job is cancelled.

Effect: the newest commit starts building at push, so its image
publishes at about gate time plus 45 s, instead of after the previous
run ends. That removes the 0–17 min queue wait, which is most of the gap
between the p50 of 29 min and the gate time of about 14 min.

### Plan: build in PR CI and promote on merge (not in this PR)

What must change:
1. Turn on the GitHub merge queue for `master`. The queue tests a
`gh-readonly-queue/...` commit and then fast-forwards `master` to that
exact commit, so the SHA stamped in the image (`PAPERCLIP_BUILD_COMMIT`,
`build-info.json`, the provider pack `runnerSourceRevision`, the UI
bundle) is the final master SHA.
2. On `merge_group`: build both architectures with **no secrets and no
registry credentials**, because the code under test is not yet on
master. Write each image as an OCI layout artifact. A second job that
runs no PR code pushes those artifacts by digest to a staging
repository, keyed by the merge-group SHA.
3. Run `cloud-readiness` on `merge_group` for the same SHA, so
exact-source proof exists before merge.
4. On the master push, `docker.yml` looks up the staged digests for
`github.sha`, checks that the tree and SHA match, retags them as
`sha-<full>`, and signs with the same identity
(`docker.yml@refs/heads/master`). If no staged image exists, it falls
back to the current build.

Guarantees to keep: never publish or sign an image without exact-source
proof for that SHA. The signer identity and the attested digest stay the
same. No secrets in any job that runs non-CODEOWNER code. The PID 1
probe runs on both architectures before signing. Superseded queue
entries do not publish.

Expected saving: from master push to signed image, about 1–2 min
(lookup, retag, attest) instead of 29 min p50 and 56 min p90 today. The
work moves before the merge, so the merge itself waits for the queue
(about the current gate time of 14 min). A staging fix then reaches
Cloud about 25 min sooner at the median.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used via Claude Code
with tool use (shell, file edits, read-only GitHub API and registry
reads). Default reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:29:11 -06:00
Nicky LeachandClaude Opus 5.5 ff8ec41e6a feat(runner): upgrade ACPX to 0.22.0 and move patched behavior into Paperclip (#20770)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Runner and the shared `adapter-utils` engine run ACP
agents through the embedded ACPX library
> - Paperclip pins ACPX 0.13.1 and keeps a large patch
(`patches/acpx@0.13.1.patch`) on it
> - ACPX 0.22.0 adds host hooks (`onAcpMessage`, `assertActive`,
`processLifecycle`, prompt-response `_meta`, `spawnAgent`-friendly
launch) that make much of that patch unnecessary
> - A large patch is expensive to port and hides behavior that belongs
to Paperclip
> - This pull request upgrades to ACPX 0.22.0 and moves most patched
behavior into Runner and `adapter-utils`
> - The benefit is a patch that is 484 lines instead of 1,334, with the
moved behavior covered by Paperclip tests

## Linked Issues or Issue Description

Refs #20675, #14430, #15075, #17942, #9980

**Proposed enhancement**
Upgrade the embedded ACPX library from 0.13.1 to 0.22.0. Keep only the
patch behavior that stock 0.22 cannot express.

**Motivation**
The 0.13.1 patch held behavior that Paperclip owns, for example usage
receipts, Copilot policy, model checks and the agent environment. Each
ACPX upgrade had to port all of it. ACPX 0.22 gives hooks that let
Paperclip do this work itself.

**Current behavior**
Runner and `adapter-utils` depend on 1,334 lines of patched ACPX 0.13.1
code.

**Expected behavior**
Runner and `adapter-utils` use ACPX 0.22.0. The patch keeps only:
extension methods, Cursor subagent routing, event `_meta` and the
message boundary chunk, the protocol guard hook, `spawnAgent`, and
`requestExtension`.

## What Changed

- ACPX 0.13.1 → 0.22.0 in Runner, `adapter-utils`, the server, the
Daytona runner images and the qualified profiles (`acpxVersion`).
`@agentclientprotocol/sdk` moves to 1.9.0.
- New `patches/acpx@0.22.0.patch` (484 lines). The 0.12.0 and 0.13.1
patches are removed.
- Delivery receipts: Runner confirms a permission reply when
`onAcpMessage` sees the exact outbound reply. Question and extension
replies count as delivered when Runner hands them back to ACPX. New
`interaction-delivery.ts`.
- Copilot policy: the existing policy function runs from `onAcpMessage`.
A violation stops the agent, cancels later permission requests and fails
the next prompt through `assertActive`.
- Model selection: stock 0.22 validation, plus a sidecar gate
(`createAcpxEffectiveModelGate`) that refuses a prompt if the agent
reports another model. Grok binds its model when the session opens.
Antigravity adds its concrete model to the model list ACPX reads.
- Typed session failures: Runner and `adapter-utils` read them from the
turn result `_meta` and fail the turn. They advertise the capability
themselves.
- Pi and Cursor usage receipts: a collector (`prompt-receipts.ts`) binds
them to the turn's own `session/prompt` reply.
- Goal capability: Runner reads it from the `initialize` reply. It keeps
it under `paperclip.goalCapability` in the saved agent capabilities
`_meta` and restores it when a saved session loads.
- Agent environment: Runner and `adapter-utils` set the agent
environment in `spawnAgent`. ACP terminals keep their current
environment. `adapter-utils` blocks inheritance of `XAI_API_KEY` and
refuses to start if the server has an `ACPX_AUTH_*` variable.
- `adapter-utils` spawn hooks (`spawnCwd`, stderr, `onAgentSpawn`): a
new `stockAcpRuntimeOptions` maps them to stock `spawnAgent` and
`processLifecycle`.
- Windows: ACPX 0.22 rejects raw command strings on Windows.
`adapter-utils` now passes an argv array there.
- Grok rules: a Runner guard (`grok-session-rules.ts`) writes
`session/new` `_meta.rules`.
- Turn-dispatch journal (#20675): ACPX 0.22 ignores errors from
`onAcpMessage`. The adapter keeps a failed `dispatched` write and throws
it from the turn's `assertActive`. ACPX calls `assertActive` before the
prompt write, so the write still stops. The real-ACPX test now checks
this route. `docs/durable-recovery.md` is updated.
- Plan events: the sidecar reads 0.22 `status` events tagged `plan`.
- CI: the backup-checkpoint lane now installs through the shared
`setup-ci-dependencies` action, like PR CI, so it resolves a stale
`pnpm-lock.yaml` in place. It runs on any root `package.json` change,
and pull requests never commit the lockfile, so its own frozen install
failed on dependency changes.
- Tests that checked removed patch internals are removed or rewritten:
the response-delivery, Cursor-usage and Pi-receipt package contracts,
and the goal-persistence packaging test. New unit tests cover the moved
behavior.

## Verification

Local runs on macOS (darwin-arm64), Node 26, on the rebased head:

- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/drivers/acpx src/cli`: 2,000 passed.
- `tsc -p packages/paperclip-runner/tsconfig.json` and
`tsconfig.test.json`: no errors, except the existing
`@paperclipai/paperclip-eval-kernel` build gap in `workflow-harness.ts`.
- Runner package contracts (`node --test
test/acpx-*-package-contract.test.mjs
test/acpx-sidecar-contract.test.mjs`), `scripts/build-inputs.test.mjs`,
`scripts/contracts/contract-fingerprints.test.mjs`,
`scripts/acpx-patch-packaging.test.mjs`: all pass. `node
scripts/generate-acpx-sidecar-contract.mjs --check` passes.
- `cargo test -p paperclip-runner-core --test acpx_park_preparation
--test acpx_park_readiness --test acpx_warm_attachment --test
native_provider_backend`: 86 passed. `cargo fmt --check` passes.
- `adapter-utils`: `tsc` clean. `vitest run src/acpx-engine
src/mcp-isolation.integration.test.ts`: 457 passed (one Gemini flag test
failed once under load and passes when run alone).
- Adapters: `claude-local` and `codex-local` `acp.quota.test.ts` (23 and
39 passed), `grok-local` (149 passed).
- Server: `scripts/prepare-runner-vendor.test.mjs`,
`native-session-executor.test.ts`, `quarantined-task-retry.test.ts`
(1,416 passed).
- `cargo test -p paperclip-runner-core --lib -- previous_acpx_release
acpx_upgrade unrestorable_saved_sessions`: 3 passed. `node --test
test/acpx-session-carry-forward-package-contract.test.mjs`: passes.
- `patch -p1 --forward` on a clean `acpx-0.22.0.tgz` gives output
identical to the port.
- Under full-suite load, some Runner live and recovery tests time out.
Each one passes when run alone. Full regression is left to CI.

Not run: live provider runs (they need provider credentials), the opt-in
Kimi live peer test, the Gemini filesystem smoke test, and Windows.

## Risks

- Previous-release story: sessions saved by the previous release (ACPX
0.13.1) carry forward. ACPX 0.22 loads the session records 0.13.1 wrote.
Runner moves a saved descriptor that names 0.13.1
(`CARRIED_ACPX_RELEASES`) to 0.22.0, and the sidecar keeps 0.13.1 as the
ACPX value in the session identity, so the saved session key and
identity record still match. The existing launch-profile rebind still
decides which sessions resume: a settled session, or the same run's
interrupted turn through lost-turn recovery. A session saved by any
other ACPX release is still refused as `provider_state_incompatible`.
`rules/acpx-profile-identity.mdc` and `docs/durable-recovery.md` now
describe this.
- Proof: `acpx-session-carry-forward-package-contract.test.mjs` resumes
a session record written by the real ACPX 0.13.1 with its Paperclip
patch. 0.22 loads the same provider session (no `session/new`), runs a
turn and keeps the earlier turn. Rust tests cover a settled 0.13.1
session that the next run attaches to, and a turn interrupted by the
upgrade. A recovery-identity test checks that the session key and
identity record match across the versions.
- `pnpm-lock.yaml` is not committed. CI resolves it for each run, and
the lockfile refresh workflow updates it after merge.
- Behavior change on the server: `adapter-utils` refuses to create an
ACPX runtime if the server environment has an `ACPX_AUTH_*` variable.
Before, ACPX ignored such variables. Stock 0.22 would use them to
authenticate agents with server credentials.
- Delivery confirmation is now Runner-owned. A permission reply is
confirmed by the exact outbound message. Question and extension replies
are confirmed when Runner hands them to ACPX, not when the pipe accepts
them.
- Users can select only models that the agent advertises. Stock 0.22
rejects other model IDs (Claude is exempt).
- ACP terminals keep the sidecar or server environment, as before. This
existing exposure is not changed by this pull request.
- The `adapter-utils` Gemini version probe now runs with the server
environment plus the run variables, not the run variables alone.
- The pack contract digest and profile `acpxVersion` change, so Daytona
needs a new runner image.
- Grok launch flags (`--model`, `--reasoning-effort`) were checked on
1.0.13 but session open needs a login, so Grok keeps the existing
`_meta` path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool
use (shell, file editing, subagents). Extended reasoning was used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 07:28:00 -08:00
DottaandClaude Opus 5.5 e2cf9a670f fix(runner-e2e): resync SDK public lock pins and make the lockfile bot keep them current (#20623)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner support suite proves the plugin SDK public types from
pinned sources in `tests/runner-e2e/source-sdk-public-inputs.json`
> - That registry pins the committed `pnpm-lock.yaml` blob and a
`lockSha256` that must equal the `outputSha256` in
`tests/runner-e2e/report-lock.json`
> - The lockfile bot refresh #20615 changed the lock and
`report-lock.json`, but the bot does not write the SDK registry
> - The bot merges without CI, so the drift guard from #20579 could not
block it, and every PR now fails `ci / Runner support suite`
> - The guard also checked only the source blobs, so it did not name the
`lockSha256` drift, and the sync script could not update `lockSha256`
> - This pull request regenerates the registry, derives `lockSha256`
from the reviewed policy, and makes the bot resync and check the
registry in its own commit
> - The benefit is that master CI is green again, and the next lockfile
refresh cannot leave the SDK proof stale

## Linked Issues or Issue Description

Refs #20615. Refs #20579. Refs #20550. Refs #20613. Overlaps #20618,
which refreshes only the registry data. This PR has the same data change
and also closes the gap that let the drift land.

**What happened?**

After #20615 merged, `ci / Runner support suite` failed on master and on
every PR (for example runs 42877750315 and 42877750957, and PR #20613).
`source-sdk-public-type.test.ts` and `source-sdk-evidence.test.ts`
failed with "Reviewed SDK reference policy does not match public proof".
`source-sdk-public-inputs.test.mjs` failed with "pnpm-lock.yaml
(changed)".

**Expected behavior**

A lockfile refresh keeps the SDK public proof registry in step with the
lock and the reviewed report-lock policy. If the registry drifts, the
failure names the stale pin and the fix command.

**Steps to reproduce**

1. Check out master at `3bf7af6c77`.
2. Run `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
source-sdk`.
3. See the "Reviewed SDK reference policy does not match public proof"
failure.

## What Changed

- Root cause: #20615 changed `report-lock.json` from output `2360d5e9…`
to `b3208306…` and changed the `pnpm-lock.yaml` blob from `851e65af…` to
`6d825f97…`. The registry kept the old values. `refresh-lockfile.yml`
commits only the lockfiles and the report-lock proof, and it merges
without CI.
- `source-sdk-public-inputs.json`: regenerated with `node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`. Only
`inputs["pnpm-lock.yaml"]` and `lockSha256` change. The compiler, parser
and Zod pins do not change.
- `source-sdk-public-inputs.mjs`: the sync now copies `lockSha256` from
the committed `report-lock.json` `outputSha256`. It refuses when the
committed lock is not that policy's reviewed input or output. This is
the same rule that `prepareReviewedSdkReference` applies. Only the
lockfile bot can write that policy, so a contributor cannot choose
`lockSha256`.
- `staleSdkPublicInputs()`: also reports a `lockSha256` that differs
from the committed report-lock output. The guard test then names both
stale pins and the `--write` command.
- `source-sdk-reviewed-fixture.ts`: the "Reviewed SDK reference policy
does not match public proof" error now names the two values and the fix
command. The check itself is unchanged.
- `refresh-lockfile.yml`: after the lock commit, the bot runs the SDK
sync with `--write`, amends the commit, and runs `--check` before it
pushes. `head_sha` is still read after the amend, so the bot merges the
exact commit it checked.
- Tests: new sync tests for a lockfile refresh and for an unreviewed
lock, and a workflow-order test in `lockfile-refresh-merge.test.mjs`.
- Docs: `doc/RELEASE-AUTOMATION-SETUP.md` §8.1 and
`tests/runner-e2e/OPERATIONS.md` describe the resync.

## Verification

- Reproduced on master: `source-sdk-public-type.test.ts` and
`source-sdk-evidence.test.ts` failed with the policy mismatch.
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
source-sdk prepare-report-lock sdk-canary workflow-security`: 9 files,
135 tests pass.
- Before the registry change, the extended guard reported
`pnpm-lock.yaml (changed), tests/runner-e2e/report-lock.json
outputSha256 (lockSha256 changed)` with the fix command.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse
HEAD) --check` reports `changed: false`.
- `node --test .github/scripts/tests/lockfile-refresh-merge.test.mjs
.github/scripts/tests/lockfile-refresh-workflows.test.mjs
.github/scripts/tests/lockfile-trust.test.mjs`: 39 tests pass.
- `pnpm test:e2e:runner:unit` on a host with load near 28: 6 files timed
out, and none of them are the SDK proof files. All 15 of those files
pass when run alone. CI runs the full suite.

## Risks

- The bot step now runs one more repository script. If that script
refuses (for example, an unreviewed committed lock), the refresh PR is
not pushed. Before this change, the bot merged a refresh that made
master red.
- `lockSha256` now comes from the reviewed policy, not from the old
registry value. The proof check is unchanged. It still needs the lock
bytes to hash to `lockSha256`.
- If #20618 merges first, this branch rebases cleanly, because the
registry change is identical.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit) and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 06:57:51 -08:00
DottaandPaperclip 81c9143fe4 fix: upgrade Better Auth to 1.7.16 with account migration (#18906)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - The board uses Better Auth for credential accounts and sessions.
> - The server pins Better Auth 1.7.2.
> - The November security release fixes 12 issues in Better Auth.
> - The new version stops writing the old required account issuer field.
> - This change updates the packages and the database contract together.
> - New users can sign up, and existing users can still sign in.

## Linked Issues or Issue Description

**What happened?**
The server uses Better Auth 1.7.2. The [security
release](https://better-auth.com/blog/security-release-2026-11-03) fixes
12 issues in 1.7.16. Earlier update PRs #17909 and #17911 target 1.7.7,
which is below this security release.

**Expected behavior**
Use the patched stable release. Keep new and existing credential
accounts usable.

**Steps to reproduce**
Read the server dependency pin. Upgrade it without a schema change.
Better Auth then rejects the required account issuer column. See the
[upgrade guide](https://better-auth.com/docs/guides/1-7-upgrade-guide).

## What Changed

- Pin Better Auth to 1.7.16. The repository refresh bot owns the
committed Core lockfile.
- Make the old issuer column nullable. Keep existing issuer values.
- Replace the issuer index with a unique provider/account index.
- Generate the migration and retain the five newest schema snapshots.
- Test new signup, new-account sign-in, legacy issuer sign-in, duplicate
rejection, and separate provider identities with real PostgreSQL.
- Update the database schema test. Test the historical issuer backfill
separately. Verify upgrades over existing rows and rollback on duplicate
provider/account keys.
- Document the duplicate check and migration order.
- Add the approved two-line dependency resolution step before the backup
job’s frozen install. Leave the frozen check and backup tests intact.

## Verification

- Core head: `af70282c7e24335b8688c8f3cc88fdf65045632c`; parent
`0f1481e4462100357a7caa496b875cf0bc500d95` includes master `84fc67317`.
- Nine changed files. The only workflow edit is the approved two-line
backup dependency resolution step; no managed lockfile edits.
- Migration `0465_better_auth_account_identity.sql` is three SQL
statements; snapshot changes only `public.account`, five newest
snapshots retained.
- DB typecheck and 23 focused tests passed on the parent. Both real
PostgreSQL migration/signup suites were rerun November 11: all 4 tests
passed. Application source and tests are unchanged in this workflow-only
commit.
- The parent’s sandbox-provider lane passed all 205 file-sync tests but
reported an unhandled EPIPE during the symlink rejection case; its
aggregate verify therefore failed. No unrelated source change was made.
The new head reruns the full applicable CI.
- Reproduced the old backup install failure locally. Resolving manifests
followed by a frozen install passed locally; restored the bot-owned
committed lock afterward. Hosted backup qualification remains required.
- Current-head CI and review are running; no final-head green claim or
merge yet.

## Risks

Apply the migration before the new server starts. Check for duplicate
provider/account pairs first. Duplicate rows stop index creation. Keep
separate users separate when resolving duplicates. The migration keeps
issuer data. Do not restore its NOT NULL constraint while new rows
contain null. No deployment is included.

## Model Used

OpenAI GPT-6 Codex, with reasoning and tool use. The runtime did not
expose a more specific model ID or context window.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-11 08:26:58 -06:00
DottaandPaperclip 282503b5b3 fix(ci): prepare native binaries for Runner chaos checks
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Release verification checks that Runner preserves work through
faults and shutdown.
> - A governed-wait recovery case now runs a real native Runner and a
controlled Codex process.
> - That case needs the workspace debug binaries before the chaos suite
starts.
> - The chaos workflow builds JavaScript contracts but does not build
those native binaries.
> - A clean release worker fails before it can verify recovery.
> - This pull request builds the native prerequisites with the existing
package command before the tests.

## Linked Issues or Issue Description

Refs #20336, which added the real-Runner recovery case. Refs #20322,
whose normal delivery exposed the missing prerequisite. Existing native
test preparation and related chaos PRs were checked; no duplicate
preparation repair was found.

**What happened?**

The [Cloud readiness chaos
job](https://github.com/paperclipai/paperclip/actions/runs/42686564449/job/129397518634)
failed on Core `c67909cb0f1fa973e0991ba068d9c8b2f8f64924`. The
governed-wait recovery test tried to open
`packages/paperclip-runner/runner/target/debug/paperclip-runnerd`, which
did not exist. The job had installed dependencies and built eval
contracts. It had not built native workspace binaries.

**Expected behavior**

The chaos job builds its native prerequisites before it starts the
Runner fault tests. A build failure must fail the job. The existing
recovery assertions must still execute.

**Steps to reproduce**

1. Use a source checkout with JavaScript dependencies and contracts
prepared but no Rust debug workspace binaries.
2. Run `src/native-session-runtime.test.ts` with the test name `settles
a governed wait through real Runner shutdown after a missing Codex
terminal`.
3. Observe a missing native artifact. On the hosted job it was the
daemon. In the local reproduction, canonical JavaScript preflight had
already built a release daemon, so the missing artifact was debug
`fake-codex-app-server`.
4. Build the native workspace binaries and run the unchanged case again.

**Paperclip version or commit**

Hosted failure: `c67909cb0f1fa973e0991ba068d9c8b2f8f64924`. This branch
is based on fresh master `8a6b08227484591a30a7adea3840f7e96755ed81`.

**Deployment mode**

GitHub Actions source readiness and reusable Runner chaos verification.

## What Changed

- Run the existing `pnpm --filter @paperclipai/paperclip-runner
build:rust` command before the Runner fault suites. This builds the
locked native workspace binaries, including the daemon and controlled
Codex server.
- Extend the existing release-workflow contract tests to require
unconditional native preparation after dependency installation and
before the real-Runner suite.

## Verification

- The new workflow contract failed against the previous job setup. It
passed after adding the prerequisite.
- The unchanged governed-wait case failed with a missing debug
fake-Codex binary before native preparation: 1 failed, 251 intentionally
skipped, 2.92 seconds.
- Complete canonical worktree preflight prepared the debug binaries in
10.8 seconds. The unchanged real-Runner case then passed: 1 passed, 251
intentionally skipped, 2.03 seconds. It verifies saved work, accepted
handoff, confirmed provider exit, durable state and a single admitted
turn.
- After rebasing onto refreshed master, the exact case passed again in
**1.84 seconds**. The three affected workflow contracts also passed.
- **412 existing workflow, routing and native-cache contract tests
passed** in 26.58 seconds. Their input files did not change in the
rebase.
- `actionlint -oneline -shellcheck= -pyflakes=` and `git diff --check`
passed.
- The test uses the real local Rust Runner and a controlled local
provider. It does not call a live model or modify a hosted instance.
- The full local chaos suite was not run. Hosted execution of the
changed reusable chaos job remains a postmerge check.
- Required CI for final head `dd751732bff287dbeb05ddadd25c3370c817dd7a`:
[workflow attempt
2](https://github.com/paperclipai/paperclip/actions/runs/42689605454/attempts/2)
**passed** at 20:58:20 UTC on November 9: all 53 jobs accounted for, 52
successful and 1 skipped. The fresh [Build
job](https://github.com/paperclipai/paperclip/actions/runs/42689605454/job/129419966076),
[server shard
4](https://github.com/paperclipai/paperclip/actions/runs/42689605454/job/129419970716),
and [final verify
aggregate](https://github.com/paperclipai/paperclip/actions/runs/42689605454/job/129431023346)
all passed. The exact-head check rollup and classic Snyk status are
successful, with no failures or checks still pending.
- Attempt 1 remains a nonpass: the clean public npm install requested an
AWS SDK package version before that version was published, and server
shard 4 exceeded an existing five-second completion poll before its test
retry collided with a retained fixed-name environment fixture. The
unchanged exact server case passed locally. One bounded failed-job retry
reran both failed leaves; neither the fixture nor assertions were
changed. The initial cause of the five-second completion miss has not
been established.
- Fresh exact-head Greptile review is **5/5** with no open inline
threads. Ponytail is **4/5, Ship**, with two explicitly nonblocking
suggestions: a separate native-build step deadline and tighter
workflow-job scoping in the contract test. The native build remains
bounded by the existing job deadline; the current contract proves
prerequisite ordering in the affected workflow. Both suggestions are
retained for follow-up rather than changing this final tested revision.

## Risks

- A cold native build adds compilation time and disk use to the chaos
worker. The existing job timeout remains 40 minutes on the protected AWS
route and 45 minutes on the hosted route.
- Native build failures stop the job. The test selection, assertions,
permissions and source-proof gates remain in force.
- This change addresses the observed missing artifacts. Earlier
runner-loss waves and the separate managed-credential assertion remain
nonpasses; this change does not establish their causes or repair them.

## Model Used

OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository
tools and code execution. Context-window size is not exposed by the
runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 15:00:19 -06:00
DottaandPaperclip 3a2f71db31 fix(ci): handle missing runner output and await import fixture readiness (#20339)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - CI checks must report real result failures after their setup phase.
> - Required summaries must run even when the routing gate has no runner
output.
> - Process-recovery tests must start their checkpoint deadline after
the child loads its modules.
> - An empty runner prevents summaries from starting. Slow child startup
can consume the checkpoint deadline.
> - This pull request adds a fallback for the two summaries and a ready
signal for the recovery fixture.
> - Existing routing, checkpoint, key and recovery assertions remain in
place.

## Linked Issues or Issue Description

Refs #20322. Related open PRs #12512 and #12513 propose a broader runner
contract and caller activation. This change does not adopt that contract
or change normal routing. No current repair for an empty aggregate
runner was found.

**What happened?**

The PR workflow can fail before starting any jobs when the routing gate
has no runner output. GitHub reports an empty `runs-on` value for the
`verify` and `e2e` summary jobs. Their `always()` conditions make this
path possible when a dependency cannot run. A real workflow rerun showed
both errors at lines 801 and 1784 of the trusted workflow.

The first run of this PR also failed two process-recovery cases with no
saved child checkpoints. The fixture starts its 15-second checkpoint
poll before a new Node process imports the server modules. A controlled
16-second pre-import delay reproduces the same zero-checkpoint
assertion. That evidence proves a deadline-boundary defect. The original
CI run has no phase events, so it does not prove which stage caused
those two CI failures.

**Expected behavior**

The summaries can start on `ubuntu-latest` when routing has no runner
output. They must still fail when routing or required work did not
succeed. The process fixture waits for a bounded child-ready signal
before its unchanged checkpoint deadline. It must still prove checkpoint
reuse, process termination and refusal under a changed master key.

**Steps to reproduce**

1. Start or rerun PR CI with no runner output from the routing gate.
2. Let the two `always()` summary jobs evaluate their runner
expressions.
3. Observe that the previous expressions produce an empty value and
GitHub refuses the jobs.
4. For the fixture defect, temporarily delay the child before imports
for 16 seconds. The previous fixture fails its 15-second checkpoint poll
before the child is ready. The updated fixture passes with the same
delay. The temporary delay is excluded from this PR.

**Paperclip version or commit**

Reproduced with the workflow from
`f56b097fbd25607b3b84e418085ab42c2cbb758d`. The caller and both reusable
workflow files are byte-identical between that revision and the affected
PR head. This fix is based on current master
`4a349ea44754f0b1fb8e5e04dc6cdcb05d1c3396`.

**Deployment mode**

GitHub Actions PR CI.

## What Changed

- Add `ubuntu-latest` as the fallback only for `verify` and `e2e` when
the routing runner is empty.
- Extend the existing workflow contract test to require a usable
fallback and execute each real summary script with a failed, cancelled
or skipped gate and absent routing outputs.
- Wait up to 30 seconds for the recovery child to finish imports and
construct storage before starting the existing 15-second checkpoint
poll. Keep the 30-second work abort, 60-second test timeout, and every
kill, key, state and proof assertion.
- Preserve normal runner selection, task selection, permissions and all
result assertions. No production code, public command or product
contract changes.

## Verification

- The new local workflow contract failed before the two
runner-expression changes. The existing summary scripts rejected failed
routing while the test identified the missing fallback.
- **148 focused tests passed** across CI planning, routing and E2E
sharding in 16.00 seconds. These execute the actual summary Node scripts
and preserve failure and cancellation checks.
- `actionlint -oneline -shellcheck= -pyflakes=` passed for the trusted
workflow, routing workflow and caller. `git diff --check` passed.
- The canonical worktree preflight completed with the pinned toolchain
and shared build cache.
- This test does not implement a GitHub expression evaluator. The real
GitHub startup error establishes the absent-output failure; the source
contract and actionlint validate the proposed expression. Hosted
execution of this fallback remains to be verified after the trusted
workflow reaches master.
- The unchanged two process cases passed locally before the fixture
change: 2 passed in 31.06 seconds. A controlled 16-second startup delay
then reproduced the zero-checkpoint failure in 21.81 seconds. The ready
handshake passed with the same delay in 37.11 seconds. The temporary
delay and phase logging were removed. These are measured test durations,
not a measurement of the original CI child startup.
- The final full nested-verification fixture passed: **38/38 cases in
132.30 seconds**, with disposable local PostgreSQL. This includes
partial checkpoint reuse, changed-key refusal and all existing recovery
controls.
- Fresh final head `da35ea034bdee603c021c4eba5b0f78ee6ef52d9`: the [full
PR
workflow](https://github.com/paperclipai/paperclip/actions/runs/42678538191),
[final ci /
verify](https://github.com/paperclipai/paperclip/actions/runs/42678538191/job/129374227690)
and [incremental
qualification](https://github.com/paperclipai/paperclip/actions/runs/42678537930)
passed. The [previously failing server
lane](https://github.com/paperclipai/paperclip/actions/runs/42678538191/job/129366881302)
passed. The final rollup has 61 successful checks, 6 expected skips, 2
neutral checks and no pending checks or failures. Snyk passed.
-
[Greptile](https://github.com/paperclipai/paperclip/pull/20339#issuecomment-6748154727)
and
[Ponytail](https://github.com/paperclipai/paperclip/pull/20339#issuecomment-6748242241)
both reviewed that exact final head at 5/5. There are no inline review
threads. Ponytail permits retaining the source check with actionlint,
which is the validation used here.
- The [first workflow
run](https://github.com/paperclipai/paperclip/actions/runs/42674612619)
failed the two cases described above; it is not counted as a pass. No CI
retry was used to obtain the final result. CI uses the merged trusted
workflow; its policy tests validate this proposed source file.

## Risks

- A routing failure now produces a normal failed summary instead of an
invalid runner expression. The unchanged assertions require successful
routing, policy, valid scope and selected suite results.
- Only lightweight summary jobs can fall back to GitHub-hosted capacity.
Heavy jobs and authorization rules retain their existing route.
- The new readiness allowance is bounded. A child that does not become
ready, fails its work deadline, or cannot produce the expected
checkpoint still fails. The original CI failure cause remains unproved
beyond the observed zero-checkpoint assertions.
- The fallback assertion is an exact source contract. A formatting
change can require an assertion update. The separate actionlint check
validates expression syntax; the text check does not emulate GitHub.
- This does not repair a GitHub internal error or runner interruption.
Those failures remain nonpasses.

## Model Used

OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository
tools and code execution. Context-window size is not exposed by the
runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 13:14:16 -06:00
DottaandClaude Opus 5.5 347d21fcbd test(runner-e2e): scope each agent's sandbox to its own run's saved state (#20253)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The runner E2E chat evals run real agents against a local Paperclip
instance on a CI host
> - PR #20044 hides the eval harness from local native agents with a
Linux Landlock domain around each runnerd launch
> - That domain is the same for every run in an attempt, so an eval
agent can still read another run's saved state
> - Recent chat-eval runs show agents that read other runs' OpenCode
session databases, another run's run log, the instance `pg_wal`,
`control-plane-state.json`, and `/proc/<pid>/cmdline` of harness
processes
> - This pull request scopes each runnerd launch to its own run and
moves the selected case id off the command line
> - The benefit is that an eval agent cannot learn about the case or
other runs from files outside its own run

## Linked Issues or Issue Description

Refs #20044

**What happened**

In local native OpenCode chat-eval cells, worker agents read files that
belong to other runs or to the harness:

- other runs' OpenCode sqlite session databases under
`runtime/paperclip-runner/durable-sessions/` (a status-cards run)
- `/proc/<pid>/cmdline` of harness processes, which shows harness paths
and the case id (`routines.pause-owned`)
- another run's run-log ndjson, and a listing of the instance
`db/pg_wal` (`routines.pause-owned`)
- a listing of `control-plane-state.json`
(`runs-adapters.list-model-options`)

The Landlock sandbox from #20044 hides only the harness (`tests/`, the
Git store, private attempt directories). All runnerd launches share one
hidden set, so the instance's per-run directories stay readable.

**Expected behavior**

An agent process sees only its own run's durable session, OpenCode
session and provider trace. Other runs' sessions, run logs and traces,
the instance database files, secrets and the shared control-plane state
in the attempt instance are denied. The case id is not on any harness
command line.

**Steps to reproduce**

1. Dispatch a local native OpenCode chat-features cell that has two
agents (for example a status-cards case).
2. In the worker's shell, run `ls
<instance>/runtime/paperclip-runner/durable-sessions/*/` and `ls
<instance>/db/pg_wal`.
3. Run `cat /proc/<launcher pid>/cmdline`.
4. Before this change, all three succeed and show other runs' files and
`--id chat-features...`.

## What Changed

- `agent-filesystem-sandbox.py`: new `--expose PATH` option. It grants a
path beneath a hidden directory. Landlock does not check traversal, so
the run uses its own entry while the parent stays unlistable. The helper
refuses a missing exposure, an exposure outside every hidden path, or
one that would reveal a hidden path. It fails the launch unless every
exposed path is readable.
- `agent-filesystem-sandbox.ts`: the launcher passes the instance root
in the sandbox configuration. `runnerdLaunchSandbox` adds per-launch
hidden paths: `durable-sessions/`, `runner/`, `control-plane/`,
`opencode/` under `runtime/paperclip-runner/`, and `data/run-logs/`,
`data/provider-traces/`, `data/backups/`, `db/`, `secrets/`. It exposes
only the launch's own durable-session directory (from `--state-dir`),
Codex runner state directory, OpenCode session directory (from
`--session-id`, named as the OpenCode driver names it) and provider
trace file.
- The wrapper creates the exposed entries, and the shared instance
directories that the product creates on first use, before runnerd
starts. It logs the run-scoped hidden and exposed paths for each launch.
- New `probeRunScopedSandbox`: at server start, a sandboxed shell runs
against a fixture instance. Another run's session, OpenCode database,
runner state, run log, trace, `pg_wal`, secrets, control-plane state and
the server's `/proc/<pid>/environ` must be denied. The run's own
session, OpenCode directory, ACPX directory and trace must be writable.
In CI any failure stops the attempt. The result goes to `server.log`.
- `selectors.ts` and `launch.ts`: the launcher reads the selected cell
from `PAPERCLIP_RUNNER_E2E_SELECTED_ID` and removes it from its
environment.
- `runner-full-stack-e2e.yml` (Run paid cell) and
`chat-catalog-testops-case.yml`: pass the cell in that variable instead
of `--id`. A target branch that does not read the variable still gets
`--id`.
- `SECURITY.md`: documents the run scope, the `/proc` limit and what
stays readable.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/agent-filesystem-sandbox.test.ts
tests/runner-e2e/workflow-security.test.ts
tests/runner-e2e/manual-private-workflow.test.ts
tests/runner-e2e/private-trust-workflow-security.test.ts` passes on
macOS. The Landlock host tests (including the new fixture-instance probe
test) run only on Linux CI.
- `npx tsc --noEmit -p tests/runner-e2e` shows no errors in the changed
files.
- Paid run
[42611358428](https://github.com/paperclipai/paperclip/actions/runs/42611358428):
`chat-features.runner-opencode.local.agents.change-manager` on the same
commit applied to `chat-eval/master-candidate` (fca4bb7ce3). Result:
passed. In `server.log` the setup probe reports `failures: []` and
`run:otherDurableSession`, `run:durableSessionListing`,
`run:otherRunnerState`, `run:controlPlaneState`,
`run:otherOpencodeSession`, `run:otherRunLog`, `run:databaseWal`,
`run:secrets`, `run:otherProviderTrace` and `run:harnessEnviron` all
`denied`; `run:ownDurableSession`, `run:ownOpencodeSession`,
`run:acpxRuntime` and `run:ownProviderTrace` `allowed`;
`run:harnessCmdline` `allowed` (recorded, Landlock limit). The runnerd
launch line lists the nine run-scoped hidden instance paths and exposes
only its own durable-session and OpenCode session directories. The agent
finished the case inside that sandbox.

## Risks

- A sandboxed runnerd cannot create new direct entries in the instance
root, `runtime/`, `runtime/paperclip-runner/` or `data/`. The wrapper
creates the known ones first (`acpx`, `companies`, `projects`,
`workspaces`, `runtime-context-assets`, `skills`,
`runtime/provider-packs`). A provider that creates a different new
directory there fails in a sandboxed local cell. The paid run covers
OpenCode only. ACPX cells keep their runtime directory readable.
- A run still reads its own session's control-plane state, and other
agents' homes and task workspaces in the same instance.
- Landlock cannot hide `/proc/<pid>/cmdline`. Harness paths stay visible
there. Only the case id moves to the environment. Scheduled multi-cell
campaigns still pass `--id`.
- Harness-only change. No product code changes.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), extended thinking, tool
use (shell, file edits, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 07:28:12 -06:00
DottaandClaude Opus 5.5 051834bdc8 feat(chaos): add the on-demand crash-point suite and its manual CI workflow (#19946)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A control-plane restart between two durable writes leaves an
in-progress record owned by a dead process
> - Random restart-chaos waves found these 20 ms–1 s windows only by
luck
> - #19937 added `crashPoint(name)`, and #19938 put it at 20 durable
write boundaries
> - Nothing yet runs a task, kills the server at a chosen point,
restarts it, and checks the plan's invariants
> - This pull request adds that on-demand suite, as a local command and
as a manual CI workflow (not a required check, per Decision 2)
> - The benefit is a repeatable work list: which boundary breaks which
invariant, for Phases 2–4

## Linked Issues or Issue Description

Refs #19831 (plan, Phase 5 and Decision 2). Refs #19861 (report-only
reconciler used for I1–I7). Stacked on #19938 (call sites), which is
stacked on #19937 (helper).

**Problem**
The named crash points cannot be exercised end to end. No command arms
one point, runs real agent tasks across it, restarts, and grades the
result.

**Expected behavior**
`node scripts/crash-points.mjs --points <glob> --kinds <id> --adapters
<list>` runs one wave per selected point and reports each point as pass,
fail (with failing R rules and violated invariants), not reached, or
harness error. A `workflow_dispatch` workflow runs the same thing on
demand.

## What Changed

- **Runtime arming.** `PAPERCLIP_CRASH_POINT_CONTROL=<file>`, in the
shared helper and the runner mirror. Each hit reads `<token>
<name>[:<nth>]`, and a new token resets the count. It is test mode only:
`NODE_ENV=production` ignores it, and the disarmed path is still one
comparison.
- **`tests/restart-chaos/main.ts --crash-point
<a>:<nth>,<b>:<nth>,...`** (config, types, ledger):
  - one booted server and no Vite UI;
- per point: take an invariant baseline, launch the point's batch, arm
the point, then on rest disarm it and check invariants as violations
that are new since the baseline;
  - every task is graded R1–R6 and T;
  - `crash-point.json` (v2) holds one entry per point.
- **`tests/restart-chaos/supervisor.ts`:**
  - every spawn gets the control file;
- a SIGKILL whose receipt names the child's pid is restarted like K1:
the control file is emptied first, the owned tree is killed, and a
replacement starts. A failed tree cleanup fails the fault;
- the isolated instance pins `PAPERCLIP_IN_WORKTREE=false` (a
`.paperclip/.env` above the checkout otherwise suppressed all
scheduling);
  - `RESTART_CHAOS_HEALTH_TIMEOUT_MS`.
- **`tests/restart-chaos/invariants.ts`:**
- the Phase 1 report-only pass with `record: false`: two real-clock
passes for I1 and I2, and one pass 15 minutes ahead for I3, I4 and I7;
  - spool kinds are read from the wave's instance;
  - an unfinished pass is a harness error;
  - each violation carries its wake and run state as evidence.
- **`tests/restart-chaos/crash-points.ts` and
`scripts/crash-points.mjs`:**
- select points by glob, kind and environment, and run them in one wave;
  - a per-point outcome: pass, fail, not_reached or harness_error;
  - `SUMMARY.md` with a violation evidence column.
- **`.github/workflows/crash-points.yml`**, manual only, under the
paid-workflow contract:
  - a credential-free `authorize` job;
  - a `suite` job in `runner-e2e-paid` whose first step reauthorizes;
  - a frozen install with `--ignore-scripts` and no shared cache;
  - only the selected adapters' keys;
  - the workflow is added to both security-test allowlists.
- **Docs:** "Crash-point suite" in `tests/restart-chaos/README.md`, and
"Built (Phase 5)" plus the first work list in the plan.

## Verification

- Focused tests:
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/crash-points.test.ts`;
- `cd packages/shared && npx vitest run src/crash-points.test.ts`: 15
tests, including runtime arming in a real child process;
- `node --test scripts/__tests__/release-verify-workflow.test.mjs`: 72
tests;
  - `tests/runner-e2e/workflow-security.test.ts`: 19 tests.
- Typecheck: `node scripts/typecheck-test-projects.mjs
tests/restart-chaos` passes.
- Loop line: question "how long is one crash-point loop?" | loop: `node
scripts/crash-points.mjs --points <one>` (codex A1) | elapsed 2.7 min
end to end: boot 9 s, crash 9 s after arming, replacement ready in 6 s,
rest about 2 min | result: one point alone takes about 3 min. Before
this change it took about 15 min (one cold boot per point, Vite UI, host
at load 77).
- Full local run, 14 points, codex A1, 11.8 min in all:
- every point was reached (the server killed itself at each one, and the
replacement came back);
  - 11 pass;
- `workspace.finalize.after_start_before_copyback` and
`workspace.finalize.after_copyback_before_done` fail I2 and R6: a
`workspace_finalize` operation stays running after the run succeeded;
- `wake.claim.after_run_running_before_wakeup_claimed` reports I4. Its
evidence shows the wake failed with "…retrying once", and the retry
delivered the work, so this is the known refusal-regex false positive.
- The 6 Daytona-only points were not run.

## Risks

- Test harness and an on-demand workflow only. No product code changes.
- `.github/**` needs a CODEOWNER review. The workflow uses provider
keys, so it refuses non-default branches and actors that are not on the
allowlist, like `runner-full-stack-e2e.yml`.
- `PAPERCLIP_IN_WORKTREE=false` is pinned in every chaos wave, not only
crash-point waves. A chaos instance is never a worktree dev instance, so
scheduling must not be suppressed.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 23:41:15 -06:00
DottaandClaude Opus 5.5 2e7c66aac9 fix(ci): stop the PR merge review notifier from flooding the Actions queue (#19630)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Its GitHub Actions CI shares the `ubuntu-latest` runner pool across
all workflows
> - #19480 added the `PR merge review notification` workflow. It
triggers on `workflow_run`, on every completed `check_run`, and on every
commit `status`
> - One PR push completes about 60 checks, so it queues about 60
notifier jobs. Each job takes a runner
> - The Actions queue now holds about 340 queued runs. About 260 of them
are this notifier. Other `ubuntu-latest` jobs wait more than one hour
(for example, the `authorize` job of Runner Full-Stack E2E image builds)
> - This pull request skips events that cannot complete a green PR state
before a runner is allocated, and it collapses the remaining events to
the newest one per PR head
> - The benefit is that this unjams the Actions queue and keeps the
advisory review notification

## Linked Issues or Issue Description

Refs #19480

**What happened?**
The `PR merge review notification` workflow from #19480 schedules one
runner job for each `workflow_run`, `check_run`, and `status` event. One
PR push makes about 60+ jobs. The repository Actions queue has about 340
queued runs (about 260 from this notifier at the time of writing). Other
`ubuntu-latest` jobs wait more than one hour.

**What did you expect?**
The notifier requests one advisory merge review when PR verification is
complete. It must not use a runner for each intermediate check.

**Steps to reproduce**
1. Push to any PR.
2. Run `gh run list --workflow pr-merge-review.yml -L 200 --json
event,status`.
3. See tens of queued `check_run`, `status`, and `workflow_run` runs for
the same push.

## What Changed

- `.github/workflows/pr-merge-review.yml`: the job `if:` now schedules
the job only for events that can complete a green PR state:
- `workflow_run` from `pull_request` or `pull_request_target` with
conclusion `success`, `skipped`, or `neutral`.
- `check_run` from a non-Actions app (app id is not `15368`) with
conclusion `success`, `skipped`, or `neutral`. GitHub Actions check runs
are already covered by the `workflow_run` list.
  - `status` with state `success`.
- GitHub cannot filter `check_run` or `status` at the trigger level. A
skipped job makes a run record but does not take a runner.
- The job has a concurrency group `pr-merge-review-<PR head SHA>` with
`cancel-in-progress: true`. The notifier reads the current PR state, so
the newest event subsumes older ones. The group is on the job, not the
workflow, so a skipped event cannot cancel a scheduled notifier.
- The environment, permissions, trusted default-branch checkout, and
secret handling do not change.
- `.github/scripts/tests/notify-pr-merge-review.test.mjs`: a new test
pins the trigger kinds, the job condition, and the job-level concurrency
group.
- `doc/PR-MERGE-REVIEW-WEBHOOK.md`: describes the event filter and the
concurrency group.

**Expected runner jobs per PR push** (typical PR: `PR`, `Storybook
Visual`, `commitperclip PR Review` workflows; Greptile, Paperclip
Review, Socket (2), Superagent (2) external checks; one Snyk status):

| | Before | After |
| --- | --- | --- |
| Eligible events that take a runner | about 60–70 (every check run + 2
statuses + 3 workflow runs) | at most about 10 (3 workflow runs + 6
external checks + 1 status, only when green) |
| Jobs that run to completion | about 60–70 | usually 1–4: events that
arrive while a job is pending or running cancel it, so a burst collapses
to the newest event. In a backed-up queue, all events for one head
collapse to one job |

## Verification

- `node --test .github/scripts/tests/notify-pr-merge-review.test.mjs`:
22 tests pass.
- The new workflow test fails on the old workflow (red) and passes on
the new workflow (green).
- `actionlint .github/workflows/pr-merge-review.yml`: no findings.
- I confirmed with `gh run list` that `workflow_run.head_sha` for the
`pull_request_target` workflow (`commitperclip PR Review`) is the PR
head SHA. So one SHA key groups all event kinds for one PR head.
- I confirmed that all workflows that make PR checks are in the
`workflow_run` list (`CI routing` and `Trusted PR CI` are reusable
workflows called by `PR`).
- After merge: check that `gh run list --workflow pr-merge-review.yml`
shows mostly `skipped` runs for each push, and a small number of runs
that execute.

## Risks

- A check provider that reports a final green state through a GitHub
Actions check run from a workflow that is not in the `workflow_run` list
does not trigger the notifier. All current PR workflows are in the list.
The doc already tells authors to add new PR workflows to the list.
- A failed or cancelled check that later succeeds on re-run still
triggers the notifier, because the re-run completes with `success`.
- `cancel-in-progress` can cancel a notifier during delivery. The newer
job evaluates the same or a more complete state, and the Paperclip
idempotency key deduplicates deliveries for the same PR, head, and base.
- Each event still makes a workflow run record. Skipped runs do not take
a runner.
- Low risk otherwise: the notifier is advisory and `continue-on-error`.
It does not affect PR verification results.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 08:24:45 -06:00
DottaandClaude Opus 5.5 b5b972ceb1 fix(runner): stop provider tool processes that leave the owned process group (#19727)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The local runner (`paperclip-runnerd`) supervises each provider
process, such as the ACPX sidecar for Claude, in its own process group
> - Run teardown stops that process group, and nothing else
> - Claude Code starts each Bash tool command in a new session
(`setsid`). A command such as `nohup ./bench.sh & echo $!` leaves a job
in that session, and the wrapper exits at once. The job is then a child
of init and outside the owned group
> - In a restart chaos storm, a run failed after a restart resume. Its
tool job kept running beside the task's next run, in the same workspace,
for more than four minutes
> - This pull request makes the supervisor record every process that its
provider starts, including processes that leave the group, and stop them
all before teardown reports success
> - The benefit is that a run's tools cannot outlive the run and cause
duplicate side effects in the next run

## Linked Issues or Issue Description

**What happened?**
A restart chaos storm ran a local ACPX Claude task. After a restart
resume, the agent ran `nohup ./bench.sh <nonce> > log 2>&1 & echo
"started pid $!"` and then ended its turn without a result. The run
failed with `native_finalization_missing`. The runner tore down the
sidecar group, and the guardian killed the provider group. `bench.sh`
was in the session that Claude Code made for the tool (pgid and sid =
the exited wrapper, ppid = 1). It was never signalled. It kept forking
`sleep 1` while the task's next run started and finished in the same
workspace.

**Steps to reproduce**
1. Run a local ACPX Claude task whose agent runs a Bash tool command
such as `nohup ./bench.sh x > log 2>&1 & echo $!`.
2. End the run (any outcome, including failure after a restart resume).
3. Run `ps -axo pid,ppid,pgid,command | grep bench.sh`. The job is still
running with ppid 1.

**Agent adapter(s) involved**
Claude Code (local ACPX). The gap is in the shared supervisor, so any
provider whose tools leave the group is affected.

**Root cause**
`SupervisedProcess`
(`packages/paperclip-runner/runner/crates/runner-core/src/process_supervisor.rs`)
spawns with `process_group(0)` and stops only that group in
`terminate_group`, `terminate_group_immediately`, `wait` and `Drop`. Its
own comment said that a caller that must exclude escaped or re-parented
descendants needs a separate lifetime fence. No caller had one. The ACPX
sidecar transport (`acpx_sidecar_transport.rs`, `shutdown` and `poison`)
and the ACPX provider guardian (`installation-integrity.ts`, which kills
only its own group) therefore left every `setsid` tool job behind. The
failure path after the resume was not special: the same gap applies to
every teardown.

**Expected behavior**
When a run ends for any reason, every process that its provider or tools
started stops, and teardown proves that they are gone. Identity comes
from PID, group, session and start time, never from a command name.

## What Changed

- Add the `paperclip-process-table` crate. It reads PID, parent, process
group and start time for every process (libproc on macOS, `/proc` on
Linux). `runner-core` forbids unsafe code, so the two checked libproc
calls live in this small crate.
- Add `process_lineage.rs` to `runner-core`. A sampler thread (every 50
ms) records each descendant of the supervised process by PID and start
time. It also records each group and session that a descendant leads,
with the leader's start time. A process belongs to the lineage when its
parent belongs, or when it is in a recorded group or session and started
no earlier than its leader. Groups and sessions that the supervised
process only inherited (runnerd's session) are never recorded. A
recorded ID whose PID now names a different process is dropped.
- `SupervisedProcess` samples the lineage just before it signals, and
after the leader is reaped it stops every live member: TERM, then KILL
after the shutdown grace. It fails with an error until it proves that no
member is alive (grace plus 5 s). This applies to `terminate_group`,
`terminate_group_immediately`, `wait` and `Drop`.
- If the lineage sampler cannot start, the supervisor stops the process
it just started and reports the failure, so a provider never runs
without tool cleanup.
- Table reads and state updates happen under one lock, so an older read
cannot erase members that a newer read recorded.
- The process table reports processes that exist but cannot be read. A
recorded member that cannot be read stays unresolved, and teardown fails
until its exit is proven.
- The Runner workspace now has two members.
`scripts/runner-release-crate-cache.mjs` and
`scripts/runner-debug-crate-cache.mjs` cache and mark current the
process-table lib unit with the Runner crate (otherwise Cargo rebuilds
the Runner crate after every restore). The debug contract accepts only
this library-only member and still refuses any other workspace change.
The workflow cache path lists and the debug clean step are updated to
match.
- `terminate_group_immediately` now accepts `EPERM` from the group
signal when the leader has already exited. macOS returns `EPERM` for a
group whose only member is the unreaped leader. The lineage pass then
proves that no other member remains.

## Verification

- `cargo test -p paperclip-runner-core --lib -- process_lineage tool_job
immediate_group_fence` (macOS, local).
- New real-process tests in `process_supervisor.rs`: a supervised shell
starts a wrapper with `setsid` (Perl `POSIX::setsid`), the wrapper runs
`nohup <loop> &`, writes the job PID and exits. The test asserts that
the job is outside the owned group and re-parented, then calls
`terminate_group`, `terminate_group_immediately` or drops the process,
and asserts that the job is gone when the call returns.
- Red: with the tracker disabled, all three fail with `tool job <pid>
outlived its provider`.
  - Green: all three pass.
- Unit tests in `process_lineage.rs` cover a re-parented job, a job that
left its group but stayed in the recorded session, a session made after
the first sighting, PID reuse, and no claim through the supervisor's
inherited session.
- `node --test scripts/__tests__/runner-release-crate-cache.test.mjs
scripts/__tests__/runner-debug-crate-cache.test.mjs` (new case: the
process-table library is cached with the crate, and any other member
still needs review).
- `cargo test -p paperclip-process-table` (includes an exited process
reported as gone), and `cargo test -p paperclip-runner-core --lib` (570
passed).
`acpx_provider_backend::tests::unconfirmed_suspension_state_becomes_recoverable_only_after_cleanup_and_suspend`
fails sometimes under full-suite load. It fails at the same rate with
the tracker disabled, so it is a flake that already existed.
- Sampler cost: one table read takes about 0.4 ms for 1,200 processes on
macOS (about 0.8% of one core for each supervised process).
- The restart-chaos replay (`--replay w-20261107T023921-s7106/t67`) did
not run. The local session was not permitted to load provider
credentials. It is not proven against a live Claude run.

## Risks

- Teardown now stops all the processes that a provider started, also
intentional background jobs. This is the intended contract: nothing
outlives the run.
- A wrapper that lives less than one sample (50 ms) and orphans a job
before the sampler sees it is not recorded. Claude Code's wrapper
sources a shell snapshot and lives for seconds. On Linux, a
child-subreaper would close this gap. That is not in this change.
- When `runnerd` itself is killed with SIGKILL, this code does not run.
The provider guardian still kills its own group, but escaped tool
sessions are not stopped in that case.
- Teardown can return a new error when it cannot prove the lineage
stopped. Callers already handle `terminate_group` errors as provider
shutdown failures.
- The server does not yet gate the next run on the tool lineage of the
prior run directly. It relies on the runner finishing its teardown
before the runner exits.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 08:24:41 -06:00
DottaandClaude Opus 5.5 efc0b359da fix(ci): rebuild and sign an unsigned reused Daytona e2e image (#19677)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paid runner E2E campaigns run in `runner-full-stack-e2e.yml`.
Daytona cells need a signed, content-addressed runner image
> - The "Publish verified Daytona image" job reuses `e2e-content-<id>`
when the tag exists, and signs only after a fresh build
> - A run that stops between the push and `cosign sign` leaves an
unsigned tag. Every later run reuses it and fails `cosign verify` with
"no signatures found"
> - That failure skips the whole test matrix, including local cells. The
first Reliability TestOps launch (run 42298071794) ran 0/7 cells for
this reason
> - This pull request reuses a tag only when its signature verifies.
Otherwise it rebuilds, pushes and signs the tag
> - The benefit is that an interrupted run cannot block every later
Daytona campaign

## Linked Issues or Issue Description

Refs #19549

**What happened?**
Run https://github.com/paperclipai/paperclip/actions/runs/42298071794
(TestOps launch of the Reliability plan) failed in "Reuse or publish
immutable image" with `Error: no signatures found` (exit code 10). No
test cell ran. The report shows 7 missing attempts.

**Expected behavior**
A campaign with Daytona cells always produces a signed, verified image,
or it fails for a real build or signing error. A stale unsigned tag must
not block it.

**Steps to reproduce**
1. A run pushes
`ghcr.io/paperclipai/paperclip-daytona-runner:e2e-content-<id>` and
stops before `cosign sign` (for example, a runner shutdown).
2. A later campaign with the same content ID finds the tag and skips the
build.
3. `cosign verify` fails with "no signatures found", and the matrix is
skipped.

**Paperclip version or commit**
master 02a2299173.

## What Changed

- `.github/workflows/runner-full-stack-e2e.yml`: add a `verify_signed`
helper. Reuse an existing content tag only when its digest has this
workflow's signature. If verification fails, warn, rebuild, push and
sign. The final `cosign verify`, logout and config checks are unchanged.
- `tests/runner-e2e/daytona-image.test.ts`: assert that reuse is gated
on signature verification and that the build-and-sign path follows it.

## Verification

- `npx vitest run daytona-image workflow-security` in
`tests/runner-e2e`: 38/38 pass.
- Red before the fix: the new assertion fails on the unchanged workflow
(1 failed, 18 passed in `daytona-image`).
- The workflow YAML parses.
- Live check after merge: relaunch the Reliability TestOps plan. The
image job should warn about the unsigned tag, rebuild it, sign it and
verify it, and then the cells run.

## Risks

- Low risk. The rebuild uses the same trusted inputs and cache, and it
only happens when verification fails. The new digest replaces the
unsigned tag. The identity and issuer checks are unchanged.
- Rebuilding an unsigned tag takes extra time on that one run.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code with tool use
(shell, browser read of the GitHub job log), standard context.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 22:29:11 -06:00
Nicky LeachandClaude Opus 5.5 fb3cf1a072 feat(ci): retry a failed Vitest case once in PR CI and report each retried pass (#19637)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Contributors wait for PR CI before merge, so CI wall-clock time sets
the pace of work
> - PR CI runs Vitest shards with 4 workers on 4-vCPU runners, so
timing-sensitive tests sometimes time out under load
> - One such timeout fails the whole run, and a re-run costs another 15
minutes or more
> - In one 7-hour window on 2026-11-06, about 30 failed jobs were test
flakes on commits that passed on a later attempt
> - This pull request retries a failed Vitest case once in PR CI only,
and reports every pass that needed the retry
> - The benefit is fewer whole-run re-runs, while each flaky test stays
visible so that we can fix it

## Linked Issues or Issue Description

**What existing behavior does this improve?**

PR CI fails the whole run when one Vitest case fails once because of
runner load.

**Subsystem affected**

CI: the PR Vitest lanes (general, serialized, workspace and Runner
package lanes).

**Current behavior**

Vitest uses `retry: 0` everywhere. One timeout under load fails the run.
A person must notice it and re-run the failed jobs. About 20% of
recently merged PRs needed at least one re-run.

**Proposed behavior**

PR CI retries a failed Vitest case one time. A test that passes only on
the retry gets a GitHub warning annotation and a row in the job summary.
Local runs, release, cloud readiness and master keep `retry: 0`.

**Reason and benefit**

A re-run of one test case costs seconds. A re-run of the CI run costs 15
minutes or more. The report keeps flaky tests visible. Separate pull
requests fix the known flaky tests.

**Breaking changes**

None.

## What Changed

- Add `scripts/ci/vitest-retry.mjs`. It sets 1 retry when
`GITHUB_ACTIONS=true` and `GITHUB_WORKFLOW=PR`, and 0 in other contexts.
`PAPERCLIP_VITEST_RETRY` overrides the value.
- Add `scripts/ci/retry-reporter.mjs`. It writes a `::warning`
annotation and a step-summary table row for each test that passed only
on a retry.
- `scripts/run-vitest-stable.mjs` passes `--retry` to `vitest run`. The
`vitest list` collection step does not retry.
- The root `vitest.config.ts` and
`packages/paperclip-runner/vitest.config.ts` add the retry reporter
beside the existing reporters. The Runner config sets `retry` from the
same gate.
- `.github/workflows/pr-trusted.yml` reads the `CI_VITEST_RETRY`
repository variable. Set it to `0` to turn retries off without a code
change.
- Add `scripts/ci/retry-reporter.test.mjs` to `test:release-registry`,
and document the behavior in `doc/DEVELOPING.md`.

## Verification

- `node --test scripts/ci/retry-reporter.test.mjs`: 11/11 pass. The
tests cover the gate matrix, the reporter output, a real Vitest run with
a test that fails once, the wrapper arguments, and the Runner config.
- `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`:
39/39 pass.
- `node --test .github/scripts/tests/*.test.mjs`: 744/744 pass.
- Manual check: a temporary test that fails once ran through
`run-vitest-stable.mjs` and through the Runner package config. With the
PR environment it passed and wrote the annotation and the summary row.
With `GITHUB_WORKFLOW=Release` it failed. The temporary test is removed.

## Risks

- A retry can hide a flaky test. The annotation and the summary row
reduce this risk.
- A test that fails for a real reason now runs twice in PR CI, so a
failed run takes slightly longer.
- `pr.yml` loads `pr-trusted.yml` from master, so the `CI_VITEST_RETRY`
switch works only after merge. The retry itself is in the scripts and
starts with this pull request.
- Low risk otherwise: release and master verification are not changed.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution. A Claude sub-agent wrote the change, and the parent
session reviewed it.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 16:43:02 -08:00
Nicky LeachandClaude Opus 5.5 97953e9b59 fix(ci): repair the weekly test-weight refresh and extend it to e2e and Runner lanes (#19650)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Contributors wait for PR CI before merge, so CI wall-clock time sets
the pace of work
> - CI test shards are balanced with per-file durations that a weekly
workflow refreshes
> - The weekly refresh has failed since 2026-11-02, and the e2e weights
have not been measured since 2026-09-15. Today e2e shards 3 and 6 take
7.0–7.7 minutes of tests, and shard 1 takes 4.3 minutes
> - A single file can also grow until it sets the time of its shard, and
nothing tells us about it
> - This pull request repairs the refresh, extends it to the e2e and
Runner package lanes, and warns about test files over 180 seconds
> - The benefit is balanced shards that stay balanced, and early notice
of oversized test files

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The weekly refresh of the test-duration weights that balance CI shards.

**Subsystem affected**

CI: `scripts/ci/refresh-test-durations.mjs`,
`.github/workflows/ci-test-durations.yml`, and the Vitest and Playwright
CI reporters.

**Current behavior**

- The 2026-11-02 refresh (run 41408033686) failed. `gh api …/logs` now
refuses output that contains terminal escape sequences.
- Most successful `cloud-readiness.yml` master runs are admission-only
and have no server jobs. The script took the newest 2 runs, so it would
fail with "No server timing jobs found" on most weeks.
- The e2e weights (`scripts/e2e-shard-durations.json`) have no refresh
and are from 2026-09-15.
- No signal tells us when one test file becomes too slow.

**Proposed behavior**

The refresh runs again, selects master runs that ran the server jobs,
refreshes the e2e and Runner package-lane weights from merged PRs, and
lists oversized files. CI writes a warning for each test file over 180
seconds.

**Reason and benefit**

Unbalanced shards add minutes to every PR. The current e2e weights put
472 seconds on the slowest shard. The new weights put about 300 seconds
on each shard.

**Breaking changes**

None. Weights change only shard placement, never which tests run.

## What Changed

- Add `scripts/ci/gh-api.mjs`. It always passes
`--allow-escape-sequences`, retries without the flag when an older `gh`
does not know it, and strips ANSI codes from logs.
- `scripts/ci/refresh-test-durations.mjs`:
- scans up to 50 master runs and uses the 2 newest that ran the server
jobs;
- adds the `--dry-run`, `--pr-body=<path>` and
`--lanes=server,e2e,runner` options;
- validates every lane before it writes a file. If a merged-PR lane has
fewer than 2 valid runs, it keeps the current weights and writes a
warning.
- Add `scripts/ci/pr-run-durations.mjs`. It uses the final passing
`pr.yml` run of each merged PR, and it validates:
  - the PR is merged into `master`;
  - the merge commit is in the checkout;
  - the run's head SHA is the PR's final head SHA;
  - every shard of the run succeeded.

  It parses Playwright per-spec times and Runner per-file times.
- Add `scripts/ci/oversized-test-files.mjs` (Vitest) and
`scripts/ci/oversized-test-files-playwright.mjs` (Playwright). In GitHub
Actions they write a `::warning` for each test file over
`PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS` (default 180). They never
fail a job. The root and Runner Vitest configs and the e2e Playwright
config load them.
- `scripts/ci/test-timing-reporter.mjs`: move the per-file duration
calculation into an exported `moduleDurationMs()` helper. The behavior
does not change.
- `.github/workflows/ci-test-durations.yml` and
`.github/ci-duration-pr.md`:
  - the bot PR can also update the e2e and Runner manifests;
  - the PR body lists oversized files;
  - the verify step also runs the new tests and the e2e shard tests.
- Refresh `scripts/e2e-shard-durations.json`: 56 of 56 specs measured
(before: 30), from merged PRs #19613, #19191 and #19176. The file now
uses the shared schema with `sources` and `coverage`.

## Verification

- `node --test scripts/ci/test-selection-pr-timings.test.mjs
scripts/ci/test-selection-timings.test.mjs
scripts/__tests__/e2e-shard.test.mjs`: 32/32 pass. These include the e2e
shard balance check with the new weights.
- `node scripts/ci/refresh-test-durations.mjs --dry-run` against real
GitHub data, all lanes:

| Manifest | Measured | Sources |
| --- | --- | --- |
| `general-server-shard-durations.json` | 1488/1534 files | master runs
42296943853, 42294279748 |
| `serialized-shard-durations.json` | 167/167 files | the same two runs
|
| `e2e-shard-durations.json` | 56/56 specs | PR runs 42296374365,
42297700476, 42296994179 |
| Runner `vitest-lane-durations.json` | 310/311 files | the same three
PR runs |

- e2e placement over 8 shards, with the new medians as the real costs:
  - with the old weights, the slowest shard has 472 seconds;
  - with the new weights, every shard has about 300 seconds.
- The dry run lists 12 files over 180 seconds. The largest is
`server/src/services/workspace-recovery.test.ts`, at 636 seconds.
- In this PR, only the e2e weight file changes. The weekly run refreshes
the server weights. The Runner manifest is added by #19646.

## Risks

- The `--allow-escape-sequences` path ran only in unit tests. The local
`gh` did not support the flag, so the real dry run used the fallback.
The next weekly run is the first real test of the flag.
- The changes to the bot-PR shell steps did not run in Actions yet.
- The e2e weights come from 3 PR runs. Real shard times vary more than
the estimate.
- This PR and #19637 both edit the `reporters` lists in
`vitest.config.ts` and `packages/paperclip-runner/vitest.config.ts`. The
second PR to merge needs a small merge resolution.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution. A Claude sub-agent wrote the change, and the parent
session reviewed it.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 15:50:02 -08:00
Nicky LeachandClaude Opus 5.5 a2aa69bc62 perf(ci): split the Runner package Vitest lane into three duration-balanced lanes (#19646)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Contributors wait for PR CI before merge, so CI wall-clock time sets
the pace of work
> - The Runner package Vitest lane is the PR CI critical path. It
finishes a median 15.3 minutes after the push
> - That lane runs all 311 package test files in one job with 2 workers:
1,381 seconds of serial work and 727 seconds of wall time
> - Ten files account for 61% of the time. Vitest's `--shard` splits by
file count, not by duration, so it cannot balance this work
> - This pull request splits the package files into three lanes balanced
by duration, and runs the native server group in its own lane
> - The benefit is about 232 seconds of tests per package lane instead
of 727 seconds

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The time that PR CI needs to run the Runner package Vitest tests.

**Subsystem affected**

CI: the `Verify Paperclip Runner` Vitest lanes in `pr-trusted.yml`, and
`packages/paperclip-runner/scripts/run-pr-vitest-lane.mjs`.

**Current behavior**

The `vitest package` lane takes a median of 13.6 minutes. It is the last
job to finish in most PR runs. The lane keeps a 2-worker limit so that
Runner and sidecar processes can start within their deadlines. Average
CPU use is 29%.

**Proposed behavior**

Three package lanes each run about one third of the files, balanced by
measured duration. Each lane keeps the 2-worker limit. The native server
group runs in its own lane, as before.

**Reason and benefit**

The Runner lane stops being the critical path. More workers per lane
would add CPU contention and timeouts, so this change adds lanes
instead. The cost is about 3 runner-minutes per run for the extra setup.

**Breaking changes**

None. The `master` ruleset requires only `ci / verify` and `ci / e2e`,
so the new job names do not change required checks.

## What Changed

- Add `packages/paperclip-runner/scripts/vitest-package-lanes.mjs`.
- It gets the file inventory from `vitest list --filesOnly --json`, so
it follows `vitest.config.ts`.
- It partitions the files with the `partitionGeneralServerSuites`
function from `scripts/general-server-shard.mjs`.
- Add `packages/paperclip-runner/scripts/vitest-lane-durations.json`.
- It uses the same schema as
`scripts/general-server-shard-durations.json`.
- It is seeded from CI run 42290920882. It measures 310 of the 311
files.
- `run-pr-vitest-lane.mjs` accepts two new arguments:
  - `--package-lane=K/N` runs one partition.
  - `--native-server` runs only the native server group.
- The `--shard=1/2` and `--shard=2/2` arguments keep their exact
behavior for the matrix that master still sends.
- `.github/ci-task-matrix.json` and the fallback matrix in
`pr-trusted.yml` have three `vitest package (K/3)` lanes and the `vitest
native server` lane. The labels keep the `vitest` prefix because
`scripts/ci-profile/trace.mjs` matches `Verify Paperclip Runner
(vitest`.
- Tests:
- the lanes cover the inventory exactly once, checked with real `vitest
list`;
  - the old `--shard` commands plan the same commands as before;
  - the manifest schema is correct;
  - bad lane arguments are refused.
- Update the Runner Vitest lane paragraph in `doc/DEVELOPING.md`.

## Verification

- `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`:
44/44 pass.
- `node --test scripts/__tests__/runner-release-crate-cache.test.mjs
.github/scripts/tests/ci-plan.test.mjs`, together with the file above:
71/71 pass.
- Related workflow, cache, routing and profile tests: 446/446, 153/153
and 14/14 pass.
- Local end-to-end run of `GITHUB_WORKFLOW=PR node
scripts/run-pr-vitest-lane.mjs --package-lane=3/3` on macOS:
- the prep steps ran, and the lane ran 104 of 311 files in 123 seconds;
- 2 tests failed: a macOS `kill EPERM` and a 5-second timeout under full
local load without the CI worker limit;
  - both files pass when run alone (35/35 tests).
- This PR's own CI runs the matrix from master, so it uses the old
`--shard` commands. The new lanes run in CI only after merge. Reviewers
can compare the job durations of the first PR runs after the merge.

| Lane | Files | Serial estimate | Wall estimate with 2 workers |
| --- | --- | --- | --- |
| 1/3 | 103 | 464.9 s | ~232 s |
| 2/3 | 104 | 465.0 s | ~232 s |
| 3/3 | 104 | 465.0 s | ~232 s |

## Risks

- The weights come from one CI log and do not include collection or
setup time. The balance can be uneven until a refresh measures real lane
runs.
- Each lane passes its file paths to Vitest as filters, and Vitest
matches a filter as a substring of the path. A path that is a substring
of another path would run a file in two lanes. No current path does
this. If one does, the script prints a warning, and no file is dropped.
- Each package lane runs `ensure:eval-build-deps` and `build:rust`
again, which adds about 3 runner-minutes per run.
- A comment in `scripts/run-vitest-stable.mjs` (about line 110) still
says that the last Runner shard runs the native group. A follow-up will
correct it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution. A Claude sub-agent wrote the change, and the parent
session reviewed it and ran the local lane check.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 15:12:05 -08:00
Nicky LeachandClaude Opus 5.5 f2299cb00f feat(ci): re-run failed PR jobs once when every failure was a lost runner (#19640)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Contributors wait for PR CI before merge, so CI wall-clock time sets
the pace of work
> - PR CI runs on self-hosted AWS runners, and these runners are
sometimes lost mid-job
> - On 2026-11-06, 21 of 51 failed jobs on commits that later passed
were runner loss. The losses came in fleet-wide bursts, for example 10
jobs on 10 instances in 51 seconds
> - After each loss, a person must notice the failure and click "Re-run
failed jobs"
> - This pull request adds a workflow that re-runs the failed jobs once,
only when every failure was a lost runner
> - The benefit is fewer manual re-runs and a shorter wait for CI to
pass

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The recovery of PR CI runs that failed only because a self-hosted runner
was lost.

**Subsystem affected**

CI: the trusted PR workflow (`pr.yml` → `pr-trusted.yml`).

**Current behavior**

When an AWS runner is lost, the job fails with "The self-hosted runner
lost communication with the server" or "The runner has received a
shutdown signal". The PR stays red until a person re-runs the failed
jobs.

**Proposed behavior**

When the `PR` workflow completes with a failure on attempt 1, a new
workflow examines the failed jobs. If every failed job lost its runner,
it re-runs the failed jobs one time. If any job failed for another
reason, it does nothing.

**Reason and benefit**

A runner loss is not a code problem. The re-run starts without waiting
for a person to notice the failure.

**Breaking changes**

None.

## What Changed

- Add `.github/workflows/ci-runner-loss-rerun.yml`. It triggers on
`workflow_run` for `PR`, and it acts only on attempt 1 failures of
`pull_request` runs in this repository. It runs on `ubuntu-latest`, so
it does not depend on the AWS fleet.
- Add `.github/scripts/runner-loss-rerun.mjs`. The script:
- lists the attempt-1 jobs and ignores the aggregate jobs `ci / verify`
and `ci / e2e`;
- finds runner loss in the check-run annotations. If the only annotation
is "The operation was canceled.", it reads the `##[error]` lines in the
job log;
- calls `rerun-failed-jobs` only if every remaining failed job lost its
runner;
- does not re-run when the gate job failed, the PR head moved, the PR is
closed, or a newer run exists for the same commit;
- writes the decision and a job table to the step summary. A `--dry-run`
mode never posts.
- Add 16 tests, with trimmed fixtures from four real runs.
- Add a paragraph about the workflow to the "Trusted PR Workflow"
section of `doc/DEVELOPING.md`.

## Verification

- `node --test .github/scripts/tests/runner-loss-rerun.test.mjs`: 16/16
pass. The tests cover:
- re-run when all jobs were lost, including the case where the signal is
only in the log;
- no re-run for mixed failures, real failures, only aggregate failures,
attempt 2, a failed gate, or a stale run;
- pagination, incomplete listings, a cross-host link that is refused,
and the workflow guards.
- `node --test '.github/scripts/tests/*.test.mjs'`: 760/760 pass.
- A read-only dry run against the real GitHub API for six recent runs
classified all six correctly.
- The workflow can run on GitHub only after it is on master. Reviewers
can make sure that the first runner-loss failure after the merge
triggers one re-run.

## Risks

- **Trust:** a re-run of failed jobs uses the outputs of the gate job
that succeeded on attempt 1, including its AWS runner admission. A
person who clicks "Re-run failed jobs" gets the same result. The only
difference is that `github-actions[bot]` triggers attempt 2. The
workflow never checks out or runs PR code.
- **Failed gate:** if the gate job itself failed, the script does
nothing. A re-run by the bot would put all jobs on GitHub-hosted
runners.
- **Profiling:** `scripts/ci-profile/export.mjs` exports only attempts
triggered by authorized users. The bot-triggered attempt 2 is not
exported to Honeycomb. Attempt 1 is still exported.
- **Events:** GitHub may not create `workflow_run` events for attempts
that `GITHUB_TOKEN` started. Then other `workflow_run` consumers do not
react to attempt 2. Required checks still update.
- **Permissions:** `actions: write`, `checks: read`, `contents: read`
and `pull-requests: read`. The last one lets the script confirm that the
run still belongs to the open PR's current head.
- **Cancelled jobs:** a runner loss where the lost jobs end as
`cancelled` and only the aggregate jobs show `failure` is not re-run.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution. A Claude sub-agent wrote the change, and the parent
session reviewed it.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 15:05:42 -08:00
DottaandPaperclip b6e9615aed feat(ci): request advisory merge review after PR verification (#19480)
## Thinking Path

> - Paperclip helps people manage AI agents and their work.
> - Pull requests already run CI and automated code review.
> - A completed check does not start a final advisory merge review.
> - A private routine webhook can connect these two steps.
> - The webhook credential must stay outside PR code and public logs.
> - This pull request adds an optional notification from trusted branch
code.
> - The routine records a recommendation. A person keeps merge control.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

It connects successful PR verification to an advisory Paperclip routine.

**Subsystem affected**

GitHub Actions and the existing routine webhook API.

**Current behavior**

A PR can finish verification without asking a routine for a final merge
recommendation.

**Proposed behavior**

A trusted workflow reads the current PR and its complete check list. It
sends the PR identity to a private webhook when required checks pass and
other reported checks finish successfully, as skipped, or as neutral.
The receiving routine runs the configured merge review skill and records
its decision on its Paperclip task.

**Reason and benefit**

This gives maintainers an advisory decision after verification. It does
not grant merge control to the routine.

**Breaking changes**

None. Missing secrets or delivery errors leave verification results
unchanged.

Searched open and closed PRs and issues for related work. No duplicate
notification workflow was found. Related API work: #18058 and #18581.
This change uses the routine API and does not alter chat credential
setup.

## What Changed

- Add a default-branch workflow for PR workflow completions, external
checks, and commit statuses.
- Use read permissions and a GitHub environment restricted to master for
the private URL and bearer token.
- Require successful `ci / verify`, `ci / e2e`, and additional
branch-rule checks. Read every status page and refresh the PR before
delivery.
- Use a stable idempotency key for each PR, head, and base. Reject stale
events and changed candidates.
- Verify the two core check runs belong to the same real GitHub Actions
suite for the unchanged PR caller and master-owned reusable CI. Reject
forged names, renamed callers, and incomplete file listings.
- Keep HTTP redirects disabled, bound request time, and print fixed
delivery states only.
- Add regression checks and document private setup, pause, rotation, and
acceptance steps.
- Grant commit-status read access. Add a separate PR API probe with the
same read permissions and the real Actions token. The probe has no
private environment or webhook secrets and sends no notification.

## Verification

- Pass: `node --test
.github/scripts/tests/notify-pr-merge-review.test.mjs` (21 tests after
the security and token-permission fixes).
- Pass: actionlint v1.7.12 on `.github/workflows/pr-merge-review.yml`.
- Pass: live GitHub query of a current PR and its complete status list.
This found a GraphQL field mismatch during development. The corrected
query passed with 62 checks.
- Pass: the configured bearer webhook returned `test_received` with
`routineStarted=false` in setup mode. Readback confirmed an active
routine and environment secrets restricted to the master branch.
- Not run: `pnpm verify --explain` could not start because this checkout
has no installed workspace dependencies. Required CI will supply broad
coverage.
- The first quality-gate run rejected a CI-only title because this
change adds a script. The feature title passed the later gate.
- The security review found that PR-controlled check names could forge
eligibility. A focused regression failed before the fix and passed after
suite, run, caller, and reusable-workflow checks were added. The
corrected provenance query also passed against a completed historical CI
run without sending a webhook.
- Earlier CI at ec2c2dc failed in `workspace-recovery.test.ts` with a
generated company-prefix collision, in
`idle-domain-deadlines-postgres.test.ts` with a deadline snapshot
mismatch and timeout, and in `CoordinatorTasks.test.tsx` with a route
query mismatch. The browser shard also recorded a CSS preload test
failure before cancellation. These failures remain recorded in [run
42245040952](https://github.com/paperclipai/paperclip/actions/runs/42245040952).
This PR does not edit those production paths. Current master is merged
and fresh CI is required; the earlier failures are not claimed resolved.
- The repeated browser, general-server, and UI lanes passed at
715cee88ce. The earlier failures remain historical; their causes are not
all established. The Runner package lane was still running before the
next fix.
- A reviewer found missing `statuses: read`, which the broader local
token did not expose. The permission regression failed before the fix
and passed after it. The new read-only API probe will test the query
with the actual Actions token.
- Pass at c3c84394b7: the real Actions-token API probe, typecheck,
build, browser aggregate, security scan, and Greptile 5/5. No unresolved
review threads remain.
- The only remaining CI failure is a 15 second timeout in the existing
retained-native-session-cleanup-work backlog test (server shard 6/12).
This file and its production implementation are unchanged from the
previous candidate whose lane passed. The one scoped rerun passed at the
same head, and the required verify aggregate now passes. The timeout
remains in [run 42250393645, attempt
1](https://github.com/paperclipai/paperclip/actions/runs/42250393645/attempts/1);
its cause is not established. This is recovery evidence, not a claimed
root-cause fix.
- A nonblocking review suggestion to defer branch-rule lookup until a PR
is matched remains a minor efficiency follow-up. It is not a correctness
or security finding.
- Pending: a real notification and resulting routine task after this
workflow reaches master. The setup test did not start a PR review.

## Risks

- This workflow uses secrets. Keep the environment restricted to the
master branch. It runs only trusted branch code and does not read PR
artifacts, caches, or executable files.
- GitHub event timing and check visibility can race. Delivery is best
effort. A missed event can omit a review; it cannot change CI results or
merge a PR.
- Optional skipped and neutral checks permit an assessment. They do not
prove that a check ran. Required checks must succeed. The routine must
assess unavailable scanner coverage and other evidence gaps before its
recommendation.
- Add any new PR verification workflow name to the completion trigger
list.
- PRs that change the primary CI caller, `.github/workflows/pr.yml`,
need a manual advisory review until their caller change reaches master.
This prevents a PR from replacing verification with a forged result.
- This PR changes CI configuration. CODEOWNERS review and normal merge
rules still apply.

## Model Used

OpenAI GPT-6 family through Codex. The exact deployment model ID and
context window are not exposed to this run. Used reasoning, tool use,
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with available version and
capability details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 15:25:57 -06:00
DottaandClaude Opus 5.5 b7e28eb2f7 fix(evals): let an allowlisted manual dispatch authorize its private evidence (#19599)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat-feature evals run in the trusted `runner-full-stack-e2e`
workflow and keep their evidence in private history
> - A manual chat run needs authority to write that private evidence
> - Today that authority is a stored approval for the exact run ID in
`RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON`
> - The run ID exists only after dispatch, so every manual run stops
until a person writes the record
> - The dispatcher is already checked against
`RUNNER_E2E_ALLOWED_ACTOR_IDS`, so the second step adds friction without
a new decision
> - This pull request lets an allowlisted manual dispatch authorize its
own private evidence, as the TestOps admission already does
> - The benefit is that an allowlisted operator can start a chat eval
and it runs, while scheduled runs still never touch chat evidence

## Linked Issues or Issue Description

Refs #18134, Refs #18170

**What existing behavior does this improve?**
A manual `runner-full-stack-e2e` run with chat-feature IDs fails at
private evidence transfer unless a person first adds an approval row for
that run ID to a repository variable.

**Subsystem affected**
Runner E2E manual workflow private evidence authorization
(`manual-private-transfer.ts`, `runner-full-stack-e2e.yml`).

**Proposed change**
Derive the evidence authority from the dispatch itself, with the same
checks the TestOps chat admission uses.

## What Changed

- New `tests/runner-e2e/manual-dispatch-authority.ts`:
`authorizeManualDispatch` admits private evidence only when all of these
are true:
- The event is `workflow_dispatch` of `runner-full-stack-e2e.yml` on the
default branch, with the `manual` cadence.
  - The run attempt is `1`.
- The actor is in `RUNNER_E2E_ALLOWED_ACTOR_IDS`, the event sender is
that actor, and the triggering actor is the same person.
  - The trusted cell selection equals the dispatch's own `id` input.
- The authority binds the workflow SHA, the target SHA from the
authorize job, the run ID and the cells. Every job derives the same
identity hash.
- `manual-private-transfer.ts`: uses the TestOps authority first, then
the dispatch authority, then the stored approval. With the dispatch
authority, `prepare` records the target launcher digest instead of
comparing it with an approved digest.
- `runner-full-stack-e2e.yml`: the 14 steps that read
`RUNNER_E2E_MANUAL_PRIVATE_APPROVALS_JSON` now set
`PAPERCLIP_MANUAL_DISPATCH_AUTHORITY: v1` and pass
`RUNNER_E2E_ALLOWED_ACTOR_IDS`.
- `doc/manual-e2e-private-evidence.md`: describes the dispatch
authority. The stored-approval shape stays documented for the TestOps
catalog case workflow.
- The new file is separate on purpose. `manual-private-retention.ts` and
the other protocol files must match each target byte for byte, so
existing candidate branches keep working.

## Verification

- `vitest run tests/runner-e2e/manual-dispatch-authority.test.ts
tests/runner-e2e/manual-private-transfer.test.ts
tests/runner-e2e/manual-private-report.test.ts
tests/runner-e2e/manual-private-report-integration.test.ts
tests/runner-e2e/manual-private-policy.test.ts`: 57 tests passed.
- New tests cover one valid dispatch and rejection of: a scheduled run,
a bounded cadence, a non-allowlisted actor, an empty allowlist, a rerun
by another person, a second attempt, a non-default or different workflow
ref, a sender who is not the actor, a missing target SHA, a missing
authority switch, and cells that differ from the dispatch input. The
eight-cell limit still applies.
- A `prepare` test runs the real transfer script: a non-allowlisted
actor produces no outputs, and an allowlisted dispatch produces the
session policy and an identity that records the target launcher digest.
- Negative control: with the previous `manual-private-transfer.ts`, the
new `prepare` test fails.
- Workflow tests that read this workflow: 111 of 116 passed. The 5
failures in `workflow-target.test.ts` (`IMAGE_ONLY: unbound variable`)
also fail on unchanged `master`. `paid-runner-routing.test.mjs` passed
(7 tests).
- I did not run a live dispatch. The first live proof is a manual
dispatch of chat IDs after merge.

## Risks

- The dispatcher now vouches for the target branch. A target branch with
a changed launcher can write private evidence if an allowlisted operator
dispatches it. The protocol files must still match the trusted copies
byte for byte.
- The eight-cell limit is unchanged. The report job has eight fixed read
slots, and each read policy must fit the 2,048-character session policy
limit. Lifting the limit needs a separate change to the report sessions.
- The `runner-e2e-paid` environment still has required reviewers. That
is a repository setting and this change does not touch it.
- The authority uses `RUNNER_E2E_ALLOWED_ACTOR_IDS`, not `CODEOWNERS`.
Keep that variable aligned with the people who may start paid runs.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 14:26:31 -06:00
DottaandClaude Opus 5.5 744056ecc8 feat(evals): run the staging-soak reliability cases from a TestOps plan (#19549)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The runner full-stack E2E lane runs real Paperclip work against real
agents, and TestOps launches its standing plans
> - PR #19455 added seven staging-soak reliability cases, but no TestOps
plan or workflow lane selects them, so they run only by hand with
personal keys
> - These cases catch restart, interrupt, queue and lost-runner
regressions, so they must run from TestOps before a release with the
standard reports
> - This pull request adds a sibling "Reliability — staging soak"
TestOps plan on the existing plan-2 lane, with native OpenCode and
DeepSeek 1017 through OpenRouter as the plan profile for six of seven
cases
> - The benefit is a cheap, repeatable pre-release reliability launch
with the same Allure/TestOps results, retained report and dashboard as
plan 2

## Linked Issues or Issue Description

Refs #19455

**What existing behavior does this improve?**
The staging-soak reliability cases from #19455 run only by a manual
local command.

**Subsystem affected**
Runner full-stack E2E (`tests/runner-e2e/`) and its TestOps lane in
`.github/workflows/runner-full-stack-e2e.yml`.

**Current behavior**
No TestOps plan includes the cases. The workflow cannot select them from
TestOps. Every cell uses native Codex (`gpt-5.6-sol`) or an ACPX facade
on Daytona, which is expensive.

**Proposed behavior**
A TestOps launch of the Reliability plan runs seven cells (six DeepSeek
1017, one native Codex for steering) through the standing plan-2 path
and reports them like plan 2.

**Reason and benefit**
The cases become a first-class, low-cost, pre-release check with
retained evidence. Plan 2 keeps its 20-case qualification semantics.

**Breaking changes**
None. Plan 2 selection, reporting and history identities stay the same.

## What Changed

- **Plan choice: a sibling plan, not an extension of plan 2.** Plan 2 is
bounded at 20 standing IDs. Its `full-run` / `test-plan-2` tag needs a
complete 20-case attempt, and subset runs do not qualify it. Adding
seven cases would change that denominator and the qualification history.
The sibling plan shares the same TestOps GitHub job, selection,
admission, projection, upload and verification code. A launch selects
cases from exactly one plan (`plan_mixed` fails), and each plan has its
own full-plan size (20 or 7).
- `tests/runner-e2e/reliability-testops.mjs` (new): the mapping module.
It has seven cases, full names `paperclip.Product
E2E.everyday-workflows.<case>`, execution IDs, soak signatures and an
empty `reliabilityRegistration`. A partial, duplicate, plan-2 or
malformed registration is rejected. An empty registration makes
selection fail closed before any spend
(`product_regression_testops_reliability_unregistered` for a bootstrap,
`case_mismatch` for a TestOps launch). No TestOps IDs are invented.
- `product-regression-testops.mjs` / `-cli.mjs`: select a plan by its
case identities or by the new bootstrap input. The projection binds each
cell's own environment (`local` or `daytona`) instead of a fixed
`local`. It adds a `soak-<signature>` tag and reports `plan` and a
per-plan `fullPlan`. Plan-2 history IDs, parameters and results do not
change.
- Catalog: `runner-opencode-deepseek-1017` is now one shared fixture
with `supportedEnvironments: ["local", "daytona"]`. The OpenCode server
driver already runs on Daytona for `runner-opencode` (task-workspaces,
core-compatibility, rolling deployment); only the model differs. The
chat-feature suite stays local, and its definition hash does not change.
`everyday-workflows` gains exactly six DeepSeek plan cells (definition
version 12, 52 + 13 + 3 cells).
- Cell matrix (plan), DeepSeek 1017 on native OpenCode:
`interrupt-redirect`, `reassign-midrun` and `tree-pause-release` run on
`local`. `three-turn-revise` runs on Daytona with the per-turn lease
(S49). `long-silent-command` (S52/S52b) and `queued-across-restart`
(S55/U9) run on the Daytona warm lease.
- `queue-steer` is the one exception. It must steer an active turn, and
the OpenCode server driver declares `steering: false`, so the route
answers `steering_unsupported` and the case cannot pass. The plan uses
the existing steering-capable
`everyday-workflows.runner-codex.daytona.queue-steer` cell for it
(Greptile finding). No DeepSeek `queue-steer` cell exists. The other
native Codex Daytona cells and the ACPX facades stay selectable through
`--group reliability`, but they are outside the plan.
- `runner-full-stack-e2e.yml`: adds the `testops_plan` bootstrap input
(`product-regression` by default), sets the plan-specific bootstrap
launch name and tags, and renames two jobs to plan-neutral names. The
OpenCode steps now match `startsWith(matrix.profileId,
'runner-opencode')`, so the DeepSeek profile gets the pinned OpenCode
binary, runner TypeScript, the local Node check and the remote provider
pack. Before this change, the DeepSeek profile could not run in this
workflow. `OPENROUTER_API_KEY`, `OPENAI_API_KEY` (only for
`queue-steer`) and `DAYTONA_API_KEY` come from the same
`matrix.credentialName` and `matrix.environmentId` gates that other
cells use.
- `runner-e2e-testops-registration.yml` +
`reliability-testops-registration.mjs` (new): a trusted, master-only,
allowlisted-actor, idempotent `workflow_dispatch`. `inspect` only reads.
`register` reuses cases with the same full name, creates missing cases
and the plan, and assigns the plan to plan 2's GitHub job with plan 2's
job parameters. It writes a receipt with a ready-to-commit
`reliabilityRegistration` snippet. Only this job gets `ALLURE_TOKEN`,
and it never runs a provider cell.
- **No schedule and not per PR.** The plan runs only from a TestOps
launch or a manual dispatch. It is not in the daily or weekly cron
(`scheduledSelection` excludes it). A weekly run would add a paid
50-minute Daytona cell every week, but these signatures move with
release-risky changes, not with time.
- Docs: `EVERYDAY-WORKFLOWS.md` (matrix, environment reasons, local
coverage limits, registration steps), `README.md`, `doc/evals.md`.

### Registration still needed (requires TestOps credentials)

1. Dispatch **Runner E2E TestOps registration** from master with
`operation=inspect`.
2. Dispatch it with `operation=register`.
3. Commit the `reliabilityRegistration` snippet from the run summary
into `tests/runner-e2e/reliability-testops.mjs` in one reviewed PR.
4. Launch the plan from TestOps. Tag only a complete seven-case attempt
`full-run`.

### What local plan cells cannot reproduce

The local cells do not reproduce the Daytona-only variants: the remote
`workspace_busy` save after Stop (S47), remote stop settlement during
reassignment (S48), and a lost or replaced remote runner during a tree
hold (S50). S51 (lost runner) has no case yet. The explicit-only native
Codex Daytona extras still cover S47, S48 and S50 on a sandbox.

## Verification

- `node --test tests/runner-e2e/product-regression-testops.test.mjs
tests/runner-e2e/reliability-testops.test.mjs`: 26 pass. These tests
cover fail-closed selection, partial or duplicate registration, plan
mixing, environment binding, per-plan denominators, preserved failure
classes, the CLI select → project → verify flow for a seven-case launch,
registration idempotency and job checks, and the workflow inputs and
credential gates.
- `vitest` for `reliability-testops-catalog`,
`product-regression-testops-catalog`, `everyday-reliability` and
`catalog.test`: 98 pass. These tests bind the mapping to real catalog
cells (profile, model, environment, credentials, single attempt). They
also check that the cells stay out of `--all`, the suite, profile and
environment selectors and the daily/weekly cron. Real grader output must
survive projection.
- `pnpm test:e2e:runner:typecheck`: pass.
- `pnpm test:e2e:runner -- --list --suite everyday-workflows --group
reliability` lists 16 cells: 6 DeepSeek plan cells, 7 native Codex cells
(one is the `queue-steer` plan cell) and 3 ACPX extras. `--matrix-json`
for the DeepSeek plan IDs emits `needsDaytona=true` and three Daytona
OpenCode cells with `credentialName=OPENROUTER_API_KEY`.
- Full `pnpm test:e2e:runner:unit`: the remaining failures happen the
same way on an unchanged master checkout. They are in
`workflow-security` (3), `controller-outage`, `operating-receipts`,
`pi-*`, `private-framework-report` and `remote-native-bootstrap`. #19529
tracks that drift. This PR adds no failures.
- Not run: paid cells, a TestOps registration and a TestOps launch. They
need provider and TestOps credentials, which exist only in protected CI
environments. Catalog and projection tests are not live qualification.

## Risks

- The registration script calls TestOps APIs (`POST /api/testcase`,
`/api/testplan`, `/assign`, `/jobparameter`) through the same endpoints
as the chat catalog reconcile. Tests use fakes, not the live API.
`inspect` comes first, and `register` is idempotent.
- DeepSeek 1017 on Daytona has not run before. The first launch is the
acceptance step for those three cells.
- If a later case needs another capability that OpenCode lacks, it must
use a capable profile in the same way as `queue-steer`. The catalog test
pins the single exception.
- The OpenCode workflow conditions now match any `runner-opencode*`
profile. Today this covers `runner-opencode` and
`runner-opencode-deepseek-1017` only.
- If someone commits a bad `reliabilityRegistration`, module load fails
for both plans. A unit test rejects any committed registration that is
not empty or complete.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use
(shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 14:19:58 -06:00
DottaandClaude Opus 5.5 840838476a perf(runner-e2e): cut the Daytona image build critical path (#19545)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner Full-Stack E2E workflow tests the Runner in Daytona
sandboxes from a signed, content-addressed image
> - Every change to an audited Runner input needs a new image before a
Daytona cell can test it
> - The image job took 10 to 16.5 minutes today, and a restart chaos
campaign waits on it after each runner-side fix
> - Measurement shows two serial waits: a full Rust dependency compile
(about 5 of 6.6 minutes) and a 3.5 to 4 minute download of cached base
layers that starts only after the builder stages finish
> - There is also no way to build only the image: every dispatch reaches
a paid gate that the operator must reject
> - This pull request removes both serial waits and adds an `image_only`
dispatch input
> - The benefit is a shorter fix-to-Daytona-test loop, with every image
security check unchanged

## Linked Issues or Issue Description

**What happened?**

The `Publish verified Daytona image` job is on the critical path of the
control-plane restart chaos campaign. Builds today took 16.5 min (run
42240434011), 15 min (42253920727), 10.5 min (42259118276) and 16 min
(42265637650). Run 42264611573 failed: its RunsOn runner lost
communication with GitHub 7.6 minutes into the build step, during the
Rust compile. GitHub has no log for that job.

**Expected behavior**

A Runner source change should not recompile the locked third-party Rust
graph. Cached image layers should download while the builder stages
compile. An operator should be able to publish only the image, without a
paid gate.

**Steps to reproduce**

Read the BuildKit timeline in the logs of job 127970125495 (run
42240434011). See the stage table below.

## What Changed

- `docker/daytona-runner/Dockerfile`: compile the locked Rust dependency
graph in its own layer against placeholder targets. The layer depends
only on `Cargo.toml`, `Cargo.lock` and the crate manifest. The step
deletes the placeholder outputs and fingerprints, so the next step
always compiles the real Runner sources. The final stage still receives
only the stripped executable.
- `docker/daytona-runner/Dockerfile`: move the content-ID /
source-revision check above the two `COPY --from` steps. It does not
depend on either builder stage, so BuildKit now downloads the cached
base and CLI layers in parallel with the compile. Both copies use `COPY
--link`.
- `.github/workflows/runner-full-stack-e2e.yml`: add the boolean
`image_only` dispatch input. `authorize`, `target_lock`, `catalog` and
`daytona_image` run unchanged. Every other campaign job gets `&&
inputs.image_only != true`. `authorize` rejects `image_only` together
with TestOps inputs.
- `tests/runner-e2e/workflow-image-only.test.ts` (new): pin the job
classification, the top-level guard on each campaign job, the guard on
every protected environment, and the TestOps rejection.
- `tests/runner-e2e/README.md`: document the layer layout and the
`image_only` command.

## Verification

Measured on the same RunsOn `m7i-flex.xlarge` (4 vCPU, 16 GiB, native
x64) runner class. "Before" is master job 127970125495 (run
42240434011). "After (cold)" is this branch, job 128077687214 (run
42271236140), with no dependency layer in the registry cache yet. Its
paid gate was rejected.

| Stage | Before | After (cold dependency cache) |
|---|---|---|
| Whole image job | 16.5 min | **11.6 min** |
| Rust dependency graph | 311 s (inside one 399 s step) | 395 s (own
layer; reused from cache after a master build) |
| `paperclip-runner-core` compile | ~87 s | 65 s |
| Cached base and CLI layer download | 236 s, **after** cargo finished |
257 s, **in parallel** with cargo |
| Copy runnerd + provider pack | 20 s + 50 s | 9 s (`--link`) |
| Provider pack stage | 143 s (parallel) | 152 s (parallel) |
| Export + push | 82 s + 10 s | 81 s + 11 s |
| Dependency layer size | n/a | `target` 852 MB + registry 333 MB
(builder stage only) |

With a warm dependency cache, a Runner source change compiles only the
65 s crate step. The critical path then becomes the ~4 min cached-layer
download or the provider pack. The expected job time is about 7 to 8
min. I measure this after merge with an `image_only` dispatch (master
first, to write the cache, then a Rust-only branch change).

Disk: the instance starts with 38.76 GiB free. The dependency layer adds
about 1.2 GB uncompressed to builder storage. The final image does not
change.


- `vitest run tests/runner-e2e/workflow-image-only.test.ts
tests/runner-e2e/workflow-security.test.ts
tests/runner-e2e/daytona-image.test.ts`: the new test and all image
tests pass. Three tests in `workflow-security.test.ts` (`gates every
provider-secret job with stable actor IDs`, `keeps provider credentials
inside explicitly gated paid workflows`, `isolates private archive trust
from workflows that execute target-branch code`) fail on `origin/master`
without this change, because of `chat-catalog-testops-case.yml`. This
pull request does not change them.
- `node --test tests/runner-e2e/product-regression-testops.test.mjs`:
15/15 pass.
- No local Docker build. The Dockerfile is measured with a
branch-targeted CI dispatch whose paid gate was rejected.

## Risks

- The dependency layer keeps the Rust `target` and registry trees in a
builder-stage layer, and default-branch builds export it to the registry
cache. This adds builder disk and cache upload when `Cargo.lock`
changes. The build prints the size (`du -sk target
/usr/local/cargo/registry`).
- The dependency cache helps only after a default-branch build writes
it. Branch builds read it but never write it. This is the existing cache
policy.
- `image_only` changes the workflow, so it works only after merge to
master.
- No check is removed. Lockfile verification, cosign sign and verify,
the anonymous pull check, the provider-pack manifest match, the
default-branch-only cache write and the actor allowlist are unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, GitHub CLI). Extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 14:15:16 -06:00
Nicky LeachandClaude Opus 5.5 79c724b292 ci: run the Runner support suite as a required PR check (#19537)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Runner support suite (`pnpm test:e2e:runner:unit`) checks the
code behind paid Runner runs: workflow security policy, evidence
oracles, cost receipts and controller recovery.
> - Only a manual Docker Runner check ran this suite. PR CI did not run
it.
> - So changes on master broke 16 of its tests, and nobody saw it.
#19527 and #19529 fix those tests.
> - This pull request runs the suite in PR CI and makes the `verify`
check require it.
> - The benefit is that a change that breaks the suite fails on its own
pull request, before merge.

## Linked Issues or Issue Description

Refs #19527 and #19529 (both merged; they repair the suite). Also
accounts for #19533, which added the sandbox provider job.

**Merge order:** #19527 and #19529 have merged, so the suite now passes
on master. This PR can merge.

**What existing behavior does this improve?**

PR CI for `pr-trusted.yml`.

**Current behavior**

`pnpm test:e2e:runner:unit` runs only in `docker-runner-check.yml`, by
manual dispatch. Master changes broke 12–13 tests by Nov 3 and 16 by Nov
6, with no signal on any pull request.

**Proposed behavior**

A required "Runner support suite" job runs the suite on pull requests.

**Reason and benefit**

Drift fails on the pull request that causes it. Authors update
deliberate pins (profile versions, suite hashes, workflow policy) in the
same pull request.

## What Changed

- `pr-trusted.yml`: new job `verify_runner_e2e_support` ("Runner support
suite"). It uses the same steps as other trusted jobs: checkout without
credentials, setup-node directly before pnpm, the npm overrides, the
shared `setup-ci-dependencies` restore, and CI profiling. Then it runs
`pnpm test:e2e:runner:unit`. It has no secrets. Timeout: 20 minutes.
- `pr-trusted.yml`, job `verify`: the job now needs the new job and
checks its result. A failed, cancelled or wrongly skipped run fails
`verify`.
- `turbo.json`: new routing task `ci:runner-e2e-support`. Its inputs are
`server/**`, `cli/**`, `packages/**` and `ui/**`. Changes to `tests/**`,
`scripts/**` and `.github/**` already force full CI, so they also run
the job.
- `.github/ci-task-matrix.json`: one lane for the new job.
- Tests: `ci-plan.test.mjs`, `ci-profile.test.mjs`, `e2e-shard.test.mjs`
and `workflow-security.test.ts` now include the new job. A UI component
change now selects this one Runner task. The suite imports UI modules
and checks that catalog-listed UI files stay tracked in git, so UI
changes can break it.
- CI profiling: when `PAPERCLIP_CI_PROFILE` is on, the suite's Vitest
config adds `scripts/ci-profile/vitest-reporter.mjs`, and `run-unit.mjs`
runs the Node tests through `scripts/ci-profile/node-test.mjs`. So the
job uploads per-test timings, like the root and Runner package suites.
The launcher loads the profiling helper only after its "Vitest is not
installed" check.
- `tests/runner-e2e/README.md`: says that PR CI runs the suite, and that
deliberate pin changes go in the same pull request.

## Verification

- `node --test .github/scripts/tests/*.test.mjs scripts/ci/*.test.mjs
scripts/__tests__/e2e-shard.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs
scripts/__tests__/runner-release-crate-cache.test.mjs`: 830 passed, 3
failed. The same 3 fail on the base branch without this change. They
need local tools that are not on this Mac.
- A local run with `PAPERCLIP_CI_PROFILE=true` wrote per-test and
per-file timing records for Vitest and for Node. Without profiling,
output and exit codes do not change. Both `unit-launcher.test.mjs` tests
pass.
- The aggregate test now covers the new job: `verify` rejects it when it
fails, is cancelled, or is skipped while selected.
- The pnpm bootstrap policy tests in `workflow-security.test.ts` pass
with 10 setups in `pr-trusted.yml`.
- On this PR's merged tree (`80b526d697`, master plus this PR): `pnpm
test:e2e:runner:unit` passed, Vitest 320 files (6,830 tests) and
node:test 373, in 58s. It also passes on master `4c1f629c31`.
- Suite runtime on a 10-core Mac: 43–63s. With `--maxWorkers=4`: Vitest
118s, about 490s of CPU in total. On a 4-vCPU GitHub runner I expect
about 3–4 minutes for the tests and 1–2 minutes for setup. The job runs
in parallel with the longer test shards, so it should not lengthen CI.
- `pr.yml` calls `pr-trusted.yml@master`. This PR's own CI does not run
the new job. The first real CI timing comes from the first pull request
after merge.

## Risks

- More runner minutes: one extra job on most pull requests (all except
documentation-only ones).
- The suite has not run on Linux CI recently: old `docker-runner-check`
logs have expired. A Linux-only failure would show up on the first pull
requests after merge. The job can be removed from the `verify` list
quickly if needed.
- #19229 removes `version: 9.15.4` from the pnpm steps in this file.
Whichever PR merges second needs a small rebase.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), by Anthropic, in Claude Code with
tool use (shell, file edits, git, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 11:50:42 -08:00
DottaandClaude Opus 5.5 187c0f6d6e fix(runner-e2e): prepare the artifact oracle for every suite that runs it (#19518)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner Full-Stack E2E workflow runs paid Daytona restart cells
for the control-plane restart chaos campaign
> - Manual run 42243751291 failed
`daytona-restart-continuity.runner-codex.daytona.recover-controller`
before task creation: `docker version` timed out after 10 seconds in the
artifact oracle preflight
> - The everyday flow requires the Python artifact oracle by case alone
(`requiresEverydayArtifactOracle(caseId)`). The workflow step that
starts Docker and pulls the oracle image also filters by suite, so
suites that reuse `recover-controller` skip it
> - On a cold RunsOn host the first `docker version` takes about 20
seconds (the everyday cell in the same run shows this), so the 10-second
preflight fails when the workflow step did not run first
> - This pull request gates the oracle step by case only, which matches
the runtime requirement
> - The benefit is that the Daytona restart cell reaches task creation
instead of failing in setup

## Linked Issues or Issue Description

**What happened?**

Run https://github.com/paperclipai/paperclip/actions/runs/42243751291
job `daytona-restart-continuity.runner-codex.daytona.recover-controller`
failed with `Artifact sandbox qualification failed before task creation:
... docker version ... timed out after 10 seconds`. The job log shows no
"Prepare pinned Python artifact oracle image" step.

**Expected behavior**

Every cell whose flow runs the artifact oracle prepares Docker and the
pinned oracle image before paid execution.

**Steps to reproduce**

Enumerate `runnerMatrix` cells where
`requiresEverydayArtifactOracle(task.id)` is true. The cells in
`daytona-restart-continuity`, `rolling-web-deployment`,
`rolling-web-deployment-warm` and `remote-workspace-storage` do not
match the old workflow condition.

## What Changed

- `.github/workflows/runner-full-stack-e2e.yml`: the "Prepare pinned
Python artifact oracle image" step is now gated by `matrix.caseId` only.
The suite filter is removed.
- `tests/runner-e2e/codex-ci-sandbox.test.ts`: the oracle gate test now
checks every catalog cell, not only `everyday-workflows` cells, and
rejects a suite filter.
- `tests/runner-e2e/workflow-security.test.ts`: the pinned condition
string follows the new gate.

## Verification

- Red: with only the test change, `npx vitest run --config
tests/runner-e2e/vitest.config.ts
tests/runner-e2e/codex-ci-sandbox.test.ts` fails on `prepares the
artifact oracle for exactly the stories that execute downloads`.
- Green: with the workflow change, the same command passes (13/13).
- `tests/runner-e2e/workflow-security.test.ts` locally shows the same 5
failures before and after this change (`pr-trusted.yml` step counts and
`chat-catalog-testops-case.yml` credential checks). They do not involve
this step.
- Live proof: a narrow manual dispatch after merge.

## Risks

- Low risk. Cells in four more suites now run a Docker version check, a
pinned image pull and an inspect before paid execution. This adds about
20 to 30 seconds per cell. The image is pinned by digest. The step has
no secrets.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, GitHub CLI, file edit).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 12:40:12 -06:00
Nicky LeachandClaude Opus 5.5 3b3037aab5 fix(ci): run the sandbox provider tests in required CI (#19533)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Sandbox provider plugins (Daytona, E2B, Cloudflare, Kubernetes,
Modal, and others) run agents in remote sandboxes
> - The providers are outside the root pnpm workspace, so `pnpm
test:run` and the CI test lanes never run their Vitest suites
> - paperclipai/paperclip#18839 typechecks those tests, but a test that
fails at runtime still fails nothing
> - The providers have about 1,040 tests, and they finish in about 15
seconds on Linux
> - This pull request adds a required `Sandbox provider tests` job that
runs them when a change can affect them
> - The benefit is that a broken provider, plugin SDK, or shared
contract fails CI on the pull request instead of in a release or a paid
end-to-end run

## Linked Issues or Issue Description

Refs #18839

**What happened?**

No CI job runs the sandbox provider tests. `runner-full-stack-e2e.yml`
builds the Daytona plugin for a paid nightly run, but it does not run
any provider's `pnpm test`.

**Expected behavior**

A pull request that can break a provider runs that provider's tests, and
a failure blocks `ci / verify`.

**Steps to reproduce**

Add `expect(1).toBe(2)` to a test in
`packages/plugins/sandbox-providers/e2b/src`. Open a pull request. CI
stays green.

## What Changed

- Added `scripts/test-sandbox-providers.mjs` and its tests. The script
installs the providers through `typecheckTestPrograms` with no programs,
because that function already owns the provider install and the plugin
SDK link. Then it runs each provider's `pnpm test`, four at a time, and
prints each provider's output as one block. It fails if a suite fails or
if no provider has a test script.
- The script skips `cloudflare/bridge-template`, because Cloudflare's
Vitest config already runs those tests.
- Added the `sandbox_provider_tests` job to
`.github/workflows/pr-trusted.yml`, and made `ci / verify` require its
result.
- Added the `//#ci:sandbox-providers` task to `turbo.json` and its lane
to `.github/ci-task-matrix.json`. The CI planner runs the job only when
a change touches `packages/plugins/sandbox-providers/**`,
`packages/plugins/sdk/**`, or `packages/shared/**`. `packages/shared` is
the plugin SDK's only workspace dependency. Global CI inputs, such as
`scripts/**`, `.github/**`, `pnpm-lock.yaml`, or any `package.json`,
still select every job.
- The job turns off core dumps. A Daytona capacity test kills a process
at a file-size limit inside the exported directory. If core dumps are
on, the core file joins the export, and the test sees 12 files instead
of 11.
- Updated the job lists in `scripts/__tests__/e2e-shard.test.mjs` and
the job count in `.github/scripts/tests/ci-profile.test.mjs`. Added
planner tests to `.github/scripts/tests/ci-plan.test.mjs`. One of these
tests fails if the plugin SDK gets a new workspace dependency that is
not in the job's inputs.
- Added the new test file to `test:release-registry`, and documented the
script in `doc/DEVELOPING.md`.

## Verification

- `node scripts/test-sandbox-providers.mjs` on macOS: 8 of 9 suites pass
in 31.6 s, including the install. Kubernetes fails 12 `file-sync` tests,
because macOS has no `/proc/self/fd`. Some Daytona tests skip, because
they need GNU `tar` and coreutils.
- The same script in a Linux container (`node:24`, a non-root user, core
dumps off): all 9 suites pass, with 1,012 tests passed and 26 skipped.
The script took 15 s after the install.
  - Kubernetes: 205 passed.
  - Daytona: 590 passed and 16 skipped.
- With core dumps on, Daytona's `preserve literal capacity repair` test
fails. The export includes a `core` file. With `--ulimit core=0`, it
passes.
- To look for timing failures under load, I ran the whole script 12 more
times in the container, limited to 2 CPUs (8 runs) and 1 CPU (4 runs).
All 12 runs passed. They took 45 to 76 s on 2 CPUs and 73 to 102 s on 1
CPU.
- `ubuntu-latest` has `zstd`, and my container did not, so some Daytona
tests skipped in those runs. With `zstd` installed, Daytona passed with
598 tests and 8 skipped, and only the live tests skipped. One earlier
`zstd` run failed while the stress runs used the same CPUs, and I did
not capture its output. The next run passed.
- `node --test` on every test that reads `pr-trusted.yml`, the CI
planner, or the task matrix: 816 passed. These include
`.github/scripts/tests/*.test.mjs`,
`scripts/__tests__/e2e-shard.test.mjs`,
`scripts/ci/test-selection-workflow.test.mjs`, and the new tests.
- `pnpm run test:release-registry` on macOS: 251 of 255 pass, including
the new tests. The 4 failures are `wait_for_npm_package_versions` tests
in `scripts/release-lib.test.mjs`. They fail the same way on master on
the same machine, and CI runs this suite on Linux.
- `actionlint` 1.7.7 reports no errors on the changed workflow.
- I called `planTasks` from `.github/scripts/ci-plan.mjs` with one
changed file at a time. `server/src/index.ts` and `ui/src/App.tsx` leave
`sandbox_provider_tests` off.
`packages/plugins/sandbox-providers/e2b/src/plugin.ts` and
`packages/shared/src/index.ts` turn it on.

This pull request cannot run its own new job. `pr.yml` calls
`pr-trusted.yml@master`, so the job first runs on pull requests after
this one merges. After merge, check the first pull request that changes
a provider or `packages/shared`.

## Risks

- **First real run is after merge.** I measured the time on Linux in
Docker, not on a CI runner. CI runners are about 2 times slower than my
machine, so I expect about 2 to 3 minutes for the job, including the
dependency restore. The job runs in parallel with the other jobs, and
the slowest required job takes about 13 minutes. So the job should not
make CI slower.
- **Open provider pull requests.** After this merges, open pull requests
that change a provider run its tests. If those tests fail, `ci / verify`
blocks them until they are fixed.
- **`packages/shared` changes often.** Any change there runs this job. A
Linux-only provider failure would block those pull requests. All suites
pass on Linux today.
- **One unexplained Daytona failure.** One Daytona run with `zstd`
failed under CPU contention, and I did not capture its output. Some
Daytona tests have short deadlines. If the job is flaky after merge,
quarantine the failing test or revert the job, and do not leave `ci /
verify` red.
- **Path rules can be too narrow.** If a provider starts to import
another workspace package, a change to that package does not run this
job. The plugin SDK dependency test covers the SDK. It does not cover a
provider that imports a workspace package directly. Today no provider
does.
- **Overlap with paperclipai/paperclip#19489 and
paperclipai/paperclip#19522.** Both pull requests change the provider
install in `scripts/typecheck-test-programs.mjs`. This pull request does
not edit that file. It calls the same install function, so it uses their
install changes when they merge. All three pull requests edit the same
section of `doc/DEVELOPING.md`, so a small text conflict is possible.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with
tool use (shell, file edits, GitHub CLI, Docker). Extended reasoning was
on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 10:34:03 -08:00
Nicky LeachandClaude Opus 5.5 35280516c0 fix(ci): set up pnpm in the Vitest selection job like other trusted jobs (#19527)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Trusted CI workflows set up pnpm in a fixed way. Node comes first,
and npm audit, fund and update-notifier calls are off.
> - A policy test in the Runner support suite checks every pnpm setup in
`pr-trusted.yml` for this shape.
> - PR #18348 added a pnpm setup to the new Vitest selection job. That
setup did not have the npm overrides, and setup-node was not directly
before it.
> - PR CI does not run the Runner support suite. So the policy test
failed on master, and nobody saw it.
> - This pull request gives the selection job the same pnpm setup as the
other eight jobs.
> - The benefit is that every trusted pnpm setup follows one rule again,
and the policy test passes.

## Linked Issues or Issue Description

Refs #18348 (added the job). Refs #18625 (its description noted the
"nine bootstraps where eight are expected" failure). Related: #19229
also edits the pnpm steps in `pr-trusted.yml`.

**What happened?**

`pnpm test:e2e:runner:unit` fails on master in two tests: "keeps pnpm
bootstrap registry telemetry out of trusted workflow setup" and
"installs a modern Node runtime before every trusted pnpm bootstrap".
`pr-trusted.yml` has nine pnpm setups. The ninth, in `test_selection`,
has no `NPM_CONFIG_AUDIT`, `NPM_CONFIG_FUND` or
`NPM_CONFIG_UPDATE_NOTIFIER` override. The step directly before it is
the eligibility check, not setup-node.

**Expected behavior**

Every trusted pnpm setup has the three overrides, and setup-node is the
step directly before it.

**Steps to reproduce**

Run `pnpm test:e2e:runner:unit` on master. Read the two failures above.

## What Changed

- `pr-trusted.yml`, job `test_selection`: add "Setup Node.js for pnpm
bootstrap" directly before "Setup pnpm". Both steps use the existing
eligibility condition. The pnpm step now sets the three npm overrides.
Both actions keep their pinned SHAs, with version comments added.
- `tests/runner-e2e/workflow-security.test.ts`: the expected count of
pnpm setups is now set per workflow: 8 for `runner-full-stack-e2e.yml`
and 9 for `pr-trusted.yml`. The checks on each setup do not change.

## Verification

- On this branch, the two pnpm tests in `workflow-security.test.ts`
pass. Before this change, they failed with "expected 8, got 9".
- The other three failures in that file are fixed separately in the
companion PR that repairs the rest of the suite.
- `node --test scripts/ci/test-selection-workflow.test.mjs`: 4 passed.
- The workflow parses as YAML. The `test_selection` steps are now:
checkout, setup-node, eligibility check, setup-node for pnpm, setup
pnpm, restore dependencies, select, upload, publish.
- `pr.yml` calls `pr-trusted.yml@master`. So this PR's own CI runs the
master copy of this workflow. The change runs in CI only after merge.

## Risks

- Low risk. The new setup-node step runs again with the same Node 24
version. It adds about one second when selection is eligible. When
selection is not eligible, both steps are skipped, as before.
- `test_selection` has `continue-on-error: true`. If the new steps fail,
the original test matrix still runs.
- #19229 removes `version: 9.15.4` from every pnpm step in this file. If
both PRs merge, the second one needs a small rebase on this step.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), by Anthropic, in Claude Code with
tool use (shell, file edits, git, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 10:14:44 -08:00
Nicky LeachandClaude Opus 5.5 e3041f5481 fix(ci): let the lockfile bot own the sandbox provider lockfiles (#19522)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The repository has a supply-chain rule: contributors do not commit
lockfiles. A bot and verified Dependabot commits write them.
> - The rule checks only the root `pnpm-lock.yaml`.
> - The sandbox providers are outside the root workspace, so they have
two more pnpm lockfiles. Pull requests edited them by hand, for example
#12791, #18839 and #19489.
> - This pull request makes the same rule and the same bot cover those
two lockfiles.
> - The benefit is one lockfile rule for every managed pnpm lockfile,
and no lockfile edits in contributor pull requests.

## Linked Issues or Issue Description

Refs #19489

**What existing behavior does this improve?**

The lockfile policy and the lockfile refresh bot.

**Subsystem affected**

CI: `.github/scripts/lockfile-trust.mjs`,
`.github/workflows/refresh-lockfile.yml`, and the sandbox provider
installs.

**Current behavior**

`lockfile-trust.mjs` blocks contributor edits to `pnpm-lock.yaml` only.
A contributor can edit
`scripts/typecheck-sandbox-providers/pnpm-lock.yaml` or
`packages/plugins/sandbox-providers/daytona/pnpm-lock.yaml`, and CI
passes. The refresh bot does not update these files.
`typecheck:test-programs` fails on a stale provider lockfile and tells
the contributor to commit a new one.

**Proposed behavior**

The policy blocks contributor edits to all three lockfiles. The refresh
bot regenerates all three after merge. CI resolves a stale provider
lockfile in place for the run, as it does for the root lockfile.

**Reason and benefit**

Lockfile content comes only from the bot or from verified Dependabot
commits. Contributors do not need to run pnpm in special folders.

**Breaking changes**

None. A pull request that edits a provider lockfile now fails the policy
check. Remove the lockfile change from the pull request. The bot commits
it after merge.

## What Changed

- `lockfile-trust.mjs` now exports `LOCKFILES`: the root lockfile, the
shared sandbox provider lockfile, and the Daytona lockfile. The policy
blocks contributor edits to each of them. A Dependabot pull request must
match a verified Dependabot commit for each lockfile that it changes.
The `merged` command and the reporting-lock proof still use only the
root lockfile.
- `check-pr-lockfile.mjs` (advisory check) uses the same list.
- `refresh-lockfile.yml`:
- A job-level `LOCKFILES` env lists the same three files. A test checks
that it equals the policy list.
- A new step regenerates the two provider lockfiles with
`--resolution-only --ignore-scripts --no-frozen-lockfile`.
- The unexpected-changes check, the change check, and `git add` cover
all three lockfiles.
  - The release-age gate checks the added versions in every lockfile.
- A held refresh saves all three lockfiles at their repository paths,
with a `source-locks.sha256` file. The next run reuses them only if
every master lockfile is unchanged. An artifact in the old format is not
reused.
  - The PR title is now `chore(lockfile): refresh pnpm lockfiles`.
- `check-lockfile-release-age.mjs` accepts more than one
`--base`/`--head` pair.
- `scripts/typecheck-test-programs.mjs`: if the frozen install of the
provider workspace fails, the script resolves the lockfile in place and
installs frozen again.
- `scripts/build-standalone-public-packages.mjs`: the same fallback
applies to a package with a committed lockfile (Daytona). `release.sh`
already restores tracked files when it exits.
- `doc/DEVELOPING.md` and `doc/RELEASE-AUTOMATION-SETUP.md` describe the
managed lockfiles. They also say that the Pi and Kimi distribution
lockfiles stay hand-managed, because they hold attested provider pins.

## Verification

- `node --test .github/scripts/tests/*.test.mjs
scripts/__tests__/build-standalone-concurrency.test.mjs`: 731 tests
pass. New tests cover:
- the policy for provider lockfiles, including a real git history with a
Dependabot commit and a later hand edit
  - the workflow and policy lists match
  - the unexpected-changes check, run with bash against a fixture
- a held refresh round trip, and its invalidation when a provider
lockfile changes on master
  - the release-age gate across lockfile pairs
  - the inline fallback in both install scripts
- `node --test scripts/typecheck-test-programs.test.mjs`: the install
tests pass. Two tests fail locally only because this worktree has no
root `node_modules` (no `tsc`). They also fail without this change.
- I ran the new bot commands on master with pnpm 9.15.4. No package
version changed. pnpm 9.15.4 removes `libc:` fields and adds
`deprecated:` notes, so the first refresh is a metadata-only change. The
release-age gate passes it because it adds no versions.
- I ran the release-age and held-lock steps locally with bash against
this commit. The gate reported `eligible=true`, and `sha256sum --check`
accepted the saved hashes.
- The workflow YAML parses, and `bash -n` accepts every `run` block.
- Not testable before merge: the policy job loads `lockfile-trust.mjs`
from the base branch, so the new rule applies only to pull requests
opened after this merges. The refresh workflow runs only on master.

## Risks

- The release build now resolves a stale Daytona lockfile in place, from
the registry. The root release install already resolves without a frozen
lockfile, so this is the same level of trust.
- The paid runner e2e job still installs Daytona with
`--frozen-lockfile`. After a Daytona manifest change merges, its Daytona
cases fail until the bot's refresh merges. This matches the rule that
paid jobs refuse a new root lockfile until its refresh merges.
- Dependabot pull requests for the provider entries in #19489 change a
provider lockfile. They pass only if each lockfile matches a verified
Dependabot commit, the same as for the root lockfile.
- Low risk otherwise: the bot keeps its existing token, environment, and
merge rules.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, and git).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 10:07:57 -08:00
Nicky LeachandClaude Opus 5.5 83c6832ccb fix(build): typecheck package, Playwright, and e2e test files in CI (#18839)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Runner, plugin, and Playwright tests verify how those agents run,
connect, and appear in the board
> - Most package tsconfigs exclude tests, and Vitest strips types, so a
type error in those tests fails nothing
> - The tests drifted to 803 real type errors, and some of those errors
hid assertions that checked nothing
> - paperclipai/paperclip#18739 and paperclipai/paperclip#18738 check
the server test tree and four test projects, but not these files
> - This pull request adds a required check for these test files, fixes
every error, and fails CI if a test file is in no program
> - The benefit is that a type error in any test file outside
`server/src` now fails CI

## Linked Issues or Issue Description

Refs #18739
Refs #18738

**What happened?**

517 test files were in no program that required CI checks. A type error
in those files did not fail CI.

**Expected behavior**

CI fails on a type error in any test file.

**Steps to reproduce**

Add `export const value: number = "text";` to a test file under
`packages/paperclip-runner/src` on master. CI stays green, because
`tsconfig.json` excludes test files and Vitest strips types.

**Measured gap**

Each row is a program with its tests included, on one thread. File and
error counts are for head `7d4b5e907a`. Memory peaks were measured
before the two latest merges of master.

| Group | Test files | Real errors | Peak |
| --- | ---: | ---: | ---: |
| `paperclip-runner` Node source | 290 | 644 | 1.09 GiB |
| `paperclip-runner` surfaces | 9 | 12 | 0.42 GiB |
| `paperclip-runner` devtools and examples | 14 | 0 | 0.68 GiB |
| `sandbox-providers/kubernetes` | 21 | 65 | 0.44 GiB |
| 8 other sandbox providers and `bridge-template` | 32 | 4 | 0.34 GiB |
| `plugins/sdk` | 22 | 15 | 0.41 GiB |
| `plugin-workspace-diff` | 5 | 0 | 0.37 GiB |
| `plugin-orchestration-smoke-example` | 1 | 1 | 0.15 GiB |
| `kv-demo-mcp-server` | 3 | 0 | 0.13 GiB |
| `packages/db` scripts | 1 | 0 | 1.01 GiB |
| 8 Playwright suites under `tests/` | 52 | 25 | 1.72 GiB |
| `tests/runner-acceptance` | 3 | 0 | 0.58 GiB |
| `tests/e2e`, through layer declarations | 63 | 37 | at most 1.59 GiB |
| `ui` `AttentionInteractionResolver.test.tsx` | 1 | 0 | n/a |
| **Total** | **517** | **803** | |

27 of the 803 errors came from master while this pull request was open.
Each merge of master brought new test code that no check had read.

Errors that were config errors, not real errors:

- One TS2688 for each provider, because the root install does not
install provider dependencies.
- 9 in the Runner Node program (TS7016 and TS6059).
- 2 in `tsconfig.browser.json`. TypeScript 7 removed `baseUrl`, so that
config was invalid.
- 213 in the Playwright suites (`@/*` aliases and Vite types).
- About 60 in `tests/e2e` (import extensions and aliases).

`packages/plugins/create-paperclip-plugin` has no gap. Its tsconfig
excludes its test, but `cli/tsconfig.json` reaches that test through an
import. `server/test-baselines` also has no gap, because
`tests/lifecycle-baseline` reads it.

## What Changed

- Added `scripts/typecheck-test-programs.mjs` and its tests. The
`Typecheck + Release Registry` job runs it. It checks each program in
`PROGRAMS` on one thread.
- The script fails if a tracked `*.test.ts(x)` or `*.spec.ts(x)` file
outside `server/src` is in no program. The compiler file list decides
this, because `include` drops `foo.test.tsx` when `foo.test.ts` exists.
- Added a `tsconfig.test.json` to each uncovered package, and
`tests/tsconfig.playwright.json`.
- Added `tests/e2e` to `PROJECTS` in
`scripts/typecheck-test-projects.mjs`, with a `tsconfig.json` and a
`tsconfig.layers.json`.
- Added `scripts/typecheck-sandbox-providers`. It is a small workspace
that installs provider SDKs from one lockfile. I did not add a lockfile
inside each provider, because
`scripts/build-standalone-public-packages.mjs` uses a frozen install
when one exists, and that changes what a release resolves.
- Corrected `packages/paperclip-runner/tsconfig.browser.json` for
TypeScript 7.
- Set `rootDir` to `..` in
`packages/paperclip-runner/tsconfig.test.json`, because one Runner test
imports from `packages/adapter-utils/src`. The program does not emit
files, so this changes no build output.
- Moved the Kubernetes provider from `vitest ^3.2.4` to `^4.1.8`, the
version the other providers use. Vitest 3 requires `tinypool ^1.1.1`,
which has two critical advisories (GHSA-85c8-ppgw-ccpr and
GHSA-5gmw-xhrv-c9v3). The first patched version is 2.1.2. Its tests have
the same result on both versions: 189 passed, and the same 12
`file-sync` tests failed.
- Moved the Kubernetes provider from `zod ^3.24.2` to `^4.6.6`, and gave
its two `z.record` calls a key schema. This is the only production
change, and it stays in this pull request. See "Production change for
review".
- Renamed `AttentionInteractionResolver.test.ts` to
`AttentionInteractionResolver.replace.test.ts`, so TypeScript reads the
`.test.tsx` beside it.
- Fixed all 803 errors, with changes to test files and test support
only. The fix for `contracts/native-execution.test.ts` is the same as
the fix in paperclipai/paperclip#18739, so those two changes merge
cleanly.
- Updated `doc/DEVELOPING.md`.

This pull request does not edit `server/scripts/typecheck-layers.mjs`.
paperclipai/paperclip#18739 and paperclipai/paperclip#18738 have both
merged, and this branch includes them.

### Production change for review

The new lockfile made dependency review flag `zod@3.25.76` as a
high-severity advisory (GHSA-mrhx-c7xc-m3w5). The first patched version
is 4.6.6, so 3.x has no fix. The published Kubernetes plugin uses zod to
validate operator config. The zod 3 source does not compile against zod
4, so a typecheck-only override cannot fix this.

The existing tests do not prove this change, because they also passed
with the invalid one-argument `z.record`. So I parsed 45 inputs with the
same schema on zod 3.25.76 and zod 4.6.7:

| Result | Inputs | Detail |
| --- | ---: | --- |
| Identical | 23 | Includes defaults, valid config, and a `__proto__`
key |
| Same outcome, new error wording | 21 | For example, `Invalid url` is
now `Invalid URL`. `validateConfig` shows this text to operators. |
| Different outcome | 1 | `jobTtlSecondsAfterFinished: 2 ** 60` was
valid and is now rejected. The new limit is `Number.MAX_SAFE_INTEGER`. |

I did not test this against a real cluster.

The decision is to keep this change in this pull request. Since then,
paperclipai/paperclip#17364 has moved the rest of the repository to zod
4.6.x, so the Kubernetes plugin now matches it.

### Behavior change for review

`reducedMotion` under `use` is not a Playwright option. It had no effect
in 11 configs and 2 `test.use` calls, so those suites ran with
`no-preference`. The options are now under `contextOptions`, so reduced
motion now takes effect. `app-shell-design.spec.ts` asserts a real
transition, so it now asks for `no-preference` itself.

The baseline manifest has `snapshotCount: 0`, so this invalidates no
published baseline. A frozen trial reference PNG can differ.

### Findings

These are product questions. I did not change them, and I did not hide
them with casts:

1. **`PrpEvent` and its schema disagree.** The TypeScript type in
`src/protocol/replay-contract.ts` requires `normalizedSessionId`. The
`required` list in `protocol/schemas/event.schema.json` does not include
it. `validatePrpEvent` accepts events that the type forbids.
2. **`requestCheckboxConfirmation` is stricter than its runtime.** It
uses the output type, so plugin authors must pass `continuationPolicy`
and `allowDeclineReason`, although the schema gives them defaults.
`z.input` would fix this.
3. **`RetainedRunnerdMaintenanceInput.backend.kind` is a free string.**
The quarantine domain comes from `descriptor.kind`, but the release uses
`proof.backend.kind`. If they differ,
`completeRetainedNativeSessionCleanup` returns 0 and the quarantine
stays. The production caller passes `"runner"` today, so this is not a
live bug.
4. **`CodexHarnessSession.close(input?)` is optional, but the contract
requires `{ reason }`.**

These are tests that verified nothing. I corrected them, and they pass:

- `backends/native-instruction-measurement.test.ts` filtered on
`placement === "core"`, which is not a valid placement. It measured an
empty tool list. It now measures 14 tools.
- `drivers/acpx/codex-acpx-driver.test.ts` checked for `usage.updated`,
which is not an event type.
- The `hostFixture()` helper in `drivers/acpx/runtime-host.test.ts`
ignored two overrides, so
`expect(prepareSandbox).not.toHaveBeenCalled()` could not fail.
- `live/live-session.test.ts` set `agent: "pi"` before a spread that
overwrote it.
- `native-distribution-integrity.test.ts` passed an AbortSignal where
`openCommand` expects `{ signal }`.

These are gaps that this pull request does not close:

- **The root lockfile still has some old zod versions.**
paperclipai/paperclip#17364 moved the direct zod dependencies to 4.6.x.
`pnpm-lock.yaml` still resolves `zod@3.25.76`, `4.1.11`, and `4.4.3`,
which are below the patched 4.6.6.
`packages/paperclip-runner/scripts/pi-distribution/package.json` pins
`zod` to `3.25.76`, and some agent SDKs resolve `4.4.3`. Dependency
review does not flag them, because this pull request does not add them.
- **Dead GitHub coverage.** `chat-adapters-ui-providers.spec.ts`
returned early for GitHub, so four later `provider === "github"`
branches never ran. I removed them. They included the private key file
and paste match, and the reconnect flow. Nothing asserts those behaviors
in that spec now.
- **Provider tests do not run in CI.** The root `vitest.config.ts` lists
only Daytona. The other 8 providers, `bridge-template` included, are
typechecked now, but CI does not run their tests.
- **One `packages/db` test never runs.**
`scripts/clean-poisoned-claude-sessions.test.ts` is outside the
`include` of `packages/db/vitest.config.ts`.
- **`plugin-orchestration-smoke-example` does not collect.**
`@paperclipai/shared` is not linked in its `node_modules`. It passes 3
of 3 with an alias.

## Verification

Run these commands:

```sh
node scripts/typecheck-test-programs.mjs --low-memory
pnpm --filter @paperclipai/server typecheck
node scripts/typecheck-test-projects.mjs --low-memory --reuse-declarations
node --test scripts/typecheck-test-programs.test.mjs
```

Results on macOS (arm64), Node 26.12. CI uses Node 24, and its
`Typecheck + Release Registry` job also passes. At head `7d4b5e907a`,
the test programs, `tests/e2e`, and the script tests pass again with 0
errors. The times and peaks below come from the run before the two
latest merges of master:

| Check | Result |
| --- | --- |
| Test programs, `--low-memory` | Exit 0, 0 errors, 42 seconds, peak
1.78 GiB |
| Server layers, low memory | Exit 0, 0 errors, 38 seconds, peak 3.41
GiB |
| Test projects, including `tests/e2e` | Exit 0, 0 errors, 13 seconds,
peak 1.59 GiB |
| Script tests | 12 passed |
| Injected type error in each of 23 places | Each one failed its program
with TS2322 |
| Coverage check | 0 unchecked files, 10 seconds |

I did not run this in a 4 GiB container. The peaks above are `maximum
resident set size` for the largest single process.

**Runtime check of the fixes.** Types do not prove that a test still
works, so I ran the suites before the two latest merges of master:

| Suite | Result |
| --- | --- |
| Runner, full suite | 4,885 passed, 3 failed, 13 skipped |
| Sandbox providers, 10 packages | 939 passed, 12 failed, 36 skipped |
| Plugin SDK | 200 passed |
| `kv-demo-mcp-server` | 12 passed |
| Renamed UI test and its sibling | 4 passed |

All 15 failures exist without these fixes or come from the local
machine:

- **3 Cursor Cloud timeouts.** `artifact-delivery.test.ts` has 1 and
`cursor-cloud-driver.test.ts` has 2. They time out in a full local run,
and all 133 tests in those files with `codex-security-config.test.ts`
pass when run alone. They fail in the same way with those two files at
their unedited versions.
- **12 Kubernetes `file-sync.test.ts` failures.** They fail in the same
way before and after the fixes. The first failure is that
`/proc/self/fd` does not exist on macOS.

After each of the two latest merges of master, I ran only the Runner
test files that I changed:

| Merge | Result |
| --- | --- |
| `885c249cb9`, 8 files | 534 passed |
| `7d4b5e907a`, 7 files | 672 passed, 16 failed |

The 16 failures are in the two `source-reconciliation` test files. They
fail in the same way with master's unedited versions of those files. 15
of them fail with `ACPX sidecar exited at initialize`, so I think my
local build of the Rust and sidecar binaries is out of date. I did not
rebuild them to check. CI builds them from scratch, and all Runner jobs
pass in CI at `7d4b5e907a`.

**Not run:**

- `tests/e2e` and the 8 Playwright suites need a server, so most of them
have only compile checks. `playwright test --list` loads 308 tests from
the main e2e config.
- `packages/plugins/sandbox-providers/kubernetes/test/integration` needs
a cluster, so it has only compile checks.
- I did not run the full `pnpm test:run` or root `pnpm build`. Required
CI runs them.

**Reduced motion.** I ran 6 motion-sensitive Storybook specs in system
Chrome with a temporary config, once with `reduce` and once with
`no-preference`. Both runs had 45 passed and the same 5 failed. So
reduced motion did not change those results. The 5 failures do not
depend on motion. At least one comes from the browser, because system
Chrome prints `oklch(55.6% 0 0)` where the test expects `oklch(0.556 0
0)`. I did not run the other Storybook specs. The `storybook-visual`
label runs that workflow.

## Risks

- **A new lockfile to maintain.** A change to a provider `package.json`
now fails the typecheck job until someone refreshes
`scripts/typecheck-sandbox-providers/pnpm-lock.yaml`. The failure
message gives the command.
- **The job has a new install.** The step downloads provider SDKs, so it
depends on the npm registry. It uses `--frozen-lockfile` and
`--ignore-scripts`.
- **The local install replaces provider `node_modules`.** A contributor
with a standalone provider install must reinstall that provider after
running the script.
- **Reduced motion can change Storybook results.** See "Behavior change
for review". That workflow is not part of required CI.
- **Kubernetes plugin validation changed.** A release of this plugin
ships zod 4. Error wording changes, and very large integers are now
rejected. See "Production change for review".
- **Master moves fast.** Each merge of master brought new test code with
type errors, 27 so far. After this merges, the check reports those
errors in the pull request that adds them.

## Model Used

- Provider: Anthropic Claude, through Claude Code
- Model: Claude Opus 5.5 (`claude-opus-5-5`)
- Capabilities: tool use, code execution, and subagents for the type
fixes in separate files

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived
details. The app generated the branch name.
- [ ] I have run tests locally and they pass. See the failures explained
in Verification.
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 07:33:59 -08:00
Nicky LeachandClaude Opus 5.5 18ff71bca3 fix(ci): load the lockfile policy from the current base-branch tip (#19390)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Only trusted environments may change `pnpm-lock.yaml`. #19356 moved
that rule into `.github/scripts/lockfile-trust.mjs`.
> - The `pr-trusted.yml` policy job loads that script from trusted code,
so a pull request cannot change the check that judges it.
> - The job reads the trusted commit from `pull_request.base.sha`.
GitHub does not move that value forward on every push.
> - #17364 showed the result. Its recorded base was still the `master`
commit from before #19356, so the job used the earlier rule.
> - The earlier rule exempts any branch named `chore/refresh-lockfile`,
including one from a fork.
> - This pull request loads the policy from the first parent of GitHub's
merge commit, which is the current base-branch tip.
> - The benefit is that every pull request gets the current lockfile
policy, including pull requests opened before #19356.

## Linked Issues or Issue Description

Refs #19356, #17364.

**What happened?**
On #17364 at `482d76686e`, the "Block untrusted lockfile edits" step
printed nothing and passed. `pull_request.base.sha` was `da3084a05d`,
the `master` commit from before #19356. That commit has no
`lockfile-trust.mjs`, so the step used the earlier rule. The merge
commit that CI tested already contained #19356.

**Expected behavior**
The policy job runs the current `lockfile-trust.mjs` from `master` for
every pull request.

**Steps to reproduce**
1. Open a pull request before #19356 merged, or keep a pull request
whose recorded base predates it.
2. Push a commit. GitHub keeps the old `pull_request.base.sha`.
3. The policy job uses the earlier rule. A fork branch named
`chore/refresh-lockfile` can still change `pnpm-lock.yaml`.

**Also found**
In a local test, the script exited 0 without any check when it ran from
a symlinked temporary directory. Its entry-point check compared a
symlinked path with a real path. GitHub-hosted Linux runners do not use
a symlinked `RUNNER_TEMP`, but the check must fail closed.

## What Changed

- `pr-trusted.yml` "Block untrusted lockfile edits":
- Load `lockfile-trust.mjs` from `$GITHUB_SHA^1`, the first parent of
GitHub's merge commit. Do this only when `$GITHUB_SHA^2` is the PR head.
Otherwise, keep `pull_request.base.sha`.
  - Give the script the same trusted base for its diff.
  - Fail the step if the script writes no `writer=` output.
- `lockfile-trust.mjs`: compare real paths in the entry-point check.
- `lockfile-trust.test.mjs`: run the real step script against a git
history with a stale recorded base and a merge commit. Check that it
blocks a contributor lockfile change on a `chore/refresh-lockfile`
branch, allows a docs-only change, and falls back to the recorded base
when no merge commit is present.
- `doc/RELEASE-AUTOMATION-SETUP.md`: state where the job loads the
policy from.

## Verification

- `node --test '.github/scripts/tests/*.test.mjs'`: 709 passed.
- The new stale-base test fails when the workflow change is removed.
- The policy-step tests run with `RUNNER_TEMP` behind a symlink. They
fail when the realpath fix is removed.
- A test loads a policy that exits 0 without writing `writer=`. It fails
when the fail-closed check is removed.
- This pull request's own CI uses `pr-trusted.yml` from `master`. The
new step takes effect after merge.

## Risks

- If the checkout is not GitHub's merge commit, the step falls back to
`pull_request.base.sha`, which is the current behavior. The
`pull_request` event always checks out the merge commit by default.
- The fail-closed check can fail a pull request if the script stops
writing `writer=`. The script writes it on every path that returns.
- Low risk otherwise. The change touches one workflow step, one script
line, tests, and docs.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 07:25:41 -08:00
Nicky LeachandClaude Opus 5.5 3a8f95d928 fix(ci): let verified Dependabot lockfiles merge through the reporting-lock gate (#19356)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Only trusted environments may change `pnpm-lock.yaml`: the lockfile
refresh workflow and Dependabot. Contributors never commit it.
> - Paid evaluation jobs also install a lock only when the
reporting-lock proof (`tests/runner-e2e/report-lock.json` and `.patch`)
admits it.
> - #19283 added a CI test that checks the committed lock against that
proof. Only the refresh workflow writes the proof, so every Dependabot
lockfile pull request now fails CI. Examples are #17364, #17172, and
#17911.
> - A merged Dependabot lockfile also breaks the refresh workflow.
`prepare-report-lock.mjs --refresh` refuses to renew the proof for a
lock that the proof does not already admit.
> - This pull request lets a verified Dependabot lockfile pass, and it
keeps both lockfile rules strict. It also closes a gap in the
contributor rule.
> - The benefit is that Dependabot security updates can merge again, and
only trusted environments write lock content.

## Linked Issues or Issue Description

Refs #19283, #17364, #17172, #17911, #19303.

**What happened?**
`server/scripts/reporting-lock-policy.test.mjs` fails on each Dependabot
pull request that changes `pnpm-lock.yaml`. The error is "Reporting lock
changed; review the dependency correction before install". A Dependabot
pull request cannot change the proof files, so it can never pass. After
a merge, the lockfile refresh also stops at "Committed report lock is
not reviewed; refusing to refresh its proof".

**Expected behavior**
- Contributors cannot change `pnpm-lock.yaml` or the reporting-lock
proof.
- Dependabot can change `pnpm-lock.yaml`.
- The lockfile refresh workflow can change `pnpm-lock.yaml` and the
proof.
- After a Dependabot lockfile merges, the refresh workflow admits it and
renews the proof.

**Steps to reproduce**
1. Open a Dependabot pull request that changes `pnpm-lock.yaml`, such as
#17364.
2. Let CI run. The `server (9/12)` shard fails on the reporting-lock
policy test.

**Security gap found during this work**
The old rule skipped the lockfile check when `github.head_ref` was
`chore/refresh-lockfile`. A contributor could open a pull request from a
fork branch with that name and commit a lockfile change. The old rule
also exempted any pull request that Dependabot opened, including commits
that other people pushed to its branch.

## What Changed

- Add `.github/scripts/lockfile-trust.mjs`. It decides which writers may
change lock content:
- The refresh workflow: the head is `chore/refresh-lockfile` in this
repository, and `github-actions[bot]` opened the pull request.
- Dependabot: Dependabot opened the pull request from this repository,
and the head `pnpm-lock.yaml` matches a Dependabot commit. That commit
must have `dependabot[bot]` as its author, GitHub (`web-flow`) as its
committer, and a verified signature. The script reads these fields from
the GitHub API.
- The reporting-lock proof files count as lock content. Only the refresh
workflow may change them.
- `pr-trusted.yml` policy job: replace the inline `head_ref` and author
exemption with `lockfile-trust.mjs pr`. The job runs the script copy
from the base commit, so a pull request cannot change the check that
judges it. If the base predates the script, the job uses the earlier
rule.
- `refresh-lockfile.yml`: add `lockfile-trust.mjs merged`. It checks
that the last lockfile change on `master` is a GitHub-signed merge of a
Dependabot pull request, and that the lockfile at the pull request head
matches a verified Dependabot commit. If both checks pass, it gives the
lockfile SHA-256 to `prepare-report-lock.mjs --refresh
--trusted-lock-sha256`.
- `refresh-lockfile.yml` PR step: check `pnpm-lock.yaml` and both proof
files before it exits as unchanged. A refresh that renews only the proof
now opens a pull request.
- `prepare-report-lock.mjs` and its `.d.mts` declaration:
`refreshReportLock` accepts a trusted lock hash. If the trusted lock
already resolves, the proof admits it exactly with an empty patch. If
not, the proof records the resolution as the reviewed correction, as
before.
- `reporting-lock-policy.test.mjs`: skip on Dependabot pull requests.
The policy job verifies those lockfiles. The general test job gets
`PAPERCLIP_CI_PR_AUTHOR` for this purpose. This keeps the test, so it
still merges cleanly with #19303.
- `check-pr-lockfile.mjs` (advisory gate): allow Dependabot lockfile
changes, and flag changes to the proof files.
- `CODEOWNERS`: add the two proof files.
- `doc/RELEASE-AUTOMATION-SETUP.md`: describe the writers, the
`@dependabot recreate` path, and the proof renewal after merge.

## Verification

- `node --test '.github/scripts/tests/*.test.mjs'`: 707 passed. This
includes 17 new tests in `lockfile-trust.test.mjs` and 3 new tests in
`check-pr-lockfile.test.mjs`. The new tests run the `pr` command against
real git histories with a stub `gh`.
- `tests/runner-e2e/prepare-report-lock.test.ts`: 10 passed, including 3
new trusted-lock refresh tests. I ran it with a minimal Vitest config,
because the runner-e2e config builds a provider bundle first.
- `server/scripts/reporting-lock-policy.test.mjs`: passes normally, and
skips with `PAPERCLIP_CI_PR_AUTHOR=dependabot[bot]`.
- Live check of the `pr` mode against #17364 with the real GitHub API:
- With the real author, it reports "pnpm-lock.yaml matches verified
Dependabot commit 83c328c15c…" and `writer=dependabot`.
- With a contributor as the author and `chore/refresh-lockfile` as the
head ref, it fails.
- Live check of the `merged` mode at the real merge commits of #16018,
#16020, and #15361. All three report a trusted lock hash. For #16018 and
#15361, `master` changed after Dependabot's commit, so GitHub's merge
produced different lockfile bytes. An exact-bytes rule would refuse
them. That is why the rule trusts GitHub-signed merges of a verified
Dependabot head.
- Greptile reviewed `513ca2def8` at 4/5 with one P1: a proof-only
refresh was never committed, because the PR step checked only
`pnpm-lock.yaml`. `edbeb1ea0d` fixes it. A new test in
`lockfile-refresh-merge.test.mjs` runs the step's change check on a
proof-only change. The test fails without the fix.
- CI `Typecheck + Release Registry` failed at `edbeb1ea0d`:
`prepare-report-lock.d.mts` did not declare the new options argument
(TS2554 twice in `prepare-report-lock.test.ts`). `9139aaad85` updates
the declaration. A local `tsc -p tests/runner-e2e/tsconfig.json` shows
the 2 errors with the old declaration and none with the new one.
- CI at `9139aaad85` failed in six general test shards. None of them run
code that this pull request changes, and PR CI loads `pr-trusted.yml`
from `master`. The failures:
- `heartbeat-plugin-environment.test.ts`: "expected 'running' to be
'succeeded'"
  - `status-card-evaluation-work.test.ts`: a deadline comparison
  - `isolated-repository-rehome.test.ts`: timed out after 15 seconds
- `chat-channels.integration.test.ts`: GitHub webhook recovery
scheduling
- `company-import-export-e2e.test.ts`: child server start timeout, which
also fails on unrelated branches
- `workspaces-b`: every test passed, but an `EPIPE` error came after
`acpx-engine/execute.test.ts`
  I started a rerun of the failed jobs.
- Superagent found a P1 at `9139aaad85`: a verified signature proves who
committed, not who authored. A collaborator could sign a commit with
their own key and set Dependabot as the author. `4fe180d6c1` also
requires GitHub (`web-flow`) as the committer. New tests cover the
spoofed-author case. #17364, #16018, #16020 and #15361 still verify
against the GitHub API.
- `git merge-file` merges this test change with #19303 without
conflicts.
- Not run: the full repository test suite and typecheck. These changes
touch only CI scripts, workflows, one helper, and tests. GitHub CI
covers full regression.

## Risks

- `pr-trusted.yml` loads from `master`, so the new policy takes effect
only after merge. This pull request's own CI uses the earlier rule.
- A Dependabot pull request that is behind `master` now fails if someone
merges `master` into it and the lockfile changes. Use `@dependabot
recreate`. The policy job error says this.
- Paid evaluation jobs refuse a merged Dependabot lockfile until the
refresh pull request merges. The refresh merges without review when it
adds no new package versions. If the release-age gate holds it, the wait
lasts until a maintainer merges it or the daily refresh runs. During
that time, the reporting-lock policy test also fails on other pull
requests, which is the current behavior for any unrenewed lock.
- The rule trusts GitHub as the signer of Dependabot commits and of
merge commits (`web-flow`). Superagent uses the same trust root.
- People with write access can still push to `chore/refresh-lockfile`.
This was already true. The bot merges only its own head commit
(`--match-head-commit`), and code-owner review covers anything else.
- A future manual proof renewal, such as #19283, now needs the refresh
workflow or an admin merge. A contributor pull request that changes the
proof files fails.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused tests; see
Verification)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:42:26 -08:00
DottaandPaperclip cd3832d235 ci(evals): route trusted catalog authorization to AWS (#19301)
## Thinking Path

> - Paperclip helps people manage AI agents at work.
> - The chat catalog workflow checks each manual dispatch before
protected work.
> - Its credential-free authorize job waits for GitHub-hosted capacity.
> - A recent run waited about 16 minutes before that short gate could
start.
> - The existing public-CI runner group already allows this catalog
workflow on master.
> - This change places only the approved first manual master attempt on
that fleet.
> - Every other context retains hosted placement and the actual dispatch
guard.

## Linked Issues or Issue Description

Refs: #16042, #18067. The latter changes preparation after authorization
in a different workflow. This change covers only the chat catalog
authorize job.

**What happened?**

The authorize job requests ubuntu-latest even for the approved
main-branch operator. Hosted runner queue time delays all later work.
The current run remains untouched.

**Expected behavior**

Use existing AWS public-CI capacity only for the exact approved first
manual master context. Retain the hosted route for all other contexts.
The existing guard must still verify the actual API identity and both
actors after checkout.

**Steps to reproduce**

1. Dispatch Chat catalog TestOps from master as the approved operator.
2. Inspect authorize placement before a runner starts.
3. Compare the branch, actor, attempt and workflow rejection controls.

## What Changed

- Add repository name and ID, exact workflow ref, master ref, event,
actor ID, attempt and triggering-actor placement fences.
- Reuse the existing placement evaluator for 15 allocation cases.
- Keep the actual dispatch guard and exact-SHA checkout. Keep select and
all reporting jobs unchanged.

## Verification

- Local focused controls: 16 passed. The parent workflow failed all 15
new placement cases because it had no conditional route.
- Existing full routing and workflow suites: 192 passed on the final
patch.
- A temporary select-to-AWS mutation failed the bounded select control;
restored placement passed.
- actionlint for chat-catalog-testops.yml and git diff --check: passed.
- The first full local run hit sandbox EPERM on existing loopback HTTP
controls. That result is retained. The same unchanged suites were then
run with listener permission.
- No application build, SDK build, model call, eval dispatch, workflow
retry or fleet change was performed.
- Latest-head checks: all 55 succeeded. Required `ci / verify` and `ci /
e2e` passed; genuine Greptile review is 5/5 on this exact head with no
unresolved threads.

## Risks

This reuses the existing trusted-main fleet boundary. It is not a
privilege-free worker. The current instance profile has IMDS enabled
with required tokens and includes SSM GetParameter/GetParameters
capability. No IAM, runner-group or network policy changes are included.
The group already permits this exact catalog workflow on master.

Placement checks run before allocation. The actual API and both-actor
guard still runs after checkout. An unauthorized context selects hosted
capacity and must pass the unchanged guard before protected work. Select
still waits for hosted capacity and keeps its protected token boundary.
AWS capacity can also queue. This change does not promise a fixed
latency gain or move a queued job.

## Model Used

OpenAI gpt-6.1-sol with reasoning, code editing and tool use. The
runtime does not report its context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-05 12:32:22 -06:00
DottaandClaude Opus 5.5 b1d00eeca0 fix(evals): derive the controlled Runner eval policy from committed defaults (#19253)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Runner protocol qualification runs real providers against 41 cases
and reports them in TestOps.
> - Controlled mode (#17889) adds a spend cap, private evidence storage,
and cost and trace accounting.
> - Operators must hand-maintain one large JSON variable for each
campaign size, price review, and lockfile change.
> - A one-cell policy blocked the first 41-case run, and its prices
expired after 24 hours.
> - This pull request derives that policy from committed defaults and
the selected cases.
> - The benefit is that a full run needs no policy edit, and all three
safety properties stay.

## Linked Issues or Issue Description

Refs #17889, #19011, #19070.

**What existing behavior does this improve?**

Controlled Runner protocol campaigns read
`RUNNER_PROTOCOL_EVAL_CONTROLLER_POLICY_JSON`. That variable holds the
cell limit, a price table with a review expiry, a pinned lockfile
digest, and self-hashed bound proofs.

**Current behavior**

A campaign with more cells than `maxCells` fails before provider work.
Expired prices also fail before provider work. Any change to the
lockfile needs a new pin. A one-cell policy made all 41 TestOps cases
fail as infrastructure, so an operator had to edit the variable by hand
before the full run.

**Proposed behavior**

The workflow derives the policy at preparation time. The cell count
comes from the selected catalog. Prices come from a committed table with
a version and a source URL. The controller installs the committed frozen
lockfile. One optional variable, `RUNNER_PROTOCOL_EVAL_MAX_USD`, can
override the committed $400 spend cap.

**Reason and benefit**

Each piece that was hand-maintained duplicated a check that already
exists. The manifest digest binds every allocation. The frozen install
binds dependencies to the reviewed commit. Under model-request
readiness, the self-hashed bounds never changed admission or accounting.

**Breaking changes**

The old variable is no longer read. It can stay set or be deleted.
Retained manifests that include the old proofs still replay.

## What Changed

- Add
`packages/paperclip-runner/scripts/controlled-evals/hosted-defaults.json`.
It holds the spend cap, duration, route, Claude Sonnet 5 prices,
allocation, overhead reservation and trace limits.
- Add `hostedPolicy()` in `hosted.mjs`. It derives `maxCells` from the
selection. It gives each campaign a validity window from preparation to
one hour after its deadline. It applies the optional spend cap override.
- Remove `capacityBound` and the policy self-hash from hosted
preparation. `manifest.mjs` needs model `provisionalBound` only when
readiness is not `model_requests`.
- Workflow: stop reading the policy variable. The controller build
installs the committed `pnpm-lock.yaml` with `--frozen-lockfile` and
fails on any lockfile drift. Preparation checks that the runtime receipt
lockfile digest equals the committed lockfile. The catalog limit is now
1 to 512 selected cells.
- Keep everything else the same: the spend-cap check before provider
work, private S3 retention of the manifest and derived
`hosted-policy.json`, receipts, tracing, first-attempt and zero-retry
gates, and paid-environment approvals.
- Update `hosted.test.mjs`, `portable.test.mjs`,
`runner-protocol-eval-workflow-security.test.mjs`, and
`docs/runner-protocol-live-evals.md`.

## Verification

- `node --test scripts/controlled-evals/hosted.test.mjs`: 8 pass. The
new tests show that `maxCells` equals the selection and that the
validity window covers the deadline with no committed expiry. They also
show that the spend cap override applies. With the committed defaults,
41 cells validate with $339 reserved, and 49 cells fail on the $400
budget.
- Negative controls still fail: drift in selection, source or attempt; a
bad lock digest; missing tracing; a duration over 4 hours; a malformed
or too-low spend cap; and zero cells.
- `node --test scripts/runner-protocol-eval-workflow-security.test.mjs`:
20 pass. Controlled admission now passes whether the old variable is
absent, still set, or malformed. A real git checkout with a modified
`pnpm-lock.yaml` fails the controller guard before install.
- `node --test scripts/controlled-evals/controller.test.mjs
scripts/controlled-evals/portable.test.mjs`: all pass.
- `actionlint .github/workflows/runner-protocol-live-evals.yml` and `git
diff --check` pass.
- `tests/runner-e2e/workflow-security.test.ts` has 5 failures that this
change does not cause. They concern `pr-trusted.yml`,
`chat-catalog-testops-case.yml` and `runner-full-stack-e2e.yml`. This PR
does not touch those files.
- I have not run a live controlled campaign on this branch. Controlled
mode runs only from the default branch. The current 41-case run uses
master and the existing variable.

## Risks

- The committed price table has no expiry. A provider price change needs
a reviewed edit to the table. The spend cap still limits the maximum
spend.
- If the committed lockfile does not match the package manifests, the
controller build fails before provider work. CI already requires a
frozen lockfile.
- The spend cap is now $400 by default. It was $339 in the last manual
policy. The cap is a reservation limit, not expected spend. The expected
spend for 41 cases is about $5–20.
- Only ACPX Claude Sonnet 5 has a committed allocation, as in the old
variable. A new controlled model needs a committed price entry and
allocation.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool
use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 10:08:10 -06:00
DottaandPaperclip 85e4d12856 fix(evals): report controlled protocol cases from safe records (#19070)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Runner protocol qualification records real provider behavior in
TestOps.
> - Controlled attempts keep raw records and accounting in private
storage.
> - A live case passed, but a later public-render command rejected its
controlled context.
> - The missing public report caused the final result to be stored as
broken.
> - This pull request creates the closed TestOps result from existing
sanitized records inside the private report job.
> - The benefit is correct result reporting while the privacy guard
stays effective.

## Linked Issues or Issue Description

Refs #18985 and #19011. The related-PR search found no separate repair
for this failure.

**What happened?**

[The live
canary](https://github.com/paperclipai/paperclip/actions/runs/41916513378)
passed all four provider checks. Its controller verified cleanup and
closed its request accounting. The private canonical report recorded one
passed case. The next command used `--public-viewer` while the
controlled manifest remained set. The eval program rejected it with
`controlled campaign evidence must remain private`. The closed exporter
then had no public payload and correctly stored a broken result.

**Expected behavior**

Report the original case grade from sanitized structured records. Keep
raw controlled records private. Keep the original checks, source
bindings, and missing-evidence rules.

**Steps to reproduce**

Run one controlled protocol case with TestOps reporting enabled. Let the
private canonical report finish. The following public-render step fails
before the closed result can read the grade.

**Paperclip version or commit**

Observed at `4740d251b7032e714d33039ba66a56948238c38f`. This repair is
based on `c6cb7e2a2d42743c0c3c4ead2cbad4a299f9a2e7`.

**Deployment mode**

GitHub-hosted Linux, controlled ACPX Claude protocol qualification,
first workflow attempt, zero automatic retries.

## What Changed

- Add a narrow `export-runs` path to the existing TestOps helper. Bind
the sanitized score and artifact to the exact attempt, case, model,
provider, driver, and public-projection schema. Reuse the existing
strict grade and check validation.
- Keep sanitization inside the private report job. Use structured
records for controlled TestOps reporting. Keep the public viewer for
legacy publication.
- Add record identity, privacy, grade, command, and workflow routing
controls. Wire the existing helper and workflow-security files into the
required Node registry. Document the reporting path.

## Verification

- 43 focused campaign/sanitizer, TestOps and workflow security tests
passed. The command regression writes a raw attempt, calls the
production sanitizer, and runs `export-runs` on its output. It also
repeats that chain after a failed-check mutation. Both new-control files
now run through required `test:typescript:prep`, not only the optional
publisher script.
- The actual required prep ran locally with provider credentials
excluded. The eval-kernel and Rust builds passed. Its Node registry
executed the new producer-to-export and security controls by name: 403
passed, 2 failed, 11 existing skips. The two original package-contract
failures cannot resolve this isolated clone's Codex ACP and Claude ACP
package links. This is not a green local full-prep claim. The first
attempt also stopped at a missing Node-types link, before an ignored
clone-local link to the locked cached version restored that
prerequisite.
- 72 release workflow tests passed with the existing locked pnpm 9.15.4
wrapper and local loopback permission. The first sandbox run failed
because loopback fixtures could not listen; the default local tool path
also failed two install fixtures. No test assertions were changed to
resolve these failures.
- The actual repaired workflow shell ran locally against the retained
canary campaign and sanitized records. It emitted one closed passed
result with all four original checks. No provider request or upload
occurred.
- Missing or failed checks still reject a pass. Private-only markers
stay absent from the closed output. Wrong model, provider, driver,
attempt, case, and schemas reject.
- The unmodified, pinned eval program still rejects a controlled
public-viewer command.
- Actionlint and `git diff --check` passed. [Full final-head
CI](https://github.com/paperclipai/paperclip/actions/runs/41927878886)
passed on its first attempt at `0d0c090b1b`. Required verify/e2e gates,
all four Runner lanes, typecheck, build, backend and browser shards
passed. Package suite: 4951 passed, 10 existing skips across 297 passed
files and one existing file skip.
- The required static log contains passing lines for both the production
sanitizer-to-export command regression and the closed-projection
security control. The prior optional-script registration did not run
those tests in required CI; the one script-entry fix now enforces them.
Provider/runtime and workflow bytes are unchanged by the
verification-only follow-ups.
- Greptile: fresh 5/5 on `0d0c090b1b`, with no actionable new issues.
The separate Paperclip Review requested a producer-to-consumer
regression. That regression and its required-CI wiring are complete. The
addressed thread is resolved; no open review threads remain.
- No fresh deployed reporting pass is claimed. The failed workflow and
its stored broken result remain unchanged.

## Risks

The exporter reads structured records inside the private job. It copies
only the closed result fields and requires the existing sanitized marker
and identity checks. The final reporting job still receives only those
files and verifies the saved result bindings. The Python privacy guard,
provider path, case assertions, budgets, deadlines, and retries are
unchanged.

## Model Used

OpenAI Codex, exact model `gpt-6.1-sol`, max reasoning effort. The
inherited runtime reports a 475,000-token context window. Capabilities
used: reasoning, code changes, and repository, shell, and GitHub tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 22:03:59 -06:00
DottaandClaude Opus 5.5 ac57b183f0 fix(evals): run campaign cells when TestOps selection is skipped (#19053)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner Full-Stack E2E workflow runs the paid Product E2E cells,
including the Product regression TestOps plan
> - #18983 added an optional `testops_selection` job and made `catalog`
depend on it
> - That job is skipped in every ordinary manual or scheduled campaign
> - GitHub's implicit `success()` also fails when an ancestor job was
skipped, so every job below `catalog` was skipped
> - Since #18983 merged, ordinary campaigns run no cells: the merge
reports every selected cell as "No result artifact was uploaded"
> - This pull request gives each job below `catalog` an explicit status
condition
> - The benefit is that ordinary, nightly and branch-targeted campaigns
run their cells again, and TestOps campaigns are unchanged

## Linked Issues or Issue Description

Refs #18983, #19000

**What happened?**
Run https://github.com/paperclipai/paperclip/actions/runs/41914430034
selected 20 Product regression cells with the `id` input. `catalog`
passed. `daytona_image`, `build_runner_artifacts`,
`build_remote_provider_pack` and every matrix job were skipped. The
merge failed with 20 "No result artifact was uploaded" rows.

**Expected behavior**
An ordinary manual campaign runs its selected cells when
`testops_selection` is skipped, as it did before #18983.

**Steps to reproduce**
Dispatch `runner-full-stack-e2e.yml` on `master` without `testops_case`
or `ALLURE_JOB_RUN_ID`. The scheduled daily campaign has the same path.

## What Changed

- `daytona_image`, `build_runner_artifacts`,
`build_remote_provider_pack`, `test`, `test_recurring` and
`test_manual_private` now start with `!cancelled()`.
- Each of those jobs now requires `needs.<job>.result == 'success'` for
each of its real dependencies. A failed or skipped build still skips the
cells, as before. The existing schedule, cadence and evidence-route
conditions are unchanged.
- A new regression test parses the workflow. It requires every job below
`testops_selection` to use a status function and to require each real
dependency result.

## Verification

- `node --test tests/runner-e2e/product-regression-testops.test.mjs`:
15/15 pass.
- Negative control: the new test fails against the current `master`
workflow.
- `daytona-image.test.ts`, `workflow-target.test.ts` and
`workflow-timeout.test.ts` pass. `workflow-security.test.ts` has 5
failures. The same 5 fail on unchanged `master` (they concern
`pr-trusted.yml` and `chat-catalog-testops-case.yml`).
- A workflow change takes effect only from `master`, because paid
campaigns run only from the default branch. After merge, dispatch an
ordinary `id=` campaign and confirm the cells run.

## Risks

- Low risk. The conditions keep the previous failure propagation: a job
runs only when each direct dependency succeeded and the run is not
cancelled.
- `catalog` success already implies `authorize` and `target_lock`
success, so those are not repeated.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 17:41:18 -06:00
c19c4348c5 fix(server): record inspector provenance for dynamic import callback errors (#17844)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/49/changes/paperclip/49
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server sends errors to Sentry when an operator sets a backend
DSN. An unhandled rejection is captured in `strict` mode, so the server
process exits after the capture.
> - A hosted server reported `TypeError
[ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING]: A dynamic import callback was
not specified.` two times, eight days apart, on two unrelated releases.
Each event stopped the process.
> - The stack has no first-party frame. An anonymous `setTimeout`
callback at `<anonymous>:1:N` calls `import()`. The column changed
between the two events, so the source text changed.
> - In local reproduction, only two paths make that frame:
`vm.compileFunction` without a filename, and an expression that a client
evaluates through the Node inspector. No shipped server code or
production dependency uses either path. The current events give no data
about the source.
> - This pull request adds bounded provenance to that one error: the
inspector state when the event is reported (never its URL), allow-listed
Node flag names, the process ids, the main-thread flag and the uptime.
> - The benefit is that the next event gives evidence about the source
of the anonymous code. A `closed` state rules out an expression from a
still-open inspector session. An `open` state needs other evidence
before a debugger is blamed.

## Linked Issues or Issue Description

No public issue exists. This section follows the bug report template.

**What happened?**

A hosted server process stopped two times with an unhandled rejection:
`TypeError [ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING]: A dynamic import
callback was not specified.` The stack is
`importModuleDynamicallyCallback (node:internal/modules/esm/utils:280)`
← `Timeout._onTimeout (<anonymous>:1:20)` (first event) or
`(<anonymous>:1:41)` (second event) ← `listOnTimeout`. The events came 3
minutes and 17 minutes after process start, on two releases that changed
only skill documents and UI files. The Sentry event has no data that
shows where the anonymous code came from.

**Expected behavior**

The error report gives enough data to investigate the source of the
anonymous code: whether an inspector session was open in the process,
and which inspector-related Node flags were set.

**Steps to reproduce**

1. `node -e "require('node:vm').compileFunction(\"setTimeout(() =>
import('node:fs'), 1)\", [])()"` gives the same error and the same
`Timeout._onTimeout (<anonymous>:1:N)` frame.
2. An expression such as `setTimeout(() => import('node:fs'), 1)`,
evaluated through the inspector (`Runtime.evaluate`), gives the same
error and frame.
3. `vm.Script` (`evalmachine.<anonymous>`), `eval` or `new Function` in
a `vm` context (`eval at …`), `new Function`, indirect `eval`, `node
-e`, `node -p` and stdin give a different frame or import successfully.

**Paperclip version or commit**

`master` at `0cb3fc7db591ba23e21fbbc70042bc44ec7d6f20`. The hosted
events came from builds `f96a1e64` and `3e7c3ee9`, on Node v24.22.0 and
v24.22.1.

**Relevant subsystem**

Server Sentry error monitoring (`server/src/sentry.ts`).

## What Changed

- New `server/src/sentry-dynamic-import-provenance.ts`:
`addDynamicImportProvenance(event, hint)` acts only when
`hint.originalException.code` is
`ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING`. It adds:
- the tag `dynamic_import_inspector` with the value `open`, `closed` or
`unavailable`
- the context `dynamic_import_provenance` with `inspector`, `nodeFlags`,
`mainThread`, `pid`, `ppid` and `uptimeSeconds`
- `nodeFlags` holds only the names of `--inspect`, `--inspect-brk`,
`--inspect-wait`, `--inspect-port`, `--debug-port`, `--disable-sigusr1`
and `--experimental-vm-modules` from `execArgv` and `NODE_OPTIONS`. Flag
values and all other flags are dropped.
- The hook never sends the inspector URL, because the URL contains a
session secret. It loads `node:inspector` lazily, so a Node build
without the inspector still starts and reports `unavailable`. The hook
never throws. Every other event is unchanged.
- The module comment and the doc say that the state is evidence at
report time, not proof of origin.
- `server/src/sentry.ts`: `beforeSend` is now `enrichServerEvent`. It
runs the existing run-link enrichment, then the new hook.
- `doc/observability.md`: the default capture set names the new tag and
context, and says what is never sent.
- `.github/workflows/sentry-contract.yml`: the Real Sentry SDK isolation
job also runs the new test file.
- Tests in
`server/src/__tests__/sentry-dynamic-import-provenance.test.ts`:
  - the real Node error from `vm.compileFunction`
- a real open inspector session: the URL and its secret are absent from
the event. The test closes only the session it opened, and it skips when
a developer already has an inspector open.
  - flag filtering with values and other flags present in `NODE_OPTIONS`
  - other codes and other hint shapes stay unchanged
  - a failed state read keeps the original event
- the real `@sentry/node@10.71.0` SDK sends the original rejection to
the server `beforeSend`, and enriches only the matching event. With
`PAPERCLIP_REQUIRE_SENTRY_TEST_SDK=1`, a missing SDK fails the file.

## Verification

- With the audited SDK on `NODE_PATH` and
`PAPERCLIP_REQUIRE_SENTRY_TEST_SDK=1`, as in the contract job:
`run-failure-sentry-real-sdk.test.ts` and the new test file pass (17
tests).
- Without the SDK: the new test file, `sentry.test.ts` and
`sentry-links.test.ts` pass (52 pass, 6 skipped).
- With `PAPERCLIP_REQUIRE_SENTRY_TEST_SDK=1` and no SDK: the new test
file fails at once, as intended.
- Red: with `beforeSend` set back to `addSentryRunLinks`, the real-SDK
test fails (`expected [] to have a length of 1`). Green with the change.
- Hosted CI on this head: typecheck, build, all test partitions and Real
Sentry SDK isolation pass.
- Not run locally: the full Vitest suite and browser suites. A live
hosted check is not possible before deployment.

## Risks

- Low risk. The hook acts only on one error code. All other events keep
their current content and grouping.
- The process still exits after the capture, as before. This change adds
diagnostics. It does not fix the source of the anonymous code.
- `pid`, `ppid` and uptime describe the server process only. They hold
no tenant, user or request data.
- If a later Node version changes the error code, the hook stops acting
and the events are the same as today.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through the Claude
Agent SDK, with tool use (shell, file edit, code execution, GitHub and
Sentry connectors).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 14:07:26 -08:00