Commit Graph
5 Commits
Author SHA1 Message Date
DottaandClaude Fable 5.2 76da2fec4f perf(runner): deliver transport events when they arrive instead of polling every 5 ms (#20862)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server runs each native agent session through a runner
transport. The transport reads the events that the runner commits to the
session's control plane and delivers them to the driver.
> - The transport found those events with a timer. Every live transport
ran a pass every 5 ms, for as long as the session was open, whether or
not an event had arrived.
> - A CPU profile of the staging server shows the cost. The main thread
was at 96% of one core, and this pass used 5.6% of it with about a dozen
live sessions. The cost grows with the number of sessions and never
stops.
> - The same timer also delayed every event. An event waited up to 5 ms
for the next pass before the driver saw it. Agent boot time and time to
first token are measured requirements.
> - This pull request makes the control plane tell the transport when a
reader can see a change, and the transport delivers in the next task.
The fixed timer is gone.
> - The benefit is that an open session with nothing to deliver costs
nothing on the main thread, and an event reaches the driver sooner than
before.

## Linked Issues or Issue Description

No public issue exists. The problem is described here.

**What happened?**

The server's main thread on a hosted instance stayed at 96% of one core,
so every request and every agent turn waited behind it. A 30 second CPU
profile of the live server (Node 24, about a dozen live native agent
sessions) attributes 5.6% of the main thread to `#pumpEventsSafely` and
4.7% self time to `#pumpEvents` in
`packages/paperclip-runner/src/live/runnerd-codex-transport.ts`. Three
places started `setInterval(() => this.#pumpEventsSafely(), 5)`. Each
pass scans the session's committed-event window, so the cost is
`sessions x 200 passes per second x window size`, with nothing to
deliver.

**Expected behavior**

An open session that has no event to deliver does no periodic work. An
event that arrives is delivered at once, not at the next timer tick.

**Steps to reproduce**

1. `cd packages/paperclip-runner`
2. `PAPERCLIP_TRANSPORT_PUMP_BENCHMARK=1 pnpm exec vitest run
src/live/runnerd-transport-pump.benchmark.test.ts`
3. Read the `TRANSPORT_PUMP_BENCHMARK` lines.
`stateReadsPerSessionPerSecond` counts how often an open, idle transport
reads the control plane. `deliveryWaitMs` is the time from the moment a
reader can see a committed event to the moment the transport delivers
it.

**Paperclip version or commit**

Present on master at `c8ae02e5ce`. The profile is from `556b249897`.

**Deployment mode**

Hosted server with native agent sessions. The same code runs in local
development.

## What Changed

- `DurablePrpControlPlane` has a new `onReadableStateChange(listener)`.
It fires when the state that a reader of `store` can see may have
changed. The control plane calls it after every stored-state replacement
or save, after a saved checkpoint releases events that were waiting for
it, and after an authenticated runner moves the recovered cursor.
- The transport subscribes when it creates its control plane. A change
wakes one delivery pass in the next task (`setImmediate`). The pass
never runs inside the control plane's own commit, and several changes in
one task share one pass. This keeps the existing order: the control
plane finishes its commit before a reader reacts.
- The three `setInterval(..., 5)` timers are gone. `#startPump` and
`#stopPump` replace them and keep the same lifecycle: the pump runs from
session start, and it stops on release, controller retirement, failure,
and close.
- The delivery pass now reports what it left behind. Events that the
transport holds back itself (a pending turn start, a pending recovery
binding, a trace write that must be retried) are retried every 5 ms, as
before, and only while they are held. This is the only timer that
remains.
- A pass that stops after it exposes `turn/started` (so the driver can
bind the turn first) now continues in the next task. Before, the events
after it waited for the next 5 ms tick.
- `attachRun` wakes the pump when it moves the transport to the new run
identity.
- New regression test `src/live/runnerd-codex-transport-pump.test.ts`,
new opt-in benchmark
`src/live/runnerd-transport-pump.benchmark.test.ts`, a paced provider
fixture, and a test-only probe that records when each committed event
becomes readable.

How this meets `rules/sleep-aware-background-work.mdc`:

| Item | Answer |
| --- | --- |
| Trigger | The control plane's readable-state change. No timer. |
| Source of truth | The control plane's committed-event window and the
transport's delivery cursor. A notification is a hint to look at them. |
| Pending-work predicate | The pass returns `held` or `next_task`, or a
trace write is waiting. |
| Next deadline or retry | 5 ms, only while the predicate is true.
`next_task` uses one `setImmediate`. |
| Quiet condition | Nothing is held. Then the transport has no timer and
no scheduled task. Release, retirement, failure, and close stop it
completely. |

## Verification

**Reproduction and before/after measurement**

Run from `packages/paperclip-runner`:

```sh
PAPERCLIP_TRANSPORT_PUMP_BENCHMARK=1 pnpm exec vitest run src/live/runnerd-transport-pump.benchmark.test.ts
```

The benchmark opens 12 real transports against `runnerd` and a
credential-free provider fixture. It measures the open sessions while
nothing arrives. Then all 12 sessions stream one turn at the same time
(40 events each, one every 15 ms). "Before" is the unchanged source at
`e2cf9a670f`, which was the base of this branch when the measurements
were taken. "After" is this change on that base. Each cell shows two
runs, and the runs alternate between before and after. The host was a
shared macOS arm64 machine with other work running (load average 8 to
10), so absolute numbers carry that noise.

External checkpoint persistence. This is the mode that the server uses.

| Measure | Before | After |
| --- | --- | --- |
| Idle: control-plane state reads per session per second | 364 / 365 | 0
/ 0 |
| Idle: event-loop utilization, 12 new sessions | 1.5% / 1.5% | 0.0% /
0.0% |
| Idle: event-loop utilization, 49 retained events per session | 2.1% /
2.2% | 0.0% / 0.0% |
| Idle: process CPU, 49 retained events per session | 12.8 / 11.8 ms per
second | 0.05 / 0.04 ms per second |
| Delivery wait, mean (480 events) | 3.48 / 4.22 ms | 1.91 / 1.81 ms |
| Delivery wait, p50 | 3.16 / 3.97 ms | 0.83 / 0.58 ms |
| Delivery wait, p95 | 7.78 / 8.49 ms | 6.36 / 7.47 ms |
| Delivery wait, max | 8.84 / 9.23 ms | 9.07 / 8.86 ms |
| Delivery wait of the turn's terminal event, p50 | 2.37 / 2.58 ms |
0.08 / 0.05 ms |

The idle cost grows with the retained event window, as the profile
predicts. With `PAPERCLIP_TRANSPORT_PUMP_DELTAS=400` (409 retained
events per session, one run each): idle event-loop utilization 3.4%
before and 0.0% after, idle process CPU 21.5 ms per second before and
0.08 after. Delivery wait in that run: mean 5.93 ms before and 3.55 ms
after, p95 16.27 ms before and 8.03 ms after, max 66.94 ms before and
14.04 ms after.

Delivery wait is the time from the moment a reader can see a committed
event to the moment the transport hands it to the driver. It is equal or
better at every point. The p95 and the max did not move much. They
belong to the first events of each turn, which wait for the turn start
response. That wait is a separate 10 ms poll (`waitForRunnerCommand`),
and this pull request does not change it.

File persistence (the local journal, which writes and syncs on the main
thread):

| Measure | Before | After |
| --- | --- | --- |
| Idle: state reads per session per second | 363 / 364 | 0 / 0 |
| Delivery wait, 12 sessions, mean | 10.33 / 13.76 ms | 11.88 / 12.73 ms
|
| Delivery wait, 12 sessions, p50 | 6.04 / 6.87 ms | 6.34 / 6.20 ms |
| Delivery wait, 4 sessions and 400 events each, mean | 4.75 ms | 2.39
ms |
| Delivery wait, 4 sessions and 400 events each, p50 / p95 | 0.07 /
22.99 ms | 0.06 / 11.80 ms |

With 12 sessions the file journal saturates the main thread with its own
synchronous writes, and the wait is the time behind those writes. Three
more before-runs of that case gave means of 9.0, 14.0 and 20.2 ms, so
the two columns are the same within the noise. With 4 sessions the
journal does not saturate and the after column is better.

**Regression test (red, then green)**

```sh
pnpm exec vitest run src/live/runnerd-codex-transport-pump.test.ts
```

- `stops looking for events while an open session has none to deliver`
(file and external persistence). On the unchanged source: `expected 708
to be less than 100` and `expected 732 to be less than 100` state reads
in a 2 second idle window. With the fix: pass.
- `delivers each committed event in the next task, without a timer tick`
(file and external persistence). The test freezes every interval timer,
runs a full turn, and checks that each event is delivered before a task
that the test queues at the moment the event becomes readable. This is
an order check and does not depend on the speed of the host. On the
unchanged source: `the turn's events were delivered without a periodic
poll: expected false to be true`. With the fix: pass.
- Result: 4 of 4 failed on the unchanged source, 4 of 4 passed three
times with the fix, and 4 of 4 passed again after the rebase.

**Other local checks**

- `node ./node_modules/typescript/bin/tsc -p tsconfig.json --noEmit` and
`-p tsconfig.test.json --noEmit` in `packages/paperclip-runner`: pass on
the rebased commit.
- `pnpm exec vitest run src/live/runnerd-codex-transport` (15 files, 455
tests). These suites cover attachment, recovery, warm transition,
suspension, and release and resume on real `runnerd` processes. Before
the rebase, on a quieter host: all passed. On the rebased commit, at a
host load average of 15 to 20: 454 passed and 1 failed. The failure is
the terminal-replay row that is described below.
- `pnpm exec vitest run src/live src/control-plane
src/native-session-runtime.test.ts src/backends
src/drivers/dot/runnerd-dot-driver.test.ts
src/drivers/muse/runnerd-muse-driver.test.ts
src/drivers/codex/codex-protocol-integrity.test.ts --maxWorkers=3` (81
files): 1566 passed, 32 failed, 6 skipped. The failures:
- 30 failures in `runnerd-acpx-permission-ordering.test.ts` and
`runnerd-acpx-permission-forwarding.test.ts`. Both replace the control
plane with a fixture that appended events in place and relied on the 5
ms poll to notice them. The fixture now implements
`onReadableStateChange` and notifies its readers when it emits. Both
files pass (30 of 30).
- 1 failure in `runnerd-codex-transport.test.ts`, `stops a lost
runnerd's provider and escaped tool job from its lineage journal
(journal=true)`: `runner did not durably suspend before checkpoint`. It
passed 3 of 3 when run again alone.
- 1 failure in `runnerd-codex-transport-settlement.test.ts`, the
terminal-replay row of `settles only retained control authority without
starting another provider turn`: `native_cleanup_maintenance_unproven`,
with events still in the runner outbox. It failed again in 2 runs of
that test group and in the rebased run above, each time while the host
load average was 11 to 20. At a load average near 8 it passed 3 of 3
alone and 1 of 1 as a group with the fix. The unchanged master source
(`c8ae02e5ce`) gives the same result: it passed 2 of 2 alone and 1 of 1
as a group at a load average near 8, and the same row failed with the
same error (`native_cleanup_maintenance_unproven`, 27 events still in
the runner outbox) in a group run at a load average of 11 to 15. So this
row is sensitive to host load with or without this change. The failing
step is the retained-session maintenance path. That path builds its own
control plane, has no transport pump, has no listener on the new
notification, and has a fixed 30 second deadline.
- `pnpm verify` returns exit 2 (broader verification required, because
of the new fixture file). CI supplies the broad coverage.
- Not run locally: the server test suites, the repository-wide
typecheck, and the build. CI ran them on `4bce83f50a`: 59 checks passed
and 4 were skipped (Storybook deploy, Storybook visual regression,
Sandbox provider tests, Select Vitest files).

## Risks

- **A change that does not notify.** With no periodic pass, an event is
delivered only when something wakes the pump. Every wake comes from one
of a small, closed set: the five places where the store replaces or
saves its state, the checkpoint publication, the recovered-cursor
update, and the transport's own identity and start points. A new code
path that changes readable state without one of these would leave an
event undelivered until the next change. The regression test runs a full
turn with every interval timer frozen, in both persistence modes, so a
missing wake on the normal path fails the test. The existing transport
suites (attachment, recovery, warm transition, suspension, release and
resume) now run with no timer that could hide a missing wake.
- **Delivery in the next task, not later.** An event now reaches the
driver up to 5 ms earlier. The pass still runs in its own task after the
commit, and the same gates hold a terminal event until the turn start
response is released. The order of events does not change.
- **No protocol, schema, Rust runner, or provider pack change.** The
contract fingerprint inputs are untouched. Saved state from an older
release resumes the same way.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, in Claude Code. Tool
use and local code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 11:08:50 -06:00
Nicky LeachandClaude Opus 5.5 ff8ec41e6a feat(runner): upgrade ACPX to 0.22.0 and move patched behavior into Paperclip (#20770)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Runner and the shared `adapter-utils` engine run ACP
agents through the embedded ACPX library
> - Paperclip pins ACPX 0.13.1 and keeps a large patch
(`patches/acpx@0.13.1.patch`) on it
> - ACPX 0.22.0 adds host hooks (`onAcpMessage`, `assertActive`,
`processLifecycle`, prompt-response `_meta`, `spawnAgent`-friendly
launch) that make much of that patch unnecessary
> - A large patch is expensive to port and hides behavior that belongs
to Paperclip
> - This pull request upgrades to ACPX 0.22.0 and moves most patched
behavior into Runner and `adapter-utils`
> - The benefit is a patch that is 484 lines instead of 1,334, with the
moved behavior covered by Paperclip tests

## Linked Issues or Issue Description

Refs #20675, #14430, #15075, #17942, #9980

**Proposed enhancement**
Upgrade the embedded ACPX library from 0.13.1 to 0.22.0. Keep only the
patch behavior that stock 0.22 cannot express.

**Motivation**
The 0.13.1 patch held behavior that Paperclip owns, for example usage
receipts, Copilot policy, model checks and the agent environment. Each
ACPX upgrade had to port all of it. ACPX 0.22 gives hooks that let
Paperclip do this work itself.

**Current behavior**
Runner and `adapter-utils` depend on 1,334 lines of patched ACPX 0.13.1
code.

**Expected behavior**
Runner and `adapter-utils` use ACPX 0.22.0. The patch keeps only:
extension methods, Cursor subagent routing, event `_meta` and the
message boundary chunk, the protocol guard hook, `spawnAgent`, and
`requestExtension`.

## What Changed

- ACPX 0.13.1 → 0.22.0 in Runner, `adapter-utils`, the server, the
Daytona runner images and the qualified profiles (`acpxVersion`).
`@agentclientprotocol/sdk` moves to 1.9.0.
- New `patches/acpx@0.22.0.patch` (484 lines). The 0.12.0 and 0.13.1
patches are removed.
- Delivery receipts: Runner confirms a permission reply when
`onAcpMessage` sees the exact outbound reply. Question and extension
replies count as delivered when Runner hands them back to ACPX. New
`interaction-delivery.ts`.
- Copilot policy: the existing policy function runs from `onAcpMessage`.
A violation stops the agent, cancels later permission requests and fails
the next prompt through `assertActive`.
- Model selection: stock 0.22 validation, plus a sidecar gate
(`createAcpxEffectiveModelGate`) that refuses a prompt if the agent
reports another model. Grok binds its model when the session opens.
Antigravity adds its concrete model to the model list ACPX reads.
- Typed session failures: Runner and `adapter-utils` read them from the
turn result `_meta` and fail the turn. They advertise the capability
themselves.
- Pi and Cursor usage receipts: a collector (`prompt-receipts.ts`) binds
them to the turn's own `session/prompt` reply.
- Goal capability: Runner reads it from the `initialize` reply. It keeps
it under `paperclip.goalCapability` in the saved agent capabilities
`_meta` and restores it when a saved session loads.
- Agent environment: Runner and `adapter-utils` set the agent
environment in `spawnAgent`. ACP terminals keep their current
environment. `adapter-utils` blocks inheritance of `XAI_API_KEY` and
refuses to start if the server has an `ACPX_AUTH_*` variable.
- `adapter-utils` spawn hooks (`spawnCwd`, stderr, `onAgentSpawn`): a
new `stockAcpRuntimeOptions` maps them to stock `spawnAgent` and
`processLifecycle`.
- Windows: ACPX 0.22 rejects raw command strings on Windows.
`adapter-utils` now passes an argv array there.
- Grok rules: a Runner guard (`grok-session-rules.ts`) writes
`session/new` `_meta.rules`.
- Turn-dispatch journal (#20675): ACPX 0.22 ignores errors from
`onAcpMessage`. The adapter keeps a failed `dispatched` write and throws
it from the turn's `assertActive`. ACPX calls `assertActive` before the
prompt write, so the write still stops. The real-ACPX test now checks
this route. `docs/durable-recovery.md` is updated.
- Plan events: the sidecar reads 0.22 `status` events tagged `plan`.
- CI: the backup-checkpoint lane now installs through the shared
`setup-ci-dependencies` action, like PR CI, so it resolves a stale
`pnpm-lock.yaml` in place. It runs on any root `package.json` change,
and pull requests never commit the lockfile, so its own frozen install
failed on dependency changes.
- Tests that checked removed patch internals are removed or rewritten:
the response-delivery, Cursor-usage and Pi-receipt package contracts,
and the goal-persistence packaging test. New unit tests cover the moved
behavior.

## Verification

Local runs on macOS (darwin-arm64), Node 26, on the rebased head:

- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/drivers/acpx src/cli`: 2,000 passed.
- `tsc -p packages/paperclip-runner/tsconfig.json` and
`tsconfig.test.json`: no errors, except the existing
`@paperclipai/paperclip-eval-kernel` build gap in `workflow-harness.ts`.
- Runner package contracts (`node --test
test/acpx-*-package-contract.test.mjs
test/acpx-sidecar-contract.test.mjs`), `scripts/build-inputs.test.mjs`,
`scripts/contracts/contract-fingerprints.test.mjs`,
`scripts/acpx-patch-packaging.test.mjs`: all pass. `node
scripts/generate-acpx-sidecar-contract.mjs --check` passes.
- `cargo test -p paperclip-runner-core --test acpx_park_preparation
--test acpx_park_readiness --test acpx_warm_attachment --test
native_provider_backend`: 86 passed. `cargo fmt --check` passes.
- `adapter-utils`: `tsc` clean. `vitest run src/acpx-engine
src/mcp-isolation.integration.test.ts`: 457 passed (one Gemini flag test
failed once under load and passes when run alone).
- Adapters: `claude-local` and `codex-local` `acp.quota.test.ts` (23 and
39 passed), `grok-local` (149 passed).
- Server: `scripts/prepare-runner-vendor.test.mjs`,
`native-session-executor.test.ts`, `quarantined-task-retry.test.ts`
(1,416 passed).
- `cargo test -p paperclip-runner-core --lib -- previous_acpx_release
acpx_upgrade unrestorable_saved_sessions`: 3 passed. `node --test
test/acpx-session-carry-forward-package-contract.test.mjs`: passes.
- `patch -p1 --forward` on a clean `acpx-0.22.0.tgz` gives output
identical to the port.
- Under full-suite load, some Runner live and recovery tests time out.
Each one passes when run alone. Full regression is left to CI.

Not run: live provider runs (they need provider credentials), the opt-in
Kimi live peer test, the Gemini filesystem smoke test, and Windows.

## Risks

- Previous-release story: sessions saved by the previous release (ACPX
0.13.1) carry forward. ACPX 0.22 loads the session records 0.13.1 wrote.
Runner moves a saved descriptor that names 0.13.1
(`CARRIED_ACPX_RELEASES`) to 0.22.0, and the sidecar keeps 0.13.1 as the
ACPX value in the session identity, so the saved session key and
identity record still match. The existing launch-profile rebind still
decides which sessions resume: a settled session, or the same run's
interrupted turn through lost-turn recovery. A session saved by any
other ACPX release is still refused as `provider_state_incompatible`.
`rules/acpx-profile-identity.mdc` and `docs/durable-recovery.md` now
describe this.
- Proof: `acpx-session-carry-forward-package-contract.test.mjs` resumes
a session record written by the real ACPX 0.13.1 with its Paperclip
patch. 0.22 loads the same provider session (no `session/new`), runs a
turn and keeps the earlier turn. Rust tests cover a settled 0.13.1
session that the next run attaches to, and a turn interrupted by the
upgrade. A recovery-identity test checks that the session key and
identity record match across the versions.
- `pnpm-lock.yaml` is not committed. CI resolves it for each run, and
the lockfile refresh workflow updates it after merge.
- Behavior change on the server: `adapter-utils` refuses to create an
ACPX runtime if the server environment has an `ACPX_AUTH_*` variable.
Before, ACPX ignored such variables. Stock 0.22 would use them to
authenticate agents with server credentials.
- Delivery confirmation is now Runner-owned. A permission reply is
confirmed by the exact outbound message. Question and extension replies
are confirmed when Runner hands them to ACPX, not when the pipe accepts
them.
- Users can select only models that the agent advertises. Stock 0.22
rejects other model IDs (Claude is exempt).
- ACP terminals keep the sidecar or server environment, as before. This
existing exposure is not changed by this pull request.
- The `adapter-utils` Gemini version probe now runs with the server
environment plus the run variables, not the run variables alone.
- The pack contract digest and profile `acpxVersion` change, so Daytona
needs a new runner image.
- Grok launch flags (`--model`, `--reasoning-effort`) were checked on
1.0.13 but session open needs a login, so Grok keeps the existing
`_meta` path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool
use (shell, file editing, subagents). Extended reasoning was used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 07:28:00 -08:00
Nicky Leach 03498f90ee Add explicit Runner session release and resume support (#19683) 2026-11-06 18:31:23 -08:00
Nicky LeachandClaude Opus 5.5 83c6832ccb fix(build): typecheck package, Playwright, and e2e test files in CI (#18839)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Runner, plugin, and Playwright tests verify how those agents run,
connect, and appear in the board
> - Most package tsconfigs exclude tests, and Vitest strips types, so a
type error in those tests fails nothing
> - The tests drifted to 803 real type errors, and some of those errors
hid assertions that checked nothing
> - paperclipai/paperclip#18739 and paperclipai/paperclip#18738 check
the server test tree and four test projects, but not these files
> - This pull request adds a required check for these test files, fixes
every error, and fails CI if a test file is in no program
> - The benefit is that a type error in any test file outside
`server/src` now fails CI

## Linked Issues or Issue Description

Refs #18739
Refs #18738

**What happened?**

517 test files were in no program that required CI checks. A type error
in those files did not fail CI.

**Expected behavior**

CI fails on a type error in any test file.

**Steps to reproduce**

Add `export const value: number = "text";` to a test file under
`packages/paperclip-runner/src` on master. CI stays green, because
`tsconfig.json` excludes test files and Vitest strips types.

**Measured gap**

Each row is a program with its tests included, on one thread. File and
error counts are for head `7d4b5e907a`. Memory peaks were measured
before the two latest merges of master.

| Group | Test files | Real errors | Peak |
| --- | ---: | ---: | ---: |
| `paperclip-runner` Node source | 290 | 644 | 1.09 GiB |
| `paperclip-runner` surfaces | 9 | 12 | 0.42 GiB |
| `paperclip-runner` devtools and examples | 14 | 0 | 0.68 GiB |
| `sandbox-providers/kubernetes` | 21 | 65 | 0.44 GiB |
| 8 other sandbox providers and `bridge-template` | 32 | 4 | 0.34 GiB |
| `plugins/sdk` | 22 | 15 | 0.41 GiB |
| `plugin-workspace-diff` | 5 | 0 | 0.37 GiB |
| `plugin-orchestration-smoke-example` | 1 | 1 | 0.15 GiB |
| `kv-demo-mcp-server` | 3 | 0 | 0.13 GiB |
| `packages/db` scripts | 1 | 0 | 1.01 GiB |
| 8 Playwright suites under `tests/` | 52 | 25 | 1.72 GiB |
| `tests/runner-acceptance` | 3 | 0 | 0.58 GiB |
| `tests/e2e`, through layer declarations | 63 | 37 | at most 1.59 GiB |
| `ui` `AttentionInteractionResolver.test.tsx` | 1 | 0 | n/a |
| **Total** | **517** | **803** | |

27 of the 803 errors came from master while this pull request was open.
Each merge of master brought new test code that no check had read.

Errors that were config errors, not real errors:

- One TS2688 for each provider, because the root install does not
install provider dependencies.
- 9 in the Runner Node program (TS7016 and TS6059).
- 2 in `tsconfig.browser.json`. TypeScript 7 removed `baseUrl`, so that
config was invalid.
- 213 in the Playwright suites (`@/*` aliases and Vite types).
- About 60 in `tests/e2e` (import extensions and aliases).

`packages/plugins/create-paperclip-plugin` has no gap. Its tsconfig
excludes its test, but `cli/tsconfig.json` reaches that test through an
import. `server/test-baselines` also has no gap, because
`tests/lifecycle-baseline` reads it.

## What Changed

- Added `scripts/typecheck-test-programs.mjs` and its tests. The
`Typecheck + Release Registry` job runs it. It checks each program in
`PROGRAMS` on one thread.
- The script fails if a tracked `*.test.ts(x)` or `*.spec.ts(x)` file
outside `server/src` is in no program. The compiler file list decides
this, because `include` drops `foo.test.tsx` when `foo.test.ts` exists.
- Added a `tsconfig.test.json` to each uncovered package, and
`tests/tsconfig.playwright.json`.
- Added `tests/e2e` to `PROJECTS` in
`scripts/typecheck-test-projects.mjs`, with a `tsconfig.json` and a
`tsconfig.layers.json`.
- Added `scripts/typecheck-sandbox-providers`. It is a small workspace
that installs provider SDKs from one lockfile. I did not add a lockfile
inside each provider, because
`scripts/build-standalone-public-packages.mjs` uses a frozen install
when one exists, and that changes what a release resolves.
- Corrected `packages/paperclip-runner/tsconfig.browser.json` for
TypeScript 7.
- Set `rootDir` to `..` in
`packages/paperclip-runner/tsconfig.test.json`, because one Runner test
imports from `packages/adapter-utils/src`. The program does not emit
files, so this changes no build output.
- Moved the Kubernetes provider from `vitest ^3.2.4` to `^4.1.8`, the
version the other providers use. Vitest 3 requires `tinypool ^1.1.1`,
which has two critical advisories (GHSA-85c8-ppgw-ccpr and
GHSA-5gmw-xhrv-c9v3). The first patched version is 2.1.2. Its tests have
the same result on both versions: 189 passed, and the same 12
`file-sync` tests failed.
- Moved the Kubernetes provider from `zod ^3.24.2` to `^4.6.6`, and gave
its two `z.record` calls a key schema. This is the only production
change, and it stays in this pull request. See "Production change for
review".
- Renamed `AttentionInteractionResolver.test.ts` to
`AttentionInteractionResolver.replace.test.ts`, so TypeScript reads the
`.test.tsx` beside it.
- Fixed all 803 errors, with changes to test files and test support
only. The fix for `contracts/native-execution.test.ts` is the same as
the fix in paperclipai/paperclip#18739, so those two changes merge
cleanly.
- Updated `doc/DEVELOPING.md`.

This pull request does not edit `server/scripts/typecheck-layers.mjs`.
paperclipai/paperclip#18739 and paperclipai/paperclip#18738 have both
merged, and this branch includes them.

### Production change for review

The new lockfile made dependency review flag `zod@3.25.76` as a
high-severity advisory (GHSA-mrhx-c7xc-m3w5). The first patched version
is 4.6.6, so 3.x has no fix. The published Kubernetes plugin uses zod to
validate operator config. The zod 3 source does not compile against zod
4, so a typecheck-only override cannot fix this.

The existing tests do not prove this change, because they also passed
with the invalid one-argument `z.record`. So I parsed 45 inputs with the
same schema on zod 3.25.76 and zod 4.6.7:

| Result | Inputs | Detail |
| --- | ---: | --- |
| Identical | 23 | Includes defaults, valid config, and a `__proto__`
key |
| Same outcome, new error wording | 21 | For example, `Invalid url` is
now `Invalid URL`. `validateConfig` shows this text to operators. |
| Different outcome | 1 | `jobTtlSecondsAfterFinished: 2 ** 60` was
valid and is now rejected. The new limit is `Number.MAX_SAFE_INTEGER`. |

I did not test this against a real cluster.

The decision is to keep this change in this pull request. Since then,
paperclipai/paperclip#17364 has moved the rest of the repository to zod
4.6.x, so the Kubernetes plugin now matches it.

### Behavior change for review

`reducedMotion` under `use` is not a Playwright option. It had no effect
in 11 configs and 2 `test.use` calls, so those suites ran with
`no-preference`. The options are now under `contextOptions`, so reduced
motion now takes effect. `app-shell-design.spec.ts` asserts a real
transition, so it now asks for `no-preference` itself.

The baseline manifest has `snapshotCount: 0`, so this invalidates no
published baseline. A frozen trial reference PNG can differ.

### Findings

These are product questions. I did not change them, and I did not hide
them with casts:

1. **`PrpEvent` and its schema disagree.** The TypeScript type in
`src/protocol/replay-contract.ts` requires `normalizedSessionId`. The
`required` list in `protocol/schemas/event.schema.json` does not include
it. `validatePrpEvent` accepts events that the type forbids.
2. **`requestCheckboxConfirmation` is stricter than its runtime.** It
uses the output type, so plugin authors must pass `continuationPolicy`
and `allowDeclineReason`, although the schema gives them defaults.
`z.input` would fix this.
3. **`RetainedRunnerdMaintenanceInput.backend.kind` is a free string.**
The quarantine domain comes from `descriptor.kind`, but the release uses
`proof.backend.kind`. If they differ,
`completeRetainedNativeSessionCleanup` returns 0 and the quarantine
stays. The production caller passes `"runner"` today, so this is not a
live bug.
4. **`CodexHarnessSession.close(input?)` is optional, but the contract
requires `{ reason }`.**

These are tests that verified nothing. I corrected them, and they pass:

- `backends/native-instruction-measurement.test.ts` filtered on
`placement === "core"`, which is not a valid placement. It measured an
empty tool list. It now measures 14 tools.
- `drivers/acpx/codex-acpx-driver.test.ts` checked for `usage.updated`,
which is not an event type.
- The `hostFixture()` helper in `drivers/acpx/runtime-host.test.ts`
ignored two overrides, so
`expect(prepareSandbox).not.toHaveBeenCalled()` could not fail.
- `live/live-session.test.ts` set `agent: "pi"` before a spread that
overwrote it.
- `native-distribution-integrity.test.ts` passed an AbortSignal where
`openCommand` expects `{ signal }`.

These are gaps that this pull request does not close:

- **The root lockfile still has some old zod versions.**
paperclipai/paperclip#17364 moved the direct zod dependencies to 4.6.x.
`pnpm-lock.yaml` still resolves `zod@3.25.76`, `4.1.11`, and `4.4.3`,
which are below the patched 4.6.6.
`packages/paperclip-runner/scripts/pi-distribution/package.json` pins
`zod` to `3.25.76`, and some agent SDKs resolve `4.4.3`. Dependency
review does not flag them, because this pull request does not add them.
- **Dead GitHub coverage.** `chat-adapters-ui-providers.spec.ts`
returned early for GitHub, so four later `provider === "github"`
branches never ran. I removed them. They included the private key file
and paste match, and the reconnect flow. Nothing asserts those behaviors
in that spec now.
- **Provider tests do not run in CI.** The root `vitest.config.ts` lists
only Daytona. The other 8 providers, `bridge-template` included, are
typechecked now, but CI does not run their tests.
- **One `packages/db` test never runs.**
`scripts/clean-poisoned-claude-sessions.test.ts` is outside the
`include` of `packages/db/vitest.config.ts`.
- **`plugin-orchestration-smoke-example` does not collect.**
`@paperclipai/shared` is not linked in its `node_modules`. It passes 3
of 3 with an alias.

## Verification

Run these commands:

```sh
node scripts/typecheck-test-programs.mjs --low-memory
pnpm --filter @paperclipai/server typecheck
node scripts/typecheck-test-projects.mjs --low-memory --reuse-declarations
node --test scripts/typecheck-test-programs.test.mjs
```

Results on macOS (arm64), Node 26.12. CI uses Node 24, and its
`Typecheck + Release Registry` job also passes. At head `7d4b5e907a`,
the test programs, `tests/e2e`, and the script tests pass again with 0
errors. The times and peaks below come from the run before the two
latest merges of master:

| Check | Result |
| --- | --- |
| Test programs, `--low-memory` | Exit 0, 0 errors, 42 seconds, peak
1.78 GiB |
| Server layers, low memory | Exit 0, 0 errors, 38 seconds, peak 3.41
GiB |
| Test projects, including `tests/e2e` | Exit 0, 0 errors, 13 seconds,
peak 1.59 GiB |
| Script tests | 12 passed |
| Injected type error in each of 23 places | Each one failed its program
with TS2322 |
| Coverage check | 0 unchecked files, 10 seconds |

I did not run this in a 4 GiB container. The peaks above are `maximum
resident set size` for the largest single process.

**Runtime check of the fixes.** Types do not prove that a test still
works, so I ran the suites before the two latest merges of master:

| Suite | Result |
| --- | --- |
| Runner, full suite | 4,885 passed, 3 failed, 13 skipped |
| Sandbox providers, 10 packages | 939 passed, 12 failed, 36 skipped |
| Plugin SDK | 200 passed |
| `kv-demo-mcp-server` | 12 passed |
| Renamed UI test and its sibling | 4 passed |

All 15 failures exist without these fixes or come from the local
machine:

- **3 Cursor Cloud timeouts.** `artifact-delivery.test.ts` has 1 and
`cursor-cloud-driver.test.ts` has 2. They time out in a full local run,
and all 133 tests in those files with `codex-security-config.test.ts`
pass when run alone. They fail in the same way with those two files at
their unedited versions.
- **12 Kubernetes `file-sync.test.ts` failures.** They fail in the same
way before and after the fixes. The first failure is that
`/proc/self/fd` does not exist on macOS.

After each of the two latest merges of master, I ran only the Runner
test files that I changed:

| Merge | Result |
| --- | --- |
| `885c249cb9`, 8 files | 534 passed |
| `7d4b5e907a`, 7 files | 672 passed, 16 failed |

The 16 failures are in the two `source-reconciliation` test files. They
fail in the same way with master's unedited versions of those files. 15
of them fail with `ACPX sidecar exited at initialize`, so I think my
local build of the Rust and sidecar binaries is out of date. I did not
rebuild them to check. CI builds them from scratch, and all Runner jobs
pass in CI at `7d4b5e907a`.

**Not run:**

- `tests/e2e` and the 8 Playwright suites need a server, so most of them
have only compile checks. `playwright test --list` loads 308 tests from
the main e2e config.
- `packages/plugins/sandbox-providers/kubernetes/test/integration` needs
a cluster, so it has only compile checks.
- I did not run the full `pnpm test:run` or root `pnpm build`. Required
CI runs them.

**Reduced motion.** I ran 6 motion-sensitive Storybook specs in system
Chrome with a temporary config, once with `reduce` and once with
`no-preference`. Both runs had 45 passed and the same 5 failed. So
reduced motion did not change those results. The 5 failures do not
depend on motion. At least one comes from the browser, because system
Chrome prints `oklch(55.6% 0 0)` where the test expects `oklch(0.556 0
0)`. I did not run the other Storybook specs. The `storybook-visual`
label runs that workflow.

## Risks

- **A new lockfile to maintain.** A change to a provider `package.json`
now fails the typecheck job until someone refreshes
`scripts/typecheck-sandbox-providers/pnpm-lock.yaml`. The failure
message gives the command.
- **The job has a new install.** The step downloads provider SDKs, so it
depends on the npm registry. It uses `--frozen-lockfile` and
`--ignore-scripts`.
- **The local install replaces provider `node_modules`.** A contributor
with a standalone provider install must reinstall that provider after
running the script.
- **Reduced motion can change Storybook results.** See "Behavior change
for review". That workflow is not part of required CI.
- **Kubernetes plugin validation changed.** A release of this plugin
ships zod 4. Error wording changes, and very large integers are now
rejected. See "Production change for review".
- **Master moves fast.** Each merge of master brought new test code with
type errors, 27 so far. After this merges, the check reports those
errors in the pull request that adds them.

## Model Used

- Provider: Anthropic Claude, through Claude Code
- Model: Claude Opus 5.5 (`claude-opus-5-5`)
- Capabilities: tool use, code execution, and subagents for the type
fixes in separate files

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived
details. The app generated the branch name.
- [ ] I have run tests locally and they pass. See the failures explained
in Verification.
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 07:33:59 -08:00
DottaandPaperclip a194166017 perf(ci): repair dependency caches and parallelize transport fixtures (#18883)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Required CI checks protect changes to its native Runner and control
plane.
> - Recent Runner jobs restored a Rust cache but still rebuilt
dependencies for 74–92 seconds.
> - New workspace install hooks disabled the materialized JavaScript
dependency cache.
> - A single transport file also kept 326 test cases on one worker.
> - This pull request corrects the cache inputs, replays known install
hooks, and separates independent transport test groups.
> - The same checks can finish sooner without removing test cases or
widening runner access.

## Linked Issues or Issue Description

Refs #18794 and #18544. Related dependency-action version update:
#17698; this change fixes the cache input contract used by the currently
pinned action.

**What happened?**

Recent PR jobs such as #18783 and #18825 restored an exact Rust
dependency cache from October 16. The virtual workspace's development
profile changed on October 30. Rust Cache 2.9.2 hashes member manifests
but omits that virtual manifest. Rebuilding cannot replace an immutable
exact cache entry. The JavaScript cache also rejected the new server and
Runner Copilot postinstall hooks.

**Expected behavior**

Changing a Cargo profile selects a new dependency cache. An exact
JavaScript dependency restore still performs required workspace install
hooks. Independent transport fixtures use the existing file workers
while retaining all assertions and real deadlines.

**Steps to reproduce**

Inspect the Rust dependency cache hit and following `build:rust` in the
recent Runner package jobs. Run `node scripts/ci-dependency-cache.mjs`
on the previous revision to see `cacheable: false`. Run the original
transport file to measure the serial test work.

## What Changed

- Add the virtual workspace manifest and ancestor Cargo configuration to
all shared Rust dependency keys. Keep reader/writer identities aligned
and preserve master-only cache writes and CODEOWNERS-only AWS routing.
- Recognize only the exact reviewed root, server, and Runner postinstall
commands. Include the server/Runner installer scripts in the dependency
key. Replay workspace postinstall hooks after a materialized restore.
Unknown lifecycle hooks still disable reuse.
- Move attachment recovery, warm attachment, live adoption, and
suspension fixtures into separate files. Keep the large event-backlog
cases in the original file to avoid starting another expensive fixture
group at the same time. Share five stateless test helpers outside
production source.
- Extend the opt-in `profile-ci` benchmark with a PR-scoped Rust cache
producer and a fresh-host consumer. Require exact dependency/workspace
hits and reject any Cargo unit that is rebuilt. Retain the JSON build
evidence. Existing setup qualification checks real JavaScript cache
restores.
- Schedule the ten measured files over 50 seconds first, in approximate
duration order. Package CI uses up to four workers, bounded by available
CPUs; local runs retain Vitest defaults. The AWS runner has four CPUs
and 16 GiB RAM; the three-worker run averaged 44% CPU and peaked at 15%
memory. Preserve Vitest selection and shard membership and default
ordering for other files. Job count and AWS authorization are unchanged.
- Scope the optional paired server benchmark to changes in its measured
inputs. Cache-only changes qualify exact restores and run the shared
package tests using restored dependencies.
- Update the cache and test documentation and the existing native cache
contract guard.

## Verification

- All 586 `.github/scripts/tests/*.test.mjs` policy tests passed,
including cache trust and runner routing guards. Actionlint passed for
all three changed workflows.
- All 169 original top-level transport test definitions remain
byte-identical. The original file passed all 326 cases in 317.99 seconds
on this macOS host. A clean preliminary split passed the exact same
named cases in 183.01 seconds with the same four-worker cap.
- Preserve the first failed experiment: its parallel run overlapped the
policy suite and timed out in the 1,024-event settled backlog case at 60
seconds; 325 cases passed. A clean repeat passed, but that does not
establish the earlier failure's sole cause. The final layout keeps the
backlog cases in the original file. The final layout passed the same 326
cases in 186.35 seconds (41.4% faster than the 317.99-second baseline).
The original test bodies and all timeouts remain unchanged.
- Fresh-host Rust qualification passed: all 308 compilation units were
fresh, and Cargo completed in 1.59 seconds on a new runner ([final-head
run](https://github.com/paperclipai/paperclip/actions/runs/41745082837/job/126296908816)).
JavaScript setup dropped from 46 seconds cold to 23 seconds with exact
dependency/build restores; all 1,052 shared-package cases passed against
the restored tree ([final-head
job](https://github.com/paperclipai/paperclip/actions/runs/41745082837/job/126295937797)).
The previous qualification also passed (43 to 19 seconds) and confirmed
all three postinstall hooks replayed. These are qualification results,
not an end-to-end PR timing claim.
- Initial full AWS package run passed 4,907 cases with 10 existing
skips. The original-file remainder fell to 118.7 seconds, but the
100-second deadline file started last, leaving the full package at
417.43 seconds. Starting only three long files first passed 4,911 cases
(including four new sequencer guards) in 422.54 seconds, so reordering
alone did not improve full-package time. Summed file execution was 1,202
seconds across three busy workers. The final four-worker configuration
passed **4,912 cases, with the same 10 existing skips**, in **324.99
seconds**, a **23.1% reduction** from 422.54 seconds. The five added
cases guard ordering, selection, sharding, and path handling. Peak
memory was 17.95% of the 16 GiB runner; average CPU was 49.14%
([final-head package
job](https://github.com/paperclipai/paperclip/actions/runs/41745082974/job/126296339480)).
- Local workspace typecheck and build passed. The complete local `pnpm
test:run` passed all six chat partitions but reported failures in one
existing OAuth redirect case and one gateway authority case in the
general-server group. Both focused repetitions passed (three OAuth cases
and one gateway case), and a complete repeat of both files passed all
593 cases in 42.89 seconds; that does not explain the original failures.
The broad local run was stopped after 61 minutes once full final-head CI
had passed; its remaining general/workspace/serialized work was not
completed locally, and it is not reported as a pass. Original failure
logs were retained. Full final-head CI provides the complete suite
validation. Sequencer selection/shard guards and the 11-case transport
suspension smoke passed. The corrected release workflow guard passed all
71 tests.
- The first CI run failed only the policy guard that still expected the
old native cache key; downstream aggregate checks also failed. That
guard now checks the new profile/config prefix and existing
host/target/protocol suffix. A second policy run found that the new
sequencer test imported Vitest in the dependency-free policy lane; it
now runs in the Runner package lane, where all four cases pass. Greptile
reviewed final head `8011f84189` at 5/5 with no actionable findings and
confirmed this issue fixed. Final-head policy, typecheck, build, and
review passed. At 23:52:46 UTC on November 3, several AWS jobs were
terminated together: the Runner package process received SIGTERM and
browser logs recorded a runner shutdown signal. The failed/cancelled
jobs passed on the same-head retry, including required `ci / verify`; no
assertion or timeout was changed in response. The final concurrency
adjustment at `fe5a69bb0b` also received Greptile 5/5 with no actionable
findings. Final-head required CI, including `ci / verify`, passed
([aggregate](https://github.com/paperclipai/paperclip/actions/runs/41745082974/job/126300554711)).
Local timings exclude install, build, cache transfer, and CI queue time.

## Risks

A new Rust key needs one successful canonical master build before
ordinary PRs can hit it. The fresh-host qualification uses separate
PR-scoped entries. Parallel test files can increase I/O contention; case
execution inside each file, timeouts, and assertions are unchanged. Two
extracted suites exceed 1,000 lines because they retain existing
multi-stage recovery fixtures unchanged; one parameterized fixture alone
is 1,059 lines. No production execution logic, lockfile, credentials,
runner trust gates, or required check is removed.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository editing, command
execution, and GitHub inspection. The host does not expose a more
precise model snapshot or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — targeted checks passed;
the broad-run failures and repetitions are documented above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 20:17:08 -06:00