From d5dd09c9662e9836e7a8f98a45a793a9fe4a1804 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 7 Nov 2026 15:24:12 -0600 Subject: [PATCH] docs(plans): design turn-scoped ACPX MCP authority; correct the Claude baseline (#19912) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Follow-up messages must start fast. #19888 made Codex and OpenCode follow-ups 2.6–5× faster by keeping their runners warm. > - Claude (ACPX) did not gain, because ACPX relaunches the Claude process and reloads its session on every follow-up (3.2–4.2 s). > - That relaunch is a safety boundary: the ACPX MCP endpoint lives as long as the process and cannot attribute a tool call to a turn, so reaping the process is the proof against late calls. > - Removing it needs a new authority model, which needs a reviewed design before any code. > - This pull request adds that design and corrects the Claude baseline. ## Linked Issues or Issue Description Refs #19888 (warm default), #19846 (TTFT baseline, finding 4). **Docs issue** The TTFT docs did not explain why Claude follow-ups relaunch, what removing that would require, or the corrected Claude numbers. **What is wrong or missing** `doc/plans/2026-11-07-warm-follow-up-ttft.md` reported a Claude follow-up at 10.5 s, measured under heavy host load. A quieter re-measurement gives 6.3 s per-turn against 7.1 s warm. There was no design for removing the per-turn relaunch. **Proposed fix** A design document for turn-scoped ACPX MCP authority, and a corrected-baseline section in the warm follow-up doc. ## What Changed - New `doc/plans/2026-11-07-turn-scoped-acpx-mcp-authority.md`, which covers: - the measured cost breakdown; - how the current boundary works (tool bridge bearer, call-ID tombstones, `restart_idle_provider`, `run.attach`); - a threat model (late tool calls from a settled turn, replay, credential replay, cross-turn and cross-run confusion, restart replay); - three candidate designs with their safety arguments: turn-scoped credentials in a live session, a proof that no calls are in flight, and an in-process session reload with per-generation credentials, which is recommended for a spike; - restart and chaos implications, the expected TTFT gain, and next steps. - `doc/plans/2026-11-07-warm-follow-up-ttft.md`: corrected Claude baseline. Per_turn stays the default; the earlier 10.5 s was host noise. Finding 4 now points to the design and drops the idle-retirement idea, which would only bring warm level with per-turn. - Docs only. No code changes until Dotta has reviewed the design and a security review is done. ## Verification - The Claude numbers come from `ttft-baseline.ts --adapters claude --samples 6 --phases coldwarm --lifecycle ` on master `fc6e1b0346` (load average about 9). - The step breakdown comes from temporary timing in runnerd's ACPX backend. It was not committed. - `node scripts/check-maps.mjs` passes. ## Risks - Docs only. The design proposes no behavior change until it has been reviewed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- ...26-11-07-turn-scoped-acpx-mcp-authority.md | 105 ++++++++++++++++++ doc/plans/2026-11-07-warm-follow-up-ttft.md | 16 ++- 2 files changed, 119 insertions(+), 2 deletions(-) create mode 100644 doc/plans/2026-11-07-turn-scoped-acpx-mcp-authority.md diff --git a/doc/plans/2026-11-07-turn-scoped-acpx-mcp-authority.md b/doc/plans/2026-11-07-turn-scoped-acpx-mcp-authority.md new file mode 100644 index 00000000000..0d0cec994bc --- /dev/null +++ b/doc/plans/2026-11-07-turn-scoped-acpx-mcp-authority.md @@ -0,0 +1,105 @@ +# Turn-scoped ACPX MCP authority (design, not implemented) + +Status: proposal, 2026-11-07. No code until Dotta has reviewed this design and a security review is done. +Context: [`2026-11-07-warm-follow-up-ttft.md`](2026-11-07-warm-follow-up-ttft.md) (finding 4) and [`2026-11-07-ttft-baseline.md`](2026-11-07-ttft-baseline.md). + +## Summary + +- **Every Claude (ACPX) follow-up relaunches the Claude provider process and reloads its session.** That is 3.2–4.2 s of each follow-up's time to first token (TTFT). +- **The relaunch is a safety boundary.** The ACPX MCP endpoint lives as long as the provider process, so it cannot tell which turn sent a tool call. Reaping the process proves that no late call from an earlier turn can execute under a later turn's authority. +- **The relaunch is also a run boundary.** Each follow-up comment is a new heartbeat run. A new run brings a new Paperclip gateway token, new instruction text and new file grants, which today reach the provider only at launch. +- **Removing the relaunch needs two things:** + - a way to bind every MCP call to the turn and run that issued it, so late calls are rejected by authentication rather than by process death; + - a way for a live provider to adopt a new run's authority. +- **This document compares three designs.** It recommends prototyping design C: an in-process session reload with per-generation credentials. It is the smallest change that keeps a process-independent proof. +- **Expected gain:** a Claude follow-up drops from about 6.3 s to about 3–4 s TTFT p50 locally, if the in-process reload is as fast as measured below. + +## Measured cost (local, current master) + +Claude follow-up TTFT p50 is 6.3 s with `per_turn` (the default) and 7.1 s with `warm` (n = 6 each, load average about 9). A new task takes 11.4 s. A per-turn runner retires the old provider generation at the end of the previous run. A warm runner does it inside `run.attach`, on the follow-up's critical path, which is why warm is slower for Claude today. + +Breakdown of a warm Claude follow-up, measured with temporary timing in runnerd's ACPX backend (2 samples): + +| Step | Time | On the critical path | +|---|---|---| +| `run.attach`: checkpoint and shut down the old provider generation | 1.16–1.18 s | warm only | +| Start the new generation (spawn the sidecar, `session/load` without initialization) | 0.73–0.99 s | yes | +| First prompt: deferred Claude ACP initialization and session load | 2.46–2.54 s | yes (`turn.submitted` → `turn.started`) | + +A per-turn follow-up pays the second and third rows. `runner.turn.submitted_to_started` is 3.16 s p50 per-turn and 2.67 s warm. The new-generation start appears as `provider.session.load` (1.2 s p50 per-turn). + +## How the boundary works today + +- **Tool calls.** The runner's tool bridge (`startRunnerToolBridge`) serves a runner-owned MCP endpoint (`url` + `bearerToken`, where the token is the bridge secret). Paperclip's native MCP gateway is a second runner-owned binding with its own token. Both bindings are passed to the provider in ACP `session/new` or `session/load` (`runtime-host.ts`, `mcpServers`). ACP has no request that changes MCP servers during a session. +- **Turn epochs.** `provider_bridge.rs` keeps a tombstone for every settled call ID, so a replayed call never executes twice. `prepare_turn` keeps the tombstones across turns. `prepare_turn_after_provider_restart` releases them, and it is legal only after the owning process was reaped. Codex continues in one process across turns because its tool calls carry turn and thread identity in the app-server protocol, so it only needs the tombstones (bounded at 65,536 call IDs, then a process-boundary rollover). +- **ACPX turns.** `acpx_provider_session.rs` `start_turn` calls `restart_idle_provider` whenever the session has settled turns. The comment there states the reason: "The MCP endpoint is session-lifetime and cannot authenticate which provider turn originated a late HTTP callback." A new generation starts with no settled turns, so the first turn after a relaunch does not relaunch again. +- **ACPX runs.** `acpx_provider_backend.rs` `attach_run` checkpoints the sidecar and lets the next `session.open` start a new generation. `run_attachment_policy` allows Claude to rotate registered asset paths across runs (`AuthenticatedAssetPaths`); the new text reaches the provider at launch. + +## Threat model + +Assets: the authority of the current turn and run. That covers its tools, its Paperclip gateway grants, its completion contract and its workspace and file grants. + +| Threat | What goes wrong | How today's design stops it | +|---|---|---| +| T1. Late call from a settled turn | The provider sends a tool call after its turn ended. Examples: an in-flight HTTP request after an interrupt, a retry, or a background subagent. If it arrives during the next turn, it executes with that turn's authority and results. | The old process is reaped before the next turn, so no such call can exist. | +| T2. Replay of a completed call | A call ID that already executed is sent again. | Call-ID tombstones, kept until the owning process is reaped. | +| T3. Credential replay | A captured bearer token is used by another process or after its owner should have lost authority. | The token is per bridge instance and dies with the process. | +| T4. Cross-turn confusion | A call is attributed to the wrong turn's completion contract or receipts. | One turn per process generation. | +| T5. Cross-run confusion | A call from run N executes under run N+1's Paperclip gateway grants, instructions or file grants. | `run.attach` relaunches the provider; the gateway token is run-scoped. | +| T6. Restart replay | After a controller or runner restart, a recovered generation accepts calls meant for the previous owner. | Generation identity, lifetime fences and the restart-recovery checks. | + +Any replacement must stop T1, T2, T4 and T5 without relying on process death, and must not weaken T3 or T6. + +## Candidate designs + +### A. Turn-scoped credentials in a live session + +Mint a new bearer token per turn and revoke the previous one at the bridge. + +- **Safety argument:** a late call from turn N carries turn N's revoked token and is rejected by authentication (T1, T4). Tombstones still stop T2. +- **Blocker:** ACP passes MCP servers only at `session/new` or `session/load`. A provider cannot learn a new token mid-session, so A reduces to design C. + +### B. Proof that no calls are in flight + +Keep the process when a turn settles cleanly, and relaunch only when the proof fails. + +- **Proof conditions:** the turn ended with ACP `end_turn` (not cancel, interrupt or failure); the bridge has no pending or completed receipts; there are no runtime requests; the goal is dormant. Between turns, the bridge also rejects every call. +- **Safety argument:** stops late calls that arrive between turns. It also relies on a provider guarantee: once Claude ACP reports `end_turn`, the provider issues no further MCP calls for that turn. +- **Gap:** that guarantee is a property of the upstream provider, not something Paperclip can verify. It would have to be qualified per provider release, and every background subagent or hook feature is a regression risk. A call that arrives during the next turn is still indistinguishable (T1, T4). +- **Cross-run:** B does not address T5. Run-scoped authority still needs a relaunch, so B helps only same-run turns, which are rare: each comment is a new run. + +### C. In-process session reload with per-generation credentials (recommended prototype) + +Keep the Claude ACP process. At each turn or run boundary, revoke the bridge credential and reload the same provider session (`session/load` with the same session ID) in the same process. The reload carries a freshly minted bridge token, the new run's Paperclip gateway token, and the new instruction and grant text. + +- **Safety argument:** every MCP binding is authenticated by a token minted for exactly one session generation (turn, plus run where it changed). The bridge revokes the previous token before the reload is acknowledged. + - T1 and T4: a late call from an earlier generation is rejected by authentication, whatever its call ID or arrival time. + - T2: call-ID tombstones still apply, and they can be released when the token epoch rotates, because no old-epoch call can authenticate. + - T3: tokens remain unguessable, per-generation and runner-owned. + - T5: the run-scoped gateway token rotates with the run. + - T6: generation identity is persisted with the token epoch, and a recovered runner mints new tokens. +- **Required: bind the epoch at authentication and recheck it before dispatch.** The bridge authenticates the bearer before it reads the request body, and the sidecar binds a call to the active turn only after that. A request that authenticated under epoch N can therefore finish arriving after rotation to N+1. It holds no pending tool receipt while its body is in transit, so the pending-call guard cannot block the next turn. The bridge must record the epoch a request authenticated with, and reject the request if the current epoch differs after the body is read and before dispatch. It must make the same check before it binds the call to a turn. Rotation must not be acknowledged while any request authenticated under the old epoch is still being read. +- **Cost:** removes the process spawn and the old process's checkpoint and shutdown. It keeps one `session/load`. The expected saving is the generation start plus the part of the deferred initialization that is process start-up rather than session load, measured below. +- **Unknowns to settle before code:** + 1. Whether Claude ACP supports a second `session/load` on a live process with different MCP servers and instructions, and tears down the first query's MCP clients, which is required so old-epoch connections cannot persist. + 2. The latency of an in-process reload compared with a relaunch. + 3. Whether the instruction and grant text can change on reload, or only the MCP servers. + +## Restart and chaos implications + +- A kept-alive Claude process becomes part of warm runner state. It follows the same reattach, fence and reconnect-grace rules as Codex and OpenCode warm runners (#19879). +- Persisted state gains the token epoch and generation ID. Previous-release story (`rules/upgrade-safe-run-state.mdc`): a session saved by the current release has no token epoch. It resumes by relaunching, as today, and the first relaunch establishes epoch 1. No refusal is needed. +- Restart recovery must never reuse a revoked token. A recovered runner rotates tokens before admitting any call. +- Chaos qualification: K1, K2 and K6 waves with Claude on warm, plus a new fault, "interrupt a turn with a tool call in flight, then send a follow-up", which must show the late call rejected with an authentication error. All existing R1–R6 checks must stay green. +- Red/green test for the authentication-to-dispatch gap: start a tool request under epoch N and hold its body; rotate to N+1 and start the next turn; then release the body. The request must be rejected and must never reach a turn. + +## Expected TTFT gain + +If an in-process reload costs about one `session/load` (an estimated 1–1.5 s, not yet measured) instead of a relaunch (3.2–4.2 s), a warm Claude follow-up drops from about 6.3 s to about 3–4 s TTFT p50 locally. Claude would then default to warm, like Codex and OpenCode. Design B, even where its proof holds, does not help cross-run follow-ups, which are the common case. + +## Next steps + +1. Dotta reviews this design. +2. Security review of the threat model and of design C. +3. Spike, no product code: measure an in-process `session/load` with rotated MCP servers against Claude ACP, and confirm the first query's MCP clients are torn down. +4. Only then, an implementation plan with red/green tests for T1–T6 and the chaos wave above. diff --git a/doc/plans/2026-11-07-warm-follow-up-ttft.md b/doc/plans/2026-11-07-warm-follow-up-ttft.md index 0b6162e956d..bc34f79c33d 100644 --- a/doc/plans/2026-11-07-warm-follow-up-ttft.md +++ b/doc/plans/2026-11-07-warm-follow-up-ttft.md @@ -5,7 +5,7 @@ Status: measured 2026-11-07. Follows [`2026-11-07-ttft-baseline.md`](2026-11-07- ## Summary - **Warm runners make Codex and OpenCode follow-ups 2.6–5× faster.** Follow-up TTFT p50 drops from 12.0 s to 4.7 s (Codex) and from 21.7 s to 4.2 s (OpenCode). A follow-up is now faster than a new task, which it was not before. -- **Claude does not gain.** Its ACPX runner relaunches the Claude process and reloads the session on every follow-up turn by design, so warm keeps runnerd alive but the follow-up still takes about 10.7 s. Claude stays per-turn by default. +- **Claude does not gain.** Its ACPX runner relaunches the Claude process and reloads the session on every follow-up turn by design, so warm keeps runnerd alive without shortening the follow-up. Claude stays per-turn by default. A quieter re-measurement (below) puts the Claude follow-up at 6.3 s per-turn against 7.1 s warm; the 10.5 s in the main table was host load. - **Restart recovery is not slower with warm.** K1 and K2 recovery latency is equal or lower, and every interrupted run recovered. - **The default is now warm for local Codex and OpenCode** agents that never chose a lifecycle, with the 5-minute idle window. Explicit choices and admitted sessions keep their policy. - **Two defects blocked warm and are fixed:** warm follow-ups kept the first run's completion contract (#19874; every OpenCode warm follow-up looped until timeout), and an idle warm runner whose controller crashed could live up to its 1-hour connection lease (now a 7-minute reconnect grace). @@ -34,6 +34,18 @@ Status: measured 2026-11-07. Follows [`2026-11-07-ttft-baseline.md`](2026-11-07- A new task is cold in both lifecycles (a new provider session), so its differences are host noise. +### Corrected Claude baseline + +The Claude rows above ran at a 1-minute load average of 13–27. A Claude-only re-measurement on master `fc6e1b0346` at a load average of about 9 (n = 6 each, `--adapters claude --samples 6 --phases coldwarm`): + +| Claude | per_turn | warm | +|---|---|---| +| New task TTFT p50 / p95 | 11.37 / 12.84 s | 9.66 / 12.52 s | +| Follow-up TTFT p50 / p95 | 6.30 / 7.28 s | 7.13 / 7.26 s | +| Follow-up `turn.submitted` → `turn.started` p50 | 3.16 s | 2.67 s | + +A Claude follow-up is already faster than a new task with `per_turn`, and `per_turn` stays the default. The earlier 10.5 s follow-up figure was host noise. Warm is slower for Claude because its `run.attach` retires the old provider generation on the critical path (about 1.2 s), which a per-turn runner does at the end of the previous run. + ### Where a per_turn follow-up spends its time (p50) | Adapter | Runner launch + provider restore | `turn.submitted` → `turn.started` | @@ -91,7 +103,7 @@ This check found that `normalizePaperclipRunnerAdapterConfig`, which runs on eve The cause is ours, not the provider's. The ACPX session reaps the sidecar and Claude process at the start of every turn after the first, then resumes the same persistent session in a fresh process generation (`acpx_provider_session.rs`, `restart_idle_provider`). The documented reason: the MCP endpoint is session-lifetime and cannot tell which provider turn sent a late HTTP callback. On a resumed session the sidecar also defers provider initialization to the first prompt (`acpx-runtime-sidecar.ts`, `await runtimeTurn.promptStarted`). Both land between `turn.submitted` and `turn.started`: 3.8 s p50 per_turn, 2.75 s warm. -A warm runner does not avoid it. `run.attach` checkpoints the ACPX sidecar and the next `session.open` starts a new process generation and loads the session (about 5 s of the warm follow-up's attach), and that resumed generation still defers its provider initialization to the first prompt (2.75 s). The relaunch itself is required: the old generation must be reaped before its call-ID tombstones are released. What can change is when it happens. Preparing the next generation while the session is idle, after the previous turn settles, would take it off the follow-up's critical path. That is a runnerd change with its own late-callback proof and timing receipts. Until then Claude stays per-turn by default. +A warm runner does not avoid it. Instrumented, a warm follow-up spends 1.2 s in `run.attach` checkpointing and shutting down the old provider generation, 0.7–1.0 s starting the new generation, and 2.5 s in the deferred initialization and session load at the first prompt. Retiring the old generation earlier would only bring warm level with per-turn. Removing the relaunch itself needs MCP authority that is scoped to a turn and run instead of to the provider process; see [`2026-11-07-turn-scoped-acpx-mcp-authority.md`](2026-11-07-turn-scoped-acpx-mcp-authority.md). Until then Claude stays per-turn by default. ## Gaps