Commit Graph
682 Commits
Author SHA1 Message Date
Nicky LeachandClaude Fable 5.2 ddfbbfff20 fix(runner): recover provider-started turns after a controller restart (#21612)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core runs Claude Code through Paperclip Runner. Claude can start a
shell command in the background and end its turn. When the command
finishes, Claude Code starts a cycle of its own to continue.
> - The Runner can keep Claude while a background shell is running and
give the cycle of Claude to the controller as a turn that the provider
started. Core can wait for that turn. #21607 lets a caller ask the
transport for it.
> - A controller can restart at any time, for example at a deploy. A
wait can last 10 minutes and a provider-started turn can last longer, so
a restart during one of them is a normal event.
> - Tests with the real `paperclip-runnerd` showed that such a restart
failed the session. The checkpoint that Core had was the checkpoint of
the turn before, and the recovery of the driver refused or mislabelled
the turns that it did not know.
> - This pull request makes that restart safe. It changes no production
run, because no caller asks for provider-started turns before part 3.
> - The benefit is that part 3 can turn the behavior on without a failed
run at each deploy.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

#21456 turned the behavior on and had grown too large to review. It is
now three stacked pull requests:

1. #21607 (merged): the transport can ask the Runner for
provider-started turns. No caller asks.
2. **This pull request:** the recovery after a controller restart during
a wait or during a provider-started turn.
3. #21456: Core sets the option for native ACPX Claude runs. Only that
part changes what a production run does.

This is part 2. Its base branch is `master`. Refs #19912 (design),
#21456.

**What existing behavior does this improve?**

None in production. It makes a controller restart safe for a run that
asked for provider-started turns.

**Subsystem affected**

Paperclip Runner package: the consumer
(`src/native-session-runtime.ts`), the session backend
(`src/backends/harness-driver-backend.ts`), the driver
(`src/drivers/codex/`) and the transport (`src/live/`). Only for a run
that asked for provider-started turns.

**Current behavior**

- The consumer saves no checkpoint when a wait starts or when a
provider-started turn starts. During a wait and during a
provider-started turn, the checkpoint that Core has is the checkpoint
from the start of the turn before. It names that turn as active.
- A restart from that checkpoint fails: the driver has the old turn as
active, gets the events of a provider-started turn, and stops the
session with `turn_binding_mismatch`.
- The recovery of the driver adopts one turn that it does not know as a
disposition recovery turn, and refuses two or more. With two or more, it
closes the transport, and that stops the Runner and Claude.
- The Runner does not send again an event that the controller
acknowledged. If the start of a provider-started turn was acknowledged
before the restart, the new driver gets the other events of that turn
with no start.
- A turn that ends waiting for a provider-started turn sets the terminal
of the session, as a turn that ends the run does. After a restart, if
the provider-started turn already has its result when recovery
completes, the consumer finds a result and a terminal, settles the run,
and does not read the events of that turn. The start, the items, the
usage event and the terminal of that turn do not reach Core.
- The recovered driver reads notifications from its construction, and
`reconcile()` maps the terminal of the saved turn only after its reads.
If the saved checkpoint is still the checkpoint of the controller turn
and the start of the provider-started turn is already queued, the driver
judges that start too early and stops the session with
`turn_binding_mismatch`.
- A checkpoint that waits can also have a consumed disposition marker (a
recovery turn after an earlier restart ended waiting). The driver
restores that session as closed and reads no notifications, and the
consumer settles the run on the earlier terminal with no result.

**Proposed behavior**

- **Checkpoints.** The consumer saves a checkpoint when a wait starts
and when a provider-started turn starts, with the function that it
already uses for a governed wait. A checkpoint that fails stops the
turn, as for a governed wait.
- **A checkpoint as of the event that Core stored.** The driver gives
each event its sequence when it reads it from the provider, before the
consumer stores it. A snapshot that is taken later can be ahead of Core.
For these two checkpoints, the driver captures its snapshot when it has
handled the notification that made the event (the `turn.completed` that
waits, or the `turn.started` of the provider), with the sequence of that
event. The consumer asks for the snapshot as of the event that it has
just appended. The saved state and cursor are then those of an event
that Core has.
- **Session terminal.** A turn that ends waiting for a provider-started
turn does not set the terminal of the session (`HarnessNativeSession`).
The terminal of the session is the end of the run, and that turn does
not end the run. A checkpoint that is saved during a wait or during a
provider-started turn has no terminal, so a recovered session settles
only from the events that it reads, in order, to the terminal of the
turn that has the result.
- **Driver, a checkpoint that waits.** Recovery keeps the wait. It does
not adopt and does not refuse the turns after the last terminal that it
knows. The Runner sends again, in order, each event that was not
acknowledged, and the driver admits each provider-started turn from that
stream, as it does with no restart.
- **Driver, a checkpoint with an active turn.** `reconcile()` accepts
one turn in progress that comes after the saved active turn, when the
saved turn ended with the wait marker. It refuses each other unexpected
active turn as before, and it refuses this one too when the reconciled
turn settled a result.
- **Driver, a start that is queued before recovery reconciled the saved
turn.** A `turn/started` of the provider that arrives while the session
does not wait for one first waits until `reconcile()` has settled, and
is then judged as before. `reconcile()` judges the live state after its
reads; if the stream already settled the saved turn into a wait, it
accepts exactly one turn in progress after the last terminal turn and
does not adopt it. The driver admits that turn at its start.
- **A wait with a consumed disposition marker.** The driver keeps
reading notifications for a restored session that waits, as it does for
a wait with no restart. It still refuses a second controller-started
turn. The consumer does not take the consumed-marker shortcut for a
recovered snapshot that waits, so the wait is resumed.
- **Transport, the start comes first.** For each driver, the transport
gives the `turn/started` of a provider-started turn before each other
notification and before a tool call of that turn. If it did not give
that start to this driver, it builds it from the `turn.started` event in
its stored events. It does not do this for the turn that the recovered
driver already has as active (`resumeActiveTurnId`). It binds the events
and the tool call of a provider-started turn to that turn, not to the
turn that the Runner reports as active at the time of the recovery read.
- **Transport, which start owns an event.** The stored events can have
gaps: the window removes old events but keeps the latest of each anchor
type, and the controller checkpoint drops streamed deltas. The owner of
an event is the last stored `turn.started` before it, if it is a
provider start, and if the first turn terminal that is stored after that
start is not before the event and is the terminal of the same provider
turn (`providerTurnId`). If the first stored terminal after the start is
the terminal of a different turn, the end of the provider turn is not
known, and the transport builds no start.

**Reason and benefit**

- The checkpoint is the cause. With a true checkpoint, the two states of
a restart are states that recovery can handle: a wait, or an active
turn.
- The Runner already sends again each event that was not acknowledged.
Recovery uses that stream and does not build the turns again from a
read.
- The window removes a terminal only while a later terminal of the same
type is stored. Thus the first stored terminal after a start shows if a
terminal of that turn was removed. A rule that only ignores the gaps is
not safe: the Runner can make a `turn.failed` for a turn that did not
emit its start (`provider_turn_lost_on_restore`).

## What Changed

Production code: 113 lines added, 19 removed.

- `src/native-session-runtime.ts` (15 added, 4 removed): the two
checkpoints in `consumeTurn`, with the sequence of the appended event,
and the resumed wait with a consumed disposition marker.
- `src/drivers/codex/codex-session-state.ts` (24 added, 1 removed),
`codex-session-terminal.ts` (2 added), `codex-session-notifications.ts`
(2 added), `codex-harness-session.ts` (24 added, 7 removed),
`codex-app-server-driver-impl.ts` (4 added, 1 removed): the snapshot at
the event, and the recovery of the driver.
- `src/contracts/harness-driver.ts` and
`src/contracts/native-session-backend.ts` (1 added each): an optional
`boundarySnapshot` on a driver session, and an optional sequence in the
snapshot options. Other drivers do not implement it and are not changed.
- `src/backends/harness-driver-backend.ts` (7 added, 5 removed): the
session terminal, and the snapshot as of a sequence.
- `src/live/runnerd-codex-transport.ts` (15 added, 1 removed) and
`src/live/provider-turn-tool-order.ts` (18 added): the start-first rule
and the owner of an event.
- Tests: `src/live/runnerd-acpx-provider-turn.test.ts` (572 added, 5
removed),
`src/drivers/codex/codex-app-server-driver.provider-turn.test.ts` (348
added), `src/live/runnerd-codex-transport.test.ts` (137 added),
`src/control-plane/durable-prp-control-plane.test.ts` (84 added),
`src/live/provider-turn-tool-order.test.ts` (new, 61),
`src/native-session-runtime.provider-turn.test.ts` (92 added, 6
removed), `src/backends/harness-driver-backend.test.ts` (56 added).
- `runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs` (test binary
only, 123 added, 14 removed): a delay before a provider-started turn, a
goal state at its terminal, a list of provider-started turns for one
session, a gap between them, a turn that stays open until the new
controller reads the snapshot, and a turn that starts after that read.
The modes that existed are not changed, and the fake reads its input on
one thread as before.

## State saved by the previous release

- No new persisted field. The driver session already has
`awaitingProviderTurn` (#21453).
- **More checkpoints:** one when a wait starts and one when a
provider-started turn starts. An ordinary turn saves the same number of
checkpoints as before (4 in the unit test, measured before and after).
- **No terminal in a checkpoint of a wait.** A checkpoint that is saved
during a wait or during a provider-started turn has `terminal: null`.
When a wait ends with no provider-started turn (the limit, a comment of
a person, or the Runner ends the run), the last checkpoint of the run
also has `terminal: null`; before, it had the terminal of the turn that
waited. One reader uses that field with no result: the text of a cleanup
failure (`nativeCheckpointRecordsCompletedTurn`), which then does not
say that the turn completed. In the tests with the real daemon (the
limit, the provider process lost, the sidecar exits), the run still ends
on the path of today (`native_finalization_missing`).
- **A checkpoint of the previous release** has no waiting state.
Recovery takes the path of today.
- **Rollback of the Runner package during a wait:** the previous code
reads the checkpoint, ignores `awaitingProviderTurn`, and takes its
recovery path of today.

## Verification

With nothing from part 3. The Rust checks and the build checks in the
first item were run before the last seven commits; these commits change
no Rust. The other items were run at the head of this branch
(`36d5e5e587`), or at an earlier commit of this branch for a suite whose
files the later commits do not change.

- Before the last seven commits: `pnpm run typecheck:typescript`, `pnpm
run build:typescript` and `node --test
test/acpx-sidecar-contract.test.mjs` (18 passed) in
`packages/paperclip-runner`; `node
scripts/verify-runner-vendor-dependencies.mjs` in `server`; `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`; `cargo fmt
--check`, `cargo build`, the 17 `acpx_*` suites (241 passed) and the
`acpx` unit tests (102 passed): all passed.
- `npx tsc --noEmit -p tsconfig.json` and `npx tsc --noEmit -p
tsconfig.test.json` in `packages/paperclip-runner`; `pnpm typecheck` in
`server`: all passed.
- `src/live/runnerd-acpx-provider-turn.test.ts`, 4 runs with the
production code of this head, 26 of 26 passed in each. The Core of these
tests follows the production append (`appendNativeSourceEvent`): it
stores an event that is out of order, keeps its contiguous cursor below
a gap, and reports a duplicate or a conflict for an event that it has.
It does not reject a gap in the sequence; production does not. The
restart cases, all with the real `paperclip-runnerd`:
- **From the checkpoint that the real consumer saved.** The first
`executeNativeSession` is fenced and detached as the server does for a
restart, and a second one recovers from the last checkpoint that the
first one saved. (a) Restart during the wait; the provider-started turn
starts after the new controller is attached. (b) Restart while a
provider-started turn is in progress. In each: the test asserts the
shape of that checkpoint, the run settles with the result of the
provider-started turn, each event is appended one time and in order, the
turn has one `turn.started` with `origin: "provider"`, and the session
closes with no error.
- **Without the two checkpoints, each of these cases fails:** the saved
checkpoint is the checkpoint of the turn before, and the second
controller fails with `turn_binding_mismatch` or with a replay conflict.
- **A slow `appendEvent`, with events that arrive behind the stored
event** (3 cases, each run 3 times with the fix and 3 times without it,
with the same result in each run). R7: the append of the
`turn.completed` that waits is slow, and the provider-started turn
starts behind it. With the snapshot at the event, the checkpoint is the
waiting state at the sequence that Core has, and Core gets the
`turn.started` of the turn one time. Without it, the checkpoint names
the provider-started turn as active, the transport builds no start, and
Core never gets that `turn.started`. R8: the first controller answers
the `paperclip_finish` call while the append of the provider
`turn.started` is slow. With the snapshot at the event, the checkpoint
has no result and the run ends with `native_finalization_missing` (see
Risks). Without it, the checkpoint has a result that Core does not have,
and recovery stops with "session cannot start another turn". R6: the
append of the provider `turn.started` is slow and a delta is queued
behind it. With the snapshot at the event, the new controller continues
the sequence that Core has. Without it, the only difference is one
sequence number that Core never gets.
- **The start was acknowledged before the restart** (3 cases: one turn
open across the restart; two turns; a turn that waits for the answer to
its `paperclip_finish` call). The new driver gets the start of each turn
one time before its other events, the finish call is answered exactly
one time, and the session closes with no error.
- **The turn ran with no controller attached** (2 cases, with and with
no result) and **two turns in one outage** (2 cases): the turns arrive
in order after the recovery.
- **A provider-started turn that is active in the checkpoint** gets no
second start, and the next provider-started turn arrives after it.
- **The turn already has its result when recovery completes** (3 cases:
the turn made its `paperclip_finish` call with no controller attached;
the turn waited for the answer to that call at the restart; two turns
across the restart). The test asserts the exact list of events that Core
gets: each provider-started turn has one start, its usage item, its
message item and its terminal, and then `run.result.accepted` and
`run.terminal`. `completeRun` is called one time with the turn that has
the result. Before the change to the session terminal, the second
controller appended only `run.result.accepted` and `run.terminal` in
each of these cases.
- **The wait checkpoint is not saved, then a restart** (R10). The last
checkpoint is the checkpoint of the controller turn, Core has the
terminal of that turn, and the start of the provider-started turn is
queued. The run settles with the result of the provider-started turn,
Core has one `turn.started` for it, and the finish call is answered one
time. Without the wait in the driver, recovery stops with
`turn_binding_mismatch`.
- **An ordinary turn**, with no option. One that waits for the answer to
its finish call at a restart: Core gets its result, its terminal and the
run terminal one time. One whose checkpoint after `startTurn` is ahead
of Core at a restart (R9): the run settles, and Core does not have only
the one event that the first controller did not store. These cases pass
with and without this pull request.
- **No checkpoint with a terminal and no result** is saved by the two
controllers in the cases above.
- `src/live/provider-turn-tool-order.test.ts` and
`src/live/runnerd-codex-transport.test.ts`: 345 passed. The helper tests
cover a stored start with removed events after it, a dropped delta, the
terminal of the turn before and after the event, a terminal of a
different turn after a removed terminal, and a controller start. With no
option, with a different agent or provider, and for a turn that the
controller started, the notifications of the transport are the same as
before this pull request.
- `src/control-plane/durable-prp-control-plane.test.ts`: 230 passed.
Three new tests use the real commit path: a full window of 4,096 events
removes the events after a provider start and the start is found; the
window removes the `turn.failed` of a provider turn after a later
`turn.failed` with no start, and no start is found; a restored
checkpoint that dropped the deltas of the turn, and the start is found.
- `src/drivers/codex` and `src/backends/harness-driver-backend.test.ts`:
491 passed. The driver tests cover a provider start that is queued
before `recoverSession()` is called, the wait state with one turn in
progress at `reconcile()` (and the two cases that it must still refuse),
a restored wait with a consumed disposition marker (the provider turn is
admitted and `startTurn` is refused), the snapshot at the two events
(also after the driver moved on, and for a restart continuation that
ends waiting), one later turn in progress, one completed and a second in
progress, two completed with a result, and the cases that `reconcile()`
must still refuse. The backend tests cover the session terminal and the
snapshot as of a sequence.
- `src/native-session-runtime.provider-turn.test.ts` and
`src/native-session-runtime.test.ts`: 333 passed. Two new cases resume a
restored wait with a consumed disposition marker: the provider-started
turn arrives and settles the run, or the wait ends at its limit and the
run takes the path of today with no new turn.
- `server`: `native-session-executor.test.ts`,
`native-provider-turn-wait.test.ts` and
`native-runner-restart-recovery.integration.test.ts`: 1,743 passed, 21
skipped.
- Mutation checks, each made named tests fail: no checkpoint at the
start of a wait; no checkpoint at the start of a provider-started turn;
the consumer does not pass the sequence (R6, R7 and R8 fail: a hole in
the sequence, no `turn.started` in Core, a different failure); the
backend ignores the sequence; the driver has no snapshot at the event;
no start-first rule; no exclusion of the active turn of the recovered
driver; the turn binding only at the first delivery; the contiguous
sequence required again; no check of the first terminal after the start;
the adoption of later turns put back; no exception in `reconcile()`; no
check after the settlement in `reconcile()`; the terminal of the session
set again by a turn that ends waiting; no wait of the provider start for
`reconcile()`; no exception in `reconcile()` for a session that waits;
the pump deferred for a restored wait; the consumed-marker shortcut
taken for a snapshot that waits.

Not run: the full suite. CI supplies that coverage.

## Risks

- **No production effect.** Each change is behind the waiting state, a
provider `turn.started`, or the option of the transport. No caller sets
the option before part 3.
- **An event that the first controller received but Core did not store
is not sent again after a restart.** This is the same on `master` and is
not changed here. Only the terminal of a turn is built again. If the
lost event is the result of a provider-started turn (the first
controller answered the `paperclip_finish` call and stopped before Core
stored the proposal), the run ends on the path of today for a turn with
no result. R8 shows this. R9 shows the same loss for one event of an
ordinary turn.
- **A second `turn.started` for one provider-started turn can reach
Core.** This occurs if the restart comes after Core stored the start and
before the checkpoint of that start is saved (the time of one checkpoint
write). The last checkpoint then waits, and the transport builds the
start again for the new driver. Seen with the real daemon when the
checkpoint write is held; not in a committed test. The usage accounting
of the server uses the turn id as its key, so there is no second count,
but the second start resets the "restored" mark of that turn. The
transcript does not read `turn.started`.
- **The older checkpoints can also save a cursor that is ahead of Core**
(measured for the checkpoint after `startTurn` of an ordinary turn).
Production accepts the gap in the sequence that follows, so this is not
a failure there. It is on `master` and is not changed here.
- **When the terminal of a provider-started turn was removed from the
stored events,** the transport builds no start for an event of that
turn, and the session stops on that event, as before. This needs a later
terminal of the same type with no start of its own (a turn that was lost
while it started) and a full window.
- **The last checkpoint of a wait that ends with no provider-started
turn has no terminal,** as the section above says. The change is only in
a run that waits, and no run waits before part 3.
- **A wait with a consumed disposition marker has no test with the real
daemon.** The state needs two restarts in one run, and the test helper
does one. Unit tests of the driver and of the consumer cover it.
- **A provider start that waits for `reconcile()`** waits as long as the
reads of `reconcile()` take. Each read ends at the command deadline of
the Runner (30 seconds by default) or when the transport closes.
- **`item.delta` events across a restart are not asserted.** A streamed
delta that the first controller received is not sent again. The
completed items are asserted.
- **The driver keeps a maximum of 8 snapshots at events** and drops each
one when the consumer asks for it or for a later one. If the consumer is
more than 8 such events behind, the checkpoint uses the current
snapshot, as before this change.
- **A second provider-started turn that also ends during the same
outage** is not tested with the real daemon: the fake gives each
terminal the same goal revision, and the Runner then holds the second
terminal until a controller is attached. The end of the reconnect time
of the Runner is not tested here.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, decided the designs, reviewed the diffs, and ran a
part of the verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as Claude Code subagents with
tool use (shell, file edits, local test runs), wrote the change and the
tests, reproduced the review findings with the real daemon, and verified
this slice alone.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 12:51:52 -08:00
DottaandClaude Opus 5.5 d4498ae7ac fix(runner): settle every recorded tool result and free sessions blocked by a closed turn's pending call (#21490)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call Paperclip tools through the Runner. The Runner records
each call and returns the controller's result to the provider (here
OpenCode)
> - In chat-eval cohort 13, a `list_workflows` result was recorded but
never reached OpenCode. OpenCode waited about 290 s, and the call stayed
pending
> - The next run's `run.attach` then failed with
`durable_pending_tool_calls`, so the person's follow-up question never
got an answer
> - This pull request fixes the result shape that caused the rejection,
makes the Runner settle every recorded result, and lets the next run
settle a call that a closed turn left pending
> - The benefit is that a tool result always reaches the agent, and a
stuck call can no longer block every later turn

## Linked Issues or Issue Description

**What happened?**
Chat-eval cohort 13 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `df1b12ad8e`). In
`cases.case-history` and `cases.create-case`, the CEO called
`list_documents` (or `list_projects`) and `list_workflows` (call id 5)
together. The server answered `GET /companies/{id}/workflows` with 200
in 584 ms. The OpenCode MCP bridge diagnostic for call 5 says `outcome:
"unanswered"`, `written: false`, `handlerMs: 289038` (290723 in
create-case). The sibling call was answered in about 1 s. The run ended
with `runner_settlement_incomplete`: `semanticTools.pending` held call 5
and its `semantic_tool.result` delivery command had status `failed`. The
next question started a run whose `run.attach` was refused: `run.attach
requires a settled Codex provider session: durable_pending_tool_calls`.
The run failed in 17 s, and the question was not answered.

Cause: `executeNativeWorkflowTool` returned the Runner API result
unchanged: `{ ok: true, status: 200, operationId: "GET
/api/companies/{companyId}/workflows", ... }`. Every other API-backed
tool renames that field to `apiOperationId`. The Rust tool bridge
(`semantic_response_value`) reads an object with a boolean `ok` and an
`operationId` as a semantic envelope that must name this tool and call.
The REST route did not match, so `apply_result` failed with `semantic
result envelope does not match its provider call`. Nothing was sent to
the provider, and the call stayed pending forever. The red test
reproduces this exact error. The parallel call is not the trigger: the
sibling call settled normally, and the OpenCode MCP bridge answered
other parallel pairs in the same runs. `list_workflows` first appears in
cohort 13, so cohorts 11 and 12 did not hit it. All four workflow tools
had this shape.

**Expected behavior**
Every recorded tool result reaches the provider. When a recorded result
cannot be delivered, the provider gets an error at once and the call
settles. A call that a closed turn left pending does not block the next
run.

**Steps to reproduce**
1. Run an OpenCode agent with Cases enabled on the durable Runner.
2. Ask a question that makes the agent call `list_workflows`.
3. See the call wait about 290 s. Then ask a follow-up question and see
`run.attach` fail with `durable_pending_tool_calls`.

**Paperclip version or commit**
`master` at `df1b12ad8e`.

## What Changed

- `server/src/services/native-runtime/native-workflow-tools.ts`:
workflow tool results name the route `apiOperationId`, like case tools
and `call_api`.
- `provider_bridge.rs`: add `settle_undeliverable_result`. When the
bridge rejects a recorded result for a call it still holds, the call
settles with a bounded, non-retryable runner error. Conflicting or
unknown results still fail as before.
- `provider_backend.rs` (`deliver_semantic_result`): use that
settlement, deliver the error to the provider, and record a
`semantic_tool_result_undeliverable` diagnostic with the call,
operation, reason and the rejected result's digest.
- `provider_bridge.rs` and `provider_backend.rs` (`attach_run`): add
`expire_closed_turn_calls`. When nothing is live (no active or ambiguous
turn, no provider request waiting), `run.attach` settles calls that a
closed turn left pending as an expired failure with an unknown outcome,
and records `semantic_tool_result_expired`. The provider stopped waiting
when its turn ended, and the new run's controller never delivers the old
run's results. A call that a live turn still waits on still blocks
attachment. This follows the existing `tool_call_interrupted_by_restart`
rule: never execute the operation again, and record an unknown outcome
when no result can ever arrive.
- Docs: `doc/run-log-events.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe both
diagnostics.

## Verification

- Red/green, Rust (`cd packages/paperclip-runner/runner && cargo test -p
paperclip-runner-core --test codex_provider --
a_rejected_recorded_result run_attach_expires`):
- Before:
`a_rejected_recorded_result_still_answers_the_provider_and_frees_the_session`
fails with `semantic result envelope does not match its provider call`
(the production error).
`run_attach_expires_a_call_left_pending_by_a_closed_turn` fails with
`run.attach requires a settled Codex provider session:
durable_pending_tool_calls` (the production refusal).
- After: both pass. The fake provider logs `tool-response:failure`, so
it got an answer at once. The next `run.attach` succeeds.
- Parallel calls (`cargo test -p paperclip-runner-core --lib --
provider_bridge`):
`a_rejected_result_in_a_parallel_pair_still_settles_its_own_call` sends
the cohort 13 pair (`list_documents` + `list_workflows`). The rejected
call settles, its sibling keeps its own result, and the next turn
starts. `only_a_closed_turn_expires_its_pending_calls` checks that an
open turn expires nothing.
- Upgrade and mid-run cutover:
`run_attach_expires_a_call_left_pending_by_a_closed_turn` saves state
with a pending call after the turn ends, stops that Runner, and attaches
with a new Runner process from the saved state.
`durable_backend_settles_pending_tools_when_recovery_finds_the_turn_ended`
(unchanged) still passes: a late result in the same run still settles as
`settled_after_turn`.
- Red/green, server (`cd server && npx vitest run
src/services/native-runtime/native-case-tools.integration.test.ts -t
workflows`): before, `expected { ok: true, status: 201, … } to not have
property "operationId"`; after, 2 passed.
- `cargo test -p paperclip-runner-core`: 643 lib tests pass. One lib
test (`fenced_receipt_output_keeps_prefix_when_stdout_never_closes`)
failed once with `EPERM` while signaling a process group under parallel
load. It passes alone 3 of 3 and does not touch this change. The
integration test binaries pass.
- Paid narrow run for `cases.case-history` and `cases.create-case`: see
the comment below.

## Risks

- Previous-release story: no persisted shape changes. State saved by the
previous Runner with a call pending after its turn closed carries
forward. The new Runner settles that call at the next `run.attach`, and
the session continues (tested from saved state with a new process). A
previous-release Runner that receives the corrected `apiOperationId`
result delivers it normally, because the result is no longer read as an
envelope.
- Behavior change: a call left by a closed turn is now settled at the
next run's attachment, not kept forever. Its outcome is recorded as
unknown, and the operation is never run again. A provider turn that is
still live keeps the old refusal (`durable_pending_tool_calls`,
`provider_side_effect_pending`), and the controller still never replaces
the session.
- A recorded result that the bridge rejects now becomes an error for the
agent, not an endless wait. The agent is told the action may have
completed and must check the current state before repeating it.
- Latency: nothing is added before turn start. The attach-time check
walks the in-memory pending calls (usually none) and adds no I/O beyond
the existing state save. The rejected-delivery path now answers in the
same command instead of after the provider's ~290 s client timeout, and
the follow-up run no longer fails after 17 s.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21398 (first release where `list_workflows` was called in this
eval), #19733 (the attach refusal), #20886 (earlier OpenCode delivery
fix).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 14:24:21 -06:00
Nicky LeachandClaude Opus 5.5 68d5c9c120 feat(runner): let a caller ask the transport for provider-started turns (#21607)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core runs Claude Code through Paperclip Runner. Claude can start a
shell command in the background and end its turn. When the command
finishes, Claude Code starts a cycle of its own to continue.
> - Today the run ends when the turn ends. The command stops, and the
run fails with `native_finalization_missing`.
> - The Runner can already keep Claude while a background shell is
running and give the cycle of Claude to the controller as a turn that
the provider started (#21423, #21316). Core can already wait for that
turn (#21453, #21454, #21455). No run does this yet, because no
controller asks the Runner for it.
> - The request goes through the transport of the Runner package. This
pull request gives the transport an option for it and the behavior that
the option needs.
> - No caller sets the option in this pull request. It changes no
production run.
> - The benefit is a small base for the two pull requests after it: the
recovery after a controller restart, and the switch that turns the
behavior on.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

#21456 turned the behavior on and had grown too large to review. It is
now three stacked pull requests:

1. **This pull request:** the transport can ask the Runner for
provider-started turns. No caller asks.
2. #21612: the recovery after a controller restart during a wait or
during a provider-started turn.
3. #21456: Core sets the option for native ACPX Claude runs. Only that
part changes what a production run does.

Refs #19912 (design), #21456.

**What existing behavior does this improve?**

None in production. It adds the support that parts 2 and 3 use.

**Subsystem affected**

Paperclip Runner package, the controller side:
`src/live/runnerd-codex-transport.ts`. ACPX Claude only, and only for a
caller that sets the new option.

**Current behavior**

- The transport has no way to ask the Runner for provider-started turns.

**Proposed behavior**

- The transport options have a new field, `providerStartedTurns`. The
transport uses it only for ACPX with the agent `claude`. In each other
case the field has no effect.
- With the option, the transport sends `providerStartedTurns: true` in
`turn.start`. It does not send it for a compaction turn.
- With the option, a tool call in a provider-started turn waits until
the transport gave each earlier event to the driver. Without this, a
`paperclip_finish` of that turn could reach the driver before the
`turn.started` of that turn, and the driver would refuse it.
- With the option, the transport gives a `run.terminal` of the Runner to
the driver as a marked internal notification (`paperclip/runTerminal`).
The driver uses it only during a wait, so a wait ends at once when the
Runner says that the run is over.
- With no option, the transport does the same as before this pull
request. Live sessions, evals, the CLI runner and the probes do not set
the option.

**Reason and benefit**

- The opt-in is a choice of the caller, not a property of the agent.
Only a consumer that can wait for a provider-started turn must ask for
one. `CapabilityLiveSession` completes a turn at the first
`turn/completed`, so it must not get the behavior because its agent is
Claude.

## What Changed

- `src/live/runnerd-codex-transport.ts` (31 lines added, 1 removed): the
option, the field in `turn.start`, the order of the tool call, and the
`paperclip/runTerminal` notification.
- `src/live/provider-turn-tool-order.ts` (new, 16 lines): finds the
source sequence of a tool call that belongs to a provider-started turn.
- `src/live/runnerd-acpx-provider-turn.test.ts` (new, 239 lines): 7
cases with the real `paperclip-runnerd`, the fake ACPX sidecar, and the
real consumer of Core.
- `src/live/live-session.test.ts` (12 lines added): a Claude live
session gives the transport no such option.
- `runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs` (test binary
only): the mode `provider-result-resolved`. In this mode the
provider-started turn calls `paperclip_finish` and waits for the answer
of Core through the Runner (`tool.resolve`), as the real sidecar does.
The mode `provider-result` is not changed.

## State saved by the previous release

- No persisted state changes in this part.
- **A Runner that does not know the field** ignores it.
- **Rollback of the Runner package:** the previous transport does not
send the field, so the Runner does not wait.

## Verification

At the head of this branch (`2f2cb14b9c`), with nothing from parts 2 and
3:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json`, `pnpm run typecheck:typescript` and `pnpm run
build:typescript` in `packages/paperclip-runner`; `pnpm typecheck` and
`node scripts/verify-runner-vendor-dependencies.mjs` in `server`; `node
--test scripts/__tests__/runner-release-crate-cache.test.mjs`; `node
--test test/acpx-sidecar-contract.test.mjs` (18 passed): all passed.
- `src/live/runnerd-acpx-provider-turn.test.ts`, 3 runs, 7 of 7 passed
in each. The cases:
- background work, then the turn of Claude calls `paperclip_finish`;
Core answers; the Runner gives the answer to the sidecar with exactly
one `tool.resolve`; the turn completes; the run finalizes with one
`run.result.accepted`;
- a Claude run with no option from the caller takes the path of today:
no `providerStartedTurns` in `turn.start`, no `paperclip/runTerminal`,
no wait, and no `session.provider_turn` to the sidecar;
- the limit ends the wait, the run takes the path of today, and the
provider processes had exited before the cleanup of the test;
- the provider process is lost during the wait, and the run settles in
seconds;
  - the sidecar exits during the wait, and the run settles in seconds;
  - a turn with no background work does not wait;
- an ACPX agent that is not Claude is never opted in, also when the
caller sets the option.
- Each case asserts the result of the close of the session: clean in 6
cases, and a named failure in the case where the sidecar is killed. Each
clean case also asserts that there is no
`native_session_settlement_incomplete` diagnostic and no "conflicting
duplicate tool result" in the durable state.
- `src/live/live-session.test.ts`,
`src/live/runnerd-codex-transport.test.ts`,
`src/live/runnerd-acpx-provider-turn-wait.test.ts`,
`src/native-session-runtime.provider-turn.test.ts`: 434 passed.
- Rust, because the fake sidecar changed: `cargo fmt --check`, `cargo
build`, the `acpx_*` suites (244 passed) and the `acpx` unit tests (102
passed).
- Mutation checks, run on the full change before the split: with the
option ignored, 6 of the 7 assertions of the case with no option fail;
with the old mode of the fake, the case with the result fails at the
close of the session.

Not run: the full suite. CI supplies that coverage.

## Risks

- **No production effect.** No caller sets the option. Core sets it in
part 3.
- **The transport is shared by each provider.** The three new behaviors
are behind one condition: the option, ACPX, and the agent `claude`. With
the condition false, the code path is the path of today.
- **The tool-call ordering uses a 10 ms poll** while earlier events are
not delivered, only in a provider-started turn.
- **A controller restart during a wait or during a provider-started turn
is not handled in this part.** Part 2 handles it. Part 3 must not merge
before part 2.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 08:16:24 -08:00
Nicky LeachandClaude Opus 5.5 b1f608296e feat(runner): keep a run open for a limited time while the Runner waits for the provider's own turn (#21454)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core runs Claude Code through Paperclip Runner. Claude can start a
shell command in the background and end its turn. When the command
finishes, Claude Code starts a cycle of its own to continue.
> - Today the run ends when the turn ends. The command stops, and the
run fails with `native_finalization_missing`.
> - The Runner part of this work is complete in #21316 and the pull
requests below it: when a controller sends `providerStartedTurns` on
`turn.start`, the Runner keeps Claude while a background shell is
running, and gives the cycle of Claude to the controller as a turn that
the provider started.
> - `consumeTurn` reads the events of one turn and returns at the
terminal. It already continues past a terminal for a Codex goal.
> - This pull request makes it continue past a terminal that waits for a
turn of the provider, and it limits that wait.
> - The benefit is that a run continues in the turn that Claude starts,
and a wait always ends.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one of four parts on the Core side. They are
stacked on #21316:

1. The driver admits a turn that the provider started.
2. The consumer waits for that turn, for a limited time.
3. The server ends the wait when a person comments, and keeps the clock
across a restart.
4. The transport sends the opt-in. Only this part changes what a
production run does.

This is part 2. Its base branch is `feat/core-provider-turn-driver`.
Refs #19912 (design).

**What existing behavior does this improve?**

Nothing in production yet. What the consumer of a native session does
after a turn that ended with no result while the Runner waits.

**Subsystem affected**

Paperclip Runner package, the controller side:
`src/native-session-runtime.ts` and a new module
`src/native-provider-turn-wait.ts`.

**Current behavior**

- `consumeTurn` returns at the first terminal. With no result, the run
is finalized as `native_finalization_missing`.

**Proposed behavior**

- At a terminal with `awaitingProviderTurn` and no result, `consumeTurn`
continues to read. It consumes the provider-started turn as the next
turn of the same run. A turn that reports a result finalizes as a normal
run.
- The wait has a limit of 10 minutes from the first terminal that waits.
A later terminal that waits does not start the clock again. The limit is
never later than 5 seconds before the timeout of the run.
- The limit and an early end stop the wait only. They never interrupt a
provider-started turn that is running; that turn stays under the timeout
of the run. If that turn ends and the Runner waits again, the run takes
the path of today at once.
- When the wait ends with no turn active, the consumer returns the last
terminal, and the existing code for a run with no result decides
(`resolveMissingResult`, then `native_finalization_missing`). No new
failure code.
- The wait ends at once when the driver gives the acknowledgement of
part 1. This covers Stop, the limit, and a Runner that says the wait is
over.
- `executeNativeSession` gets one new option, `providerTurnWait`: the
start time of a wait that an earlier controller process saw, and a hook
that ends the wait early.
- When the wait ends a turn, the consumer first waits for the receipts
of the text deltas that are committed in batches (#21460), as each other
return path of `consumeTurn` does. Without this, a delta that arrived
during the wait was not in the event count of the turn, and a delta that
failed to commit was not reported.
- When the interrupt that ends the wait is not available or fails, the
limit forces the end of the wait. If that forced end starts a new turn
(the continuation that a restart owes), the consumer continues to read
that turn. The forced-end signal is used one time only, and the read
that is open at that moment is kept until it gets its event. Before this
fix, the consumer returned the terminal of the old turn and lost the
first event of the new turn, and the run could end as
`native_finalization_missing`.
- If the controller restarts during the wait, the run continues to wait.
When the saved checkpoint still shows the first turn as active, the
recovered session gives the waiting terminal again, and the wait starts
again with the start time that was recorded. When the checkpoint was
saved after the waiting terminal, the consumer finds that terminal in
the journal, starts the wait again, and does not send the
disposition-only recovery prompt. The consumer does this only if two
conditions are true: the recovered session has the waiting state
(`awaitingProviderTurn`, saved by the driver since #21453), and the
terminal in the journal has the marker. If one condition is false, the
run uses the recovery path of today. When a wait that started again ends
with no turn, the existing code for a run with no result decides, as for
each other wait.

**Reason and benefit**

- The 10 minutes are the maximum time of a foreground command in the
pinned Claude Code.
- A limit on the wait, not on the work, means that a long test run in
the turn of Claude is not cut.

## What Changed

- `src/native-provider-turn-wait.ts` (new, 129 lines) and
`src/native-session-runtime.ts` (58 lines added, 4 removed),
`src/index.ts` (1 export).
- `src/native-session-runtime.provider-turn.test.ts`: written again for
the new behavior (557 lines added, 61 removed).

## State saved by the previous release

- This part adds no persisted state. The wait is in memory. Part 3 reads
its start time from an event row that is already saved.

## Verification

At the head of this branch:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json` and `pnpm run build:typescript` in
`packages/paperclip-runner`; `node
scripts/verify-runner-vendor-dependencies.mjs` and `pnpm typecheck` in
`server`; `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`: all passed.
- `src/native-session-runtime.provider-turn.test.ts`: 30 tests. They
cover: a controller restart with the first turn still active in the
checkpoint (the wait continues; a provider-started turn settles its
result; with no turn the limit ends the wait on the recorded clock); a
controller restart with the checkpoint saved after the waiting terminal
(the same two results, no recovery prompt, no event of the journal
appended again); a limit that passed before the restart ends the wait at
once, with and without an interrupt; the recovery prompt of today when
the terminal in the journal has no marker, when the journal has no
terminal, and when the recovered session has no waiting state; a second
restart during the wait; a forced end that starts a new turn settles the
result of the new turn, with each event of that turn appended one time
and in order; an event that reaches the open read after the forced end
is appended exactly one time; a delta that arrives during the wait is
counted when the limit ends the turn (the test fails when the new wait
for receipts is removed); continue and settle the result of the
provider-started turn; a second wait does not start the clock again; the
limit with no turn active; the limit and a comment during a running turn
(not interrupted); a running turn that ends and waits again after the
limit; Stop; a `run.terminal` that the Runner held; the Runner stream
ends during the wait; the limit under the timeout of the run; a start
time from an earlier controller; a plan that a provider-started turn
proposes settles as the plan review wait.
- `src/native-session-runtime.test.ts`, not changed by this PR: 297
passed.
- Mutation checks, each made tests fail: no continue at a waiting
terminal; no limit; no early end; no clock from an earlier controller;
the clock starts again at each wait; a running turn is interrupted at
the limit.
- Mutation checks for the forced end, each made the two new tests fail:
the forced-end signal used on each loop (the run ends as
`native_finalization_missing`); a new read started after the forced end
(the first event of the new turn is lost).
- Mutation checks for the restart, each made named tests fail: the
recovery prompt not held back; the wait not started again in
`consumeTurn`; the wait started again on the session state alone (the
test with no marker in the journal fails).

Not run: the full suite. CI supplies that coverage.

## Risks

- **No production effect.** No terminal has the marker before part 4.
- **`consumeTurn` runs for each native turn of each agent.** It makes
one small object for each turn and reads one more payload field at a
terminal.
- **If the interrupt that ends the wait is not available or fails,** the
limit forces the end: the consumer returns the last terminal, or
continues to read when the forced end starts a new turn. This is a
fallback path. On a forced return, one read of the event stream stays
open until the session closes, as before this PR.
- **A wait that starts again after a restart holds back the recovery
prompt.** This is the one change to the recovery flow. It needs the two
conditions above, and both exist only for a run that asked for
provider-started turns.
- **Known limit.** The driver does not clear its waiting state when the
consumer ends a wait with an interrupt. A restart in the short time
after that can start the wait again; the recorded clock and the comment
check then end it.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 05:59:42 +00:00
Nicky LeachandClaude Fable 5.2 82ce8e64ba feat(runner): let the session driver admit a turn that the provider started (#21453)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core runs Claude Code through Paperclip Runner. Claude can start a
shell command in the background and end its turn. When the command
finishes, Claude Code starts a cycle of its own to continue.
> - Today the run ends when the turn ends. The command stops, and the
run fails with `native_finalization_missing`.
> - The Runner part of this work is complete in #21316 and the pull
requests below it: when a controller sends `providerStartedTurns` on
`turn.start`, the Runner keeps Claude while a background shell is
running, and gives the cycle of Claude to the controller as a turn that
the provider started.
> - Core reads Runner events through the Codex app-server driver, also
for ACPX agents. The driver refuses a turn that the controller did not
start, except a goal turn of Codex.
> - This pull request lets the driver accept a turn that the provider
started, only directly after a turn that the Runner marked as waiting.
> - The benefit is that the next parts can use that turn. Nothing
reaches this code until the last part sends the opt-in.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one of four parts on the Core side. They are
stacked on #21316:

1. The driver admits a turn that the provider started.
2. The consumer waits for that turn, for a limited time.
3. The server ends the wait when a person comments, and keeps the clock
across a restart.
4. The transport sends the opt-in. Only this part changes what a
production run does.

This is part 1. Its base branch is
`feat/adopt-claude-provider-started-turn`. Refs #19912 (design).

**What existing behavior does this improve?**

Nothing in production yet. What the driver does with a turn that the
provider started.

**Subsystem affected**

Paperclip Runner package, the controller side: `src/drivers/codex` (the
driver that Core uses to read a native session).

**Current behavior**

- A `turn/started` that the controller did not ask for fails the session
with `turn_binding_mismatch`, except for an active Codex goal.
- An interrupt with no active turn fails with "already terminal".

**Proposed behavior**

- A turn terminal that the Runner marked with `awaitingProviderTurn` and
that has no result puts the driver in a waiting state. The driver event
`turn.completed` carries the marker.
- In that state, a `turn/started` with `origin: "provider"` is admitted
the same way as a goal turn, and the driver event carries `origin:
"provider"`. In each other state it still fails closed.
- In that state with no active turn, an interrupt gives an
acknowledgement item with the status `provider_turn_wait_ended`. It
sends nothing to the Runner. The driver also gives that acknowledgement
when it gets the internal `paperclip/runTerminal` notification with a
private marker; a notification without the marker does nothing.
- An interrupt for a provider-started turn that is running is sent to
the Runner only when it names that turn.
- The driver saves the waiting state in the persisted session
(`awaitingProviderTurn`, written only during a wait). After a controller
restart, the restored session still admits the provider-started turn and
still acknowledges an interrupt during the wait. The driver restores the
state only when the session has no active turn, no stored result and at
least one finished turn, so a new run cannot take the wait of the run
before it.
- When a restart finds that the saved active turn is finished, the turn
read of the transport and `reconcile()` of the driver carry the same
marker as the live `turn/completed` notification.
`runnerd-codex-transport.ts` copies `awaitingProviderTurn: true` from
the saved `turn.completed` event to the turn object of a read.
- The driver ends the waiting state when the Runner reports the run
terminal.

**Reason and benefit**

The consumer in the next part needs these three signals: a marked
terminal, an admitted turn, and a clear end of the wait.

## What Changed

- `codex-session-terminal.ts`, `codex-session-notifications.ts`,
`codex-session-state.ts`, `codex-harness-session.ts`,
`codex-driver-values.ts`: 37 lines added, 4 removed.
- `codex-app-server-driver.provider-turn.test.ts` (new, 232 lines).
- `contracts/harness-driver.ts`, `contracts/native-session-backend.ts`,
`backends/harness-driver-backend.ts`, `codex-app-server-driver-impl.ts`:
the optional field in the two persisted shapes and its copy in each
direction.
- `live/runnerd-codex-transport.ts`: the marker on a recovered turn
terminal. The loop that builds the earlier terminals of a `thread/read`
is now the function `runnerdPriorTurnTerminals`, with the same logic, so
that a test can call it.

## State saved by the previous release

- One new optional field in the persisted driver session:
`awaitingProviderTurn`. A snapshot outside a wait has the shape of the
previous release.
- **State from the previous release, read by this release:** the field
is absent, so the driver does not wait. This is the behavior of today.
- **State from this release, read by the previous release:** the
previous reader copies the fields that it knows and ignores this one. No
reader rejects it, and the integrity digest covers the stored object as
written.

## Verification

At the head of this branch:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json` and `pnpm run build:typescript` in
`packages/paperclip-runner`; `node
scripts/verify-runner-vendor-dependencies.mjs` and `pnpm typecheck` in
`server`; `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`: all passed.
- 6 driver tests: a provider-started turn after a marked terminal is
admitted and can report a result; a provider-started turn with no marked
terminal fails closed; an interrupt during the wait is acknowledged; a
running provider-started turn is not interrupted by the end of the wait;
the marked notification ends the wait and a notification with no marker
does not; a normal terminal is not changed.
- Restart tests: a session restored from a snapshot with the waiting
state admits the provider-started turn and settles its result; a
snapshot outside a wait has no new field; a restored session with no
waiting state refuses a provider-started turn as today; an interrupt
during a restored wait is acknowledged. A saved active turn that the
read reports finished with the marker gives a `turn.completed` with
`awaitingProviderTurn: true`; with no marker it is an ordinary terminal.
- `harness-driver-backend.test.ts`: the field goes through a serialized
checkpoint only during a wait. `runnerd-codex-transport.test.ts` (324
passed): the marker is on a recovered terminal only for a
`turn.completed` with a boolean `true`.
- `src/native-session-runtime.test.ts`: 297 passed, with the real
`paperclip-runnerd`.
- Mutation checks for the restart work, each made a named test fail: no
restore of the state; no copy in the backend; no copy in the transport;
no marker argument in `reconcile()`; no end of the wait at the run
terminal.
- Mutation checks: the acknowledgement branch removed, the admission
removed, and the marker check removed each made a test fail.

Not run: the full suite. CI supplies that coverage.

## Risks

- **No production effect.** The Runner sends the marker only after the
opt-in of part 4.
- **The admission rule is shared with Codex goal turns.** The existing
goal tests pass with no change.
- **Known limit after a controller restart.** If the Runner already runs
a provider-started turn and the saved active turn is the turn before it,
recovery fails closed, as today for each unexpected active turn.
- **A controller turn that starts during a wait** clears the waiting
state when its `turn/started` arrives, not when it is sent. A snapshot
in that short time can hold an active turn and the field together; the
restore rule drops the field.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 20:45:49 -08:00
Nicky LeachandClaude Fable 5.2 f7b1651799 feat(acpx-sidecar): adopt Claude's own cycle as a provider-started turn when the Runner arms it (#21316)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner runs Claude Code through the ACPX sidecar.
> - Claude can start a shell command in the background and end its turn.
When the command finishes, Claude Code starts a cycle of its own to
continue. Today the Runner stops Claude at the end of the turn, so the
command stops and the run fails with `native_finalization_missing`.
> - A stack of pull requests lets the Runner accept that cycle as a new
turn of the same run, when a controller asks for it.
> - The parts before this one give the adapter an ordered stream of
those cycles (#21421), declare the contract (#21422), and make
`paperclip-runnerd` wait for such a turn (#21423).
> - This pull request is the last part in the Runner. The sidecar reads
the stream, and gives a cycle to `paperclip-runnerd` as a
provider-started turn only when `paperclip-runnerd` armed it.
> - The benefit is that a later change in Core can let a run continue
when background work finishes. Until Core sends the opt-in, a production
run does the same as today.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request was first opened with the full change (49 files). It
is now split. The parts, in stack order after #21311:

1. #21419: sidecar preparation (a refactor).
2. #21420: converter for raw ACP session updates.
3. #21421: cycle stream at the adapter.
4. #21422: contract for provider-started turns.
5. #21423: `paperclip-runnerd` waits for a provider-started turn.
6. This pull request: the sidecar adopts a cycle.

Its base branch is `feat/runnerd-await-provider-turn` (#21423). The code
at the head of this branch is the same as before the split; only the
base changed. Review comments made before the split are about code that
is now in the parts above. Refs #19912 (design).

**What existing behavior does this improve?**

What the ACPX sidecar does with a cycle that Claude starts by itself.

**Subsystem affected**

Paperclip Runner: `src/cli/acpx-runtime-sidecar.ts`. Claude only.

**Current behavior**

- The sidecar does not read the cycle stream of the adapter (#21421), so
each cycle is declined and the background state becomes unknown.
- The sidecar does not know the command `session.provider_turn`, so
`paperclip-runnerd` (#21423) never waits.

**Proposed behavior**

- The command `session.provider_turn` with `armed: true` arms the
sidecar for one turn. The sidecar accepts it only for the terminal that
reported a running background shell, with no active turn and a provider
that is alive. One flag holds the arming. It is cleared at each turn
start and turn end, on suspend and close, and when the provider is lost.
- One loop, `consumeProviderCycles`, reads the stream. For each cycle:
- if the sidecar is armed and no turn is active, the sidecar adopts the
cycle: it emits `runtime.turn_started` with `{ "origin": "provider" }`,
then the events, the usage and the terminal of the cycle, as for a
normal turn;
- if the cycle cannot be decided yet (the prompt turn is still settling,
or `paperclip-runnerd` did not arm yet), the sidecar holds it for a
maximum of 5 seconds, and then adopts or drops it;
- in each other case the sidecar drops the cycle. The stream ends, and
the background state becomes unknown, so `paperclip-runnerd` restarts
the provider at the end of the turn as today.
- A Paperclip tool call or a permission request of a held cycle waits
for the decision of its own cycle. It is never bound to the prompt turn
or to a different cycle.
- If an adopted cycle proposes a plan (`ExitPlanMode`), the sidecar
publishes the same `plan.proposed` event as for a controller turn, and
the turn ends with the same terminal: completed, with no
`backgroundTasks`, so `paperclip-runnerd` does not wait again. A
controller turn is not changed.
- If the provider exits while the sidecar is armed with no turn active,
the sidecar sends a `runtime.diagnostic` with the code
`ACPX_PROVIDER_PROCESS_LOST`, and `paperclip-runnerd` ends the wait.
- `turn.cancel` for a provider-started turn cancels the cycle and closes
the provider.
- With no arming, nothing is shown. No controller sends the opt-in yet,
so `paperclip-runnerd` never arms.

**Reason and benefit**

- Only `paperclip-runnerd` decides that a cycle becomes a turn. The
sidecar cannot start a turn by itself.
- One reader of one ordered stream means that two cycles cannot overlap.

## What Changed

- `src/cli/acpx-runtime-sidecar.ts` (127 lines added): the command, the
arming flag, `consumeProviderCycles`, `adoptProviderCycle`,
`pumpProviderTurn`, and the wait of `waitForTool` for a cycle that is
not decided.
- `src/cli/acpx-runtime-sidecar.provider-turn.test.ts` (new, 320 lines)
and two fake hosts in `src/test-support` (227 lines). One host follows a
script. The second runs the real cycle stream and the real background
task tracker.
- `src/cli/acpx-runtime-sidecar.test.ts` (2 lines): one more parameter
for a test harness.

No schema, no Rust and no persisted state changes in this part.

## Verification

At the head of this branch, from `packages/paperclip-runner`:

- `pnpm run typecheck:typescript`, `npx tsc --noEmit -p
tsconfig.test.json` and `pnpm run build:typescript`: no errors. The
server check `node scripts/verify-runner-vendor-dependencies.mjs`, the
cache guard and the contract tests: passed.
- `npx vitest run src/cli src/drivers/acpx
src/live/runnerd-acpx-provider-turn-wait.test.ts
src/drivers/runner-tool-bridge.test.ts
src/native-session-runtime.provider-turn.test.ts`: `128 passed (128)`
files, `2147 passed`, 9 skipped.
- `cargo test -p paperclip-runner-core --lib` and the `acpx_*` suites:
all passed. No Rust changed in this part.
- The sidecar tests of this part: a cycle that starts before the arming
command, or while the prompt turn settles, is held and then adopted with
its tool call; a cycle is never shown when `paperclip-runnerd` did not
arm, disarmed, armed for a different terminal, armed for a quiet
terminal, or armed after the provider exited; two cycles in sequence
with `{ live: 2 }`, then `{ live: 1 }`, then an unknown state; cancel; a
lost provider; Codex gets no change.
- Mutation checks, each made the named tests fail: the sidecar arms with
no matching terminal; no hold; the old hold condition that drops a cycle
while the prompt turn settles; no wait of a tool call for the decision.
- Live run with a real Claude account, not in CI, with the real ACP
runtime and the real Anthropic API (model `claude-sonnet-5-5`), at the
adapter level: one background shell gave `{ live: 1 }`, one cycle with
its own usage, then an unknown state. Two shells of 8 s and 28 s gave `{
live: 2 }`, `{ live: 1 }`, then unknown. All Paperclip tool calls were
admitted, and none was refused.

Not run: a live run through the real sidecar process,
`paperclip-runnerd` and Core (that path needs an API key, not a local
sign-in); Linux; `pnpm verify --full`. CI supplies the broad coverage.

## Risks

- **The feature is off.** No controller sends the opt-in, so the sidecar
is never armed. The new code that runs today is the loop that reads the
stream: with no arming it drops each cycle, and the background state
becomes unknown as before.
- **A follow-up for a command that finishes while Claude reports on an
earlier one is lost.** The Runner waits again only while a background
command is still running. The run ends as it does today. The reason is
in #21421.
- **A cycle that starts in the last second of a prompt turn** can have
its first messages in the prompt turn and the rest in the
provider-started turn. Tool calls stay safe: the admission rule of
#21278 refuses a call that the active turn did not announce.
- **The 5 second hold** is not tested under load. A cycle that is held
longer is dropped, and the run takes the path of today.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 19:50:22 -08:00
Nicky LeachandClaude Opus 5.5 4a78124a3e feat(runner): wait for a provider-started Claude turn when the controller asks (#21423)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner runs Claude Code through the ACPX sidecar.
> - Claude can start a shell command in the background and end its turn.
When the command finishes, Claude Code starts a cycle of its own to
continue. Today the Runner stops Claude at the end of the turn, so the
command stops and the run fails with `native_finalization_missing`.
> - A stack of pull requests lets the Runner accept that cycle as a new
turn of the same run, when a controller asks for it.
> - `paperclip-runnerd` ends a run when a turn ends. For a Claude turn
that ended with no result while a background shell is running, this
stops the shell and the run fails.
> - #21278 binds each tool call to its turn, and #21293 reports how many
background shells are running at the end of a turn. Part 4 declared the
contract.
> - This pull request makes `paperclip-runnerd` wait for a turn that
Claude starts by itself, when the controller asks for it, and admit that
turn into the same run.
> - The benefit is that the Runner has one place that decides to wait
and one place that ends a wait.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one part of a change that was first opened as one
large pull request (#21316). The parts, in stack order after #21311:

1. #21419: sidecar preparation (a refactor).
2. #21420: converter for raw ACP session updates.
3. #21421: cycle stream at the adapter.
4. #21422: contract for provider-started turns.
5. #21423: `paperclip-runnerd` waits for a provider-started turn.
6. #21316: the sidecar adopts a cycle.

This is part 5. Its base branch is `feat/prp-provider-turn-contract`.
After each part, a production run does the same as today, because no
controller sends the opt-in. Refs #19912 (design).

**What existing behavior does this improve?**

What `paperclip-runnerd` does after a Claude turn ends with no result
while a background shell is running. Nothing changes until a controller
sends the opt-in.

**Subsystem affected**

Paperclip Runner: the ACPX provider backend in `runner-core` (Rust).
Claude only.

**Current behavior**

- After each completed turn the Runner emits `run.terminal`. A turn with
a running background shell gets the provider restart, and the shell
stops.

**Proposed behavior**

Nothing changes unless the controller sends `providerStartedTurns: true`
in the payload of `turn.start`. No controller sends it yet.

With that field:

- The Runner keeps the provider and waits when all of these are true:
the agent is Claude; the turn completed; the run has no accepted
`paperclip_finish` or `paperclip_block` result; the terminal event
reported `backgroundTasks` with `live` above zero; each condition of
#21293 for a kept process is true; the sidecar accepted
`session.provider_turn` with `armed: true`.
- `turn.completed` then has `awaitingProviderTurn: true`, and no
`run.terminal` follows. The Runner holds that `run.terminal`.
- When the sidecar reports a turn that the provider started, the Runner
admits it only during a wait and publishes `turn.started` with `origin:
"provider"`. In each other case that report fails the session closed.
- When a wait ends with no turn active, the Runner publishes the held
`run.terminal`. This applies when the provider process exits, the
sidecar exits, the controller cancels or stops the run, the session is
closed or suspended, and after a Runner restart.
- The held `run.terminal` and the removal of the wait are saved in one
write, and the controller gets the event from the next poll. A crash
after a cancel or a stop cannot lose it, and it is never sent two times.
- A controller `turn.start` during a wait replaces the provider first,
as before, and the held event is dropped.
- A `run.attach` from a different run during a wait ends the wait when
the attach succeeds. The Runner then discards the held `run.terminal`,
because that event is of the old run and must not end the new run. This
is the same rule that a Runner restart applies. The wait is removed only
in the state that the attach saves for the new run. If the attach fails,
for example when the old session does not stop, the old run keeps the
wait and gets its held `run.terminal` one time, from the next poll or
from a cancel or a stop. A `run.attach` from the same run publishes the
held event to that run.
- The Runner waits at most 5 seconds for the sidecar to answer a request
to arm or to disarm a provider-started turn. This is the limit that turn
cancellation already uses. Before, the limit was the 30 seconds of an
ordinary request, and a cancel, a stop or a shutdown had to wait behind
it. No answer is handled as a lost sidecar: the session fails closed,
and the Runner still publishes the held `run.terminal` one time.
- The sidecar of this branch does not know `session.provider_turn`, so
the arming request fails and the Runner does not wait. The behavior
comes with part 6.

**Reason and benefit**

- The wait is one entry in the saved session state. One rule ends it at
each save, so each new way to leave the open session ends the wait with
no new code.
- A wait must not outlive the provider or the sidecar, and a cancel must
stop it.

## What Changed

- `acpx_provider_turn_wait.rs` (new, 230 lines with its unit tests):
`provider_turn_wait { turn_id, held_run_terminal }` in the saved state;
the rule that ends a wait at save; `settle_turn_events`, which decides
at one place to arm, to create the wait and to put
`awaitingProviderTurn` on `turn.completed`.
- `acpx_provider_turn.rs` (new, 119 lines): the offer of a turn at its
terminal, the arming request, and `admit_provider_turn`.
- `acpx_provider_session.rs`, `acpx_provider_backend.rs`,
`acpx_provider_state.rs`, `provider_events.rs`,
`acpx_park_preparation.rs`: the opt-in from `turn.start`; the record
that the run has a result; the provider-started turn in the state and in
the PRP events; cancel and stop with no turn; no turn-end restart and no
park during a wait.
- `acpx_sidecar_transport.rs` (11 lines): during a wait, a poll with no
event also checks that the sidecar process is alive. A child process can
keep the output stream of a dead sidecar open.
- `bin/fake-acpx-sidecar.rs` (232 lines): provider-started turns and
failures for tests.
- Tests (1,164 lines): `tests/acpx_quiet_turn_continuation.rs`,
`tests/support/acpx_quiet_turn_restart.rs`,
`src/live/runnerd-acpx-provider-turn-wait.test.ts` (the real daemon),
`src/native-session-runtime.provider-turn.test.ts` (the controller of
today).
- `runner/DURABLE_TRANSPORT.md`: the rules.

## State saved by the previous release

- One new optional field in the saved ACPX session state:
`providerTurnWait`. It is written only during a wait. In each other
state the saved file has the shape of the previous release.
- **State from the previous release, read by this release:** the field
is absent, so there is no wait. A unit test covers this.
- **State from this release, read by the previous release (a
rollback):** outside a wait, the file loads as before. During a wait,
the previous Runner refuses the file with its "ACPX provider state is
malformed" error, because its loader refuses each unknown field. Only a
run that waits at the moment of a rollback is in this case, and no run
waits before a controller sends the opt-in.
- **Runner restart during a wait:** the wait is restored. The provider
is a new process, so the wait is over, and the controller receives the
held `run.terminal` one time. A test with the real daemon covers this.
- **Controller of today or of the previous release:** `turn.start` has
no opt-in. A test runs a turn with a running shell and a turn with an
unknown background state, and compares the events, the commands after
the turn, the restart, and the park state before and after the restart.
They are the same.

## Verification

At the head of this branch, from `packages/paperclip-runner`:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run
build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs
test/protocol-contract.test.mjs`, the server check `node
scripts/verify-runner-vendor-dependencies.mjs`, `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt
--check` and `cargo build -p paperclip-runner-core --bins` (no new
warning): all passed.
- `cargo test -p paperclip-runner-core --lib`: 644 passed, 4 ignored.
All 17 `acpx_*` suites passed.
- `src/live/runnerd-acpx-provider-turn-wait.test.ts`, 5 cases with the
real `paperclip-runnerd` and the fake sidecar through PRP: a wait ends
with exactly one `run.terminal` when the sidecar exits while a child
process keeps its output open, when the provider is lost, when the
controller cancels, and after a Runner restart; with no opt-in,
`run.terminal` follows the turn and the sidecar is never armed.
- Rust tests: wait and admit; no wait with a result; no wait with no
opt-in; an active goal neither marks nor arms a wait; a sidecar that
refuses to arm; a provider-started turn that the Runner did not wait for
fails the session closed; cancel and stop during a wait; provider and
sidecar loss; restart; the saved shape in both directions.
- `src/native-session-runtime.provider-turn.test.ts`: the real
`consumeTurn` of today gets the event sequence with no opt-in and
finalizes it as `native_finalization_missing`, as today.
- Attach during a wait, in `tests/support/acpx_quiet_turn_restart.rs`: a
different run attaches after the snapshot reports `warmAttachReady:
true`; the new run gets no `run.terminal`, the saved state has no wait,
and the old session is closed. A same-run attach gives exactly one
`run.terminal`. An attach that fails when the old sidecar is dead leaves
the wait with the old run, and the old run then gets exactly one
`run.terminal` from a poll, `run.cancel`, `turn.interrupt` or
`turn.stop`. Mutation checks: with no fix, the old `run.terminal` is in
the pending events of the new run; with the wait ended before the attach
is complete, the old run gets no `run.terminal` after a failed attach.
- No answer from the sidecar, in
`tests/support/acpx_quiet_turn_restart.rs`: the fake sidecar does not
answer the disarm request; `run.cancel` and `turn.interrupt` each return
`already_settled` in less than 10 seconds, exactly one `run.terminal`
reaches the run, `session.close` completes in less than 2 seconds with
no second `run.terminal`, and the provider process is gone. With no
answer to the arm request, the turn ends with its `run.terminal` and
with no wait. With the limit removed, each test fails at 30 seconds.
- Mutation checks, each made the named tests fail: the rule that ends a
wait at save; no `run.terminal` after a restart; arming with no opt-in;
the marker set with no wait; the sidecar process check removed.

Not run: a rollback with the binary of the previous release; a run with
the opt-in and a sidecar that answers the arming request with an error
(the code reads each error as not armed); Linux; the full suite. CI
supplies the broad coverage.

## Risks

- **The feature is off.** No controller sends the opt-in.
- **A rollback of the Runner during a wait** makes the previous Runner
refuse that saved session. See "State saved by the previous release".
- **The wait has no time limit in the Runner.** The controller must
close, suspend or cancel. Core will own that limit.
- **Park is refused during a wait.** The controller must end the wait
before it parks.
- **A new run can attach during a wait.** The attach is not blocked,
because the Runner has no time limit on the wait and a new run can send
only `run.attach`. After an attach that succeeds, the old run gets no
`run.terminal` from the Runner; the controller must not depend on that
event. The request that disarms a sidecar that the attach keeps has no
test, because no test can keep a session across an attach during a wait.
- **After a sidecar that does not answer, `runner.suspend` returns an
error at once** ("provider lifetime cleanup is not yet proven"), and
`session.close` completes. This is the behavior of today for each lost
sidecar; this PR does not change it.
- **The shared code that changed** runs for each Claude turn: the record
of a run result, and the settle step at each turn end.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:21:10 -08:00
Devin FoleyandPaperclip 493519fc88 fix(runner): poll a live ACPX turn without waiting out the idle read (#21525)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runner (runnerd) relays provider events to the server and
waits for cumulative ACKs before it polls the provider again.
> - After a control frame, runnerd uses a 1 ms queued read only when the
executor reports a live provider poll. Otherwise it waits up to the 250
ms runtime read timeout, and any frame that arrives in that time skips
the poll.
> - Only the Codex facade reports a live poll. ACPX runs (Claude, Codex
through ACPX and others) always wait the full 250 ms after the last ACK
of every batch, and steady control traffic starves their provider polls.
> - This pull request lets an ACPX executor report a live poll while its
restored session has an active turn.
> - The benefit is that ACPX output is relayed without the 250 ms idle
wait, and tool-heavy turns no longer starve provider polls.

## Linked Issues or Issue Description

**What happened?**
A routine run started several subagents that made about 190 tool calls
in about 10 minutes. On the server, consecutive runner events arrived at
least 576 ms apart (median about 1.5 seconds), one event at a time,
while the time from each event's `emittedAt` to its insert was 0 to 0.1
seconds. The runner fell several minutes behind the provider. The
backlog then overflowed the transport's event buffer (fixed separately
in #21524) and the turn failed.

**Expected behavior**
A live ACPX turn is polled again as soon as its outstanding events are
acknowledged and no control frame is queued, the same as a Codex facade
turn.

**Steps to reproduce**
1. Run an ACPX turn that produces events while the controller sends
commands and ACKs.
2. After each ACK, runnerd calls `receive_json`, which waits up to 250
ms before it polls the provider. A frame that arrives in that time skips
the poll.

**Relevant logs or output**
```
min gap between consecutive runner events: 576 ms, median about 1.5 s, never batched
insert time - emittedAt: 0.0 to 0.1 s
```

Related: #21524 bounds the receive buffer that this backlog overflowed.

## What Changed

- `acpx_provider_backend.rs`:
`AcpxCommandExecutor::provider_poll_is_live` returns true only when the
session was restored, has no persistence error, does not wait for a
next-turn generation, and its state is `turn_starting` or `turn_active`
with an active turn, no unconfirmed provider exit, and the run it owns.
- `native_provider_backend.rs`: `SelectedExecutor` forwards
`provider_poll_is_live` to the ACPX executor.
- `tests/acpx_warm_attachment.rs`: new test through
`NativeProviderCommandExecutor`. The session is not live before
`run.prepare` or after `session.open`, is live after `turn.start`, and a
new executor that has not restored its session is not live.

What does not change:
- runnerd still polls only when the read returns no frame, so a queued
stop or suspend still runs before the next provider poll.
- The ACK gate (no poll while sent events are not acknowledged) does not
change.
- The 1 ms read is used only while the source sequence advanced since
the last quiet poll, as for Codex today, so an idle turn does not
busy-loop.

## Verification

- `cargo test --release --manifest-path runner/Cargo.toml --locked -p
paperclip-runner-core --test acpx_warm_attachment` (in
`packages/paperclip-runner`): 7 passed. The new test fails without the
new method, because the default returns false.
- `cargo fmt --all -- --check` and `cargo check --locked --workspace`
pass.
- `cargo test --release --locked -p paperclip-runner-core
--no-fail-fast`: 1,127 passed, 3 failed. The failures are
`codex_provider::tests::fenced_receipt_output_reports_capacity_without_dropping_known_frames`
("failed to signal owned process group: Operation not permitted") and
`stdio_mcp::lost_acknowledgement_blocks_calls_until_socket_replacement`
and `metadata_only_socket_changes_do_not_bypass_unknown_outcome` (relay
exit timing). They also fail without this change in the macOS sandbox
where I ran the suite, and none of them uses the ACPX executor. CI
covers them on Linux.
- The existing `durable::tests::ack_release_poll` cases cover the runner
side of the live read: an ACK releases the next poll without the read
timeout, and a command queued behind the ACK runs first.
- I did not run a live provider. After the deploy, the gap between
runner events of an ACPX run with queued output should drop below 250
ms.

## Risks

- Low risk. The change only selects the existing 1 ms queued read for
ACPX during an active turn.
- A turn that produces events continuously now polls more often. Each
poll still writes the same durable state as before, so disk writes per
event do not change.
- Analysis also found that runnerd waits for every sent event to be
acknowledged before it polls again. A bounded in-flight window would
raise throughput further, but it changes the tested rule that control
queued behind an ACK runs before the next poll. That is left out of this
pull request on purpose.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 16:46:05 -08:00
Devin FoleyandPaperclip 254744f77c fix(runner): bound the ACPX receive buffer by bytes, not 512 events (#21524)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runner talks to ACPX providers through a sidecar process.
The sidecar writes events and command replies as JSON lines on stdout.
> - While the transport waits for a command reply, it buffers every
event that arrives before the reply. It failed when 512 events were
buffered.
> - The sidecar allows up to 4 MiB of unflushed stdout. A backlog of
small events passes 512 events long before any memory concern, for
example during a long turn with several subagents while the runner
relays events slowly.
> - When the limit hits, the transport is poisoned and the turn fails
with `acpx_provider_transport_failed`, which is not recoverable. The
task is then blocked.
> - This pull request bounds the receive buffer by frame bytes,
consistent with the sidecar's own output bound, and keeps a count
backstop for floods of tiny frames.
> - The benefit is that a normal event backlog no longer kills the turn,
and the buffer stays bounded in memory.

## Linked Issues or Issue Description

**What happened?**
A routine run started several subagents, which made about 190 GitHub
tool calls in about 10 minutes. The runner relays provider events at
about one per controller round trip, so the sidecar output fell several
minutes behind. A semantic tool result was then delivered through
`tool.resolve`. While it waited for the reply, the transport read more
than 512 backlogged events, failed with `ACPX sidecar exceeded the
buffered event limit`, and poisoned the transport. The turn failed with
`acpx_provider_transport_failed`, and recovery blocked the task.

**Expected behavior**
A backlog that the sidecar itself accepted does not fail a request. Only
a backlog that is too large for memory fails.

**Steps to reproduce**
1. Let a sidecar write more than 512 small events before it writes the
reply to a pending command.
2. The command fails with `ACPX sidecar exceeded the buffered event
limit`, and the transport is poisoned.

**Relevant logs or output**
```
ACPX sidecar exceeded the buffered event limit
turn.failed code=acpx_provider_transport_failed recoverable=false
```

No related public PR exists. I searched open and closed PRs for this
limit. The slow event relay that builds the backlog is a separate
problem.

## What Changed

- `acpx_sidecar_transport.rs`:
- Each buffered event keeps the byte length of its frame. The transport
keeps a running byte total.
- The buffer fails above 16 MiB (`MAX_BUFFERED_EVENT_BYTES`, four times
the sidecar's 4 MiB unflushed output bound
`SIDECAR_MAX_UNFLUSHED_OUTPUT_BYTES`), or above 65,536 events (one event
per 256 bytes of the budget) as a backstop for tiny frames.
- `poll_event` and `defer_polled_event` keep the byte total correct. A
poisoned transport resets it.
- `fake-acpx-sidecar.rs`: new `backlog`, `large-backlog` and
`count-flood` modes. `flood` now writes 18 MiB of 64 KiB events instead
of 513 small events.
- `tests/acpx_sidecar_transport.rs`: new tests for a backlog of 2,048
small events, for freeing the byte budget as events drain, and for a
flood of tiny events. The existing flood test still checks the error.

## Verification

- `cargo test --release --manifest-path runner/Cargo.toml --locked -p
paperclip-runner-core --test acpx_sidecar_transport` (in
`packages/paperclip-runner`): 23 passed.
- `buffers_a_backlog_of_small_events_before_a_response` fails before
this change (2,048 events are above the old 512 limit).
`frees_the_byte_budget_as_buffered_events_drain` needs correct
accounting: two backlogs of 12 MiB pass only when drained events free
their bytes.
- `cargo fmt --all -- --check` and `cargo check --locked --workspace`
pass.
- `cargo test --release --locked -p paperclip-runner-core
--no-fail-fast`: 1,128 passed, 4 failed. The failures are
`codex_provider::tests::fenced_receipt_output_keeps_prefix_when_stdout_never_closes`
and
`fenced_receipt_output_reports_capacity_without_dropping_known_frames`
("failed to signal owned process group: Operation not permitted"), and
`stdio_mcp::lost_acknowledgement_blocks_calls_until_socket_replacement`
and `metadata_only_socket_changes_do_not_bypass_unknown_outcome` (relay
exit timing). None of them uses the ACPX sidecar transport. They come
from the macOS sandbox where I ran the suite. CI covers them on Linux.
- `cargo test --release -p paperclip-runner-core --lib acpx`: 99 passed.
`--test acpx_provider_resolutions --test acpx_sidecar_stopped_time`:
passed.
- I did not run a live provider. After the deploy, a long turn with many
tool events should not fail with `buffered event limit`.

## Risks

- Low risk. The change is local to the transport receive buffer.
- The memory bound is now 16 MiB of frames instead of 512 events. Parsed
events use more memory than their frames, so the worst case is a few
times 16 MiB in the runner process.
- The sidecar's own 4 MiB unflushed output bound does not change. A
burst that is longer than that still fails in the sidecar with
`output_buffer_limit_exceeded`. The slow relay that builds the backlog
is a separate fix.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 16:45:58 -08:00
DottaandClaude Opus 5.5 e82c4f0a31 fix(runner): prove a warm Runner's suspension on a slow sandbox host so the next run does not fail (#21472)
**DO NOT MERGE until the coordinator confirms the Daytona validation.**

This PR extends the failed-close retry that #21442 (merged) added. It is
based on `master`, which contains #21442.

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm remote sandbox is reused by the next run of the same task.
Before the next run starts, the previous run's Runner must be retired:
suspended, with durable proof.
> - An idle warm close had 10 s for everything. On a loaded sandbox host
one liveness probe or one read of the durable state used that up. The
suspend was never sent, or it completed but was never proven.
> - The next run then replayed that cached close failure, failed before
any provider work, and the task showed a failed run with Retry
> - For a follow-up on a task blocked on its subtasks, the automatic
continuation was refused and the Retry stayed until another wake
> - This pull request bounds each probe, gives the suspend proof its own
read window, lets a later close prove the suspension again, and lets
that follow-up continue by itself
> - The benefit is that a slow host no longer turns a completed
suspension into a failed task, and the person sees no Retry for it

## Linked Issues or Issue Description

**What happened?**

Local release-swap cells on Daytona (G5c and G6c, 2026-11-13) each
showed one Retry banner: "previous run's session could not be retired
(runner did not durably suspend before checkpoint)". G6c had no fault
injection, so this is a real class.

- In the G6c case, the Runner completed its suspend and exited. The
server's proof read (a sandbox exec under load) ran out of the remaining
0.6 s of the close grace, so the suspension was "not proven".
- In the G5c case (an adopted Runner), the liveness probe
`adoptedRunner.isAlive()` took about 220 s and used the whole close
deadline. The suspend was never queued. Five adopted sessions failed the
same way in the same second.
- The successor run's retirement called `transport.close()`, which
returned the cached rejection of the predecessor's failed close. It did
not send a new suspend and did not read the state again.
- The successor failed before provider work. Its fresh-sandbox
continuation (`interrupted_preparation`) was refused as "blocked after
recovery", because the parent task was blocked on its subtasks before
the follow-up started. The task showed a failed run with Retry for 27 s,
until a blocker-stall wake.

**Expected behavior**

A completed suspend is proven, even on a slow host. A follow-up that
fails before any provider work on a task that was already blocked
continues by itself, and the task shows as working while that
continuation is queued.

**Steps to reproduce**

1. Close an idle warm Runner whose durable state read takes longer than
the close grace, or whose liveness probe stalls.
2. On master the close fails and the next close replays the same
failure.
3. The new tests in `runnerd-codex-transport-close-proof.test.ts` and
`heartbeat-native-predispatch-retry.test.ts` do this.

**Paperclip version or commit**

`master` at c6b497d831 (contains #21442).

## What Changed

- `runnerd-codex-transport.ts` (controller-side TypeScript):
- Each liveness probe in the close is bounded. A probe that does not
answer counts as possibly alive, never as gone, and the suspend goes
over the live connection. A connected Runner's probe gets a 1 s slice,
so drain and suspend keep their budget.
- Once the suspend command's completion is observed, what the
durable-state read shows decides how long the proof may take:

    | Read after the completion | Proof window | Maximum close time |
    | --- | --- | --- |
    | Suspended, right Runner | proven at once | unchanged |
| Another run's or Runner's state | ends at the close deadline, as on
master | close grace (10 s; 45 s for Pi) |
| Right Runner, not yet `suspended` | settle window of 3 s from the
first such read (`RUNNER_SUSPENSION_PROOF_SETTLE_MS`, test option
`closeProofSettleMs`) | close grace + 3 s |
| Unreadable or slow read | full proof window of 30 s from the
completion (`RUNNER_SUSPENSION_PROOF_READ_MS`, `closeProofReadMs`) |
close grace + 30 s |

All rows add the existing bounded exit-probe slices. runner-core saves
the `suspended` lifecycle before it delivers the suspend result
(`persist_lifecycle_before_command_delivery` in `durable/runner.rs`), so
a readable non-suspended state after the completion can only be a stale
remote read; the settle window covers a few read attempts and is not a
write-ordering wait.
- The settlement diagnostic names the branch that ended the window
(`proofWindowEnd`: `proven`, `wrong_identity`,
`not_suspended_after_settle`, `unreadable_after_window`,
`suspend_not_completed`).
- A close that failed only because readable proof was missing is retried
by the next close: it reads the durable state again with a fresh bound
and finishes settled. A close whose proof was refuted stays memoized, as
on master. This extends the failed-close retry from #21442.
  - The close failure carries per-step timings.
- `native-runtime/predecessor-retirement.ts`,
`heartbeat/run-completion.ts`: the predecessor-retirement fallback
records the task's status when the run failed.
- `modules/run-dispatch/domain/policy.ts`,
`modules/run-dispatch/adapters/postgres.ts`: a continuation of a run
that did no provider work, on a task that was already blocked then, is
not treated as blocked after recovery. Its dependency check follows the
wake it continues. The continuation keeps a person's comment wake.
- `task-interruption-presentation.ts`: while that run's failure handling
queues the continuation (up to 30 s), the task shows as working with no
Retry. The window ends when the continuation is scheduled, when the run
records `continuationScheduling: "refused"` (written by
`run-completion.ts` when the fallback does not schedule it), or after 30
s. The fallback record alone no longer ends it.

## Verification

Red on master, green on this branch:

- `runnerd-codex-transport-close-proof.test.ts`
- `proves a completed suspend on its own read window when the durable
state read outlasts the close grace`
- `suspends five adopted idle Runners whose liveness probes stall
together, instead of spending the close on the probe`
- `heartbeat-native-predispatch-retry.test.ts`
- `continues a follow-up on a blocked task whose predecessor could not
be retired`
- `does not continue a task that was blocked only after its run failed`
- `task-interruption-presentation.test.ts`
- `shows the task as working while its fresh-sandbox continuation is
being queued`
- `keeps the task working after the fallback is recorded while its
continuation is still being scheduled` (review fix)
  - `offers Retry when the continuation was refused or never queued`
- `runnerd-codex-transport.test.ts` (review fixes)
- `gives a suspend that completes near the close deadline its whole
proof window from the completion`
- `bounds the proof window from the observed completion, so a proof read
that never answers still ends`
- `proves a suspension whose suspended state appears 1.5 s after the
command completed`
- `fails a suspension whose state stays ready after the settle window,
not after the whole proof window`
- `fails at the close deadline when the durable state belongs to another
run`
- The three existing tests in
`runnerd-codex-transport-suspension.test.ts` that expect a conclusive
negative to fail within the close grace pass unchanged.

Local runs on this branch: `tsc --noEmit` is clean for `server` and
`packages/paperclip-runner`. Passing on this branch:
`runnerd-codex-transport-suspension`, `-close-proof`, `-adoption`,
`-remote-retention`, `runnerd-detach-durability` (all files), the
proof-window cases of `runnerd-codex-transport.test.ts`,
`task-interruption-presentation.test.ts` (72), `run-dispatch` policy and
Postgres adapter tests and `heartbeat-native-predispatch-retry.test.ts`
(166 together).

Conflict resolved: `task-interruption-presentation.ts` gained
`WAITING_CONTINUATION_MOVE_FORWARD_MS` on master (#21385). Both
constants are kept.

Runner content ID: the Runner change is controller-side TypeScript under
`packages/paperclip-runner/src` only. There is no Rust, `protocol/` or
`generated/` change. Existing Runner images work with this controller.

Daytona: this fix has not run on Daytona at all. The two banners above
are the evidence of the class (G5c and G6c ran on 588c6ba267, without
this fix).

## Risks

- Not proven on Daytona.
- A close can now take up to 30 s longer when the proof read is
unreadable or slow, and up to 3 s longer when the read is stale. This is
on the close path only, not before `turn.start`.
- A probe that does not answer now counts as possibly alive, so the
close sends a suspend instead of treating the Runner as gone.
- Ownership: this PR changes
`server/src/modules/run-dispatch/domain/policy.ts`,
`server/src/modules/run-dispatch/adapters/postgres.ts` and
`server/src/services/heartbeat/run-completion.ts`. CODEOWNERS does not
list these files. Their recent changes come from the restart-recovery
and Retry work (#21348, #21256, #20863, #21302). That area should review
the dispatch-policy change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:32:22 -06:00
Nicky LeachandClaude Opus 5.5 1332ab987f feat(prp): declare the contract for provider-started turns (#21422)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner runs Claude Code through the ACPX sidecar.
> - Claude can start a shell command in the background and end its turn.
When the command finishes, Claude Code starts a cycle of its own to
continue. Today the Runner stops Claude at the end of the turn, so the
command stops and the run fails with `native_finalization_missing`.
> - A stack of pull requests lets the Runner accept that cycle as a new
turn of the same run, when a controller asks for it.
> - The Runner, its sidecar and the controller speak two protocols with
schemas: PRP between the controller and the Runner, and the sidecar
protocol between `paperclip-runnerd` and the ACPX sidecar.
> - A turn that the provider starts needs new fields and one new command
in those protocols.
> - This pull request declares them and generates the readers again. It
changes no behavior.
> - The benefit is that the next two pull requests show behavior only,
with the contract already in review.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one part of a change that was first opened as one
large pull request (#21316). The parts, in stack order after #21311:

1. #21419: sidecar preparation (a refactor).
2. #21420: converter for raw ACP session updates.
3. #21421: cycle stream at the adapter.
4. #21422: contract for provider-started turns.
5. #21423: `paperclip-runnerd` waits for a provider-started turn.
6. #21316: the sidecar adopts a cycle.

This is part 4. Its base branch is `feat/claude-provider-cycle-stream`.
After each part, a production run does the same as today, because no
controller sends the opt-in. Refs #19912 (design).

**What existing behavior does this improve?**

Nothing in production. The protocol schemas.

**Subsystem affected**

Paperclip Runner: `protocol/` and the generated TypeScript and Rust
contract code.

**Current behavior**

- PRP has no way to say that a turn was started by the provider, or that
the Runner waits for such a turn.
- The sidecar protocol has no command to arm the sidecar and no event
for a turn that the provider starts.

**Proposed behavior**

- PRP (`schemas/provider-turn.schema.json`, new): `providerStartedTurns`
(boolean) in the payload of `turn.start`, `origin: "provider"` on
`turn.started`, and `awaitingProviderTurn: true` on `turn.completed`.
`command-v3` and `event-v3` refer to it.
- Sidecar protocol (`provider-schemas/acpx-sidecar.schema.json`): the
command `session.provider_turn` with `{ turnId, armed }`, the event
`runtime.turn_started` with `{ origin: "provider" }`, and the payload of
the diagnostic for a lost provider. The schema enforces that payload for
a `runtime.diagnostic` with the code `ACPX_PROVIDER_PROCESS_LOST`.
Diagnostics with other codes keep their open payload.
- `paperclip-runnerd` decodes `runtime.turn_started` and refuses it. No
sidecar sends it yet. The Runner of today refuses an unknown event too.
- No controller, Runner or sidecar uses the new fields in this pull
request.

**Reason and benefit**

The repository rule for Runner contracts says that new fields must be in
the schema with generated readers. A separate pull request lets a
reviewer read the contract alone.

## What Changed

- `protocol/schemas/provider-turn.schema.json` (new),
`command-v3.schema.json`, `event-v3.schema.json`,
`protocol/provider-schemas/acpx-sidecar.schema.json`,
`protocol/manifest.json`.
- `scripts/generate-acpx-sidecar-contract.mjs` and
`scripts/generate-protocol-schema-module.mjs`.
- Generated: `src/drivers/acpx/generated-sidecar-contract.ts`,
`src/protocol/generated/schema-bundle.ts`,
`src/protocol/generated/standalone-validators.ts`,
`runner/crates/runner-core/src/generated_acpx_sidecar_contract.rs`.
- `runner/crates/runner-core/src/acpx_event_payload.rs` (3 lines): the
Rust `match` on the event type must name the new event to compile.
- `test/protocol-contract.test.mjs`.
- The contract digests of the pack change (`prp.acpx-sidecar`,
`prp.command-v3`, `prp.event-v3`, `rust`, and the new
`prp.provider-turn`). No stored file refers to these digests. No
provider `commandDigest` and no profile version changes.

## Verification

At the head of this branch, from `packages/paperclip-runner`:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run
build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs
test/protocol-contract.test.mjs`, the server check `node
scripts/verify-runner-vendor-dependencies.mjs`, `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt
--check` and `cargo build -p paperclip-runner-core --bins` (no new
warning): all passed. The contract tests: 32 passed.
- The sidecar suite and the tracker tests: 141 passed.

Not run: the full suite. CI supplies that coverage.

## Risks

- **The pack contract digest changes.** Older image packs are not used
again.
- **The opt-in field is accepted by the schema but has no effect yet.**
A controller that sends it before part 5 gets the behavior of today.
- **State saved by the previous release:** this pull request changes no
persisted shape.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 00:20:45 +00:00
Nicky LeachandClaude Opus 5.5 e5ad224cc5 feat(acpx): give Claude's own cycles to the adapter as one ordered stream (#21421)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner runs Claude Code through the ACPX sidecar.
> - Claude can start a shell command in the background and end its turn.
When the command finishes, Claude Code starts a cycle of its own to
continue. Today the Runner stops Claude at the end of the turn, so the
command stops and the run fails with `native_finalization_missing`.
> - A stack of pull requests lets the Runner accept that cycle as a new
turn of the same run, when a controller asks for it.
> - Claude Code starts a cycle of its own when a background task
finishes and no prompt is in flight. The Runner sees the raw messages of
that cycle.
> - #21293 only marks the background state as unknown when it sees such
activity.
> - This pull request gives each such cycle to the caller of the adapter
as one ordered stream: started, its events, ended with its result.
Nothing reads the stream yet.
> - The benefit is one place that knows where a cycle starts and ends,
with cycles that cannot overlap.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one part of a change that was first opened as one
large pull request (#21316). The parts, in stack order after #21311:

1. #21419: sidecar preparation (a refactor).
2. #21420: converter for raw ACP session updates.
3. #21421: cycle stream at the adapter.
4. #21422: contract for provider-started turns.
5. #21423: `paperclip-runnerd` waits for a provider-started turn.
6. #21316: the sidecar adopts a cycle.

This is part 3. Its base branch is
`feat/acp-session-update-event-converter`. After each part, a production
run does the same as today, because no controller sends the opt-in. Refs
#19912 (design).

**What existing behavior does this improve?**

Nothing in production yet. How the ACPX adapter sees the cycles that
Claude starts by itself.

**Subsystem affected**

Paperclip Runner: `src/drivers/acpx` (the adapter, the runtime host, the
Claude background task tracker). Claude only.

**Current behavior**

- The raw ACP observer gives each message to the background task
tracker.
- Activity of Claude with no client request in flight makes the
background state unknown. The messages of that cycle are not kept.

**Proposed behavior**

- `ClaudeProviderCycles` puts each cycle on one ordered stream. A cycle
starts at the first update of the own session with no client request and
no prompt in flight. It ends at the `usage_update` that has
`_meta._claude/origin`.
- A caller gets the stream with `providerCycles()`, reads it with
`next()`, and calls `adopt()` or `drop()` for each cycle. A permission
request waits for the decision of its own cycle. The stream records the
cycle of each tool call, and `decisionFor(toolCallId)` gives that
decision. The binding that a caller gives to `adopt()` can also carry
`onPlanProposal`, so a plan that an adopted cycle proposes goes to the
caller as it does for a normal turn.
- With no caller, a cycle is declined and the background state becomes
unknown, as before. The sidecar does not call `providerCycles()` in this
pull request, so production does the same as today.
- The tracker reports `{ live: N }` only while N shells are running, `{
live: 0 }` only if the process never had a background task and Claude
never acted between prompts, and unknown in each other case. A dropped,
failed or cancelled cycle makes the state unknown.
- An empty result that directly follows the end of a cycle (zero usage,
same context use and cost) is ignored. Claude Code sometimes sends it.
It must not become a cycle, and its usage must not be reported two
times.

**Reason and benefit**

- One producer and one reader make the order of cycles a property of the
structure.
- The tracker does not read the order of messages to decide which
finished task a cycle reported on. Captures showed that Claude Code can
put two notifications in one cycle, start a second cycle, or give a
notification to a cycle in progress, and the ACP stream does not say
which.

## What Changed

- `src/drivers/acpx/claude-provider-turns.ts` (new, 303 lines):
`ClaudeProviderCycles`.
- `acpx-runtime-adapter.ts` (78 added, 8 removed) and `runtime-host.ts`
(24 added): `providerCycles()` and `providerTurnsAvailable()`, and the
permission path that waits for the decision.
- `claude-background-tasks.ts` (16 added, 7 removed): the rule above.
- Tests (764 lines): `claude-provider-turns.test.ts`,
`claude-provider-turn-tasks.test.ts`, `claude-background-tasks.test.ts`,
a permission test in `acpx-runtime-adapter.test.ts`, two tests with the
real `claude-agent-acp`
(`acpx-runtime-adapter.claude-provider-turn.test.ts`,
`acpx-runtime-adapter.claude-overlapping-shells.test.ts`), and a
captured raw stream as a fixture.

## Verification

At the head of this branch, from `packages/paperclip-runner`:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run
build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs
test/protocol-contract.test.mjs`, the server check `node
scripts/verify-runner-vendor-dependencies.mjs`, `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt
--check` and `cargo build -p paperclip-runner-core --bins` (no new
warning): all passed.
- 10 test files, 294 passed. This includes the test of #21293 with the
real `claude-agent-acp`, not changed: with no caller of the stream it
still gets `{ live: 1 }` at the end of the turn and an unknown state
after the cycle of Claude.
- Real `claude-agent-acp` with a local fake Anthropic API: a background
shell finishes, and the stream gives one cycle with its own usage. With
two shells where the second finishes before the first cycle sends its
first update, the first cycle ends with an unknown state.
- Mutation checks, each made the named tests fail: an end item before
the last message of a cycle; a drop that does not make the state
unknown; a finished task that reports `{ live: 0 }`; the empty result
not ignored; a permission request that does not wait.
- Live run with a real Claude account, not in CI, on the full change:
one background shell gave `{ live: 1 }`, one cycle, then unknown. Two
shells of 8 s and 28 s gave `{ live: 2 }`, `{ live: 1 }`, then unknown.

Not run: the full suite. CI supplies that coverage.

## Risks

- **The time at which the state becomes unknown changes a little.** With
no caller, a cycle makes the state unknown at its first message. Before,
it was at the first message of an activity kind. The result for the
Runner is the same: a restart at the end of the turn.
- **Limit, by design.** After a cycle, `{ live: N }` counts only running
shells. A command that finishes while Claude already reports on an
earlier one gets no cycle that the Runner waits for.
- **The rule for the empty result is narrow** and comes from captures of
one Claude Code version. The reason that Claude Code sends it is not
known.
- **A queue of 4,096 messages for one cycle.** A cycle that is larger
fails, and the state becomes unknown.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:48:53 -08:00
Nicky LeachandClaude Opus 5.5 1d112a504f feat(acpx): convert raw Claude session updates the way ACPX does (#21420)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner runs Claude Code through the ACPX sidecar.
> - Claude can start a shell command in the background and end its turn.
When the command finishes, Claude Code starts a cycle of its own to
continue. Today the Runner stops Claude at the end of the turn, so the
command stops and the run fails with `native_finalization_missing`.
> - A stack of pull requests lets the Runner accept that cycle as a new
turn of the same run, when a controller asks for it.
> - The Runner sees each raw ACP message before ACPX does. A cycle that
Claude starts by itself is not a part of an ACPX prompt, so ACPX gives
no events for it.
> - To show that cycle in the same form as a normal turn, the Runner
must convert the raw messages itself. ACPX 0.22 does not export its
converter.
> - This pull request adds a copy of that converter with a test that
compares it with ACPX. Nothing uses it yet.
> - The benefit is that a later pull request can show the cycle of
Claude with the same events as a normal turn.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one part of a change that was first opened as one
large pull request (#21316). The parts, in stack order after #21311:

1. #21419: sidecar preparation (a refactor).
2. #21420: converter for raw ACP session updates.
3. #21421: cycle stream at the adapter.
4. #21422: contract for provider-started turns.
5. #21423: `paperclip-runnerd` waits for a provider-started turn.
6. #21316: the sidecar adopts a cycle.

This is part 2. Its base branch is
`refactor/acpx-sidecar-runtime-event-step`. After each part, a
production run does the same as today, because no controller sends the
opt-in. Refs #19912 (design).

**What existing behavior does this improve?**

Nothing yet. It adds a module that a later part uses.

**Subsystem affected**

Paperclip Runner: `src/drivers/acpx`. One new direct dependency in the
Runner and the server.

**Current behavior**

- ACPX converts `session/update` messages to runtime events inside a
prompt. The function is private.
- The Runner has no way to make the same events from a raw message.

**Proposed behavior**

- `acp-session-update-event.ts` converts a raw `session/update` to the
event that ACPX 0.22 makes for the same update.
- It validates the message with `@agentclientprotocol/sdk` 1.9.0. This
is the version that ACPX 0.22 already installs.
- No production code imports the module in this pull request.

**Reason and benefit**

A copy with a comparison test is a smaller risk than a patch of ACPX.
The test fails when the two differ.

## What Changed

- `src/drivers/acpx/acp-session-update-event.ts` (new, 367 lines).
- `acp-session-update-event.test.ts` and
`acpx-runtime-adapter.claude-session-update-events.test.ts` (new, 363
lines).
- `packages/paperclip-runner/package.json` and `server/package.json`:
`@agentclientprotocol/sdk` 1.9.0 as a direct dependency. The server
needs the line because it copies the Runner build with no
`node_modules`.
- `pnpm-lock.yaml` is not in this pull request. The lockfile workflow
owns it.

## Verification

At the head of this branch, from `packages/paperclip-runner`:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run
build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs
test/protocol-contract.test.mjs`, the server check `node
scripts/verify-runner-vendor-dependencies.mjs`, `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt
--check` and `cargo build -p paperclip-runner-core --bins` (no new
warning): all passed.
- The two new test files and the sidecar suite: 130 passed.
- The comparison test runs the real ACPX runtime with the pinned
`claude-agent-acp` and a local fake Anthropic API, and gives the same
raw updates to the copy. The events are the same.
- A second test fails if the SDK version of the Runner is not the
version that ACPX resolves. With 1.4.0 in `package.json` it fails.

Not run: the full suite. CI supplies that coverage.

## Risks

- **A new direct dependency.** It is already installed through ACPX, so
no new package is downloaded. The version test keeps the two the same.
- **The converter is a copy of ACPX code.** An ACPX upgrade must run the
comparison test.
- **The module is not used yet.** It is used in part 3.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:14:59 -08:00
Nicky LeachandClaude Opus 5.5 055a0a3c54 refactor(acpx-sidecar): extract the runtime event step and the test process helper (#21419)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner runs Claude Code through the ACPX sidecar.
> - Claude can start a shell command in the background and end its turn.
When the command finishes, Claude Code starts a cycle of its own to
continue. Today the Runner stops Claude at the end of the turn, so the
command stops and the run fails with `native_finalization_missing`.
> - A stack of pull requests lets the Runner accept that cycle as a new
turn of the same run, when a controller asks for it.
> - The last part of that stack adds a second kind of turn to the
sidecar. It needs two pieces of the code of a normal turn.
> - This pull request moves those two pieces into functions, and moves a
test helper to a file of its own. It changes no behavior.
> - The benefit is that the later pull request shows only new behavior,
not moved code.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one part of a change that was first opened as one
large pull request (#21316). The parts, in stack order after #21311:

1. #21419: sidecar preparation (a refactor).
2. #21420: converter for raw ACP session updates.
3. #21421: cycle stream at the adapter.
4. #21422: contract for provider-started turns.
5. #21423: `paperclip-runnerd` waits for a provider-started turn.
6. #21316: the sidecar adopts a cycle.

This is part 1. Its base branch is
`refactor/remove-direct-gateway-path`. After each part, a production run
does the same as today, because no controller sends the opt-in. Refs
#19912 (design).

**What existing behavior does this improve?**

Nothing for a user. The structure of the ACPX sidecar code.

**Subsystem affected**

Paperclip Runner: `src/cli/acpx-runtime-sidecar.ts` and its test.

**Current behavior**

- `pumpTurn` has the code that prepares each runtime event for the
Runner, and the code that sends the terminal frame of a turn, inline.
- The sidecar test file has the `SidecarProcess` helper that starts a
sidecar process.

**Proposed behavior**

- `createRuntimeEventStep` has the event code. `emitTurnTerminal` has
the terminal code. `pumpTurn` calls both.
- `src/test-support/acpx-sidecar-process.ts` has the helper, so a second
test file can use it.
- No protocol, event or timing change.

**Reason and benefit**

A later pull request adds a turn that Claude starts by itself. It uses
the same two functions, and its tests use the same helper.

## What Changed

- `src/cli/acpx-runtime-sidecar.ts`: 74 lines added and 56 removed. The
code moved with its comments, with no change.
- `src/cli/acpx-runtime-sidecar.test.ts` and
`src/test-support/acpx-sidecar-process.ts`: the helper moved, 138 lines
added and 132 removed.

## Verification

At the head of this branch, from `packages/paperclip-runner`:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run
build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs
test/protocol-contract.test.mjs`, the server check `node
scripts/verify-runner-vendor-dependencies.mjs`, `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt
--check` and `cargo build -p paperclip-runner-core --bins` (no new
warning): all passed.
- `npx vitest run src/cli/acpx-runtime-sidecar.test.ts`: 125 passed. The
tests are the same as before, apart from the import of the helper.

Not run: the full suite. CI supplies that coverage.

## Risks

- **A refactor of the turn loop of each ACPX agent.** The moved code
runs for Claude, Codex, Pi, Kimi, Cursor, Copilot and Grok. The existing
sidecar suite covers it. No new test is necessary, because no behavior
changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:39:55 -08:00
DottaandClaude Opus 5.5 342fb308bd perf(run-events): commit streaming text deltas in batches (#21460)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process stores every event of every running native agent
in `heartbeat_run_events`
> - Text deltas (`item.delta`) are about 85% of that process's SQL
statements: each delta is one transaction of four statements on the
shared 10-connection pool
> - At 50 agents and 20 deltas per second, the process falls behind and
unrelated agent requests wait for the pool
> - This pull request commits text deltas in batches: one transaction
for all deltas that arrive within 250 ms, with one multi-row insert
> - The benefit is about 7 times fewer transactions and 5–6 times fewer
statements per delta, with no delay for the first text of an item

## Linked Issues or Issue Description

Refs #21435 (this pull request is stacked on it). Refs #21131. Item 1a
of the agent scaling plan in #21418.

**What existing behavior does this improve?**
The persistence of native text deltas. `native-session-runtime` awaited
`PaperclipControlPlanePort.appendEvent` for each event, and the port ran
one transaction per event.

**Subsystem affected**
Server run event ingest (`paperclip-control-plane-port.ts`,
`heartbeat-run-events.ts`) and the runner package's turn consumer
(`native-session-runtime.ts`).

**Current behavior**
Each delta takes one transaction: BEGIN, run lock, append, COMMIT. In
the load model with 50 agents (deltas only), that is about 4.0
statements and 1.0 transaction per delta.

**Proposed behavior**
The turn consumer starts the next append without waiting for a delta's
receipt. The port collects deltas and commits them in one transaction
with one insert. In the same load model, that is about 0.7 statements
and 0.14 transactions per delta.

**Reason and benefit**
Fewer transactions free the shared pool and the main thread for other
agents' requests. This is the largest per-agent cost in the scaling
plan.

**Breaking changes**
None. The row shape does not change. A port without the new capability
keeps the earlier serial behavior.

## What Changed

- `appendHeartbeatRunEventBatch` (`heartbeat-run-events.ts`) appends
native source events of one run and one source in one transaction. When
the events continue the stored source cursor (from #21435), it sends
four statements for the whole batch: the run lock, one replay probe, one
`insert ... select from jsonb_array_elements(...)` of every row, and the
commit. Any other batch (a replay, a gap, an event with a wake, or a run
without a cursor) appends its events one by one in the same transaction,
with the single-append checks.
- **Storage choice: one row per delta, written by one statement per
batch.** I did not store one row per batch. The transcript, live events,
run log export, chat progress, replay (`replayEvents`), recovery cursors
and the UI all read one row per delta. One statement per batch keeps
every reader unchanged, and rows written before this change read the
same.
- `PaperclipControlPlanePort` sets `pipelinesStreamingEvents`. An
`item.delta` appended with the new `streaming` option goes into a queue:
- It commits at once until a delta with text commits after the latest
other event. So the first text of each item, and the time to first
token, never wait for the timer.
- Later deltas commit after at most 250 ms, or at once when the batch
has 100 deltas.
- Every other event, `checkpointSession`, `completeRun`, `replayEvents`
and `provesNoRunProviderWork` first commit the queue. A non-streaming
event is never committed before the deltas that preceded it.
- A delta appended without `streaming` commits at once, so callers that
wait for each receipt are not slowed.
- The receipt of a delta settles only after its batch commits and its
observers (live event, executor callback) run, in order.
- If a batch fails, every delta in it rejects and nothing from it is
stored.
- `native-session-runtime` `consumeTurn` appends deltas with `streaming`
and does not wait for their receipts (at most 256 pending). Before any
other event, any result, and the governed-wait settlement, it calls
`flushStreamingEvents()` and then waits for all pending receipts, so a
boundary never waits for the timer. A failed delta stops the turn before
a later event is appended. The consumer joins every pending delta append
before it settles, also when the provider stream fails.
- If one observer of a committed batch fails, only that delta's receipt
rejects. The other rows are still published and observed.
- `ControlPlanePort` contract: optional `pipelinesStreamingEvents`,
`flushStreamingEvents()` and `AppendControlPlaneEventOptions.streaming`.

## Verification

- Red/green. The new tests failed before the change and pass after it:
- Runner: "starts text delta appends without waiting and flushes them
before any other event" (`native-session-runtime.test.ts`). The port in
the test has no timer, so the turn finishes only if the boundary
flushes. Before: the appends were serial, and later the boundary waited
for the timer.
- Runner: "joins pending delta appends before a failed turn settles".
Before: the turn settled while a delta commit still ran.
- Port: "observes every committed delta of a batch when one observer
fails".
- Port: "commits the first text delta at once and later deltas in one
batch before the next other event". It checks one transaction and one
insert for five deltas, flush before a non-streaming event, receipts
only after commit, and immediate commit of the first delta with text.
Before: each delta committed alone.
- Port: "flushes queued deltas on the timer and before a replay read,
and rejects every delta of a failed batch". It checks the
crash-before-commit case: a revoked writer fails the batch, nothing is
stored, and the resent deltas commit idempotently (the stored one is a
duplicate).
- Store: four new tests in `heartbeat-run-event-atomic-append.test.ts`:
one lock, one probe and one insert per batch; replay idempotency and
payload conflict; a batch that fails before commit stores nothing and
its resend commits; the one-by-one path for no cursor and for gaps.
- Focused suites pass: `heartbeat-run-event-atomic-append`,
`heartbeat-run-event-sequencing`, `paperclip-control-plane-port`
(including the package conformance suite),
`paperclip-semantic-conformance`, `native-observer-replay`,
`native-event-boundary-golden`, `live-recovery-work`,
`native-interaction-bridge`, `chat-native-progress-work`, and
`native-session-runtime.test.ts` in the runner package.
- `pnpm --filter @paperclipai/paperclip-runner run typecheck:typescript`
and `pnpm --filter @paperclipai/server run typecheck` pass. `pnpm
check:module-boundaries` passes.
- Load model (`scripts/benchmark-agent-scaling-load.ts` from #21418,
`AGENTS=50 SECONDS=20`, local embedded PostgreSQL). The load model calls
the store directly, so for this run I added a local `BATCH_MS` mode that
applies the port's batching rules (first delta at once, 250 ms timer,
100 cap, flush before other events) and does not wait for delta
receipts. That mode is not in this pull request. The machine was very
busy (load average 40–60), so absolute figures are noisy; per-delta
ratios are the stable result:

| Deltas only | Statements/s | Transactions/s | Deltas done of offered |
Statements per delta | Transactions per delta | Lateness p50 / p99 |
  | --- | --- | --- | --- | --- | --- | --- |
| One transaction per delta | 2,920 | 730 | 14,668 of 20,000 | 3.98 |
1.00 | 3,074 / 5,384 ms |
| Batched (250 ms) | 704 | 141 | 19,940 of 20,000 | 0.71 | 0.14 | 668 /
2,858 ms |
| One transaction per delta (repeat) | 1,712 | 428 | 8,575 of 20,000 |
3.99 | 1.00 | 6,558 / 11,464 ms |
| Batched (250 ms, repeat) | 856 | 172 | 20,000 of 20,000 | 0.86 | 0.17
| 187 / 556 ms |

With all sources on, the machine was saturated (event-loop utilization
0.96–0.98) in both modes and the results were not usable (764 and 12,800
of 20,000 deltas done).

## Risks

- **Ack and crash (read first).** In the live runnerd path the runner's
ack does not wait for the `heartbeat_run_events` row today.
`DurablePrpControlPlane.#event` acks after the in-memory journal commit.
Streaming events are not in the controller checkpoint. So a delta that
is acked but not yet appended is lost if the server crashes, and the
runner does not resend it. This is the earlier behavior. This pull
request makes that window up to 250 ms longer. The final text still
survives through `item.completed`, which commits immediately and in
order. This pull request does not change `durable-prp-control-plane.ts`.
- Live text in the UI arrives in steps of up to 250 ms after the first
text of an item.
- A delta failure now surfaces at the next non-delta event or at the end
of the turn, not at the delta. The turn stops before it appends a later
event.
- The governed-wait drain and the control-event replay still append one
event at a time. They do not carry deltas.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:34:01 -06:00
Nicky LeachandClaude Fable 5.2 0aaab644fe refactor(runner): remove the direct assigned-gateway path (#21311)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent reaches the tools of its app connections through an MCP
gateway.
> - Before #21274, some native harnesses got the gateway URL and a
bearer token and connected to the gateway directly. The Runner has code
for that: an HTTP MCP server for ACPX providers, a server entry in the
Codex `config.toml`, a server entry in the OpenCode config, and the
lists that pass the three `PAPERCLIP_NATIVE_MCP_*` variables to a
sidecar.
> - #21274 gives each native harness its gateway tools through the PRP
relay. Core removes the three variables from each Runner environment.
The direct code path gets no binding.
> - Code that cannot run still needs maintenance, and it can still put a
token in a provider process if a variable arrives by another way.
> - This pull request deletes the direct path from the Runner.
> - The benefit is one path for assigned gateway tools, and no Runner
code that can give a gateway token to a provider.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is in a stack. Its base branch is
`fix/keep-quiet-claude-provider-between-turns` (#21293). It depends on
#21274, which is the bottom of the stack. On `master` without #21274,
the direct path is still in use.

**What existing behavior does this improve?**

The Runner code that connects a provider directly to the assigned MCP
gateway.

**Subsystem affected**

Paperclip Runner: the TypeScript drivers and the Rust `runner-core`
environment lists.

**Current behavior**

After #21274, Core never gives `PAPERCLIP_NATIVE_MCP_NAME`,
`PAPERCLIP_NATIVE_MCP_URL` or `PAPERCLIP_NATIVE_MCP_TOKEN` to a Runner.
The Runner still has the code that reads them and configures a direct
gateway server for ACPX providers, Codex and OpenCode.

**Proposed behavior**

The Runner has no code that reads the three variables to configure a
provider. If the variables are in the environment of a Runner, no
provider environment, sidecar, Codex `config.toml` or OpenCode config
gets them.

**Reason and benefit**

Less code, one path, and no way for a gateway token to reach a provider
process through the Runner.

## What Changed

- Deleted `packages/paperclip-runner/src/drivers/native-mcp.ts`
(`nativeMcpLaunchBinding`).
- `drivers/acpx/runtime-host.ts`: removed the native gateway server
entry and the name conflict check. The admission gate keeps its message.
A run with a binding id is admitted when the sidecar uses the PRP
transport with its tool bridge, as the production sidecar always does.
- `drivers/acpx/environment.ts`: the name and the URL are not in the
provider allowlist.
- `drivers/runtime-context-materializer.ts` and
`live/runnerd-codex-transport.ts`: the Codex home functions have no
`nativeMcp` parameter. The sidecar token pass-through is gone.
- `drivers/opencode/opencode-server-driver.ts`: removed the read of the
binding, its trace redaction value, and the gateway server entry in the
OpenCode config.
- `drivers/stdio-mcp.ts` and `runner-core/src/stdio_mcp.rs`: removed the
reservation of the gateway server name. No gateway server exists, so no
name can conflict. The names `paperclip` and `paperclip_runner_*` stay
reserved.
- `runner-core/src/acpx_sidecar_transport.rs` and `codex_provider.rs`:
the three variables are not in the pass-through lists.
- `control-plane/durable-prp-control-plane.ts`: the three variables are
not in the platform environment list.
- `packages/paperclip-runner/README.md`: the text about direct callers
and assigned servers.
- Kept: the three names stay in the reserved lists
(`src/configured-environment.ts`,
`runner-core/src/configured_environment.rs`). A user cannot configure
them. Tests in both languages now check all three names.
- No server code changed. The relay needs the gateway token and URL on
the control plane.

23 files, 73 lines added and 211 lines removed.

## State saved by the previous release

- This pull request changes no persisted shape. The provider server
lists, the launch profile digests, the protocol versions and the sidecar
contract are the same as with #21274.
- A Runner or sidecar that the previous release started keeps its own
code until its run ends, as #21274 describes.
- A kept Codex home that has the old gateway server in `config.toml` is
written again at the next start and at a recovery of the same run. A
test covers this.
- If the old variables reach a new Runner, no provider gets them. Tests
cover this for ACPX, Codex and OpenCode.

## Verification

From `packages/paperclip-runner`:

- `npx vitest run src/drivers/acpx/runtime-host.test.ts
src/live/runnerd-codex-transport.test.ts
src/drivers/acpx/environment.test.ts
src/drivers/opencode/opencode-server-driver.test.ts
src/live/runnerd-codex-transport-warm-attachment.test.ts
src/drivers/runtime-context-materializer.test.ts
src/drivers/agent-observability.test.ts src/drivers/stdio-mcp.test.ts
src/control-plane/durable-prp-control-plane.test.ts
src/configured-environment.test.ts`: `10 passed (10)` files, `807
passed` tests.
- `npx tsc --noEmit -p tsconfig.json` and `-p tsconfig.test.json`: no
errors.
- `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`: no
changes.
- `cargo test -p paperclip-runner-core --test acpx_sidecar_transport`:
20 passed. `cargo test -p paperclip-runner-core --lib`: 639 passed, 4
ignored.

Tests that failed with the old code and pass now:

- `runtime-host.test.ts`: "admits a relayed … run with a binding id and
ignores a previous-release gateway binding", for Claude, Codex and
Cursor.
- `runnerd-codex-transport.test.ts`: "drops a previous-release gateway
binding from the … runner environment", for ACPX Claude, Codex, Pi,
Cursor, Copilot and Kimi, and for OpenCode.
- `opencode-server-driver.test.ts`: the OpenCode config has only the
task bridge.
- `runnerd-codex-transport-warm-attachment.test.ts`: a restored Codex
home has no gateway server and no gateway credential in `config.toml`.
- `environment.test.ts`: a provider environment has none of the three
variables.
- Rust
`previous_release_gateway_binding_and_unrelated_secrets_never_reach_the_sidecar`:
a Runner that starts with the old variables gives none of them to the
sidecar, for five agents.

Mutation check: with the three names put back in the Rust pass-through
list, the Rust test fails.

Not run: a live run with a real provider. Not run: `pnpm verify` and the
full suite. CI supplies that coverage.

## Risks

- **This pull request is safe only after #21274.** The stack order keeps
that.
- **A developer who sets the three variables by hand for
`eval-session`** no longer gets direct gateway tools. No document
describes that use.
- **The warm attachment test uses the staged Runner binary from
preflight.** That binary does not have the Rust changes of this pull
request. The Codex path of that test does not use the changed lists.
- The evidence that no entry point supplies a binding comes from a read
of the code: Core removes the variables before it builds each Runner
environment, and a native run cannot use the in-process path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change, reviewed the diff, and ran the verification and a mutation check
again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), traced each code path and
wrote the change and the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 13:33:47 -08:00
Devin FoleyandPaperclip 12aa08721c feat(ssh): use the agent AI account in new environments (#21248)
Use the agent-selected AI account for new SSH environments. Preserve
remote-machine authentication for existing environments and expose it as
an advanced option. Keep managed credentials private while preserving
supported conversation state and credential refreshes across turns.

Validated with focused route, adapter, lifecycle, and real-process tests;
final CI passed on d634944df1.
Greptile Apex reviewed that revision at 5/5 with no unresolved threads.
External neutral security reports remain documented in the pull request.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-13 13:20:13 -08:00
DottaandClaude Opus 5.5 2a8b485140 fix(runner): accept object fields sent as JSON text and name wrong-type rejections (#21485)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip runner checks each tool call from the model against
the tool's input schema before it sends the call to the control plane
> - Some model hosts send an object argument as its JSON text. On the
DigitalOcean host, `paperclip_finish` arrived with `continuation` as a
string
> - The runner rejected these calls, and the repeated-rejection guard
then told the model that its arguments "did not arrive", which is wrong
when every field arrived with the wrong type
> - This pull request makes the runner parse such a string back into its
object when that is safe, records each coercion as a run event, and
makes the guard name the actual problem
> - The benefit is fewer stuck finish calls, and guidance that points
the model at the real fault

## Linked Issues or Issue Description

Refs #21398

**What happened?**
Chat-eval cohort 13 ran OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017` on master `df1b12ad8e`.
On the DigitalOcean host, 6 of the 8 remaining "Invalid tool input"
rejections were `paperclip_finish` calls with `continuation` sent as a
JSON string (`/continuation must be object,null`). Examples:
`gateways-profiles.rotate-token`, `instance-operations.restart-service`,
and `questions-and-approvals.multi-part-question`. The run diagnostic
recorded `continuationShape: "string"`. After the second identical
rejection, the guard from #21398 said "If you meant to send other
arguments, they did not arrive". Every argument had arrived; one had the
wrong type.

**Expected behavior**
When a schema field is object-typed, does not allow text, and the text
parses to a plain JSON object that makes the call valid, the runner
accepts the call with the parsed object and logs a run event. When a
call stays invalid, the guard says which required fields are missing and
which fields have the wrong type.

**Steps to reproduce**
1. Start the OpenCode MCP bridge.
2. Call `paperclip_finish` with `reportedWorkDisposition: "yielded"` and
`continuation` as `JSON.stringify({ kind: "response_wake", ... })`.
3. Before this change, the bridge returns `Invalid tool input:
/continuation must be object,null`.

## What Changed

- Added `packages/paperclip-runner/src/drivers/tool-input-coercion.ts`.
`admitToolInput` validates the arguments. If they are invalid, it parses
a string in an object-typed field back into its object. It does this
only when the field's schema declares `object`, does not allow `string`,
the text parses to a plain JSON object, and the whole coerced call
validates. It applies to any object-typed field at any depth, not only
`continuation`.
- The OpenCode MCP bridge and the runner tool bridge use
`admitToolInput`. They dispatch the coerced arguments, and they send one
`paperclip.runner.tool-input-coerced/v1` diagnostic per accepted
coercion. The OpenCode driver already records bridge diagnostics as run
events. The diagnostic names the tool and the field paths. It never
contains the values. The runner tool bridge has a new optional
`onDiagnostic` option.
- When parsed text is still invalid, the rejection describes what is
wrong inside the parsed object, for example `/options/mode must be
string`.
- The repeated-rejection guard now says "Required fields X did not
arrive" only for missing required fields. For fields with the wrong
type, it says "Every field you sent arrived, but one has the wrong type:
continuation must be an object or null, not text".

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/drivers/opencode/mcp-bridge.test.ts
src/drivers/runner-tool-bridge.test.ts` (63 tests pass).
- Red/green: with the source changes reverted and the new tests kept, 5
tests fail: coercion acceptance and its run event (both bridges), no
coercion for text fields or non-object text, the wrong-type guard
wording, and the missing-field guard wording. With the change, all of
them pass.
- `npx tsc --noEmit -p packages/paperclip-runner` passes.
- A narrow paid run of `questions-and-approvals.multi-part-question`,
`runs-adapters.inspect-failed-run` and `budgets-costs.explain-hard-stop`
ran on a branch that combines this PR with the two sibling cohort-13
PRs. The results are in a PR comment.

## Risks

- Low risk. Coercion runs only after a call fails validation, and the
call is accepted only if the coerced form validates. A field that allows
text is never parsed, so a valid string never changes meaning.
- Previous-release story (upgrade-safe run state): this change adds no
persisted run, session, launch or pack state. It changes only how one
in-flight tool call is admitted. Saved state carries forward unchanged.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:06:37 -06:00
Nicky LeachandClaude Fable 5.2 0ebe6d846d perf(runner): keep a quiet Claude provider between turns of a run (#21293)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner runs Claude Code through the ACPX sidecar. The
Runner stops and starts the sidecar and Claude at each turn boundary.
> - The reason was that a late tool call from an earlier turn could run
in a later turn. #21278 removes that reason for Claude: the sidecar
refuses a call that the active turn did not announce.
> - A second reason remains. Claude Code works by itself between
prompts. When a background task finishes, Claude starts a cycle of its
own. If the process stayed alive with a live background task, that cycle
could start between turns or mix with the next turn.
> - So the Runner can keep the process only when it knows that Claude
has no background work.
> - This pull request adds that knowledge and uses it. After a Claude
turn that completed with no background work, the Runner keeps the
process for the next turn of the same run.
> - The benefit is that a later turn of the same run starts with no
provider restart. This is also the base for the next step, which lets a
run continue when background work finishes.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is stacked on #21278. Its base branch is
`fix/bind-claude-tool-calls-to-turn`. Refs #19912 (design), #12419
(restart at turn start), #20145 (restart at park) and #20190 (restart at
turn end).

**What existing behavior does this improve?**

The provider restart between turns of the same run, for Claude through
ACPX.

**Subsystem affected**

Paperclip Runner: the ACPX sidecar (TypeScript) and the ACPX provider
session in `runner-core` (Rust). Claude only.

**Current behavior**

- When a completed turn settles, the Runner restarts the sidecar and
Claude (`restart_session_after_turn`, #20190).
- If that restart did not run, the next turn restarts them at its start
(`start_turn`).
- The Runner does not know if Claude has background tasks.

**Proposed behavior**

- The sidecar asks `claude-agent-acp` for its async task reports. This
is an optional capability of the pinned `claude-agent-acp` 0.73.0. The
patch does not change.
- A tracker reads those reports and the id of each background Bash
command. It reports `{ live: 0 }` only when it knows that this process
never had a background task and that Claude did not act between prompts.
In all other cases the state is unknown.
- The sidecar adds `backgroundTasks` to the terminal event of a
completed Claude turn, and to `session.snapshot`.
- The Runner keeps the process between two turns only when all of these
are true:
  - the agent is Claude;
  - the turn completed (not failed, cancelled or interrupted);
  - the terminal event reported `backgroundTasks: { "live": 0 }`;
  - each existing condition of the turn-end restart is true;
- before the next turn, a new `session.snapshot` reports the same
identity and run, no active turn, no pending work, and `{ "live": 0 }`;
  - the settled-turn ledger and the tool replay ledgers are not full.
- In each other case the Runner restarts as before.
- A kept process keeps its tool call-id tombstones and its settled turn
identities.

**Reason and benefit**

- A turn that follows a quiet turn in the same run starts without a
restart.
- The decision is safe by default: unknown state means a restart.

## What Changed

TypeScript (`packages/paperclip-runner/src`):

- `drivers/acpx/claude-background-tasks.ts` (new):
`ClaudeBackgroundTaskTracker`. It reads raw ACP messages. It reports
unknown when the capability is not negotiated, when a task finished,
when a live task is not a shell command, when an update is malformed,
when a limit is reached, when Claude acts with no client request in
flight, or when Claude calls `CronCreate` or `ScheduleWakeup`.
- `drivers/acpx/provider-failure.ts` and `acpx-runtime-adapter.ts`:
Claude sessions also advertise the `asyncTasks` capability. The raw ACP
observer gives each message to the tracker. ACPX 0.22 drops the
`async_task_*` updates before the turn event stream, so only the raw
observer sees them.
- `drivers/acpx/runtime-host.ts`: `backgroundTaskState()`.
- `cli/acpx-runtime-sidecar.ts`: the terminal event of a completed
Claude turn carries `backgroundTasks` in the same payload as
`goalAtTerminal`, with the same size limit. `session.snapshot` carries
it when no turn is active.
- `drivers/acpx/claude-tool-use-admission.ts`:
`hasRoomForAnotherTurn()`. When the admitted-id ledger cannot hold one
more full turn, the sidecar reports unknown, so the Runner restarts
before the ledger is full.

Rust (`packages/paperclip-runner/runner/crates/runner-core`):

- `acpx_provider_session.rs`: the session records a quiet turn from the
terminal event. `start_turn` continues in the same process only under
the conditions above, and then uses `prepare_turn`, which keeps the
tombstones.
- `acpx_provider_generation.rs`: `can_continue_after_turn`,
`continue_after_quiet_turn`, and the snapshot check
`background_work_is_quiet`.
- `acpx_park_preparation.rs`: the turn-end restart does not run for a
quiet Claude turn.
- `acpx_provider_backend.rs`: `session.reconciled` reports a replaced
process only when the turn start replaced it.
- `bin/fake-acpx-sidecar.rs`: new option `--background-tasks` for tests.
- `runner/DURABLE_TRANSPORT.md`: the rule for a kept sidecar.

Contract (`packages/paperclip-runner/protocol`):

- The `backgroundTasks` report is `$defs.backgroundTasks` in
`provider-schemas/acpx-sidecar.schema.json`: an object with only `live`,
a non-negative integer. The schema validates it on
`runtime.turn_terminal`.
- `scripts/generate-acpx-sidecar-contract.mjs` generates the TypeScript
type and reader and the Rust reader from it. The tracker, the sidecar
and the Rust session use the generated code.
- `fixtures/provider-boundary/acpx-background-tasks.json` has shared
test documents. The TypeScript reader, the Rust reader and the schema
are tested with the same accepted and refused shapes.
- This changes the contract digest of the provider pack
(`prp.acpx-sidecar` and `rust`). Older image packs are not reused. No
provider `commandDigest` and no profile version changes.

Not changed: park and run attachment. A process that ran a turn is still
restarted at park, and only a process that ran no turn is bound to the
next run. Other ACPX agents and Codex do not change. The
`claude-agent-acp` patch does not change.

## State saved by the previous release

- This pull request adds no persisted state. The three new fields of the
session are in memory only.
- A terminal event or a snapshot without `backgroundTasks` means
unknown. An older sidecar therefore gets the restart, as before.
- An older Runner with a new pack ignores the field and restarts at each
turn, as before.
- After a Runner restart, the in-memory state is gone. The next turn
restarts the provider, as before. A test covers this.

## Verification

From `packages/paperclip-runner`:

- `npx vitest run src/drivers/acpx/claude-background-tasks.test.ts
src/drivers/acpx/acpx-runtime-adapter.claude-background-tasks.test.ts
src/drivers/acpx/acpx-runtime-adapter.claude-tool-use.test.ts
src/drivers/acpx/claude-tool-use-admission.test.ts
src/cli/acpx-runtime-sidecar.test.ts
src/drivers/acpx/acpx-runtime-adapter.test.ts
src/drivers/acpx/runtime-host.test.ts`: `7 passed (7)` files, `363
passed` tests.
- `npx tsc --noEmit -p tsconfig.json` and `npx tsc --noEmit -p
tsconfig.test.json`: no errors.
- `pnpm run typecheck:typescript` (it includes the contract checks): no
errors. `node --test test/acpx-sidecar-contract.test.mjs`: 17 passed.
- `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`: no
changes.
- `cargo test -p paperclip-runner-core --test
acpx_quiet_turn_continuation --test acpx_park_preparation --test
acpx_park_readiness --test acpx_provider_turns --test
acpx_provider_session --test acpx_warm_attachment --test
native_provider_backend`: 4, 33, 8, 31, 12, 6 and 47 passed. No test
failed.
- `cargo test -p paperclip-runner-core --lib`: 637 passed, 4 ignored.
- Mutation check: when the Rust code accepts each live count as quiet,
two tests in `acpx_park_preparation` fail.

New tests:

- **Real ACP runtime**
(`acpx-runtime-adapter.claude-background-tasks.test.ts`), with the
pinned `claude-agent-acp`, the pinned Claude Code client and a local
fake Anthropic API. A foreground Bash turn reports `{ live: 0 }`. A
background Bash in the same process reports `{ live: 1 }` at the
terminal event, and unknown after Claude ran its own cycle for the
finished task. Each turn has its own usage receipt. One process serves
all turns.
- **Tracker unit tests** (13): spawn, completion, failure, stop, pause,
a task that is not a shell, a subagent, no negotiation, malformed
updates, activity between prompts, scheduling tools, session load
replay, limits.
- **Sidecar tests**: the terminal event carries the state for Claude and
not for Codex. A full ledger reports unknown.
- **Rust** `tests/acpx_quiet_turn_continuation.rs` (4) and
`tests/support/acpx_quiet_turn_restart.rs` (7):
- a quiet Claude turn continues in the same process, and a replay of a
settled call id is refused;
  - live, unknown and malformed reports restart;
- background work that appears after the terminal event restarts at turn
start;
  - failed, cancelled and interrupted turns restart;
  - Codex with a quiet report restarts;
  - an active goal restarts;
  - 4,096 kept turns, then the next turn restarts;
- park after a kept process still prepares a new process, and the next
run attaches to it;
  - a Runner restart after a kept process resumes in a new process.
- The existing restart tests did not change. The fake sidecar sends no
report by default, so they are the tests for the unknown case.

Live run with a real Claude account, not in CI: the real ACP runtime
(ACPX 0.22, the patched `claude-agent-acp`, the pinned Claude Code
client), a real tool bridge, and the real Anthropic API with model
`claude-sonnet-5-5`, on the code at the top of this stack. A turn that
started one background shell reported `{ live: 1 }`. A turn that started
two reported `{ live: 2 }`, then `{ live: 1 }` and `{ live: 0 }` as each
shell finished. A turn in which Claude waited for its background shell
in the foreground reported the unknown state.

Not run: server integration tests (the fake sidecar there sends no
report, so their process counts do not change), Linux, and a live run
through the real sidecar process and runnerd.

## Risks

- **All Claude sessions now negotiate the `asyncTasks` capability.**
`claude-agent-acp` then sends `async_task_*` updates and adds a marker
to a background Bash update. ACPX drops the updates. No other behavior
of `claude-agent-acp` changes, from a read of its code.
- **Park after a kept process pays the restart again**, as it did before
#20190. With real Claude that restart was about 1.5 s.
- **A small window remains** between the snapshot and the prompt of the
next turn. Claude can start a cycle of its own in that window only
without a background task, because a process that had a background task
is not kept.
- **Activity between prompts is found only when no client request is in
flight.** A request that never gets a response would hide such activity.
- **Background watchers cannot be told apart from other tasks**, so each
task that is not a shell command makes the state unknown.
- **Each kept turn adds one `session.snapshot` request.**
- A multi-turn run with real Claude now uses one provider process. This
path has tests with the fake sidecar and with the real ACP runtime. It
has no test with the real sidecar host.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change, reviewed the diff, and ran the TypeScript and Rust verification
again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), investigated the signals
and wrote the change and the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 12:54:43 -08:00
7773b52f05 fix(codex): preserve sandbox certificate trust in Codex shell commands (#20364)
<!-- Written in ASD-STE100 style: short sentences, active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner and the Codex adapters run agent shell commands
inside a sandboxed Codex session with a strict environment allowlist
> - The controller builds `shell_environment_policy.include_only` before
the sandbox injects certificate-trust variables such as `SSL_CERT_FILE`
and `GIT_SSL_CAINFO`
> - The allowlist drops the injected variables, so managed `git` and
`gh` commands cannot verify HTTPS against a runtime-provided CA
> - This pull request allows the certificate-trust variable names
through every environment policy stage, from the Rust process supervisor
to the shell allowlist
> - The benefit is that managed `git` and `gh` work again in sandboxes
that require a runtime CA, while provider-credential isolation and the
bounded allowlist stay in place

## Linked Issues or Issue Description

- Fixes: #20019
- Refs #20038 — community pull request for the same defect. This change
reviews the approach independently. The commit credits the community
author as co-author.

## What Changed

- Add `CERTIFICATE_ENVIRONMENT_KEYS` (`SSL_CERT_FILE`, `SSL_CERT_DIR`,
`GIT_SSL_CAINFO`, `NODE_EXTRA_CA_CERTS`, `DENO_CERT`,
`REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`) to
`packages/paperclip-runner/src/configured-environment.ts`
- Native Codex shell policy (`codex-security-config.ts`): always put the
certificate names in `include_only`; set inheritance to `"all"` with
`ignore_default_excludes=true`, bounded by the explicit allowlist
- Pass the certificate names through the app-server environment filter,
the ACPX spawn environment filter, the ACPX runtime sandbox
`include_only`, the runner platform environment keys, and the Rust
process supervisor base key list
- Legacy and ACPX-engine Codex shell allowlists
(`packages/adapter-utils/src/codex-shell-environment.ts`): always
include the certificate names
- Add a synthetic private-CA HTTPS journey test: a real `git ls-remote`
fails TLS verification without the injected CA and succeeds with it,
through the policy-filtered child environment
- Document the contract in `doc/sandbox-shell-environment.md` and link
it from `doc/DEVELOPING.md`

## Verification

- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/drivers/codex/codex-security-config.test.ts
src/configured-environment.test.ts
src/drivers/codex/codex-shell-certificate-trust.test.ts
src/drivers/codex/codex-app-server-driver.context-control.test.ts
src/drivers/acpx/runtime-sandbox.test.ts` — 104 tests pass
- `pnpm --filter @paperclipai/adapter-utils exec vitest run
src/codex-shell-environment.test.ts src/acpx-engine/execute.test.ts` —
270 tests pass
- `pnpm --filter @paperclipai/adapter-codex-local exec vitest run
src/server/execute.test.ts` — 10 tests pass
- Codex driver lifecycle, conformance, recovery, and boundaries suites
pass (103 tests); environment-builder suites pass (307 tests)
- Red proof: with only the source fix reverted, the new regressions
fail. The journey test shows `GIT_SSL_CAINFO` is undefined in the shell
environment, and `git` fails TLS verification against the private-CA
fixture
- `cargo check` passes for `paperclip-runner-core`; `rustfmt --check`
passes for the edited Rust file
- The journey test needs `git` and `openssl`; it skips when either tool
is absent

## Risks

- The native shell policy now always sets
`shell_environment_policy.inherit="all"` with
`ignore_default_excludes=true`. Before, it set `"none"` when no
credentials or task keys were projected. The explicit `include_only`
allowlist still bounds every inherited name, so no new secret names can
cross
- The allowlist grows by seven non-secret certificate-trust names. Their
values point at CA files. The sandbox filesystem policy still controls
access to those files; this change grants no filesystem access
- No persisted run, session, or launch identity changes shape. Sessions
saved by the previous release resume unchanged; the new policy applies
when the next launch builds its arguments
- Provider credentials keep their existing exclusions at every stage

## Model Used

- Claude Fable 5 (Anthropic, model ID `claude-fable-5`), extended
thinking, agentic tool use, run through Claude Code. The implementation
was written and reviewed independently against the community design in
#20038; the community author is credited as co-author in the commit.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (pending: checks run after this
PR opens)
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(pending review)
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: everlastingcrown <everlastingcrown@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-11-13 14:13:01 -06:00
Nicky LeachandClaude Fable 5.2 7866e9ef05 fix(acpx): admit a Claude tool call only for a tool use its turn announced (#21278)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner runs Claude Code through the ACPX sidecar. Claude
calls Paperclip tools through a Runner-owned MCP endpoint, the tool
bridge.
> - The bridge identifies a call by its JSON-RPC request id. That id
does not say which turn sent the call. The sidecar binds a call to the
turn that is active when the call arrives.
> - Because of this, the Runner stops and starts the Claude process at
each turn boundary. A late call from an earlier turn then cannot run in
a later turn. This restart also stops all background work that Claude
started.
> - Claude Code already sends the missing identity. Each MCP
`tools/call` has `_meta["claudecode/toolUseId"]`. It is the id of the
ACP `tool_call` that the provider announces first, inside one turn.
> - This pull request uses that id. The sidecar admits a Claude tool
call only if the active turn announced its tool use.
> - The benefit is that a call from an earlier turn is refused by its
identity. This is the first step to keep the Claude process alive
between turns. This pull request does not change the restart.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

Refs #19912 (the design for turn-scoped ACPX MCP authority), #12419 (the
restart at turn start) and #20190 (the restart at turn end).

This pull request is the bottom of a stack. #21274 is merged, so the
base branch is `master`. It uses no code from #21274. The stack order
lets a later pull request remove the direct gateway path after both.

**What existing behavior does this improve?**

How the Runner decides which turn a Claude tool call belongs to.

**Subsystem affected**

Paperclip Runner: the tool bridge and the ACPX sidecar, for the Claude
provider only.

**Current behavior**

- The bridge keys a call by the JSON-RPC request id
(`runner-tool-bridge.ts`).
- The sidecar gives the call to the active turn (`waitForTool` in
`acpx-runtime-sidecar.ts`).
- After `session/cancel`, Claude Code does not stop an MCP request that
is in flight. A test run showed two calls from two turns live at the
same time on one MCP session. Request ids cannot tell them apart.

**Proposed behavior**

- The bridge can read a provider tool-use id from `params._meta` and
give it to the handler. It keys replays by that id.
- For Claude, the sidecar admits a call only when the turn event stream
of the active turn contains the ACP `tool_call` with the same id, and no
earlier call used that id.
- The announcement and the call use different channels. If the call
arrives first, the sidecar waits for the announcement for a maximum of 5
seconds. The wait stops when the turn ends or the call is cancelled.
- A call with no id, an id from an earlier turn, an id that was never
announced, or an id that was used before gets a tool error. There is no
fallback.
- The call id that the sidecar sends to runnerd is `tool-use:<id>` for
Claude.

**Reason and benefit**

- A late call from an earlier turn cannot run in a later turn. The
identity of the call prevents this, not only the process restart.
- A later change can stop the restart between turns.

## What Changed

- `packages/paperclip-runner/src/drivers/runner-tool-bridge.ts`
- New option `providerToolUseIdMetaKey`. Without it the bridge does not
change.
- With it, the bridge reads `params._meta[key]`. It refuses a value that
is not a bounded string. It gives a valid value to the handler as
`providerToolUseId`.
- The replay key is the tool-use id. A retry with a new request id
replays the first result. The same id with different arguments is
refused.
  - `notifications/cancelled` still finds the call by its request id.
-
`packages/paperclip-runner/src/drivers/acpx/claude-tool-use-admission.ts`
(new): `ClaudeToolUseAdmission`.
- `observe` records an id only from an event with `type === "tool_call"`
and `tag === "tool_call"`. A `tool_call_update` does not count, so a
late update about an old tool use cannot admit it.
- `admit` applies the rule above. An id is used when it is admitted.
Limits: 4,096 announcements for each turn and 8,192 admitted ids. At a
limit, new calls are refused.
- `packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts`
- The sidecar creates the admission object only for `agent === "claude"`
and sets the bridge option only for Claude.
- `pumpTurn` gives each turn event to `observe`. `waitForTool` calls
`admit` and then checks the turn and run again.
- The internal `get_task_context` call for a completion continuation
does not go through admission.
- The Claude call id does not use the per-process namespace, because a
tool-use id is unique.
- No change to Rust, the sidecar contract, the `claude-agent-acp` patch,
the server, or restart, park and turn lifecycle code. Other ACPX agents
and Cursor Cloud use the same code path as before.

Why 5 seconds: the measured time between the announcement and the call
is 1.1 ms to 14.7 ms. The limit must also include queue and event-loop
delay. 5 seconds is far above that, and a call that cannot be attributed
still gets a fast answer.

## Verification

Commands, from `packages/paperclip-runner`:

- `npx vitest run
src/drivers/acpx/acpx-runtime-adapter.claude-tool-use.test.ts
src/drivers/acpx/claude-tool-use-admission.test.ts
src/drivers/runner-tool-bridge.test.ts
src/drivers/runner-tool-bridge.claude-tool-use.test.ts
src/drivers/runner-tool-bridge.claude-schema.test.ts
src/cli/acpx-runtime-sidecar.test.ts
src/drivers/acpx/acpx-runtime-adapter.test.ts
src/drivers/cursor-cloud/cursor-cloud-driver.test.ts`: `8 passed (8)`
files, `384 passed` tests.
- `npx tsc --noEmit -p tsconfig.json`: no errors.

New tests:

- **Real ACP runtime** (`acpx-runtime-adapter.claude-tool-use.test.ts`).
It uses the real `openCodexAcpxRuntime`, ACPX 0.22, the pinned and
patched `claude-agent-acp`, the pinned Claude Code client and a real
tool bridge, against a local fake Anthropic API.
- Turn 1: two parallel tool uses are admitted as `tool-use:<id>`. The
model receives both results. The turn completes.
- Turn 2 in the same session: a replay of a turn 1 id returns the cached
result and does not dispatch again. The same id with different arguments
is refused. `admit` refuses the old id.
- In five runs each call arrived 6.9 ms to 13.2 ms after its
announcement. No call waited.
- **Qualification** (`runner-tool-bridge.claude-tool-use.test.ts`). The
pinned Claude Code client sends `_meta["claudecode/toolUseId"]` on each
`tools/call`, with the id that the API issued. This test must pass
before a Claude Code version change.
- **Admission unit tests** (19): announced then called, called then
announced, never announced, turn end during the wait, cancel, an id from
the previous turn, reuse, two concurrent calls with one id, two turns
with live calls, a subagent call, a bad id.
- **Sidecar process tests** (2), with a fake host, for Claude and for
Codex. The Codex run shows no change.
- **Bridge unit tests** (5).

Mutation checks:

- Change the announcement predicate so that it matches nothing: the real
ACP test and one unit test fail.
- Remove the bridge option in the real ACP test: the test fails.
- Count only `tool_call_update`: the unit test "counts only an opening
ACP tool_call event as an announcement" fails.

Live run with a real Claude account, not in CI: the real ACP runtime
(ACPX 0.22, the patched `claude-agent-acp`, the pinned Claude Code
client), a real tool bridge, and the real Anthropic API with model
`claude-sonnet-5-5`, on the code at the top of this stack. In three
sessions, 8 Paperclip tool calls were admitted and none was refused.
This includes calls in a prompt turn, in a cycle that Claude started
itself, and in a later turn of the same process. Each call arrived 0.8 s
to 1.7 s after its announcement, so no call waited for a late
announcement.

Not run: a turn through the real sidecar process with the real host.
That needs a qualified provider installation, the runtime sandbox and
the provider lifetime lease. Not run: the Linux builds of the pinned
client and a test under load. Not run: `pnpm verify` and the full suite.
CI supplies that coverage.

## Risks

- **This changes each Paperclip tool call that Claude makes.** If the
announcement does not reach the turn event stream in production, the
sidecar refuses the call after 5 seconds. The real ACP test covers the
stream of ACPX 0.22 with the pinned provider. It does not cover the real
sidecar host.
- **The 5 second limit is not tested under load.** A slow host can delay
the announcement. The call is then refused, and Claude sees a tool
error.
- **A Claude Code version that stops sending the id breaks all Claude
tool calls.** The qualification test finds this before the version
changes.
- **The admitted-id ledger is in sidecar memory.** Today the sidecar
restarts at each turn boundary, so the ledger is small. The limit of
8,192 ids refuses new calls when it is full.
- **Subagent calls and calls from a cycle that Claude starts itself**
are covered by unit tests and fake-host tests, not by the real ACP
stream.
- **An admitted call that is in flight when its turn ends** is refused
by the existing turn cleanup. Claude Code can leave the HTTP request
open. The bridge stops it only on its own cancel or close, as before.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change, reviewed the diff, asked for the real ACP runtime test, and ran
the verification and a mutation check again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 11:56:42 -08:00
DottaandClaude Opus 5.5 38821e061f perf(runner): serialise only changed journal records for a checkpoint (#21437)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every running agent, and its main thread
limits how many agents can run at the same time
> - Each agent has a controller journal. The controller saves a
checkpoint of this journal after each control event and before each
frame it sends
> - Each checkpoint converted every command and every event of the
journal to JSON again, also when only one record changed. The cost grew
with the journal length (up to 500 commands and 4,096 events)
> - This pull request keeps the JSON of each record and uses it again
while the record does not change
> - The benefit is that a checkpoint converts only the records that
changed, so the main thread does less work for each agent

## Linked Issues or Issue Description

Refs #21411 (this change sits on top of its checkpoint queue).

**What existing behavior does this improve?**
The controller journal checkpoint in
`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`.
Each flush calls `captureCheckpoint`, which runs `JSON.stringify` on
every command and every recovery event of the journal.

**Current behavior**
A flush after one new event converts the full journal to JSON. With
4,096 events, one flush made about 5 MB of new JSON strings and used
about 14 ms of CPU on the event loop. A hosted CPU profile with about 50
agents showed about 3% of the main thread in this path.

**Proposed behavior**
A flush converts only the records that changed since the last flush. The
other records use the JSON that the controller made before. The saved
records are byte-identical to the records of the full conversion.

**Reason and benefit**
The target is 1,000 agents in one controller process. Checkpoint CPU
must not grow with journal length. After this change, one flush with
4,096 events makes about 2.5 KB of new JSON and uses about 2 ms of CPU.

## What Changed

- Add `CheckpointRecordCache` to `durable-prp-control-plane.ts`. It
keeps the JSON and the part name of each command and event record, keyed
by the record object. It uses the JSON again only when the record has
the same keys, in the same order, with the same values (`===`). Commands
change some fields in place (`status`, `result`, `controllerSeq`), and
this comparison finds these changes.
- `captureCheckpoint` accepts the cache as an optional fourth argument.
Only `setCheckpointPersistence` and `flushCheckpoint` pass it. All other
callers (decode, legacy import, bootstrap inspection) do the full
conversion as before.
- `checkpointChanges` uses loops instead of array copies. Records that
did not change keep the same string, so the comparison is by reference.
- `queueCommand` stores a copy of the command payload
(`structuredClone`). The journal must own its nested objects, because
the cache compares nested objects by reference.
- `durableRecoveryInternals` exposes `captureCheckpoint` and
`createCheckpointRecordCache` for tests.
- Regions changed: `checkpointChanges` and the new cache above
`captureCheckpoint`; `captureCheckpoint`; `durableRecoveryInternals`;
one field of `DurablePrpControlPlane`; the `captureCheckpoint` calls in
`setCheckpointPersistence` and `flushCheckpoint`; the `payload` field in
`queueCommand`. The `#event` path is not changed.
- No change to what a checkpoint saves, when a flush is awaited, or the
order of event commit, acknowledgement and checkpoint. The "at most one
waiting checkpoint" behavior from #21411 is not changed.

## Verification

- New `durable-prp-checkpoint-records.test.ts`:
- A journal with 499 commands and 4,000 events gets one new command. The
test counts `JSON.stringify` calls on journal records during the flush.
Before: 4,499. After: 0 calls on existing records, at most 2 in total
(the size check of the new command and its checkpoint record). The delta
contains only `command:500`. The test then restores a new controller
from the checkpoint and compares its records with a full conversion
without the cache. I saw this test fail on `origin/master`.
- A property-style test with 5 seeds and 600 random steps each: add,
settle and remove commands (with `controllerSeq` renumbering), add,
replay and evict events, streaming and usage events, `structuredClone`
and spread copies of the state, header counters, the same value written
again, and a key-order change. After each step, the records with the
cache must be byte-identical to the records without the cache, in the
same order. I checked that this test fails when the field comparison is
removed (cache by object only).
- Existing `durable-prp-control-plane.test.ts` passes. On this loaded
machine, some 5-second timing tests time out with and without this
change (they also time out on `origin/master`), and pass when I run them
alone.
- Benchmark: `PAPERCLIP_CHECKPOINT_FLUSH_BENCHMARK=1 pnpm exec vitest
run src/control-plane/durable-prp-checkpoint.benchmark.test.ts` (from
`packages/paperclip-runner`). Each case commits one new event and
flushes with a no-op save, 200 times. CPU time is noisy (load average
about 40), the byte and record counts are exact.

| Journal | CPU per flush, before | CPU per flush, after | JSON bytes
per flush, before | after | Records converted per flush, before | after
|
| --- | --- | --- | --- | --- | --- | --- |
| 10 commands, 10 events | 207 µs | 64–80 µs | 20,975 | 2,465 | 20 | 1 |
| 500 commands, 10 events | 1,874 µs | 505–511 µs | 375,068 | 2,466 |
510 | 1 |
| 10 commands, 4,096 events | 14,013 µs | 1,655–2,343 µs | 5,174,786 |
2,473 | 4,106 | 1 |

- `pnpm check:module-boundaries` passes. `tsc -p
packages/paperclip-runner/tsconfig.json --noEmit` passes.

## Risks

- The cache assumes that nested `payload`, `result` and `envelope`
objects of journal records are not edited in place after they enter the
journal. I checked the module: results are stored with
`structuredClone`, envelopes with `structuredClone`, and events are
replaced, not edited. This change also copies the payload in
`queueCommand`. If a later change edits a nested object in place, the
checkpoint could keep old JSON. The property test shows top-level edits
are found.
- Some per-record work is still proportional to the journal (a Map of
part names and a field comparison for each record). It is about 0.4 µs
for each record. Removing it needs dirty tracking in every state
mutator, which this change does not do.
- No change to the saved format, so old and new servers read the same
checkpoints during a deploy overlap.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edits) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:21:51 -06:00
Dotta 1e33415f67 test(runner): run Kimi layout suites once in prep (#20487)
<!-- All text in Simplified Technical English. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Runner package has node:test suites that verify the pinned Kimi
Code native runtime layout.
> - `test:typescript:prep` selects each Kimi layout suite directly. The
same command also selects the `build-verified-provider-entrypoints`
aggregator. The aggregator imported the same four Kimi suites.
> - One command therefore registered every Kimi layout test twice.
Commit b866fa4bd (#18028) added both routes in the same change. The
duplicate has no second contract: both routes run the same files,
assertions, and fixtures in the same required run.
> - This pull request removes the aggregator imports. The direct prep
selection stays. Each Kimi suite now runs once per command.
> - The benefit is less test work and less CI time in each prep run,
with the same assertions.

## Linked Issues or Issue Description

No public issue exists. Problem description:

**What existing behavior does this improve?**

The `test:typescript:prep` command in
`packages/paperclip-runner/package.json`. It selects test files for
`pnpm test:typescript` and `check:protocol` (release verification).

**Current behavior**

The command passes `build-verified-provider-entrypoints.test.mjs` and
the four Kimi suites to one `node --test` run. The aggregator file also
imports the four Kimi suites. Each Kimi test file then registers its
tests twice: once inside the aggregator process, and once in its own
process. Local measurement on the exact prep selection: 81 test
registrations, 71 pass, 10 skip, about 6.6 seconds.

**Proposed behavior**

The aggregator keeps its own two tests and its imports of suites that
prep does not select (`provision-pi-package`, the four Antigravity
suites). The four Kimi suites run only through their direct prep
selection. Same selection of the aggregator, same files, same
assertions. Local measurement after the change: 62 test registrations,
56 pass, 6 skip, 0 fail, about 4.6 seconds. The removed 19 registrations
are exact duplicates; each Kimi test still runs once.

**Reason and benefit**

Double registration wastes CI and developer time on each prep run and
makes test counts misleading. One run of each suite keeps the same
regression proof.

**Breaking changes**

None. No production code changes. The suites and their assertions are
unchanged.

## What Changed

- Removed the four Kimi suite imports from
`packages/paperclip-runner/scripts/build-verified-provider-entrypoints.test.mjs`
(`materialize-kimi-distribution`, `provision-kimi-package`,
`patch-kimi-runtime`, `kimi-metadata-probe`).
- Added a comment that says the Kimi suites are selected directly by
`test:typescript:prep`, so this aggregator must not import them again.
- No changes to `package.json`, routing, or the suites themselves.

## Verification

- Checked the routing: `test:typescript:prep` in
`packages/paperclip-runner/package.json` lists all four Kimi suites
directly, so each suite stays selected with multiplicity one after this
change. The aggregator is only run through that same command; no
workflow or script runs it alone.
- Baseline (master `233f61564`): `node --test` with the exact prep
selection of the aggregator plus the four Kimi suites -> 81 tests, 71
pass, 0 fail, 10 skipped, 6.6 s.
- After the change (this head): same command -> 62 tests, 56 pass, 0
fail, 6 skipped, 4.6 s. 19 exact duplicate registrations removed; the
Kimi suites still run and pass once each.
- Timing is a single local sample; treat the speed difference as
indicative, not proven. The structural change (19 duplicate
registrations removed) is exact.
- Not run: full `pnpm -r typecheck`, full workspace test suite, Rust
checks. A filtered dependency install was used because the workspace
disk is 10 GB and a full workspace install exceeds it. Required PR CI
supplies the typecheck, build, and vitest coverage for this head.

## Risks

Low risk. The aggregator still loads and its own two tests pass. If a
future lane runs the aggregator alone, the Kimi suites would not run
there; today no such lane exists, and the prep command still selects
them directly.

## Model Used

- Provider: Z.ai, model GLM (assisted by Paperclip's Test Gardener
agent).
- Exact model ID: `openrouter/z-ai/glm-5.3`.
- Reasoning/thinking mode: standard chain-of-thought; no extra thinking
mode.
- Capabilities: tool use (shell, file edit, GitHub CLI), repo test
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
2026-11-13 12:55:31 -06:00
DottaandClaude Opus 5.5 d9b977baf2 fix(runner): answer a live Runner's commands while it replays its backlog after a reconnect (#21442)
**DO NOT MERGE until the coordinator confirms the Daytona validation.**

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox is driven by a Runner process. The
server talks to it over PRP and saves the controller state in a
checkpoint.
> - After a release swap, the incoming server reconnects to Runners that
kept working while no server was attached. Each Runner first replays the
events it built up.
> - The server answered commands one at a time and only after every
event queued ahead of them. It wrote one checkpoint and one
acknowledgement per event. A recovery read sent at reconnect therefore
timed out at 30 s behind the replay, although the Runner was alive and
had answered.
> - A tool call that the previous server started and never answered
blocked every later command, and a failed close was replayed forever.
> - This pull request lets read and barrier answers pass the replay,
extends a wait while the replay still commits, acknowledges a replayed
burst once per batch, delivers every pending command at attach, settles
an orphaned tool call at attach, and retries a close whose drain timed
out.
> - The benefit is that a live Runner is reached after a late reconnect
instead of being declared unanswered and then replaced, which lost its
turn.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13 (06:05Z to 09:45Z). For each
run that crossed a swap we measured the gap from the last Runner output
to the moment the new server recorded the Runner's process identity:

- gap under 150 s: 0 of 9 runs failed
- gap over 420 s: 14 of 15 runs failed
- median gap: 180 s for runs that resumed, 470 s for runs that failed

Authentication took under 1 s. The identity was recorded 4 to 7 minutes
later, because it is written only after the Runner answers `run.attach`,
`runner.drain` and the thread read. Those answers waited behind the
replayed event backlog. The failure codes on the first attempt were:

- `PRP command session.snapshot timed out` and
`codex_history_read_failed: thread/turns/list read failed` (all 10
`ingress` runs in the sample)
- `PRP command runner.drain timed out (runner connected; pending ahead 2
[semantic_tool.result] oldest 1023s)` on the retry

A separate class, `cleanup_quarantined` (23 failures in 3 hours, no swap
involved), showed "prior session cleanup remains incomplete" on every
retry, because a close that failed once kept replaying the same
rejection.

A live Daytona capture showed the stale `semantic_tool.result`: Codex
waited for the result of an exec tool call that the stopped server
owned. It was not an unacknowledged frame.

**Expected behavior**

A live Runner answers recovery commands within its deadline after a late
reconnect. A tool call left by an earlier controller is settled once
(with its durable receipt, or as interrupted with an unknown outcome,
never executed again). A close that failed only because its journal
drain timed out reaches its real outcome on the next close.

**Steps to reproduce**

1. Start a control plane with a commit callback that takes 50 ms per
event.
2. Connect a Runner that replays 1,000 events, then send
`session.snapshot`.
3. On master the snapshot times out at its 30 s deadline. The new test
`answers a command issued during a 1,000-event replay within its
deadline` does this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

All changes are controller-side TypeScript under
`packages/paperclip-runner/src`.

- `durable-prp-control-plane.ts`: a `command_result` for
`session.snapshot`, `session.goal.get`, `runner.drain` or
`semantic_tool.result` that reports completed or rejected is applied
when it arrives, not after the events queued ahead of it. Failed
results, lifecycle, attachment and turn commands keep their place.
- `durable-prp-control-plane.ts`: the welcome carries every consecutive
pending command (up to 32, within the existing byte bound). A lifecycle
command or `run.attach` that is not first is never batched.
- `durable-prp-control-plane.ts`: an ordinary event inside a replayed
burst is committed at once but checkpointed and acknowledged once per
batch (when the queue drains, when a critical event arrives, or after 64
events). An ACK is still sent only after the checkpoint that contains
its cursor. The owner fence in `flushCheckpoint` is unchanged. Critical
events keep their own checkpoint and ACK.
- `runnerd-codex-transport.ts`: `waitForRunnerCommand` also extends a
deadline while the controller keeps committing the Runner's events. A
Runner with no committed event for 30 s still times out at the original
deadline. A Runner that keeps emitting but never answers times out after
at most 5 more minutes.
- `runnerd-codex-transport.ts`: recovery of a semantic tool input that
an earlier generation dispatched and never answered now also runs when
the controller attaches, not only after startup completes.
`paperclip_finish` and `paperclip_block` still wait for startup.
- `runnerd-codex-transport.ts`: a close that failed only because the
controller journal drain timed out is no longer cached. The next close
drains again with a fresh bound and finishes as settled or as a typed
`NativeSessionCloseUnrecoverableError`.

## Verification

Red on master, green on this branch:

- `durable-prp-control-plane.test.ts`
- `answers a command issued during a 1,000-event replay within its
deadline`
- `keeps a failed command result behind the events the Runner sent
before it`
- `delivers every command pending at attach so an earlier generation's
result cannot hold the recovery probe`
  - `never delivers a lifecycle command ahead of its turn at attach`
- `acknowledges a replayed burst once per batch, never ahead of its
durable checkpoint`
- `still acknowledges a lifecycle event inside a burst before the next
frame`
- `answers a snapshot from a Runner whose provider stays wedged until
its whole replay is acknowledged` (models the Daytona capture: the Codex
Runner stops reading provider output while it holds unacknowledged
events)
- `runnerd-codex-transport-reattach-carried-commands.test.ts`
- `keeps waiting past the deadline while the controller is still
committing the Runner's replayed events`
- `times out a Runner that keeps replaying but never answers once the
replay extension is spent`
- `does not extend a deadline for a Runner whose replay has stopped
advancing`
- `runnerd-codex-transport-orphaned-tool-at-attach.test.ts`: `settles an
earlier generation's in-flight tool call at attach even when that attach
never finishes startup`
- `runnerd-detach-durability.test.ts`: new case `close_timeout` in
`joins an output flush before controller retirement permits a successor
writer`

Measured answer latency for a snapshot sent during a replay (50 ms
commit callback): 100 / 1,000 / 10,000 events took 13.95 s / more than
900 s / did not finish on master, and 26 ms / 26 ms / 170 ms with this
branch.

Local runs on this branch: `tsc --noEmit` is clean for
`packages/paperclip-runner` and `server`.
`durable-prp-control-plane.test.ts`: 227 of 227 pass.
`runnerd-codex-transport-reattach-carried-commands`, `-adoption`,
`-orphaned-tool-at-attach` and `runnerd-detach-durability`: all pass.
`runnerd-codex-transport-settlement` and `-deadlines`: the run on this
shared machine (load 50 to 96) had 8 tests fail. A rerun of those two
files alone (load 40 to 72) still had 7 failures: one timing assertion
(`expected 7419 to be less than 3500`) and six settlement cases that
ended with `native_cleanup_maintenance_unproven` after timeouts. The set
of failing cases changed between the two runs. The CI Runner support
suite ran both files on this head and passed, so I rely on CI for them.

Runner content ID: this PR changes only controller-side TypeScript under
`packages/paperclip-runner/src`. There is no Rust, `protocol/` or
`generated/` change. The Runner content ID changes because its hash
covers that source, but existing Runner images work with this
controller. The Daytona cells below ran this controller against an image
built for an earlier content ID.

Daytona results that exist (local servers with Daytona sandboxes,
release-swap cells). They ran on a combined branch with this PR and
three others (the reattach-first PR (`fix/reattach-live-runs-first`),
the adopt-live-Runner PR (`fix/adopt-live-runner-on-retry`), and #21362,
which is merged), so they cannot be credited to this PR alone:

- G5 baseline (master d1d71a7755; 480 s delay injected on the first
identity write per run): one Claude run reproduced the staging chain:
authentication 419 ms, identity +423.5 s, attempt 1 `native session
provider recovery timed out after 900000ms`, then the retry failed with
`runner_harness_state_mismatch: resumed_not_suspended`.
- G5c (same cell, combined branch 588c6ba267): 6 of 6 tasks finished
hands-off. That chain did not recur. All 6 runs that crossed swap 2 were
adopted on the same Runner process, with no retry and no 900 s timeout.
- Caveat: G5c ran before the reattach-phase timing fix in the
reattach-first PR (`fix/reattach-live-runs-first`). The command fixes in
this PR were present in G5c except the test-only fix for the
wedged-provider test.
- G6c (12 live runs and 20 queued, two graceful swaps, no injection): 0
failed runs at either swap; 11 of 11 survivors at both. Swap 2 was slow
(see the reattach-first PR (`fix/reattach-live-runs-first`)).

## Risks

- Not proven: the Runner-side stall. A Codex Runner stops reading
provider output while no controller is attached (bounded 256-line
channel, provider stdout pipe full). This PR makes the controller
consume the replay quickly, so the stall clears in about 2 to 6 s once
the controller drives it. The durable fix is in the Runner (keep
draining provider output to the durable outbox while disconnected, save
once per ACK batch, run commands ahead of ACKs). That needs a new Runner
image and is not in this PR.
- Not proven: why a connected Runner with an empty outbox sometimes did
not answer for more than 30 s. The G5 capture points to the server
stalling past its own deadline (a `session.open` answered and
acknowledged 8 minutes before the server reported it timed out), not to
the Runner.
- Not done: cleanup takeover for a same-run retry when
`remoteCleanupScope` is absent, and for a predecessor run that shares
the warm sandbox. This PR only stops the replayed close failure.
- Time to reach a wedged Codex Runner still grows with its backlog:
about backlog x per-event commit / database concurrency (40 runs x 1,000
events x 20 ms / 8 is about 100 s).
- Behavior shift: a crash during a batched burst replays at most the
unacknowledged batch (up to 64 events). The Runner still holds those
events, and commit order is unchanged.
- A Runner that keeps emitting but never answers now fails after at most
5 more minutes instead of 30 s.
- The predecessor-retirement PR (`fix/retire-predecessor-session`)
builds on the close path changed here (`retry a close whose controller
journal drain timed out`).

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code. The fixes were written by Claude agents in the
same session family.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:34:31 -06:00
DottaandClaude Opus 5.5 df1b12ad8e fix(runner): advertise flat tool schemas and stop repeated runner-side input rejections (#21398)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents act through Paperclip tools. The runner shows each tool to
the model with an input schema and checks every call against that schema
before it runs
> - Some tool schemas had a root `oneOf`, `anyOf`, `allOf` or
`if`/`then`. `handoff_case` had no top-level `properties` at all
> - Some model hosts convert such a schema into an empty argument list.
The model then calls the tool with no arguments, and the runner rejects
the call as "Invalid tool input"
> - The repeated-failure guard from #21360 does not see these
rejections, because they happen in the runner before the call reaches
the control plane. The model sent the same rejected call again and again
> - This pull request gives every agent-facing tool a plain object
schema with named properties, checks the old cross-field rules with
named errors, and adds a runner-side guard for identical "Invalid tool
input" rejections
> - The benefit is that models on these hosts can see and send the
arguments, and a dropped argument ends in a clear correction or a stop,
not a long loop

## Linked Issues or Issue Description

**What happened?**
In a chat eval cohort with OpenCode and
`openrouter/deepseek/deepseek-v4.1-flash-1017`, the hosts Wafer and
DigitalOcean served most model calls. In the case handoff case,
`handoff_case` arrived with no arguments on all 10 calls across two
cases. The runner rejected each one with "must have required property
idempotencyKey, case, toAgentId, ...". The model's reasoning said that
the tool showed an empty property list. The model fell back to
`call_api`, but the reply waiter had already expired, so the case
failed. In the preserve-draft case, `request_human_input` was rejected 9
times in a row with the same root `then` error.

**Expected behavior**
Every tool that an agent sees has a plain object schema with named
properties, so every host can show its arguments. When the same call is
rejected twice with the same arguments, the model gets a correction that
lists the expected properties. On the third identical call, the model is
told to change the arguments or stop.

**Paperclip version or commit**
`master` at f7ddab8e94.

**Steps to reproduce**
1. Run an OpenCode agent with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, where OpenRouter routes
to Wafer or DigitalOcean.
2. Ask the agent to hand a case to another agent with `handoff_case`.
3. See `handoff_case` arrive with no arguments and get rejected on every
call.

## What Changed

- Add `providerToolSchemaShapeIssues`
(`packages/paperclip-runner/src/contracts/provider-tool-schema-shape.ts`).
It reports a root that is not `type: "object"`, has no `properties`, or
uses `oneOf`, `anyOf`, `allOf`, `not`, `if`, `then`, `else`,
`dependent*` or `$ref`.
- Add inventory tests that run this check on every agent-facing tool:
the live semantic catalog, the semantic action catalog, the Codex tool
specs, `paperclip_finish` and `paperclip_block` (runner), and the
server-only tool definitions (server). Before this change, the runner
test listed 16 entries for 8 tools: `handoff_case`,
`request_human_input`, `read_case`, `update_case`, `create_case`,
`read_workflow`, `update_workflow` and `paperclip_finish`.
- Flatten these 8 schemas. `handoff_case` is now one object with `to`
(optional, `"done"`), and `case` is one object with optional `id` and
`create`. Descriptions on the properties state which fields each variant
needs.
- Add `toolInputVariantError`
(`packages/paperclip-runner/src/contracts/tool-input-variants.ts`). It
checks the former cross-field rules and returns an error that names what
is wrong and how to send the call correctly. For example: `Invalid tool
input for handoff_case: An agent handoff requires toAgentId, phase. To
finish an existing case instead, send to: "done" with
expectedHandoffVersion and expectedIssueStatusVersion.`
- The server runs this check first in
`PaperclipRunnerToolAuthority.execute` and returns `422`. The #21360
guard therefore counts these errors too. Assigned MCP tools and
connector tools are not checked. The runner semantic dispatchers run the
same check after schema validation. The runner MCP bridges run it for
`paperclip_finish`, which never reaches the control plane.
- Add `RepeatedToolInputRejectionGuard`
(`packages/paperclip-runner/src/drivers/repeated-tool-input-rejection.ts`)
and use it in the OpenCode MCP bridge and the runner tool bridge, where
runner-side validation happens. The key is the tool, the canonical
arguments and the error text, separated per MCP session. The second
identical rejection adds the arguments that arrived and the expected
required and optional properties. The third and later identical calls
say that the call was not accepted and that the model must change the
arguments or stop and report the blocker. State is in memory, per
bridge, and capped at 64 entries.
- Regenerate `semantic-tool-contracts.json`,
`semantic-action-catalog.json`, `protocol/manifest.json` and the seeded
eval fixture.
- Add `spec/fixtures/pre-flat-tool-input-schemas.json`: the 8 strict
schemas from the previous release, for the upgrade test.

## Verification

- `cd packages/paperclip-runner && npx vitest run src/contracts
src/catalog src/protocol-actions src/semantic-tools
src/drivers/opencode/mcp-bridge.test.ts
src/drivers/runner-tool-bridge.test.ts`: all pass.
- `cd server && npx vitest run
src/services/native-runtime/tool-input-variant-admission.test.ts
src/services/native-runtime/repeated-tool-failure-guard.test.ts`: 9
passed.
- Red/green:
- The runner inventory test fails on `master` and lists the 16 entries.
It passes after the change.
- The guard test replays the recorded empty `handoff_case` call 4 times
through the real OpenCode bridge. Without the guard, the second call has
no correction and the test fails. With the guard, the second call lists
`required idempotencyKey, case, instructions; optional to, toAgentId,
phase, ...`, and the third says the call was not accepted.
- Negative controls: changed arguments get a fresh error, and a valid
call still runs once.
- Upgrade safety: `tool-input-variants.test.ts` compiles the previous
release's strict schemas from the fixture. For a corpus of valid and
invalid inputs per tool (27 for `handoff_case`), it checks that "flat
schema accepts and `toolInputVariantError` is null" gives exactly the
same result as the old schema. Each tool has accepted and rejected
inputs.
- Generated outputs: `check:semantic-contracts`,
`check:capability-contract`, `check:protocol-coverage`,
`check:capability-inventory`, `check:protocol-manifest` and the replay
golden check pass.
- Rust tests that read the generated contract: see the CI `runner`
checks.
- Paid narrow run: see the comment below.

## Risks

- Previous-release story: tool schemas are not saved run identity. The
OpenCode bridge lists tools again when a session resumes. A durable Rust
run that is already attached keeps the tool catalog it saved, because
re-attaching the same run does not replace the catalog. Every input the
old strict schemas accepted is still accepted, and every input they
rejected is still rejected, so saved calls, receipts and replays carry
forward. No session is reset, and no profile version or provider
`commandDigest` changes. The pack `contractDigest` changes because
`generated/` changed, as it does for any tool contract change.
- A rejected variant now arrives later for the semantic tools: the
server returns `422` instead of the runner bridge rejecting it. The
error text names the problem.
- For `paperclip_finish` on Codex, the advertised schema no longer has
the root conditional. The Rust bridge accepts a `done` result with a
continuation object, and canonical result validation still rejects it,
as before.
- Latency: the advertised schemas of the 8 tools are 1,876 bytes smaller
(23,785 to 21,909), about 470 fewer prompt tokens when all are listed.
Compiling their validators in the bridge takes 164 ms instead of 214 ms
(median of 15). The guard adds one SHA-256 of the canonical arguments
for each rejected call only. Nothing is added before turn start.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:08:58 -06:00
Nicky LeachandClaude Fable 5.2 84a4d345b3 refactor(docker): take the application image's Runner from the runner-artifacts stage (#21332)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work. `master` publishes
it alone as the image `ghcr.io/paperclipai/paperclip-runner`, built by
the `runner-artifacts` stages of the `Dockerfile`
> - The application image does not use those stages. It has five stages
of its own that compile the daemon, it builds the Runner again inside
the server build, and it builds a second provider pack
> - So one `Dockerfile` holds two recipes for the Runner, and the two
results differ: another base image and another daemon binary
> - This pull request removes the second recipe. The application image
copies the Runner out of the `runner-artifacts` stage, which the same
build runs once
> - The benefit is one Runner recipe, and an application image whose
Runner is the Runner the contract describes. A later pull request can
replace the stage with the published image

## Linked Issues or Issue Description

This is the second part of a chain. The first part is #21331, which
removes the cache check that cannot run on this `Dockerfile`. Refs
#21204, #21229, #21230 (the Runner image), #21306 (the no-rebuild mode
of the vendor step), #21307 (the pull-request check of the application
image) and #21312. No issue exists for this change.

**Problem or motivation**

The stages `rust-toolchain`, `rust-chef`, `runner-plan`, `runner-deps`
and `runner-build` compile the daemon on Debian trixie. The server build
then runs the full build of the Runner package. The
`runner-provider-pack` stage builds the provider pack. The
`runner-artifacts` stages do all of that again for the Runner image, on
Debian bookworm, with a stripped daemon.

**Proposed solution**

Read only the `runner-artifacts` stage from the application stages.
Build it inline in every build. Do not pass a published image yet.

**Alternatives considered**

- Pass the published Runner image to the build in this pull request.
That needs the resolver, labels and a rule for commits without an image.
It is the next step, and this one must be safe alone.
- Keep the old stages as a fallback. Two recipes is the problem.

## What Changed

- `Dockerfile`: remove the five stages. `build` starts from `base`. It
copies `/runner-package/dist` and the build stamp from
`runner-artifacts`, sets `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` and
runs the vendor check before the first application build.
- `Dockerfile`: `runner-provider-pack` starts from `base`. One `RUN`
with a bind mount of `runner-artifacts` copies the pack, the receipt and
the archives, and moves the large files to their own tree. `build` and
`runner-provider-pack` now come after the stages they read.
- `Dockerfile`: the three `COPY` instructions of `production` that take
the pack are byte-identical. A test pins all three.
- `Dockerfile`: `build` declares the build argument
`PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST`, empty by default, and one step
runs the new `scripts/check-runner-image-expectation.mjs` on
`/runner-image.json` of `runner-artifacts`. **This pull request declares
the argument. No build sets it. The next pull request, which gives a
published Runner image to the build, is its first user.** With the
argument set, the build fails unless `runner-artifacts` is the published
image of exactly those inputs. `docker-app-check.yml` also runs when the
script changes.
- Comments: the `Dockerfile` has 220 comment lines, down from 247. The
comments of the two moved stages moved with them, unchanged, so the diff
shows them as added lines. Eight comments that named a removed stage or
a step that moved get the smallest correction. Two comments in
`prepare-runner-vendor.mjs` and `build-inputs.mjs` said that the
`Dockerfile` relies on the standalone rebuild; that clause is removed.
- Tests: replace the two cargo-chef tests of
`docker-build-stamp.test.ts` with three tests of the new shape. Update
the pack test, `docker-image-layer-reuse.test.ts`,
`vite-build-commit.test.ts`, the stage rules of
`runner-image-identity.test.ts`, and the include-layout test of
`runner-release-crate-cache.test.mjs`, which now reads
`runner-artifacts-runnerd`.
- Docs: `doc/DOCKER.md`, `doc/DEVELOPING.md`, and a new section "The
application image" in `doc/runner-image-contract.md`.

## Verification

All builds: `linux/arm64`, local, `--build-arg
PAPERCLIP_BUILD_COMMIT=<sha>` with one commit for every build. The
builds ran before the rebase onto `master`, on the head of #21310 as it
was before it merged. `master` changed neither the `Dockerfile` nor an
input of the Runner in between.

- `docker buildx build --target production`, cold, before and after: 506
s and 410 s. Compressed size: 2,971 MB and 2,900 MB.
- The image config and the text of all 33 history rows are identical.
The row `COPY /provider-pack-transport/ /opt/paperclip-runner/ #
buildkit` matches the pattern of `retained-provider-pack-oci.ts` exactly
once.
- A comparison of every file of the two images (path, type, size, mode,
link target, and SHA-256 under `/app` and `/opt`). The differences are
in "Risks". All other files are byte-identical, the root `node_modules`
included.
- From the image after: the four `/app/packages/paperclip-runner/...`
paths and the contents of `/opt/paperclip-runner/` exist.
`paperclip-runnerd --build-metadata` runs. `provider-pack.json` has
`runnerSourceRevision` equal to the build commit and no
`runnerInputsDigest`. The entry points `index.js`, `live/index.js` and
`testing.js` of the Runner import.
- The vendor step logged `reusing the current Runner build` in both of
its runs. Every `Compiling` line of the log is in
`runner-artifacts-runnerd`.
- `--target build`, `--target runner-provider-pack` and `--target cloud`
built. The `build` stage has no `gcc`, `cc`, `cargo`, `rustc` or
`pkg-config`.
- `--target runner-artifacts` and `runner-artifacts-slim`, before and
after: `runner-image-layout.mjs check` and `runner-image-oci.mjs
check-layers` pass on both, in `node:24.22.3-bookworm`. The daemon,
`provider-pack.json` and the two stable layers have the same digests
before and after.
- Without a build commit: `build` and `runner-provider-pack` built, and
the pack was skipped with the same message. This run was on the next
branch of this chain, after the rebase. The last steps of `production`
were not run for this case.
- The expectation argument, on this `Dockerfile`, `--target build
--output type=cacheonly`, after the rebase:
- No argument, inline Runner: the build passes (233 s). The step passes.
- Argument, inline Runner: the build fails at the step. The message says
that the `runner-artifacts` of the build are not a published Runner
image.
- Argument `d4df566f…` and the published image of those inputs as
`--build-context
runner-artifacts=docker-image://ghcr.io/paperclipai/paperclip-runner@sha256:0cd8250b…`:
the step passes, and the build runs no Runner stage. The vendor step
then fails with `inputs changed since the recorded build`, because this
tree differs from the tree of that image by one comment line in
`build-inputs.mjs`. With that line put back, the same build completes,
and the vendor step reuses the published build.
- Another digest with the same context: the build fails at the step. The
message names the inputs the image has.
- A tree whose lockfile is behind the manifests of the Runner fails at
the frozen install of the Runner stages, after 169 s, with the message
that stage already had.
- `docker buildx build --check`: no warning, before and after.
`scripts/check-docker-deps-stage.mjs` and
`scripts/check-node-version-policy.mjs` pass.
- Tests, on macOS and in `node:24.22.3-bookworm`, after the rebase: the
changed server test files with the other tests that read the
`Dockerfile` or the image workflows (231 pass in 10 files on Linux),
`vite-build-commit.test.ts` (10 pass),
`runner-release-crate-cache.test.mjs` (11 pass),
`docker-deps-postinstall.test.mjs`. The changed server test files
typecheck.
- Not run: the full test suite, any `linux/amd64` build, `podman`, the
workflows and the release pipeline. Only CI can prove amd64,
`docker.yml`, `docker-merge-group.yml`, `release.yml`,
`docker-app-check.yml` and `runner-image.yml`.

## Risks

- **The digest of the Runner image changes.** Its identity covers the
whole `Dockerfile`. The next `master` push publishes a new
`inputs-<digest>`. The contents do not change: see the comparison above.
- **A build that is given a Runner of other inputs fails; it does not
ship that Runner.** With the argument, it fails at the expectation step.
Without the argument, it fails at the vendor step when the build inputs
of the Runner differ. One comment line in `build-inputs.mjs` changes
those build inputs in this pull request, so the image published before
this pull request cannot be vendored into a tree after it.
- **What a consumer of the application image sees** (measured on
`linux/arm64`):
- The daemon is the stripped binary compiled on Debian bookworm: 23.2
MB, was 36.4 MB, at
`/app/packages/paperclip-runner/dist/bin/paperclip-runnerd` and at the
vendored copy in `server/dist`. Its hash changes. It has no symbols for
a crash backtrace.
- `/app/packages/paperclip-eval-kernel/dist` is absent. Only
`dist/eval/workflow-harness.js` of the Runner imports that package. Only
`eval/index.js` exports the harness again, and no module imports that
file. The server runtime bundle does not name the package.
- The provider pack: `provider-pack.json` is byte-identical. Two
provider executables and one licence file are hard links to their other
copy in the pack, where they were separate files. One empty directory is
absent. The pack is 195 MB smaller, and the layer of its large files
gets a new digest once.
- The transport archives have other bytes, because the Runner build
gives large files a fixed timestamp. The receipt names the same files
with the same digests.
- Paths, modes, owners, the image config and the layer order do not
change.
- **What a contributor and a maintainer of the image build see:**
- **The image build does not run the drift checks of the Runner.**
Before, the server build ran the full `build` script of the Runner, so a
failed drift check failed the image build on `master`. Now the Runner
stages run only the output steps of that script. Repository CI runs the
checks on every pull request. `.github/docker-context-checks.Dockerfile`
and three exceptions in `.dockerignore` exist for those checks in the
image build; a separate follow-up removes them, and this chain does not
edit them.
- **The dependency-caching layer of the daemon is gone.** Before,
cargo-chef compiled the Rust dependencies in a layer of their own, and a
change to the Rust source compiled only the Runner's own crates. Now
such a change compiles the dependencies again (146 s here for the whole
daemon). A change to the server, the UI or the Runner's TypeScript still
reuses the compiled daemon.
- **A lockfile behind the manifests of the Runner fails the build.**
Before, `docker build` resolved it inside the image. The image workflows
refresh the lockfile first. A local build prints the command to run. The
next pull request makes this failure early and clear.
- **The build needs more disk**: about 25 GB of builder storage here,
was about 20 GB, with the image export.
- Rollback: revert this pull request. Images that exist do not change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude
Code. The model used tools: it edited files, ran the tests and ran the
Docker builds that this description reports.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 09:48:49 -08:00
DottaandClaude Opus 5.5 594f9d3ec2 perf(runner): keep at most one checkpoint waiting behind a slow write (#21411)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native Runner control plane saves a checkpoint of each run's
controller state before it acknowledges an event or sends a command
> - Each new state revision captured a full snapshot and queued it
behind the write in progress
> - Inbound events wait for their own checkpoint, so they do not pile
up. Commands that the server queues do not wait, so they can
> - When the database or the event loop is slow, the queue of full
snapshots grows without limit, and events wait behind the whole queue
> - This pull request keeps at most one snapshot waiting per run. A
newer flush replaces the waiting snapshot of the same run
> - The benefit is bounded memory and fewer database writes under slow
persistence, with the same durability guarantee

## Linked Issues or Issue Description

**What happened?**

A hosted server ran out of JavaScript heap at intervals of 47 to 105
minutes. A retained-memory sample attributed about 16% of the memory
that stayed alive to `captureCheckpoint` (from `flushCheckpoint`) and to
`#capturePublishedState` in
`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`.

**Expected behavior**

Memory for checkpoints stays proportional to the number of live runs,
not to the number of state revisions that wait for a slow write.

**Steps to reproduce**

1. Create control planes with checkpoint persistence that takes 1 second
per write.
2. Commit events at 13 per second and queue commands at 2 per second for
each run.
3. Measure the heap in the middle of the run.

`node --import ./server/node_modules/tsx/dist/loader.mjs
scripts/benchmark-runner-checkpoint-backlog.ts` does these steps.

**Paperclip version or commit**

`master` at 2659ff1d22.

## What Changed

- `DurablePrpControlPlane.flushCheckpoint` keeps one snapshot that waits
for the write in progress. A newer flush for the same run replaces that
snapshot and returns the same promise. A flush for a different run (a
warm handoff) still queues its own write.
- New tests in `durable-prp-control-plane.test.ts`:
- 40 commands queued behind a slow write produce one more write, not 40.
Each caller resolves only after the stored records contain its command.
The stored records decode to the last written checkpoint.
- When the combined write fails, every caller that waited on it rejects,
and the next flush writes a full reset that contains the commands.
- New `scripts/benchmark-runner-checkpoint-backlog.ts` measures retained
heap per run, checkpoint writes, and whether finished control planes are
collected.

Why this is safe:

- A waiting snapshot has not started its write. No caller has received a
result for it, so no acknowledgement or command send depends on it.
- The replacement is captured from a later state of the same store. It
contains every change that the replaced snapshot contained.
- Every caller of the replaced snapshot waits on the same promise. The
promise resolves only after the newer snapshot is durable. A failure
rejects all of them, and the existing retry writes a full reset.
- Writes stay on one chain, in order. The server fence, the encryption,
and the stored checkpoint format do not change.
- A flush for a different run ID never replaces a waiting snapshot, so
the earlier run still gets its last state.

## Verification

- Red/green: the two new tests fail on `master` (42 writes instead of 3;
the waiting callers resolve instead of rejecting) and pass with the fix.
- `npx vitest run src/control-plane/durable-prp-control-plane.test.ts`
in `packages/paperclip-runner`: 219 of 219 pass.
- `node scripts/check-module-boundaries.mjs` passes.
- Benchmark, 8 runs, 30 s, 13 events/s and 2 commands/s per run, 1 s
writes, 300 events of 2 KB already in each run:

| | `master` | this PR |
| --- | --- | --- |
| Retained heap per run in the middle of the load | 15,009 KB | 2,447 KB
|
| Retained heap per idle live run | 1,296 KB | 1,415 KB |
| Checkpoint writes | 515 (one per revision) | 248 (for 720 revisions) |
| New events committed per run in 30 s | 4 | 30 |
| Control planes collected after stop | 8 of 8 | 8 of 8 |

- With commands at 0 per second, `master` has no backlog: an inbound
event waits for its own checkpoint before the next frame is read. The
backlog comes from flushes that do not wait.

## Risks

- Behavioral shift: when a combined write fails, callers whose own
earlier revision would have been written alone now see that failure. A
checkpoint failure already disconnects the runner and the next flush
writes a full reset, so recovery is the same.
- Fewer intermediate states reach the database. A crash can only lose a
state that no caller has seen as saved, which is the same as before.
- Not covered by a test: a warm handoff whose run ID changes while a
snapshot waits. The code does not combine those; the existing
warm-transition tests pass.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:08:15 -06:00
DottaandClaude Opus 5.5 681af53fa6 fix(runner): ask once more for a text answer when a turn that did tool work ends empty twice (#21359)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner drives OpenCode turns and decides when a turn has a final
answer
> - A turn can do real tool work and then end with no text and no
completion call
> - The driver asks once for a final response. That request can also
come back empty, and then the run fails
> - The person then sees a failed run, although the requested work was
saved
> - This pull request adds one bounded request, text answer first, for a
turn that already did tool work
> - The benefit is that finished work gets a reply instead of a failed
run

## Linked Issues or Issue Description

**What happened?**
An agent saved a requested document with a tool. Its next response had
no text and no completion call. The driver asked once for a final
response, and that response was empty too. The run failed with
`opencode_empty_response` and showed a failure message to the person.

**Expected behavior**
When the work is done, the person gets a short final answer. An empty
response is a recoverable state, not a hard failure.

**Steps to reproduce**
1. Use an OpenCode agent on an OpenRouter model.
2. Ask it to write and save a plan document.
3. Let the provider host return an empty response after the save, and
again for the correction.
4. See the run fail although the document exists.

**Paperclip version or commit**
`master` at ff971149c2.

**Agent adapter(s) involved**
OpenCode through the native Runner.

**Additional context**
The model answered with only a completion tool call, and the serving
host returned no text and no tool call. The correction asked for the
same tool call, so it was lost again. The turn had no other recovery.

## What Changed

- `OpenCodeServerDriver`: when a turn already did tool work and the
correction comes back empty, the driver sends one more request. That
request asks for a short plain-text answer first and the completion call
after it. The text does not depend on tool-call delivery, so a chat
reply arrives even if the host drops the call again. An execution task
still gets its completion call.
- A turn makes at most two correction requests. A turn without tool work
keeps the existing single correction.
- Only tool work that the active turn reports counts. A late tool update
from an earlier turn does not earn an extra request.
- The final failure message now says how many corrections ran.
- The driver emits a separate diagnostic,
`opencode_empty_response_text_correction`, for the second request.
- `empty-response.test.ts`: two new cases from the recorded failure. One
case answers on the plain-text request and completes. One case stays
empty and fails after exactly three prompts. One case sends a late tool
update from an earlier turn and keeps the single correction.

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/drivers/opencode/empty-response.test.ts
src/drivers/opencode/interrupt-empty-response.test.ts
src/drivers/opencode/interrupt-pending-tool-call.test.ts
src/drivers/opencode/output-limit.test.ts` passes. All 19 OpenCode
driver test files pass (241 tests).
- Red/green: with the driver change removed, `tool-work-text` and
`tool-work-empty` fail. With a gate that counts any tool update,
`stale-tool-work` fails. With the change, both pass.
- The existing `empty-again` case (no tool work) still fails after two
prompts, so the bound is unchanged for that case.
- `npx tsc --noEmit -p packages/paperclip-runner` passes.

Latency:
- The change runs only after an empty response. It adds no work before
the turn starts and no work on a turn with a normal answer. Boot time
and time to first token do not change.
- On the recovery path, it adds one short model request. In the recorded
failure, one correction request took about 2.4 s.

## Risks

- Low risk. The extra request is bounded to one per turn and only after
tool work.
- In a chat, a text answer without a completion call uses the existing
chat-reply path.
- An execution task needs its completion call. If the host drops that
call again, the run fails as it did before this change, and the existing
failed-turn continuation resumes it. This change does not make that case
worse.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code
with tool use (shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:57:33 -06:00
DottaandClaude Opus 5.5 9fbfd93c9b fix(runner): record an OpenCode call's serving host when the call ends (#21358)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode runs that use OpenRouter record which host served each
model call in the run log
> - Operators use this host to compare quality and failures between
OpenRouter hosts
> - About one in three of the first model calls in a turn had no
recorded host, while later calls had a host about 93% of the time
> - The provider proxy cleared the recorded host when each request
started, and this clear raced the usage report of the step before
> - This pull request records a call's host when the call ends, not when
the next call starts
> - The benefit is that each step's usage names the host that served it,
so host comparisons are complete

## Linked Issues or Issue Description

**What happened?**

In an evaluation cohort of OpenCode runs through OpenRouter, 128 of 356
first usage events in a run (36%) had no `usage.upstreamProvider`. For
later usage events the rate was 168 of 2,362 (7%).

Root cause:
- OpenCode 1.18.34 sends a step's final token count only 14 to 20 ms
before it sends the next model request. We measured this three times
with the real OpenCode binary and a local fake provider.
- The provider proxy cleared the recorded host when each request
started. The driver reads OpenCode's event stream. When the next request
started before the driver handled the token event, the clear won, and
the step's usage had no host.
- If the next call's first chunk had arrived first, the step's usage
would show the next call's host.
- The first call of a turn lost the race most often, because the driver
is busiest at turn start.

**Expected behavior**

Each step's usage names the host that served that step's model call. A
call whose host the proxy cannot read carries no host. A call never
inherits another call's host.

**Steps to reproduce**

1. Start an OpenCode session through the OpenRouter provider proxy.
2. Complete one model call that names a host.
3. Start a second model call and keep its response open.
4. Let OpenCode report the first step's usage.
5. See that the usage has no `upstreamProvider`.

**Paperclip version or commit**

origin/master ff971149c2.

**Deployment mode**

Any deployment that runs OpenCode with an OpenRouter model.

**Agent adapter(s) involved**

OpenCode (Paperclip Runner OpenCode server driver).

## What Changed

- `opencode-server-driver.ts`: the provider proxy keeps the host it
reads for each call and records it when that call's response ends. It no
longer clears the recorded host when a request starts. A call that ends
without a readable host (skipped, non-2xx, redirect, or failed) records
no host.
- `opencode-server-driver.test.ts`: a new red/green test completes one
call. Then it starts a second call that has already sent its first chunk
with another host, and keeps it open. The test checks that the turn's
usage names the first call's host. Then it ends the second call and
checks that the next turn names the second call's host.
- `doc/run-log-events.md`: the "OpenRouter serving host" section now
says the host is recorded when a call ends, and why.

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/drivers/opencode/opencode-server-driver.test.ts
src/drivers/opencode/upstream-served.test.ts` (91 and 8 passed).
- Red/green: without the source change, the new test "keeps a finished
call's host on its usage while OpenCode's next model call is already in
flight" fails. The first step's usage names the second call's host. With
the second call's first chunk held back, the old code instead gave a
usage with no `upstreamProvider`, which is the symptom we saw in the
cohort. With the change, the test passes.
- The existing test "does not carry an earlier call's host onto a call
the tap skips, or into another run" still passes.
- `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` passes.
- Latency: the change adds no `await` on the request path. It removes
one synchronous callback before the upstream request and adds one after
the response ends. The tap still forwards each chunk before it reads it,
so time to first token and boot time do not change.

## Risks

- Low risk. The change only affects run-log fields (`upstreamProvider`,
`providerRequestId`). These fields stay in the instance database and are
not sent to Paperclip Telemetry or OpenTelemetry.
- A small race remains if the driver handles a step's usage after the
whole next call has ended. That window is now one full model call
instead of about 15 ms.
- If two model calls run at the same time in one session, the usage
names the call that ended last. This was also true before.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edits, code execution). It did the
investigation, the change, the tests, and this description.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:15:42 -06:00
DottaandClaude Opus 5.5 3fc2198759 fix(runner): treat tool-call markup in OpenCode text as an unexecuted call (#21361)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner drives OpenCode and turns its event stream into
the run's progress, reply, and completion result
> - Some model hosts return a tool call as raw tool-call syntax inside
the assistant text, not as a structured tool call
> - The OpenCode driver streamed that syntax as progress and accepted it
as the final answer, so the run succeeded with markup as its reply and
the intended tool call never ran
> - This pull request treats leaked tool-call markup as an unexecuted,
malformed tool call
> - The benefit is that the person never sees raw call syntax as a
reply, and the agent gets one bounded correction to make the call
properly

## Linked Issues or Issue Description

**What happened?**
An agent run that uses an OpenRouter model can end with raw tool-call
markup as its reply. The model meant to call the completion tool. The
host put the call into the message text as DeepSeek DSML markup (for
example `<|DSML| invoke name="...">`). The runner accepted that text as
the final answer and settled the run as succeeded. The completion call
did not run, so follow-up automation repeated the work.

**Steps to reproduce**
1. Run an OpenCode agent with a DeepSeek model through an OpenRouter
host that leaks tool-call syntax into text.
2. Let the model end its turn with a completion call that the host
returns as text.
3. Look at the run reply.

**Expected behavior**
The runner does not show the markup and does not accept it as the reply.
It tells the model that the tool did not run and asks it once to call
the tool through the tool interface.

**Paperclip version or commit**
`master` at the time of this pull request.

**Additional context**
Today the markup streams as progress and becomes the run's final reply.
The run succeeds without its completion call.

## What Changed

- Added `tool-call-markup.ts` in the OpenCode driver. It detects leaked
tool-call markup in one linear scan: DSML and DeepSeek call-token
sentinels anywhere, and `<function_calls>` / `<tool_call>` wrappers only
at a line start with a call body. It ignores fenced code blocks.
- The driver holds markup (and a trailing fragment that can still become
markup) out of streamed progress text.
- A completed text part that contains markup is not a final-answer
candidate. The driver emits the `opencode_tool_call_markup_discarded`
diagnostic and records the position, so earlier text cannot become the
reply either.
- The turn then goes through the existing one-shot empty-response
correction. The correction text says the tool did not run and asks the
model to call tools only through the tool interface.
- Regression tests: detection cases (recorded DSML shape, other
families, prose and code-fence negatives, streaming monotonicity) and a
driver case that replays the recorded markup.

## Verification

- `cd packages/paperclip-runner && npx vitest run src/drivers/opencode/`
passes (20 files, 263 tests).
- Red/green: the new `tool-call-markup` case in `empty-response.test.ts`
fails on the unchanged driver (the markup is selected as the final
reply) and passes with this change.
- `npx tsc --noEmit -p packages/paperclip-runner` passes.
- Cost: the scan adds about 22 µs for each streamed update of a 20 KB
answer with `<` on every line (43 ms over 1,960 updates), and 0.05 ms
for each completed part. It adds no waits and does not touch boot or the
start of a turn.

## Risks

- Low risk. A real answer that puts DSML sentinels or a
`<function_calls>`/`<tool_call>` wrapper with a call body outside a code
fence is now treated as an unexecuted call. The correction then asks
once for a proper answer.
- If the model leaks markup again after the correction, the existing
empty-response failure path applies.
- No schema, API, or saved run-state change.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 06:00:29 -06:00
DottaandClaude Opus 5.5 cb94108484 fix(native): finish the run from a provider turn a timed-out start left running (#21338)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native Codex run talks to a remote Runner through a controller. A
deploy or a same-run retry can reattach a new controller to the live
Runner.
> - A slow turn start can make the first controller give up with
"runnerd did not report the provider turn identity". Codex still runs
that turn.
> - The retry reattaches the Runner and adopts that turn, because the
Runner accepted it for this run's latest turn.start.
> - If the adopted turn finishes while recovery still runs, the saved
checkpoint has no active turn. The runtime then tries to start a fresh
turn, and the run fails with "session cannot start another turn". The
Runner is lost and the task needs a Retry.
> - This pull request makes the runtime treat a terminal that the
recovered session newly settled as the adopted provider terminal when
the saved checkpoint bound no turn.
> - The benefit is that a turn which a timed-out start left running
finishes the same run on the reattached Runner, with its session
context, and with no second turn/start.

## Linked Issues or Issue Description

**What happened?**
On staging, Codex runs on a remote Runner lost the Runner after a
same-run reattachment. Each chain started with a turn.start that timed
out ("runnerd did not report the provider turn identity") while Codex
ran the turn anyway. After the reattachment, the run failed with a
protocol integrity fault (thread_binding_mismatch,
tool_binding_mismatch, or native_event_replay_conflict "Provider
authoritative notification item/started did not name its execution
owner"). The task then fell back to a fresh session and lost its
context. A local reproduction with the real runnerd and the fake Codex
app server shows the same trigger also fails when the abandoned turn
finishes during the reattachment: the runtime tries a new turn and fails
with "session cannot start another turn".

**Expected behavior**
A turn that a timed-out turn.start left running belongs to the same run.
The reattached run adopts it, binds its late notifications, and finishes
from its outcome. Recovery does not kill the Runner and does not need a
manual Retry. A real cross-run or cross-task binding violation still
fails closed.

**Steps to reproduce**
1. Start a native Codex run on a remote Runner whose provider is slow to
emit turn/started after it accepts turn/start.
2. Let the controller give up with "runnerd did not report the provider
turn identity" while the Runner stays alive.
3. Let the same-run retry reattach the Runner.
4. Let the provider turn start and finish after the reattachment.
5. See the run fail instead of finishing from the adopted turn.

**Paperclip version or commit**
master at 22c9300dac (after #21321 and #21329)

**Deployment mode**
Paperclip Cloud staging, remote Runner in a Daytona sandbox

**Agent adapter(s) involved**
Codex (codex_app_server through paperclip-runnerd)

## What Changed

- `native-session-runtime.ts`: a terminal that the recovered session
newly settled now counts as the adopted provider terminal when the saved
checkpoint had no active turn. The runtime consumes that terminal
instead of starting a fresh turn.
- New `runnerd-abandoned-turn-start-reattach.test.ts` with the real
runnerd and the fake Codex app server. Each attempt runs in a fresh
module graph, as each attempt runs in its own executor process in
production. The first attempt times out on turn.start against an
unreachable remote Runner and leaves it alive.
- Red/green: the provider turn starts and finishes only after the
reattachment. The run now succeeds from `provider-turn-1` with one
turn/start and no duplicate Runner launch. Before the fix it failed with
"session cannot start another turn".
- Guard: output the abandoned turn reported while no controller was
attached binds to the reattached run's adopted turn with no
`session.failed`, and the Runner stays alive.

## Verification

- `npx vitest run
src/live/runnerd-abandoned-turn-start-reattach.test.ts` in
`packages/paperclip-runner`: red without the runtime change ("session
cannot start another turn"), green with it.
- `npx vitest run src/native-session-runtime.test.ts
src/live/runnerd-restart-continuation.test.ts
src/live/runnerd-abandoned-turn-start-reattach.test.ts
src/live/runnerd-turn-start-timeout-retention.test.ts
src/live/runnerd-codex-transport-cut-turn-start.test.ts
src/drivers/codex/codex-app-server-driver.terminal-replay.test.ts
src/drivers/codex/codex-app-server-driver.recovery.test.ts`: 7 files,
386 tests passed.
- `tsc -p tsconfig.test.json --noEmit` in `packages/paperclip-runner`:
no new errors (the existing `src/eval/workflow-harness.ts` errors need
the eval kernel build).

## Risks

- The runtime now adopts a newly settled terminal when the saved
checkpoint bound no turn. The driver records a turn terminal only for a
turn that it bound as active (the persisted active turn, the
uncheckpointed disposition turn, or this run's own accepted turn). A
turn of another run or task is never bound, so it still fails closed in
the driver.
- The staging chains failed with binding-mismatch codes. The local
reproduction reaches the same trigger and the adoption path, but the
fake provider cannot hold a live turn while it answers other requests,
so the exact staging codes did not reproduce locally. Late items that
reach a reattached controller before it knows the provider turn identity
can still fail closed. That path needs a separate fix if it recurs after
this change.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5) in Claude Code, with tool use
(shell, file edits, browser reads of staging run events).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21321, #21329

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:43:41 -06:00
DottaandClaude Opus 5.5 43aabb7080 fix(runner): keep configured secret values out of OpenCode output, storage and model requests (#21330)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can configure a secret that Paperclip delivers to an
agent's commands as an environment variable
> - The value must stay out of what the agent runtime stores and what it
sends to the model
> - With the OpenCode runtime, a command that printed the variable put
the value in OpenCode's tool output. OpenCode stored that output in its
session database and sent it to the model. Paperclip's own run log
already redacted it
> - Paperclip did not tell the runner which configured values are
secrets, so the runner could not protect them without also hiding
ordinary settings
> - This pull request sends secret names (never values) to the runner,
filters exact secret values before OpenCode captures any tool output,
redacts them from model requests, and tells the agent how to check a
secret without printing it
> - The benefit is that a configured secret stays out of model context
and out of runtime storage, even when an agent prints it

## Linked Issues or Issue Description

**What happened?**
The `chat-features.runner-opencode.local.secrets.personal-secret` eval
failed because the persisted-state scan found a configured secret value
in OpenCode's session database. The person entered the value through the
normal secret entry form. Nobody pasted it in chat. The agent ran a
shell command that printed the environment variable it was given.
Paperclip's run log redacted the value. OpenCode stored the raw output
in its database and included it in the next model request.

**Expected behavior**
A configured secret must never reach model context or runtime
persistence. Ordinary configured values must not be redacted.

**Steps to reproduce**
Configure a secret environment variable for an OpenCode agent. Have the
agent run a command that prints the variable. Then search the OpenCode
session data and the model requests for the value.

## What Changed

- **Secret names cross to the runner.**
`configuredEnvironmentProjection` takes the controller's secret key
names and adds `PAPERCLIP_SECRET_ENV_KEYS`. This marker holds only the
names of configured entries that came from secrets. It travels with the
configured projection across the TypeScript and Rust runner boundaries
(`configured-environment.ts`, `configured_environment.rs`). The server
passes `secretKeys` in `runtime-execution.ts`.
- **Command output is filtered before OpenCode reads it.** For a launch
that has configured secrets, the runner sets OpenCode's `shell` to a
small POSIX wrapper (`drivers/opencode/output-custody.ts`). The wrapper
runs the shell OpenCode would use anyway. It sends stdout and stderr
through an `awk` filter that replaces exact secret values. OpenCode
never receives the raw bytes, so its streamed previews, its database and
its model requests stay clean. The wrapper keeps the exit status. It
does not wait for background processes. The values come from the
environment that OpenCode already holds and are never written to disk.
- **Other tool results are rewritten before they are saved.** A local
OpenCode plugin rewrites completed non-shell results (file reads,
searches, MCP results) in `tool.execute.after`. OpenCode runs that hook
before it stores or returns the result. The wrapper refuses to run any
command until the plugin has started for the current launch (per-launch
nonce). So no command can write a secret to a file that a later read
returns without filtering.
- **Model requests are redacted as a second layer.** The OpenCode
provider proxy removes exact secret values from the string values of
each parsed model request. Field names and other literals are never
changed. A quick byte scan for each value, raw and JSON-escaped, sends
the original bytes unchanged when no value is present.
- **The agent gets guidance.** Native task prompts list the configured
secret names. They say to check a secret with `[ -n "$VAR" ]` and never
to print, echo, log or write a secret value.
- **No added startup delay.** The wrapper is content-addressed under the
runtime directory and reused across launches, so it is not rewritten on
every launch. OpenCode's plugin-dependency install is marked as already
done, so loading the plugin does not add wait time to the first tool
call.
- **Only exact values are redacted.** Command output is filtered line by
line, so each line of a multi-line value that has at least 4 characters
is redacted too. If a configured secret has no line of 4 or more
characters, the launch is refused with
`opencode_output_custody_unavailable` instead of running unprotected.

## Verification

-
`packages/paperclip-runner/src/drivers/opencode/output-custody.test.ts`
(10 tests): the wrapper filters stdout, stderr and a partial last line;
keeps exit codes (7, and 143 for SIGTERM); filters each line of a
multi-line value; refuses to run before plugin readiness or with an old
nonce; returns at once when a background process keeps running; leaves
ordinary values unchanged. The plugin rewrites nested results.
Model-request redaction covers raw and JSON-escaped values and never
changes field names or literals (a secret equal to `true` leaves
`"stream": true` intact). Secrets the filter cannot match are refused.
No value is written to disk.
- `opencode-server-driver.test.ts`: one saved session launches three
times. First with no secret marker (older controller), then with the
marker, then with no marker again (rollback). The first launch is
unchanged, the second applies custody, and the third launch still
starts. This test fails without the driver change.
- `configured-environment.test.ts`: the marker names only secret
entries, crosses the runner spawn and OpenCode boundaries, reads as "no
secrets" when absent, and is refused when malformed. Rust:
`secret_name_marker_crosses_only_with_its_projection`.
- `server/src/__tests__/configured-secret-guidance.test.ts`: guidance
content, and no guidance when there are no secrets.
- Real OpenCode 1.18.34 qualification (`PAPERCLIP_OPENCODE_QUALIFY=1
node --test scripts/qualify-opencode-runtime.test.mjs`). A fake model
asks for a command that prints the secret and writes it to a file, then
a read of that file. Unprotected baseline: the value is in the session
database and in model requests (red). With custody: it is in neither,
and the exit status is kept (green).
- **Latency** (local, macOS, real OpenCode, 8 runs each): boot 629 ms vs
697 ms baseline; first model request after the prompt 678 ms vs 698 ms
(no regression); first command 139 ms vs 191 ms; each later command +9
ms (36 vs 27 ms). Proxy cost per request when no secret is present (the
normal case): 33 µs at 50 KiB, 164 µs at 300 KiB, 0.55 ms at 1 MiB, 2.3
ms at 4 MiB. When a secret is present: 91 µs, 0.48 ms, 1.6 ms and 6.8
ms. The scan runs only for launches that have configured secrets, and it
adds nothing to OpenCode's own startup.
- A narrow paid run of
`chat-features.runner-opencode.local.secrets.personal-secret` on this
branch is linked in a comment.

## Risks

- **Previous-release story.** The marker is new and optional. A runner
that receives no marker (older controller) launches exactly as before.
An older runner drops the marker at its environment boundary. The change
saves no new run or session state. OpenCode configuration is regenerated
on every launch, so sessions saved before this change resume under
custody (covered by the cutover test). Warm-process reuse keys include
the marker, so a warm process from before the deploy is not reused for a
run that has secrets.
- **Commands go through the wrapper.** Output arrives line by line.
Output written after the shell exits is dropped, which matches how
OpenCode stops reading at exit. If `awk`, `mkfifo` or `rm` is missing,
the launch is refused with `opencode_output_custody_unavailable` and
does not run unprotected.
- **Scope.** Only exact values are redacted. An agent that encodes a
value (for example base64) is out of scope. The model-request layer
applies when Paperclip routes the model through its proxy (OpenRouter
and Paperclip connections).

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:23:56 -06:00
DottaandClaude Opus 5.5 0986c3eb3a fix(native): let a same-run reattachment take over its own retained remote Runner cleanup (#21328)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a recoverable failure, the server resumes the
same run and reattaches to that Runner
> - #20923 keeps an unreachable remote Runner alive when the failed
attempt closes, so that the same-run reattachment can adopt it
> - The same close also leaves the session in the in-process cleanup
quarantine. The quarantine refuses every new session of the task, and
that includes the reattachment that the Runner was kept for
> - Close retries cannot settle it. After the first close fails, the
Runner transport returns the same rejected close for every retry
> - On staging, this was the largest failure class: 14 Codex runs in one
hour failed with `native_session_cleanup_quarantined`, while only 5 runs
succeeded. Each task then needed Retry
> - This pull request lets the same run's reattachment to that Runner,
in the same remote sandbox, take over its own retained cleanup
> - The benefit is that the task continues on its own, with no spent
attempt, no failed run and no Retry

## Linked Issues or Issue Description

Refs #20923, #21031

**What happened?**
A remote native run (Codex on Daytona) failed a turn with a recoverable
error, for example `runnerd did not report the provider turn identity`
or `native_finalization_missing: session returned no semantic result`.
The close then logged `runner did not prove durable suspension before
checkpoint` and `remote runner left alive unsettled for same-run
reattachment`. Each later same-run reattachment was refused at
admission:
- When the close error was retryable, admission waited out the grace and
failed with `native_session_cleanup_quarantined: prior session cleanup
remains incomplete`. That spent a same-run attempt. After the attempt
budget, the run failed. All close retries replayed the same failed close
within milliseconds (`runner transport close requested: native session
quarantined cleanup recovery`), so they could never settle the
quarantine.
- When the close error was `native_session_close_unrecoverable`
(operator-required), the next attempt failed at once with
`native_session_cleanup_quarantined: the previous session is still
shutting down; Paperclip finishes the cleanup automatically and the
saved work is kept.` That code is a permanent failure.

"Paperclip finishes the cleanup automatically" did not happen in the
process. Only a server restart (a deploy) removed the in-memory
quarantine, and by then the run had failed and the task waited for
Retry. Staging restarts every 20 to 45 minutes, so attempts often end
mid-cleanup and meet this gate.

**Expected behavior**
The same run's reattachment adopts the Runner that its earlier attempt
kept for it. It does not wait for a close that cannot settle, and it
does not spend an attempt. The task continues without Retry. Other runs
of the task and other sandboxes stay gated.

**Steps to reproduce**
1. Start a remote native run, and let its turn fail with a recoverable
transport error while the Runner cannot prove suspension.
2. The close keeps the Runner alive and quarantines the session.
3. The server schedules the same-run reattachment 30 s later.
4. The reattachment fails with `native_session_cleanup_quarantined`, and
the task needs Retry.

**Paperclip version or commit**
Deployed Core `c730872974` on staging. Reproduced on `master` at
`49f0030f06`.

**Deployment mode**
Hosted cloud (staging), Daytona remote sandboxes.

**Agent adapter(s) involved**
`paperclip_runner` native runtime with the Codex app server backend.

## What Changed

- `packages/paperclip-runner/src/native-session-runtime.ts`:
`executeNativeSession` takes a new option `retainedRunnerReattachment`.
With a remote cleanup scope, admission retires quarantine entries that
belong to the same run in the same remote scope. Removal stops later
close retries. A close attempt that is still in flight stays an
admission owner until it settles, so the old and new controllers do not
overlap. Other runs, other sandboxes and local backends keep the gate.
- `server/src/services/native-runtime/native-session-executor.ts`: sets
`retainedRunnerReattachment` only for a `reattach_remote_runner`
recovery of the same run.
- `doc/execution-semantics.md`: records the takeover.
- Tests:
- `packages/paperclip-runner/src/native-session-runtime.test.ts`: four
new cases. An operator-required close with a retained Runner. A
retryable close whose retries replay one failure. A close that a deploy
interrupts while it is in flight (the reattachment waits for it, then
takes over). Controls: another run, a run without the takeover option
and a local backend stay refused.
- `server/src/services/native-runtime/native-session-executor.test.ts`:
asserts that the option is set for a same-run remote reattachment and is
not set for a fresh sandbox run.

## Verification

- Red on `master` (runtime source restored): 3 of the 4 new runtime
cases fail with exactly the two staging errors (`the previous session is
still shutting down…` and `prior session cleanup remains incomplete`).
The control case passes.
- Red on `master` (executor source restored): the 3 executor assertions
fail.
- Green with this change:
- `cd packages/paperclip-runner && npx vitest run
src/native-session-runtime.test.ts` (294 passed)
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "exact
Stop cleanup|fresh sandbox|retirement"` (46 passed)
- `cd server && npx vitest run
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts
src/services/quarantined-task-retry.test.ts
src/__tests__/completed-turn-cleanup-hold-settle.test.ts` (61 passed)
- `npx tsc --noEmit` in `packages/paperclip-runner` and `server` passes.
- Staging evidence: runs `7b7ded99` and `27c487ef` show the sequence
above in their events and logs.

## Risks

- The takeover applies only when the server recovers the same run with
`reattach_remote_runner` in the same remote cleanup scope. This is the
case that #20923 keeps the Runner alive for. After a deploy the
quarantine is already gone and the same reattachment runs today, so the
takeover gives the in-process retry the same behavior as a restart.
- A close attempt that is in flight is never overlapped. The
reattachment waits for it within the existing admission grace.
- The reattachment still verifies Runner ownership and its controller
generation before it adopts the Runner. If the Runner is gone, the
existing relaunch or reconciliation path handles it.
- No persisted shape changes. Saved state from the previous release is
not affected because the quarantine lives only in process memory.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:15:40 -06:00
DottaandClaude Opus 5.5 7e7452e88f fix(native): keep a reattached Runner alive when its provider recovery deadline ends (#21321)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs on remote sandbox Runners. A same-run reattachment
adopts a live Runner after a failed attempt or a server deploy.
> - On staging, a remote Codex Runner is often gone when the next
attempt reattaches (`process_verification` / `process_not_running`). The
task then continues in a fresh session and loses its context.
> - The deploy does not stop the Runner. The logs show `controller
authority detached for restart; durable runner left alive`, and the new
controller adopts the same Runner after the deploy.
> - Our own cleanup stops it. The adopted Runner answers its recovery
reads slowly while its in-flight turn runs. The 900 s provider recovery
deadline then aborts the recovery, and the abort closes the transport
with `turn.stop` and `runner.suspend` and signals the process.
> - This pull request makes the recovery deadline detach from the
Runner, as a recovery read timeout already does. The server marks the
failure as an unanswered adopted Runner, so the next attempt replaces
the Runner from its saved journal and resumes the same provider thread.
> - The benefit is that a deadline no longer kills a live Runner and no
longer forces a fresh session. The task keeps its session context.

## Linked Issues or Issue Description

Refs #21314, #21272, #21161, #21023

**What happened?**
Codex runs on Daytona remote Runners failed often on staging. In three
sampled chains (attempt 2 of each), the same-run reattachment adopted
the live Runner, authenticated in under 1 s, and then waited the full
deadline: `native session provider recovery timed out after 900000ms`
(909 s, 930 s and 915 s). The deadline cleanup sent `turn.stop` and
`runner.suspend` (`provider turn stop timed out before runner
suspension`). The next attempt found the Runner gone 25–180 s later
(`process_not_running`, `runnerLifecycle: ready`, empty stderr) and
continued in a fresh session. During the 900 s wait, the in-flight turn
was still running Paperclip tools (`tool.execute` spans at 04:43 in a
recovery that started at 04:29).

**Expected behavior**
A remote Runner survives a recovery deadline and a control-plane deploy.
The next attempt continues the same provider session. A Stop still stops
the Runner.

**Steps to reproduce**
1. Start a Codex run on a remote Runner and let a turn start.
2. Fail the attempt so that its close keeps the Runner (for example, a
turn start that outlives its bound).
3. Let the same-run reattachment adopt the Runner, and make the Runner
stop answering recovery commands.
4. Let the provider recovery deadline expire.
5. See that the Runner process is gone and the next attempt reports
`process_not_running`.

**Paperclip version or commit**
master at b555087f0f (after #21314).

**Deployment mode**
Hosted staging (cloud), remote Daytona sandbox Runners, deploys every
20–45 minutes.

**Agent adapter(s) involved**
Native Codex (`paperclip_runner`, Codex app-server through
`paperclip-runnerd`).

## What Changed

- Add `NativeProviderRecoveryDeadlineError` (same message as before) and
use it for the provider recovery deadline in the native session runtime.
- In the Codex driver, a recovery aborted by this deadline detaches the
controller (no `turn.stop`, no suspend, no signal). A Stop and every
other abort still close and stop the Runner.
- On the server, `nativeRecoveryCommandUnanswered` recognizes the
deadline. The failure records `adoptedRunnerUnanswered`, so the next
claim carries `replaceUnansweredRunner`. That attempt fences the old
Runner lineage, launches a fresh Runner from the saved journal and
resumes the same provider thread (the #21272 path). It does not adopt
the slow Runner again for another 900 s.
- Update `doc/execution-semantics.md`.

## Verification

- Real `paperclip-runnerd` + fake Codex
(`runnerd-codex-transport-remote-retention.test.ts`): a retained Runner
is adopted, authenticates, then stops answering. A deadline abort leaves
it alive with no pending `turn.stop`/`runner.suspend`/`runner.shutdown`.
A Stop abort still stops it.
- Red/green: without the fix, the real-runnerd deadline test fails
(`expected 'gone' to be 'alive'`), the driver unit test fails (detach
called 0 times), and the server classification test fails. With the fix,
all pass.
- Measurements on real runnerd: the deadline cleanup took 10.0 s and
killed the Runner before. Now it returns in 0 ms and the Runner stays
alive. The replacement over the kept Runner's journal resumes the same
provider thread in about 2.2 s with one `thread/start` and one
`turn/start` (no replay).
- `npx vitest run
src/live/runnerd-codex-transport-remote-retention.test.ts
src/drivers/codex/codex-app-server-driver.recovery.test.ts
src/native-session-runtime.test.ts` (368 passed).
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "never
answered its recovery read"` (passed).
- `tsc --noEmit` for `packages/paperclip-runner` and `server` (clean).

## Risks

- A deadline now leaves a live Runner behind. The next attempt replaces
it through the launch fence, which fails closed when it cannot prove the
old lineage gone. This is the same risk as the existing recovery read
timeout detach.
- When the provider attempt budget is already spent, the kept Runner is
not replaced by this run. This also matches the existing timeout detach.
- Not in this PR: the upstream trigger. The first attempt abandons a
slow turn start (`runnerd did not report the provider turn identity` or
`turn.start timed out`) while Codex later runs that turn. Two sampled
chains then stopped the Runner on
`protocol_failure:tool_binding_mismatch` / `thread_binding_mismatch`
after a deploy reattachment. These need a separate fix.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:30:22 -06:00
DottaandClaude Opus 5.5 d9a4bb23fe fix(runner): wait for a slow Codex turn start while it reports progress (#21329)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Codex agents run through the Paperclip runner, and the server-side
transport starts each provider turn
> - After Codex accepts a turn, the transport waits a fixed 30 s for
that turn's `turn/started` before it reports the turn identity
> - Codex starts its MCP servers before it starts the turn, so a healthy
cold start in a loaded sandbox often takes longer than 30 s
> - The transport then failed a live turn with `runnerd did not report
the provider turn identity`, and the abandoned turn's late notifications
broke the reattachment (`did not name its execution owner`,
`runner_harness_state_mismatch: resumed_not_suspended`)
> - This pull request sizes the start window to the measured cold start
and extends it while the provider keeps reporting startup progress
> - The benefit is that slow but healthy Codex starts succeed on the
first attempt, and a dead start still fails at a bound

## Linked Issues or Issue Description

**What happened?**

On staging, Codex runs on Daytona failed at turn start with
`native_session_interrupted: runnerd did not report the provider turn
identity`. The run log shows the same sequence each time. The runner
accepts the turn. Codex then reports startup progress (a bubblewrap
config warning, `account/updated`, and `mcpServer/startupStatus/updated`
failures for an MCP server that is not logged in) for about 30 s. The
transport gives up 30 s after acceptance. A few seconds later, Codex
sends the turn's `turn/started`, which now arrives at a closed turn
(`protocol_failure:turn_binding_mismatch`). The same-run reattachment
then hits that orphaned turn. It fails with `Provider authoritative
notification item/started did not name its execution owner.` or
`runner_harness_state_mismatch: resumed_not_suspended`.

Of 198 recent runs in the staging company, 52 contain this turn identity
failure. In 68 successful starts, submit to `turn/started` took p50 25
s, p75 37 s, p90 50 s, p95 54 s and max 76 s. The fixed 30 s wait sits
near the median.

**Expected behavior**

A slow but healthy Codex start succeeds. A start that makes no progress,
whose Runner exits, or that the runner fails still fails at a bound and
recovers through the normal same-run resume.

**Steps to reproduce**

1. Start a Codex turn whose provider accepts the turn but sends
`turn/started` after the 30 s default window (on staging: several MCP
servers and a loaded sandbox).
2. Let the provider report startup notices during the wait, and let the
runner not prove the accepted turn live in time (on staging no turn was
adopted).
3. See `runnerd did not report the provider turn identity`, then the
reattachment conflict.

The new tests reproduce this with the fake Codex app server
(`--turn-started-delay-ms` with `--turn-started-progress-ms`) and a
runner snapshot that cannot prove liveness.

**Paperclip version or commit**

Staging Core `c730872974` (`2026.1106.0+2253`). Base of this PR:
`origin/master`.

**Deployment mode**

Authenticated, public, managed cloud (staging). Remote runner on
Daytona.

**Agent adapter(s) involved**

`paperclip_runner` with the Codex provider (`gpt-6.1-sol`).

## What Changed

- `runnerd-codex-transport.ts`: the default wait for an accepted turn's
`turn/started` is now 90 s (measured max 76 s), not 30 s. An explicit
`turnStartTimeoutMs` keeps its old bound.
- The wait restarts its window each time the runner commits a provider
notice, or a runner diagnostic that names the accepted turn. The total
wait stops at 180 s (option `turnStartProgressCeilingMs`, so tests can
lower it). A provider relaunch after the accepted `turn.start` does not
extend it.
- The adoption check (runner snapshot proves the accepted turn live) now
runs before each window end, not only before the first one. A Runner
exit and a runner failure of the pending turn still fail at once.
- `fake-codex-app-server`: new `--turn-started-progress-ms` flag sends
`mcpServer/startupStatus/updated` failures while it delays
`turn/started`.
- `doc/execution-semantics.md`: documents the start window, the progress
rule and the staging measurements.

## Verification

- Red on `origin/master` transport (with the new fake flag): `keeps
waiting for an accepted Codex turn while its startup keeps reporting
progress` failed after 4.6 s, and `gives a silent Codex cold start the
measured default start window` failed after 30.9 s. Both failed with
`runnerd did not report the provider turn identity`.
- Green: `npx vitest run
src/live/runnerd-codex-transport-deadlines.test.ts` passes 9/9. This
includes a ceiling test: provider notices every 1 s and no
`turn/started` fail at an 8 s test ceiling, and the test goes red with
the ceiling cap removed. It also includes the existing adoption,
no-adoption-within-bound and runner-failure tests, and a new test that a
silent start still fails within its bound.
- `npx vitest run` on the adoption, cut-turn-start, stall,
turn-start-timeout-retention, remote-retention, restart-continuation and
controller-outage transport suites passes 48/48.
- `npx vitest run src/live/runnerd-codex-transport.test.ts
src/live/live-session.test.ts src/live/turn-stream.test.ts`: 391/392
pass. The one failure (`stops a lost runnerd's provider and escaped tool
job from its lineage journal (journal=true)`, 1 s, process-lineage
timing under full-suite load) passes when run alone, both on this branch
and on `origin/master`.
- `tsc --noEmit` for `packages/paperclip-runner` and `cargo fmt --check`
pass.

## Risks

- A silent dead start (no Runner exit, no provider output, no runner
failure) now fails after 90 s, not 30 s. The Runner exit and
runner-failure paths still fail at once, so this applies only to a
provider that stops without a trace.
- A provider that keeps sending notices but never starts the turn waits
at most 180 s.
- No schema, API or UI change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, browser reads of staging run data).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Refs #21272, #21120

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:21:37 -06:00
Nicky LeachandClaude Fable 5.2 314b893678 fix(native-runtime): deliver assigned gateway tools to every native harness through the PRP relay (#21274)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent can have app connections. Paperclip gives the agent the
tools of those apps through an MCP gateway, with its approval and audit
rules.
> - Native harnesses reach the gateway in two ways today. In the relay,
the server keeps the gateway token and offers the tools over the
authenticated Runner channel. In the direct way, the server gives the
gateway URL and a bearer token to the harness.
> - Which way a harness uses comes from a list of special cases that
grew one harness at a time. Local Codex and all ACPX agents use the
direct way. The others use the relay.
> - The direct way puts a run-scoped token in a provider process. A
process that the agent starts can read it. A provider process that
serves more than one run can hold a revoked token.
> - This pull request replaces the list with one rule. Each native
harness gets its assigned gateway tools through the relay.
> - The benefit is one behavior for all native harnesses, and no gateway
token in a provider process.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

Refs #13942 (the relay), #17149 (ACPX accepts relayed tools) and #19912
(the design for turn-scoped ACPX MCP authority).

**What existing behavior does this improve?**

How a native harness reaches the tools of its assigned MCP gateway.

**Subsystem affected**

Native runtime in the server (`native-session-executor.ts`,
`native-session-resume.ts`). One Runner test file. One Rust test.

**Current behavior**

- `relayAssignedMcp` is true for plugin tasks, Cursor Cloud, OpenCode,
Codex on a remote target, and Dot.
- For local Codex and for each ACPX agent (Claude, Codex, Cursor, Pi,
Kimi, Grok, Copilot, Antigravity), Core puts
`PAPERCLIP_NATIVE_MCP_NAME`, `PAPERCLIP_NATIVE_MCP_URL` and
`PAPERCLIP_NATIVE_MCP_TOKEN` in the Runner environment. The ACPX host
gives them to the provider as an HTTP MCP server. The local Codex home
gets them in `config.toml`.
- The token is run-scoped. Core revokes it when the run ends.

**Proposed behavior**

- A native run that has an assigned gateway uses the relay. The
condition names no provider.
- Core builds the gateway tools on the control plane, offers them with
the other Runner tools, dispatches each call over PRP, and authorizes
each call again through the gateway.
- Core removes the three `PAPERCLIP_NATIVE_MCP_*` variables from each
Runner environment.
- The tool contract fingerprint has the relay marker for each target,
not only for remote targets.

**Reason and benefit**

- All native harnesses have the same behavior.
- No provider process holds a gateway token.
- A sandbox does not need to reach the gateway URL of the host.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`
- The list of special cases is gone. Core validates the binding and
builds the relayed tools when a binding is present.
- Core does not list the gateway for a review run, Grok Bot or Muse. The
tool authority never offered app tools to those. A gateway error can
therefore not fail those runs.
- OpenCode keeps its `on_demand` catalog. That option depends on the
provider, not on the relay.
  - Core removes the three variables from each Runner environment.
- `server/src/services/native-runtime/native-session-resume.ts`: the
relay marker in the tool contract fingerprint is unconditional.
- `doc/execution-semantics.md`: the text about the tool contract
fingerprint now states the rule, the tool names and the cutover.
- App tool schemas. New module
`packages/paperclip-runner/src/drivers/tool-input-schema.ts` compiles
each schema by itself, in its declared dialect (draft-07, 2019-09 or
2020-12), with no shared `$id` registry. Core uses it when it builds the
relayed tools (`assigned-mcp-tools.ts`). Core does not offer a tool
whose schema has an unsupported dialect or does not compile, and writes
a log entry with the tool name and the reason. The run does not fail.
The ACPX tool bridge and the OpenCode MCP bridge use the same function.
The tools of Paperclip itself stay strict.
- Large arguments. The ACPX sidecar sends the arguments of a tool call
to runnerd with no display limit. The limit is the tool-call admission
limit of runnerd, 256 KiB. A larger call fails with a tool error that
gives the limit. The sidecar does not dispatch it.
- Tests in `native-session-executor.test.ts`,
`native-session-resume.test.ts`, `runtime-context-materializer.test.ts`
and `acpx_provider_backend.rs`.

Not changed in this pull request: the direct code path. It is still in
the Runner, but it receives no binding. #21311 deletes it.

## Behavior that each harness sees

| Harness | Before | After |
|---|---|---|
| Codex on a remote target, OpenCode, Cursor Cloud, Dot, plugin tasks |
Relay | No change |
| Local Codex | Direct: a `paperclip-assigned` server in `config.toml` |
Relay. App tools are dynamic tools named `app_<name>_<digest>` |
| ACPX agents | Direct: a second HTTP MCP server | Relay. App tools are
on the `paperclip` bridge as `mcp__paperclip__app_<name>_<digest>` |
| ACPX with a preconfigured home | No gateway | No change |
| Grok Bot, Muse | Got the variables and ignored them | Do not get the
variables |

- A relayed tool name has a maximum of 64 characters. With the
`mcp__paperclip__` prefix it has a maximum of 80.
- Permission behavior keeps the same intent. For Claude in the
`approve-reads` and `approve-paperclip` modes, app tools ask for
approval, as before. For the other ACPX agents the permission decision
does not use the server name.
- In a planning run or an ask run, the relay offers only tools with read
risk. The direct gateway offered all tools. This is stricter.
- When the catalog is too large, each harness gets the two compact tools
`paperclip_search_assigned_tools` and `paperclip_call_assigned_tool`.
- The four helper tools for resources and prompts
(`paperclip_list_resources`, `paperclip_read_resource`,
`paperclip_list_prompts`, `paperclip_get_prompt`) are not in the relay.
No harness loses them: a run token permits only `tools/list` and
`tools/call` (`run-preparation.ts`), so these four tools already fail
with `gateway_token_action_denied` on the direct path.

## State saved by the previous release

- **Local sessions of each provider start again one time.** The tool
contract fingerprint changes for local targets. The next run sees a
different fingerprint, starts a new provider session with the full task
context in the same workspace, and keeps the old saved state. This is
the existing rotation for a tool contract change. Codex needs it,
because Codex cannot add tools to a thread that it resumes. For the
local providers that already used the relay, the restart is not
necessary. It is accepted so that the rule names no provider.
- **Remote sessions continue.** Their fingerprint does not change. For
ACPX, the attach of the next run replaces the saved tool set.
- **A kept Codex home.** Paperclip writes `config.toml` again at each
start and recovery. The old gateway server and its token are removed.
- **A deploy lands during a run, and the Runner survives.** The
controller attaches to the same Runner for the same run. The old
provider keeps its direct gateway connection. The controller detach
keeps the gateway token of that run, and Paperclip revokes it when the
run ends.
- **A deploy lands during a run, and the Runner is dead.** The Runner
starts again for the same run with its saved tool set and no direct
gateway. The assigned app tools are not available for the remainder of
that one run. The next run gets them through the relay. This applies to
each harness that moves to the relay. It has no test.
- Paperclip refuses no saved state and discards none.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-resume.test.ts`: `1508 passed
| 7 skipped`, 1 failed. The failed test is "archives a damaged prior
epoch before completing guarded same-task replacement with real runnerd"
(`ENOENT … authority-epochs`). It fails in the same way on `master` at
53608bd1d6 and with the product change removed, on macOS.
- `cd server && npx tsc --noEmit -p tsconfig.json`: no errors.
- `cd packages/paperclip-runner && npx vitest run
src/drivers/runtime-context-materializer.test.ts`: `18 passed`. `npx tsc
--noEmit -p tsconfig.test.json`: no errors.
- `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`: no
changes. `cargo test -p paperclip-runner-core --lib
acpx_provider_backend`: `50 passed`.

- `npx vitest run src/drivers/tool-input-schema.test.ts
src/drivers/runner-tool-bridge.schemas.test.ts
src/drivers/runner-tool-bridge.test.ts
src/cli/acpx-runtime-sidecar.test.ts src/drivers/opencode` in
`packages/paperclip-runner`: `21 passed (21)` files, `377 passed` tests.
`pnpm run typecheck:typescript`: no errors.
- `cargo test -p paperclip-runner-core --test acpx_event_payload --test
tool_input_schema_fixtures`: 10 and 3 passed.
- Schema tests use one shared fixture
(`tool-input-schema-fixtures.json`) for the TypeScript compiler, the
tool bridge and the Rust `begin_call`: a draft-07 schema, a second
schema with the same `$id`, 2019-09, 2020-12, draft-04, an unknown
dialect, a `$ref` with no target, and an invalid keyword. Before the fix
the bridge failed with `no schema with key or ref
"http://json-schema.org/draft-07/schema#"`.
- A tool schema whose `$id` is the URI of a meta-schema cannot damage
the shared compiler. Before this fix, `removeSchema` deleted that
meta-schema, and later schemas failed to compile for each run in the
process. A test with 7 reserved ids failed before the fix and passes
now.
- Core limits the compile work of each run start. A schema over 512 KiB
is not compiled, and one listing compiles a maximum of 768 KiB of
schemas. These are the limits of the Runner. A tool over a limit is
served only through the search and call tools. Core keeps the outcome
for each schema digest (1,024 entries) and yields to the event loop
before each compile. Four tests failed before this change.
- A sidecar test sends a tool call with 200 KiB of arguments and
receives them byte for byte, refuses a call with 300 KiB with the named
error, and keeps the replacement of a lone surrogate.

Tests, each seen to fail without the product change:

- "keeps assigned MCP credentials on the control plane for %s" now runs
for remote Codex, local Codex, OpenCode and ACPX Claude, Pi, Kimi,
Copilot and Cursor. It checks that Core builds the relay, offers the
tool, dispatches a call, and gives no `PAPERCLIP_NATIVE_MCP_*` variable
to the Runner. Without the change, the five new harness cases fail.
- "relays the assigned gateway when a deploy lands mid-run…" runs for
Claude and Pi.
- The fingerprint tests keep the old local fingerprint as a retained
value and check that the new one is different. The remote fingerprints
for Codex and OpenCode do not change.
- "drops a retained direct gateway server when the relayed launch has no
binding": a Codex home with the old `config.toml` entry loses that entry
at the next start and at a recovery of the same run.
-
`next_run_attach_offers_relayed_gateway_tools_to_a_direct_gateway_checkpoint`
runs for Claude and Pi.

Not run: a live run with a real provider and a real app connection for
each harness. Not run: `pnpm verify` and the full suite. CI supplies
that coverage.

## Risks

- **An ACPX app call has a limit of 256 KiB for its arguments.** The
direct gateway accepted a request body of 10 MB. Codex and OpenCode
permit 480 KiB. To get nearer to 10 MB, runnerd needs a larger admission
limit for tool calls or a protocol that sends arguments in parts. That
is not in this pull request.
- **A tool with a draft-04 or draft-06 schema is not offered.** Ajv 8
does not support draft-04, and the TypeScript and Rust validators must
agree. On the direct path the provider got such a tool.
- **Core compiles app schemas on its event loop.** Before this pull
request Core did not compile them. With the limits above, one valid
schema near 512 KiB can hold the loop for about 0.65 s, and one listing
for about 1.2 s in total (measured on a developer machine). An app that
changes its schemas at each listing gets no help from the cache. A
worker thread with a timeout would remove this; it is not in this pull
request.
- **Tool names change for local Codex and all ACPX agents.** A saved
agent instruction that names a gateway tool directly does not match the
new name.
- **Each local native session starts again one time after the deploy.**
The provider conversation of that session does not continue. The task
context and the workspace stay.
- **A run that is in flight at deploy, with a dead Runner, has no app
tools until it ends.**
- **Names of 80 characters are not tested live** with Kimi, Copilot,
Grok, Antigravity and Codex through ACPX. Pi already shortens a name
longer than 64 characters.
- **Managed Claude and AWS AgentCore now receive relayed app tools.**
This is not tested live.
- **A gateway token expires after 1 hour.** The relay keeps the token on
the controller for the whole run, as OpenCode and remote Codex do today.
A longer run loses its app tools on both paths. This pull request does
not change that.
- **The gateway URL must have the path `/mcp/gateways/<id>`.** With a
path prefix in the API base URL, a run with an assigned gateway fails
with `assigned native MCP gateway path is invalid`. This was true for
the relayed harnesses before. It is now true for all.
- No test covers the rule that Core does not list the gateway for Grok
Bot and Muse.

## Follow-up

The direct path is now unreachable. #21311 deletes it. That pull request
is in the same stack, above the pull requests that change the same ACPX
host files.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change, reviewed the diff and ran the verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), investigated each harness
and wrote the change and the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 00:16:58 -08:00
Nicky LeachandClaude Opus 5.5 2483cb46c2 feat(plugins): deliver configured environment to plugin Runner tasks (#21280)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent gets environment variables from its execution environment,
its own configuration, its project, and its routine. Secrets are
resolved for each run.
> - A plugin environment can run the agent as a native Runner task on
its own host, through the environment task contract.
> - That contract has no field for the agent's variables. Runs on a
plugin environment get only the variables that the plugin sets itself.
> - This pull request adds a versioned contract field for the values,
sends them on a session handoff, lets Codex commands inherit the
variables in an external sandbox, and lets a plugin protect the
variables that it sets itself.
> - The benefit is that plain and secret variables work on plugin
environments in the same way as on local environments.

## Linked Issues or Issue Description

Refs #21233. That pull request by @lucktastic finds the same gap and
proposes the same capability and field names
(`configuredEnvironmentVersion: 1`, `configuredEnvironment`). This pull
request uses the same names, so providers implement one contract. It
differs in these places:

- Codex commands get the variables through an external-sandbox denylist,
not through key names sent at `prepare`. A retained Runner does not need
a new preparation when a key is added or removed.
- A session handoff `resume` gets the values, because it starts a new
Runner on another computer. Value changes on the same computer need no
new field: Core's process reuse key includes the values, so Core starts
a new task.
- A driver can declare `protectedEnvironmentKeys`. Paperclip rejects or
warns when one is saved.
- The run log names skipped variables only for an older provider.

Thank you to @lucktastic for the diagnosis and the contract shape.

**What happened?**
An agent has environment variables, plain and secret. Its runs on a
plugin task environment do not get them. `PAPERCLIP_CONFIGURED_ENV_KEYS`
in the run lists only the variables that the plugin sets.

**Expected behavior**
Each run gets the agent's configured variables.

**Steps to reproduce**
1. Set a plain and a secret environment variable on an agent.
2. Assign the agent to a plugin environment that provides Runner tasks.
3. Start a run. Read the names in `PAPERCLIP_CONFIGURED_ENV_KEYS` in a
command that the agent runs.
4. The agent's names are not in the list.

## What Changed

- **SDK contract** (`packages/plugins/sdk/src/environment-tasks.ts`):
- A provider advertises `configuredEnvironmentVersion: 1` on `describe`.
- `submit` gets `configuredEnvironment`. `resume` gets it only for a
session handoff.
- Names, count, and sizes have the same limits as the Runner projection.
- **Server**:
- `createRunnerdBackendWithinSessionClaim` gives the run's configured
projection to the plugin task owner. The projection has no provider
credentials and no generated runtime keys.
- `createPluginRunnerTaskOwner` sends the fields only to a provider with
the capability. An older provider gets no new field, and the run log
names the variables that were not set. The log never prints values.
- **Codex** (`codex-security-config.ts`):
- An externally sandboxed launch uses `inherit="all"`,
`ignore_default_excludes=true`, and an `exclude` list of Codex's own
credentials, instead of an `include_only` list fixed at preparation
time.
- A test reads the Runner's Codex environment keys, and fails if a new
key is not classified.
  - Host launches keep the bounded allowlist.
- **runnerd** (`codex_provider.rs`): `runtime_notification_codex_args`
still grants the notification files to read-only profiles when no
allowlist is present. Plan mode in an external sandbox needs these
grants.
- **Protected keys**:
- An environment driver can declare `protectedEnvironmentKeys` in its
manifest.
- Paperclip rejects a protected key on an agent or environment that uses
the driver. The 422 has code `protected_environment_variable`.
- On a project, a routine, or another agent, Paperclip saves the key.
The response has a `configurationWarnings` notice that names the
environments. The project and routine pages show the notice as a toast.
The agent pages already show configuration warnings.
- **Docs**: `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md` describes the
provider obligations.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the reviewed
SDK source pin.

## Verification

- `packages/plugins/sdk`: `vitest run tests/environment-tasks.test.ts`
(38 pass). Accepts bounded values. Rejects bad names, NUL bytes, too
many variables, and oversized values.
- `server`: `vitest run
src/services/native-runtime/plugin-runner-task.test.ts` (95 pass).
Values go only to a provider with the capability, never in `prepare`.
`resume` sends values only for a handoff.
- `server`: `vitest run src/services/protected-environment-keys.test.ts
src/__tests__/project-routes-env.test.ts` (10 pass). Rejection, notices,
and the project route response.
- `server`: all route suites that import the project, routine,
environment, or agent routes (45 files, 1689 pass).
- `packages/shared`: `vitest run src/validators/plugin.test.ts` (64
pass).
- `packages/paperclip-runner`: `vitest run src/drivers/codex
src/configured-environment.test.ts
src/live/runner-provider-launch.test.ts` (444 pass).
- `packages/paperclip-runner/runner`: `cargo test -p
paperclip-runner-core --lib runtime_notification` (9 pass, 2 ignored as
before).
- `pnpm typecheck` in `packages/shared`, `packages/plugins/sdk`,
`packages/paperclip-runner`, `server`, and `ui`.
- Not run locally: `pnpm verify --full` and browser suites. Required CI
covers the final revision.

## Risks

- **Existing providers:** low risk. Core sends no new field to a
provider without the capability. Their runs behave as before, with one
added log line when variables were skipped.
- **Codex in an external sandbox:** commands now see every variable in
the Runner environment except Codex's own credentials. The execution
environment builds that environment from an allowlist. If the Runner
gives Codex a new credential, the drift test fails until the key is
classified.
- **Saving env:** new rejections happen only when an installed driver
declares `protectedEnvironmentKeys`. No driver declares them today.
- **Providers:** they must keep the values transient and never save or
log them. The doc lists these obligations.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:11:23 +00:00
Nicky LeachandClaude Fable 5.2 ff91bfe2dc fix(runner): withhold Claude's self-scheduling tools from Paperclip ACPX sessions (#21276)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip owns scheduling. An agent uses Paperclip tools such as
task monitors and routines to continue work later.
> - Paperclip Runner runs Claude Code through the ACPX sidecar.
> - Claude Code offers its own scheduling tools to the model:
`CronCreate`, `CronDelete`, `CronList` and `ScheduleWakeup`. With these
tools the provider can wake itself. Paperclip does not see or govern
that work.
> - Today the Runner stops the Claude process when a turn ends, so these
tools have little effect. A later change keeps the Claude process alive
between turns.
> - This pull request removes the four tools from Claude sessions that
Paperclip starts through ACPX.
> - The benefit is that a Claude agent schedules work only through
Paperclip.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

Refs #19912 (the design for turn-scoped ACPX MCP authority).

**What existing behavior does this improve?**

The set of tools that Claude Code offers the model in a Paperclip ACPX
session.

**Subsystem affected**

Paperclip Runner, ACPX provider (Claude).

**Current behavior**

A Paperclip ACPX Claude session offers the model 26 built-in tools. Four
of them are `CronCreate`, `CronDelete`, `CronList` and `ScheduleWakeup`.

**Proposed behavior**

- The Claude settings that Paperclip writes into the session home deny
the four tools. Claude Code then does not offer them.
- Each Claude launch sets `CLAUDE_CODE_DISABLE_CRON=1`. This stops the
cron scheduler of Claude Code and removes the three cron tools.
- All other tools stay. `SendMessage`, `ListAgents`, `Workflow`, `Agent`
and `TaskStop` do not change.

**Reason and benefit**

The provider cannot schedule its own wake-ups. Paperclip tools stay the
only way to schedule work.

## What Changed

- `packages/paperclip-runner/src/drivers/acpx/permission-policy.ts`: new
constant `CLAUDE_SELF_SCHEDULING_TOOLS` with the four tool names.
- `packages/paperclip-runner/src/drivers/acpx/runtime-sandbox.ts`: the
Claude `settings.json` now always has `permissions.deny` with the four
tools, in each permission mode. The `allow` rules do not change.
- `packages/paperclip-runner/src/drivers/acpx/environment.ts`:
`createSanitizedAcpxSpawnInput` sets `CLAUDE_CODE_DISABLE_CRON=1` for
each Claude launch. The host cannot override it.
- New test `claude-scheduling-tools.claude-acp.test.ts`. It prepares the
session home with the production `prepareAcpxRuntimeSandbox`. It then
runs the pinned `claude-agent-acp` and the pinned Claude Code client
against a local fake Anthropic API. It reads the tools that the client
offers the model.
- Unit tests in `environment.test.ts` and `runtime-sandbox.test.ts`. One
assertion in `runtime-host.test.ts` now expects the `deny` list.
- No change to the `claude-agent-acp` patch, the lockfile or a pinned
digest. No change to Codex.

## State saved by the previous release

- A session that the previous release saved continues with no change.
The new restriction is not part of a saved identity or digest.
- Each time Paperclip starts Claude for a session, it writes its own
`settings.json` again with the deny list and sets
`CLAUDE_CODE_DISABLE_CRON`. This includes a resumed session and a start
after an upgrade.
- A resumed session keeps its identity, its home and its transcript. It
loses the four tools from its next start.
- A Claude process that the previous release started keeps its tools
until it stops. Paperclip does not refuse it and does not classify it
differently. Its next start gets the restriction.
- No refusal code is necessary, because no saved state becomes
incompatible.

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/drivers/acpx/claude-scheduling-tools.claude-acp.test.ts`
- Without the product change: `2 failed` (`expected [ Array(26) ] to not
include 'CronCreate'`).
  - With the product change: `2 passed`.
- `npx vitest run
src/drivers/acpx/claude-scheduling-tools.claude-acp.test.ts
src/drivers/acpx/environment.test.ts
src/drivers/acpx/runtime-sandbox.test.ts
src/drivers/acpx/runtime-host.test.ts`: `4 passed (4)` files, `147
passed` tests.
- `npx tsc --noEmit -p tsconfig.json` in `packages/paperclip-runner`: no
errors.
- Tools offered after the change, isolated home: the same 26 tools
without `CronCreate`, `CronDelete`, `CronList` and `ScheduleWakeup`.
- `npx vitest run src/drivers/acpx/runtime-host.test.ts -t "previous
release"`: the new test "resumes a Claude session saved by the previous
release and withholds self-scheduling tools on relaunch" fails without
the product change and passes with it. The test saves a session, puts
its home into the shape of the previous release, and resumes it through
`AcpxRuntimeHost.open`.
- `npx tsc --noEmit -p tsconfig.test.json` in
`packages/paperclip-runner`: no errors.
- The fixture in `claude-scheduling-tools.claude-acp.test.ts` now stops
the process group of the provider and waits for it before it removes the
fixture directory.
- Not run: a live run with a real Claude account, and a remote Runner.
Not run: `pnpm verify` and the full suite. CI supplies that coverage.

## Risks

- **A preconfigured home still offers `ScheduleWakeup`.** In that mode
Paperclip does not write the Claude settings of the user, so the deny
list does not apply. The environment variable removes only the three
cron tools. To remove `ScheduleWakeup` there, the Runner must pass
`disallowedTools` to the session. ACPX 0.22 does not forward that
option. This is a follow-up.
- **The new test runs the real pinned Claude Code client.** It needs the
platform binary, as `runner-tool-bridge.claude-schema.test.ts` does
today. It uses no credentials and no network.
- **A prompt or skill that tells Claude to use these tools no longer
works.** No Paperclip instruction refers to them.
- The prepared harness home and remote Runners use the same code path.
They are not tested separately.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change, reviewed the diff and ran the verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), investigated the options
and wrote the change and the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 22:17:24 -08:00
DottaandClaude Opus 5.5 bcadbde449 feat(runner): record the OpenRouter host that served each OpenCode model call (#21298)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode agents can use models through OpenRouter, and the Runner
proxies those model calls
> - OpenRouter routes one model across many hosts, and these hosts use
different quantizations (fp4, mxfp4, fp8)
> - Chat evaluations with identical prompts behave very differently
between runs, but the run log does not record which host served a call
> - The run usage shows `provider: "deepseek"` and `biller:
"openrouter"`, and `providerRequestId` is always `null`
> - This pull request records the serving host and the OpenRouter
generation ID for each model call in the run log
> - The benefit is that we can compare evaluation cohorts by host, and
the cost event names a real OpenRouter generation

## Linked Issues or Issue Description

**Subsystem affected**
Runner OpenCode provider proxy, runnerd usage normalization, and native
run usage accounting.

**Problem or motivation**
OpenRouter's default routing sends a request to one of many hosts. Each
host can use a different quantization. Chat evaluation results change a
lot between runs with identical prompts. Paperclip does not record which
host served a call, so we cannot compare results by host.
`usageJson.providerRequestId` is always `null` for OpenCode runs.

**Proposed solution**
Every OpenRouter completion response has a top-level `provider` field
and an `id` (the generation ID). Each streamed chunk has them too. The
Runner proxy reads them from the first chunk after it forwards the
bytes. The Runner attaches them to the usage of that call. The values go
to the run log, and the run's first generation ID goes to
`usageJson.providerRequestId`.

**Alternatives considered**
A new PRP event type such as `provider.upstream.served` would need a new
event enum value. An older control plane rejects an unknown event type.
A `harness.diagnostic` event shows in the task chat as a diagnostic. The
usage payload already flows to every consumer, and older consumers
ignore extra fields. Thus this change uses the usage payload.

**Additional context**
Run-log change only (AGENTS.md section 7). The data stays in
`heartbeat_run_events` and `heartbeat_runs.usageJson`. It does not touch
`packages/shared/src/telemetry/` or OpenTelemetry.

## What Changed

- New `createUpstreamServedTap`
(`packages/paperclip-runner/src/drivers/opencode/upstream-served.ts`).
It is a pass-through stream. It pushes each chunk downstream first, then
it reads `provider` and `id` from the first SSE `data:` event (or from a
JSON body at its end). It scans at most 64 KiB of a stream and 1 MiB of
a JSON body. It skips encoded bodies and unknown content types. Any
parse problem or recorder error stops the scan and leaves the response
unchanged.
- `startOpenCodeProviderProxy` uses the tap only for 2xx responses. The
OpenCode session keeps the most recent served call. Each
`item.completed` usage payload and `session.usage()` now carry
`upstreamProvider` and `providerRequestId`. The usage dedupe fingerprint
does not include these fields, so the number of usage events does not
change.
- The OpenCode app-server proxy adds the two fields to
`thread/tokenUsage/updated`. Rust `usage.reported` now fills its
existing `providerRequestId` field and adds an optional
`upstreamProvider`. The TypeScript runnerd transport keeps both fields
when it rehydrates the usage notification. `usage.schema.json` documents
`upstreamProvider` (the protocol manifest and schema bundle are
regenerated).
- Server native turn accounting keeps the first generation ID the run
reports. The usage checkpoint and the native adapter result carry it as
`providerRequestId`, so `usageJson.providerRequestId` and the final cost
event name it.
- `doc/run-log-events.md` documents the fields in a new "OpenRouter
serving host" section.

## Verification

- Red/green: `opencode-server-driver.test.ts` "passes OpenRouter's
stream through unchanged and records the host that served it on the
turn's usage". The test sends a streamed fixture with a comment line, a
host chunk, a malformed chunk, and `[DONE]` through the proxy. It checks
that the bytes are identical and that the turn's usage names `Novita`
and the generation ID. The test fails without the proxy wiring.
- `upstream-served.test.ts`: streamed fixture, chunk split inside a
multibyte character, non-streamed JSON, malformed chunks, a truncated
JSON body, an encoded body, the 64 KiB limit, and a recorder that
throws.
- `opencode-proxy-usage.test.ts`, `runnerd-codex-transport.test.ts`
(rehydration), Rust
`provider_events::tests::carries_the_serving_host_and_generation_on_proxy_usage`,
and server `native-usage-accounting.test.ts` (first generation ID stays
stable across turns).
- Local: OpenCode driver, OpenCode proxy, and runnerd transport suites
(31 files, 596 tests) pass. `cargo test -p paperclip-runner-core` passes
(one timing test failed once under load and passed alone). Runner and
server `tsc --noEmit` pass. Protocol manifest, type, and coverage checks
pass.
- Latency: a local benchmark proxied a 400-chunk SSE stream 200 times
with the tap and 200 times without it. The median time to first byte was
292 µs with the tap and 259 µs without it (+33 µs, which is the extra
stream stage). The total stream time changed from 491.4 ms to 492.5 ms.
The tap spends about 5.5 µs of CPU to inspect the first chunk, after it
has forwarded that chunk. OpenRouter time to first token is hundreds of
milliseconds.
- Paid evidence: one single-case run of
`chat-features.runner-opencode.local.skills.fork-read-only` on this
branch. The result is in a comment on this pull request.

## Risks

- Low risk. All new fields are optional and additive. An older control
plane ignores the extra usage fields. `usage.schema.json` is not bound
by the event schema, and no validator enforces it on `usage.reported`.
- Saved state from the previous release: this change adds no persisted
run, session, checkpoint, or launch state. A saved session resumes
unchanged. A run that is in flight during a deploy reports usage without
the new fields until its Runner is new. `usageJson.providerRequestId`
then stays `null`, as before.
- `providerRequestId` on the run and its cost event is the run's first
generation ID, not each call's ID. The value is stable, so a late usage
receipt still matches the posted charge. Per-call IDs and hosts are in
the run log.
- If OpenCode starts a new model call before it reports the usage of the
previous call, a usage event can name the newer call. OpenCode reports a
message's usage when its step ends, so this is not expected.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code as an agent with tool use (shell, file edits, test runs). Extended
reasoning was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:15:53 -06:00
Nicky LeachandClaude Opus 5.5 e65a3095a7 fix(runner): quiesce the Core-dialed route once the Runner commits its suspend (#21245)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Core runs agent sessions through the Paperclip Runner. With provider
ingress, Core dials the Runner over a PRP WebSocket.
> - When Core closes a session, it asks the Runner to suspend. The
Runner suspends, reports the result, and exits.
> - Core's ingress connection did not know that this exit was expected.
It treated the drop as a transport loss and redialed the Runner. Each
redial got a 404 until the close released the route.
> - The release (park) path already quiesces the connection before the
Runner goes away. The suspend path did not.
> - This pull request makes the suspend close quiesce the route in the
same way, once the Runner commits its suspend.
> - The benefit is that a clean suspend no longer looks like a lost
Runner. A drop before the suspend is committed still redials and
recovers as before.

## Linked Issues or Issue Description

**What happened?**
After a provider-ingress session closed cleanly, Core redialed the
Runner's ingress endpoint several times and got 404 each time. The
Runner state was `suspended`, `runner.drain` and `runner.suspend` had
both completed, and the Runner had exited with code 0. The redials
followed the outbound reconnect backoff (about 250, 500, 1000 and 2000
ms) and stopped when the registration release closed the outbound.

**Expected behavior**
After Core commits to `runner.suspend` and the Runner reports it
completed, the Runner's exit is expected. Core must not redial or
classify the drop as a transport loss. The release path already works
this way: it calls `outbound.quiesce()` before it releases the task.

**Steps to reproduce**
1. Open a session through a Core-dialed route (plugin provider ingress
or the `listen_ws` driver).
2. Close the session. Core drains, queues `runner.suspend`, and the
Runner suspends and exits.
3. See that the outbound reconnect loop redials the gone Runner until
the registration release closes it.

**Paperclip version or commit**
`master` at b4ca19f3a8.

**Agent adapter(s) involved**
Not adapter-specific (core bug). It affects any Core-dialed Runner
route: plugin provider ingress and the `listen_ws` driver.

**Additional context**
The redials do not fail the session. The plugin outbound uses a 7-minute
recovery grace, and `#failTransport` ignores failures after
`#closeAndSettle` sets `#closed`. The new test confirms this: without
the fix, the close still settles, but the route redials. This PR removes
the misleading redial. It does not change how a run's own failure is
classified.

## What Changed

- `RunnerExecutionDriver.controlPlaneRegistration` results can now
include an optional `quiesce()`. Core calls it when it has committed the
Runner's exit. The route then stops liveness and reconnect, and it keeps
the established receive/ACK path.
- `DurablePrpControlPlane.onRunnerExitCommitted(listener)` is new. The
authority calls its listeners after it saves a completed
`runner.suspend` or `runner.shutdown` result and before it sends the
acknowledgement. The Runner exits when it gets that acknowledgement, so
a route quiesced here never sees the exit as a loss.
- `awaitRunnerSuspensionBarrier` calls a new `onExitCommitted` callback
once, when its suspend command is `completed`. It also calls the
callback when it reuses a completed receipt and the durable state is
still `suspended`. A rejected, failed or unanswered suspend never calls
it.
- `DurablePrpCodexTransport` registers each control-plane route through
one `#registerControlPlane` helper. The helper records each route's
`quiesce` until that route is released. The suspend close subscribes to
`onRunnerExitCommitted` while its barrier runs and quiesces all of these
routes from that notification. The barrier's own `onExitCommitted`
covers a completed receipt that was acknowledged before this close.
Quiescing clears the tracked routes, so it runs once per close. The
helper copies property descriptors, so getters such as the `listen_ws`
driver's lazy `failure` stay live.
- The plugin task registration (`plugin-runner-task.ts`) and the
`listen_ws` driver registration (`native-runner-execution-driver.ts`)
implement `quiesce` with `outbound?.quiesce()`. This is the same call
that the release path makes.

## Verification

- New transport test with a real runnerd:
`packages/paperclip-runner/src/live/runnerd-codex-transport-suspend-route.test.ts`.
The test route acts like a Core-dialed provider ingress: on a socket
drop that it was not told to expect, it records a redial into a 404.
Before the fix the events were `connected, redial: 404, release`. After
the fix they are `connected, quiesce after runner.suspend completed,
release`. The close settles, and the durable Runner state is
`suspended`. A second case delays the durable-state read by 1.5 s, so
the Runner exits while the barrier still reads. That case fails without
the authority hook and passes with it.
- New barrier unit test in `runnerd-codex-transport.test.ts`. It shows
that the exit is committed only after the retried suspend completes,
never on a rejected or unanswered suspend, and on a reused receipt only
when the state is `suspended`.
- New server tests for the plugin task and `listen_ws` registrations.
`quiesce` reaches the active outbound and does not close it. Release
still closes it.
- Existing tests pass. These include the outbound release-path quiesce
test, the half-open and lease-expiry failure tests (a drop before the
commit still fails after its grace), and the plugin release/resume
facade tests.
- `pnpm verify` exited 2 (the change touches files outside the resolved
test graphs), so I ran focused suites. Locally: 8 runner transport files
and the 4 control-plane test files, 667 tests
(`runnerd-codex-transport*.test.ts` for suspension, release/resume,
remote retention, settlement, warm attachment and attachment recovery,
plus the new suspend-route test), and 5 server files, 148 tests (both
plugin Runner integration tests, the plugin task, the execution driver
and the outbound). Required CI supplies the broad coverage.
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- Low risk. `quiesce` is optional. Registrations that do not supply it
behave as before.
- The quiesce happens only after the Runner reports a completed suspend.
A socket drop before that point still redials and recovers.
- The quiesced route keeps its receive/ACK path, so the Runner's last
frames before exit are still acknowledged.
- No persisted state or protocol change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:19:24 -08:00
DottaandClaude Opus 5.5 497ccba8f1 fix(native): replace a reattached Runner that left its recovery read unanswered (#21272)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs execute in a remote Runner that the server
controls over the durable PRP channel
> - When a run is interrupted, a same-run retry reattaches the live
Runner and reads its session (`session.snapshot`) before it continues
> - On staging, the reattached Runner authenticated and restored its
session, and then did not answer the read
> - The recovery kept that Runner, so each later attempt adopted the
same Runner and failed the same way until the attempt budget ended the
run
> - This pull request makes the next attempt replace that Runner with a
fresh Runner over its saved journal, and makes a failed Codex history
read say what failed
> - The benefit is that an interrupted Codex run continues without a
manual Retry, and its provider turn is not sent again

## Linked Issues or Issue Description

Refs #21161, Refs #21053, Refs #20853

**What happened?**

A Codex agent on Daytona ran recovery runs (wake
`issue_recovery_action_restored`). In one hour, 11 of about 35 finished
runs failed. All failed the same way:

- `native_session_recovery_failed: HarnessCapabilityUnavailableError:
read is unavailable: Error: runnerd exited while waiting for
session.snapshot` (8 runs)
- `native_session_recovery_failed: HarnessReconciliationError:
codex_history_read_failed: thread/turns/list requires supported
paginated history; Error: provider_transport_failed: PRP command
session.snapshot timed out (runner connected; pending ahead 0; last
runner event seq 338 76s ago)` (3 runs)

The run log of each run shows one sequence:

1. Attempt 1 fails with `runnerd did not report the provider turn
identity`. The close leaves the Runner alive and unsettled for the
same-run reattachment.
2. Attempt 2 adopts that Runner (same PID). The Runner authenticates.
The log shows `runnerd restored its durable PRP session and provider
thread`. Then the driver's recovery read (`thread/read` or
`thread/turns/list`, which both send `session.snapshot`) gets no answer.
The Runner still answers WebSocket pings, but it sends no command result
and almost no events (`pending ahead 0`, last event 76 s to 391 s
earlier). The recovery treats this as transient and detaches. The Runner
stays alive.
3. Attempt 3 adopts the same Runner again and fails the same way. In
some runs a protocol fault (`thread_binding_mismatch`) closes the
transport during the read, so the message is `runnerd exited while
waiting for session.snapshot`.
4. The provider attempt budget ends the run.

#21161 did not help. Its wait extension applies only while a queued
answer is being committed. Here the Runner sent no answer at all.

The `paginated history` text was also wrong. `codex-history.ts` called
every failed history read unsupported, also a transport timeout.

**Expected behavior**

A Runner that left a recovery read unanswered is not adopted again. The
next attempt starts a fresh Runner from the saved journal, resumes the
provider session from the saved checkpoint, and does not send the
provider turn again. A failed history read reports the real failure. A
Codex build that refuses paginated history falls back to one complete
thread read.

**Steps to reproduce**

1. Start a Codex run in a remote (Daytona) Runner.
2. Let the first attempt fail at turn start, so that the close keeps the
Runner alive.
3. Let the reattached Runner stop answering commands after it
authenticates (for example, stop the Runner process after its first
answer).
4. See `PRP command session.snapshot timed out (runner connected;
pending ahead 0; ...)` on attempt 2 and the same failure on attempt 3,
with the same Runner PID.

**Paperclip version or commit**

Staging Core `57aeda190b` (contains #21161 and #21120). Fixed on
`master` after `53608bd1d6`.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`, `codex_app_server`).

## What Changed

- `nativeRecoveryCommandUnanswered` (executor) finds a
`native_session_recovery_failed` whose Runner did not answer a recovery
command (`PRP command <type> timed out` or `runnerd exited while waiting
for <type>`). Identity, authority, integrity and incompatibility
refusals do not match.
- When such a failure ends a `reattach_remote_runner` attempt on a
sandbox target, the failure detail records `adoptedRunnerUnanswered:
true`.
- `claimNativeRestartRecoveries` copies it to the next claim as
`replaceUnansweredRunner: true`, in the same way as
`exitedBeforeSessionOpen`.
- `verifyRemoteRunnerOrSettledExit` with `replaceUnansweredRunner` keeps
all identity checks. When the old Runner is still alive, it returns no
adopted process, and the attempt launches a fresh Runner from the saved
journal. The existing launch fence stops the old `paperclip-runnerd` on
that state folder and its whole lineage, and fails the launch closed
when it cannot prove them gone. A Runner that already exited mid-turn
keeps the existing fail-closed path, because its provider or tool
processes can survive where the fence cannot find them. The task then
continues in a fresh session from the saved work. The attempt budget is
unchanged.
- `codex-history.ts`: a history read that fails without a provider
answer is now `codex_history_read_failed: <method> read failed;
<error>`. A provider refusal of the method (`-32601` or `-32600`) on the
first page falls back to one `thread/read` with `includeTurns: true`.
That fallback fails closed when the thread differs, or when turns, items
or turn identities are missing. A refusal on a later page still fails
closed.
- `doc/execution-semantics.md` describes both rules. The launch-fence
code comment now names the replacement case.

### Changed existing test

`codex-history.test.ts` "does not turn an interrupted page or
unsupported method into idle history" used a plain `Error("method not
found")` on page 2 and expected the `supported paginated history` text.
A plain error is now a failed read, not an unsupported method. The test
now uses a real provider refusal (`CodexRpcError` `-32601`) on page 2
and still expects a fail-closed rejection. A new test covers the
failed-read message.

## Verification

- Red before the fix, green after:
- `server/src/services/native-runtime/native-session-executor.test.ts` ›
"a same-run reattachment whose adopted Runner never answered its
recovery read": the replacement and staging-message tests failed on the
old code (the Runner was adopted again, the three staging messages were
not recognized). Two further tests ("never resumes beside an exited
unanswered Runner's lineage") failed on the first revision of this PR,
which relaunched beside an exited mid-turn Runner, and pass now. All 15
pass.
- `native-runner-restart-recovery.integration.test.ts` › "asks the
same-run reattachment to replace a Runner that left its recovery read
unanswered" (embedded Postgres): the claim had no
`replaceUnansweredRunner` on the old code. Now passes.
- `codex-history.test.ts`: 7 new tests failed on the old code. All 16
pass.
- Real `paperclip-runnerd` and fake Codex:
`runnerd-codex-transport-remote-retention.test.ts` › "continues the same
provider thread from a fresh Runner over the journal of a retained
Runner that stopped answering, without replaying its turn". It runs the
staging sequence: turn started, close keeps the stopped Runner, a second
controller adopts it, the Runner stops answering and the controller
detaches, the Runner and provider are killed (the launch fence), and a
fresh Runner over the same journal returns the same thread. The fake
Codex call log shows one `thread/start` and one `turn/start`.
- Focused suites: `packages/paperclip-runner` `src/drivers/codex/` plus
the remote-retention, adoption and cold-restore transport suites (442
tests). Two suites timed out under parallel load and pass alone. Server:
`native-session-executor`, `native-restart-recovery`,
`native-runner-restart-recovery.integration`,
`runner-lost-mid-turn-continuation` (1714 passed, 21 skipped).
- `pnpm typecheck` in `server` (all layers) and `tsc` for
`paperclip-runner` source and tests pass.
- Not run locally: the container-gated remote Runner suites
(`PAPERCLIP_TEST_RUNNER_CONTAINER`) and the full suite. CI covers the
full suite.

## Risks

- Medium. The replacement relaunches a Runner over a journal that is not
settled, but only while the old Runner is alive. Safety comes from three
facts: the launch fence stops the old Runner and its lineage with proof
or fails the launch; the Runner restores its own durable journal (as
local `resume_dead_runner` does); and the real-runnerd test shows that
the turn is not sent again. An exited Runner is never relaunched beside
its possible survivors.
- If the previous close left a `runner.suspend` pending, a fresh Runner
can execute it and exit. The attempt then fails as it does today. The
usual detach path after an unanswered read sends no suspend.
- The replacement uses the existing attempt budget. It does not add
attempts.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code
with tool use (shell, file edits, browser reads of staging run events
and logs), extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 22:06:24 -06:00
Nicky LeachandClaude Fable 5.2 9b613562b7 feat(docker): split the Runner image into stable layers and add a slim variant (#21229)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - Part 2 of this stack builds the Runner alone as the image
`runner-artifacts`, in one layer of about 850 MB
> - Most of those bytes are provider binaries and Node. They change only
with a provider version or the Node pin, but one layer means that every
Runner source change makes all 850 MB new
> - About half of the image is the pack's transport archives. A consumer
that copies the pack as files does not need them
> - This pull request splits the image into layers by how often their
content changes, and adds a slim variant without the transport archives
> - The benefit is that a new image shares about 93% of its bytes with
the one before it, and a consumer that needs no transport archives pulls
half as much

## Linked Issues or Issue Description

Refs #21204 (part 2 of this stack) and #21167 (provider pack layers). No
issue exists for this change.

**Problem or motivation**

The image from part 2 is one layer. A registry and a builder store and
move layers, not files. A change to one Runner source file therefore
makes a new 850 MB layer, although 790 MB of its content is the same as
before. The image also holds the provider pack two times: as files, and
as transport archives of those files. Some consumers need only the
files.

**Proposed solution**

Write the image as four layers: the pack's large dependency files,
everything else, the transport archives of those large files, and the
other transport content. Make the two large layers byte-identical from
build to build. Add the target `runner-artifacts-slim`, which is the
first two layers. The full image is the slim image plus the two
transport layers, as the same layer blobs. Check all of this on the
built images.

**Alternatives considered**

- Two unrelated images. Then the registry stores the shared content two
times unless the layers happen to match, and nothing ties the two
together.
- Split by path only, with all files of 1 MiB or more in the first
layer. Six of those files are the Runner's own bundles under `dist`.
They change with the Runner's source, so the first layer would change
with it too.
- Leave the transport archives as they were written. Their bytes then
depended on timing (see "What Changed"), and the archive layer never
repeated.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 3 of 4. Each part is on top of the one before it:

1. #21228 (merged): an optional field in the provider pack manifest
2. #21204: build and check the Runner image, with no publication
3. #21229: layers by change rate, and the slim variant (this pull
request)
4. #21230: publish both variants from `master`

## What Changed

- `Dockerfile`: two final stages. `runner-artifacts-slim` is two `COPY
--link` layers. `runner-artifacts` starts from it and adds two more. The
build stage sorts its output into the four layers. Every path is where
it was in part 2: consumers copy by path and see no difference.

  | Layer | Content | Changes with |
  | --- | --- | --- |
| 1 | The pack's dependency files of 1 MiB or more: provider binaries
and Node | A provider version or the Node pin |
| 2 | Everything else, with `runner-image.json` | Every Runner source
change |
| 3 (full only) | The transport archive of each file in layer 1 | The
same as layer 1 |
| 4 (full only) | The other transport archives, the overlay archive and
the receipt | The same as layer 2 |

- Stable bytes. The files of layers 1 and 3 get timestamp 0, and the
large dependency files get it before their transport archives are
written. A file under the pack's `dist` is the Runner's own output, so
it and its archive are in layers 2 and 4.
- `scripts/create-provider-pack-archive.mjs`, two separate commits.
**This script also writes the application image's archives.**
- The same file now gives the same archive every time. The script piped
tar into Node's zstd compressor. When the pipe ended while the
compressor still had its last chunk queued, Node finished the zstd
stream two times, and the archive got an empty frame of nine bytes at
its end. That depended on timing. In the pinned Linux base, 37 of 60
runs of the old code had the frame. The compressor is now ended only
after it has taken every chunk.
- Archive N holds file N of the receipt, and the receipt listed the
files in path order. `dist` sorts before `node_modules`, so one more
large file in the Runner's own output gave every dependency file another
archive name, and layer 3 became a new blob with the same contents. The
receipt now lists the files outside `dist` first. The readers
(`provider-pack-layers.ts`, `retained-provider-pack-source.ts`) pair
file N with `N.tar.zst` and do not use the path order, so they need no
change. For the application image, the archive numbers change one time,
and the bytes of each archive do not change.
- `runner-image.json` is the same file in both variants. Its new field
`fullImageOnly` names the two paths that only the full image has.
`paths` did not change.
- `scripts/runner-image-layout.mjs check` takes `--variant slim|full`.
It is required and has no default. A slim tree that holds a transport
path fails. A full tree without them fails.
- `scripts/runner-image-oci.mjs` (new, 585 lines). It is not an input of
the image, because the image build does not run it.
- `unpack <oci archive> --out <tree>` writes the tree of an image from
the layers of its archive, in order. It checks every blob against its
digest. It refuses a whiteout, an absolute path, a path with `..`, a
path that is written two times, an entry behind a link, and every entry
that is not a file, a directory or a link.
- `check-layers --slim <oci archive> --full <oci archive>` checks the
table above on the built images. It reads the receipt to decide which
archive belongs in which layer. It fails when the full image does not
start with the slim image's layer blobs in the same order, when a path
that is not a directory is in two layers, and when a shared layer holds
transport content under any name or directory.
- The script reads the archive and its layers itself. It does not use
the system's `tar`, so every platform reads them the same way. It finds
a member only under the name that buildx gives it.
- `.github/workflows/runner-image.yml`: each leg builds each variant one
time, as an OCI archive and nothing else. It unpacks both archives, runs
`check` on each tree with its variant, runs `check-layers`, and runs the
smoke matrix on the slim tree. The tree that is checked is therefore the
bytes of the image, not a second export of the same build. Every step
that builds has a time limit. The workflow still has no registry access.
- Why one export: in the first runs of this pull request, the slim build
stalled on all four legs. Each build command had two exports, a tree and
an archive. The slim command, with every stage from the cache, wrote its
archive and then made no progress on the tree for 50 minutes. A test now
holds the rule of one export and a time limit for each build command.
- The smoke matrix runs on the slim tree. The daemon is in a layer that
both variants share, so one run covers both.
- `server/src/__tests__/runner-image-identity.test.ts`: the test that
holds the rules for the Runner stages now finds a stage wherever
BuildKit starts one. It fails for `FROM x as name`, a flag on `FROM`, a
lowercase or indented `COPY`, an `ADD` or `ONBUILD`, a `COPY` with
another flag, and a `COPY` that continues on the next line. Nine samples
show each case.
- Two comments are corrected. The comment on
`PAPERCLIP_RUNNER_INPUTS_DIGEST` in `build-provider-pack.mjs` said that
no check reads the field. The `Dockerfile` comment on the publication
argument named a workflow "that builds the image for publication".
- `doc/runner-image-contract.md` (159 lines): the two variants, the
section "Layers", how the checks get their tree, and the new checks.
`doc/DOCKER.md` names the slim target.

Script sizes: `runner-image-layout.mjs` 969 lines,
`runner-image-oci.mjs` 585, `runner-image-identity.mjs` 166,
`create-provider-pack-archive.mjs` 114.

## Verification

Image identity at this commit:
`a544fc9dde780963419ede0f34252f3a57de43a760d41d3eb8fcd4f577504f6a`.

Linux (the pinned `node:24.22.3-bookworm` base with GNU tar 1.34 and
zstd, the worktree mounted read-only):

- `runner-image-identity.test.ts` (41), `runner-image-layout.test.ts`
(100), `runner-image-oci.test.ts` (50), `runner-image-workflow.test.ts`
(8), `provider-pack-archive.test.ts` (47, none skipped),
`retained-provider-pack-source.test.ts` (19),
`remote-provider-pack-verification.test.ts`,
`docker-image-layer-reuse.test.ts`, `docker-build-stamp.test.ts`: 293
tests pass in 9 files.
- The same files pass on macOS. The tests of `check-layers` failed on
Linux in the first runs of this pull request: their fixtures named the
archive members `./index.json`, which bsdtar finds and GNU tar does not.
The fixtures now name the members as buildx does.

`linux/arm64`, on a local Docker host, with the new export path:

- Each variant builds as an OCI archive only: the full target in 156
seconds without a cache, the slim target in 5 seconds from the cache.
- `unpack` writes the full image (10,519 entries that are not
directories) in 4 seconds, and the slim image in 4 seconds.
- `check <tree> --variant <variant> --expect-inputs-digest <digest>
--expect-source-revision <sha> --expect-architecture arm64` passes on
each unpacked tree. `runner-image.json` is the same file in both.
- `check-layers --slim <slim> --full <full>` passes.
- The smoke matrix passes on all five base images, from the unpacked
slim tree.

| Layer | Digest | Bytes | In |
| --- | --- | --- | --- |
| 1 | `sha256:3f21b15b3a99…70ac1c` | 379,609,589 | slim and full |
| 2 | `sha256:d5567acc977f…b4f12a` | 43,906,956 | slim and full |
| 3 | `sha256:6a340088ca2c…6ba5b5` | 410,938,259 | full |
| 4 | `sha256:789e97f5db88…3eab44` | 15,656,249 | full |

- The slim image is 423,516,545 bytes in two layers. The full image is
850,111,053 bytes in four.

Stability, with three builds that share no cache for the build step:

| Build | Source revision | Runner build output | Layer 1 | Layer 2 |
Layer 3 | Layer 4 |
| --- | --- | --- | --- | --- | --- | --- |
| A | `aaaa…` | this commit: 6 files of 1 MiB or more under `dist` |
`3f21b15b…` | `d5567acc…` | `6a340088…` | `789e97f5…` |
| B | `bbbb…` | this commit | `3f21b15b…` | `a2f8a4fe…` | `6a340088…` |
`4e6c492d…` |
| D | `dddd…` | one new source file that adds two large files under
`dist` | `3f21b15b…` | `33b0bdaf…` | `6a340088…` | `0ea322bf…` |

- Layers 1 and 3 are byte-identical in all three: 790,547,848 of
850,111,053 bytes, 93%. Build D has two more large files under `dist`,
eight in place of six, and another image identity. Its dependency files
keep the archive numbers 0 to 20. With the earlier numbering, those
numbers moved by two, and layer 3 was a new blob with the same contents.
- Layer 1 has the same digest as before this round. Layer 3 has a new
digest one time, because the archives have new numbers.

Other checks:

- `tests/runner-e2e` workflow tests: 39 pass. `node --test
.github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image
contract scripts: 33 pass. `check-docker-deps-stage.mjs`,
`check-node-version-policy.mjs` and `docker buildx build --check` report
nothing.

GitHub, at this head:

- The `Runner image` workflow passes on both architectures with the new
export path: `Build and check (amd64)` in 14m41s and `Build and check
(arm64)` in 8m41s. Each leg builds both variants, checks both layouts
and the layer rule, and runs the smoke matrix. The first runs of this
pull request stalled in the slim export before any check.

Never run:

- `native-session-executor.test.ts`. It does not load on this host, with
or without this change, because `@paperclipai/plugin-sdk` is not built
here. It imports the reader of the retained pack and no archive number.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- The two changes to `create-provider-pack-archive.mjs` change the
application image's transport archives: about half of them lose nine
bytes, and all of them get new numbers one time. An archive's content
does not change. A reader pairs file N of a receipt with archive N of
the same build.
- The large dependency files in the pack, and in their transport
archives, now have the modification time 0 (1970). Before, they had the
build time. Nothing in this repository reads those times.
- `unpack` is a reader of tar and gzip that this repository owns. It
reads the layers that BuildKit, GNU tar and bsdtar write (tested with
each), and it fails for a header it does not know.
- The check on the built images, not the recipe, is what proves the
layer rule; it runs in each leg of the workflow.
- `check` now needs `--variant`. A caller that does not pass it fails
with a clear message. No caller exists outside this repository yet.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 19:52:02 -08:00
DottaandClaude Opus 5.5 a298ca7fbc fix(runner): prove ownership right before starting a turn in a recovered session (#21260)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Runner runs native agent sessions (Codex, Claude, and
others) and keeps them alive across server swaps. A new server
generation recovers the session and continues it.
> - Only the generation that owns the run may submit a turn into the
provider session. The Runner proves ownership with a fenced checkpoint
(controller boot, generation and lease, judged by the database clock).
> - On a recovered session, the startup turn and the restart
continuation took that checkpoint right after recovery, then prepared
the turn's input (attachments, queued input, steered messages), then
submitted.
> - A generation that lost the run during input preparation still
submitted a turn into the shared provider session.
> - This pull request repeats the fenced checkpoint right before
`turn.start` on those two recovered-session paths.
> - The benefit is that a server generation that no longer owns a run
cannot add a turn to its provider session. New sessions are unchanged,
so nothing is added before their first turn.

## Linked Issues or Issue Description

**What happened?**
On staging, a Codex run crossed three server swaps. After its turn
finished without a result, a turn that no generation could account for
appeared in the same provider thread. Every path that submits a turn
into an existing session is preceded by the fenced checkpoint, but the
two startup paths took it before input preparation, not right before the
submit. A generation that lost the run in that window still submitted.

**Expected behavior**
A generation submits a turn into a recovered session only if it still
owns the run at the moment of the submit.

**Steps to reproduce**
1. Recover a native session on a new server generation.
2. While it prepares the startup turn's input, fence the generation
(another generation takes the lease).
3. Before this change, the turn is still submitted. After it, the
checkpoint fails and no turn starts.

## What Changed

- `packages/paperclip-runner/src/native-session-runtime.ts`:
- The restart continuation calls the fenced `checkpoint` right before
`session.startTurn`, after attachments and steered messages are
prepared.
- The startup turn on a recovered session (`recovered === true`) calls
`checkpoint` right before `session.startTurn`. A new session's normal
path is unchanged.
- `packages/paperclip-runner/src/native-session-runtime.test.ts`: new
tests "proves it still owns a recovered session right before starting a
turn in it" (startup turn) and "proves it still owns a recovered session
right before a restart continuation" at startup and when the lost turn's
terminal arrives (`resumeInterruptedTurn`).

Turn starts into an existing session, and how each proves ownership
right before the submit:
- Startup turn on a recovered session: new checkpoint, covered by the
first test.
- Restart continuation (`startRestartContinuation`), at startup and via
`resumeInterruptedTurn`: new checkpoint, covered by the second test
(both cases).
- Continuation disposition (`requestContinuationDisposition`): already
persists the fenced checkpoint immediately before `startTurn`, with only
a snapshot read between them and no input preparation. No change.
- Compaction turn: started right after the fenced recovery checkpoint
with an empty message; there is no input preparation between them (the
only possible await is the disposition-replay read, which applies only
when a disposition turn was submitted, and then no compaction turn
starts). No change.
- Runner-only change. No change to `protocol/`, `generated/`, Rust
shapes or the ACPX patch, so the provider-pack `contractDigest` and
interface revisions are unchanged and older image packs stay compatible.
The Runner build and image content for this source change are produced
by CI (`runner-artifacts`); nothing is deployed by this PR.

## Verification

- Red-to-green: "proves it still owns a recovered session right before
starting a turn in it" fails without the startup-turn checkpoint (the
turn is submitted after the generation is fenced) and passes with it.
Both restart-continuation cases fail with the checkpoint removed from
`startRestartContinuation` and pass with it.
- Local: `cd packages/paperclip-runner && pnpm exec vitest run
src/native-session-runtime.test.ts`: 290 passed.
- Local: `node scripts/typecheck-test-programs.mjs
packages/paperclip-runner/tsconfig.test.json`: passed (the first CI run
failed on the test's attachment prompt returning `undefined`; it now
returns a string).
- **Not run locally:** one test case that needs the Rust
`paperclip-runnerd` binary, and
`src/live/runnerd-restart-continuation.test.ts` (10 cases). The Rust
binary was not built on this machine. CI must cover them.

## Risks

- One extra checkpoint round trip before `turn.start` on a recovered
session and on a restart continuation. These are recovery paths; a new
session's first turn gets no added wait, so boot and time-to-first-token
for new runs are unchanged.
- If the checkpoint fails (the generation lost the run), the turn is not
submitted. That is the intended behaviour; the owning generation
continues the run.
- No migration. No schema change. No pack contract change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:08:19 -06:00
DottaandClaude Opus 5.5 52817dff55 fix(runner): name missing tool fields and what the call included (#21252)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents act through Paperclip tools, and the runner validates each
tool call against its input schema
> - When a call is invalid, the runner returns an "Invalid tool input"
error to the model
> - For a call that left out required fields, that error was a list of
JSON Schema messages such as `/ must have required property 'kickoff'`
> - A model sent `paperclip_create_case` with only
`workflow_document_text` for about 5 minutes, believed it had sent every
field, and never replied to the person
> - This pull request makes the error name the missing fields, the
fields the call did include, and a short example of the missing fields
> - The benefit is that a model can see its own mistake and correct the
call on the next attempt

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `cases.create-case` (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`), the agent called
`paperclip_create_case` about 20 times with only
`workflow_document_text`. Each call returned `Invalid tool input: / must
have required property 'idempotencyKey'; / must have required property
'caseType'; ...`. The model's reasoning said that the other fields were
being stripped. It never created the case and never replied.

**Steps to reproduce**
1. Validate `{ "workflow_document_text": "..." }` against the
`create_case` input schema.
2. Read `toolInputError(validator)`.

**Expected behavior**
The error says which required fields are missing, says that the call
included only `workflow_document_text`, and shows a short example of the
missing fields.

## What Changed

- `packages/paperclip-runner/src/drivers/tool-input-error.ts`: for
missing top-level required fields, the error now says `missing required
fields: idempotencyKey, caseType, title, kickoff.`, `This call included
only: workflow_document_text.`, and a placeholder example built from the
schema, such as
`{"idempotencyKey":"<idempotencyKey>",...,"kickoff":{"toAgentId":"<toAgentId>","phase":"<phase>","instructions":"<instructions>"}}`.
Other validation errors follow after `Also:`. Field-description guidance
is unchanged.
- `packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts` and
`runner-tool-bridge.ts`: pass the call arguments to the error builder.
Only argument names are read; values are never echoed.
- `packages/paperclip-runner/src/drivers/tool-input-error.test.ts`: a
test on the real `create_case` schema and a test that other errors still
follow the missing fields.
- The `create_case` schema is not flattened. The model also left out the
flat fields (`idempotencyKey`, `caseType`, `title`), so nesting was not
the cause, and `kickoff` is shared with `handoff_case`.

## Verification

- `cd packages/paperclip-runner && npx vitest run
src/drivers/tool-input-error.test.ts
src/drivers/opencode/mcp-bridge.test.ts
src/drivers/opencode/opencode-server-driver.test.ts` passes (115 tests).
The two new tests fail on master.

## Risks

- Low risk. Only the error text for invalid calls changes. Validation
and dispatch are unchanged. The example is bounded to 600 characters and
the field lists to 12 names.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:06:48 -06:00
Nicky LeachandClaude Opus 5.5 45523d21d0 fix(runner): re-bind any settled ACPX checkpoint across a Runner upgrade (#21220)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Runner runs Claude Code through the ACPX backend and saves
a durable checkpoint so a later run can continue the same conversation
> - Each Runner release changes the ACPX launch profile digest that the
checkpoint stores
> - Restore re-binds a saved checkpoint to the new digest only when it
is `suspended`, or when the same run resumes an interrupted turn
> - A retired or parked Claude Runner leaves an idle `session_open`
checkpoint, so a follow-up from the next run fails with
`provider_state_incompatible` before its first turn
> - `run.attach` already accepts that checkpoint. The restore gate is
stricter than the attach it serves
> - This pull request defines a settled checkpoint once and uses the
same rule for attach and for the upgrade re-bind
> - The benefit is that a completed Claude conversation continues after
a Runner upgrade, and a refused re-bind names its reason

## Linked Issues or Issue Description

Fixes #21212

Refs #20112 (the same-run interrupted-turn case, which this change keeps
as is)

## What Changed

- Added `settled_checkpoint_blockers`. A settled checkpoint is not
closed, has a provider identity, has no active turn, has no unconfirmed
provider exit, and has no undelivered event except a `session.resumed`
notice. `run.attach` now admits with this function. The warm-attach
readiness probe also uses it, and still waits for a pending
`session.resumed` notice.
- Restore re-binds any settled checkpoint to the current launch profile
digest, whichever run saved it and whatever its lifecycle
(`session_open`, `prepared` or `suspended`). Candidate admission, a
changed launch shape and pending runtime requests still block the
re-bind.
- `rebinds_interrupted_turn` keeps only its distinct case: the same
run's `turn_starting` or `turn_active` turn, under the lost-turn
recovery preconditions from #20112. The settled rule now covers its
`session_open` case.
- A digest mismatch that is not re-bound now names each reason, for
example `(not re-bound: durable_active_turn)` or `(not re-bound:
launch_shape_changed)`. The classification stays
`provider_state_incompatible`.
- Updated "Settled ACPX sessions across a Runner upgrade" in
`packages/paperclip-runner/docs/durable-recovery.md`.

## Verification

From `packages/paperclip-runner/runner`:

- `cargo test -p paperclip-runner-core --lib acpx_provider_backend`: 48
passed
- `cargo test -p paperclip-runner-core --test acpx_warm_attachment`: 6
passed
- `cargo test --locked -p paperclip-runner-core --no-fail-fast`: 1108
passed, 0 failed, 6 ignored
- `cargo fmt --all -- --check` and `cargo check --locked --workspace`:
clean

New and changed tests:

-
`settled_checkpoints_of_a_retired_runner_rebind_for_the_next_run_after_an_upgrade`
covers `session_open` with a semantic result, `prepared` with an
identity, and a pending `session.resumed` notice. Each checkpoint is
saved by run 1 and restored by run 2 under a new digest. Restore does
not write the checkpoint, `run.attach` succeeds, and the saved digest
becomes the current one. This test fails before the fix.
-
`continues_a_settled_conversation_in_the_next_run_after_a_runner_upgrade`
runs end to end with the fake sidecar. The replacement Runner attaches,
resumes the same provider session, and emits exactly one `turn.started`.
This test fails before the fix with the production error.
-
`unrestorable_saved_sessions_keep_strict_bindings_and_name_their_class`
no longer lists `session_open` as strict. Each remaining strict case
asserts the reason in its error.

I did not run clippy because CI does not run it. I did not run `pnpm
verify` because only Rust and docs changed.

## Risks

- The re-bind now admits more checkpoints. It admits only the states
that `run.attach` already admits on the same build, so no in-flight
provider work moves to a new build. An active turn, a pending runtime
request, an unconfirmed provider exit, or an undelivered non-resume
event still blocks the re-bind.
- A same-run `session_open` checkpoint that has a semantic result or an
active goal now re-binds. The same build already re-opens this state.
The result and goal checks gate lost-turn recovery only, and that
recovery path is unchanged.
- Session start still verifies every launch artifact, the command digest
and the restored provider identity after the re-bind.
- No migration and no API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, sub-agents) and extended thinking. One sub-agent
wrote the change. A second session reviewed it before this pull request.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 17:44:58 -08:00
Nicky LeachandClaude Fable 5.2 5d3fa712aa feat(docker): build and check the Runner alone as the runner-artifacts target (#21204)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - Today the application image build compiles the Runner. Other images
that need the Runner build it again, or pull the whole application image
to copy it out
> - Each of those builds can produce a slightly different Runner, and
each one costs time
> - The Runner should be built one time for each change to its inputs,
and every image should copy the same bytes
> - This pull request adds the build target `runner-artifacts`, a
contract for its layout, and a workflow that builds and checks it. It
publishes nothing
> - Nothing consumes the target in this pull request. The application
image build does not change
> - The benefit is one checked Runner build that later changes can
publish and copy by digest

## Linked Issues or Issue Description

Refs #21096 (the merge-queue image build) and #21167 (provider pack
layers). Those are earlier work on the same build. No issue exists for
this change.

**Problem or motivation**

The Runner is compiled inside the application image build. The daemon
comes from the `runner-build` stage. The provider pack is built on top
of the full server and UI build. Downstream sandbox and VM images need
the same Runner. They rebuild it from source, or they pull the
application image and copy paths out of it. There is no contract for
those paths and no single identity for a Runner build.

**Proposed solution**

Build the Runner alone, from its own inputs, as a plain file tree. Give
each build an identity: a digest of its inputs. Document the layout as a
contract, and check every built tree against it. Keep every check that
proves the Runner is good in this one build.

**Alternatives considered**

- Keep the current builds and only document the paths in the application
image. This does not remove the duplicate builds, and consumers still
need the full application image.
- Build the Runner in `docker.yml`. That workflow has no pull-request
trigger, so a change to the Runner build would have no check before it
merges.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 2 of 4. Each part is on top of the one before it:

1. `runner/pack-inputs-digest`: an optional field in the provider pack
manifest
2. `runner/runner-image-build` (#21204): build and check the Runner
image, with no publication (this pull request)
3. `runner/runner-image-layers`: layers by change rate, and the slim
variant
4. `runner/runner-image-publish`: publish both variants from `master`

This pull request had one larger form before. Publication, the image
layers and the second variant are now parts 3 and 4.

## What Changed

- `Dockerfile`: eight new stages that end in the target
`runner-artifacts` (`FROM scratch`). They install with a filtered,
frozen lockfile from a context that holds only the Runner's inputs. The
base is a Node image on Debian bookworm, pinned by digest. No existing
stage changed. `cloud` is still the default target.
- The tree holds: the stripped daemon, the Runner package's complete
`dist` with the build stamp that its own build records, the provider
pack with its transport archives, the Grok provisioning inputs, the
transport contract source and a bundled transport module, the lockfile,
and `runner-image.json`.
- `scripts/runner-image-identity.mjs`: computes the image identity. It
hashes every file under the paths that the build copies from the build
context (path, executable bit, content; symbolic links by target), the
whole `Dockerfile`, and the two platforms. The build runs it on the
files that the build context supplies after `.dockerignore`.
- `scripts/runner-image-layout.mjs`: the layout contract. `write`
assembles the tree in the build. `check` verifies a built tree: every
documented path, the build stamp, every pack artifact and tree digest in
`provider-pack.json`, and every transport archive. It also extracts the
transport archives the way a sandbox does and compares the result with
the pack. No checked path may resolve outside the tree.
- Publication form. The build argument
`PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1` is off by default. Only with it
does the pack manifest carry `payload.runnerInputsDigest` (part 1), and
`runner-image.json` records `publication: true`. Another image build
that runs the stage inline builds the same Runner, but its pack names no
image. `check` requires the field in a publication build and refuses it
in any other. `--expect-inputs-digest` passes only for a publication
build.
- `docker/runner-image/Dockerfile.smoke`: runs `paperclip-runnerd
--build-metadata` on each base image that a consumer uses. Each base is
pinned by digest.
- `.github/workflows/runner-image.yml`: two jobs, on pull requests that
touch an input and on manual dispatch. `identity` resolves the lockfile
and computes the digest in the build context. `build` builds each
architecture on a native runner, checks the layout, and runs the smoke
matrix. The workflow has `contents: read` only. It holds no credential,
reads no registry, and publishes nothing. It builds the publication
form, so the check can bind the tree to the digest that the identity job
resolved. The tree stays on the runner.
- `packages/paperclip-runner/scripts/provider-pack-runtime-tree.mjs`:
the tree digest function moves here from `build-provider-pack.mjs`
without a change in behavior, so the layout check can use it. A test
pins its value.
- `.github/workflows/pr-merge-review.yml`: lists the new workflow, as
`doc/PR-MERGE-REVIEW-WEBHOOK.md` requires.
- `doc/runner-image-contract.md` (127 lines): the contract for
consumers. `doc/DOCKER.md` links to it.

Changes from the earlier form of this pull request:

- The identity hashes the whole `Dockerfile`. The script no longer
parses stages, and it no longer refuses recipes.
`scripts/runner-image-identity.mjs` went from 397 lines to 166. A short
test keeps the three rules that make the hash enough: the Runner stages
start only from pinned images, only `runner-artifacts-context` reads the
build context, and it copies only the paths that the identity hashes.
- The contract document went from 384 lines to 127.
- The workflow tests keep only the security properties: job permissions,
no credential and no registry access, triggers, actions pinned by
commit, and no expression inside a shell script. They no longer pin step
names, order, or wording.

## Verification

Verified on `linux/arm64` only, on a local Docker host:

- `docker buildx build --target runner-artifacts --platform linux/arm64
--build-arg PAPERCLIP_RUNNER_SOURCE_REVISION=<sha> --build-arg
PAPERCLIP_RUNNER_IMAGE_PUBLICATION=1 --output type=local,dest=<tree> .`
builds in 166 seconds without a cache. The image is one layer of
850,121,093 bytes. The tree is 1,755,345,443 bytes in 9,969 files.
- `node scripts/runner-image-layout.mjs check <tree>
--expect-inputs-digest <digest> --expect-source-revision <sha>
--expect-architecture arm64` passes on the exported tree.
- `docker buildx build --file docker/runner-image/Dockerfile.smoke
--build-context runner-artifacts=<tree> docker/runner-image` passes on
all five base images.
- The digest from `--target runner-artifacts-identity` equals the digest
in the built `runner-image.json`:
`d05f51f075cb8fec687a1d27f574ba7cb41cd8ebbe9927c5cbb2cfc13b147880`.
- The same target with `PAPERCLIP_BUILD_COMMIT=<sha>` and no publication
argument builds. Its pack has `runnerSourceRevision` and no
`runnerInputsDigest`. `check` passes on it, and `check
--expect-inputs-digest` fails with "this is not a publication build".
- An edit to `ui/package.json` and to `server/src` leaves the digest
unchanged. A comment added to an application stage of the `Dockerfile`
changes it.
- The two points above ran one commit state earlier. The only difference
to this commit is one type annotation in a comment of
`scripts/runner-image-layout.mjs`.
- A named build context replaces the `runner-artifacts` stage, and no
Runner stage runs. A build with the application build's view of the
context accepts the tree's `dist` and build stamp as current
(`assessRunnerBuild`).
- The daemon has the same bytes in the five cold builds of this stack
that were compared.
- Tests: `cd server && node node_modules/vitest/vitest.mjs run` on
`runner-image-identity.test.ts`, `runner-image-layout.test.ts` and
`runner-image-workflow.test.ts` (132 tests), with
`remote-provider-pack-verification.test.ts` and the existing Dockerfile
tests: 186 tests pass. `tests/runner-e2e` workflow tests: 39 pass. `node
--test .github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the
image contract scripts and `provider-pack-runtime-tree.test.mjs`: 33
pass. `check-docker-deps-stage.mjs`, `check-node-version-policy.mjs` and
`docker buildx build --check` report nothing.

Never run:

- The `linux/amd64` build. It is the leg that downloads the Cursor and
Copilot assets for the pack.
- The workflow. No job has run on GitHub.
- The transport archive round trip with GNU tar outside the build. The
build runs it with GNU tar. The local host ran it with bsdtar.
- `actionlint` and `hadolint`. They are not installed on the local host.
- The full test suite and the full typecheck. CI must supply them.

## Risks

- Low risk to the application image. The change adds stages and files.
No existing target reaches the new stages.
- The first run of the workflow is the first test of the amd64 build.
This pull request triggers it.
- The `Dockerfile` is one file for all images. Any edit to it, also to
an application stage, changes the Runner image identity. That is the
price of not parsing it.
- The Dockerfile frontend (`# syntax=docker/dockerfile:1.20`) is a tag,
not a digest. A new frontend release under that tag does not change the
identity.
- The image does not run the drift checks of the Runner's `build`
script. They need files outside the Runner's inputs, and repository CI
runs them. A test fails when the `build` script gains a step.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 17:30:51 -08:00
DottaandClaude Opus 5.5 2d1c2db64a feat(mcp): host the MCP Skills extension for connected apps (#21219)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use connected apps through MCP. Paperclip already hosts MCP
Apps views and speaks stateless MCP 2026-07-28.
> - The MCP Skills extension (SEP-2640, Final) lets an app publish
skills (instructions and files) next to its tools. Hugging Face already
serves 25.
> - Paperclip did not read them, so agents missed each app's own
workflow guidance.
> - Skills are a prompt-injection surface. The spec makes manifest
binding, verification, origin tagging and approval revocation host
requirements.
> - This pull request makes Paperclip a Skills host: discover, review,
verify, and deliver app skills to the agents that use the app.
> - The benefit is that connected apps can teach agents their workflows,
under board control.

## Linked Issues or Issue Description

**Problem**
Connected MCP servers can publish skills with the
`io.modelcontextprotocol/skills` extension. Paperclip ignored them. A
board could not see, review or approve them, and agents never received
them.

**Proposed solution**
Implement the host side of SEP-2640. List skills during catalog refresh,
record them per connection and company, bind board approval to the full
file manifest, and load approved skills into agent runs with verified,
lazily fetched content.

**Why it matters**
Apps ship workflow guidance with their tools. Without the extension,
each team must copy that guidance into company skills by hand, and it
goes stale.

Refs #21119, Refs #21140 (the MCP Apps and stateless MCP client work
this builds on).

## What Changed

- **Client capability.** Every stateless request declares
`io.modelcontextprotocol/skills`. Legacy handshake sessions do not. The
capabilities that `server/discover` advertises are remembered beside the
protocol era.
- **Discovery.** Catalog refresh calls `skills/list` (paged) only when
the server declared the extension. A listing inside its `ttlMs` is
reused. A skills failure never fails the tool refresh. Nothing runs
while a run starts.
- **Schema.** New `tool_mcp_skills` (company, connection, skill URI;
frontmatter, sorted manifest, manifest digest, review state) and
`tool_mcp_skill_files` (verified bytes keyed by connection, URI and
digest). Both cascade with the connection.
- **Governance.** New skills wait for review. Approval binds to the
order-free digest of every `{uri, digest, size}`; a request for any
other digest gets 409. A changed manifest or frontmatter revokes
approval and logs `tool_mcp_skill.approval_revoked`. Dynamic, oversized
and invalid skills are declined with a reason. Approve and decline use
the company skill policy (`skills.install`, `external_package`,
`mcp-connection:<id>`).
- **Run time.** Approved skills of connections the agent can use load in
parallel with other skill preparation, inside a 3 s budget. `SKILL.md`
is read with the agent's grant and checked for size, SHA-256 and
frontmatter. A failure calls `skills/get` and takes the skill out of
use. Each skill becomes a per-run bundle on the connector skill path,
named in its app's namespace, tagged with its origin. Paperclip writes
the frontmatter: `allowed-tools` only if the board approved it.
- **Supporting files.** `GET
/api/agents/me/mcp-skills/:skillId/files?path=` serves manifest-listed
files to an agent in an active run, through the skill's own connection.
The sandbox bridge allows the route.
- **Board API and UI.** `GET/POST /api/companies/:companyId/mcp-skills…`
(list, read instructions, approve, decline). The Skills page shows
"Skills from connected apps" with app, state, size, requested tools and
plain-language notes. It renders nothing when no app publishes skills.
- **Docs.** `doc/connections/MCP-SKILLS.md`, linked from
`doc/connections/README.md`.
- **Shared frontmatter parser.** It now decodes YAML single-quoted
scalars. Before, 5 of Hugging Face's 25 skills kept their quotes and
would fail the frontmatter check; now all 25 match. The generated runner
copy is regenerated.

File size notes: `server/src/services/tool-gateway.ts` grows by about
200 lines. The run-time reads must use its grant resolution and
`callAssignedConnectionProtocol`, like MCP Apps view reads. Splitting
the gateway is a separate refactor. `CompanySkills.tsx` and
`CompanySkills.production.tsx` gain 2–4 lines each; the section lives in
its own component.

## Verification

- `server/src/__tests__/mcp-skills.test.ts` (15 tests) uses a fake
stateless skills server and embedded Postgres. It covers: discovery only
after the server declares the extension; the capability in `_meta`;
`ttlMs` reuse; lazy fetch (no reads at listing or approval; `SKILL.md`
at load; supporting files on read); a stale-digest approval refused; a
reorder that keeps approval; a changed or added file that revokes it;
dynamic skills declined; a same-size tampered `SKILL.md` rejected,
refreshed with `skills/get`, and taken out of use; mismatched
frontmatter rejected; reads outside the manifest refused; the same URI
and name from two servers kept apart; company scoping; a slow app
skipped within the budget and cached later; HTTP authorization (board
only, agent needs an active run, run header must match) and activity
logging.
- Red/green: with the digest check disabled, the tamper test fails; with
the manifest comparison disabled, the revocation test fails. Both pass
with the code.
- `server/src/__tests__/mcp-protocol-versions.test.ts`,
`mcp-apps.test.ts`, `connection-instructions.test.ts`,
`tool-gateway-discovery-http.test.ts`: 47 pass (two capability
assertions updated for the new extension).
- `packages/shared/src/frontmatter.test.ts` adds a single-quoted scalar
case; the runner contract test passes (23 tests).
- `ui/src/components/skills/ConnectedAppSkills.test.tsx` (3) and
`ui/src/pages/CompanySkills.test.tsx` (44) pass.
`packages/adapter-utils/src/sandbox-callback-bridge.test.ts` (89)
passes.
- `pnpm --filter @paperclipai/server typecheck` passes. UI `tsc` passes
except the known missing generated Storybook catalog files.
- `pnpm verify` returned exit 2 (doc change is a global CI input);
required CI supplies broad coverage.
- Live check against `https://huggingface.co/mcp` without credentials,
through Paperclip's own services and the real egress guard: 25 skills
discovered as pending; three approved; cold load 135 ms, cached load 13
ms; a supporting file read through the manifest; the bundle tagged with
its origin.
- Storybook: `Agents and reusable capabilities/Skills and Skill
Studio/Connected app skills` (needs review, all states, tool request,
name collision, reading instructions).

## Risks

- **Migration:** two new tables only. No changes to existing tables.
Safe with the previous release running.
- **Run start:** companies without an approved app skill pay one indexed
read. With approved skills the wait is capped at 3 s; late or failing
skills are left out with a `skills.mcp_unavailable` run warning and
never fail the run.
- **Agent harness enforcement:** Paperclip cannot gate each
code-execution tool call an agent makes while following an app's skill.
Approval is explicit and per skill, the approval note says agents may
run commands the skill describes, and `allowed-tools` is never granted
without a separate tick.
- **Shared-home adapters** receive app skills like company skills. A
revoked skill leaves on the next skill sync.
- Not in this PR: per-connection opt-in for dynamic skills; serving
Paperclip's own skills from its public MCP endpoint;
`resources/directory/read`; app skills on an agent's skills tab.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test runs, live HTTP checks).
Extended reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:18:06 -06:00
Nicky LeachandClaude Fable 5.2 2dcfdabc79 feat(runner): let a provider pack record the Runner image it was built in (#21228)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work: a daemon
(`paperclip-runnerd`) and a provider pack of agent tools
> - A provider pack has a manifest, `provider-pack.json`. It records the
commit the pack was built from as `payload.runnerSourceRevision`
> - The next pull requests build the Runner alone as an image that has
an identity of its own: a digest of the Runner's inputs
> - A pack from that image must be able to say which image it came from.
The commit does not say that, because many commits have the same inputs
> - This pull request adds one optional field to the pack manifest for
that digest
> - The benefit is that a reader of a pack can find the exact image that
built it. Packs that are built today do not change

## Linked Issues or Issue Description

Refs #21204 (the Runner image, part 2 of this stack). No issue exists
for this change.

**Problem or motivation**

The provider pack manifest records a source revision. It has no place
for the identity of a Runner image. The Runner image in #21204 is named
by a digest of its inputs, and one image serves every commit that has
those inputs. Without a field in the manifest, a pack cannot name the
image that built it.

**Proposed solution**

Add the optional field `payload.runnerInputsDigest`. The pack builder
writes it only when the environment variable
`PAPERCLIP_RUNNER_INPUTS_DIGEST` is set. The value must be 64 lowercase
hexadecimal characters. The pack builder fails for another value.
Nothing sets the variable in this pull request.

**Alternatives considered**

- Put the digest in a file beside the pack. Then the pack digest does
not cover it, and a copy of the pack loses it.
- Reuse `runnerSourceRevision`. That field is a commit, and code
compares it with other commits.

**Roadmap alignment**

This is build infrastructure. It adds no product feature and does not
overlap an item in `ROADMAP.md`.

This is part 1 of 4. Each part is on top of the one before it:

1. `runner/pack-inputs-digest`: an optional field in the provider pack
manifest (this pull request)
2. `runner/runner-image-build` (#21204): build and check the Runner
image, with no publication
3. `runner/runner-image-layers`: layers by change rate, and the slim
variant
4. `runner/runner-image-publish`: publish both variants from `master`

## What Changed

- `packages/paperclip-runner/scripts/build-provider-pack.mjs`: reads
`PAPERCLIP_RUNNER_INPUTS_DIGEST`. When it is set, the script checks its
form and writes `payload.runnerInputsDigest` after
`runnerSourceRevision`. When it is not set, the manifest is the same as
before, byte for byte.
-
`server/src/services/native-runtime/remote-provider-pack-verification.ts`:
the manifest type has the optional field `runnerInputsDigest`. No check
reads it.
-
`server/src/services/native-runtime/remote-provider-pack-verification.test.ts`:
one test. A sealed manifest with the field passes verification. A
manifest whose field was changed after it was sealed fails, because the
pack digest covers the field.

The change is 20 added lines in 3 files.

## Verification

- `cd server && node node_modules/vitest/vitest.mjs run
src/services/native-runtime/remote-provider-pack-verification.test.ts`:
19 tests pass.
- Part 2 builds a pack with the variable set and a pack without it, on
`linux/arm64`. The first has the field and the second does not.

Never run:

- The full test suite and the full typecheck. CI must supply them.

## Risks

- Low risk. The field is optional and additive, and nothing sets the
variable yet.
- The pack digest covers the payload. A pack that has the field
therefore has another digest than the same pack without it. No pack has
the field until part 2 builds one.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (shell, file edits, local Docker builds).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 16:58:14 -08:00
DottaandClaude Opus 5.5 ef6d347f79 fix(runner): keep reattached runners answering after a restart (#21161)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs execute in a remote Runner that the server controls over
the durable PRP channel
> - After a server restart, the next server reattaches to the live
Runner and sends recovery barriers (`runner.drain`, `session.snapshot`)
> - These barriers timed out while the Runner stayed connected, and each
same-run retry failed the same way
> - After three attempts the run stopped with
`native_session_retry_exhausted` and needed a manual Retry
> - This pull request finds the two causes and fixes both in the
controller and the transport
> - The benefit is that restart recovery finishes on its own, without a
manual Retry

## Linked Issues or Issue Description

**What happened?**

After a server restart, a run reattached to its live remote Runner. The
Runner authenticated. Then a recovery command got no recorded answer for
30 s or more, although the Runner stayed connected. Errors on staging:

- `PRP command session.snapshot timed out (runner connected; pending
ahead 0; last runner event seq 869 55s ago)`
- `PRP command runner.drain timed out (runner connected; pending ahead 1
[semantic_tool.result] oldest 170s ...)`
- `runnerd exited while waiting for runner.drain`

Each reattach failed the same way. After three attempts the run ended
with "Workspace repair required" (`native_session_retry_exhausted`).

Root causes, proven with the real Runner and the fake Codex harness:

1. The Runner answered in about 40 ms. At reconnect it also sends the
event backlog that it collected while the server was down. The
controller handles inbound frames of a connection in order, and it
commits each event (accounting write and checkpoint flush) before the
next frame. So the answer waited behind the whole backlog. With 400
events at 100 ms each, the answer was handled after 44 s, and the 30 s
barrier had already failed. The `semantic_tool.result` case is the same:
its answer waited behind the backlog, and the next command is sent only
after that answer is handled. The Runner code was not the cause.
2. When a Runner did not answer during close, the close path queued
`runner.drain` and `runner.suspend`, kept the Runner alive for
reattachment, and saved them in the journal. The next attempt delivered
that old `runner.suspend` to the reattached Runner, and the Runner
suspended and exited.

**Expected behavior**

A reattach to a live Runner completes recovery. A Runner answer that is
queued behind earlier frames is not a timeout. A reattach never delivers
a terminal command from an earlier attempt.

**Steps to reproduce**

1. Start a Codex run in a remote Runner with a turn that streams output.
2. Restart the server. Let the Runner collect a few hundred events while
it is disconnected.
3. Let the next server reattach while event commits are slow, as on a
loaded database.
4. See `session.snapshot timed out (runner connected; pending ahead 0)`.
5. Alternatively, let a close path leave `runner.drain` and
`runner.suspend` pending, then reattach. See `runnerd exited while
waiting for runner.drain`.

**Paperclip version or commit**

`master` at 3fb35fcf13.

**Deployment mode**

Hosted (staging), remote Daytona sandbox Runner.

**Agent adapter(s) involved**

Codex (native Runner, `paperclip-runnerd`).

## What Changed

- `DurablePrpControlPlane` records each Runner `command_result` frame
when it arrives, before the ordered frame chain handles it.
`queuedCommandResultProgressAt(commandId)` returns the latest time that
the chain made progress while that answer is queued.
- `waitForRunnerCommand` keeps waiting past its deadline while the
answer to the command, or to a pending command ahead of it, is queued
and the chain advanced in the last 30 s. When the chain stops for 30 s,
the barrier fails as before. A wait that starts after its deadline still
rejects a late ACK. Frame order does not change.
- When a barrier completes after such an extension, it credits the time
past the deadline (plus the 5 s post-stop grace) to the admission clock
(`RunnerStallLedger.credit`). Later barriers that share a cold admission
deadline (Pi, OpenCode, ACPX) keep their time.
`#waitForProviderIdentityWork` checks the ready state once before it
rejects an expired deadline.
- `DurablePrpControlPlane.withdrawUndeliveredTerminalCommands()` runs
only when the transport adopts a live Runner. The controller sends only
the first pending command, so no command after it reached the Runner,
and a live, serving Runner cannot have executed a suspend. The method
removes every pending `runner.suspend` and `runner.shutdown` from that
pending suffix. It keeps every other pending command, for example a
`semantic_tool.result` that a still-running tool callback queued after
the close, and renumbers the kept commands so the Runner's command
sequence stays contiguous. Pending drains stay, because they have no
effect.
- A generated command id now skips an id that a renumbered command
already holds.
- `doc/execution-semantics.md` describes both rules.

### Changed existing test

`runnerd-codex-transport-remote-retention.test.ts` asserted that the
reattaching controller delivers the failed close's `runner.suspend`, so
the adopted Runner suspends and exits, and a later attempt relaunches it
(#20923). On staging that path failed the attempt with `runnerd exited
while waiting for runner.drain` (run 3e28c2b0) and did not recover. This
PR requires that a reattach does not inherit that suspend. The test now
asserts that the adopted Runner stays alive, answers `thread/read`,
keeps a `ready` journal, gets no suspend, and no duplicate Runner is
launched. The retention part of the test (the close keeps the frozen
Runner alive) is unchanged.

## Verification

- Red before the fix, green after (real `paperclip-runnerd` and fake
Codex):
- `... past an event backlog ahead of its recovery snapshot result`
failed with the staging error `session.snapshot timed out (runner
connected; pending ahead 0; ...)`. Debug timing showed the Runner
answered in 40 ms and the controller handled the answer 44 s later.
- `... past a runner.suspend left by an earlier attempt's close path`
failed with the staging error `runnerd exited while waiting for
runner.drain`.
- `... past a runner.suspend with a tool result queued after it` failed
the same way with tail-only withdrawal (the Greptile case), and passes
now. The tool result is delivered once and all sequence numbers stay
contiguous.
- Unit tests for `waitForRunnerCommand`: it waits while a queued answer
advances, it times out when progress stops, and a later barrier that
shares the admission deadline still completes after an extended one.
- `... past a tool result its predecessor queued after the restart`
guards delivery of a `semantic_tool.result` queued between controller
generations.
- The existing `shares Pi's absolute cold deadline across recovery
barriers and rejects a late ACK` passes unchanged.
- `cd packages/paperclip-runner && pnpm exec vitest run
src/live/runnerd-codex-transport.test.ts
src/live/runnerd-codex-transport-remote-retention.test.ts
src/live/runnerd-codex-transport-reattach-carried-commands.test.ts`
passes (320 tests).
- Related suites pass: adoption, carried suspend, stale controller
journal, stall, deadlines,
`src/backends/harness-driver-backend.test.ts`, and all
`src/control-plane` tests (330 tests).
- `tsc --noEmit -p tsconfig.test.json` passes in
`packages/paperclip-runner`.
- `pnpm verify` asked for broader coverage (exit 2). Required CI
supplies it.

## Risks

- A barrier can now wait longer than its deadline when the Runner has
answered and the controller still commits a large backlog. The wait ends
30 s after progress stops. This trades a slower recovery for no failed
run.
- The withdrawal applies only to a live adopted Runner. If such a Runner
had somehow journaled a withdrawn suspend, the next command gets a
`controller_sequence_mismatch` rejection, and the Runner executes
nothing.
- Renumbering changes the `controllerSeq` of pending commands that never
reached the Runner. Their command ids do not change.
- Not in this PR: a fallback that stops a connected Runner that never
answers and resumes it from its journal. With these fixes, the observed
staging failures no longer reach that state.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool
use (shell, file editing, browser for read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:50:43 -06:00
DottaandClaude Opus 5.5 c5e4b46d3f fix(acpx): shut down the per-port ACPX runtime when its session closes (#21151)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The experimental Runner starts ACP providers through ACPX, and each
runtime port owns one ACPX runtime
> - When the native provider dies during a turn, the port retires the
turn at once, but the turn stays queued inside the ACPX runtime
> - `port.close()` closed only the session handle, so that queued turn
wrote its final session record after close had returned
> - Any caller that removes the state directory after close races that
write; CI showed it as `ENOTEMPTY ... state/sessions` in an unrelated
test on 5 branches
> - This pull request makes close shut down the port's own ACPX runtime
after the session closes, inside the existing close bound
> - The benefit is that a closed port no longer writes into its state
directory, and the flaky test is fixed at its cause

## Linked Issues or Issue Description

**What happened?**
`packages/paperclip-runner` test `native-distribution-integrity.test.ts
> native ACPX execution closure > expires a real ACPX permission on
native child death without replaying its mutation` fails intermittently
with `Error: ENOTEMPTY: directory not empty, rmdir
'/tmp/native-acpx-test-XXXX/state/sessions'`. It failed 5 times on 5
unrelated branches in 48 hours.

**Steps to reproduce**
1. Run the test 30 times while two CPU-bound shell loops run.
2. 2 of 30 runs fail with ENOTEMPTY in `afterEach`.
3. Instrumenting the session store shows a `saveSessionOwner` write from
`finalizeRuntimeTurn` that starts after `port.close()` and `afterEach`
have begun.

**Paperclip version or commit**
`master` at the base of this branch (CI on unrelated pull requests,
2026-11-10 to 2026-11-12).

**Expected behavior**
After `port.close()` resolves, the ACPX runtime owned by that port does
no more work and writes no more state.

**Actual behavior**
The turn that native provider death retired is still queued in the ACPX
runtime manager. It finalizes after close and writes the session file
while the caller removes the directory.

## What Changed

- `acpx-runtime-adapter.ts`: after the final successful close, the port
calls `runtime.shutdown()` within `runtimeCloseTimeoutMs`. ACPX shutdown
waits for its pending turn tasks, so the abandoned turn finishes its
writes first.
- A close that succeeds late, after its bound, also starts the shutdown.
A concurrent or repeated `close()` waits for the same shutdown and gets
its failure too. A shutdown failure is reported in the existing `ACPX
runtime and provider cleanup failed` error. The shutdown attempt stays
in the retained cleanup owners.
- `acpx-runtime-adapter.test.ts`: regression tests. Close does not
resolve before runtime shutdown settles. A shutdown failure is reported
to every closer. A late close success starts the shutdown.

## Verification

- Red/green: both new tests fail on master (`expected "vi.fn()" to be
called once, but got 0 times`; `promise resolved "undefined" instead of
rejecting`) and pass with this change.
- Loop of the flaky test, 30 runs each, with 2 extra CPU-bound shell
loops on an already loaded 18-core Mac (load average about 30):
  - Before: 28 pass, 2 fail (both `ENOTEMPTY ... state/sessions`).
  - After: 30 pass, 0 fail.
- `npx vitest run src/drivers/acpx` in `packages/paperclip-runner`: 92
files, 1743 tests pass.
- `tsc --noEmit -p packages/paperclip-runner`: clean.

## Risks

- Low risk. The runtime is created per port and is not used after a
successful final close. Close can now take up to one more close bound if
the runtime's pending work is slow. A late reconciliation close never
runs after this point, because shutdown only starts when the port is
finally closed.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:29:36 -06:00