Commit Graph
283 Commits
Author SHA1 Message Date
Nicky LeachandClaude Fable 5.2 ddfbbfff20 fix(runner): recover provider-started turns after a controller restart (#21612)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core runs Claude Code through Paperclip Runner. Claude can start a
shell command in the background and end its turn. When the command
finishes, Claude Code starts a cycle of its own to continue.
> - The Runner can keep Claude while a background shell is running and
give the cycle of Claude to the controller as a turn that the provider
started. Core can wait for that turn. #21607 lets a caller ask the
transport for it.
> - A controller can restart at any time, for example at a deploy. A
wait can last 10 minutes and a provider-started turn can last longer, so
a restart during one of them is a normal event.
> - Tests with the real `paperclip-runnerd` showed that such a restart
failed the session. The checkpoint that Core had was the checkpoint of
the turn before, and the recovery of the driver refused or mislabelled
the turns that it did not know.
> - This pull request makes that restart safe. It changes no production
run, because no caller asks for provider-started turns before part 3.
> - The benefit is that part 3 can turn the behavior on without a failed
run at each deploy.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

#21456 turned the behavior on and had grown too large to review. It is
now three stacked pull requests:

1. #21607 (merged): the transport can ask the Runner for
provider-started turns. No caller asks.
2. **This pull request:** the recovery after a controller restart during
a wait or during a provider-started turn.
3. #21456: Core sets the option for native ACPX Claude runs. Only that
part changes what a production run does.

This is part 2. Its base branch is `master`. Refs #19912 (design),
#21456.

**What existing behavior does this improve?**

None in production. It makes a controller restart safe for a run that
asked for provider-started turns.

**Subsystem affected**

Paperclip Runner package: the consumer
(`src/native-session-runtime.ts`), the session backend
(`src/backends/harness-driver-backend.ts`), the driver
(`src/drivers/codex/`) and the transport (`src/live/`). Only for a run
that asked for provider-started turns.

**Current behavior**

- The consumer saves no checkpoint when a wait starts or when a
provider-started turn starts. During a wait and during a
provider-started turn, the checkpoint that Core has is the checkpoint
from the start of the turn before. It names that turn as active.
- A restart from that checkpoint fails: the driver has the old turn as
active, gets the events of a provider-started turn, and stops the
session with `turn_binding_mismatch`.
- The recovery of the driver adopts one turn that it does not know as a
disposition recovery turn, and refuses two or more. With two or more, it
closes the transport, and that stops the Runner and Claude.
- The Runner does not send again an event that the controller
acknowledged. If the start of a provider-started turn was acknowledged
before the restart, the new driver gets the other events of that turn
with no start.
- A turn that ends waiting for a provider-started turn sets the terminal
of the session, as a turn that ends the run does. After a restart, if
the provider-started turn already has its result when recovery
completes, the consumer finds a result and a terminal, settles the run,
and does not read the events of that turn. The start, the items, the
usage event and the terminal of that turn do not reach Core.
- The recovered driver reads notifications from its construction, and
`reconcile()` maps the terminal of the saved turn only after its reads.
If the saved checkpoint is still the checkpoint of the controller turn
and the start of the provider-started turn is already queued, the driver
judges that start too early and stops the session with
`turn_binding_mismatch`.
- A checkpoint that waits can also have a consumed disposition marker (a
recovery turn after an earlier restart ended waiting). The driver
restores that session as closed and reads no notifications, and the
consumer settles the run on the earlier terminal with no result.

**Proposed behavior**

- **Checkpoints.** The consumer saves a checkpoint when a wait starts
and when a provider-started turn starts, with the function that it
already uses for a governed wait. A checkpoint that fails stops the
turn, as for a governed wait.
- **A checkpoint as of the event that Core stored.** The driver gives
each event its sequence when it reads it from the provider, before the
consumer stores it. A snapshot that is taken later can be ahead of Core.
For these two checkpoints, the driver captures its snapshot when it has
handled the notification that made the event (the `turn.completed` that
waits, or the `turn.started` of the provider), with the sequence of that
event. The consumer asks for the snapshot as of the event that it has
just appended. The saved state and cursor are then those of an event
that Core has.
- **Session terminal.** A turn that ends waiting for a provider-started
turn does not set the terminal of the session (`HarnessNativeSession`).
The terminal of the session is the end of the run, and that turn does
not end the run. A checkpoint that is saved during a wait or during a
provider-started turn has no terminal, so a recovered session settles
only from the events that it reads, in order, to the terminal of the
turn that has the result.
- **Driver, a checkpoint that waits.** Recovery keeps the wait. It does
not adopt and does not refuse the turns after the last terminal that it
knows. The Runner sends again, in order, each event that was not
acknowledged, and the driver admits each provider-started turn from that
stream, as it does with no restart.
- **Driver, a checkpoint with an active turn.** `reconcile()` accepts
one turn in progress that comes after the saved active turn, when the
saved turn ended with the wait marker. It refuses each other unexpected
active turn as before, and it refuses this one too when the reconciled
turn settled a result.
- **Driver, a start that is queued before recovery reconciled the saved
turn.** A `turn/started` of the provider that arrives while the session
does not wait for one first waits until `reconcile()` has settled, and
is then judged as before. `reconcile()` judges the live state after its
reads; if the stream already settled the saved turn into a wait, it
accepts exactly one turn in progress after the last terminal turn and
does not adopt it. The driver admits that turn at its start.
- **A wait with a consumed disposition marker.** The driver keeps
reading notifications for a restored session that waits, as it does for
a wait with no restart. It still refuses a second controller-started
turn. The consumer does not take the consumed-marker shortcut for a
recovered snapshot that waits, so the wait is resumed.
- **Transport, the start comes first.** For each driver, the transport
gives the `turn/started` of a provider-started turn before each other
notification and before a tool call of that turn. If it did not give
that start to this driver, it builds it from the `turn.started` event in
its stored events. It does not do this for the turn that the recovered
driver already has as active (`resumeActiveTurnId`). It binds the events
and the tool call of a provider-started turn to that turn, not to the
turn that the Runner reports as active at the time of the recovery read.
- **Transport, which start owns an event.** The stored events can have
gaps: the window removes old events but keeps the latest of each anchor
type, and the controller checkpoint drops streamed deltas. The owner of
an event is the last stored `turn.started` before it, if it is a
provider start, and if the first turn terminal that is stored after that
start is not before the event and is the terminal of the same provider
turn (`providerTurnId`). If the first stored terminal after the start is
the terminal of a different turn, the end of the provider turn is not
known, and the transport builds no start.

**Reason and benefit**

- The checkpoint is the cause. With a true checkpoint, the two states of
a restart are states that recovery can handle: a wait, or an active
turn.
- The Runner already sends again each event that was not acknowledged.
Recovery uses that stream and does not build the turns again from a
read.
- The window removes a terminal only while a later terminal of the same
type is stored. Thus the first stored terminal after a start shows if a
terminal of that turn was removed. A rule that only ignores the gaps is
not safe: the Runner can make a `turn.failed` for a turn that did not
emit its start (`provider_turn_lost_on_restore`).

## What Changed

Production code: 113 lines added, 19 removed.

- `src/native-session-runtime.ts` (15 added, 4 removed): the two
checkpoints in `consumeTurn`, with the sequence of the appended event,
and the resumed wait with a consumed disposition marker.
- `src/drivers/codex/codex-session-state.ts` (24 added, 1 removed),
`codex-session-terminal.ts` (2 added), `codex-session-notifications.ts`
(2 added), `codex-harness-session.ts` (24 added, 7 removed),
`codex-app-server-driver-impl.ts` (4 added, 1 removed): the snapshot at
the event, and the recovery of the driver.
- `src/contracts/harness-driver.ts` and
`src/contracts/native-session-backend.ts` (1 added each): an optional
`boundarySnapshot` on a driver session, and an optional sequence in the
snapshot options. Other drivers do not implement it and are not changed.
- `src/backends/harness-driver-backend.ts` (7 added, 5 removed): the
session terminal, and the snapshot as of a sequence.
- `src/live/runnerd-codex-transport.ts` (15 added, 1 removed) and
`src/live/provider-turn-tool-order.ts` (18 added): the start-first rule
and the owner of an event.
- Tests: `src/live/runnerd-acpx-provider-turn.test.ts` (572 added, 5
removed),
`src/drivers/codex/codex-app-server-driver.provider-turn.test.ts` (348
added), `src/live/runnerd-codex-transport.test.ts` (137 added),
`src/control-plane/durable-prp-control-plane.test.ts` (84 added),
`src/live/provider-turn-tool-order.test.ts` (new, 61),
`src/native-session-runtime.provider-turn.test.ts` (92 added, 6
removed), `src/backends/harness-driver-backend.test.ts` (56 added).
- `runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs` (test binary
only, 123 added, 14 removed): a delay before a provider-started turn, a
goal state at its terminal, a list of provider-started turns for one
session, a gap between them, a turn that stays open until the new
controller reads the snapshot, and a turn that starts after that read.
The modes that existed are not changed, and the fake reads its input on
one thread as before.

## State saved by the previous release

- No new persisted field. The driver session already has
`awaitingProviderTurn` (#21453).
- **More checkpoints:** one when a wait starts and one when a
provider-started turn starts. An ordinary turn saves the same number of
checkpoints as before (4 in the unit test, measured before and after).
- **No terminal in a checkpoint of a wait.** A checkpoint that is saved
during a wait or during a provider-started turn has `terminal: null`.
When a wait ends with no provider-started turn (the limit, a comment of
a person, or the Runner ends the run), the last checkpoint of the run
also has `terminal: null`; before, it had the terminal of the turn that
waited. One reader uses that field with no result: the text of a cleanup
failure (`nativeCheckpointRecordsCompletedTurn`), which then does not
say that the turn completed. In the tests with the real daemon (the
limit, the provider process lost, the sidecar exits), the run still ends
on the path of today (`native_finalization_missing`).
- **A checkpoint of the previous release** has no waiting state.
Recovery takes the path of today.
- **Rollback of the Runner package during a wait:** the previous code
reads the checkpoint, ignores `awaitingProviderTurn`, and takes its
recovery path of today.

## Verification

With nothing from part 3. The Rust checks and the build checks in the
first item were run before the last seven commits; these commits change
no Rust. The other items were run at the head of this branch
(`36d5e5e587`), or at an earlier commit of this branch for a suite whose
files the later commits do not change.

- Before the last seven commits: `pnpm run typecheck:typescript`, `pnpm
run build:typescript` and `node --test
test/acpx-sidecar-contract.test.mjs` (18 passed) in
`packages/paperclip-runner`; `node
scripts/verify-runner-vendor-dependencies.mjs` in `server`; `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`; `cargo fmt
--check`, `cargo build`, the 17 `acpx_*` suites (241 passed) and the
`acpx` unit tests (102 passed): all passed.
- `npx tsc --noEmit -p tsconfig.json` and `npx tsc --noEmit -p
tsconfig.test.json` in `packages/paperclip-runner`; `pnpm typecheck` in
`server`: all passed.
- `src/live/runnerd-acpx-provider-turn.test.ts`, 4 runs with the
production code of this head, 26 of 26 passed in each. The Core of these
tests follows the production append (`appendNativeSourceEvent`): it
stores an event that is out of order, keeps its contiguous cursor below
a gap, and reports a duplicate or a conflict for an event that it has.
It does not reject a gap in the sequence; production does not. The
restart cases, all with the real `paperclip-runnerd`:
- **From the checkpoint that the real consumer saved.** The first
`executeNativeSession` is fenced and detached as the server does for a
restart, and a second one recovers from the last checkpoint that the
first one saved. (a) Restart during the wait; the provider-started turn
starts after the new controller is attached. (b) Restart while a
provider-started turn is in progress. In each: the test asserts the
shape of that checkpoint, the run settles with the result of the
provider-started turn, each event is appended one time and in order, the
turn has one `turn.started` with `origin: "provider"`, and the session
closes with no error.
- **Without the two checkpoints, each of these cases fails:** the saved
checkpoint is the checkpoint of the turn before, and the second
controller fails with `turn_binding_mismatch` or with a replay conflict.
- **A slow `appendEvent`, with events that arrive behind the stored
event** (3 cases, each run 3 times with the fix and 3 times without it,
with the same result in each run). R7: the append of the
`turn.completed` that waits is slow, and the provider-started turn
starts behind it. With the snapshot at the event, the checkpoint is the
waiting state at the sequence that Core has, and Core gets the
`turn.started` of the turn one time. Without it, the checkpoint names
the provider-started turn as active, the transport builds no start, and
Core never gets that `turn.started`. R8: the first controller answers
the `paperclip_finish` call while the append of the provider
`turn.started` is slow. With the snapshot at the event, the checkpoint
has no result and the run ends with `native_finalization_missing` (see
Risks). Without it, the checkpoint has a result that Core does not have,
and recovery stops with "session cannot start another turn". R6: the
append of the provider `turn.started` is slow and a delta is queued
behind it. With the snapshot at the event, the new controller continues
the sequence that Core has. Without it, the only difference is one
sequence number that Core never gets.
- **The start was acknowledged before the restart** (3 cases: one turn
open across the restart; two turns; a turn that waits for the answer to
its `paperclip_finish` call). The new driver gets the start of each turn
one time before its other events, the finish call is answered exactly
one time, and the session closes with no error.
- **The turn ran with no controller attached** (2 cases, with and with
no result) and **two turns in one outage** (2 cases): the turns arrive
in order after the recovery.
- **A provider-started turn that is active in the checkpoint** gets no
second start, and the next provider-started turn arrives after it.
- **The turn already has its result when recovery completes** (3 cases:
the turn made its `paperclip_finish` call with no controller attached;
the turn waited for the answer to that call at the restart; two turns
across the restart). The test asserts the exact list of events that Core
gets: each provider-started turn has one start, its usage item, its
message item and its terminal, and then `run.result.accepted` and
`run.terminal`. `completeRun` is called one time with the turn that has
the result. Before the change to the session terminal, the second
controller appended only `run.result.accepted` and `run.terminal` in
each of these cases.
- **The wait checkpoint is not saved, then a restart** (R10). The last
checkpoint is the checkpoint of the controller turn, Core has the
terminal of that turn, and the start of the provider-started turn is
queued. The run settles with the result of the provider-started turn,
Core has one `turn.started` for it, and the finish call is answered one
time. Without the wait in the driver, recovery stops with
`turn_binding_mismatch`.
- **An ordinary turn**, with no option. One that waits for the answer to
its finish call at a restart: Core gets its result, its terminal and the
run terminal one time. One whose checkpoint after `startTurn` is ahead
of Core at a restart (R9): the run settles, and Core does not have only
the one event that the first controller did not store. These cases pass
with and without this pull request.
- **No checkpoint with a terminal and no result** is saved by the two
controllers in the cases above.
- `src/live/provider-turn-tool-order.test.ts` and
`src/live/runnerd-codex-transport.test.ts`: 345 passed. The helper tests
cover a stored start with removed events after it, a dropped delta, the
terminal of the turn before and after the event, a terminal of a
different turn after a removed terminal, and a controller start. With no
option, with a different agent or provider, and for a turn that the
controller started, the notifications of the transport are the same as
before this pull request.
- `src/control-plane/durable-prp-control-plane.test.ts`: 230 passed.
Three new tests use the real commit path: a full window of 4,096 events
removes the events after a provider start and the start is found; the
window removes the `turn.failed` of a provider turn after a later
`turn.failed` with no start, and no start is found; a restored
checkpoint that dropped the deltas of the turn, and the start is found.
- `src/drivers/codex` and `src/backends/harness-driver-backend.test.ts`:
491 passed. The driver tests cover a provider start that is queued
before `recoverSession()` is called, the wait state with one turn in
progress at `reconcile()` (and the two cases that it must still refuse),
a restored wait with a consumed disposition marker (the provider turn is
admitted and `startTurn` is refused), the snapshot at the two events
(also after the driver moved on, and for a restart continuation that
ends waiting), one later turn in progress, one completed and a second in
progress, two completed with a result, and the cases that `reconcile()`
must still refuse. The backend tests cover the session terminal and the
snapshot as of a sequence.
- `src/native-session-runtime.provider-turn.test.ts` and
`src/native-session-runtime.test.ts`: 333 passed. Two new cases resume a
restored wait with a consumed disposition marker: the provider-started
turn arrives and settles the run, or the wait ends at its limit and the
run takes the path of today with no new turn.
- `server`: `native-session-executor.test.ts`,
`native-provider-turn-wait.test.ts` and
`native-runner-restart-recovery.integration.test.ts`: 1,743 passed, 21
skipped.
- Mutation checks, each made named tests fail: no checkpoint at the
start of a wait; no checkpoint at the start of a provider-started turn;
the consumer does not pass the sequence (R6, R7 and R8 fail: a hole in
the sequence, no `turn.started` in Core, a different failure); the
backend ignores the sequence; the driver has no snapshot at the event;
no start-first rule; no exclusion of the active turn of the recovered
driver; the turn binding only at the first delivery; the contiguous
sequence required again; no check of the first terminal after the start;
the adoption of later turns put back; no exception in `reconcile()`; no
check after the settlement in `reconcile()`; the terminal of the session
set again by a turn that ends waiting; no wait of the provider start for
`reconcile()`; no exception in `reconcile()` for a session that waits;
the pump deferred for a restored wait; the consumed-marker shortcut
taken for a snapshot that waits.

Not run: the full suite. CI supplies that coverage.

## Risks

- **No production effect.** Each change is behind the waiting state, a
provider `turn.started`, or the option of the transport. No caller sets
the option before part 3.
- **An event that the first controller received but Core did not store
is not sent again after a restart.** This is the same on `master` and is
not changed here. Only the terminal of a turn is built again. If the
lost event is the result of a provider-started turn (the first
controller answered the `paperclip_finish` call and stopped before Core
stored the proposal), the run ends on the path of today for a turn with
no result. R8 shows this. R9 shows the same loss for one event of an
ordinary turn.
- **A second `turn.started` for one provider-started turn can reach
Core.** This occurs if the restart comes after Core stored the start and
before the checkpoint of that start is saved (the time of one checkpoint
write). The last checkpoint then waits, and the transport builds the
start again for the new driver. Seen with the real daemon when the
checkpoint write is held; not in a committed test. The usage accounting
of the server uses the turn id as its key, so there is no second count,
but the second start resets the "restored" mark of that turn. The
transcript does not read `turn.started`.
- **The older checkpoints can also save a cursor that is ahead of Core**
(measured for the checkpoint after `startTurn` of an ordinary turn).
Production accepts the gap in the sequence that follows, so this is not
a failure there. It is on `master` and is not changed here.
- **When the terminal of a provider-started turn was removed from the
stored events,** the transport builds no start for an event of that
turn, and the session stops on that event, as before. This needs a later
terminal of the same type with no start of its own (a turn that was lost
while it started) and a full window.
- **The last checkpoint of a wait that ends with no provider-started
turn has no terminal,** as the section above says. The change is only in
a run that waits, and no run waits before part 3.
- **A wait with a consumed disposition marker has no test with the real
daemon.** The state needs two restarts in one run, and the test helper
does one. Unit tests of the driver and of the consumer cover it.
- **A provider start that waits for `reconcile()`** waits as long as the
reads of `reconcile()` take. Each read ends at the command deadline of
the Runner (30 seconds by default) or when the transport closes.
- **`item.delta` events across a restart are not asserted.** A streamed
delta that the first controller received is not sent again. The
completed items are asserted.
- **The driver keeps a maximum of 8 snapshots at events** and drops each
one when the consumer asks for it or for a later one. If the consumer is
more than 8 such events behind, the checkpoint uses the current
snapshot, as before this change.
- **A second provider-started turn that also ends during the same
outage** is not tested with the real daemon: the fake gives each
terminal the same goal revision, and the Runner then holds the second
terminal until a controller is attached. The end of the reconnect time
of the Runner is not tested here.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, decided the designs, reviewed the diffs, and ran a
part of the verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as Claude Code subagents with
tool use (shell, file edits, local test runs), wrote the change and the
tests, reproduced the review findings with the real daemon, and verified
this slice alone.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 12:51:52 -08:00
DottaandClaude Opus 5.5 d4498ae7ac fix(runner): settle every recorded tool result and free sessions blocked by a closed turn's pending call (#21490)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call Paperclip tools through the Runner. The Runner records
each call and returns the controller's result to the provider (here
OpenCode)
> - In chat-eval cohort 13, a `list_workflows` result was recorded but
never reached OpenCode. OpenCode waited about 290 s, and the call stayed
pending
> - The next run's `run.attach` then failed with
`durable_pending_tool_calls`, so the person's follow-up question never
got an answer
> - This pull request fixes the result shape that caused the rejection,
makes the Runner settle every recorded result, and lets the next run
settle a call that a closed turn left pending
> - The benefit is that a tool result always reaches the agent, and a
stuck call can no longer block every later turn

## Linked Issues or Issue Description

**What happened?**
Chat-eval cohort 13 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `df1b12ad8e`). In
`cases.case-history` and `cases.create-case`, the CEO called
`list_documents` (or `list_projects`) and `list_workflows` (call id 5)
together. The server answered `GET /companies/{id}/workflows` with 200
in 584 ms. The OpenCode MCP bridge diagnostic for call 5 says `outcome:
"unanswered"`, `written: false`, `handlerMs: 289038` (290723 in
create-case). The sibling call was answered in about 1 s. The run ended
with `runner_settlement_incomplete`: `semanticTools.pending` held call 5
and its `semantic_tool.result` delivery command had status `failed`. The
next question started a run whose `run.attach` was refused: `run.attach
requires a settled Codex provider session: durable_pending_tool_calls`.
The run failed in 17 s, and the question was not answered.

Cause: `executeNativeWorkflowTool` returned the Runner API result
unchanged: `{ ok: true, status: 200, operationId: "GET
/api/companies/{companyId}/workflows", ... }`. Every other API-backed
tool renames that field to `apiOperationId`. The Rust tool bridge
(`semantic_response_value`) reads an object with a boolean `ok` and an
`operationId` as a semantic envelope that must name this tool and call.
The REST route did not match, so `apply_result` failed with `semantic
result envelope does not match its provider call`. Nothing was sent to
the provider, and the call stayed pending forever. The red test
reproduces this exact error. The parallel call is not the trigger: the
sibling call settled normally, and the OpenCode MCP bridge answered
other parallel pairs in the same runs. `list_workflows` first appears in
cohort 13, so cohorts 11 and 12 did not hit it. All four workflow tools
had this shape.

**Expected behavior**
Every recorded tool result reaches the provider. When a recorded result
cannot be delivered, the provider gets an error at once and the call
settles. A call that a closed turn left pending does not block the next
run.

**Steps to reproduce**
1. Run an OpenCode agent with Cases enabled on the durable Runner.
2. Ask a question that makes the agent call `list_workflows`.
3. See the call wait about 290 s. Then ask a follow-up question and see
`run.attach` fail with `durable_pending_tool_calls`.

**Paperclip version or commit**
`master` at `df1b12ad8e`.

## What Changed

- `server/src/services/native-runtime/native-workflow-tools.ts`:
workflow tool results name the route `apiOperationId`, like case tools
and `call_api`.
- `provider_bridge.rs`: add `settle_undeliverable_result`. When the
bridge rejects a recorded result for a call it still holds, the call
settles with a bounded, non-retryable runner error. Conflicting or
unknown results still fail as before.
- `provider_backend.rs` (`deliver_semantic_result`): use that
settlement, deliver the error to the provider, and record a
`semantic_tool_result_undeliverable` diagnostic with the call,
operation, reason and the rejected result's digest.
- `provider_bridge.rs` and `provider_backend.rs` (`attach_run`): add
`expire_closed_turn_calls`. When nothing is live (no active or ambiguous
turn, no provider request waiting), `run.attach` settles calls that a
closed turn left pending as an expired failure with an unknown outcome,
and records `semantic_tool_result_expired`. The provider stopped waiting
when its turn ended, and the new run's controller never delivers the old
run's results. A call that a live turn still waits on still blocks
attachment. This follows the existing `tool_call_interrupted_by_restart`
rule: never execute the operation again, and record an unknown outcome
when no result can ever arrive.
- Docs: `doc/run-log-events.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe both
diagnostics.

## Verification

- Red/green, Rust (`cd packages/paperclip-runner/runner && cargo test -p
paperclip-runner-core --test codex_provider --
a_rejected_recorded_result run_attach_expires`):
- Before:
`a_rejected_recorded_result_still_answers_the_provider_and_frees_the_session`
fails with `semantic result envelope does not match its provider call`
(the production error).
`run_attach_expires_a_call_left_pending_by_a_closed_turn` fails with
`run.attach requires a settled Codex provider session:
durable_pending_tool_calls` (the production refusal).
- After: both pass. The fake provider logs `tool-response:failure`, so
it got an answer at once. The next `run.attach` succeeds.
- Parallel calls (`cargo test -p paperclip-runner-core --lib --
provider_bridge`):
`a_rejected_result_in_a_parallel_pair_still_settles_its_own_call` sends
the cohort 13 pair (`list_documents` + `list_workflows`). The rejected
call settles, its sibling keeps its own result, and the next turn
starts. `only_a_closed_turn_expires_its_pending_calls` checks that an
open turn expires nothing.
- Upgrade and mid-run cutover:
`run_attach_expires_a_call_left_pending_by_a_closed_turn` saves state
with a pending call after the turn ends, stops that Runner, and attaches
with a new Runner process from the saved state.
`durable_backend_settles_pending_tools_when_recovery_finds_the_turn_ended`
(unchanged) still passes: a late result in the same run still settles as
`settled_after_turn`.
- Red/green, server (`cd server && npx vitest run
src/services/native-runtime/native-case-tools.integration.test.ts -t
workflows`): before, `expected { ok: true, status: 201, … } to not have
property "operationId"`; after, 2 passed.
- `cargo test -p paperclip-runner-core`: 643 lib tests pass. One lib
test (`fenced_receipt_output_keeps_prefix_when_stdout_never_closes`)
failed once with `EPERM` while signaling a process group under parallel
load. It passes alone 3 of 3 and does not touch this change. The
integration test binaries pass.
- Paid narrow run for `cases.case-history` and `cases.create-case`: see
the comment below.

## Risks

- Previous-release story: no persisted shape changes. State saved by the
previous Runner with a call pending after its turn closed carries
forward. The new Runner settles that call at the next `run.attach`, and
the session continues (tested from saved state with a new process). A
previous-release Runner that receives the corrected `apiOperationId`
result delivers it normally, because the result is no longer read as an
envelope.
- Behavior change: a call left by a closed turn is now settled at the
next run's attachment, not kept forever. Its outcome is recorded as
unknown, and the operation is never run again. A provider turn that is
still live keeps the old refusal (`durable_pending_tool_calls`,
`provider_side_effect_pending`), and the controller still never replaces
the session.
- A recorded result that the bridge rejects now becomes an error for the
agent, not an endless wait. The agent is told the action may have
completed and must check the current state before repeating it.
- Latency: nothing is added before turn start. The attach-time check
walks the in-memory pending calls (usually none) and adds no I/O beyond
the existing state save. The rejected-delivery path now answers in the
same command instead of after the provider's ~290 s client timeout, and
the follow-up run no longer fails after 17 s.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21398 (first release where `list_workflows` was called in this
eval), #19733 (the attach refusal), #20886 (earlier OpenCode delivery
fix).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 14:24:21 -06:00
Nicky LeachandClaude Opus 5.5 68d5c9c120 feat(runner): let a caller ask the transport for provider-started turns (#21607)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core runs Claude Code through Paperclip Runner. Claude can start a
shell command in the background and end its turn. When the command
finishes, Claude Code starts a cycle of its own to continue.
> - Today the run ends when the turn ends. The command stops, and the
run fails with `native_finalization_missing`.
> - The Runner can already keep Claude while a background shell is
running and give the cycle of Claude to the controller as a turn that
the provider started (#21423, #21316). Core can already wait for that
turn (#21453, #21454, #21455). No run does this yet, because no
controller asks the Runner for it.
> - The request goes through the transport of the Runner package. This
pull request gives the transport an option for it and the behavior that
the option needs.
> - No caller sets the option in this pull request. It changes no
production run.
> - The benefit is a small base for the two pull requests after it: the
recovery after a controller restart, and the switch that turns the
behavior on.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

#21456 turned the behavior on and had grown too large to review. It is
now three stacked pull requests:

1. **This pull request:** the transport can ask the Runner for
provider-started turns. No caller asks.
2. #21612: the recovery after a controller restart during a wait or
during a provider-started turn.
3. #21456: Core sets the option for native ACPX Claude runs. Only that
part changes what a production run does.

Refs #19912 (design), #21456.

**What existing behavior does this improve?**

None in production. It adds the support that parts 2 and 3 use.

**Subsystem affected**

Paperclip Runner package, the controller side:
`src/live/runnerd-codex-transport.ts`. ACPX Claude only, and only for a
caller that sets the new option.

**Current behavior**

- The transport has no way to ask the Runner for provider-started turns.

**Proposed behavior**

- The transport options have a new field, `providerStartedTurns`. The
transport uses it only for ACPX with the agent `claude`. In each other
case the field has no effect.
- With the option, the transport sends `providerStartedTurns: true` in
`turn.start`. It does not send it for a compaction turn.
- With the option, a tool call in a provider-started turn waits until
the transport gave each earlier event to the driver. Without this, a
`paperclip_finish` of that turn could reach the driver before the
`turn.started` of that turn, and the driver would refuse it.
- With the option, the transport gives a `run.terminal` of the Runner to
the driver as a marked internal notification (`paperclip/runTerminal`).
The driver uses it only during a wait, so a wait ends at once when the
Runner says that the run is over.
- With no option, the transport does the same as before this pull
request. Live sessions, evals, the CLI runner and the probes do not set
the option.

**Reason and benefit**

- The opt-in is a choice of the caller, not a property of the agent.
Only a consumer that can wait for a provider-started turn must ask for
one. `CapabilityLiveSession` completes a turn at the first
`turn/completed`, so it must not get the behavior because its agent is
Claude.

## What Changed

- `src/live/runnerd-codex-transport.ts` (31 lines added, 1 removed): the
option, the field in `turn.start`, the order of the tool call, and the
`paperclip/runTerminal` notification.
- `src/live/provider-turn-tool-order.ts` (new, 16 lines): finds the
source sequence of a tool call that belongs to a provider-started turn.
- `src/live/runnerd-acpx-provider-turn.test.ts` (new, 239 lines): 7
cases with the real `paperclip-runnerd`, the fake ACPX sidecar, and the
real consumer of Core.
- `src/live/live-session.test.ts` (12 lines added): a Claude live
session gives the transport no such option.
- `runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs` (test binary
only): the mode `provider-result-resolved`. In this mode the
provider-started turn calls `paperclip_finish` and waits for the answer
of Core through the Runner (`tool.resolve`), as the real sidecar does.
The mode `provider-result` is not changed.

## State saved by the previous release

- No persisted state changes in this part.
- **A Runner that does not know the field** ignores it.
- **Rollback of the Runner package:** the previous transport does not
send the field, so the Runner does not wait.

## Verification

At the head of this branch (`2f2cb14b9c`), with nothing from parts 2 and
3:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json`, `pnpm run typecheck:typescript` and `pnpm run
build:typescript` in `packages/paperclip-runner`; `pnpm typecheck` and
`node scripts/verify-runner-vendor-dependencies.mjs` in `server`; `node
--test scripts/__tests__/runner-release-crate-cache.test.mjs`; `node
--test test/acpx-sidecar-contract.test.mjs` (18 passed): all passed.
- `src/live/runnerd-acpx-provider-turn.test.ts`, 3 runs, 7 of 7 passed
in each. The cases:
- background work, then the turn of Claude calls `paperclip_finish`;
Core answers; the Runner gives the answer to the sidecar with exactly
one `tool.resolve`; the turn completes; the run finalizes with one
`run.result.accepted`;
- a Claude run with no option from the caller takes the path of today:
no `providerStartedTurns` in `turn.start`, no `paperclip/runTerminal`,
no wait, and no `session.provider_turn` to the sidecar;
- the limit ends the wait, the run takes the path of today, and the
provider processes had exited before the cleanup of the test;
- the provider process is lost during the wait, and the run settles in
seconds;
  - the sidecar exits during the wait, and the run settles in seconds;
  - a turn with no background work does not wait;
- an ACPX agent that is not Claude is never opted in, also when the
caller sets the option.
- Each case asserts the result of the close of the session: clean in 6
cases, and a named failure in the case where the sidecar is killed. Each
clean case also asserts that there is no
`native_session_settlement_incomplete` diagnostic and no "conflicting
duplicate tool result" in the durable state.
- `src/live/live-session.test.ts`,
`src/live/runnerd-codex-transport.test.ts`,
`src/live/runnerd-acpx-provider-turn-wait.test.ts`,
`src/native-session-runtime.provider-turn.test.ts`: 434 passed.
- Rust, because the fake sidecar changed: `cargo fmt --check`, `cargo
build`, the `acpx_*` suites (244 passed) and the `acpx` unit tests (102
passed).
- Mutation checks, run on the full change before the split: with the
option ignored, 6 of the 7 assertions of the case with no option fail;
with the old mode of the fake, the case with the result fails at the
close of the session.

Not run: the full suite. CI supplies that coverage.

## Risks

- **No production effect.** No caller sets the option. Core sets it in
part 3.
- **The transport is shared by each provider.** The three new behaviors
are behind one condition: the option, ACPX, and the agent `claude`. With
the condition false, the code path is the path of today.
- **The tool-call ordering uses a 10 ms poll** while earlier events are
not delivered, only in a provider-started turn.
- **A controller restart during a wait or during a provider-started turn
is not handled in this part.** Part 2 handles it. Part 3 must not merge
before part 2.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 08:16:24 -08:00
Nicky LeachandClaude Opus 5.5 4a78124a3e feat(runner): wait for a provider-started Claude turn when the controller asks (#21423)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner runs Claude Code through the ACPX sidecar.
> - Claude can start a shell command in the background and end its turn.
When the command finishes, Claude Code starts a cycle of its own to
continue. Today the Runner stops Claude at the end of the turn, so the
command stops and the run fails with `native_finalization_missing`.
> - A stack of pull requests lets the Runner accept that cycle as a new
turn of the same run, when a controller asks for it.
> - `paperclip-runnerd` ends a run when a turn ends. For a Claude turn
that ended with no result while a background shell is running, this
stops the shell and the run fails.
> - #21278 binds each tool call to its turn, and #21293 reports how many
background shells are running at the end of a turn. Part 4 declared the
contract.
> - This pull request makes `paperclip-runnerd` wait for a turn that
Claude starts by itself, when the controller asks for it, and admit that
turn into the same run.
> - The benefit is that the Runner has one place that decides to wait
and one place that ends a wait.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one part of a change that was first opened as one
large pull request (#21316). The parts, in stack order after #21311:

1. #21419: sidecar preparation (a refactor).
2. #21420: converter for raw ACP session updates.
3. #21421: cycle stream at the adapter.
4. #21422: contract for provider-started turns.
5. #21423: `paperclip-runnerd` waits for a provider-started turn.
6. #21316: the sidecar adopts a cycle.

This is part 5. Its base branch is `feat/prp-provider-turn-contract`.
After each part, a production run does the same as today, because no
controller sends the opt-in. Refs #19912 (design).

**What existing behavior does this improve?**

What `paperclip-runnerd` does after a Claude turn ends with no result
while a background shell is running. Nothing changes until a controller
sends the opt-in.

**Subsystem affected**

Paperclip Runner: the ACPX provider backend in `runner-core` (Rust).
Claude only.

**Current behavior**

- After each completed turn the Runner emits `run.terminal`. A turn with
a running background shell gets the provider restart, and the shell
stops.

**Proposed behavior**

Nothing changes unless the controller sends `providerStartedTurns: true`
in the payload of `turn.start`. No controller sends it yet.

With that field:

- The Runner keeps the provider and waits when all of these are true:
the agent is Claude; the turn completed; the run has no accepted
`paperclip_finish` or `paperclip_block` result; the terminal event
reported `backgroundTasks` with `live` above zero; each condition of
#21293 for a kept process is true; the sidecar accepted
`session.provider_turn` with `armed: true`.
- `turn.completed` then has `awaitingProviderTurn: true`, and no
`run.terminal` follows. The Runner holds that `run.terminal`.
- When the sidecar reports a turn that the provider started, the Runner
admits it only during a wait and publishes `turn.started` with `origin:
"provider"`. In each other case that report fails the session closed.
- When a wait ends with no turn active, the Runner publishes the held
`run.terminal`. This applies when the provider process exits, the
sidecar exits, the controller cancels or stops the run, the session is
closed or suspended, and after a Runner restart.
- The held `run.terminal` and the removal of the wait are saved in one
write, and the controller gets the event from the next poll. A crash
after a cancel or a stop cannot lose it, and it is never sent two times.
- A controller `turn.start` during a wait replaces the provider first,
as before, and the held event is dropped.
- A `run.attach` from a different run during a wait ends the wait when
the attach succeeds. The Runner then discards the held `run.terminal`,
because that event is of the old run and must not end the new run. This
is the same rule that a Runner restart applies. The wait is removed only
in the state that the attach saves for the new run. If the attach fails,
for example when the old session does not stop, the old run keeps the
wait and gets its held `run.terminal` one time, from the next poll or
from a cancel or a stop. A `run.attach` from the same run publishes the
held event to that run.
- The Runner waits at most 5 seconds for the sidecar to answer a request
to arm or to disarm a provider-started turn. This is the limit that turn
cancellation already uses. Before, the limit was the 30 seconds of an
ordinary request, and a cancel, a stop or a shutdown had to wait behind
it. No answer is handled as a lost sidecar: the session fails closed,
and the Runner still publishes the held `run.terminal` one time.
- The sidecar of this branch does not know `session.provider_turn`, so
the arming request fails and the Runner does not wait. The behavior
comes with part 6.

**Reason and benefit**

- The wait is one entry in the saved session state. One rule ends it at
each save, so each new way to leave the open session ends the wait with
no new code.
- A wait must not outlive the provider or the sidecar, and a cancel must
stop it.

## What Changed

- `acpx_provider_turn_wait.rs` (new, 230 lines with its unit tests):
`provider_turn_wait { turn_id, held_run_terminal }` in the saved state;
the rule that ends a wait at save; `settle_turn_events`, which decides
at one place to arm, to create the wait and to put
`awaitingProviderTurn` on `turn.completed`.
- `acpx_provider_turn.rs` (new, 119 lines): the offer of a turn at its
terminal, the arming request, and `admit_provider_turn`.
- `acpx_provider_session.rs`, `acpx_provider_backend.rs`,
`acpx_provider_state.rs`, `provider_events.rs`,
`acpx_park_preparation.rs`: the opt-in from `turn.start`; the record
that the run has a result; the provider-started turn in the state and in
the PRP events; cancel and stop with no turn; no turn-end restart and no
park during a wait.
- `acpx_sidecar_transport.rs` (11 lines): during a wait, a poll with no
event also checks that the sidecar process is alive. A child process can
keep the output stream of a dead sidecar open.
- `bin/fake-acpx-sidecar.rs` (232 lines): provider-started turns and
failures for tests.
- Tests (1,164 lines): `tests/acpx_quiet_turn_continuation.rs`,
`tests/support/acpx_quiet_turn_restart.rs`,
`src/live/runnerd-acpx-provider-turn-wait.test.ts` (the real daemon),
`src/native-session-runtime.provider-turn.test.ts` (the controller of
today).
- `runner/DURABLE_TRANSPORT.md`: the rules.

## State saved by the previous release

- One new optional field in the saved ACPX session state:
`providerTurnWait`. It is written only during a wait. In each other
state the saved file has the shape of the previous release.
- **State from the previous release, read by this release:** the field
is absent, so there is no wait. A unit test covers this.
- **State from this release, read by the previous release (a
rollback):** outside a wait, the file loads as before. During a wait,
the previous Runner refuses the file with its "ACPX provider state is
malformed" error, because its loader refuses each unknown field. Only a
run that waits at the moment of a rollback is in this case, and no run
waits before a controller sends the opt-in.
- **Runner restart during a wait:** the wait is restored. The provider
is a new process, so the wait is over, and the controller receives the
held `run.terminal` one time. A test with the real daemon covers this.
- **Controller of today or of the previous release:** `turn.start` has
no opt-in. A test runs a turn with a running shell and a turn with an
unknown background state, and compares the events, the commands after
the turn, the restart, and the park state before and after the restart.
They are the same.

## Verification

At the head of this branch, from `packages/paperclip-runner`:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run
build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs
test/protocol-contract.test.mjs`, the server check `node
scripts/verify-runner-vendor-dependencies.mjs`, `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt
--check` and `cargo build -p paperclip-runner-core --bins` (no new
warning): all passed.
- `cargo test -p paperclip-runner-core --lib`: 644 passed, 4 ignored.
All 17 `acpx_*` suites passed.
- `src/live/runnerd-acpx-provider-turn-wait.test.ts`, 5 cases with the
real `paperclip-runnerd` and the fake sidecar through PRP: a wait ends
with exactly one `run.terminal` when the sidecar exits while a child
process keeps its output open, when the provider is lost, when the
controller cancels, and after a Runner restart; with no opt-in,
`run.terminal` follows the turn and the sidecar is never armed.
- Rust tests: wait and admit; no wait with a result; no wait with no
opt-in; an active goal neither marks nor arms a wait; a sidecar that
refuses to arm; a provider-started turn that the Runner did not wait for
fails the session closed; cancel and stop during a wait; provider and
sidecar loss; restart; the saved shape in both directions.
- `src/native-session-runtime.provider-turn.test.ts`: the real
`consumeTurn` of today gets the event sequence with no opt-in and
finalizes it as `native_finalization_missing`, as today.
- Attach during a wait, in `tests/support/acpx_quiet_turn_restart.rs`: a
different run attaches after the snapshot reports `warmAttachReady:
true`; the new run gets no `run.terminal`, the saved state has no wait,
and the old session is closed. A same-run attach gives exactly one
`run.terminal`. An attach that fails when the old sidecar is dead leaves
the wait with the old run, and the old run then gets exactly one
`run.terminal` from a poll, `run.cancel`, `turn.interrupt` or
`turn.stop`. Mutation checks: with no fix, the old `run.terminal` is in
the pending events of the new run; with the wait ended before the attach
is complete, the old run gets no `run.terminal` after a failed attach.
- No answer from the sidecar, in
`tests/support/acpx_quiet_turn_restart.rs`: the fake sidecar does not
answer the disarm request; `run.cancel` and `turn.interrupt` each return
`already_settled` in less than 10 seconds, exactly one `run.terminal`
reaches the run, `session.close` completes in less than 2 seconds with
no second `run.terminal`, and the provider process is gone. With no
answer to the arm request, the turn ends with its `run.terminal` and
with no wait. With the limit removed, each test fails at 30 seconds.
- Mutation checks, each made the named tests fail: the rule that ends a
wait at save; no `run.terminal` after a restart; arming with no opt-in;
the marker set with no wait; the sidecar process check removed.

Not run: a rollback with the binary of the previous release; a run with
the opt-in and a sidecar that answers the arming request with an error
(the code reads each error as not armed); Linux; the full suite. CI
supplies the broad coverage.

## Risks

- **The feature is off.** No controller sends the opt-in.
- **A rollback of the Runner during a wait** makes the previous Runner
refuse that saved session. See "State saved by the previous release".
- **The wait has no time limit in the Runner.** The controller must
close, suspend or cancel. Core will own that limit.
- **Park is refused during a wait.** The controller must end the wait
before it parks.
- **A new run can attach during a wait.** The attach is not blocked,
because the Runner has no time limit on the wait and a new run can send
only `run.attach`. After an attach that succeeds, the old run gets no
`run.terminal` from the Runner; the controller must not depend on that
event. The request that disarms a sidecar that the attach keeps has no
test, because no test can keep a session across an attach during a wait.
- **After a sidecar that does not answer, `runner.suspend` returns an
error at once** ("provider lifetime cleanup is not yet proven"), and
`session.close` completes. This is the behavior of today for each lost
sidecar; this PR does not change it.
- **The shared code that changed** runs for each Claude turn: the record
of a run result, and the settle step at each turn end.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:21:10 -08:00
Devin FoleyandPaperclip 493519fc88 fix(runner): poll a live ACPX turn without waiting out the idle read (#21525)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runner (runnerd) relays provider events to the server and
waits for cumulative ACKs before it polls the provider again.
> - After a control frame, runnerd uses a 1 ms queued read only when the
executor reports a live provider poll. Otherwise it waits up to the 250
ms runtime read timeout, and any frame that arrives in that time skips
the poll.
> - Only the Codex facade reports a live poll. ACPX runs (Claude, Codex
through ACPX and others) always wait the full 250 ms after the last ACK
of every batch, and steady control traffic starves their provider polls.
> - This pull request lets an ACPX executor report a live poll while its
restored session has an active turn.
> - The benefit is that ACPX output is relayed without the 250 ms idle
wait, and tool-heavy turns no longer starve provider polls.

## Linked Issues or Issue Description

**What happened?**
A routine run started several subagents that made about 190 tool calls
in about 10 minutes. On the server, consecutive runner events arrived at
least 576 ms apart (median about 1.5 seconds), one event at a time,
while the time from each event's `emittedAt` to its insert was 0 to 0.1
seconds. The runner fell several minutes behind the provider. The
backlog then overflowed the transport's event buffer (fixed separately
in #21524) and the turn failed.

**Expected behavior**
A live ACPX turn is polled again as soon as its outstanding events are
acknowledged and no control frame is queued, the same as a Codex facade
turn.

**Steps to reproduce**
1. Run an ACPX turn that produces events while the controller sends
commands and ACKs.
2. After each ACK, runnerd calls `receive_json`, which waits up to 250
ms before it polls the provider. A frame that arrives in that time skips
the poll.

**Relevant logs or output**
```
min gap between consecutive runner events: 576 ms, median about 1.5 s, never batched
insert time - emittedAt: 0.0 to 0.1 s
```

Related: #21524 bounds the receive buffer that this backlog overflowed.

## What Changed

- `acpx_provider_backend.rs`:
`AcpxCommandExecutor::provider_poll_is_live` returns true only when the
session was restored, has no persistence error, does not wait for a
next-turn generation, and its state is `turn_starting` or `turn_active`
with an active turn, no unconfirmed provider exit, and the run it owns.
- `native_provider_backend.rs`: `SelectedExecutor` forwards
`provider_poll_is_live` to the ACPX executor.
- `tests/acpx_warm_attachment.rs`: new test through
`NativeProviderCommandExecutor`. The session is not live before
`run.prepare` or after `session.open`, is live after `turn.start`, and a
new executor that has not restored its session is not live.

What does not change:
- runnerd still polls only when the read returns no frame, so a queued
stop or suspend still runs before the next provider poll.
- The ACK gate (no poll while sent events are not acknowledged) does not
change.
- The 1 ms read is used only while the source sequence advanced since
the last quiet poll, as for Codex today, so an idle turn does not
busy-loop.

## Verification

- `cargo test --release --manifest-path runner/Cargo.toml --locked -p
paperclip-runner-core --test acpx_warm_attachment` (in
`packages/paperclip-runner`): 7 passed. The new test fails without the
new method, because the default returns false.
- `cargo fmt --all -- --check` and `cargo check --locked --workspace`
pass.
- `cargo test --release --locked -p paperclip-runner-core
--no-fail-fast`: 1,127 passed, 3 failed. The failures are
`codex_provider::tests::fenced_receipt_output_reports_capacity_without_dropping_known_frames`
("failed to signal owned process group: Operation not permitted") and
`stdio_mcp::lost_acknowledgement_blocks_calls_until_socket_replacement`
and `metadata_only_socket_changes_do_not_bypass_unknown_outcome` (relay
exit timing). They also fail without this change in the macOS sandbox
where I ran the suite, and none of them uses the ACPX executor. CI
covers them on Linux.
- The existing `durable::tests::ack_release_poll` cases cover the runner
side of the live read: an ACK releases the next poll without the read
timeout, and a command queued behind the ACK runs first.
- I did not run a live provider. After the deploy, the gap between
runner events of an ACPX run with queued output should drop below 250
ms.

## Risks

- Low risk. The change only selects the existing 1 ms queued read for
ACPX during an active turn.
- A turn that produces events continuously now polls more often. Each
poll still writes the same durable state as before, so disk writes per
event do not change.
- Analysis also found that runnerd waits for every sent event to be
acknowledged before it polls again. A bounded in-flight window would
raise throughput further, but it changes the tested rule that control
queued behind an ACK runs before the next poll. That is left out of this
pull request on purpose.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 16:46:05 -08:00
Devin FoleyandPaperclip 254744f77c fix(runner): bound the ACPX receive buffer by bytes, not 512 events (#21524)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runner talks to ACPX providers through a sidecar process.
The sidecar writes events and command replies as JSON lines on stdout.
> - While the transport waits for a command reply, it buffers every
event that arrives before the reply. It failed when 512 events were
buffered.
> - The sidecar allows up to 4 MiB of unflushed stdout. A backlog of
small events passes 512 events long before any memory concern, for
example during a long turn with several subagents while the runner
relays events slowly.
> - When the limit hits, the transport is poisoned and the turn fails
with `acpx_provider_transport_failed`, which is not recoverable. The
task is then blocked.
> - This pull request bounds the receive buffer by frame bytes,
consistent with the sidecar's own output bound, and keeps a count
backstop for floods of tiny frames.
> - The benefit is that a normal event backlog no longer kills the turn,
and the buffer stays bounded in memory.

## Linked Issues or Issue Description

**What happened?**
A routine run started several subagents, which made about 190 GitHub
tool calls in about 10 minutes. The runner relays provider events at
about one per controller round trip, so the sidecar output fell several
minutes behind. A semantic tool result was then delivered through
`tool.resolve`. While it waited for the reply, the transport read more
than 512 backlogged events, failed with `ACPX sidecar exceeded the
buffered event limit`, and poisoned the transport. The turn failed with
`acpx_provider_transport_failed`, and recovery blocked the task.

**Expected behavior**
A backlog that the sidecar itself accepted does not fail a request. Only
a backlog that is too large for memory fails.

**Steps to reproduce**
1. Let a sidecar write more than 512 small events before it writes the
reply to a pending command.
2. The command fails with `ACPX sidecar exceeded the buffered event
limit`, and the transport is poisoned.

**Relevant logs or output**
```
ACPX sidecar exceeded the buffered event limit
turn.failed code=acpx_provider_transport_failed recoverable=false
```

No related public PR exists. I searched open and closed PRs for this
limit. The slow event relay that builds the backlog is a separate
problem.

## What Changed

- `acpx_sidecar_transport.rs`:
- Each buffered event keeps the byte length of its frame. The transport
keeps a running byte total.
- The buffer fails above 16 MiB (`MAX_BUFFERED_EVENT_BYTES`, four times
the sidecar's 4 MiB unflushed output bound
`SIDECAR_MAX_UNFLUSHED_OUTPUT_BYTES`), or above 65,536 events (one event
per 256 bytes of the budget) as a backstop for tiny frames.
- `poll_event` and `defer_polled_event` keep the byte total correct. A
poisoned transport resets it.
- `fake-acpx-sidecar.rs`: new `backlog`, `large-backlog` and
`count-flood` modes. `flood` now writes 18 MiB of 64 KiB events instead
of 513 small events.
- `tests/acpx_sidecar_transport.rs`: new tests for a backlog of 2,048
small events, for freeing the byte budget as events drain, and for a
flood of tiny events. The existing flood test still checks the error.

## Verification

- `cargo test --release --manifest-path runner/Cargo.toml --locked -p
paperclip-runner-core --test acpx_sidecar_transport` (in
`packages/paperclip-runner`): 23 passed.
- `buffers_a_backlog_of_small_events_before_a_response` fails before
this change (2,048 events are above the old 512 limit).
`frees_the_byte_budget_as_buffered_events_drain` needs correct
accounting: two backlogs of 12 MiB pass only when drained events free
their bytes.
- `cargo fmt --all -- --check` and `cargo check --locked --workspace`
pass.
- `cargo test --release --locked -p paperclip-runner-core
--no-fail-fast`: 1,128 passed, 4 failed. The failures are
`codex_provider::tests::fenced_receipt_output_keeps_prefix_when_stdout_never_closes`
and
`fenced_receipt_output_reports_capacity_without_dropping_known_frames`
("failed to signal owned process group: Operation not permitted"), and
`stdio_mcp::lost_acknowledgement_blocks_calls_until_socket_replacement`
and `metadata_only_socket_changes_do_not_bypass_unknown_outcome` (relay
exit timing). None of them uses the ACPX sidecar transport. They come
from the macOS sandbox where I ran the suite. CI covers them on Linux.
- `cargo test --release -p paperclip-runner-core --lib acpx`: 99 passed.
`--test acpx_provider_resolutions --test acpx_sidecar_stopped_time`:
passed.
- I did not run a live provider. After the deploy, a long turn with many
tool events should not fail with `buffered event limit`.

## Risks

- Low risk. The change is local to the transport receive buffer.
- The memory bound is now 16 MiB of frames instead of 512 events. Parsed
events use more memory than their frames, so the worst case is a few
times 16 MiB in the runner process.
- The sidecar's own 4 MiB unflushed output bound does not change. A
burst that is longer than that still fails in the sidecar with
`output_buffer_limit_exceeded`. The slow relay that builds the backlog
is a separate fix.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 16:45:58 -08:00
Nicky LeachandClaude Opus 5.5 1332ab987f feat(prp): declare the contract for provider-started turns (#21422)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner runs Claude Code through the ACPX sidecar.
> - Claude can start a shell command in the background and end its turn.
When the command finishes, Claude Code starts a cycle of its own to
continue. Today the Runner stops Claude at the end of the turn, so the
command stops and the run fails with `native_finalization_missing`.
> - A stack of pull requests lets the Runner accept that cycle as a new
turn of the same run, when a controller asks for it.
> - The Runner, its sidecar and the controller speak two protocols with
schemas: PRP between the controller and the Runner, and the sidecar
protocol between `paperclip-runnerd` and the ACPX sidecar.
> - A turn that the provider starts needs new fields and one new command
in those protocols.
> - This pull request declares them and generates the readers again. It
changes no behavior.
> - The benefit is that the next two pull requests show behavior only,
with the contract already in review.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is one part of a change that was first opened as one
large pull request (#21316). The parts, in stack order after #21311:

1. #21419: sidecar preparation (a refactor).
2. #21420: converter for raw ACP session updates.
3. #21421: cycle stream at the adapter.
4. #21422: contract for provider-started turns.
5. #21423: `paperclip-runnerd` waits for a provider-started turn.
6. #21316: the sidecar adopts a cycle.

This is part 4. Its base branch is `feat/claude-provider-cycle-stream`.
After each part, a production run does the same as today, because no
controller sends the opt-in. Refs #19912 (design).

**What existing behavior does this improve?**

Nothing in production. The protocol schemas.

**Subsystem affected**

Paperclip Runner: `protocol/` and the generated TypeScript and Rust
contract code.

**Current behavior**

- PRP has no way to say that a turn was started by the provider, or that
the Runner waits for such a turn.
- The sidecar protocol has no command to arm the sidecar and no event
for a turn that the provider starts.

**Proposed behavior**

- PRP (`schemas/provider-turn.schema.json`, new): `providerStartedTurns`
(boolean) in the payload of `turn.start`, `origin: "provider"` on
`turn.started`, and `awaitingProviderTurn: true` on `turn.completed`.
`command-v3` and `event-v3` refer to it.
- Sidecar protocol (`provider-schemas/acpx-sidecar.schema.json`): the
command `session.provider_turn` with `{ turnId, armed }`, the event
`runtime.turn_started` with `{ origin: "provider" }`, and the payload of
the diagnostic for a lost provider. The schema enforces that payload for
a `runtime.diagnostic` with the code `ACPX_PROVIDER_PROCESS_LOST`.
Diagnostics with other codes keep their open payload.
- `paperclip-runnerd` decodes `runtime.turn_started` and refuses it. No
sidecar sends it yet. The Runner of today refuses an unknown event too.
- No controller, Runner or sidecar uses the new fields in this pull
request.

**Reason and benefit**

The repository rule for Runner contracts says that new fields must be in
the schema with generated readers. A separate pull request lets a
reviewer read the contract alone.

## What Changed

- `protocol/schemas/provider-turn.schema.json` (new),
`command-v3.schema.json`, `event-v3.schema.json`,
`protocol/provider-schemas/acpx-sidecar.schema.json`,
`protocol/manifest.json`.
- `scripts/generate-acpx-sidecar-contract.mjs` and
`scripts/generate-protocol-schema-module.mjs`.
- Generated: `src/drivers/acpx/generated-sidecar-contract.ts`,
`src/protocol/generated/schema-bundle.ts`,
`src/protocol/generated/standalone-validators.ts`,
`runner/crates/runner-core/src/generated_acpx_sidecar_contract.rs`.
- `runner/crates/runner-core/src/acpx_event_payload.rs` (3 lines): the
Rust `match` on the event type must name the new event to compile.
- `test/protocol-contract.test.mjs`.
- The contract digests of the pack change (`prp.acpx-sidecar`,
`prp.command-v3`, `prp.event-v3`, `rust`, and the new
`prp.provider-turn`). No stored file refers to these digests. No
provider `commandDigest` and no profile version changes.

## Verification

At the head of this branch, from `packages/paperclip-runner`:

- `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p
tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run
build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs
test/protocol-contract.test.mjs`, the server check `node
scripts/verify-runner-vendor-dependencies.mjs`, `node --test
scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt
--check` and `cargo build -p paperclip-runner-core --bins` (no new
warning): all passed. The contract tests: 32 passed.
- The sidecar suite and the tracker tests: 141 passed.

Not run: the full suite. CI supplies that coverage.

## Risks

- **The pack contract digest changes.** Older image packs are not used
again.
- **The opt-in field is accepted by the schema but has no effect yet.**
A controller that sends it before part 5 gets the behavior of today.
- **State saved by the previous release:** this pull request changes no
persisted shape.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change and the split, reviewed the diff, and ran a part of the
verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), wrote the change and the
tests and cut this slice from the full change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 00:20:45 +00:00
Nicky LeachandClaude Fable 5.2 0aaab644fe refactor(runner): remove the direct assigned-gateway path (#21311)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent reaches the tools of its app connections through an MCP
gateway.
> - Before #21274, some native harnesses got the gateway URL and a
bearer token and connected to the gateway directly. The Runner has code
for that: an HTTP MCP server for ACPX providers, a server entry in the
Codex `config.toml`, a server entry in the OpenCode config, and the
lists that pass the three `PAPERCLIP_NATIVE_MCP_*` variables to a
sidecar.
> - #21274 gives each native harness its gateway tools through the PRP
relay. Core removes the three variables from each Runner environment.
The direct code path gets no binding.
> - Code that cannot run still needs maintenance, and it can still put a
token in a provider process if a variable arrives by another way.
> - This pull request deletes the direct path from the Runner.
> - The benefit is one path for assigned gateway tools, and no Runner
code that can give a gateway token to a provider.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is in a stack. Its base branch is
`fix/keep-quiet-claude-provider-between-turns` (#21293). It depends on
#21274, which is the bottom of the stack. On `master` without #21274,
the direct path is still in use.

**What existing behavior does this improve?**

The Runner code that connects a provider directly to the assigned MCP
gateway.

**Subsystem affected**

Paperclip Runner: the TypeScript drivers and the Rust `runner-core`
environment lists.

**Current behavior**

After #21274, Core never gives `PAPERCLIP_NATIVE_MCP_NAME`,
`PAPERCLIP_NATIVE_MCP_URL` or `PAPERCLIP_NATIVE_MCP_TOKEN` to a Runner.
The Runner still has the code that reads them and configures a direct
gateway server for ACPX providers, Codex and OpenCode.

**Proposed behavior**

The Runner has no code that reads the three variables to configure a
provider. If the variables are in the environment of a Runner, no
provider environment, sidecar, Codex `config.toml` or OpenCode config
gets them.

**Reason and benefit**

Less code, one path, and no way for a gateway token to reach a provider
process through the Runner.

## What Changed

- Deleted `packages/paperclip-runner/src/drivers/native-mcp.ts`
(`nativeMcpLaunchBinding`).
- `drivers/acpx/runtime-host.ts`: removed the native gateway server
entry and the name conflict check. The admission gate keeps its message.
A run with a binding id is admitted when the sidecar uses the PRP
transport with its tool bridge, as the production sidecar always does.
- `drivers/acpx/environment.ts`: the name and the URL are not in the
provider allowlist.
- `drivers/runtime-context-materializer.ts` and
`live/runnerd-codex-transport.ts`: the Codex home functions have no
`nativeMcp` parameter. The sidecar token pass-through is gone.
- `drivers/opencode/opencode-server-driver.ts`: removed the read of the
binding, its trace redaction value, and the gateway server entry in the
OpenCode config.
- `drivers/stdio-mcp.ts` and `runner-core/src/stdio_mcp.rs`: removed the
reservation of the gateway server name. No gateway server exists, so no
name can conflict. The names `paperclip` and `paperclip_runner_*` stay
reserved.
- `runner-core/src/acpx_sidecar_transport.rs` and `codex_provider.rs`:
the three variables are not in the pass-through lists.
- `control-plane/durable-prp-control-plane.ts`: the three variables are
not in the platform environment list.
- `packages/paperclip-runner/README.md`: the text about direct callers
and assigned servers.
- Kept: the three names stay in the reserved lists
(`src/configured-environment.ts`,
`runner-core/src/configured_environment.rs`). A user cannot configure
them. Tests in both languages now check all three names.
- No server code changed. The relay needs the gateway token and URL on
the control plane.

23 files, 73 lines added and 211 lines removed.

## State saved by the previous release

- This pull request changes no persisted shape. The provider server
lists, the launch profile digests, the protocol versions and the sidecar
contract are the same as with #21274.
- A Runner or sidecar that the previous release started keeps its own
code until its run ends, as #21274 describes.
- A kept Codex home that has the old gateway server in `config.toml` is
written again at the next start and at a recovery of the same run. A
test covers this.
- If the old variables reach a new Runner, no provider gets them. Tests
cover this for ACPX, Codex and OpenCode.

## Verification

From `packages/paperclip-runner`:

- `npx vitest run src/drivers/acpx/runtime-host.test.ts
src/live/runnerd-codex-transport.test.ts
src/drivers/acpx/environment.test.ts
src/drivers/opencode/opencode-server-driver.test.ts
src/live/runnerd-codex-transport-warm-attachment.test.ts
src/drivers/runtime-context-materializer.test.ts
src/drivers/agent-observability.test.ts src/drivers/stdio-mcp.test.ts
src/control-plane/durable-prp-control-plane.test.ts
src/configured-environment.test.ts`: `10 passed (10)` files, `807
passed` tests.
- `npx tsc --noEmit -p tsconfig.json` and `-p tsconfig.test.json`: no
errors.
- `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`: no
changes.
- `cargo test -p paperclip-runner-core --test acpx_sidecar_transport`:
20 passed. `cargo test -p paperclip-runner-core --lib`: 639 passed, 4
ignored.

Tests that failed with the old code and pass now:

- `runtime-host.test.ts`: "admits a relayed … run with a binding id and
ignores a previous-release gateway binding", for Claude, Codex and
Cursor.
- `runnerd-codex-transport.test.ts`: "drops a previous-release gateway
binding from the … runner environment", for ACPX Claude, Codex, Pi,
Cursor, Copilot and Kimi, and for OpenCode.
- `opencode-server-driver.test.ts`: the OpenCode config has only the
task bridge.
- `runnerd-codex-transport-warm-attachment.test.ts`: a restored Codex
home has no gateway server and no gateway credential in `config.toml`.
- `environment.test.ts`: a provider environment has none of the three
variables.
- Rust
`previous_release_gateway_binding_and_unrelated_secrets_never_reach_the_sidecar`:
a Runner that starts with the old variables gives none of them to the
sidecar, for five agents.

Mutation check: with the three names put back in the Rust pass-through
list, the Rust test fails.

Not run: a live run with a real provider. Not run: `pnpm verify` and the
full suite. CI supplies that coverage.

## Risks

- **This pull request is safe only after #21274.** The stack order keeps
that.
- **A developer who sets the three variables by hand for
`eval-session`** no longer gets direct gateway tools. No document
describes that use.
- **The warm attachment test uses the staged Runner binary from
preflight.** That binary does not have the Rust changes of this pull
request. The Codex path of that test does not use the changed lists.
- The evidence that no entry point supplies a binding comes from a read
of the code: Core removes the variables before it builds each Runner
environment, and a native run cannot use the in-process path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change, reviewed the diff, and ran the verification and a mutation check
again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), traced each code path and
wrote the change and the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 13:33:47 -08:00
Nicky LeachandClaude Fable 5.2 0ebe6d846d perf(runner): keep a quiet Claude provider between turns of a run (#21293)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner runs Claude Code through the ACPX sidecar. The
Runner stops and starts the sidecar and Claude at each turn boundary.
> - The reason was that a late tool call from an earlier turn could run
in a later turn. #21278 removes that reason for Claude: the sidecar
refuses a call that the active turn did not announce.
> - A second reason remains. Claude Code works by itself between
prompts. When a background task finishes, Claude starts a cycle of its
own. If the process stayed alive with a live background task, that cycle
could start between turns or mix with the next turn.
> - So the Runner can keep the process only when it knows that Claude
has no background work.
> - This pull request adds that knowledge and uses it. After a Claude
turn that completed with no background work, the Runner keeps the
process for the next turn of the same run.
> - The benefit is that a later turn of the same run starts with no
provider restart. This is also the base for the next step, which lets a
run continue when background work finishes.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

This pull request is stacked on #21278. Its base branch is
`fix/bind-claude-tool-calls-to-turn`. Refs #19912 (design), #12419
(restart at turn start), #20145 (restart at park) and #20190 (restart at
turn end).

**What existing behavior does this improve?**

The provider restart between turns of the same run, for Claude through
ACPX.

**Subsystem affected**

Paperclip Runner: the ACPX sidecar (TypeScript) and the ACPX provider
session in `runner-core` (Rust). Claude only.

**Current behavior**

- When a completed turn settles, the Runner restarts the sidecar and
Claude (`restart_session_after_turn`, #20190).
- If that restart did not run, the next turn restarts them at its start
(`start_turn`).
- The Runner does not know if Claude has background tasks.

**Proposed behavior**

- The sidecar asks `claude-agent-acp` for its async task reports. This
is an optional capability of the pinned `claude-agent-acp` 0.73.0. The
patch does not change.
- A tracker reads those reports and the id of each background Bash
command. It reports `{ live: 0 }` only when it knows that this process
never had a background task and that Claude did not act between prompts.
In all other cases the state is unknown.
- The sidecar adds `backgroundTasks` to the terminal event of a
completed Claude turn, and to `session.snapshot`.
- The Runner keeps the process between two turns only when all of these
are true:
  - the agent is Claude;
  - the turn completed (not failed, cancelled or interrupted);
  - the terminal event reported `backgroundTasks: { "live": 0 }`;
  - each existing condition of the turn-end restart is true;
- before the next turn, a new `session.snapshot` reports the same
identity and run, no active turn, no pending work, and `{ "live": 0 }`;
  - the settled-turn ledger and the tool replay ledgers are not full.
- In each other case the Runner restarts as before.
- A kept process keeps its tool call-id tombstones and its settled turn
identities.

**Reason and benefit**

- A turn that follows a quiet turn in the same run starts without a
restart.
- The decision is safe by default: unknown state means a restart.

## What Changed

TypeScript (`packages/paperclip-runner/src`):

- `drivers/acpx/claude-background-tasks.ts` (new):
`ClaudeBackgroundTaskTracker`. It reads raw ACP messages. It reports
unknown when the capability is not negotiated, when a task finished,
when a live task is not a shell command, when an update is malformed,
when a limit is reached, when Claude acts with no client request in
flight, or when Claude calls `CronCreate` or `ScheduleWakeup`.
- `drivers/acpx/provider-failure.ts` and `acpx-runtime-adapter.ts`:
Claude sessions also advertise the `asyncTasks` capability. The raw ACP
observer gives each message to the tracker. ACPX 0.22 drops the
`async_task_*` updates before the turn event stream, so only the raw
observer sees them.
- `drivers/acpx/runtime-host.ts`: `backgroundTaskState()`.
- `cli/acpx-runtime-sidecar.ts`: the terminal event of a completed
Claude turn carries `backgroundTasks` in the same payload as
`goalAtTerminal`, with the same size limit. `session.snapshot` carries
it when no turn is active.
- `drivers/acpx/claude-tool-use-admission.ts`:
`hasRoomForAnotherTurn()`. When the admitted-id ledger cannot hold one
more full turn, the sidecar reports unknown, so the Runner restarts
before the ledger is full.

Rust (`packages/paperclip-runner/runner/crates/runner-core`):

- `acpx_provider_session.rs`: the session records a quiet turn from the
terminal event. `start_turn` continues in the same process only under
the conditions above, and then uses `prepare_turn`, which keeps the
tombstones.
- `acpx_provider_generation.rs`: `can_continue_after_turn`,
`continue_after_quiet_turn`, and the snapshot check
`background_work_is_quiet`.
- `acpx_park_preparation.rs`: the turn-end restart does not run for a
quiet Claude turn.
- `acpx_provider_backend.rs`: `session.reconciled` reports a replaced
process only when the turn start replaced it.
- `bin/fake-acpx-sidecar.rs`: new option `--background-tasks` for tests.
- `runner/DURABLE_TRANSPORT.md`: the rule for a kept sidecar.

Contract (`packages/paperclip-runner/protocol`):

- The `backgroundTasks` report is `$defs.backgroundTasks` in
`provider-schemas/acpx-sidecar.schema.json`: an object with only `live`,
a non-negative integer. The schema validates it on
`runtime.turn_terminal`.
- `scripts/generate-acpx-sidecar-contract.mjs` generates the TypeScript
type and reader and the Rust reader from it. The tracker, the sidecar
and the Rust session use the generated code.
- `fixtures/provider-boundary/acpx-background-tasks.json` has shared
test documents. The TypeScript reader, the Rust reader and the schema
are tested with the same accepted and refused shapes.
- This changes the contract digest of the provider pack
(`prp.acpx-sidecar` and `rust`). Older image packs are not reused. No
provider `commandDigest` and no profile version changes.

Not changed: park and run attachment. A process that ran a turn is still
restarted at park, and only a process that ran no turn is bound to the
next run. Other ACPX agents and Codex do not change. The
`claude-agent-acp` patch does not change.

## State saved by the previous release

- This pull request adds no persisted state. The three new fields of the
session are in memory only.
- A terminal event or a snapshot without `backgroundTasks` means
unknown. An older sidecar therefore gets the restart, as before.
- An older Runner with a new pack ignores the field and restarts at each
turn, as before.
- After a Runner restart, the in-memory state is gone. The next turn
restarts the provider, as before. A test covers this.

## Verification

From `packages/paperclip-runner`:

- `npx vitest run src/drivers/acpx/claude-background-tasks.test.ts
src/drivers/acpx/acpx-runtime-adapter.claude-background-tasks.test.ts
src/drivers/acpx/acpx-runtime-adapter.claude-tool-use.test.ts
src/drivers/acpx/claude-tool-use-admission.test.ts
src/cli/acpx-runtime-sidecar.test.ts
src/drivers/acpx/acpx-runtime-adapter.test.ts
src/drivers/acpx/runtime-host.test.ts`: `7 passed (7)` files, `363
passed` tests.
- `npx tsc --noEmit -p tsconfig.json` and `npx tsc --noEmit -p
tsconfig.test.json`: no errors.
- `pnpm run typecheck:typescript` (it includes the contract checks): no
errors. `node --test test/acpx-sidecar-contract.test.mjs`: 17 passed.
- `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`: no
changes.
- `cargo test -p paperclip-runner-core --test
acpx_quiet_turn_continuation --test acpx_park_preparation --test
acpx_park_readiness --test acpx_provider_turns --test
acpx_provider_session --test acpx_warm_attachment --test
native_provider_backend`: 4, 33, 8, 31, 12, 6 and 47 passed. No test
failed.
- `cargo test -p paperclip-runner-core --lib`: 637 passed, 4 ignored.
- Mutation check: when the Rust code accepts each live count as quiet,
two tests in `acpx_park_preparation` fail.

New tests:

- **Real ACP runtime**
(`acpx-runtime-adapter.claude-background-tasks.test.ts`), with the
pinned `claude-agent-acp`, the pinned Claude Code client and a local
fake Anthropic API. A foreground Bash turn reports `{ live: 0 }`. A
background Bash in the same process reports `{ live: 1 }` at the
terminal event, and unknown after Claude ran its own cycle for the
finished task. Each turn has its own usage receipt. One process serves
all turns.
- **Tracker unit tests** (13): spawn, completion, failure, stop, pause,
a task that is not a shell, a subagent, no negotiation, malformed
updates, activity between prompts, scheduling tools, session load
replay, limits.
- **Sidecar tests**: the terminal event carries the state for Claude and
not for Codex. A full ledger reports unknown.
- **Rust** `tests/acpx_quiet_turn_continuation.rs` (4) and
`tests/support/acpx_quiet_turn_restart.rs` (7):
- a quiet Claude turn continues in the same process, and a replay of a
settled call id is refused;
  - live, unknown and malformed reports restart;
- background work that appears after the terminal event restarts at turn
start;
  - failed, cancelled and interrupted turns restart;
  - Codex with a quiet report restarts;
  - an active goal restarts;
  - 4,096 kept turns, then the next turn restarts;
- park after a kept process still prepares a new process, and the next
run attaches to it;
  - a Runner restart after a kept process resumes in a new process.
- The existing restart tests did not change. The fake sidecar sends no
report by default, so they are the tests for the unknown case.

Live run with a real Claude account, not in CI: the real ACP runtime
(ACPX 0.22, the patched `claude-agent-acp`, the pinned Claude Code
client), a real tool bridge, and the real Anthropic API with model
`claude-sonnet-5-5`, on the code at the top of this stack. A turn that
started one background shell reported `{ live: 1 }`. A turn that started
two reported `{ live: 2 }`, then `{ live: 1 }` and `{ live: 0 }` as each
shell finished. A turn in which Claude waited for its background shell
in the foreground reported the unknown state.

Not run: server integration tests (the fake sidecar there sends no
report, so their process counts do not change), Linux, and a live run
through the real sidecar process and runnerd.

## Risks

- **All Claude sessions now negotiate the `asyncTasks` capability.**
`claude-agent-acp` then sends `async_task_*` updates and adds a marker
to a background Bash update. ACPX drops the updates. No other behavior
of `claude-agent-acp` changes, from a read of its code.
- **Park after a kept process pays the restart again**, as it did before
#20190. With real Claude that restart was about 1.5 s.
- **A small window remains** between the snapshot and the prompt of the
next turn. Claude can start a cycle of its own in that window only
without a background task, because a process that had a background task
is not kept.
- **Activity between prompts is found only when no client request is in
flight.** A request that never gets a response would hide such activity.
- **Background watchers cannot be told apart from other tasks**, so each
task that is not a shell command makes the state unknown.
- **Each kept turn adds one `session.snapshot` request.**
- A multi-turn run with real Claude now uses one provider process. This
path has tests with the fake sidecar and with the real ACP runtime. It
has no test with the real sidecar host.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change, reviewed the diff, and ran the TypeScript and Rust verification
again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), investigated the signals
and wrote the change and the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 12:54:43 -08:00
7773b52f05 fix(codex): preserve sandbox certificate trust in Codex shell commands (#20364)
<!-- Written in ASD-STE100 style: short sentences, active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner and the Codex adapters run agent shell commands
inside a sandboxed Codex session with a strict environment allowlist
> - The controller builds `shell_environment_policy.include_only` before
the sandbox injects certificate-trust variables such as `SSL_CERT_FILE`
and `GIT_SSL_CAINFO`
> - The allowlist drops the injected variables, so managed `git` and
`gh` commands cannot verify HTTPS against a runtime-provided CA
> - This pull request allows the certificate-trust variable names
through every environment policy stage, from the Rust process supervisor
to the shell allowlist
> - The benefit is that managed `git` and `gh` work again in sandboxes
that require a runtime CA, while provider-credential isolation and the
bounded allowlist stay in place

## Linked Issues or Issue Description

- Fixes: #20019
- Refs #20038 — community pull request for the same defect. This change
reviews the approach independently. The commit credits the community
author as co-author.

## What Changed

- Add `CERTIFICATE_ENVIRONMENT_KEYS` (`SSL_CERT_FILE`, `SSL_CERT_DIR`,
`GIT_SSL_CAINFO`, `NODE_EXTRA_CA_CERTS`, `DENO_CERT`,
`REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`) to
`packages/paperclip-runner/src/configured-environment.ts`
- Native Codex shell policy (`codex-security-config.ts`): always put the
certificate names in `include_only`; set inheritance to `"all"` with
`ignore_default_excludes=true`, bounded by the explicit allowlist
- Pass the certificate names through the app-server environment filter,
the ACPX spawn environment filter, the ACPX runtime sandbox
`include_only`, the runner platform environment keys, and the Rust
process supervisor base key list
- Legacy and ACPX-engine Codex shell allowlists
(`packages/adapter-utils/src/codex-shell-environment.ts`): always
include the certificate names
- Add a synthetic private-CA HTTPS journey test: a real `git ls-remote`
fails TLS verification without the injected CA and succeeds with it,
through the policy-filtered child environment
- Document the contract in `doc/sandbox-shell-environment.md` and link
it from `doc/DEVELOPING.md`

## Verification

- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/drivers/codex/codex-security-config.test.ts
src/configured-environment.test.ts
src/drivers/codex/codex-shell-certificate-trust.test.ts
src/drivers/codex/codex-app-server-driver.context-control.test.ts
src/drivers/acpx/runtime-sandbox.test.ts` — 104 tests pass
- `pnpm --filter @paperclipai/adapter-utils exec vitest run
src/codex-shell-environment.test.ts src/acpx-engine/execute.test.ts` —
270 tests pass
- `pnpm --filter @paperclipai/adapter-codex-local exec vitest run
src/server/execute.test.ts` — 10 tests pass
- Codex driver lifecycle, conformance, recovery, and boundaries suites
pass (103 tests); environment-builder suites pass (307 tests)
- Red proof: with only the source fix reverted, the new regressions
fail. The journey test shows `GIT_SSL_CAINFO` is undefined in the shell
environment, and `git` fails TLS verification against the private-CA
fixture
- `cargo check` passes for `paperclip-runner-core`; `rustfmt --check`
passes for the edited Rust file
- The journey test needs `git` and `openssl`; it skips when either tool
is absent

## Risks

- The native shell policy now always sets
`shell_environment_policy.inherit="all"` with
`ignore_default_excludes=true`. Before, it set `"none"` when no
credentials or task keys were projected. The explicit `include_only`
allowlist still bounds every inherited name, so no new secret names can
cross
- The allowlist grows by seven non-secret certificate-trust names. Their
values point at CA files. The sandbox filesystem policy still controls
access to those files; this change grants no filesystem access
- No persisted run, session, or launch identity changes shape. Sessions
saved by the previous release resume unchanged; the new policy applies
when the next launch builds its arguments
- Provider credentials keep their existing exclusions at every stage

## Model Used

- Claude Fable 5 (Anthropic, model ID `claude-fable-5`), extended
thinking, agentic tool use, run through Claude Code. The implementation
was written and reviewed independently against the community design in
#20038; the community author is credited as co-author in the commit.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (pending: checks run after this
PR opens)
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(pending review)
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: everlastingcrown <everlastingcrown@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-11-13 14:13:01 -06:00
DottaandClaude Opus 5.5 43aabb7080 fix(runner): keep configured secret values out of OpenCode output, storage and model requests (#21330)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can configure a secret that Paperclip delivers to an
agent's commands as an environment variable
> - The value must stay out of what the agent runtime stores and what it
sends to the model
> - With the OpenCode runtime, a command that printed the variable put
the value in OpenCode's tool output. OpenCode stored that output in its
session database and sent it to the model. Paperclip's own run log
already redacted it
> - Paperclip did not tell the runner which configured values are
secrets, so the runner could not protect them without also hiding
ordinary settings
> - This pull request sends secret names (never values) to the runner,
filters exact secret values before OpenCode captures any tool output,
redacts them from model requests, and tells the agent how to check a
secret without printing it
> - The benefit is that a configured secret stays out of model context
and out of runtime storage, even when an agent prints it

## Linked Issues or Issue Description

**What happened?**
The `chat-features.runner-opencode.local.secrets.personal-secret` eval
failed because the persisted-state scan found a configured secret value
in OpenCode's session database. The person entered the value through the
normal secret entry form. Nobody pasted it in chat. The agent ran a
shell command that printed the environment variable it was given.
Paperclip's run log redacted the value. OpenCode stored the raw output
in its database and included it in the next model request.

**Expected behavior**
A configured secret must never reach model context or runtime
persistence. Ordinary configured values must not be redacted.

**Steps to reproduce**
Configure a secret environment variable for an OpenCode agent. Have the
agent run a command that prints the variable. Then search the OpenCode
session data and the model requests for the value.

## What Changed

- **Secret names cross to the runner.**
`configuredEnvironmentProjection` takes the controller's secret key
names and adds `PAPERCLIP_SECRET_ENV_KEYS`. This marker holds only the
names of configured entries that came from secrets. It travels with the
configured projection across the TypeScript and Rust runner boundaries
(`configured-environment.ts`, `configured_environment.rs`). The server
passes `secretKeys` in `runtime-execution.ts`.
- **Command output is filtered before OpenCode reads it.** For a launch
that has configured secrets, the runner sets OpenCode's `shell` to a
small POSIX wrapper (`drivers/opencode/output-custody.ts`). The wrapper
runs the shell OpenCode would use anyway. It sends stdout and stderr
through an `awk` filter that replaces exact secret values. OpenCode
never receives the raw bytes, so its streamed previews, its database and
its model requests stay clean. The wrapper keeps the exit status. It
does not wait for background processes. The values come from the
environment that OpenCode already holds and are never written to disk.
- **Other tool results are rewritten before they are saved.** A local
OpenCode plugin rewrites completed non-shell results (file reads,
searches, MCP results) in `tool.execute.after`. OpenCode runs that hook
before it stores or returns the result. The wrapper refuses to run any
command until the plugin has started for the current launch (per-launch
nonce). So no command can write a secret to a file that a later read
returns without filtering.
- **Model requests are redacted as a second layer.** The OpenCode
provider proxy removes exact secret values from the string values of
each parsed model request. Field names and other literals are never
changed. A quick byte scan for each value, raw and JSON-escaped, sends
the original bytes unchanged when no value is present.
- **The agent gets guidance.** Native task prompts list the configured
secret names. They say to check a secret with `[ -n "$VAR" ]` and never
to print, echo, log or write a secret value.
- **No added startup delay.** The wrapper is content-addressed under the
runtime directory and reused across launches, so it is not rewritten on
every launch. OpenCode's plugin-dependency install is marked as already
done, so loading the plugin does not add wait time to the first tool
call.
- **Only exact values are redacted.** Command output is filtered line by
line, so each line of a multi-line value that has at least 4 characters
is redacted too. If a configured secret has no line of 4 or more
characters, the launch is refused with
`opencode_output_custody_unavailable` instead of running unprotected.

## Verification

-
`packages/paperclip-runner/src/drivers/opencode/output-custody.test.ts`
(10 tests): the wrapper filters stdout, stderr and a partial last line;
keeps exit codes (7, and 143 for SIGTERM); filters each line of a
multi-line value; refuses to run before plugin readiness or with an old
nonce; returns at once when a background process keeps running; leaves
ordinary values unchanged. The plugin rewrites nested results.
Model-request redaction covers raw and JSON-escaped values and never
changes field names or literals (a secret equal to `true` leaves
`"stream": true` intact). Secrets the filter cannot match are refused.
No value is written to disk.
- `opencode-server-driver.test.ts`: one saved session launches three
times. First with no secret marker (older controller), then with the
marker, then with no marker again (rollback). The first launch is
unchanged, the second applies custody, and the third launch still
starts. This test fails without the driver change.
- `configured-environment.test.ts`: the marker names only secret
entries, crosses the runner spawn and OpenCode boundaries, reads as "no
secrets" when absent, and is refused when malformed. Rust:
`secret_name_marker_crosses_only_with_its_projection`.
- `server/src/__tests__/configured-secret-guidance.test.ts`: guidance
content, and no guidance when there are no secrets.
- Real OpenCode 1.18.34 qualification (`PAPERCLIP_OPENCODE_QUALIFY=1
node --test scripts/qualify-opencode-runtime.test.mjs`). A fake model
asks for a command that prints the secret and writes it to a file, then
a read of that file. Unprotected baseline: the value is in the session
database and in model requests (red). With custody: it is in neither,
and the exit status is kept (green).
- **Latency** (local, macOS, real OpenCode, 8 runs each): boot 629 ms vs
697 ms baseline; first model request after the prompt 678 ms vs 698 ms
(no regression); first command 139 ms vs 191 ms; each later command +9
ms (36 vs 27 ms). Proxy cost per request when no secret is present (the
normal case): 33 µs at 50 KiB, 164 µs at 300 KiB, 0.55 ms at 1 MiB, 2.3
ms at 4 MiB. When a secret is present: 91 µs, 0.48 ms, 1.6 ms and 6.8
ms. The scan runs only for launches that have configured secrets, and it
adds nothing to OpenCode's own startup.
- A narrow paid run of
`chat-features.runner-opencode.local.secrets.personal-secret` on this
branch is linked in a comment.

## Risks

- **Previous-release story.** The marker is new and optional. A runner
that receives no marker (older controller) launches exactly as before.
An older runner drops the marker at its environment boundary. The change
saves no new run or session state. OpenCode configuration is regenerated
on every launch, so sessions saved before this change resume under
custody (covered by the cutover test). Warm-process reuse keys include
the marker, so a warm process from before the deploy is not reused for a
run that has secrets.
- **Commands go through the wrapper.** Output arrives line by line.
Output written after the shell exits is dropped, which matches how
OpenCode stops reading at exit. If `awk`, `mkfifo` or `rm` is missing,
the launch is refused with `opencode_output_custody_unavailable` and
does not run unprotected.
- **Scope.** Only exact values are redacted. An agent that encodes a
value (for example base64) is out of scope. The model-request layer
applies when Paperclip routes the model through its proxy (OpenRouter
and Paperclip connections).

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 03:23:56 -06:00
DottaandClaude Opus 5.5 d9a4bb23fe fix(runner): wait for a slow Codex turn start while it reports progress (#21329)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Codex agents run through the Paperclip runner, and the server-side
transport starts each provider turn
> - After Codex accepts a turn, the transport waits a fixed 30 s for
that turn's `turn/started` before it reports the turn identity
> - Codex starts its MCP servers before it starts the turn, so a healthy
cold start in a loaded sandbox often takes longer than 30 s
> - The transport then failed a live turn with `runnerd did not report
the provider turn identity`, and the abandoned turn's late notifications
broke the reattachment (`did not name its execution owner`,
`runner_harness_state_mismatch: resumed_not_suspended`)
> - This pull request sizes the start window to the measured cold start
and extends it while the provider keeps reporting startup progress
> - The benefit is that slow but healthy Codex starts succeed on the
first attempt, and a dead start still fails at a bound

## Linked Issues or Issue Description

**What happened?**

On staging, Codex runs on Daytona failed at turn start with
`native_session_interrupted: runnerd did not report the provider turn
identity`. The run log shows the same sequence each time. The runner
accepts the turn. Codex then reports startup progress (a bubblewrap
config warning, `account/updated`, and `mcpServer/startupStatus/updated`
failures for an MCP server that is not logged in) for about 30 s. The
transport gives up 30 s after acceptance. A few seconds later, Codex
sends the turn's `turn/started`, which now arrives at a closed turn
(`protocol_failure:turn_binding_mismatch`). The same-run reattachment
then hits that orphaned turn. It fails with `Provider authoritative
notification item/started did not name its execution owner.` or
`runner_harness_state_mismatch: resumed_not_suspended`.

Of 198 recent runs in the staging company, 52 contain this turn identity
failure. In 68 successful starts, submit to `turn/started` took p50 25
s, p75 37 s, p90 50 s, p95 54 s and max 76 s. The fixed 30 s wait sits
near the median.

**Expected behavior**

A slow but healthy Codex start succeeds. A start that makes no progress,
whose Runner exits, or that the runner fails still fails at a bound and
recovers through the normal same-run resume.

**Steps to reproduce**

1. Start a Codex turn whose provider accepts the turn but sends
`turn/started` after the 30 s default window (on staging: several MCP
servers and a loaded sandbox).
2. Let the provider report startup notices during the wait, and let the
runner not prove the accepted turn live in time (on staging no turn was
adopted).
3. See `runnerd did not report the provider turn identity`, then the
reattachment conflict.

The new tests reproduce this with the fake Codex app server
(`--turn-started-delay-ms` with `--turn-started-progress-ms`) and a
runner snapshot that cannot prove liveness.

**Paperclip version or commit**

Staging Core `c730872974` (`2026.1106.0+2253`). Base of this PR:
`origin/master`.

**Deployment mode**

Authenticated, public, managed cloud (staging). Remote runner on
Daytona.

**Agent adapter(s) involved**

`paperclip_runner` with the Codex provider (`gpt-6.1-sol`).

## What Changed

- `runnerd-codex-transport.ts`: the default wait for an accepted turn's
`turn/started` is now 90 s (measured max 76 s), not 30 s. An explicit
`turnStartTimeoutMs` keeps its old bound.
- The wait restarts its window each time the runner commits a provider
notice, or a runner diagnostic that names the accepted turn. The total
wait stops at 180 s (option `turnStartProgressCeilingMs`, so tests can
lower it). A provider relaunch after the accepted `turn.start` does not
extend it.
- The adoption check (runner snapshot proves the accepted turn live) now
runs before each window end, not only before the first one. A Runner
exit and a runner failure of the pending turn still fail at once.
- `fake-codex-app-server`: new `--turn-started-progress-ms` flag sends
`mcpServer/startupStatus/updated` failures while it delays
`turn/started`.
- `doc/execution-semantics.md`: documents the start window, the progress
rule and the staging measurements.

## Verification

- Red on `origin/master` transport (with the new fake flag): `keeps
waiting for an accepted Codex turn while its startup keeps reporting
progress` failed after 4.6 s, and `gives a silent Codex cold start the
measured default start window` failed after 30.9 s. Both failed with
`runnerd did not report the provider turn identity`.
- Green: `npx vitest run
src/live/runnerd-codex-transport-deadlines.test.ts` passes 9/9. This
includes a ceiling test: provider notices every 1 s and no
`turn/started` fail at an 8 s test ceiling, and the test goes red with
the ceiling cap removed. It also includes the existing adoption,
no-adoption-within-bound and runner-failure tests, and a new test that a
silent start still fails within its bound.
- `npx vitest run` on the adoption, cut-turn-start, stall,
turn-start-timeout-retention, remote-retention, restart-continuation and
controller-outage transport suites passes 48/48.
- `npx vitest run src/live/runnerd-codex-transport.test.ts
src/live/live-session.test.ts src/live/turn-stream.test.ts`: 391/392
pass. The one failure (`stops a lost runnerd's provider and escaped tool
job from its lineage journal (journal=true)`, 1 s, process-lineage
timing under full-suite load) passes when run alone, both on this branch
and on `origin/master`.
- `tsc --noEmit` for `packages/paperclip-runner` and `cargo fmt --check`
pass.

## Risks

- A silent dead start (no Runner exit, no provider output, no runner
failure) now fails after 90 s, not 30 s. The Runner exit and
runner-failure paths still fail at once, so this applies only to a
provider that stops without a trace.
- A provider that keeps sending notices but never starts the turn waits
at most 180 s.
- No schema, API or UI change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, browser reads of staging run data).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Refs #21272, #21120

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 02:21:37 -06:00
Nicky LeachandClaude Fable 5.2 314b893678 fix(native-runtime): deliver assigned gateway tools to every native harness through the PRP relay (#21274)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent can have app connections. Paperclip gives the agent the
tools of those apps through an MCP gateway, with its approval and audit
rules.
> - Native harnesses reach the gateway in two ways today. In the relay,
the server keeps the gateway token and offers the tools over the
authenticated Runner channel. In the direct way, the server gives the
gateway URL and a bearer token to the harness.
> - Which way a harness uses comes from a list of special cases that
grew one harness at a time. Local Codex and all ACPX agents use the
direct way. The others use the relay.
> - The direct way puts a run-scoped token in a provider process. A
process that the agent starts can read it. A provider process that
serves more than one run can hold a revoked token.
> - This pull request replaces the list with one rule. Each native
harness gets its assigned gateway tools through the relay.
> - The benefit is one behavior for all native harnesses, and no gateway
token in a provider process.

## Linked Issues or Issue Description

No public issue exists. The issue is described here.

Refs #13942 (the relay), #17149 (ACPX accepts relayed tools) and #19912
(the design for turn-scoped ACPX MCP authority).

**What existing behavior does this improve?**

How a native harness reaches the tools of its assigned MCP gateway.

**Subsystem affected**

Native runtime in the server (`native-session-executor.ts`,
`native-session-resume.ts`). One Runner test file. One Rust test.

**Current behavior**

- `relayAssignedMcp` is true for plugin tasks, Cursor Cloud, OpenCode,
Codex on a remote target, and Dot.
- For local Codex and for each ACPX agent (Claude, Codex, Cursor, Pi,
Kimi, Grok, Copilot, Antigravity), Core puts
`PAPERCLIP_NATIVE_MCP_NAME`, `PAPERCLIP_NATIVE_MCP_URL` and
`PAPERCLIP_NATIVE_MCP_TOKEN` in the Runner environment. The ACPX host
gives them to the provider as an HTTP MCP server. The local Codex home
gets them in `config.toml`.
- The token is run-scoped. Core revokes it when the run ends.

**Proposed behavior**

- A native run that has an assigned gateway uses the relay. The
condition names no provider.
- Core builds the gateway tools on the control plane, offers them with
the other Runner tools, dispatches each call over PRP, and authorizes
each call again through the gateway.
- Core removes the three `PAPERCLIP_NATIVE_MCP_*` variables from each
Runner environment.
- The tool contract fingerprint has the relay marker for each target,
not only for remote targets.

**Reason and benefit**

- All native harnesses have the same behavior.
- No provider process holds a gateway token.
- A sandbox does not need to reach the gateway URL of the host.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`
- The list of special cases is gone. Core validates the binding and
builds the relayed tools when a binding is present.
- Core does not list the gateway for a review run, Grok Bot or Muse. The
tool authority never offered app tools to those. A gateway error can
therefore not fail those runs.
- OpenCode keeps its `on_demand` catalog. That option depends on the
provider, not on the relay.
  - Core removes the three variables from each Runner environment.
- `server/src/services/native-runtime/native-session-resume.ts`: the
relay marker in the tool contract fingerprint is unconditional.
- `doc/execution-semantics.md`: the text about the tool contract
fingerprint now states the rule, the tool names and the cutover.
- App tool schemas. New module
`packages/paperclip-runner/src/drivers/tool-input-schema.ts` compiles
each schema by itself, in its declared dialect (draft-07, 2019-09 or
2020-12), with no shared `$id` registry. Core uses it when it builds the
relayed tools (`assigned-mcp-tools.ts`). Core does not offer a tool
whose schema has an unsupported dialect or does not compile, and writes
a log entry with the tool name and the reason. The run does not fail.
The ACPX tool bridge and the OpenCode MCP bridge use the same function.
The tools of Paperclip itself stay strict.
- Large arguments. The ACPX sidecar sends the arguments of a tool call
to runnerd with no display limit. The limit is the tool-call admission
limit of runnerd, 256 KiB. A larger call fails with a tool error that
gives the limit. The sidecar does not dispatch it.
- Tests in `native-session-executor.test.ts`,
`native-session-resume.test.ts`, `runtime-context-materializer.test.ts`
and `acpx_provider_backend.rs`.

Not changed in this pull request: the direct code path. It is still in
the Runner, but it receives no binding. #21311 deletes it.

## Behavior that each harness sees

| Harness | Before | After |
|---|---|---|
| Codex on a remote target, OpenCode, Cursor Cloud, Dot, plugin tasks |
Relay | No change |
| Local Codex | Direct: a `paperclip-assigned` server in `config.toml` |
Relay. App tools are dynamic tools named `app_<name>_<digest>` |
| ACPX agents | Direct: a second HTTP MCP server | Relay. App tools are
on the `paperclip` bridge as `mcp__paperclip__app_<name>_<digest>` |
| ACPX with a preconfigured home | No gateway | No change |
| Grok Bot, Muse | Got the variables and ignored them | Do not get the
variables |

- A relayed tool name has a maximum of 64 characters. With the
`mcp__paperclip__` prefix it has a maximum of 80.
- Permission behavior keeps the same intent. For Claude in the
`approve-reads` and `approve-paperclip` modes, app tools ask for
approval, as before. For the other ACPX agents the permission decision
does not use the server name.
- In a planning run or an ask run, the relay offers only tools with read
risk. The direct gateway offered all tools. This is stricter.
- When the catalog is too large, each harness gets the two compact tools
`paperclip_search_assigned_tools` and `paperclip_call_assigned_tool`.
- The four helper tools for resources and prompts
(`paperclip_list_resources`, `paperclip_read_resource`,
`paperclip_list_prompts`, `paperclip_get_prompt`) are not in the relay.
No harness loses them: a run token permits only `tools/list` and
`tools/call` (`run-preparation.ts`), so these four tools already fail
with `gateway_token_action_denied` on the direct path.

## State saved by the previous release

- **Local sessions of each provider start again one time.** The tool
contract fingerprint changes for local targets. The next run sees a
different fingerprint, starts a new provider session with the full task
context in the same workspace, and keeps the old saved state. This is
the existing rotation for a tool contract change. Codex needs it,
because Codex cannot add tools to a thread that it resumes. For the
local providers that already used the relay, the restart is not
necessary. It is accepted so that the rule names no provider.
- **Remote sessions continue.** Their fingerprint does not change. For
ACPX, the attach of the next run replaces the saved tool set.
- **A kept Codex home.** Paperclip writes `config.toml` again at each
start and recovery. The old gateway server and its token are removed.
- **A deploy lands during a run, and the Runner survives.** The
controller attaches to the same Runner for the same run. The old
provider keeps its direct gateway connection. The controller detach
keeps the gateway token of that run, and Paperclip revokes it when the
run ends.
- **A deploy lands during a run, and the Runner is dead.** The Runner
starts again for the same run with its saved tool set and no direct
gateway. The assigned app tools are not available for the remainder of
that one run. The next run gets them through the relay. This applies to
each harness that moves to the relay. It has no test.
- Paperclip refuses no saved state and discards none.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-resume.test.ts`: `1508 passed
| 7 skipped`, 1 failed. The failed test is "archives a damaged prior
epoch before completing guarded same-task replacement with real runnerd"
(`ENOENT … authority-epochs`). It fails in the same way on `master` at
53608bd1d6 and with the product change removed, on macOS.
- `cd server && npx tsc --noEmit -p tsconfig.json`: no errors.
- `cd packages/paperclip-runner && npx vitest run
src/drivers/runtime-context-materializer.test.ts`: `18 passed`. `npx tsc
--noEmit -p tsconfig.test.json`: no errors.
- `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`: no
changes. `cargo test -p paperclip-runner-core --lib
acpx_provider_backend`: `50 passed`.

- `npx vitest run src/drivers/tool-input-schema.test.ts
src/drivers/runner-tool-bridge.schemas.test.ts
src/drivers/runner-tool-bridge.test.ts
src/cli/acpx-runtime-sidecar.test.ts src/drivers/opencode` in
`packages/paperclip-runner`: `21 passed (21)` files, `377 passed` tests.
`pnpm run typecheck:typescript`: no errors.
- `cargo test -p paperclip-runner-core --test acpx_event_payload --test
tool_input_schema_fixtures`: 10 and 3 passed.
- Schema tests use one shared fixture
(`tool-input-schema-fixtures.json`) for the TypeScript compiler, the
tool bridge and the Rust `begin_call`: a draft-07 schema, a second
schema with the same `$id`, 2019-09, 2020-12, draft-04, an unknown
dialect, a `$ref` with no target, and an invalid keyword. Before the fix
the bridge failed with `no schema with key or ref
"http://json-schema.org/draft-07/schema#"`.
- A tool schema whose `$id` is the URI of a meta-schema cannot damage
the shared compiler. Before this fix, `removeSchema` deleted that
meta-schema, and later schemas failed to compile for each run in the
process. A test with 7 reserved ids failed before the fix and passes
now.
- Core limits the compile work of each run start. A schema over 512 KiB
is not compiled, and one listing compiles a maximum of 768 KiB of
schemas. These are the limits of the Runner. A tool over a limit is
served only through the search and call tools. Core keeps the outcome
for each schema digest (1,024 entries) and yields to the event loop
before each compile. Four tests failed before this change.
- A sidecar test sends a tool call with 200 KiB of arguments and
receives them byte for byte, refuses a call with 300 KiB with the named
error, and keeps the replacement of a lone surrogate.

Tests, each seen to fail without the product change:

- "keeps assigned MCP credentials on the control plane for %s" now runs
for remote Codex, local Codex, OpenCode and ACPX Claude, Pi, Kimi,
Copilot and Cursor. It checks that Core builds the relay, offers the
tool, dispatches a call, and gives no `PAPERCLIP_NATIVE_MCP_*` variable
to the Runner. Without the change, the five new harness cases fail.
- "relays the assigned gateway when a deploy lands mid-run…" runs for
Claude and Pi.
- The fingerprint tests keep the old local fingerprint as a retained
value and check that the new one is different. The remote fingerprints
for Codex and OpenCode do not change.
- "drops a retained direct gateway server when the relayed launch has no
binding": a Codex home with the old `config.toml` entry loses that entry
at the next start and at a recovery of the same run.
-
`next_run_attach_offers_relayed_gateway_tools_to_a_direct_gateway_checkpoint`
runs for Claude and Pi.

Not run: a live run with a real provider and a real app connection for
each harness. Not run: `pnpm verify` and the full suite. CI supplies
that coverage.

## Risks

- **An ACPX app call has a limit of 256 KiB for its arguments.** The
direct gateway accepted a request body of 10 MB. Codex and OpenCode
permit 480 KiB. To get nearer to 10 MB, runnerd needs a larger admission
limit for tool calls or a protocol that sends arguments in parts. That
is not in this pull request.
- **A tool with a draft-04 or draft-06 schema is not offered.** Ajv 8
does not support draft-04, and the TypeScript and Rust validators must
agree. On the direct path the provider got such a tool.
- **Core compiles app schemas on its event loop.** Before this pull
request Core did not compile them. With the limits above, one valid
schema near 512 KiB can hold the loop for about 0.65 s, and one listing
for about 1.2 s in total (measured on a developer machine). An app that
changes its schemas at each listing gets no help from the cache. A
worker thread with a timeout would remove this; it is not in this pull
request.
- **Tool names change for local Codex and all ACPX agents.** A saved
agent instruction that names a gateway tool directly does not match the
new name.
- **Each local native session starts again one time after the deploy.**
The provider conversation of that session does not continue. The task
context and the workspace stay.
- **A run that is in flight at deploy, with a dead Runner, has no app
tools until it ends.**
- **Names of 80 characters are not tested live** with Kimi, Copilot,
Grok, Antigravity and Codex through ACPX. Pi already shortens a name
longer than 64 characters.
- **Managed Claude and AWS AgentCore now receive relayed app tools.**
This is not tested live.
- **A gateway token expires after 1 hour.** The relay keeps the token on
the controller for the whole run, as OpenCode and remote Codex do today.
A longer run loses its app tools on both paths. This pull request does
not change that.
- **The gateway URL must have the path `/mcp/gateways/<id>`.** With a
path prefix in the API base URL, a run with an assigned gateway fails
with `assigned native MCP gateway path is invalid`. This was true for
the relayed harnesses before. It is now true for all.
- No test covers the rule that Core does not list the gateway for Grok
Bot and Muse.

## Follow-up

The direct path is now unreachable. #21311 deletes it. That pull request
is in the same stack, above the pull requests that change the same ACPX
host files.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the
change, reviewed the diff and ran the verification again.
- Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with
tool use (shell, file edits, local test runs), investigated each harness
and wrote the change and the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 00:16:58 -08:00
Nicky LeachandClaude Opus 5.5 2483cb46c2 feat(plugins): deliver configured environment to plugin Runner tasks (#21280)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent gets environment variables from its execution environment,
its own configuration, its project, and its routine. Secrets are
resolved for each run.
> - A plugin environment can run the agent as a native Runner task on
its own host, through the environment task contract.
> - That contract has no field for the agent's variables. Runs on a
plugin environment get only the variables that the plugin sets itself.
> - This pull request adds a versioned contract field for the values,
sends them on a session handoff, lets Codex commands inherit the
variables in an external sandbox, and lets a plugin protect the
variables that it sets itself.
> - The benefit is that plain and secret variables work on plugin
environments in the same way as on local environments.

## Linked Issues or Issue Description

Refs #21233. That pull request by @lucktastic finds the same gap and
proposes the same capability and field names
(`configuredEnvironmentVersion: 1`, `configuredEnvironment`). This pull
request uses the same names, so providers implement one contract. It
differs in these places:

- Codex commands get the variables through an external-sandbox denylist,
not through key names sent at `prepare`. A retained Runner does not need
a new preparation when a key is added or removed.
- A session handoff `resume` gets the values, because it starts a new
Runner on another computer. Value changes on the same computer need no
new field: Core's process reuse key includes the values, so Core starts
a new task.
- A driver can declare `protectedEnvironmentKeys`. Paperclip rejects or
warns when one is saved.
- The run log names skipped variables only for an older provider.

Thank you to @lucktastic for the diagnosis and the contract shape.

**What happened?**
An agent has environment variables, plain and secret. Its runs on a
plugin task environment do not get them. `PAPERCLIP_CONFIGURED_ENV_KEYS`
in the run lists only the variables that the plugin sets.

**Expected behavior**
Each run gets the agent's configured variables.

**Steps to reproduce**
1. Set a plain and a secret environment variable on an agent.
2. Assign the agent to a plugin environment that provides Runner tasks.
3. Start a run. Read the names in `PAPERCLIP_CONFIGURED_ENV_KEYS` in a
command that the agent runs.
4. The agent's names are not in the list.

## What Changed

- **SDK contract** (`packages/plugins/sdk/src/environment-tasks.ts`):
- A provider advertises `configuredEnvironmentVersion: 1` on `describe`.
- `submit` gets `configuredEnvironment`. `resume` gets it only for a
session handoff.
- Names, count, and sizes have the same limits as the Runner projection.
- **Server**:
- `createRunnerdBackendWithinSessionClaim` gives the run's configured
projection to the plugin task owner. The projection has no provider
credentials and no generated runtime keys.
- `createPluginRunnerTaskOwner` sends the fields only to a provider with
the capability. An older provider gets no new field, and the run log
names the variables that were not set. The log never prints values.
- **Codex** (`codex-security-config.ts`):
- An externally sandboxed launch uses `inherit="all"`,
`ignore_default_excludes=true`, and an `exclude` list of Codex's own
credentials, instead of an `include_only` list fixed at preparation
time.
- A test reads the Runner's Codex environment keys, and fails if a new
key is not classified.
  - Host launches keep the bounded allowlist.
- **runnerd** (`codex_provider.rs`): `runtime_notification_codex_args`
still grants the notification files to read-only profiles when no
allowlist is present. Plan mode in an external sandbox needs these
grants.
- **Protected keys**:
- An environment driver can declare `protectedEnvironmentKeys` in its
manifest.
- Paperclip rejects a protected key on an agent or environment that uses
the driver. The 422 has code `protected_environment_variable`.
- On a project, a routine, or another agent, Paperclip saves the key.
The response has a `configurationWarnings` notice that names the
environments. The project and routine pages show the notice as a toast.
The agent pages already show configuration warnings.
- **Docs**: `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md` describes the
provider obligations.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the reviewed
SDK source pin.

## Verification

- `packages/plugins/sdk`: `vitest run tests/environment-tasks.test.ts`
(38 pass). Accepts bounded values. Rejects bad names, NUL bytes, too
many variables, and oversized values.
- `server`: `vitest run
src/services/native-runtime/plugin-runner-task.test.ts` (95 pass).
Values go only to a provider with the capability, never in `prepare`.
`resume` sends values only for a handoff.
- `server`: `vitest run src/services/protected-environment-keys.test.ts
src/__tests__/project-routes-env.test.ts` (10 pass). Rejection, notices,
and the project route response.
- `server`: all route suites that import the project, routine,
environment, or agent routes (45 files, 1689 pass).
- `packages/shared`: `vitest run src/validators/plugin.test.ts` (64
pass).
- `packages/paperclip-runner`: `vitest run src/drivers/codex
src/configured-environment.test.ts
src/live/runner-provider-launch.test.ts` (444 pass).
- `packages/paperclip-runner/runner`: `cargo test -p
paperclip-runner-core --lib runtime_notification` (9 pass, 2 ignored as
before).
- `pnpm typecheck` in `packages/shared`, `packages/plugins/sdk`,
`packages/paperclip-runner`, `server`, and `ui`.
- Not run locally: `pnpm verify --full` and browser suites. Required CI
covers the final revision.

## Risks

- **Existing providers:** low risk. Core sends no new field to a
provider without the capability. Their runs behave as before, with one
added log line when variables were skipped.
- **Codex in an external sandbox:** commands now see every variable in
the Runner environment except Codex's own credentials. The execution
environment builds that environment from an allowlist. If the Runner
gives Codex a new credential, the drift test fails until the key is
classified.
- **Saving env:** new rejections happen only when an installed driver
declares `protectedEnvironmentKeys`. No driver declares them today.
- **Providers:** they must keep the values transient and never save or
log them. The doc lists these obligations.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:11:23 +00:00
DottaandClaude Opus 5.5 bcadbde449 feat(runner): record the OpenRouter host that served each OpenCode model call (#21298)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode agents can use models through OpenRouter, and the Runner
proxies those model calls
> - OpenRouter routes one model across many hosts, and these hosts use
different quantizations (fp4, mxfp4, fp8)
> - Chat evaluations with identical prompts behave very differently
between runs, but the run log does not record which host served a call
> - The run usage shows `provider: "deepseek"` and `biller:
"openrouter"`, and `providerRequestId` is always `null`
> - This pull request records the serving host and the OpenRouter
generation ID for each model call in the run log
> - The benefit is that we can compare evaluation cohorts by host, and
the cost event names a real OpenRouter generation

## Linked Issues or Issue Description

**Subsystem affected**
Runner OpenCode provider proxy, runnerd usage normalization, and native
run usage accounting.

**Problem or motivation**
OpenRouter's default routing sends a request to one of many hosts. Each
host can use a different quantization. Chat evaluation results change a
lot between runs with identical prompts. Paperclip does not record which
host served a call, so we cannot compare results by host.
`usageJson.providerRequestId` is always `null` for OpenCode runs.

**Proposed solution**
Every OpenRouter completion response has a top-level `provider` field
and an `id` (the generation ID). Each streamed chunk has them too. The
Runner proxy reads them from the first chunk after it forwards the
bytes. The Runner attaches them to the usage of that call. The values go
to the run log, and the run's first generation ID goes to
`usageJson.providerRequestId`.

**Alternatives considered**
A new PRP event type such as `provider.upstream.served` would need a new
event enum value. An older control plane rejects an unknown event type.
A `harness.diagnostic` event shows in the task chat as a diagnostic. The
usage payload already flows to every consumer, and older consumers
ignore extra fields. Thus this change uses the usage payload.

**Additional context**
Run-log change only (AGENTS.md section 7). The data stays in
`heartbeat_run_events` and `heartbeat_runs.usageJson`. It does not touch
`packages/shared/src/telemetry/` or OpenTelemetry.

## What Changed

- New `createUpstreamServedTap`
(`packages/paperclip-runner/src/drivers/opencode/upstream-served.ts`).
It is a pass-through stream. It pushes each chunk downstream first, then
it reads `provider` and `id` from the first SSE `data:` event (or from a
JSON body at its end). It scans at most 64 KiB of a stream and 1 MiB of
a JSON body. It skips encoded bodies and unknown content types. Any
parse problem or recorder error stops the scan and leaves the response
unchanged.
- `startOpenCodeProviderProxy` uses the tap only for 2xx responses. The
OpenCode session keeps the most recent served call. Each
`item.completed` usage payload and `session.usage()` now carry
`upstreamProvider` and `providerRequestId`. The usage dedupe fingerprint
does not include these fields, so the number of usage events does not
change.
- The OpenCode app-server proxy adds the two fields to
`thread/tokenUsage/updated`. Rust `usage.reported` now fills its
existing `providerRequestId` field and adds an optional
`upstreamProvider`. The TypeScript runnerd transport keeps both fields
when it rehydrates the usage notification. `usage.schema.json` documents
`upstreamProvider` (the protocol manifest and schema bundle are
regenerated).
- Server native turn accounting keeps the first generation ID the run
reports. The usage checkpoint and the native adapter result carry it as
`providerRequestId`, so `usageJson.providerRequestId` and the final cost
event name it.
- `doc/run-log-events.md` documents the fields in a new "OpenRouter
serving host" section.

## Verification

- Red/green: `opencode-server-driver.test.ts` "passes OpenRouter's
stream through unchanged and records the host that served it on the
turn's usage". The test sends a streamed fixture with a comment line, a
host chunk, a malformed chunk, and `[DONE]` through the proxy. It checks
that the bytes are identical and that the turn's usage names `Novita`
and the generation ID. The test fails without the proxy wiring.
- `upstream-served.test.ts`: streamed fixture, chunk split inside a
multibyte character, non-streamed JSON, malformed chunks, a truncated
JSON body, an encoded body, the 64 KiB limit, and a recorder that
throws.
- `opencode-proxy-usage.test.ts`, `runnerd-codex-transport.test.ts`
(rehydration), Rust
`provider_events::tests::carries_the_serving_host_and_generation_on_proxy_usage`,
and server `native-usage-accounting.test.ts` (first generation ID stays
stable across turns).
- Local: OpenCode driver, OpenCode proxy, and runnerd transport suites
(31 files, 596 tests) pass. `cargo test -p paperclip-runner-core` passes
(one timing test failed once under load and passed alone). Runner and
server `tsc --noEmit` pass. Protocol manifest, type, and coverage checks
pass.
- Latency: a local benchmark proxied a 400-chunk SSE stream 200 times
with the tap and 200 times without it. The median time to first byte was
292 µs with the tap and 259 µs without it (+33 µs, which is the extra
stream stage). The total stream time changed from 491.4 ms to 492.5 ms.
The tap spends about 5.5 µs of CPU to inspect the first chunk, after it
has forwarded that chunk. OpenRouter time to first token is hundreds of
milliseconds.
- Paid evidence: one single-case run of
`chat-features.runner-opencode.local.skills.fork-read-only` on this
branch. The result is in a comment on this pull request.

## Risks

- Low risk. All new fields are optional and additive. An older control
plane ignores the extra usage fields. `usage.schema.json` is not bound
by the event schema, and no validator enforces it on `usage.reported`.
- Saved state from the previous release: this change adds no persisted
run, session, checkpoint, or launch state. A saved session resumes
unchanged. A run that is in flight during a deploy reports usage without
the new fields until its Runner is new. `usageJson.providerRequestId`
then stays `null`, as before.
- `providerRequestId` on the run and its cost event is the run's first
generation ID, not each call's ID. The value is stable, so a late usage
receipt still matches the posted charge. Per-call IDs and hosts are in
the run log.
- If OpenCode starts a new model call before it reports the usage of the
previous call, a usage event can name the newer call. OpenCode reports a
message's usage when its step ends, so this is not expected.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code as an agent with tool use (shell, file edits, test runs). Extended
reasoning was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:15:53 -06:00
Nicky LeachandClaude Opus 5.5 45523d21d0 fix(runner): re-bind any settled ACPX checkpoint across a Runner upgrade (#21220)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Runner runs Claude Code through the ACPX backend and saves
a durable checkpoint so a later run can continue the same conversation
> - Each Runner release changes the ACPX launch profile digest that the
checkpoint stores
> - Restore re-binds a saved checkpoint to the new digest only when it
is `suspended`, or when the same run resumes an interrupted turn
> - A retired or parked Claude Runner leaves an idle `session_open`
checkpoint, so a follow-up from the next run fails with
`provider_state_incompatible` before its first turn
> - `run.attach` already accepts that checkpoint. The restore gate is
stricter than the attach it serves
> - This pull request defines a settled checkpoint once and uses the
same rule for attach and for the upgrade re-bind
> - The benefit is that a completed Claude conversation continues after
a Runner upgrade, and a refused re-bind names its reason

## Linked Issues or Issue Description

Fixes #21212

Refs #20112 (the same-run interrupted-turn case, which this change keeps
as is)

## What Changed

- Added `settled_checkpoint_blockers`. A settled checkpoint is not
closed, has a provider identity, has no active turn, has no unconfirmed
provider exit, and has no undelivered event except a `session.resumed`
notice. `run.attach` now admits with this function. The warm-attach
readiness probe also uses it, and still waits for a pending
`session.resumed` notice.
- Restore re-binds any settled checkpoint to the current launch profile
digest, whichever run saved it and whatever its lifecycle
(`session_open`, `prepared` or `suspended`). Candidate admission, a
changed launch shape and pending runtime requests still block the
re-bind.
- `rebinds_interrupted_turn` keeps only its distinct case: the same
run's `turn_starting` or `turn_active` turn, under the lost-turn
recovery preconditions from #20112. The settled rule now covers its
`session_open` case.
- A digest mismatch that is not re-bound now names each reason, for
example `(not re-bound: durable_active_turn)` or `(not re-bound:
launch_shape_changed)`. The classification stays
`provider_state_incompatible`.
- Updated "Settled ACPX sessions across a Runner upgrade" in
`packages/paperclip-runner/docs/durable-recovery.md`.

## Verification

From `packages/paperclip-runner/runner`:

- `cargo test -p paperclip-runner-core --lib acpx_provider_backend`: 48
passed
- `cargo test -p paperclip-runner-core --test acpx_warm_attachment`: 6
passed
- `cargo test --locked -p paperclip-runner-core --no-fail-fast`: 1108
passed, 0 failed, 6 ignored
- `cargo fmt --all -- --check` and `cargo check --locked --workspace`:
clean

New and changed tests:

-
`settled_checkpoints_of_a_retired_runner_rebind_for_the_next_run_after_an_upgrade`
covers `session_open` with a semantic result, `prepared` with an
identity, and a pending `session.resumed` notice. Each checkpoint is
saved by run 1 and restored by run 2 under a new digest. Restore does
not write the checkpoint, `run.attach` succeeds, and the saved digest
becomes the current one. This test fails before the fix.
-
`continues_a_settled_conversation_in_the_next_run_after_a_runner_upgrade`
runs end to end with the fake sidecar. The replacement Runner attaches,
resumes the same provider session, and emits exactly one `turn.started`.
This test fails before the fix with the production error.
-
`unrestorable_saved_sessions_keep_strict_bindings_and_name_their_class`
no longer lists `session_open` as strict. Each remaining strict case
asserts the reason in its error.

I did not run clippy because CI does not run it. I did not run `pnpm
verify` because only Rust and docs changed.

## Risks

- The re-bind now admits more checkpoints. It admits only the states
that `run.attach` already admits on the same build, so no in-flight
provider work moves to a new build. An active turn, a pending runtime
request, an unconfirmed provider exit, or an undelivered non-resume
event still blocks the re-bind.
- A same-run `session_open` checkpoint that has a semantic result or an
active goal now re-binds. The same build already re-opens this state.
The result and goal checks gate lost-turn recovery only, and that
recovery path is unchanged.
- Session start still verifies every launch artifact, the command digest
and the restored provider identity after the re-bind.
- No migration and no API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, sub-agents) and extended thinking. One sub-agent
wrote the change. A second session reviewed it before this pull request.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 17:44:58 -08:00
Nicky LeachandPaperclip 38501b1170 Add turn-scoped attachment downloads to native Runner (#20956)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native agents open input images with a file reader.
> - Plugin Runner tasks use an authenticated Runner connection.
> - Runner needs a way to fetch an authorized file from the tenant app.
> - The tenant grants access to the exact attachment for the active run.
> - Runner downloads and verifies it before it returns a local path.

## Linked Issues or Issue Description

Refs #19740. Builds on the attachment authorization and command
transport added in #20939.

## What Changed

- Add the PRP v3 `runner.attachment.download` command for native
workspace providers.
- Issue short-lived grants for exact authorized wake and historical
attachments. Recheck the active run, source, size, and SHA-256 at the
tenant endpoint.
- Download current-wake inputs before provider dispatch. Download
historical files when the authorized chat reader asks for them. Preserve
SSH and sandbox command staging as a fallback.
- Run HTTPS transfers off the Runner control loop. Report completion
with one PRP event per transfer. Drop pending results on cancellation
and clear files at turn settlement, authority rotation, and restart.
- Grant restricted Codex sessions read-only access to the attachment
directory.

## Verification

- 69 targeted server attachment tests passed. They cover wake and
historical downloads, authorization failures, and command fallback.
- 22 focused historical attachment tests passed, including grant
redemption under a held database lock.
- Six focused Rust attachment tests passed, including a source-pinned
Codex sandbox check. The sandbox could read the staged file but could
not modify it or read adjacent Runner state.
- The real Runner transport test confirmed one durable command and one
completion event for a transfer. Restart-continuation tests confirmed
fresh attachment paths appear before dispatch.
- Server and Runner TypeScript checks, Rust formatting, and PRP
generated schema checks passed.
- Required GitHub CI passed on `7a33752bd7`. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/43122967837)
- Greptile scored the same revision 5/5 with no unresolved review
threads.
[Review](https://github.com/paperclipai/paperclip/pull/20956#issuecomment-6808582134)

## Risks

- Saved provider and Runner identities keep their existing formats. A
previous Runner falls back to command staging where the target supports
it. A restart continuation refreshes temporary attachment paths before
dispatch.
- Plugin Runner tasks require a compatible Runner ROM and outbound HTTPS
access to the configured tenant origin.
- Live agent-computer qualification remains a release check.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, shell tools, and
local test execution. Exact variant and context window are unavailable
in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #123` /
`Closes #123` / `Refs #123` or (b) described the issue in-PR following
the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-11 16:28:02 -08:00
DottaandClaude Opus 5.5 d9d5958d09 fix(runner): reject an out-of-order controller command instead of exiting (#20946)
## Thinking Path

- Paperclip's controller and the sandbox Runner exchange numbered
commands; the Runner refuses a command whose number is not exactly one
past the last it processed.
- Staging's most common "Runner gone" failure was a follow-up controller
whose journal was behind the Runner. #20928 removed the cause on the
server (a warm follow-up now launches from the journal its retired
predecessor saved) and added a defence that reconciles a journal behind
only by the predecessor's settled close.
- The Runner still treated an out-of-order command as fatal:
`durable/state.rs` returned an error and the process exited. A
controller with a stale journal from any other cause (state saved by an
older release, a mid-run deploy) therefore lost the Runner process and
the run could only fail as "Runner gone"
(`runner_remote_process_identity_unavailable`).
- This pull request makes the Runner refuse such a command with a
precise result instead of exiting, and makes the server rest that run
with a precise code and a Retry whose next run reconciles the journal
(the #20928 defence) before it opens the session.

## Linked Issues or Issue Description

- Refs #20928 (the server-side root-cause fix and the journal
reconciliation this relies on)
- Refs #20907 (makes runnerd exit text visible in the run log)
- Duplicate search: searched PRs for "controller sequence contiguous",
"controller_sequence_mismatch" and "out-of-order command"; none changes
the Runner's handling.

## What Changed

- `runner-core/src/durable/state.rs`: a new command whose
`controllerSeq` is not `lastControllerCommandSeq + 1` (behind or a gap)
now yields `CommandDisposition::Reject` with `status: "rejected"` and `{
code: "controller_sequence_mismatch", message, expectedControllerSeq,
receivedControllerSeq }`. The command is not executed or journaled and
the last sequence is unchanged; the process keeps running. Replay of an
already-processed command and the compacted-history rejection are
unchanged.
- `native-session-executor.ts`: a failure carrying
`controller_sequence_mismatch` is classified as
`native_controller_journal_behind_runner`, a permanent (terminal)
failure with a Retry action that says no provider work began and that
Retry reconciles the journal from the saved checkpoint or stops with a
precise error.
- Tests:
- Rust `tests/durable_recovery.rs`: out-of-order (behind and gap)
commands are rejected with the precise code, the journal is unchanged,
and the next contiguous command executes. The existing
`command_gaps_and_identifier_reuse_fail_closed` unit test now asserts
the rejection instead of an error.
- Real runnerd `src/live/runnerd-stale-controller-journal.test.ts`: a
follow-up controller with a journal missing the predecessor's close now
gets `"code":"controller_sequence_mismatch"`, the Runner process stays
alive and its durable sequence is unchanged; with the current journal it
resumes.
  - Server: the classification and terminal disposition of the new code.

## Verification

- Red (master Runner): the Rust test panics on `expect("an out-of-order
command is not fatal")`; the real-runnerd test receives `runnerd exited
unexpectedly with code 1: paperclip-runnerd: controller sequence must be
contiguous: expected 6, received 4`.
- Green: `cargo test -p paperclip-runner-core --test durable_recovery`
(4 passed); `cargo test -p paperclip-runner-core` lib suite (the only
other failure, `task_memory::tests::monitor_bounds_notices`, is a timing
test that passes alone); `npx vitest run
src/live/runnerd-stale-controller-journal.test.ts` with a runnerd built
from this branch (2 passed); server `native-session-executor.test.ts`
focused tests (71 passed).
- `pnpm --filter @paperclipai/paperclip-runner check:acpx-profiles`:
profiles match the release manifest. No ACPX profile, attestation, pin
or pack contract schema changed (`rules/acpx-profile-identity.mdc`): the
change is Runner source behaviour with an existing command-result shape,
so no profile version or `commandDigest` changes.

## Risks

- A controller that is *ahead* of the Runner (a gap) is now rejected
instead of fatal. Neither case executes the command; the Runner journal
is unchanged, so this is strictly safer than exiting.
- The rested run needs a Retry. Before this change the same run failed
after its retries as "Runner gone" with no precise cause.
- Runner source changes the Daytona image content ID; a new image is
needed for chaos waves on this revision.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), 1M-token context, tool
use (shell, file edits, tests) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 16:14:31 -06:00
DottaandClaude Opus 5.5 a52f622be8 fix(native): retry a remote provider startup failure once its runner exit is proven (#20874)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) start a Runner in the
sandbox, and the Runner opens the provider session
> - When the provider rejects `session.open` at startup, no provider
work began
> - #20808 ends every such remote rejection at once with Retry, because
restart recovery could not prove that the failed runner was gone. A
fault that clears by itself therefore needs one human click
> - The Runner transport already settles the failed runner on close and
observes its exit. That is the missing proof
> - This pull request records that proof on the failure and lets the
same-run bootstrap retry use it. It stops a startup error that repeats,
so an image fault that a retry cannot fix does not loop
> - The benefit is that a transient remote startup failure recovers
without a person, and a permanent one still rests at once with a working
Retry that names the error

## Linked Issues or Issue Description

Refs #20808
Refs #20832
Refs #20820

**What happened?**
A remote (Daytona) native run whose provider rejected `session.open` at
startup ended as `native_provider_startup_rejected` with Retry. This
happened even when the Runner had settled and exited, and no provider
session, event or checkpoint existed. A person had to click Retry for a
fault that the durable record proves did no provider work.

**Expected behavior**
With a proven runner exit and no provider work, the run retries its
bootstrap on the same run automatically. A startup error that repeats on
the next attempt stops at once with Retry and names that error. Failures
saved without the proof keep today's behaviour.

**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-startup-rejected-retry.integration.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "remote startup rejection|session.open rejection"`
2. On master, 3 cases fail: the proven remote rejection is
`terminal_failure`, the repeat does not name its error, and restart
recovery holds the proven failure on
`ambiguous_remote_recovery_evidence`.

**Paperclip version or commit**
master at 3ded7bbd91.

## What Changed

- Runner transport (`runnerd-codex-transport.ts`): new
`runnerExitConfirmed()`. It is true only after a completed close settled
the runner (provider stopped, runner suspended) and the runner process
exit was observed.
- Codex driver (`codex-app-server-driver-impl.ts`): after it closes the
transport for a failed session open, it sets `runnerExitConfirmed` on
the error when the transport proves the exit.
- Executor (`native-session-executor.ts`):
- A remote `provider_session_open_rejected` with a startup
`providerFailure` keeps the same-run `bootstrap_retry` only with that
proof. The saved failure records `runnerExitConfirmed: true`.
- Each failed attempt's recovery history entry records a startup failure
signature: provider, category and message, with numbers ignored. When
the next attempt fails with the same signature, it stops as
`native_provider_startup_rejected`. Its next action names the startup
error and keeps "Retry starts a new run".
- Deterministic rejections (`requires_action` authentication, permission
or invalid request) stop at once as before.
  - Local targets are unchanged.
- Restart recovery (`native-restart-recovery.ts`):
`remoteRunnerExitedBeforeProviderWork` also accepts a settled
`provider_session_open_rejected` with `runnerExitConfirmed: true`. All
its other checks still apply: no provider session, no provider event, no
checkpoint, no runner-sent event, no successor.
- Docs: `doc/execution-semantics.md` and
`packages/paperclip-runner/docs/durable-recovery.md`.

## Verification

- Red on master, green here:
- `native-startup-rejected-retry.integration.test.ts`: proven remote
rejection → `retryable_failure` with `runnerExitConfirmed`; unproven →
terminal; repeat → terminal, the next action names `bwrap: loopback`,
and Retry is offered.
- `native-runner-restart-recovery.integration.test.ts`: proven failure →
`bootstrap_incomplete` (`remote_runner_exited_before_provider_work`); an
older release's failure without the proof stays held.
- `codex-app-server-driver.lifecycle.test.ts`: the driver records the
proof only when the transport reports it after close.
- `native-session-executor.test.ts`: unit cases for the rule, the
`cause` chain, the repeat signature and deterministic rejections.
- Local runs: the 3 server files above pass (1380 tests); restart
recovery `-t remote` 75 pass; driver lifecycle 25 pass.
- `tsc --noEmit` clean for `server` and `packages/paperclip-runner`.
- Not run: a Daytona chaos wave. There is no in-sandbox startup fault
seam yet (see Risks).

## Risks

- The proof depends on the transport's close settling the runner after a
failed `session.open`. The Rust runner keeps serving after a failed
`session.open` (lifecycle `Continue`) and accepts `runner.suspend`, so
close should settle. If it does not settle, there is no proof and the
run ends at once with Retry, which is today's behaviour. It is never
less safe than today.
- Retries stay bounded: the startup retry budget (3 attempts) and the
repeat rule both apply. A repeated identical error stops after one
automatic retry.
- Saved state: failures saved by older releases have no
`runnerExitConfirmed` and keep the old rule. History entries without a
signature never match a repeat. A mid-run deploy is safe: an old
executor never writes the proof.
- Not covered by chaos: a remote "Codex fails at startup N times" fault
needs a hook inside the sandbox Runner or Codex. DF1's seam is
server-side only, and no open chaos PR adds a seam inside the sandbox.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) Opus 5.5, model ID `claude-opus-5-5`, run in Claude
Code with tool use (shell, file edit) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 11:55:15 -06:00
Nicky LeachandClaude Opus 5.5 a219dd7499 fix(runner): keep orphan-leaving tests apart from the verifying lineage reap (#20849)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Runner stops a dead runnerd's provider and tool processes
from its process lineage journal
(`process_lineage::journal::reap_journals`)
> - The reap refuses its proof (`unattributed_orphan`) while any live
orphan (parent PID 1) of this user started after the dead runnerd's last
sample, whoever started it. This rule is deliberate
> - `cargo test` runs the runner-core lib tests in parallel in one
process, and several of those tests leave orphans of this user on
purpose
> - When one of those orphans starts during the reap in
`reaping_stops_a_dead_runners_provider_and_its_escaped_tool_job`, the
reap correctly refuses, and `ci / Verify Paperclip Runner (rust)` fails
> - This pull request keeps the orphan-leaving tests apart from that
reap with a test-only gate, and fixes a related zombie race in the
supervisor integration tests
> - The benefit is that the Runner Rust job stops failing on unrelated
pull requests

## Linked Issues or Issue Description

Refs #20834. Refs #20770.

**What happened?**

`ci / Verify Paperclip Runner (rust)` failed intermittently in
`process_lineage::journal::tests::reaping_stops_a_dead_runners_provider_and_its_escaped_tool_job`
(journal.rs:649), for example in run 43044317310 on #20834. The panic
was `ReapReport { verified: false, reason: Some("unattributed_orphan"),
journals: 1, signalled: 4, remaining: [4178] }`. The test's own thread
was TID 4142, so the leftover process started while the test ran. The
same test failed once on #20770 and passed on re-run.

**Expected behavior**

The reap proves the test's provider and escaped tool job gone, and no
other test can make it refuse.

**Steps to reproduce**

1. Build the runner-core lib tests for Linux (`cargo test --release -p
paperclip-runner-core --lib --no-run`).
2. Run the binary 40 times in `docker run --init --cpus 2 -e
RUST_TEST_THREADS=8`.
3. See `reaping_stops_a_dead_runners_provider_and_its_escaped_tool_job`
fail with `unattributed_orphan` (3 of 40 runs here).

## What Changed

- Root cause: the reap's orphan rule covers every process of this user
on the host. The reaping test and the orphan-leaving tests run in the
same test process at the same time. The test's own processes are always
attributed through the recorded group and session, so the leftover was
another test's orphan.
- `process_lineage::host_orphans` (test-only): a gate. A test that can
leave an orphan of this user holds `may_orphan()` until the orphan is
gone or a zombie. A test whose reap must verify holds
`no_foreign_orphans()`. Holders of `may_orphan()` never wait for each
other, so a test can hold several without a deadlock.
`wait_until_group_gone()` waits until a killed group has exited, because
a signal takes effect after `kill` returns.
- `reaping_stops_a_dead_runners_provider_and_its_escaped_tool_job` holds
`no_foreign_orphans()`.
- These lib tests now hold `may_orphan()`:
- `reaping_refuses_an_orphan_born_after_the_last_sample` (deliberate
orphan)
- the three `process_supervisor`
`*_stops_a_tool_job_that_left_its_session` tests (escaped tool job)
-
`orphaned_startup::restart_keeps_the_fence_while_a_child_survives_its_exited_leader`
(`sleep 30` outlives its leader)
- `exited_child_reader_deadline_bounds_continuous_descendant_stdout`
(flooding subshell outlives the Codex provider)
- `a_timed_out_turn_start_never_establishes_a_resumable_rejection`
(`sleep 10` can be orphaned for a moment during shutdown)
- the five `provider_backend` tests that use
`disconnected_receipt_limit_fixture` (`while :; do sleep 1; done`,
killed through `fence_local_receipt_limit`). The fixture returns its
guard first, so the guard is released after the executor stops the
provider
- the two `late_park_observation_*` tests (`cat >/dev/null` runs until
shutdown)
-
`a_held_cancellation_that_cannot_reach_the_provider_keeps_the_completed_turn`
(shutdown during `sleep 2`)
- New test
`a_verifying_reap_waits_for_held_orphans_and_holds_new_ones_back` checks
the gate's ordering.
- `tests/process_supervisor.rs`:
`forced_process_group_cleanup_stops_harness_and_worker` and
`natural_harness_exit_also_cleans_up_workers` slept 20 ms and then used
`kill -0` on a worker that was re-parented to init. `kill -0` succeeds
for a zombie, and init reaps it on its own schedule. The tests now check
that the process is not running (gone or a zombie), with no sleep.
Cleanup already proves that before it returns.
- No product code changes. The reap's orphan rule is unchanged.

Audit of the other tests in the job (`check:runner`):

- Only the runner-core lib binary can overlap the reap. `cargo test
--workspace` runs test binaries one after another.
- A second sweep, after review, checked every `/bin/sh` launch and every
external command (`sleep`, `cat`, `perl`, `nohup`, `touch`, `mv`) in
runner-core's `src/`, including `src/tests/*.rs`. The first sweep had
missed tests that start a provider through `CodexProvider::start` or
that live in `src/tests/`.
- Other lib tests that start processes are not affected. They run shells
that use only builtins (`configured_environment`, the other
`process_supervisor` and `codex_provider` tests,
`acpx_provider_session`), or run a short foreground `touch` that
finishes long before teardown
(`disconnected_codex_output_is_durable...`), or they start direct
children that the test waits for (`oom_score`, `exited_writer`,
`process_group_leader`). `publisher_crosses_real_codex_tool_boundary...`
is `#[ignore]`.
- The other `reap_journals` tests refuse before the orphan rule applies,
or (the orphan test) require only their own orphan in `remaining`.
- The `kill -0` checks in `tests/codex_provider.rs` and
`tests/acpx_park_preparation.rs` target direct children that runner-core
waits for before it returns, so they are deterministic.

## Verification

Linux runs used Colima on macOS (linux/arm64), `rust:1.97.1-bookworm`,
`--release`, and `docker run --init`, so orphans go to an init at PID 1,
as on CI.

- Full runner-core lib suite, `--cpus 2`, `RUST_TEST_THREADS=8`, 40
runs: before the fix, 3 runs failed with the CI signature. After the
fix, 0 runs failed, and no other test failed.
- The reaping test beside a temporary test that keeps leaving
short-lived orphans (not committed), `--test-threads 2`, 10 runs: before
the fix, 10 failed. After the fix, with the temporary test holding
`may_orphan()`, 0 failed.
- An orphan-leaving shell outside the test binary made the unfixed test
fail in 9 of 10 runs with the same signature. This confirms the
mechanism.
- After the review fix: macOS `cargo test --release -p
paperclip-runner-core --lib -- receipt_limit late_park_observation
a_held_cancellation process_lineage process_supervisor orphaned_startup
exited_child_reader_deadline a_timed_out_turn_start`: 65 passed. Linux
full lib suite (same settings) ×20: 0 failures.
- macOS (darwin-arm64): `cargo fmt --all -- --check` passes. `cargo test
--release -p paperclip-runner-core --lib -- process_lineage
process_supervisor orphaned_startup exited_child_reader_deadline
a_timed_out_turn_start`: 51 passed. `cargo test --release -p
paperclip-runner-core --test process_supervisor`: 7 passed.
- Worker cleanup tests: with `tini` as init, 60 unfixed runs on 0.5 CPU
with busy loops all passed, because `tini` reaps at once. In a container
whose PID 1 never reaps (`sleep infinity`), the unfixed tests fail at
`assertion failed: !process_exists(worker_pid)` because the killed
worker stays a zombie. The fixed tests pass. The unfixed tests depend on
how fast init reaps.

## Risks

- Low risk. All changes are in tests or `#[cfg(test)]` code.
- The gate covers the orphan producers that exist now. A new lib test
that leaves an orphan of this user must hold `may_orphan()`. The
`host_orphans` doc comment says so. Otherwise, it can make the verifying
reap refuse again.
- A process outside the test binary that leaves an orphan of this user
during the reap still makes the test refuse. This is correct product
behavior. CI runs no such process during `cargo test`.
- `no_foreign_orphans()` waits until every `may_orphan()` guard is
released, so the reaping test can wait a few seconds behind those tests.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool
use (shell, file editing, Docker). Extended reasoning was used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 08:55:30 -08:00
DottaandClaude Opus 5.5 4529dfb77f fix(runner): keep Codex planning read-only on Daytona without network namespaces (#20820)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A planning task runs a native Codex agent in read-only planning mode
before the user approves a Plan
> - On a Daytona sandbox, every Codex planning run failed before its
first turn: `error creating thread: failed to load AGENTS.md
instructions for environment 'local': fs sandbox helper failed ...
bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted`
> - Codex 0.160.0 reads files through a filesystem helper when the
permission profile restricts reads. That helper always runs bubblewrap
with `--unshare-net`. Daytona has no `CAP_NET_ADMIN`, so bubblewrap
cannot bring up loopback in the new network namespace
> - The planning profile must stay read-only. Codex's plan collaboration
mode is only a prompt, so the external-sandbox profile (#20808 tried it,
then reverted it) would let planning runs write files
> - This pull request keeps the read-only profile, but inside an
external sandbox it gives the profile full-disk read, no writes and
network. Codex then reads files directly and runs commands without a new
network namespace, and it still refuses writes
> - The benefit is that Codex planning works on Daytona and stays
read-only

## Linked Issues or Issue Description

Refs #20808

**What happened?**
A Codex planning task on a Daytona execution environment failed at
startup. The run ended with `provider_session_open_rejected` and the
Codex message `bwrap: loopback: Failed RTM_NEWADDR: Operation not
permitted`. Standard and ask runs on the same sandbox worked.

**Expected behavior**
A planning run on Daytona starts, writes its Plan and asks for approval.
Codex must still refuse file writes and mutating commands during
planning.

**Steps to reproduce**
1. Configure a native Codex agent with a Daytona execution environment.
2. Create a task in planning mode.
3. The first run fails before its first turn with the bubblewrap
loopback error.

## What Changed

- `createIsolatedCodexAppServerArgs`: when
`PAPERCLIP_RUNNER_EXTERNAL_SANDBOX=1`, the
`paperclip-runner-workspace-read-only` profile is now `{":root"="read"}`
(plus the brokered `GH_CONFIG_DIR` write that planning already had). Its
network still follows `PAPERCLIP_RUNNER_NETWORK_ACCESS`. It has no deny
entries, so Codex 0.160.0 treats it as full-disk read and skips its
filesystem helper, which always unshares the network. Local and SSH
planning keep the narrower profile unchanged.
- The Codex session context now records `rootAccess` `read` (planning)
or `write` (default) in an external sandbox, instead of always `none`.
- The Runner still sends `permissions:
"paperclip-runner-workspace-read-only"` and the plan collaboration mode
on `thread/start` and `turn/start` for planning. A new Rust test pins
that for an externally sandboxed read-only launch (no `sandbox:
danger-full-access`, no `externalSandbox` `sandboxPolicy`).
- A new Rust test proves that a launch saved with the previous planning
profile rules has the same stable launch identity as a launch with the
new rules, so saved sessions carry forward.
- New manual Product E2E suite `codex-planning-read-only`
(`runner-codex` on local and Daytona). The planning turn runs one shell
write, writes a Plan whose first line records the outcome, asks for
approval, and finishes after approval. A grader reads the run events and
fails if no write attempt was seen or if any write succeeded.
- `docs/codex-driver.md` describes the external-sandbox planning
profile, why it avoids the helper, and its read scope.

## Verification

- Red on Daytona, master `123ee31552`: [run
43029207500](https://github.com/paperclipai/paperclip/actions/runs/43029207500),
cell `core-compatibility.runner-codex.daytona.plan-revise-accept` failed
twice with `error creating thread: failed to load AGENTS.md instructions
for environment 'local': fs sandbox helper failed with status exit
status: 1: bwrap: loopback: Failed RTM_NEWADDR: Operation not
permitted`.
- Planning turn completes on Daytona: [run
43031462803](https://github.com/paperclipai/paperclip/actions/runs/43031462803),
cell `core-compatibility.runner-codex.daytona.plan-revise-accept`. The
planning run succeeded, saved a two-step Plan with the draft marker and
opened a confirmation for that revision (all API checks passed). The
cell then failed a later browser check: the Plan preview rendered as the
`task-chat-plan-preview-fallback` card, which the harness does not
accept. That is a separate UI placement gap.
- Write attempt on Daytona: [run
43038547892](https://github.com/paperclipai/paperclip/actions/runs/43038547892),
cell `codex-planning-read-only.runner-codex.daytona.plan-write-refused`
on `f30aeb3e72`. The planning run succeeded and ran the shell write
`printf probe > plan-write-probe-....txt || echo WRITE_REFUSED`. It
ended `failed` with exit 1 and no file, because bubblewrap could not
start the command (`bwrap: setting up uid map: Permission denied`). The
write was refused fail-closed by the sandbox launcher, not by a
read-only filesystem error. The stricter grader in a3d43b7bfd reports
this case as 'sandbox failed to start' rather than a pass. Shell
commands in Daytona planning therefore do not run on the current image;
the driver doc records this as a remaining gap.
- Red/green unit test: `npx vitest run
src/drivers/codex/codex-security-config.test.ts` (in
`packages/paperclip-runner`). The new test failed on the unfixed code
(the profile was `":root"="none"` with `CODEX_HOME` denied and network
off) and passes with the fix.
- `cargo test --locked -p paperclip-runner-core --test codex_provider --
external_sandbox` and `cargo test --locked -p paperclip-runner-core
--lib -- external_sandbox_planning_profile_rules
runtime_launch_rebinding`: pass.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/codex-planning-read-only.test.ts
tests/runner-e2e/catalog.test.ts`: pass. `pnpm
test:e2e:runner:typecheck`: pass.
- Real Codex 0.160.0 app-server probe (macOS) with the generated launch
arguments and `PAPERCLIP_RUNNER_EXTERNAL_SANDBOX=1`: `thread/start`
returned sandbox `{"type":"readOnly","networkAccess":true}` and loaded
the workspace `AGENTS.md`; `command/exec` with the planning profile
refused `printf x > plan-write-probe.txt` (exit 1, `Operation not
permitted`, no file) and allowed `cat AGENTS.md`. Trace logs showed 0
filesystem-helper invocations with the new profile and 336 with the
previous restricted profile.
- Codex 0.160.0 source: `exec-server/src/fs_sandbox.rs` always sets
`NetworkSandboxPolicy::Restricted` for the helper,
`core/src/agents_md.rs` uses the helper only without full-disk read, and
`linux-sandbox` adds `--unshare-net` only for a restricted or proxy-only
network.

## Risks

- Inside an external sandbox, planning commands can now read the whole
sandbox filesystem, including the isolated Codex home. Default-mode
turns in the same sandbox already have full read and write access, so
this does not widen what the sandbox exposes. Local and SSH planning are
unchanged.
- Inside an external sandbox, planning shell commands fail closed on the
current Daytona image because bubblewrap cannot map user IDs there.
Planning still reads files and writes the Plan through Codex and
Paperclip tools.
- If a future Codex release adds deny-read entries or makes the helper
mandatory, planning on Daytona could fail again. The new Daytona E2E
suite detects that.
- Upgrade safety: the profile filesystem and network keys are excluded
from the stable launch identity, so saved sessions resume with the new
rules (new Rust regression). No persisted schema, pack contract or ACPX
profile changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, GitHub CLI) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 09:44:28 -06:00
Nicky LeachandClaude Opus 5.5 ff8ec41e6a feat(runner): upgrade ACPX to 0.22.0 and move patched behavior into Paperclip (#20770)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Runner and the shared `adapter-utils` engine run ACP
agents through the embedded ACPX library
> - Paperclip pins ACPX 0.13.1 and keeps a large patch
(`patches/acpx@0.13.1.patch`) on it
> - ACPX 0.22.0 adds host hooks (`onAcpMessage`, `assertActive`,
`processLifecycle`, prompt-response `_meta`, `spawnAgent`-friendly
launch) that make much of that patch unnecessary
> - A large patch is expensive to port and hides behavior that belongs
to Paperclip
> - This pull request upgrades to ACPX 0.22.0 and moves most patched
behavior into Runner and `adapter-utils`
> - The benefit is a patch that is 484 lines instead of 1,334, with the
moved behavior covered by Paperclip tests

## Linked Issues or Issue Description

Refs #20675, #14430, #15075, #17942, #9980

**Proposed enhancement**
Upgrade the embedded ACPX library from 0.13.1 to 0.22.0. Keep only the
patch behavior that stock 0.22 cannot express.

**Motivation**
The 0.13.1 patch held behavior that Paperclip owns, for example usage
receipts, Copilot policy, model checks and the agent environment. Each
ACPX upgrade had to port all of it. ACPX 0.22 gives hooks that let
Paperclip do this work itself.

**Current behavior**
Runner and `adapter-utils` depend on 1,334 lines of patched ACPX 0.13.1
code.

**Expected behavior**
Runner and `adapter-utils` use ACPX 0.22.0. The patch keeps only:
extension methods, Cursor subagent routing, event `_meta` and the
message boundary chunk, the protocol guard hook, `spawnAgent`, and
`requestExtension`.

## What Changed

- ACPX 0.13.1 → 0.22.0 in Runner, `adapter-utils`, the server, the
Daytona runner images and the qualified profiles (`acpxVersion`).
`@agentclientprotocol/sdk` moves to 1.9.0.
- New `patches/acpx@0.22.0.patch` (484 lines). The 0.12.0 and 0.13.1
patches are removed.
- Delivery receipts: Runner confirms a permission reply when
`onAcpMessage` sees the exact outbound reply. Question and extension
replies count as delivered when Runner hands them back to ACPX. New
`interaction-delivery.ts`.
- Copilot policy: the existing policy function runs from `onAcpMessage`.
A violation stops the agent, cancels later permission requests and fails
the next prompt through `assertActive`.
- Model selection: stock 0.22 validation, plus a sidecar gate
(`createAcpxEffectiveModelGate`) that refuses a prompt if the agent
reports another model. Grok binds its model when the session opens.
Antigravity adds its concrete model to the model list ACPX reads.
- Typed session failures: Runner and `adapter-utils` read them from the
turn result `_meta` and fail the turn. They advertise the capability
themselves.
- Pi and Cursor usage receipts: a collector (`prompt-receipts.ts`) binds
them to the turn's own `session/prompt` reply.
- Goal capability: Runner reads it from the `initialize` reply. It keeps
it under `paperclip.goalCapability` in the saved agent capabilities
`_meta` and restores it when a saved session loads.
- Agent environment: Runner and `adapter-utils` set the agent
environment in `spawnAgent`. ACP terminals keep their current
environment. `adapter-utils` blocks inheritance of `XAI_API_KEY` and
refuses to start if the server has an `ACPX_AUTH_*` variable.
- `adapter-utils` spawn hooks (`spawnCwd`, stderr, `onAgentSpawn`): a
new `stockAcpRuntimeOptions` maps them to stock `spawnAgent` and
`processLifecycle`.
- Windows: ACPX 0.22 rejects raw command strings on Windows.
`adapter-utils` now passes an argv array there.
- Grok rules: a Runner guard (`grok-session-rules.ts`) writes
`session/new` `_meta.rules`.
- Turn-dispatch journal (#20675): ACPX 0.22 ignores errors from
`onAcpMessage`. The adapter keeps a failed `dispatched` write and throws
it from the turn's `assertActive`. ACPX calls `assertActive` before the
prompt write, so the write still stops. The real-ACPX test now checks
this route. `docs/durable-recovery.md` is updated.
- Plan events: the sidecar reads 0.22 `status` events tagged `plan`.
- CI: the backup-checkpoint lane now installs through the shared
`setup-ci-dependencies` action, like PR CI, so it resolves a stale
`pnpm-lock.yaml` in place. It runs on any root `package.json` change,
and pull requests never commit the lockfile, so its own frozen install
failed on dependency changes.
- Tests that checked removed patch internals are removed or rewritten:
the response-delivery, Cursor-usage and Pi-receipt package contracts,
and the goal-persistence packaging test. New unit tests cover the moved
behavior.

## Verification

Local runs on macOS (darwin-arm64), Node 26, on the rebased head:

- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/drivers/acpx src/cli`: 2,000 passed.
- `tsc -p packages/paperclip-runner/tsconfig.json` and
`tsconfig.test.json`: no errors, except the existing
`@paperclipai/paperclip-eval-kernel` build gap in `workflow-harness.ts`.
- Runner package contracts (`node --test
test/acpx-*-package-contract.test.mjs
test/acpx-sidecar-contract.test.mjs`), `scripts/build-inputs.test.mjs`,
`scripts/contracts/contract-fingerprints.test.mjs`,
`scripts/acpx-patch-packaging.test.mjs`: all pass. `node
scripts/generate-acpx-sidecar-contract.mjs --check` passes.
- `cargo test -p paperclip-runner-core --test acpx_park_preparation
--test acpx_park_readiness --test acpx_warm_attachment --test
native_provider_backend`: 86 passed. `cargo fmt --check` passes.
- `adapter-utils`: `tsc` clean. `vitest run src/acpx-engine
src/mcp-isolation.integration.test.ts`: 457 passed (one Gemini flag test
failed once under load and passes when run alone).
- Adapters: `claude-local` and `codex-local` `acp.quota.test.ts` (23 and
39 passed), `grok-local` (149 passed).
- Server: `scripts/prepare-runner-vendor.test.mjs`,
`native-session-executor.test.ts`, `quarantined-task-retry.test.ts`
(1,416 passed).
- `cargo test -p paperclip-runner-core --lib -- previous_acpx_release
acpx_upgrade unrestorable_saved_sessions`: 3 passed. `node --test
test/acpx-session-carry-forward-package-contract.test.mjs`: passes.
- `patch -p1 --forward` on a clean `acpx-0.22.0.tgz` gives output
identical to the port.
- Under full-suite load, some Runner live and recovery tests time out.
Each one passes when run alone. Full regression is left to CI.

Not run: live provider runs (they need provider credentials), the opt-in
Kimi live peer test, the Gemini filesystem smoke test, and Windows.

## Risks

- Previous-release story: sessions saved by the previous release (ACPX
0.13.1) carry forward. ACPX 0.22 loads the session records 0.13.1 wrote.
Runner moves a saved descriptor that names 0.13.1
(`CARRIED_ACPX_RELEASES`) to 0.22.0, and the sidecar keeps 0.13.1 as the
ACPX value in the session identity, so the saved session key and
identity record still match. The existing launch-profile rebind still
decides which sessions resume: a settled session, or the same run's
interrupted turn through lost-turn recovery. A session saved by any
other ACPX release is still refused as `provider_state_incompatible`.
`rules/acpx-profile-identity.mdc` and `docs/durable-recovery.md` now
describe this.
- Proof: `acpx-session-carry-forward-package-contract.test.mjs` resumes
a session record written by the real ACPX 0.13.1 with its Paperclip
patch. 0.22 loads the same provider session (no `session/new`), runs a
turn and keeps the earlier turn. Rust tests cover a settled 0.13.1
session that the next run attaches to, and a turn interrupted by the
upgrade. A recovery-identity test checks that the session key and
identity record match across the versions.
- `pnpm-lock.yaml` is not committed. CI resolves it for each run, and
the lockfile refresh workflow updates it after merge.
- Behavior change on the server: `adapter-utils` refuses to create an
ACPX runtime if the server environment has an `ACPX_AUTH_*` variable.
Before, ACPX ignored such variables. Stock 0.22 would use them to
authenticate agents with server credentials.
- Delivery confirmation is now Runner-owned. A permission reply is
confirmed by the exact outbound message. Question and extension replies
are confirmed when Runner hands them to ACPX, not when the pipe accepts
them.
- Users can select only models that the agent advertises. Stock 0.22
rejects other model IDs (Claude is exempt).
- ACP terminals keep the sidecar or server environment, as before. This
existing exposure is not changed by this pull request.
- The `adapter-utils` Gemini version probe now runs with the server
environment plus the run variables, not the run variables alone.
- The pack contract digest and profile `acpxVersion` change, so Daytona
needs a new runner image.
- Grok launch flags (`--model`, `--reasoning-effort`) were checked on
1.0.13 but session open needs a login, so Grok keeps the existing
`_meta` path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool
use (shell, file editing, subagents). Extended reasoning was used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 07:28:00 -08:00
DottaandClaude Opus 5.5 fa3cd6f330 fix(native): end provider startup rejections with Retry instead of an ownership hold (#20808)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner starts Codex for native runs. On a remote
sandbox such as Daytona, the sandbox is the OS boundary.
> - A Codex planning run on Daytona failed in `session.open`. Codex's
read-only sandbox helper could not set up loopback in the container
(`bwrap: loopback: Failed RTM_NEWADDR`), and Codex loads `AGENTS.md`
through that helper.
> - The server treated that definite provider rejection as an
interrupted transport. Its same-run bootstrap retry went through remote
restart recovery and stopped on `ambiguous_remote_recovery_evidence`.
The task showed "Recovering saved work…" and no Retry, although no
provider work was done.
> - This pull request gives such startup rejections a named, permanent
outcome with a working Retry, and pins ask and standard runs to the
external sandbox policy.
> - The benefit is that a provider startup refusal ends as a clear
failure with Retry, never as an ownership hold.

## Linked Issues or Issue Description

**What happened?**
A Codex planning run on a Daytona sandbox failed in `session.open` with
`error creating thread: failed to load AGENTS.md instructions ... bwrap:
loopback: Failed RTM_NEWADDR: Operation not permitted`. The server
recorded `native_session_interrupted` with recovery mode
`bootstrap_retry`. Thirty seconds later, remote restart recovery blocked
the run with `native_restart_recovery_blocked` /
`ambiguous_remote_recovery_evidence`. No server restart happened. All
default-mode Codex runs on the same image succeeded that day; only the
planning run failed.

**Expected behavior**
A provider startup failure with no provider work must end as a clear
failure with a working Retry, or get a bounded automatic retry that also
ends with Retry. It must never become an ambiguous-ownership hold.

**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-startup-rejected-retry.integration.test.ts`.
On master the deterministic case is `retryable_failure` /
`native_session_interrupted`.
2. `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "provider
itself rejected"`. On master the new code is not produced.

**Paperclip version or commit**
master at 84fc67317e.

## What Changed

- Server (`native-session-executor.ts`): new permanent code
`native_provider_startup_rejected`.
- A definite `session.open` failure maps to it when its startup
`providerFailure` is `requires_action` for `authentication`,
`permission` or `invalid_request`.
- On a remote execution target (`nativeRemoteStartupRejectionCode`), any
definite `session.open` failure with a startup `providerFailure` maps to
it. A remote same-run bootstrap after the runner launched cannot prove
ownership and stops on `ambiguous_remote_recovery_evidence`.
- The run ends as `terminal_failure` with a board owner and a plain next
action. The board offers Retry, which starts one new run with a fresh
provider session.
- On a local target, unknown, transient, rate-limit, quota and model
startup failures keep the bounded same-run retries and end as
`native_session_retry_exhausted` with Retry. That code is unchanged
because the chat-channel failed-run retry depends on it.
- Runner: planning runs keep Codex's
`paperclip-runner-workspace-read-only` profile, also in an external
sandbox. Codex 0.160.0's plan collaboration mode is a prompt ("Plan Mode
(Conversational)") and does not block writes or commands, so it cannot
replace that enforcement.
- Ask mode: an ask task builds `executionMode: "default"`. Its launch
has `readOnly: false`, the external sandbox profile, and no
collaboration mode. Ask mode restricts only Paperclip mutation tools,
not the Codex sandbox.
- Tests:
- `native-startup-rejected-retry.integration.test.ts` (new, embedded
Postgres, Codex provider): a deterministic rejection ends at once with
`canRetry: true`, and Retry starts exactly one new run. Unknown and
transient local failures stay `retryable_failure`. A spent retry budget
ends `terminal_failure` with `canRetry: true`.
  - Unit tests for the deterministic, local-retry and remote rules.
- `runner-provider-launch.test.ts`: ask, standard and planning launches
in an external sandbox.
- The Rust external-sandbox test now fails if a non-planning turn sends
a collaboration mode (`--forbid-plan-collaboration` switch on the fake
app server).
- Docs: `packages/paperclip-runner/docs/codex-driver.md`,
`packages/paperclip-runner/docs/durable-recovery.md`,
`doc/execution-semantics.md`.

## Verification

- Red on master, green here: the commands in "Steps to reproduce".
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts`: 1364 pass.
- `npx vitest run src/live/runner-provider-launch.test.ts`
(packages/paperclip-runner): 22 pass.
- `cargo test -p paperclip-runner-core --test codex_provider
external_sandbox`: pass. `cargo fmt --check`: clean.
- `npx tsc --noEmit -p server/tsconfig.json`: clean.
- `pnpm verify` returned exit 2 (a global CI input changed). Required CI
supplies the broad coverage.

## Risks

- Codex planning on Daytona still fails at thread creation. With this
change it ends at once with the provider message and Retry instead of an
ownership hold. A follow-up must make Codex's read-only sandbox work in
that container, or add a verified Paperclip read-only guard. This PR
does not relax read-only enforcement.
- A remote startup failure that could clear by itself now needs a Retry
click instead of an automatic same-run retry. Before this change, that
retry ended in an ownership hold, so no working automatic path is lost.
- A false `native_provider_startup_rejected` costs automatic retries and
shows Retry earlier. It never replays provider work, because
`session.open` precedes every prompt.
- Saved state: no persisted shape or launch argument changes. A run that
an older release held on `ambiguous_remote_recovery_evidence` after the
same rejection already offers Retry once its saved workspace is
recovered. No migration is needed.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) Opus 5.5, model ID `claude-opus-5-5`, run in Claude
Code with tool use (shell, file edit, browser reads) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 07:48:18 -06:00
DottaandClaude 08ff0828d6 fix(native): offer Retry at once when the Runner refuses a turn start on a closed session (#20729)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native ACPX run (for example Claude) starts each turn with
`turn.start`. When a start is cut off, the Runner reads the sidecar's
dispatch journal (#20675).
> - If the prompt never left, the Runner keeps the session reopenable,
and the same-run resume sends the prompt once. If the prompt may have
left, the Runner closes the session, because a resend could duplicate
it.
> - A closed session cannot reopen for the same run. Every same-run
resume got `ACPX session is not open`. The server treated that as a
generic interruption and spent three attempts (about 95 s) before the
board offered Retry.
> - This pull request gives that refusal a closed class in the Runner
and a permanent, named failure code in the server.
> - The benefit is that the board offers a working Retry on the first
refusal, while every recoverable case keeps recovering by itself.

## Linked Issues or Issue Description

**What happened?**
Restart-chaos wave `w-20261111T011704-s9107`, task t05 (local, claude,
workload C1, fault TM2). A freeze cut off a turn start, and the Runner
saved the session `closed`. Three same-run resumes each failed with `PRP
command turn.start failed: … "message":"ACPX session is not open"`.
Recovery stopped with `native_session_retry_exhausted` at 01:25:14, and
the board then offered Retry. The Retry run succeeded.

**Expected behavior**
When the Runner proves the session is closed for this run, the run stops
at the first refusal with a named failure, and the board offers a Retry
that works. A start that the Runner proved never reached the provider
still recovers by itself.

**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-closed-session-retry.integration.test.ts`
2. On master, "offers Retry on the first refusal, and Retry starts a new
run" fails: the coordinator is `retryable_failure` /
`native_session_interrupted` (a same-run retry in 30 s).

Refs #20675, #20720

## What Changed

- Runner (`durable/mod.rs`, `acpx_provider_backend.rs`): new closed
class `provider_session_closed`. The Runner sets it on these refusals:
- a turn start that saves the session `closed` (`ACPX turn start failed
closed: …`), but not the rejected-checkpoint case, which stays
`prepared` and reopens;
- `ACPX session is not open` when the saved lifecycle is `closed` for
this run;
- the saved turn-end-restart refusal for this run (asserted in
`acpx_turn_end_restart`).
A session that waits for provider-exit proof can reopen after a restart.
Its refusal gets no class.
- Server (`native-session-executor.ts`): a `turn.start` failure with
that class maps to the new permanent code
`native_provider_session_closed`. The server reads the class from the
authenticated command result, and from the command-failure text if the
error was rewrapped. `nativeSessionFailureDisposition` treats the code
as permanent, so the first refusal ends the run as `terminal_failure`
with a board owner and a plain next action.
- Docs: `packages/paperclip-runner/docs/durable-recovery.md` states the
class and the older-Runner behavior.
- Tests:
- `native-closed-session-retry.integration.test.ts` (new, embedded
Postgres) drives `executePaperclipNativeSession` with the Runner
refusal. On the first refusal, the board shows `canRetry: true`.
`requestTaskRetry` is accepted, and `resumeTaskRetry` starts exactly one
new run with a fresh provider session. A start proved undispatched stays
`retryable_failure`.
- Unit tests in `native-session-executor.test.ts`: both closed refusals,
a rewrapped error, an undispatched start, an unclassified older-Runner
refusal, and a class on another command.
- Rust `acpx_cut_turn_start`: a start cut off before dispatch has no
class. A start cut off after dispatch, and the later refusal, carry
`provider_session_closed`.

## Verification

- Red on master, green here: step 1 above.
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts`: 1358/1358
pass.
- `cargo test -p paperclip-runner-core --test acpx_park_preparation` (26
pass), `--lib acpx` (94 pass), `--test acpx_provider_turns --test
acpx_provider_session --test acpx_warm_attachment` (all pass), `cargo
fmt --check` clean.
- `npx tsc --noEmit -p server/tsconfig.json`: clean.
- Not run: a chaos wave.

## Risks

- This changes the Rust Runner. Daytona needs a new runner image before
remote runs get the class. Until then, an older Runner sends the refusal
without the class, and the server keeps today's bounded same-run
retries. That path is unchanged and ends with a working Retry. The
server does not trust the text alone, because an older Runner sends the
same text for a session that can still reopen.
- The class is set only where the Runner's saved state proves that no
later start of this run can reopen the session. A false class would cost
one automatic resume and show Retry earlier. It never replays provider
work.
- No persisted shape changes. The class is an optional field of a failed
command result. Old saved state resumes unchanged. Nothing new runs
before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, run in Claude Code
with tool use (shell, file edit) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-11 00:12:41 -06:00
DottaandClaude fcf4e39b41 fix(runner): do not count stopped time against a turn start or an ACPX sidecar request (#20720)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run starts each provider turn with `turn.start`. The
control plane waits for the Runner, and an ACPX Runner waits for its
sidecar, which starts the provider and sends the prompt.
> - Both waits use a 30 s bound. When the host or container stops the
whole tree (SIGSTOP, a host sleep), the clocks keep moving while no
process can act.
> - On thaw both bounds have passed. The control plane reports `PRP
command turn.start timed out`, and the Runner fails the sidecar request
and closes the ACPX session.
> - #20704 added a stopped-time clock for startup and recovery. This
pull request puts `turn.start` and its follow-up waits on the same
clock, and adds the same rule to the Runner's sidecar response wait.
> - The benefit is that a freeze during a turn start no longer fails the
turn or closes the session.

## Linked Issues or Issue Description

**What happened?**
Restart-chaos wave `w-20261111T011704-s9107`, task t05 (local, claude,
workload C1, fault TM2). A follow-up run sent `turn.start` at 01:22:21.
The harness then stopped the server, Runner, sidecar and Claude twice:
28 s, then 33 s after a 1 s thaw. At 01:23:25 the control plane reported
`provider_transport_failed: PRP command turn.start timed out`. Three
same-run resumes then failed with `ACPX session is not open`, recovery
stopped with `native_session_retry_exhausted`, and the task needed a
manual Retry. The Retry run succeeded.

**Expected behavior**
Time that the processes spend stopped does not count against the
turn-start bound or the sidecar request bound. A real timeout still
fires while the processes run.

**Steps to reproduce**
1. `cd packages/paperclip-runner && npx vitest run
src/live/runnerd-codex-transport-stall.test.ts`. On master it fails with
`provider_transport_failed: PRP command turn.start timed out`.
2. `cargo test --manifest-path
packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core
--test acpx_sidecar_stopped_time`. On master it fails with `ACPX sidecar
request timed out at initialize`. It stops the process early in the wait
and again in its last half second.

Note on the repeated `ACPX session is not open`: the wave ran
`eafd5853d9`, which does not contain #20675. On master, #20675 settles a
cut-off start from the sidecar's dispatch journal. If the prompt never
left, the Runner reopens the session and sends it once. If the journal
shows the prompt left, or cannot prove it, the session stays closed and
the run ends with a working Retry, as the class rule requires. This pull
request stops the freeze from cutting off the start in the first place.

Refs #20704, #20675

## What Changed

- `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`:
`#startTurn` runs under `#withAdmissionClock`. The turn-start deadline,
the held-tool-call wait, the start-event wait, the adoption check time
and the `session.snapshot` wait all read `#waitNow()`, so they share one
stopped-time credit.
-
`packages/paperclip-runner/runner/crates/runner-core/src/acpx_sidecar_transport.rs`:
`receive_response` blocks in 1 s slices. `credit_stopped_time` treats a
slice overshoot beyond 2 s as a stop. It moves the deadline by the
overshoot, and to at least 5 s after the thaw, so a stop in the last
slice still leaves the sidecar time to answer. Total credit per request
is capped at 15 min.
- The control-plane clock (`RunnerStallLedger`) now uses the same rule:
a stop is credited in full plus a 5 s grace, capped at 15 min per
admission.
-
`packages/paperclip-runner/src/live/runnerd-codex-transport-stall.test.ts`
(new): drives the real transport and Runner. It SIGSTOPs the Runner
tree, jumps `Date.now` by 40 s mid-wait, then resumes the Runner. It
fails if the turn start leaves the clock, or if the command wait stops
reading it (checked by removing `clock: this.#admissionClock`). This is
the transport-level test requested in the #20704 review.
-
`packages/paperclip-runner/runner/crates/runner-core/tests/acpx_sidecar_stopped_time.rs`
(new, its own test binary): a sidecar stops the test process for 6 s
against a 3 s bound and answers after the thaw.
- Unit test for `credit_stopped_time`.

## Verification

- Red on master, green here: both commands in "Steps to reproduce".
- `npx vitest run src/live/runnerd-codex-transport.test.ts
src/live/runnerd-codex-transport-stall.test.ts
src/live/runnerd-acpx-source-reconciliation.test.ts
src/live/runnerd-codex-transport-adoption.test.ts`: 342/342 pass.
- `cargo test -p paperclip-runner-core --lib acpx_sidecar_transport` (12
pass), `--test acpx_sidecar_transport --test acpx_park_preparation
--test acpx_provider_turns` (all pass), `cargo fmt --check` clean.
- `npx tsc --noEmit -p tsconfig.json` and `-p tsconfig.test.json`:
clean.
- Not run: a new chaos wave.

## Risks

- A real stall longer than 2 s in the Runner (or 5 s in the control
plane) during a turn start or a sidecar request now extends that wait by
the stall plus 5 s. The waited-on process could not be observed during
the stall either. The 15 min cap bounds a process that keeps stalling.
- The Rust wait now wakes once per second while it waits for a sidecar
reply. This is an active request with no I/O per wake.
- The 1 s control-plane ticker from #20704 now also runs during a turn
start, and stops when the start returns.
- No persisted state, launch identity or saved-session shape changes.
Old saved runs resume through the same code, and an older Runner keeps
its old wait. Nothing new runs before `turn.start`, and no wait is added
on the normal path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, run in Claude Code
with tool use (shell, file edit) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 23:25:32 -06:00
DottaandClaude e719d42557 fix(runner): accept the acknowledgement of an indeterminate suspend replay (#20714)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run talks to a Runner process (runnerd). The controller in
the server sends commands. The Runner journals each command before it
acts and replies with a result.
> - `runner.suspend` and `runner.shutdown` are terminal commands. The
Runner waits for a `command_result_ack` before it exits. It compares
that acknowledgement with the receipt that it journaled.
> - A Runner that is killed inside `runner.suspend` journals the command
but not its result. The next Runner replays the command as
`indeterminate`. It sends it on the wire as `failed` with
`execution_indeterminate`, and it still treats it as terminal.
> - The controller acknowledged the `failed` status that it received.
The Runner compared that with its journal status `indeterminate` and
exited with "terminal command result acknowledgement changed its durable
identity". The controller also lifted its terminal fence for a failed
suspend and sent the next command to a Runner that was about to exit.
> - This pull request makes both sides agree on the status of that one
receipt, and keeps the fence.
> - The benefit is that a freeze or restart during a suspend no longer
burns the same-run retries and leaves the task at a manual Retry.

## Linked Issues or Issue Description

**What happened?**

A restart-chaos task (local Runner, Claude, a 28-second freeze of the
server during a live turn) needed a manual Retry. Retry worked.

1. The turn completed. The server then sent `runner.suspend`. The server
froze before the Runner reported the result.
2. The close found the Runner not durably suspended (`commandStatus:
pending`, `runnerLifecycle: ready`). The run failed with
`native_runner_checkpoint_owner_lost`, and same-run recovery stopped the
old runnerd. The suspend was journaled but not finished.
3. Attempt 2 started a new runnerd on the same state. The welcome
delivered the stale suspend. The Runner replayed it as `indeterminate`
and sent `failed`. The controller acknowledged `failed`. The Runner
exited with code 1: "terminal command result acknowledgement changed its
durable identity". The controller had recorded the suspend as `failed`.
4. Attempt 3 found the fence for a suspend that the controller no longer
listed as pending. The Runner reconciled it and exited with code 0. The
server reported "runnerd exited unexpectedly with code 0", and
`native_session_retry_exhausted` stopped automatic recovery.

**Expected behavior**

The controller acknowledges exactly the receipt that the Runner sent.
The Runner accepts that acknowledgement, records `suspended`, and the
next attempt starts from a clean suspended Runner.

**Steps to reproduce**

1. Start a native run with a local Runner. Let the turn complete so that
the server sends `runner.suspend`.
2. Freeze the server (SIGSTOP) for about 30 seconds before the suspend
result arrives, so that recovery stops the Runner inside the suspend.
3. Let same-run recovery start a new Runner. It exits with "terminal
command result acknowledgement changed its durable identity". The unit
tests in this PR reproduce the same sequence without a live run.

**Paperclip version or commit**

`master` at 200c6e40e2. The failing wave ran on 9296c74e8d plus #20656.

**Root cause**

`command_result_envelope` changes the journal status `indeterminate` to
`failed` on the wire. `wait_for_terminal_result_ack` compared the
acknowledgement with the journal status, not with the status that it
sent. The controller also treated a `failed` suspend as a refused
suspend (the Runner keeps serving), but the Runner treats an
indeterminate suspend as terminal.

## What Changed

- Runner (`runner.rs`): a new `wire_command_status` gives the status
that goes on the wire. `wait_for_terminal_result_ack` accepts the
acknowledgement when its status is the wire status (`failed`) or the
journal status (`indeterminate`). Command id, type, and controller
sequence must still match exactly.
- Controller (`durable-prp-control-plane.ts`): for a result with
`execution_indeterminate`, the `command_result_ack` names
`indeterminate`. Runners that shipped before this change accept only
that status. The controller keeps the terminal-lifecycle fence for an
indeterminate suspend, so it does not send the next command to a Runner
that is exiting. The command stays `failed` in the controller journal,
as before.
- Runner Rust test
`indeterminate_suspend_replay_accepts_the_acknowledged_wire_status`: it
seeds a journaled suspend, starts the Runner against a fake controller
that acknowledges `failed` and then `indeterminate`, and checks that the
Runner finishes as `suspended` with no fence. Before the fix it failed
with the production message.
- Controller test: it checks the `indeterminate` acknowledgement and
that the next queued command is not delivered on that connection.
- `docs/durable-recovery.md`: it describes this case.

## Verification

- Red, then green: `cargo test -p paperclip-runner-core --lib
indeterminate_suspend_replay` failed with `DurableRunnerError("terminal
command result acknowledgement changed its durable identity")` before
the Runner change and passes after it.
- Red, then green: `npx vitest run
src/control-plane/durable-prp-control-plane.test.ts -t "indeterminate
suspension"` failed (`status: "failed"`, expected `"indeterminate"`)
before the controller change and passes after it.
- `cargo test -p paperclip-runner-core --lib durable`: 152 passed.
- `npx vitest run src/control-plane/durable-prp-control-plane.test.ts`:
215 passed.
- `npx vitest run src/live/runnerd-codex-transport-settlement.test.ts
src/live/runnerd-codex-transport-suspension.test.ts`: 30 passed. These
use the real runnerd binary built from `master`, so they also cover the
new controller with an old Runner.
- `npx tsc --noEmit -p packages/paperclip-runner`: clean.
- `pnpm verify` returned exit 2 (broad input). Required CI supplies the
broad coverage.
- Not run: a chaos wave. The unit tests reproduce the exact sequence at
the Runner and controller boundary.

## Risks

- Version mix. Old Runner with new server: the acknowledgement says
`indeterminate`, which is the old Runner's journal status, so it is
accepted. New Runner with old server: the acknowledgement says `failed`,
which is the new Runner's wire status, so it is accepted. New with new:
accepted.
- No command runs again. The indeterminate suspend is never executed
again; only its acknowledgement changes. The Runner still refuses a
different command id, type, or controller sequence.
- The Runner change takes effect only after a new Runner image. The
controller change alone removes the failure for Runners that already
shipped.
- Remaining gap: the attempt that delivers the stale suspend still ends
when that Runner exits as suspended. The next same-run attempt then
starts from a clean suspended Runner. A later change can restart the
Runner inside the same attempt.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, test runs). Extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 22:53:53 -06:00
DottaandClaude 3fdcc2a28c fix(runner): resend an ACPX turn whose start was cut off before the prompt left (#20675)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native ACPX run (for example Claude) sends `turn.start` to its
Runner. The Runner asks the ACPX sidecar to start the turn, and the
sidecar sends the prompt to the provider
> - A start can fail without a definite answer. A host freeze can
outlast the sidecar request deadline, or the sidecar or Runner can die
during the start
> - The Runner then saved the session `closed`. Every same-run resume
failed with `ACPX session is not open`, and the task needed Retry
> - The Runner cannot tell from its own state whether the provider has
the prompt. Resending could duplicate it. A continuation could lose it
> - This pull request adds a durable record that answers that question:
a sidecar prompt-dispatch journal, written before the prompt reaches the
provider
> - The benefit is that a start cut off before the prompt left resends
the prompt exactly once, with no Retry. Every case the record cannot
prove keeps today's behavior

## Linked Issues or Issue Description

**What happened?**
Local restart-chaos wave `w-20261110T185315-s9107`, task t05 (claude,
workload C1, fault TM2). A follow-up run reopened the parked Claude
session and sent `turn.start`. The sidecar first starts Claude and loads
the conversation; the prompt goes out only after that. After 24 s
without a reply, the harness froze the server, Runner, sidecar and
Claude (SIGSTOP) for 33 s. On thaw the Runner's 30 s sidecar deadline
had passed. It reaped the sidecar and saved the session `closed`. Three
same-run resumes failed with `PRP command turn.start failed: … ACPX
session is not open`, recovery stopped with
`native_session_retry_exhausted`, and a person had to click Retry.

**Expected behavior**
When a durable record proves the prompt never reached the provider, the
run resends it once by itself. When the record cannot prove it, the run
behaves as today.

**Steps to reproduce**
1. On master, run `cargo test --manifest-path
packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core
--test acpx_park_preparation cut_turn_start`.
2.
`a_start_cut_off_before_the_prompt_left_reopens_and_sends_the_prompt_once`
fails: the session is `closed` and the next start fails with `ACPX
session is not open`.

Refs #20507, #20645, #20674

## What Changed

- New file contract in
`protocol/provider-schemas/acpx-sidecar.schema.json`:
`turnDispatchJournalFile` (`paperclip-turn-dispatch.jsonl`) and
`turnDispatchJournalRecord` (`received` / `dispatched`). The generator
emits the file name and schema into TypeScript and Rust, so the pack
contract digest covers them.
- Sidecar (`turn-dispatch-journal.ts`, `codex-runtime-adapter.ts`,
`runtime-host.ts`, `acpx-runtime-sidecar.ts`): when the runtime port
starts a turn, before any provider work, it truncates the journal and
writes `received`. ACPX calls `onAcpMessage("outbound")` synchronously
before it writes a message to the provider. On the turn's first
`session/prompt` request the port appends `dispatched` and fsyncs it
there, so the record exists before the bytes leave. A failed dispatch
write aborts the prompt write. A failed `received` write removes the
file and runs the turn unjournaled.
- Runner (`acpx_turn_dispatch.rs`, `acpx_provider_backend.rs`): the
journal is read only after the provider lifetime fence proves that the
sidecar and provider are gone. A lone complete `received` record for the
turn proves the prompt never left.
- A start that failed in flight saves the session `prepared` with no
active turn and returns `ACPX turn start failed before its prompt
reached the provider`. The run's next `turn.start` reopens the
conversation and sends the prompt once. Until the proof settles, the
saved state stays `turn_starting`, so a Runner killed during the proof
goes through the restore check below.
- A Runner killed while the session was `turn_starting` restores the
session idle (`session.resumed`, no `turn.failed`). The same-run resume
then starts the turn once instead of sending a continuation that would
lose the prompt.
- A `dispatched` record, another turn's record, a torn or malformed
file, or no file keeps today's behavior (closed session, or lost-turn
recovery).
- Fake sidecar: writes the same journal and adds `hang-before-dispatch`
/ `hang-after-dispatch` faults.
- Docs: `packages/paperclip-runner/docs/durable-recovery.md` ("ACPX turn
starts cut off before the prompt left"). The design note is
`doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md` (in
#20674).

## Verification

- Red on master, green here: `cargo test -p paperclip-runner-core --test
acpx_park_preparation cut_turn_start` (4 tests: cut before dispatch
reopens and sends once; cut after dispatch stays closed and never
resends; Runner killed before dispatch restores idle; Runner killed
after dispatch or with no journal reports the lost turn). The two
"before" cases fail on master.
- `cargo test -p paperclip-runner-core`: all pass. `cargo fmt --check`
clean.
- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/drivers/acpx/turn-dispatch-journal.test.ts
src/drivers/acpx/codex-runtime-adapter.test.ts`: 121 passed. One test
uses a real ACPX runtime and an ACP peer that refuses a prompt arriving
before its `dispatched` record; it fails if the hook is removed.
- `node scripts/generate-acpx-sidecar-contract.mjs --check`, `node
--test scripts/contracts/contract-fingerprints.test.mjs`, and `tsc -p
tsconfig.test.json` pass.
- Time to first token, measured with the same open/write/fsync sequence
on this Mac (APFS, load average ~11, 300 samples): `received` p50 0.24
ms, p95 0.59 ms; `dispatched` (with fsync) p50 4.7 ms, p95 11.5 ms.
Against a Claude TTFT budget of 9.5–16.5 s p50 this is under 0.1%. No
wait, round trip or file transfer is added. Recovery work (fence,
journal read) runs only after a failed start.
- Live restart-chaos wave and runner image: see the comments on this
pull request.

## Risks

- A false "not dispatched" proof would duplicate a prompt. It needs a
lone, complete `received` record for this exact turn ID, read after the
fence proves the sidecar is gone. `dispatched` is fsynced before the
prompt write, and a failed write aborts the send. A failed `received`
write removes the file, so it cannot leave a lone record.
- Upgrade safety: saved provider state gains no field (its decoder
denies unknown fields), and only lifecycles an older Runner already
handles are written. An older Runner ignores the journal file. State
saved by an older Runner has no journal, which reads as unknown and
keeps today's behavior. A new Runner with an older sidecar never sees a
`received` record, so no proof forms.
- The changed sidecar schema changes the pack contract digest, so
Daytona needs a new runner image. Older image packs are not reused.
- A failed start now waits for the provider lifetime fence (at most 5 s,
usually immediate) before it settles. This is only on a failure path.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file editing) in Claude Code. Extended reasoning was used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 20:15:48 -06:00
DottaandPaperclip 76c787fffc fix(runner): report monitor notes length limits (#20604)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use task monitors to resume a future check.
> - The monitor schema limits notes to 500 characters.
> - The field description did not state the limit. Rejection showed only
the schema path.
> - This pull request adds the limit to the description and to safe
length errors.
> - Agents can correct notes before they retry a rejected call.

## Linked Issues or Issue Description

Related prior fix: #16868.

**What happened?**

A native `set_task_monitor` call with 501 characters in `monitor.notes`
failed with a path that ended in `notes/maxLength`. The error did not
state 500. The schema already had `maxLength: 500`, but the field
description did not state it.

**Expected behavior**

State the limit in the field description and in the rejection. Accept
500 characters. Reject 501 characters before dispatch. Preserve a saved
monitor when a replacement is invalid.

**Steps to reproduce**

1. Supply a future `monitor.nextCheckAt` and 501 characters in
`monitor.notes`.
2. Read the native provider error.
3. Before this fix, it shows the constraint path without the maximum.

**Paperclip version or commit**

Reproduced on `7368a0b15c2f02107778a961144d86e2eb59a952` in a Linux
source checkout, with Node 25.9.0 and Rust 1.97.1.

## What Changed

- Use one local value for the published notes limit and its description.
The server test checks agreement with the persisted shared validator.
Keep the standalone Runner package boundary.
- Read the numeric maximum from the authorized JSON Schema validation
error. Keep submitted values private and preserve the 512-byte feedback
bound.
- Test the published schema and real bridge dispatch path at 500 and 501
characters. Test saved contents and rejection without monitor changes
with real PostgreSQL.
- Refresh generated contracts and document the limit.
- Sync two SDK proof lock identities after the reviewed master lock
refresh in #20615. This fixes the remaining CI proof mismatch. Keep
compiler, parser, Zod, and all rejection checks unchanged.

## Verification

- Final head: `bac60234ab3a8fc7043a90c7f9c81b02e250428b`.
- Red: the server regression failed because the field description
omitted `500`. The Rust regression failed because the error omitted
`maximum 500 characters`.
- Local passes after the final base update: 44 PostgreSQL
monitor/scheduler tests, 14 real-source Rust bridge tests, 11
cache/Docker guard tests, and 56 SDK proof tests. Runner TypeScript
build and generated contract checks pass.
- The focused Rust check compiles the actual source in a small Cargo
workspace. The full Rust CI suites also pass on Linux and Windows.
- Final-head [PR CI
run](https://github.com/paperclipai/paperclip/actions/runs/42881633978)
passes. This includes typecheck, build, Runner support, Runner Rust and
Vitest, server tests, serialized tests, and all eight e2e shards.
- Final-head [Windows Rust
run](https://github.com/paperclipai/paperclip/actions/runs/42881633419)
passes.
- Greptile: 5/5 on the final head, with no open findings. There are 61
successful checks and five conditional skips. The separate Paperclip
Review remains queued; its comment says work is paused. No CI or test
check is pending or failed.
- `pnpm verify` returned exit 2 and selected broader CI. The full local
suite was not run. Final-head CI supplies that coverage. Earlier failed
attempts were repaired and are not counted as passes.

## Risks

- Low risk. The existing 500-character bound, scheduling semantics, and
permissions stay in place.
- Length feedback uses only a schema path and numeric bound. It does not
print submitted text.
- The Runner binary and tool contracts must reach an instance for the
new guidance to apply. This PR does not deploy or merge.

## Model Used

OpenAI Codex, GPT-6, with reasoning, tool use, and code execution. The
runtime does not expose a more specific model ID or context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 13:59:58 -06:00
Devin FoleyandPaperclip d2832a5a0b fix(runner): keep redacted display events inside their schema bounds (#20607)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs execute through Paperclip Runner. Its daemon, runnerd,
stores provider events durably before the server receives them
> - Before storing rich display events, such as tool output and provider
notices, runnerd redacts credential-shaped values. It then checks the
result against the event's protocol schema
> - Redaction can make a string longer. A field already at its schema
limit then goes over the limit, runnerd rejects the event, and the
daemon exits
> - The event is replayed after each restart, so every replacement
runner exits the same way. The run loops until something outside it
stops the run
> - This pull request trims a field that redaction pushed past its
schema limit, in both places that redact rich display events, so the
event is accepted and the runner keeps working
> - The benefit is that long tool output that contains a secret no
longer crashes the runner

## Linked Issues or Issue Description

**What happened?**

An agent run read GitHub Actions output and, after a context compaction,
entered a loop. About every 14 seconds runnerd started, restored its
durable PRP session, and exited with `paperclip-runnerd: redacted
display activity exceeds its schema bounds`. This continued for 15
minutes (62 launches) until a server restart ended the run. Every
reattach after that failed the same way, so the task was blocked.

**Expected behavior**

Redaction never makes a valid display event invalid. runnerd stores the
redacted event and continues.

**Steps to reproduce**

1. Produce a tool output longer than 64 KiB that contains a
credential-shaped pair, for example a GitHub Actions log line
`GITHUB_TOKEN: ***`. The ACPX sidecar keeps its last 64 KiB, which is
exactly the `toolExecution.output` limit of 65,536 characters.
2. runnerd redacts the pair to `GITHUB_TOKEN: [REDACTED]`, which is
longer, so `output` now has more than 65,536 characters.
3. `preserve_bounded_display_content` returns `redacted display activity
exceeds its schema bounds`, and runnerd exits.

A probe test with 64 KiB of output that contains `GITHUB_TOKEN: ***`,
`token=abc`, `Authorization: Bearer x`, or `password: hunter2` gets
exactly this error on `master`.

**Paperclip version or commit**

`master` at 491dfa1c6d.

## What Changed

- `acpx_event_payload.rs`: new `fit_redacted_display_to_length_bounds`.
- It asks the rich-event validator which fields fail `maxLength`, and
trims each one to its limit.
- `output` keeps its tail, like the sidecar's own truncation, and sets
`outputTruncated: true` on tool execution events. Other fields keep
their start.
- Fields within their limits do not change. A payload that still fails
its schema for another reason (type, pattern, or enum) is rejected as
before.
- Both places that redact rich display events call it before their
post-redaction schema check:
  - `decode_rich_event`, for ACPX `runtime.rich_event` payloads;
- `preserve_bounded_display_content` in `durable/state.rs`, before
durable storage.
- A new `tests/acpx_rich_events.rs` test decodes a 60,000-character
`plan.proposed` (Claude ExitPlanMode) with `token=abc`, and a
65,536-character tool output, through `decode_acpx_event`.
- Three new tests in `durable/tests/redaction_display_bounds.rs`,
included by the `durable/state.rs` test module:
- A full-length tool output stays within its bound for each of the four
credential shapes. It is the tail of the redacted text, no secret
remains, `outputTruncated` is set, and the stored event passes its
schema.
- A provider notice `summary` at its 4,000-character limit keeps its
start and gets no `outputTruncated` field.
- A short tool output that redaction makes longer, but still within its
limit, is not trimmed.

## Verification

- `cargo fmt --all -- --check`: clean.
- `cargo check --locked --workspace`: passes, with no new warnings.
- `cargo test -p paperclip-runner-core --lib -- redaction_
durable_display_content keeps_a_long_plan governance_display`: 16 pass,
including the three new tests.
- `cargo test -p paperclip-runner-core --test acpx_rich_events --test
acpx_event_payload`: 6 and 9 pass. Without the decoder call, the new
decoder test fails with `ACPX rich event redaction exceeded its schema
bounds`.
- Red check: before the fix, the probe described above failed with
`redacted display activity exceeds its schema bounds` for all four
credential shapes.
- Full `cargo test -p paperclip-runner-core --lib`: 620 pass and 2 fail.
The 2 failures are
`codex_provider::tests::fenced_receipt_output_reports_capacity_without_dropping_known_frames`
and `fenced_tool_rejection_keeps_identity_and_real_delivery_strict`.
They fail only under the full parallel run. In isolation they pass on
`master` and pass 3 of 3 times with this change.

## Risks

- Low. Only payloads that runnerd rejects today change, and those
payloads stop the runner. A trimmed display field loses some characters
at the limit. The tail of tool output is kept and `outputTruncated` is
set.
- Redaction still runs before trimming. Trimming only removes
characters, so it cannot expose a secret.
- Companion pull request #20608 bounds the transport's restore loop, so
any other event that runnerd rejects fails the run quickly instead of
looping.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) via Claude Code, with tool use
(shell, file editing, cargo builds and tests, read-only log analysis).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 11:48:27 -08:00
Nicky Leach 17136a6c25 fix(runtime): persist managed Runner credential refreshes across execution targets (#20580) 2026-11-10 10:15:29 -08:00
DottaandClaude 16c30ac32b fix(runner): relaunch a provider whose startup only timed out without a reply (#20521)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs execute through `paperclip-runnerd`. After a restart, a
new runner relaunches the provider (for example the Codex app server). A
durable startup fence stops a second runner from launching a provider
while an unadmitted one may still run.
> - #19784 lets a new runner retire a fence whose recorded provider
process and group have provably exited. That rule covers a launch that a
kill cut off. It deliberately keeps a failed initialization closed,
because that can be a real provider failure.
> - A relaunch whose reply never came within the runner's 30 s request
deadline is saved as the same failed initialization. A stalled or frozen
host causes this. The restart-chaos fault TM2 SIGSTOPs the runner and
Codex, and their monotonic clocks keep running while they are stopped.
> - Every later runner then refused to restore with "provider startup
ownership remains unadmitted", and the user had to click Retry. The
owner's rule is that a run interrupted by a restart or sleep must resume
by itself.
> - This pull request records whether initialization failed only because
the provider gave no reply, and lets a later runner retire that fence
once the provider is provably gone.
> - The benefit is that a stall during a provider relaunch continues
automatically, while a real provider rejection still fails as before.

## Linked Issues or Issue Description

Refs #19784

**What happened?**
Restart-chaos wave `w-20261110T091152-s426`, task t03 (Codex, B5), run
`c0bbce18-836b-43af-8195-92ebe3e21cca`. A TM3 restart relaunched the
Codex provider. A TM2 freeze stopped the tree 1 s later for 45 s, and a
TM3 kill followed the thaw. The next three runners each failed
`runner.drain` with "provider startup ownership remains unadmitted".
Recovery stopped with `native_session_retry_exhausted`, and the task
needed Retry.

**What did you expect?**
The new runner relaunches the provider and the run continues.

**Steps to reproduce**
The new test
`restart_relaunches_a_provider_whose_relaunch_timed_out_without_a_reply`
(runner-core `tests/codex_provider.rs`). The fake Codex stalls its first
`thread/resume` reply past the 30 s deadline. On `origin/master`, the
saved attempt is `initialization_failed` / `thread_open`, and the next
runner fails with "provider startup ownership remains unadmitted".

## What Changed

Design:
- The deadline that passes is the runner's own 30 s request deadline.
When a startup request (`initialize`, `thread/start`/`thread/resume`,
`thread/read`) reaches that deadline with the provider still running and
silent, the runner sets `noReply: true` on the durable
`initialization_failed` startup record.
- Not `noReply`: a provider error response, a provider that exited
before it replied, or closed stdout.
- `retire_orphaned_startup_attempt` retires an `initialization_failed`
fence only when `noReply` is set and the recorded process and its
process group are both gone (`ESRCH`). This is the same proof as
`spawned` in #19784. A rejection, an exit, a live process or group, and
spawn or admission failures stay fenced.
- Validation accepts `noReply` only on `initialization_failed` at the
`initialize`, `thread_open` or `thread_read` stage.
- Bounded: a runner restores once (`restore_checked` / `restore_error`),
so the loop is one relaunch per recovery attempt. Recovery's existing
resume budget (`native_session_retry_exhausted`, seen after three
attempts in the wave) stops a provider that never replies.
- Upgrade: `noReply` is `#[serde(default, skip_serializing_if =
"Not::not")]`. A new runner reads state saved by master as `noReply:
false`, so that state keeps today's behaviour and stays fenced.
- Rollback: an old runner (`deny_unknown_fields`) rejects state that
holds a `noReply` fence and fails closed. That run needs Retry, which is
today's outcome for the same fence. The field is written only while a
timed-out fence is present, and admission clears it, so all other state
is unchanged for old runners.
- Normal path: nothing is added before `turn.start`. The cost is one
in-memory flag per request; no extra I/O.

Files:
- `runner-core/src/codex_provider.rs`: `last_request_got_no_reply` and
the `no_reply` field on `ProviderStartupObservation::Failed`.
- `runner-core/src/provider_backend.rs`: the `noReply` field, its
validation, and the retirement rule, with unit tests:
- a no-reply fence is retired at `initialize`, `thread_open` and
`thread_read`;
  - it is kept while the provider may still run;
  - a rejection or a state saved without the field stays fenced.
- `runner-core/src/bin/fake-codex-app-server.rs`: the test-only switch
`--stall-first-thread-resume-ms`.
- `runner-core/tests/codex_provider.rs`: the end-to-end red/green test.
- `tests/restart-chaos/README.md` and `time-faults.ts`: correct the TM2
claim. A stopped process's monotonic clock keeps running, so TM2 is also
a long process stall.

## Verification

- `cargo test -p paperclip-runner-core --test codex_provider
restart_relaunches_a_provider_whose_relaunch_timed_out` is red on
`origin/master` (the next runner refuses restore) and green here, in
about 31 s.
- `cargo test -p paperclip-runner-core`: 1080 passed, 0 failed. This
includes
`failed_autonomous_restore_and_rollover_keep_their_exact_startup_origin`,
where a real "no rollout found" rejection still never restarts.
- `cargo fmt -p paperclip-runner-core -- --check` is clean.
- A restart-chaos proof wave on seed 426 will follow in a comment.

## Risks

- A provider that hangs for good is relaunched once per recovery attempt
until the existing resume budget stops it. Before, it failed on the
first attempt. This costs at most a few extra 30 s attempts, and they
are bounded.
- Rollback to an older runner while a `noReply` fence exists fails
closed for that one run, which needs Retry, as it does today.
- No database, server or API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, cargo and test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 05:23:30 -06:00
DottaandClaude a09093a5c5 fix(native): resume a run whose turn.start a restart cut off before Codex created the turn (#20507)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native Codex run sends `turn.start` to its runner, and the runner
sends `turn/start` to Codex. The runner marks the start "ambiguous"
before it sends the request
> - A control-plane restart can kill the runner and Codex while that
request is in flight. After the restart, neither journal says whether
Codex created the turn, and the resumed thread has no active turn
> - The restored runner then failed closed on every same-run resume with
"cannot safely recover an ambiguous Codex turn start without an active
replacement turn". The answer never reached the agent
> - The next run on the task was refused at the runner-state fence
(`prior_owner_terminal_state_indeterminate;
retirement_refused=journal_commands_unsettled(turn.start.failed)`)
before it asked for a runner. Retry after that run then waited forever
on `process_identity_missing` for a process that never existed
> - This pull request lets the runner prove from Codex's own thread
history that the start never created a turn, and then resume and start
the turn once. It also lets Retry continue after a run that the
runner-state fence refused before launch
> - The benefit is that a turn start cut off by a restart continues by
itself when Codex never saw it, and a person's Retry is no longer a dead
end

## Linked Issues or Issue Description

**What happened?**
Local restart-chaos wave `w-20261110T033426-s108`. Command: `--env local
--adapters codex --workloads Q1,Q2 --faults K1 --triggers answered
--tasks 4 --concurrency 2 --seed 108`. Task t03 (Q2): the answer run
`92a57f50` resumed after the first restart. A second K1 restart landed
0.25 s after `session.resumed`, while its `turn.start` was in flight.
Each same-run resume then failed with `PRP command session.snapshot
failed: cannot safely recover an ambiguous Codex turn start without an
active replacement turn`. When the prerequisite finished, the next run
`0e6aa9d1` failed before launch with `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate;
retirement_refused=journal_commands_unsettled(turn.start.failed)`. The
accepted Retry (202) never created a run: the task stayed on
`retryWait.reason=process_identity_missing` until the wave timed out
(`R3:retry_no_new_run`).

**Expected behavior**
If Codex never created the turn, the run resumes and starts it once by
itself. If that cannot be proven, a person's Retry starts a new run
instead of waiting forever.

**Steps to reproduce**
1. Run the wave command above on master.
2. Or run `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/live/runnerd-codex-transport-cut-turn-start.test.ts` on master (case
"before" fails with the wave error).

## What Changed

- Runner (`codex_provider.rs`, `provider_backend.rs`): when the restored
provider state has an ambiguous turn start, no active turn before the
loss, and the resumed thread has no active turn, the runner reads the
newest turn of the thread (`thread/turns/list`, limit 1, metadata only,
the same request `read_thread` already sends). If there is no turn, or
the newest turn is one Paperclip already settled, Codex never created
the turn. The runner clears the ambiguity, emits a `harness.diagnostic`
with code `ambiguous_turn_start_not_created`, and restores an idle
session. The old `turn.start` stays `execution_indeterminate` and is
never executed again. The same-run resume then starts the run's turn
once, through the existing "recovered session without an active turn"
path.
- Any other history (a turn Paperclip never settled, an in-progress
turn, an incomplete page, a non-Codex provider) keeps today's
fail-closed error. The probe runs only on this recovery path. The normal
turn path does no new work before `turn.start`.
- Server (`native-local-process-stop.ts`): `nativeRunNeverLaunched` also
accepts a failed native run whose error is the runner-state fence
refusal `runner_state_identity_mismatch: prior_owner_*` (or
`legacy_prior_owner_*`). The executor throws it only while it migrates
another run's saved state, before it requests a launch. The existing
evidence check still applies: any launch request, identity record,
`adapter.invoke` or runner event keeps the run held. Other runner-state
errors (for example `authority_indeterminate`) stay held.
- Fake Codex (`fake-codex-app-server.rs`): keeps the thread's turn
history in a `<state-file>.turns` sidecar, so `thread/turns/list`
returns history like real Codex (newest first, honours `limit`). New
`--hang-turn-start <file>` holds `turn/start` "before" or "after" the
thread records the turn, to model a kill in flight.
- Tests:
- New real-runnerd test
`runnerd-codex-transport-cut-turn-start.test.ts`. A real runner and fake
Codex start a turn; both processes are killed while `turn/start` is in
flight. "before": the resumed run completes, the journal holds the old
`turn.start` (`execution_indeterminate`) and one new completed
`turn.start` with the same task text, and Codex saw exactly one more
`turn/start`. "after": recovery still fails closed and no new
`turn/start` is sent.
- Rust:
`ambiguous_turn_start_that_never_created_a_turn_recovers_and_starts_once`
and
`ambiguous_turn_start_that_created_an_unsettled_turn_still_fails_closed`.
Both rewrite saved provider state the way a master runner wrote it, so
they also cover state saved before this change.
- `explicit-native-continuation.test.ts`: five new rows for the
runner-state fence. Admitted: refused before launch (with and without a
coordinator). Held on `process_identity_missing`: refused after a launch
request, refused after a runner event, and a run that failed its own
`authority_indeterminate` check.
- `docs/durable-recovery.md`: describes both behaviours.

## Verification

- Red/green, real runnerd: on master (the staged master runnerd),
`runnerd-codex-transport-cut-turn-start.test.ts` "before" fails with
`PRP command runner.drain failed: ... cannot safely recover an ambiguous
Codex turn start without an active replacement turn`, the wave error.
With this change, both cases pass.
- Red/green, Rust: without the `provider_backend.rs` change, `cargo test
-p paperclip-runner-core --test codex_provider
ambiguous_turn_start_that` fails the "never created" test. With it, both
pass. The full `cargo test -p paperclip-runner-core` passes.
- Red/green, server: without the `native-local-process-stop.ts` change,
the two "refused at the prior-owner runner-state fence" rows fail with
`process_identity_missing`. With it, all 14 rows pass.
- Related suites pass: 11 Runner transport files, 419 tests
(`runnerd-codex-transport`, `-attachment-recovery`, `-cold-restore`,
`-warm-attachment`, `-release-resume`, `-cut-attach`, `-cut-turn-start`,
`-settlement`, `-suspension`, `runnerd-restart-continuation`,
`runnerd-saved-continuation`); server `explicit-native-continuation`,
`quarantined-task-retry` and `issue-confirm-run-stopped-routes`, 3
files, 470 tests; full `cargo test -p paperclip-runner-core`.
- `pnpm --filter @paperclipai/paperclip-runner exec tsc --noEmit` and
server `tsc --noEmit` pass.
- Pack `contractDigest` is unchanged: `sha256:3164357d…` before and
after (`scripts/contracts/contract-fingerprints.mjs`).
- Live proof: the wave command above on this branch. Results follow in a
comment.

**No duplicate provider work.** The runner starts a new turn only when
Codex's own history proves the cut-off start created nothing. The old
command is never executed again (the runner refuses an `indeterminate`
command). A recorded turn that Paperclip never settled keeps the run
failed closed.

## Risks

- Low-medium risk. The runner change runs only while it restores a
session whose saved state has an ambiguous turn start and no active
turn. Before this change, that run could never continue.
- The proof relies on Codex recording a turn in the thread before it
runs anything. If Codex accepted the request but died before it wrote
the turn, the runner treats the start as not created and starts the turn
once. Codex writes the user turn to the thread when it starts the turn,
before it asks the model for output, so no model output or tool call is
expected without a recorded turn.
- If the thread's newest turn is not in the runner's settled set (for
example after a provider identity epoch rollover forgot older
identities), the runner keeps failing closed, as today.
- **Open decision (not changed here):** when Codex did record the
cut-off turn, it may have done work. This pull request keeps the run
failed closed and does not replay or adopt that turn. Continuing is a
person's decision through Retry or a new message.
- The server change trusts the executor's own error text
`runner_state_identity_mismatch: prior_owner_*`, together with the
absence of every launch and runner event. A run that requested a launch
keeps every stop proof.
- **Runner release:** the pack `contractDigest` does not change. Local
runners use the runnerd built with the app, so the fix applies with this
release.
- Not covered: the related `authority_indeterminate` variant in the same
wave (t04, Q1) is a separate cause; its Retry already worked in that
wave.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #20477, #20482

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 03:49:22 -06:00
DottaandClaude 914e3ae14f fix(runner): settle ACPX turns when the cancel reply is lost (#20498)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Claude agents run through the native Runner and the ACPX sidecar
> - A saved question or a board Stop cancels the active turn with
`turn.cancel`
> - The Runner waited 30 s for the sidecar's reply to that cancel, and
the Runner serves controller commands one at a time
> - When that reply was lost, the controller's shutdown commands timed
out first, so the run failed and the task needed a human Retry
> - This pull request bounds that wait, keeps the transport open, and
lets the provider's own turn terminal settle the turn
> - The benefit is that a lost cancel reply on Claude settles the same
way it already does on Codex (#20336)

## Linked Issues or Issue Description

Refs #20336 (Codex equivalent: settle a saved handoff after verified
shutdown). Related, no overlap: #20477, #20482, #20483.

**What happened?**

The restart-chaos harness dropped the ACPX sidecar's reply to the
Runner's `turn.cancel` (fault PV2, target `interrupt_ack`).
A Claude task that saved a question (Q1) failed with "The waiting state
is saved; confirming shutdown of the previous execution is still
pending". Its next attempt failed `native_session_cleanup_quarantined`.
A Claude task stopped by the board (S1) ended `blocked`. Retry was
accepted but started no run.

Cause, in
`packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs`
(`interrupt_turn`, master line ~800): the cancel used the ordinary
sidecar request with the 30 s bound from `acpx_provider_backend.rs:539`.
While it waited:
- `turn.stop` and `runner.suspend` stayed queued. The controller's 15 s
governed drain plus its 10 s close grace ran out first (`runner did not
durably suspend before checkpoint`, suspend command `pending`).
- At 30 s, `fail_closed` killed the sidecar and dropped the provider's
buffered turn terminal. The poll path then recorded `turn.failed`
(`acpx_provider_transport_failed`) and `provider_exit_unconfirmed`
(`acpx_provider_backend.rs:2716-2738`). `turn.stop` returned
`already_settled`, and `runner.suspend` refused with `ACPX provider
lifetime cleanup is not yet proven`
(`acpx_provider_backend.rs:2613-2621`).

Codex passes the same fault because its Runner does not block on the
interrupt reply.

**Expected behavior**

A lost cancel reply must not fail the run. The saved question must
settle as `succeeded`. A board Stop must end the run as cancelled with a
proven suspension, so the next turn can start.

**Steps to reproduce**

`node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts
--env local --adapters claude --workloads Q1,S1 --faults PV2
--provider-fault-message interrupt_ack --concurrency 1 --tasks 4`
Failing wave on master: `w-20261110T060120-s207` (t04, t05).

**Deployment mode**

Native Runner with the Claude ACPX profile, local. Hosted sandboxes run
the same Runner code.

## What Changed

- `AcpxSidecarTransport::request_turn_cancel`: sends `turn.cancel` and
waits only to a caller deadline. On a missing reply it keeps the
transport usable and retains the request ID, so a late reply is skipped
in both the response and the event-poll paths. A rejected reply and
transport errors keep their existing fail-closed result.
- `AcpxProviderSession::interrupt_turn`: waits at most 5 s (or the
shorter configured request bound) for the cancel reply. On a missing
reply it retires the runtime from warm reuse and returns. The provider's
turn terminal still arrives through polling, and `turn.stop` still reaps
a provider that never stops.
- `AcpxProviderSession::deliver_tool_result`: a board Stop can land
while a semantic tool runs. The session's own `turn.cancel` then retires
the call in the sidecar before its result arrives, and the turn terminal
may still be buffered. The sidecar rejects the result as stale
(`ACPX_TOOL_CALL_STALE`). When this session sent `turn.cancel` for that
turn, the Runner now keeps the durable result, like a result for an
already closed turn. Every other rejection still fails closed.
(`AcpxSidecarTransport::request_tool_resolve` reports whether a
rejection was the stale-call case.)
- A repeated interruption while a cancel is unanswered sends nothing and
returns at once (review finding).
- `fake-acpx-sidecar`: new `--cancel-reply lost|late` flag. After that
cancel, `tool.resolve` is rejected as stale. The provider ends the turn,
and the reply is dropped or written just before the next request.
- Three regression tests (below) and a short section in
`packages/paperclip-runner/docs/durable-recovery.md`.

No TypeScript, server, schema or protocol change. The normal start path
is unchanged; nothing waits before turn start.

## Verification

Red/green (red runs used the master versions of
`acpx_provider_session.rs` and `acpx_sidecar_transport.rs`):
- `cargo test -p paperclip-runner-core --test acpx_warm_attachment
a_lost_cancellation_reply_still_settles_the_turn_and_suspends_the_runner`
- Red: after 30.4 s, `ACPX turn interrupt failed: ACPX sidecar request
timed out at turn.cancel` (the live error).
- Green: `turn.interrupt` returns in about 5 s, then `turn.interrupted`
and a `cancelled` `run.terminal` (no `turn.failed`), `turn.stop`, and
`runner.suspend` completes.
- `cargo test -p paperclip-runner-core --test acpx_provider_turns
lost_cancellation_reply_keeps_the_turn_terminal_and_the_transport`
  - Red: `interrupt_turn` fails at the request bound.
- Green, for a lost reply and for a late reply: the terminal is polled,
warm reuse is refused, and the next request (`session.close`) succeeds
after the late reply arrives. A late reply causes no second settle and
no second prompt.
- `cargo test -p paperclip-runner-core --test acpx_provider_turns
a_tool_result_for_a_call_retired_by_our_cancellation_does_not_fail_the_transport`
- Red (guard disabled): `ACPX sidecar command tool.resolve was rejected
(retryable=false, classification=provider_tool_call_retired)`, the same
error as live wave task t03 below.
- Green: the result is kept, the pending tool clears, and the turn
terminal is polled after it.
- `cargo test -p paperclip-runner-core` (whole crate): all pass.
Existing `turns-tool-resolution-rejected` coverage still fails closed
when no cancel was sent. `cargo fmt --check` passes.
- Pack contract digest: unchanged (`sha256:3164357d…` before and after,
from `scripts/contracts/contract-fingerprints.mjs`).
- Live wave 1, `w-20261110T072706-s211` (commit `d91bff1e9f`, the first
fix only), `--adapters claude --workloads Q1,S1 --faults PV2
--provider-fault-message interrupt_ack --tasks 4`: 3 of 4 pass. Both Q1
tasks pass (the asking run `succeeded`, no failed run, answer in the
brief). One S1 passes. S1 task t03 failed: Stop landed during a
`write_task_document` call and hit the stale tool-result error above.
The second commit fixes that case.
- Live wave 2, `w-20261110T082702-s212` on head `76dd57515d`, same
flags: every task passes R1–R6 and U1. Both Q1 tasks pass with the fault
applied (asking run `succeeded`, no failed run). S1 t01 passes with the
fault applied (Stop took effect, task `done`, no Retry). S1 t03 fails
only `provider_fault_applied` (`applied=0`): the agent finished before
the Stop window, so nothing was injected and the task proves nothing
either way.
- Live wave 3, `w-20261110T091057-s213` on head `76dd57515d`, same
flags: the same result. t01 (S1), t02 and t04 (Q1) pass with the fault
applied; t01's Stop took effect (`status=cancelled`), task `done`, no
Retry. t03 again fails only `provider_fault_applied` (`applied=0`, the
run finished before the Stop window); all its R1–R6 and U checks pass.
- Across waves 2 and 3 on the final head: 6 of 6 tasks with the fault
applied pass (4 Q1, 2 S1). No product rule failed. The two misses are
harness timing (no injection), not product failures.

## Risks

- A healthy cancel reply that takes longer than 5 s now retires the warm
runtime. The next turn starts a fresh provider process for that session.
The observed healthy reply time is about 1 s.
- If the provider never stops and the reply is lost, the turn stays
active until `turn.stop` terminates the provider and proves the lifetime
fence. If that proof fails, the existing settlement-pending and
quarantine states are unchanged; this PR adds no success path without
proof.
- Runner binary change: hosted Daytona images need a normal image
rebuild to pick up the fix. The pack `contractDigest` does not change,
so existing image packs stay reusable.

## Model Used

- Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code
with tool use (shell, file edit, code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 03:17:14 -06:00
DottaandClaude 3e572475c8 fix(native): re-attach an answer run whose run.attach a restart cut off (#20477)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person answers an agent's saved question, Paperclip starts an
answer run. For a per-turn session, that run starts a new runner and
sends `run.attach` to bind the saved conversation to the run
> - The runner journals each command before its effect. A restart that
kills it inside that window leaves `run.attach` with an unknown outcome.
The restored runner settles it as `execution_indeterminate` and never
executes it again
> - The controller's same-run resume waited on that old command, so
every resume failed with "PRP command run.attach failed". The answer
never reached the agent. A Retry or a stall followed
> - A kill during the attachment's provider launch can also leave a
startup fence that no later runner retires, so every later command fails
with "provider startup ownership remains unadmitted"
> - This pull request attaches again with a new command when nothing
after the cut-off attachment started provider work. It also lets the
runner retire that orphaned startup fence
> - The benefit is that an answer given just before a restart continues
on its own

## Linked Issues or Issue Description

**What happened?**
Local restart-chaos journey wave `w-20261110T054406-s108` on master
`2c2186706e`. Command: `--adapters codex --workloads Q1,Q2 --faults K1
--triggers answered --tasks 4 --concurrency 2 --seed 108`. Result: 1 of
4 passed.
- t02 (Q1, K1 2.7 s after the answer). Answer run `0b357a1f` resumed
after the restart. Each resume failed with
`native_session_recovery_failed: PRP command run.attach failed:
execution_indeterminate`. Then "Automatic recovery stopped", and the
task needed a Retry.
- t01 (Q2, K1 1.1 s after the answer). Answer run `82c60af9` failed the
same way on three same-run resumes. Its provider state kept `lifecycle:
"prepared"` with a `spawned` startup attempt for `run.attach`, and that
provider process was gone. Each later drain and suspend failed with
"provider startup ownership remains unadmitted". The successor run then
refused with
`retirement_refused=journal_commands_unsettled(run.attach.failed)`. The
accepted Retry created no run (`retry_no_new_run`).

The earlier wave `w-20261110T033426-s108` hit the same window at
different points. It showed `authority_indeterminate` (Q1) and an
ambiguous `turn.start` stall (Q2). See Risks for what this pull request
does not cover.

**Expected behavior**
An answer run that a restart interrupts before it starts provider work
resumes by itself. It does not need a Retry.

**Steps to reproduce**
1. Run the wave command above on master.
2. Or run `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/live/runnerd-codex-transport-cut-attach.test.ts` on master. Both
cases fail with the error from the wave.

## What Changed

- `runnerd-codex-transport.ts`: the same-run resume waits on the
recovered `run.attach` as before. If that wait fails, the new
`cutOffRunAttachmentPayload` checks the journal. It queues one new
`run.attach` with the same payload, and waits on it, only when all of
these are true:
- the latest `run.attach` has an unknown outcome (`indeterminate`, or
`failed` with `execution_indeterminate`);
- it is a cold attachment (no `paperclipNextAuthority`), with no warm
transition and no staged attachment;
- every later command is recovery maintenance, as the control plane's
shared `isRunnerRecoveryMaintenanceCommand` defines it
(`session.snapshot`, `session.goal.get`, `runner.drain`,
`runner.suspend`, `runner.shutdown`, `turn.stop`);
- no committed event shows provider work (`turn.accepted`,
`turn.started`, `item.started`, `tool.execution.started`,
`semantic_tool.input`, `runtime_request.created` or
`mcp_app.tool_input`).
  Any other failure is thrown unchanged.
- `provider_backend.rs` (Runner): `retire_orphaned_startup_attempt` also
treats a `prepared` state with a saved thread and a nonzero process
generation as restorable. `attach_run` already documents this state as
the settled checkpoint that `run.attach` restores. The other fence rules
do not change. The fence is retired only before the spawn, or when the
recorded process and its group have exited.
- Tests:
- New real-runnerd test `runnerd-codex-transport-cut-attach.test.ts`. A
real runner and the fake Codex server run an asking run, then a cold
answer-run attachment. The test kills both processes and rewinds both
journals to "`run.attach` journaled, outcome not recorded", then
resumes. One case has a startup fence and one does not.
- A unit test for `cutOffRunAttachmentPayload`. It covers each
maintenance command, and the cases where it must not re-attach (a
definite answer, a pending command, a warm or staged attachment, any
work command, a turn event).
- The Rust orphaned-startup test also covers `prepared`. A new test
shows that a `prepared` state with no saved thread keeps its fence.
- `docs/durable-recovery.md`: describes both behaviors.

## Verification

- Red/green, real runnerd:
- On master, `runnerd-codex-transport-cut-attach.test.ts` fails both
cases with `PRP command run.attach failed: ... execution_indeterminate`,
the error from the wave.
- With only the transport change, the case without a fence passes. The
fence case fails with `provider startup ownership remains unadmitted`,
the t01 error.
  - With both changes, both cases pass.
- Red/green, Rust: `cargo test -p paperclip-runner-core --lib orphaned`
fails `restart_retires_a_spawned_startup_whose_provider_process_exited`
for `prepared` without the runner change. With it, all 10 tests pass.
- Related transport suites, 8 files and 387 tests, pass:
`runnerd-codex-transport`, `-attachment-recovery`, `-cold-restore`,
`-warm-attachment`, `-release-resume`, `-cut-attach`,
`runnerd-restart-continuation` and `runnerd-saved-continuation`. `tsc
--noEmit` for the Runner package passes.
- Live proof: the same wave command on this branch. Results follow in a
comment.

**No duplicate provider work.** The new attachment is sent only when the
journal proves that no provider work followed the cut-off one. No
command after it can start or steer work: only the shared
recovery-maintenance commands are allowed, and none of them starts a
turn. No turn, item, tool-input or runtime-request event is committed.
`run.attach` itself starts no turn. In the Codex and ACPX runners,
`attach_run` refuses to attach unless the session is settled. That means
no active or ambiguous turn, no pending tool call and no live active
goal. The old command is never executed again, because the runner
refuses an `indeterminate` command. The new attachment is a new command
with a new ID.

## Risks

- Low-medium risk. The new path runs only after a same-run resume has
already failed on an unknown-outcome attachment. Before this change,
that run could not continue.
- If an attachment's partial effect had changed provider state, the
runner's own settled-session check refuses the new attachment. The run
then fails as it does today.
- **Runner release:** the Runner change does not change the pack
`contractDigest`. It is `sha256:3164357d…` before and after
(`scripts/contracts/contract-fingerprints.mjs`). Local runners use the
runnerd built with the app, so the fix takes effect with this release. A
remote image pack (for example Daytona) that was built before this
change keeps the old fence rule until a new image or pack is built and
qualified. No release or pin bump is part of this pull request.
- **Mixed versions during an upgrade:**
- New server with an old runner: the server re-attaches. A run without a
startup fence continues. With a fence, the old runner refuses the new
attachment, the same failure as today.
- Old server with a new runner: the runner retires the fence, but the
old server still fails on the unknown-outcome attachment, as today.
- Neither mix adds a new failure. Saved state from older builds needs no
migration.
- Not covered here, and still open:
- A `turn.start` with an unknown outcome where the resumed thread has no
active turn (wave `w-20261110T033426-s108` t03). The runner deliberately
fails closed for that case.
- A warm-runner attachment that may already have been delivered
(`authority_indeterminate`).
- A Retry that waits on `process_identity_missing` for a run that failed
before any process started. A separate pull request will cover it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #20146, #19784, #20148

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 02:20:14 -06:00
Devin FoleyandPaperclip 549ca46dae fix(runner): retain Claude GitHub broker settings through the Rust sidecar (#20463)
## Thinking Path

> - Paperclip lets people manage AI agents for work.
> - Shared agents use the instructing person's GitHub identity for
managed commands.
> - Native Claude starts through the Rust daemon and a TypeScript ACPX
sidecar.
> - The Rust launch filter kept the command launchers but removed their
broker transport.
> - Shell commands then reported `GitHub capability_missing`, although
connector tools worked.
> - This pull request retains that transport for managed Claude
launches.
> - Each command can acquire its current identity through the existing
broker checks.

## Linked Issues or Issue Description

**What happened?**

Native Claude could use GitHub connector tools but could not
authenticate `git` or `gh`. The Rust daemon removed the broker URL and
session/bridge capabilities before starting the ACPX sidecar.
Reconnecting the account did not repair this process environment.

**Expected behavior**

Managed shell commands use the current authorized GitHub identity. The
provider receives broker capabilities, while raw GitHub tokens remain
confined to individual commands.

**Steps to reproduce**

1. Start native Claude through the Rust Runner with managed GitHub
launchers.
2. Run `gh api user` and a private repository ref lookup through Bash.
3. Observe `capability_missing` and no request to the broker.

**Paperclip version or commit**

Reproduced from source at `2c2186706ef4bb352b662f9d8faebefe47fb52ac`.
The repair is based on `a5e01634f69f18dfe10a0bf709df02d9e6a2135b`.

**Deployment mode**

Native Claude through ACPX. Local acceptance used macOS arm64 and Node
25.6.1. The reported failure also affected a Linux sandbox.

Related: #18212 preserved launcher paths. #18850 repaired the TypeScript
filter. This change repairs the preceding Rust filter. The duplicate
search found no open repair for this boundary. ROADMAP.md already lists
managed personal GitHub identities.

## What Changed

- Retain the broker URL and both broker capabilities when Claude has a
nonempty managed launcher directory.
- Keep raw GitHub tokens, unrelated secrets, empty launcher
configurations, and sibling ACPX providers outside that exception.
- Exercise the regression through the public sidecar initialize protocol
in the existing Rust integration suite. Document the restart
requirement.

The Rust sidecar transport owns this filter. Credential resolution,
company access, approval, and run identity policies keep their existing
owners. No schema or API changes are needed.

## Verification

- Red/green: the subprocess regression failed on the missing
`PAPERCLIP_GITHUB_BROKER_URL` before the fix, then passed with all three
broker settings and the negative controls.
- Passed before moving the regression: 615 Rust core tests, with 3
existing ignored tests. After moving it into the integration suite, all
20 sidecar integration tests and the 9 remaining sidecar unit controls
pass. The moved regression was also run against the unfixed source and
failed on the missing broker URL.
- Passed: 85 TypeScript environment, launcher, sandbox, and
configured-environment controls. These include operation identity
changes, revocation, recovery, and exclusion of capabilities from
persisted environment data.
- Passed: Runner Rust typecheck and format check, plus the Runner build
through `node scripts/agent-preflight.mjs --skip-rust-debug`.
- Live before/after: real `claude-opus-5-5` through the actual Rust
daemon and built ACPX sidecar reproduced the failure with the old
binary. Neither shell command reached the broker. With the fixed binary,
`gh` identified the connected account and `git` read private repository
refs. A second turn retained a workspace file and authenticated again.
After the controlled broker denied access, the next command failed
without reusing the prior token. No GitHub writes were made.
- Live release cutover: saved a session with the unfixed daemon,
suspended it, and restored it with the repaired daemon on
`aa9f5b30614c011ebc3fe4509e11173b766c1fea`. The same session and
workspace marker survived. `gh` authenticated after restoration, then
failed when the controlled broker revoked access.
- Live acceptance used an Anthropic API credential and a loopback broker
with an authorized operator GitHub credential. This isolates the
repaired transport; it does not qualify the hosted subscription
credential or production grant resolver.
- The first PR review gate did not recognize the in-source Rust test.
The regression now lives in the existing integration test file, and the
same test-coverage gate passes locally.
- `pnpm verify` returned its documented exit 2 and required broad CI
coverage. The final head passed the full CI typecheck, build, and test
lanes.

- Greptile reviewed `64cca69b76073cc3185b28334465dee48fbadb29` at 5/5
with no actionable findings or open review threads. Its check passed on
that head. The same head is mergeable with no conflicts.

- Final head `64cca69b76073cc3185b28334465dee48fbadb29` passed [full PR
CI](https://github.com/paperclipai/paperclip/actions/runs/42761401396),
[Windows
Rust](https://github.com/paperclipai/paperclip/actions/runs/42761401117),
and [isolated Runner
compilation](https://github.com/paperclipai/paperclip/actions/runs/42761400973).
Typecheck, build, all test shards, Runner checks, browser checks, and
aggregate verification passed. No checks are pending or failed; five
conditional jobs were skipped.

## Risks

- The Claude sidecar now receives the intended broker capabilities. The
existing broker must continue to enforce active-run identity and current
connection access. Raw GitHub tokens remain excluded from the provider
launch environment.
- Previous-release checkpoint and profile formats carry forward
unchanged. A running provider must retire and restart normally to
inherit the repaired environment. No profile reset or data migration is
required.
- Hosted recovery requires delivery of the updated Runner binary. This
PR does not deploy or retry existing tasks. Live acceptance has not
exercised Linux sandbox confinement or a GitHub write.

## Model Used

OpenAI `gpt-6.1-sol` through Codex, with xhigh reasoning. Used
repository inspection, code editing, and shell execution. The context
window size is not exposed. Anthropic `claude-opus-5-5` was exercised as
the acceptance-test provider.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 21:42:46 -08:00
Nicky LeachandClaude Opus 5.5 ff4ba7d319 perf(runner): run launch artifacts in place from a trusted read-only root (#20428)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Rust Runner (`paperclip-runnerd`) starts provider sidecars and
proxies from qualified launch artifacts: a Node runtime and the scripts
it runs
> - Since #12699, Runner copies each artifact into a sealed memfd and
checks its SHA-256 on every verification. This defends against a path
swap or an in-place rewrite between the check and the exec
> - On a hosted agent computer (4 vCPU), the copy and digest of the 127
MB Node takes about 0.26–0.30 s warm and about 1.2 s right after a wake.
This is on the path from `session.open` to the sidecar spawn, and again
on every warm `run.attach`. Each Runner also holds about 125 MiB of
shmem for the copy
> - On that computer, the artifacts live on the Runner ROM: a read-only
erofs image that the deploy pins by digest. The threat that the memfd
defends against does not exist there. On local hosts the artifacts are
writable, so the memfd must stay
> - This pull request adds an explicit, path-scoped, default-off
`--trusted-launch-root <dir>`. Runner runs an artifact in place only
when its open descriptor proves that it is on an immutable mount under
that root. Everything else keeps the sealed, hashed snapshot
> - The benefit is a shorter `session.open` and `run.attach` on agent
computers, and about 125 MiB less shmem per Runner, with no change where
the flag is absent

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Launch-artifact verification in `paperclip-runnerd`
(`qualified_launch.rs`). Its callers are the ACPX `verified_transport`
(every `session_config`, including the park rebind) and the OpenCode and
Cursor Cloud proxy launches.

**Subsystem affected**
Rust Runner (`packages/paperclip-runner/runner/crates/runner-core`).

**Current behavior**
Every verification opens the artifact, copies all bytes into a sealed
memfd, and hashes them. Node is 127 MB. The cost is described in the
Thinking Path.

**Proposed behavior**
When the host passes `--trusted-launch-root /opt/paperclip/runner` and
the root passes its checks, Runner holds each admitted artifact open in
place and executes it through `/proc/self/fd/N`. It does not copy or
re-hash it. Without the flag, nothing changes.

**Reason and benefit**
See the Thinking Path.

Refs #12699 (introduced the sealed memfd). Refs #20423 (overlap, see
below).

## What Changed

- `qualified_launch.rs`: new `TrustedLaunchRoot` and
`set_trusted_launch_root` (process-global, set once at startup, like
`--task-memory-cgroup`).
- **Root checks at startup.** The path must be absolute, already
canonical (no symlink or `..`), a directory, and on a read-only,
exec-permitted erofs or squashfs mount. If a check fails, runnerd logs a
warning and keeps today's behavior.
- **Per-artifact checks, on the open descriptor.** After today's checks
(regular file, not a symlink, not group- or world-writable, open-time
identity), the descriptor must be a regular file on the root's device.
Its kernel path (`readlink /proc/self/fd/N`) must lie under the root by
path components, so `/opt/paperclip/runner2` does not match. Its mount
must pass `fstatvfs` (`ST_RDONLY` set, `ST_NOEXEC` clear) and `fstatfs`
(erofs or squashfs magic). If any check fails, Runner logs the reason
and uses the sealed snapshot.
- **Admitted artifacts** become
`VerifiedProcessArtifact::in_place_on_trusted_root` (crate-private). The
held file goes through the existing `inherited_artifact` path, so the
exec is `/proc/self/fd/N` and `PAPERCLIP_VERIFIED_RUNTIME_EXECUTABLE`
still names a descriptor that nested launches can reopen.
- New `verify_assigned_executable_launch_artifact` never uses the root.
`stdio_mcp.rs` now calls it, so host-assigned stdio MCP binaries always
keep the sealed snapshot.
- `paperclip-runnerd.rs`: parse `--trusted-launch-root <dir>`. Older
runnerd builds ignore unknown flags, so a host can pass it before every
Runner understands it.
- The trusted path is Linux-only. On macOS and Windows,
`TrustedLaunchRoot::open` always refuses.
- `tests/runner-e2e/SECURITY.md`: documents how the Kimi and Pi
provider-death fixtures behave under a trusted root.

### Decision: the trusted path drops the digest

The trusted path does not hash the artifact, not even once per process:
- The ROM image is pinned by digest at deploy time.
- On the agent computer, the Launcher computes the expected
`--acpx-sidecar-command-sha256` and `--acpx-sidecar-script-sha256` from
the same ROM files immediately before it starts Runner. A re-hash of an
immutable file can only repeat that result.
- One hash per process would not help `session.open`, because each task
starts a new Runner process.
- The mount check is the guard: erofs and squashfs have no write path,
so the bytes cannot change while the descriptor is open.

### Fixtures that assert the sealed memfd

- Untrusted-mode assertions are unchanged. The fixtures do not pass the
flag.
- Kimi (`kimi-provider-fault.py` `checked_parent_descriptor`): with a
trusted root, the parent descriptor is the ROM's Node file (one link).
The existing linked-descriptor branch handles it: same inode as
`/proc/<pid>/exe`, and the complete bytes hashed against `nodeSha256`.
That branch never accepts the file as a memfd. The existing test
`test_linked_descriptor_keeps_regular_single_link_checks_without_memfd_claim`
covers it.
- Pi provider-death admission binds `/proc/self/fd/3|7` to the native
distribution snapshot's Node. The TypeScript ACPX closure copies that
file whether or not a trusted root is set, so the flag does not affect
it.

### Overlap with #20423

#20423 (draft) moves the same copy and hash to Runner boot and reuses
the result. Where this flag is set (Cloud agent computers), there is
nothing left to precompute, so #20423 is mostly redundant there. #20423
still helps hosts without a trusted root (local and sandbox Runners).
This PR does not change #20423.

## Verification

All runs are on Linux (aarch64, Rust 1.97.1, in a container):
- `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`
- `cargo check --manifest-path runner/Cargo.toml --locked --workspace`
- `cargo test -p paperclip-runner-core --lib -- qualified_launch
trusted_root commonjs`: 12 passed. The new tests are:
-
`an_artifact_under_an_immutable_trusted_root_runs_in_place_without_a_digest`:
trusted-root hit, with no memfd, the same inode as the source, and no
re-hash.
- `an_artifact_outside_the_trusted_root_keeps_the_sealed_snapshot`: a
path outside the root is sealed, and a wrong digest still fails.
- `a_sibling_with_the_root_as_a_name_prefix_is_outside_the_root`:
`rom2/` does not match `rom`.
- `a_root_that_is_not_read_only_falls_back_to_the_sealed_snapshot`: the
real probe refuses a writable root at startup. A per-artifact mount
failure falls back, and a wrong digest still fails.
  - `the_real_mount_probe_rejects_a_writable_filesystem`
- `symlink_and_parent_escapes_from_the_root_fall_back`: a directory
symlink out of the root, a `..` path out of the root, and a
final-component symlink (still refused).
  - `a_root_must_be_absolute_canonical_and_a_directory`
- `without_a_trusted_root_launch_artifacts_are_sealed_and_hashed`: flag
absent, behavior unchanged.
  - `host_assigned_executables_ignore_the_trusted_root`
-
`trusted_root_artifacts_execute_the_held_file_and_reopen_it_for_nested_launches`:
Runner execs from the held file, and a child re-execs
`$PAPERCLIP_VERIFIED_RUNTIME_EXECUTABLE`. Both report the source file as
`/proc/self/exe`.
- `cargo test -p paperclip-runner-core --test stdio_mcp --test
process_supervisor`: 4 and 8 passed.
- Ad hoc, not committed: a privileged container with a real loop-mounted
squashfs. The real `fstatvfs`/`fstatfs` probe admitted a `ro` mount and
refused a `ro,noexec` mount. The unit tests replace only that mount
probe, because CI cannot mount filesystems.
- Not verified: a run on an agent computer with the real erofs ROM, and
the latency and memory savings there. The Cloud Launcher change that
passes the flag is a separate follow-up.

## Risks

- **This change needs review from a security owner before merge.** It
removes the sealed copy and the per-launch digest for admitted
artifacts. That relies on the host's assertion that the root is a
digest-pinned immutable image, plus the descriptor checks above.
- **Misconfiguration fails closed.** A wrong, writable, `noexec`,
non-canonical or missing root only loses the speedup, and runnerd logs a
warning. There is no global "trust everything" switch, and Runner never
selects a root from the filesystem it finds.
- **Residual trust.** On an admitted file, Runner no longer detects a
ROM image that differs from what the launch profile expected. On the
agent computer, the Launcher derives the expected digests from the same
ROM. Another host that passes the flag must pin its image the same way.
An attacker with root in the guest (who could write the backing block
device or remount) is out of scope; such an attacker can already control
runnerd.
- **Allowlist.** Only erofs and squashfs are accepted, because they
cannot be written through any mount. A read-only bind mount of a
writable filesystem is refused. A ROM that changes to another filesystem
type will silently fall back to the sealed path (slower, still safe).
- No migration. Behavior is unchanged when the flag is absent. Older
runnerd builds ignore the flag.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, container builds). Extended reasoning was
on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 17:46:33 -08:00
Nicky LeachandClaude Opus 5.5 5858cef8f9 perf(runner): poll the provider right after an ACK releases it (#20374)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner (`paperclip-runnerd`) sends provider events to the
controller over one authenticated socket
> - Its main loop reads one control frame, then polls the provider only
after a read finds no frame
> - After the controller ACKs the last sent event, the next read waits
the full 250 ms `RUNTIME_READ_TIMEOUT` before the Runner polls the
provider again
> - A Codex turn start needs one or two such rounds before the Runner
reads `turn/started`, so every Codex turn waits 250 ms or 500 ms for no
reason
> - This pull request reads only the frames that are already queued
after a consumed control frame, and then polls at once, when this
connection produced the unacknowledged output
> - The benefit is 250 ms less time to `turn/started` for each ACK
round, and faster event delivery after each ACK

## Linked Issues or Issue Description

This is an enhancement. The fields follow
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**
Codex turns wait for the Runner read timeout before the Runner sends
`turn/started`.

**Subsystem affected**
The Runner (`paperclip-runnerd`) durable loop in
`packages/paperclip-runner/runner/crates/runner-core/src/durable/`, for
the local Codex provider.

**Current behavior**
On a hosted test deployment (48 h, 114 Codex turns),
`runner.transport.started_event` for Codex has two modes:

| Codex turn | n | `started_event` p50 |
| --- | --- | --- |
| no provider cold start in the trace | 57 | 468 ms (all < 700 ms) |
| provider cold start in the trace | 57 | 918 ms (51 of 57 ≥ 700 ms) |

For Claude (ACPX) turns, the p50 is 90 ms. ACPX returns `turn.started`
with the command result. Codex reads `turn/started` from the provider in
a later poll.

**Proposed behavior**
The Runner polls the provider as soon as the controller ACK releases the
poll gate. The time to `turn/started` then follows the controller round
trip, not the read timeout.

**Reason and benefit**
- Real Codex (`codex app-server` 0.157, local) sends `turn/started` 2–16
ms after its `turn/start` response. The provider is not the cause.
- `runner.transport.started_event` starts when the `turn.start` command
result arrives. It ends when the controller commits `turn.started`. In
this time the Runner sends `turn.accepted`, waits for its ACK, waits 250
ms, polls, and sends `turn.started`.
- On the first turn after a provider launch, the first poll returns only
the launch diagnostic that the executor retained. A second ACK round and
a second 250 ms wait come before the poll that reads `turn/started`.
This is the difference between the two modes above (about 450 ms: 250 ms
wait and one event round).
- So 250 ms (warm turn) or 500 ms (first turn) of each Codex
`started_event` is the read timeout. The rest is one or two controller
event rounds (about 200 ms each on that deployment), less than 20 ms of
Codex time, and the 10 ms pump interval of the TypeScript transport.
- The expected benefit is 250 ms less per warm Codex turn start and 500
ms less for the first turn after a provider launch, plus faster Codex
event delivery after each ACK.

**Breaking changes**
None. No protocol, schema, or controller change.

## What Changed

- `AuthenticatedTransport::receive_queued_json` reads with a 1 ms
timeout and then restores `RUNTIME_READ_TIMEOUT`.
- New `CommandExecutor::provider_poll_is_live` hook (default `false`).
The local Codex executor returns `true` only when it owns a running
provider with no open startup attempt, so a poll cannot restore or
launch a provider. The native executor forwards it for the local Codex
facade only.
- In the steady-state loop, after a consumed control frame, the Runner
uses `receive_queued_json` when it has sent output that this connection
produced (a command or a provider poll) and the executor reports a live
provider. If no frame is queued, it polls the provider at once.
- When a provider poll returns no output, or when no frame was consumed,
the Runner uses the 250 ms read as before. A 1 ms read follows only a
consumed frame, so an idle loop does not spin.
- The replay boundary is the highest source sequence before the
welcome's pending commands run. A fresh command delivered in the welcome
counts as this connection's output. Replayed commands only resend their
stored results.
- Replayed output from an earlier connection or process, ACPX, and the
other providers keep the 250 ms read. The controller can still send its
next command before a poll restores a provider. (Without the
live-provider gate, six ACPX source-recovery tests in
`runnerd-acpx-source-reconciliation.test.ts` failed: the fast poll
restored the ACPX sidecar before the next recovery command.)
- Control frames that are already queued behind the ACK are still read
and executed before the provider poll. The ACK gate in
`poll_executor_events_after_controller_ack` is unchanged.

## Verification

- `cargo test -p paperclip-runner-core --lib durable::` (121 passed).
Five new tests in `src/durable/tests/ack_release_poll.rs`, included by
the transport test module like `hosted_retention.rs`:
-
`acknowledged_output_polls_the_provider_without_waiting_out_the_read_timeout`:
the provider poll follows the ACK read in less than
`RUNTIME_READ_TIMEOUT`. Without the change it fails (the poll waited
280–356 ms).
- `control_queued_behind_an_ack_runs_before_the_next_provider_poll`: a
command sent in the same write as the ACK executes before the provider
poll.
-
`welcome_command_output_polls_the_provider_without_waiting_out_the_read_timeout`:
the output of a fresh command in the welcome also releases the poll at
once. With the replay boundary taken after the welcome commands, it
fails.
- `replayed_output_ack_keeps_the_read_timeout_before_the_provider_poll`:
after a reconnect, a command sent 50 ms after the ACK of replayed output
executes before the provider poll. Without the replay condition it
fails.
-
`restorable_provider_ack_keeps_the_read_timeout_before_the_provider_poll`:
when the executor does not report a live provider, a command sent 50 ms
after the ACK executes before the provider poll.
- `npx vitest run src/live/runnerd-` in `packages/paperclip-runner` with
the new debug `paperclip-runnerd`: run with `--no-file-parallelism`.
Before the welcome-boundary fix: 576 passed, 3 skipped. After it: 575
passed, 3 skipped, and 1 failure once (`stops a lost runnerd's provider
and escaped tool job from its lineage journal (journal=true)`, a SIGKILL
timing test) while `cargo test` ran at the same time.
`runnerd-codex-transport.test.ts` then passed 293 of 293 three times
with this build, and three times with the base build.
- Local before and after measurement with the real Runner, the
TypeScript transport, and the fake Codex app-server (8 turns, 2 runs
each, debug build):

  | span | before p50 | after p50 |
  | --- | --- | --- |
| `runner.transport.started_event` | 377 ms (first turn 682–695 ms) |
119–122 ms (first turn 117–119 ms) |
| `runner.transport.turn_start` | 462–471 ms (first turn 775–780 ms) |
204–212 ms (first turn 247–251 ms) |

- The saving on the hosted deployment is not measured. From the analysis
above, the expected saving is about 250 ms for a warm Codex turn and
about 500 ms for the first turn after a provider launch.

## Risks

- Only local Codex gets the fast path. ACPX streaming keeps the old 250
ms wait after each ACK. A later change can opt ACPX in when its poll can
prove that it does not restore the sidecar.
- More provider polls during Codex streaming: a poll can now follow each
ACK at once instead of after 250 ms. Batches can become smaller, so the
provider state save for each batch can happen more often. Each event
already has its own durable save.
- A command that the controller sends more than 1 ms after its ACK can
now arrive after a provider poll. This was already possible after 250
ms. The change applies only when this connection produced the output, so
the provider is already active.
- If the Runner cannot restore the 250 ms read timeout, it treats the
socket as failed and reconnects. It does not continue with a 1 ms
timeout.
- No protocol, schema, or controller change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, code search, Honeycomb queries) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 15:38:58 -08:00
Nicky LeachandPaperclip 79bf733012 fix(runner): share the bounded ACPX cleanup proof (#20332)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Runner keeps a lifetime fence until the original ACPX provider
stops.
> - A recovered Runner must prove that fence is free before it clears
saved cleanup state.
> - Suspension and other recovery paths already wait for transient
listener holders. Recovered shutdown used one immediate probe.
> - That difference made parallel Rust tests fail after they dropped
their original listeners.
> - This pull request uses the existing bounded proof for recovered
shutdown and removes the repeated test retry.
> - The benefit is one cleanup rule, with the same fail-closed proof and
no new work on normal task dispatch.

## Linked Issues or Issue Description

Fixes: #20210

Thanks to @genius-0963 for the test-only fix proposed in #20321. That
related alternative adds a test-only shutdown retry. This change applies
the existing bounded cleanup proof in the shared shutdown path and
removes the same retry pattern from native-provider integration tests.
It also tests a single shutdown call while the original quorum is still
held.

## What Changed

- Use `await_provider_lifetime_fence` for a recovered executor with no
live session and unconfirmed provider exit. Delete the unused one-shot
helper.
- Keep the acquired quorum held through the existing durable state
write. A failed proof leaves `providerExitUnconfirmed` set.
- Give the shared timeout message wording that applies to shutdown and
suspension.
- Extend the recovery test with a worker-start signal. Hold the original
listeners until the single shutdown call is waiting, then release them
and assert successful cleanup without a provider session.
- Use the same bounded proof in the retired-quorum fixture test. Remove
the duplicate retry loop from the native-provider test helper.
- Document the bounded recovery cleanup wait.

## Verification

- macOS arm64, Rust `1.97.1`, source base `3a2f71db31`, head
`368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b`, isolated worktree. Only
targeted checks ran locally.
- `cargo fmt --manifest-path packages/paperclip-runner/runner/Cargo.toml
--all -- --check`: passed.
- `cargo test --manifest-path
packages/paperclip-runner/runner/Cargo.toml --locked -p
paperclip-runner-core --lib acpx_provider_backend::tests`: 45 passed on
the rebased head (5.21 seconds). The initial head also passed five more
parallel-test runs (45 tests per run), with no failure.
- `cargo test --manifest-path
packages/paperclip-runner/runner/Cargo.toml --locked -p
paperclip-runner-core --test native_provider_backend acpx_`: 14 passed
on the rebased head (9.43 seconds). These tests use the real Rust
executor and controlled sidecars.
- Negative control: restore only shutdown's original one-shot proof. The
new regression fails at `shutdown must wait while the original lifetime
still holds a quorum`. Restore the fix and the affected module passes.
- Independent parent verification on current commit `368fc814f7`: the
three exact recovery, held-quorum, and retired-quorum tests passed (3
tests, 5.06 seconds). Independent source review confirmed the unchanged
three-file scope and found no actionable issue.
- The active-turn recovery test still holds a live quorum through the
five-second bound. It checks that shutdown refuses cleanup, saved state
remains unconfirmed, and the provider never starts. It then releases the
quorum and confirms cleanup.
- Standard Greptile scored current commit
`368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b` 5/5 with no review threads.
- Initial GitHub general-server shard 7 failed in unchanged
`remote-workspace-nested-verification.test.ts`, at its 15-second wait
for a child-process import checkpoint (expected one row, got zero; both
Vitest attempts). [Failed
job](https://github.com/paperclipai/paperclip/actions/runs/42668667033/job/129339371815).
That fixture runs Node, PostgreSQL, Git, and storage migration; it does
not run the native Runner or the changed Rust cleanup. The cause of the
checkpoint timing miss remains unknown. The focused retry passed on the
unchanged source ([successful
retry](https://github.com/paperclipai/paperclip/actions/runs/42668667033/job/129347569863)).
This proves recovery, not the cause of the original timing miss. GitHub
kept the reusable-workflow aggregate failed after that job-only retry
and rejected both a final-gate retry and a failed-job retry. A
subsequent run-level retry materialized only the gate and aggregate
jobs; its aggregate inputs reported prior jobs as `abandoned`, so it
failed without a new test failure. A new `full-ci` label event requests
a fresh workflow graph on the same commit. The reason for those
rerun-state results is not established.
- The fresh `full-ci` run
[42674033208](https://github.com/paperclipai/paperclip/actions/runs/42674033208)
then lost several runners mid-job. The maintained [runner-loss recovery
42677096525](https://github.com/paperclipai/paperclip/actions/runs/42677096525)
classified all four failed jobs as runner loss: typecheck, E2E shards 2
and 8, and server shard 4. The evidence was a shutdown log or
lost-communication annotation. It retried the failed jobs together;
canceled sibling jobs were also requeued. This is runner-interruption
evidence, separate from the earlier checkpoint assertion.
- The maintained recovery attempt again failed the same
partial-checkpoint assertion in unchanged
`remote-workspace-nested-verification.test.ts`, twice ([job
129361525237](https://github.com/paperclipai/paperclip/actions/runs/42674033208/job/129361525237)).
Neither checkpoint-failure job logged runner shutdown or lost
communication; their resource summaries reported no significant
constraint. The child phase at the timeout remains unknown. No further
blind retry or TypeScript change is included here. Related
startup-handshake work exists in #20339; its controlled-delay evidence
establishes a deadline-boundary defect but does not establish the phase
of these original CI failures.
- Rebased onto current master after upstream #20339 merged. That
separate change fixes a demonstrated child-startup/checkpoint-deadline
boundary. It does not prove which child phase caused the earlier
zero-checkpoint assertions. This PR still changes only its original Rust
and documentation files. Fresh GitHub CI on the new base will establish
recovery, not retrospective cause.
- Full local tests and a local release build were not run. Current-head
[PR CI
42681719411](https://github.com/paperclipai/paperclip/actions/runs/42681719411),
[Runner Windows
42681719803](https://github.com/paperclipai/paperclip/actions/runs/42681719803),
and [Docker Runner
42681719601](https://github.com/paperclipai/paperclip/actions/runs/42681719601)
all passed on `368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b`. The full check
list has 59 successes and 5 intentional skips. GitHub supplied broad
typecheck, build, and optimized Rust coverage. This is new-base
recovery; the cause of the original checkpoint assertion failures
remains unproved.

## Risks

- A genuinely unconfirmed original provider can now hold recovered
shutdown for up to the existing five-second proof deadline. The proof
returns immediately when two candidate ports are free.
- The changed branch requires both no in-memory provider session and
unconfirmed saved cleanup. Live-session shutdown and ordinary dispatch
keep their existing paths. Outer controller close deadlines remain
authoritative.
- State saved by the previous release carries forward unchanged. No
schema, launch identity, profile digest, protocol, or retry-budget
change.
- No live provider session or hosted deployment was tested. The tests
cover cleanup proof, refusal, durable state, and controlled provider
execution.

## Model Used

- OpenAI GPT-6 through Codex, with reasoning, repository inspection,
code editing, and targeted test execution. The exact serving revision
and context window are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub references)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 11:36:44 -08:00
DottaandClaude Opus 5.5 97188f276c perf(runner): skip the turn-end provider restart in a per-turn Runner (#20312)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner runs each agent's provider (for Claude, an ACPX
sidecar) and keeps its session across runs.
> - #20190 restarts the ACPX provider when a turn settles, so a warm
follow-up does not pay the respawn on its critical path.
> - A per-turn Runner (`lifecycleMode: per_turn`, the default) exits
with its run. Nothing reuses the generation that restart prepares: the
next run starts a new Runner process and loads the session again.
> - In that mode the restart can only spawn a provider for teardown. On
a loaded host it can also fail, which was the cause of the closed
sessions that #20285 fixed.
> - This pull request skips the turn-end restart when the controller
launched the Runner as per-turn.
> - The benefit is that per-turn runs never spend a provider spawn they
cannot use, and never risk a restart failure at the end of their run.

## Linked Issues or Issue Description

Refs #20190, #20285

**What happened?**
A per-turn Runner restarted its ACPX provider after its turn settled,
although the Runner exits with its run. The next run launches a new
Runner process (`runner.process.launch mode=resume`, then
`provider.session.load`), so the restarted generation is torn down
unused. In an earlier A/B on a host with load average about 17–27, this
restart fired and failed, and the failure closed the session (fixed in
#20285).

**Steps to reproduce**
1. Run a Claude task with the default per-turn lifecycle.
2. Let the turn settle and the controller acknowledge its events before
it shuts the Runner down (more likely on a loaded host).
3. The Runner reaps the provider, spawns a new generation, and then
exits with the run, which reaps the new generation too.

**Expected behavior**
A per-turn Runner does not restart its provider after its turn. A warm
Runner keeps the #20190 restart. A later turn of the same run still
restarts the provider at its start.

## What Changed

- `DurableRunnerConfig` has a new `turn_scoped` field.
- `paperclip-runnerd` sets it from `--lifecycle-mode`, which the
controller already passes (`durable-prp-control-plane.ts` pushes
`--lifecycle-mode <mode>`), but which `runnerd` ignored until now. Only
`per_turn` sets it. `warm`, and launches without the flag (older
controllers), keep today's behavior.
- `NativeProviderCommandExecutor::prepare_next_turn` returns early when
`turn_scoped` is set. The lazy restart at the next turn start is
unchanged.
- Tests:
- New
`a_per_turn_runner_skips_the_turn_end_restart_and_restarts_a_later_turn_at_its_start`:
no new provider process and no sidecar command after the turn settles; a
later turn of the same run still restarts at its start
(`session.suspend`, `initialize`). It fails without the skip (2 process
starts instead of 1).
  - New `only_a_per_turn_lifecycle_is_turn_scoped` for the flag parsing.
  - The other `DurableRunnerConfig` literals set `turn_scoped: false`.

## Verification

- `cargo test -p paperclip-runner-core`:
  - `--lib`: 602 passed.
  - `--test acpx_park_preparation`: 22 passed.
  - `acpx_park_readiness`: 8 passed.
  - `acpx_warm_attachment`: 4 passed.
  - `acpx_provider_suspend`: 5 passed.
  - `acpx_provider_session`: 12 passed.
  - `acpx_provider_turns`: 29 passed.
  - `native_provider_backend`: 47 passed.
  - `durable_recovery`: 3 passed.
  - `--bin paperclip-runnerd turn_scoped`: 1 passed.
- `cargo build --workspace --all-targets` passes.
- Local live A/B with `ttft-baseline --adapters claude --samples 3
--restarts 0 --phases coldwarm`. Control is master after #20285
(7fe9345030); the other arm is this branch. Host load average was about
6–9.

| | Cold first activity p50 | Cold TTFT p50 | Warm first activity p50 |
Warm TTFT p50 | Result-accepted → exit |
  |---|---|---|---|---|---|
  | Control | 5.61 s | 8.86 s | 4.95 s | 5.00 s | 1.49–1.84 s |
| This branch | 6.72 s | 9.83 s | 5.52 s | 5.57 s | 1.45–1.63 s (one
cold run 3.64 s) |

- At this load the difference is within run-to-run noise. Both arms
passed 6/6 samples.
- A process sampler saw no short-lived provider process in either arm.
Every provider process lived 11–16 s, about one run. So at load ~8 the
per-turn Runner exited before the turn-end restart ran, and this change
had nothing to remove.
- The restart did run, and fail, in the earlier A/B at load ~17–27
(#20285). This change removes that path for per-turn Runners. I did not
reproduce that load level here, so this PR has no measured speed gain to
claim.

## Risks

- **Low risk.** The change only removes work in per-turn Runners. Warm
Runners and launches without `--lifecycle-mode` behave as before.
- If a per-turn run starts a second turn, that turn restarts its
provider at its start, as before #20190. That costs the respawn on its
critical path; per-turn runs normally have one turn.
- Remote launchers that build their own `runnerd` arguments and do not
pass `--lifecycle-mode` keep today's restart. That is the safe default.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 10:50:25 -06:00
DottaandClaude Opus 5.5 fd2e7482c0 fix(runner): stop a lost runnerd's provider and tools from a durable lineage journal (#20270)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A local native run uses one runner daemon (`paperclip-runnerd`).
runnerd starts the provider (for example OpenCode), and the provider
starts tool processes
> - When runnerd dies in the middle of a tool call, the provider and its
tools are re-parented. The only record of that process tree was in
runnerd's memory
> - Close then cannot prove that those processes stopped. It quarantines
the session, and the next attempt stops with
`native_session_cleanup_quarantined`: "Automatic recovery stopped /
operator recovery required"
> - Restarts and runner loss are product failures. Interrupted work must
recover by itself, and it must never get two writers in one workspace
> - This pull request makes runnerd journal its process lineage to disk,
and makes close stop and prove gone every journalled process after
runnerd is lost
> - The benefit is that a killed runnerd leads to an automatic, safe
retry within the normal attempt budget, instead of an operator stop

## Linked Issues or Issue Description

Refs #20259 (runner loss classification and the lineage sampler panic).
Refs #20260 (automatic continuation of an inert cleanup quarantine).

**What happened?**
A chat eval killed only the owned local `runnerd` process (SIGKILL)
during an OpenCode tool call
(`chat-features.runner-opencode.local.recovery.retry-exhausted`,
`recovery.explain-stall` and `recovery.source-action`). Attempt 1
recorded `native_runner_process_exited` with "Retry scheduled 1/3".
Attempt 2 stopped with `native_session_cleanup_quarantined`. The task
was blocked with no owner, and the next action was operator recovery.

**Expected behavior**
After runnerd is lost, the product stops the provider and tool processes
that runnerd started, proves them gone, and continues the run within its
retry budget. When it cannot prove that, it keeps the fail-closed
quarantine.

**Steps to reproduce**
1. Start a local native OpenCode (or Codex) run.
2. Let the agent run a tool that leaves the provider's session (for
example `nohup job &`).
3. Send SIGKILL to the `runnerd` PID only.
4. Close rejects with `runner did not durably suspend before
checkpoint`, and the next attempt fails with
`native_session_cleanup_quarantined`.

**Paperclip version or commit**
master at 4c15eea770

## What Changed

- `process_lineage.rs`: runnerd journals each supervised process's
lineage to `<runner state>/process-lineage/lineage-<runnerd
pid>-<n>.json`. The journal holds the root, every member, and every
group and session a member leads, each with its start time. The first
record is written when the provider is spawned, before any tool can run.
Later records are written by the existing 50 ms sampler only when the
record changes. A teardown that proves the lineage gone removes the
journal.
- `process_lineage/journal.rs` (new module): the on-disk format, the
write rules and the reaper `reap_journals`. The reaper refuses when
runnerd never journalled (no writer marker) or when runnerd is still
alive. It restores each journal into the same `LineageState` rules,
sends TERM and then KILL only to processes whose PID and start time
still match, and reports `verified` only when none remain. A journal
that it cannot read refuses the proof.
- Unsampled escapes: each record stores `sampled_through`, the newest
process start that its read listed. The value is refreshed at most once
a second while the membership is unchanged. The reaper refuses
(`unattributed_orphan`) when a same-user orphan (parent PID 1) that is
in no journal started after that point. Such a process can be a tool
that started after the last read, so the reaper neither signals it nor
ignores it.
- A failed journal write removes the writer marker. Removal needs no
free space. After that, no reap can verify from a record that may be
missing a member.
- `paperclip-runnerd`: durable mode enables the journal before any
provider starts. A new `reap-lineage --state-dir --runner-pid` mode
prints the report and exits 0 only when verified.
- `runnerd-codex-transport.ts`: close samples whether runnerd exited
unsuspended before close sent any signal. In that case, for a runnerd
that this transport spawned on this host, it runs the reaper from the
same executable. A verified report ends the session as a crash stop with
its durable record intact. Recovery then continues the run. Any other
result keeps the existing `NativeSessionCloseUnrecoverableError`
quarantine. Launcher-owned (remote) and adopted runners do not use this
path.
- `runnerd-codex-transport.ts`: close does not queue a durable
`runner.suspend` for a runnerd that it already knows is lost. Before,
that suspend stayed pending in the saved control-plane state. A later
runnerd on the same root then failed at startup with `terminal command
result acknowledgement changed its durable identity`. The shared
suspension barrier is unchanged for remote and adopted runners.
- `runnerd-codex-transport.ts`: a runnerd that exits during close (for
example because its provider snapshot is unreadable) is not a loss. It
keeps the quarantine.
- `runnerd-codex-transport.ts`: `#runnerHasExited` treated a
signal-killed runnerd as alive, because `exitCode` is `null` and only
`signalCode` is set. Close then spent its whole deadline trying to
suspend a dead runner. It now checks `signalCode` too.
- `fake-codex-app-server`: `--spawn-escaped-tool-job <pidfile>` starts a
tool job in its own session at turn start.
- `doc/execution-semantics.md`: describes the journal, the reaper and
the fallback.

## Verification

- Rust, `cargo test -p paperclip-runner-core --lib process_lineage` (14
pass) and `process_supervisor` (20 pass). New cases:
- `reaping_refuses_an_orphan_born_after_the_last_sample`: a tool starts
after the last read, starts a new session and exits, and leaves a
non-leader orphan job. The reap refuses and does not signal the job. It
fails without the guard.
- `a_failed_journal_write_withdraws_the_writer_marker` and
`an_unchanged_record_is_rewritten_only_to_advance_sampled_through`.
- `reaping_stops_a_dead_runners_provider_and_its_escaped_tool_job`: a
real provider and a real `setsid` tool job, with a dead writer. The reap
stops both, proves them gone and retires the journal and marker.
- `reaping_refuses_without_a_marker_or_while_the_writer_lives` and
`reaping_refuses_an_unreadable_journal`.
- `a_journalled_record_restores_the_same_lineage`: after a round trip,
the restored record still finds a re-parented job and its later child,
and ignores an unrelated process.
- TypeScript, live with the real debug runnerd: `stops a lost runnerd's
provider and escaped tool job from its lineage journal`.
- `journal=true`: SIGKILL runnerd mid-turn. The escaped tool job
outlives runnerd. Close resolves, the job is gone, and the report is
`verified`. Before the fix, close rejected with `runner did not durably
suspend before checkpoint` and the job stayed alive.
- `journal=false` (an older runnerd with no journal): close still
rejects, and the job is not signalled.
- `runner-lineage-reap.test.ts`: a report from an older runnerd, a
malformed report, or an inconsistent `verified` report never counts as
proof.
- Turn-start cost: the journal adds one single-process read and one
small file write when runnerd spawns its provider (about 1.5 ms in a
macOS debug build), once for each runnerd. Nothing is added for each
turn.
- Paid runner E2E,
`chat-features.runner-opencode.local.recovery.retry-exhausted`, on the
eval branch with this fix (runs 42637596571 and 42644400434). Before the
fix (run 42615758956), attempt 2 stopped with
`native_session_cleanup_quarantined`. Now each of the three runnerd
kills records `native_runner_process_exited`. Attempts 1 and 2 are
`retryable_failure`, and attempt 3 is `terminal_failure` with cause
`native_session_retry_exhausted` (attempt 3 of 3). The reaper ran after
the first kill and verified. The cell still fails one eval setup check.
The check waits for an *active* exhausted recovery action, but
`execution-recovery-resolution.ts` settles that action as resolved and
blocks the task (`preserve_without_replay_v1`). That is a separate
grader decision, outside this PR.

## Risks

- Medium. The change only affects a close where runnerd already exited
without suspending and without a signal from close. Every other close is
unchanged.
- The reaper signals processes by PID. Each signal re-checks the PID's
start time first, and a group or session is used only while its leader
PID still names the recorded process (the same rules as the existing
in-process teardown).
- A crash stop leaves the durable root unsuspended. The coordinator
already scheduled a retry for `native_runner_process_exited`. That retry
now runs instead of failing on the quarantine. The paid E2E shows that
attempts 2 and 3 start on the same run.
- The escape guard can refuse when an unrelated same-user process
becomes an orphan in the window of about one second after the last read.
The fallback is today's quarantine. An orphan of a subreaper other than
init is not seen by the guard.
- The journal adds at most one small write each second for each runnerd
while processes start on the host. This happens on the background
sampler, not before turn start.
- Previous-release story: no database or saved-session schema change.
The journal is a new file in the runner state directory. A runnerd from
an older release writes no journal and has no `reap-lineage` mode. Its
loss keeps today's quarantine, and the existing board Retry and
stopped-session proofs still apply.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 10:19:41 -06:00
DottaandClaude Opus 5.5 e015c2ef21 fix(runner): keep a Claude session resumable when its turn-end provider restart fails (#20285)
## Thinking Path

- #20145 prepares the next Claude provider generation at park, and
#20190 restarts the ACPX provider when a turn settles. They move the
sidecar respawn off the follow-up's critical path.
- When that turn-end restart fails, the runner "fails closed":
`restart_session_after_turn` saves the session `closed` and drops it.
Within the run, the next turn fails exactly like a failed restart at
turn start. That's intended.
- Across runs it breaks the conversation:
- the next run's `run.attach` refuses a `closed` session ("run.attach
requires the same settled ACPX provider profile and session");
  - it reconnects for about 10 s;
- the controller then records a continuity break and starts a fresh
Claude conversation (`run.startup.reuse provider_session:
fresh/created`).
- A local A/B (`ttft-baseline --adapters claude --samples 3 --phases
coldwarm`, on a host with load average about 17–27) confirmed it:

  | | d40a7494 (before #20145/#20190) | a3a25ce6 (after) |
  |---|---|---|
| Follow-ups that resumed the conversation | 3 of 3 (`verified_reuse`,
resume 4.4–8.2 s) | 1 of 3 |
| Follow-ups that started a fresh conversation | 0 of 3 | 2 of 3
(bootstrap 16–17 s) |

The kept instance shows each lost follow-up's runner received provider
state `lifecycle: "closed"`, with `run.attach` refused.
- A failed park preparation already recovers safely: it waits for the
reaped provider's lifetime fence, then saves `suspended`. The turn-end
restart should do the same.

## Linked Issues or Issue Description

Refs #20190, #20145

**What happened?**
When the post-turn ACPX provider restart failed, the next Claude run
could not attach. It spent about 10 s and then started a new
conversation, losing the task's conversation context.

**Steps to reproduce**
1. Run a Claude task on a busy host so the turn-end provider restart
fails (the fixture `reject-open` fault reproduces it).
2. Post a follow-up comment (a new run).
3. The follow-up's `run.attach` is refused, then `run.startup.reuse`
reports `provider_session: fresh/created`.

**Expected behavior**
The follow-up resumes the same Claude conversation. Within the failing
run, later turns still fail closed.

## What Changed

- `acpx_park_preparation.rs`:
- A failed `restart_session_after_turn` now calls the shared
`recover_failed_generation_restart`, renamed from
`recover_failed_park_preparation`.
- That recovery drops the session, marks it `prepared` with the provider
exit unconfirmed, waits for the provider-lifetime fence, then saves
`suspended`.
- If the fence or the save fails, the state stays exit-unconfirmed,
which is the safe direction.
- `acpx_provider_backend.rs`:
- Later turns of the failing run still fail closed: the first with "ACPX
turn start failed closed: <cause>", then "ACPX session is not open".
That's the same as a failed restart at turn start.
- The restriction is saved in the ACPX state as
`turnEndRestartFailedRun` (serialized only while set), so it outlasts a
Runner process restart inside the same run. Same-run restore doesn't
reopen the session while it's set.
- Only a `run.attach` from a different run clears it; a same-run attach
keeps it. State validation requires it to match the descriptor's run.
- The cause is handed to the durable Runner once
(`take_turn_end_restart_failure`), which records it as a redacted
`turn-end provider restart failed: <cause>` diagnostic in
`runner-state.json`. That gives the under-load cause a first recorded
sample.
- Tests (`tests/support/acpx_turn_end_restart.rs`):
- New:
`the_next_run_resumes_the_conversation_after_a_failed_turn_end_restart`,
red on master (the state is `closed`). It covers the same Runner process
with a warm attach, and a new Runner process with a cold attach. The
conversation id is preserved, and the next run completes a turn.
- New:
`a_runner_restart_in_the_same_run_keeps_the_failed_turn_end_restart_closed`,
red without the saved field. A new Runner process with the same run id
fails `turn.start` without spawning a provider; then a new run
cold-attaches and resumes the same conversation.
- The existing
`a_failed_turn_end_restart_fails_the_next_turn_like_a_failed_turn_start_restart`
now also asserts that the turn-end failure saves `suspended` (the
turn-start failure stays `closed`). Everything else about the run's
failing turns is unchanged.

## Verification

- `cargo test -p paperclip-runner-core`:
  - `--test acpx_park_preparation`: 21 passed.
  - `--lib acpx`: 93 passed.
  - `acpx_park_readiness`: 8 passed.
  - `acpx_warm_attachment`: 4 passed.
  - `acpx_provider_suspend`: 5 passed.
  - `acpx_provider_session`: 12 passed.
  - `acpx_provider_turns`: 29 passed.
  - `native_provider_backend`: 47 passed.
  - `durable_recovery`: 3 passed.
- Live A/B on this branch (`ttft-baseline --adapters claude --samples 3
--phases coldwarm`): follow-ups resumed 3/3 (1/3 before), follow-up
first activity p50 6.2 s (about 20–23 s before).
- `cargo clippy --all-targets`: no findings on changed lines.

## Risks

- **Late-callback guarantee:** unchanged. The failed restart already
reaped the old generation, and the next generation starts only via
`run.attach` or a later reopen. That's the same resume path a failed
park preparation uses.
- **Fence wait:** the failure path now waits for the reaped provider's
lifetime fence before saving, matching park recovery. If the fence never
clears, the state stays exit-unconfirmed and fails closed.
- **Rollback:** `AcpxDurableState` denies unknown fields. The new field
is written only between a failed turn-end restart and the next run's
attach. A rollback inside that window makes the older Runner refuse the
saved state, which still fails closed, as master's `closed` state does
today.
- **Scope:** the cause of restart failures under load isn't changed
here. This makes their failure mode safe.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 10:15:25 -06:00
DottaandClaude Opus 5.5 3374ca54f5 fix(runner): record a runner exit mid-turn as runner loss, and stop the lineage sampler panicking (#20259)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runner (`runnerd`) runs each provider turn, and the
server records why a run stopped and what the person can do next
> - When `runnerd` exits during a provider turn, the server saved the
run as `native_provider_terminal_failed`: "The provider session is
permanently unusable", with automatic retries stopped
> - The real cause was the runner process, and the transport already
names it (`native_runner_process_exited`) at the start of the
closed-stream message
> - Separately, the process-lineage sampler added in #19727 panicked on
Linux debug builds when a provider tool started its own session, and
that panic text became the run's visible failure
> - This pull request records a runner exit mid-turn as runner loss, and
reads session IDs without the assertion that panicked
> - The benefit is that a person sees the true cause of a stopped run,
and recovery treats a lost runner like every other runner exit

## Linked Issues or Issue Description

Refs #19727

**What happened**

A chat eval killed only the owned `runnerd` process (SIGKILL) while a
provider tool was running. The run failed with `errorCode:
native_provider_terminal_failed` and the next action "The provider
session is permanently unusable". The error text was `Notification
transport failed closed: native_runner_process_exited: runnerd exited
unexpectedly: paperclip-runnerd panic: panicked at
rustix-1.1.4/src/pid.rs:61:9: assertion failed: raw > 0`. The panic came
from the lineage sampler thread before the kill. It called
`rustix::process::getsid`, which asserts a nonzero session. A kernel
thread, or a process whose session leader is outside the PID namespace,
reports session 0.

**Expected behavior**

A runner process exit is recorded as `native_runner_process_exited`, the
same cause the server already uses for a runner exit outside a turn.
Recovery then applies its normal runner-loss rules (checkpoint evidence,
bounded attempts, cleanup quarantine). The lineage sampler does not
panic, and its diagnostics do not replace the real failure text.

**Steps to reproduce**

1. Start a native OpenCode run on Linux with a debug `runnerd`.
2. Let the agent run a shell tool that starts its own session (for
example a long `node -e` command).
3. Send SIGKILL to the `runnerd` PID only.
4. Read `GET /api/heartbeat-runs/<runId>`: `errorCode` is
`native_provider_terminal_failed` and the error names the rustix panic.

## What Changed

- `nativeSessionFailureSourceCode` returns
`native_runner_process_exited` for a `NativeProviderTerminalFailure`
with code `notification_transport_failed` when its message starts with
`Notification transport failed closed: native_runner_process_exited: `.
Only the runner transport writes that prefix. Other closed-stream causes
stay `native_provider_terminal_failed`.
- `paperclip-process-table` has a new `session(pid)` function. Linux
reads the session field of `/proc/<pid>/stat`. macOS calls `getsid`. A
session of 0 or an error gives `None`.
- `process_lineage` reads sessions through
`paperclip_process_table::session` instead of rustix `getsid`.
- Regression tests: the runner-loss classification (attempts 1 and 3,
and three negative shapes), the stat session parser with a kernel-thread
line, and a live read of this process's session.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "runner
process exit that closed|preserves stable provider and runner|model
rejection from a provider terminal|operator-owned through terminal"` (8
passed)
- `cd packages/paperclip-runner/runner && cargo test -p
paperclip-process-table` (macOS: 3 passed; the Linux-only parser tests
run in CI)
- `cd packages/paperclip-runner/runner && cargo test -p
paperclip-runner-core --lib process_lineage` (6 passed)
- End to end: the `recovery.*` chat eval cells that kill `runnerd` are
run against this change on a separate eval branch.

## Risks

- Behavior shift: a runner exit during a turn now follows runner-loss
recovery instead of stopping as a permanent provider failure. Recovery
still decides from durable evidence. Ambiguous state stays terminal, and
an unverified cleanup stays quarantined.
- Previous-release story: no persisted shape changes. Runs saved earlier
with `native_provider_terminal_failed` keep that code. Only new failures
are classified with the new rule.
- The session read on Linux now uses `/proc` instead of a syscall. It
gives the same value, and it reports 0 (now `None`) in the same cases.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 06:38:24 -06:00
DottaandClaude Opus 5.5 465da4227a fix(runner): drop a tool target path that breaks a line so the controller can acknowledge its event (#20238)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner turns provider tool calls into durable PRP
events. The controller acknowledges each event only after it validates
and commits it.
> - The controller's event schema forbids line terminators in a safe
path such as `tool.execution` `target`.
> - The ACPX sidecar and the Rust runner accepted a location path that
contained a newline. Claude sometimes emits a malformed Write call that
puts the file content into the path.
> - The controller refused that event without an acknowledgement. The
runner redelivered the same bytes, and the run stopped making progress
for 14 minutes (2,848 reconnects).
> - This pull request makes both producers drop such a path, so the
event carries a null target and the controller accepts it.
> - The benefit is that a malformed provider path can no longer wedge a
run's event delivery.

## Linked Issues or Issue Description

**What happened?**
In restart-chaos wave seed 9807 (claude, R1, local), the run stopped
receiving events after runner source seq 33. The runner reconnected
2,848 times in 14 minutes. Offline replay of the saved runner outbox
through the controller's `validatePrpEvent` showed that seq 34 fails
`/payload/target` (the safePath pattern), because the target is
`packages/shared/src/chaos/slug-2c5c3c8f.ts\n<parameter
name="content">export function chaosSlug…`. Seq 35, 39 and 40 (the same
tool call) fail the same way.

**Expected behavior**
The runner never emits an event that the controller's schema refuses. A
malformed tool path becomes a null target, and the tool event still
arrives.

**Steps to reproduce**
1. Have an ACPX provider send a tool call whose location path contains a
newline (for example, a Write call with the content in the path).
2. See that the runner emits `tool.execution.progressed` with that
target.
3. See that the controller closes the connection without an
acknowledgement on every redelivery.

**Paperclip version or commit**
master at 3963d38560

## What Changed

- `runner-core/src/tool_file_changes.rs`: new `is_single_line_path`,
which refuses control characters and U+2028/U+2029 (the characters
ECMAScript `.` does not match). `normalize_relative` uses it, so
file-change paths are covered too.
- `runner-core/src/provider_events.rs`: `safe_acpx_location` uses the
same check (it replaces the NUL-only check).
- `src/drivers/acpx/safe-locations.ts`: the sidecar drops a location
whose path contains a control character or U+2028/U+2029.
- Tests: a Rust test covers the observed path, `\r`, U+2028, U+2029 and
tab, plus a full ACPX tool-call normalization with a broken location and
file change. A TS sidecar test covers the same paths.

The controller-side detection and reason capture for a refused event is
in paperclipai/paperclip#20235 (Phase 5). This PR does not touch the
controller.

## Verification

- Red before the fix: the Rust test
`drops_a_location_whose_path_breaks_a_line` and the TS test `drops a
path that breaks a line…` both failed. Both pass after it.
- `npx vitest run src/cli/acpx-sidecar-locations.test.ts
src/cli/acpx-runtime-sidecar.test.ts src/drivers/acpx` in
`packages/paperclip-runner`: 1,818 passed.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in
`packages/paperclip-runner`: passes.
- `cargo test --lib` in `runner-core`: 598 passed. Four
`task_memory::tests::monitor_*` tests fail on this macOS host in
isolation too. That file is unchanged here (last changed in #18801), and
the CI Rust job is the authority.

## Risks

- Low. Only a path that the controller already refuses is dropped. A
valid POSIX filename with a tab is now dropped as a display target too,
which is consistent with the existing ACPX rich-event check
(`char::is_control`).
- Previous-release story: a runner outbox already saved with such an
event (from an older runner) still holds it. That case needs the
controller detector in paperclipai/paperclip#20235, which fails the run
closed instead of looping. No saved state, schema or pack contract
changes here.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 05:06:43 -06:00
DottaandClaude Opus 5.5 d7d0f96f3f perf(runner): save provider state once per poll instead of once per frame (#20217)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native OpenCode, Codex and Cursor Cloud chat turns run through
runnerd. runnerd reads provider frames in order and keeps the run's
provider state in a durable file
> - For every frame that it reads, runnerd rewrote the whole provider
state file and called `fsync` on the file and on its directory. The
state holds every tool result kept in the turn, so each save got more
expensive as the turn went on
> - A tool call request is just the next frame. It waited behind one
full save for every frame before it. #20209 measured tool-call
turnaround that grew from about 1.4 s to about 15 s in one turn
> - #20209 stopped the OpenCode proxy from sending frames that runnerd
discards. Every frame that runnerd still reads still paid a full save,
and other proxies (for example Cursor Cloud) still send such frames
> - This pull request makes runnerd save the frames of one poll once,
before anything derived from them leaves runnerd. A save that would
write the bytes already on disk is skipped
> - The benefit is that the number of saves before a tool call no longer
grows with the number of streamed frames. In a local measurement, a tool
call after 400 frames with 830 KB of kept state took 0.57 s instead of
more than 30 s

## Linked Issues or Issue Description

Refs #20209
Related: #17514 (the same per-event full-state rewrite, on the server
control-plane side)

**What happened**

During a native OpenCode chat turn, tool-call turnaround grew from about
1.4 s to about 15 s as the turn went on. runnerd saved its complete
durable provider state, with `fsync`, after every provider frame. That
state includes the tool results kept so far in the turn. A tool call
request waited behind one full save for each earlier frame.

**Expected behavior**

The time to deliver a tool call must not grow with the number of
assistant-text, reasoning or item frames that came before it. A crash or
restart must still never lose state that a resumed run needs.

**Steps to reproduce**

1. Start a native OpenCode (or Codex app-server or Cursor Cloud) turn
that keeps a few tool results and then streams many text or reasoning
frames before its next tool call.
2. Measure the time from the tool call frame to the
`semantic_tool.input` event.
3. See that the time grows with the number of earlier frames and with
the size of the kept state. Locally, `cargo test -p
paperclip-runner-core --lib measure_streamed_frame -- --ignored
--nocapture` on `master` reports 102 saves and 9.3 s for 100 frames with
830 KB of kept state, and more than 30 s for 400 frames.

## What Changed

-
`packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs`
and `codex_provider.rs`:
- The provider frame loop moves into `poll_provider_frames`. An ordinary
notification frame and a descendant notification frame no longer save
the state. They mark it as unsaved.
- After the loop, `poll_current_provider_with_control` saves the unsaved
state once. It also saves when a frame fails, before it returns the
error.
- Frames that already saved at once still do so: turn terminals, Cursor
Cloud terminal deliveries, `reconciliation_required`, protocol failures,
MCP permission and runtime requests, process exit, and new tool calls.
- A tool call that runnerd answers at once (an exact completed replay or
the receipt-limit stop) saves unsaved state before it writes to the
provider. A rejected tool call already saved first.
- `retained_events`, `execute` and `host_park_ready` save unsaved state
first, so no event, command result or park decision can expose state
that is not on disk.
- Courtesy rejections that `CodexProvider::poll` writes by itself
(unsupported requests, descendant requests, delayed requests after a
terminal, full runtime-request capacity) are held while runnerd reads a
batch (`hold_courtesy_responses`). runnerd sends them only after it
saves the state that covers every earlier frame
(`save_before_provider_writes`). A terminal frame also saves and
releases them before runnerd asks the provider for goal state or drains
the accounting tail. Released courtesy replies are best-effort, as
courtesy replies were before: a provider that already closed stdin
cannot turn a received terminal into a polling failure. The
receipt-limit stop sends held cancellations before its `turn/interrupt`
request, so their order to the provider does not change.
- The trace records for runnerd's normalization
(`rust_durable_normalization`), which name emitted event ids, are kept
until the state that holds those ids is saved. The trace therefore only
names saved events. Saved mappings are written before an exited or
failed provider is dropped, so they are not lost. Raw frame records
still go to the trace when a frame is read; they name no event ids.
- `save_unsaved_frame_state` does not write when the serialized state is
byte-for-byte the same as the saved file. All other saves write as
before.
- Serialization moves to `serialize_provider_state`, and the file write
moves to `write_state_bytes`. The write path (private temporary file,
`fsync`, rename, directory `fsync`, `0600`) does not change.
- New tests in
`packages/paperclip-runner/runner/crates/runner-core/src/tests/provider_frame_save_batching.rs`:
- `streamed_frames_do_not_multiply_durable_saves_before_a_tool_call`: 8
and 48 streamed frame pairs before a tool call need the same number of
saves (1). On `master` they need 18 and 98.
-
`batched_frame_state_is_saved_before_events_leave_and_survives_a_restart`:
at every poll, the saved file is equal to the in-memory state and holds
exactly the delivered events. A new executor loads the file after the
old one stops and gets the same state, the same unacknowledged events
and the pending tool call.
- `batched_frame_state_is_saved_when_a_later_frame_fails`: a frame that
fails after 20 batched frames still leaves those 20 frames on disk.
- `previous_release_state_is_rewritten_by_the_first_batched_save`: a
state file in an older shape (compact JSON without newer default fields)
loads, and the first batched save rewrites it. Unchanged state is not
rewritten. Changed state is.
-
`courtesy_rejections_reach_the_provider_only_after_earlier_frames_are_saved`:
when the courtesy rejection reaches the provider, the saved file (what a
restarted runnerd would load) already holds the 20 earlier frames and
equals the in-memory state. It fails if the hold is removed.
- `trace_interpretations_name_only_saved_events`: every event id that a
trace record names is in saved state at the moment the record is
written. It fails if records are written before the save.
-
`a_crash_inside_an_unsaved_batch_leaves_saved_state_and_provider_output_unchanged`:
after 122 frames are read and not yet saved, the saved file is
byte-for-byte unchanged and runnerd has written nothing to the provider.
A reload gets the pre-batch event sequence and no pending events.
-
`a_held_cancellation_that_cannot_reach_the_provider_keeps_the_completed_turn`:
the provider closes stdin and then completes the turn with a tool call
still pending. The held cancellation cannot be written, and the turn
still completes and is saved.
- `trace_interpretations_survive_a_provider_exit_in_the_same_poll`: the
trace keeps the mappings for frames that arrived in the same poll as the
provider exit.
- `measure_streamed_frame_cost_before_a_tool_call` (ignored): prints
saves and tool-call latency for different frame counts and kept-state
sizes.

## Verification

- `cargo test --locked -p paperclip-runner-core` (debug, macOS arm64):
library 595 passed, 1 failed; all 35 integration test binaries passed
(`--no-fail-fast`). The one library failure is
`task_memory::tests::monitor_bounds_notices`. It also fails when run
alone, takes about 9 minutes, and waits on file-watch timing in
`task_memory.rs`, which this PR does not touch. In one earlier run, two
ACPX park tests in `acpx_park_preparation` failed. The same binary also
failed on unchanged `master` (with a different test), and it passed in
the final run.
- `cargo test -p paperclip-runner-core --lib measure_streamed_frame --
--ignored --nocapture`, debug build, macOS APFS (`sync_all` is
`F_FULLFSYNC`). Each row is one turn: frames streamed, then one tool
call.

| Kept state | Frames | Saves (master) | Tool-call latency (master) |
Saves (this PR) | Tool-call latency (this PR) |
| --- | --- | --- | --- | --- | --- |
| 4 KB | 100 | 102 | 946 ms | 1 | 18 ms |
| 4 KB | 400 | 405 | 3,791 ms | 7 | 70 ms |
| 211 KB | 100 | 102 | 3,207 ms | 1 | 37 ms |
| 211 KB | 400 | 405 | 12,803 ms | 7 | 193 ms |
| 833 KB | 100 | 102 | 9,251 ms | 1 | 93 ms |
| 833 KB | 400 | about 405 | more than 30 s (test deadline) | 7 | 567 ms
|

With 400 frames, the remaining saves are one per poll of up to 128
frames and one per controller acknowledgement.
- With the debug `paperclip-runnerd` and `fake-codex-app-server`
binaries from this branch: `npx vitest run` on
`src/live/runnerd-opencode-streaming-tools`,
`runnerd-opencode-assigned-tools`, `runnerd-codex-transport`,
`runnerd-codex-transport-settlement`,
`runnerd-codex-transport-cold-restore`, `runnerd-restart-continuation`,
`runnerd-saved-continuation`, `runnerd-detach-durability`, and
`src/cli/opencode-proxy-output`, `opencode-proxy-events`,
`opencode-app-server-proxy`: 11 files, 353 tests passed.

## Risks

- Durability. The rule is: nothing derived from a frame leaves runnerd
before the state that holds that frame is on disk. That covers delivered
events, command results, park decisions, tool results and rejections,
courtesy rejections, Cursor Cloud terminal acknowledgements, and trace
records that name event ids. The tests above check each of these.
- The unsaved window is at most one poll (128 frames). Every frame in it
is undelivered: no event from it reached the controller, and runnerd
wrote nothing to the provider while it was unsaved.
`a_crash_inside_an_unsaved_batch_leaves_saved_state_and_provider_output_unchanged`
proves this. A crash in the window therefore leaves the saved state and
everything the controller and provider have seen exactly as a crash just
before those frames were read. The resumed run starts from the same
event sequence, so it cannot deliver an event id twice. Its normal
recovery (thread resume and `thread/read` reconciliation) handles those
frames in the same way as frames still in the provider pipe. On
`master`, a crash could already lose the one frame being processed.
- One provider write can still happen in the window: `thread/read`,
which `CodexProvider::poll` sends to check the lineage of an unknown
helper thread. It is a read-only query that waits for its answer, and it
changes nothing on the provider side.
- Courtesy rejections now reach the provider at the end of the poll
instead of during it. The delay is one batch. The provider gets the same
replies.
- Previous-release story: the saved file shape does not change, and no
new field is saved. A file from the previous release loads as before.
The skip check compares exact bytes, so an older file is never treated
as already saved. The first save after an upgrade rewrites it in the
current shape. A deploy that lands mid-turn sees a file that the
previous release saved after each frame. That file is valid input for
this code.
- The ACPX backend has its own frame loop and is not changed here.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file editing) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 03:43:27 -06:00
Nicky LeachandClaude Opus 5.5 92a18bb0f3 perf(runner): restart the ACPX provider when a turn settles (#20190)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner drives Claude through the ACPX sidecar. The
sidecar's MCP endpoint lasts for the whole session, so it cannot tell
which turn sent a late tool callback.
> - Because of this, the Runner reaps the sidecar and provider of a
generation that ran a turn before it releases call-ID tombstones and
rotates receipt epochs. Today this restart runs at the start of the next
turn.
> - Every follow-up turn in a run pays that restart on its critical
path: about 1.5 s with real Claude. #20145 moves the restart to park,
but a second turn in the same run, after the first turn that follows an
attach, still restarts cold.
> - The restart does not have to wait for the next turn. When a turn
settles and the host has its terminal result, the Runner is idle.
> - This pull request restarts the provider at that point. The next
turn, or the next park, then finds a fresh generation.
> - The benefit: same-run follow-up turns start without a restart, park
does no restart after a settled turn, and the late-callback guarantee is
unchanged.

## Linked Issues or Issue Description

Refs #20145, which moved the restart to park. This pull request builds
on it. No public issue exists, so the problem is described here.

**Subsystem affected**

Paperclip Runner, ACPX provider (Claude).

**Current behavior**

`AcpxProviderSession::start_turn_timed` restarts the provider before any
turn whose generation already ran a turn. A follow-up turn in the same
run waits for `session.suspend`, sidecar shutdown, a new sidecar,
`initialize`, `session.open` and `run.attach` before `turn.start`. Since
#20145, park does the restart instead, but only for the first turn after
an attach.

**Proposed behavior**

When a turn settles and the controller has ACKed every committed event,
the Runner restarts the provider. The next `turn.start` uses the fresh
generation and only rotates receipts. Park reuses the fresh generation
and does no second restart.

**Reason and benefit**

The restart leaves the turn's critical path, both for same-run
follow-ups and for the turns after a park. The order of reap and
rotation stays the same, so a late callback from the reaped generation
still cannot reach a later turn.

## What Changed

- **Runner loop** (`durable/runner.rs`): new
`CommandExecutor::prepare_next_turn` hook, a no-op by default. The
runner calls it after the idle provider poll, only when no control frame
is waiting and the controller has ACKed every committed event
(`acked_source_seq >= highest_source_seq`). So the restart always runs
after the terminal result reaches the host.
- **ACPX turn-end restart** (`acpx_park_preparation.rs`,
`acpx_provider_generation.rs`):
- `restart_session_after_turn` restarts a Claude session that has a
completed turn, nothing pending, no buffered events, a dormant goal and
its whole event tail ACKed. It first settles an abandoned park
observation. If that settlement reads more events, it skips this round,
and the next idle round restarts after those events are ACKed.
- `AcpxProviderSession::restart_after_turn` does the same suspend → reap
→ start → bootstrap → identity check as the turn-start restart. It
records the reaped process id in `restarted_after_turn_from`. This field
is the explicit "fresh because of a turn-end restart" marker.
- The session does not poll a fresh generation until its next turn or
rebind, the same as a prepared one. Its startup output is published with
the next turn, as today. Polling it early would add events after the
host computed its park cut.
- **Next turn** (`acpx_provider_session.rs`): `start_turn_timed`
restarts only when the generation ran a turn. When
`restarted_after_turn_from` is set, it does the post-restart receipt and
ledger rotation without a second restart. `session.reconciled` reports
the reaped process as `previousProcessId`.
- **Park**: `prepare_next_generation` restarts (with the 20 s park
deadline) only if no turn-end restart ran. Otherwise it marks the
generation for `run.attach` rebind and returns `false`, so the runner
does not ask for a second park proof. The park-time receipt rotation
moves to the next turn start, where the turn-end path does it too.
- **Restart failure**: the executor sets the same state that a failed
turn-start restart leaves (`closed`, no session, provider exit
confirmed). The next `turn.start` fails with the same `ACPX turn start
failed closed: …` error that the turn-start restart gives today.
- **Fake sidecar**: new `pending-input` fault, which opens an input
request in a turn that then completes.

### Races

| Race | Handling |
| --- | --- |
| A turn or steer arrives during the turn-end restart | The runner loop
is single-threaded. The command waits in the socket until the restart
returns, then starts on the fresh generation without a restart. Only one
generation is live. |
| A turn arrives before the hook runs (for example, in the same read as
the ACK) | Control frames go first, so the turn restarts at its start,
as today. The hook then restarts after that turn. |
| Park arrives during the turn-end restart | Host control is read at the
top of the loop, so park waits. Park then reuses the fresh generation.
No event is published in between, so the host's park cut still matches.
|
| Park arrives before the hook runs | Park restarts with the 20 s
deadline, as in #20145. The hook then finds a fresh generation and does
nothing. |
| Eviction, `runner.suspend` or retire after the restart | They reap the
fresh generation through the ordinary paths. |
| The restart fails | The session is `closed`. The next turn fails with
the same error and the same durable state as a failed turn-start
restart. Park returns conflict, as it does for a closed session today. |
| Cancellation | `run.cancel`, `turn.interrupt` and `turn.stop` after
the restart return `already_settled`, as after any settled turn. A
cancellation that retires the runtime closes the session, so no restart
runs. |
| Interrupted, cancelled or failed turn | `completed_turn_observed` is
false, so no turn-end restart runs. The next turn restarts at its start,
as today. |
| Pending permission or input requests at the terminal | The reducer
ends them at the terminal, so the restart runs. A later
`request.resolve` fails with `no active turn`, with or without the
restart. |
| Active session goal at the terminal | No restart, because the goal can
keep working after its prompt. |

### Removed as redundant, and what stays

- **Removed:** the park-time restart and dormant verify when a turn-end
restart already made the generation fresh. The runner's
`host_park_ready` proves dormancy just before `prepare_for_park`, and
nothing runs in between.
- **Removed:** the runner's second park proof in that case, because the
provider did not change.
- **Removed:** the park-time receipt and ledger rotation. The next turn
start does it once, for both paths.
- **Kept: the turn-start restart.** Without it, a turn queued before the
hook, a turn after an interrupted or cancelled turn, a turn after an
active goal, and any non-Claude ACPX turn would run in the generation
that ran the previous turn. Tests cover the queued-turn and interrupted
cases.
- **Kept: the park-time restart.** Without it, a park that arrives
before the hook would leave the next run to restart at attach. A test
covers this.
- **Kept: the settlement of an abandoned park observation.** Without it,
the restart drops events that the old generation sent before the
abandoned response. A test covers this, and it fails with the settlement
removed.
- **Kept: the goal-dormancy check.** Without it, the restart kills an
active goal's autonomous work. A test covers this, and it fails with the
check removed.
- **Kept: the ACK gate.** Without it, the restart can run before the
host has the terminal result. A runner test covers this.
- **Kept: deferred polling of the fresh generation.** Without it, its
startup event lands after the host computed its park cut, and the park
conflicts.
- **Kept: the kept failure message.** Without it, the next turn fails
with `ACPX session is not open`, and the restart error is lost.

## Verification

- `cargo test -p paperclip-runner-core --lib` and every
`tests/acpx_*.rs`, `native_provider_backend` and `durable_recovery`
suite: all pass (debug).
- New tests (`tests/support/acpx_turn_end_restart.rs`, run from
`acpx_park_preparation`) prove these points:
- A settled turn reaps the old pid before it spawns the new one, and no
predecessor is still live. A same-run follow-up then sends only
`turn.start` and reports the reaped pid in `session.reconciled`.
- Park after a turn-end restart sends no sidecar command and returns
`false`. Attach rebinds the same pid. The run's second turn restarts at
its end, and the third turn starts without a restart.
  - The reaped generation's late callback never reaches the next turn.
- A turn that arrives before the hook restarts at its start. A park
before the hook restarts once.
- A failed turn-end restart gives the same next-turn error, retry error,
durable state and process count as a failed turn-start restart.
- Cancellation after the restart is `already_settled`. `runner.suspend`
reaps the fresh generation.
- An active goal skips the restart. An input request open at the
terminal behaves the same with or without the restart.
- `acpx_provider_turns`: a session-level turn-end restart drops the old
generation's late `issues.delete` callback, which reuses the previous
turn's call ID. The next turn then admits that call ID from the fresh
provider. Interrupted and retired turns do not restart at turn end.
- `acpx_park_readiness`: a turn-end-restarted generation is park-ready.
An abandoned park observation tail is delivered before the restart.
- `durable::runner`: the hook runs only after the terminal is ACKed and
when control is idle. It never runs during ACK debt, a waiting control
frame or a disconnect.
- Server integration tests (`plugin-runner-release-resume`,
`plugin-runner-eviction` and `plugin-runner-task-warm`) with this
branch's Runner: 49 passed. The provider process counts do not change:
each released run still adds one process start, now at turn end instead
of at park.
- `cargo fmt --check`: clean.
- Fake-sidecar timing (release build, 7 runs, medians):

  | | before (`master` with #20145) | this PR |
  | --- | --- | --- |
| same-run follow-up `turn.start` → `turn.started` | 51.5 ms | 22.0 ms |
| second turn after an attach, `turn.start` → `turn.started` | 270.3 ms
| 23.5 ms |
  | turn-end restart, after the terminal is ACKed | — | 29.0 ms |
  | park preparation after a settled turn | 34.1 ms | 0.03 ms |

- Real Claude: not run for this pull request. With real Claude, the
restart took about 1.5 s on the turn-start path, and this change takes
it off that path. That number is not measured on this branch.

## Risks

- A run that ends after one turn without a park now starts one extra
sidecar that no turn uses. The Claude CLI itself starts only at the
first prompt, so the cost is the sidecar's start and bootstrap.
- The restart blocks the Runner loop for its length (about 1.5 s with
real Claude), as the turn-start restart does today. It has no deadline,
as before. A park that arrives during it waits, inside the host's 30 s
park wait.
- The fresh generation is not polled until its next turn. If it exits
while idle, the next turn finds out at `turn.start` and fails closed.
Park finds out at its observation.
- The turn-end restart is for Claude only, the same as park preparation.
Other ACPX agents keep restarting at turn start.
- If the Runner crashes during the restart, the durable state still says
`session_open`. Recovery is the same as for a crash while idle.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use
(shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:34:26 -08:00
Nicky LeachandClaude Opus 5.5 2d93413085 perf(runner): prepare the next Claude provider generation at park (#20145)
## Thinking Path

> - Paperclip runs agents through Paperclip Runner, and a host can park
an idle Runner between runs and resume it for the next run on the same
task.
> - For Claude (ACPX), a follow-up run on a parked Runner pays about
1.5–2.3 s at `run.attach` before the provider can start the turn. The
attachment closes the Claude sidecar and spawns a new one: initialize,
`session.open` with `session/load`, `run.attach`, then a dormant verify.
> - The respawn is required. The tool-bridge MCP endpoint lasts for the
whole session and can't tell which turn a late callback came from, so a
new turn must not run in a process that ran an earlier one.
> - Nothing requires the respawn to happen at the *start* of the next
run. A turn that ends 2 s later is a better experience than one that
starts 2 s later.
> - This pull request does the respawn while parking. The next run's
`run.attach` then only rebinds the run id on a fresh, never-used
generation.
> - Follow-up Claude runs on a parked Runner reach the provider about
1.5–2 s sooner, and the late-callback guarantees are unchanged.

## Linked Issues or Issue Description

No public issue exists. The issue is described inline.

### What happened?

On a hosted deployment with parked Runners, Claude follow-up runs
reached the provider about 1.8 s later than Codex follow-ups.
`runner.session.resume` medians were 5.2 s for Claude and 3.4 s for
Codex. The extra time is the ACPX respawn inside `run.attach`. Codex
keeps its process in place.

### Expected behavior

The respawn happens while the Runner parks, so the next run starts on a
prepared generation.

### Steps to reproduce

1. Run a Claude (ACPX) task on a host that parks Runners between runs.
2. Let the run finish and the Runner park.
3. Send a follow-up. `run.attach` closes the provider and starts,
initializes, opens and attaches a new sidecar before `turn.start`. The
Claude CLI itself starts at the first prompt on either path.

### Paperclip version or commit

`master` at e194e193.

### Deployment mode

Hosted, with parked Runner tasks.

## What Changed

- **`durable/runner.rs`:**
- New `CommandExecutor::prepare_for_park(deadline)` hook, a no-op by
default. The runner passes a deadline 20 s after the park request; the
host waits 30 s for the reply.
- The Park host-control arm calls it after `host_park_ready()`,
re-proves readiness, and only then replies `parked`.
- A failure records a diagnostic and replies `conflict`, the existing
park-failure path.
- **ACPX, park side** (`acpx_park_preparation.rs`,
`acpx_provider_generation.rs`):
- For Claude, with the session open, nothing pending and nothing queued:
reap the old generation, using the same suspend-and-reap sequence a
second turn uses.
- Start the replacement with the current run attached, and prove it
dormant.
  - Roll the tool receipt epochs and the turn-identity ledger over.
- Record a digest of the session configuration it was opened with,
ignoring the MCP binding id and the aggregate digest.
- Every sidecar request during preparation is capped at the park
deadline. Running out of time fails like any other preparation failure.
- **ACPX, attach side:**
- A prepared, never-used generation whose digest matches the next run is
rebound: the sidecar gets `run.attach` with the new run id, the reducer
and tool bridges are reset, and the snapshot is checked for the new run
id.
- Anything else takes today's cold path: a different tool set, skill,
prompt or launch; a buffered callback; or any rebind error.
- **ACPX, turn start:** the provider restarts when its generation has
run a turn, instead of when the run has settled turns.
`has_settled_turns` is removed.
- **ACPX, polling:** polling skips a prepared generation until it is
rebound or starts a turn, so nothing it emits is attributed to the
parked run.
- **Sidecar** (`acpx-runtime-sidecar.ts`): rejects `run.attach` once a
turn has started in that process.
- **Fake sidecar:** reports live predecessors at process start, and adds
the faults `reject-open`, `late-tool-after-suspend` and `pre-turn-tool`.

## Verification

- `cargo test -p paperclip-runner-core`: 1029 passed (debug).
  - New `tests/acpx_park_preparation.rs` (10 tests). These prove:
- park reaps the old pid before spawning the new one, and acks only
after the dormant verify;
- a late callback from the old generation after park is rejected, and so
is a pre-turn call from the new generation;
- a compatible attach keeps the pid, and the sidecar and snapshot carry
the new run id;
- changed instructions, tools, skills or runtime context take the cold
path;
    - a used generation is never rebound;
- a prepare failure produces a park conflict and a recoverable state;
    - Retire works after a prepared park;
- a replacement whose `session.open` stalls fails at the park deadline,
leaves the session suspended, and the next run attaches and completes a
turn.
- `cargo fmt --check` and `cargo check --workspace --locked`: clean.
- `vitest` `acpx-runtime-sidecar.test.ts` and
`acpx-sidecar-input.test.ts`: 126 passed. The new re-attach guard test
fails if the guard is removed.
- `vitest` `plugin-runner-release-resume.integration.test.ts` and
`plugin-runner-eviction.integration.test.ts` with the branch's Runner:
48 passed. Both now expect the provider process to be replaced at park
instead of at attach.
- Fake-sidecar timing (release build, 3 runs): `run.attach` took 230–260
ms cold and 40–80 ms after a prepared park.
- Real Claude (claude-agent-acp 0.73.0, release build on a hosted test
VM, 3 runs each way plus one with a 180 s cgroup freeze between park and
attach):

  | | before | this PR |
  |---|---|---|
  | `run.attach` | 1691 ms mean | 212 ms mean |
| `run.attach` start → follow-up `turn.started` | 2565 ms mean | 1065 ms
mean |
  | `turn.start` → first provider output | 2261 ms mean | 2232 ms mean |
  | park preparation added to park | — | about 1450 ms |

Session recall, an authorized tool call after the rebind, and the
absence of any tool call or event from the old run all passed in every
run, including after the freeze. The prepared generation holds only the
sidecar: the Claude CLI starts at the follow-up prompt on both paths, so
a parked Runner also holds one node process instead of the whole Claude
tree.
- Hit rate: on a hosted test deployment, all 15 consecutive Claude runs
across two tasks carried identical content-addressed instruction and
skill bundle paths, so the prepared digest would have matched.
- Not run: the hosted park, claim and attach loop with real Claude, a
native MCP binding or skills with real Claude, `cargo test --release`,
smoke tests.
- Pre-existing flake:
`unconfirmed_suspension_state_becomes_recoverable_only_after_cleanup_and_suspend`
fails intermittently on clean `master` when the `hosted acpx durable::`
subset runs in parallel. It passed in both full runs.

## Risks

- **Park takes about 1.5 s longer** before it replies with real Claude.
Preparation is capped at 20 s; the host's park wait is 30 s.
- **Configuration changes:** a change between park and the next run
(registered asset paths, tools, prompt) takes the cold path. That's
safe, but saves nothing.
- **Stale environment:** a prepared process was opened with the previous
run's environment and disk state. If credential or configuration files
change on disk while the Runner is parked, the prepared process won't
see them, whereas the cold path would.
- **Failed prepare:** a park that fails after the reap leaves no live
provider. The next turn or `run.attach` starts Claude cold, as today.
- **Scope:** only ACPX Claude prepares. Other providers park exactly as
before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, git, GitHub CLI) and subagents.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 21:38:57 -08:00
DottaandClaude Opus 5.5 412d49e48d fix(runner): resume an ACPX turn interrupted by a deploy instead of refusing the launch digest (#20112)
## Thinking Path

- The rule for upgrade-safe saved state: work saved by the old build
must resume under the new build.
- The K14 upgrade-mid-run chaos wave (`838e9bf3d2` → `f789f3e2af`) broke
that rule for Claude. The same heartbeat run resumed after the upgrade
restart, and the runner's resume probe (`runner.drain`) failed with
`ACPX durable launch profile digest does not match runner startup`
(`provider_state_incompatible`). t03 and t04's Claude subtask both
needed a board Retry. Codex crossed the upgrade cleanly.
- The ACPX launch profile digest covers the provider-pack authority and
the sidecar artifact hashes, so every deploy changes it.
- #18879 already re-binds a suspended session to the new digest. A
deploy that restarts the runner mid-turn leaves the same run with an
active turn, or with an open session, which is the state a same-build
lost-turn restore leaves behind. Neither was re-bound.
- A same-build restart already recovers both states safely: it reopens
an open session, and recovers an active turn through
`restore_conversation_after_lost_turn`, which is guarded by the
provider-lifetime fence. An upgraded build should take the same path.

## Linked Issues or Issue Description

Refs #18879

**What happened?**
A deploy restarted the runner while a Claude (ACPX) task was mid-run.
When the same run resumed under the new build, restore refused the saved
state with `provider_state_incompatible`. The task stopped and needed a
manual Retry, which started a fresh provider session and dropped the
conversation.

**Steps to reproduce**
1. Start a Claude task on build A and let it reach an active turn.
2. Upgrade to build B, which ships a new provider pack or sidecar build,
with a graceful restart while the turn runs.
3. The same run's resume fails with `ACPX durable launch profile digest
does not match runner startup`.

**Expected behavior**
The resumed run recovers exactly as it would after a same-build restart.
The conversation is restored, or the lost turn is closed through the
normal recovery path, with no Retry.

## What Changed

-
`packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs`:
- `rebinds_settled_session` now also re-binds the saved digest for an
**interrupted turn or open session of the same heartbeat run**
(`rebinds_interrupted_turn`).
- The re-bind requires exactly the same-build recovery preconditions: no
accepted semantic result, no pending runtime request, no unconfirmed
provider exit, and a dormant goal. It also keeps the existing
requirements: no candidate admission, and the same sidecar command and
arguments.
- Recovery then runs unchanged. The provider-lifetime fence still
decides whether the conversation is restored or the lost turn is closed.
- **Relocated installs:** an upgrade can install the same launch files
under a new directory, as an `npx` versioned cache, a new image layout
or a new checkout does. The K14 harness moves between worktrees. The
re-bind now accepts a launch of the same shape: the same executable and
sidecar file names, and identical non-path arguments. It then records
this startup's runner-owned paths in the saved descriptor. A renamed
executable or a changed argument stays strict.
- An active turn or open session saved by a *different* run stays
strict, as before. So do suspended sessions with undelivered events, and
every qualified-release binding (ACPX, agent server and command digest).
- Tests (Rust, `acpx_provider_backend`):
-
`interrupted_turn_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
red on master. An old-digest saved state with an active turn restores,
the digest is re-bound, and identity and command digest are kept.
-
`open_session_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
restore gets past the saved-state check, to provider start.
- `saved_sessions_rebind_to_an_upgraded_build_installed_at_a_new_path`:
red on master and on the first fix, with the production error. It covers
an interrupted turn of the same run and a settled session attached by
the next run, both moved to a new install directory.
-
`interrupted_turn_after_upgrade_stays_strict_when_lost_turn_recovery_is_unsafe`:
each unsafe guard still fails with `provider_state_incompatible` and
leaves the saved state untouched.
- The existing strict tests for other runs are unchanged and still pass.
- `packages/paperclip-runner/docs/durable-recovery.md`: documents the
extended re-bind.
- `tests/acpx_warm_attachment.rs`: the end-to-end test
`restores_a_turn_lost_to_a_runner_upgrade_under_the_new_launch_profile`,
from the Phase 2 session's parallel fix. It fails on master with the
production error and passes here.

## Verification

- `cargo test -p paperclip-runner-core --lib acpx_provider_backend`: 44
passed.
- The full `paperclip-runner-core` lib suite: 585 passed. The 3 failures
are timing-based `task_memory` kill-monitor tests on macOS; that file is
untouched (last changed 5 days ago). CI runs them on Linux.
- `cargo fmt --check`: clean.
- Live: the first K14 rerun, at 070365404e, still failed for 3 Claude
runs. The cause was the install path: the saved sidecar path pointed
into the old build's worktree. Fixed in 72e020e13b; a second K14 rerun
has been requested.

## Risks

- **The restored conversation runs on the new provider pack.** That is
already the case for re-bound suspended sessions. Session start still
verifies every artifact and the restored provider identity against the
qualified command digest. A changed upstream release still fails strict.
- **Old provider process.** For an active turn, lost-turn recovery waits
for the provider-lifetime fence before restoring. For an open session,
behavior matches the same-build restart, which reopens without the
fence.
- **A state that really is incompatible** (an upstream provider release
change):
- **Between turns**, the common upgrade case: the next run's checkpoint
carries `allow_replacement_after_resume_failure`, so the refusal opens a
fresh provider session automatically, with no Retry. A new test,
`continues in a fresh session when a new run's saved session is
incompatible…`, pins this in `native-session-runtime.test.ts`.
- **Mid-turn, same run:** this still stops for a board Retry, by design.
The effects of the lost turn are unknown, and the automatic replacement
sweep (`decideNativeReplacement`) only replaces a session when it has a
complete effect inventory.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 16:34:23 -06:00
DottaandClaude Opus 5.5 a8bbdf237c fix(runner): reopen the ACPX session at turn.start when it can be safely reopened (#20110)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native Claude ACPX run must survive restarts and suspensions. A
turn stopped for suspension drops the live ACPX session and leaves the
saved lifecycle `prepared`
> - The next `turn.start` checked for a live session before anything
else, so it failed with "ACPX session is not open". That happened even
though the saved session was safe to reopen
> - In restart chaos this made a Claude approval task (G1 t03) need a
Retry after repeated restores
> - This pull request makes `turn.start` reopen a reopenable session
before it starts the turn
> - The benefit is that a suspended or restored Claude run keeps going
on its own, with no Retry

## Linked Issues or Issue Description

Refs #20089

**What happened?**
In a local restart-chaos G1 (approval) wave, Claude ACPX task t03 failed
after repeated restores with `ACPX session is not open` and needed a
Retry. The Retry overwrote the session directory, so the exact sequence
could not be read back. The same error reproduces directly with
`turn.stop` (suspension) followed by `turn.start` on the same session:
the stop clears the live session and sets the lifecycle to `prepared`,
and `turn.start` never reopens it.

**Expected behavior**
When the saved session can be reopened, `turn.start` reopens it, the
same way `session.open` does, and then starts the turn. "Can be
reopened" means: the conversation identity is recorded, the provider
exit is confirmed, the session belongs to this run, and the lifecycle is
`prepared`, `suspended` or `session_open`. A session that was closed on
purpose, or whose provider exit is unconfirmed, still fails.

**Steps to reproduce**
1. `run.prepare`, `session.open`, then `turn.start`.
2. `turn.stop` (suspension stop).
3. `turn.start` again. Before this change, it fails with "ACPX session
is not open".

**Paperclip version or commit**
master at f789f3e2af. Branch rebased on current master.

## What Changed

- `acpx_provider_backend.rs`: `turn.start` checks whether there is no
live session and the saved state is reopenable. If so, it reopens
through `open_session` first, and the reopen's `session.resumed` event
comes before the turn's events. The previous body moved unchanged into
`start_turn_in_open_session`.
- `tests/acpx_warm_attachment.rs`: a new test,
`reopens_the_conversation_for_a_turn_after_a_suspension_stop`. It drives
prepare, open, turn, stop, turn and expects the second turn to be
accepted with `session.resumed`. It also checks that after
`session.close`, `turn.start` still fails.

## Verification

- Red/green: the new test fails on master with "ACPX session is not
open" and passes with the change. `cargo test -p runner-core --test
acpx_warm_attachment`: 3/3 passed.
- I also ran the full `runner-core` crate suite. Five tests failed: four
`task_memory` monitor tests and `unconfirmed_suspension_state...`. Each
of them passes on an unchanged master checkout, and the ACPX one also
passes when rerun in this branch, so they are timing-dependent and not
caused by this change.

## Risks

- Low risk. A reopen happens only under the same conditions in which
`session.open` would already succeed for this run. A closed session and
an unconfirmed provider exit stay fail-closed. The order of events
changes only by adding `session.resumed` ahead of the turn's events.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 15:46:11 -06:00
DottaandClaude Opus 5.5 c55af26821 fix(runner): treat a replayed start of a turn lost on restore as history (t48) (#20074)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native Codex run must survive a control-plane restart and continue
its work
> - #20057 made recovery adopt an in-flight turn, or start a new turn
when the restored runner reports the old turn lost
> - One case was not covered. A graceful drain can detach the controller
after the runner accepted `turn.start` but before the provider started
the turn
> - The runner then records the provider's `turn.started` while no
controller is attached. After a restart it replays that start, and the
Codex session rejects it as an unexpected turn
> - This pull request treats that replayed start as history from the
earlier generation
> - The benefit is that the run starts its continuation turn instead of
stopping with `native_event_replay_conflict`

## Linked Issues or Issue Description

Refs #20057

**What happened?**
A restart chaos wave ended with one failed task (t48, Codex). A graceful
SIGTERM suspended the controller 300 ms after the runner accepted
`turn.start`. The provider's `turn.started` came after the controller
detached. Recovery classified the run as `resume_dead_runner`. The
restored runner reported the turn as `provider_turn_lost_on_restore`,
which is correct. But it also replayed the undelivered `turn.started`.
The Codex session failed with "Provider started an unexpected turn."
(`turn_binding_mismatch`), and recovery stopped with
`native_event_replay_conflict`.

**Expected behavior**
Recovery starts a new continuation turn. The replayed start of the lost
turn is recorded as history, and the run completes.

**Steps to reproduce**
1. Start a Codex native run.
2. Detach the controller gracefully after the runner accepts
`turn.start` and before the provider emits `turn/started`.
3. Let the runner record `turn.started` while it is unattached, then
kill the runner and the provider.
4. Restore the run. Before this change, the session fails with
`turn_binding_mismatch`.

**Paperclip version or commit**
master at a9cf4ee731 (the wave ran on 66a2cbab98).

## What Changed

- `codex-app-server-driver-impl.ts`: recovery identifies the turn that
this run's latest `turn.start` had the runner accept, when recovery did
not adopt it and provider history does not show it live. Recovery passes
it to the session as `lostAcceptedTurnId`.
- `codex-session-state.ts`: the session keeps `lostAcceptedTurnId` as an
unconfirmed lost turn. The stale-event rule treats its replayed start
and items as history.
- `codex-session-notifications.ts`: `turn/started` applies the
stale-event rule to earlier-generation and unconfirmed lost turns. The
runner's restart-loss terminal confirms the lost turn. Any other
terminal, or a semantic result, from that turn fails closed, so a real
outcome is never dropped. A start for a turn that this generation
settled, or for a turn the session never saw, still fails closed.
- `runnerd-restart-continuation.test.ts`: a new connected test with the
real runner and the fake Codex server reproduces the t48 shape. A new
fake option, `--turn-started-gate`, holds the provider's start until the
test detaches the controller, so the test does not depend on timing.
- `codex-app-server-driver.terminal-replay.test.ts`: driver tests for
the replayed start (history), and for an ordinary failure or a result
from the lost turn (fail closed).

## Verification

- Red: before the fix, the new test failed with
`NativeProviderTerminalFailure: Provider started an unexpected turn.` at
`codex-harness-session.ts:211`. This is the same error and stack as the
chaos run.
- Green: `npx vitest run src/drivers/codex
src/live/runnerd-restart-continuation.test.ts` (in
`packages/paperclip-runner`) passed 392 of 392.
- The wider Codex, restart and recovery test files (45 files, 1153
tests) passed on the final revision. The two fail-closed driver tests
fail on the first revision, where the lost turn's outcome was dropped as
history. The one failure in an earlier run ("still rejects history for a
turn that became terminal in the live session") showed that the first
version was too broad. The rule now applies only to earlier-generation
turns, and that test passes.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in
`packages/paperclip-runner` passed.

## Risks

- Low risk. The change only applies after a restore, to a turn that the
runner accepted for this run's latest `turn.start`, and only when
recovery did not adopt that turn and provider history does not show it
in progress.
- A turn that the session never saw still fails closed. A re-start of a
turn this generation settled still fails closed. A real failure or a
result from the lost turn fails closed instead of being dropped.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 11:14:24 -06:00
DottaandClaude Opus 5.5 c0c26188db fix(runner): stamp event generations and record an earlier generation's events as history (#19960)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs execute through the Paperclip Runner, which turns
provider output (Codex, OpenCode, Claude through ACPX) into durable
runner events
> - In the restart chaos campaign, events from an earlier turn or runner
generation failed the current turn about ten times (RC-15, 20, 20b, 20c,
33, 33b, 59 and 61)
> - #19718, #19722, #19772 and #19783 each special-cased one variant,
and other variants still fail the run
> - The durable-ownership plan (section G, invariant 6) asks for one
rule: every event carries its generation and turn, and an earlier
generation's events are recorded as history and never fail the current
turn
> - This pull request stamps the runner generation on every event and
applies one stale-event rule in the ACPX provider state, the Codex
driver (which Runnerd also uses for OpenCode) and the pending
`turn.start` check
> - The benefit is that a late report about a settled or restored turn
no longer stops a healthy turn, while events for a turn the runner never
saw still fail closed

## Linked Issues or Issue Description

Refs #19718, #19722, #19772, #19783

**What happened?**
A runner or provider can report a turn after that turn settled: a
resumed provider reports the lost turn's usage or terminal, an ACPX
sidecar emits a last delta after the turn terminal, or a held failure of
the previous turn sits in front of a new `turn.start`. Several of these
paths fail the current turn: ACPX `AcpxEventScope` rejects any event for
a non-active turn ("named a stale turn"), the Codex driver fails
`turn_binding_mismatch` for a settled turn's late item between turns,
and `runnerdPendingTurnStartFailure` reports any held non-recoverable
failure, whatever turn it names. Runner events carried no generation.

**Expected behavior**
An event for a turn that is not active, and that an earlier generation
accepted or this generation already settled, is recorded as history and
never fails the current turn. An event for a turn the runner never saw
still fails closed.

**Steps to reproduce**
ACPX: settle turn 1, start turn 2, and deliver a late `runtime.event`
for turn 1. Before this change, the provider state returns an error and
the run fails.

## What Changed

- Runner (`durable/state.rs`, `durable/runner.rs`): every event payload
carries `generation`, the runner process generation that emitted it. A
runner process that recovers state starts the next generation
(`begin_recovered_generation`); it is saved with the next normal write,
so recovery adds no write before the first turn. Loading stays pure. Old
saved state loads as generation 0. Outbox events keep their original
generation.
- PRP event schemas v1–v3 declare the optional `generation` property;
generated validators and the manifest are regenerated.
- ACPX (`acpx_event_scope.rs`, `acpx_provider_state.rs`): an event that
names a settled turn of this run becomes a bounded
`stale_generation_event` diagnostic (32 per state) and changes no state.
Unknown turns still fail.
- Codex driver (`codex-session-state.ts`,
`codex-session-notifications.ts`): one predicate, `isHistoryTurn()`: the
turn is not active, and it was restored from the checkpoint (an earlier
generation) or settled in this generation. The classifier's stale-turn
branch, the resume-usage branch, restored-terminal replay and the main
fail point (`notificationNamesActiveTurn`) all use it. A terminal for an
earlier generation's turn is history even when it disagrees with the
checkpoint. A terminal that contradicts a turn this same generation
settled is still a `conflicting_turn_terminal` integrity failure, and a
provider that starts a settled turn again is still rejected.
- Transport (`runnerd-codex-transport.ts`): a pending `turn.start`
ignores a held failure that names another provider turn.
- `DURABLE_TRANSPORT.md` documents the stamp and the rule.

## Verification

- `cargo test -p paperclip-runner-core`: new
`records_late_events_for_a_settled_turn_as_history` (red before),
`stamps_every_event_with_the_runner_process_generation`; two existing
ACPX tests now assert "recorded as history, no state change" instead of
"rejected". Durable tests: 112 passed. `cargo fmt` clean.
- `codex-app-server-driver.stale-generation.test.ts` (new, 9 tests):
late item, delta, terminal and workspace change for a settled turn
during and before the next turn, an earlier generation's conflicting
terminal after recovery, and an unknown turn that still fails. Red on
master: 8 of 9 (the during-turn cases on master already dropped the
event, with a different diagnostic; the between-turn and recovery cases
failed the session).
- `runnerd-pending-turn-start-failure.test.ts` (new): an earlier turn's
held failure no longer fails the pending start; the pending turn's own
failure and a failed run terminal still do.
- `vitest run src/drivers/codex src/live/runnerd-codex-transport`: 787
passed; 2 Codex executable-resolution tests timed out under host load
and pass when run alone.
- `tsc -p tsconfig.json` for the runner package, `check:protocol-types`,
`check:replay-goldens`: clean.
- Not run: chaos waves (the coordinator schedules them on this host).

## Risks

- A late report about a settled turn is now dropped with a diagnostic
instead of failing. A provider that keeps working on a settled turn
could have its output ignored silently; a provider that starts a settled
turn again, or names a turn the runner never saw, still fails closed.
- `generation` is a new optional field. Consumers that do not know it
ignore it, as the PRP contract requires.
- The generation counter is written with the next normal state save, so
there is no new write before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 23:16:00 -06:00