mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 03:29:18 +01:00
ddfbbfff202dfba53b39b342ec4e8a5afbf584ae
283
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
ddfbbfff20 |
fix(runner): recover provider-started turns after a controller restart (#21612)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - The Runner can keep Claude while a background shell is running and give the cycle of Claude to the controller as a turn that the provider started. Core can wait for that turn. #21607 lets a caller ask the transport for it. > - A controller can restart at any time, for example at a deploy. A wait can last 10 minutes and a provider-started turn can last longer, so a restart during one of them is a normal event. > - Tests with the real `paperclip-runnerd` showed that such a restart failed the session. The checkpoint that Core had was the checkpoint of the turn before, and the recovery of the driver refused or mislabelled the turns that it did not know. > - This pull request makes that restart safe. It changes no production run, because no caller asks for provider-started turns before part 3. > - The benefit is that part 3 can turn the behavior on without a failed run at each deploy. ## Linked Issues or Issue Description No public issue exists. The issue is described here. #21456 turned the behavior on and had grown too large to review. It is now three stacked pull requests: 1. #21607 (merged): the transport can ask the Runner for provider-started turns. No caller asks. 2. **This pull request:** the recovery after a controller restart during a wait or during a provider-started turn. 3. #21456: Core sets the option for native ACPX Claude runs. Only that part changes what a production run does. This is part 2. Its base branch is `master`. Refs #19912 (design), #21456. **What existing behavior does this improve?** None in production. It makes a controller restart safe for a run that asked for provider-started turns. **Subsystem affected** Paperclip Runner package: the consumer (`src/native-session-runtime.ts`), the session backend (`src/backends/harness-driver-backend.ts`), the driver (`src/drivers/codex/`) and the transport (`src/live/`). Only for a run that asked for provider-started turns. **Current behavior** - The consumer saves no checkpoint when a wait starts or when a provider-started turn starts. During a wait and during a provider-started turn, the checkpoint that Core has is the checkpoint from the start of the turn before. It names that turn as active. - A restart from that checkpoint fails: the driver has the old turn as active, gets the events of a provider-started turn, and stops the session with `turn_binding_mismatch`. - The recovery of the driver adopts one turn that it does not know as a disposition recovery turn, and refuses two or more. With two or more, it closes the transport, and that stops the Runner and Claude. - The Runner does not send again an event that the controller acknowledged. If the start of a provider-started turn was acknowledged before the restart, the new driver gets the other events of that turn with no start. - A turn that ends waiting for a provider-started turn sets the terminal of the session, as a turn that ends the run does. After a restart, if the provider-started turn already has its result when recovery completes, the consumer finds a result and a terminal, settles the run, and does not read the events of that turn. The start, the items, the usage event and the terminal of that turn do not reach Core. - The recovered driver reads notifications from its construction, and `reconcile()` maps the terminal of the saved turn only after its reads. If the saved checkpoint is still the checkpoint of the controller turn and the start of the provider-started turn is already queued, the driver judges that start too early and stops the session with `turn_binding_mismatch`. - A checkpoint that waits can also have a consumed disposition marker (a recovery turn after an earlier restart ended waiting). The driver restores that session as closed and reads no notifications, and the consumer settles the run on the earlier terminal with no result. **Proposed behavior** - **Checkpoints.** The consumer saves a checkpoint when a wait starts and when a provider-started turn starts, with the function that it already uses for a governed wait. A checkpoint that fails stops the turn, as for a governed wait. - **A checkpoint as of the event that Core stored.** The driver gives each event its sequence when it reads it from the provider, before the consumer stores it. A snapshot that is taken later can be ahead of Core. For these two checkpoints, the driver captures its snapshot when it has handled the notification that made the event (the `turn.completed` that waits, or the `turn.started` of the provider), with the sequence of that event. The consumer asks for the snapshot as of the event that it has just appended. The saved state and cursor are then those of an event that Core has. - **Session terminal.** A turn that ends waiting for a provider-started turn does not set the terminal of the session (`HarnessNativeSession`). The terminal of the session is the end of the run, and that turn does not end the run. A checkpoint that is saved during a wait or during a provider-started turn has no terminal, so a recovered session settles only from the events that it reads, in order, to the terminal of the turn that has the result. - **Driver, a checkpoint that waits.** Recovery keeps the wait. It does not adopt and does not refuse the turns after the last terminal that it knows. The Runner sends again, in order, each event that was not acknowledged, and the driver admits each provider-started turn from that stream, as it does with no restart. - **Driver, a checkpoint with an active turn.** `reconcile()` accepts one turn in progress that comes after the saved active turn, when the saved turn ended with the wait marker. It refuses each other unexpected active turn as before, and it refuses this one too when the reconciled turn settled a result. - **Driver, a start that is queued before recovery reconciled the saved turn.** A `turn/started` of the provider that arrives while the session does not wait for one first waits until `reconcile()` has settled, and is then judged as before. `reconcile()` judges the live state after its reads; if the stream already settled the saved turn into a wait, it accepts exactly one turn in progress after the last terminal turn and does not adopt it. The driver admits that turn at its start. - **A wait with a consumed disposition marker.** The driver keeps reading notifications for a restored session that waits, as it does for a wait with no restart. It still refuses a second controller-started turn. The consumer does not take the consumed-marker shortcut for a recovered snapshot that waits, so the wait is resumed. - **Transport, the start comes first.** For each driver, the transport gives the `turn/started` of a provider-started turn before each other notification and before a tool call of that turn. If it did not give that start to this driver, it builds it from the `turn.started` event in its stored events. It does not do this for the turn that the recovered driver already has as active (`resumeActiveTurnId`). It binds the events and the tool call of a provider-started turn to that turn, not to the turn that the Runner reports as active at the time of the recovery read. - **Transport, which start owns an event.** The stored events can have gaps: the window removes old events but keeps the latest of each anchor type, and the controller checkpoint drops streamed deltas. The owner of an event is the last stored `turn.started` before it, if it is a provider start, and if the first turn terminal that is stored after that start is not before the event and is the terminal of the same provider turn (`providerTurnId`). If the first stored terminal after the start is the terminal of a different turn, the end of the provider turn is not known, and the transport builds no start. **Reason and benefit** - The checkpoint is the cause. With a true checkpoint, the two states of a restart are states that recovery can handle: a wait, or an active turn. - The Runner already sends again each event that was not acknowledged. Recovery uses that stream and does not build the turns again from a read. - The window removes a terminal only while a later terminal of the same type is stored. Thus the first stored terminal after a start shows if a terminal of that turn was removed. A rule that only ignores the gaps is not safe: the Runner can make a `turn.failed` for a turn that did not emit its start (`provider_turn_lost_on_restore`). ## What Changed Production code: 113 lines added, 19 removed. - `src/native-session-runtime.ts` (15 added, 4 removed): the two checkpoints in `consumeTurn`, with the sequence of the appended event, and the resumed wait with a consumed disposition marker. - `src/drivers/codex/codex-session-state.ts` (24 added, 1 removed), `codex-session-terminal.ts` (2 added), `codex-session-notifications.ts` (2 added), `codex-harness-session.ts` (24 added, 7 removed), `codex-app-server-driver-impl.ts` (4 added, 1 removed): the snapshot at the event, and the recovery of the driver. - `src/contracts/harness-driver.ts` and `src/contracts/native-session-backend.ts` (1 added each): an optional `boundarySnapshot` on a driver session, and an optional sequence in the snapshot options. Other drivers do not implement it and are not changed. - `src/backends/harness-driver-backend.ts` (7 added, 5 removed): the session terminal, and the snapshot as of a sequence. - `src/live/runnerd-codex-transport.ts` (15 added, 1 removed) and `src/live/provider-turn-tool-order.ts` (18 added): the start-first rule and the owner of an event. - Tests: `src/live/runnerd-acpx-provider-turn.test.ts` (572 added, 5 removed), `src/drivers/codex/codex-app-server-driver.provider-turn.test.ts` (348 added), `src/live/runnerd-codex-transport.test.ts` (137 added), `src/control-plane/durable-prp-control-plane.test.ts` (84 added), `src/live/provider-turn-tool-order.test.ts` (new, 61), `src/native-session-runtime.provider-turn.test.ts` (92 added, 6 removed), `src/backends/harness-driver-backend.test.ts` (56 added). - `runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs` (test binary only, 123 added, 14 removed): a delay before a provider-started turn, a goal state at its terminal, a list of provider-started turns for one session, a gap between them, a turn that stays open until the new controller reads the snapshot, and a turn that starts after that read. The modes that existed are not changed, and the fake reads its input on one thread as before. ## State saved by the previous release - No new persisted field. The driver session already has `awaitingProviderTurn` (#21453). - **More checkpoints:** one when a wait starts and one when a provider-started turn starts. An ordinary turn saves the same number of checkpoints as before (4 in the unit test, measured before and after). - **No terminal in a checkpoint of a wait.** A checkpoint that is saved during a wait or during a provider-started turn has `terminal: null`. When a wait ends with no provider-started turn (the limit, a comment of a person, or the Runner ends the run), the last checkpoint of the run also has `terminal: null`; before, it had the terminal of the turn that waited. One reader uses that field with no result: the text of a cleanup failure (`nativeCheckpointRecordsCompletedTurn`), which then does not say that the turn completed. In the tests with the real daemon (the limit, the provider process lost, the sidecar exits), the run still ends on the path of today (`native_finalization_missing`). - **A checkpoint of the previous release** has no waiting state. Recovery takes the path of today. - **Rollback of the Runner package during a wait:** the previous code reads the checkpoint, ignores `awaitingProviderTurn`, and takes its recovery path of today. ## Verification With nothing from part 3. The Rust checks and the build checks in the first item were run before the last seven commits; these commits change no Rust. The other items were run at the head of this branch (`36d5e5e587`), or at an earlier commit of this branch for a suite whose files the later commits do not change. - Before the last seven commits: `pnpm run typecheck:typescript`, `pnpm run build:typescript` and `node --test test/acpx-sidecar-contract.test.mjs` (18 passed) in `packages/paperclip-runner`; `node scripts/verify-runner-vendor-dependencies.mjs` in `server`; `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`; `cargo fmt --check`, `cargo build`, the 17 `acpx_*` suites (241 passed) and the `acpx` unit tests (102 passed): all passed. - `npx tsc --noEmit -p tsconfig.json` and `npx tsc --noEmit -p tsconfig.test.json` in `packages/paperclip-runner`; `pnpm typecheck` in `server`: all passed. - `src/live/runnerd-acpx-provider-turn.test.ts`, 4 runs with the production code of this head, 26 of 26 passed in each. The Core of these tests follows the production append (`appendNativeSourceEvent`): it stores an event that is out of order, keeps its contiguous cursor below a gap, and reports a duplicate or a conflict for an event that it has. It does not reject a gap in the sequence; production does not. The restart cases, all with the real `paperclip-runnerd`: - **From the checkpoint that the real consumer saved.** The first `executeNativeSession` is fenced and detached as the server does for a restart, and a second one recovers from the last checkpoint that the first one saved. (a) Restart during the wait; the provider-started turn starts after the new controller is attached. (b) Restart while a provider-started turn is in progress. In each: the test asserts the shape of that checkpoint, the run settles with the result of the provider-started turn, each event is appended one time and in order, the turn has one `turn.started` with `origin: "provider"`, and the session closes with no error. - **Without the two checkpoints, each of these cases fails:** the saved checkpoint is the checkpoint of the turn before, and the second controller fails with `turn_binding_mismatch` or with a replay conflict. - **A slow `appendEvent`, with events that arrive behind the stored event** (3 cases, each run 3 times with the fix and 3 times without it, with the same result in each run). R7: the append of the `turn.completed` that waits is slow, and the provider-started turn starts behind it. With the snapshot at the event, the checkpoint is the waiting state at the sequence that Core has, and Core gets the `turn.started` of the turn one time. Without it, the checkpoint names the provider-started turn as active, the transport builds no start, and Core never gets that `turn.started`. R8: the first controller answers the `paperclip_finish` call while the append of the provider `turn.started` is slow. With the snapshot at the event, the checkpoint has no result and the run ends with `native_finalization_missing` (see Risks). Without it, the checkpoint has a result that Core does not have, and recovery stops with "session cannot start another turn". R6: the append of the provider `turn.started` is slow and a delta is queued behind it. With the snapshot at the event, the new controller continues the sequence that Core has. Without it, the only difference is one sequence number that Core never gets. - **The start was acknowledged before the restart** (3 cases: one turn open across the restart; two turns; a turn that waits for the answer to its `paperclip_finish` call). The new driver gets the start of each turn one time before its other events, the finish call is answered exactly one time, and the session closes with no error. - **The turn ran with no controller attached** (2 cases, with and with no result) and **two turns in one outage** (2 cases): the turns arrive in order after the recovery. - **A provider-started turn that is active in the checkpoint** gets no second start, and the next provider-started turn arrives after it. - **The turn already has its result when recovery completes** (3 cases: the turn made its `paperclip_finish` call with no controller attached; the turn waited for the answer to that call at the restart; two turns across the restart). The test asserts the exact list of events that Core gets: each provider-started turn has one start, its usage item, its message item and its terminal, and then `run.result.accepted` and `run.terminal`. `completeRun` is called one time with the turn that has the result. Before the change to the session terminal, the second controller appended only `run.result.accepted` and `run.terminal` in each of these cases. - **The wait checkpoint is not saved, then a restart** (R10). The last checkpoint is the checkpoint of the controller turn, Core has the terminal of that turn, and the start of the provider-started turn is queued. The run settles with the result of the provider-started turn, Core has one `turn.started` for it, and the finish call is answered one time. Without the wait in the driver, recovery stops with `turn_binding_mismatch`. - **An ordinary turn**, with no option. One that waits for the answer to its finish call at a restart: Core gets its result, its terminal and the run terminal one time. One whose checkpoint after `startTurn` is ahead of Core at a restart (R9): the run settles, and Core does not have only the one event that the first controller did not store. These cases pass with and without this pull request. - **No checkpoint with a terminal and no result** is saved by the two controllers in the cases above. - `src/live/provider-turn-tool-order.test.ts` and `src/live/runnerd-codex-transport.test.ts`: 345 passed. The helper tests cover a stored start with removed events after it, a dropped delta, the terminal of the turn before and after the event, a terminal of a different turn after a removed terminal, and a controller start. With no option, with a different agent or provider, and for a turn that the controller started, the notifications of the transport are the same as before this pull request. - `src/control-plane/durable-prp-control-plane.test.ts`: 230 passed. Three new tests use the real commit path: a full window of 4,096 events removes the events after a provider start and the start is found; the window removes the `turn.failed` of a provider turn after a later `turn.failed` with no start, and no start is found; a restored checkpoint that dropped the deltas of the turn, and the start is found. - `src/drivers/codex` and `src/backends/harness-driver-backend.test.ts`: 491 passed. The driver tests cover a provider start that is queued before `recoverSession()` is called, the wait state with one turn in progress at `reconcile()` (and the two cases that it must still refuse), a restored wait with a consumed disposition marker (the provider turn is admitted and `startTurn` is refused), the snapshot at the two events (also after the driver moved on, and for a restart continuation that ends waiting), one later turn in progress, one completed and a second in progress, two completed with a result, and the cases that `reconcile()` must still refuse. The backend tests cover the session terminal and the snapshot as of a sequence. - `src/native-session-runtime.provider-turn.test.ts` and `src/native-session-runtime.test.ts`: 333 passed. Two new cases resume a restored wait with a consumed disposition marker: the provider-started turn arrives and settles the run, or the wait ends at its limit and the run takes the path of today with no new turn. - `server`: `native-session-executor.test.ts`, `native-provider-turn-wait.test.ts` and `native-runner-restart-recovery.integration.test.ts`: 1,743 passed, 21 skipped. - Mutation checks, each made named tests fail: no checkpoint at the start of a wait; no checkpoint at the start of a provider-started turn; the consumer does not pass the sequence (R6, R7 and R8 fail: a hole in the sequence, no `turn.started` in Core, a different failure); the backend ignores the sequence; the driver has no snapshot at the event; no start-first rule; no exclusion of the active turn of the recovered driver; the turn binding only at the first delivery; the contiguous sequence required again; no check of the first terminal after the start; the adoption of later turns put back; no exception in `reconcile()`; no check after the settlement in `reconcile()`; the terminal of the session set again by a turn that ends waiting; no wait of the provider start for `reconcile()`; no exception in `reconcile()` for a session that waits; the pump deferred for a restored wait; the consumed-marker shortcut taken for a snapshot that waits. Not run: the full suite. CI supplies that coverage. ## Risks - **No production effect.** Each change is behind the waiting state, a provider `turn.started`, or the option of the transport. No caller sets the option before part 3. - **An event that the first controller received but Core did not store is not sent again after a restart.** This is the same on `master` and is not changed here. Only the terminal of a turn is built again. If the lost event is the result of a provider-started turn (the first controller answered the `paperclip_finish` call and stopped before Core stored the proposal), the run ends on the path of today for a turn with no result. R8 shows this. R9 shows the same loss for one event of an ordinary turn. - **A second `turn.started` for one provider-started turn can reach Core.** This occurs if the restart comes after Core stored the start and before the checkpoint of that start is saved (the time of one checkpoint write). The last checkpoint then waits, and the transport builds the start again for the new driver. Seen with the real daemon when the checkpoint write is held; not in a committed test. The usage accounting of the server uses the turn id as its key, so there is no second count, but the second start resets the "restored" mark of that turn. The transcript does not read `turn.started`. - **The older checkpoints can also save a cursor that is ahead of Core** (measured for the checkpoint after `startTurn` of an ordinary turn). Production accepts the gap in the sequence that follows, so this is not a failure there. It is on `master` and is not changed here. - **When the terminal of a provider-started turn was removed from the stored events,** the transport builds no start for an event of that turn, and the session stops on that event, as before. This needs a later terminal of the same type with no start of its own (a turn that was lost while it started) and a full window. - **The last checkpoint of a wait that ends with no provider-started turn has no terminal,** as the section above says. The change is only in a run that waits, and no run waits before part 3. - **A wait with a consumed disposition marker has no test with the real daemon.** The state needs two restarts in one run, and the test helper does one. Unit tests of the driver and of the consumer cover it. - **A provider start that waits for `reconcile()`** waits as long as the reads of `reconcile()` take. Each read ends at the command deadline of the Runner (30 seconds by default) or when the transport closes. - **`item.delta` events across a restart are not asserted.** A streamed delta that the first controller received is not sent again. The completed items are asserted. - **The driver keeps a maximum of 8 snapshots at events** and drops each one when the consumer asks for it or for a later one. If the consumer is more than 8 such events behind, the checkpoint uses the current snapshot, as before this change. - **A second provider-started turn that also ends during the same outage** is not tested with the real daemon: the fake gives each terminal the same goal revision, and the Runner then holds the second terminal until a controller is attached. The end of the reconnect time of the Runner is not tested here. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, decided the designs, reviewed the diffs, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as Claude Code subagents with tool use (shell, file edits, local test runs), wrote the change and the tests, reproduced the review findings with the real daemon, and verified this slice alone. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
d4498ae7ac |
fix(runner): settle every recorded tool result and free sessions blocked by a closed turn's pending call (#21490)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call Paperclip tools through the Runner. The Runner records
each call and returns the controller's result to the provider (here
OpenCode)
> - In chat-eval cohort 13, a `list_workflows` result was recorded but
never reached OpenCode. OpenCode waited about 290 s, and the call stayed
pending
> - The next run's `run.attach` then failed with
`durable_pending_tool_calls`, so the person's follow-up question never
got an answer
> - This pull request fixes the result shape that caused the rejection,
makes the Runner settle every recorded result, and lets the next run
settle a call that a closed turn left pending
> - The benefit is that a tool result always reaches the agent, and a
stuck call can no longer block every later turn
## Linked Issues or Issue Description
**What happened?**
Chat-eval cohort 13 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `df1b12ad8e`). In
`cases.case-history` and `cases.create-case`, the CEO called
`list_documents` (or `list_projects`) and `list_workflows` (call id 5)
together. The server answered `GET /companies/{id}/workflows` with 200
in 584 ms. The OpenCode MCP bridge diagnostic for call 5 says `outcome:
"unanswered"`, `written: false`, `handlerMs: 289038` (290723 in
create-case). The sibling call was answered in about 1 s. The run ended
with `runner_settlement_incomplete`: `semanticTools.pending` held call 5
and its `semantic_tool.result` delivery command had status `failed`. The
next question started a run whose `run.attach` was refused: `run.attach
requires a settled Codex provider session: durable_pending_tool_calls`.
The run failed in 17 s, and the question was not answered.
Cause: `executeNativeWorkflowTool` returned the Runner API result
unchanged: `{ ok: true, status: 200, operationId: "GET
/api/companies/{companyId}/workflows", ... }`. Every other API-backed
tool renames that field to `apiOperationId`. The Rust tool bridge
(`semantic_response_value`) reads an object with a boolean `ok` and an
`operationId` as a semantic envelope that must name this tool and call.
The REST route did not match, so `apply_result` failed with `semantic
result envelope does not match its provider call`. Nothing was sent to
the provider, and the call stayed pending forever. The red test
reproduces this exact error. The parallel call is not the trigger: the
sibling call settled normally, and the OpenCode MCP bridge answered
other parallel pairs in the same runs. `list_workflows` first appears in
cohort 13, so cohorts 11 and 12 did not hit it. All four workflow tools
had this shape.
**Expected behavior**
Every recorded tool result reaches the provider. When a recorded result
cannot be delivered, the provider gets an error at once and the call
settles. A call that a closed turn left pending does not block the next
run.
**Steps to reproduce**
1. Run an OpenCode agent with Cases enabled on the durable Runner.
2. Ask a question that makes the agent call `list_workflows`.
3. See the call wait about 290 s. Then ask a follow-up question and see
`run.attach` fail with `durable_pending_tool_calls`.
**Paperclip version or commit**
`master` at `df1b12ad8e`.
## What Changed
- `server/src/services/native-runtime/native-workflow-tools.ts`:
workflow tool results name the route `apiOperationId`, like case tools
and `call_api`.
- `provider_bridge.rs`: add `settle_undeliverable_result`. When the
bridge rejects a recorded result for a call it still holds, the call
settles with a bounded, non-retryable runner error. Conflicting or
unknown results still fail as before.
- `provider_backend.rs` (`deliver_semantic_result`): use that
settlement, deliver the error to the provider, and record a
`semantic_tool_result_undeliverable` diagnostic with the call,
operation, reason and the rejected result's digest.
- `provider_bridge.rs` and `provider_backend.rs` (`attach_run`): add
`expire_closed_turn_calls`. When nothing is live (no active or ambiguous
turn, no provider request waiting), `run.attach` settles calls that a
closed turn left pending as an expired failure with an unknown outcome,
and records `semantic_tool_result_expired`. The provider stopped waiting
when its turn ended, and the new run's controller never delivers the old
run's results. A call that a live turn still waits on still blocks
attachment. This follows the existing `tool_call_interrupted_by_restart`
rule: never execute the operation again, and record an unknown outcome
when no result can ever arrive.
- Docs: `doc/run-log-events.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe both
diagnostics.
## Verification
- Red/green, Rust (`cd packages/paperclip-runner/runner && cargo test -p
paperclip-runner-core --test codex_provider --
a_rejected_recorded_result run_attach_expires`):
- Before:
`a_rejected_recorded_result_still_answers_the_provider_and_frees_the_session`
fails with `semantic result envelope does not match its provider call`
(the production error).
`run_attach_expires_a_call_left_pending_by_a_closed_turn` fails with
`run.attach requires a settled Codex provider session:
durable_pending_tool_calls` (the production refusal).
- After: both pass. The fake provider logs `tool-response:failure`, so
it got an answer at once. The next `run.attach` succeeds.
- Parallel calls (`cargo test -p paperclip-runner-core --lib --
provider_bridge`):
`a_rejected_result_in_a_parallel_pair_still_settles_its_own_call` sends
the cohort 13 pair (`list_documents` + `list_workflows`). The rejected
call settles, its sibling keeps its own result, and the next turn
starts. `only_a_closed_turn_expires_its_pending_calls` checks that an
open turn expires nothing.
- Upgrade and mid-run cutover:
`run_attach_expires_a_call_left_pending_by_a_closed_turn` saves state
with a pending call after the turn ends, stops that Runner, and attaches
with a new Runner process from the saved state.
`durable_backend_settles_pending_tools_when_recovery_finds_the_turn_ended`
(unchanged) still passes: a late result in the same run still settles as
`settled_after_turn`.
- Red/green, server (`cd server && npx vitest run
src/services/native-runtime/native-case-tools.integration.test.ts -t
workflows`): before, `expected { ok: true, status: 201, … } to not have
property "operationId"`; after, 2 passed.
- `cargo test -p paperclip-runner-core`: 643 lib tests pass. One lib
test (`fenced_receipt_output_keeps_prefix_when_stdout_never_closes`)
failed once with `EPERM` while signaling a process group under parallel
load. It passes alone 3 of 3 and does not touch this change. The
integration test binaries pass.
- Paid narrow run for `cases.case-history` and `cases.create-case`: see
the comment below.
## Risks
- Previous-release story: no persisted shape changes. State saved by the
previous Runner with a call pending after its turn closed carries
forward. The new Runner settles that call at the next `run.attach`, and
the session continues (tested from saved state with a new process). A
previous-release Runner that receives the corrected `apiOperationId`
result delivers it normally, because the result is no longer read as an
envelope.
- Behavior change: a call left by a closed turn is now settled at the
next run's attachment, not kept forever. Its outcome is recorded as
unknown, and the operation is never run again. A provider turn that is
still live keeps the old refusal (`durable_pending_tool_calls`,
`provider_side_effect_pending`), and the controller still never replaces
the session.
- A recorded result that the bridge rejects now becomes an error for the
agent, not an endless wait. The agent is told the action may have
completed and must check the current state before repeating it.
- Latency: nothing is added before turn start. The attach-time check
walks the in-memory pending calls (usually none) and adds no I/O beyond
the existing state save. The rejected-delivery path now answers in the
same command instead of after the provider's ~290 s client timeout, and
the follow-up run no longer fails after 17 s.
## Model Used
- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Refs #21398 (first release where `list_workflows` was called in this
eval), #19733 (the attach refusal), #20886 (earlier OpenCode delivery
fix).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
68d5c9c120 |
feat(runner): let a caller ask the transport for provider-started turns (#21607)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - Today the run ends when the turn ends. The command stops, and the run fails with `native_finalization_missing`. > - The Runner can already keep Claude while a background shell is running and give the cycle of Claude to the controller as a turn that the provider started (#21423, #21316). Core can already wait for that turn (#21453, #21454, #21455). No run does this yet, because no controller asks the Runner for it. > - The request goes through the transport of the Runner package. This pull request gives the transport an option for it and the behavior that the option needs. > - No caller sets the option in this pull request. It changes no production run. > - The benefit is a small base for the two pull requests after it: the recovery after a controller restart, and the switch that turns the behavior on. ## Linked Issues or Issue Description No public issue exists. The issue is described here. #21456 turned the behavior on and had grown too large to review. It is now three stacked pull requests: 1. **This pull request:** the transport can ask the Runner for provider-started turns. No caller asks. 2. #21612: the recovery after a controller restart during a wait or during a provider-started turn. 3. #21456: Core sets the option for native ACPX Claude runs. Only that part changes what a production run does. Refs #19912 (design), #21456. **What existing behavior does this improve?** None in production. It adds the support that parts 2 and 3 use. **Subsystem affected** Paperclip Runner package, the controller side: `src/live/runnerd-codex-transport.ts`. ACPX Claude only, and only for a caller that sets the new option. **Current behavior** - The transport has no way to ask the Runner for provider-started turns. **Proposed behavior** - The transport options have a new field, `providerStartedTurns`. The transport uses it only for ACPX with the agent `claude`. In each other case the field has no effect. - With the option, the transport sends `providerStartedTurns: true` in `turn.start`. It does not send it for a compaction turn. - With the option, a tool call in a provider-started turn waits until the transport gave each earlier event to the driver. Without this, a `paperclip_finish` of that turn could reach the driver before the `turn.started` of that turn, and the driver would refuse it. - With the option, the transport gives a `run.terminal` of the Runner to the driver as a marked internal notification (`paperclip/runTerminal`). The driver uses it only during a wait, so a wait ends at once when the Runner says that the run is over. - With no option, the transport does the same as before this pull request. Live sessions, evals, the CLI runner and the probes do not set the option. **Reason and benefit** - The opt-in is a choice of the caller, not a property of the agent. Only a consumer that can wait for a provider-started turn must ask for one. `CapabilityLiveSession` completes a turn at the first `turn/completed`, so it must not get the behavior because its agent is Claude. ## What Changed - `src/live/runnerd-codex-transport.ts` (31 lines added, 1 removed): the option, the field in `turn.start`, the order of the tool call, and the `paperclip/runTerminal` notification. - `src/live/provider-turn-tool-order.ts` (new, 16 lines): finds the source sequence of a tool call that belongs to a provider-started turn. - `src/live/runnerd-acpx-provider-turn.test.ts` (new, 239 lines): 7 cases with the real `paperclip-runnerd`, the fake ACPX sidecar, and the real consumer of Core. - `src/live/live-session.test.ts` (12 lines added): a Claude live session gives the transport no such option. - `runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs` (test binary only): the mode `provider-result-resolved`. In this mode the provider-started turn calls `paperclip_finish` and waits for the answer of Core through the Runner (`tool.resolve`), as the real sidecar does. The mode `provider-result` is not changed. ## State saved by the previous release - No persisted state changes in this part. - **A Runner that does not know the field** ignores it. - **Rollback of the Runner package:** the previous transport does not send the field, so the Runner does not wait. ## Verification At the head of this branch (`2f2cb14b9c`), with nothing from parts 2 and 3: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript` and `pnpm run build:typescript` in `packages/paperclip-runner`; `pnpm typecheck` and `node scripts/verify-runner-vendor-dependencies.mjs` in `server`; `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`; `node --test test/acpx-sidecar-contract.test.mjs` (18 passed): all passed. - `src/live/runnerd-acpx-provider-turn.test.ts`, 3 runs, 7 of 7 passed in each. The cases: - background work, then the turn of Claude calls `paperclip_finish`; Core answers; the Runner gives the answer to the sidecar with exactly one `tool.resolve`; the turn completes; the run finalizes with one `run.result.accepted`; - a Claude run with no option from the caller takes the path of today: no `providerStartedTurns` in `turn.start`, no `paperclip/runTerminal`, no wait, and no `session.provider_turn` to the sidecar; - the limit ends the wait, the run takes the path of today, and the provider processes had exited before the cleanup of the test; - the provider process is lost during the wait, and the run settles in seconds; - the sidecar exits during the wait, and the run settles in seconds; - a turn with no background work does not wait; - an ACPX agent that is not Claude is never opted in, also when the caller sets the option. - Each case asserts the result of the close of the session: clean in 6 cases, and a named failure in the case where the sidecar is killed. Each clean case also asserts that there is no `native_session_settlement_incomplete` diagnostic and no "conflicting duplicate tool result" in the durable state. - `src/live/live-session.test.ts`, `src/live/runnerd-codex-transport.test.ts`, `src/live/runnerd-acpx-provider-turn-wait.test.ts`, `src/native-session-runtime.provider-turn.test.ts`: 434 passed. - Rust, because the fake sidecar changed: `cargo fmt --check`, `cargo build`, the `acpx_*` suites (244 passed) and the `acpx` unit tests (102 passed). - Mutation checks, run on the full change before the split: with the option ignored, 6 of the 7 assertions of the case with no option fail; with the old mode of the fake, the case with the result fails at the close of the session. Not run: the full suite. CI supplies that coverage. ## Risks - **No production effect.** No caller sets the option. Core sets it in part 3. - **The transport is shared by each provider.** The three new behaviors are behind one condition: the option, ACPX, and the agent `claude`. With the condition false, the code path is the path of today. - **The tool-call ordering uses a 10 ms poll** while earlier events are not delivered, only in a provider-started turn. - **A controller restart during a wait or during a provider-started turn is not handled in this part.** Part 2 handles it. Part 3 must not merge before part 2. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4a78124a3e |
feat(runner): wait for a provider-started Claude turn when the controller asks (#21423)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. > - Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. Today the Runner stops Claude at the end of the turn, so the command stops and the run fails with `native_finalization_missing`. > - A stack of pull requests lets the Runner accept that cycle as a new turn of the same run, when a controller asks for it. > - `paperclip-runnerd` ends a run when a turn ends. For a Claude turn that ended with no result while a background shell is running, this stops the shell and the run fails. > - #21278 binds each tool call to its turn, and #21293 reports how many background shells are running at the end of a turn. Part 4 declared the contract. > - This pull request makes `paperclip-runnerd` wait for a turn that Claude starts by itself, when the controller asks for it, and admit that turn into the same run. > - The benefit is that the Runner has one place that decides to wait and one place that ends a wait. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one part of a change that was first opened as one large pull request (#21316). The parts, in stack order after #21311: 1. #21419: sidecar preparation (a refactor). 2. #21420: converter for raw ACP session updates. 3. #21421: cycle stream at the adapter. 4. #21422: contract for provider-started turns. 5. #21423: `paperclip-runnerd` waits for a provider-started turn. 6. #21316: the sidecar adopts a cycle. This is part 5. Its base branch is `feat/prp-provider-turn-contract`. After each part, a production run does the same as today, because no controller sends the opt-in. Refs #19912 (design). **What existing behavior does this improve?** What `paperclip-runnerd` does after a Claude turn ends with no result while a background shell is running. Nothing changes until a controller sends the opt-in. **Subsystem affected** Paperclip Runner: the ACPX provider backend in `runner-core` (Rust). Claude only. **Current behavior** - After each completed turn the Runner emits `run.terminal`. A turn with a running background shell gets the provider restart, and the shell stops. **Proposed behavior** Nothing changes unless the controller sends `providerStartedTurns: true` in the payload of `turn.start`. No controller sends it yet. With that field: - The Runner keeps the provider and waits when all of these are true: the agent is Claude; the turn completed; the run has no accepted `paperclip_finish` or `paperclip_block` result; the terminal event reported `backgroundTasks` with `live` above zero; each condition of #21293 for a kept process is true; the sidecar accepted `session.provider_turn` with `armed: true`. - `turn.completed` then has `awaitingProviderTurn: true`, and no `run.terminal` follows. The Runner holds that `run.terminal`. - When the sidecar reports a turn that the provider started, the Runner admits it only during a wait and publishes `turn.started` with `origin: "provider"`. In each other case that report fails the session closed. - When a wait ends with no turn active, the Runner publishes the held `run.terminal`. This applies when the provider process exits, the sidecar exits, the controller cancels or stops the run, the session is closed or suspended, and after a Runner restart. - The held `run.terminal` and the removal of the wait are saved in one write, and the controller gets the event from the next poll. A crash after a cancel or a stop cannot lose it, and it is never sent two times. - A controller `turn.start` during a wait replaces the provider first, as before, and the held event is dropped. - A `run.attach` from a different run during a wait ends the wait when the attach succeeds. The Runner then discards the held `run.terminal`, because that event is of the old run and must not end the new run. This is the same rule that a Runner restart applies. The wait is removed only in the state that the attach saves for the new run. If the attach fails, for example when the old session does not stop, the old run keeps the wait and gets its held `run.terminal` one time, from the next poll or from a cancel or a stop. A `run.attach` from the same run publishes the held event to that run. - The Runner waits at most 5 seconds for the sidecar to answer a request to arm or to disarm a provider-started turn. This is the limit that turn cancellation already uses. Before, the limit was the 30 seconds of an ordinary request, and a cancel, a stop or a shutdown had to wait behind it. No answer is handled as a lost sidecar: the session fails closed, and the Runner still publishes the held `run.terminal` one time. - The sidecar of this branch does not know `session.provider_turn`, so the arming request fails and the Runner does not wait. The behavior comes with part 6. **Reason and benefit** - The wait is one entry in the saved session state. One rule ends it at each save, so each new way to leave the open session ends the wait with no new code. - A wait must not outlive the provider or the sidecar, and a cancel must stop it. ## What Changed - `acpx_provider_turn_wait.rs` (new, 230 lines with its unit tests): `provider_turn_wait { turn_id, held_run_terminal }` in the saved state; the rule that ends a wait at save; `settle_turn_events`, which decides at one place to arm, to create the wait and to put `awaitingProviderTurn` on `turn.completed`. - `acpx_provider_turn.rs` (new, 119 lines): the offer of a turn at its terminal, the arming request, and `admit_provider_turn`. - `acpx_provider_session.rs`, `acpx_provider_backend.rs`, `acpx_provider_state.rs`, `provider_events.rs`, `acpx_park_preparation.rs`: the opt-in from `turn.start`; the record that the run has a result; the provider-started turn in the state and in the PRP events; cancel and stop with no turn; no turn-end restart and no park during a wait. - `acpx_sidecar_transport.rs` (11 lines): during a wait, a poll with no event also checks that the sidecar process is alive. A child process can keep the output stream of a dead sidecar open. - `bin/fake-acpx-sidecar.rs` (232 lines): provider-started turns and failures for tests. - Tests (1,164 lines): `tests/acpx_quiet_turn_continuation.rs`, `tests/support/acpx_quiet_turn_restart.rs`, `src/live/runnerd-acpx-provider-turn-wait.test.ts` (the real daemon), `src/native-session-runtime.provider-turn.test.ts` (the controller of today). - `runner/DURABLE_TRANSPORT.md`: the rules. ## State saved by the previous release - One new optional field in the saved ACPX session state: `providerTurnWait`. It is written only during a wait. In each other state the saved file has the shape of the previous release. - **State from the previous release, read by this release:** the field is absent, so there is no wait. A unit test covers this. - **State from this release, read by the previous release (a rollback):** outside a wait, the file loads as before. During a wait, the previous Runner refuses the file with its "ACPX provider state is malformed" error, because its loader refuses each unknown field. Only a run that waits at the moment of a rollback is in this case, and no run waits before a controller sends the opt-in. - **Runner restart during a wait:** the wait is restored. The provider is a new process, so the wait is over, and the controller receives the held `run.terminal` one time. A test with the real daemon covers this. - **Controller of today or of the previous release:** `turn.start` has no opt-in. A test runs a turn with a running shell and a turn with an unknown background state, and compares the events, the commands after the turn, the restart, and the park state before and after the restart. They are the same. ## Verification At the head of this branch, from `packages/paperclip-runner`: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs test/protocol-contract.test.mjs`, the server check `node scripts/verify-runner-vendor-dependencies.mjs`, `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt --check` and `cargo build -p paperclip-runner-core --bins` (no new warning): all passed. - `cargo test -p paperclip-runner-core --lib`: 644 passed, 4 ignored. All 17 `acpx_*` suites passed. - `src/live/runnerd-acpx-provider-turn-wait.test.ts`, 5 cases with the real `paperclip-runnerd` and the fake sidecar through PRP: a wait ends with exactly one `run.terminal` when the sidecar exits while a child process keeps its output open, when the provider is lost, when the controller cancels, and after a Runner restart; with no opt-in, `run.terminal` follows the turn and the sidecar is never armed. - Rust tests: wait and admit; no wait with a result; no wait with no opt-in; an active goal neither marks nor arms a wait; a sidecar that refuses to arm; a provider-started turn that the Runner did not wait for fails the session closed; cancel and stop during a wait; provider and sidecar loss; restart; the saved shape in both directions. - `src/native-session-runtime.provider-turn.test.ts`: the real `consumeTurn` of today gets the event sequence with no opt-in and finalizes it as `native_finalization_missing`, as today. - Attach during a wait, in `tests/support/acpx_quiet_turn_restart.rs`: a different run attaches after the snapshot reports `warmAttachReady: true`; the new run gets no `run.terminal`, the saved state has no wait, and the old session is closed. A same-run attach gives exactly one `run.terminal`. An attach that fails when the old sidecar is dead leaves the wait with the old run, and the old run then gets exactly one `run.terminal` from a poll, `run.cancel`, `turn.interrupt` or `turn.stop`. Mutation checks: with no fix, the old `run.terminal` is in the pending events of the new run; with the wait ended before the attach is complete, the old run gets no `run.terminal` after a failed attach. - No answer from the sidecar, in `tests/support/acpx_quiet_turn_restart.rs`: the fake sidecar does not answer the disarm request; `run.cancel` and `turn.interrupt` each return `already_settled` in less than 10 seconds, exactly one `run.terminal` reaches the run, `session.close` completes in less than 2 seconds with no second `run.terminal`, and the provider process is gone. With no answer to the arm request, the turn ends with its `run.terminal` and with no wait. With the limit removed, each test fails at 30 seconds. - Mutation checks, each made the named tests fail: the rule that ends a wait at save; no `run.terminal` after a restart; arming with no opt-in; the marker set with no wait; the sidecar process check removed. Not run: a rollback with the binary of the previous release; a run with the opt-in and a sidecar that answers the arming request with an error (the code reads each error as not armed); Linux; the full suite. CI supplies the broad coverage. ## Risks - **The feature is off.** No controller sends the opt-in. - **A rollback of the Runner during a wait** makes the previous Runner refuse that saved session. See "State saved by the previous release". - **The wait has no time limit in the Runner.** The controller must close, suspend or cancel. Core will own that limit. - **Park is refused during a wait.** The controller must end the wait before it parks. - **A new run can attach during a wait.** The attach is not blocked, because the Runner has no time limit on the wait and a new run can send only `run.attach`. After an attach that succeeds, the old run gets no `run.terminal` from the Runner; the controller must not depend on that event. The request that disarms a sidecar that the attach keeps has no test, because no test can keep a session across an attach during a wait. - **After a sidecar that does not answer, `runner.suspend` returns an error at once** ("provider lifetime cleanup is not yet proven"), and `session.close` completes. This is the behavior of today for each lost sidecar; this PR does not change it. - **The shared code that changed** runs for each Claude turn: the record of a run result, and the settle step at each turn end. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
493519fc88 |
fix(runner): poll a live ACPX turn without waiting out the idle read (#21525)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runner (runnerd) relays provider events to the server and waits for cumulative ACKs before it polls the provider again. > - After a control frame, runnerd uses a 1 ms queued read only when the executor reports a live provider poll. Otherwise it waits up to the 250 ms runtime read timeout, and any frame that arrives in that time skips the poll. > - Only the Codex facade reports a live poll. ACPX runs (Claude, Codex through ACPX and others) always wait the full 250 ms after the last ACK of every batch, and steady control traffic starves their provider polls. > - This pull request lets an ACPX executor report a live poll while its restored session has an active turn. > - The benefit is that ACPX output is relayed without the 250 ms idle wait, and tool-heavy turns no longer starve provider polls. ## Linked Issues or Issue Description **What happened?** A routine run started several subagents that made about 190 tool calls in about 10 minutes. On the server, consecutive runner events arrived at least 576 ms apart (median about 1.5 seconds), one event at a time, while the time from each event's `emittedAt` to its insert was 0 to 0.1 seconds. The runner fell several minutes behind the provider. The backlog then overflowed the transport's event buffer (fixed separately in #21524) and the turn failed. **Expected behavior** A live ACPX turn is polled again as soon as its outstanding events are acknowledged and no control frame is queued, the same as a Codex facade turn. **Steps to reproduce** 1. Run an ACPX turn that produces events while the controller sends commands and ACKs. 2. After each ACK, runnerd calls `receive_json`, which waits up to 250 ms before it polls the provider. A frame that arrives in that time skips the poll. **Relevant logs or output** ``` min gap between consecutive runner events: 576 ms, median about 1.5 s, never batched insert time - emittedAt: 0.0 to 0.1 s ``` Related: #21524 bounds the receive buffer that this backlog overflowed. ## What Changed - `acpx_provider_backend.rs`: `AcpxCommandExecutor::provider_poll_is_live` returns true only when the session was restored, has no persistence error, does not wait for a next-turn generation, and its state is `turn_starting` or `turn_active` with an active turn, no unconfirmed provider exit, and the run it owns. - `native_provider_backend.rs`: `SelectedExecutor` forwards `provider_poll_is_live` to the ACPX executor. - `tests/acpx_warm_attachment.rs`: new test through `NativeProviderCommandExecutor`. The session is not live before `run.prepare` or after `session.open`, is live after `turn.start`, and a new executor that has not restored its session is not live. What does not change: - runnerd still polls only when the read returns no frame, so a queued stop or suspend still runs before the next provider poll. - The ACK gate (no poll while sent events are not acknowledged) does not change. - The 1 ms read is used only while the source sequence advanced since the last quiet poll, as for Codex today, so an idle turn does not busy-loop. ## Verification - `cargo test --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --test acpx_warm_attachment` (in `packages/paperclip-runner`): 7 passed. The new test fails without the new method, because the default returns false. - `cargo fmt --all -- --check` and `cargo check --locked --workspace` pass. - `cargo test --release --locked -p paperclip-runner-core --no-fail-fast`: 1,127 passed, 3 failed. The failures are `codex_provider::tests::fenced_receipt_output_reports_capacity_without_dropping_known_frames` ("failed to signal owned process group: Operation not permitted") and `stdio_mcp::lost_acknowledgement_blocks_calls_until_socket_replacement` and `metadata_only_socket_changes_do_not_bypass_unknown_outcome` (relay exit timing). They also fail without this change in the macOS sandbox where I ran the suite, and none of them uses the ACPX executor. CI covers them on Linux. - The existing `durable::tests::ack_release_poll` cases cover the runner side of the live read: an ACK releases the next poll without the read timeout, and a command queued behind the ACK runs first. - I did not run a live provider. After the deploy, the gap between runner events of an ACPX run with queued output should drop below 250 ms. ## Risks - Low risk. The change only selects the existing 1 ms queued read for ACPX during an active turn. - A turn that produces events continuously now polls more often. Each poll still writes the same durable state as before, so disk writes per event do not change. - Analysis also found that runnerd waits for every sent event to be acknowledged before it polls again. A bounded in-flight window would raise throughput further, but it changes the tested rule that control queued behind an ACK runs before the next poll. That is left out of this pull request on purpose. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
254744f77c |
fix(runner): bound the ACPX receive buffer by bytes, not 512 events (#21524)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runner talks to ACPX providers through a sidecar process.
The sidecar writes events and command replies as JSON lines on stdout.
> - While the transport waits for a command reply, it buffers every
event that arrives before the reply. It failed when 512 events were
buffered.
> - The sidecar allows up to 4 MiB of unflushed stdout. A backlog of
small events passes 512 events long before any memory concern, for
example during a long turn with several subagents while the runner
relays events slowly.
> - When the limit hits, the transport is poisoned and the turn fails
with `acpx_provider_transport_failed`, which is not recoverable. The
task is then blocked.
> - This pull request bounds the receive buffer by frame bytes,
consistent with the sidecar's own output bound, and keeps a count
backstop for floods of tiny frames.
> - The benefit is that a normal event backlog no longer kills the turn,
and the buffer stays bounded in memory.
## Linked Issues or Issue Description
**What happened?**
A routine run started several subagents, which made about 190 GitHub
tool calls in about 10 minutes. The runner relays provider events at
about one per controller round trip, so the sidecar output fell several
minutes behind. A semantic tool result was then delivered through
`tool.resolve`. While it waited for the reply, the transport read more
than 512 backlogged events, failed with `ACPX sidecar exceeded the
buffered event limit`, and poisoned the transport. The turn failed with
`acpx_provider_transport_failed`, and recovery blocked the task.
**Expected behavior**
A backlog that the sidecar itself accepted does not fail a request. Only
a backlog that is too large for memory fails.
**Steps to reproduce**
1. Let a sidecar write more than 512 small events before it writes the
reply to a pending command.
2. The command fails with `ACPX sidecar exceeded the buffered event
limit`, and the transport is poisoned.
**Relevant logs or output**
```
ACPX sidecar exceeded the buffered event limit
turn.failed code=acpx_provider_transport_failed recoverable=false
```
No related public PR exists. I searched open and closed PRs for this
limit. The slow event relay that builds the backlog is a separate
problem.
## What Changed
- `acpx_sidecar_transport.rs`:
- Each buffered event keeps the byte length of its frame. The transport
keeps a running byte total.
- The buffer fails above 16 MiB (`MAX_BUFFERED_EVENT_BYTES`, four times
the sidecar's 4 MiB unflushed output bound
`SIDECAR_MAX_UNFLUSHED_OUTPUT_BYTES`), or above 65,536 events (one event
per 256 bytes of the budget) as a backstop for tiny frames.
- `poll_event` and `defer_polled_event` keep the byte total correct. A
poisoned transport resets it.
- `fake-acpx-sidecar.rs`: new `backlog`, `large-backlog` and
`count-flood` modes. `flood` now writes 18 MiB of 64 KiB events instead
of 513 small events.
- `tests/acpx_sidecar_transport.rs`: new tests for a backlog of 2,048
small events, for freeing the byte budget as events drain, and for a
flood of tiny events. The existing flood test still checks the error.
## Verification
- `cargo test --release --manifest-path runner/Cargo.toml --locked -p
paperclip-runner-core --test acpx_sidecar_transport` (in
`packages/paperclip-runner`): 23 passed.
- `buffers_a_backlog_of_small_events_before_a_response` fails before
this change (2,048 events are above the old 512 limit).
`frees_the_byte_budget_as_buffered_events_drain` needs correct
accounting: two backlogs of 12 MiB pass only when drained events free
their bytes.
- `cargo fmt --all -- --check` and `cargo check --locked --workspace`
pass.
- `cargo test --release --locked -p paperclip-runner-core
--no-fail-fast`: 1,128 passed, 4 failed. The failures are
`codex_provider::tests::fenced_receipt_output_keeps_prefix_when_stdout_never_closes`
and
`fenced_receipt_output_reports_capacity_without_dropping_known_frames`
("failed to signal owned process group: Operation not permitted"), and
`stdio_mcp::lost_acknowledgement_blocks_calls_until_socket_replacement`
and `metadata_only_socket_changes_do_not_bypass_unknown_outcome` (relay
exit timing). None of them uses the ACPX sidecar transport. They come
from the macOS sandbox where I ran the suite. CI covers them on Linux.
- `cargo test --release -p paperclip-runner-core --lib acpx`: 99 passed.
`--test acpx_provider_resolutions --test acpx_sidecar_stopped_time`:
passed.
- I did not run a live provider. After the deploy, a long turn with many
tool events should not fail with `buffered event limit`.
## Risks
- Low risk. The change is local to the transport receive buffer.
- The memory bound is now 16 MiB of frames instead of 512 events. Parsed
events use more memory than their frames, so the worst case is a few
times 16 MiB in the runner process.
- The sidecar's own 4 MiB unflushed output bound does not change. A
burst that is longer than that still fails in the sidecar with
`output_buffer_limit_exceeded`. The slow relay that builds the backlog
is a separate fix.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
|
||
|
|
1332ab987f |
feat(prp): declare the contract for provider-started turns (#21422)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. > - Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. Today the Runner stops Claude at the end of the turn, so the command stops and the run fails with `native_finalization_missing`. > - A stack of pull requests lets the Runner accept that cycle as a new turn of the same run, when a controller asks for it. > - The Runner, its sidecar and the controller speak two protocols with schemas: PRP between the controller and the Runner, and the sidecar protocol between `paperclip-runnerd` and the ACPX sidecar. > - A turn that the provider starts needs new fields and one new command in those protocols. > - This pull request declares them and generates the readers again. It changes no behavior. > - The benefit is that the next two pull requests show behavior only, with the contract already in review. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one part of a change that was first opened as one large pull request (#21316). The parts, in stack order after #21311: 1. #21419: sidecar preparation (a refactor). 2. #21420: converter for raw ACP session updates. 3. #21421: cycle stream at the adapter. 4. #21422: contract for provider-started turns. 5. #21423: `paperclip-runnerd` waits for a provider-started turn. 6. #21316: the sidecar adopts a cycle. This is part 4. Its base branch is `feat/claude-provider-cycle-stream`. After each part, a production run does the same as today, because no controller sends the opt-in. Refs #19912 (design). **What existing behavior does this improve?** Nothing in production. The protocol schemas. **Subsystem affected** Paperclip Runner: `protocol/` and the generated TypeScript and Rust contract code. **Current behavior** - PRP has no way to say that a turn was started by the provider, or that the Runner waits for such a turn. - The sidecar protocol has no command to arm the sidecar and no event for a turn that the provider starts. **Proposed behavior** - PRP (`schemas/provider-turn.schema.json`, new): `providerStartedTurns` (boolean) in the payload of `turn.start`, `origin: "provider"` on `turn.started`, and `awaitingProviderTurn: true` on `turn.completed`. `command-v3` and `event-v3` refer to it. - Sidecar protocol (`provider-schemas/acpx-sidecar.schema.json`): the command `session.provider_turn` with `{ turnId, armed }`, the event `runtime.turn_started` with `{ origin: "provider" }`, and the payload of the diagnostic for a lost provider. The schema enforces that payload for a `runtime.diagnostic` with the code `ACPX_PROVIDER_PROCESS_LOST`. Diagnostics with other codes keep their open payload. - `paperclip-runnerd` decodes `runtime.turn_started` and refuses it. No sidecar sends it yet. The Runner of today refuses an unknown event too. - No controller, Runner or sidecar uses the new fields in this pull request. **Reason and benefit** The repository rule for Runner contracts says that new fields must be in the schema with generated readers. A separate pull request lets a reviewer read the contract alone. ## What Changed - `protocol/schemas/provider-turn.schema.json` (new), `command-v3.schema.json`, `event-v3.schema.json`, `protocol/provider-schemas/acpx-sidecar.schema.json`, `protocol/manifest.json`. - `scripts/generate-acpx-sidecar-contract.mjs` and `scripts/generate-protocol-schema-module.mjs`. - Generated: `src/drivers/acpx/generated-sidecar-contract.ts`, `src/protocol/generated/schema-bundle.ts`, `src/protocol/generated/standalone-validators.ts`, `runner/crates/runner-core/src/generated_acpx_sidecar_contract.rs`. - `runner/crates/runner-core/src/acpx_event_payload.rs` (3 lines): the Rust `match` on the event type must name the new event to compile. - `test/protocol-contract.test.mjs`. - The contract digests of the pack change (`prp.acpx-sidecar`, `prp.command-v3`, `prp.event-v3`, `rust`, and the new `prp.provider-turn`). No stored file refers to these digests. No provider `commandDigest` and no profile version changes. ## Verification At the head of this branch, from `packages/paperclip-runner`: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json`, `pnpm run typecheck:typescript`, `pnpm run build:typescript`, `node --test test/acpx-sidecar-contract.test.mjs test/protocol-contract.test.mjs`, the server check `node scripts/verify-runner-vendor-dependencies.mjs`, `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`, `cargo fmt --check` and `cargo build -p paperclip-runner-core --bins` (no new warning): all passed. The contract tests: 32 passed. - The sidecar suite and the tracker tests: 141 passed. Not run: the full suite. CI supplies that coverage. ## Risks - **The pack contract digest changes.** Older image packs are not used again. - **The opt-in field is accepted by the schema but has no effect yet.** A controller that sends it before part 5 gets the behavior of today. - **State saved by the previous release:** this pull request changes no persisted shape. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0aaab644fe |
refactor(runner): remove the direct assigned-gateway path (#21311)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An agent reaches the tools of its app connections through an MCP gateway. > - Before #21274, some native harnesses got the gateway URL and a bearer token and connected to the gateway directly. The Runner has code for that: an HTTP MCP server for ACPX providers, a server entry in the Codex `config.toml`, a server entry in the OpenCode config, and the lists that pass the three `PAPERCLIP_NATIVE_MCP_*` variables to a sidecar. > - #21274 gives each native harness its gateway tools through the PRP relay. Core removes the three variables from each Runner environment. The direct code path gets no binding. > - Code that cannot run still needs maintenance, and it can still put a token in a provider process if a variable arrives by another way. > - This pull request deletes the direct path from the Runner. > - The benefit is one path for assigned gateway tools, and no Runner code that can give a gateway token to a provider. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is in a stack. Its base branch is `fix/keep-quiet-claude-provider-between-turns` (#21293). It depends on #21274, which is the bottom of the stack. On `master` without #21274, the direct path is still in use. **What existing behavior does this improve?** The Runner code that connects a provider directly to the assigned MCP gateway. **Subsystem affected** Paperclip Runner: the TypeScript drivers and the Rust `runner-core` environment lists. **Current behavior** After #21274, Core never gives `PAPERCLIP_NATIVE_MCP_NAME`, `PAPERCLIP_NATIVE_MCP_URL` or `PAPERCLIP_NATIVE_MCP_TOKEN` to a Runner. The Runner still has the code that reads them and configures a direct gateway server for ACPX providers, Codex and OpenCode. **Proposed behavior** The Runner has no code that reads the three variables to configure a provider. If the variables are in the environment of a Runner, no provider environment, sidecar, Codex `config.toml` or OpenCode config gets them. **Reason and benefit** Less code, one path, and no way for a gateway token to reach a provider process through the Runner. ## What Changed - Deleted `packages/paperclip-runner/src/drivers/native-mcp.ts` (`nativeMcpLaunchBinding`). - `drivers/acpx/runtime-host.ts`: removed the native gateway server entry and the name conflict check. The admission gate keeps its message. A run with a binding id is admitted when the sidecar uses the PRP transport with its tool bridge, as the production sidecar always does. - `drivers/acpx/environment.ts`: the name and the URL are not in the provider allowlist. - `drivers/runtime-context-materializer.ts` and `live/runnerd-codex-transport.ts`: the Codex home functions have no `nativeMcp` parameter. The sidecar token pass-through is gone. - `drivers/opencode/opencode-server-driver.ts`: removed the read of the binding, its trace redaction value, and the gateway server entry in the OpenCode config. - `drivers/stdio-mcp.ts` and `runner-core/src/stdio_mcp.rs`: removed the reservation of the gateway server name. No gateway server exists, so no name can conflict. The names `paperclip` and `paperclip_runner_*` stay reserved. - `runner-core/src/acpx_sidecar_transport.rs` and `codex_provider.rs`: the three variables are not in the pass-through lists. - `control-plane/durable-prp-control-plane.ts`: the three variables are not in the platform environment list. - `packages/paperclip-runner/README.md`: the text about direct callers and assigned servers. - Kept: the three names stay in the reserved lists (`src/configured-environment.ts`, `runner-core/src/configured_environment.rs`). A user cannot configure them. Tests in both languages now check all three names. - No server code changed. The relay needs the gateway token and URL on the control plane. 23 files, 73 lines added and 211 lines removed. ## State saved by the previous release - This pull request changes no persisted shape. The provider server lists, the launch profile digests, the protocol versions and the sidecar contract are the same as with #21274. - A Runner or sidecar that the previous release started keeps its own code until its run ends, as #21274 describes. - A kept Codex home that has the old gateway server in `config.toml` is written again at the next start and at a recovery of the same run. A test covers this. - If the old variables reach a new Runner, no provider gets them. Tests cover this for ACPX, Codex and OpenCode. ## Verification From `packages/paperclip-runner`: - `npx vitest run src/drivers/acpx/runtime-host.test.ts src/live/runnerd-codex-transport.test.ts src/drivers/acpx/environment.test.ts src/drivers/opencode/opencode-server-driver.test.ts src/live/runnerd-codex-transport-warm-attachment.test.ts src/drivers/runtime-context-materializer.test.ts src/drivers/agent-observability.test.ts src/drivers/stdio-mcp.test.ts src/control-plane/durable-prp-control-plane.test.ts src/configured-environment.test.ts`: `10 passed (10)` files, `807 passed` tests. - `npx tsc --noEmit -p tsconfig.json` and `-p tsconfig.test.json`: no errors. - `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`: no changes. - `cargo test -p paperclip-runner-core --test acpx_sidecar_transport`: 20 passed. `cargo test -p paperclip-runner-core --lib`: 639 passed, 4 ignored. Tests that failed with the old code and pass now: - `runtime-host.test.ts`: "admits a relayed … run with a binding id and ignores a previous-release gateway binding", for Claude, Codex and Cursor. - `runnerd-codex-transport.test.ts`: "drops a previous-release gateway binding from the … runner environment", for ACPX Claude, Codex, Pi, Cursor, Copilot and Kimi, and for OpenCode. - `opencode-server-driver.test.ts`: the OpenCode config has only the task bridge. - `runnerd-codex-transport-warm-attachment.test.ts`: a restored Codex home has no gateway server and no gateway credential in `config.toml`. - `environment.test.ts`: a provider environment has none of the three variables. - Rust `previous_release_gateway_binding_and_unrelated_secrets_never_reach_the_sidecar`: a Runner that starts with the old variables gives none of them to the sidecar, for five agents. Mutation check: with the three names put back in the Rust pass-through list, the Rust test fails. Not run: a live run with a real provider. Not run: `pnpm verify` and the full suite. CI supplies that coverage. ## Risks - **This pull request is safe only after #21274.** The stack order keeps that. - **A developer who sets the three variables by hand for `eval-session`** no longer gets direct gateway tools. No document describes that use. - **The warm attachment test uses the staged Runner binary from preflight.** That binary does not have the Rust changes of this pull request. The Codex path of that test does not use the changed lists. - The evidence that no entry point supplies a binding comes from a read of the code: Core removes the variables before it builds each Runner environment, and a native run cannot use the in-process path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change, reviewed the diff, and ran the verification and a mutation check again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), traced each code path and wrote the change and the tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
0ebe6d846d |
perf(runner): keep a quiet Claude provider between turns of a run (#21293)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner runs Claude Code through the ACPX sidecar. The Runner stops and starts the sidecar and Claude at each turn boundary. > - The reason was that a late tool call from an earlier turn could run in a later turn. #21278 removes that reason for Claude: the sidecar refuses a call that the active turn did not announce. > - A second reason remains. Claude Code works by itself between prompts. When a background task finishes, Claude starts a cycle of its own. If the process stayed alive with a live background task, that cycle could start between turns or mix with the next turn. > - So the Runner can keep the process only when it knows that Claude has no background work. > - This pull request adds that knowledge and uses it. After a Claude turn that completed with no background work, the Runner keeps the process for the next turn of the same run. > - The benefit is that a later turn of the same run starts with no provider restart. This is also the base for the next step, which lets a run continue when background work finishes. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is stacked on #21278. Its base branch is `fix/bind-claude-tool-calls-to-turn`. Refs #19912 (design), #12419 (restart at turn start), #20145 (restart at park) and #20190 (restart at turn end). **What existing behavior does this improve?** The provider restart between turns of the same run, for Claude through ACPX. **Subsystem affected** Paperclip Runner: the ACPX sidecar (TypeScript) and the ACPX provider session in `runner-core` (Rust). Claude only. **Current behavior** - When a completed turn settles, the Runner restarts the sidecar and Claude (`restart_session_after_turn`, #20190). - If that restart did not run, the next turn restarts them at its start (`start_turn`). - The Runner does not know if Claude has background tasks. **Proposed behavior** - The sidecar asks `claude-agent-acp` for its async task reports. This is an optional capability of the pinned `claude-agent-acp` 0.73.0. The patch does not change. - A tracker reads those reports and the id of each background Bash command. It reports `{ live: 0 }` only when it knows that this process never had a background task and that Claude did not act between prompts. In all other cases the state is unknown. - The sidecar adds `backgroundTasks` to the terminal event of a completed Claude turn, and to `session.snapshot`. - The Runner keeps the process between two turns only when all of these are true: - the agent is Claude; - the turn completed (not failed, cancelled or interrupted); - the terminal event reported `backgroundTasks: { "live": 0 }`; - each existing condition of the turn-end restart is true; - before the next turn, a new `session.snapshot` reports the same identity and run, no active turn, no pending work, and `{ "live": 0 }`; - the settled-turn ledger and the tool replay ledgers are not full. - In each other case the Runner restarts as before. - A kept process keeps its tool call-id tombstones and its settled turn identities. **Reason and benefit** - A turn that follows a quiet turn in the same run starts without a restart. - The decision is safe by default: unknown state means a restart. ## What Changed TypeScript (`packages/paperclip-runner/src`): - `drivers/acpx/claude-background-tasks.ts` (new): `ClaudeBackgroundTaskTracker`. It reads raw ACP messages. It reports unknown when the capability is not negotiated, when a task finished, when a live task is not a shell command, when an update is malformed, when a limit is reached, when Claude acts with no client request in flight, or when Claude calls `CronCreate` or `ScheduleWakeup`. - `drivers/acpx/provider-failure.ts` and `acpx-runtime-adapter.ts`: Claude sessions also advertise the `asyncTasks` capability. The raw ACP observer gives each message to the tracker. ACPX 0.22 drops the `async_task_*` updates before the turn event stream, so only the raw observer sees them. - `drivers/acpx/runtime-host.ts`: `backgroundTaskState()`. - `cli/acpx-runtime-sidecar.ts`: the terminal event of a completed Claude turn carries `backgroundTasks` in the same payload as `goalAtTerminal`, with the same size limit. `session.snapshot` carries it when no turn is active. - `drivers/acpx/claude-tool-use-admission.ts`: `hasRoomForAnotherTurn()`. When the admitted-id ledger cannot hold one more full turn, the sidecar reports unknown, so the Runner restarts before the ledger is full. Rust (`packages/paperclip-runner/runner/crates/runner-core`): - `acpx_provider_session.rs`: the session records a quiet turn from the terminal event. `start_turn` continues in the same process only under the conditions above, and then uses `prepare_turn`, which keeps the tombstones. - `acpx_provider_generation.rs`: `can_continue_after_turn`, `continue_after_quiet_turn`, and the snapshot check `background_work_is_quiet`. - `acpx_park_preparation.rs`: the turn-end restart does not run for a quiet Claude turn. - `acpx_provider_backend.rs`: `session.reconciled` reports a replaced process only when the turn start replaced it. - `bin/fake-acpx-sidecar.rs`: new option `--background-tasks` for tests. - `runner/DURABLE_TRANSPORT.md`: the rule for a kept sidecar. Contract (`packages/paperclip-runner/protocol`): - The `backgroundTasks` report is `$defs.backgroundTasks` in `provider-schemas/acpx-sidecar.schema.json`: an object with only `live`, a non-negative integer. The schema validates it on `runtime.turn_terminal`. - `scripts/generate-acpx-sidecar-contract.mjs` generates the TypeScript type and reader and the Rust reader from it. The tracker, the sidecar and the Rust session use the generated code. - `fixtures/provider-boundary/acpx-background-tasks.json` has shared test documents. The TypeScript reader, the Rust reader and the schema are tested with the same accepted and refused shapes. - This changes the contract digest of the provider pack (`prp.acpx-sidecar` and `rust`). Older image packs are not reused. No provider `commandDigest` and no profile version changes. Not changed: park and run attachment. A process that ran a turn is still restarted at park, and only a process that ran no turn is bound to the next run. Other ACPX agents and Codex do not change. The `claude-agent-acp` patch does not change. ## State saved by the previous release - This pull request adds no persisted state. The three new fields of the session are in memory only. - A terminal event or a snapshot without `backgroundTasks` means unknown. An older sidecar therefore gets the restart, as before. - An older Runner with a new pack ignores the field and restarts at each turn, as before. - After a Runner restart, the in-memory state is gone. The next turn restarts the provider, as before. A test covers this. ## Verification From `packages/paperclip-runner`: - `npx vitest run src/drivers/acpx/claude-background-tasks.test.ts src/drivers/acpx/acpx-runtime-adapter.claude-background-tasks.test.ts src/drivers/acpx/acpx-runtime-adapter.claude-tool-use.test.ts src/drivers/acpx/claude-tool-use-admission.test.ts src/cli/acpx-runtime-sidecar.test.ts src/drivers/acpx/acpx-runtime-adapter.test.ts src/drivers/acpx/runtime-host.test.ts`: `7 passed (7)` files, `363 passed` tests. - `npx tsc --noEmit -p tsconfig.json` and `npx tsc --noEmit -p tsconfig.test.json`: no errors. - `pnpm run typecheck:typescript` (it includes the contract checks): no errors. `node --test test/acpx-sidecar-contract.test.mjs`: 17 passed. - `cargo fmt --manifest-path runner/Cargo.toml --all -- --check`: no changes. - `cargo test -p paperclip-runner-core --test acpx_quiet_turn_continuation --test acpx_park_preparation --test acpx_park_readiness --test acpx_provider_turns --test acpx_provider_session --test acpx_warm_attachment --test native_provider_backend`: 4, 33, 8, 31, 12, 6 and 47 passed. No test failed. - `cargo test -p paperclip-runner-core --lib`: 637 passed, 4 ignored. - Mutation check: when the Rust code accepts each live count as quiet, two tests in `acpx_park_preparation` fail. New tests: - **Real ACP runtime** (`acpx-runtime-adapter.claude-background-tasks.test.ts`), with the pinned `claude-agent-acp`, the pinned Claude Code client and a local fake Anthropic API. A foreground Bash turn reports `{ live: 0 }`. A background Bash in the same process reports `{ live: 1 }` at the terminal event, and unknown after Claude ran its own cycle for the finished task. Each turn has its own usage receipt. One process serves all turns. - **Tracker unit tests** (13): spawn, completion, failure, stop, pause, a task that is not a shell, a subagent, no negotiation, malformed updates, activity between prompts, scheduling tools, session load replay, limits. - **Sidecar tests**: the terminal event carries the state for Claude and not for Codex. A full ledger reports unknown. - **Rust** `tests/acpx_quiet_turn_continuation.rs` (4) and `tests/support/acpx_quiet_turn_restart.rs` (7): - a quiet Claude turn continues in the same process, and a replay of a settled call id is refused; - live, unknown and malformed reports restart; - background work that appears after the terminal event restarts at turn start; - failed, cancelled and interrupted turns restart; - Codex with a quiet report restarts; - an active goal restarts; - 4,096 kept turns, then the next turn restarts; - park after a kept process still prepares a new process, and the next run attaches to it; - a Runner restart after a kept process resumes in a new process. - The existing restart tests did not change. The fake sidecar sends no report by default, so they are the tests for the unknown case. Live run with a real Claude account, not in CI: the real ACP runtime (ACPX 0.22, the patched `claude-agent-acp`, the pinned Claude Code client), a real tool bridge, and the real Anthropic API with model `claude-sonnet-5-5`, on the code at the top of this stack. A turn that started one background shell reported `{ live: 1 }`. A turn that started two reported `{ live: 2 }`, then `{ live: 1 }` and `{ live: 0 }` as each shell finished. A turn in which Claude waited for its background shell in the foreground reported the unknown state. Not run: server integration tests (the fake sidecar there sends no report, so their process counts do not change), Linux, and a live run through the real sidecar process and runnerd. ## Risks - **All Claude sessions now negotiate the `asyncTasks` capability.** `claude-agent-acp` then sends `async_task_*` updates and adds a marker to a background Bash update. ACPX drops the updates. No other behavior of `claude-agent-acp` changes, from a read of its code. - **Park after a kept process pays the restart again**, as it did before #20190. With real Claude that restart was about 1.5 s. - **A small window remains** between the snapshot and the prompt of the next turn. Claude can start a cycle of its own in that window only without a background task, because a process that had a background task is not kept. - **Activity between prompts is found only when no client request is in flight.** A request that never gets a response would hide such activity. - **Background watchers cannot be told apart from other tasks**, so each task that is not a shell command makes the state unknown. - **Each kept turn adds one `session.snapshot` request.** - A multi-turn run with real Claude now uses one provider process. This path has tests with the fake sidecar and with the real ACP runtime. It has no test with the real sidecar host. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change, reviewed the diff, and ran the TypeScript and Rust verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), investigated the signals and wrote the change and the tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
7773b52f05 |
fix(codex): preserve sandbox certificate trust in Codex shell commands (#20364)
<!-- Written in ASD-STE100 style: short sentences, active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Paperclip Runner and the Codex adapters run agent shell commands inside a sandboxed Codex session with a strict environment allowlist > - The controller builds `shell_environment_policy.include_only` before the sandbox injects certificate-trust variables such as `SSL_CERT_FILE` and `GIT_SSL_CAINFO` > - The allowlist drops the injected variables, so managed `git` and `gh` commands cannot verify HTTPS against a runtime-provided CA > - This pull request allows the certificate-trust variable names through every environment policy stage, from the Rust process supervisor to the shell allowlist > - The benefit is that managed `git` and `gh` work again in sandboxes that require a runtime CA, while provider-credential isolation and the bounded allowlist stay in place ## Linked Issues or Issue Description - Fixes: #20019 - Refs #20038 — community pull request for the same defect. This change reviews the approach independently. The commit credits the community author as co-author. ## What Changed - Add `CERTIFICATE_ENVIRONMENT_KEYS` (`SSL_CERT_FILE`, `SSL_CERT_DIR`, `GIT_SSL_CAINFO`, `NODE_EXTRA_CA_CERTS`, `DENO_CERT`, `REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`) to `packages/paperclip-runner/src/configured-environment.ts` - Native Codex shell policy (`codex-security-config.ts`): always put the certificate names in `include_only`; set inheritance to `"all"` with `ignore_default_excludes=true`, bounded by the explicit allowlist - Pass the certificate names through the app-server environment filter, the ACPX spawn environment filter, the ACPX runtime sandbox `include_only`, the runner platform environment keys, and the Rust process supervisor base key list - Legacy and ACPX-engine Codex shell allowlists (`packages/adapter-utils/src/codex-shell-environment.ts`): always include the certificate names - Add a synthetic private-CA HTTPS journey test: a real `git ls-remote` fails TLS verification without the injected CA and succeeds with it, through the policy-filtered child environment - Document the contract in `doc/sandbox-shell-environment.md` and link it from `doc/DEVELOPING.md` ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/drivers/codex/codex-security-config.test.ts src/configured-environment.test.ts src/drivers/codex/codex-shell-certificate-trust.test.ts src/drivers/codex/codex-app-server-driver.context-control.test.ts src/drivers/acpx/runtime-sandbox.test.ts` — 104 tests pass - `pnpm --filter @paperclipai/adapter-utils exec vitest run src/codex-shell-environment.test.ts src/acpx-engine/execute.test.ts` — 270 tests pass - `pnpm --filter @paperclipai/adapter-codex-local exec vitest run src/server/execute.test.ts` — 10 tests pass - Codex driver lifecycle, conformance, recovery, and boundaries suites pass (103 tests); environment-builder suites pass (307 tests) - Red proof: with only the source fix reverted, the new regressions fail. The journey test shows `GIT_SSL_CAINFO` is undefined in the shell environment, and `git` fails TLS verification against the private-CA fixture - `cargo check` passes for `paperclip-runner-core`; `rustfmt --check` passes for the edited Rust file - The journey test needs `git` and `openssl`; it skips when either tool is absent ## Risks - The native shell policy now always sets `shell_environment_policy.inherit="all"` with `ignore_default_excludes=true`. Before, it set `"none"` when no credentials or task keys were projected. The explicit `include_only` allowlist still bounds every inherited name, so no new secret names can cross - The allowlist grows by seven non-secret certificate-trust names. Their values point at CA files. The sandbox filesystem policy still controls access to those files; this change grants no filesystem access - No persisted run, session, or launch identity changes shape. Sessions saved by the previous release resume unchanged; the new policy applies when the next launch builds its arguments - Provider credentials keep their existing exclusions at every stage ## Model Used - Claude Fable 5 (Anthropic, model ID `claude-fable-5`), extended thinking, agentic tool use, run through Claude Code. The implementation was written and reviewed independently against the community design in #20038; the community author is credited as co-author in the commit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (pending: checks run after this PR opens) - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (pending review) - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: everlastingcrown <everlastingcrown@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
43aabb7080 |
fix(runner): keep configured secret values out of OpenCode output, storage and model requests (#21330)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person can configure a secret that Paperclip delivers to an agent's commands as an environment variable > - The value must stay out of what the agent runtime stores and what it sends to the model > - With the OpenCode runtime, a command that printed the variable put the value in OpenCode's tool output. OpenCode stored that output in its session database and sent it to the model. Paperclip's own run log already redacted it > - Paperclip did not tell the runner which configured values are secrets, so the runner could not protect them without also hiding ordinary settings > - This pull request sends secret names (never values) to the runner, filters exact secret values before OpenCode captures any tool output, redacts them from model requests, and tells the agent how to check a secret without printing it > - The benefit is that a configured secret stays out of model context and out of runtime storage, even when an agent prints it ## Linked Issues or Issue Description **What happened?** The `chat-features.runner-opencode.local.secrets.personal-secret` eval failed because the persisted-state scan found a configured secret value in OpenCode's session database. The person entered the value through the normal secret entry form. Nobody pasted it in chat. The agent ran a shell command that printed the environment variable it was given. Paperclip's run log redacted the value. OpenCode stored the raw output in its database and included it in the next model request. **Expected behavior** A configured secret must never reach model context or runtime persistence. Ordinary configured values must not be redacted. **Steps to reproduce** Configure a secret environment variable for an OpenCode agent. Have the agent run a command that prints the variable. Then search the OpenCode session data and the model requests for the value. ## What Changed - **Secret names cross to the runner.** `configuredEnvironmentProjection` takes the controller's secret key names and adds `PAPERCLIP_SECRET_ENV_KEYS`. This marker holds only the names of configured entries that came from secrets. It travels with the configured projection across the TypeScript and Rust runner boundaries (`configured-environment.ts`, `configured_environment.rs`). The server passes `secretKeys` in `runtime-execution.ts`. - **Command output is filtered before OpenCode reads it.** For a launch that has configured secrets, the runner sets OpenCode's `shell` to a small POSIX wrapper (`drivers/opencode/output-custody.ts`). The wrapper runs the shell OpenCode would use anyway. It sends stdout and stderr through an `awk` filter that replaces exact secret values. OpenCode never receives the raw bytes, so its streamed previews, its database and its model requests stay clean. The wrapper keeps the exit status. It does not wait for background processes. The values come from the environment that OpenCode already holds and are never written to disk. - **Other tool results are rewritten before they are saved.** A local OpenCode plugin rewrites completed non-shell results (file reads, searches, MCP results) in `tool.execute.after`. OpenCode runs that hook before it stores or returns the result. The wrapper refuses to run any command until the plugin has started for the current launch (per-launch nonce). So no command can write a secret to a file that a later read returns without filtering. - **Model requests are redacted as a second layer.** The OpenCode provider proxy removes exact secret values from the string values of each parsed model request. Field names and other literals are never changed. A quick byte scan for each value, raw and JSON-escaped, sends the original bytes unchanged when no value is present. - **The agent gets guidance.** Native task prompts list the configured secret names. They say to check a secret with `[ -n "$VAR" ]` and never to print, echo, log or write a secret value. - **No added startup delay.** The wrapper is content-addressed under the runtime directory and reused across launches, so it is not rewritten on every launch. OpenCode's plugin-dependency install is marked as already done, so loading the plugin does not add wait time to the first tool call. - **Only exact values are redacted.** Command output is filtered line by line, so each line of a multi-line value that has at least 4 characters is redacted too. If a configured secret has no line of 4 or more characters, the launch is refused with `opencode_output_custody_unavailable` instead of running unprotected. ## Verification - `packages/paperclip-runner/src/drivers/opencode/output-custody.test.ts` (10 tests): the wrapper filters stdout, stderr and a partial last line; keeps exit codes (7, and 143 for SIGTERM); filters each line of a multi-line value; refuses to run before plugin readiness or with an old nonce; returns at once when a background process keeps running; leaves ordinary values unchanged. The plugin rewrites nested results. Model-request redaction covers raw and JSON-escaped values and never changes field names or literals (a secret equal to `true` leaves `"stream": true` intact). Secrets the filter cannot match are refused. No value is written to disk. - `opencode-server-driver.test.ts`: one saved session launches three times. First with no secret marker (older controller), then with the marker, then with no marker again (rollback). The first launch is unchanged, the second applies custody, and the third launch still starts. This test fails without the driver change. - `configured-environment.test.ts`: the marker names only secret entries, crosses the runner spawn and OpenCode boundaries, reads as "no secrets" when absent, and is refused when malformed. Rust: `secret_name_marker_crosses_only_with_its_projection`. - `server/src/__tests__/configured-secret-guidance.test.ts`: guidance content, and no guidance when there are no secrets. - Real OpenCode 1.18.34 qualification (`PAPERCLIP_OPENCODE_QUALIFY=1 node --test scripts/qualify-opencode-runtime.test.mjs`). A fake model asks for a command that prints the secret and writes it to a file, then a read of that file. Unprotected baseline: the value is in the session database and in model requests (red). With custody: it is in neither, and the exit status is kept (green). - **Latency** (local, macOS, real OpenCode, 8 runs each): boot 629 ms vs 697 ms baseline; first model request after the prompt 678 ms vs 698 ms (no regression); first command 139 ms vs 191 ms; each later command +9 ms (36 vs 27 ms). Proxy cost per request when no secret is present (the normal case): 33 µs at 50 KiB, 164 µs at 300 KiB, 0.55 ms at 1 MiB, 2.3 ms at 4 MiB. When a secret is present: 91 µs, 0.48 ms, 1.6 ms and 6.8 ms. The scan runs only for launches that have configured secrets, and it adds nothing to OpenCode's own startup. - A narrow paid run of `chat-features.runner-opencode.local.secrets.personal-secret` on this branch is linked in a comment. ## Risks - **Previous-release story.** The marker is new and optional. A runner that receives no marker (older controller) launches exactly as before. An older runner drops the marker at its environment boundary. The change saves no new run or session state. OpenCode configuration is regenerated on every launch, so sessions saved before this change resume under custody (covered by the cutover test). Warm-process reuse keys include the marker, so a warm process from before the deploy is not reused for a run that has secrets. - **Commands go through the wrapper.** Output arrives line by line. Output written after the shell exits is dropped, which matches how OpenCode stops reading at exit. If `awk`, `mkfifo` or `rm` is missing, the launch is refused with `opencode_output_custody_unavailable` and does not run unprotected. - **Scope.** Only exact values are redacted. An agent that encodes a value (for example base64) is out of scope. The model-request layer applies when Paperclip routes the model through its proxy (OpenRouter and Paperclip connections). ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d9a4bb23fe |
fix(runner): wait for a slow Codex turn start while it reports progress (#21329)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Codex agents run through the Paperclip runner, and the server-side transport starts each provider turn > - After Codex accepts a turn, the transport waits a fixed 30 s for that turn's `turn/started` before it reports the turn identity > - Codex starts its MCP servers before it starts the turn, so a healthy cold start in a loaded sandbox often takes longer than 30 s > - The transport then failed a live turn with `runnerd did not report the provider turn identity`, and the abandoned turn's late notifications broke the reattachment (`did not name its execution owner`, `runner_harness_state_mismatch: resumed_not_suspended`) > - This pull request sizes the start window to the measured cold start and extends it while the provider keeps reporting startup progress > - The benefit is that slow but healthy Codex starts succeed on the first attempt, and a dead start still fails at a bound ## Linked Issues or Issue Description **What happened?** On staging, Codex runs on Daytona failed at turn start with `native_session_interrupted: runnerd did not report the provider turn identity`. The run log shows the same sequence each time. The runner accepts the turn. Codex then reports startup progress (a bubblewrap config warning, `account/updated`, and `mcpServer/startupStatus/updated` failures for an MCP server that is not logged in) for about 30 s. The transport gives up 30 s after acceptance. A few seconds later, Codex sends the turn's `turn/started`, which now arrives at a closed turn (`protocol_failure:turn_binding_mismatch`). The same-run reattachment then hits that orphaned turn. It fails with `Provider authoritative notification item/started did not name its execution owner.` or `runner_harness_state_mismatch: resumed_not_suspended`. Of 198 recent runs in the staging company, 52 contain this turn identity failure. In 68 successful starts, submit to `turn/started` took p50 25 s, p75 37 s, p90 50 s, p95 54 s and max 76 s. The fixed 30 s wait sits near the median. **Expected behavior** A slow but healthy Codex start succeeds. A start that makes no progress, whose Runner exits, or that the runner fails still fails at a bound and recovers through the normal same-run resume. **Steps to reproduce** 1. Start a Codex turn whose provider accepts the turn but sends `turn/started` after the 30 s default window (on staging: several MCP servers and a loaded sandbox). 2. Let the provider report startup notices during the wait, and let the runner not prove the accepted turn live in time (on staging no turn was adopted). 3. See `runnerd did not report the provider turn identity`, then the reattachment conflict. The new tests reproduce this with the fake Codex app server (`--turn-started-delay-ms` with `--turn-started-progress-ms`) and a runner snapshot that cannot prove liveness. **Paperclip version or commit** Staging Core `c730872974` (`2026.1106.0+2253`). Base of this PR: `origin/master`. **Deployment mode** Authenticated, public, managed cloud (staging). Remote runner on Daytona. **Agent adapter(s) involved** `paperclip_runner` with the Codex provider (`gpt-6.1-sol`). ## What Changed - `runnerd-codex-transport.ts`: the default wait for an accepted turn's `turn/started` is now 90 s (measured max 76 s), not 30 s. An explicit `turnStartTimeoutMs` keeps its old bound. - The wait restarts its window each time the runner commits a provider notice, or a runner diagnostic that names the accepted turn. The total wait stops at 180 s (option `turnStartProgressCeilingMs`, so tests can lower it). A provider relaunch after the accepted `turn.start` does not extend it. - The adoption check (runner snapshot proves the accepted turn live) now runs before each window end, not only before the first one. A Runner exit and a runner failure of the pending turn still fail at once. - `fake-codex-app-server`: new `--turn-started-progress-ms` flag sends `mcpServer/startupStatus/updated` failures while it delays `turn/started`. - `doc/execution-semantics.md`: documents the start window, the progress rule and the staging measurements. ## Verification - Red on `origin/master` transport (with the new fake flag): `keeps waiting for an accepted Codex turn while its startup keeps reporting progress` failed after 4.6 s, and `gives a silent Codex cold start the measured default start window` failed after 30.9 s. Both failed with `runnerd did not report the provider turn identity`. - Green: `npx vitest run src/live/runnerd-codex-transport-deadlines.test.ts` passes 9/9. This includes a ceiling test: provider notices every 1 s and no `turn/started` fail at an 8 s test ceiling, and the test goes red with the ceiling cap removed. It also includes the existing adoption, no-adoption-within-bound and runner-failure tests, and a new test that a silent start still fails within its bound. - `npx vitest run` on the adoption, cut-turn-start, stall, turn-start-timeout-retention, remote-retention, restart-continuation and controller-outage transport suites passes 48/48. - `npx vitest run src/live/runnerd-codex-transport.test.ts src/live/live-session.test.ts src/live/turn-stream.test.ts`: 391/392 pass. The one failure (`stops a lost runnerd's provider and escaped tool job from its lineage journal (journal=true)`, 1 s, process-lineage timing under full-suite load) passes when run alone, both on this branch and on `origin/master`. - `tsc --noEmit` for `packages/paperclip-runner` and `cargo fmt --check` pass. ## Risks - A silent dead start (no Runner exit, no provider output, no runner failure) now fails after 90 s, not 30 s. The Runner exit and runner-failure paths still fail at once, so this applies only to a provider that stops without a trace. - A provider that keeps sending notices but never starts the turn waits at most 180 s. - No schema, API or UI change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, browser reads of staging run data). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Refs #21272, #21120 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
314b893678 |
fix(native-runtime): deliver assigned gateway tools to every native harness through the PRP relay (#21274)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent can have app connections. Paperclip gives the agent the
tools of those apps through an MCP gateway, with its approval and audit
rules.
> - Native harnesses reach the gateway in two ways today. In the relay,
the server keeps the gateway token and offers the tools over the
authenticated Runner channel. In the direct way, the server gives the
gateway URL and a bearer token to the harness.
> - Which way a harness uses comes from a list of special cases that
grew one harness at a time. Local Codex and all ACPX agents use the
direct way. The others use the relay.
> - The direct way puts a run-scoped token in a provider process. A
process that the agent starts can read it. A provider process that
serves more than one run can hold a revoked token.
> - This pull request replaces the list with one rule. Each native
harness gets its assigned gateway tools through the relay.
> - The benefit is one behavior for all native harnesses, and no gateway
token in a provider process.
## Linked Issues or Issue Description
No public issue exists. The issue is described here.
Refs #13942 (the relay), #17149 (ACPX accepts relayed tools) and #19912
(the design for turn-scoped ACPX MCP authority).
**What existing behavior does this improve?**
How a native harness reaches the tools of its assigned MCP gateway.
**Subsystem affected**
Native runtime in the server (`native-session-executor.ts`,
`native-session-resume.ts`). One Runner test file. One Rust test.
**Current behavior**
- `relayAssignedMcp` is true for plugin tasks, Cursor Cloud, OpenCode,
Codex on a remote target, and Dot.
- For local Codex and for each ACPX agent (Claude, Codex, Cursor, Pi,
Kimi, Grok, Copilot, Antigravity), Core puts
`PAPERCLIP_NATIVE_MCP_NAME`, `PAPERCLIP_NATIVE_MCP_URL` and
`PAPERCLIP_NATIVE_MCP_TOKEN` in the Runner environment. The ACPX host
gives them to the provider as an HTTP MCP server. The local Codex home
gets them in `config.toml`.
- The token is run-scoped. Core revokes it when the run ends.
**Proposed behavior**
- A native run that has an assigned gateway uses the relay. The
condition names no provider.
- Core builds the gateway tools on the control plane, offers them with
the other Runner tools, dispatches each call over PRP, and authorizes
each call again through the gateway.
- Core removes the three `PAPERCLIP_NATIVE_MCP_*` variables from each
Runner environment.
- The tool contract fingerprint has the relay marker for each target,
not only for remote targets.
**Reason and benefit**
- All native harnesses have the same behavior.
- No provider process holds a gateway token.
- A sandbox does not need to reach the gateway URL of the host.
## What Changed
- `server/src/services/native-runtime/native-session-executor.ts`
- The list of special cases is gone. Core validates the binding and
builds the relayed tools when a binding is present.
- Core does not list the gateway for a review run, Grok Bot or Muse. The
tool authority never offered app tools to those. A gateway error can
therefore not fail those runs.
- OpenCode keeps its `on_demand` catalog. That option depends on the
provider, not on the relay.
- Core removes the three variables from each Runner environment.
- `server/src/services/native-runtime/native-session-resume.ts`: the
relay marker in the tool contract fingerprint is unconditional.
- `doc/execution-semantics.md`: the text about the tool contract
fingerprint now states the rule, the tool names and the cutover.
- App tool schemas. New module
`packages/paperclip-runner/src/drivers/tool-input-schema.ts` compiles
each schema by itself, in its declared dialect (draft-07, 2019-09 or
2020-12), with no shared `$id` registry. Core uses it when it builds the
relayed tools (`assigned-mcp-tools.ts`). Core does not offer a tool
whose schema has an unsupported dialect or does not compile, and writes
a log entry with the tool name and the reason. The run does not fail.
The ACPX tool bridge and the OpenCode MCP bridge use the same function.
The tools of Paperclip itself stay strict.
- Large arguments. The ACPX sidecar sends the arguments of a tool call
to runnerd with no display limit. The limit is the tool-call admission
limit of runnerd, 256 KiB. A larger call fails with a tool error that
gives the limit. The sidecar does not dispatch it.
- Tests in `native-session-executor.test.ts`,
`native-session-resume.test.ts`, `runtime-context-materializer.test.ts`
and `acpx_provider_backend.rs`.
Not changed in this pull request: the direct code path. It is still in
the Runner, but it receives no binding. #21311 deletes it.
## Behavior that each harness sees
| Harness | Before | After |
|---|---|---|
| Codex on a remote target, OpenCode, Cursor Cloud, Dot, plugin tasks |
Relay | No change |
| Local Codex | Direct: a `paperclip-assigned` server in `config.toml` |
Relay. App tools are dynamic tools named `app_<name>_<digest>` |
| ACPX agents | Direct: a second HTTP MCP server | Relay. App tools are
on the `paperclip` bridge as `mcp__paperclip__app_<name>_<digest>` |
| ACPX with a preconfigured home | No gateway | No change |
| Grok Bot, Muse | Got the variables and ignored them | Do not get the
variables |
- A relayed tool name has a maximum of 64 characters. With the
`mcp__paperclip__` prefix it has a maximum of 80.
- Permission behavior keeps the same intent. For Claude in the
`approve-reads` and `approve-paperclip` modes, app tools ask for
approval, as before. For the other ACPX agents the permission decision
does not use the server name.
- In a planning run or an ask run, the relay offers only tools with read
risk. The direct gateway offered all tools. This is stricter.
- When the catalog is too large, each harness gets the two compact tools
`paperclip_search_assigned_tools` and `paperclip_call_assigned_tool`.
- The four helper tools for resources and prompts
(`paperclip_list_resources`, `paperclip_read_resource`,
`paperclip_list_prompts`, `paperclip_get_prompt`) are not in the relay.
No harness loses them: a run token permits only `tools/list` and
`tools/call` (`run-preparation.ts`), so these four tools already fail
with `gateway_token_action_denied` on the direct path.
## State saved by the previous release
- **Local sessions of each provider start again one time.** The tool
contract fingerprint changes for local targets. The next run sees a
different fingerprint, starts a new provider session with the full task
context in the same workspace, and keeps the old saved state. This is
the existing rotation for a tool contract change. Codex needs it,
because Codex cannot add tools to a thread that it resumes. For the
local providers that already used the relay, the restart is not
necessary. It is accepted so that the rule names no provider.
- **Remote sessions continue.** Their fingerprint does not change. For
ACPX, the attach of the next run replaces the saved tool set.
- **A kept Codex home.** Paperclip writes `config.toml` again at each
start and recovery. The old gateway server and its token are removed.
- **A deploy lands during a run, and the Runner survives.** The
controller attaches to the same Runner for the same run. The old
provider keeps its direct gateway connection. The controller detach
keeps the gateway token of that run, and Paperclip revokes it when the
run ends.
- **A deploy lands during a run, and the Runner is dead.** The Runner
starts again for the same run with its saved tool set and no direct
gateway. The assigned app tools are not available for the remainder of
that one run. The next run gets them through the relay. This applies to
each harness that moves to the relay. It has no test.
- Paperclip refuses no saved state and discards none.
## Verification
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-resume.test.ts`: `1508 passed
| 7 skipped`, 1 failed. The failed test is "archives a damaged prior
epoch before completing guarded same-task replacement with real runnerd"
(`ENOENT … authority-epochs`). It fails in the same way on `master` at
|
||
|
|
2483cb46c2 |
feat(plugins): deliver configured environment to plugin Runner tasks (#21280)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - An agent gets environment variables from its execution environment, its own configuration, its project, and its routine. Secrets are resolved for each run. > - A plugin environment can run the agent as a native Runner task on its own host, through the environment task contract. > - That contract has no field for the agent's variables. Runs on a plugin environment get only the variables that the plugin sets itself. > - This pull request adds a versioned contract field for the values, sends them on a session handoff, lets Codex commands inherit the variables in an external sandbox, and lets a plugin protect the variables that it sets itself. > - The benefit is that plain and secret variables work on plugin environments in the same way as on local environments. ## Linked Issues or Issue Description Refs #21233. That pull request by @lucktastic finds the same gap and proposes the same capability and field names (`configuredEnvironmentVersion: 1`, `configuredEnvironment`). This pull request uses the same names, so providers implement one contract. It differs in these places: - Codex commands get the variables through an external-sandbox denylist, not through key names sent at `prepare`. A retained Runner does not need a new preparation when a key is added or removed. - A session handoff `resume` gets the values, because it starts a new Runner on another computer. Value changes on the same computer need no new field: Core's process reuse key includes the values, so Core starts a new task. - A driver can declare `protectedEnvironmentKeys`. Paperclip rejects or warns when one is saved. - The run log names skipped variables only for an older provider. Thank you to @lucktastic for the diagnosis and the contract shape. **What happened?** An agent has environment variables, plain and secret. Its runs on a plugin task environment do not get them. `PAPERCLIP_CONFIGURED_ENV_KEYS` in the run lists only the variables that the plugin sets. **Expected behavior** Each run gets the agent's configured variables. **Steps to reproduce** 1. Set a plain and a secret environment variable on an agent. 2. Assign the agent to a plugin environment that provides Runner tasks. 3. Start a run. Read the names in `PAPERCLIP_CONFIGURED_ENV_KEYS` in a command that the agent runs. 4. The agent's names are not in the list. ## What Changed - **SDK contract** (`packages/plugins/sdk/src/environment-tasks.ts`): - A provider advertises `configuredEnvironmentVersion: 1` on `describe`. - `submit` gets `configuredEnvironment`. `resume` gets it only for a session handoff. - Names, count, and sizes have the same limits as the Runner projection. - **Server**: - `createRunnerdBackendWithinSessionClaim` gives the run's configured projection to the plugin task owner. The projection has no provider credentials and no generated runtime keys. - `createPluginRunnerTaskOwner` sends the fields only to a provider with the capability. An older provider gets no new field, and the run log names the variables that were not set. The log never prints values. - **Codex** (`codex-security-config.ts`): - An externally sandboxed launch uses `inherit="all"`, `ignore_default_excludes=true`, and an `exclude` list of Codex's own credentials, instead of an `include_only` list fixed at preparation time. - A test reads the Runner's Codex environment keys, and fails if a new key is not classified. - Host launches keep the bounded allowlist. - **runnerd** (`codex_provider.rs`): `runtime_notification_codex_args` still grants the notification files to read-only profiles when no allowlist is present. Plan mode in an external sandbox needs these grants. - **Protected keys**: - An environment driver can declare `protectedEnvironmentKeys` in its manifest. - Paperclip rejects a protected key on an agent or environment that uses the driver. The 422 has code `protected_environment_variable`. - On a project, a routine, or another agent, Paperclip saves the key. The response has a `configurationWarnings` notice that names the environments. The project and routine pages show the notice as a toast. The agent pages already show configuration warnings. - **Docs**: `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md` describes the provider obligations. - `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the reviewed SDK source pin. ## Verification - `packages/plugins/sdk`: `vitest run tests/environment-tasks.test.ts` (38 pass). Accepts bounded values. Rejects bad names, NUL bytes, too many variables, and oversized values. - `server`: `vitest run src/services/native-runtime/plugin-runner-task.test.ts` (95 pass). Values go only to a provider with the capability, never in `prepare`. `resume` sends values only for a handoff. - `server`: `vitest run src/services/protected-environment-keys.test.ts src/__tests__/project-routes-env.test.ts` (10 pass). Rejection, notices, and the project route response. - `server`: all route suites that import the project, routine, environment, or agent routes (45 files, 1689 pass). - `packages/shared`: `vitest run src/validators/plugin.test.ts` (64 pass). - `packages/paperclip-runner`: `vitest run src/drivers/codex src/configured-environment.test.ts src/live/runner-provider-launch.test.ts` (444 pass). - `packages/paperclip-runner/runner`: `cargo test -p paperclip-runner-core --lib runtime_notification` (9 pass, 2 ignored as before). - `pnpm typecheck` in `packages/shared`, `packages/plugins/sdk`, `packages/paperclip-runner`, `server`, and `ui`. - Not run locally: `pnpm verify --full` and browser suites. Required CI covers the final revision. ## Risks - **Existing providers:** low risk. Core sends no new field to a provider without the capability. Their runs behave as before, with one added log line when variables were skipped. - **Codex in an external sandbox:** commands now see every variable in the Runner environment except Codex's own credentials. The execution environment builds that environment from an allowlist. If the Runner gives Codex a new credential, the drift test fails until the key is classified. - **Saving env:** new rejections happen only when an installed driver declares `protectedEnvironmentKeys`. No driver declares them today. - **Providers:** they must keep the values transient and never save or log them. The doc lists these obligations. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
bcadbde449 |
feat(runner): record the OpenRouter host that served each OpenCode model call (#21298)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - OpenCode agents can use models through OpenRouter, and the Runner proxies those model calls > - OpenRouter routes one model across many hosts, and these hosts use different quantizations (fp4, mxfp4, fp8) > - Chat evaluations with identical prompts behave very differently between runs, but the run log does not record which host served a call > - The run usage shows `provider: "deepseek"` and `biller: "openrouter"`, and `providerRequestId` is always `null` > - This pull request records the serving host and the OpenRouter generation ID for each model call in the run log > - The benefit is that we can compare evaluation cohorts by host, and the cost event names a real OpenRouter generation ## Linked Issues or Issue Description **Subsystem affected** Runner OpenCode provider proxy, runnerd usage normalization, and native run usage accounting. **Problem or motivation** OpenRouter's default routing sends a request to one of many hosts. Each host can use a different quantization. Chat evaluation results change a lot between runs with identical prompts. Paperclip does not record which host served a call, so we cannot compare results by host. `usageJson.providerRequestId` is always `null` for OpenCode runs. **Proposed solution** Every OpenRouter completion response has a top-level `provider` field and an `id` (the generation ID). Each streamed chunk has them too. The Runner proxy reads them from the first chunk after it forwards the bytes. The Runner attaches them to the usage of that call. The values go to the run log, and the run's first generation ID goes to `usageJson.providerRequestId`. **Alternatives considered** A new PRP event type such as `provider.upstream.served` would need a new event enum value. An older control plane rejects an unknown event type. A `harness.diagnostic` event shows in the task chat as a diagnostic. The usage payload already flows to every consumer, and older consumers ignore extra fields. Thus this change uses the usage payload. **Additional context** Run-log change only (AGENTS.md section 7). The data stays in `heartbeat_run_events` and `heartbeat_runs.usageJson`. It does not touch `packages/shared/src/telemetry/` or OpenTelemetry. ## What Changed - New `createUpstreamServedTap` (`packages/paperclip-runner/src/drivers/opencode/upstream-served.ts`). It is a pass-through stream. It pushes each chunk downstream first, then it reads `provider` and `id` from the first SSE `data:` event (or from a JSON body at its end). It scans at most 64 KiB of a stream and 1 MiB of a JSON body. It skips encoded bodies and unknown content types. Any parse problem or recorder error stops the scan and leaves the response unchanged. - `startOpenCodeProviderProxy` uses the tap only for 2xx responses. The OpenCode session keeps the most recent served call. Each `item.completed` usage payload and `session.usage()` now carry `upstreamProvider` and `providerRequestId`. The usage dedupe fingerprint does not include these fields, so the number of usage events does not change. - The OpenCode app-server proxy adds the two fields to `thread/tokenUsage/updated`. Rust `usage.reported` now fills its existing `providerRequestId` field and adds an optional `upstreamProvider`. The TypeScript runnerd transport keeps both fields when it rehydrates the usage notification. `usage.schema.json` documents `upstreamProvider` (the protocol manifest and schema bundle are regenerated). - Server native turn accounting keeps the first generation ID the run reports. The usage checkpoint and the native adapter result carry it as `providerRequestId`, so `usageJson.providerRequestId` and the final cost event name it. - `doc/run-log-events.md` documents the fields in a new "OpenRouter serving host" section. ## Verification - Red/green: `opencode-server-driver.test.ts` "passes OpenRouter's stream through unchanged and records the host that served it on the turn's usage". The test sends a streamed fixture with a comment line, a host chunk, a malformed chunk, and `[DONE]` through the proxy. It checks that the bytes are identical and that the turn's usage names `Novita` and the generation ID. The test fails without the proxy wiring. - `upstream-served.test.ts`: streamed fixture, chunk split inside a multibyte character, non-streamed JSON, malformed chunks, a truncated JSON body, an encoded body, the 64 KiB limit, and a recorder that throws. - `opencode-proxy-usage.test.ts`, `runnerd-codex-transport.test.ts` (rehydration), Rust `provider_events::tests::carries_the_serving_host_and_generation_on_proxy_usage`, and server `native-usage-accounting.test.ts` (first generation ID stays stable across turns). - Local: OpenCode driver, OpenCode proxy, and runnerd transport suites (31 files, 596 tests) pass. `cargo test -p paperclip-runner-core` passes (one timing test failed once under load and passed alone). Runner and server `tsc --noEmit` pass. Protocol manifest, type, and coverage checks pass. - Latency: a local benchmark proxied a 400-chunk SSE stream 200 times with the tap and 200 times without it. The median time to first byte was 292 µs with the tap and 259 µs without it (+33 µs, which is the extra stream stage). The total stream time changed from 491.4 ms to 492.5 ms. The tap spends about 5.5 µs of CPU to inspect the first chunk, after it has forwarded that chunk. OpenRouter time to first token is hundreds of milliseconds. - Paid evidence: one single-case run of `chat-features.runner-opencode.local.skills.fork-read-only` on this branch. The result is in a comment on this pull request. ## Risks - Low risk. All new fields are optional and additive. An older control plane ignores the extra usage fields. `usage.schema.json` is not bound by the event schema, and no validator enforces it on `usage.reported`. - Saved state from the previous release: this change adds no persisted run, session, checkpoint, or launch state. A saved session resumes unchanged. A run that is in flight during a deploy reports usage without the new fields until its Runner is new. `usageJson.providerRequestId` then stays `null`, as before. - `providerRequestId` on the run and its cost event is the run's first generation ID, not each call's ID. The value is stable, so a late usage receipt still matches the posted charge. Per-call IDs and hosts are in the run log. - If OpenCode starts a new model call before it reports the usage of the previous call, a usage event can name the newer call. OpenCode reports a message's usage when its step ends, so this is not expected. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code as an agent with tool use (shell, file edits, test runs). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
45523d21d0 |
fix(runner): re-bind any settled ACPX checkpoint across a Runner upgrade (#21220)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Runner runs Claude Code through the ACPX backend and saves a durable checkpoint so a later run can continue the same conversation > - Each Runner release changes the ACPX launch profile digest that the checkpoint stores > - Restore re-binds a saved checkpoint to the new digest only when it is `suspended`, or when the same run resumes an interrupted turn > - A retired or parked Claude Runner leaves an idle `session_open` checkpoint, so a follow-up from the next run fails with `provider_state_incompatible` before its first turn > - `run.attach` already accepts that checkpoint. The restore gate is stricter than the attach it serves > - This pull request defines a settled checkpoint once and uses the same rule for attach and for the upgrade re-bind > - The benefit is that a completed Claude conversation continues after a Runner upgrade, and a refused re-bind names its reason ## Linked Issues or Issue Description Fixes #21212 Refs #20112 (the same-run interrupted-turn case, which this change keeps as is) ## What Changed - Added `settled_checkpoint_blockers`. A settled checkpoint is not closed, has a provider identity, has no active turn, has no unconfirmed provider exit, and has no undelivered event except a `session.resumed` notice. `run.attach` now admits with this function. The warm-attach readiness probe also uses it, and still waits for a pending `session.resumed` notice. - Restore re-binds any settled checkpoint to the current launch profile digest, whichever run saved it and whatever its lifecycle (`session_open`, `prepared` or `suspended`). Candidate admission, a changed launch shape and pending runtime requests still block the re-bind. - `rebinds_interrupted_turn` keeps only its distinct case: the same run's `turn_starting` or `turn_active` turn, under the lost-turn recovery preconditions from #20112. The settled rule now covers its `session_open` case. - A digest mismatch that is not re-bound now names each reason, for example `(not re-bound: durable_active_turn)` or `(not re-bound: launch_shape_changed)`. The classification stays `provider_state_incompatible`. - Updated "Settled ACPX sessions across a Runner upgrade" in `packages/paperclip-runner/docs/durable-recovery.md`. ## Verification From `packages/paperclip-runner/runner`: - `cargo test -p paperclip-runner-core --lib acpx_provider_backend`: 48 passed - `cargo test -p paperclip-runner-core --test acpx_warm_attachment`: 6 passed - `cargo test --locked -p paperclip-runner-core --no-fail-fast`: 1108 passed, 0 failed, 6 ignored - `cargo fmt --all -- --check` and `cargo check --locked --workspace`: clean New and changed tests: - `settled_checkpoints_of_a_retired_runner_rebind_for_the_next_run_after_an_upgrade` covers `session_open` with a semantic result, `prepared` with an identity, and a pending `session.resumed` notice. Each checkpoint is saved by run 1 and restored by run 2 under a new digest. Restore does not write the checkpoint, `run.attach` succeeds, and the saved digest becomes the current one. This test fails before the fix. - `continues_a_settled_conversation_in_the_next_run_after_a_runner_upgrade` runs end to end with the fake sidecar. The replacement Runner attaches, resumes the same provider session, and emits exactly one `turn.started`. This test fails before the fix with the production error. - `unrestorable_saved_sessions_keep_strict_bindings_and_name_their_class` no longer lists `session_open` as strict. Each remaining strict case asserts the reason in its error. I did not run clippy because CI does not run it. I did not run `pnpm verify` because only Rust and docs changed. ## Risks - The re-bind now admits more checkpoints. It admits only the states that `run.attach` already admits on the same build, so no in-flight provider work moves to a new build. An active turn, a pending runtime request, an unconfirmed provider exit, or an undelivered non-resume event still blocks the re-bind. - A same-run `session_open` checkpoint that has a semantic result or an active goal now re-binds. The same build already re-opens this state. The result and goal checks gate lost-turn recovery only, and that recovery path is unchanged. - Session start still verifies every launch artifact, the command digest and the restored provider identity after the re-bind. - No migration and no API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, sub-agents) and extended thinking. One sub-agent wrote the change. A second session reviewed it before this pull request. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
38501b1170 |
Add turn-scoped attachment downloads to native Runner (#20956)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native agents open input images with a file reader. > - Plugin Runner tasks use an authenticated Runner connection. > - Runner needs a way to fetch an authorized file from the tenant app. > - The tenant grants access to the exact attachment for the active run. > - Runner downloads and verifies it before it returns a local path. ## Linked Issues or Issue Description Refs #19740. Builds on the attachment authorization and command transport added in #20939. ## What Changed - Add the PRP v3 `runner.attachment.download` command for native workspace providers. - Issue short-lived grants for exact authorized wake and historical attachments. Recheck the active run, source, size, and SHA-256 at the tenant endpoint. - Download current-wake inputs before provider dispatch. Download historical files when the authorized chat reader asks for them. Preserve SSH and sandbox command staging as a fallback. - Run HTTPS transfers off the Runner control loop. Report completion with one PRP event per transfer. Drop pending results on cancellation and clear files at turn settlement, authority rotation, and restart. - Grant restricted Codex sessions read-only access to the attachment directory. ## Verification - 69 targeted server attachment tests passed. They cover wake and historical downloads, authorization failures, and command fallback. - 22 focused historical attachment tests passed, including grant redemption under a held database lock. - Six focused Rust attachment tests passed, including a source-pinned Codex sandbox check. The sandbox could read the staged file but could not modify it or read adjacent Runner state. - The real Runner transport test confirmed one durable command and one completion event for a transfer. Restart-continuation tests confirmed fresh attachment paths appear before dispatch. - Server and Runner TypeScript checks, Rust formatting, and PRP generated schema checks passed. - Required GitHub CI passed on `7a33752bd7`. [CI run](https://github.com/paperclipai/paperclip/actions/runs/43122967837) - Greptile scored the same revision 5/5 with no unresolved review threads. [Review](https://github.com/paperclipai/paperclip/pull/20956#issuecomment-6808582134) ## Risks - Saved provider and Runner identities keep their existing formats. A previous Runner falls back to command staging where the target supports it. A restart continuation refreshes temporary attachment paths before dispatch. - Plugin Runner tasks require a compatible Runner ROM and outbound HTTPS access to the configured tenant origin. - Live agent-computer qualification remains a release check. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, shell tools, and local test execution. Exact variant and context window are unavailable in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #123` / `Closes #123` / `Refs #123` or (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
d9d5958d09 |
fix(runner): reject an out-of-order controller command instead of exiting (#20946)
## Thinking Path - Paperclip's controller and the sandbox Runner exchange numbered commands; the Runner refuses a command whose number is not exactly one past the last it processed. - Staging's most common "Runner gone" failure was a follow-up controller whose journal was behind the Runner. #20928 removed the cause on the server (a warm follow-up now launches from the journal its retired predecessor saved) and added a defence that reconciles a journal behind only by the predecessor's settled close. - The Runner still treated an out-of-order command as fatal: `durable/state.rs` returned an error and the process exited. A controller with a stale journal from any other cause (state saved by an older release, a mid-run deploy) therefore lost the Runner process and the run could only fail as "Runner gone" (`runner_remote_process_identity_unavailable`). - This pull request makes the Runner refuse such a command with a precise result instead of exiting, and makes the server rest that run with a precise code and a Retry whose next run reconciles the journal (the #20928 defence) before it opens the session. ## Linked Issues or Issue Description - Refs #20928 (the server-side root-cause fix and the journal reconciliation this relies on) - Refs #20907 (makes runnerd exit text visible in the run log) - Duplicate search: searched PRs for "controller sequence contiguous", "controller_sequence_mismatch" and "out-of-order command"; none changes the Runner's handling. ## What Changed - `runner-core/src/durable/state.rs`: a new command whose `controllerSeq` is not `lastControllerCommandSeq + 1` (behind or a gap) now yields `CommandDisposition::Reject` with `status: "rejected"` and `{ code: "controller_sequence_mismatch", message, expectedControllerSeq, receivedControllerSeq }`. The command is not executed or journaled and the last sequence is unchanged; the process keeps running. Replay of an already-processed command and the compacted-history rejection are unchanged. - `native-session-executor.ts`: a failure carrying `controller_sequence_mismatch` is classified as `native_controller_journal_behind_runner`, a permanent (terminal) failure with a Retry action that says no provider work began and that Retry reconciles the journal from the saved checkpoint or stops with a precise error. - Tests: - Rust `tests/durable_recovery.rs`: out-of-order (behind and gap) commands are rejected with the precise code, the journal is unchanged, and the next contiguous command executes. The existing `command_gaps_and_identifier_reuse_fail_closed` unit test now asserts the rejection instead of an error. - Real runnerd `src/live/runnerd-stale-controller-journal.test.ts`: a follow-up controller with a journal missing the predecessor's close now gets `"code":"controller_sequence_mismatch"`, the Runner process stays alive and its durable sequence is unchanged; with the current journal it resumes. - Server: the classification and terminal disposition of the new code. ## Verification - Red (master Runner): the Rust test panics on `expect("an out-of-order command is not fatal")`; the real-runnerd test receives `runnerd exited unexpectedly with code 1: paperclip-runnerd: controller sequence must be contiguous: expected 6, received 4`. - Green: `cargo test -p paperclip-runner-core --test durable_recovery` (4 passed); `cargo test -p paperclip-runner-core` lib suite (the only other failure, `task_memory::tests::monitor_bounds_notices`, is a timing test that passes alone); `npx vitest run src/live/runnerd-stale-controller-journal.test.ts` with a runnerd built from this branch (2 passed); server `native-session-executor.test.ts` focused tests (71 passed). - `pnpm --filter @paperclipai/paperclip-runner check:acpx-profiles`: profiles match the release manifest. No ACPX profile, attestation, pin or pack contract schema changed (`rules/acpx-profile-identity.mdc`): the change is Runner source behaviour with an existing command-result shape, so no profile version or `commandDigest` changes. ## Risks - A controller that is *ahead* of the Runner (a gap) is now rejected instead of fatal. Neither case executes the command; the Runner journal is unchanged, so this is strictly safer than exiting. - The rested run needs a Retry. Before this change the same run failed after its retries as "Runner gone" with no precise cause. - Runner source changes the Daytona image content ID; a new image is needed for chaos waves on this revision. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), 1M-token context, tool use (shell, file edits, tests) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a52f622be8 |
fix(native): retry a remote provider startup failure once its runner exit is proven (#20874)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) start a Runner in the
sandbox, and the Runner opens the provider session
> - When the provider rejects `session.open` at startup, no provider
work began
> - #20808 ends every such remote rejection at once with Retry, because
restart recovery could not prove that the failed runner was gone. A
fault that clears by itself therefore needs one human click
> - The Runner transport already settles the failed runner on close and
observes its exit. That is the missing proof
> - This pull request records that proof on the failure and lets the
same-run bootstrap retry use it. It stops a startup error that repeats,
so an image fault that a retry cannot fix does not loop
> - The benefit is that a transient remote startup failure recovers
without a person, and a permanent one still rests at once with a working
Retry that names the error
## Linked Issues or Issue Description
Refs #20808
Refs #20832
Refs #20820
**What happened?**
A remote (Daytona) native run whose provider rejected `session.open` at
startup ended as `native_provider_startup_rejected` with Retry. This
happened even when the Runner had settled and exited, and no provider
session, event or checkpoint existed. A person had to click Retry for a
fault that the durable record proves did no provider work.
**Expected behavior**
With a proven runner exit and no provider work, the run retries its
bootstrap on the same run automatically. A startup error that repeats on
the next attempt stops at once with Retry and names that error. Failures
saved without the proof keep today's behaviour.
**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-startup-rejected-retry.integration.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "remote startup rejection|session.open rejection"`
2. On master, 3 cases fail: the proven remote rejection is
`terminal_failure`, the repeat does not name its error, and restart
recovery holds the proven failure on
`ambiguous_remote_recovery_evidence`.
**Paperclip version or commit**
master at
|
||
|
|
a219dd7499 |
fix(runner): keep orphan-leaving tests apart from the verifying lineage reap (#20849)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Runner stops a dead runnerd's provider and tool processes from its process lineage journal (`process_lineage::journal::reap_journals`) > - The reap refuses its proof (`unattributed_orphan`) while any live orphan (parent PID 1) of this user started after the dead runnerd's last sample, whoever started it. This rule is deliberate > - `cargo test` runs the runner-core lib tests in parallel in one process, and several of those tests leave orphans of this user on purpose > - When one of those orphans starts during the reap in `reaping_stops_a_dead_runners_provider_and_its_escaped_tool_job`, the reap correctly refuses, and `ci / Verify Paperclip Runner (rust)` fails > - This pull request keeps the orphan-leaving tests apart from that reap with a test-only gate, and fixes a related zombie race in the supervisor integration tests > - The benefit is that the Runner Rust job stops failing on unrelated pull requests ## Linked Issues or Issue Description Refs #20834. Refs #20770. **What happened?** `ci / Verify Paperclip Runner (rust)` failed intermittently in `process_lineage::journal::tests::reaping_stops_a_dead_runners_provider_and_its_escaped_tool_job` (journal.rs:649), for example in run 43044317310 on #20834. The panic was `ReapReport { verified: false, reason: Some("unattributed_orphan"), journals: 1, signalled: 4, remaining: [4178] }`. The test's own thread was TID 4142, so the leftover process started while the test ran. The same test failed once on #20770 and passed on re-run. **Expected behavior** The reap proves the test's provider and escaped tool job gone, and no other test can make it refuse. **Steps to reproduce** 1. Build the runner-core lib tests for Linux (`cargo test --release -p paperclip-runner-core --lib --no-run`). 2. Run the binary 40 times in `docker run --init --cpus 2 -e RUST_TEST_THREADS=8`. 3. See `reaping_stops_a_dead_runners_provider_and_its_escaped_tool_job` fail with `unattributed_orphan` (3 of 40 runs here). ## What Changed - Root cause: the reap's orphan rule covers every process of this user on the host. The reaping test and the orphan-leaving tests run in the same test process at the same time. The test's own processes are always attributed through the recorded group and session, so the leftover was another test's orphan. - `process_lineage::host_orphans` (test-only): a gate. A test that can leave an orphan of this user holds `may_orphan()` until the orphan is gone or a zombie. A test whose reap must verify holds `no_foreign_orphans()`. Holders of `may_orphan()` never wait for each other, so a test can hold several without a deadlock. `wait_until_group_gone()` waits until a killed group has exited, because a signal takes effect after `kill` returns. - `reaping_stops_a_dead_runners_provider_and_its_escaped_tool_job` holds `no_foreign_orphans()`. - These lib tests now hold `may_orphan()`: - `reaping_refuses_an_orphan_born_after_the_last_sample` (deliberate orphan) - the three `process_supervisor` `*_stops_a_tool_job_that_left_its_session` tests (escaped tool job) - `orphaned_startup::restart_keeps_the_fence_while_a_child_survives_its_exited_leader` (`sleep 30` outlives its leader) - `exited_child_reader_deadline_bounds_continuous_descendant_stdout` (flooding subshell outlives the Codex provider) - `a_timed_out_turn_start_never_establishes_a_resumable_rejection` (`sleep 10` can be orphaned for a moment during shutdown) - the five `provider_backend` tests that use `disconnected_receipt_limit_fixture` (`while :; do sleep 1; done`, killed through `fence_local_receipt_limit`). The fixture returns its guard first, so the guard is released after the executor stops the provider - the two `late_park_observation_*` tests (`cat >/dev/null` runs until shutdown) - `a_held_cancellation_that_cannot_reach_the_provider_keeps_the_completed_turn` (shutdown during `sleep 2`) - New test `a_verifying_reap_waits_for_held_orphans_and_holds_new_ones_back` checks the gate's ordering. - `tests/process_supervisor.rs`: `forced_process_group_cleanup_stops_harness_and_worker` and `natural_harness_exit_also_cleans_up_workers` slept 20 ms and then used `kill -0` on a worker that was re-parented to init. `kill -0` succeeds for a zombie, and init reaps it on its own schedule. The tests now check that the process is not running (gone or a zombie), with no sleep. Cleanup already proves that before it returns. - No product code changes. The reap's orphan rule is unchanged. Audit of the other tests in the job (`check:runner`): - Only the runner-core lib binary can overlap the reap. `cargo test --workspace` runs test binaries one after another. - A second sweep, after review, checked every `/bin/sh` launch and every external command (`sleep`, `cat`, `perl`, `nohup`, `touch`, `mv`) in runner-core's `src/`, including `src/tests/*.rs`. The first sweep had missed tests that start a provider through `CodexProvider::start` or that live in `src/tests/`. - Other lib tests that start processes are not affected. They run shells that use only builtins (`configured_environment`, the other `process_supervisor` and `codex_provider` tests, `acpx_provider_session`), or run a short foreground `touch` that finishes long before teardown (`disconnected_codex_output_is_durable...`), or they start direct children that the test waits for (`oom_score`, `exited_writer`, `process_group_leader`). `publisher_crosses_real_codex_tool_boundary...` is `#[ignore]`. - The other `reap_journals` tests refuse before the orphan rule applies, or (the orphan test) require only their own orphan in `remaining`. - The `kill -0` checks in `tests/codex_provider.rs` and `tests/acpx_park_preparation.rs` target direct children that runner-core waits for before it returns, so they are deterministic. ## Verification Linux runs used Colima on macOS (linux/arm64), `rust:1.97.1-bookworm`, `--release`, and `docker run --init`, so orphans go to an init at PID 1, as on CI. - Full runner-core lib suite, `--cpus 2`, `RUST_TEST_THREADS=8`, 40 runs: before the fix, 3 runs failed with the CI signature. After the fix, 0 runs failed, and no other test failed. - The reaping test beside a temporary test that keeps leaving short-lived orphans (not committed), `--test-threads 2`, 10 runs: before the fix, 10 failed. After the fix, with the temporary test holding `may_orphan()`, 0 failed. - An orphan-leaving shell outside the test binary made the unfixed test fail in 9 of 10 runs with the same signature. This confirms the mechanism. - After the review fix: macOS `cargo test --release -p paperclip-runner-core --lib -- receipt_limit late_park_observation a_held_cancellation process_lineage process_supervisor orphaned_startup exited_child_reader_deadline a_timed_out_turn_start`: 65 passed. Linux full lib suite (same settings) ×20: 0 failures. - macOS (darwin-arm64): `cargo fmt --all -- --check` passes. `cargo test --release -p paperclip-runner-core --lib -- process_lineage process_supervisor orphaned_startup exited_child_reader_deadline a_timed_out_turn_start`: 51 passed. `cargo test --release -p paperclip-runner-core --test process_supervisor`: 7 passed. - Worker cleanup tests: with `tini` as init, 60 unfixed runs on 0.5 CPU with busy loops all passed, because `tini` reaps at once. In a container whose PID 1 never reaps (`sleep infinity`), the unfixed tests fail at `assertion failed: !process_exists(worker_pid)` because the killed worker stays a zombie. The fixed tests pass. The unfixed tests depend on how fast init reaps. ## Risks - Low risk. All changes are in tests or `#[cfg(test)]` code. - The gate covers the orphan producers that exist now. A new lib test that leaves an orphan of this user must hold `may_orphan()`. The `host_orphans` doc comment says so. Otherwise, it can make the verifying reap refuse again. - A process outside the test binary that leaves an orphan of this user during the reap still makes the test refuse. This is correct product behavior. CI runs no such process during `cargo test`. - `no_foreign_orphans()` waits until every `may_orphan()` guard is released, so the reaping test can wait a few seconds behind those tests. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file editing, Docker). Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4529dfb77f |
fix(runner): keep Codex planning read-only on Daytona without network namespaces (#20820)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A planning task runs a native Codex agent in read-only planning mode
before the user approves a Plan
> - On a Daytona sandbox, every Codex planning run failed before its
first turn: `error creating thread: failed to load AGENTS.md
instructions for environment 'local': fs sandbox helper failed ...
bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted`
> - Codex 0.160.0 reads files through a filesystem helper when the
permission profile restricts reads. That helper always runs bubblewrap
with `--unshare-net`. Daytona has no `CAP_NET_ADMIN`, so bubblewrap
cannot bring up loopback in the new network namespace
> - The planning profile must stay read-only. Codex's plan collaboration
mode is only a prompt, so the external-sandbox profile (#20808 tried it,
then reverted it) would let planning runs write files
> - This pull request keeps the read-only profile, but inside an
external sandbox it gives the profile full-disk read, no writes and
network. Codex then reads files directly and runs commands without a new
network namespace, and it still refuses writes
> - The benefit is that Codex planning works on Daytona and stays
read-only
## Linked Issues or Issue Description
Refs #20808
**What happened?**
A Codex planning task on a Daytona execution environment failed at
startup. The run ended with `provider_session_open_rejected` and the
Codex message `bwrap: loopback: Failed RTM_NEWADDR: Operation not
permitted`. Standard and ask runs on the same sandbox worked.
**Expected behavior**
A planning run on Daytona starts, writes its Plan and asks for approval.
Codex must still refuse file writes and mutating commands during
planning.
**Steps to reproduce**
1. Configure a native Codex agent with a Daytona execution environment.
2. Create a task in planning mode.
3. The first run fails before its first turn with the bubblewrap
loopback error.
## What Changed
- `createIsolatedCodexAppServerArgs`: when
`PAPERCLIP_RUNNER_EXTERNAL_SANDBOX=1`, the
`paperclip-runner-workspace-read-only` profile is now `{":root"="read"}`
(plus the brokered `GH_CONFIG_DIR` write that planning already had). Its
network still follows `PAPERCLIP_RUNNER_NETWORK_ACCESS`. It has no deny
entries, so Codex 0.160.0 treats it as full-disk read and skips its
filesystem helper, which always unshares the network. Local and SSH
planning keep the narrower profile unchanged.
- The Codex session context now records `rootAccess` `read` (planning)
or `write` (default) in an external sandbox, instead of always `none`.
- The Runner still sends `permissions:
"paperclip-runner-workspace-read-only"` and the plan collaboration mode
on `thread/start` and `turn/start` for planning. A new Rust test pins
that for an externally sandboxed read-only launch (no `sandbox:
danger-full-access`, no `externalSandbox` `sandboxPolicy`).
- A new Rust test proves that a launch saved with the previous planning
profile rules has the same stable launch identity as a launch with the
new rules, so saved sessions carry forward.
- New manual Product E2E suite `codex-planning-read-only`
(`runner-codex` on local and Daytona). The planning turn runs one shell
write, writes a Plan whose first line records the outcome, asks for
approval, and finishes after approval. A grader reads the run events and
fails if no write attempt was seen or if any write succeeded.
- `docs/codex-driver.md` describes the external-sandbox planning
profile, why it avoids the helper, and its read scope.
## Verification
- Red on Daytona, master `123ee31552`: [run
43029207500](https://github.com/paperclipai/paperclip/actions/runs/43029207500),
cell `core-compatibility.runner-codex.daytona.plan-revise-accept` failed
twice with `error creating thread: failed to load AGENTS.md instructions
for environment 'local': fs sandbox helper failed with status exit
status: 1: bwrap: loopback: Failed RTM_NEWADDR: Operation not
permitted`.
- Planning turn completes on Daytona: [run
43031462803](https://github.com/paperclipai/paperclip/actions/runs/43031462803),
cell `core-compatibility.runner-codex.daytona.plan-revise-accept`. The
planning run succeeded, saved a two-step Plan with the draft marker and
opened a confirmation for that revision (all API checks passed). The
cell then failed a later browser check: the Plan preview rendered as the
`task-chat-plan-preview-fallback` card, which the harness does not
accept. That is a separate UI placement gap.
- Write attempt on Daytona: [run
43038547892](https://github.com/paperclipai/paperclip/actions/runs/43038547892),
cell `codex-planning-read-only.runner-codex.daytona.plan-write-refused`
on `f30aeb3e72`. The planning run succeeded and ran the shell write
`printf probe > plan-write-probe-....txt || echo WRITE_REFUSED`. It
ended `failed` with exit 1 and no file, because bubblewrap could not
start the command (`bwrap: setting up uid map: Permission denied`). The
write was refused fail-closed by the sandbox launcher, not by a
read-only filesystem error. The stricter grader in
|
||
|
|
ff8ec41e6a |
feat(runner): upgrade ACPX to 0.22.0 and move patched behavior into Paperclip (#20770)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Runner and the shared `adapter-utils` engine run ACP agents through the embedded ACPX library > - Paperclip pins ACPX 0.13.1 and keeps a large patch (`patches/acpx@0.13.1.patch`) on it > - ACPX 0.22.0 adds host hooks (`onAcpMessage`, `assertActive`, `processLifecycle`, prompt-response `_meta`, `spawnAgent`-friendly launch) that make much of that patch unnecessary > - A large patch is expensive to port and hides behavior that belongs to Paperclip > - This pull request upgrades to ACPX 0.22.0 and moves most patched behavior into Runner and `adapter-utils` > - The benefit is a patch that is 484 lines instead of 1,334, with the moved behavior covered by Paperclip tests ## Linked Issues or Issue Description Refs #20675, #14430, #15075, #17942, #9980 **Proposed enhancement** Upgrade the embedded ACPX library from 0.13.1 to 0.22.0. Keep only the patch behavior that stock 0.22 cannot express. **Motivation** The 0.13.1 patch held behavior that Paperclip owns, for example usage receipts, Copilot policy, model checks and the agent environment. Each ACPX upgrade had to port all of it. ACPX 0.22 gives hooks that let Paperclip do this work itself. **Current behavior** Runner and `adapter-utils` depend on 1,334 lines of patched ACPX 0.13.1 code. **Expected behavior** Runner and `adapter-utils` use ACPX 0.22.0. The patch keeps only: extension methods, Cursor subagent routing, event `_meta` and the message boundary chunk, the protocol guard hook, `spawnAgent`, and `requestExtension`. ## What Changed - ACPX 0.13.1 → 0.22.0 in Runner, `adapter-utils`, the server, the Daytona runner images and the qualified profiles (`acpxVersion`). `@agentclientprotocol/sdk` moves to 1.9.0. - New `patches/acpx@0.22.0.patch` (484 lines). The 0.12.0 and 0.13.1 patches are removed. - Delivery receipts: Runner confirms a permission reply when `onAcpMessage` sees the exact outbound reply. Question and extension replies count as delivered when Runner hands them back to ACPX. New `interaction-delivery.ts`. - Copilot policy: the existing policy function runs from `onAcpMessage`. A violation stops the agent, cancels later permission requests and fails the next prompt through `assertActive`. - Model selection: stock 0.22 validation, plus a sidecar gate (`createAcpxEffectiveModelGate`) that refuses a prompt if the agent reports another model. Grok binds its model when the session opens. Antigravity adds its concrete model to the model list ACPX reads. - Typed session failures: Runner and `adapter-utils` read them from the turn result `_meta` and fail the turn. They advertise the capability themselves. - Pi and Cursor usage receipts: a collector (`prompt-receipts.ts`) binds them to the turn's own `session/prompt` reply. - Goal capability: Runner reads it from the `initialize` reply. It keeps it under `paperclip.goalCapability` in the saved agent capabilities `_meta` and restores it when a saved session loads. - Agent environment: Runner and `adapter-utils` set the agent environment in `spawnAgent`. ACP terminals keep their current environment. `adapter-utils` blocks inheritance of `XAI_API_KEY` and refuses to start if the server has an `ACPX_AUTH_*` variable. - `adapter-utils` spawn hooks (`spawnCwd`, stderr, `onAgentSpawn`): a new `stockAcpRuntimeOptions` maps them to stock `spawnAgent` and `processLifecycle`. - Windows: ACPX 0.22 rejects raw command strings on Windows. `adapter-utils` now passes an argv array there. - Grok rules: a Runner guard (`grok-session-rules.ts`) writes `session/new` `_meta.rules`. - Turn-dispatch journal (#20675): ACPX 0.22 ignores errors from `onAcpMessage`. The adapter keeps a failed `dispatched` write and throws it from the turn's `assertActive`. ACPX calls `assertActive` before the prompt write, so the write still stops. The real-ACPX test now checks this route. `docs/durable-recovery.md` is updated. - Plan events: the sidecar reads 0.22 `status` events tagged `plan`. - CI: the backup-checkpoint lane now installs through the shared `setup-ci-dependencies` action, like PR CI, so it resolves a stale `pnpm-lock.yaml` in place. It runs on any root `package.json` change, and pull requests never commit the lockfile, so its own frozen install failed on dependency changes. - Tests that checked removed patch internals are removed or rewritten: the response-delivery, Cursor-usage and Pi-receipt package contracts, and the goal-persistence packaging test. New unit tests cover the moved behavior. ## Verification Local runs on macOS (darwin-arm64), Node 26, on the rebased head: - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/drivers/acpx src/cli`: 2,000 passed. - `tsc -p packages/paperclip-runner/tsconfig.json` and `tsconfig.test.json`: no errors, except the existing `@paperclipai/paperclip-eval-kernel` build gap in `workflow-harness.ts`. - Runner package contracts (`node --test test/acpx-*-package-contract.test.mjs test/acpx-sidecar-contract.test.mjs`), `scripts/build-inputs.test.mjs`, `scripts/contracts/contract-fingerprints.test.mjs`, `scripts/acpx-patch-packaging.test.mjs`: all pass. `node scripts/generate-acpx-sidecar-contract.mjs --check` passes. - `cargo test -p paperclip-runner-core --test acpx_park_preparation --test acpx_park_readiness --test acpx_warm_attachment --test native_provider_backend`: 86 passed. `cargo fmt --check` passes. - `adapter-utils`: `tsc` clean. `vitest run src/acpx-engine src/mcp-isolation.integration.test.ts`: 457 passed (one Gemini flag test failed once under load and passes when run alone). - Adapters: `claude-local` and `codex-local` `acp.quota.test.ts` (23 and 39 passed), `grok-local` (149 passed). - Server: `scripts/prepare-runner-vendor.test.mjs`, `native-session-executor.test.ts`, `quarantined-task-retry.test.ts` (1,416 passed). - `cargo test -p paperclip-runner-core --lib -- previous_acpx_release acpx_upgrade unrestorable_saved_sessions`: 3 passed. `node --test test/acpx-session-carry-forward-package-contract.test.mjs`: passes. - `patch -p1 --forward` on a clean `acpx-0.22.0.tgz` gives output identical to the port. - Under full-suite load, some Runner live and recovery tests time out. Each one passes when run alone. Full regression is left to CI. Not run: live provider runs (they need provider credentials), the opt-in Kimi live peer test, the Gemini filesystem smoke test, and Windows. ## Risks - Previous-release story: sessions saved by the previous release (ACPX 0.13.1) carry forward. ACPX 0.22 loads the session records 0.13.1 wrote. Runner moves a saved descriptor that names 0.13.1 (`CARRIED_ACPX_RELEASES`) to 0.22.0, and the sidecar keeps 0.13.1 as the ACPX value in the session identity, so the saved session key and identity record still match. The existing launch-profile rebind still decides which sessions resume: a settled session, or the same run's interrupted turn through lost-turn recovery. A session saved by any other ACPX release is still refused as `provider_state_incompatible`. `rules/acpx-profile-identity.mdc` and `docs/durable-recovery.md` now describe this. - Proof: `acpx-session-carry-forward-package-contract.test.mjs` resumes a session record written by the real ACPX 0.13.1 with its Paperclip patch. 0.22 loads the same provider session (no `session/new`), runs a turn and keeps the earlier turn. Rust tests cover a settled 0.13.1 session that the next run attaches to, and a turn interrupted by the upgrade. A recovery-identity test checks that the session key and identity record match across the versions. - `pnpm-lock.yaml` is not committed. CI resolves it for each run, and the lockfile refresh workflow updates it after merge. - Behavior change on the server: `adapter-utils` refuses to create an ACPX runtime if the server environment has an `ACPX_AUTH_*` variable. Before, ACPX ignored such variables. Stock 0.22 would use them to authenticate agents with server credentials. - Delivery confirmation is now Runner-owned. A permission reply is confirmed by the exact outbound message. Question and extension replies are confirmed when Runner hands them to ACPX, not when the pipe accepts them. - Users can select only models that the agent advertises. Stock 0.22 rejects other model IDs (Claude is exempt). - ACP terminals keep the sidecar or server environment, as before. This existing exposure is not changed by this pull request. - The `adapter-utils` Gemini version probe now runs with the server environment plus the run variables, not the run variables alone. - The pack contract digest and profile `acpxVersion` change, so Daytona needs a new runner image. - Grok launch flags (`--model`, `--reasoning-effort`) were checked on 1.0.13 but session open needs a login, so Grok keeps the existing `_meta` path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file editing, subagents). Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
fa3cd6f330 |
fix(native): end provider startup rejections with Retry instead of an ownership hold (#20808)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner starts Codex for native runs. On a remote
sandbox such as Daytona, the sandbox is the OS boundary.
> - A Codex planning run on Daytona failed in `session.open`. Codex's
read-only sandbox helper could not set up loopback in the container
(`bwrap: loopback: Failed RTM_NEWADDR`), and Codex loads `AGENTS.md`
through that helper.
> - The server treated that definite provider rejection as an
interrupted transport. Its same-run bootstrap retry went through remote
restart recovery and stopped on `ambiguous_remote_recovery_evidence`.
The task showed "Recovering saved work…" and no Retry, although no
provider work was done.
> - This pull request gives such startup rejections a named, permanent
outcome with a working Retry, and pins ask and standard runs to the
external sandbox policy.
> - The benefit is that a provider startup refusal ends as a clear
failure with Retry, never as an ownership hold.
## Linked Issues or Issue Description
**What happened?**
A Codex planning run on a Daytona sandbox failed in `session.open` with
`error creating thread: failed to load AGENTS.md instructions ... bwrap:
loopback: Failed RTM_NEWADDR: Operation not permitted`. The server
recorded `native_session_interrupted` with recovery mode
`bootstrap_retry`. Thirty seconds later, remote restart recovery blocked
the run with `native_restart_recovery_blocked` /
`ambiguous_remote_recovery_evidence`. No server restart happened. All
default-mode Codex runs on the same image succeeded that day; only the
planning run failed.
**Expected behavior**
A provider startup failure with no provider work must end as a clear
failure with a working Retry, or get a bounded automatic retry that also
ends with Retry. It must never become an ambiguous-ownership hold.
**Steps to reproduce**
1. `cd server && npx vitest run
src/services/native-runtime/native-startup-rejected-retry.integration.test.ts`.
On master the deterministic case is `retryable_failure` /
`native_session_interrupted`.
2. `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "provider
itself rejected"`. On master the new code is not produced.
**Paperclip version or commit**
master at
|
||
|
|
08ff0828d6 |
fix(native): offer Retry at once when the Runner refuses a turn start on a closed session (#20729)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native ACPX run (for example Claude) starts each turn with `turn.start`. When a start is cut off, the Runner reads the sidecar's dispatch journal (#20675). > - If the prompt never left, the Runner keeps the session reopenable, and the same-run resume sends the prompt once. If the prompt may have left, the Runner closes the session, because a resend could duplicate it. > - A closed session cannot reopen for the same run. Every same-run resume got `ACPX session is not open`. The server treated that as a generic interruption and spent three attempts (about 95 s) before the board offered Retry. > - This pull request gives that refusal a closed class in the Runner and a permanent, named failure code in the server. > - The benefit is that the board offers a working Retry on the first refusal, while every recoverable case keeps recovering by itself. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T011704-s9107`, task t05 (local, claude, workload C1, fault TM2). A freeze cut off a turn start, and the Runner saved the session `closed`. Three same-run resumes each failed with `PRP command turn.start failed: … "message":"ACPX session is not open"`. Recovery stopped with `native_session_retry_exhausted` at 01:25:14, and the board then offered Retry. The Retry run succeeded. **Expected behavior** When the Runner proves the session is closed for this run, the run stops at the first refusal with a named failure, and the board offers a Retry that works. A start that the Runner proved never reached the provider still recovers by itself. **Steps to reproduce** 1. `cd server && npx vitest run src/services/native-runtime/native-closed-session-retry.integration.test.ts` 2. On master, "offers Retry on the first refusal, and Retry starts a new run" fails: the coordinator is `retryable_failure` / `native_session_interrupted` (a same-run retry in 30 s). Refs #20675, #20720 ## What Changed - Runner (`durable/mod.rs`, `acpx_provider_backend.rs`): new closed class `provider_session_closed`. The Runner sets it on these refusals: - a turn start that saves the session `closed` (`ACPX turn start failed closed: …`), but not the rejected-checkpoint case, which stays `prepared` and reopens; - `ACPX session is not open` when the saved lifecycle is `closed` for this run; - the saved turn-end-restart refusal for this run (asserted in `acpx_turn_end_restart`). A session that waits for provider-exit proof can reopen after a restart. Its refusal gets no class. - Server (`native-session-executor.ts`): a `turn.start` failure with that class maps to the new permanent code `native_provider_session_closed`. The server reads the class from the authenticated command result, and from the command-failure text if the error was rewrapped. `nativeSessionFailureDisposition` treats the code as permanent, so the first refusal ends the run as `terminal_failure` with a board owner and a plain next action. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` states the class and the older-Runner behavior. - Tests: - `native-closed-session-retry.integration.test.ts` (new, embedded Postgres) drives `executePaperclipNativeSession` with the Runner refusal. On the first refusal, the board shows `canRetry: true`. `requestTaskRetry` is accepted, and `resumeTaskRetry` starts exactly one new run with a fresh provider session. A start proved undispatched stays `retryable_failure`. - Unit tests in `native-session-executor.test.ts`: both closed refusals, a rewrapped error, an undispatched start, an unclassified older-Runner refusal, and a class on another command. - Rust `acpx_cut_turn_start`: a start cut off before dispatch has no class. A start cut off after dispatch, and the later refusal, carry `provider_session_closed`. ## Verification - Red on master, green here: step 1 above. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts`: 1358/1358 pass. - `cargo test -p paperclip-runner-core --test acpx_park_preparation` (26 pass), `--lib acpx` (94 pass), `--test acpx_provider_turns --test acpx_provider_session --test acpx_warm_attachment` (all pass), `cargo fmt --check` clean. - `npx tsc --noEmit -p server/tsconfig.json`: clean. - Not run: a chaos wave. ## Risks - This changes the Rust Runner. Daytona needs a new runner image before remote runs get the class. Until then, an older Runner sends the refusal without the class, and the server keeps today's bounded same-run retries. That path is unchanged and ends with a working Retry. The server does not trust the text alone, because an older Runner sends the same text for a session that can still reopen. - The class is set only where the Runner's saved state proves that no later start of this run can reopen the session. A false class would cost one automatic resume and show Retry earlier. It never replays provider work. - No persisted shape changes. The class is an optional field of a failed command result. Old saved state resumes unchanged. Nothing new runs before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
fcf4e39b41 |
fix(runner): do not count stopped time against a turn start or an ACPX sidecar request (#20720)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run starts each provider turn with `turn.start`. The control plane waits for the Runner, and an ACPX Runner waits for its sidecar, which starts the provider and sends the prompt. > - Both waits use a 30 s bound. When the host or container stops the whole tree (SIGSTOP, a host sleep), the clocks keep moving while no process can act. > - On thaw both bounds have passed. The control plane reports `PRP command turn.start timed out`, and the Runner fails the sidecar request and closes the ACPX session. > - #20704 added a stopped-time clock for startup and recovery. This pull request puts `turn.start` and its follow-up waits on the same clock, and adds the same rule to the Runner's sidecar response wait. > - The benefit is that a freeze during a turn start no longer fails the turn or closes the session. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T011704-s9107`, task t05 (local, claude, workload C1, fault TM2). A follow-up run sent `turn.start` at 01:22:21. The harness then stopped the server, Runner, sidecar and Claude twice: 28 s, then 33 s after a 1 s thaw. At 01:23:25 the control plane reported `provider_transport_failed: PRP command turn.start timed out`. Three same-run resumes then failed with `ACPX session is not open`, recovery stopped with `native_session_retry_exhausted`, and the task needed a manual Retry. The Retry run succeeded. **Expected behavior** Time that the processes spend stopped does not count against the turn-start bound or the sidecar request bound. A real timeout still fires while the processes run. **Steps to reproduce** 1. `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport-stall.test.ts`. On master it fails with `provider_transport_failed: PRP command turn.start timed out`. 2. `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core --test acpx_sidecar_stopped_time`. On master it fails with `ACPX sidecar request timed out at initialize`. It stops the process early in the wait and again in its last half second. Note on the repeated `ACPX session is not open`: the wave ran `eafd5853d9`, which does not contain #20675. On master, #20675 settles a cut-off start from the sidecar's dispatch journal. If the prompt never left, the Runner reopens the session and sends it once. If the journal shows the prompt left, or cannot prove it, the session stays closed and the run ends with a working Retry, as the class rule requires. This pull request stops the freeze from cutting off the start in the first place. Refs #20704, #20675 ## What Changed - `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`: `#startTurn` runs under `#withAdmissionClock`. The turn-start deadline, the held-tool-call wait, the start-event wait, the adoption check time and the `session.snapshot` wait all read `#waitNow()`, so they share one stopped-time credit. - `packages/paperclip-runner/runner/crates/runner-core/src/acpx_sidecar_transport.rs`: `receive_response` blocks in 1 s slices. `credit_stopped_time` treats a slice overshoot beyond 2 s as a stop. It moves the deadline by the overshoot, and to at least 5 s after the thaw, so a stop in the last slice still leaves the sidecar time to answer. Total credit per request is capped at 15 min. - The control-plane clock (`RunnerStallLedger`) now uses the same rule: a stop is credited in full plus a 5 s grace, capped at 15 min per admission. - `packages/paperclip-runner/src/live/runnerd-codex-transport-stall.test.ts` (new): drives the real transport and Runner. It SIGSTOPs the Runner tree, jumps `Date.now` by 40 s mid-wait, then resumes the Runner. It fails if the turn start leaves the clock, or if the command wait stops reading it (checked by removing `clock: this.#admissionClock`). This is the transport-level test requested in the #20704 review. - `packages/paperclip-runner/runner/crates/runner-core/tests/acpx_sidecar_stopped_time.rs` (new, its own test binary): a sidecar stops the test process for 6 s against a 3 s bound and answers after the thaw. - Unit test for `credit_stopped_time`. ## Verification - Red on master, green here: both commands in "Steps to reproduce". - `npx vitest run src/live/runnerd-codex-transport.test.ts src/live/runnerd-codex-transport-stall.test.ts src/live/runnerd-acpx-source-reconciliation.test.ts src/live/runnerd-codex-transport-adoption.test.ts`: 342/342 pass. - `cargo test -p paperclip-runner-core --lib acpx_sidecar_transport` (12 pass), `--test acpx_sidecar_transport --test acpx_park_preparation --test acpx_provider_turns` (all pass), `cargo fmt --check` clean. - `npx tsc --noEmit -p tsconfig.json` and `-p tsconfig.test.json`: clean. - Not run: a new chaos wave. ## Risks - A real stall longer than 2 s in the Runner (or 5 s in the control plane) during a turn start or a sidecar request now extends that wait by the stall plus 5 s. The waited-on process could not be observed during the stall either. The 15 min cap bounds a process that keeps stalling. - The Rust wait now wakes once per second while it waits for a sidecar reply. This is an active request with no I/O per wake. - The 1 s control-plane ticker from #20704 now also runs during a turn start, and stops when the start returns. - No persisted state, launch identity or saved-session shape changes. Old saved runs resume through the same code, and an older Runner keeps its old wait. Nothing new runs before `turn.start`, and no wait is added on the normal path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
e719d42557 |
fix(runner): accept the acknowledgement of an indeterminate suspend replay (#20714)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run talks to a Runner process (runnerd). The controller in the server sends commands. The Runner journals each command before it acts and replies with a result. > - `runner.suspend` and `runner.shutdown` are terminal commands. The Runner waits for a `command_result_ack` before it exits. It compares that acknowledgement with the receipt that it journaled. > - A Runner that is killed inside `runner.suspend` journals the command but not its result. The next Runner replays the command as `indeterminate`. It sends it on the wire as `failed` with `execution_indeterminate`, and it still treats it as terminal. > - The controller acknowledged the `failed` status that it received. The Runner compared that with its journal status `indeterminate` and exited with "terminal command result acknowledgement changed its durable identity". The controller also lifted its terminal fence for a failed suspend and sent the next command to a Runner that was about to exit. > - This pull request makes both sides agree on the status of that one receipt, and keeps the fence. > - The benefit is that a freeze or restart during a suspend no longer burns the same-run retries and leaves the task at a manual Retry. ## Linked Issues or Issue Description **What happened?** A restart-chaos task (local Runner, Claude, a 28-second freeze of the server during a live turn) needed a manual Retry. Retry worked. 1. The turn completed. The server then sent `runner.suspend`. The server froze before the Runner reported the result. 2. The close found the Runner not durably suspended (`commandStatus: pending`, `runnerLifecycle: ready`). The run failed with `native_runner_checkpoint_owner_lost`, and same-run recovery stopped the old runnerd. The suspend was journaled but not finished. 3. Attempt 2 started a new runnerd on the same state. The welcome delivered the stale suspend. The Runner replayed it as `indeterminate` and sent `failed`. The controller acknowledged `failed`. The Runner exited with code 1: "terminal command result acknowledgement changed its durable identity". The controller had recorded the suspend as `failed`. 4. Attempt 3 found the fence for a suspend that the controller no longer listed as pending. The Runner reconciled it and exited with code 0. The server reported "runnerd exited unexpectedly with code 0", and `native_session_retry_exhausted` stopped automatic recovery. **Expected behavior** The controller acknowledges exactly the receipt that the Runner sent. The Runner accepts that acknowledgement, records `suspended`, and the next attempt starts from a clean suspended Runner. **Steps to reproduce** 1. Start a native run with a local Runner. Let the turn complete so that the server sends `runner.suspend`. 2. Freeze the server (SIGSTOP) for about 30 seconds before the suspend result arrives, so that recovery stops the Runner inside the suspend. 3. Let same-run recovery start a new Runner. It exits with "terminal command result acknowledgement changed its durable identity". The unit tests in this PR reproduce the same sequence without a live run. **Paperclip version or commit** `master` at |
||
|
|
3fdcc2a28c |
fix(runner): resend an ACPX turn whose start was cut off before the prompt left (#20675)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native ACPX run (for example Claude) sends `turn.start` to its Runner. The Runner asks the ACPX sidecar to start the turn, and the sidecar sends the prompt to the provider > - A start can fail without a definite answer. A host freeze can outlast the sidecar request deadline, or the sidecar or Runner can die during the start > - The Runner then saved the session `closed`. Every same-run resume failed with `ACPX session is not open`, and the task needed Retry > - The Runner cannot tell from its own state whether the provider has the prompt. Resending could duplicate it. A continuation could lose it > - This pull request adds a durable record that answers that question: a sidecar prompt-dispatch journal, written before the prompt reaches the provider > - The benefit is that a start cut off before the prompt left resends the prompt exactly once, with no Retry. Every case the record cannot prove keeps today's behavior ## Linked Issues or Issue Description **What happened?** Local restart-chaos wave `w-20261110T185315-s9107`, task t05 (claude, workload C1, fault TM2). A follow-up run reopened the parked Claude session and sent `turn.start`. The sidecar first starts Claude and loads the conversation; the prompt goes out only after that. After 24 s without a reply, the harness froze the server, Runner, sidecar and Claude (SIGSTOP) for 33 s. On thaw the Runner's 30 s sidecar deadline had passed. It reaped the sidecar and saved the session `closed`. Three same-run resumes failed with `PRP command turn.start failed: … ACPX session is not open`, recovery stopped with `native_session_retry_exhausted`, and a person had to click Retry. **Expected behavior** When a durable record proves the prompt never reached the provider, the run resends it once by itself. When the record cannot prove it, the run behaves as today. **Steps to reproduce** 1. On master, run `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core --test acpx_park_preparation cut_turn_start`. 2. `a_start_cut_off_before_the_prompt_left_reopens_and_sends_the_prompt_once` fails: the session is `closed` and the next start fails with `ACPX session is not open`. Refs #20507, #20645, #20674 ## What Changed - New file contract in `protocol/provider-schemas/acpx-sidecar.schema.json`: `turnDispatchJournalFile` (`paperclip-turn-dispatch.jsonl`) and `turnDispatchJournalRecord` (`received` / `dispatched`). The generator emits the file name and schema into TypeScript and Rust, so the pack contract digest covers them. - Sidecar (`turn-dispatch-journal.ts`, `codex-runtime-adapter.ts`, `runtime-host.ts`, `acpx-runtime-sidecar.ts`): when the runtime port starts a turn, before any provider work, it truncates the journal and writes `received`. ACPX calls `onAcpMessage("outbound")` synchronously before it writes a message to the provider. On the turn's first `session/prompt` request the port appends `dispatched` and fsyncs it there, so the record exists before the bytes leave. A failed dispatch write aborts the prompt write. A failed `received` write removes the file and runs the turn unjournaled. - Runner (`acpx_turn_dispatch.rs`, `acpx_provider_backend.rs`): the journal is read only after the provider lifetime fence proves that the sidecar and provider are gone. A lone complete `received` record for the turn proves the prompt never left. - A start that failed in flight saves the session `prepared` with no active turn and returns `ACPX turn start failed before its prompt reached the provider`. The run's next `turn.start` reopens the conversation and sends the prompt once. Until the proof settles, the saved state stays `turn_starting`, so a Runner killed during the proof goes through the restore check below. - A Runner killed while the session was `turn_starting` restores the session idle (`session.resumed`, no `turn.failed`). The same-run resume then starts the turn once instead of sending a continuation that would lose the prompt. - A `dispatched` record, another turn's record, a torn or malformed file, or no file keeps today's behavior (closed session, or lost-turn recovery). - Fake sidecar: writes the same journal and adds `hang-before-dispatch` / `hang-after-dispatch` faults. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` ("ACPX turn starts cut off before the prompt left"). The design note is `doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md` (in #20674). ## Verification - Red on master, green here: `cargo test -p paperclip-runner-core --test acpx_park_preparation cut_turn_start` (4 tests: cut before dispatch reopens and sends once; cut after dispatch stays closed and never resends; Runner killed before dispatch restores idle; Runner killed after dispatch or with no journal reports the lost turn). The two "before" cases fail on master. - `cargo test -p paperclip-runner-core`: all pass. `cargo fmt --check` clean. - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/drivers/acpx/turn-dispatch-journal.test.ts src/drivers/acpx/codex-runtime-adapter.test.ts`: 121 passed. One test uses a real ACPX runtime and an ACP peer that refuses a prompt arriving before its `dispatched` record; it fails if the hook is removed. - `node scripts/generate-acpx-sidecar-contract.mjs --check`, `node --test scripts/contracts/contract-fingerprints.test.mjs`, and `tsc -p tsconfig.test.json` pass. - Time to first token, measured with the same open/write/fsync sequence on this Mac (APFS, load average ~11, 300 samples): `received` p50 0.24 ms, p95 0.59 ms; `dispatched` (with fsync) p50 4.7 ms, p95 11.5 ms. Against a Claude TTFT budget of 9.5–16.5 s p50 this is under 0.1%. No wait, round trip or file transfer is added. Recovery work (fence, journal read) runs only after a failed start. - Live restart-chaos wave and runner image: see the comments on this pull request. ## Risks - A false "not dispatched" proof would duplicate a prompt. It needs a lone, complete `received` record for this exact turn ID, read after the fence proves the sidecar is gone. `dispatched` is fsynced before the prompt write, and a failed write aborts the send. A failed `received` write removes the file, so it cannot leave a lone record. - Upgrade safety: saved provider state gains no field (its decoder denies unknown fields), and only lifecycles an older Runner already handles are written. An older Runner ignores the journal file. State saved by an older Runner has no journal, which reads as unknown and keeps today's behavior. A new Runner with an older sidecar never sees a `received` record, so no proof forms. - The changed sidecar schema changes the pack contract digest, so Daytona needs a new runner image. Older image packs are not reused. - A failed start now waits for the provider lifetime fence (at most 5 s, usually immediate) before it settles. This is only on a failure path. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file editing) in Claude Code. Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
76c787fffc |
fix(runner): report monitor notes length limits (#20604)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents use task monitors to resume a future check. > - The monitor schema limits notes to 500 characters. > - The field description did not state the limit. Rejection showed only the schema path. > - This pull request adds the limit to the description and to safe length errors. > - Agents can correct notes before they retry a rejected call. ## Linked Issues or Issue Description Related prior fix: #16868. **What happened?** A native `set_task_monitor` call with 501 characters in `monitor.notes` failed with a path that ended in `notes/maxLength`. The error did not state 500. The schema already had `maxLength: 500`, but the field description did not state it. **Expected behavior** State the limit in the field description and in the rejection. Accept 500 characters. Reject 501 characters before dispatch. Preserve a saved monitor when a replacement is invalid. **Steps to reproduce** 1. Supply a future `monitor.nextCheckAt` and 501 characters in `monitor.notes`. 2. Read the native provider error. 3. Before this fix, it shows the constraint path without the maximum. **Paperclip version or commit** Reproduced on `7368a0b15c2f02107778a961144d86e2eb59a952` in a Linux source checkout, with Node 25.9.0 and Rust 1.97.1. ## What Changed - Use one local value for the published notes limit and its description. The server test checks agreement with the persisted shared validator. Keep the standalone Runner package boundary. - Read the numeric maximum from the authorized JSON Schema validation error. Keep submitted values private and preserve the 512-byte feedback bound. - Test the published schema and real bridge dispatch path at 500 and 501 characters. Test saved contents and rejection without monitor changes with real PostgreSQL. - Refresh generated contracts and document the limit. - Sync two SDK proof lock identities after the reviewed master lock refresh in #20615. This fixes the remaining CI proof mismatch. Keep compiler, parser, Zod, and all rejection checks unchanged. ## Verification - Final head: `bac60234ab3a8fc7043a90c7f9c81b02e250428b`. - Red: the server regression failed because the field description omitted `500`. The Rust regression failed because the error omitted `maximum 500 characters`. - Local passes after the final base update: 44 PostgreSQL monitor/scheduler tests, 14 real-source Rust bridge tests, 11 cache/Docker guard tests, and 56 SDK proof tests. Runner TypeScript build and generated contract checks pass. - The focused Rust check compiles the actual source in a small Cargo workspace. The full Rust CI suites also pass on Linux and Windows. - Final-head [PR CI run](https://github.com/paperclipai/paperclip/actions/runs/42881633978) passes. This includes typecheck, build, Runner support, Runner Rust and Vitest, server tests, serialized tests, and all eight e2e shards. - Final-head [Windows Rust run](https://github.com/paperclipai/paperclip/actions/runs/42881633419) passes. - Greptile: 5/5 on the final head, with no open findings. There are 61 successful checks and five conditional skips. The separate Paperclip Review remains queued; its comment says work is paused. No CI or test check is pending or failed. - `pnpm verify` returned exit 2 and selected broader CI. The full local suite was not run. Final-head CI supplies that coverage. Earlier failed attempts were repaired and are not counted as passes. ## Risks - Low risk. The existing 500-character bound, scheduling semantics, and permissions stay in place. - Length feedback uses only a schema path and numeric bound. It does not print submitted text. - The Runner binary and tool contracts must reach an instance for the new guidance to apply. This PR does not deploy or merge. ## Model Used OpenAI Codex, GPT-6, with reasoning, tool use, and code execution. The runtime does not expose a more specific model ID or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
d2832a5a0b |
fix(runner): keep redacted display events inside their schema bounds (#20607)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs execute through Paperclip Runner. Its daemon, runnerd,
stores provider events durably before the server receives them
> - Before storing rich display events, such as tool output and provider
notices, runnerd redacts credential-shaped values. It then checks the
result against the event's protocol schema
> - Redaction can make a string longer. A field already at its schema
limit then goes over the limit, runnerd rejects the event, and the
daemon exits
> - The event is replayed after each restart, so every replacement
runner exits the same way. The run loops until something outside it
stops the run
> - This pull request trims a field that redaction pushed past its
schema limit, in both places that redact rich display events, so the
event is accepted and the runner keeps working
> - The benefit is that long tool output that contains a secret no
longer crashes the runner
## Linked Issues or Issue Description
**What happened?**
An agent run read GitHub Actions output and, after a context compaction,
entered a loop. About every 14 seconds runnerd started, restored its
durable PRP session, and exited with `paperclip-runnerd: redacted
display activity exceeds its schema bounds`. This continued for 15
minutes (62 launches) until a server restart ended the run. Every
reattach after that failed the same way, so the task was blocked.
**Expected behavior**
Redaction never makes a valid display event invalid. runnerd stores the
redacted event and continues.
**Steps to reproduce**
1. Produce a tool output longer than 64 KiB that contains a
credential-shaped pair, for example a GitHub Actions log line
`GITHUB_TOKEN: ***`. The ACPX sidecar keeps its last 64 KiB, which is
exactly the `toolExecution.output` limit of 65,536 characters.
2. runnerd redacts the pair to `GITHUB_TOKEN: [REDACTED]`, which is
longer, so `output` now has more than 65,536 characters.
3. `preserve_bounded_display_content` returns `redacted display activity
exceeds its schema bounds`, and runnerd exits.
A probe test with 64 KiB of output that contains `GITHUB_TOKEN: ***`,
`token=abc`, `Authorization: Bearer x`, or `password: hunter2` gets
exactly this error on `master`.
**Paperclip version or commit**
`master` at
|
||
|
|
17136a6c25 | fix(runtime): persist managed Runner credential refreshes across execution targets (#20580) | ||
|
|
16c30ac32b |
fix(runner): relaunch a provider whose startup only timed out without a reply (#20521)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute through `paperclip-runnerd`. After a restart, a new runner relaunches the provider (for example the Codex app server). A durable startup fence stops a second runner from launching a provider while an unadmitted one may still run. > - #19784 lets a new runner retire a fence whose recorded provider process and group have provably exited. That rule covers a launch that a kill cut off. It deliberately keeps a failed initialization closed, because that can be a real provider failure. > - A relaunch whose reply never came within the runner's 30 s request deadline is saved as the same failed initialization. A stalled or frozen host causes this. The restart-chaos fault TM2 SIGSTOPs the runner and Codex, and their monotonic clocks keep running while they are stopped. > - Every later runner then refused to restore with "provider startup ownership remains unadmitted", and the user had to click Retry. The owner's rule is that a run interrupted by a restart or sleep must resume by itself. > - This pull request records whether initialization failed only because the provider gave no reply, and lets a later runner retire that fence once the provider is provably gone. > - The benefit is that a stall during a provider relaunch continues automatically, while a real provider rejection still fails as before. ## Linked Issues or Issue Description Refs #19784 **What happened?** Restart-chaos wave `w-20261110T091152-s426`, task t03 (Codex, B5), run `c0bbce18-836b-43af-8195-92ebe3e21cca`. A TM3 restart relaunched the Codex provider. A TM2 freeze stopped the tree 1 s later for 45 s, and a TM3 kill followed the thaw. The next three runners each failed `runner.drain` with "provider startup ownership remains unadmitted". Recovery stopped with `native_session_retry_exhausted`, and the task needed Retry. **What did you expect?** The new runner relaunches the provider and the run continues. **Steps to reproduce** The new test `restart_relaunches_a_provider_whose_relaunch_timed_out_without_a_reply` (runner-core `tests/codex_provider.rs`). The fake Codex stalls its first `thread/resume` reply past the 30 s deadline. On `origin/master`, the saved attempt is `initialization_failed` / `thread_open`, and the next runner fails with "provider startup ownership remains unadmitted". ## What Changed Design: - The deadline that passes is the runner's own 30 s request deadline. When a startup request (`initialize`, `thread/start`/`thread/resume`, `thread/read`) reaches that deadline with the provider still running and silent, the runner sets `noReply: true` on the durable `initialization_failed` startup record. - Not `noReply`: a provider error response, a provider that exited before it replied, or closed stdout. - `retire_orphaned_startup_attempt` retires an `initialization_failed` fence only when `noReply` is set and the recorded process and its process group are both gone (`ESRCH`). This is the same proof as `spawned` in #19784. A rejection, an exit, a live process or group, and spawn or admission failures stay fenced. - Validation accepts `noReply` only on `initialization_failed` at the `initialize`, `thread_open` or `thread_read` stage. - Bounded: a runner restores once (`restore_checked` / `restore_error`), so the loop is one relaunch per recovery attempt. Recovery's existing resume budget (`native_session_retry_exhausted`, seen after three attempts in the wave) stops a provider that never replies. - Upgrade: `noReply` is `#[serde(default, skip_serializing_if = "Not::not")]`. A new runner reads state saved by master as `noReply: false`, so that state keeps today's behaviour and stays fenced. - Rollback: an old runner (`deny_unknown_fields`) rejects state that holds a `noReply` fence and fails closed. That run needs Retry, which is today's outcome for the same fence. The field is written only while a timed-out fence is present, and admission clears it, so all other state is unchanged for old runners. - Normal path: nothing is added before `turn.start`. The cost is one in-memory flag per request; no extra I/O. Files: - `runner-core/src/codex_provider.rs`: `last_request_got_no_reply` and the `no_reply` field on `ProviderStartupObservation::Failed`. - `runner-core/src/provider_backend.rs`: the `noReply` field, its validation, and the retirement rule, with unit tests: - a no-reply fence is retired at `initialize`, `thread_open` and `thread_read`; - it is kept while the provider may still run; - a rejection or a state saved without the field stays fenced. - `runner-core/src/bin/fake-codex-app-server.rs`: the test-only switch `--stall-first-thread-resume-ms`. - `runner-core/tests/codex_provider.rs`: the end-to-end red/green test. - `tests/restart-chaos/README.md` and `time-faults.ts`: correct the TM2 claim. A stopped process's monotonic clock keeps running, so TM2 is also a long process stall. ## Verification - `cargo test -p paperclip-runner-core --test codex_provider restart_relaunches_a_provider_whose_relaunch_timed_out` is red on `origin/master` (the next runner refuses restore) and green here, in about 31 s. - `cargo test -p paperclip-runner-core`: 1080 passed, 0 failed. This includes `failed_autonomous_restore_and_rollover_keep_their_exact_startup_origin`, where a real "no rollout found" rejection still never restarts. - `cargo fmt -p paperclip-runner-core -- --check` is clean. - A restart-chaos proof wave on seed 426 will follow in a comment. ## Risks - A provider that hangs for good is relaunched once per recovery attempt until the existing resume budget stops it. Before, it failed on the first attempt. This costs at most a few extra 30 s attempts, and they are bounded. - Rollback to an older runner while a `noReply` fence exists fails closed for that one run, which needs Retry, as it does today. - No database, server or API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, cargo and test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
a09093a5c5 |
fix(native): resume a run whose turn.start a restart cut off before Codex created the turn (#20507)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native Codex run sends `turn.start` to its runner, and the runner sends `turn/start` to Codex. The runner marks the start "ambiguous" before it sends the request > - A control-plane restart can kill the runner and Codex while that request is in flight. After the restart, neither journal says whether Codex created the turn, and the resumed thread has no active turn > - The restored runner then failed closed on every same-run resume with "cannot safely recover an ambiguous Codex turn start without an active replacement turn". The answer never reached the agent > - The next run on the task was refused at the runner-state fence (`prior_owner_terminal_state_indeterminate; retirement_refused=journal_commands_unsettled(turn.start.failed)`) before it asked for a runner. Retry after that run then waited forever on `process_identity_missing` for a process that never existed > - This pull request lets the runner prove from Codex's own thread history that the start never created a turn, and then resume and start the turn once. It also lets Retry continue after a run that the runner-state fence refused before launch > - The benefit is that a turn start cut off by a restart continues by itself when Codex never saw it, and a person's Retry is no longer a dead end ## Linked Issues or Issue Description **What happened?** Local restart-chaos wave `w-20261110T033426-s108`. Command: `--env local --adapters codex --workloads Q1,Q2 --faults K1 --triggers answered --tasks 4 --concurrency 2 --seed 108`. Task t03 (Q2): the answer run `92a57f50` resumed after the first restart. A second K1 restart landed 0.25 s after `session.resumed`, while its `turn.start` was in flight. Each same-run resume then failed with `PRP command session.snapshot failed: cannot safely recover an ambiguous Codex turn start without an active replacement turn`. When the prerequisite finished, the next run `0e6aa9d1` failed before launch with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=journal_commands_unsettled(turn.start.failed)`. The accepted Retry (202) never created a run: the task stayed on `retryWait.reason=process_identity_missing` until the wave timed out (`R3:retry_no_new_run`). **Expected behavior** If Codex never created the turn, the run resumes and starts it once by itself. If that cannot be proven, a person's Retry starts a new run instead of waiting forever. **Steps to reproduce** 1. Run the wave command above on master. 2. Or run `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-codex-transport-cut-turn-start.test.ts` on master (case "before" fails with the wave error). ## What Changed - Runner (`codex_provider.rs`, `provider_backend.rs`): when the restored provider state has an ambiguous turn start, no active turn before the loss, and the resumed thread has no active turn, the runner reads the newest turn of the thread (`thread/turns/list`, limit 1, metadata only, the same request `read_thread` already sends). If there is no turn, or the newest turn is one Paperclip already settled, Codex never created the turn. The runner clears the ambiguity, emits a `harness.diagnostic` with code `ambiguous_turn_start_not_created`, and restores an idle session. The old `turn.start` stays `execution_indeterminate` and is never executed again. The same-run resume then starts the run's turn once, through the existing "recovered session without an active turn" path. - Any other history (a turn Paperclip never settled, an in-progress turn, an incomplete page, a non-Codex provider) keeps today's fail-closed error. The probe runs only on this recovery path. The normal turn path does no new work before `turn.start`. - Server (`native-local-process-stop.ts`): `nativeRunNeverLaunched` also accepts a failed native run whose error is the runner-state fence refusal `runner_state_identity_mismatch: prior_owner_*` (or `legacy_prior_owner_*`). The executor throws it only while it migrates another run's saved state, before it requests a launch. The existing evidence check still applies: any launch request, identity record, `adapter.invoke` or runner event keeps the run held. Other runner-state errors (for example `authority_indeterminate`) stay held. - Fake Codex (`fake-codex-app-server.rs`): keeps the thread's turn history in a `<state-file>.turns` sidecar, so `thread/turns/list` returns history like real Codex (newest first, honours `limit`). New `--hang-turn-start <file>` holds `turn/start` "before" or "after" the thread records the turn, to model a kill in flight. - Tests: - New real-runnerd test `runnerd-codex-transport-cut-turn-start.test.ts`. A real runner and fake Codex start a turn; both processes are killed while `turn/start` is in flight. "before": the resumed run completes, the journal holds the old `turn.start` (`execution_indeterminate`) and one new completed `turn.start` with the same task text, and Codex saw exactly one more `turn/start`. "after": recovery still fails closed and no new `turn/start` is sent. - Rust: `ambiguous_turn_start_that_never_created_a_turn_recovers_and_starts_once` and `ambiguous_turn_start_that_created_an_unsettled_turn_still_fails_closed`. Both rewrite saved provider state the way a master runner wrote it, so they also cover state saved before this change. - `explicit-native-continuation.test.ts`: five new rows for the runner-state fence. Admitted: refused before launch (with and without a coordinator). Held on `process_identity_missing`: refused after a launch request, refused after a runner event, and a run that failed its own `authority_indeterminate` check. - `docs/durable-recovery.md`: describes both behaviours. ## Verification - Red/green, real runnerd: on master (the staged master runnerd), `runnerd-codex-transport-cut-turn-start.test.ts` "before" fails with `PRP command runner.drain failed: ... cannot safely recover an ambiguous Codex turn start without an active replacement turn`, the wave error. With this change, both cases pass. - Red/green, Rust: without the `provider_backend.rs` change, `cargo test -p paperclip-runner-core --test codex_provider ambiguous_turn_start_that` fails the "never created" test. With it, both pass. The full `cargo test -p paperclip-runner-core` passes. - Red/green, server: without the `native-local-process-stop.ts` change, the two "refused at the prior-owner runner-state fence" rows fail with `process_identity_missing`. With it, all 14 rows pass. - Related suites pass: 11 Runner transport files, 419 tests (`runnerd-codex-transport`, `-attachment-recovery`, `-cold-restore`, `-warm-attachment`, `-release-resume`, `-cut-attach`, `-cut-turn-start`, `-settlement`, `-suspension`, `runnerd-restart-continuation`, `runnerd-saved-continuation`); server `explicit-native-continuation`, `quarantined-task-retry` and `issue-confirm-run-stopped-routes`, 3 files, 470 tests; full `cargo test -p paperclip-runner-core`. - `pnpm --filter @paperclipai/paperclip-runner exec tsc --noEmit` and server `tsc --noEmit` pass. - Pack `contractDigest` is unchanged: `sha256:3164357d…` before and after (`scripts/contracts/contract-fingerprints.mjs`). - Live proof: the wave command above on this branch. Results follow in a comment. **No duplicate provider work.** The runner starts a new turn only when Codex's own history proves the cut-off start created nothing. The old command is never executed again (the runner refuses an `indeterminate` command). A recorded turn that Paperclip never settled keeps the run failed closed. ## Risks - Low-medium risk. The runner change runs only while it restores a session whose saved state has an ambiguous turn start and no active turn. Before this change, that run could never continue. - The proof relies on Codex recording a turn in the thread before it runs anything. If Codex accepted the request but died before it wrote the turn, the runner treats the start as not created and starts the turn once. Codex writes the user turn to the thread when it starts the turn, before it asks the model for output, so no model output or tool call is expected without a recorded turn. - If the thread's newest turn is not in the runner's settled set (for example after a provider identity epoch rollover forgot older identities), the runner keeps failing closed, as today. - **Open decision (not changed here):** when Codex did record the cut-off turn, it may have done work. This pull request keeps the run failed closed and does not replay or adopt that turn. Continuing is a person's decision through Retry or a new message. - The server change trusts the executor's own error text `runner_state_identity_mismatch: prior_owner_*`, together with the absence of every launch and runner event. A run that requested a launch keeps every stop proof. - **Runner release:** the pack `contractDigest` does not change. Local runners use the runnerd built with the app, so the fix applies with this release. - Not covered: the related `authority_indeterminate` variant in the same wave (t04, Q1) is a separate cause; its Retry already worked in that wave. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #20477, #20482 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
914e3ae14f |
fix(runner): settle ACPX turns when the cancel reply is lost (#20498)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Claude agents run through the native Runner and the ACPX sidecar > - A saved question or a board Stop cancels the active turn with `turn.cancel` > - The Runner waited 30 s for the sidecar's reply to that cancel, and the Runner serves controller commands one at a time > - When that reply was lost, the controller's shutdown commands timed out first, so the run failed and the task needed a human Retry > - This pull request bounds that wait, keeps the transport open, and lets the provider's own turn terminal settle the turn > - The benefit is that a lost cancel reply on Claude settles the same way it already does on Codex (#20336) ## Linked Issues or Issue Description Refs #20336 (Codex equivalent: settle a saved handoff after verified shutdown). Related, no overlap: #20477, #20482, #20483. **What happened?** The restart-chaos harness dropped the ACPX sidecar's reply to the Runner's `turn.cancel` (fault PV2, target `interrupt_ack`). A Claude task that saved a question (Q1) failed with "The waiting state is saved; confirming shutdown of the previous execution is still pending". Its next attempt failed `native_session_cleanup_quarantined`. A Claude task stopped by the board (S1) ended `blocked`. Retry was accepted but started no run. Cause, in `packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs` (`interrupt_turn`, master line ~800): the cancel used the ordinary sidecar request with the 30 s bound from `acpx_provider_backend.rs:539`. While it waited: - `turn.stop` and `runner.suspend` stayed queued. The controller's 15 s governed drain plus its 10 s close grace ran out first (`runner did not durably suspend before checkpoint`, suspend command `pending`). - At 30 s, `fail_closed` killed the sidecar and dropped the provider's buffered turn terminal. The poll path then recorded `turn.failed` (`acpx_provider_transport_failed`) and `provider_exit_unconfirmed` (`acpx_provider_backend.rs:2716-2738`). `turn.stop` returned `already_settled`, and `runner.suspend` refused with `ACPX provider lifetime cleanup is not yet proven` (`acpx_provider_backend.rs:2613-2621`). Codex passes the same fault because its Runner does not block on the interrupt reply. **Expected behavior** A lost cancel reply must not fail the run. The saved question must settle as `succeeded`. A board Stop must end the run as cancelled with a proven suspension, so the next turn can start. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters claude --workloads Q1,S1 --faults PV2 --provider-fault-message interrupt_ack --concurrency 1 --tasks 4` Failing wave on master: `w-20261110T060120-s207` (t04, t05). **Deployment mode** Native Runner with the Claude ACPX profile, local. Hosted sandboxes run the same Runner code. ## What Changed - `AcpxSidecarTransport::request_turn_cancel`: sends `turn.cancel` and waits only to a caller deadline. On a missing reply it keeps the transport usable and retains the request ID, so a late reply is skipped in both the response and the event-poll paths. A rejected reply and transport errors keep their existing fail-closed result. - `AcpxProviderSession::interrupt_turn`: waits at most 5 s (or the shorter configured request bound) for the cancel reply. On a missing reply it retires the runtime from warm reuse and returns. The provider's turn terminal still arrives through polling, and `turn.stop` still reaps a provider that never stops. - `AcpxProviderSession::deliver_tool_result`: a board Stop can land while a semantic tool runs. The session's own `turn.cancel` then retires the call in the sidecar before its result arrives, and the turn terminal may still be buffered. The sidecar rejects the result as stale (`ACPX_TOOL_CALL_STALE`). When this session sent `turn.cancel` for that turn, the Runner now keeps the durable result, like a result for an already closed turn. Every other rejection still fails closed. (`AcpxSidecarTransport::request_tool_resolve` reports whether a rejection was the stale-call case.) - A repeated interruption while a cancel is unanswered sends nothing and returns at once (review finding). - `fake-acpx-sidecar`: new `--cancel-reply lost|late` flag. After that cancel, `tool.resolve` is rejected as stale. The provider ends the turn, and the reply is dropped or written just before the next request. - Three regression tests (below) and a short section in `packages/paperclip-runner/docs/durable-recovery.md`. No TypeScript, server, schema or protocol change. The normal start path is unchanged; nothing waits before turn start. ## Verification Red/green (red runs used the master versions of `acpx_provider_session.rs` and `acpx_sidecar_transport.rs`): - `cargo test -p paperclip-runner-core --test acpx_warm_attachment a_lost_cancellation_reply_still_settles_the_turn_and_suspends_the_runner` - Red: after 30.4 s, `ACPX turn interrupt failed: ACPX sidecar request timed out at turn.cancel` (the live error). - Green: `turn.interrupt` returns in about 5 s, then `turn.interrupted` and a `cancelled` `run.terminal` (no `turn.failed`), `turn.stop`, and `runner.suspend` completes. - `cargo test -p paperclip-runner-core --test acpx_provider_turns lost_cancellation_reply_keeps_the_turn_terminal_and_the_transport` - Red: `interrupt_turn` fails at the request bound. - Green, for a lost reply and for a late reply: the terminal is polled, warm reuse is refused, and the next request (`session.close`) succeeds after the late reply arrives. A late reply causes no second settle and no second prompt. - `cargo test -p paperclip-runner-core --test acpx_provider_turns a_tool_result_for_a_call_retired_by_our_cancellation_does_not_fail_the_transport` - Red (guard disabled): `ACPX sidecar command tool.resolve was rejected (retryable=false, classification=provider_tool_call_retired)`, the same error as live wave task t03 below. - Green: the result is kept, the pending tool clears, and the turn terminal is polled after it. - `cargo test -p paperclip-runner-core` (whole crate): all pass. Existing `turns-tool-resolution-rejected` coverage still fails closed when no cancel was sent. `cargo fmt --check` passes. - Pack contract digest: unchanged (`sha256:3164357d…` before and after, from `scripts/contracts/contract-fingerprints.mjs`). - Live wave 1, `w-20261110T072706-s211` (commit `d91bff1e9f`, the first fix only), `--adapters claude --workloads Q1,S1 --faults PV2 --provider-fault-message interrupt_ack --tasks 4`: 3 of 4 pass. Both Q1 tasks pass (the asking run `succeeded`, no failed run, answer in the brief). One S1 passes. S1 task t03 failed: Stop landed during a `write_task_document` call and hit the stale tool-result error above. The second commit fixes that case. - Live wave 2, `w-20261110T082702-s212` on head `76dd57515d`, same flags: every task passes R1–R6 and U1. Both Q1 tasks pass with the fault applied (asking run `succeeded`, no failed run). S1 t01 passes with the fault applied (Stop took effect, task `done`, no Retry). S1 t03 fails only `provider_fault_applied` (`applied=0`): the agent finished before the Stop window, so nothing was injected and the task proves nothing either way. - Live wave 3, `w-20261110T091057-s213` on head `76dd57515d`, same flags: the same result. t01 (S1), t02 and t04 (Q1) pass with the fault applied; t01's Stop took effect (`status=cancelled`), task `done`, no Retry. t03 again fails only `provider_fault_applied` (`applied=0`, the run finished before the Stop window); all its R1–R6 and U checks pass. - Across waves 2 and 3 on the final head: 6 of 6 tasks with the fault applied pass (4 Q1, 2 S1). No product rule failed. The two misses are harness timing (no injection), not product failures. ## Risks - A healthy cancel reply that takes longer than 5 s now retires the warm runtime. The next turn starts a fresh provider process for that session. The observed healthy reply time is about 1 s. - If the provider never stops and the reply is lost, the turn stays active until `turn.stop` terminates the provider and proves the lifetime fence. If that proof fails, the existing settlement-pending and quarantine states are unchanged; this PR adds no success path without proof. - Runner binary change: hosted Daytona images need a normal image rebuild to pick up the fix. The pack `contractDigest` does not change, so existing image packs stay reusable. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edit, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
3e572475c8 |
fix(native): re-attach an answer run whose run.attach a restart cut off (#20477)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a person answers an agent's saved question, Paperclip starts an answer run. For a per-turn session, that run starts a new runner and sends `run.attach` to bind the saved conversation to the run > - The runner journals each command before its effect. A restart that kills it inside that window leaves `run.attach` with an unknown outcome. The restored runner settles it as `execution_indeterminate` and never executes it again > - The controller's same-run resume waited on that old command, so every resume failed with "PRP command run.attach failed". The answer never reached the agent. A Retry or a stall followed > - A kill during the attachment's provider launch can also leave a startup fence that no later runner retires, so every later command fails with "provider startup ownership remains unadmitted" > - This pull request attaches again with a new command when nothing after the cut-off attachment started provider work. It also lets the runner retire that orphaned startup fence > - The benefit is that an answer given just before a restart continues on its own ## Linked Issues or Issue Description **What happened?** Local restart-chaos journey wave `w-20261110T054406-s108` on master `2c2186706e`. Command: `--adapters codex --workloads Q1,Q2 --faults K1 --triggers answered --tasks 4 --concurrency 2 --seed 108`. Result: 1 of 4 passed. - t02 (Q1, K1 2.7 s after the answer). Answer run `0b357a1f` resumed after the restart. Each resume failed with `native_session_recovery_failed: PRP command run.attach failed: execution_indeterminate`. Then "Automatic recovery stopped", and the task needed a Retry. - t01 (Q2, K1 1.1 s after the answer). Answer run `82c60af9` failed the same way on three same-run resumes. Its provider state kept `lifecycle: "prepared"` with a `spawned` startup attempt for `run.attach`, and that provider process was gone. Each later drain and suspend failed with "provider startup ownership remains unadmitted". The successor run then refused with `retirement_refused=journal_commands_unsettled(run.attach.failed)`. The accepted Retry created no run (`retry_no_new_run`). The earlier wave `w-20261110T033426-s108` hit the same window at different points. It showed `authority_indeterminate` (Q1) and an ambiguous `turn.start` stall (Q2). See Risks for what this pull request does not cover. **Expected behavior** An answer run that a restart interrupts before it starts provider work resumes by itself. It does not need a Retry. **Steps to reproduce** 1. Run the wave command above on master. 2. Or run `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-codex-transport-cut-attach.test.ts` on master. Both cases fail with the error from the wave. ## What Changed - `runnerd-codex-transport.ts`: the same-run resume waits on the recovered `run.attach` as before. If that wait fails, the new `cutOffRunAttachmentPayload` checks the journal. It queues one new `run.attach` with the same payload, and waits on it, only when all of these are true: - the latest `run.attach` has an unknown outcome (`indeterminate`, or `failed` with `execution_indeterminate`); - it is a cold attachment (no `paperclipNextAuthority`), with no warm transition and no staged attachment; - every later command is recovery maintenance, as the control plane's shared `isRunnerRecoveryMaintenanceCommand` defines it (`session.snapshot`, `session.goal.get`, `runner.drain`, `runner.suspend`, `runner.shutdown`, `turn.stop`); - no committed event shows provider work (`turn.accepted`, `turn.started`, `item.started`, `tool.execution.started`, `semantic_tool.input`, `runtime_request.created` or `mcp_app.tool_input`). Any other failure is thrown unchanged. - `provider_backend.rs` (Runner): `retire_orphaned_startup_attempt` also treats a `prepared` state with a saved thread and a nonzero process generation as restorable. `attach_run` already documents this state as the settled checkpoint that `run.attach` restores. The other fence rules do not change. The fence is retired only before the spawn, or when the recorded process and its group have exited. - Tests: - New real-runnerd test `runnerd-codex-transport-cut-attach.test.ts`. A real runner and the fake Codex server run an asking run, then a cold answer-run attachment. The test kills both processes and rewinds both journals to "`run.attach` journaled, outcome not recorded", then resumes. One case has a startup fence and one does not. - A unit test for `cutOffRunAttachmentPayload`. It covers each maintenance command, and the cases where it must not re-attach (a definite answer, a pending command, a warm or staged attachment, any work command, a turn event). - The Rust orphaned-startup test also covers `prepared`. A new test shows that a `prepared` state with no saved thread keeps its fence. - `docs/durable-recovery.md`: describes both behaviors. ## Verification - Red/green, real runnerd: - On master, `runnerd-codex-transport-cut-attach.test.ts` fails both cases with `PRP command run.attach failed: ... execution_indeterminate`, the error from the wave. - With only the transport change, the case without a fence passes. The fence case fails with `provider startup ownership remains unadmitted`, the t01 error. - With both changes, both cases pass. - Red/green, Rust: `cargo test -p paperclip-runner-core --lib orphaned` fails `restart_retires_a_spawned_startup_whose_provider_process_exited` for `prepared` without the runner change. With it, all 10 tests pass. - Related transport suites, 8 files and 387 tests, pass: `runnerd-codex-transport`, `-attachment-recovery`, `-cold-restore`, `-warm-attachment`, `-release-resume`, `-cut-attach`, `runnerd-restart-continuation` and `runnerd-saved-continuation`. `tsc --noEmit` for the Runner package passes. - Live proof: the same wave command on this branch. Results follow in a comment. **No duplicate provider work.** The new attachment is sent only when the journal proves that no provider work followed the cut-off one. No command after it can start or steer work: only the shared recovery-maintenance commands are allowed, and none of them starts a turn. No turn, item, tool-input or runtime-request event is committed. `run.attach` itself starts no turn. In the Codex and ACPX runners, `attach_run` refuses to attach unless the session is settled. That means no active or ambiguous turn, no pending tool call and no live active goal. The old command is never executed again, because the runner refuses an `indeterminate` command. The new attachment is a new command with a new ID. ## Risks - Low-medium risk. The new path runs only after a same-run resume has already failed on an unknown-outcome attachment. Before this change, that run could not continue. - If an attachment's partial effect had changed provider state, the runner's own settled-session check refuses the new attachment. The run then fails as it does today. - **Runner release:** the Runner change does not change the pack `contractDigest`. It is `sha256:3164357d…` before and after (`scripts/contracts/contract-fingerprints.mjs`). Local runners use the runnerd built with the app, so the fix takes effect with this release. A remote image pack (for example Daytona) that was built before this change keeps the old fence rule until a new image or pack is built and qualified. No release or pin bump is part of this pull request. - **Mixed versions during an upgrade:** - New server with an old runner: the server re-attaches. A run without a startup fence continues. With a fence, the old runner refuses the new attachment, the same failure as today. - Old server with a new runner: the runner retires the fence, but the old server still fails on the unknown-outcome attachment, as today. - Neither mix adds a new failure. Saved state from older builds needs no migration. - Not covered here, and still open: - A `turn.start` with an unknown outcome where the resumed thread has no active turn (wave `w-20261110T033426-s108` t03). The runner deliberately fails closed for that case. - A warm-runner attachment that may already have been delivered (`authority_indeterminate`). - A Retry that waits on `process_identity_missing` for a run that failed before any process started. A separate pull request will cover it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #20146, #19784, #20148 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
549ca46dae |
fix(runner): retain Claude GitHub broker settings through the Rust sidecar (#20463)
## Thinking Path > - Paperclip lets people manage AI agents for work. > - Shared agents use the instructing person's GitHub identity for managed commands. > - Native Claude starts through the Rust daemon and a TypeScript ACPX sidecar. > - The Rust launch filter kept the command launchers but removed their broker transport. > - Shell commands then reported `GitHub capability_missing`, although connector tools worked. > - This pull request retains that transport for managed Claude launches. > - Each command can acquire its current identity through the existing broker checks. ## Linked Issues or Issue Description **What happened?** Native Claude could use GitHub connector tools but could not authenticate `git` or `gh`. The Rust daemon removed the broker URL and session/bridge capabilities before starting the ACPX sidecar. Reconnecting the account did not repair this process environment. **Expected behavior** Managed shell commands use the current authorized GitHub identity. The provider receives broker capabilities, while raw GitHub tokens remain confined to individual commands. **Steps to reproduce** 1. Start native Claude through the Rust Runner with managed GitHub launchers. 2. Run `gh api user` and a private repository ref lookup through Bash. 3. Observe `capability_missing` and no request to the broker. **Paperclip version or commit** Reproduced from source at `2c2186706ef4bb352b662f9d8faebefe47fb52ac`. The repair is based on `a5e01634f69f18dfe10a0bf709df02d9e6a2135b`. **Deployment mode** Native Claude through ACPX. Local acceptance used macOS arm64 and Node 25.6.1. The reported failure also affected a Linux sandbox. Related: #18212 preserved launcher paths. #18850 repaired the TypeScript filter. This change repairs the preceding Rust filter. The duplicate search found no open repair for this boundary. ROADMAP.md already lists managed personal GitHub identities. ## What Changed - Retain the broker URL and both broker capabilities when Claude has a nonempty managed launcher directory. - Keep raw GitHub tokens, unrelated secrets, empty launcher configurations, and sibling ACPX providers outside that exception. - Exercise the regression through the public sidecar initialize protocol in the existing Rust integration suite. Document the restart requirement. The Rust sidecar transport owns this filter. Credential resolution, company access, approval, and run identity policies keep their existing owners. No schema or API changes are needed. ## Verification - Red/green: the subprocess regression failed on the missing `PAPERCLIP_GITHUB_BROKER_URL` before the fix, then passed with all three broker settings and the negative controls. - Passed before moving the regression: 615 Rust core tests, with 3 existing ignored tests. After moving it into the integration suite, all 20 sidecar integration tests and the 9 remaining sidecar unit controls pass. The moved regression was also run against the unfixed source and failed on the missing broker URL. - Passed: 85 TypeScript environment, launcher, sandbox, and configured-environment controls. These include operation identity changes, revocation, recovery, and exclusion of capabilities from persisted environment data. - Passed: Runner Rust typecheck and format check, plus the Runner build through `node scripts/agent-preflight.mjs --skip-rust-debug`. - Live before/after: real `claude-opus-5-5` through the actual Rust daemon and built ACPX sidecar reproduced the failure with the old binary. Neither shell command reached the broker. With the fixed binary, `gh` identified the connected account and `git` read private repository refs. A second turn retained a workspace file and authenticated again. After the controlled broker denied access, the next command failed without reusing the prior token. No GitHub writes were made. - Live release cutover: saved a session with the unfixed daemon, suspended it, and restored it with the repaired daemon on `aa9f5b30614c011ebc3fe4509e11173b766c1fea`. The same session and workspace marker survived. `gh` authenticated after restoration, then failed when the controlled broker revoked access. - Live acceptance used an Anthropic API credential and a loopback broker with an authorized operator GitHub credential. This isolates the repaired transport; it does not qualify the hosted subscription credential or production grant resolver. - The first PR review gate did not recognize the in-source Rust test. The regression now lives in the existing integration test file, and the same test-coverage gate passes locally. - `pnpm verify` returned its documented exit 2 and required broad CI coverage. The final head passed the full CI typecheck, build, and test lanes. - Greptile reviewed `64cca69b76073cc3185b28334465dee48fbadb29` at 5/5 with no actionable findings or open review threads. Its check passed on that head. The same head is mergeable with no conflicts. - Final head `64cca69b76073cc3185b28334465dee48fbadb29` passed [full PR CI](https://github.com/paperclipai/paperclip/actions/runs/42761401396), [Windows Rust](https://github.com/paperclipai/paperclip/actions/runs/42761401117), and [isolated Runner compilation](https://github.com/paperclipai/paperclip/actions/runs/42761400973). Typecheck, build, all test shards, Runner checks, browser checks, and aggregate verification passed. No checks are pending or failed; five conditional jobs were skipped. ## Risks - The Claude sidecar now receives the intended broker capabilities. The existing broker must continue to enforce active-run identity and current connection access. Raw GitHub tokens remain excluded from the provider launch environment. - Previous-release checkpoint and profile formats carry forward unchanged. A running provider must retire and restart normally to inherit the repaired environment. No profile reset or data migration is required. - Hosted recovery requires delivery of the updated Runner binary. This PR does not deploy or retry existing tasks. Live acceptance has not exercised Linux sandbox confinement or a GitHub write. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with xhigh reasoning. Used repository inspection, code editing, and shell execution. The context window size is not exposed. Anthropic `claude-opus-5-5` was exercised as the acceptance-test provider. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
ff4ba7d319 |
perf(runner): run launch artifacts in place from a trusted read-only root (#20428)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Rust Runner (`paperclip-runnerd`) starts provider sidecars and proxies from qualified launch artifacts: a Node runtime and the scripts it runs > - Since #12699, Runner copies each artifact into a sealed memfd and checks its SHA-256 on every verification. This defends against a path swap or an in-place rewrite between the check and the exec > - On a hosted agent computer (4 vCPU), the copy and digest of the 127 MB Node takes about 0.26–0.30 s warm and about 1.2 s right after a wake. This is on the path from `session.open` to the sidecar spawn, and again on every warm `run.attach`. Each Runner also holds about 125 MiB of shmem for the copy > - On that computer, the artifacts live on the Runner ROM: a read-only erofs image that the deploy pins by digest. The threat that the memfd defends against does not exist there. On local hosts the artifacts are writable, so the memfd must stay > - This pull request adds an explicit, path-scoped, default-off `--trusted-launch-root <dir>`. Runner runs an artifact in place only when its open descriptor proves that it is on an immutable mount under that root. Everything else keeps the sealed, hashed snapshot > - The benefit is a shorter `session.open` and `run.attach` on agent computers, and about 125 MiB less shmem per Runner, with no change where the flag is absent ## Linked Issues or Issue Description **What existing behavior does this improve?** Launch-artifact verification in `paperclip-runnerd` (`qualified_launch.rs`). Its callers are the ACPX `verified_transport` (every `session_config`, including the park rebind) and the OpenCode and Cursor Cloud proxy launches. **Subsystem affected** Rust Runner (`packages/paperclip-runner/runner/crates/runner-core`). **Current behavior** Every verification opens the artifact, copies all bytes into a sealed memfd, and hashes them. Node is 127 MB. The cost is described in the Thinking Path. **Proposed behavior** When the host passes `--trusted-launch-root /opt/paperclip/runner` and the root passes its checks, Runner holds each admitted artifact open in place and executes it through `/proc/self/fd/N`. It does not copy or re-hash it. Without the flag, nothing changes. **Reason and benefit** See the Thinking Path. Refs #12699 (introduced the sealed memfd). Refs #20423 (overlap, see below). ## What Changed - `qualified_launch.rs`: new `TrustedLaunchRoot` and `set_trusted_launch_root` (process-global, set once at startup, like `--task-memory-cgroup`). - **Root checks at startup.** The path must be absolute, already canonical (no symlink or `..`), a directory, and on a read-only, exec-permitted erofs or squashfs mount. If a check fails, runnerd logs a warning and keeps today's behavior. - **Per-artifact checks, on the open descriptor.** After today's checks (regular file, not a symlink, not group- or world-writable, open-time identity), the descriptor must be a regular file on the root's device. Its kernel path (`readlink /proc/self/fd/N`) must lie under the root by path components, so `/opt/paperclip/runner2` does not match. Its mount must pass `fstatvfs` (`ST_RDONLY` set, `ST_NOEXEC` clear) and `fstatfs` (erofs or squashfs magic). If any check fails, Runner logs the reason and uses the sealed snapshot. - **Admitted artifacts** become `VerifiedProcessArtifact::in_place_on_trusted_root` (crate-private). The held file goes through the existing `inherited_artifact` path, so the exec is `/proc/self/fd/N` and `PAPERCLIP_VERIFIED_RUNTIME_EXECUTABLE` still names a descriptor that nested launches can reopen. - New `verify_assigned_executable_launch_artifact` never uses the root. `stdio_mcp.rs` now calls it, so host-assigned stdio MCP binaries always keep the sealed snapshot. - `paperclip-runnerd.rs`: parse `--trusted-launch-root <dir>`. Older runnerd builds ignore unknown flags, so a host can pass it before every Runner understands it. - The trusted path is Linux-only. On macOS and Windows, `TrustedLaunchRoot::open` always refuses. - `tests/runner-e2e/SECURITY.md`: documents how the Kimi and Pi provider-death fixtures behave under a trusted root. ### Decision: the trusted path drops the digest The trusted path does not hash the artifact, not even once per process: - The ROM image is pinned by digest at deploy time. - On the agent computer, the Launcher computes the expected `--acpx-sidecar-command-sha256` and `--acpx-sidecar-script-sha256` from the same ROM files immediately before it starts Runner. A re-hash of an immutable file can only repeat that result. - One hash per process would not help `session.open`, because each task starts a new Runner process. - The mount check is the guard: erofs and squashfs have no write path, so the bytes cannot change while the descriptor is open. ### Fixtures that assert the sealed memfd - Untrusted-mode assertions are unchanged. The fixtures do not pass the flag. - Kimi (`kimi-provider-fault.py` `checked_parent_descriptor`): with a trusted root, the parent descriptor is the ROM's Node file (one link). The existing linked-descriptor branch handles it: same inode as `/proc/<pid>/exe`, and the complete bytes hashed against `nodeSha256`. That branch never accepts the file as a memfd. The existing test `test_linked_descriptor_keeps_regular_single_link_checks_without_memfd_claim` covers it. - Pi provider-death admission binds `/proc/self/fd/3|7` to the native distribution snapshot's Node. The TypeScript ACPX closure copies that file whether or not a trusted root is set, so the flag does not affect it. ### Overlap with #20423 #20423 (draft) moves the same copy and hash to Runner boot and reuses the result. Where this flag is set (Cloud agent computers), there is nothing left to precompute, so #20423 is mostly redundant there. #20423 still helps hosts without a trusted root (local and sandbox Runners). This PR does not change #20423. ## Verification All runs are on Linux (aarch64, Rust 1.97.1, in a container): - `cargo fmt --manifest-path runner/Cargo.toml --all -- --check` - `cargo check --manifest-path runner/Cargo.toml --locked --workspace` - `cargo test -p paperclip-runner-core --lib -- qualified_launch trusted_root commonjs`: 12 passed. The new tests are: - `an_artifact_under_an_immutable_trusted_root_runs_in_place_without_a_digest`: trusted-root hit, with no memfd, the same inode as the source, and no re-hash. - `an_artifact_outside_the_trusted_root_keeps_the_sealed_snapshot`: a path outside the root is sealed, and a wrong digest still fails. - `a_sibling_with_the_root_as_a_name_prefix_is_outside_the_root`: `rom2/` does not match `rom`. - `a_root_that_is_not_read_only_falls_back_to_the_sealed_snapshot`: the real probe refuses a writable root at startup. A per-artifact mount failure falls back, and a wrong digest still fails. - `the_real_mount_probe_rejects_a_writable_filesystem` - `symlink_and_parent_escapes_from_the_root_fall_back`: a directory symlink out of the root, a `..` path out of the root, and a final-component symlink (still refused). - `a_root_must_be_absolute_canonical_and_a_directory` - `without_a_trusted_root_launch_artifacts_are_sealed_and_hashed`: flag absent, behavior unchanged. - `host_assigned_executables_ignore_the_trusted_root` - `trusted_root_artifacts_execute_the_held_file_and_reopen_it_for_nested_launches`: Runner execs from the held file, and a child re-execs `$PAPERCLIP_VERIFIED_RUNTIME_EXECUTABLE`. Both report the source file as `/proc/self/exe`. - `cargo test -p paperclip-runner-core --test stdio_mcp --test process_supervisor`: 4 and 8 passed. - Ad hoc, not committed: a privileged container with a real loop-mounted squashfs. The real `fstatvfs`/`fstatfs` probe admitted a `ro` mount and refused a `ro,noexec` mount. The unit tests replace only that mount probe, because CI cannot mount filesystems. - Not verified: a run on an agent computer with the real erofs ROM, and the latency and memory savings there. The Cloud Launcher change that passes the flag is a separate follow-up. ## Risks - **This change needs review from a security owner before merge.** It removes the sealed copy and the per-launch digest for admitted artifacts. That relies on the host's assertion that the root is a digest-pinned immutable image, plus the descriptor checks above. - **Misconfiguration fails closed.** A wrong, writable, `noexec`, non-canonical or missing root only loses the speedup, and runnerd logs a warning. There is no global "trust everything" switch, and Runner never selects a root from the filesystem it finds. - **Residual trust.** On an admitted file, Runner no longer detects a ROM image that differs from what the launch profile expected. On the agent computer, the Launcher derives the expected digests from the same ROM. Another host that passes the flag must pin its image the same way. An attacker with root in the guest (who could write the backing block device or remount) is out of scope; such an attacker can already control runnerd. - **Allowlist.** Only erofs and squashfs are accepted, because they cannot be written through any mount. A read-only bind mount of a writable filesystem is refused. A ROM that changes to another filesystem type will silently fall back to the sealed path (slower, still safe). - No migration. Behavior is unchanged when the flag is absent. Older runnerd builds ignore the flag. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, container builds). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5858cef8f9 |
perf(runner): poll the provider right after an ACK releases it (#20374)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner (`paperclip-runnerd`) sends provider events to the controller over one authenticated socket > - Its main loop reads one control frame, then polls the provider only after a read finds no frame > - After the controller ACKs the last sent event, the next read waits the full 250 ms `RUNTIME_READ_TIMEOUT` before the Runner polls the provider again > - A Codex turn start needs one or two such rounds before the Runner reads `turn/started`, so every Codex turn waits 250 ms or 500 ms for no reason > - This pull request reads only the frames that are already queued after a consumed control frame, and then polls at once, when this connection produced the unacknowledged output > - The benefit is 250 ms less time to `turn/started` for each ACK round, and faster event delivery after each ACK ## Linked Issues or Issue Description This is an enhancement. The fields follow `.github/ISSUE_TEMPLATE/enhancement.yml`. **What existing behavior does this improve?** Codex turns wait for the Runner read timeout before the Runner sends `turn/started`. **Subsystem affected** The Runner (`paperclip-runnerd`) durable loop in `packages/paperclip-runner/runner/crates/runner-core/src/durable/`, for the local Codex provider. **Current behavior** On a hosted test deployment (48 h, 114 Codex turns), `runner.transport.started_event` for Codex has two modes: | Codex turn | n | `started_event` p50 | | --- | --- | --- | | no provider cold start in the trace | 57 | 468 ms (all < 700 ms) | | provider cold start in the trace | 57 | 918 ms (51 of 57 ≥ 700 ms) | For Claude (ACPX) turns, the p50 is 90 ms. ACPX returns `turn.started` with the command result. Codex reads `turn/started` from the provider in a later poll. **Proposed behavior** The Runner polls the provider as soon as the controller ACK releases the poll gate. The time to `turn/started` then follows the controller round trip, not the read timeout. **Reason and benefit** - Real Codex (`codex app-server` 0.157, local) sends `turn/started` 2–16 ms after its `turn/start` response. The provider is not the cause. - `runner.transport.started_event` starts when the `turn.start` command result arrives. It ends when the controller commits `turn.started`. In this time the Runner sends `turn.accepted`, waits for its ACK, waits 250 ms, polls, and sends `turn.started`. - On the first turn after a provider launch, the first poll returns only the launch diagnostic that the executor retained. A second ACK round and a second 250 ms wait come before the poll that reads `turn/started`. This is the difference between the two modes above (about 450 ms: 250 ms wait and one event round). - So 250 ms (warm turn) or 500 ms (first turn) of each Codex `started_event` is the read timeout. The rest is one or two controller event rounds (about 200 ms each on that deployment), less than 20 ms of Codex time, and the 10 ms pump interval of the TypeScript transport. - The expected benefit is 250 ms less per warm Codex turn start and 500 ms less for the first turn after a provider launch, plus faster Codex event delivery after each ACK. **Breaking changes** None. No protocol, schema, or controller change. ## What Changed - `AuthenticatedTransport::receive_queued_json` reads with a 1 ms timeout and then restores `RUNTIME_READ_TIMEOUT`. - New `CommandExecutor::provider_poll_is_live` hook (default `false`). The local Codex executor returns `true` only when it owns a running provider with no open startup attempt, so a poll cannot restore or launch a provider. The native executor forwards it for the local Codex facade only. - In the steady-state loop, after a consumed control frame, the Runner uses `receive_queued_json` when it has sent output that this connection produced (a command or a provider poll) and the executor reports a live provider. If no frame is queued, it polls the provider at once. - When a provider poll returns no output, or when no frame was consumed, the Runner uses the 250 ms read as before. A 1 ms read follows only a consumed frame, so an idle loop does not spin. - The replay boundary is the highest source sequence before the welcome's pending commands run. A fresh command delivered in the welcome counts as this connection's output. Replayed commands only resend their stored results. - Replayed output from an earlier connection or process, ACPX, and the other providers keep the 250 ms read. The controller can still send its next command before a poll restores a provider. (Without the live-provider gate, six ACPX source-recovery tests in `runnerd-acpx-source-reconciliation.test.ts` failed: the fast poll restored the ACPX sidecar before the next recovery command.) - Control frames that are already queued behind the ACK are still read and executed before the provider poll. The ACK gate in `poll_executor_events_after_controller_ack` is unchanged. ## Verification - `cargo test -p paperclip-runner-core --lib durable::` (121 passed). Five new tests in `src/durable/tests/ack_release_poll.rs`, included by the transport test module like `hosted_retention.rs`: - `acknowledged_output_polls_the_provider_without_waiting_out_the_read_timeout`: the provider poll follows the ACK read in less than `RUNTIME_READ_TIMEOUT`. Without the change it fails (the poll waited 280–356 ms). - `control_queued_behind_an_ack_runs_before_the_next_provider_poll`: a command sent in the same write as the ACK executes before the provider poll. - `welcome_command_output_polls_the_provider_without_waiting_out_the_read_timeout`: the output of a fresh command in the welcome also releases the poll at once. With the replay boundary taken after the welcome commands, it fails. - `replayed_output_ack_keeps_the_read_timeout_before_the_provider_poll`: after a reconnect, a command sent 50 ms after the ACK of replayed output executes before the provider poll. Without the replay condition it fails. - `restorable_provider_ack_keeps_the_read_timeout_before_the_provider_poll`: when the executor does not report a live provider, a command sent 50 ms after the ACK executes before the provider poll. - `npx vitest run src/live/runnerd-` in `packages/paperclip-runner` with the new debug `paperclip-runnerd`: run with `--no-file-parallelism`. Before the welcome-boundary fix: 576 passed, 3 skipped. After it: 575 passed, 3 skipped, and 1 failure once (`stops a lost runnerd's provider and escaped tool job from its lineage journal (journal=true)`, a SIGKILL timing test) while `cargo test` ran at the same time. `runnerd-codex-transport.test.ts` then passed 293 of 293 three times with this build, and three times with the base build. - Local before and after measurement with the real Runner, the TypeScript transport, and the fake Codex app-server (8 turns, 2 runs each, debug build): | span | before p50 | after p50 | | --- | --- | --- | | `runner.transport.started_event` | 377 ms (first turn 682–695 ms) | 119–122 ms (first turn 117–119 ms) | | `runner.transport.turn_start` | 462–471 ms (first turn 775–780 ms) | 204–212 ms (first turn 247–251 ms) | - The saving on the hosted deployment is not measured. From the analysis above, the expected saving is about 250 ms for a warm Codex turn and about 500 ms for the first turn after a provider launch. ## Risks - Only local Codex gets the fast path. ACPX streaming keeps the old 250 ms wait after each ACK. A later change can opt ACPX in when its poll can prove that it does not restore the sidecar. - More provider polls during Codex streaming: a poll can now follow each ACK at once instead of after 250 ms. Batches can become smaller, so the provider state save for each batch can happen more often. Each event already has its own durable save. - A command that the controller sends more than 1 ms after its ACK can now arrive after a provider poll. This was already possible after 250 ms. The change applies only when this connection produced the output, so the provider is already active. - If the Runner cannot restore the 250 ms read timeout, it treats the socket as failed and reconnects. It does not continue with a 1 ms timeout. - No protocol, schema, or controller change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, code search, Honeycomb queries) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
79bf733012 |
fix(runner): share the bounded ACPX cleanup proof (#20332)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner keeps a lifetime fence until the original ACPX provider stops. > - A recovered Runner must prove that fence is free before it clears saved cleanup state. > - Suspension and other recovery paths already wait for transient listener holders. Recovered shutdown used one immediate probe. > - That difference made parallel Rust tests fail after they dropped their original listeners. > - This pull request uses the existing bounded proof for recovered shutdown and removes the repeated test retry. > - The benefit is one cleanup rule, with the same fail-closed proof and no new work on normal task dispatch. ## Linked Issues or Issue Description Fixes: #20210 Thanks to @genius-0963 for the test-only fix proposed in #20321. That related alternative adds a test-only shutdown retry. This change applies the existing bounded cleanup proof in the shared shutdown path and removes the same retry pattern from native-provider integration tests. It also tests a single shutdown call while the original quorum is still held. ## What Changed - Use `await_provider_lifetime_fence` for a recovered executor with no live session and unconfirmed provider exit. Delete the unused one-shot helper. - Keep the acquired quorum held through the existing durable state write. A failed proof leaves `providerExitUnconfirmed` set. - Give the shared timeout message wording that applies to shutdown and suspension. - Extend the recovery test with a worker-start signal. Hold the original listeners until the single shutdown call is waiting, then release them and assert successful cleanup without a provider session. - Use the same bounded proof in the retired-quorum fixture test. Remove the duplicate retry loop from the native-provider test helper. - Document the bounded recovery cleanup wait. ## Verification - macOS arm64, Rust `1.97.1`, source base `3a2f71db31`, head `368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b`, isolated worktree. Only targeted checks ran locally. - `cargo fmt --manifest-path packages/paperclip-runner/runner/Cargo.toml --all -- --check`: passed. - `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml --locked -p paperclip-runner-core --lib acpx_provider_backend::tests`: 45 passed on the rebased head (5.21 seconds). The initial head also passed five more parallel-test runs (45 tests per run), with no failure. - `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml --locked -p paperclip-runner-core --test native_provider_backend acpx_`: 14 passed on the rebased head (9.43 seconds). These tests use the real Rust executor and controlled sidecars. - Negative control: restore only shutdown's original one-shot proof. The new regression fails at `shutdown must wait while the original lifetime still holds a quorum`. Restore the fix and the affected module passes. - Independent parent verification on current commit `368fc814f7`: the three exact recovery, held-quorum, and retired-quorum tests passed (3 tests, 5.06 seconds). Independent source review confirmed the unchanged three-file scope and found no actionable issue. - The active-turn recovery test still holds a live quorum through the five-second bound. It checks that shutdown refuses cleanup, saved state remains unconfirmed, and the provider never starts. It then releases the quorum and confirms cleanup. - Standard Greptile scored current commit `368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b` 5/5 with no review threads. - Initial GitHub general-server shard 7 failed in unchanged `remote-workspace-nested-verification.test.ts`, at its 15-second wait for a child-process import checkpoint (expected one row, got zero; both Vitest attempts). [Failed job](https://github.com/paperclipai/paperclip/actions/runs/42668667033/job/129339371815). That fixture runs Node, PostgreSQL, Git, and storage migration; it does not run the native Runner or the changed Rust cleanup. The cause of the checkpoint timing miss remains unknown. The focused retry passed on the unchanged source ([successful retry](https://github.com/paperclipai/paperclip/actions/runs/42668667033/job/129347569863)). This proves recovery, not the cause of the original timing miss. GitHub kept the reusable-workflow aggregate failed after that job-only retry and rejected both a final-gate retry and a failed-job retry. A subsequent run-level retry materialized only the gate and aggregate jobs; its aggregate inputs reported prior jobs as `abandoned`, so it failed without a new test failure. A new `full-ci` label event requests a fresh workflow graph on the same commit. The reason for those rerun-state results is not established. - The fresh `full-ci` run [42674033208](https://github.com/paperclipai/paperclip/actions/runs/42674033208) then lost several runners mid-job. The maintained [runner-loss recovery 42677096525](https://github.com/paperclipai/paperclip/actions/runs/42677096525) classified all four failed jobs as runner loss: typecheck, E2E shards 2 and 8, and server shard 4. The evidence was a shutdown log or lost-communication annotation. It retried the failed jobs together; canceled sibling jobs were also requeued. This is runner-interruption evidence, separate from the earlier checkpoint assertion. - The maintained recovery attempt again failed the same partial-checkpoint assertion in unchanged `remote-workspace-nested-verification.test.ts`, twice ([job 129361525237](https://github.com/paperclipai/paperclip/actions/runs/42674033208/job/129361525237)). Neither checkpoint-failure job logged runner shutdown or lost communication; their resource summaries reported no significant constraint. The child phase at the timeout remains unknown. No further blind retry or TypeScript change is included here. Related startup-handshake work exists in #20339; its controlled-delay evidence establishes a deadline-boundary defect but does not establish the phase of these original CI failures. - Rebased onto current master after upstream #20339 merged. That separate change fixes a demonstrated child-startup/checkpoint-deadline boundary. It does not prove which child phase caused the earlier zero-checkpoint assertions. This PR still changes only its original Rust and documentation files. Fresh GitHub CI on the new base will establish recovery, not retrospective cause. - Full local tests and a local release build were not run. Current-head [PR CI 42681719411](https://github.com/paperclipai/paperclip/actions/runs/42681719411), [Runner Windows 42681719803](https://github.com/paperclipai/paperclip/actions/runs/42681719803), and [Docker Runner 42681719601](https://github.com/paperclipai/paperclip/actions/runs/42681719601) all passed on `368fc814f7ccf88ac4bf68ca9eb7f1a6c294307b`. The full check list has 59 successes and 5 intentional skips. GitHub supplied broad typecheck, build, and optimized Rust coverage. This is new-base recovery; the cause of the original checkpoint assertion failures remains unproved. ## Risks - A genuinely unconfirmed original provider can now hold recovered shutdown for up to the existing five-second proof deadline. The proof returns immediately when two candidate ports are free. - The changed branch requires both no in-memory provider session and unconfirmed saved cleanup. Live-session shutdown and ordinary dispatch keep their existing paths. Outer controller close deadlines remain authoritative. - State saved by the previous release carries forward unchanged. No schema, launch identity, profile digest, protocol, or retry-budget change. - No live provider session or hosted deployment was tested. The tests cover cleanup proof, refusal, durable state, and controlled provider execution. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, repository inspection, code editing, and targeted test execution. The exact serving revision and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
97188f276c |
perf(runner): skip the turn-end provider restart in a per-turn Runner (#20312)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner runs each agent's provider (for Claude, an ACPX
sidecar) and keeps its session across runs.
> - #20190 restarts the ACPX provider when a turn settles, so a warm
follow-up does not pay the respawn on its critical path.
> - A per-turn Runner (`lifecycleMode: per_turn`, the default) exits
with its run. Nothing reuses the generation that restart prepares: the
next run starts a new Runner process and loads the session again.
> - In that mode the restart can only spawn a provider for teardown. On
a loaded host it can also fail, which was the cause of the closed
sessions that #20285 fixed.
> - This pull request skips the turn-end restart when the controller
launched the Runner as per-turn.
> - The benefit is that per-turn runs never spend a provider spawn they
cannot use, and never risk a restart failure at the end of their run.
## Linked Issues or Issue Description
Refs #20190, #20285
**What happened?**
A per-turn Runner restarted its ACPX provider after its turn settled,
although the Runner exits with its run. The next run launches a new
Runner process (`runner.process.launch mode=resume`, then
`provider.session.load`), so the restarted generation is torn down
unused. In an earlier A/B on a host with load average about 17–27, this
restart fired and failed, and the failure closed the session (fixed in
#20285).
**Steps to reproduce**
1. Run a Claude task with the default per-turn lifecycle.
2. Let the turn settle and the controller acknowledge its events before
it shuts the Runner down (more likely on a loaded host).
3. The Runner reaps the provider, spawns a new generation, and then
exits with the run, which reaps the new generation too.
**Expected behavior**
A per-turn Runner does not restart its provider after its turn. A warm
Runner keeps the #20190 restart. A later turn of the same run still
restarts the provider at its start.
## What Changed
- `DurableRunnerConfig` has a new `turn_scoped` field.
- `paperclip-runnerd` sets it from `--lifecycle-mode`, which the
controller already passes (`durable-prp-control-plane.ts` pushes
`--lifecycle-mode <mode>`), but which `runnerd` ignored until now. Only
`per_turn` sets it. `warm`, and launches without the flag (older
controllers), keep today's behavior.
- `NativeProviderCommandExecutor::prepare_next_turn` returns early when
`turn_scoped` is set. The lazy restart at the next turn start is
unchanged.
- Tests:
- New
`a_per_turn_runner_skips_the_turn_end_restart_and_restarts_a_later_turn_at_its_start`:
no new provider process and no sidecar command after the turn settles; a
later turn of the same run still restarts at its start
(`session.suspend`, `initialize`). It fails without the skip (2 process
starts instead of 1).
- New `only_a_per_turn_lifecycle_is_turn_scoped` for the flag parsing.
- The other `DurableRunnerConfig` literals set `turn_scoped: false`.
## Verification
- `cargo test -p paperclip-runner-core`:
- `--lib`: 602 passed.
- `--test acpx_park_preparation`: 22 passed.
- `acpx_park_readiness`: 8 passed.
- `acpx_warm_attachment`: 4 passed.
- `acpx_provider_suspend`: 5 passed.
- `acpx_provider_session`: 12 passed.
- `acpx_provider_turns`: 29 passed.
- `native_provider_backend`: 47 passed.
- `durable_recovery`: 3 passed.
- `--bin paperclip-runnerd turn_scoped`: 1 passed.
- `cargo build --workspace --all-targets` passes.
- Local live A/B with `ttft-baseline --adapters claude --samples 3
--restarts 0 --phases coldwarm`. Control is master after #20285
(
|
||
|
|
fd2e7482c0 |
fix(runner): stop a lost runnerd's provider and tools from a durable lineage journal (#20270)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A local native run uses one runner daemon (`paperclip-runnerd`).
runnerd starts the provider (for example OpenCode), and the provider
starts tool processes
> - When runnerd dies in the middle of a tool call, the provider and its
tools are re-parented. The only record of that process tree was in
runnerd's memory
> - Close then cannot prove that those processes stopped. It quarantines
the session, and the next attempt stops with
`native_session_cleanup_quarantined`: "Automatic recovery stopped /
operator recovery required"
> - Restarts and runner loss are product failures. Interrupted work must
recover by itself, and it must never get two writers in one workspace
> - This pull request makes runnerd journal its process lineage to disk,
and makes close stop and prove gone every journalled process after
runnerd is lost
> - The benefit is that a killed runnerd leads to an automatic, safe
retry within the normal attempt budget, instead of an operator stop
## Linked Issues or Issue Description
Refs #20259 (runner loss classification and the lineage sampler panic).
Refs #20260 (automatic continuation of an inert cleanup quarantine).
**What happened?**
A chat eval killed only the owned local `runnerd` process (SIGKILL)
during an OpenCode tool call
(`chat-features.runner-opencode.local.recovery.retry-exhausted`,
`recovery.explain-stall` and `recovery.source-action`). Attempt 1
recorded `native_runner_process_exited` with "Retry scheduled 1/3".
Attempt 2 stopped with `native_session_cleanup_quarantined`. The task
was blocked with no owner, and the next action was operator recovery.
**Expected behavior**
After runnerd is lost, the product stops the provider and tool processes
that runnerd started, proves them gone, and continues the run within its
retry budget. When it cannot prove that, it keeps the fail-closed
quarantine.
**Steps to reproduce**
1. Start a local native OpenCode (or Codex) run.
2. Let the agent run a tool that leaves the provider's session (for
example `nohup job &`).
3. Send SIGKILL to the `runnerd` PID only.
4. Close rejects with `runner did not durably suspend before
checkpoint`, and the next attempt fails with
`native_session_cleanup_quarantined`.
**Paperclip version or commit**
master at
|
||
|
|
e015c2ef21 |
fix(runner): keep a Claude session resumable when its turn-end provider restart fails (#20285)
## Thinking Path - #20145 prepares the next Claude provider generation at park, and #20190 restarts the ACPX provider when a turn settles. They move the sidecar respawn off the follow-up's critical path. - When that turn-end restart fails, the runner "fails closed": `restart_session_after_turn` saves the session `closed` and drops it. Within the run, the next turn fails exactly like a failed restart at turn start. That's intended. - Across runs it breaks the conversation: - the next run's `run.attach` refuses a `closed` session ("run.attach requires the same settled ACPX provider profile and session"); - it reconnects for about 10 s; - the controller then records a continuity break and starts a fresh Claude conversation (`run.startup.reuse provider_session: fresh/created`). - A local A/B (`ttft-baseline --adapters claude --samples 3 --phases coldwarm`, on a host with load average about 17–27) confirmed it: | | |
||
|
|
3374ca54f5 |
fix(runner): record a runner exit mid-turn as runner loss, and stop the lineage sampler panicking (#20259)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runner (`runnerd`) runs each provider turn, and the server records why a run stopped and what the person can do next > - When `runnerd` exits during a provider turn, the server saved the run as `native_provider_terminal_failed`: "The provider session is permanently unusable", with automatic retries stopped > - The real cause was the runner process, and the transport already names it (`native_runner_process_exited`) at the start of the closed-stream message > - Separately, the process-lineage sampler added in #19727 panicked on Linux debug builds when a provider tool started its own session, and that panic text became the run's visible failure > - This pull request records a runner exit mid-turn as runner loss, and reads session IDs without the assertion that panicked > - The benefit is that a person sees the true cause of a stopped run, and recovery treats a lost runner like every other runner exit ## Linked Issues or Issue Description Refs #19727 **What happened** A chat eval killed only the owned `runnerd` process (SIGKILL) while a provider tool was running. The run failed with `errorCode: native_provider_terminal_failed` and the next action "The provider session is permanently unusable". The error text was `Notification transport failed closed: native_runner_process_exited: runnerd exited unexpectedly: paperclip-runnerd panic: panicked at rustix-1.1.4/src/pid.rs:61:9: assertion failed: raw > 0`. The panic came from the lineage sampler thread before the kill. It called `rustix::process::getsid`, which asserts a nonzero session. A kernel thread, or a process whose session leader is outside the PID namespace, reports session 0. **Expected behavior** A runner process exit is recorded as `native_runner_process_exited`, the same cause the server already uses for a runner exit outside a turn. Recovery then applies its normal runner-loss rules (checkpoint evidence, bounded attempts, cleanup quarantine). The lineage sampler does not panic, and its diagnostics do not replace the real failure text. **Steps to reproduce** 1. Start a native OpenCode run on Linux with a debug `runnerd`. 2. Let the agent run a shell tool that starts its own session (for example a long `node -e` command). 3. Send SIGKILL to the `runnerd` PID only. 4. Read `GET /api/heartbeat-runs/<runId>`: `errorCode` is `native_provider_terminal_failed` and the error names the rustix panic. ## What Changed - `nativeSessionFailureSourceCode` returns `native_runner_process_exited` for a `NativeProviderTerminalFailure` with code `notification_transport_failed` when its message starts with `Notification transport failed closed: native_runner_process_exited: `. Only the runner transport writes that prefix. Other closed-stream causes stay `native_provider_terminal_failed`. - `paperclip-process-table` has a new `session(pid)` function. Linux reads the session field of `/proc/<pid>/stat`. macOS calls `getsid`. A session of 0 or an error gives `None`. - `process_lineage` reads sessions through `paperclip_process_table::session` instead of rustix `getsid`. - Regression tests: the runner-loss classification (attempts 1 and 3, and three negative shapes), the stat session parser with a kernel-thread line, and a live read of this process's session. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "runner process exit that closed|preserves stable provider and runner|model rejection from a provider terminal|operator-owned through terminal"` (8 passed) - `cd packages/paperclip-runner/runner && cargo test -p paperclip-process-table` (macOS: 3 passed; the Linux-only parser tests run in CI) - `cd packages/paperclip-runner/runner && cargo test -p paperclip-runner-core --lib process_lineage` (6 passed) - End to end: the `recovery.*` chat eval cells that kill `runnerd` are run against this change on a separate eval branch. ## Risks - Behavior shift: a runner exit during a turn now follows runner-loss recovery instead of stopping as a permanent provider failure. Recovery still decides from durable evidence. Ambiguous state stays terminal, and an unverified cleanup stays quarantined. - Previous-release story: no persisted shape changes. Runs saved earlier with `native_provider_terminal_failed` keep that code. Only new failures are classified with the new rule. - The session read on Linux now uses `/proc` instead of a syscall. It gives the same value, and it reports 0 (now `None`) in the same cases. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
465da4227a |
fix(runner): drop a tool target path that breaks a line so the controller can acknowledge its event (#20238)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner turns provider tool calls into durable PRP
events. The controller acknowledges each event only after it validates
and commits it.
> - The controller's event schema forbids line terminators in a safe
path such as `tool.execution` `target`.
> - The ACPX sidecar and the Rust runner accepted a location path that
contained a newline. Claude sometimes emits a malformed Write call that
puts the file content into the path.
> - The controller refused that event without an acknowledgement. The
runner redelivered the same bytes, and the run stopped making progress
for 14 minutes (2,848 reconnects).
> - This pull request makes both producers drop such a path, so the
event carries a null target and the controller accepts it.
> - The benefit is that a malformed provider path can no longer wedge a
run's event delivery.
## Linked Issues or Issue Description
**What happened?**
In restart-chaos wave seed 9807 (claude, R1, local), the run stopped
receiving events after runner source seq 33. The runner reconnected
2,848 times in 14 minutes. Offline replay of the saved runner outbox
through the controller's `validatePrpEvent` showed that seq 34 fails
`/payload/target` (the safePath pattern), because the target is
`packages/shared/src/chaos/slug-2c5c3c8f.ts\n<parameter
name="content">export function chaosSlug…`. Seq 35, 39 and 40 (the same
tool call) fail the same way.
**Expected behavior**
The runner never emits an event that the controller's schema refuses. A
malformed tool path becomes a null target, and the tool event still
arrives.
**Steps to reproduce**
1. Have an ACPX provider send a tool call whose location path contains a
newline (for example, a Write call with the content in the path).
2. See that the runner emits `tool.execution.progressed` with that
target.
3. See that the controller closes the connection without an
acknowledgement on every redelivery.
**Paperclip version or commit**
master at
|
||
|
|
d7d0f96f3f |
perf(runner): save provider state once per poll instead of once per frame (#20217)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native OpenCode, Codex and Cursor Cloud chat turns run through runnerd. runnerd reads provider frames in order and keeps the run's provider state in a durable file > - For every frame that it reads, runnerd rewrote the whole provider state file and called `fsync` on the file and on its directory. The state holds every tool result kept in the turn, so each save got more expensive as the turn went on > - A tool call request is just the next frame. It waited behind one full save for every frame before it. #20209 measured tool-call turnaround that grew from about 1.4 s to about 15 s in one turn > - #20209 stopped the OpenCode proxy from sending frames that runnerd discards. Every frame that runnerd still reads still paid a full save, and other proxies (for example Cursor Cloud) still send such frames > - This pull request makes runnerd save the frames of one poll once, before anything derived from them leaves runnerd. A save that would write the bytes already on disk is skipped > - The benefit is that the number of saves before a tool call no longer grows with the number of streamed frames. In a local measurement, a tool call after 400 frames with 830 KB of kept state took 0.57 s instead of more than 30 s ## Linked Issues or Issue Description Refs #20209 Related: #17514 (the same per-event full-state rewrite, on the server control-plane side) **What happened** During a native OpenCode chat turn, tool-call turnaround grew from about 1.4 s to about 15 s as the turn went on. runnerd saved its complete durable provider state, with `fsync`, after every provider frame. That state includes the tool results kept so far in the turn. A tool call request waited behind one full save for each earlier frame. **Expected behavior** The time to deliver a tool call must not grow with the number of assistant-text, reasoning or item frames that came before it. A crash or restart must still never lose state that a resumed run needs. **Steps to reproduce** 1. Start a native OpenCode (or Codex app-server or Cursor Cloud) turn that keeps a few tool results and then streams many text or reasoning frames before its next tool call. 2. Measure the time from the tool call frame to the `semantic_tool.input` event. 3. See that the time grows with the number of earlier frames and with the size of the kept state. Locally, `cargo test -p paperclip-runner-core --lib measure_streamed_frame -- --ignored --nocapture` on `master` reports 102 saves and 9.3 s for 100 frames with 830 KB of kept state, and more than 30 s for 400 frames. ## What Changed - `packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs` and `codex_provider.rs`: - The provider frame loop moves into `poll_provider_frames`. An ordinary notification frame and a descendant notification frame no longer save the state. They mark it as unsaved. - After the loop, `poll_current_provider_with_control` saves the unsaved state once. It also saves when a frame fails, before it returns the error. - Frames that already saved at once still do so: turn terminals, Cursor Cloud terminal deliveries, `reconciliation_required`, protocol failures, MCP permission and runtime requests, process exit, and new tool calls. - A tool call that runnerd answers at once (an exact completed replay or the receipt-limit stop) saves unsaved state before it writes to the provider. A rejected tool call already saved first. - `retained_events`, `execute` and `host_park_ready` save unsaved state first, so no event, command result or park decision can expose state that is not on disk. - Courtesy rejections that `CodexProvider::poll` writes by itself (unsupported requests, descendant requests, delayed requests after a terminal, full runtime-request capacity) are held while runnerd reads a batch (`hold_courtesy_responses`). runnerd sends them only after it saves the state that covers every earlier frame (`save_before_provider_writes`). A terminal frame also saves and releases them before runnerd asks the provider for goal state or drains the accounting tail. Released courtesy replies are best-effort, as courtesy replies were before: a provider that already closed stdin cannot turn a received terminal into a polling failure. The receipt-limit stop sends held cancellations before its `turn/interrupt` request, so their order to the provider does not change. - The trace records for runnerd's normalization (`rust_durable_normalization`), which name emitted event ids, are kept until the state that holds those ids is saved. The trace therefore only names saved events. Saved mappings are written before an exited or failed provider is dropped, so they are not lost. Raw frame records still go to the trace when a frame is read; they name no event ids. - `save_unsaved_frame_state` does not write when the serialized state is byte-for-byte the same as the saved file. All other saves write as before. - Serialization moves to `serialize_provider_state`, and the file write moves to `write_state_bytes`. The write path (private temporary file, `fsync`, rename, directory `fsync`, `0600`) does not change. - New tests in `packages/paperclip-runner/runner/crates/runner-core/src/tests/provider_frame_save_batching.rs`: - `streamed_frames_do_not_multiply_durable_saves_before_a_tool_call`: 8 and 48 streamed frame pairs before a tool call need the same number of saves (1). On `master` they need 18 and 98. - `batched_frame_state_is_saved_before_events_leave_and_survives_a_restart`: at every poll, the saved file is equal to the in-memory state and holds exactly the delivered events. A new executor loads the file after the old one stops and gets the same state, the same unacknowledged events and the pending tool call. - `batched_frame_state_is_saved_when_a_later_frame_fails`: a frame that fails after 20 batched frames still leaves those 20 frames on disk. - `previous_release_state_is_rewritten_by_the_first_batched_save`: a state file in an older shape (compact JSON without newer default fields) loads, and the first batched save rewrites it. Unchanged state is not rewritten. Changed state is. - `courtesy_rejections_reach_the_provider_only_after_earlier_frames_are_saved`: when the courtesy rejection reaches the provider, the saved file (what a restarted runnerd would load) already holds the 20 earlier frames and equals the in-memory state. It fails if the hold is removed. - `trace_interpretations_name_only_saved_events`: every event id that a trace record names is in saved state at the moment the record is written. It fails if records are written before the save. - `a_crash_inside_an_unsaved_batch_leaves_saved_state_and_provider_output_unchanged`: after 122 frames are read and not yet saved, the saved file is byte-for-byte unchanged and runnerd has written nothing to the provider. A reload gets the pre-batch event sequence and no pending events. - `a_held_cancellation_that_cannot_reach_the_provider_keeps_the_completed_turn`: the provider closes stdin and then completes the turn with a tool call still pending. The held cancellation cannot be written, and the turn still completes and is saved. - `trace_interpretations_survive_a_provider_exit_in_the_same_poll`: the trace keeps the mappings for frames that arrived in the same poll as the provider exit. - `measure_streamed_frame_cost_before_a_tool_call` (ignored): prints saves and tool-call latency for different frame counts and kept-state sizes. ## Verification - `cargo test --locked -p paperclip-runner-core` (debug, macOS arm64): library 595 passed, 1 failed; all 35 integration test binaries passed (`--no-fail-fast`). The one library failure is `task_memory::tests::monitor_bounds_notices`. It also fails when run alone, takes about 9 minutes, and waits on file-watch timing in `task_memory.rs`, which this PR does not touch. In one earlier run, two ACPX park tests in `acpx_park_preparation` failed. The same binary also failed on unchanged `master` (with a different test), and it passed in the final run. - `cargo test -p paperclip-runner-core --lib measure_streamed_frame -- --ignored --nocapture`, debug build, macOS APFS (`sync_all` is `F_FULLFSYNC`). Each row is one turn: frames streamed, then one tool call. | Kept state | Frames | Saves (master) | Tool-call latency (master) | Saves (this PR) | Tool-call latency (this PR) | | --- | --- | --- | --- | --- | --- | | 4 KB | 100 | 102 | 946 ms | 1 | 18 ms | | 4 KB | 400 | 405 | 3,791 ms | 7 | 70 ms | | 211 KB | 100 | 102 | 3,207 ms | 1 | 37 ms | | 211 KB | 400 | 405 | 12,803 ms | 7 | 193 ms | | 833 KB | 100 | 102 | 9,251 ms | 1 | 93 ms | | 833 KB | 400 | about 405 | more than 30 s (test deadline) | 7 | 567 ms | With 400 frames, the remaining saves are one per poll of up to 128 frames and one per controller acknowledgement. - With the debug `paperclip-runnerd` and `fake-codex-app-server` binaries from this branch: `npx vitest run` on `src/live/runnerd-opencode-streaming-tools`, `runnerd-opencode-assigned-tools`, `runnerd-codex-transport`, `runnerd-codex-transport-settlement`, `runnerd-codex-transport-cold-restore`, `runnerd-restart-continuation`, `runnerd-saved-continuation`, `runnerd-detach-durability`, and `src/cli/opencode-proxy-output`, `opencode-proxy-events`, `opencode-app-server-proxy`: 11 files, 353 tests passed. ## Risks - Durability. The rule is: nothing derived from a frame leaves runnerd before the state that holds that frame is on disk. That covers delivered events, command results, park decisions, tool results and rejections, courtesy rejections, Cursor Cloud terminal acknowledgements, and trace records that name event ids. The tests above check each of these. - The unsaved window is at most one poll (128 frames). Every frame in it is undelivered: no event from it reached the controller, and runnerd wrote nothing to the provider while it was unsaved. `a_crash_inside_an_unsaved_batch_leaves_saved_state_and_provider_output_unchanged` proves this. A crash in the window therefore leaves the saved state and everything the controller and provider have seen exactly as a crash just before those frames were read. The resumed run starts from the same event sequence, so it cannot deliver an event id twice. Its normal recovery (thread resume and `thread/read` reconciliation) handles those frames in the same way as frames still in the provider pipe. On `master`, a crash could already lose the one frame being processed. - One provider write can still happen in the window: `thread/read`, which `CodexProvider::poll` sends to check the lineage of an unknown helper thread. It is a read-only query that waits for its answer, and it changes nothing on the provider side. - Courtesy rejections now reach the provider at the end of the poll instead of during it. The delay is one batch. The provider gets the same replies. - Previous-release story: the saved file shape does not change, and no new field is saved. A file from the previous release loads as before. The skip check compares exact bytes, so an older file is never treated as already saved. The first save after an upgrade rewrites it in the current shape. A deploy that lands mid-turn sees a file that the previous release saved after each frame. That file is valid input for this code. - The ACPX backend has its own frame loop and is not changed here. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file editing) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
92a18bb0f3 |
perf(runner): restart the ACPX provider when a turn settles (#20190)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner drives Claude through the ACPX sidecar. The sidecar's MCP endpoint lasts for the whole session, so it cannot tell which turn sent a late tool callback. > - Because of this, the Runner reaps the sidecar and provider of a generation that ran a turn before it releases call-ID tombstones and rotates receipt epochs. Today this restart runs at the start of the next turn. > - Every follow-up turn in a run pays that restart on its critical path: about 1.5 s with real Claude. #20145 moves the restart to park, but a second turn in the same run, after the first turn that follows an attach, still restarts cold. > - The restart does not have to wait for the next turn. When a turn settles and the host has its terminal result, the Runner is idle. > - This pull request restarts the provider at that point. The next turn, or the next park, then finds a fresh generation. > - The benefit: same-run follow-up turns start without a restart, park does no restart after a settled turn, and the late-callback guarantee is unchanged. ## Linked Issues or Issue Description Refs #20145, which moved the restart to park. This pull request builds on it. No public issue exists, so the problem is described here. **Subsystem affected** Paperclip Runner, ACPX provider (Claude). **Current behavior** `AcpxProviderSession::start_turn_timed` restarts the provider before any turn whose generation already ran a turn. A follow-up turn in the same run waits for `session.suspend`, sidecar shutdown, a new sidecar, `initialize`, `session.open` and `run.attach` before `turn.start`. Since #20145, park does the restart instead, but only for the first turn after an attach. **Proposed behavior** When a turn settles and the controller has ACKed every committed event, the Runner restarts the provider. The next `turn.start` uses the fresh generation and only rotates receipts. Park reuses the fresh generation and does no second restart. **Reason and benefit** The restart leaves the turn's critical path, both for same-run follow-ups and for the turns after a park. The order of reap and rotation stays the same, so a late callback from the reaped generation still cannot reach a later turn. ## What Changed - **Runner loop** (`durable/runner.rs`): new `CommandExecutor::prepare_next_turn` hook, a no-op by default. The runner calls it after the idle provider poll, only when no control frame is waiting and the controller has ACKed every committed event (`acked_source_seq >= highest_source_seq`). So the restart always runs after the terminal result reaches the host. - **ACPX turn-end restart** (`acpx_park_preparation.rs`, `acpx_provider_generation.rs`): - `restart_session_after_turn` restarts a Claude session that has a completed turn, nothing pending, no buffered events, a dormant goal and its whole event tail ACKed. It first settles an abandoned park observation. If that settlement reads more events, it skips this round, and the next idle round restarts after those events are ACKed. - `AcpxProviderSession::restart_after_turn` does the same suspend → reap → start → bootstrap → identity check as the turn-start restart. It records the reaped process id in `restarted_after_turn_from`. This field is the explicit "fresh because of a turn-end restart" marker. - The session does not poll a fresh generation until its next turn or rebind, the same as a prepared one. Its startup output is published with the next turn, as today. Polling it early would add events after the host computed its park cut. - **Next turn** (`acpx_provider_session.rs`): `start_turn_timed` restarts only when the generation ran a turn. When `restarted_after_turn_from` is set, it does the post-restart receipt and ledger rotation without a second restart. `session.reconciled` reports the reaped process as `previousProcessId`. - **Park**: `prepare_next_generation` restarts (with the 20 s park deadline) only if no turn-end restart ran. Otherwise it marks the generation for `run.attach` rebind and returns `false`, so the runner does not ask for a second park proof. The park-time receipt rotation moves to the next turn start, where the turn-end path does it too. - **Restart failure**: the executor sets the same state that a failed turn-start restart leaves (`closed`, no session, provider exit confirmed). The next `turn.start` fails with the same `ACPX turn start failed closed: …` error that the turn-start restart gives today. - **Fake sidecar**: new `pending-input` fault, which opens an input request in a turn that then completes. ### Races | Race | Handling | | --- | --- | | A turn or steer arrives during the turn-end restart | The runner loop is single-threaded. The command waits in the socket until the restart returns, then starts on the fresh generation without a restart. Only one generation is live. | | A turn arrives before the hook runs (for example, in the same read as the ACK) | Control frames go first, so the turn restarts at its start, as today. The hook then restarts after that turn. | | Park arrives during the turn-end restart | Host control is read at the top of the loop, so park waits. Park then reuses the fresh generation. No event is published in between, so the host's park cut still matches. | | Park arrives before the hook runs | Park restarts with the 20 s deadline, as in #20145. The hook then finds a fresh generation and does nothing. | | Eviction, `runner.suspend` or retire after the restart | They reap the fresh generation through the ordinary paths. | | The restart fails | The session is `closed`. The next turn fails with the same error and the same durable state as a failed turn-start restart. Park returns conflict, as it does for a closed session today. | | Cancellation | `run.cancel`, `turn.interrupt` and `turn.stop` after the restart return `already_settled`, as after any settled turn. A cancellation that retires the runtime closes the session, so no restart runs. | | Interrupted, cancelled or failed turn | `completed_turn_observed` is false, so no turn-end restart runs. The next turn restarts at its start, as today. | | Pending permission or input requests at the terminal | The reducer ends them at the terminal, so the restart runs. A later `request.resolve` fails with `no active turn`, with or without the restart. | | Active session goal at the terminal | No restart, because the goal can keep working after its prompt. | ### Removed as redundant, and what stays - **Removed:** the park-time restart and dormant verify when a turn-end restart already made the generation fresh. The runner's `host_park_ready` proves dormancy just before `prepare_for_park`, and nothing runs in between. - **Removed:** the runner's second park proof in that case, because the provider did not change. - **Removed:** the park-time receipt and ledger rotation. The next turn start does it once, for both paths. - **Kept: the turn-start restart.** Without it, a turn queued before the hook, a turn after an interrupted or cancelled turn, a turn after an active goal, and any non-Claude ACPX turn would run in the generation that ran the previous turn. Tests cover the queued-turn and interrupted cases. - **Kept: the park-time restart.** Without it, a park that arrives before the hook would leave the next run to restart at attach. A test covers this. - **Kept: the settlement of an abandoned park observation.** Without it, the restart drops events that the old generation sent before the abandoned response. A test covers this, and it fails with the settlement removed. - **Kept: the goal-dormancy check.** Without it, the restart kills an active goal's autonomous work. A test covers this, and it fails with the check removed. - **Kept: the ACK gate.** Without it, the restart can run before the host has the terminal result. A runner test covers this. - **Kept: deferred polling of the fresh generation.** Without it, its startup event lands after the host computed its park cut, and the park conflicts. - **Kept: the kept failure message.** Without it, the next turn fails with `ACPX session is not open`, and the restart error is lost. ## Verification - `cargo test -p paperclip-runner-core --lib` and every `tests/acpx_*.rs`, `native_provider_backend` and `durable_recovery` suite: all pass (debug). - New tests (`tests/support/acpx_turn_end_restart.rs`, run from `acpx_park_preparation`) prove these points: - A settled turn reaps the old pid before it spawns the new one, and no predecessor is still live. A same-run follow-up then sends only `turn.start` and reports the reaped pid in `session.reconciled`. - Park after a turn-end restart sends no sidecar command and returns `false`. Attach rebinds the same pid. The run's second turn restarts at its end, and the third turn starts without a restart. - The reaped generation's late callback never reaches the next turn. - A turn that arrives before the hook restarts at its start. A park before the hook restarts once. - A failed turn-end restart gives the same next-turn error, retry error, durable state and process count as a failed turn-start restart. - Cancellation after the restart is `already_settled`. `runner.suspend` reaps the fresh generation. - An active goal skips the restart. An input request open at the terminal behaves the same with or without the restart. - `acpx_provider_turns`: a session-level turn-end restart drops the old generation's late `issues.delete` callback, which reuses the previous turn's call ID. The next turn then admits that call ID from the fresh provider. Interrupted and retired turns do not restart at turn end. - `acpx_park_readiness`: a turn-end-restarted generation is park-ready. An abandoned park observation tail is delivered before the restart. - `durable::runner`: the hook runs only after the terminal is ACKed and when control is idle. It never runs during ACK debt, a waiting control frame or a disconnect. - Server integration tests (`plugin-runner-release-resume`, `plugin-runner-eviction` and `plugin-runner-task-warm`) with this branch's Runner: 49 passed. The provider process counts do not change: each released run still adds one process start, now at turn end instead of at park. - `cargo fmt --check`: clean. - Fake-sidecar timing (release build, 7 runs, medians): | | before (`master` with #20145) | this PR | | --- | --- | --- | | same-run follow-up `turn.start` → `turn.started` | 51.5 ms | 22.0 ms | | second turn after an attach, `turn.start` → `turn.started` | 270.3 ms | 23.5 ms | | turn-end restart, after the terminal is ACKed | — | 29.0 ms | | park preparation after a settled turn | 34.1 ms | 0.03 ms | - Real Claude: not run for this pull request. With real Claude, the restart took about 1.5 s on the turn-start path, and this change takes it off that path. That number is not measured on this branch. ## Risks - A run that ends after one turn without a park now starts one extra sidecar that no turn uses. The Claude CLI itself starts only at the first prompt, so the cost is the sidecar's start and bootstrap. - The restart blocks the Runner loop for its length (about 1.5 s with real Claude), as the turn-start restart does today. It has no deadline, as before. A park that arrives during it waits, inside the host's 30 s park wait. - The fresh generation is not polled until its next turn. If it exits while idle, the next turn finds out at `turn.start` and fails closed. Park finds out at its observation. - The turn-end restart is for Claude only, the same as park preparation. Other ACPX agents keep restarting at turn start. - If the Runner crashes during the restart, the durable state still says `session_open`. Recovery is the same as for a crash while idle. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2d93413085 |
perf(runner): prepare the next Claude provider generation at park (#20145)
## Thinking Path
> - Paperclip runs agents through Paperclip Runner, and a host can park
an idle Runner between runs and resume it for the next run on the same
task.
> - For Claude (ACPX), a follow-up run on a parked Runner pays about
1.5–2.3 s at `run.attach` before the provider can start the turn. The
attachment closes the Claude sidecar and spawns a new one: initialize,
`session.open` with `session/load`, `run.attach`, then a dormant verify.
> - The respawn is required. The tool-bridge MCP endpoint lasts for the
whole session and can't tell which turn a late callback came from, so a
new turn must not run in a process that ran an earlier one.
> - Nothing requires the respawn to happen at the *start* of the next
run. A turn that ends 2 s later is a better experience than one that
starts 2 s later.
> - This pull request does the respawn while parking. The next run's
`run.attach` then only rebinds the run id on a fresh, never-used
generation.
> - Follow-up Claude runs on a parked Runner reach the provider about
1.5–2 s sooner, and the late-callback guarantees are unchanged.
## Linked Issues or Issue Description
No public issue exists. The issue is described inline.
### What happened?
On a hosted deployment with parked Runners, Claude follow-up runs
reached the provider about 1.8 s later than Codex follow-ups.
`runner.session.resume` medians were 5.2 s for Claude and 3.4 s for
Codex. The extra time is the ACPX respawn inside `run.attach`. Codex
keeps its process in place.
### Expected behavior
The respawn happens while the Runner parks, so the next run starts on a
prepared generation.
### Steps to reproduce
1. Run a Claude (ACPX) task on a host that parks Runners between runs.
2. Let the run finish and the Runner park.
3. Send a follow-up. `run.attach` closes the provider and starts,
initializes, opens and attaches a new sidecar before `turn.start`. The
Claude CLI itself starts at the first prompt on either path.
### Paperclip version or commit
`master` at
|
||
|
|
412d49e48d |
fix(runner): resume an ACPX turn interrupted by a deploy instead of refusing the launch digest (#20112)
## Thinking Path
- The rule for upgrade-safe saved state: work saved by the old build
must resume under the new build.
- The K14 upgrade-mid-run chaos wave (`838e9bf3d2` → `f789f3e2af`) broke
that rule for Claude. The same heartbeat run resumed after the upgrade
restart, and the runner's resume probe (`runner.drain`) failed with
`ACPX durable launch profile digest does not match runner startup`
(`provider_state_incompatible`). t03 and t04's Claude subtask both
needed a board Retry. Codex crossed the upgrade cleanly.
- The ACPX launch profile digest covers the provider-pack authority and
the sidecar artifact hashes, so every deploy changes it.
- #18879 already re-binds a suspended session to the new digest. A
deploy that restarts the runner mid-turn leaves the same run with an
active turn, or with an open session, which is the state a same-build
lost-turn restore leaves behind. Neither was re-bound.
- A same-build restart already recovers both states safely: it reopens
an open session, and recovers an active turn through
`restore_conversation_after_lost_turn`, which is guarded by the
provider-lifetime fence. An upgraded build should take the same path.
## Linked Issues or Issue Description
Refs #18879
**What happened?**
A deploy restarted the runner while a Claude (ACPX) task was mid-run.
When the same run resumed under the new build, restore refused the saved
state with `provider_state_incompatible`. The task stopped and needed a
manual Retry, which started a fresh provider session and dropped the
conversation.
**Steps to reproduce**
1. Start a Claude task on build A and let it reach an active turn.
2. Upgrade to build B, which ships a new provider pack or sidecar build,
with a graceful restart while the turn runs.
3. The same run's resume fails with `ACPX durable launch profile digest
does not match runner startup`.
**Expected behavior**
The resumed run recovers exactly as it would after a same-build restart.
The conversation is restored, or the lost turn is closed through the
normal recovery path, with no Retry.
## What Changed
-
`packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs`:
- `rebinds_settled_session` now also re-binds the saved digest for an
**interrupted turn or open session of the same heartbeat run**
(`rebinds_interrupted_turn`).
- The re-bind requires exactly the same-build recovery preconditions: no
accepted semantic result, no pending runtime request, no unconfirmed
provider exit, and a dormant goal. It also keeps the existing
requirements: no candidate admission, and the same sidecar command and
arguments.
- Recovery then runs unchanged. The provider-lifetime fence still
decides whether the conversation is restored or the lost turn is closed.
- **Relocated installs:** an upgrade can install the same launch files
under a new directory, as an `npx` versioned cache, a new image layout
or a new checkout does. The K14 harness moves between worktrees. The
re-bind now accepts a launch of the same shape: the same executable and
sidecar file names, and identical non-path arguments. It then records
this startup's runner-owned paths in the saved descriptor. A renamed
executable or a changed argument stays strict.
- An active turn or open session saved by a *different* run stays
strict, as before. So do suspended sessions with undelivered events, and
every qualified-release binding (ACPX, agent server and command digest).
- Tests (Rust, `acpx_provider_backend`):
-
`interrupted_turn_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
red on master. An old-digest saved state with an active turn restores,
the digest is re-bound, and identity and command digest are kept.
-
`open_session_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
restore gets past the saved-state check, to provider start.
- `saved_sessions_rebind_to_an_upgraded_build_installed_at_a_new_path`:
red on master and on the first fix, with the production error. It covers
an interrupted turn of the same run and a settled session attached by
the next run, both moved to a new install directory.
-
`interrupted_turn_after_upgrade_stays_strict_when_lost_turn_recovery_is_unsafe`:
each unsafe guard still fails with `provider_state_incompatible` and
leaves the saved state untouched.
- The existing strict tests for other runs are unchanged and still pass.
- `packages/paperclip-runner/docs/durable-recovery.md`: documents the
extended re-bind.
- `tests/acpx_warm_attachment.rs`: the end-to-end test
`restores_a_turn_lost_to_a_runner_upgrade_under_the_new_launch_profile`,
from the Phase 2 session's parallel fix. It fails on master with the
production error and passes here.
## Verification
- `cargo test -p paperclip-runner-core --lib acpx_provider_backend`: 44
passed.
- The full `paperclip-runner-core` lib suite: 585 passed. The 3 failures
are timing-based `task_memory` kill-monitor tests on macOS; that file is
untouched (last changed 5 days ago). CI runs them on Linux.
- `cargo fmt --check`: clean.
- Live: the first K14 rerun, at
|
||
|
|
a8bbdf237c |
fix(runner): reopen the ACPX session at turn.start when it can be safely reopened (#20110)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native Claude ACPX run must survive restarts and suspensions. A
turn stopped for suspension drops the live ACPX session and leaves the
saved lifecycle `prepared`
> - The next `turn.start` checked for a live session before anything
else, so it failed with "ACPX session is not open". That happened even
though the saved session was safe to reopen
> - In restart chaos this made a Claude approval task (G1 t03) need a
Retry after repeated restores
> - This pull request makes `turn.start` reopen a reopenable session
before it starts the turn
> - The benefit is that a suspended or restored Claude run keeps going
on its own, with no Retry
## Linked Issues or Issue Description
Refs #20089
**What happened?**
In a local restart-chaos G1 (approval) wave, Claude ACPX task t03 failed
after repeated restores with `ACPX session is not open` and needed a
Retry. The Retry overwrote the session directory, so the exact sequence
could not be read back. The same error reproduces directly with
`turn.stop` (suspension) followed by `turn.start` on the same session:
the stop clears the live session and sets the lifecycle to `prepared`,
and `turn.start` never reopens it.
**Expected behavior**
When the saved session can be reopened, `turn.start` reopens it, the
same way `session.open` does, and then starts the turn. "Can be
reopened" means: the conversation identity is recorded, the provider
exit is confirmed, the session belongs to this run, and the lifecycle is
`prepared`, `suspended` or `session_open`. A session that was closed on
purpose, or whose provider exit is unconfirmed, still fails.
**Steps to reproduce**
1. `run.prepare`, `session.open`, then `turn.start`.
2. `turn.stop` (suspension stop).
3. `turn.start` again. Before this change, it fails with "ACPX session
is not open".
**Paperclip version or commit**
master at
|
||
|
|
c55af26821 |
fix(runner): treat a replayed start of a turn lost on restore as history (t48) (#20074)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native Codex run must survive a control-plane restart and continue its work > - #20057 made recovery adopt an in-flight turn, or start a new turn when the restored runner reports the old turn lost > - One case was not covered. A graceful drain can detach the controller after the runner accepted `turn.start` but before the provider started the turn > - The runner then records the provider's `turn.started` while no controller is attached. After a restart it replays that start, and the Codex session rejects it as an unexpected turn > - This pull request treats that replayed start as history from the earlier generation > - The benefit is that the run starts its continuation turn instead of stopping with `native_event_replay_conflict` ## Linked Issues or Issue Description Refs #20057 **What happened?** A restart chaos wave ended with one failed task (t48, Codex). A graceful SIGTERM suspended the controller 300 ms after the runner accepted `turn.start`. The provider's `turn.started` came after the controller detached. Recovery classified the run as `resume_dead_runner`. The restored runner reported the turn as `provider_turn_lost_on_restore`, which is correct. But it also replayed the undelivered `turn.started`. The Codex session failed with "Provider started an unexpected turn." (`turn_binding_mismatch`), and recovery stopped with `native_event_replay_conflict`. **Expected behavior** Recovery starts a new continuation turn. The replayed start of the lost turn is recorded as history, and the run completes. **Steps to reproduce** 1. Start a Codex native run. 2. Detach the controller gracefully after the runner accepts `turn.start` and before the provider emits `turn/started`. 3. Let the runner record `turn.started` while it is unattached, then kill the runner and the provider. 4. Restore the run. Before this change, the session fails with `turn_binding_mismatch`. **Paperclip version or commit** master at |
||
|
|
c0c26188db |
fix(runner): stamp event generations and record an earlier generation's events as history (#19960)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute through the Paperclip Runner, which turns provider output (Codex, OpenCode, Claude through ACPX) into durable runner events > - In the restart chaos campaign, events from an earlier turn or runner generation failed the current turn about ten times (RC-15, 20, 20b, 20c, 33, 33b, 59 and 61) > - #19718, #19722, #19772 and #19783 each special-cased one variant, and other variants still fail the run > - The durable-ownership plan (section G, invariant 6) asks for one rule: every event carries its generation and turn, and an earlier generation's events are recorded as history and never fail the current turn > - This pull request stamps the runner generation on every event and applies one stale-event rule in the ACPX provider state, the Codex driver (which Runnerd also uses for OpenCode) and the pending `turn.start` check > - The benefit is that a late report about a settled or restored turn no longer stops a healthy turn, while events for a turn the runner never saw still fail closed ## Linked Issues or Issue Description Refs #19718, #19722, #19772, #19783 **What happened?** A runner or provider can report a turn after that turn settled: a resumed provider reports the lost turn's usage or terminal, an ACPX sidecar emits a last delta after the turn terminal, or a held failure of the previous turn sits in front of a new `turn.start`. Several of these paths fail the current turn: ACPX `AcpxEventScope` rejects any event for a non-active turn ("named a stale turn"), the Codex driver fails `turn_binding_mismatch` for a settled turn's late item between turns, and `runnerdPendingTurnStartFailure` reports any held non-recoverable failure, whatever turn it names. Runner events carried no generation. **Expected behavior** An event for a turn that is not active, and that an earlier generation accepted or this generation already settled, is recorded as history and never fails the current turn. An event for a turn the runner never saw still fails closed. **Steps to reproduce** ACPX: settle turn 1, start turn 2, and deliver a late `runtime.event` for turn 1. Before this change, the provider state returns an error and the run fails. ## What Changed - Runner (`durable/state.rs`, `durable/runner.rs`): every event payload carries `generation`, the runner process generation that emitted it. A runner process that recovers state starts the next generation (`begin_recovered_generation`); it is saved with the next normal write, so recovery adds no write before the first turn. Loading stays pure. Old saved state loads as generation 0. Outbox events keep their original generation. - PRP event schemas v1–v3 declare the optional `generation` property; generated validators and the manifest are regenerated. - ACPX (`acpx_event_scope.rs`, `acpx_provider_state.rs`): an event that names a settled turn of this run becomes a bounded `stale_generation_event` diagnostic (32 per state) and changes no state. Unknown turns still fail. - Codex driver (`codex-session-state.ts`, `codex-session-notifications.ts`): one predicate, `isHistoryTurn()`: the turn is not active, and it was restored from the checkpoint (an earlier generation) or settled in this generation. The classifier's stale-turn branch, the resume-usage branch, restored-terminal replay and the main fail point (`notificationNamesActiveTurn`) all use it. A terminal for an earlier generation's turn is history even when it disagrees with the checkpoint. A terminal that contradicts a turn this same generation settled is still a `conflicting_turn_terminal` integrity failure, and a provider that starts a settled turn again is still rejected. - Transport (`runnerd-codex-transport.ts`): a pending `turn.start` ignores a held failure that names another provider turn. - `DURABLE_TRANSPORT.md` documents the stamp and the rule. ## Verification - `cargo test -p paperclip-runner-core`: new `records_late_events_for_a_settled_turn_as_history` (red before), `stamps_every_event_with_the_runner_process_generation`; two existing ACPX tests now assert "recorded as history, no state change" instead of "rejected". Durable tests: 112 passed. `cargo fmt` clean. - `codex-app-server-driver.stale-generation.test.ts` (new, 9 tests): late item, delta, terminal and workspace change for a settled turn during and before the next turn, an earlier generation's conflicting terminal after recovery, and an unknown turn that still fails. Red on master: 8 of 9 (the during-turn cases on master already dropped the event, with a different diagnostic; the between-turn and recovery cases failed the session). - `runnerd-pending-turn-start-failure.test.ts` (new): an earlier turn's held failure no longer fails the pending start; the pending turn's own failure and a failed run terminal still do. - `vitest run src/drivers/codex src/live/runnerd-codex-transport`: 787 passed; 2 Codex executable-resolution tests timed out under host load and pass when run alone. - `tsc -p tsconfig.json` for the runner package, `check:protocol-types`, `check:replay-goldens`: clean. - Not run: chaos waves (the coordinator schedules them on this host). ## Risks - A late report about a settled turn is now dropped with a diagnostic instead of failing. A provider that keeps working on a settled turn could have its output ignored silently; a provider that starts a settled turn again, or names a turn the runner never saw, still fails closed. - `generation` is a new optional field. Consumers that do not know it ignore it, as the PRP contract requires. - The generation counter is written with the next normal state save, so there is no new write before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |