mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 06:10:44 +01:00
42bc238aa5a1208cabecba76773e0500fcd33959
239
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Nicky LeachandClaude Opus 5.5
|
92a18bb0f3 |
perf(runner): restart the ACPX provider when a turn settles (#20190)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner drives Claude through the ACPX sidecar. The sidecar's MCP endpoint lasts for the whole session, so it cannot tell which turn sent a late tool callback. > - Because of this, the Runner reaps the sidecar and provider of a generation that ran a turn before it releases call-ID tombstones and rotates receipt epochs. Today this restart runs at the start of the next turn. > - Every follow-up turn in a run pays that restart on its critical path: about 1.5 s with real Claude. #20145 moves the restart to park, but a second turn in the same run, after the first turn that follows an attach, still restarts cold. > - The restart does not have to wait for the next turn. When a turn settles and the host has its terminal result, the Runner is idle. > - This pull request restarts the provider at that point. The next turn, or the next park, then finds a fresh generation. > - The benefit: same-run follow-up turns start without a restart, park does no restart after a settled turn, and the late-callback guarantee is unchanged. ## Linked Issues or Issue Description Refs #20145, which moved the restart to park. This pull request builds on it. No public issue exists, so the problem is described here. **Subsystem affected** Paperclip Runner, ACPX provider (Claude). **Current behavior** `AcpxProviderSession::start_turn_timed` restarts the provider before any turn whose generation already ran a turn. A follow-up turn in the same run waits for `session.suspend`, sidecar shutdown, a new sidecar, `initialize`, `session.open` and `run.attach` before `turn.start`. Since #20145, park does the restart instead, but only for the first turn after an attach. **Proposed behavior** When a turn settles and the controller has ACKed every committed event, the Runner restarts the provider. The next `turn.start` uses the fresh generation and only rotates receipts. Park reuses the fresh generation and does no second restart. **Reason and benefit** The restart leaves the turn's critical path, both for same-run follow-ups and for the turns after a park. The order of reap and rotation stays the same, so a late callback from the reaped generation still cannot reach a later turn. ## What Changed - **Runner loop** (`durable/runner.rs`): new `CommandExecutor::prepare_next_turn` hook, a no-op by default. The runner calls it after the idle provider poll, only when no control frame is waiting and the controller has ACKed every committed event (`acked_source_seq >= highest_source_seq`). So the restart always runs after the terminal result reaches the host. - **ACPX turn-end restart** (`acpx_park_preparation.rs`, `acpx_provider_generation.rs`): - `restart_session_after_turn` restarts a Claude session that has a completed turn, nothing pending, no buffered events, a dormant goal and its whole event tail ACKed. It first settles an abandoned park observation. If that settlement reads more events, it skips this round, and the next idle round restarts after those events are ACKed. - `AcpxProviderSession::restart_after_turn` does the same suspend → reap → start → bootstrap → identity check as the turn-start restart. It records the reaped process id in `restarted_after_turn_from`. This field is the explicit "fresh because of a turn-end restart" marker. - The session does not poll a fresh generation until its next turn or rebind, the same as a prepared one. Its startup output is published with the next turn, as today. Polling it early would add events after the host computed its park cut. - **Next turn** (`acpx_provider_session.rs`): `start_turn_timed` restarts only when the generation ran a turn. When `restarted_after_turn_from` is set, it does the post-restart receipt and ledger rotation without a second restart. `session.reconciled` reports the reaped process as `previousProcessId`. - **Park**: `prepare_next_generation` restarts (with the 20 s park deadline) only if no turn-end restart ran. Otherwise it marks the generation for `run.attach` rebind and returns `false`, so the runner does not ask for a second park proof. The park-time receipt rotation moves to the next turn start, where the turn-end path does it too. - **Restart failure**: the executor sets the same state that a failed turn-start restart leaves (`closed`, no session, provider exit confirmed). The next `turn.start` fails with the same `ACPX turn start failed closed: …` error that the turn-start restart gives today. - **Fake sidecar**: new `pending-input` fault, which opens an input request in a turn that then completes. ### Races | Race | Handling | | --- | --- | | A turn or steer arrives during the turn-end restart | The runner loop is single-threaded. The command waits in the socket until the restart returns, then starts on the fresh generation without a restart. Only one generation is live. | | A turn arrives before the hook runs (for example, in the same read as the ACK) | Control frames go first, so the turn restarts at its start, as today. The hook then restarts after that turn. | | Park arrives during the turn-end restart | Host control is read at the top of the loop, so park waits. Park then reuses the fresh generation. No event is published in between, so the host's park cut still matches. | | Park arrives before the hook runs | Park restarts with the 20 s deadline, as in #20145. The hook then finds a fresh generation and does nothing. | | Eviction, `runner.suspend` or retire after the restart | They reap the fresh generation through the ordinary paths. | | The restart fails | The session is `closed`. The next turn fails with the same error and the same durable state as a failed turn-start restart. Park returns conflict, as it does for a closed session today. | | Cancellation | `run.cancel`, `turn.interrupt` and `turn.stop` after the restart return `already_settled`, as after any settled turn. A cancellation that retires the runtime closes the session, so no restart runs. | | Interrupted, cancelled or failed turn | `completed_turn_observed` is false, so no turn-end restart runs. The next turn restarts at its start, as today. | | Pending permission or input requests at the terminal | The reducer ends them at the terminal, so the restart runs. A later `request.resolve` fails with `no active turn`, with or without the restart. | | Active session goal at the terminal | No restart, because the goal can keep working after its prompt. | ### Removed as redundant, and what stays - **Removed:** the park-time restart and dormant verify when a turn-end restart already made the generation fresh. The runner's `host_park_ready` proves dormancy just before `prepare_for_park`, and nothing runs in between. - **Removed:** the runner's second park proof in that case, because the provider did not change. - **Removed:** the park-time receipt and ledger rotation. The next turn start does it once, for both paths. - **Kept: the turn-start restart.** Without it, a turn queued before the hook, a turn after an interrupted or cancelled turn, a turn after an active goal, and any non-Claude ACPX turn would run in the generation that ran the previous turn. Tests cover the queued-turn and interrupted cases. - **Kept: the park-time restart.** Without it, a park that arrives before the hook would leave the next run to restart at attach. A test covers this. - **Kept: the settlement of an abandoned park observation.** Without it, the restart drops events that the old generation sent before the abandoned response. A test covers this, and it fails with the settlement removed. - **Kept: the goal-dormancy check.** Without it, the restart kills an active goal's autonomous work. A test covers this, and it fails with the check removed. - **Kept: the ACK gate.** Without it, the restart can run before the host has the terminal result. A runner test covers this. - **Kept: deferred polling of the fresh generation.** Without it, its startup event lands after the host computed its park cut, and the park conflicts. - **Kept: the kept failure message.** Without it, the next turn fails with `ACPX session is not open`, and the restart error is lost. ## Verification - `cargo test -p paperclip-runner-core --lib` and every `tests/acpx_*.rs`, `native_provider_backend` and `durable_recovery` suite: all pass (debug). - New tests (`tests/support/acpx_turn_end_restart.rs`, run from `acpx_park_preparation`) prove these points: - A settled turn reaps the old pid before it spawns the new one, and no predecessor is still live. A same-run follow-up then sends only `turn.start` and reports the reaped pid in `session.reconciled`. - Park after a turn-end restart sends no sidecar command and returns `false`. Attach rebinds the same pid. The run's second turn restarts at its end, and the third turn starts without a restart. - The reaped generation's late callback never reaches the next turn. - A turn that arrives before the hook restarts at its start. A park before the hook restarts once. - A failed turn-end restart gives the same next-turn error, retry error, durable state and process count as a failed turn-start restart. - Cancellation after the restart is `already_settled`. `runner.suspend` reaps the fresh generation. - An active goal skips the restart. An input request open at the terminal behaves the same with or without the restart. - `acpx_provider_turns`: a session-level turn-end restart drops the old generation's late `issues.delete` callback, which reuses the previous turn's call ID. The next turn then admits that call ID from the fresh provider. Interrupted and retired turns do not restart at turn end. - `acpx_park_readiness`: a turn-end-restarted generation is park-ready. An abandoned park observation tail is delivered before the restart. - `durable::runner`: the hook runs only after the terminal is ACKed and when control is idle. It never runs during ACK debt, a waiting control frame or a disconnect. - Server integration tests (`plugin-runner-release-resume`, `plugin-runner-eviction` and `plugin-runner-task-warm`) with this branch's Runner: 49 passed. The provider process counts do not change: each released run still adds one process start, now at turn end instead of at park. - `cargo fmt --check`: clean. - Fake-sidecar timing (release build, 7 runs, medians): | | before (`master` with #20145) | this PR | | --- | --- | --- | | same-run follow-up `turn.start` → `turn.started` | 51.5 ms | 22.0 ms | | second turn after an attach, `turn.start` → `turn.started` | 270.3 ms | 23.5 ms | | turn-end restart, after the terminal is ACKed | — | 29.0 ms | | park preparation after a settled turn | 34.1 ms | 0.03 ms | - Real Claude: not run for this pull request. With real Claude, the restart took about 1.5 s on the turn-start path, and this change takes it off that path. That number is not measured on this branch. ## Risks - A run that ends after one turn without a park now starts one extra sidecar that no turn uses. The Claude CLI itself starts only at the first prompt, so the cost is the sidecar's start and bootstrap. - The restart blocks the Runner loop for its length (about 1.5 s with real Claude), as the turn-start restart does today. It has no deadline, as before. A park that arrives during it waits, inside the host's 30 s park wait. - The fresh generation is not polled until its next turn. If it exits while idle, the next turn finds out at `turn.start` and fails closed. Park finds out at its observation. - The turn-end restart is for Claude only, the same as park preparation. Other ACPX agents keep restarting at turn start. - If the Runner crashes during the restart, the durable state still says `session_open`. Recovery is the same as for a crash while idle. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
2d93413085 |
perf(runner): prepare the next Claude provider generation at park (#20145)
## Thinking Path
> - Paperclip runs agents through Paperclip Runner, and a host can park
an idle Runner between runs and resume it for the next run on the same
task.
> - For Claude (ACPX), a follow-up run on a parked Runner pays about
1.5–2.3 s at `run.attach` before the provider can start the turn. The
attachment closes the Claude sidecar and spawns a new one: initialize,
`session.open` with `session/load`, `run.attach`, then a dormant verify.
> - The respawn is required. The tool-bridge MCP endpoint lasts for the
whole session and can't tell which turn a late callback came from, so a
new turn must not run in a process that ran an earlier one.
> - Nothing requires the respawn to happen at the *start* of the next
run. A turn that ends 2 s later is a better experience than one that
starts 2 s later.
> - This pull request does the respawn while parking. The next run's
`run.attach` then only rebinds the run id on a fresh, never-used
generation.
> - Follow-up Claude runs on a parked Runner reach the provider about
1.5–2 s sooner, and the late-callback guarantees are unchanged.
## Linked Issues or Issue Description
No public issue exists. The issue is described inline.
### What happened?
On a hosted deployment with parked Runners, Claude follow-up runs
reached the provider about 1.8 s later than Codex follow-ups.
`runner.session.resume` medians were 5.2 s for Claude and 3.4 s for
Codex. The extra time is the ACPX respawn inside `run.attach`. Codex
keeps its process in place.
### Expected behavior
The respawn happens while the Runner parks, so the next run starts on a
prepared generation.
### Steps to reproduce
1. Run a Claude (ACPX) task on a host that parks Runners between runs.
2. Let the run finish and the Runner park.
3. Send a follow-up. `run.attach` closes the provider and starts,
initializes, opens and attaches a new sidecar before `turn.start`. The
Claude CLI itself starts at the first prompt on either path.
### Paperclip version or commit
`master` at
|
||
|
DottaandClaude Opus 5.5
|
412d49e48d |
fix(runner): resume an ACPX turn interrupted by a deploy instead of refusing the launch digest (#20112)
## Thinking Path
- The rule for upgrade-safe saved state: work saved by the old build
must resume under the new build.
- The K14 upgrade-mid-run chaos wave (`838e9bf3d2` → `f789f3e2af`) broke
that rule for Claude. The same heartbeat run resumed after the upgrade
restart, and the runner's resume probe (`runner.drain`) failed with
`ACPX durable launch profile digest does not match runner startup`
(`provider_state_incompatible`). t03 and t04's Claude subtask both
needed a board Retry. Codex crossed the upgrade cleanly.
- The ACPX launch profile digest covers the provider-pack authority and
the sidecar artifact hashes, so every deploy changes it.
- #18879 already re-binds a suspended session to the new digest. A
deploy that restarts the runner mid-turn leaves the same run with an
active turn, or with an open session, which is the state a same-build
lost-turn restore leaves behind. Neither was re-bound.
- A same-build restart already recovers both states safely: it reopens
an open session, and recovers an active turn through
`restore_conversation_after_lost_turn`, which is guarded by the
provider-lifetime fence. An upgraded build should take the same path.
## Linked Issues or Issue Description
Refs #18879
**What happened?**
A deploy restarted the runner while a Claude (ACPX) task was mid-run.
When the same run resumed under the new build, restore refused the saved
state with `provider_state_incompatible`. The task stopped and needed a
manual Retry, which started a fresh provider session and dropped the
conversation.
**Steps to reproduce**
1. Start a Claude task on build A and let it reach an active turn.
2. Upgrade to build B, which ships a new provider pack or sidecar build,
with a graceful restart while the turn runs.
3. The same run's resume fails with `ACPX durable launch profile digest
does not match runner startup`.
**Expected behavior**
The resumed run recovers exactly as it would after a same-build restart.
The conversation is restored, or the lost turn is closed through the
normal recovery path, with no Retry.
## What Changed
-
`packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs`:
- `rebinds_settled_session` now also re-binds the saved digest for an
**interrupted turn or open session of the same heartbeat run**
(`rebinds_interrupted_turn`).
- The re-bind requires exactly the same-build recovery preconditions: no
accepted semantic result, no pending runtime request, no unconfirmed
provider exit, and a dormant goal. It also keeps the existing
requirements: no candidate admission, and the same sidecar command and
arguments.
- Recovery then runs unchanged. The provider-lifetime fence still
decides whether the conversation is restored or the lost turn is closed.
- **Relocated installs:** an upgrade can install the same launch files
under a new directory, as an `npx` versioned cache, a new image layout
or a new checkout does. The K14 harness moves between worktrees. The
re-bind now accepts a launch of the same shape: the same executable and
sidecar file names, and identical non-path arguments. It then records
this startup's runner-owned paths in the saved descriptor. A renamed
executable or a changed argument stays strict.
- An active turn or open session saved by a *different* run stays
strict, as before. So do suspended sessions with undelivered events, and
every qualified-release binding (ACPX, agent server and command digest).
- Tests (Rust, `acpx_provider_backend`):
-
`interrupted_turn_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
red on master. An old-digest saved state with an active turn restores,
the digest is re-bound, and identity and command digest are kept.
-
`open_session_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
restore gets past the saved-state check, to provider start.
- `saved_sessions_rebind_to_an_upgraded_build_installed_at_a_new_path`:
red on master and on the first fix, with the production error. It covers
an interrupted turn of the same run and a settled session attached by
the next run, both moved to a new install directory.
-
`interrupted_turn_after_upgrade_stays_strict_when_lost_turn_recovery_is_unsafe`:
each unsafe guard still fails with `provider_state_incompatible` and
leaves the saved state untouched.
- The existing strict tests for other runs are unchanged and still pass.
- `packages/paperclip-runner/docs/durable-recovery.md`: documents the
extended re-bind.
- `tests/acpx_warm_attachment.rs`: the end-to-end test
`restores_a_turn_lost_to_a_runner_upgrade_under_the_new_launch_profile`,
from the Phase 2 session's parallel fix. It fails on master with the
production error and passes here.
## Verification
- `cargo test -p paperclip-runner-core --lib acpx_provider_backend`: 44
passed.
- The full `paperclip-runner-core` lib suite: 585 passed. The 3 failures
are timing-based `task_memory` kill-monitor tests on macOS; that file is
untouched (last changed 5 days ago). CI runs them on Linux.
- `cargo fmt --check`: clean.
- Live: the first K14 rerun, at
|
||
|
DottaandClaude Opus 5.5
|
a8bbdf237c |
fix(runner): reopen the ACPX session at turn.start when it can be safely reopened (#20110)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native Claude ACPX run must survive restarts and suspensions. A
turn stopped for suspension drops the live ACPX session and leaves the
saved lifecycle `prepared`
> - The next `turn.start` checked for a live session before anything
else, so it failed with "ACPX session is not open". That happened even
though the saved session was safe to reopen
> - In restart chaos this made a Claude approval task (G1 t03) need a
Retry after repeated restores
> - This pull request makes `turn.start` reopen a reopenable session
before it starts the turn
> - The benefit is that a suspended or restored Claude run keeps going
on its own, with no Retry
## Linked Issues or Issue Description
Refs #20089
**What happened?**
In a local restart-chaos G1 (approval) wave, Claude ACPX task t03 failed
after repeated restores with `ACPX session is not open` and needed a
Retry. The Retry overwrote the session directory, so the exact sequence
could not be read back. The same error reproduces directly with
`turn.stop` (suspension) followed by `turn.start` on the same session:
the stop clears the live session and sets the lifecycle to `prepared`,
and `turn.start` never reopens it.
**Expected behavior**
When the saved session can be reopened, `turn.start` reopens it, the
same way `session.open` does, and then starts the turn. "Can be
reopened" means: the conversation identity is recorded, the provider
exit is confirmed, the session belongs to this run, and the lifecycle is
`prepared`, `suspended` or `session_open`. A session that was closed on
purpose, or whose provider exit is unconfirmed, still fails.
**Steps to reproduce**
1. `run.prepare`, `session.open`, then `turn.start`.
2. `turn.stop` (suspension stop).
3. `turn.start` again. Before this change, it fails with "ACPX session
is not open".
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
c55af26821 |
fix(runner): treat a replayed start of a turn lost on restore as history (t48) (#20074)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native Codex run must survive a control-plane restart and continue its work > - #20057 made recovery adopt an in-flight turn, or start a new turn when the restored runner reports the old turn lost > - One case was not covered. A graceful drain can detach the controller after the runner accepted `turn.start` but before the provider started the turn > - The runner then records the provider's `turn.started` while no controller is attached. After a restart it replays that start, and the Codex session rejects it as an unexpected turn > - This pull request treats that replayed start as history from the earlier generation > - The benefit is that the run starts its continuation turn instead of stopping with `native_event_replay_conflict` ## Linked Issues or Issue Description Refs #20057 **What happened?** A restart chaos wave ended with one failed task (t48, Codex). A graceful SIGTERM suspended the controller 300 ms after the runner accepted `turn.start`. The provider's `turn.started` came after the controller detached. Recovery classified the run as `resume_dead_runner`. The restored runner reported the turn as `provider_turn_lost_on_restore`, which is correct. But it also replayed the undelivered `turn.started`. The Codex session failed with "Provider started an unexpected turn." (`turn_binding_mismatch`), and recovery stopped with `native_event_replay_conflict`. **Expected behavior** Recovery starts a new continuation turn. The replayed start of the lost turn is recorded as history, and the run completes. **Steps to reproduce** 1. Start a Codex native run. 2. Detach the controller gracefully after the runner accepts `turn.start` and before the provider emits `turn/started`. 3. Let the runner record `turn.started` while it is unattached, then kill the runner and the provider. 4. Restore the run. Before this change, the session fails with `turn_binding_mismatch`. **Paperclip version or commit** master at |
||
|
DottaandClaude Opus 5.5
|
c0c26188db |
fix(runner): stamp event generations and record an earlier generation's events as history (#19960)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute through the Paperclip Runner, which turns provider output (Codex, OpenCode, Claude through ACPX) into durable runner events > - In the restart chaos campaign, events from an earlier turn or runner generation failed the current turn about ten times (RC-15, 20, 20b, 20c, 33, 33b, 59 and 61) > - #19718, #19722, #19772 and #19783 each special-cased one variant, and other variants still fail the run > - The durable-ownership plan (section G, invariant 6) asks for one rule: every event carries its generation and turn, and an earlier generation's events are recorded as history and never fail the current turn > - This pull request stamps the runner generation on every event and applies one stale-event rule in the ACPX provider state, the Codex driver (which Runnerd also uses for OpenCode) and the pending `turn.start` check > - The benefit is that a late report about a settled or restored turn no longer stops a healthy turn, while events for a turn the runner never saw still fail closed ## Linked Issues or Issue Description Refs #19718, #19722, #19772, #19783 **What happened?** A runner or provider can report a turn after that turn settled: a resumed provider reports the lost turn's usage or terminal, an ACPX sidecar emits a last delta after the turn terminal, or a held failure of the previous turn sits in front of a new `turn.start`. Several of these paths fail the current turn: ACPX `AcpxEventScope` rejects any event for a non-active turn ("named a stale turn"), the Codex driver fails `turn_binding_mismatch` for a settled turn's late item between turns, and `runnerdPendingTurnStartFailure` reports any held non-recoverable failure, whatever turn it names. Runner events carried no generation. **Expected behavior** An event for a turn that is not active, and that an earlier generation accepted or this generation already settled, is recorded as history and never fails the current turn. An event for a turn the runner never saw still fails closed. **Steps to reproduce** ACPX: settle turn 1, start turn 2, and deliver a late `runtime.event` for turn 1. Before this change, the provider state returns an error and the run fails. ## What Changed - Runner (`durable/state.rs`, `durable/runner.rs`): every event payload carries `generation`, the runner process generation that emitted it. A runner process that recovers state starts the next generation (`begin_recovered_generation`); it is saved with the next normal write, so recovery adds no write before the first turn. Loading stays pure. Old saved state loads as generation 0. Outbox events keep their original generation. - PRP event schemas v1–v3 declare the optional `generation` property; generated validators and the manifest are regenerated. - ACPX (`acpx_event_scope.rs`, `acpx_provider_state.rs`): an event that names a settled turn of this run becomes a bounded `stale_generation_event` diagnostic (32 per state) and changes no state. Unknown turns still fail. - Codex driver (`codex-session-state.ts`, `codex-session-notifications.ts`): one predicate, `isHistoryTurn()`: the turn is not active, and it was restored from the checkpoint (an earlier generation) or settled in this generation. The classifier's stale-turn branch, the resume-usage branch, restored-terminal replay and the main fail point (`notificationNamesActiveTurn`) all use it. A terminal for an earlier generation's turn is history even when it disagrees with the checkpoint. A terminal that contradicts a turn this same generation settled is still a `conflicting_turn_terminal` integrity failure, and a provider that starts a settled turn again is still rejected. - Transport (`runnerd-codex-transport.ts`): a pending `turn.start` ignores a held failure that names another provider turn. - `DURABLE_TRANSPORT.md` documents the stamp and the rule. ## Verification - `cargo test -p paperclip-runner-core`: new `records_late_events_for_a_settled_turn_as_history` (red before), `stamps_every_event_with_the_runner_process_generation`; two existing ACPX tests now assert "recorded as history, no state change" instead of "rejected". Durable tests: 112 passed. `cargo fmt` clean. - `codex-app-server-driver.stale-generation.test.ts` (new, 9 tests): late item, delta, terminal and workspace change for a settled turn during and before the next turn, an earlier generation's conflicting terminal after recovery, and an unknown turn that still fails. Red on master: 8 of 9 (the during-turn cases on master already dropped the event, with a different diagnostic; the between-turn and recovery cases failed the session). - `runnerd-pending-turn-start-failure.test.ts` (new): an earlier turn's held failure no longer fails the pending start; the pending turn's own failure and a failed run terminal still do. - `vitest run src/drivers/codex src/live/runnerd-codex-transport`: 787 passed; 2 Codex executable-resolution tests timed out under host load and pass when run alone. - `tsc -p tsconfig.json` for the runner package, `check:protocol-types`, `check:replay-goldens`: clean. - Not run: chaos waves (the coordinator schedules them on this host). ## Risks - A late report about a settled turn is now dropped with a diagnostic instead of failing. A provider that keeps working on a settled turn could have its output ignored silently; a provider that starts a settled turn again, or names a turn the runner never saw, still fails closed. - `generation` is a new optional field. Consumers that do not know it ignore it, as the PRP contract requires. - The generation counter is written with the next normal state save, so there is no new write before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
fda58d8cc1 |
fix(chat): post the complete final reply, not a 4,000-character run-log preview (#19971)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People talk to agents in chat. The agent's final message becomes the chat reply. > - The runner cuts every completed agent message to a 4,000-character run-log preview. The server then posts that preview as the reply. > - A long answer, for example a list of 501 open tasks, reaches the person cut off with "…[truncated]". > - PR #19959 fixed only the case where the accepted summary starts with the exact preview text. It missed runs where the final prose differs from the summary. > - This pull request keeps the complete final answer in the runner, and makes the server never prefer a preview over a complete source. > - The benefit is that the person receives the agent's complete reply. ## Linked Issues or Issue Description Supersedes #19959. **What happened** A chat agent wrote a final answer of about 4,985 characters (all 501 tasks). The posted chat reply was a 4,000-character preview that ended with "…[truncated]". **Expected behavior** The person receives the agent's complete final message. A preview of a message is never posted when a complete text exists. **Steps to reproduce** 1. Ask a chat agent (OpenCode or Codex) to list a large set of open work, so that its final answer is longer than 4,000 characters. 2. Let the run finish. 3. Read the chat reply. It ends with "…[truncated]" at 4,000 characters. Root cause: - `packages/paperclip-runner/runner/crates/runner-core/src/provider_events.rs`: the codex/OpenCode facade projection of `item/completed` and the ACPX `AssistantMessage` projection cut the final answer with `bounded_text(text, MAX_TEXT_CHARS)` (4,000 characters). - `durable/state.rs` `sanitize_value` caps every string again at 4,096 bytes before the event goes to the server. - `server/src/services/heartbeat-run-summary.ts` `resolveHeartbeatRunResponse` gives an explicit `final` agent message top precedence, so it posts the preview. ## What Changed - Runner (`provider_events.rs`): a completed `final_answer` agent message keeps its complete redacted text, up to a new 32,000-character `MAX_FINAL_AGENT_MESSAGE_CHARS` bound. Progress prose and in-flight items keep the 4,000-character preview. - Runner (`durable/state.rs`): `preserve_final_agent_message_text` keeps the complete redacted text of a completed final agent message when it fits in half of the transport frame. Otherwise the marked preview stays, so a large message never fails the frame check. - Server (`heartbeat-run-summary.ts`): `readCompletedAssistantMessageCandidate` marks a stored preview as `bounded`. The resolver never posts a bounded preview while a complete source exists. It posts a marked adapter final response or the accepted semantic summary instead (`bounded_final_agent_message_replaced_by_semantic_summary`). It posts the marked preview only when no complete text exists (`bounded_final_agent_message_without_complete_source`). This also fixes replies from runners that are already deployed. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-run-summary.test.ts`: 86 passed. Six new tests: a complete final message longer than 4,000 characters is posted in full; the old exact-prefix case posts the complete accepted answer; a preview whose prose differs from the summary posts the complete summary (also with chat precedence); a marked adapter final response wins over a preview; the marked preview is posted only when nothing complete exists; a stored preview candidate is marked `bounded`. Five of the six fail against the `master` resolver. - `cargo test -p paperclip-runner-core --lib -- final_answer complete_final`: 3 passed. Both new runner tests fail when the fix is reverted. - `cargo test -p paperclip-runner-core --lib`: 581 passed, 3 failed. The failures are `task_memory::tests::monitor_*` (OOM-kill monitor timing tests). They do not touch the changed code. - `cargo test -p paperclip-runner-core --test acpx_event_projection --test acpx_rich_events --test local_integrity_boundary_golden --test acpx_provider_events`: all passed. `cargo fmt --check` passes. - Evidence from the failed eval run: the `item.completed` final `agentMessage` text was exactly 4,000 characters (3,988 retained plus the marker). The `run.result.proposed` summary was complete at 4,985 characters. ## Risks - Larger durable events: a final answer can now be up to 32,000 characters (at most about 128 KB). The outbox keeps the preview when the text exceeds half of the frame size, so the frame check does not fail. - Redaction stays the same. The complete text goes through `redact_sensitive_text_values`, which the preview also used. - The server treats any final message that ends with "…[truncated]" as a preview. If an agent writes that text itself, the server posts the complete accepted summary instead. That summary is also the agent's own complete answer. - The runner part reaches users only after a runner release. The server part fixes replies at once when an accepted summary exists. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
f755361ffe |
fix(recovery): deliver a steered message the restart lost before the model read it (#19962)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can steer a running Codex or Pi turn: a comment goes into the live turn instead of waiting for the next run > - The provider acknowledges the steer at once, but the model reads it only at its next step, after the in-flight tool call > - A control-plane restart in that window kills the provider with the message in memory. Restart recovery starts a continuation turn that does not carry the message. The message is lost and no error shows > - The restart chaos harness showed this in 6 of 6 steered runs (Codex, K1, K2 and K6, 2 samples each): steer acknowledged, run succeeded, no reply to the steered message > - This pull request makes restart recovery deliver every acknowledged steer that the model did not read, exactly once > - The benefit is that manual Steer no longer loses messages on restart, and the planned automatic steering can ship on top of it ## Linked Issues or Issue Description **What happened?** A comment steered into a running Codex turn got no reply when a control-plane restart (SIGKILL, SIGTERM or crash loop) happened during the agent's tool call. The run succeeded, and the steered comment had no effect. **Steps to reproduce** 1. Start a Codex task that runs a 40-second command. 2. During the command, post a comment and click Steer. The provider acknowledges it. 3. Restart the control plane before the command ends. 4. Recovery resumes the turn with a continuation. The agent finishes the task and ignores the comment. **Expected behavior** The agent reads and handles the steered comment once, after recovery. **Root cause** The provider keeps acknowledged steering input in memory until its next model step. The restart continuation (`paperclip.native-continuation.v1`) only told the model to continue the current request, so the acknowledged input was gone. ## What Changed - `unreadSteeredMessageIds` (runner package): derives acknowledged-but-unread steers from run-log events in order. A `steering_acknowledgement` item adds the message id (once per id). A `userMessage` item in the same turn reads the oldest unread steer. Codex records each steered input as one `userMessage` item when it enters the conversation. A `userMessage` item in a later turn reads all earlier steers, because the continuation carries them. `turn.completed` reads all steers. - `readUnreadSteeredMessages` (server): reads those facts from `heartbeat_run_events` (only transport-validated events with a `sourceEventId`). It also reads the comment or interaction text for the same company and issue. The run log is the durable record of both "acknowledged" and "read", so no new column is needed. - The restart continuation now includes `steeredMessages` and one instruction to handle them, oldest first. The runtime reads them through a lazy `getUnreadSteeredMessages` option, only when a restart continuation starts. Native recovery wires it for every run. - `queuedInteractionSteeringBody` is extracted, so recovery rebuilds a steered interaction response with the same text the steer sent. - Fake Codex gets `--consume-steer`, which emits the `userMessage` item for an accepted steer, like real Codex. ## Verification - Red/green live test with real runnerd and fake Codex (`runnerd-restart-continuation.test.ts`, "delivers a steered message the restart lost exactly once"): start a held turn, steer it, SIGKILL runner and provider, restore and recover. - "unread": the continuation carries the message exactly once. Without the runtime change this test fails. - "read" (`--consume-steer`): the continuation does not carry it again. - Both: after the continuation completes, a later restart has nothing to deliver. - `steering-delivery.test.ts`: ordering, dedupe, initial-input, later-turn and completed-turn rules. - `unread-steered-messages.test.ts` (embedded Postgres): ignores events without a source id and messages from other issues. It reads interaction text, ignores model-output items as read evidence, and returns nothing after the continuation input. - `pnpm vitest run src/steering-delivery.test.ts src/live/runnerd-restart-continuation.test.ts` (runner) and `pnpm vitest run src/services/native-runtime/unread-steered-messages.test.ts` (server): pass on the rebased revision. - `issue-queued-comments-routes.test.ts` and `native-session-runtime.test.ts -t restart`: pass. `native-session-resume.test.ts` "archives a damaged prior epoch…" fails locally on plain `origin/master` too (local runnerd binary); CI runs it with a fresh build. - Restart chaos (Codex steer mode, K1/K2/K6) runs after this lands in the harness slot. Before: 0 of 6 steered messages answered. ## Risks - Duplicate delivery: a provider that records steered input without a `userMessage` item, or a `userMessage` event lost in the crash window before persistence, makes recovery send the message again. The model then sees it twice. That is safer than losing it, and the instruction asks the model to handle each one once. - Pi (ACPX) steering has no `userMessage` read evidence today, so a Pi steer that the model already read before a restart may arrive again in the continuation. - Only the restart continuation carries the messages. Other recovery paths start after a completed turn, which already read every steer. - No schema or migration change. A session saved by an earlier release resumes normally; its run log already holds the acknowledgement events. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Can H. Tartanoglu
|
410aa6d1b4 |
test(runner): resolve provider fixture signals through PATH (#14476)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its native runner tests verify exact provider shutdown and recovery. > - Four fixture call sites invoke `/bin/kill` directly. > - NixOS supplies `kill` through PATH, so those tests fail before checking process state. > - This pull request resolves `kill` through PATH, as the neighboring runner tests already do. > - The same stop, probe, and resume assertions can run on non-FHS hosts. ## Linked Issues or Issue Description **What happened?** Four Codex provider tests fail with `No such file or directory` on NixOS. Each failure comes from a test-owned `/bin/kill` invocation. The executable is installed on PATH. **Expected behavior** Tests should probe and signal their disposable provider processes with the installed `kill` command. **Steps to reproduce** On a NixOS host with `kill` on PATH and no `/bin/kill`, run the `codex_provider` Rust integration target. Three `durable_stop_*` cases and `structured_question_round_trips_through_the_normalized_backend` fail. All four failures were reproduced separately on unmodified `e9debd3eac328e6abe87ab89e9ff40ee2bb56b73`. Related: these failures also blocked the broader local qualification of #14474. This fixture fix has no dependency on that runtime change. ## What Changed - Replace four test-only `/bin/kill` executable paths with `kill`. - Retain every signal, PID/process-group operand, assertion, and unwind cleanup. - Match existing PATH lookup in this file and `tests/process_supervisor.rs`. ## Verification Rebased onto master `820c458465` (single commit `ad1fb1043`). - `cd packages/paperclip-runner/runner && cargo test -p paperclip-runner-core --test codex_provider --locked -j4 -- --test-threads=2`: 94 passed, 0 failed, 2 ignored (subprocess helper entries that their parent tests run). The four tests that used `/bin/kill` pass on a host without `/bin/kill`. - No production code or documentation changes. - Hosted CI: `General tests (workspaces-b)` timed out in `packages/db/src/backup-lib.test.ts` and `e2e shard (1/8)` hit `ERR_STREAM_UNABLE_TO_PIPE`. Neither area is touched by this test-only diff. Master pushes currently run the "CI profile skipped" workflow, so there is no master run of those jobs to compare against. ## Risks Low risk. Tests require a working `kill` on their test PATH. Fixture PIDs and arguments stay unchanged. ## Model Used - Anthropic Claude Sonnet 5.5 (`claude-sonnet-5-5`) through Claude Code with tool use and code execution, for the rebase and verification. Original change: OpenAI `gpt-6-astra` through OpenCode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandClaude Opus 5.5
|
dcaada5f59 |
fix(native): carry each warm run's own completion contract (#19874)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A warm runner (`lifecycleMode: warm`) keeps runnerd and the provider
process alive between runs, so a follow-up message can start fast.
> - Each run has its own completion contract. The revision changes with
each follow-up, and the agent must name that revision when it finishes.
> - On a warm attachment the transport rebuilt the `run.attach` payload
from its constructor options. Those options belong to the first run.
Every later run kept the first run's contract.
> - The OpenCode proxy checks the finish claim locally against that
stale contract. Every warm OpenCode follow-up was refused ("Semantic
result completion contract revision does not match"), and the agent
looped until the run timed out.
> - This pull request carries each run's own contract into the warm
attachment, and lets runnerd adopt it without a cold provider resume.
> - The benefit: warm follow-ups finish correctly, and they keep the
live provider process, which is the reason to be warm.
## Linked Issues or Issue Description
Refs #19852 (startup sub-spans), #19846 (TTFT baseline).
**What existing behavior does this improve?**
Warm cross-run attachment for native Codex and OpenCode agents.
**Current behavior**
In a local measurement wave with `lifecycleMode: warm`, 4 of 4 OpenCode
follow-ups looped for 8+ minutes. Each finish call was refused with
"Semantic result completion contract revision does not match". The saved
`runAttachTemplate` after the attachment still had revision `1`, while
the follow-up turn named revision `2`. Codex hid the defect because its
finish claim goes through the controller's tool path, which uses the
current input.
**Proposed behavior**
The attachment carries the attaching run's contract. Runnerd updates the
contract in place for Codex and OpenCode and keeps the provider process.
The OpenCode proxy receives the current contract with the run identity
on `turn/start`.
**Reason and benefit**
Warm follow-ups are the largest TTFT lever in the baseline. A
correctness-only fix (cold-resume the provider when the contract
changes) would make every warm follow-up relaunch Codex or OpenCode,
because the revision changes on every follow-up.
## What Changed
- `attachRun` carries `completionContract` through `NativeSession`, the
harness backend, the Codex harness session and the runnerd transport.
`executeNativeSession` passes the current input's contract. The
transport uses it instead of its first-run `resumeCompletionContract`.
- Runnerd (`codex_provider.rs`, `attach_run_in_place`): a changed
contract no longer forces a cold resume for `codex` and `opencode`.
Codex app-server never receives the contract, and the OpenCode proxy now
receives it per turn. Other providers keep the cold-resume boundary. A
changed tool set still forces it.
- Runnerd sends `paperclipCompletionContract` with each OpenCode
`turn/start`, next to the existing `paperclipRunId`.
- The OpenCode proxy passes that contract to `attachRun` when the run
changes. `OpenCodeHarnessSession.attachRun` replaces its envelope's
contract and validates it.
## Verification
- `src/live/runnerd-codex-transport-warm-attachment.test.ts` (new case):
real runnerd and fake Codex. After a warm attachment with a new
contract, the saved `runAttachTemplate` has the new revision and the
Codex pid is unchanged.
- `src/cli/opencode-proxy-input.test.ts`: the real proxy bundle serves a
warm second run with revision `proxy-v2`. The second
`paperclip/runResult` is produced and names `proxy-v2`. Without the fix
the finish call is refused.
- Each test fails without its part of the fix: the proxy test without
the proxy change; the transport test without the transport change (stale
saved revision); and, with `PAPERCLIP_ATTACH_TRANSITION_RUNNER` pointed
at a runnerd built without the `codex_provider.rs` change, the Codex pid
changes.
- `cargo test -p paperclip-runner-core --lib warm` (11 tests),
`native-session-runtime.test.ts` (232), the OpenCode driver, backend and
warm-attachment suites, and `tsc --noEmit` for the runner package pass.
## Risks
- Previous-release story (`rules/upgrade-safe-run-state.mdc`): no
persisted shape changes. Saved `runAttachTemplate` values carry forward,
and the next attachment writes the current contract. A new runnerd with
an old OpenCode proxy behaves as today: the proxy ignores the new
parameter and keeps the stale contract. An old runnerd sends no
parameter, so a new proxy keeps its launch contract.
- In-place adoption widens the warm-retention path for Codex and
OpenCode to changed contracts. The idle, accounting and pending-work
checks still apply first.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edit, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
a9ac2e2039 |
feat(native): split local runner startup into stage, launch and session-load spans (#19852)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a person messages an agent, the agent must start its turn fast. Time to first token (TTFT) and agent boot are first-class metrics. > - The Phase 0 baseline (#19846) found that local runner startup costs 1.6–17 s at p50 and up to 50 s at p95. It is the largest part of TTFT. > - The run log records that startup as one span (`runner.session.bootstrap` or `runner.session.resume`). We cannot see if the time goes to staging, to the `paperclip-runnerd` process launch, or to the provider session load. > - This pull request splits local runner startup into three sub-spans in the run log. > - The benefit: the follow-up work to cut follow-up-turn TTFT can measure each step before and after. ## Linked Issues or Issue Description Refs #19846 (TTFT baseline, finding 1 and gap "No sub-spans exist inside local runner startup"). Refs #19838. **What existing behavior does this improve?** The run-log performance spans for local native runner startup. **Current behavior** The local path records one `runner.session.bootstrap` or `runner.session.resume` span that covers runtime staging, the runner process launch and the provider handshake together. The remote path already records `runner.runtime.stage` and `runner.process.launch`; the local path does not. **Proposed behavior** The local path records three sequential child spans of `runner.session.startup`: `runner.runtime.stage`, `runner.process.launch` and `provider.session.load`. **Reason and benefit** The TTFT work needs to know which step dominates before it changes the runner lifecycle. Without these spans, a change to warm runner reuse cannot show which cost it removes. ## What Changed - `RunnerdCodexTransport` has a new optional `onStartupStep` callback. `#start` and `#resume` time three milestones and call it synchronously. A failing callback never changes startup. A nested startup (exact adoption from `thread/start`) keeps the outer clock. - `runner.runtime.stage`: launch preparation before the runner process exists. - `runner.process.launch`: spawn until runnerd accepts the run (`run.prepare`, or the recovery attachment on resume). An adopted live runner records no launch. This keeps the existing meaning in `summarizeStartupReuse`: a launch span means a fresh runner. - `provider.session.load`: `session.open` (or the recovery probe) until the provider identity is confirmed. - `recordNativeRunnerStartupStep` (`native-run-trace.ts`) records each step as a `run.performance.span` row with `mode` and `target: "local"`. It uses `trace.defer`, so the run-log write never delays `turn.start`. - The native executor passes the callback only for a local target. Remote launchers keep their own spans, so there are no duplicates. - `redaction.ts`: allowlist `runner.process.launch` and `provider.session.load` as native span names. - `doc/run-log-events.md`: new section "Local runner startup sub-steps". Data path (AGENTS.md §5.7): run log only, plus Observability spans when an operator sets an OTLP endpoint. The attributes (`mode`, `target`, `outcome`) are already in the closed span-attribute allowlist. No Telemetry change. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-codex-transport-startup-steps.test.ts`: a real runnerd and the fake Codex fixture. Bootstrap and a cold resume each report stage, launch and session load in order, with `ok` outcomes and no overlap. A throwing sink does not change startup. - `cd server && npx vitest run src/services/native-runtime/native-run-trace.test.ts src/__tests__/redaction.test.ts`: the rows have the correct parent, offsets, durations and failed level. A stalled run-log sink does not block the caller. The rows settle before `task.provider_session.measured`. The names survive redaction. - The existing `runnerd-codex-transport*` suites pass (412 tests). Under host load (load average 16–27), the first full run had 7 timing failures in suites that do not set the callback. Each failing file passed when I ran it again alone. `runnerd-codex-transport-settlement` passed 3/3 for the failing case alone. - `tsc --noEmit` passes for `packages/paperclip-runner` and `server`. ## Risks - Low risk. Without `onStartupStep`, the transport behaves as before: the clock is not created and each milestone call is an optional-chain no-op. - More run-log rows: three per local runner startup. - The step boundaries are controller-side observations. `runner.process.launch` includes the runnerd authentication and the `run.prepare` acknowledgement, not only the OS spawn. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
b5b972ceb1 |
fix(runner): stop provider tool processes that leave the owned process group (#19727)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The local runner (`paperclip-runnerd`) supervises each provider process, such as the ACPX sidecar for Claude, in its own process group > - Run teardown stops that process group, and nothing else > - Claude Code starts each Bash tool command in a new session (`setsid`). A command such as `nohup ./bench.sh & echo $!` leaves a job in that session, and the wrapper exits at once. The job is then a child of init and outside the owned group > - In a restart chaos storm, a run failed after a restart resume. Its tool job kept running beside the task's next run, in the same workspace, for more than four minutes > - This pull request makes the supervisor record every process that its provider starts, including processes that leave the group, and stop them all before teardown reports success > - The benefit is that a run's tools cannot outlive the run and cause duplicate side effects in the next run ## Linked Issues or Issue Description **What happened?** A restart chaos storm ran a local ACPX Claude task. After a restart resume, the agent ran `nohup ./bench.sh <nonce> > log 2>&1 & echo "started pid $!"` and then ended its turn without a result. The run failed with `native_finalization_missing`. The runner tore down the sidecar group, and the guardian killed the provider group. `bench.sh` was in the session that Claude Code made for the tool (pgid and sid = the exited wrapper, ppid = 1). It was never signalled. It kept forking `sleep 1` while the task's next run started and finished in the same workspace. **Steps to reproduce** 1. Run a local ACPX Claude task whose agent runs a Bash tool command such as `nohup ./bench.sh x > log 2>&1 & echo $!`. 2. End the run (any outcome, including failure after a restart resume). 3. Run `ps -axo pid,ppid,pgid,command | grep bench.sh`. The job is still running with ppid 1. **Agent adapter(s) involved** Claude Code (local ACPX). The gap is in the shared supervisor, so any provider whose tools leave the group is affected. **Root cause** `SupervisedProcess` (`packages/paperclip-runner/runner/crates/runner-core/src/process_supervisor.rs`) spawns with `process_group(0)` and stops only that group in `terminate_group`, `terminate_group_immediately`, `wait` and `Drop`. Its own comment said that a caller that must exclude escaped or re-parented descendants needs a separate lifetime fence. No caller had one. The ACPX sidecar transport (`acpx_sidecar_transport.rs`, `shutdown` and `poison`) and the ACPX provider guardian (`installation-integrity.ts`, which kills only its own group) therefore left every `setsid` tool job behind. The failure path after the resume was not special: the same gap applies to every teardown. **Expected behavior** When a run ends for any reason, every process that its provider or tools started stops, and teardown proves that they are gone. Identity comes from PID, group, session and start time, never from a command name. ## What Changed - Add the `paperclip-process-table` crate. It reads PID, parent, process group and start time for every process (libproc on macOS, `/proc` on Linux). `runner-core` forbids unsafe code, so the two checked libproc calls live in this small crate. - Add `process_lineage.rs` to `runner-core`. A sampler thread (every 50 ms) records each descendant of the supervised process by PID and start time. It also records each group and session that a descendant leads, with the leader's start time. A process belongs to the lineage when its parent belongs, or when it is in a recorded group or session and started no earlier than its leader. Groups and sessions that the supervised process only inherited (runnerd's session) are never recorded. A recorded ID whose PID now names a different process is dropped. - `SupervisedProcess` samples the lineage just before it signals, and after the leader is reaped it stops every live member: TERM, then KILL after the shutdown grace. It fails with an error until it proves that no member is alive (grace plus 5 s). This applies to `terminate_group`, `terminate_group_immediately`, `wait` and `Drop`. - If the lineage sampler cannot start, the supervisor stops the process it just started and reports the failure, so a provider never runs without tool cleanup. - Table reads and state updates happen under one lock, so an older read cannot erase members that a newer read recorded. - The process table reports processes that exist but cannot be read. A recorded member that cannot be read stays unresolved, and teardown fails until its exit is proven. - The Runner workspace now has two members. `scripts/runner-release-crate-cache.mjs` and `scripts/runner-debug-crate-cache.mjs` cache and mark current the process-table lib unit with the Runner crate (otherwise Cargo rebuilds the Runner crate after every restore). The debug contract accepts only this library-only member and still refuses any other workspace change. The workflow cache path lists and the debug clean step are updated to match. - `terminate_group_immediately` now accepts `EPERM` from the group signal when the leader has already exited. macOS returns `EPERM` for a group whose only member is the unreaped leader. The lineage pass then proves that no other member remains. ## Verification - `cargo test -p paperclip-runner-core --lib -- process_lineage tool_job immediate_group_fence` (macOS, local). - New real-process tests in `process_supervisor.rs`: a supervised shell starts a wrapper with `setsid` (Perl `POSIX::setsid`), the wrapper runs `nohup <loop> &`, writes the job PID and exits. The test asserts that the job is outside the owned group and re-parented, then calls `terminate_group`, `terminate_group_immediately` or drops the process, and asserts that the job is gone when the call returns. - Red: with the tracker disabled, all three fail with `tool job <pid> outlived its provider`. - Green: all three pass. - Unit tests in `process_lineage.rs` cover a re-parented job, a job that left its group but stayed in the recorded session, a session made after the first sighting, PID reuse, and no claim through the supervisor's inherited session. - `node --test scripts/__tests__/runner-release-crate-cache.test.mjs scripts/__tests__/runner-debug-crate-cache.test.mjs` (new case: the process-table library is cached with the crate, and any other member still needs review). - `cargo test -p paperclip-process-table` (includes an exited process reported as gone), and `cargo test -p paperclip-runner-core --lib` (570 passed). `acpx_provider_backend::tests::unconfirmed_suspension_state_becomes_recoverable_only_after_cleanup_and_suspend` fails sometimes under full-suite load. It fails at the same rate with the tracker disabled, so it is a flake that already existed. - Sampler cost: one table read takes about 0.4 ms for 1,200 processes on macOS (about 0.8% of one core for each supervised process). - The restart-chaos replay (`--replay w-20261107T023921-s7106/t67`) did not run. The local session was not permitted to load provider credentials. It is not proven against a live Claude run. ## Risks - Teardown now stops all the processes that a provider started, also intentional background jobs. This is the intended contract: nothing outlives the run. - A wrapper that lives less than one sample (50 ms) and orphans a job before the sampler sees it is not recorded. Claude Code's wrapper sources a shell snapshot and lives for seconds. On Linux, a child-subreaper would close this gap. That is not in this change. - When `runnerd` itself is killed with SIGKILL, this code does not run. The provider guardian still kills its own group, but escaped tool sessions are not stopped in that case. - Teardown can return a new error when it cannot prove the lineage stopped. Callers already handle `terminate_group` errors as provider shutdown failures. - The server does not yet gate the next run on the tool lineage of the prior run directly. It relies on the runner finishing its teardown before the runner exits. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
fff70fe49e |
fix(runner): relaunch a provider whose runner died before admitting it (#19784)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute through `paperclip-runnerd`. When the control plane restarts, recovery starts a new runnerd process, and that process restores the provider (for example the Codex app server) from durable state. > - Each provider launch writes a durable startup fence before it spawns the process. Admission clears the fence. The fence stops a second runner from launching a provider while an unadmitted one may still run. > - A restart chaos wave sent SIGTERM and then SIGKILL to the control plane in quick succession. The second kill landed after the restored Codex provider was spawned and before it was admitted. > - The fence stayed in state with phase `spawned`. Every later runner refused to restore. The recovery probe's `runner.drain` failed with "provider startup ownership remains unadmitted", and the task needed a manual Retry. > - This pull request lets a new runner retire that fence when the recorded provider process and its process group have provably exited. It keeps every other fence closed. > - The benefit is that a double restart during provider relaunch continues automatically. ## Linked Issues or Issue Description **What happened?** A Codex run was active. A graceful restart (K2) suspended it, and recovery started a new runnerd that began restoring the provider. That runnerd spawned the Codex app server (pid recorded, phase `spawned`). A hard restart (K1) 9 s later killed it before admission. At the next boot, recovery chose `resume_dead_runner` and queued its `runner.drain` probe. The new runnerd failed the probe with `command_execution_failed: provider startup ownership remains unadmitted`. The run ended `native_session_recovery_failed`, and the task needed a Retry. The kept provider state had `lifecycle: "turn_active"` and a `startupAttempt` with `phase: "spawned"`, `trigger: "restore"`, `processId` = `processGroupId`, and no recorded exit. That process no longer existed. **Steps to reproduce** 1. Start a Codex task on a local runner. 2. Restart the control plane gracefully during the turn. 3. Kill the control plane again while recovery relaunches the provider, before the provider is admitted. **Expected behavior** Recovery relaunches the provider and continues the run. **Actual behavior** Every later runner refuses to restore, and the run fails. ## What Changed - `provider_backend.rs`: after a runner loads durable provider state, `retire_orphaned_startup_attempt` clears the startup fence only when all of these hold: - the attempt is in phase `spawned` (neither admitted nor failed); - the session is still restorable (`session_open`, `turn_active` or `provider_exited`); - signal 0 returns `ESRCH` for both the recorded process and its process group. It records a `provider_startup_retired` diagnostic event with the attempt, persists the state, and lets the normal restore path relaunch the provider. - Intent-phase fences (OpenCode). Three OpenCode runs after a K6 crash loop, and other runs on master, stopped at phase `intent` with no recorded process. Under load, the runner spends seconds verifying launch artifacts after it records intent. An intent alone could not show whether a process was started. - A new intent now carries `beforeSpawn: true`. - A new `SpawnStarting` observation clears the mark immediately before every provider spawn site in `start_with_tools_observed`. That clear is fsynced, but it emits no extra diagnostic event. - An intent still marked `beforeSpawn` (in a restorable session) proves that no process was started, so the fence is retired. - An intent that reached the spawn, or one written before this mark existed, stays fenced. - A runner that predates the field rejects the state as unknown (`deny_unknown_fields`), which fails closed. - `validate()` only allows the mark on an intent or on a spawn-stage failure. - The fence stays closed for: - an intent that reached the spawn, or one without the before-spawn mark; - a failed startup (initialization or spawn); - a session that recovery closed; - a process or group that may still run; - a permission error or an invalid id. - The existing test `failed_authenticated_identity_commit_keeps_the_durable_startup_fence` now uses a live process group instead of the fixed pid 123. That pid might not exist on a CI host, so the test would no longer prove a running provider keeps the fence. ## Verification - Red before the fix: the new integration test `restart_relaunches_a_provider_whose_runner_died_before_admitting_it` (fake Codex app server; turn active; a `spawned` restore fence with an exited process group; then the recovery `runner.drain` probe) failed with `provider startup ownership remains unadmitted`, the incident error. After the fix it passes. It shows one more `thread/resume`, the retired diagnostic, and a cleared fence. - Unit tests (unix-only module): retirement for each restorable lifecycle, and for an intent still before the spawn. A `SpawnStarting` outside its intent is rejected. The fence stays closed for: - an intent that reached the spawn; - an intent without the before-spawn mark; - a real child that survives its killed leader (process-group probe, no stubs); - a failed spawn; - a failed admission; - a closed session; - a live provider; - a surviving process group. - `cargo test -p paperclip-runner-core --test codex_provider`: 119 passed. The startup event counts are unchanged, because the spawn-start mark emits no event. The existing fence tests (`failed_provider_startup_is_persistently_fenced_before_another_process_can_resume`, `durable_recovery_closes_a_provider_that_reopens_a_settled_turn`) still pass. - `cargo test -p paperclip-runner-core --lib`: 570 passed. `cargo check --workspace --tests` passes. - `cargo fmt --check` passes. ## Risks - Process ids can be reused. A reused id only makes a gone process look alive, so the fence stays closed. Retirement needs `ESRCH` for both the pid and the group. - A provider process that moved to a new process group is outside this check. Before admission the runner has sent it no turn, so it cannot repeat provider work. - Intent fences written by older runners, including the kept incident states, have no before-spawn mark. They stay fenced and still need the Retry. New launches carry the mark. - A runner-side change takes effect on local runners after the runner build. Daytona needs a new image. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, test execution). Default context window. Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
35d41f11bb |
fix(runner): keep Codex thread usage reported before a run's first turn as history (#19783)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A new Codex run can attach to a Codex thread that earlier runs used > - In restart storm 9, a comment-wake run attached to that thread after a restart. Before its first turn, it received the thread's usage from the earlier run > - The runnerd transport gave that usage an empty turn owner. The driver rejected it as an unowned authoritative notification and failed the run > - The task then stopped with `native_event_replay_conflict` and needed the next run to finish > - This pull request keeps usage that arrives before a run's first turn as history > - The benefit is that a run is not lost to earlier work's usage, and it is not billed for that usage ## Linked Issues or Issue Description Refs #19718, #19722 **What happened?** Restart storm 9 (seed 7111), task CHA-5 (Codex). Run A succeeded. A K2 restart fired. The comment-wake run B then started on a fresh runnerd and attached to the same durable Codex session. Before run B started a turn, runnerd committed a `usage.reported` event for the thread's earlier usage. The transport rehydrated it as `thread/tokenUsage/updated` with `turnId: ""`, because no turn was bound yet. `classifyCodexNotification` classifies an empty id as `invalid_authority`. The driver failed the run with "Provider authoritative notification thread/tokenUsage/updated did not name its execution owner." Recovery then stopped with `native_event_replay_conflict`. **Expected behavior** Usage that arrives before a run's first turn does not fail the run. If the session has no usage baseline yet, that usage is history, so the run is not billed for it. **Steps to reproduce** 1. Use fake Codex with `--usage-before-first-turn`. It reports thread usage right after `thread/start`. 2. Open a driver session over the real runnerd transport. Wait until runnerd commits the usage, then start a turn. 3. Before this change, the run fails with "did not name its execution owner". ## What Changed - `runnerd-codex-transport.ts` (`rehydrateRunnerdUsageNotification`): the transport omits `turnId` when no turn is bound. It no longer sends an empty owner. - `codex-session-notifications.ts`: usage reported before the session's first turn, with no baseline yet, sets the baseline (history). Usage during or after a turn keeps its existing accounting. - Fake Codex: new `--usage-before-first-turn` flag. - New test `src/live/runnerd-codex-pre-turn-usage.test.ts` with real runnerd, the real Codex driver and fake Codex. ## Verification - Red, run twice before the fix: the test failed with `NativeProviderTerminalFailure: Provider authoritative notification thread/tokenUsage/updated did not name its execution owner.` This is the live message. - Green, run three times after the fix: the turn completes. The run's usage `runDelta` is 7/2 (thread total 12/3 minus the 5/1 reported before the turn). - `npx vitest run src/drivers/codex src/live/runnerd-codex-transport src/native-session-runtime`: 991 of 992 tests pass. One settlement test timed out under parallel load. It passes alone (19/19), and this change does not touch it. - `pnpm run typecheck` in `packages/paperclip-runner` passes. `cargo fmt` is clean. - Not run: the live restart-chaos replay. It needs provider credentials, which this session does not load. ## Risks - Low risk. The accounting change applies only when no turn is active, no turn is pending, the session has no settled turns and there is no baseline. In that state, the usage cannot belong to this run's work. - The test reproduces the failing shape (usage committed before the first turn) with a fresh session. It does not reproduce the exact restart and run-attach sequence from the storm. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file editing). Default context window and reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
ee1e1a3994 |
fix(runner): adopt the turn a dying controller's turn.start already started (#19772)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote Runner hosts the provider (here Claude through ACPX) and
keeps working while the control plane restarts
> - A controller can die after the Runner accepts its `turn.start` but
before the controller checkpoints the accepted turn or emits
`turn.accepted`
> - The provider keeps running that turn. On reattach, recovery saw no
checkpointed active turn, and the Runner's replayed `turn/started`
failed the session with "Provider started an unexpected turn"
> - The run then failed with `native_event_replay_conflict`, replay was
blocked, and a finished issue moved from done to blocked
> - This pull request lets recovery adopt that turn when the Runner's
authenticated command result proves it is the controller's own turn
> - The benefit is that a controller crash loop no longer fails a run
that the provider is still doing correctly, and a foreign turn still
fails closed
## Linked Issues or Issue Description
**What happened?**
In a restart chaos run (Claude through ACPX on a remote Runner, crash
loop fault), a follow-up comment woke a new run. The controller sent
`turn.start` and the crash loop began about one second later. After the
restarts, the controller reattached to the same Runner and failed with
"Provider started an unexpected turn". The run failed with
`native_event_replay_conflict` and recovery required workspace repair.
**What did you expect?**
The new controller generation adopts the turn that the old generation
started, waits for its terminal, and finishes the run.
**Evidence**
The durable controller journal for the run shows command 4, `turn.start`
with `turnId: turn_lab_18ac…`, completed with `{ providerTurnId:
turn_lab_18ac…, status: accepted }`. The Runner emitted `turn.started`
for that turn 1.4 s after the command, after the old controller died.
The replacement's `session.snapshot` returned `activeProviderTurnId:
turn_lab_18ac…`. The controller's own checkpoint had no active turn and
the run had no `turn.accepted` event.
Related: #19711, #19691, #19718
## What Changed
- `codex-app-server-driver-impl.ts` `recoverSession`: when the
checkpoint has no active turn, no semantic result, and no autonomous
goal or disposition turn applies, adopt the single in-progress provider
turn if the transport proves that this run's latest completed
`turn.start` accepted that exact turn id. Any other active turn is not
adopted and still fails closed.
- `codex-session-state.ts`: a recovered session with an adopted turn
emits the missing `turn.accepted` before any of that turn's
notifications.
- `app-server-transport.ts`: new optional transport method
`controllerAcceptedTurnId()`.
- `runnerd-codex-transport.ts`: implement it from the durable command
journal (`controllerAcceptedTurnStartId`). The latest `turn.start` must
be completed and accepted, its result must name the requested turn id,
and no run attachment may come after it.
- `DURABLE_TRANSPORT.md`: document the adoption rule.
## Verification
- Red/green with the real `paperclip-runnerd` and the scripted ACPX
Claude sidecar: `npx vitest run
src/control-plane/acpx-controller-recovery.test.ts -t RC-59`. The old
controller sends `turn.start` and stops before the provider accepts it.
The provider accepts and starts the turn while no controller is
attached. A replacement controller adopts the live Runner through the
production transport and `CodexAppServerDriver.recoverSession`. Before
the fix: `activeTurnId` is null (the session then fails on
`turn/started`). After the fix: events are `turn.accepted`,
`turn.started`, `turn.completed` for the same turn, with one
`turn.start` to the provider.
- Driver tests: `npx vitest run
src/drivers/codex/codex-app-server-driver.recovery.test.ts -t
"controller that died"`. Adoption passes. With no accepted result, or a
result for a different turn, `startTurn` still fails with "Provider
started an unexpected turn."
- Journal unit tests: `npx vitest run
src/live/runnerd-codex-transport.test.ts -t
controllerAcceptedTurnStartId` (pending, failed, rejected, mismatched
turn, later attachment, later pending start all return null).
- Regression: `src/drivers/codex` (all files),
`src/live/runnerd-restart-continuation.test.ts`,
`src/live/runnerd-codex-transport-adoption.test.ts`,
`src/live/runnerd-codex-transport-release-resume.test.ts`,
`src/control-plane/acpx-controller-recovery.test.ts` pass. `pnpm run
typecheck:typescript` passes in `packages/paperclip-runner`.
- `pnpm verify` asks for broader coverage (exit 2). Required CI supplies
it.
## Risks
- Low to medium. The change only acts when the checkpoint has no active
turn and the Runner reports exactly one active turn. Adoption needs the
Runner's authenticated, run-scoped `turn.start` result for that exact
id. Without that proof, behavior does not change.
- If a `turn.start` result is still pending when the replacement reads
the thread, recovery does not adopt and fails closed as before.
- Not proven on Daytona. I did not run the paid replay.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Devin FoleyandPaperclip
|
2d4f80504e |
fix(runner): retain exact ACPX checkpoints after rejected turn starts (#19762)
Preserve the exact idle ACPX conversation when a matching turn.start response rejects work with retryable SESSION_RESUME_REQUIRED. Require verified identity, no pending work, confirmed process cleanup and the provider lifetime fence before normal drain and suspension can publish a reusable checkpoint. Unknown acceptance and arbitrary closed states remain closed. The real Runner regression completes the next turn on attempts 2 and 3, requires successful terminal status for the exact submitted turn, and checks expectedIdentity on both resume requests. Controlled wire faults reject failed, cancelled and missing-identity recovery. Validation: 193 affected Rust tests on unchanged Rust source/fixtures; 134 recovery integration tests, 14 environment-gated skips; 21 Claude effort tests; all server typecheck layers. Exact-head CI: 59 successful checks and statuses, 6 intentional skips. Greptile Apex 5/5, no unresolved findings. Hosted deployment and task acceptance remain separate. Fixes: #19751 Refs: #19366 Refs: #19745 Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Nicky Leach
|
167e39c4ef | fix(runner): keep warm run.attach events on ordinary accounting (#19707) | ||
|
DottaandClaude Opus 5.5
|
fc7e3a6503 |
fix(recovery): never replace a native session after a deterministic refusal (#19733)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs keep a saved provider session, and same-run
recovery resumes it after a failure
> - When resume fails and the checkpoint allows it, recovery archives
the saved continuity state and opens a fresh replacement session
> - Recovery used that path for every resume failure, including
deterministic refusals that left the saved state unchanged and repeat on
every retry
> - In five staging tasks a refusal (a protected launch argument, or a
predecessor tool call with no result) escalated into a replacement. The
remote archive move failed part way, the next attempt found no runner
state, and each task was blocked for workspace repair
> - This pull request stops before replacement after a closed-class
refusal, keeps the saved state, and hands the run to an operator with a
named reason
> - The benefit is that a deterministic refusal can no longer damage
saved state or open a replacement over an unsettled external action
## Linked Issues or Issue Description
Refs #19659, Refs #19516, Refs #19334
**What happened?**
Five Codex tasks on Daytona sandboxes failed on staging. The first
attempt failed with a deterministic refusal:
- `run.attach cannot change protected launch arguments` (four tasks,
fixed separately in #19659), or
- `native_runner_source_semantic_reconciliation_required` with a
predecessor `paperclip_call_assigned_tool` call still pending (one
task).
The same-run retry then opened a replacement session.
`archiveContinuityState` failed with
`runner_continuity_break_archive_failed` and no detail. The third
attempt failed with `runner_remote_state_unavailable: exit=1`, and the
task was blocked with "Workspace repair required".
**Steps to reproduce**
1. Save a native session whose checkpoint policy is
`allow_replacement_after_resume_failure`.
2. Make the resume fail with a refusal that does not change state (an
attach refusal, or a pending tool call).
3. Same-run recovery opens a replacement session and archives the saved
continuity state.
**Expected behavior**
A deterministic refusal stops the run for an operator. The saved state
stays intact. A run with a pending side-effecting call never reaches
replacement. Transport drops keep the existing retry behavior.
**Root cause**
`executeNativeSession` checked only the checkpoint's replacement policy.
The closed failure class (#19334) only had an effect when replacement
was forbidden. Attach refusals and pending tool calls had no class at
all, and the source-reconciliation refusal was a plain `Error`.
## What Changed
- Runner (`provider_backend.rs`, `durable/mod.rs`): `run.attach`
refusals that would change the durable Codex profile or a protected
launch argument carry the closed classification
`provider_attach_refused`. An attach blocked by
`durable_pending_tool_calls` carries `provider_side_effect_pending`.
These refusals already leave the saved state unchanged.
- Contracts (`harness-driver.ts`): new closed classes
`PROVIDER_ATTACH_REFUSED` and `PROVIDER_SIDE_EFFECT_PENDING`. A typed
`NativeRunnerSourceReconciliationRequiredError` replaces the plain
`Error` at the five `native_runner_source_*_reconciliation_required`
throw sites. `runnerRecoveryFailureClass` reads the authenticated
command-result classification or the typed error, never message text.
- Runtime (`native-session-runtime.ts`): after either class, recovery
never opens a replacement session, whatever the checkpoint policy.
- An attach refusal ends with the existing
`NativeProviderStateIncompatibleError`, which maps to the existing
operator path `native_provider_state_incompatible`.
- A pending side effect ends with the new
`NativeProviderSideEffectPendingError`.
- Server (`native-session-executor.ts`, `native-session-resume.ts`,
shared `EXECUTION_RECONCILIATION_CAUSES`):
`native_provider_side_effect_pending` is a permanent failure and an
execution-reconciliation cause, so the board gets a reconciliation
(record whether the action ran) instead of a generic Retry. The
reconciled continuation does not rebind the saved session, which still
refuses attachment. It starts a fresh provider session in the same
workspace. The saved session and workspace are unchanged.
- Archive (`archiveContinuityState`): a failed remote move now records
the bounded (512 characters), redacted exit status and output in
`runner_continuity_break_archive_failed: exit=… timedOut=… <output>`.
The remote move already ran before the local move on `master`, so a
failed remote move leaves the local state in place.
`restart-interrupted-turn.ts` accepts the detailed message for its
reattachment-loss retry.
- Docs: `doc/execution-semantics.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe the new
rule.
## Verification
Red before the fix, green after (each test first run against unchanged
source from `origin/master`
|
||
|
DottaandClaude Opus 5.5
|
d1ee25d0f7 |
fix(runner): fail a pending Codex turn start at once with the runner's own terminal failure (#19722)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane starts each Codex turn through runnerd and waits for the turn's start event (`#startTurnWork` in `runnerd-codex-transport.ts`) > - In restart storm 5, runnerd failed an accepted continuation turn before it started (`turn_binding_mismatch`, fixed at its source in #19718) > - The transport held that terminal for the pending start and waited the full 30 s bound > - It then reported "runnerd did not report the provider turn identity", which hid the real cause > - This pull request makes a pending start fail at once with the runner's own terminal failure > - The benefit is a faster failure that names its real cause, so recovery and operators act on the correct fact ## Linked Issues or Issue Description Refs #19718, #19699, #19608 **What happened?** Restart storm 5 (seed 7106), task t01 (Codex). Runnerd accepted the restart continuation turn. About 0.5 s later it committed a `harness.diagnostic`, a non-recoverable `turn.failed` (`turn_binding_mismatch`) and a failed `run.terminal`. The transport's event pump holds a turn terminal while a turn start is pending, so the start loop never saw it. The start waited 30 s, the #19608 adoption check found no active turn, and the run failed with "runnerd did not report the provider turn identity". **Expected behavior** When runnerd reports a final failure of the pending turn (a non-recoverable `turn.failed` or a failed `run.terminal`) before the turn starts, the start fails at once. The error names the runner's code and message. **Steps to reproduce** 1. Use fake Codex with `--foreign-usage-before-turn-started --turn-started-delay-ms 20000`. 2. Start a turn through the runnerd Codex transport. 3. Before this change, `turn/start` rejects after the full bound with "runnerd did not report the provider turn identity". ## What Changed - `runnerd-codex-transport.ts`: new `runnerdPendingTurnStartFailure()`. It finds a non-recoverable `turn.failed` or a failed `run.terminal` among the terminals held for the pending start and the committed events after the cursor. - `#startTurnWork` checks it while it waits for `turn/started`, before the adoption snapshot, and in the post-adoption wait. It throws `runnerd failed the provider turn before it started: <code>: <message>`. - The check runs only when the turn has not started. A recoverable terminal does not trigger it. The #19608 late-start adoption path is unchanged. - Fake Codex: new `--foreign-usage-before-turn-started` flag. It sends usage for a turn the runner does not own right after it accepts a turn. This makes runnerd fail the turn with `turn_binding_mismatch` before it starts. - New test in `runnerd-codex-transport-deadlines.test.ts` with real runnerd and fake Codex. ## Verification - Red before the fix: `npx vitest run src/live/runnerd-codex-transport-deadlines.test.ts` → the new test failed after 16.4 s with "runnerd did not report the provider turn identity". - Green after the fix: the start rejects in under 5 s with `turn_binding_mismatch`, and fake Codex saw exactly one `turn/start`. - The #19608 tests ("adopts a durably accepted turn whose start notification outlives the turn-start bound" and "does not adopt an accepted turn the runner no longer reports active") still pass. - `npx vitest run src/live/runnerd-codex-transport`: 9 files, 399 tests pass. - `pnpm run typecheck` in `packages/paperclip-runner` passes. `cargo test --bin fake-codex-app-server` passes. `cargo fmt` is clean. - `pnpm verify` exits 2 (broad input: the fake Codex binary). Required CI supplies the broad coverage. ## Risks - Low risk. The new failure fires only when runnerd itself already reported a final failure and no turn start was observed. In that state the start could not succeed, and before this change it failed after the bound anyway. - If a provider sends a non-recoverable terminal before its own turn start for a turn that later starts, the start now fails earlier. Codex sends `turn/started` before its terminal, and runnerd's terminal for a protocol failure is final, so I do not expect this. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file editing). Default context window and reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
42f6b3a870 |
fix(runner): keep a restart-lost Codex turn's late usage from failing its continuation (#19718)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Codex runs continue a turn lost to a control-plane restart once, in the same conversation (#19699) > - In restart storm 5, a Codex run crossed one graceful restart. The continuation started, but the task stopped and needed a Retry > - The runner log shows the cause. The resumed Codex thread sent the usage of the lost turn after it accepted the continuation turn. The runner rejected that usage as `turn_binding_mismatch` and failed the run > - Recovery marks the lost turn as settled in the durable ledger. It does not tell the live Codex provider, so the provider treats the old turn's usage as a foreign event > - This pull request records the lost turn as settled in the live provider too > - The benefit is that a restart-lost Codex turn continues without a Retry ## Linked Issues or Issue Description Refs #19699, #19608, #19501, #15297 **What happened?** Restart storm 5 (seed 7106). Task t01 (Codex, B5) crossed one K2 SIGTERM restart. Recovery resumed the dead runner. Runnerd restored the conversation and recorded `turn.failed` with `provider_turn_lost_on_restore` (recoverable). The control plane started the continuation (#19699). Runnerd accepted it as a new provider turn. About 0.5 s later, Codex sent `thread/tokenUsage/updated` for the lost turn id. The runner's durable state records these events: - `turn.accepted` for the new turn `01a1b3e2-…` - `harness.diagnostic` `turn_binding_mismatch`, method `thread/tokenUsage/updated`, `expectedTurnId` = new turn, `receivedTurnId` = lost turn `01a1b3e0-…` - `turn.failed` (`recoverable: false`) and `run.terminal` failed The control plane did not see `turn/started` and failed after 30 s with "runnerd did not report the provider turn identity". The task needed a manual Retry. **Expected behavior** Usage for the lost turn is history. The runner keeps it as a resume usage snapshot. The continuation runs, and the task continues without a Retry. **Steps to reproduce** 1. Start a Codex turn through the durable runner. 2. Kill the runner while the turn is active. Restore so that the provider reports no active turn (`provider_turn_lost_on_restore`). 3. Start the continuation turn. Let the provider send `thread/tokenUsage/updated` for the lost turn after it accepts the new turn. 4. Before this change, the runner fails the continuation with `turn_binding_mismatch`. ## What Changed - `provider_backend.rs`: when recovery finds that the previous active turn ended while the runner was offline, it now also restores that turn id into the live provider's settled-turn set. The durable ledger already settled it. The existing settled-turn path then turns its late usage into `paperclip/resumeUsageSnapshot` and does not bill its `last` usage to the run. - The code skips this step when the settled-turn epoch needs a rollover, the same as the durable ledger. - Fake Codex: new `--stale-usage-after-turn-start` flag. With durable turn ids, it sends usage for the previous turn right after it accepts the next turn, before `turn/started`. This matches the live Codex order. - New test `restart_continuation_ignores_late_usage_for_the_turn_lost_on_restore` in `tests/codex_provider.rs`. It uses the real durable Codex executor with fake Codex. ## Verification - Red before the fix: `cargo test -p paperclip-runner-core --test codex_provider restart_continuation_ignores_late_usage` failed with the same diagnostic as the live run (`turn_binding_mismatch`, method `thread/tokenUsage/updated`, `receivedTurnId: provider-turn-1`, `expectedTurnId: provider-turn-2`). - Green after the fix: the same test passes. The continuation emits `turn.started` for `provider-turn-2`, there is no `turn.failed` or `run.terminal`, and fake Codex saw exactly two `turn/start` calls (the original and one continuation). - `cargo test -p paperclip-runner-core`: all suites pass. `acpx_park_readiness::hosted_park_acpx_preserves_late_goals_and_rejects_foreign_or_busy_observations` failed once and passed on rerun. This change does not touch ACPX. - `cargo fmt --check` is clean. - Not run: the live restart-chaos replay of `w-20261107T023921-s7106/t01`. The local harness needs provider credentials that were not available to this session. ## Risks - Low risk. The change adds one already-ended turn id to the live provider's settled set. That set is only used to ignore stale events and to reject reuse of a settled id. So a provider that later reports the lost turn id as a new active turn is now rejected, as the durable ledger already does. - The change does not address a second, separate problem: after the runner failed the continuation, the control-plane transport waited the full 30 s turn-start bound and reported "runnerd did not report the provider turn identity" instead of the runner's `turn_binding_mismatch`. That message only applies when such a failure occurs. This PR removes this specific trigger. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file editing). Default context window and reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky Leach
|
03498f90ee | Add explicit Runner session release and resume support (#19683) | ||
|
DottaandClaude Opus 5.5
|
a894b11a2b |
fix(runner): retry a rejected runner suspension and log every cleanup quarantine (#19653)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run's close must prove that its runner durably suspended
before the session's next run may start. If it cannot, the runtime
quarantines the session (`NativeSessionCloseUnrecoverableError`)
> - In a restart chaos storm, an ACPX Claude run succeeded and its
normal close quarantined its own session. The follow-up run on that
session failed 2 s later with `native_session_cleanup_quarantined`, and
no control-plane restart came between the two runs
> - The close sent `runner.suspend` once, the ACPX sidecar rejected it
right after the turn completed, and the barrier gave up at the first
rejected command. No log line showed the quarantine or its cause
> - This pull request retries a rejected suspension within the close
deadline, and logs every cleanup quarantine with its cause when it is
created
> - The benefit is that a successful run no longer blocks its own
follow-up because of a one-time suspension rejection, and any remaining
quarantine can be traced
## Linked Issues or Issue Description
**What happened?**
Run A (ACPX Claude) succeeded. The run log of its close shows:
- `runner did not prove durable suspension before checkpoint`
- `native_session_settlement_incomplete
{"runnerSuspended":false,"providerDrained":true,"suspensionState":{"commandStatus":"failed","runnerLifecycle":"suspended",...},"semanticTools":{"pending":[]},"finalProviderState":{"activeProviderTurnId":null,"providerSettled":true}}`
The runner journal shows `runner.drain` completed, then `runner.suspend`
failed with `failed to suspend ACPX provider: ACPX sidecar command
session.suspend was rejected (retryable=false,
classification=unclassified)`. The follow-up run B on the same session
was created 200 ms after A finished and failed 2 s later at admission
(`retryQuarantinedSessionCleanups`, operator recovery branch). The
server log had no line for the quarantine. One Retry completed the task.
**Steps to reproduce**
1. Run an ACPX task whose sidecar rejects the first `session.suspend`
after the turn completes.
2. Start a follow-up run on the same session.
**Expected behavior**
The close asks again, proves suspension, and the follow-up runs. If a
quarantine is still needed, it is logged with its cause.
**Actual behavior**
The close fails on the first rejection, quarantines the session, and the
follow-up fails with no logged cause.
## What Changed
- `runner/.../durable/runner.rs` (`process_command`): a failed
`runner.suspend`, on first execution or replay, now keeps the runner in
`Continue` instead of `Suspend`. Before, the runner recorded `suspended`
over a provider whose session was still open, then exited. The RC-34
journal shows exactly that: runner lifecycle `suspended`, ACPX provider
`session_open`. A failed `runner.shutdown` still stops the runner.
- `durable-prp-control-plane.ts`: a failed or rejected `runner.suspend`
now lifts the connection's terminal-lifecycle fence, so the next pending
command (the fresh suspend) is delivered on the same connection.
Completed suspensions and every shutdown keep the fence. New test:
"delivers a fresh suspension after a rejected one on the same
connection". Before the fix, the retry was never sent and the test timed
out waiting for the frame.
- `runnerd-codex-transport.ts` (`awaitRunnerSuspensionBarrier`): after a
rejected `runner.suspend`, the barrier sends a fresh suspend command, up
to 3 attempts, 250 ms apart, inside the same close deadline. An exited
runner or a spent budget still fails closed. A rejection changes no
state: the runner returns before it records suspension.
- `native-session-runtime.ts`: new
`onNativeSessionCleanupQuarantined(listener)`. It is called when a
cleanup quarantine is created or escalates to operator recovery, with
company, run, issue, session, `operatorRecoveryRequired`, a bounded
cause (error name, code, message) and bounded settlement facts.
- `server`: `logNativeCleanupQuarantines` logs each quarantine as a
warning. The subscription is process-wide and installed once, so closes
that fail during shutdown, after HTTP closes, are still logged.
## Verification
- Red before the change: `pnpm --filter @paperclipai/paperclip-runner
exec vitest run src/live/runnerd-codex-transport.test.ts -t "rejected
suspension"`.
- "retries a rejected suspension…" got `false`.
- "stops retrying…" saw 1 attempt instead of 3.
- Green after the change: both pass.
- Rust: new test
`rejected_suspend_keeps_the_runner_serving_its_controller` covers a
failed suspend, its replay, and an accepted retry with a fresh command.
`failed_lifecycle_commands_replay_their_terminal_transition` now expects
`Continue` for suspend and `Shutdown` for shutdown. The `pnpm run
check:runner` unit suites pass (559). `acpx_park_readiness` has two
timing tests (`< 500ms`) that fail at this host load on unchanged master
as well.
- New runtime test: "announces every cleanup quarantine with its cause
when it is created" (operator-required quarantine, cause and settlement
reported once). `src/native-session-runtime.test.ts` passes (212).
- The suspension cases in `runnerd-codex-transport.test.ts` pass, except
"proves local suspension after an event backlog… (1024 suffix deltas)".
That case also times out on unchanged master at this host load (27–35);
I confirmed this by stashing the change and rerunning.
- Server `tsc --noEmit` passes after the runner package build.
## Risks
- A provider that keeps rejecting suspension now gets three tries
instead of one, within the existing close deadline. The final outcome
and quarantine are unchanged.
- The cause of the ACPX sidecar's first rejection is not proven. The
logged evidence fits the sidecar's "not at a safe suspension point"
guard (`turnId`, tools or inputs still set) racing the just-completed
turn. This change makes close tolerate it. It does not change the
sidecar.
- Log volume: one warning per quarantine.
- The Rust change ships in the runner binary. Hosted images need a
rebuild to pick it up.
- After a failed suspend, the runner stays alive with its provider. If
every retry fails, close still treats suspension as unproven and kills
the owned runner process, exactly as before.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Refs #19532, #19609.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
9a52a2dfeb |
fix(native): retry ACPX session.open startup timeouts and give cold Claude a 60 s bound (#19661)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs start an ACPX provider through Runner. `session.open` opens the provider session before any prompt. > - On a loaded host, a cold ACPX Claude `session.open` took longer than its 30-second bound. The controller reported `runner_local_connect_failed: PRP command session.open timed out`. > - The server classified this as the generic `native_session_interrupted` with `ambiguous_state`, so the task was blocked at once. A person had to click Retry, sometimes several times. In one restart-chaos storm, one task needed three Retry clicks, and on another task both Retry runs timed out too. > - The server already has a bounded same-run bootstrap retry for failures that come before provider work (#18854). It did not recognize this timeout. The 30-second bound also cut off normal slow startups. > - This pull request gives cold ACPX Claude a 60-second `session.open` bound, like Pi and OpenCode (#19487). It also retries a startup `session.open` timeout automatically on the same run, from clean state. > - The benefit is that a slow cold start on a busy host completes, or retries itself, instead of waiting for a person. ## Linked Issues or Issue Description Refs #18854 (bounded bootstrap retry for definite `session.open` failures), #19487 (60-second cold OpenCode admission), #19665 (the bootstrap proof that this change reuses). **What happened?** ACPX Claude runs failed with `runner_local_connect_failed: PRP command session.open timed out` about 35 to 40 seconds after they started. The coordinator recorded `native_session_interrupted` with `recoveryMode: "ambiguous_state"` and `providerSessionEstablished: false`. The task was blocked with a board-owned action. No provider event and no checkpoint existed. The host 1-minute load was 17 to 44, sometimes with no restart nearby. **Expected behavior** A normal slow cold start completes. A `session.open` that still times out during runner startup, with no provider event, no checkpoint and no turn, retries on the same run with backoff. At most two automatic attempts follow the first. A run where provider work may have started is never retried automatically. **Steps to reproduce** 1. Run ACPX Claude tasks on a host with a 1-minute load of about 25 or more. 2. Observe a run fail after about 35 to 40 seconds with `PRP command session.open timed out`. 3. Observe the task blocked with `ambiguous_state` and a manual Retry. **Paperclip version or commit** `master` at `6c7cc2c5d6`. ## What Changed The bound (evidence below): - `coldAdmissionTimeoutMs` gives cold ACPX Claude 60 seconds (live adoption keeps 30 seconds). The runner's Rust sidecar `session_open_timeout` and the sidecar's provider handshake (`acpxSessionHandshakeTimeoutMs`) give Claude and Pi 60 seconds too. All three bounds must agree, or the shortest one fails first. Claude's warm run attachment uses the same cold budget, like Pi. The automatic retry: - `nativeSessionFailureSourceCode` returns `provider_session_open_timeout` only for exactly `<startup failure code>: PRP command session.open timed out`. The startup failure codes are `runner_local_connect_failed`, `runner_direct_wss_failed` and `runner_ingress_unavailable`. The transport uses them only before startup completes. After startup the code is `provider_transport_failed`, which is unchanged. - `nativeProviderRecoveryEvidence` treats the cause as pre-session. Durable evidence still decides: a bootstrap retry needs no provider event (`session.started`, `turn.started` and the others) and no checkpoint. The existing budget applies: three attempts with backoff of about 30 seconds, then about 60 seconds. When the budget is used, the task is blocked with the existing "Provider startup failed on 3 attempts before any provider work began" action. - A timeout is indeterminate: the runner can open a provider session after the controller stops waiting. Live runs showed both leftovers, so the retry now starts from clean state: - `prepareNativeSessionOpenTimeoutRetry` (server) runs when the failure settles. It quarantines a local runner root only when the restart-recovery proof (`runnerdStateProvesIncompleteBootstrap`, #19665) shows that no turn began and every recorded process is gone. Any other root, and any remote runner, stays `ambiguous_state` as before. - `resetAbandonedBootstrapRecord` (runner ACPX adapter) runs on a fresh open (no runner-recorded identity). If ACPX has a saved record for the session key with no messages, no prompt and no events, it is marked `reset_on_next_ensure`. Claude saves nothing before the first prompt, so reusing such a record fails the first turn with `session_resume_required`. A record that holds any conversation is unchanged. - A chat retry of an exhausted `provider_session_open_timeout` run is accepted like the other pre-provider causes. - `doc/execution-semantics.md` and the Runner README describe the cause and the bound. ## Verification **Measured startup distribution** (ACPX Claude `runner.session.bootstrap` / `runner.session.startup` spans from restart-chaos run events, 1-minute load 8 to 44): | Source | Bound | Successful cold bootstraps (s) | Failures | | --- | --- | --- | --- | | Earlier storm evidence | 30 s | 2.9, 4.7, 7.4, 8.6, 8.7, 26.2, 27.4, 27.6, 29.2 | startups ended at 34.9 to 40.3 s (the 30 s `session.open` bound plus spawn and `run.prepare`) | | This branch, wave s3601 | 60 s | 6.3, 7.8, 8.1, 13.4, 14.4, 22.3, 22.8, 22.9, **37.3** | 0 `session.open` timeouts | | This branch, wave s3602 | 60 s | 13 samples, p50 10.5, p90 17.9, max 25.7 | 0 `session.open` timeouts | | This branch, wave s3610 (final head, load 28 to 42) | 60 s | 10.4, 11.3, 11.7, 12.0, 13.3, 14.2, 29.6, **30.4**, **30.9** | 0 `session.open` timeouts | Under load, successes clustered just under 30 s and failures sat just above it, so the 30 s bound was cutting the distribution off. With 60 s, cold starts of 30.4, 30.9 and 37.3 s succeeded that the old bound would have failed. 60 s is about 1.6 times the largest observed success, and it matches Pi and OpenCode. **Live restart-chaos waves** (local, ACPX Claude only, 4 in flight, `--rc06-masked`, real provider, K1/K2 restarts): - s3601 (8 tasks, load 19 to 32): 7/8 PASS. No `session.open` timeout. The one failure is an R3 Retry signature after K1 (`native_restart_recovery_blocked`), not this cause. - s3602 (8 tasks, load 17 to 39): 8/8 PASS. No `session.open` timeout. Three Retry clicks were for `native_provider_terminal_failed` (`provider_turn_lost_on_restore`), a different restart signature. - Forced timeout, s3609 (4 tasks, B5, K2): a local, uncommitted fault injection gave every other cold startup a 1.5 s `session.open` deadline. Result 4/4 PASS. Run 4fe6537a timed out, retried itself on the same run, and then succeeded through four K2 restarts with no person involved. Run 02799a85 hit the injection three times in a row; the budget stopped it after attempt 3, and one Retry completed it. - Before the two clean-state fixes, the same forced wave (s3607 and s3608) showed the retry reusing leftover state: `turn.start` failed with `session_resume_required`, and then "ACPX session is not open". Those runs are the red evidence for the two clean-state fixes. - Final head, s3610 (8 tasks, K1/K2, no injection, load 28 to 42): **8/8 PASS**, 0 `session.open` timeouts. The one Retry click (t06) was for `provider_turn_lost_on_restore`, a different restart signature. **Tests:** - Red before the fix: `native-startup-retry.integration.test.ts -t timeout` (embedded PostgreSQL, real executor, real resume claim). The 3 retry cases failed with `native_session_interrupted` / `ambiguous_state`. - Green: `native-startup-retry.integration.test.ts` passes 36/36. New cases: - for each startup failure code, a timeout retries on the same run with backoff, keeps the task `in_progress` with an agent-owned `resume_native_run` wake, and blocks after attempt 3 with a board-owned action; - no automatic retry when `session.started` or `turn.started` is durable, when the code is `provider_transport_failed`, or when another command timed out; - a real runner root with an opened session and no turn is quarantined; a root with a turn command stays `ambiguous_state` and intact; a remote runner is never prepared. - `codex-runtime-adapter.test.ts`: 113 pass. New cases: a 45 s delayed handshake is admitted for Claude and rejected for Codex; a never-prompted record is reset only on a fresh open; records with messages, a prompt or events are kept. - `abandoned-bootstrap-record.test.ts` (real `createAcpRuntime` with `createRuntimeStore` and a controlled ACP peer that, like Claude, cannot load a never-prompted session): without the reset, a fresh open reuses the record and the first turn fails with SESSION_RESUME_REQUIRED after `session/load`. With the reset, the provider receives `session/new` and the turn completes. - `vitest run src/drivers/acpx src/cli` in the runner: 107 files, 1,935 tests pass. - `native-session-executor.test.ts -t "session.open|startup|bootstrap"`: 144 pass. `runnerd-codex-transport.test.ts -t "cold process admission"`: 4 pass. `cargo test -p paperclip-runner-core --lib admission_budget` passes. - `npx tsc --noEmit -p server` and the runner `tsc -p tsconfig.test.json` pass. ## Risks - **This changes Runner Rust code. The Daytona runner image must be rebuilt after merge** for remote sandboxes to get the 60-second sidecar bound. Until then, a remote Claude `session.open` still fails at 30 seconds in the sidecar (as a definite failure, which already retries). A remote timeout stays operator-owned, because a remote root cannot be proven clean here. - A cold ACPX Claude start that hangs now fails after about 60 seconds instead of 30 seconds. - A timed-out `session.open` may have created an empty provider session. No prompt was sent. The retry quarantines that runner root and asks ACPX for a new session. - Same-run retries of a fresh-attempt Retry run can still fail with `isolated_task_directory_binding_mismatch`. This is RC-35 (workspace re-preparation), which has its own fix in progress. - No schema or API change. The new error code appears on runs and in run error codes. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
20fc37778f |
fix(runner): keep settled OpenCode sessions across a controller proxy rebuild (#19641)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native Runner keeps an OpenCode conversation in durable state,
so a task can continue after a control-plane restart.
> - Saved OpenCode state binds one digest of the launch profile: the
Node command, the proxy bundle and the OpenCode executable, each by path
and SHA-256.
> - The Node command and the proxy bundle are controller runtime. A
rebuild or deploy between a hard kill and recovery replaces their bytes.
> - The recovery probe then refused a settled conversation with
`provider_state_incompatible`, although the OpenCode release and the
conversation were unchanged.
> - ACPX already re-binds a settled session after a Runner upgrade
(#18879). OpenCode did not.
> - This pull request records the OpenCode release digest in saved state
and re-binds a settled session when that release is unchanged.
> - The benefit is that a task continues automatically after a restart
that also changed the controller build. A real release change still
fails closed.
## Linked Issues or Issue Description
Refs #19558, #19501, #18879.
**What happened?**
A restart-chaos wave killed the control plane with SIGKILL while an
OpenCode run was settled between turns. The first recovery attempt
restored the session. The controller's runner `dist` was then rebuilt in
the same checkout. The next recovery attempt failed in the
`runner.drain` resume probe with `OpenCode runner launch profile changed
across durable recovery` (`native_provider_state_incompatible`). The
saved state matched the original proxy bytes exactly. Only the proxy
bundle bytes had changed. The OpenCode executable, the paths and the
model were the same.
**Expected behavior**
A settled OpenCode session continues after a restart when only
controller runtime bytes changed. A changed OpenCode release, or a
session with work in flight, fails closed. The existing disposition then
offers one Retry that continues in a fresh provider session.
**Steps to reproduce**
1. Start an OpenCode run and let it settle between turns.
2. Kill the runner and the controller.
3. Replace the qualified proxy bundle with valid new bytes (rebuild or
deploy).
4. Recover the run. The resume probe fails with
`provider_state_incompatible`.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
e05bced39f |
fix(runner): adopt an accepted Codex turn whose start outlives the bound (#19608)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex runs submit each turn through runnerd. The transport
answers `turn/start` only after it sees the provider's `turn/started`
for the turn the durable command result named
> - That wait has a fixed 30 s bound
> - At host load around 50, after two hard restarts, Codex accepted a
turn and started it about 27 s after the command, beyond the bound once
scheduling delay was added. The transport failed the start, but the
provider kept running the turn and made a tool call
> - No controller owned that turn. Its close could not settle, and the
task's Retry was refused at once
> - This pull request adopts the accepted turn when the bound expires,
but only when the runner still reports that turn active
> - The benefit is that a slow but live provider turn stays owned, so
its events, Stop and close keep working
## Linked Issues or Issue Description
**What happened?**
In a restart chaos replay (two SIGKILLs about 20 s apart, local runner,
real Codex, load 50), a resumed run failed with `runnerd did not report
the provider turn identity`. The runner journal shows the durable
`turn.start` result with the provider turn id, then `turn.started` for
that same turn 27 s later, followed by items and a semantic tool call.
The run's close then stayed incomplete (`semanticResultsSettled:
false`), and a Retry of the task failed with
`native_session_cleanup_quarantined: prior session cleanup remains
incomplete`.
**Steps to reproduce**
1. Use a Codex provider that answers `turn/start` and emits
`turn/started` later than the transport's turn-start bound.
2. Start a turn through the runnerd transport.
**Expected behavior**
The accepted turn is owned and observed by the controller.
**Actual behavior**
The start fails after the bound, while the provider still runs the turn.
## What Changed
- `runnerd-codex-transport.ts` (`#startTurnWork`):
- Near the end of the bound, if `turn/started` has not arrived, the
transport reads an authenticated `session.snapshot`. It reserves at most
5 s (or a quarter of the bound) for this read.
- It binds the accepted turn only when the turn is provably live:
- the runner reports that exact provider turn as active;
- the runner attributed no provider launch
(`provider_startup_ownership`) to a command after this `turn.start`;
- for a local provider, the same process (pid and start time) that
accepted the turn is still alive.
- Otherwise it keeps waiting for `turn/started` until the configured
bound and fails exactly as before. A turn lost across a provider
relaunch keeps the same-run resume path.
- `fake-codex-app-server.rs`: new test options `--turn-started-delay-ms`
(a delay between the `turn/start` response and `turn/started`) and
`--exit-before-turn-started`.
## Verification
- Red before the fix: `pnpm --filter @paperclipai/paperclip-runner exec
vitest run src/live/runnerd-codex-transport-deadlines.test.ts -t "adopts
a durably accepted"` (real runnerd and fake Codex, 12 s start delay, 8 s
bound) failed with `runnerd did not report the provider turn identity`.
This is the replay's error.
- Green after the fix: the start resolves with `provider-turn-1`, and
`turn/started` and the turn's terminal follow in order. There is one
provider `turn/start`.
- Negative case (`does not adopt an accepted turn the runner no longer
reports active`): the provider accepts the turn and exits before
starting it. The runner relaunches the provider and still lists the
turn, but the turn is not adopted. The start fails within the bound plus
scheduling slack. A snapshot-only version of this change adopted that
dead turn, and it hung for 20 s with no terminal.
- `src/native-session-runtime.test.ts`,
`src/live/runnerd-restart-continuation.test.ts` and
`src/live/runnerd-codex-transport-deadlines.test.ts` pass. The package
typecheck passes.
- Live chaos replays: results are added in a comment.
## Risks
- A provably live provider that accepts a turn and never starts it now
waits on the turn's own timeout and Stop paths instead of failing after
30 s. A dead or relaunched provider still fails at the bound.
- Remote runners can adopt only on the runner's own evidence (active
turn and no later provider launch), because their provider pid is not a
local process.
- A stale `turn/started` for a previous turn that arrives after the new
command result is still rejected. The transport cannot attribute that
turn's following events, so rejecting it fails only the transport, and
the same-run resume continues the task (seen in the replay). Delivering
it would attach the old turn's items and usage to the new turn.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Refs #19532, #19539.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
e33aa8a46a |
fix(runner): let saved Codex threads follow the current feature policy (#19659)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip runner keeps a saved Codex thread so the next run of a
task continues the same conversation
> - On each new run, the controller sends `run.attach` with the current
Codex launch arguments, and the runner refuses any change to "protected"
arguments
> - #19516 removed `--disable image_generation` from the Codex launch
arguments, but every thread saved before that deploy still records the
flag
> - The runner treated the feature toggle as protected, so every
pre-deploy Codex thread failed with `run.attach cannot change protected
launch arguments`
> - Same-run recovery then tried to replace the thread, the replacement
failed, and the tasks became blocked for an operator
> - This pull request treats Codex feature toggles as controller launch
policy, so a saved thread adopts the current toggles and restarts on the
same thread
> - The benefit is that a release that turns a Codex feature on or off
no longer strands every saved Codex thread
## Linked Issues or Issue Description
Refs #19516
**What happened?**
After a staging deploy that included #19516, Codex agent runs on tasks
with a saved thread failed at the start. The run log showed
`native_session_recovery_failed: PRP command run.attach failed:
run.attach cannot change protected launch arguments` (error code
`native_session_interrupted`). Same-run retries then failed with
`runner_continuity_break_archive_failed` and
`runner_remote_state_unavailable`, and the tasks were blocked with
"Workspace repair required". Four tasks on one Codex agent showed this
first failure. Each one last ran successfully before the deploy.
**Steps to reproduce**
1. Run a Codex task so that its thread is saved with the launch
arguments from before #19516 (`--disable image_generation`).
2. Deploy a revision that includes #19516.
3. Wake the same task again. `run.attach` sends the current arguments
without the toggle, and the runner rejects them.
**Expected behavior**
The saved thread continues under the current launch policy.
**Root cause**
`CodexCommandExecutor::stable_launch_args` drops only the per-run
filesystem and shell-environment entries before it compares saved and
current arguments. All other arguments, including Codex
`--enable`/`--disable` feature toggles, must be identical. The
controller computes the toggles from Paperclip policy for every launch,
so any change to that policy made every saved thread impossible to
attach.
## What Changed
- `provider_backend.rs`: `stable_launch_args` also skips `--enable
<feature>` and `--disable <feature>` pairs and the
`--enable=<feature>`/`--disable=<feature>` form. The function contract
comment now lists the rebindable arguments. `run.attach` then adopts the
controller's current toggles. Because the arguments changed, the
existing path stops the provider and restarts it on the same thread.
- New test
`saved_thread_adopts_the_controllers_current_codex_feature_policy`:
- A thread saved with `--disable image_generation` attaches with the
current arguments, keeps its thread ID, and records the new arguments.
- A new `--disable` toggle (a tighter policy) is also adopted.
- A new `--enable` toggle is adopted in both flag forms, and the thread
ID stays the same.
- A sandbox bypass flag, a different `default_permissions` profile, and
a different subcommand are still rejected with "protected launch
arguments", and the saved state does not change.
## Verification
- Red on `master` (
|
||
|
DottaandClaude Opus 5.5
|
f4b19e4231 |
fix(runner): finalize a durable proposed result when restore loses its Codex turn (#19658)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Codex runs survive control-plane restarts: runnerd restores the provider conversation, and the control plane continues the task from its checkpoint > - A hard kill can land after the agent proposed its result (`run.result.proposed`) but before the provider turn ended > - The restored runner then proves that the turn ended with the accepted result. It records `session.reconciled` and `run.terminal`, but no provider turn terminal > - The Codex recovery read did not see that turn as ended, so recovery failed with "persisted active turn … is missing from thread/read". Retries ran out, and the Retry re-ran finished work > - This pull request lets recovery settle that turn with its durable result > - The benefit is that a task that already finished its work continues to finalization after a restart, instead of failing and redoing the work ## Linked Issues or Issue Description Refs #19525, #19591, #19539, #19501 **What happened?** In a restart storm, a native Codex task (B1) proposed a complete `paperclip.run_result.v1` with passing tests. The control plane accepted the completion tool call. About 9 s later, a K1 (SIGKILL and restart of the control plane) killed the controller, runnerd, and the Codex app-server. After startup, `resume_dead_runner` resumed the same run two times. Each resume spent about 25 s in `runner.session.startup` and failed with `native_session_recovery_failed: HarnessReconciliationError: persisted active turn <id> is missing from thread/read`. Then `native_session_retry_exhausted`, replay was blocked (`preserve_without_replay_v1`), and the run ended `native_session_interrupted`. The task needed a Retry, which re-ran the finished work. **Steps to reproduce** 1. Start a native Codex task through runnerd. 2. Let the agent call `paperclip_finish`, and let Paperclip accept the result. 3. Kill the controller, runnerd, and Codex before the provider turn ends. 4. Let restart recovery resume the run. Durable evidence from the kept instance: - The run events have `run.result.proposed` for the turn, then a `completion.tool_accepted` span, then the kill. - The saved checkpoint has `activeTurnId` set to that turn and `semanticResult: null`. - The Codex rollout on disk does contain the turn. The provider did persist it. - The runnerd journal for the first resume has `session.reconciled {previousProviderTurnId: <turn>, activeProviderTurnId: null}` and then `run.terminal {turnTerminalState: completed, runTerminalState: succeeded, reportedWorkDisposition: done}`. It has no `turn.*` event for that turn. The runnerd Codex provider state shows `completedTurnAuthoritative: true` and `completedProviderTurnId: <turn>`. **Expected behavior** Recovery finalizes the accepted result. It does not re-run the task. **Actual behavior** The runnerd Codex transport builds the recovery `thread/read` reply from runnerd's journal. It looked only for `turn.completed|failed|interrupted|cancelled` events that name the turn. Runnerd's "turn ended with an accepted result" branch (`provider_backend.rs`, `recovered_turn_ended_with_result`) emits only the reconciliation and the run terminal. So the reply omitted the turn, and `CodexHarnessSession.reconcile()` threw at `codex-harness-session.ts:796`. The second resume read the same journal and failed the same way. ## What Changed - `runnerd-codex-transport.ts`: a new `runnerdRecoveredTurnTerminal` helper selects the terminal for a restored turn. A provider `turn.*` terminal that names the turn wins, as before, with its cause. A `run.terminal` that follows a `session.reconciled` that names the turn as the ended previous turn (no active turn, no later `turn.started`) now also settles that turn, with the run terminal's `turnTerminalState`. The `thread/read` wait loop uses this helper. - `codex-harness-session.ts`: `reconcile()` now settles a checkpointed active turn that provider history no longer contains when the session holds the semantic result that this exact turn proposed (`resultTurnId` equals the turn). Recovery restores that result from the durable `run.result.proposed` event. Without such a result, a missing turn still fails closed. - `codex-harness-session.ts`: the same rule applies when the restored runner reports that turn `failed` with `provider_turn_lost_on_restore`. This happens when the runner cannot prove the accepted result itself. Later recoveries keep the settled outcome. Without a held result, a restore-lost turn keeps its existing failure and same-conversation continuation. - `fake-codex-app-server`: a test-only `--hold-after-terminal-result` flag keeps the turn open after an accepted completion report. - `doc/execution-semantics.md`: one additive paragraph on this recovery rule. ## Verification - Red before the fix, real `paperclip-runnerd` and fake Codex: `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-restart-continuation.test.ts -t "durable proposed result"` failed with `native_session_recovery_failed: HarnessReconciliationError: persisted active turn provider-turn-1 is missing from thread/read` at `native-session-runtime.ts:2267`. This is the same stack as the incident. - Green after the fix: the run finalizes on `provider-turn-1` with the proposed result and `runTerminalState: succeeded`. It has no `session.failed` or `turn.failed`, and Codex received only one `turn/start` (no re-run). With the transport fix alone, the test passes in about 5 s. With only the session fix, it also passes (about 30 s, through the 5 s read wait). - Driver red/green: `src/drivers/codex/codex-app-server-driver.recovery.test.ts -t "missing persisted turn"`. The settle case failed before the fix (`recovered: false`) and passes after it. The guard case (the held result belongs to another turn) still fails closed. The existing "fails recovery explicitly when the persisted active turn is missing or replaced" test still passes. - Driver red/green for the restore-lost case: `-t "restore-lost turn"`. The held=true case failed before the fix and passes after it. The held=false case still maps `turn.failed` with the lost cause. - Transport unit test: `runnerd-codex-transport.test.ts -t "reconciled result terminal"` covers the run-terminal rule and its negative cases. - Suites: `src/drivers/codex/`, `src/native-session-runtime.test.ts`, `src/backends/`, `src/live/runnerd-restart-continuation.test.ts`, and `src/live/runnerd-codex-transport.test.ts` pass. Some load-sensitive tests time out at host load 25–30 and pass when rerun alone. The two `proves local suspension after an event backlog … (1024 suffix deltas)` cases also time out on unmodified master. - `pnpm --filter @paperclipai/paperclip-runner typecheck:typescript` passes. - Live chaos replay with real Codex: K1 0–3 s after `run.result.proposed`, 3/3 PASS (R1). Each case resumed the same run, which ended `succeeded`. No Retry was needed and no work was re-run. Details are in the comment below. ## Risks - The transport rule accepts a run terminal only directly after runnerd's own reconciliation that names the ended turn with no active turn. Runnerd emits that pair only after it restored the conversation and proved the turn ended. A provider turn terminal still wins. Other run terminals do not settle a turn. - The session rules use only a result that Paperclip already holds for the exact turn. The control plane accepted that result before the kill. A later provider terminal for that turn cannot change the outcome. The finalizer can already interrupt a provider after an accepted result. - A missing turn without a durable result still fails closed. Its outcome is unknown, and the existing Retry stays available. - No profile, schema, pack contract, or database changes. The Rust change touches only the fake Codex test binary. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
374a918d67 |
fix(runner): retain qualified OpenCode artifacts for cold recovery (#19622)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native recovery must keep admitted provider state and useful workspace files. > - OpenCode binds its saved session to exact executable paths and hashes. > - A replacement image can remove those files even when session data is saved. > - Retaining a manifest alone cannot restore the missing launch files. > - This pull request saves the qualified files before provider work and restores the same owned paths. > - The benefit is cold continuity without relaxing the Runner guard or replaying a prior turn. ## Linked Issues or Issue Description Refs #19558. That fix retains the selected pack for ordinary continuation. This change adds durable file retention for a cold allocation. Related #19487 adjusts the admission budget; it does not retain launch files. A search of open and closed OpenCode cold/durable artifact PRs found no duplicate. **What happened?** A saved OpenCode session could outlive the image that supplied its Node executable, proxy, and native executable. A new image supplied different bytes. The Runner then correctly refused the changed launch profile. The saved harness did not contain the old files. **Expected behavior** Core must save qualified launch files before the first turn. It must restore those exact files under the same owned account and absolute paths. Unknown ownership, missing files, and changed hashes must still stop recovery. **Steps to reproduce** Start an admitted remote OpenCode session. Save its workspace, encrypted checkpoint, and harness data. Destroy the original allocation and controller files. Restore into a new allocation with an image whose qualified proxy has different bytes. Resume the same session. The new opt-in local qualification command automates this path with actual OpenCode and a local synthetic model. ## What Changed - Store company-bound, encrypted launch objects and canonical run/backup references. Generate migration 0438 through the normal database workflow. - Stage qualified Node, proxy, and native OpenCode files under a private content-addressed OS-account home. Record the account UID and exact launch paths. Preserve the Rust profile guard. - Gate first provider work on durable capture under the current native claim. Bound Stop/deadline waits while the actual late transfer keeps its worker and scratch ownership until settlement. Restore cold caches with verified staging and atomic rename. Keep live reattachment and historical image-rooted sessions on their existing paths. - Reuse bounded canonical expiration owners. Atomically fence old checkpoint authority with a strict company/run-scoped retirement marker before external deletion. Keep its indexed cleanup anchor across failed deletes and restart. Preserve earlier resume handles and reject later header or source-consumer writes. Add no idle scan or network call inside a transaction. - Add storage, race, executor, filesystem, and actual Rust resume regressions. Add the opt-in local Linux qualification command and operating limits in `doc/opencode-cold-recovery.md`. ## Verification - RED before repair: real PostgreSQL/AES cold restore lacked the launch files; normal executor preparation chose the new image pack. Separate regressions exposed lost upload receipts, missing cache repair, last-owner retirement, an account-UID mismatch, and a resume-handle race during external deletion. Each expected failure was captured before its repair. - Local qualification before the final master integration: 1,169 executor tests, 233 storage/archive/admission/expiration tests, all seven server typecheck layers, migration safety, and the actual Rust cold-resume test passed. - The maintained Linux command passed with two cached immutable Paperclip images whose proxy hashes differ. It removed the original account/controller trees, restored the production published cwd under a new lease, resumed the same actual OpenCode session, kept a useful file and prior model context, and made one new synthetic model request. No external model call or network was available in either container. - The object-store fixture uses real encrypted StorageService streams with an independent owned local directory. It is not hosted S3 acceptance or a complete task HTTP/actor workflow. Historical tasks still require normal effects reconciliation and recovery after deployment. - Review regressions reproduced partial-cache publication, resume-versus-deletion, fresh-writer and source-consumer resurrection, and unbounded Stop/deadline waits before each repair. They now pass through real filesystem, encrypted storage, PostgreSQL, and SDK boundaries. File validation is split from database ownership (384/831 source lines). - Clean runtime head `aadab7acc69ff959678030db46ffe2dcfbefe28f`: the maintained actual OpenCode cold qualification passed with `dirty:false`. It preserved the provider session and recorded launch paths, restored useful workspace content and prior model context, and made exactly one resumed synthetic model request. The 314,560,000-byte launch capture took 23.118 seconds; the later harness was 833,536 bytes. Subsequent repairs change backup-reference retirement and governed continuity-cache classification; capture, restore, launch, and published-workspace inputs are unchanged, so that runtime receipt remains scoped to those paths. The continuity-break consumer is verified separately. - Retention repair `c14d386cecf7123257918ba6a8d47307df2da397`: 68/68 affected real PostgreSQL/AES storage tests and 15/15 focused executor authority/admission cases pass. All seven server typecheck layers pass (source layers and unchanged test shards, with the final 590-file test layer repeated after fixture edits). All review threads have evidence-backed resolutions. Its subsequent CI caught the separate lifecycle-classification gap described below. The refreshed Paperclip review reported paused work, not a completed new review. - Final head `4d1cd3e3309e328b11abc85adc6d78c6d0c7f55d`: all 19 focused root-entry/executor cases and all three source declaration layers pass. [Greptile reviewed this exact head at 5/5](https://github.com/paperclipai/paperclip/pull/19622#issuecomment-6695984874). Final-head [CI verification](https://github.com/paperclipai/paperclip/actions/runs/42308908167/job/128214100339) passes: 60 successful checks, five conditional skips, and the explicitly neutral paused Paperclip review. No check is pending or failing. - CI also caught the launch cache missing from the session-root lifecycle contract. Both the writer scan and a real governed-replacement consumer failed locally before repair: old launch bytes remained in the active root. The cache now archives with the prior session state; the consumer verifies exact bytes are preserved in the archive and absent from the active root. - The latest review found that backup retirement could delete an independent run pin while a selected session retained the checkpoint. A real PostgreSQL/AES regression failed before the repair. Backup-only ownership now uses a NULL-safe exact backup predicate; the run pin belongs to fenced checkpoint expiration. The regression verifies checkpoint decryption and launch-file restoration after pending-backup cleanup, followed by deletion only after removing the final resume handle. - An executor first-session case checks new-session durable capture through the real capture helper and receipt propagation into the actual harness manifest at the backup boundary. Real storage cases verify the run-profile receipt and canonical backup/run pins. - Sequential closure reuse and simultaneous first admission are different scopes. A deterministic two-uploader test measures both independent uploads, one verified ready winner, loser-only deletion, and usable restoration for both runs. Each capture is bounded to 512 MiB and five minutes; up to three pending receipts per run are retained. There is no claim of global single-flight capture or a company-wide egress bound. - Earlier optional warm-ownership verification encountered missing local binaries. After exact-source setup, all nine ownership assertions passed; two other suites encountered local PostgreSQL setup/teardown contention, so those interrupted attempts are not passes. Required current-head CI supplies the broad final coverage. No deployment or hosted historical recovery is claimed. ## Risks - This adds two tables and a forward migration. The file object is bounded to 512 MiB. Capture adds one initial transfer; later harness archives exclude those large files. Company-scoped ready objects are shared and retained until the last canonical owner retires. Concurrent initial admissions can duplicate bounded transfers; each keeps independent ownership and cancellation rather than waiting on another controller. A continuity break retains the controller cache with its prior state until normal session-root cleanup, adding up to one bounded launch closure per retired state. - Cold continuity requires the same qualified Linux platform, supported runtime dependencies, owned account UID/home, and published absolute cwd. A mismatch fails closed. No alias or Rust guard change is included. - Existing image-rooted profiles have no retained file object. This change cannot backfill missing historical binaries. A fresh governed session must reconcile prior effects first. - Live reattachment does not read artifact objects. Storage or provenance failures before a new turn hold admission. They do not grant new execution, model, tool, or replay authority. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution, and local browser tools. The exact served model ID and context window are not exposed in this session. No claim is made for unavailable model metadata. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b55d8a9201 |
fix(runner): retain admitted OpenCode pack on ordinary continuation (#19558)
## Thinking Path > - Paperclip manages AI agents and preserves their work across runs. > - Native OpenCode turns can restore a saved provider session after the prior daemon stops. > - The Rust provider binds that saved state to exact qualified launch paths and artifact bytes. > - Ordinary continuation selected the current provider pack, although live reattachment retained the admitted pack. > - A new proxy or a diagnostics setting could therefore replace the launch profile behind saved state. > - This pull request retains the recorded pack for ordinary OpenCode continuation with a verified durable binding. > - The benefit is compatible continuation without weakening artifact integrity or granting replay authority. ## Linked Issues or Issue Description Refs: #17512, #18237. These added saved authority and compatible pack reuse. This change covers ordinary OpenCode continuation. **What happened?** An ordinary remote OpenCode continuation could select a newer pack while restoring its existing provider session. The Rust recovery guard then rejected the changed launch profile. The same error can occur after a controller release changes the bundled proxy. **Expected behavior** A continuation with a valid saved binding and pack receipt uses the same admitted artifacts. If those artifacts are unavailable, it holds the saved state instead of selecting another pack or creating a provider session. **Steps to reproduce** 1. Admit a remote OpenCode session with a qualified protected provider pack. 2. Persist its pack receipt and suspended durable Runner state. 3. Change the controller bundle to a valid pack with a different proxy digest. 4. Start an ordinary continuation with diagnostics enabled. The new executor regression fails before the repair because selection rejects the available original pack in favor of the new bundle. It passes after the repair. ## What Changed - Retain the saved provider pack for ordinary OpenCode continuation with a verified durable session binding. - Keep provider-pack retention independent of surviving-daemon identity: a new daemon records its current binary, and an authorized continuity break restores normal compatible pack selection. - Test retained pack selection after a proxy upgrade and fail closed when only a newer pack is available. - Test the real Rust executor with changed, valid proxy file bytes. It rejects drift without changing saved state and accepts the original artifacts again. - Document the selection rule and its limits. ## Verification - Initial prevention focused Vitest: 220 passed across native session execution, pack authority, pack verification and pack preparation. The executor regression was observed failing before the production change. - Rust native provider tests: all five OpenCode tests passed. These include the existing drift guard, actual changed proxy bytes, original artifact restoration and restart-lost turn behavior. - Advisory review follow-up: reproduced stale daemon identity in the actual executor and incompatible pack selection after a governed continuity archive. Both fail before their corresponding repairs and pass after them. Added an executor regression for a missing authority receipt; it remains at the current release integrity gate without creating a new receipt. - Updated head `639c080856` integrates master `17901d74ba` by normal merge. Affected Vitest: 202 passed across four files, including all new review regressions; all seven server typecheck layers passed. Exact-head CI and fresh Greptile/Paperclip review are pending. The prior head had 57 successful checks plus successful Snyk and Greptile 5/5; those results do not qualify this revision. - Environment: macOS arm64, Node 26.4.0, locked pnpm 9.15.4 dependencies, controlled remote command boundary and real Rust provider executor. No paid provider calls or hosted retries. - This is a bounded prevention change. It does not deliver historical provider artifacts to a replacement sandbox or replacement controller. Missing saved receipts or unavailable old artifacts remain a separate cold-recovery gap. No full hosted recovery is claimed. ## Risks - A continuation now requires its exact recorded pack when it has a durable binding. A newer installed pack cannot substitute for missing historical bytes. - Fresh launches, other providers and explicit continuity replacement keep their existing paths. Current ownership, company, model, permission and receipt binding checks remain in effect. - The repair does not relax the Rust launch-profile guard, replay prior tool effects, or authorize a new provider conversation. Existing published review receipts remain history. ## Model Used OpenAI Codex, GPT-6 family. The host does not expose the exact model ID or context window. Capabilities used: reasoning, code editing, shell execution, tests and authenticated read-only browser diagnostics. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
a0cba70f18 |
fix(runner): continue restart-interrupted ACPX and OpenCode turns (#19501)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs keep their provider conversation when the control plane restarts > - A local Codex turn lost in a restart already continues once in the same conversation (#15297) > - ACPX (Claude and the other ACPX profiles) and OpenCode took a different path: the runner closed the lost turn as a provider failure > - The task then blocked with "Provider turn failed", and the user's queued follow-up stayed unread > - A restart must not need a user action when continuing is safe > - This pull request gives ACPX and OpenCode the same recoverable restart-loss cause and the same single continuation as Codex > - The benefit is that work interrupted by a controller restart finishes on its own ## Linked Issues or Issue Description Refs #15297 (Codex continuation), #19399 (restart-interrupted turn hook), #19471 (harness fix used for the live reruns). **What happened?** After a control-plane restart during an active native ACPX Claude turn, the task ended `blocked` with `native_provider_terminal_failed: Provider turn failed`. The queued follow-up comment was not consumed. No continuation started. The next run then failed with `native_session_cleanup_quarantined`. A native OpenCode run killed mid-turn ended the same way (`turn.failed` with `error: null`). **Expected behavior** When the old provider process is gone and the conversation can be restored, continue the task once in the same conversation. When that is not safe, record an honest restart cause instead of a generic provider failure. **Steps to reproduce** 1. Start a native ACPX Claude task on a local runner. 2. Submit a follow-up comment while the turn is active. 3. Restart the control plane while the turn is still running. 4. Run `pnpm test:e2e:runner -- --max-automatic-retries 0 --id everyday-workflows.runner-acpx-claude.local.recover-controller`. **Root cause** - `runner-core/src/acpx_provider_backend.rs` `restore_session_if_needed`: a saved `turn_active` state always closed the session. It recorded `turn.failed` with only a top-level `code: acpx_active_turn_recovery_closed` and no `error` object. It also emitted `run.terminal`. - The runnerd transport forwards only `error`, so the controller saw `error: null` and reported generic `provider_turn_failed`. - `runner-core/src/provider_backend.rs` gave the recoverable `provider_turn_lost_on_restore` cause only to `provider_label == "codex"`. OpenCode shares that backend. - `nativeRestartInterruptedTurnId` accepted only `codex_app_server`. The runtime continuation gate accepted only `provider.kind === "codex"`. - Two follow-on defects appeared once the continuation ran live: - The restored ACPX provider rejected the controller's result for the lost turn's open semantic call. That call never settled. - The restored provider's MCP request ids restart. The controller keys semantic calls by `hash(runId, callId)`. The continuation's first Paperclip tool call reused an earlier id, was treated as already answered, and hung. After the result was accepted, `runner_settlement_incomplete` failed the run. ## What Changed - ACPX runner: for an active turn lost to a runner restart, prove that the original provider lifetime lost its fence quorum. Then restore the same conversation and record `turn.failed` with `error: { code: "provider_turn_lost_on_restore", recoverable: true }`. Keep the run open for the controller's continuation. An accepted result, an unanswered runtime request, or an active goal keeps the old close behavior. So do an unproven lifetime fence and a failed restore. - ACPX runner fallback close: add a structured `error` (`acpx_active_turn_recovery_closed`, not recoverable, with an honest restart message), so the cause survives the transport. - ACPX runner: track open semantic calls durably. On restore, retire the lost turn's open calls. Acknowledge their controller results as `retired` and do not send them to the new provider. - ACPX sidecar: when the sidecar restores an existing session, prefix forwarded semantic `callId`s with a per-process namespace (`r<hex>:<id>`). Fresh sessions keep raw ids. - OpenCode runner: give OpenCode the same recoverable cause and open run as Codex (`restart_lost_turn_is_continuable`). Cursor Cloud stays excluded because its agent runs remotely. - Runner TS: `nativeRestartInterruptedTurnId` accepts `acpx_runtime` and `opencode_server`. The runtime continuation gate accepts `acpx` and `opencode`. All other Codex safety rules are unchanged: one continuation, admitted dead-runner restart only, no compaction, goal, semantic result, or pending runtime request. - Docs: `doc/execution-semantics.md` and `doc/run-log-events.md`. ## Verification Red, then green, at each layer: - Rust `tests/acpx_warm_attachment.rs` `restores_the_conversation_after_losing_an_active_turn_to_a_runner_restart`. Red: status `closed`. Then red on the lost call's result: "tool result does not match a pending provider call". Green after the fix. - Rust `tests/native_provider_backend.rs` `restart_lost_opencode_turn_records_a_recoverable_cause`, with the qualified OpenCode launch fixture. Red: `error` was `null`. Green after the fix. - Rust unit test `restart_lost_turns_continue_only_for_locally_restored_providers` (Codex, OpenCode yes; Cursor Cloud no). - TS `native-session-runtime.test.ts` "continues a restart-lost acpx_runtime / opencode_server turn once". Red: `NativeProviderTerminalFailure provider_turn_lost_on_restore`. Green: exactly one continuation, which does not resend the task. - TS `acpx-runtime-sidecar.test.ts`: a restored provider's call ids are namespaced and resolve. - Server `native-runner-restart-recovery.integration.test.ts`: a checkpointed `acpx_runtime` or `opencode_server` restart interruption is claimed as `resume_dead_runner`. Red without the driver gate. Green with it. - Two `native_provider_backend.rs` tests asserted the old close-on-restore result. They now assert the new contract. Their real invariant still holds: the delivered response is never replayed. Suites run locally on the final revision: - runner-core: `cargo test --no-fail-fast acpx`, `--test native_provider_backend` (38), `--test acpx_warm_attachment`, `--test codex_provider restart_lost` - runner TS: `native-session-runtime`, `runnerd-restart-continuation`, Codex driver recovery, contracts, and `acpx-runtime-sidecar` (all pass) - server: `native-runner-restart-recovery.integration.test.ts` (80 passed) - `pnpm --filter "@paperclipai/paperclip-runner..." build` (all generated-contract checks pass) Live runner E2E (local, real Claude via ACPX, macOS arm64): - `everyday-workflows.runner-acpx-claude.local.recover-controller`: PASS twice in a row on the final runner code. The restart crossed an active turn. The task finished `done`, the queued follow-up was consumed, the ZIP download was verified, and there were no failure codes. - `stock-harness.runner-acpx-claude.local.continuity-restart` regression: PASS. - Earlier failing reruns are kept as evidence of the follow-on defects: one harness `kill EPERM` while stopping the server (unrelated to this change), one hang on the colliding call id, and one `runner_settlement_incomplete`. ## Risks - A continuation can run more work in the restored conversation. Its prompt requires inspecting state before repeating an uncertain action, as for Codex. - The ACPX restore waits up to 5 s for the original provider lifetime fence before it falls back to the old close. - The ACPX provider state file gains two fields with defaults. Older saved states still load. A downgraded runner cannot read state that a newer runner wrote with these fields. - Restored ACPX sessions now forward prefixed call ids. Only fresh sessions keep raw ids. Live proof covers Claude. Codex-ACP, Pi, Cursor, Copilot, Grok, Kimi, and Antigravity share this path, but this PR proves them only by unit tests and code review. - OpenCode is covered by the runner fixture and gate tests. The live OpenCode restart-chaos cell was not rerun here. - `runner-core` `unconfirmed_suspension_state_becomes_recoverable_only_after_cleanup_and_suspend` is flaky when run in parallel. It also fails with the new path disabled, so it predates this change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, code execution, and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandPaperclip
|
5e62d8ee82 |
Report the native private host-control protocol version (#19488)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner executes agent work and reports its binary contract through build metadata. > - Unix Runner already supports a private host-control socket for park, claim, and retire requests. > - A host cannot identify that protocol from the current metadata. > - This PR adds an explicit version field for that existing private protocol. > - Hosts can verify the selected binary before opening the private channel. ## Linked Issues or Issue Description Refs #19140 and #19129, which added the existing native lifecycle and private channel. **What existing behavior does this improve?** The `paperclip-runnerd --build-metadata` response. **Current behavior** The native binary implements the private fd 4 protocol on Unix. Its metadata has no field that identifies this protocol. Recovery versions describe different contracts. **Proposed behavior** Report `privateHostControlVersion: 1` on Unix builds. Omit the field on other builds. Keep ordinary artifact checks and runtime behavior unchanged. **Reason and benefit** A host can verify protocol support on the selected binary before using the channel. This field does not mean a provider is ready to park or that a host implements public release/resume or qualified sleep/wake. **Breaking changes** The JSON response gains one optional field on Unix. Current artifact parsers and launch checks accept extra fields. Binary contract version 2 stays unchanged. ## What Changed - Add the Unix-only private host-control metadata version. - Check the exact version and non-Unix absence in metadata unit and real CLI tests. Keep hosted parking and public release/resume fields absent. - Add a regression test that proves ordinary artifact parsing accepts and ignores the new field. - Document the field, exact-version checks, and separate provider and host qualification. ## Verification - On macOS arm64 with Rust 1.97.1, passed narrow offline, locked Cargo tests for the metadata unit, six existing private-protocol cases, wrong-fd CLI rejection, and real CLI metadata output. - Passed `cargo fmt --manifest-path packages/paperclip-runner/runner/Cargo.toml --all -- --check`. - Passed `vitest run src/evals/runnerd-artifact.test.ts` from the Runner package: 5 tests. Initial collection could not resolve Ajv in the fresh worktree. Connecting existing local dependencies resolved that setup failure; no source change was needed. - `git diff --check` passed. Reviewed the diff for secrets, personal data, and private references. - GitHub CI passed on commit `0790af690afaff10f3b69fd9130b5be1c5bab38c`: [main PR workflow](https://github.com/paperclipai/paperclip/actions/runs/42248006277), [Docker Runner check](https://github.com/paperclipai/paperclip/actions/runs/42248006207), and [Windows Runner](https://github.com/paperclipai/paperclip/actions/runs/42248006518). Windows tests cover omission of the field on non-Unix builds. All 61 current-head check runs settled without failure; the latest status context passed. - [Standard Greptile review](https://github.com/paperclipai/paperclip/pull/19488#issuecomment-6689260747) scored that same commit 5/5 with no actionable findings or unresolved threads. - No live provider or host sleep/wake test was run for this metadata-only change. ## Risks The main risk is treating protocol support as permission to park a provider or as proof that a host supports release/resume. The documentation keeps those checks separate. No caller enables the channel from this field in this PR. Current parsers check named fields or project known fields, so the optional field does not change ordinary artifact checks. No migrations, new dependencies, runtime flags, checkpoints, or provider pins. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), High reasoning, through Codex with tool use and code execution. Independent agent review checked the metadata boundary and compatibility. The context window size is not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky Leach
|
205ffcb606 | Verify live Claude and Codex ACPX eligibility before parking (#19295) | ||
|
DottaandClaude Opus 5.5
|
6ca7b151f7 |
feat: compact an agent's context from the task Properties panel (#19378)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A task's Properties panel shows the assignee agent's context window
(paperclipai/paperclip#19273)
> - The board can see that the context is nearly full, but it cannot act
on it
> - A full context makes the agent compact at an unplanned time, often
in the middle of work
> - This pull request adds "Compact now" to the Context hover card
> - The benefit is that the board can compact the agent's context
between runs, at a time it chooses
## Linked Issues or Issue Description
Refs paperclipai/paperclip#19273
**Problem**
The Context row shows how full the agent's context window is. The board
cannot ask the agent to compact it.
**Proposed solution**
A "Compact now" button in the Context hover card. It runs the harness's
own compaction: Codex `thread/compact/start`, and Claude's `/compact`
command through ACP.
**Alternatives considered**
A separate PRP `session.compact` command. Both harnesses run compaction
as a turn, so a compaction turn keeps the existing durable turn binding,
interruption, and recovery. A separate command needs a second turn
lifecycle.
## What Changed
- UI: the Context hover card shows Compact now with three states:
available, Compacting…, and unavailable with a reason (agent mid-run,
harness unsupported, task closed). The Storybook story shows each state.
- Server: `POST /api/issues/:id/context/compact` (board only,
`decideIssueAccess` `issue:mutate`, activity log
`issue.context_compaction_requested`). It queues a short run marked by
`contextSnapshot.contextCompactionRequestId`. `GET
/issues/:id/context-usage` returns the `compaction` state.
- Server: a compaction run does no task work. The native status arbiter
preserves the task status. The lifecycle skips handoff, liveness
continuation, review-path disposition and the issue comment, as it does
for goal-control runs.
- Runner TS: `startTurn({ compact: true })`. The Codex harness session
sends `thread/compact/start`. The runnerd transport maps it to PRP
`turn.start` with `compact: true` (Codex) or `text: "/compact"`
(Claude/ACPX). The native runtime option `sessionCompaction` requires a
resumed session and settles with a synthesized yielded result.
- Runner Rust: `turn.start` accepts `compact: true` for Codex.
`CodexProvider::start_compaction_turn` sends `thread/compact/start` and
binds the turn from the `turn/started` that follows the empty response.
- Fix (from #19273): Codex emitted `context.compacted` on both the start
and the end of one compaction item. It now emits it once, and the count
is distinct by `compactionId`.
- Fix (from #19273): the conversation event view dropped
`context.compacted`, so a finished run never showed the divider. The
view now keeps compactions, plus context readings for runs that
compacted. A finished run's compactions show as thread dividers.
- Fix: the Codex harness stamps before/after sizes on the compaction
event. Codex reports the compacted size before the compaction item ends,
so the old neighbour-based guess showed the wrong size.
## Verification
- `pnpm -r typecheck`, `pnpm build`, `pnpm check:token-gates`, `node
scripts/storybook-catalog.mjs`: pass.
- New tests: `server/src/__tests__/context-compaction.test.ts`,
`status-arbiter.test.ts`, `issue-context-usage.test.ts` (duplicate
compaction counts once), `ui/.../context-usage-row.test.tsx`,
`native-session-runtime.test.ts` (context compaction), Codex driver and
runnerd transport tests, Rust
`codex_compaction_turn_binds_the_turn_reported_after_its_empty_response`.
Rust `codex_provider` suite: 115 passed.
- `pnpm test:run` locally: one failure in
`chat-channels.integration.test.ts`. It reads
`.claude/skills/slack/SKILL.md` from the main checkout, which is not
present in this worktree. It is not related to this change.
- Live, local worktree instance, Codex agent on the Paperclip runner:
clicked Compact now. The run succeeded. Context went from 35k to 9.4k
tokens, and later 9.4k to 9.2k. The task stayed In Review. The thread
shows "Context compacted · 9.4k → 9.2k tokens" and the meter dropped.
- Live Claude proof: pending. The local Claude agent needs an Anthropic
API key. claude-agent-acp handles `/compact` as an echo-less turn, and
the runner already maps its "Compacting completed." notice to
`context.compacted`.
## Risks
- A compaction run resumes the agent's session. If the session cannot
resume, the run fails with `native_context_compaction_unavailable` and
the task is unchanged.
- Compact now is not available while the agent runs on the task, because
compaction interrupts the active turn.
- The conversation view now returns context readings for runs that
compacted. Runs that never compacted return the same events as before.
- An old runner binary ignores `compact` and rejects the `turn.start`
because it has no text. It does not start an ordinary turn.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Nicky Leach
|
879140cbbd | Retain ACPX read events before idle sidecar replacement (#19290) | ||
|
Nicky Leach
|
5ab7b78f2b | Runner: retain the provider across hosted park and reclaim (#19140) | ||
|
Nicky Leach
|
96516a8622 | Keep ACPX output while Core reconnects (#19365) | ||
|
Nicky LeachandPaperclip
|
08f3e7ebec |
Retain passive Codex output while Core reconnects (#19029)
## Thinking Path > - Paperclip manages work that AI agents execute through a Runner. > - Core controls the work while the Runner owns the provider process. > - Core can restart while an admitted provider turn is active. > - Reconnect waits used to leave provider output in the process pipe. > - Ordinary provider polling can also wait on a provider RPC before Core reconnects. > - This pull request drains passive local Codex output into bounded durable storage between reconnect attempts. > - Core receives the original events after recovery and keeps authority over tools and approvals. ## Linked Issues or Issue Description Refs #17594. Related work: merged #19353 supplies the Codex receipt-limit fence and deadline-aware reconnect scheduler. This PR now contains only shared passive intake and the already-live native Codex adapter. A controller outage must preserve admitted work and event identity without restoring a provider, sending a provider RPC, or executing a tool during reconnect. ## What Changed - Add a default no-op disconnected-poll hook. Managed providers and ACPX retain that default. - Drain passive frames from the existing local Codex process between reconnect attempts. Requests, terminal reconciliation and lineage checks wait for the connected path. - Persist each accepted event before acknowledging the provider queue. Soft-limit pressure retains the rejected suffix until Core ACKs free space. Hard failures stop the owned provider and preserve the durable prefix. - Preserve credential observations and the existing lease, runtime, grace and provider deadline bounds. Document the intake contract and test replay through the real PRP controller. ## Verification The slice was replayed onto master `bf10bc695a`. `git range-diff f453509e6f..20626230b9 bf10bc695a..HEAD` reports an unchanged patch. The diff has 11 files and contains no ACPX adapter or sidecar changes. Targeted local checks on this head: - `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml --locked -p paperclip-runner-core --lib disconnected -- --test-threads=1`: 12 passed. These cover passive deferral, no restoration, the real managed HTTP worker, durable backpressure, oversized output and existing receipt-limit cleanup. - The 10 `credential_observer::tests::` checks passed. The Runner daemon and recovery example build, and workspace Rust formatting passes. - `vitest run src/control-plane/runner-recovery-fixture.test.ts --maxWorkers=1`: four passed. Both controller replacement modes retain the same process and replay event identities. The deadline and silent-handshake cases also pass. - `vitest run src/live/runnerd-codex-transport.test.ts -t late-remote-release-error --maxWorkers=1`: one passed. Whitespace and added-line secret checks pass. Earlier review found that disconnected ordinary polling could perform synchronous managed history reads. The real-worker regression retained here proves no history or stream-reopen request occurs until connected polling. Earlier slice preparation hit a sandbox loopback denial and a missing local daemon; the permitted loopback test and the test after building the daemon both passed. The current-base checks above run with those prerequisites in place. Historical CI on the old combined head failed an unordered AI-connections migration comparison. The faithful full-file local check passed 158 tests, and the failure remains recorded [here](https://github.com/paperclipai/paperclip/actions/runs/42112385671/job/127543554397). Earlier receipt-limit delivery failures are addressed by merged #19353. Previous review scores belong to the old combined diff. On current head `b92d4b604ee6992eeefbc02c8d41ec4f2a608a4d`, [CI run 42125840239](https://github.com/paperclipai/paperclip/actions/runs/42125840239) passed. All 59 checks completed with 55 successes and four intentional skips. Standard review 13 and Apex review 14 each scored 5/5 on this exact head, with zero comments and no unresolved threads. The single Apex request is [recorded here](https://github.com/paperclipai/paperclip/pull/19029#issuecomment-6671993072). ## Risks During an outage, native Codex intake uses 25 ms ticks between reconnect attempts. A provider request or terminal waits for Core. Saturation can pause intake or produce a failure. Managed remote streams retain their existing limits. Tests use controlled local providers; live provider qualification and deployment are separate. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning effort, through Codex. Used source inspection, local native and PRP tests, and code review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
bf10bc695a |
feat(task): show the agent's context window in task properties (#19273)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run through a harness (Codex app-server, or Claude through ACPX), and the runner turns harness events into Paperclip events > - Both harnesses report how full the model's context window is. Codex sends `modelContextWindow` with its token usage. Claude sends `used` and `size` in ACP `usage_update` > - Paperclip dropped both values. The task view could not show how close an agent is to compaction, and a compaction left no visible mark in the task chat > - This pull request carries the context-window occupancy through the runner, stores it with the existing usage events, and shows it as a "Context" row in the task Properties panel > - It also shows a "Context compacted · 230k → 38k tokens" divider in the task chat when the agent compacts its context > - The benefit is that operators can see when a long task is near its context limit and when the agent lost detail to a compaction ## Linked Issues or Issue Description Refs #19252 (same pattern: a canonical harness event turned into task-chat UI). Refs #19267 (status notices; it turns Claude's compaction banner into a notice, and this PR reads that notice too). **What existing behavior does this improve?** The task view shows no context-window information, and a compaction is invisible in the task chat. **Subsystem affected** Runner usage normalization (Rust and TypeScript transport), the PRP usage schema, a read-only issue API, the task Properties panel and the task-chat transcript. **Current behavior** Codex `modelContextWindow` and ACP `used`/`size` are dropped in the runner. `context.compacted` always has null sizes, Claude compactions emit no `context.compacted` event at all, and the live runner turn hides every marker except "interrupted". **Proposed behavior** `usage.reported` carries an optional `context {usedTokens, windowTokens}`. The task Properties panel shows a Context row ("10% of 258k") with a hover breakdown (tokens used, input and cached input, output, cost, compactions). The transcript shows a compaction divider, with sizes from the usage before and after the compaction. **Reason and benefit** Operators can see when a long task is close to its context limit, and they can see where the agent's earlier context was summarized. **Breaking changes** None. The new schema field is optional, and the new endpoint is read-only. ## What Changed - PRP `usage.schema.json`: add an optional `context {usedTokens, windowTokens}` object. Regenerate the schema bundle and the protocol manifest. - Rust `provider_events.rs`: add `codex_context_usage` (last request `totalTokens` and `modelContextWindow`) and `acpx_context_usage` (ACP `used`/`size`). ACPX `usage_update` now sets `context`. - Rust `provider_backend.rs`: Codex `thread/tokenUsage/updated` sets `context` on live (non-historical) usage. - Rust `provider_events.rs`: emit `context.compacted` for the completed `context_compaction` provider notice that #19267 produces from Claude's compaction banner. Only this path marks a Claude compaction, so the server counts each compaction once. - `runnerd-codex-transport.ts`: `rehydrateRunnerdUsageNotification` keeps a valid `context` on `tokenUsage`, so it reaches the stored `item.completed` usage item. - Server: new `GET /api/issues/:id/context-usage` (issue read access is required) and `issueContextUsage` service. It reads only the assignee's current session (the session of its newest usage report), and returns that session's latest context reading, non-zero totals, cost, and the count of compactions across all of the session's runs. Added to OpenAPI. Shared type `IssueContextUsage`. - DB: migration 0432 adds two partial indexes on `heartbeat_run_events`, built concurrently. One indexes usage items by run and time. The other indexes `context.compacted` events by company and session. `EXPLAIN` shows that both lookups use them. - UI: new `ContextUsageMeter` (bar, percentage, hover card) and `IssueContextUsageRow` in the Execution section of Properties. The row refreshes every 8 s while a run is active, and once more when the run ends. - UI transcript: `native-run-events.ts` maps `context.compacted` to a `context` activity entry with pre/post sizes from neighbouring usage items. `transcript-adapter.ts` turns it into a new `context_compacted` marker variant. The live runner turn keeps that marker as its own timeline row. - Storybook: `task-context-window.stories.tsx` (Properties panel in a task, all states, hover breakdowns, divider). ## Verification - `cargo test -p paperclip-runner-core --lib provider_events` (new tests: ACPX context, Codex context, banner compaction, notice compaction) - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-codex-transport.test.ts` - `cd ui && npx vitest run src/components/ContextUsageMeter.test.tsx src/components/transcript src/components/task-chat src/components/issue-properties` - `cd server && npx vitest run src/__tests__/issue-context-usage.test.ts` (embedded Postgres) - `node scripts/typecheck-low-memory.mjs`, `pnpm check:token-gates`, `pnpm --filter @paperclipai/paperclip-runner run check:protocol-types` - Live check on a local worktree instance through the runnerd path: - A Codex `gpt-5.5` task showed "10% of 258k" (27k of 258k, input 77k with 49k cached, output 1.1k). - A Claude Sonnet 4.6 task showed "17% of 200k" (34k of 200k). - Storybook: Tasks and conversations / … / Task chat / Context window. - Not verified live: a real compaction. The divider is covered by unit tests through the same projector path the task chat uses. ## Risks - The schema field is optional, and older runners omit it. The UI renders nothing until a harness reports a window. - Migration 0432 builds two partial indexes concurrently on `heartbeat_run_events`, which is a large table. The indexes cover only usage items and compaction events, so they stay small. They add a little write overhead to those events. - Claude in-progress usage rows still record zero tokens with `requests: 1`. The final turn usage is correct. This PR does not change accounting. - A "Compact now" action is out of scope. It needs a compaction-only run that does not produce a run result. It will come in a follow-up. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, in-app browser for Storybook and the live instance). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandPaperclip
|
b61a815d3d |
Enforce Codex receipt-limit deadlines while Core is offline (#19353)
## Thinking Path > - Paperclip uses a Runner to execute agent work. > - The Runner keeps a record of each tool call, called a receipt. > - When that storage reaches its limit, the Runner asks Codex to stop. > - Core can stop responding while the stop request is pending. > - Codex may already have written its completion behind earlier output. > - The Runner must keep its stop deadline and preserve that real completion. ## Linked Issues or Issue Description Refs #19029. This is its first isolated slice; #19029 remains unmerged. Related #17594 covers controller replacement. Related #17516 retains receipts during explicit stop; this change handles the autonomous receipt-limit deadline. ## What Changed - Keep the existing 2s/30s stop deadlines while Core is offline, including when a connection waits for a reply. Other lease and runtime deadlines still apply. The Runner sleeps between connection attempts and deadline checks. - Stop the owned local Codex process when its deadline expires. Read its remaining output within the existing size and message limits, then process it in order. Only after output has ended and all earlier messages have been processed can the Runner report a missing completion. - After the local process has stopped as requested, skip replies that only reject or cancel old requests. Commands, user responses and actual tool results still require successful delivery. - Keep the stop behavior, regression tests and transport documentation in this slice. ## Verification Standalone checks on this branch: - Seven existing Codex tool-record limit integration cases pass. - Twenty focused checks cover deadline cleanup and retained output, including 273 distinct messages followed by one real completion, missing completion, and a pipe that stays open. - Separate checks cover strict tool-result delivery, three process-stop cases, and lease/runtime/reconnect time limits. - Four controller process tests pass. They include Core accepting a connection but never replying, and a reconnect delay longer than the allowed recovery period. - Existing warm attachment still retains the completed turn's remaining output and usage. - Rust formatting, diff checks and the local secret scan pass. The extraction first omitted an existing test helper and the connected retry preamble. Standalone compilation and receipt tests caught both omissions. Restoring them produced the results above. Full regression checks run in GitHub CI. Hosted CI passed both native platforms and typecheck, but [server11 failed](https://github.com/paperclipai/paperclip/actions/runs/42118725143/job/127565828671) one unchanged status-card test. Its fake clock advances while real database work runs; the final clock assertion was 50 ms past its limit. The identical test, service and scheduler passed all 36 cases in [the earlier baseline shard](https://github.com/paperclipai/paperclip/actions/runs/42111889388/job/127541685625). The local comparison first needed a built SDK, then stopped at embedded Postgres startup before any test ran. It did not reproduce the assertion. The single failed-jobs-only retry passed. The original failure and the blocked local comparison remain recorded here; the retry does not establish a precise timing cause. At exact head `f453509e6f`, all 59 hosted checks completed: 55 succeeded and four were skipped. Snyk also passed. Standard review round 1 and selected review round 2 each scored 5/5 on this head, with no unresolved threads. ## Risks The Runner now stops its local Codex process at the original deadline even while Core is unavailable. If another program keeps the output pipe open, or queued output exceeds the existing limits, the Runner cannot confirm the outcome. It reports that uncertainty and keeps the records already written to disk. Tests use real child processes with controlled provider fixtures. Live provider and deployment qualification remain separate. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning effort, through Codex. Used source inspection, native subprocess tests and targeted TypeScript tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
2506bcdaf4 |
Admit exact stopped Runner sources under the current run lease (#19134)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution keeps provider state and a durable controller journal. > - A completed process can stop before the next run attaches. > - The current run must own recovery of that exact source. > - An old controller must lose permission to change the consumed source journal. > - This pull request adds explicit source admission and protected task dispatch. > - The benefit is a narrow recovery boundary that preserves the original successful result. ## Linked Issues or Issue Description Native prerequisites #19132 and #19153 are merged into master. This PR targets master and updates the TypeScript Runner driver contract alongside Core and SDK admission. This change adds the SDK and Core admission boundary. Executor selection and fresh source preparation are a separate change. Related #17916 supplies the workspace export holds preserved here. ## What Changed - Plugins advertise `codexSourceRecoveryVersion: 1` for Codex App Server and `acpxSourceRecoveryVersion: 1` for ACPX. Hosts can qualify either, both, or neither. Core selects the matching field, because a stopped-source check for one provider cannot establish the other provider’s recovery guarantees. ACPX orchestration is tracked separately in #19196. Use the existing exact source identity and `resumeFrom` under the authenticated current run. No public recovery mode is added. Require the claimed source, current account and prepared launch at protected dispatch. - Keep the first source binding pinned on the consumer lease, including after uncertain dispatch. A newly acquired coordinator can refresh only the same source. - Claim one stopped successful source under the current run's acquired native lease. Store the token-free consumer reference in existing lease metadata. Preserve the source status and result. - Check the exact six source identity fields, encrypted journal header, company, task, agent, environment, driver, workspace, account selection, and current coordinator generation. Keep workspace export and legacy settlement holds. - Serialize claims and checkpoint writers in the same coordinator lock order. Reject stale source writes after consumption. Check the current consumer before recovered source writes. - Check source admission before and after protected worker dispatch. Keep the request and lease owned by the current run while the native identity remains the source. - Allow the same owner to prepare B after a fresh source submission, using the exact consumed-source proof instead of a generic warm-reuse flag. Recheck that proof after dispatch. - Preserve ordinary cached adoption and repeated preparation after launch. The current authenticated owner can differ from the adopted PRP identity without requesting a new source launch. Keep ordinary suspended recovery and live task attachment checks. Extract the shared source resolver to avoid a module cycle, and make ordinary selection honor the same durable source consumer. ## Verification - Current head `cac9cf188a` includes merged #19300 and the symmetric capabilities introduced in `b7e25a64b4`. The task-owner and warm-rebind fixtures now identify their physical task. All 29 SDK schema cases, 95 Core ownership and source-admission cases, 25 native Codex and ACPX recovery cases, and the binary metadata check passed on this head. The SDK and Runner builds and server typecheck passed. Both native providers reject the other provider’s capability. GitHub CI is green and [standard Greptile scored this commit 5/5](https://github.com/paperclipai/paperclip/pull/19134#issuecomment-6645574114) with no open threads. - Previous head `bda49271a4ae362a15baf3fe6d6d297e25fd2000` is directly based on master after #19132 and #19153 landed. [GitHub workflow 42072192023](https://github.com/paperclipai/paperclip/actions/runs/42072192023) passed on its first attempt. All check runs completed successfully or as intentional workflow skips; Snyk passed. Standard Greptile scored this exact head 5/5. That review had no unresolved findings. - Runner and SDK TypeScript builds and server typecheck passed. The focused SDK, task-owner, task-runtime, PostgreSQL source and authority suites passed all 161 cases. The production transport-factory getter regression and 14 native-lane selection checks also passed. - The Linux native lane executed the new ordinary warm A→B process case. It kept one Runner/provider process and completed two turns on the same thread. The eager-getter mutation fails with `native_runner_source_attachment_pending`; preserving the live getter passes. The production factory test changes the current owner after transport creation and checks that same options object. - PostgreSQL cases cover both concurrent source-writer/claim orders, exact source submission under the current caller, source-switch denial after uncertain dispatch, same-source coordinator refresh, stale same-boot generation, corrupted journal headers, competing consumers, account changes and export holds. - The protected dispatch sequence submits A, prepares B and retries B preparation. It rejects undispatched sources, changed accounts and foreign target bindings. Source status and accepted result remain unchanged. - Ordinary adopted A preparation under current B, repeated ordinary preparation after launch and ordinary suspended continuation retain their existing path. Absent or false claims preserve ordinary identity denial; malformed claims fail closed. - Earlier development attempts exposed ordinary preparation regressions and the live-getter omission; the final regressions cover those fixes. The first embedded PostgreSQL attempt needed missing library aliases repaired in ignored dependencies. Those attempts are not passing evidence. - Historical, superseded CI heads had unchanged UI fake-timer and Playwright health-response-lifetime failures. Their isolated cases and scoped retries passed. The response-lifetime cause remains unproven. No UI/test timeout was changed. The final rebased-head workflow above passed without a retry. - Whitespace and added-line secret/PII scans passed. Full workspace checks ran on GitHub. No live vendor probe or deployment was performed. ## Risks - Any driver that used the earlier Codex TypeScript option must use `codexSourceRecoveryVersion`. The new plugin description rejects the generic spelling. The Launcher’s existing Codex-specific HTTP header can map to this property when the plugin adopts the new SDK. The change requires no journal migration or ROM change. - This is an inactive admission foundation. Executor orchestration is separate in #19157. Native restore integrity and provider stopped-source/account validation must be present before executor activation. The native prerequisite uses ordinary recovery followed by attachment. The composed executor proofs are in #19157. - The source claim is durable. A later controller must recover the same consumer or keep the hold. It cannot select another source after an ambiguous launch. - Admission adds bounded database checks around protected startup calls. Busy issue locks fail closed; other lock waits have a one-second bound. - No migration, snapshot format, proactive receipt, recurring worker, or deployment is included. Full workspace checks and platform coverage run in CI. These tests do not prove vendor history durability or a hosted rollout. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning, through Codex with repository tools and code execution. The runtime does not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green on the current head - [x] Standard Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
22281d4b3a |
feat(task-chat): approve harness-proposed plans (Codex plan mode, Claude ExitPlanMode) (#19291)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run through the native runner. Codex and Claude Code each have their own plan mode, and each produces a proposed-plan document. > - Paperclip already has a planning work mode, an issue `plan` document, and a `request_confirmation` approval gate for plans. > - These parts were not connected. The Codex proposed-plan item was transcript text only. The server path never put Codex into plan mode. Claude was never put into plan mode, and its ExitPlanMode plan was dropped. A planning run could close its task as done with no approval. > - This pull request turns a harness's own proposed plan into the issue's Plan document and the existing approval gate, for Codex and Claude. > - The benefit is that users approve or send back the agent's real plan from the task chat. Approve resumes the agent. Request changes sends the reason back to the agent. ## Linked Issues or Issue Description Refs #19252 **Subsystem affected** Native runner provider events (Codex and ACPX Claude), the runner Codex provider (Rust), server native plan synchronization, and the task chat UI. **Problem or motivation** Codex and Claude both produce a proposed-plan document in their native plan mode. Paperclip did not use it. In live tests on the server path, a Codex planning run was not in Codex plan mode, saved its own `implementation-plan` document, and marked the task done without approval. Claude planning runs were not in Claude plan mode, and the ExitPlanMode plan text never reached Paperclip. **Proposed solution** Add a canonical `plan.proposed` event for a harness's proposed plan. Put Codex and Claude into their native plan mode for planning runs. Save a completed proposal as the issue `plan` document with the existing `request_confirmation` gate. Agents can also propose a plan on a standard task; the work mode stays standard. **Alternatives considered** Reuse `plan.updated`. Its schema is the within-turn checklist and is not the durable Plan document, so a separate event keeps the two clear. Stronger prompt text without native plan mode. Live tests showed that the model kept its own document workflow. **Roadmap alignment** This reuses the issue plan document, planning work mode and approval gate that already exist. It adds one canonical provider event. ## What Changed - `packages/paperclip-runner/protocol/schemas/*`: add the `plan.proposed` event (`paperclip.plan.proposed.v1`: `proposalId`, `status`, `markdown` up to 60k characters, `source`). Regenerate the schema bundle, validators and manifest. - `packages/paperclip-runner/src/provider-events.ts`: map the Codex `plan` item and a `<proposed_plan>` block in a completed Codex message to `plan.proposed`. Add `claudePlanProposalEvent`. - `packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs`: send Codex's native `collaborationMode: plan` on `turn/start` for a read-only (planning) Codex run. Before this change the server path never sent it. - `runner-core/src/acpx_event_payload.rs`: admit `plan.proposed` as ACPX rich display activity. - `drivers/acpx/codex-runtime-adapter.ts`: set Claude Code to native plan mode for a read-only run. Capture Claude's ExitPlanMode plan from its permission request, record it, and decline the native "Ready to code?" prompt. - `drivers/acpx/codex-acpx-driver.ts`, `cli/acpx-runtime-sidecar.ts`: emit the captured proposal as a rich event. Complete that turn instead of reporting it as cancelled. - `backends/codex-native-backend.ts`: per-harness plan-mode instructions, and a short opt-in hint for Claude on standard tasks. - `server/src/services/native-runtime/native-session-executor.ts`: synchronize a completed `plan.proposed` into the `plan` document and a `request_confirmation` in planning and standard mode. Do not change the issue status during the run (that revoked the run's lease). The run finalizer turns the pending confirmation into the review wait. - `server/src/services/heartbeat-run-summary.ts`: keep a `<proposed_plan>` block out of the posted reply. - UI: new `plan_proposal` provider family. The task chat shows a live "Writing plan" card while a proposal streams and a one-line record when it completes. Resolved plan reviews keep a receipt ("Approved Plan v1", "Requested changes to Plan v1: reason"). Plan previews skip section headings. - Storybook: `Tasks and conversations/.../Task chat/Proposed plan approval` shows the journey. - `doc/SPEC-implementation.md`: describe harness-proposed plan approval. ## Verification - `cargo test -p paperclip-runner-core --lib` (new: `read_only_codex_turns_use_native_plan_collaboration_mode`) - `cd packages/paperclip-runner && npx vitest run src/provider-events.test.ts src/drivers/acpx/codex-runtime-adapter.test.ts` - `cd server && npx vitest run src/__tests__/native-plan-proposal-sync.test.ts src/__tests__/heartbeat-run-summary.test.ts` - `cd ui && npx vitest run src/components/transcript/native-run-events.test.ts src/components/task-chat` - Live test on a local worktree instance with real harnesses: - Codex (`gpt-5.6-sol`), planning task: Codex asked one question, then produced a native plan item. The task showed Plan v1 with Approve plan / Request changes and moved to in review. Approve changed the work mode to standard. Codex then implemented the plan and its tests passed. - Codex, Request changes with a reason: Codex woke in planning mode and produced Plan v2 that includes the change. - Claude (`claude-opus-5-5`), planning task: Claude was in plan mode. ExitPlanMode became Plan v1 and the turn completed. Approve resumed Claude in standard mode, and Claude implemented the plan. - Claude, standard task that asks for a plan first: Claude stopped before code changes and asked for plan approval. The work mode stayed standard. - Storybook: open `Proposed plan approval` and check each stage. ## Risks - Codex planning runs now really use Codex plan mode on the server path. Codex can ask a clarifying question before it proposes. This matches Codex plan mode. - Claude planning runs now use Claude plan mode. A declined ExitPlanMode ends the Claude turn; the runner reports it as completed. Claude's tool row for that request records a rejection from Claude's side. - Codex does not stream the proposed plan on the server path yet, so the live card shows a placeholder until the plan completes. - A `<proposed_plan>` block in any completed Codex message on a standard task now asks for approval. This is the documented opt-in path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, browser). Long context. No extended thinking mode was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
2575b941d0 |
feat(task-chat): show hook and auto-review decisions as governance rows (#19266)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run on harnesses such as Codex and Claude, and the task chat shows what each run did > - Harnesses make automatic decisions: hooks block or allow tool calls, and Codex auto-review approves or denies actions > - Today Paperclip drops these decisions or shows them as generic rows. Codex `safety.review.*` always says `decision: "unknown"`, the native Rust runner drops `hook/*` and `item/autoApprovalReview/*`, and claude-agent-acp drops Claude hook messages > - An operator cannot see why an agent changed course or which action ran without a person's approval. This conflicts with Paperclip's approval-gate model > - This pull request records each hook and review decision with its reason and shows it as a small governance row in the task chat > - The benefit is that automatic decisions become auditable, and a blocked action explains itself without opening run logs ## Linked Issues or Issue Description Refs #19252 (precedent: canonical `plan.updated` events rendered as a task-chat card). Builds on #19259 (ACP `_meta` passthrough from ACPX), now merged. **Subsystem affected** Runner provider events (Rust native Codex normalizer, TS drivers, ACPX sidecar), the claude-agent-acp patch, the PRP provider-event schema, and the task chat activity UI. **Problem or motivation** Hook and auto-review outcomes reach Paperclip but are lost or flattened. Codex reviews always record an unknown decision. Native Codex runs drop hook and review notifications in the Rust normalizer. Claude hook messages never leave the adapter. Codex reuses one `run.id` for every call of a hook, and the id contains a host path. **Proposed solution** Map the real outcome, decision, action, risk and reason into additive optional fields on `paperclip.hook.v1` and `paperclip.safety.review.v1`. Forward Claude SDK hook messages. Render small grey rows: a stopped action is tinted and shows its reason, an approved review folds into the command it approved as an "auto-approved" mark, and routine hook passes stay out of activity summaries. **Alternatives considered** Separate rows for every approval. This duplicated each approved command, so approvals fold into the command row instead. ## What Changed - Runner (Rust): new `governance_events.rs`. `normalize_codex_notification` now maps `hook/started|completed` and `item/autoApprovalReview/started|completed`. Each hook call gets its own id (a digest of the run id and start time), so no host path is stored. - Runner (TS): new `governance-events.ts` with the same mapping for the TS Codex driver, and the Claude hook mapping. The ACPX sidecar publishes Claude hook updates as schema-validated `hook.*` rich events. - claude-agent-acp patch: turns on `includeHookEvents` and forwards `hook_started` / `hook_response` on `session_info_update._meta.claudeCode.hook`, with bounded output. - Protocol: additive optional fields. Hook records get `outcome`, `message`, `source` and `toolName`, and the `blocked` and `stopped` statuses. Review records get `actionKind`, `action`, `riskLevel`, `userConsent` (the reviewer's user-authorization level; keys that end in "authorization" are redacted as secrets), `rationale` and `decidedBy`, and the `timed_out` and `aborted` decisions. Regenerated the manifest and validators. - Fixed: the hook `blocking` flag read `executionMode === "blocking"`, but Codex sends `sync` / `async`. - UI: `native-run-events` keeps the full hook and review payloads. `governance-presentation.ts` writes the row labels. `fold-auto-reviews.ts` attaches an approved review to its command row. The rows show the tint, the reason line and the "auto-approved" mark. Activity summaries lead with a block or denial. - Storybook: "Task chat / Governance rows" shows every outcome, a Codex run, a Claude run and the live line. - Docs: added a governance activity section to `packages/paperclip-runner/docs/protocol-compatibility.md`. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/governance-events.test.ts src/provider-events.test.ts src/drivers/codex src/drivers/acpx src/cli`. 2305 tests pass. This includes a replay of real Codex 0.160 notifications from a session with hooks and auto-review (`test-fixtures/governance/codex-0.160-hooks-auto-review.json`, sanitized). - `cargo test --release -p paperclip-runner-core`. The full suite passes, and `governance_events::tests` replays the same fixture. - `cd ui && npx vitest run`. 11,047 tests pass. - `node scripts/typecheck-low-memory.mjs`, `pnpm check:token-gates`, and the runner contract checks (`check:protocol-manifest`, `check:protocol-types`, `check:capability-contract`, `check:protocol-coverage`, replay goldens, contract fingerprints) pass. - Live test on a local isolated instance. A Codex agent and a Claude agent each used a preconfigured home with a PreToolUse hook that blocks edits to `0001_init.sql`, plus UserPromptSubmit and PostToolUse hooks. Both runs showed "PreToolUse hook blocked a tool call" (Codex) or "PreToolUse hook blocked Bash" (Claude), with the hook's message. They also showed "UserPromptSubmit hook added context", and the group summary started with the block. - Auto-review cannot run in Paperclip today, because the native Codex runner only allows `approvalPolicy: never`. The review mapping and the approval fold are proven against real Codex 0.160 app-server output (fixture above) and in Storybook. ## Risks - Low risk to execution. These records are display-only and carry no authority. All new schema fields are optional, so older records stay valid and older UIs ignore the new fields. - The `auto-approved` fold depends on `targetItemId` matching the command's execution id. Real Codex 0.160 data confirms that it matches. When there is no match, the review stays as its own row. - In Paperclip's isolated homes, hooks do not run, because the runner never copies hooks into an isolated home. Rows appear only for agents with a preconfigured home, or with future managed hooks. - The claude-agent-acp patch changes. The ACPX profile check still passes, because the patch is Paperclip's ACPX layer and not the upstream release. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, browser and file editing) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
be5709a25c |
fix(runner): classify incompatible saved provider profiles (#19334)
Classify deterministic OpenCode and Cursor Cloud saved-profile mismatches with the existing permanent provider-state class. Preserve saved state and stop same-run retries after the runner refuses that state. Verified 37 native-provider backend tests, durable classified-result replay, the server policy test, rustfmt, workspace typecheck and build. Complete hosted CI passed with Greptile 5/5 and no unresolved threads. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
dd889e6a0b |
feat: show the assignee's provider usage limit in task properties (#19270)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run on the native Runner through the Codex app-server and the Claude ACP harness > - Both harnesses report the provider account's rate-limit windows: Codex sends `account/rateLimits/updated`, and Claude sends its SDK `rate_limit_event` as `usage_update._meta["_claude/rateLimit"]` > - Paperclip dropped both signals, so an operator could not see that an agent was near or past its provider limit > - This pull request normalizes both signals into one account-scoped snapshot, keeps the latest snapshot per agent, and shows it as one "Usage limit" line in the task properties pane > - The benefit is that an operator sees "5h 78% · resets 15:40" (amber above 80%, "Limited until …" when the provider refuses work, "using credits" when a used-up window falls back to credits) next to the assignee, and can click it for every window ## Linked Issues or Issue Description Refs #19246 Refs #19259 **Is your feature request related to a problem?** Codex and Claude report provider rate-limit windows during a run. Paperclip dropped them: the Codex notification fell through the Rust normalizer with no branch, and ACPX dropped Claude's `_meta`. Operators could not see how close an agent was to its provider limit. **Describe the solution you'd like** Store the latest reported windows per agent and show one compact line in the task properties pane, with a popover that lists every window. **Describe alternatives you've considered** A full gauge in the agent list and the run header. A design review chose one property line with a details popover instead. ## What Changed - Runner (Rust, the runnerd path): new `provider_rate_limits.rs`. Codex `account/rateLimits/updated` becomes a `provider.notice.recorded` event (category `provider.rate_limits`, scope `account`). `CodexProviderState` merges sparse updates and emits a notice only when the windows change. ACPX `usage_update` with a sanitized `rateLimit` field also emits the notice. - Runner (sidecar): `acpx-runtime-sidecar.ts` forwards only `status`, `rateLimitType`, `utilization` and `resetsAt` from `_meta["_claude/rateLimit"]`. This needs the ACPX `_meta` passthrough from #19259. Until that merges, Claude emits nothing. - Runner (TypeScript in-process drivers): the same mapping in `codex-session-notifications.ts` and `codex-acpx-driver.ts` through `drivers/provider-rate-limits.ts`. - Shared: `provider-rate-limits.ts` holds the snapshot type, the parser, the merge, and `summarizeProviderRateLimits` (fullest current window, expiry by reset time, limited versus using credits). - Server: `agent-provider-rate-limits.ts` merges each notice into `agent_runtime_state.state_json.providerRateLimits`, scoped by company and agent. A session reset keeps the snapshot. No migration. - UI: `AgentRateLimitPropertyValue` adds a "Usage limit" row under the assignee in `IssueProperties`. Transcripts treat the notice as run-log only. A run status event refreshes the assignee's runtime state. - Storybook: `Tasks and conversations/Task creation and lifecycle/Usage limit property`. - Docs: `doc/run-log-events.md` describes the notice and its storage. ## Verification - `cd packages/paperclip-runner/runner && cargo test -p paperclip-runner-core --lib rate_limit` and `--test acpx_provider_events`: sparse Codex merge, dedupe, reached window, checkpoint round trip, Claude utilization and rejection, notice shape. - `cd packages/paperclip-runner && npx vitest run src/drivers/provider-rate-limits.test.ts src/drivers/codex` - `cd packages/shared && npx vitest run src/provider-rate-limits.test.ts`: summary, expiry, limited, using credits, parse and merge. - `cd server && npx vitest run src/__tests__/agent-provider-rate-limits.test.ts`: create and merge, out-of-order snapshot, company isolation, unrelated notices, session reset keeps the snapshot. - `cd ui && npx vitest run src/components/transcript src/components/issue-properties src/context/LiveUpdatesProvider` - `pnpm -r typecheck` and `pnpm check:token-gates` pass. - Live test on a local worktree instance: a native Codex agent on a ChatGPT subscription login ran a real task. The runner recorded one `provider.rate_limits` notice. The server stored it. The task properties pane showed "7d 100% · using credits" (this account's weekly window was used up and had credits, and the run succeeded). The transcript showed no extra row. - Not verified live: Claude. The local Claude CLI was not signed in, and the Claude path also needs #19259. The Rust and sidecar paths have tests. - Not run locally: the full `pnpm test:run` and `pnpm build`. CI runs them. ## Risks - Low risk. The snapshot is display data. It does not hold or start work. The existing provider-limit wait (#19246) still reads typed run failures. - Codex can send a rate-limit update often. The runner emits a notice only when the windows change, so run-log growth stays small. - With API-key auth, the harnesses report no windows, so no row appears. This is the intended result. - `CodexProviderState` gains two optional serde fields with defaults, so older checkpoints still load. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, browser) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
86a9d3ec6c |
feat(task-chat): tool rows that say what ran and how it went (#19265)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The task chat shows each tool call an agent makes as a short activity row. > - Real Claude runs showed Bash rows named "Terminal" with no command and Write rows stuck at "Preparing file…". No row showed an exit code or a duration, and a tool refused by policy looked like any other failure. > - Codex already reported exit codes, durations and parsed command actions, but the UI dropped them, and every Codex read looked like a shell command. > - This pull request gives both harnesses one row grammar: a verb icon, the real subject, an exit chip and a duration. > - The benefit is that a reviewer can see what ran and how it went without opening each row. ## Linked Issues or Issue Description Refs #19259 (this branch includes its `_meta` passthrough commit; merge #19259 first or together). **Problem or motivation** Claude tool rows hide what ran: the runner keeps the first ACP title ("Terminal", "Preparing file…") and drops the refined one. `exitCode` and `durationMs` are null on every Claude tool event. The UI ignores `tool.execution.progressed`, `exitCode` and `durationMs` for every harness. **Proposed solution** Read real tool facts in the runner (Claude `_meta.claudeCode`, raw input and result text; Codex `commandActions`), carry them in the canonical tool event, and render one row grammar in the task chat. **Alternatives considered** Enable Claude's `terminal_output` client capability to get `terminal_exit`. That changes how command output arrives, and Claude's own exit code is derived from the error flag anyway, so this PR reads the error flag and the "Exit code N" text instead. ## What Changed - **Runner (TS):** `createAcpxToolEventNormalizer` keeps the latest concrete title instead of the first placeholder. It merges tool facts across the lifecycle and measures the duration. New `acpxToolDetailFromRuntimeEvent` reads the native tool name, the command or pattern, the description, the exit code and a "Permission denied" reason. Exit 0 is inferred only for Claude `Bash`, where a non-zero exit is always a failed result. - **ACPX sidecar:** reads the facts before `rawInput` is dropped and sends a bounded `toolDetail`. A file path argument becomes an attested workspace-relative location. - **runner-core (Rust):** validates `toolDetail`. `normalize_acpx_tool_call` uses the native name, exit code, duration, subject, description and blocked reason. Codex `commandExecution` gets its operation, a single-file target and a subject from `commandActions`. The shell wrapper is removed, and a single search reads as `"TODO" in src`. - **Protocol:** `toolExecution` gains the optional fields `subject`, `description` and `blockedReason`. Generated validators and the manifest are regenerated. - **UI:** - `nativeRunEventsToTranscript` handles `tool.execution.progressed` and passes a canonical `detail` on `tool_call` entries. - The task-chat row shows `$ command`, a path or a pattern, a red `exit N` chip (exit 0 only for commands) and a duration (0 is hidden). - Three or more consecutive reads fold into "Read N files a, b, c +1". - A refused tool shows "Blocked" with its reason. - **Storybook:** `Tasks/Tool rows` compares the data recorded today with the new runner data for Claude and Codex. ## Verification - `cd packages/paperclip-runner && npx vitest run src/provider-events.test.ts src/cli src/drivers/acpx` (1,957 pass) - `cd packages/paperclip-runner/runner && cargo test -p paperclip-runner-core --lib provider_events` - `cd ui && npx vitest run src/components/transcript src/components/task-chat` (973 pass) - `pnpm check:token-gates`, `pnpm --filter paperclip-runner build` (includes the protocol coverage and replay-golden checks) - Live runs on a local instance with real harnesses: - Codex (gpt-5.6-sol): the rows read `$ node --test` with `exit 1` and then `exit 0`, plus real durations. A three-file `sed` shows as one command and does not claim a single file. - Claude (claude-sonnet-5-5 through ACPX): the rows show `Bash` and `Edit` instead of "Terminal" and "Preparing file…". Commands show their real text, `exit 0` and durations, and edits show `Edited temp.js`. - Not seen live: a real Claude `Write`, a folded read run and a Claude policy denial. Unit and Rust tests cover them. ## Risks - The tool-execution schema gains only optional fields, so older events stay valid. Older runs keep rendering with what they recorded. - Commands now cross the sidecar boundary as `subject` (bounded to 4,000 characters). Codex already recorded commands as the tool name, and server redaction still applies. - The tool lifecycle now keeps the latest concrete title. A provider that retitles a call to something less useful would show the newer title. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, code execution and browser automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandPaperclip
|
a6f786479e |
Runner: verify provider eligibility before parking (#19133)
## Thinking Path > - Paperclip coordinates AI agents and their execution. > - Hosted Runners can retain a provider process between turns. > - The host must prove that process is idle before allowing its VM to sleep. > - A completed turn alone does not rule out an active goal or trailing provider work. > - This PR adds a conservative park-readiness hook for the existing Codex process. > - The later host-control consumer can use this proof with its identity and sequence fences. ## Linked Issues or Issue Description **Current behavior** There is no provider eligibility hook for host parking. The earlier implementation in #19060 also missed an active goal that arrived during its tail drain. **Proposed behavior** Default every executor to ineligible. Permit the surviving native Codex process only when its completed-turn authority still matches and it has no live or unsettled work. **Reason and benefit** Keep active goals and unresolved provider obligations awake. Review the provider proof separately from socket handling and the native park/reclaim loop. ## What Changed - Add `host_park_ready` with a default false result. - Refresh the configured credential observer before checking pending delivery. Refuse parking during credential delivery, without selecting or restoring a provider. - Verify the current Codex process and completed-turn identity; reject startup, active turns/goals/tools, and unsettled event queues. - Require completion actually observed by that process. Accepted-finish reconciliation still preserves ordinary result authority without qualifying a restored process to park. - Refuse known helper lineage, whose diagnostics do not prove that helper work has finished. - Drain bounded trailing output through the existing accounting path. Retain goal updates and clears in durable state and normal event delivery, then recheck live goal state. - Add process tests for normal eligibility and the late-goal race. Assert that a restored process is ineligible until it completes an authoritative turn in its new generation. This PR is based on master, which contains the merged Codex (#19132) and ACPX (#19153) recovery foundations. The private socket codec in #19129 is merged; #19140 consumes this proof in the shared native loop. This PR does not advertise parking support. A separate ACPX readiness change must prove its live sidecar/session has no active or unsettled work. Both Codex App Server and qualified ACPX are required in the remaining park/reclaim, Core restart, eviction, and live sleep/wake acceptance phases. ## Verification - `cargo test --offline --locked --manifest-path packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core --test codex_provider hosted_park` — 5 passed. - `ordinary_recovery_restores_a_completed_idle_session` passes and asserts that the genuinely restored provider cannot park before it completes an authoritative turn in its new generation. The fixture proves two provider starts and only one admitted turn. - The initial live-goal recheck mutation failed as expected. Review then identified that goal changes also needed durable retention. The new activation and clear cases verify saved goal state, retained events, and eligibility after delivery. - Existing goal, active-goal attachment, warm-tail, and late-accounting process regressions passed after sharing the goal event path. - The initial late-goal fixture timed out because it omitted `--durable-turn-ids`, which enables its existing first-turn gate. Added that fixture switch; both park tests then passed. - The credential-observation regression fails before the fix and passes afterward. Three focused credential cases prove durable replay and acknowledgement, fresh ACPX startup, and readiness without provider selection. The three provider parking checks also pass after the fix. - Full regression testing is left to GitHub CI. - Changed lines scanned for secrets, personal data, and private deployment references before push. Current head `28d41b083f58ee93db4305b1217719796bf3171d` adds one scoped review-fix commit to readiness `e8ba82374e0f72c82ac3195d4b1b3798c9072bab`, based on master `3d0d178ff7051aa44e07cd873fd4281357e0d815` without duplicating either recovery foundation. The PR remains eight parking-specific files. Review identified quiet helper work and reconstructed accepted-finish authority that could incorrectly qualify. The provider now records its actually observed terminal authority separately in memory, and parking refuses known helper lineage in both the saved state and the live inventory after draining. No durable field, lifecycle, public API, or ordinary recovery policy changed. Both process regressions fail when the parking guards are removed and pass when restored. Seventeen focused checks pass, including ordinary recovery, helper requests, receipt replay, credentials, and ACPX ineligibility. Rustfmt, diff and added-line scans passed. All exact-head CI is green: [PR run](https://github.com/paperclipai/paperclip/actions/runs/42087773928), [Runner Windows](https://github.com/paperclipai/paperclip/actions/runs/42087773720), and [Docker Runner](https://github.com/paperclipai/paperclip/actions/runs/42087773712) passed; Storybook was skipped. Standard review 15 scored 5/5 before the authorized follow-up [Apex review](https://github.com/paperclipai/paperclip/runs/127471966630). Apex review 16 also scored [5/5 on this exact head](https://github.com/paperclipai/paperclip/pull/19133#issuecomment-6645371152), with zero new comments and zero unresolved threads. On this current head, the first server shard 11/12 [job 127458190155](https://github.com/paperclipai/paperclip/actions/runs/42087773928/job/127458190155) failed the unchanged decision-model concurrency test: after releasing the database lock, its `waitFor` observed one provider call instead of two. The identical file passed all 37 tests on the prior reviewed head. One targeted unchanged-head retry [job 127466020677](https://github.com/paperclipai/paperclip/actions/runs/42087773928/job/127466020677) passed all 1,779 tests, including those 37 decision-model cases; 13 tests were skipped. No code, test timeout, or CI configuration changed between attempts. The original failure's cause remains unestablished. On prior reviewed head `e8ba82374e0f72c82ac3195d4b1b3798c9072bab`, the first `General tests (workspaces-c)` job [127389771745](https://github.com/paperclipai/paperclip/actions/runs/42068371976/job/127389771745) failed the unchanged database SIGTERM test at `packages/db/src/test-postgres-command.test.ts:44`: `JSON.parse` read empty readiness-file content immediately after an existence check. The fixture writes directly to that path, leaving a create-before-content race. One targeted unchanged-head retry [127398676711](https://github.com/paperclipai/paperclip/actions/runs/42068371976/job/127398676711) passed all **313 tests in 72 files**, including SIGTERM and SIGINT. No source or timeout changed between attempts. On the superseded `ca088abd552b` head, one Cloud sign-in recovery case returned to the expected task URL but missed its heading within the five-second assertion window; the other 40 cases passed. One targeted unchanged-head retry (job 127136265280) passed all **41 cases**. No test timeout or application code changed between attempts, and the first failure's cause remains unestablished. The superseded c7 parent failed eight retained-cleanup settlement cases because prepared restoration started a provider during a drain-only epoch. The corrected parent preserves ordinary prepared behavior; the new completed-source path excludes prepared providers. The parent reports all 19 settlement rows passing. This PR adds no prepared-session proof or restoration policy. ## Risks Known helpers remain conservatively ineligible even after their progress diagnostics: this slice has no authoritative helper-completion ledger. The eligibility check drains the existing bounded provider tail and can persist late accounting. That accounting must settle before parking succeeds. This hook alone does not prove VM suspension or vendor goal behavior across a process restart. Live sleep/wake qualification remains a later rollout check. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), High reasoning, through Codex with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
03eb49053f |
test(runner): bound Codex provider event waits by a hang ceiling, not 5s (#19306)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Rust runner drives provider processes such as Codex, and CI tests it on Linux, macOS and Windows > - The "Runner Rust (Windows x64)" job fails at random on pull requests that do not touch the runner > - The failures come from Codex provider integration tests that wait for fake-provider events under a fixed five-second deadline > - On a loaded Windows runner, process spawn, antivirus scans and fsyncs of durable state make these exchanges take longer than five seconds, so the wait stops before the event arrives > - This pull request keeps each wait on its condition and changes the deadline to a shared hang ceiling > - The benefit is that unrelated pull requests do not fail on this job, and a real hang still fails the test ## Linked Issues or Issue Description **What happened?** The "Runner Rust (Windows x64)" job failed on pull requests that do not change the runner. Examples: - Run 42073331106 (PR #19273) and run 42058757139: `rejected_declared_completion_stays_failed_when_the_provider_completes` panicked with `called Option::unwrap() on a None value`. The loop ended at its 5 s deadline before `session.failed` and `run.terminal` arrived. - Run 41892735530 (master): the same test failed in the same way. - Runs 41718881566 and 41670873220: `durable_descendant_lineage_survives_capacity_and_provider_restoration` failed with "capacity exhaustion must become a specific durable recovery reason". Its third phase waited only 5 s. An earlier phase of this test already had a 30 s deadline with a comment about this contention. **What did you expect?** The tests wait until the fake provider produces the event. They fail only when the provider hangs. **Steps to reproduce** Run `cargo test --release --workspace` for the runner on a contended Windows x64 runner. The suite takes about 380 s there, and some tests run for more than 60 s. The failures are intermittent. ## What Changed - `tests/codex_provider.rs`: add `EVENT_WAIT_TIMEOUT` (120 s) and `event_deadline()`. Replace all 50 fixed 5 s and 30 s wait deadlines with `event_deadline()`. Each wait already stops when its condition holds, so passing tests do not get slower. - `rejected_declared_completion_stays_failed_when_the_provider_completes`: assert that the run reached `run.terminal`, and print the observed event types if it did not. Replace the bare `unwrap()` on `session.failed` with a message that names the scenario. - `src/bin/fake-codex-app-server.rs`: the two background gate threads used 5 s and 10 s bounds. A test releases a gate only after its own waits succeed, so these bounds had the same problem. They now use one `TEST_GATE_TIMEOUT` of 120 s. - No production code changes. The two remaining `from_secs(5)` values are the unused `max_runtime` in the test config and a Unix-only latency assertion. Both stay the same. ## Verification - `cargo test --release -p paperclip-runner-core --test codex_provider --locked`: 109 passed, 0 failed, 2 ignored. - I ran the two flaky tests 48 times, with 6 copies in parallel per round to load the machine. All 48 runs passed. - `cargo fmt --all --check`: clean. - `cargo clippy -p paperclip-runner-core --test codex_provider --bin fake-codex-app-server`: no new warnings in the changed lines. The lib target already has clippy findings from the local toolchain (Rust 1.97). They are not in the changed files, and CI does not run clippy. - I did not reproduce the original failure on macOS. The slow condition occurs only on the loaded Windows runner. The Windows CI job on this pull request is the main check. ## Risks - Low risk. The change affects only tests and the test fixture binary. - If a test hangs for real, it now fails after 120 s and not after 5 s. This makes a broken run slower, but the failure message stays the same. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
1eff07f29e |
feat(task-chat): show Claude subagents as a collapsible tree in the runner turn (#19264)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The task chat shows what an agent does in a run. The Paperclip Runner sends this activity as canonical provider events. > - Claude runs subagents through its Agent (Task) tool. A run that delegates showed a confusing burst of unrelated tool calls. > - The subagent data was already sent, but it was lost: ACPX dropped `_meta`, the hardened sidecar dropped `rawInput`, and async subagents report "launched" as the tool result. > - This pull request maps each Agent call to a canonical `delegation` event, links each child tool to its subagent, and shows subagents as a collapsible card or tree. > - The benefit is that operators can see what each subagent was asked, what it is doing now, which tools it used, and what it returned. ## Linked Issues or Issue Description Refs #19252 (the docked plan card used the same canonical-event pattern) Refs #19259 (ACPX `_meta` passthrough, merged). This PR builds on it. **What happened?** When a Claude agent used its Agent tool, the task chat showed one generic "Task" row and then the subagents' tool calls as flat rows in the parent turn. The subagent type, prompt, progress and result were not visible. **Expected behavior** Each subagent shows as its own card with its type, description, prompt, live progress summary, nested tool rows and result. Subagents that start together show as one tree with status dots. **Steps to reproduce** 1. Assign a Claude (Paperclip Runner, ACPX) agent a task that asks it to launch three Explore subagents in parallel. 2. Open the task while the run is live. 3. Before this change, the subagents' tool calls appear flat in the parent turn. ## What Changed - **claude-agent-acp patch:** Under the isolated runner context, the patch turns on the SDK option `agentProgressSummaries`. It forwards each subagent's `task_progress` summary and its terminal state and summary onto the spawning tool call as `_meta.claudeCode.subagentProgress` and `subagentState`. Native subagent sessions stay off, because ACPX cannot follow child ACP sessions. - **Runner:** The new file `claude-subagent-lineage.ts` tracks each Agent/Task call across ACP update deltas. It maps the call to `delegation.started|updated|completed` events and stamps child tools with `parentItemId`. An async subagent stays running after its "Async agent launched" result until its own state settles. - **Sidecar path:** `acpx-runtime-sidecar.ts` reads the lineage before it drops `_meta` and `rawInput`. Delegations go out as schema-checked rich events. Child tools carry `parentToolCallId`, which Rust (`acpx_event_payload.rs`, `provider_events.rs`) maps to `parentItemId`. - **Contracts:** The provider-event schema adds optional `prompt` and `result` to delegation children and optional `parentItemId` to tool executions. The protocol modules are regenerated. ACPX Claude now advertises the `delegation` event family. - **UI:** `native-run-events.ts` keeps delegation payloads and the tool parent link. `buildTurnTimelineRows` groups adjacent delegations into a new `subagent_group` row, nests child tool rows (plain and provider-activity shapes), and hides the spawning call's own tool row. The new `TaskChatSubagentCard.tsx` renders the card and tree. A new size token caps the result height. - **Storybook:** The new story `Tasks and conversations/…/Task chat/Subagents` has 7 states. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/cli src/drivers/acpx src/provider-events.test.ts src/claude-subagent-lineage.test.ts src/protocol` (2069 passed) - `cargo test --release --locked -p paperclip-runner-core --lib` in `packages/paperclip-runner/runner` (488 passed, including the new parent-link test) - `node --test test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs` (48 passed) - `pnpm --filter @paperclipai/ui exec vitest run src/components/task-chat src/components/transcript`, including the new `subagent-timeline.test.ts` - `check:protocol-types`, `check:protocol-manifest`, `check:acpx-profiles` and `pnpm check:token-gates` pass. - **Live test:** A local test-drive instance ran a Claude Opus 5.5 agent on the Paperclip Runner (hardened sidecar path). The agent ran real tasks that launched three parallel Explore subagents. The live view showed "Running 3 subagents · 2 done" with a progress summary. The settled view showed "Ran 3 subagents", each with its prompt, nested tools and its own result text. ## Risks - `agentProgressSummaries` makes Claude Code generate a short progress summary for each running subagent about every 30 seconds. This adds a small model cost to runs that delegate. - The adapter patch reads `task_notification` and terminal `task_updated` messages. A future claude-agent-acp version can change these. The patch hunks then fail to apply, and the dependency bump must update them. - If a delegation event is lost, child tools stay at the top level as before. Nothing is hidden without a matching delegation. - Codex is not changed. Codex subagents need `multi_agent`, which Paperclip keeps off on purpose. Changing that is a separate policy decision. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, browser automation). It did the investigation, implementation, tests and live test. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
d361f92071 |
feat(task-chat): show harness status notices as chips, out of the answer (#19267)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run through the native runner. The runner turns harness events from Codex and Claude into canonical provider events, and the task chat shows them. > - Harnesses report their own lifecycle: context compaction, retries, model fallback, and warnings. Paperclip lost or mangled these reports. > - Claude (claude-agent-acp 0.73) sends compaction, model-fallback and warning banners as ordinary message text, so "Compacting..." ended up inside the agent's answer and the issue comment. > - Codex sends `error` with the text in `error.message` and a `willRetry` flag. Paperclip read `params.message`, so every Codex error said "Provider notice", and every retry looked fatal. > - This pull request keeps these reports out of the answer and shows them as small status chips between the turn's activity. > - The benefit is that users can see why an agent paused, retried or forgot earlier context, and the answer text stays clean. ## Linked Issues or Issue Description **Subsystem affected** Native runner provider-event mapping (TypeScript driver, ACPX sidecar and Rust runner-core) and the task chat transcript. **Problem or motivation** Claude status banners are mixed into assistant answer text. Codex error notices lose their message and their retry state. The task chat hid all of these notices, and settled turns stripped every provider notice. **Proposed solution** Classify Claude banners as provider notices at the ACPX boundary. Map Codex `TurnError` fields and `willRetry` correctly. Render status notices as inline chips. A retry chip fades after later provider work. A final error shows the real provider message. **Alternatives considered** Patch claude-agent-acp to tag banners in `_meta`. ACPX keeps only `origin`, `kind` and `source` from `_meta`, and other sessions own the ACPX `_meta` passthrough work. Banner chunks already have no `messageId`, and model output always has one, so the classification needs no adapter patch. **Roadmap alignment** This makes existing harness signals visible. It adds no new protocol schema. It uses the existing `paperclip.provider.notice.v1` `category` and `details` fields. ## What Changed - Codex `error` notifications (TypeScript `provider-events.ts` and Rust `runner-core/src/provider_events.rs`): read `error.message`, map `willRetry: true` to a recoverable `provider_retry` notice, keep category `error` for a final error, and add the `CodexErrorInfo` kind, HTTP status, and the "Reconnecting... n/m" attempt count as notice details. Retries in one turn share one notice id. - New `claude-status-banners.ts`: classifies the claude-agent-acp compaction, model-fallback and SDK informational banners. A chunk must have no `messageId` and match a known banner exactly. - ACPX sidecar and the TypeScript ACPX driver emit these banners as `provider_notice` events instead of answer text. Compaction start and end share one notice key and carry a `Phase` detail. - Rust runner-core accepts an optional `noticeKey` and bounded `details` on ACPX `provider_notice` events. - UI: new `TaskChatStatusNotice` chip and `status-notice-presentation.ts`. Status notices become top-level timeline rows, render as chips wherever protocol cards render, and stay visible in settled (historical) turns. Generic warnings stay hidden as before. - New tokens `--tc-status-notice-warning-*` in `ui/src/index.css`. - Storybook story `Tasks and conversations/.../Task chat/Status notices` with eight states. - `doc/run-log-events.md`: describes the status notice categories. ## Verification - `pnpm --filter @paperclipai/ui exec vitest run src/components/task-chat src/components/transcript` - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/provider-events.test.ts src/drivers/acpx/claude-status-banners.test.ts src/drivers/acpx/codex-acpx-driver.test.ts src/cli/acpx-runtime-sidecar.test.ts` - `cargo test -p paperclip-runner-core --lib provider_events` and `--test acpx_provider_events` - `pnpm check:token-gates` - Storybook: open `product-task-chat-status-notices--all-states-gallery`. - Live test-drive on an isolated worktree instance: - Codex (gpt-5.6-sol, app-server 0.160): a local pass-through proxy in front of the real OpenAI API cut the first two response streams. Codex ran its own retries. The chat showed an amber "Reconnecting" chip live, then "Recovered after retry" once work continued. With every stream cut, Codex made five retries and then sent a final error with the real message. - Claude (claude-haiku-5-5 through claude-agent-acp 0.73): a long reading task triggered Claude's own auto-compaction twice. The chat showed "Compacting context…" live and "Context compacted" afterwards. The final answer and the issue comment contained no compaction text. - Not verified live: model fallback and Claude informational banners. They need a model refusal or a blocking hook. Unit tests cover them. ## Risks - Low to medium. A Claude model message that is exactly a banner text and has no `messageId` would become a notice. Model output from claude-agent-acp always carries a `messageId`. - Older stored Codex `error` notices (category `error`, not recoverable) now render as a red row instead of being hidden. - Claude retries (`api_retry`) are still not visible. claude-agent-acp reports them only to clients that negotiate session-failure records, and ACPX drops `_meta`. This is a follow-up that depends on the ACPX `_meta` passthrough work. - On a failed Codex turn the thread collapses to a summary, so the red row does not show there. The existing task banner already shows the real provider message. - Some local suites failed on this machine. They do not touch the changed code: Cursor Cloud staging-custody tests, the ACPX Claude dependency-graph pin test (a timeout), and the Rust `task_memory` monitor tests. CI is the reference for these. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, code execution and browser automation. Default context window. Extended reasoning was not enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs paperclipai/paperclip#19252 (plan card precedent). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
cb0b7525e0 |
feat(task-chat): warn at the top of a run when an MCP server failed to start (#19289)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Each agent run shows its activity in the task chat. The Paperclip runner turns harness events into that activity. > - Codex reports `mcpServer/startupStatus/updated` when an MCP server fails to start. The runner dropped this event at the notification allowlist and in the Rust normalizer. > - When an MCP server fails, the agent runs without its tools. The agent then says "I could not reach the server", and the user cannot see why. > - This pull request maps a failed MCP start to a user-facing provider notice and shows it as one warning line at the top of the run. > - The benefit is that the cause of a "cannot reach my tools" answer is visible next to that answer. ## Linked Issues or Issue Description Refs #19252 (the plan card, which set the pattern for promoting a dropped harness signal to a dedicated task-chat surface). **Problem description** Codex sends `mcpServer/startupStatus/updated` (`name`, `status`, `error`, `failureReason`) during `thread/start`. The TypeScript allowlist (`codex-thread-normalization.ts`) and the Rust normalizer (`runner-core/src/provider_events.rs`) both dropped it. A run with a broken MCP server showed no sign of the failure. The agent only said that it could not reach the tools. **Proposed solution** Map `status: "failed"` to `provider.notice.recorded` with `category: "mcp_server_startup_failed"`, `severity: "warning"`, `scope: "session"` and `userActionable: true`. Show these notices as one run-health line under the runner header: "github MCP server failed to start: No such file or directory (os error 2)". **Alternatives considered** An inline notice row in the timeline. It scrolls away and sits among tool rows. A run-level line next to the header is easier to connect to the agent's answer. ## What Changed - Runner (Rust, production Codex path): `normalize_codex_notification` maps a failed `mcpServer/startupStatus/updated` to a `provider.notice.recorded` notice. `starting`, `ready` and `cancelled` stay out of the transcript. The new helper removes the Codex wrapper text ("MCP client for \`x\` failed to start: MCP startup failed:") so that the line shows the real reason. `reauthenticationRequired` becomes "needs re-authentication". - Runner (TypeScript): the same mapping in `canonicalProviderEventsFromCodex`, and the method is added to `isSupportedCodexNotificationMethod`. - Server: the `view=conversation` run-event query keeps `mcp_server_startup_failed` notices. A settled run can then still show the warning. Other notices stay out of that view. - UI: new `TaskChatRunHealthBanner`. `partitionRunHealthItems` takes MCP failures out of the run's items, with one line per server where the latest report wins. `TaskChatRunnerTurn` (live runs) and `TaskChatTurn` (settled, historical and merged turns) show it under the header. `coalesceSettledTurns` merges the failures when it merges turns. - Storybook: `Tasks and conversations/…/Task chat/Run health` shows one failure, several failures, a finished run and a healthy run, all through the production runner turn. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/provider-events.test.ts src/drivers/codex src/protocol`: 508 tests pass. - `cargo test --release -p paperclip-runner-core --lib -- provider_events`: 20 tests pass. - `pnpm --filter @paperclipai/ui exec vitest run src/components/task-chat src/components/transcript src/components/TaskChatThread.test.tsx`: 1340 tests pass, including the new `TaskChatRunHealthBanner.test.tsx` (persisted event to rendered banner, deduplication, settled historical turn, merged turns). - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/task-cold-open.test.ts`: the conversation view keeps the MCP notice and still excludes other notices. - `pnpm check:token-gates` is clean. `node scripts/typecheck-low-memory.mjs` passes. - Live test on a local instance with Codex 0.160: a Codex agent with a preconfigured Codex home whose `config.toml` has `[mcp_servers.github] command = "github-mcp-server-not-installed"`. Codex reported the failure during `thread/start`. The persisted event had the summary "github MCP server failed to start: No such file or directory (os error 2)". The banner showed while the run was live. It also showed on the settled turn after "More run output", above the agent's answer that it could not reach the GitHub MCP server. ## Risks - Low risk. The change adds one notice category. It does not change how existing notices map or render. - A settled run whose transcript is deferred on a cold open (a successful run with a posted agent comment) shows the warning only after "More run output". This is the same as all other run activity under the current compact history. - Claude does not report MCP startup failures over ACP. `claude-agent-acp` reads MCP status only for OAuth. This change is therefore Codex-only. - Provisional task titles from provider session names are not part of this change. Codex app-server does not generate thread names (only its TUI does). Claude generates its title after the turn ends, and ACPX drops the title text. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use (shell, file edits, browser automation for Storybook and the live test). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |