fix(runner): poll a live ACPX turn without waiting out the idle read (#21525)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runner (runnerd) relays provider events to the server and
waits for cumulative ACKs before it polls the provider again.
> - After a control frame, runnerd uses a 1 ms queued read only when the
executor reports a live provider poll. Otherwise it waits up to the 250
ms runtime read timeout, and any frame that arrives in that time skips
the poll.
> - Only the Codex facade reports a live poll. ACPX runs (Claude, Codex
through ACPX and others) always wait the full 250 ms after the last ACK
of every batch, and steady control traffic starves their provider polls.
> - This pull request lets an ACPX executor report a live poll while its
restored session has an active turn.
> - The benefit is that ACPX output is relayed without the 250 ms idle
wait, and tool-heavy turns no longer starve provider polls.

## Linked Issues or Issue Description

**What happened?**
A routine run started several subagents that made about 190 tool calls
in about 10 minutes. On the server, consecutive runner events arrived at
least 576 ms apart (median about 1.5 seconds), one event at a time,
while the time from each event's `emittedAt` to its insert was 0 to 0.1
seconds. The runner fell several minutes behind the provider. The
backlog then overflowed the transport's event buffer (fixed separately
in #21524) and the turn failed.

**Expected behavior**
A live ACPX turn is polled again as soon as its outstanding events are
acknowledged and no control frame is queued, the same as a Codex facade
turn.

**Steps to reproduce**
1. Run an ACPX turn that produces events while the controller sends
commands and ACKs.
2. After each ACK, runnerd calls `receive_json`, which waits up to 250
ms before it polls the provider. A frame that arrives in that time skips
the poll.

**Relevant logs or output**
```
min gap between consecutive runner events: 576 ms, median about 1.5 s, never batched
insert time - emittedAt: 0.0 to 0.1 s
```

Related: #21524 bounds the receive buffer that this backlog overflowed.

## What Changed

- `acpx_provider_backend.rs`:
`AcpxCommandExecutor::provider_poll_is_live` returns true only when the
session was restored, has no persistence error, does not wait for a
next-turn generation, and its state is `turn_starting` or `turn_active`
with an active turn, no unconfirmed provider exit, and the run it owns.
- `native_provider_backend.rs`: `SelectedExecutor` forwards
`provider_poll_is_live` to the ACPX executor.
- `tests/acpx_warm_attachment.rs`: new test through
`NativeProviderCommandExecutor`. The session is not live before
`run.prepare` or after `session.open`, is live after `turn.start`, and a
new executor that has not restored its session is not live.

What does not change:
- runnerd still polls only when the read returns no frame, so a queued
stop or suspend still runs before the next provider poll.
- The ACK gate (no poll while sent events are not acknowledged) does not
change.
- The 1 ms read is used only while the source sequence advanced since
the last quiet poll, as for Codex today, so an idle turn does not
busy-loop.

## Verification

- `cargo test --release --manifest-path runner/Cargo.toml --locked -p
paperclip-runner-core --test acpx_warm_attachment` (in
`packages/paperclip-runner`): 7 passed. The new test fails without the
new method, because the default returns false.
- `cargo fmt --all -- --check` and `cargo check --locked --workspace`
pass.
- `cargo test --release --locked -p paperclip-runner-core
--no-fail-fast`: 1,127 passed, 3 failed. The failures are
`codex_provider::tests::fenced_receipt_output_reports_capacity_without_dropping_known_frames`
("failed to signal owned process group: Operation not permitted") and
`stdio_mcp::lost_acknowledgement_blocks_calls_until_socket_replacement`
and `metadata_only_socket_changes_do_not_bypass_unknown_outcome` (relay
exit timing). They also fail without this change in the macOS sandbox
where I ran the suite, and none of them uses the ACPX executor. CI
covers them on Linux.
- The existing `durable::tests::ack_release_poll` cases cover the runner
side of the live read: an ACK releases the next poll without the read
timeout, and a command queued behind the ACK runs first.
- I did not run a live provider. After the deploy, the gap between
runner events of an ACPX run with queued output should drop below 250
ms.

## Risks

- Low risk. The change only selects the existing 1 ms queued read for
ACPX during an active turn.
- A turn that produces events continuously now polls more often. Each
poll still writes the same durable state as before, so disk writes per
event do not change.
- Analysis also found that runnerd waits for every sent event to be
acknowledged before it polls again. A bounded in-flight window would
raise throughput further, but it changes the tested rule that control
queued behind an ACK runs before the next poll. That is left out of this
pull request on purpose.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin FoleyandPaperclip authored and GitHub committed 2026-11-13 16:46:05 -08:00
1 parent 254744f77c
commit 493519fc88
3 files changed
+102

No files matched your search

@@ -3244,6 +3244,21 @@ impl CommandExecutor for AcpxCommandExecutor {
self.retained_events()
}
fn provider_poll_is_live(&self) -> bool {
self.restore_checked
&& self.persistence_error.is_none()
&& self
.session
.as_ref()
.is_some_and(|session| !session.awaits_next_turn())
&& self.state.as_ref().is_some_and(|state| {
matches!(state.lifecycle.as_str(), "turn_starting" | "turn_active")
&& state.active_turn_id.is_some()
&& !state.provider_exit_unconfirmed
&& state.descriptor.run_id == self.context.run_id
})
}
fn poll_disconnected_events(&mut self) -> Result<Vec<PolledEvent>, DurableRunnerError> {
self.ensure_persistence_healthy()?;
if self.session.is_none()
@@ -99,6 +99,7 @@ impl CommandExecutor for SelectedExecutor {
fn provider_poll_is_live(&self) -> bool {
match self {
Self::LocalFacade(executor) => executor.provider_poll_is_live(),
Self::Acpx(executor) => executor.provider_poll_is_live(),
_ => false,
}
}
@@ -9,6 +9,7 @@ use paperclip_runner_core::acpx_provider_backend::AcpxCommandExecutor;
use paperclip_runner_core::durable::{
AcpxLaunchProfile, Command, CommandExecutor, DurableRunnerConfig, QualifiedLaunchArtifact,
};
use paperclip_runner_core::native_provider_backend::NativeProviderCommandExecutor;
use serde_json::{json, Value};
use sha2::{Digest, Sha256};
@@ -309,6 +310,91 @@ fn reopens_the_conversation_for_a_turn_after_a_suspension_stop() {
original.shutdown().unwrap();
}
#[test]
fn only_a_restored_session_with_an_active_turn_polls_as_live() {
let fixture =
Fixture(std::env::temp_dir().join(format!("acpx-live-poll-{}", uuid::Uuid::new_v4())));
fs::create_dir_all(&fixture.0).unwrap();
fs::set_permissions(&fixture.0, fs::Permissions::from_mode(0o700)).unwrap();
let sidecar = fixture.0.join("sidecar");
fs::copy(env!("CARGO_BIN_EXE_fake-acpx-sidecar"), &sidecar).unwrap();
fs::set_permissions(&sidecar, fs::Permissions::from_mode(0o700)).unwrap();
let ports = unused_fence_ports();
let args = vec![
"--mode".to_owned(),
"suspend".to_owned(),
"--profile-digest".to_owned(),
PROFILE_DIGEST.to_owned(),
"--lifetime-fence-ports".to_owned(),
format!("{},{},{}", ports[0], ports[1], ports[2]),
];
let config = DurableRunnerConfig {
connect_url: "ws://127.0.0.1/runner".to_owned(),
ca_bundle_path: None,
state_dir: fixture.0.clone(),
runner_instance_id: "runner-1".to_owned(),
environment_lease_id: "lease-1".to_owned(),
run_id: "run-1".to_owned(),
normalized_session_id: "session-1".to_owned(),
turn_id: "turn-1".to_owned(),
item_id: "item-1".to_owned(),
runner_version: "0.0.0".to_owned(),
runner_digest: format!("sha256:{}", "a".repeat(64)),
acpx_launch_profile: Some(AcpxLaunchProfile {
candidate_admission: None,
authority_digest: format!("sha256:{}", "d".repeat(64)),
command: sidecar.clone(),
args: args.clone(),
artifacts: vec![QualifiedLaunchArtifact {
path: sidecar.clone(),
sha256: format!("sha256:{:x}", Sha256::digest(fs::read(&sidecar).unwrap())),
}],
}),
opencode_launch_profile: None,
cursor_cloud_launch_profile: None,
max_outbox_bytes: 1024 * 1024,
p0_reserve_bytes: 64 * 1024,
max_frame_bytes: 1024 * 1024,
reconnect_delay: Duration::from_millis(1),
reconnect_grace: None,
max_runtime: Duration::from_secs(30),
turn_scoped: false,
};
let descriptor = json!({
"kind": "acpx", "provider": "acpx", "driver": "acpx_runtime",
"providerVersion": "0.22.0", "agent": "codex", "model": "gpt-5.6-sol",
"acpxVersion": "0.22.0", "agentServerPackage": "@agentclientprotocol/codex-acp",
"agentServerVersion": "1.6.2", "agentRuntimePackage": "@openai/codex",
"agentRuntimeVersion": "0.160.0", "commandDigest": PROFILE_DIGEST,
"sidecarCommand": sidecar, "sidecarArgs": args, "runtimeDirectory": fixture.0,
"normalizedSessionId": "session-1", "runId": "run-1", "cwd": fixture.0,
"instructions": "Complete the supplied work.",
"permissionMode": "approve-all", "permissionModePinned": true,
});
let mut executor = NativeProviderCommandExecutor::with_runner_config(&fixture.0, &config);
assert!(!executor.provider_poll_is_live());
executor
.execute(&command(1, "run.prepare", json!({"provider": descriptor})))
.unwrap();
executor
.execute(&command(2, "session.open", json!({})))
.unwrap();
assert!(!executor.provider_poll_is_live());
let started = executor
.execute(&command(
3,
"turn.start",
json!({"turnId": "provider-turn-1", "text": "Do the work."}),
))
.unwrap();
assert_eq!(started.result["status"], "accepted");
assert!(executor.provider_poll_is_live());
drop(executor);
let restored = NativeProviderCommandExecutor::with_runner_config(&fixture.0, &config);
assert!(!restored.provider_poll_is_live());
}
#[test]
fn restores_the_conversation_after_losing_an_active_turn_to_a_runner_restart() {
let fixture =