mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(runner): poll a live ACPX turn without waiting out the idle read (#21525)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runner (runnerd) relays provider events to the server and waits for cumulative ACKs before it polls the provider again. > - After a control frame, runnerd uses a 1 ms queued read only when the executor reports a live provider poll. Otherwise it waits up to the 250 ms runtime read timeout, and any frame that arrives in that time skips the poll. > - Only the Codex facade reports a live poll. ACPX runs (Claude, Codex through ACPX and others) always wait the full 250 ms after the last ACK of every batch, and steady control traffic starves their provider polls. > - This pull request lets an ACPX executor report a live poll while its restored session has an active turn. > - The benefit is that ACPX output is relayed without the 250 ms idle wait, and tool-heavy turns no longer starve provider polls. ## Linked Issues or Issue Description **What happened?** A routine run started several subagents that made about 190 tool calls in about 10 minutes. On the server, consecutive runner events arrived at least 576 ms apart (median about 1.5 seconds), one event at a time, while the time from each event's `emittedAt` to its insert was 0 to 0.1 seconds. The runner fell several minutes behind the provider. The backlog then overflowed the transport's event buffer (fixed separately in #21524) and the turn failed. **Expected behavior** A live ACPX turn is polled again as soon as its outstanding events are acknowledged and no control frame is queued, the same as a Codex facade turn. **Steps to reproduce** 1. Run an ACPX turn that produces events while the controller sends commands and ACKs. 2. After each ACK, runnerd calls `receive_json`, which waits up to 250 ms before it polls the provider. A frame that arrives in that time skips the poll. **Relevant logs or output** ``` min gap between consecutive runner events: 576 ms, median about 1.5 s, never batched insert time - emittedAt: 0.0 to 0.1 s ``` Related: #21524 bounds the receive buffer that this backlog overflowed. ## What Changed - `acpx_provider_backend.rs`: `AcpxCommandExecutor::provider_poll_is_live` returns true only when the session was restored, has no persistence error, does not wait for a next-turn generation, and its state is `turn_starting` or `turn_active` with an active turn, no unconfirmed provider exit, and the run it owns. - `native_provider_backend.rs`: `SelectedExecutor` forwards `provider_poll_is_live` to the ACPX executor. - `tests/acpx_warm_attachment.rs`: new test through `NativeProviderCommandExecutor`. The session is not live before `run.prepare` or after `session.open`, is live after `turn.start`, and a new executor that has not restored its session is not live. What does not change: - runnerd still polls only when the read returns no frame, so a queued stop or suspend still runs before the next provider poll. - The ACK gate (no poll while sent events are not acknowledged) does not change. - The 1 ms read is used only while the source sequence advanced since the last quiet poll, as for Codex today, so an idle turn does not busy-loop. ## Verification - `cargo test --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --test acpx_warm_attachment` (in `packages/paperclip-runner`): 7 passed. The new test fails without the new method, because the default returns false. - `cargo fmt --all -- --check` and `cargo check --locked --workspace` pass. - `cargo test --release --locked -p paperclip-runner-core --no-fail-fast`: 1,127 passed, 3 failed. The failures are `codex_provider::tests::fenced_receipt_output_reports_capacity_without_dropping_known_frames` ("failed to signal owned process group: Operation not permitted") and `stdio_mcp::lost_acknowledgement_blocks_calls_until_socket_replacement` and `metadata_only_socket_changes_do_not_bypass_unknown_outcome` (relay exit timing). They also fail without this change in the macOS sandbox where I ran the suite, and none of them uses the ACPX executor. CI covers them on Linux. - The existing `durable::tests::ack_release_poll` cases cover the runner side of the live read: an ACK releases the next poll without the read timeout, and a command queued behind the ACK runs first. - I did not run a live provider. After the deploy, the gap between runner events of an ACPX run with queued output should drop below 250 ms. ## Risks - Low risk. The change only selects the existing 1 ms queued read for ACPX during an active turn. - A turn that produces events continuously now polls more often. Each poll still writes the same durable state as before, so disk writes per event do not change. - Analysis also found that runnerd waits for every sent event to be acknowledged before it polls again. A bounded in-flight window would raise throughput further, but it changes the tested rule that control queued behind an ACK runs before the next poll. That is left out of this pull request on purpose. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
254744f77c
commit
493519fc88
3 files changed
+102
No files matched your search
@@ -3244,6 +3244,21 @@ impl CommandExecutor for AcpxCommandExecutor {
|
||||
self.retained_events()
|
||||
}
|
||||
|
||||
fn provider_poll_is_live(&self) -> bool {
|
||||
self.restore_checked
|
||||
&& self.persistence_error.is_none()
|
||||
&& self
|
||||
.session
|
||||
.as_ref()
|
||||
.is_some_and(|session| !session.awaits_next_turn())
|
||||
&& self.state.as_ref().is_some_and(|state| {
|
||||
matches!(state.lifecycle.as_str(), "turn_starting" | "turn_active")
|
||||
&& state.active_turn_id.is_some()
|
||||
&& !state.provider_exit_unconfirmed
|
||||
&& state.descriptor.run_id == self.context.run_id
|
||||
})
|
||||
}
|
||||
|
||||
fn poll_disconnected_events(&mut self) -> Result<Vec<PolledEvent>, DurableRunnerError> {
|
||||
self.ensure_persistence_healthy()?;
|
||||
if self.session.is_none()
|
||||
|
||||
@@ -99,6 +99,7 @@ impl CommandExecutor for SelectedExecutor {
|
||||
fn provider_poll_is_live(&self) -> bool {
|
||||
match self {
|
||||
Self::LocalFacade(executor) => executor.provider_poll_is_live(),
|
||||
Self::Acpx(executor) => executor.provider_poll_is_live(),
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -9,6 +9,7 @@ use paperclip_runner_core::acpx_provider_backend::AcpxCommandExecutor;
|
||||
use paperclip_runner_core::durable::{
|
||||
AcpxLaunchProfile, Command, CommandExecutor, DurableRunnerConfig, QualifiedLaunchArtifact,
|
||||
};
|
||||
use paperclip_runner_core::native_provider_backend::NativeProviderCommandExecutor;
|
||||
use serde_json::{json, Value};
|
||||
use sha2::{Digest, Sha256};
|
||||
|
||||
@@ -309,6 +310,91 @@ fn reopens_the_conversation_for_a_turn_after_a_suspension_stop() {
|
||||
original.shutdown().unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn only_a_restored_session_with_an_active_turn_polls_as_live() {
|
||||
let fixture =
|
||||
Fixture(std::env::temp_dir().join(format!("acpx-live-poll-{}", uuid::Uuid::new_v4())));
|
||||
fs::create_dir_all(&fixture.0).unwrap();
|
||||
fs::set_permissions(&fixture.0, fs::Permissions::from_mode(0o700)).unwrap();
|
||||
let sidecar = fixture.0.join("sidecar");
|
||||
fs::copy(env!("CARGO_BIN_EXE_fake-acpx-sidecar"), &sidecar).unwrap();
|
||||
fs::set_permissions(&sidecar, fs::Permissions::from_mode(0o700)).unwrap();
|
||||
let ports = unused_fence_ports();
|
||||
let args = vec![
|
||||
"--mode".to_owned(),
|
||||
"suspend".to_owned(),
|
||||
"--profile-digest".to_owned(),
|
||||
PROFILE_DIGEST.to_owned(),
|
||||
"--lifetime-fence-ports".to_owned(),
|
||||
format!("{},{},{}", ports[0], ports[1], ports[2]),
|
||||
];
|
||||
let config = DurableRunnerConfig {
|
||||
connect_url: "ws://127.0.0.1/runner".to_owned(),
|
||||
ca_bundle_path: None,
|
||||
state_dir: fixture.0.clone(),
|
||||
runner_instance_id: "runner-1".to_owned(),
|
||||
environment_lease_id: "lease-1".to_owned(),
|
||||
run_id: "run-1".to_owned(),
|
||||
normalized_session_id: "session-1".to_owned(),
|
||||
turn_id: "turn-1".to_owned(),
|
||||
item_id: "item-1".to_owned(),
|
||||
runner_version: "0.0.0".to_owned(),
|
||||
runner_digest: format!("sha256:{}", "a".repeat(64)),
|
||||
acpx_launch_profile: Some(AcpxLaunchProfile {
|
||||
candidate_admission: None,
|
||||
authority_digest: format!("sha256:{}", "d".repeat(64)),
|
||||
command: sidecar.clone(),
|
||||
args: args.clone(),
|
||||
artifacts: vec![QualifiedLaunchArtifact {
|
||||
path: sidecar.clone(),
|
||||
sha256: format!("sha256:{:x}", Sha256::digest(fs::read(&sidecar).unwrap())),
|
||||
}],
|
||||
}),
|
||||
opencode_launch_profile: None,
|
||||
cursor_cloud_launch_profile: None,
|
||||
max_outbox_bytes: 1024 * 1024,
|
||||
p0_reserve_bytes: 64 * 1024,
|
||||
max_frame_bytes: 1024 * 1024,
|
||||
reconnect_delay: Duration::from_millis(1),
|
||||
reconnect_grace: None,
|
||||
max_runtime: Duration::from_secs(30),
|
||||
turn_scoped: false,
|
||||
};
|
||||
let descriptor = json!({
|
||||
"kind": "acpx", "provider": "acpx", "driver": "acpx_runtime",
|
||||
"providerVersion": "0.22.0", "agent": "codex", "model": "gpt-5.6-sol",
|
||||
"acpxVersion": "0.22.0", "agentServerPackage": "@agentclientprotocol/codex-acp",
|
||||
"agentServerVersion": "1.6.2", "agentRuntimePackage": "@openai/codex",
|
||||
"agentRuntimeVersion": "0.160.0", "commandDigest": PROFILE_DIGEST,
|
||||
"sidecarCommand": sidecar, "sidecarArgs": args, "runtimeDirectory": fixture.0,
|
||||
"normalizedSessionId": "session-1", "runId": "run-1", "cwd": fixture.0,
|
||||
"instructions": "Complete the supplied work.",
|
||||
"permissionMode": "approve-all", "permissionModePinned": true,
|
||||
});
|
||||
let mut executor = NativeProviderCommandExecutor::with_runner_config(&fixture.0, &config);
|
||||
assert!(!executor.provider_poll_is_live());
|
||||
executor
|
||||
.execute(&command(1, "run.prepare", json!({"provider": descriptor})))
|
||||
.unwrap();
|
||||
executor
|
||||
.execute(&command(2, "session.open", json!({})))
|
||||
.unwrap();
|
||||
assert!(!executor.provider_poll_is_live());
|
||||
let started = executor
|
||||
.execute(&command(
|
||||
3,
|
||||
"turn.start",
|
||||
json!({"turnId": "provider-turn-1", "text": "Do the work."}),
|
||||
))
|
||||
.unwrap();
|
||||
assert_eq!(started.result["status"], "accepted");
|
||||
assert!(executor.provider_poll_is_live());
|
||||
drop(executor);
|
||||
|
||||
let restored = NativeProviderCommandExecutor::with_runner_config(&fixture.0, &config);
|
||||
assert!(!restored.provider_poll_is_live());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn restores_the_conversation_after_losing_an_active_turn_to_a_runner_restart() {
|
||||
let fixture =
|
||||
|
||||
Reference in new issue
Block a user