From a6f786479e46ad70da4d2fce3b13171771e1ebda Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Thu, 5 Nov 2026 11:24:36 -0800 Subject: [PATCH] Runner: verify provider eligibility before parking (#19133) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip coordinates AI agents and their execution. > - Hosted Runners can retain a provider process between turns. > - The host must prove that process is idle before allowing its VM to sleep. > - A completed turn alone does not rule out an active goal or trailing provider work. > - This PR adds a conservative park-readiness hook for the existing Codex process. > - The later host-control consumer can use this proof with its identity and sequence fences. ## Linked Issues or Issue Description **Current behavior** There is no provider eligibility hook for host parking. The earlier implementation in #19060 also missed an active goal that arrived during its tail drain. **Proposed behavior** Default every executor to ineligible. Permit the surviving native Codex process only when its completed-turn authority still matches and it has no live or unsettled work. **Reason and benefit** Keep active goals and unresolved provider obligations awake. Review the provider proof separately from socket handling and the native park/reclaim loop. ## What Changed - Add `host_park_ready` with a default false result. - Refresh the configured credential observer before checking pending delivery. Refuse parking during credential delivery, without selecting or restoring a provider. - Verify the current Codex process and completed-turn identity; reject startup, active turns/goals/tools, and unsettled event queues. - Require completion actually observed by that process. Accepted-finish reconciliation still preserves ordinary result authority without qualifying a restored process to park. - Refuse known helper lineage, whose diagnostics do not prove that helper work has finished. - Drain bounded trailing output through the existing accounting path. Retain goal updates and clears in durable state and normal event delivery, then recheck live goal state. - Add process tests for normal eligibility and the late-goal race. Assert that a restored process is ineligible until it completes an authoritative turn in its new generation. This PR is based on master, which contains the merged Codex (#19132) and ACPX (#19153) recovery foundations. The private socket codec in #19129 is merged; #19140 consumes this proof in the shared native loop. This PR does not advertise parking support. A separate ACPX readiness change must prove its live sidecar/session has no active or unsettled work. Both Codex App Server and qualified ACPX are required in the remaining park/reclaim, Core restart, eviction, and live sleep/wake acceptance phases. ## Verification - `cargo test --offline --locked --manifest-path packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core --test codex_provider hosted_park` — 5 passed. - `ordinary_recovery_restores_a_completed_idle_session` passes and asserts that the genuinely restored provider cannot park before it completes an authoritative turn in its new generation. The fixture proves two provider starts and only one admitted turn. - The initial live-goal recheck mutation failed as expected. Review then identified that goal changes also needed durable retention. The new activation and clear cases verify saved goal state, retained events, and eligibility after delivery. - Existing goal, active-goal attachment, warm-tail, and late-accounting process regressions passed after sharing the goal event path. - The initial late-goal fixture timed out because it omitted `--durable-turn-ids`, which enables its existing first-turn gate. Added that fixture switch; both park tests then passed. - The credential-observation regression fails before the fix and passes afterward. Three focused credential cases prove durable replay and acknowledgement, fresh ACPX startup, and readiness without provider selection. The three provider parking checks also pass after the fix. - Full regression testing is left to GitHub CI. - Changed lines scanned for secrets, personal data, and private deployment references before push. Current head `28d41b083f58ee93db4305b1217719796bf3171d` adds one scoped review-fix commit to readiness `e8ba82374e0f72c82ac3195d4b1b3798c9072bab`, based on master `3d0d178ff7051aa44e07cd873fd4281357e0d815` without duplicating either recovery foundation. The PR remains eight parking-specific files. Review identified quiet helper work and reconstructed accepted-finish authority that could incorrectly qualify. The provider now records its actually observed terminal authority separately in memory, and parking refuses known helper lineage in both the saved state and the live inventory after draining. No durable field, lifecycle, public API, or ordinary recovery policy changed. Both process regressions fail when the parking guards are removed and pass when restored. Seventeen focused checks pass, including ordinary recovery, helper requests, receipt replay, credentials, and ACPX ineligibility. Rustfmt, diff and added-line scans passed. All exact-head CI is green: [PR run](https://github.com/paperclipai/paperclip/actions/runs/42087773928), [Runner Windows](https://github.com/paperclipai/paperclip/actions/runs/42087773720), and [Docker Runner](https://github.com/paperclipai/paperclip/actions/runs/42087773712) passed; Storybook was skipped. Standard review 15 scored 5/5 before the authorized follow-up [Apex review](https://github.com/paperclipai/paperclip/runs/127471966630). Apex review 16 also scored [5/5 on this exact head](https://github.com/paperclipai/paperclip/pull/19133#issuecomment-6645371152), with zero new comments and zero unresolved threads. On this current head, the first server shard 11/12 [job 127458190155](https://github.com/paperclipai/paperclip/actions/runs/42087773928/job/127458190155) failed the unchanged decision-model concurrency test: after releasing the database lock, its `waitFor` observed one provider call instead of two. The identical file passed all 37 tests on the prior reviewed head. One targeted unchanged-head retry [job 127466020677](https://github.com/paperclipai/paperclip/actions/runs/42087773928/job/127466020677) passed all 1,779 tests, including those 37 decision-model cases; 13 tests were skipped. No code, test timeout, or CI configuration changed between attempts. The original failure's cause remains unestablished. On prior reviewed head `e8ba82374e0f72c82ac3195d4b1b3798c9072bab`, the first `General tests (workspaces-c)` job [127389771745](https://github.com/paperclipai/paperclip/actions/runs/42068371976/job/127389771745) failed the unchanged database SIGTERM test at `packages/db/src/test-postgres-command.test.ts:44`: `JSON.parse` read empty readiness-file content immediately after an existence check. The fixture writes directly to that path, leaving a create-before-content race. One targeted unchanged-head retry [127398676711](https://github.com/paperclipai/paperclip/actions/runs/42068371976/job/127398676711) passed all **313 tests in 72 files**, including SIGTERM and SIGINT. No source or timeout changed between attempts. On the superseded `ca088abd552b` head, one Cloud sign-in recovery case returned to the expected task URL but missed its heading within the five-second assertion window; the other 40 cases passed. One targeted unchanged-head retry (job 127136265280) passed all **41 cases**. No test timeout or application code changed between attempts, and the first failure's cause remains unestablished. The superseded c7 parent failed eight retained-cleanup settlement cases because prepared restoration started a provider during a drain-only epoch. The corrected parent preserves ordinary prepared behavior; the new completed-source path excludes prepared providers. The parent reports all 19 settlement rows passing. This PR adds no prepared-session proof or restoration policy. ## Risks Known helpers remain conservatively ineligible even after their progress diagnostics: this slice has no authoritative helper-completion ledger. The eligibility check drains the existing bounded provider tail and can persist late accounting. That accounting must settle before parking succeeds. This hook alone does not prove VM suspension or vendor goal behavior across a process restart. Live sleep/wake qualification remains a later rollout check. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), High reasoning, through Codex with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../runner/DURABLE_TRANSPORT.md | 19 ++ .../src/bin/fake-codex-app-server.rs | 30 ++ .../crates/runner-core/src/codex_provider.rs | 28 +- .../crates/runner-core/src/durable/runner.rs | 5 + .../src/native_provider_backend.rs | 12 + .../runner-core/src/provider_backend.rs | 111 ++++-- .../runner-core/tests/codex_provider.rs | 320 ++++++++++++++++++ .../tests/native_provider_backend.rs | 40 ++- 8 files changed, 529 insertions(+), 36 deletions(-) diff --git a/packages/paperclip-runner/runner/DURABLE_TRANSPORT.md b/packages/paperclip-runner/runner/DURABLE_TRANSPORT.md index 832375b1170..c2d35cd32f8 100644 --- a/packages/paperclip-runner/runner/DURABLE_TRANSPORT.md +++ b/packages/paperclip-runner/runner/DURABLE_TRANSPORT.md @@ -120,3 +120,22 @@ and goal with A’s saved idle state. After reading the goal, it checks updates that arrive during the final status read and keeps them queued for delivery. As with Codex, a prepared session can continue or be cleaned up. This handoff requires a completed turn. Suspended sessions use their usual recovery path. + +## Host park eligibility + +`CommandExecutor::host_park_ready` defaults to false. Native Codex opts in only +for the existing process that completed the current turn. An eligibility check +never selects or restores a provider. A restored process must complete its own +authoritative turn before it can be eligible. Reconstructing an accepted finish +during recovery preserves result delivery but does not qualify that process. +ACPX remains ineligible in this slice, including after qualified recovery. +Active turns, goals, startup attempts, tool calls, pending credentials, and +unsettled events block parking. Known helper threads also block it: their +diagnostic lineage does not prove that their work finished. The bounded +completed-turn tail drain retains late accounting and goal changes for durable +state and normal event delivery. +The check also inspects the live provider +after that drain, so a late active-goal notification cannot be hidden by the +older executor journal. This hook does not detach a controller, stop a process, +or advertise hosted parking support. The host-control consumer must separately +validate the exact Runner identity and acknowledged sequence cut. diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs b/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs index e8962412e77..ed44bfc42ee 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs @@ -1788,6 +1788,17 @@ fn run() -> Result<(), Box> { )?; continue; } + if args.iter().any(|value| value == "--quiet-helper-turn") { + send(json!({"method":"item/completed","params":{ + "threadId":state.thread_id,"turnId":provider_turn_id, + "item":{"id":"spawn-quiet-helper","type":"collabAgentToolCall", + "tool":"spawnAgent","status":"completed", + "senderThreadId":state.thread_id,"receiverThreadIds":["descendant-0"]} + }}))?; + send(json!({"method":"turn/started","params":{ + "threadId":"descendant-0","turnId":"quiet-helper-turn" + }}))?; + } if descendant_notifications { // Real Codex announces a child's MCP startup before its // thread/started notification establishes parent lineage. @@ -2024,6 +2035,8 @@ fn run() -> Result<(), Box> { let turn_id = provider_turn_id.clone(); let unsafe_tail = args.iter().any(|arg| arg == "--unsafe-after-late-usage"); + let active_goal = args.iter().any(|arg| arg == "--late-active-goal"); + let clear_goal = args.iter().any(|arg| arg == "--late-goal-cleared"); let duplicate_burst = args .iter() .any(|arg| arg == "--late-accounting-duplicate-burst"); @@ -2033,6 +2046,23 @@ fn run() -> Result<(), Box> { thread::sleep(Duration::from_millis(1)); } if gate.is_file() { + if active_goal || clear_goal { + let method = if clear_goal { + "thread/goal/cleared" + } else { + "thread/goal/updated" + }; + let goal = if clear_goal { + Value::Null + } else { + json!({"objective":"Continue work","status":"active"}) + }; + let _ = send( + json!({"method":method,"params":{"threadId":thread_id,"goal":goal}}), + ); + let _ = fs::write(gate.with_extension("sent"), "sent"); + return; + } if duplicate_burst { for _ in 0..33 { let _ = send( diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs b/packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs index 3e1d6cb6095..bef7c38c84f 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs @@ -781,6 +781,7 @@ pub struct CodexProvider { expected_shutdown: bool, process_generation: u64, completed_turn_authority: Option, + observed_completed_turn_authority: Option, active_turn_result_authoritative: bool, completion_reconciliation_pending: bool, goal_allows_autonomous_turns: bool, @@ -1201,6 +1202,7 @@ impl CodexProvider { expected_shutdown: false, process_generation, completed_turn_authority: None, + observed_completed_turn_authority: None, active_turn_result_authoritative: false, completion_reconciliation_pending: false, goal_allows_autonomous_turns: false, @@ -1518,7 +1520,10 @@ impl CodexProvider { } if matches!( method.as_str(), - "thread/tokenUsage/updated" | "paperclip/resumeUsageSnapshot" + "thread/tokenUsage/updated" + | "paperclip/resumeUsageSnapshot" + | "thread/goal/updated" + | "thread/goal/cleared" ) { accounting.push((method, params)); continue; @@ -1702,6 +1707,21 @@ impl CodexProvider { }) } + pub(crate) fn observed_completed_turn_authority(&self) -> Option<(u64, &str)> { + self.observed_completed_turn_authority + .as_ref() + .map(|authority| { + ( + authority.process_generation, + authority.provider_turn_id.as_str(), + ) + }) + } + + pub(crate) fn has_known_descendants(&self) -> bool { + !self.descendant_thread_ids.is_empty() + } + pub(crate) fn mark_active_turn_result_authoritative(&mut self) -> Result<(), LocalRunnerError> { if self.active_provider_turn_id.is_none() || self.ambiguous_turn_start_pending { return Err(LocalRunnerError::invalid( @@ -1819,6 +1839,10 @@ impl CodexProvider { self.restart_idle_identity_epoch() } + pub(crate) fn has_active_goal(&self) -> bool { + self.goal_allows_autonomous_turns + } + pub fn get_goal(&mut self) -> Result { let result = self.request("thread/goal/get", json!({"threadId": self.thread_id}))?; self.goal_allows_autonomous_turns = @@ -3214,6 +3238,8 @@ impl CodexProvider { self.active_turn_result_authoritative = false; self.expected_shutdown = true; self.completed_turn_authority = completed_turn_authority; + // Restored result authority cannot prove this process saw a terminal. + self.observed_completed_turn_authority = self.completed_turn_authority.clone(); self.completion_reconciliation_pending = terminal_event_type == "turn.completed" || result_authoritative; // The provider terminal is authoritative once received. Clear diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/durable/runner.rs b/packages/paperclip-runner/runner/crates/runner-core/src/durable/runner.rs index 98f9967e400..d399b88db34 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/durable/runner.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/durable/runner.rs @@ -306,6 +306,11 @@ pub enum TerminalDeliveryReconciliation { } pub trait CommandExecutor { + /// Prove the existing provider is idle without restoring or starting it. + fn host_park_ready(&mut self) -> Result { + Ok(false) + } + fn execute(&mut self, command: &Command) -> Result; fn observe_recovery_state(&mut self, _state: &DurableState) -> Result<(), DurableRunnerError> { diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/native_provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/src/native_provider_backend.rs index 5bcc344f329..6949763f04a 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/native_provider_backend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/native_provider_backend.rs @@ -278,6 +278,18 @@ impl NativeProviderCommandExecutor { } impl CommandExecutor for NativeProviderCommandExecutor { + fn host_park_ready(&mut self) -> Result { + // Observe credentials without selecting or restoring a provider. + self.credentials.poll(true)?; + if self.credentials.pending().is_some() { + return Ok(false); + } + match self.selected.as_mut() { + Some(SelectedExecutor::LocalFacade(executor)) => executor.host_park_ready(), + _ => Ok(false), + } + } + fn observe_recovery_state( &mut self, state: &crate::durable::DurableState, diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs index 595d51cdff7..17e8f5bd760 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs @@ -761,6 +761,35 @@ fn terminal_completion_rejected(state: &CodexProviderState) -> bool { state.active_terminal_tool_rejected && state.active_provider_result_fingerprint.is_none() } +fn retain_goal_notification( + state: &mut CodexProviderState, + method: &str, + params: &Value, + working_now: bool, +) -> Result<(), DurableRunnerError> { + let event_type = match method { + "thread/goal/updated" => { + state.goal = normalize_codex_goal(params, working_now); + "session.goal.updated" + } + "thread/goal/cleared" => { + state.goal = None; + "session.goal.cleared" + } + _ => return Ok(()), + }; + state.goal_revision = state.goal_revision.saturating_add(1); + state.push_event(NormalizedProviderEvent { + event_type: event_type.to_owned(), + priority: EventPriority::P0, + payload: goal_event_payload( + state.goal.as_ref(), + state.goal_capability.as_ref(), + state.goal_revision, + ), + }) +} + fn goal_continues_after_turn(state: &CodexProviderState, goal_active: bool) -> bool { goal_active && !terminal_completion_rejected(state) } @@ -2966,6 +2995,7 @@ impl CodexCommandExecutor { DurableRunnerError::invalid("Codex provider has not been prepared") })?; for (method, params) in tail { + retain_goal_notification(state, &method, ¶ms, false)?; let events = normalize_provider_notification(state, &method, ¶ms)?; state.extend_events(events)?; } @@ -5035,31 +5065,7 @@ impl CodexCommandExecutor { }); } } - if method == "thread/goal/updated" { - state.goal = normalize_codex_goal(¶ms, working_now); - state.goal_revision = state.goal_revision.saturating_add(1); - state.push_event(NormalizedProviderEvent { - event_type: "session.goal.updated".to_owned(), - priority: EventPriority::P0, - payload: goal_event_payload( - state.goal.as_ref(), - state.goal_capability.as_ref(), - state.goal_revision, - ), - })?; - } else if method == "thread/goal/cleared" { - state.goal = None; - state.goal_revision = state.goal_revision.saturating_add(1); - state.push_event(NormalizedProviderEvent { - event_type: "session.goal.cleared".to_owned(), - priority: EventPriority::P0, - payload: goal_event_payload( - None, - state.goal_capability.as_ref(), - state.goal_revision, - ), - })?; - } + retain_goal_notification(state, &method, ¶ms, working_now)?; if let Some(reconciliation) = goal_reconciliation { match reconciliation { Ok(snapshot) => { @@ -5272,6 +5278,61 @@ impl CodexCommandExecutor { } impl CommandExecutor for CodexCommandExecutor { + fn host_park_ready(&mut self) -> Result { + let Some(state) = self.state.as_ref() else { + return Ok(false); + }; + if state.config.provider != "codex" + || state.config.driver != "codex_app_server" + || self.provider.is_none() + || !state.completed_turn_authoritative + || state.completed_turn_process_generation != Some(state.provider_process_generation) + || state.completed_provider_turn_id.is_none() + || state.active_provider_turn_id.is_some() + || state.ambiguous_turn_start_pending + || state.startup_attempt.is_some() + || state.receipt_limit_interrupt_pending + || state + .goal + .as_ref() + .is_some_and(|goal| goal.status == "active" || goal.working_now) + || state.tool_bridge.pending_calls().next().is_some() + || !state.descendant_thread_ids.is_empty() + || !state.pending_events.is_empty() + || !state.queued_events.is_empty() + || state.lifecycle == "reconciliation_required" + { + return Ok(false); + } + self.retain_completed_accounting_tail()?; + let state = self + .state + .as_ref() + .expect("surviving native provider state"); + if !state.pending_events.is_empty() || !state.queued_events.is_empty() { + return Ok(false); + } + let provider = self + .provider + .as_mut() + .expect("checked surviving native provider"); + // Tail draining can consume a late goal update before the executor + // journal sees it. Check the live provider after that drain. + if provider.has_active_goal() + || provider.has_known_descendants() + || provider.observed_completed_turn_authority() != provider.completed_turn_authority() + || provider.completed_turn_authority() + != state + .completed_turn_process_generation + .zip(state.completed_provider_turn_id.as_deref()) + { + return Ok(false); + } + Ok(provider + .warm_run_attachment_blockers() + .map_err(|_| DurableRunnerError::invalid("native Codex parking proof failed"))? + .is_empty()) + } fn can_reconcile_result_delivery(&mut self) -> Result { // ProviderToolBridge persists the exact call/result and rejects changed // receipts. Delivery cannot execute a server-side semantic operation. diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs index 604a2ad99b0..ca1e43808ac 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs @@ -147,6 +147,9 @@ fn ordinary_recovery_restores_a_completed_idle_session() { ); assert_eq!(call_count(&directory, "process-start"), 2); assert_eq!(call_count(&directory, "turn/start"), 1); + // Warm attachment readiness does not prove this new provider generation + // completed its own authoritative turn. + assert!(!executor.host_park_ready().unwrap()); executor.shutdown().unwrap(); fs::remove_dir_all(directory).unwrap(); } @@ -525,6 +528,323 @@ fn call_count(directory: &Path, method: &str) -> usize { .count() } +#[test] +fn hosted_park_proof_requires_the_surviving_settled_native_codex_process() { + let directory = temporary_directory("hosted-park-proof"); + let config = provider_config(&directory, &["--record-process-start"]); + let mut executor = + CodexCommandExecutor::with_runner_config(&directory, &durable_config(&directory)); + assert!(!executor.host_park_ready().unwrap()); + executor + .execute(&command( + "prepare", + 1, + "run.prepare", + json!({"provider":config}), + )) + .unwrap(); + assert!(!executor.host_park_ready().unwrap()); + executor + .execute(&command("open", 2, "session.open", json!({}))) + .unwrap(); + assert!(!executor.host_park_ready().unwrap()); + executor + .execute(&command( + "turn", + 3, + "turn.start", + json!({"text":"Complete the admitted native turn."}), + )) + .unwrap(); + assert!(!executor.host_park_ready().unwrap()); + wait_for_executor_event(&mut executor, "turn.completed"); + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(5); + while !executor.host_park_ready().unwrap() { + assert!(std::time::Instant::now() < deadline); + poll_and_ack(&mut executor).unwrap(); + } + assert_eq!(call_count(&directory, "process-start"), 1); + assert_eq!(call_count(&directory, "turn/start"), 1); + executor + .execute(&command( + "goal", + 4, + "session.goal.set", + json!({"objective":"Continue autonomous work.","status":"active"}), + )) + .unwrap(); + for _ in 0..8 { + poll_and_ack(&mut executor).unwrap(); + } + assert!(!executor.host_park_ready().unwrap()); + executor.shutdown().unwrap(); + assert!(!executor.host_park_ready().unwrap()); + assert_eq!(call_count(&directory, "process-start"), 1); + fs::remove_dir_all(directory).unwrap(); +} + +#[test] +fn hosted_park_rejects_a_quiet_helper_after_parent_completion() { + let directory = temporary_directory("hosted-park-helper"); + let config = provider_config(&directory, &["--quiet-helper-turn"]); + let mut executor = + CodexCommandExecutor::with_runner_config(&directory, &durable_config(&directory)); + executor + .execute(&command( + "prepare", + 1, + "run.prepare", + json!({"provider":config}), + )) + .unwrap(); + executor + .execute(&command("open", 2, "session.open", json!({}))) + .unwrap(); + executor + .execute(&command( + "turn", + 3, + "turn.start", + json!({"text":"Complete the parent while its helper keeps working."}), + )) + .unwrap(); + wait_for_executor_event(&mut executor, "turn.completed"); + while !poll_and_ack(&mut executor).unwrap().is_empty() {} + let state: Value = + serde_json::from_slice(&fs::read(directory.join("codex-provider-state.json")).unwrap()) + .unwrap(); + assert_eq!(state["completedTurnAuthoritative"], true); + assert!(state["descendantThreadIds"] + .as_array() + .unwrap() + .iter() + .any(|id| id == "descendant-0")); + assert!(!executor.host_park_ready().unwrap()); + executor.shutdown().unwrap(); + fs::remove_dir_all(directory).unwrap(); +} + +#[test] +fn hosted_park_rejects_reconciled_finish_without_a_live_completion() { + let directory = temporary_directory("hosted-park-reconciled-finish"); + let result_schema: Value = serde_json::from_slice( + &fs::read( + PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("../../../protocol/schemas/result.schema.json"), + ) + .unwrap(), + ) + .unwrap(); + let input = json!({"schema":"paperclip.run_result.v1", "reportedWorkDisposition":"done", "summary":"Completed once", + "completionClaim":{"contractRevision":"contract-1", "objectiveSatisfied":true, + "criteria":[{"criterionId":"output", "status":"satisfied", "evidenceRefs":[]}], "remainingWork":[]}, + "evidence":[], "verification":[], "attentionRequests":[], "artifacts":[]}); + let input_path = directory.join("finish-input.json"); + fs::write(&input_path, serde_json::to_vec(&input).unwrap()).unwrap(); + let config = provider_config( + &directory, + &[ + "--record-process-start", + "--durable-turn-ids", + "--terminal-result-file", + input_path.to_str().unwrap(), + ], + ); + let operation = AuthorizedTool { + operation_id: "paperclip_finish".to_owned(), + version: 1, + description: "Report completed work.".to_owned(), + input_schema: result_schema, + response_schema: json!({"type":"object"}), + }; + let tools = AuthorizedToolSet { + schema: TOOL_SET_SCHEMA.to_owned(), + schema_version: 1, + catalog_digest: authorized_tool_catalog_digest(std::slice::from_ref(&operation)).unwrap(), + operations: vec![operation], + }; + let runner_config = durable_config(&directory); + let mut first = CodexCommandExecutor::with_runner_config(&directory, &runner_config); + first + .execute(&command( + "prepare", + 1, + "run.prepare", + json!({"provider":config, "authorizedTools":tools, + "completionContract":{"revision":"contract-1", "criterionIds":["output"]}}), + )) + .unwrap(); + first + .execute(&command("open", 2, "session.open", json!({}))) + .unwrap(); + first + .execute(&command( + "turn", + 3, + "turn.start", + json!({"text":"Finish before the completion is journaled."}), + )) + .unwrap(); + wait_for_executor_event(&mut first, "semantic_tool.input"); + first + .execute(&command( + "accept", + 4, + "semantic_tool.result", + json!({"callId":"terminal-result-call", + "operationId":"paperclip_finish", "result":{"accepted":true}, "isError":false}), + )) + .unwrap(); + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(5); + loop { + let ended = fs::read(directory.join("fake-state.json")) + .ok() + .and_then(|bytes| serde_json::from_slice::(&bytes).ok()) + .is_some_and(|live| live.get("activeTurnId") == Some(&Value::Null)); + if ended { + break; + } + assert!(std::time::Instant::now() < deadline); + std::thread::sleep(std::time::Duration::from_millis(1)); + } + let state: Value = + serde_json::from_slice(&fs::read(directory.join("codex-provider-state.json")).unwrap()) + .unwrap(); + assert!(state["activeProviderResultFingerprint"].is_string()); + assert_eq!(state["completedTurnAuthoritative"], false); + drop(first); + let mut recovered = CodexCommandExecutor::with_runner_config(&directory, &runner_config); + recovered + .execute(&command("restore", 5, "session.snapshot", json!({}))) + .unwrap(); + let mut events = Vec::new(); + loop { + let next = poll_and_ack(&mut recovered).unwrap(); + if next.is_empty() { + break; + } + events.extend(next); + } + assert_eq!( + events + .iter() + .filter(|event| event.event_type == "run.terminal") + .count(), + 1 + ); + let snapshot = recovered + .execute(&command("snapshot", 6, "session.snapshot", json!({}))) + .unwrap(); + assert_eq!(snapshot.result["warmAttachReady"], true); + assert_eq!(call_count(&directory, "process-start"), 2); + assert_eq!(call_count(&directory, "turn/start"), 1); + assert!(!recovered.host_park_ready().unwrap()); + recovered.shutdown().unwrap(); + fs::remove_dir_all(directory).unwrap(); +} + +#[test] +fn hosted_park_preserves_goal_activation_during_tail_drain() { + verify_late_goal_retention(false); +} + +#[test] +fn hosted_park_preserves_goal_clear_during_tail_drain() { + verify_late_goal_retention(true); +} + +fn verify_late_goal_retention(clear_goal: bool) { + let directory = temporary_directory("hosted-park-late-goal"); + let gate = directory.join("release-goal"); + let gate_arg = gate.to_string_lossy().into_owned(); + let config = provider_config( + &directory, + &[ + "--record-process-start", + "--durable-turn-ids", + "--late-accounting-tail-gate", + &gate_arg, + if clear_goal { + "--late-goal-cleared" + } else { + "--late-active-goal" + }, + ], + ); + let mut executor = + CodexCommandExecutor::with_runner_config(&directory, &durable_config(&directory)); + executor + .execute(&command( + "prepare", + 1, + "run.prepare", + json!({"provider":config}), + )) + .unwrap(); + executor + .execute(&command("open", 2, "session.open", json!({}))) + .unwrap(); + executor + .execute(&command( + "turn", + 3, + "turn.start", + json!({"text":"Complete one turn."}), + )) + .unwrap(); + wait_for_executor_event(&mut executor, "turn.completed"); + if clear_goal { + executor + .execute(&command( + "paused-goal", + 4, + "session.goal.set", + json!({"objective":"Wait for work", "status":"paused"}), + )) + .unwrap(); + } + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(5); + while !executor.host_park_ready().unwrap() { + assert!(std::time::Instant::now() < deadline); + poll_and_ack(&mut executor).unwrap(); + } + fs::write(&gate, "release").unwrap(); + while !gate.with_extension("sent").is_file() { + assert!(std::time::Instant::now() < deadline); + std::thread::sleep(std::time::Duration::from_millis(1)); + } + // Do not poll the executor: the goal must first be observed by the park + // probe's tail drain, after its initial saved-state eligibility check. + assert!(!executor.host_park_ready().unwrap()); + assert!(!executor.host_park_ready().unwrap()); + let saved: Value = + serde_json::from_slice(&fs::read(directory.join("codex-provider-state.json")).unwrap()) + .unwrap(); + let expected_status = if clear_goal { + Value::Null + } else { + json!("active") + }; + let expected_event = if clear_goal { + "session.goal.cleared" + } else { + "session.goal.updated" + }; + assert_eq!(saved["goal"]["status"], expected_status); + let events = poll_and_ack(&mut executor).unwrap(); + assert!(events.iter().any(|event| event.event_type == expected_event + && event.payload["goal"]["status"] == expected_status)); + assert_eq!( + executor.host_park_ready().unwrap(), + clear_goal, + "only the cleared goal becomes parkable after event delivery" + ); + assert_eq!(call_count(&directory, "process-start"), 1); + assert_eq!(call_count(&directory, "turn/start"), 1); + executor.shutdown().unwrap(); + fs::remove_dir_all(directory).unwrap(); +} + #[test] fn failed_provider_startup_is_persistently_fenced_before_another_process_can_resume() { let directory = temporary_directory("failed-startup-fence"); diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs index 8e74238970e..b5a0a5c3d39 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs @@ -62,6 +62,17 @@ fn credential_observation_allows_fresh_acpx_session_startup() { #[test] #[cfg(unix)] fn credential_updates_replay_through_the_native_executor_until_durable_ack() { + exercise_credential_updates(false); +} + +#[test] +#[cfg(unix)] +fn hosted_park_refreshes_credentials_without_selecting_a_provider() { + exercise_credential_updates(true); +} + +#[cfg(unix)] +fn exercise_credential_updates(parking: bool) { let root = temporary_directory("credential-updates"); let root = fs::canonicalize(root).unwrap(); let auth = root.join("auth.json"); @@ -88,16 +99,21 @@ fn credential_updates_replay_through_the_native_executor_until_durable_ack() { let mut executor = NativeProviderCommandExecutor::with_runner_config(&root, &runner_config); assert!(executor.poll_events().unwrap().is_empty()); write_auth("refreshed"); - let _ = executor.execute(&Command { - schema: "paperclip.prp.command.v1".to_owned(), - command_id: "close-unselected".to_owned(), - controller_seq: 1, - command_type: "session.close".to_owned(), - issued_at: "2026-10-15T12:00:00.000Z".to_owned(), - deadline_at: None, - precondition: None, - payload: json!({}), - }); + if parking { + assert!(!executor.host_park_ready().unwrap()); + } else { + let _ = executor.execute(&Command { + schema: "paperclip.prp.command.v1".to_owned(), + command_id: "close-unselected".to_owned(), + controller_seq: 1, + command_type: "session.close".to_owned(), + issued_at: "2026-10-15T12:00:00.000Z".to_owned(), + deadline_at: None, + precondition: None, + payload: json!({}), + }); + } + assert!(!root.join("codex-provider-state.json").exists()); let updates = executor.retained_events().unwrap(); assert_eq!(updates.len(), 1); assert_eq!(updates[0].event_type, "auth.credentials.updated"); @@ -1521,6 +1537,7 @@ fn resumes_an_idle_acpx_session_in_a_cold_replacement_runner() { let original = executor .execute(&command(2, "session.open", json!({}))) .unwrap(); + assert!(!executor.host_park_ready().unwrap()); executor .execute(&command( 3, @@ -1530,6 +1547,8 @@ fn resumes_an_idle_acpx_session_in_a_cold_replacement_runner() { .unwrap(); let events = executor.poll_events().unwrap(); executor.acknowledge_events(events.len()).unwrap(); + // Generic ACPX recovery does not opt this provider into host parking. + assert!(!executor.host_park_ready().unwrap()); executor .execute(&command(4, "runner.suspend", json!({}))) .unwrap(); @@ -1547,6 +1566,7 @@ fn resumes_an_idle_acpx_session_in_a_cold_replacement_runner() { .execute(&command(1, "run.attach", payload)) .unwrap(); assert_eq!(resumed.result["status"], "resumed"); + assert!(!replacement.host_park_ready().unwrap()); assert_eq!( resumed.result["providerSessionId"], original.result["providerSessionId"]