diff --git a/DESIGN.md b/DESIGN.md index cff31c9a04c..7b860708c8e 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -153,7 +153,7 @@ results in place; keep failures actionable inline. Notifications for other work remain useful. Repeated delivery of the same run outcome must refresh cached state without repeating its toast, including after reconnecting. A terminal outcome delivered more than five minutes after the run finished is historical and should -refresh state silently. Expected cancellation is neutral gray, not an error. The composer's Stop action stops the current response and leaves the composer available for a new message. Pause work is a separate explicit task or subtree action. A paused task replaces the composer with an amber takeover. It says “Task is +refresh state silently. Expected cancellation is neutral gray, not an error. A busy workspace is a transient system wait with no user action. Show it only as a quiet status in the composer strip and the Workspace property row, never as a pinned banner, conversation notice or toast. The composer's Stop action stops the current response and leaves the composer available for a new message. Pause work is a separate explicit task or subtree action. A paused task replaces the composer with an amber takeover. It says “Task is paused.” and “Resume this task to send a message.” with a “Resume task” action. Subtrees use “Subtree is paused.” and “Resume subtree.” The takeover cannot be dismissed, retains drafts, and hides message inputs until the pause is released. diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 5704355c9fc..5a7659c4ee9 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1919,6 +1919,37 @@ that the whole web server is disposable or that all execution and recovery paths survive web-server replacement. Run-log/trace storage and producer retention are separate changes. +### Managed environments + +The `environments` list in `PAPERCLIP_MANAGED_CONFIG` declares instance-level +environments that boot provisions and tenants cannot edit. It holds at most one +sandbox entry (no `driver`, or `driver: "sandbox"`) and any number of +`driver: "plugin"` entries. A plugin entry names a plugin by its manifest id +(`pluginKey`) and one of its environment drivers: + +```json +{ + "name": "Remote Host", + "description": "Runs the agent on a remote host", + "driver": "plugin", + "pluginKey": "example.remote-host-plugin", + "driverKey": "tasks", + "driverConfig": {} +} +``` + +Names and `pluginKey`–`driverKey` pairs must be unique, and `driverConfig` must +not contain secret-like keys. Any other key refuses startup. `plugins.autoInstall` +holds catalog keys, not manifest ids, so boot also refuses to start unless an +auto-installed bundled or distribution catalog entry installs the entry's +`pluginKey`. + +Boot validates the driver config through the plugin, marks the row +`managedPluginEnvironment: ":"`, and adopts an existing +unmarked row for the same plugin and driver. While the plugin is down, the row is +archived; it comes back when the worker is ready. Removing an entry never archives +its row. Plugin entries do not change the instance default. + ### Experimental native remote workspace storage `experimental.remoteWorkspaceStorageEnabled` is a managed feature, on by default diff --git a/doc/dependency-wake-recovery.md b/doc/dependency-wake-recovery.md index 0187b99c6f3..41e94ef5ed5 100644 --- a/doc/dependency-wake-recovery.md +++ b/doc/dependency-wake-recovery.md @@ -13,6 +13,13 @@ unattributed operation in the same shared workspace does not remove its wait. Normal readiness, duplicate-wake, interaction, pause, and execution gates still apply. A board transition into blocked keeps the existing recovery behavior. +A dependency completion can arrive while the assignee answers a question about +the blocked task. Its saved wake waits behind that turn. Deferred-receipt recovery +reconstructs these dependency wakes after the turn and its cleanup finish, even +when they contain no comment or interaction response. It uses the existing +release and dispatch checks; task cancellation, agent pause, and pending cleanup +still prevent execution. + Cancelling a run or pausing its agent suppresses immediate stranded-work recovery. A queued user message remains a separate continuation request. The release path still preserves independently authorized deferred input; stopping the old run diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 289b51720c8..a2eebcada7c 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -2415,8 +2415,11 @@ run, release its environment, or start another provider. Lease expiry alone does not authorize that stale preparation to take ownership; normal recovery must establish a new claim first. -An exact local Codex restart can restore the conversation after losing its active -turn. Runnerd records this as `provider_turn_lost_on_restore`, with no invented +An exact local Codex, OpenCode, or ACPX restart can restore the conversation after +losing its active turn. ACPX first requires the original provider lifetime to lose +its fence quorum; without that proof, or with an accepted result, unanswered +runtime request, or active goal, it closes the session as before. Cursor Cloud +agents run remotely and are excluded. Runnerd records this as `provider_turn_lost_on_restore`, with no invented task result. The admitted restart attempt may send one continuation in that same conversation. It preserves the workspace and asks the agent to reconcile unfinished commands and external actions before proceeding. It does not resend the task or @@ -3630,6 +3633,34 @@ re-read dependency readiness, since new dependencies need not change the displayed task status. Old blocked rows without a receipt remain held; no historical status backfill is performed. +Older native executors recorded the local callback-listener startup guard as +`native_session_interrupted`. A saved exact guard outcome can enter the same +bounded safe-replacement path after restart. This requires the first attempt, +an unused provider-session identity, matching saved execution bindings, no +provider events, checkpoint, admission receipt, tool call, or cancellation, and +the existing current snapshot and confirmed stop receipts. A generic error or +a pending provider-pack receipt alone grants no recovery authority. The same +transaction can retire that run's completed file-preservation hold only when +its bound recovery and finalization receipts prove completion. Unrelated holds, +later task blocks, normal dispatch admission, and the three-attempt budget remain +effective. + +Historical replacement discovery reads at most 100 candidates per pass and +keeps a cursor only for the remaining scan. A work notification preserves forward +progress and requests a final rescan, so held earlier rows cannot hide a later +eligible run. Completed scans of held history become quiet. Final snapshot +publication, verified lease cleanup, local process-stop proof, completed workspace +recovery, and dispatch eligibility changes notify the existing replacement worker +in their transactions. A cleared historical interaction only requests another +proof check; it grants no new continuation authority. Source provider effects, +task pause or cancellation, and the original authority checks still prevent replay. +Temporary decision failures retain the existing worker's retry and advance the +bounded cursor, with a forced rescan of earlier failures. A failed page cannot +hide later ready runs. During drain, the initialized replacement worker can read +readiness once per notification generation to discharge an empty notification +without starting replacement. Concrete pending work and changes arriving during +that read remain outstanding. Restart reconstructs the scan from durable records. + For legacy GitHub input without a saved App ID or its own response receipt, Task Retry may prove the endpoint's earlier non-null App claim from a processed response ledger created before that input. The proof stays within the same diff --git a/doc/run-log-events.md b/doc/run-log-events.md index 916a3a6d0a0..29e7669f97e 100644 --- a/doc/run-log-events.md +++ b/doc/run-log-events.md @@ -348,13 +348,24 @@ that ledger records `remote_execution_never_claimed` with a `bootstrap_incomplete` disposition. The provider attempt remains zero during classification; the executor increments it before starting provider work. -For a local Codex turn lost while runnerd was stopped, the canonical `turn.failed` +For a local Codex, OpenCode, or ACPX turn lost while runnerd was stopped, the canonical `turn.failed` event retains `error.code: "provider_turn_lost_on_restore"` and `error.recoverable: true`. It records the interrupted turn, not a final task outcome. There is no synthesized result for that interruption. An admitted same-session continuation emits its own `turn.submitted`, accepted turn ID, and terminal events; only the final outcome closes the run. Reconstructed thread history retains the original error so reconciliation and live delivery agree. +An ACPX session records this only after the original provider lifetime loses +its fence quorum and the same conversation is restored. Otherwise its +`turn.failed` carries `error.code: "acpx_active_turn_recovery_closed"` with +`error.recoverable: false`, and the run closes with an honest restart message. +After an ACPX restore, the controller can still deliver results for semantic +calls of the lost turn. Runnerd acknowledges each one as `retired` without +sending it to the restored provider, so the calls settle and live semantic +dispatch resumes for the continuation. A restored ACPX provider is a new process +whose MCP request ids restart, so its semantic `callId` values carry a +per-process prefix (`r:`). They never reuse an earlier call's id in +the same run. ## Native Local Process Stop Evidence diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs index 65647805628..b3ade8b5308 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs @@ -1,7 +1,7 @@ use crate::generated_acpx_profiles::{ acpx_release_profile, ACPX_DRIVER_KIND, QUALIFIED_ACPX_VERSION, }; -use std::collections::{BTreeMap, HashMap, HashSet, VecDeque}; +use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet, VecDeque}; use std::fs::{self, DirBuilder, File}; use std::io::{Read, Write}; use std::net::TcpListener; @@ -70,6 +70,8 @@ pub const ACPX_PROVIDER_STATE_FILE: &str = "acpx-provider-state.json"; const ACPX_PROVIDER_STATE_SCHEMA: &str = "paperclip.runner.acpx-provider-state.v3"; const MAX_PROVIDER_STATE_BYTES: u64 = 16 * 1024 * 1024; const MAX_PENDING_EVENTS: usize = 8_320; +// Matches the bounded number of pending ACPX tool calls. +const MAX_OPEN_SEMANTIC_CALLS: usize = 4_096; const MAX_EVENTS_PER_POLL: usize = 128; const PROVIDER_LIFETIME_CONFIRMATION_TIMEOUT: Duration = Duration::from_secs(5); const PROVIDER_LIFETIME_CONFIRMATION_RETRY: Duration = Duration::from_millis(10); @@ -597,6 +599,13 @@ struct AcpxDurableState { pending_runtime_requests: BTreeMap, #[serde(default = "initial_event_sequence")] next_event_sequence: u64, + /// Semantic tool calls emitted to the controller without a result yet. + #[serde(default)] + open_semantic_calls: BTreeSet, + /// Open calls of a turn lost to a runner restart. The restored provider + /// never sees their results; the controller may still deliver them. + #[serde(default)] + retired_semantic_calls: BTreeSet, } impl AcpxDurableState { @@ -623,6 +632,8 @@ impl AcpxDurableState { pending_events: VecDeque::new(), pending_runtime_requests: BTreeMap::new(), next_event_sequence: initial_event_sequence(), + open_semantic_calls: BTreeSet::new(), + retired_semantic_calls: BTreeSet::new(), } } @@ -789,6 +800,31 @@ impl AcpxDurableState { if let Some(id) = event.payload.get("requestId").and_then(Value::as_str) { self.pending_runtime_requests.remove(id); } + } else if let Some(call_id) = event + .payload + .pointer("/semantic_tool/callId") + .and_then(Value::as_str) + { + // Remember semantic calls the controller may still answer. A lost + // turn's open calls become retired calls on restore, so a later + // controller result for them settles instead of failing. + match event.event_type.as_str() { + "semantic_tool.input" + if self.open_semantic_calls.len() < MAX_OPEN_SEMANTIC_CALLS => + { + self.open_semantic_calls.insert(call_id.to_owned()); + } + "semantic_tool.result" => { + self.open_semantic_calls.remove(call_id); + } + _ => {} + } + } else if matches!( + event.event_type.as_str(), + "turn.completed" | "turn.failed" | "turn.cancelled" | "turn.interrupted" + ) { + // A provider-observed terminal settles its calls in the live bridge. + self.open_semantic_calls.clear(); } self.next_event_sequence = next_sequence; self.pending_events.push_back(PolledEvent { @@ -1116,6 +1152,9 @@ impl AcpxCommandExecutor { let previous_turn = state.active_turn_id.clone(); if unsafe_active { self.context.provider_turn_id = None; + if self.restore_conversation_after_lost_turn(previous_turn.clone())? { + return Ok(()); + } let state = self .state .as_mut() @@ -1134,6 +1173,13 @@ impl AcpxCommandExecutor { "providerTerminalObserved": false, "code": "acpx_active_turn_recovery_closed", "providerShutdownFailed": true, + // Keep the cause through the controller facade, which + // forwards only the structured error of a failed turn. + "error": { + "code": "acpx_active_turn_recovery_closed", + "recoverable": false, + "message": "The runner restarted during this turn and could not restore the conversation.", + }, }), })?; state.push(NormalizedProviderEvent { @@ -1171,6 +1217,82 @@ impl AcpxCommandExecutor { self.save_state() } + /// Restores the conversation of a turn that was active when the runner + /// died, without inventing its outcome. The original provider lifetime + /// must have lost its fence quorum first, so no second provider can act on + /// the same session. The lost turn is recorded with a recoverable + /// `provider_turn_lost_on_restore` cause and the run stays open, so the + /// controller can send its single same-conversation continuation. Returns + /// false, leaving state untouched, when restoration is not safe. + fn restore_conversation_after_lost_turn( + &mut self, + previous_turn: Option, + ) -> Result { + let Some(state) = self.state.as_ref() else { + return Ok(false); + }; + // An accepted result, an unanswered runtime request, or an active goal + // needs its own reconciliation; a plain continuation cannot settle it. + let Some(identity) = state.identity.as_ref() else { + return Ok(false); + }; + if previous_turn.is_none() + || state.provider_exit_unconfirmed + || state.semantic_result.is_some() + || !state.pending_runtime_requests.is_empty() + || !(state.goal_projection.is_null() || goal_is_dormant(&state.goal_projection)) + { + return Ok(false); + } + match await_provider_lifetime_fence(identity.provider_lifetime_fence_candidates) { + // Release the quorum so the restored provider can inherit it. + Ok(listeners) => drop(listeners), + Err(_) => return Ok(false), + } + let Ok(session) = self.start_session(true) else { + return Ok(false); + }; + let identity = session.identity().clone(); + let process_id = session.process_id(); + let turn_controls = session.turn_control_capabilities(); + let state = self + .state + .as_mut() + .expect("ACPX state remains available during recovery"); + state.lifecycle = "session_open".to_owned(); + state.active_turn_id = None; + state.turn_controls = turn_controls; + // The controller may still answer the lost turn's semantic calls. + // The restored provider never issued them, so settle them on arrival. + let open = std::mem::take(&mut state.open_semantic_calls); + state.retired_semantic_calls.extend(open); + state.push(NormalizedProviderEvent { + event_type: "session.resumed".to_owned(), + priority: EventPriority::P0, + payload: session_event_payload(&state.descriptor, &identity, process_id, turn_controls), + })?; + state.push(NormalizedProviderEvent { + event_type: "turn.failed".to_owned(), + priority: EventPriority::P0, + payload: json!({ + "provider": "acpx", + "providerTurnId": previous_turn, + "status": "failed", + "providerTerminalObserved": false, + "code": "provider_turn_lost_on_restore", + "error": { + "code": "provider_turn_lost_on_restore", + "recoverable": true, + "message": "The runner restored the conversation after process loss, but the previous turn is no longer active.", + }, + }), + })?; + self.recovery_pending_effects = None; + self.session = Some(session); + self.save_state()?; + Ok(true) + } + fn start_session(&self, recovering: bool) -> Result { let state = self .state @@ -2123,6 +2245,20 @@ impl AcpxCommandExecutor { let result: ToolResult = serde_json::from_value(payload.clone()).map_err(|error| { DurableRunnerError::invalid(format!("semantic tool result is invalid: {error}")) })?; + if let Some(state) = self.state.as_ref() { + if state.retired_semantic_calls.contains(&result.call_id) { + // The call belonged to a turn lost to a runner restart. Its + // provider process is gone; acknowledge the controller result + // so the call settles, without sending it to the new provider. + // The retained receipt makes this idempotent: a delivery left + // indeterminate by a crash replays to the same acknowledgement. + return Ok(CommandExecution::result(json!({ + "status": "delivered", + "callId": result.call_id, + "retired": true, + }))); + } + } self.session .as_mut() .ok_or_else(|| DurableRunnerError::invalid("ACPX session is unavailable"))? @@ -2521,6 +2657,18 @@ fn goal_terminal_disposition<'a>( } impl CommandExecutor for AcpxCommandExecutor { + /// A restored session keeps durable receipts for the lost turn's calls. + /// Acknowledging one is idempotent and never reaches the provider or the + /// business operation, so an indeterminate delivery may replay. Live + /// calls have no durable receipt; without retired receipts this stays off. + fn can_reconcile_result_delivery(&mut self) -> Result { + self.restore()?; + Ok(self + .state + .as_ref() + .is_some_and(|state| !state.retired_semantic_calls.is_empty())) + } + fn host_park_ready(&mut self) -> Result { self.ensure_persistence_healthy()?; let Some(state) = self.state.as_ref() else { @@ -5020,6 +5168,14 @@ mod tests { assert_eq!(events[0].payload["replayAllowed"], false); assert_eq!(events[1].event_type, "turn.failed"); assert_eq!(events[1].payload["providerShutdownFailed"], true); + // An unanswered runtime request (and an unproven original lifetime) + // keeps the close path. Its cause survives the controller facade and + // is not offered for the restart continuation. + assert_eq!( + events[1].payload["error"]["code"], + "acpx_active_turn_recovery_closed" + ); + assert_eq!(events[1].payload["error"]["recoverable"], false); assert_eq!(events[2].event_type, "run.terminal"); assert_eq!(events[2].payload["schema"], "paperclip.prp.terminal.v1"); assert_eq!(events[2].payload["turnTerminalState"], "failed"); @@ -5049,6 +5205,74 @@ mod tests { fs::remove_dir_all(directory).unwrap(); } + #[test] + fn lost_turn_restoration_refuses_unreconciled_state_before_touching_the_provider() { + let directory = temporary_directory("lost-turn-guards"); + let provider_descriptor: AcpxProviderDescriptor = + serde_json::from_value(descriptor("claude")).unwrap(); + let operations = Vec::new(); + let tool_set = AuthorizedToolSet { + schema: TOOL_SET_SCHEMA.to_owned(), + schema_version: 1, + catalog_digest: authorized_tool_catalog_digest(&operations).unwrap(), + operations, + }; + let identity = AcpxProviderSessionIdentity { + kind: "acpx".to_owned(), + normalized_session_id: "session-1".to_owned(), + acpx_record_id: "record-1".to_owned(), + backend_session_id: "backend-1".to_owned(), + agent_session_id: "agent-1".to_owned(), + profile_digest: provider_descriptor.command_digest.clone(), + workspace_digest: format!("sha256:{}", "a".repeat(64)), + requested_model: provider_descriptor.model.clone(), + effective_model: provider_descriptor.model.clone(), + permission_mode: Some(provider_descriptor.permission_mode), + mode: provider_descriptor.mode.clone(), + pi_thinking_level: provider_descriptor.pi_thinking_level, + claude_effort: provider_descriptor.claude_effort.clone(), + grok_effort: provider_descriptor.grok_effort.clone(), + grok_disable_web_search: provider_descriptor.grok_disable_web_search.clone(), + provider_lifetime_fence_candidates: [60_001, 60_002, 60_003], + }; + let guards: [(&str, fn(&mut AcpxDurableState)); 4] = [ + ("provider_exit_unconfirmed", |state| { + state.provider_exit_unconfirmed = true + }), + ("accepted_semantic_result", |state| { + state.semantic_result = Some(json!({"reportedWorkDisposition": "done"})) + }), + ("unanswered_runtime_request", |state| { + state + .pending_runtime_requests + .insert("input-1".to_owned(), pending_input_request("input-1")); + }), + ("active_goal", |state| { + state.goal_projection = json!({"goal": {"status": "active"}}) + }), + ]; + for (label, seed) in guards { + let mut state = + AcpxDurableState::new(provider_descriptor.clone(), tool_set.clone(), "d".into()); + state.lifecycle = "turn_active".to_owned(); + state.identity = Some(identity.clone()); + state.active_turn_id = Some("turn-1".to_owned()); + seed(&mut state); + let mut executor = + AcpxCommandExecutor::with_runner_config(&directory, &test_config(&directory, None)); + executor.state = Some(state.clone()); + assert!( + !executor + .restore_conversation_after_lost_turn(Some("turn-1".to_owned())) + .unwrap(), + "{label} must keep the close path" + ); + assert!(executor.session.is_none(), "{label} started a provider"); + assert_eq!(executor.state, Some(state), "{label} changed saved state"); + } + fs::remove_dir_all(directory).unwrap(); + } + #[test] fn suspension_waits_for_the_original_provider_lifetime_quorum() { let (candidates, original_lifetime_fence) = reserve_provider_lifetime_fence(); diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs index 2588c4d049c..6ac3fe4dbbe 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs @@ -421,6 +421,14 @@ fn semantic_result_event( } } +/// Local Codex app-server and OpenCode server restart their own process and +/// restore the same conversation after runner loss, so a turn that vanished +/// meanwhile is a restart interruption the controller may continue once. A +/// Cursor Cloud agent runs remotely; its lost turn is not a process restart. +fn restart_lost_turn_is_continuable(provider: &str) -> bool { + matches!(provider, "codex" | "opencode") +} + fn facade_provider_name(provider: &str) -> &'static str { match provider { "opencode" => "OpenCode", @@ -2693,6 +2701,7 @@ impl CodexCommandExecutor { // trustworthy success notification to replay. Terminate it // conservatively so the controller cannot wait forever or // mistake an unknown outcome for success. + let restart_continuable = restart_lost_turn_is_continuable(&provider_label); let provider_terminal = NormalizedProviderEvent { event_type: "turn.failed".to_owned(), priority: EventPriority::P0, @@ -2701,14 +2710,14 @@ impl CodexCommandExecutor { "providerTurnId": previous_active_turn_id, "status": "failed", "providerTerminalObserved": false, - "error": if provider_label == "codex" { json!({ + "error": if restart_continuable { json!({ "code": "provider_turn_lost_on_restore", "recoverable": true, "message": "The runner restored the conversation after process loss, but the previous turn is no longer active.", }) } else { Value::Null }, }), }; - if provider_label == "codex" && !state.active_terminal_tool_rejected { + if restart_continuable && !state.active_terminal_tool_rejected { // No provider result was observed. Preserve the lost // turn as a fact, but leave the run open for the // controller's bounded same-conversation recovery. @@ -5700,6 +5709,13 @@ impl CommandExecutor for CodexCommandExecutor { #[cfg(test)] mod tests { + #[test] + fn restart_lost_turns_continue_only_for_locally_restored_providers() { + assert!(super::restart_lost_turn_is_continuable("codex")); + assert!(super::restart_lost_turn_is_continuable("opencode")); + assert!(!super::restart_lost_turn_is_continuable("cursor_cloud")); + } + #[test] fn mcp_permission_requests_use_durable_turn_and_item_identity() { let identity = ProviderEventIdentity { diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_warm_attachment.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_warm_attachment.rs index 608809e807e..168ba6e0a36 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_warm_attachment.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_warm_attachment.rs @@ -166,3 +166,168 @@ fn checkpoints_the_sidecar_and_rebinds_consecutive_warm_runs_before_accepting_wo assert_eq!(started.result["status"], "accepted"); executor.shutdown().unwrap(); } + +fn unused_fence_ports() -> [u16; 3] { + // Lifetime fence candidates must be dynamic ports (49152 and above). An + // OS-assigned port can be lower, for example on Linux. + let mut listeners = Vec::new(); + for port in (49_152..=u16::MAX).rev() { + if let Ok(listener) = std::net::TcpListener::bind(("127.0.0.1", port)) { + listeners.push(listener); + if listeners.len() == 3 { + break; + } + } + } + let ports = listeners + .iter() + .map(|listener| listener.local_addr().unwrap().port()) + .collect::>(); + [ports[0], ports[1], ports[2]] +} + +/// A runner that dies during an active ACPX turn (for example, when the +/// control plane restarts) must restore the same conversation once the old +/// provider lifetime is proven gone. It records a recoverable restart-loss +/// cause instead of a terminal provider failure. +#[test] +fn restores_the_conversation_after_losing_an_active_turn_to_a_runner_restart() { + let fixture = + Fixture(std::env::temp_dir().join(format!("acpx-restart-{}", uuid::Uuid::new_v4()))); + fs::create_dir_all(&fixture.0).unwrap(); + fs::set_permissions(&fixture.0, fs::Permissions::from_mode(0o700)).unwrap(); + let sidecar = fixture.0.join("sidecar"); + fs::copy(env!("CARGO_BIN_EXE_fake-acpx-sidecar"), &sidecar).unwrap(); + fs::set_permissions(&sidecar, fs::Permissions::from_mode(0o700)).unwrap(); + let ports = unused_fence_ports(); + let args = vec![ + "--mode".to_owned(), + "suspend".to_owned(), + "--profile-digest".to_owned(), + PROFILE_DIGEST.to_owned(), + "--lifetime-fence-ports".to_owned(), + format!("{},{},{}", ports[0], ports[1], ports[2]), + ]; + let config = DurableRunnerConfig { + connect_url: "ws://127.0.0.1/runner".to_owned(), + ca_bundle_path: None, + state_dir: fixture.0.clone(), + runner_instance_id: "runner-1".to_owned(), + environment_lease_id: "lease-1".to_owned(), + run_id: "run-1".to_owned(), + normalized_session_id: "session-1".to_owned(), + turn_id: "turn-1".to_owned(), + item_id: "item-1".to_owned(), + runner_version: "0.0.0".to_owned(), + runner_digest: format!("sha256:{}", "a".repeat(64)), + acpx_launch_profile: Some(AcpxLaunchProfile { + candidate_admission: None, + authority_digest: format!("sha256:{}", "d".repeat(64)), + command: sidecar.clone(), + args: args.clone(), + artifacts: vec![QualifiedLaunchArtifact { + path: sidecar.clone(), + sha256: format!("sha256:{:x}", Sha256::digest(fs::read(&sidecar).unwrap())), + }], + }), + opencode_launch_profile: None, + cursor_cloud_launch_profile: None, + max_outbox_bytes: 1024 * 1024, + p0_reserve_bytes: 64 * 1024, + max_frame_bytes: 1024 * 1024, + reconnect_delay: Duration::from_millis(1), + reconnect_grace: None, + max_runtime: Duration::from_secs(30), + }; + let descriptor = json!({ + "kind": "acpx", "provider": "acpx", "driver": "acpx_runtime", + "providerVersion": "0.13.1", "agent": "codex", "model": "gpt-5.6-sol", + "acpxVersion": "0.13.1", "agentServerPackage": "@agentclientprotocol/codex-acp", + "agentServerVersion": "1.6.2", "agentRuntimePackage": "@openai/codex", + "agentRuntimeVersion": "0.160.0", "commandDigest": PROFILE_DIGEST, + "sidecarCommand": sidecar, "sidecarArgs": args, "runtimeDirectory": fixture.0, + "normalizedSessionId": "session-1", "runId": "run-1", "cwd": fixture.0, + "instructions": "Complete the supplied work.", + "permissionMode": "approve-all", "permissionModePinned": true, + }); + let mut original = AcpxCommandExecutor::with_runner_config(&fixture.0, &config); + original + .execute(&command(1, "run.prepare", json!({"provider": descriptor}))) + .unwrap(); + let opened = original + .execute(&command(2, "session.open", json!({}))) + .unwrap(); + let provider_session = opened.result["providerSessionId"].clone(); + let started = original + .execute(&command( + 3, + "turn.start", + json!({"turnId": "provider-turn-1", "text": "Do the work."}), + )) + .unwrap(); + assert_eq!(started.result["status"], "accepted"); + // The runner process dies with the turn still active, after the turn sent + // the controller a semantic call that has no result yet. + drop(original); + let state_path = fixture.0.join("acpx-provider-state.json"); + let mut saved: Value = serde_json::from_slice(&fs::read(&state_path).unwrap()).unwrap(); + saved["openSemanticCalls"] = json!(["call-lost"]); + fs::write(&state_path, serde_json::to_vec(&saved).unwrap()).unwrap(); + + let mut restored = AcpxCommandExecutor::with_runner_config(&fixture.0, &config); + let snapshot = restored + .execute(&command(4, "session.snapshot", json!({}))) + .unwrap(); + assert_ne!(snapshot.result["status"], "closed"); + assert_eq!(snapshot.result["providerSessionId"], provider_session); + let events = restored.poll_events().unwrap(); + let failed = events + .iter() + .find(|event| event.event_type == "turn.failed") + .expect("the lost turn is terminated"); + assert_eq!(failed.payload["providerTurnId"], "provider-turn-1"); + assert_eq!(failed.payload["providerTerminalObserved"], false); + assert_eq!( + failed.payload["error"]["code"], + "provider_turn_lost_on_restore" + ); + assert_eq!(failed.payload["error"]["recoverable"], true); + // The run stays open for the controller's single same-conversation continuation. + assert!(!events + .iter() + .any(|event| event.event_type == "run.terminal")); + // The controller can still answer the lost call. That result settles the + // call instead of failing against the restored provider, which never saw it. + let lost_result = json!({ + "callId": "call-lost", "operationId": "register_deliverable", + "result": {"ok": true}, "isError": false, + }); + let settled = restored + .execute(&command(5, "semantic_tool.result", lost_result.clone())) + .unwrap(); + assert_eq!(settled.result["retired"], true); + // The receipt is durable and idempotent, so a delivery left indeterminate + // by a crash may replay to the same acknowledgement. + assert!(restored.can_reconcile_result_delivery().unwrap()); + let replayed = restored + .execute(&command(6, "semantic_tool.result", lost_result)) + .unwrap(); + assert_eq!(replayed.result["retired"], true); + // Calls the lost turn never opened still fail closed. + let unknown = json!({ + "callId": "call-unknown", "operationId": "register_deliverable", + "result": {"ok": true}, "isError": false, + }); + assert!(restored + .execute(&command(7, "semantic_tool.result", unknown)) + .is_err()); + let continued = restored + .execute(&command( + 8, + "turn.start", + json!({"turnId": "provider-turn-2", "text": "Continue."}), + )) + .unwrap(); + assert_eq!(continued.result["status"], "accepted"); + restored.shutdown().unwrap(); +} diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs index 8439dd9642f..e83b70871aa 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs @@ -7,7 +7,7 @@ use std::os::unix::fs::PermissionsExt; use paperclip_runner_core::durable::{ AcpxLaunchProfile, Command, CommandExecutor, CursorCloudLaunchProfile, DurableRunnerConfig, - OpenCodeLaunchProfile, QualifiedLaunchArtifact, PROVIDER_STATE_INCOMPATIBLE, + OpenCodeLaunchProfile, PolledEvent, QualifiedLaunchArtifact, PROVIDER_STATE_INCOMPATIBLE, }; use paperclip_runner_core::native_provider_backend::NativeProviderCommandExecutor; use paperclip_runner_core::provider_bridge::{authorized_tool_catalog_digest, AuthorizedTool}; @@ -579,6 +579,30 @@ fn acpx_runtime_resolution(mode: &str, request_id: &str) -> Value { }) } +/// A turn lost to a runner restart is recorded once as a recoverable loss. +fn assert_restart_lost_turn(events: &[PolledEvent]) { + let lost = events + .iter() + .filter(|event| event.event_type == "turn.failed") + .collect::>(); + assert_eq!(lost.len(), 1); + assert_eq!( + lost[0].payload["error"]["code"], + "provider_turn_lost_on_restore" + ); + assert_eq!(lost[0].payload["error"]["recoverable"], true); + assert!(!events + .iter() + .any(|event| event.event_type == "run.terminal")); +} + +/// Restarting the restored, idle session again replays no turn or request. +fn assert_no_replayed_turn_or_request(events: &[PolledEvent]) { + assert!(events + .iter() + .all(|event| event.event_type == "session.resumed")); +} + fn shutdown_recovered_acpx_fixture(executor: &mut NativeProviderCommandExecutor, directory: &Path) { // The fake sidecar advertises the same three lifetime-fence ports for all // fixtures. Parallel recovered shutdowns hold a quorum through state fsync, @@ -679,20 +703,23 @@ fn acpx_response_delivery_survives_crash_before_journaling_without_replaying_the .execute(&command(6, "request.resolve", resolve.payload)) .is_err()); assert_eq!(recovered.poll_events().unwrap(), events); + // The lost turn's conversation is restored for one continuation. Its + // outcome is a recoverable restart loss; the response is not replayed. assert_eq!( recovered .execute(&command(7, "session.snapshot", json!({}))) .unwrap() .result["status"], - "closed" + "session_open" ); + assert_restart_lost_turn(&events); // Polling is not an ACK; only the journal's ACK retires the event. recovered.acknowledge_events(events.len()).unwrap(); assert!(recovered.poll_events().unwrap().is_empty()); drop(recovered); let mut recovered_again = NativeProviderCommandExecutor::with_runner_config(&directory, &config); - assert!(recovered_again.poll_events().unwrap().is_empty()); + assert_no_replayed_turn_or_request(&recovered_again.poll_events().unwrap()); shutdown_recovered_acpx_fixture(&mut recovered_again, &directory); fs::remove_dir_all(directory).unwrap(); } @@ -864,14 +891,15 @@ fn acpx_failed_settlement_ack_replays_the_same_delivered_event_without_expiry() .execute(&command(6, "request.resolve", response)) .is_err()); assert_eq!(recovered.poll_events().unwrap(), events); + assert_restart_lost_turn(&events); recovered.acknowledge_events(events.len()).unwrap(); assert!(recovered.poll_events().unwrap().is_empty()); shutdown_recovered_acpx_fixture(&mut recovered, &directory); drop(recovered); let mut recovered_again = NativeProviderCommandExecutor::with_runner_config(&directory, &config); - assert!(recovered_again.poll_events().unwrap().is_empty()); - recovered_again.shutdown().unwrap(); + assert_no_replayed_turn_or_request(&recovered_again.poll_events().unwrap()); + shutdown_recovered_acpx_fixture(&mut recovered_again, &directory); fs::remove_dir_all(directory).unwrap(); } } @@ -2278,6 +2306,77 @@ fn rejects_a_mutable_opencode_command_outside_the_runner_launch_profile() { } #[test] +#[cfg_attr(windows, ignore = "verified launch is not supported on Windows yet")] +fn restart_lost_opencode_turn_records_a_recoverable_cause() { + // A local OpenCode server restarts with runnerd, like Codex app-server: + // the lost turn keeps its recoverable cause and the run stays open for + // the controller's single same-conversation continuation. + let directory = temporary_directory("opencode-restart-lost-turn"); + let mut config = opencode_config(&directory); + let proxy_script = directory.join("qualified-opencode-proxy-script"); + #[cfg(unix)] + fs::set_permissions(&proxy_script, fs::Permissions::from_mode(0o700)).unwrap(); + fs::write( + &proxy_script, + format!( + "#!/bin/sh\nexec '{}' --state-file '{}' --call-log '{}' --require-completion-contract --durable-turn-ids --linger-after-turn-start\n", + env!("CARGO_BIN_EXE_fake-codex-app-server"), + directory.join("fake-opencode-state.json").display(), + directory.join("fake-opencode-calls.log").display(), + ), + ) + .unwrap(); + #[cfg(unix)] + fs::set_permissions(&proxy_script, fs::Permissions::from_mode(0o500)).unwrap(); + config + .opencode_launch_profile + .as_mut() + .unwrap() + .proxy_script = qualified_artifact(proxy_script); + let mut first = NativeProviderCommandExecutor::with_runner_config(&directory, &config); + first + .execute(&command( + 1, + "run.prepare", + opencode_prepare_payload(&directory), + )) + .unwrap(); + first + .execute(&command(2, "session.open", json!({}))) + .unwrap(); + first + .execute(&command(3, "turn.start", json!({"text": "Edit the file."}))) + .unwrap(); + drop(first); + let fake_state = directory.join("fake-opencode-state.json"); + let mut saved: Value = serde_json::from_slice(&fs::read(&fake_state).unwrap()).unwrap(); + saved["activeTurnId"] = Value::Null; + fs::write(&fake_state, serde_json::to_vec(&saved).unwrap()).unwrap(); + + let mut restored = NativeProviderCommandExecutor::with_runner_config(&directory, &config); + restored + .execute(&command(4, "session.snapshot", json!({}))) + .unwrap(); + let events = restored.poll_events().unwrap(); + let lost = events + .iter() + .find(|event| event.event_type == "turn.failed") + .expect("the lost turn is terminated"); + assert_eq!( + lost.payload["error"]["code"], + "provider_turn_lost_on_restore" + ); + assert_eq!(lost.payload["error"]["recoverable"], true); + assert!(!events + .iter() + .any(|event| event.event_type == "run.terminal")); + assert_eq!(opencode_call_count(&directory, "turn/start"), 1); + restored.shutdown().unwrap(); + fs::remove_dir_all(directory).unwrap(); +} + +#[test] +#[cfg_attr(windows, ignore = "verified launch is not supported on Windows yet")] fn rejects_opencode_launch_profile_drift_across_fresh_recovery() { let directory = temporary_directory("opencode-profile-recovery"); let config = opencode_config(&directory); diff --git a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts index 676dabac7cb..db33af5ffd6 100644 --- a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts +++ b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts @@ -467,6 +467,30 @@ describe("qualified ACPX runtime sidecar", () => { } }); + it("keeps a restored provider's restarted call ids distinct from earlier calls in the run", async () => { + // A restored session starts a new provider process whose MCP request ids + // restart. The controller keys calls by run and call id, so a reused id + // would be mistaken for an earlier, already answered call. + const raw = { reportedWorkDisposition: "done", summary: "Complete", evidence: [], verification: [], + completionClaim: { contractRevision: "1", objectiveSatisfied: true, criteria: [], remainingWork: [] } }; + const tools = new Map(), emitted: any[] = []; + const handler = loadWaitForTool({ tools, emitted, validate: validatePrpStructuredRunResult, callIdNamespace: "rgeneration1" }); + const commands = loadActualToolCommands(tools); + const bridge = await startRunnerToolBridge({ handler, captureSemanticReceipt: () => () => {} }); + try { + const response = fetch(bridge.url, { method: "POST", headers: { Authorization: `Bearer ${bridge.secret}`, "Content-Type": "application/json" }, + body: JSON.stringify({ jsonrpc: "2.0", id: 2, method: "tools/call", params: { name: "paperclip_finish", arguments: raw } }) }); + await vi.waitFor(() => expect(tools.has("rgeneration1:2")).toBe(true)); + expect(emitted).toEqual([expect.objectContaining({ callId: "rgeneration1:2", operationId: "paperclip_finish" })]); + await commands.resolve({ callId: "rgeneration1:2", turnId: "test-turn", result: { accepted: true }, error: null }); + const body = await (await response).json() as ToolCallResponseBody; + expect(body.result.isError).not.toBe(true); + } finally { + for (const pending of tools.values()) pending.cleanup(); + await bridge.close(); + } + }); + it.each([["monitor", true], ["monitor", false], ["same_agent", true], ["same_agent", false]] as const)("resolves an omitted continuation before the production sidecar validator with saved %s authority=%s", async (kind, eligible) => { const raw = { reportedWorkDisposition: "yielded", summary: "The review fix is published.", evidence: [], verification: [], completionClaim: { contractRevision: "1", objectiveSatisfied: false, criteria: [], remainingWork: [{ description: "Wait for checks.", blocksCompletion: true }] } }; @@ -1752,6 +1776,7 @@ function loadWaitForTool(input: { emitted: unknown[]; validate?: typeof validatePrpStructuredRunResult; emit?: (eventType: string, payload: unknown) => boolean; + callIdNamespace?: string | null; }): (call: RunnerToolCall) => Promise { const source = readFileSync( fileURLToPath(new URL("./acpx-runtime-sidecar.ts", import.meta.url)), @@ -1764,7 +1789,7 @@ function loadWaitForTool(input: { const factory = new Function( "boundedIdentity", "tools", "turnId", "emit", "PRP_COMPLETION_TOOL_NAME", "PRP_BLOCK_TOOL_NAME", "validatePrpStructuredRunResult", "boundedSidecarValue", "record", "MAX_PENDING_TOOLS", "stringifyAcpxSidecarFrame", - "resolveCompletionContinuation", "runId", "normalizingTools", "randomUUID", + "resolveCompletionContinuation", "runId", "normalizingTools", "randomUUID", "callIdNamespace", `return (${functionSource});`, ); return factory( @@ -1786,6 +1811,7 @@ function loadWaitForTool(input: { "test-run", new Set(), randomUUID, + input.callIdNamespace ?? null, ); } diff --git a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts index 082a9e93ed3..a95d6a4f7a6 100644 --- a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts +++ b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts @@ -176,6 +176,12 @@ let openParams: AcpxSidecarOpenParams | null = null; let claudeCompactionCount = 0; let runId: string | null = null; let turnId: string | null = null; +/** + * A restored session runs in a new provider process whose MCP request ids + * restart. The controller keys semantic calls by run and call id, so prefix + * this process's ids to keep them distinct from calls earlier in the run. + */ +let callIdNamespace: string | null = null; let activeCopilotEvidence: CopilotToolEvidence | undefined; let sequence = 0; let requestSequence = 0; @@ -374,6 +380,9 @@ async function dispatch( ); host = openedHost; openParams = params; + callIdNamespace = params.expectedIdentity === undefined + ? null + : `r${randomUUID().replaceAll("-", "").slice(0, 12)}`; emit("runtime.process", { role: "sidecar", pid: process.pid, @@ -821,7 +830,10 @@ async function waitForTool(call: RunnerToolCall): Promise { if (!activeTurnId || !activeRunId || call.signal.aborted) { throw new Error("ACPX tool call is not bound to an active turn"); } - const callId = boundedIdentity(call.callId, "callId"); + const providerCallId = boundedIdentity(call.callId, "callId"); + const callId = callIdNamespace === null + ? providerCallId + : boundedIdentity(`${callIdNamespace}:${providerCallId}`, "callId"); if (tools.has(callId) || normalizingTools.has(callId)) throw new Error("ACPX tool call is duplicated"); const operationId = boundedIdentity(call.tool, "operationId"); if (tools.size + normalizingTools.size >= MAX_PENDING_TOOLS) { diff --git a/packages/paperclip-runner/src/cli/eval-gateway-runtime.test.ts b/packages/paperclip-runner/src/cli/eval-gateway-runtime.test.ts index 0ef09abd57f..dfdad937057 100644 --- a/packages/paperclip-runner/src/cli/eval-gateway-runtime.test.ts +++ b/packages/paperclip-runner/src/cli/eval-gateway-runtime.test.ts @@ -73,6 +73,7 @@ it.each([["codex", undefined], ["acpx", "claude"]])("routes %s/%s with only its expect(config).toContain('env_key = "PAPERCLIP_AI_PROVIDER_KEY"'); expect(config).toContain("requires_openai_auth = false"); expect(config).toContain("shell_snapshot = false"); + expect(config).toContain("image_generation = false"); expect(config).toContain('web_search = "disabled"'); expect(config).not.toContain(token); expect((await stat(join(env.CODEX_HOME!, "config.toml"))).mode & 0o777).toBe(0o600); diff --git a/packages/paperclip-runner/src/cli/eval-gateway-runtime.ts b/packages/paperclip-runner/src/cli/eval-gateway-runtime.ts index a4e27f9c180..0dbf4d7cba6 100644 --- a/packages/paperclip-runner/src/cli/eval-gateway-runtime.ts +++ b/packages/paperclip-runner/src/cli/eval-gateway-runtime.ts @@ -76,7 +76,9 @@ export async function prepareEvalGatewayRuntime( 'name = "Paperclip eval controller"', `base_url = ${JSON.stringify(descriptor.baseUrl + "/v1")}`, 'wire_api = "responses"', 'requires_openai_auth = false', 'env_key = "PAPERCLIP_AI_PROVIDER_KEY"', - '[features]', 'shell_snapshot = false', '', + // The controller gateway admits no hosted tools, so qualified evals keep + // Codex's hosted image tool off even though product runs enable it. + '[features]', 'shell_snapshot = false', 'image_generation = false', '', ].join("\n"), { mode: 0o600, flag: "wx" }); } else { Object.assign(environment, { diff --git a/packages/paperclip-runner/src/contracts/native-session-backend.ts b/packages/paperclip-runner/src/contracts/native-session-backend.ts index aefb325d705..fdb0f7461b7 100644 --- a/packages/paperclip-runner/src/contracts/native-session-backend.ts +++ b/packages/paperclip-runner/src/contracts/native-session-backend.ts @@ -95,11 +95,15 @@ export function isNativeRestartInterruption(error: unknown): boolean { return failure.code === NATIVE_RESTART_INTERRUPTION_CODE && failure.recoverable === true; } +const RESTART_CONTINUATION_DRIVER_KINDS: ReadonlySet = new Set(["codex_app_server", "acpx_runtime", "opencode_server"]); + /** The terminal fingerprint survives a crash between interruption and continuation. */ export function nativeRestartInterruptedTurnId(snapshot: Pick): string | null { - if (snapshot.driverKind !== "codex_app_server" || snapshot.activeTurnId != null || + // Runnerd records this cause for Codex app-server and ACPX sessions after + // proving the old provider lifetime ended and restoring the conversation. + if (!RESTART_CONTINUATION_DRIVER_KINDS.has(snapshot.driverKind ?? "") || snapshot.activeTurnId != null || snapshot.semanticResult != null || snapshot.dispositionOnlyRecoveryConsumed || snapshot.goal != null || snapshot.pendingRuntimeRequests?.length) return null; const terminal = Array.isArray(snapshot.terminalTurns) ? snapshot.terminalTurns.at(-1) : null; diff --git a/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.context-control.test.ts b/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.context-control.test.ts index 13e83500aba..f8f9d55e27d 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.context-control.test.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.context-control.test.ts @@ -129,7 +129,6 @@ describe("Codex app-server Codex driver", () => { "skills.include_instructions": false, include_apps_instructions: false, include_collaboration_mode_instructions: true, - "features.image_generation": false, }, permissions: "paperclip-runner-workspace-only", runtimeWorkspaceRoots: [WORKSPACE], @@ -160,8 +159,6 @@ describe("Codex app-server Codex driver", () => { expect.stringContaining( 'shell_environment_policy.set={PATH="/bin",LANG="C.UTF-8"}', ), - "--disable", - "image_generation", ]), ); expect( diff --git a/packages/paperclip-runner/src/drivers/codex/codex-security-config.test.ts b/packages/paperclip-runner/src/drivers/codex/codex-security-config.test.ts index b1b9f42cb3a..c8e15ed4d5c 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-security-config.test.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-security-config.test.ts @@ -1,6 +1,8 @@ import { createRequire } from "node:module"; import { execFileSync } from "node:child_process"; -import { dirname, resolve } from "node:path"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join, resolve } from "node:path"; import { resolvePinnedCodexCommand } from "./codex-command.js"; import { evalProviderTransportOptions } from "../../cli/eval-provider-runtime.js"; import { describe, expect, it } from "vitest"; @@ -358,10 +360,59 @@ describe("Codex security configuration", () => { "features.plugins": false, "features.multi_agent": false, "features.memories": false, - "features.image_generation": false, }); }); + it("N-12: lets an API-key Codex agent generate images with its existing provider key instead of asking for another", () => { + // Codex API path: the agent's OpenAI key reaches only the trusted + // app-server process. Codex's built-in image tool uses that provider auth; + // with it disabled the agent falls back to an OPENAI_API_KEY script that + // the sandboxed shell can never see, so it asks the user for a second key. + const source = { + HOME: "/host/home", CODEX_HOME: "/host/codex", PATH: "/safe/bin", + OPENAI_API_KEY: "agent-provider-key", + }; + const args = createIsolatedCodexAppServerArgs(source); + const serialized = args.join("\n"); + expect(args).not.toContain("image_generation"); + for (const mode of ["default", "plan"] as const) { + expect(createSecuredCodexThreadParams("/workspace", mode, true, false, source).config) + .not.toHaveProperty("features.image_generation"); + } + // The tool saves results under CODEX_HOME/generated_images; the agent must + // be able to read (and so copy) them into the workspace, nothing else there. + for (const profile of ["paperclip-runner-workspace-only", "paperclip-runner-workspace-read-only"]) { + const rules = args.find((arg) => arg.startsWith(`permissions.${profile}.filesystem=`))!; + expect(rules).toContain('"/host/codex"="none"'); + expect(rules).toContain('"/host/codex/generated_images"="read"'); + } + // The existing key is reused by the provider, never exposed to the shell. + expect(serialized).not.toContain("OPENAI_API_KEY"); + expect(serialized).not.toContain("agent-provider-key"); + }); + + it("N-12: grants generated images in preconfigured mode only when CODEX_HOME is outside HOME", () => { + const inside = createIsolatedCodexAppServerArgs({ HOME: "/users/executor", CODEX_HOME: "/users/executor/.codex", PAPERCLIP_RUNNER_HOME_MODE: "preconfigured" }).join("\n"); + expect(inside).not.toContain("generated_images"); + const outside = createIsolatedCodexAppServerArgs({ HOME: "/users/executor", CODEX_HOME: "/var/lib/codex-home", PAPERCLIP_RUNNER_HOME_MODE: "preconfigured" }).join("\n"); + expect(outside.match(/"\/var\/lib\/codex-home\/generated_images"="read"/g)).toHaveLength(2); + expect(outside).not.toContain('"/var/lib/codex-home"='); + }); + + it("N-12: the pinned Codex binary keeps built-in image generation enabled under the isolated launch", () => { + const codexHome = mkdtempSync(join(tmpdir(), "paperclip-n12-codex-home-")); + try { + const args = createIsolatedCodexAppServerArgs({ PATH: process.env.PATH, HOME: codexHome, CODEX_HOME: codexHome }); + const output = execFileSync(resolvePinnedCodexCommand(), [...args.slice(0, -1), "features", "list"], { + env: { PATH: process.env.PATH, HOME: codexHome, CODEX_HOME: codexHome }, + encoding: "utf8", timeout: 30_000, maxBuffer: 256 * 1024, + }); + expect(output.split("\n").find((line) => /^image_generation\s/.test(line))).toMatch(/\strue\s*$/); + } finally { + rmSync(codexHome, { recursive: true, force: true }); + } + }); + it("keeps automatic execution inside the workspace without host credentials and with normal network access", () => { const args = createIsolatedCodexAppServerArgs( { diff --git a/packages/paperclip-runner/src/drivers/codex/codex-security-config.ts b/packages/paperclip-runner/src/drivers/codex/codex-security-config.ts index 0121dce2e86..0130c7ba024 100644 --- a/packages/paperclip-runner/src/drivers/codex/codex-security-config.ts +++ b/packages/paperclip-runner/src/drivers/codex/codex-security-config.ts @@ -190,7 +190,10 @@ const SKILLLESS_BASE_CONFIG = { "features.plugins": false, "features.multi_agent": false, "features.memories": false, - "features.image_generation": false, + // Image generation stays on: Codex's built-in tool runs in the trusted + // app-server with the agent's existing provider auth. Disabling it makes the + // agent fall back to an OPENAI_API_KEY script the sandboxed shell cannot see, + // so it asks the user for a second key (N-12). } as const; export function codexCommandEnvironment( @@ -288,6 +291,16 @@ export function createIsolatedCodexAppServerArgs( .map((value) => resolve(value)), ), ]; + // Codex saves built-in image results under CODEX_HOME/generated_images and + // tells the agent to copy them into the workspace. Expose only that folder. + // A preconfigured HOME is already granted, so add the rule only when the + // Codex home sits outside it (never narrow an existing HOME write grant). + const generatedImagesRoot = typeof source.CODEX_HOME === "string" && source.CODEX_HOME.trim() + ? join(resolve(source.CODEX_HOME), "generated_images") : undefined; + const preconfiguredHome = usesPreconfiguredHome(source) && source.HOME ? resolve(source.HOME) : undefined; + const generatedImageRoots = generatedImagesRoot + && !(preconfiguredHome && codexPathInside(preconfiguredHome, generatedImagesRoot)) + ? [generatedImagesRoot] : []; const filesystemRules = [ `":root"="none"`, `":minimal"="read"`, @@ -296,6 +309,7 @@ export function createIsolatedCodexAppServerArgs( ...protectedPaths.map((path) => `${tomlString(resolve(path))}="none"`), ...(usesPreconfiguredHome(source) && source.HOME ? [`${tomlString(source.HOME)}="write"`] : []), ...readOnlyRoots.map((path) => `${tomlString(resolve(path))}="read"`), + ...generatedImageRoots.map((path) => `${tomlString(path)}="read"`), ...gitRoots.read.map((path) => `${tomlString(path)}="read"`), ...gitRoots.write.map((path) => `${tomlString(path)}="write"`), ...(instructionWorkingCopyRoot ? [`${tomlString(resolve(instructionWorkingCopyRoot))}="write"`] : []), @@ -311,6 +325,7 @@ export function createIsolatedCodexAppServerArgs( ...protectedPaths.map((path) => `${tomlString(resolve(path))}="none"`), ...(usesPreconfiguredHome(source) && source.HOME ? [`${tomlString(source.HOME)}="read"`] : []), ...readOnlyRoots.map((path) => `${tomlString(resolve(path))}="read"`), + ...generatedImageRoots.map((path) => `${tomlString(path)}="read"`), ...[...gitRoots.read, ...gitRoots.write].map((path) => `${tomlString(path)}="read"`), ...(instructionWorkingCopyRoot ? [`${tomlString(resolve(instructionWorkingCopyRoot))}="read"`] : []), ...(source.PAPERCLIP_GITHUB_BROKER_TOKEN && source.GH_CONFIG_DIR @@ -358,8 +373,6 @@ export function createIsolatedCodexAppServerArgs( // as the turn plan (`turn/plan/updated` -> `plan.updated`). "-c", CODEX_UPDATE_PLAN_TOOL_CONFIG, - "--disable", - "image_generation", ...(externalRunnerSandbox ? ["--dangerously-bypass-approvals-and-sandbox"] : []), diff --git a/packages/paperclip-runner/src/native-session-runtime.test.ts b/packages/paperclip-runner/src/native-session-runtime.test.ts index de63070094e..7cae1d4d807 100644 --- a/packages/paperclip-runner/src/native-session-runtime.test.ts +++ b/packages/paperclip-runner/src/native-session-runtime.test.ts @@ -20,6 +20,7 @@ import { NativeSessionCloseUnrecoverableError, NativeSessionCleanupQuarantinedError, NativeSessionProtocolIntegrityError, + nativeRestartInterruptedTurnId, } from "./contracts/native-session-backend.js"; import { PROVIDER_STATE_INCOMPATIBLE } from "./contracts/harness-driver.js"; import type { @@ -508,6 +509,71 @@ describe("executeNativeSession recovery", () => { expect(checkpoints.some(snapshot => snapshot.terminalTurns?.some(turn => turn.turnId === "turn-recovery"))).toBe(true); }); + it.each(["acpx_runtime", "opencode_server"] as const)("continues a restart-lost %s turn once in the same conversation", async (driverKind) => { + const qualified = resolveQualifiedAcpxProfile("claude", "claude-sonnet-5"); + const acpxInput: NativeExecutionInputV1 = (driverKind === "acpx_runtime" ? { + ...input, + session: { ...input.session, driverKind }, + provider: { kind: "acpx", agent: "claude", model: "claude-sonnet-5", permissionPolicy: "interactive", profile: { + driverKind: qualified.driverKind, protocolVersion: qualified.protocolVersion, acpxVersion: qualified.acpxVersion, + agent: qualified.agent, agentProfileVersion: qualified.agentProfileVersion, + agentServerPackage: qualified.agentServerPackage, agentServerVersion: qualified.agentServerVersion, + agentRuntimePackage: qualified.agentRuntimePackage, agentRuntimeVersion: qualified.agentRuntimeVersion, + commandDigest: qualified.commandDigest, + } }, + } : { + ...input, + session: { ...input.session, driverKind }, + provider: { kind: "opencode", model: "openrouter/deepseek/deepseek-v4-flash-0731" }, + }) as NativeExecutionInputV1; + const failure = { code: "provider_turn_lost_on_restore", recoverable: true }; + const checkpoint: PersistedNativeSession = { + backendKind: "runner", driverKind, sessionId: "record-recovery", identity, + providerSessionId: "provider-recovery", cursor: "0", activeTurnId: "turn-recovery", + terminalTurns: [], semanticResult: null, pendingRuntimeRequests: [], lineage: [], + }; + const interrupted: PersistedNativeSession = { + ...checkpoint, activeTurnId: null, + terminalTurns: [{ turnId: "turn-recovery", fingerprint: JSON.stringify({ terminalState: "failed", error: failure, result: null }) }], + }; + const startTurn = vi.fn(async () => ({ turnId: "turn-continuation" })); + const session: NativeSession = { + identity: () => identity, + async capabilities() { return { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }; }, + async *events() { + yield runnerEvent(1, "turn.failed", { status: "failed", error: failure }); + if (!startTurn.mock.calls.length) return; + yield { ...runnerEvent(2, "turn.completed", { status: "completed" }), turnId: "turn-continuation" }; + }, + startTurn, + async result() { return startTurn.mock.calls.length ? { result, terminal, turnId: "turn-continuation" } : null; }, + async snapshot() { return structuredClone(interrupted); }, + async close() {}, + }; + const backend: NativeSessionBackend = { + async descriptor() { return { kind: "runner", name: "restart", version: "1", capabilities: await session.capabilities() }; }, + async openSession() { throw new Error("must preserve the provider session"); }, + async recoverSession() { return { recovered: true, session }; }, + }; + const port: ControlPlanePort = { + async openRun() {}, + async checkpointSession() {}, + async appendEvent(event: PrpEvent) { return { cursor: event.sourceSeq, highestContiguousSourceSeq: event.sourceSeq, disposition: "committed" }; }, + async replayEvents() { return { events: [], highestContiguousSourceSeq: 0 }; }, + async completeRun() {}, + }; + await expect(executeNativeSession({ + input: acpxInput, backend, controlPlane: port, persistedSession: checkpoint, + runnerInstanceId: "runner-recovery", controlPlaneInstanceId: "control-recovery", + resumeInterruptedTurn: true, + })).resolves.toMatchObject({ result, turnId: "turn-continuation" }); + expect(startTurn).toHaveBeenCalledOnce(); + const submitted = startTurn.mock.calls[0]![0] as Parameters[0]; + expect(submitted.continuation).toBe(true); + expect(submitted.message.text).toContain("Reconcile any unfinished tool or command"); + expect(nativeRestartInterruptedTurnId(interrupted)).toBe("turn-recovery"); + }); + it("never continues task work when a restart loses a compaction turn", async () => { const failure = { code: "provider_turn_lost_on_restore", recoverable: true }; const checkpoint: PersistedNativeSession = { diff --git a/packages/paperclip-runner/src/native-session-runtime.ts b/packages/paperclip-runner/src/native-session-runtime.ts index 273f84ba33a..55fb486d92f 100644 --- a/packages/paperclip-runner/src/native-session-runtime.ts +++ b/packages/paperclip-runner/src/native-session-runtime.ts @@ -2448,7 +2448,9 @@ export async function executeNativeSession( const resumeInterruptedTurn = async (event: PrpEvent, signal: AbortSignal) => { if (restartContinuationStarted || !recovered || !options.resumeInterruptedTurn || // A compaction-only run never continues task work after a restart. - input.provider.kind !== "codex" || options.sessionCompaction || options.sessionGoalControl || options.resumeSessionGoalHeartbeat || + // Runnerd records a restart-lost turn only for these providers, after + // proving the old process ended and restoring the same conversation. + !["codex", "acpx", "opencode"].includes(input.provider.kind) || options.sessionCompaction || options.sessionGoalControl || options.resumeSessionGoalHeartbeat || recoveredSnapshot.goal || persistedSession?.semanticResult || persistedSession?.pendingRuntimeRequests?.length || event.eventType !== "turn.failed" || !isNativeRestartInterruption(event.payload.error) || event.turnId !== persistedSession?.activeTurnId) return false; diff --git a/packages/shared/src/validators/tool-access.ts b/packages/shared/src/validators/tool-access.ts index 914af74e7ab..f55163c22eb 100644 --- a/packages/shared/src/validators/tool-access.ts +++ b/packages/shared/src/validators/tool-access.ts @@ -191,7 +191,8 @@ export const createToolConnectionSchema = z.object({ status: toolConnectionStatusSchema.optional(), connectionKind: toolConnectionKindSchema.default("managed"), config: toolTransportConfigSchema.optional(), - transportConfig: toolTransportConfigSchema.default({}), + // No default: when the caller omits it, the service stores `config` here. + transportConfig: toolTransportConfigSchema.optional(), credentialRefs: z.array(mcpConnectionCredentialRefSchema).optional(), credentialSecretRefs: z.array(toolCredentialSecretRefSchema).default([]), enabled: z.boolean().optional(), diff --git a/server/src/__tests__/deferred-execution-receipt-work.test.ts b/server/src/__tests__/deferred-execution-receipt-work.test.ts index be0135fc64e..414ac9d5abf 100644 --- a/server/src/__tests__/deferred-execution-receipt-work.test.ts +++ b/server/src/__tests__/deferred-execution-receipt-work.test.ts @@ -15,7 +15,7 @@ import { issueTreeControlService } from "../services/issue-tree-control.js"; import { randomUUID } from "node:crypto"; import { and, eq, sql } from "drizzle-orm"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; -import { activityLog, agents, agentWakeupRequests, authUsers, chatActions, environmentLeases, nativeRunFinalizations, chatEndpoints, toolApplications, toolConnections, companies, companyMemberships, costEvents, createDb, projects, reconcileDatabaseWork, heartbeatRuns, issueComments, issues } from "@paperclipai/db"; +import { activityLog, agents, agentWakeupRequests, authUsers, chatActions, environmentLeases, nativeRunFinalizations, chatEndpoints, toolApplications, toolConnections, companies, companyMemberships, costEvents, createDb, projects, reconcileDatabaseWork, heartbeatRuns, issueComments, issueRelations, issues } from "@paperclipai/db"; import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; @@ -158,6 +158,96 @@ describe("deferred receipt durable dispatch", () => { source: "on_demand", status: "completed", reason: "task_retry", payload: { issueId: f.issue.id } }); await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready; await quiet(); }); + async function strandedDependency() { + const f = await fixture(); + await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: f.issue.id, + status: "succeeded", errorCode: null, contextSnapshot: { issueId: f.issue.id, + wakeReason: "issue_commented", dependencyBlockedInteraction: true } }).where(eq(heartbeatRuns.id, f.source.id)); + await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issue.id, + runId: f.source.id, phase: "committed" }); + await db.update(issues).set({ status: "blocked", executionRunId: null, checkoutRunId: null }).where(eq(issues.id, f.issue.id)); + const [blocker] = await db.insert(issues).values({ companyId: f.companyId, title: "Completed review", + status: "done", completedAt: new Date() }).returning(); + await db.insert(issueRelations).values({ companyId: f.companyId, issueId: blocker.id, + relatedIssueId: f.issue.id, type: "blocks" }); + const [intent] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agent.id, + source: "automation", reason: "issue_blockers_resolved", status: "coalesced", + requestedByActorType: "system", requestedByActorId: "native-status-committer", + payload: { issueId: f.issue.id, resolvedBlockerIssueId: blocker.id }, + }).returning(); + const [receipt] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agent.id, + source: "automation", reason: "issue_execution_deferred", status: "deferred_issue_execution", + requestedByActorType: "system", requestedByActorId: `native-status-wake-dispatch:${intent.id}`, + payload: { issueId: f.issue.id, resolvedBlockerIssueId: blocker.id, _paperclipWakeContext: { + issueId: f.issue.id, wakeReason: "issue_blockers_resolved", statusDecisionSource: "native_status_decision", + nativeStatusWakeIntentId: intent.id } }, + }).returning(); + return { ...f, receipt, blocker }; + } + + it.each(["restart", "cleanup"] as const)("recovers a dependency wake after native audit %s into one continuation", async event => { + const f = await strandedDependency(); + const { receipt } = f; + const [lease] = event === "cleanup" ? await db.insert(environmentLeases).values({ companyId: f.companyId, + heartbeatRunId: f.source.id, issueId: f.issue.id, provider: "daytona", providerLeaseId: "dependency-finalizing", + status: "active", leasePolicy: "reuse_by_environment" }).returning() : []; + await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready; + await heartbeat.registerRunDispatchWork(coordinator, () => !drain).ready; + if (lease) { + await waitForReal(async () => { + await Promise.allSettled([...passes]); + expect(coordinator.nextWakeAt()).toBeNull(); expect(idleWorkSnapshot().active).toBe(0); + }, { timeout: 10_000 }); + expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.id)))[0]) + .toMatchObject({ status: "deferred_issue_execution", runId: null }); + await environmentService(db).releaseLease(lease.id, "retained", { cleanupStatus: "success" }); + } + await waitForReal(async () => { + await Promise.allSettled([...passes]); await heartbeat.drainActiveRunExecutions(); + const [saved] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.id)); + const pending = saved.runId ? (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, saved.runId)))[0] : null; + expect(saved.status, JSON.stringify({ status: pending?.status, error: pending?.error, errors })).toBe("completed"); + expect(saved.runId).not.toBeNull(); + }, { timeout: 15_000 }); + await finish(f.issue.id); + const runs = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId)); + const dependencyRuns = runs.filter(run => run.contextSnapshot?.wakeReason === "issue_blockers_resolved"); + expect(dependencyRuns).toHaveLength(1); + const [resumed] = dependencyRuns; + expect(resumed).toMatchObject({ status: "succeeded", stdoutExcerpt: expect.stringContaining("receipt-proof"), + contextSnapshot: { wakeReason: "issue_blockers_resolved" } }); + expect(resumed.contextSnapshot?.dependencyBlockedInteraction).not.toBe(true); + await heartbeat.resumeQueuedRuns(); + const after = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId)); + expect(after.filter(run => run.contextSnapshot?.wakeReason === "issue_blockers_resolved")).toHaveLength(1); + expect(after.find(run => run.id === f.source.id)?.status).toBe("succeeded"); + expect((await db.select().from(issues).where(eq(issues.id, f.issue.id)))[0]) + .toMatchObject({ status: "done", executionRunId: null, checkoutRunId: null }); + await quiet(); + }); + it.each(["paused", "cleanup", "cancelled"] as const)("keeps the %s gate when recovering a dependency wake", async gate => { + const f = await strandedDependency(); + if (gate === "paused") await db.update(agents).set({ status: "paused" }).where(eq(agents.id, f.agent.id)); + if (gate === "cancelled") await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, f.issue.id)); + if (gate === "cleanup") await db.insert(environmentLeases).values({ companyId: f.companyId, + heartbeatRunId: f.source.id, issueId: f.issue.id, provider: "daytona", providerLeaseId: "dependency-cleanup", + status: "active", leasePolicy: "reuse_by_environment" }); + await heartbeat.resumeQueuedRuns(); + const [saved] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.receipt.id)); + expect(saved).toMatchObject({ status: gate === "cancelled" ? "cancelled" : "deferred_issue_execution", runId: null }); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1); + }); + it("checks unresolved blockers before a recovered dependency wake can execute", async () => { + const f = await strandedDependency(); + await db.update(issues).set({ status: "blocked", completedAt: null }).where(eq(issues.id, f.blocker.id)); + await heartbeat.resumeQueuedRuns(); + await heartbeat.drainActiveRunExecutions(); + const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.receipt.id)); + expect(receipt).toMatchObject({ status: "skipped", runId: expect.any(String) }); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, receipt.runId!)); + expect(run).toMatchObject({ status: "cancelled", errorCode: "issue_dependencies_blocked", stdoutExcerpt: null }); + expect((await db.select().from(issues).where(eq(issues.id, f.issue.id)))[0].status).toBe("blocked"); + }); it.each([false, true])("reconsiders every saved receipt during repeated company eligibility changes (expired wait: %s)", async expiredWait => { const f = await fixture(); const ids = Array.from({ length: 178 }, (_, index) => `${(index + 1).toString(16).padStart(8, "0")}-0000-4000-8000-000000000000`); diff --git a/server/src/__tests__/environment-routes.test.ts b/server/src/__tests__/environment-routes.test.ts index 786e97d6ea2..0274007ac2d 100644 --- a/server/src/__tests__/environment-routes.test.ts +++ b/server/src/__tests__/environment-routes.test.ts @@ -1022,6 +1022,20 @@ describe("environment routes", () => { describe("managed plugin environment rows", () => { const PLUGIN_KEY = "example.remote-host-plugin"; + const MANAGED_CONFIG_WITH_PLUGIN_ENTRY = JSON.stringify({ + v: 1, + mode: "cloud", + catalogVersion: "2026.1106.0", + features: {}, + plugins: { autoInstall: ["remote-host"] }, + environments: [{ + name: "Remote Host", + driver: "plugin", + pluginKey: PLUGIN_KEY, + driverKey: "tasks", + driverConfig: {}, + }], + }); const pluginRow = () => ({ ...createPlatformSandboxEnvironment(), id: "env-plugin-1", @@ -1058,20 +1072,46 @@ describe("environment routes", () => { expect(res.body.details).toMatchObject({ code: "environment_platform_managed" }); }); - it("allows the marker clear while no managed-config entry can own the row", async () => { - // Nothing provisions marked plugin rows yet, so a marker is stale and - // stays recoverable. The managed-config plugin entry PR refuses this - // while the document declares the row's entry. - const row = pluginRow(); - mockEnvironmentService.getById.mockResolvedValue(row); - mockEnvironmentService.update.mockResolvedValue({ ...row, metadata: {} }); + it("refuses a marker clear while the document declares the entry", async () => { + process.env.PAPERCLIP_MANAGED_CONFIG = MANAGED_CONFIG_WITH_PLUGIN_ENTRY; + try { + mockEnvironmentService.getById.mockResolvedValue(pluginRow()); + const app = createApp(ownerAdminActor); - const res = await request(createApp(ownerAdminActor)) - .patch("/api/environments/env-plugin-1") - .send({ metadata: { managedByPaperclip: null, managedPluginEnvironment: null } }); + for (const metadata of [ + { managedByPaperclip: false }, + { managedPluginEnvironment: null }, + { managedByPaperclip: null, managedPluginEnvironment: null }, + ]) { + const res = await request(app).patch("/api/environments/env-plugin-1").send({ metadata }); + expect(res.status).toBe(403); + expect(res.body.details).toMatchObject({ code: "environment_platform_managed" }); + } + expect(mockEnvironmentService.update).not.toHaveBeenCalled(); + } finally { + delete process.env.PAPERCLIP_MANAGED_CONFIG; + } + }); - expect(res.status).toBe(200); - expect(mockEnvironmentService.update).toHaveBeenCalled(); + it("allows the marker clear once the entry left the document", async () => { + process.env.PAPERCLIP_MANAGED_CONFIG = JSON.stringify({ + ...JSON.parse(MANAGED_CONFIG_WITH_PLUGIN_ENTRY), + environments: [], + }); + try { + const row = pluginRow(); + mockEnvironmentService.getById.mockResolvedValue(row); + mockEnvironmentService.update.mockResolvedValue({ ...row, metadata: {} }); + + const res = await request(createApp(ownerAdminActor)) + .patch("/api/environments/env-plugin-1") + .send({ metadata: { managedByPaperclip: null, managedPluginEnvironment: null } }); + + expect(res.status).toBe(200); + expect(mockEnvironmentService.update).toHaveBeenCalled(); + } finally { + delete process.env.PAPERCLIP_MANAGED_CONFIG; + } }); it("refuses a client that stamps the plugin marker on its own row", async () => { diff --git a/server/src/__tests__/execution-control-work.test.ts b/server/src/__tests__/execution-control-work.test.ts index 584eeafb6af..345bf605776 100644 --- a/server/src/__tests__/execution-control-work.test.ts +++ b/server/src/__tests__/execution-control-work.test.ts @@ -61,6 +61,41 @@ describe("execution-control lifecycle", () => { expect(run).toHaveBeenCalledTimes(runs + 1); expect(c.nextWakeAt()).toBeNull(); }); + it.each(["empty", "pending", "concurrent"] as const)("settles only proved-empty replacement notifications during drain (%s)", async kind => { + vi.useFakeTimers(); const owner = fakeOwner(); let draining = false; + const c = coordinator(owner, () => !draining); + const run = vi.fn(async () => {}), next = vi.fn(async (): Promise => null); + await registerExecutionControlWork(c, [{ queue: DELIVERY_QUEUES.executionReplacement, run, next, + settleEmptyWhilePaused: true }]).ready; + const before = idleWorkSnapshot().active; + draining = true; + let release: (() => void) | undefined; + if (kind === "pending") next.mockResolvedValue(Date.now() + 15_000); + if (kind === "concurrent") next.mockImplementationOnce(() => new Promise(resolve => { + release = () => resolve(null); + })); + await owner.transaction(tx => notifyDeliveryWork(tx, DELIVERY_QUEUES.executionReplacement)); + await vi.advanceTimersByTimeAsync(15_000); + if (kind === "concurrent") { + expect(release).toBeTypeOf("function"); + next.mockResolvedValue(Date.now() + 15_000); + await owner.transaction(tx => notifyDeliveryWork(tx, DELIVERY_QUEUES.executionReplacement)); + release!(); await vi.advanceTimersByTimeAsync(1); + } + expect(run).toHaveBeenCalledTimes(1); + expect(idleWorkSnapshot().active).toBe(before + (kind === "empty" ? 0 : 1)); + const reads = next.mock.calls.length; + await vi.advanceTimersByTimeAsync(60_000); + expect(next).toHaveBeenCalledTimes(reads); + expect(run).toHaveBeenCalledTimes(1); + if (kind !== "empty") { + draining = false; next.mockResolvedValue(null); + await vi.advanceTimersByTimeAsync(15_000); + expect(run).toHaveBeenCalledTimes(2); + } + expect(idleWorkSnapshot().active).toBe(before); + expect(c.nextWakeAt()).toBeNull(); + }); it("keeps a change arriving during a sweep and does not overlap it", async () => { vi.useFakeTimers(); const owner = fakeOwner(); const c = coordinator(owner); let release!: () => void; diff --git a/server/src/__tests__/helpers/historical-listener-continuation.ts b/server/src/__tests__/helpers/historical-listener-continuation.ts new file mode 100644 index 00000000000..332003b42ee --- /dev/null +++ b/server/src/__tests__/helpers/historical-listener-continuation.ts @@ -0,0 +1,134 @@ +import { createServer } from "node:http"; +import { spawn } from "node:child_process"; +import { createRequire } from "node:module"; +import { join, resolve } from "node:path"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { eq } from "drizzle-orm"; +import { completionContracts, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db"; +import { expect } from "vitest"; +import { createRunDispatch } from "../../modules/run-dispatch/index.js"; +import { prepareNativeHeartbeatRun } from "../../services/native-runtime/prepare-native-run.js"; +import { buildNativeExecutionInput } from "../../services/native-runtime/native-execution-input.js"; +import { buildNativeCompletionContract } from "../../services/native-runtime/completion-contracts.js"; +import { nativeRuntimeContextFixture } from "../../services/native-runtime/runtime-context.test-fixture.js"; +import { materializeAsset } from "../../services/native-runtime/runtime-context.js"; +import { executePaperclipNativeSession } from "../../services/native-runtime/native-session-executor.js"; +import { reconcileNativeFinalizations } from "../../services/native-runtime/native-finalization-reconciler.js"; +import { canonicalNativeRuntimeContextDigest, createNativeSessionBackend, createRunnerdCodexTransport, defaultCapabilityRunnerdBinary } from "../../vendor/paperclip-runner/index.js"; +import { registerLocalRuntimeApiServer } from "../../runtime-api.js"; +import { registerRunnerPrpAuthority, runnerPrpWebSocketInternals, setupRunnerPrpWebSocketServer } from "../../realtime/runner-prp-ws.js"; + +async function prepare(db: Db, runId: string, cwd: string, environmentLeaseId: string) { + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + const [issue] = await db.select().from(issues).where(eq(issues.id, run.nativeIssueId ?? String(run.contextSnapshot?.issueId))); + await db.update(issues).set({ executionRunId: run.id }).where(eq(issues.id, issue.id)); + const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId }); + const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issue.id)); + const runtimeContext = nativeRuntimeContextFixture(); + runtimeContext.instructions.bundle = await materializeAsset([{ path: "AGENTS.md", mode: 0o444, + content: Buffer.from("Complete the saved task once.\n") }]); + runtimeContext.aggregateDigest = canonicalNativeRuntimeContextDigest(runtimeContext); + const execution = buildNativeExecutionInput({ companyId: run.companyId, agentId: run.agentId, runId, issue, + taskPrompt: "Preserve the saved file and complete the task once.", provider: "codex", + workspace: { id: issue.executionWorkspaceId!, cwd, repoUrl: null, repoRef: null, branchName: null }, + normalizedSessionId: native.normalizedSessionId, runtimeContext, + completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1", + contract: buildNativeCompletionContract(issue, { revision: contract.revision }) }, + }); + const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + await db.update(heartbeatRuns).set({ runnerProfileJson: { ...run.runnerProfileJson, ...current.runnerProfileJson, nativeExecutionInput: execution } }) + .where(eq(heartbeatRuns.id, runId)); + return { run: current, native, execution }; +} + +export async function recordHistoricalListenerFailure(input: { + db: Db; connectionString: string; runId: string; cwd: string; environmentId: string; leaseId: string; +}) { + await input.db.update(heartbeatRuns).set({ status: "running", finishedAt: null, error: null, errorCode: null }) + .where(eq(heartbeatRuns.id, input.runId)); + const { execution, native } = await prepare(input.db, input.runId, input.cwd, input.leaseId); + await input.db.insert(nativeRunFinalizations).values({ runId: input.runId, companyId: execution.binding.companyId, + issueId: execution.binding.issueId, phase: "observed" }); + const child = spawn(process.execPath, ["--import", createRequire(import.meta.url).resolve("tsx"), + resolve(import.meta.dirname, "native-drain-controller.test-worker.ts")], { stdio: ["ignore", "pipe", "pipe", "ipc"], env: process.env }); + const logs: string[] = []; + child.stdout?.on("data", data => logs.push(String(data))); + child.stderr?.on("data", data => logs.push(String(data))); + const exited = new Promise(resolveExit => child.once("exit", resolveExit)); + try { + const result = await new Promise((resolveResult, reject) => { + const timeout = setTimeout(() => reject(new Error(logs.join("").slice(-3000))), 60_000); + child.once("error", reject); + child.on("message", (message: { ready?: boolean; done?: boolean; error?: string }) => { + if (message.ready) child.send({ connectionString: input.connectionString, options: { execution, + runnerInstanceId: native.runnerInstanceId, useRunnerd: true, + runnerExecutionTarget: { kind: "remote", transport: "sandbox", environmentId: input.environmentId, + leaseId: input.leaseId, remoteCwd: input.cwd } } }); + if (message.done || message.error) { clearTimeout(timeout); message.error ? reject(new Error(message.error)) : resolveResult(message); } + }); + }); + expect(result).toMatchObject({ commands: 0, errorMessage: "local_runtime_api_listener_unavailable" }); + expect(await exited).toBe(0); + const [failed] = await input.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, input.runId)); + expect(failed).toMatchObject({ phase: "terminal_failure", attempt: 1, failureCode: "native_session_interrupted", + failureDetail: { message: "local_runtime_api_listener_unavailable", recoveryMode: "ambiguous_state", + providerSessionEstablished: false, providerEventsExist: false, checkpointExists: false } }); + } finally { child.kill("SIGKILL"); await exited; } +} + +export async function completeHistoricalListenerSuccessor(input: { db: Db; runId: string; root: string; + prepareWorkspace: () => Promise<{ cwd: string; finalize: () => Promise }> }) { + const [scheduled] = await input.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId)); + const dispatch = createRunDispatch(input.db); + expect(await dispatch.promoteScheduledRetry({ companyId: scheduled.companyId, runId: scheduled.id, + now: new Date(scheduled.scheduledRetryAt!.getTime() + 1) })).toMatchObject({ outcome: "promoted" }); + expect(await dispatch.cancelStaleQueuedRun({ companyId: scheduled.companyId, runId: scheduled.id, expectedStatus: "queued" })) + .toMatchObject({ outcome: "not_stale" }); + await input.db.update(heartbeatRuns).set({ status: "running", runtimeMode: "native", + nativeIssueId: String(scheduled.contextSnapshot?.issueId) }).where(eq(heartbeatRuns.id, scheduled.id)); + await input.db.update(issues).set({ executionRunId: scheduled.id }).where(eq(issues.id, String(scheduled.contextSnapshot?.issueId))); + const workspace = await input.prepareWorkspace(); + const { execution, native } = await prepare(input.db, scheduled.id, workspace.cwd, `controlled-${scheduled.id}`); + await input.db.insert(nativeRunFinalizations).values({ runId: scheduled.id, companyId: scheduled.companyId, + issueId: execution.binding.issueId, phase: "observed" }); + const home = join(input.root, "codex-home"), report = join(input.root, "result.json"), calls = join(input.root, "calls.log"); + await mkdir(home, { recursive: true }); + await writeFile(report, JSON.stringify({ schema: "paperclip.run_result.v1", reportedWorkDisposition: "done", + summary: "Saved task completed after historical controller failure.", completionClaim: { + contractRevision: execution.completionContract.contract.revision, objectiveSatisfied: true, + criteria: execution.completionContract.contract.criteria.map(criterion => ({ criterionId: criterion.id, status: "satisfied", evidenceRefs: [] })), + remainingWork: [] }, evidence: [], verification: [], artifacts: [], attentionRequests: [] })); + const server = createServer(); + registerLocalRuntimeApiServer(server); + await new Promise(resolveListen => server.listen(0, "127.0.0.1", resolveListen)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("TCP listener required"); + setupRunnerPrpWebSocketServer(server, { apiUrl: `http://127.0.0.1:${address.port}` }); + const bundle = createRunnerdCodexTransport({ runnerBinary: defaultCapabilityRunnerdBinary(), + codexCommand: resolve(import.meta.dirname, "../../../../packages/paperclip-runner/runner/target/debug/fake-codex-app-server"), + codexArgs: ["--state-file", join(input.root, "provider.json"), "--terminal-result-file", report, "--call-log", calls, "--record-process-start"], + stateDirectory: input.root, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null }, + runtimeContext: execution.runtimeContext, runnerRuntimeContext: execution.runtimeContext, + sourceCodexHome: home, environment: { CODEX_HOME: home, PATH: process.env.PATH }, + prpIdentity: { runnerInstanceId: native.runnerInstanceId, environmentLeaseId: native.environmentLeaseId, + runId: scheduled.id, normalizedSessionId: native.normalizedSessionId, turnId: native.turnId, itemId: native.itemId }, + controlPlaneRegistration: authority => registerRunnerPrpAuthority({ companyId: scheduled.companyId, runId: scheduled.id, authority }), + }); + const backend = createNativeSessionBackend(execution, { runnerInstanceId: native.runnerInstanceId, + environment: { CODEX_HOME: home, PATH: process.env.PATH }, codexTransportFactory: () => bundle.transport }); + try { + expect(await executePaperclipNativeSession({ db: input.db, execution, backend, useRunnerd: true, + runnerInstanceId: native.runnerInstanceId, turnTimeoutMs: 15_000 })).toMatchObject({ exitCode: 0, timedOut: false }); + await workspace.finalize(); + await reconcileNativeFinalizations(input.db, [scheduled.id]); + const [finished] = await input.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, scheduled.id)); + expect(finished, JSON.stringify(finished.failureDetail)).toMatchObject({ phase: "committed", attempt: 2 }); + expect(await input.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, scheduled.id))) + .toMatchObject([{ schemaStatus: "accepted" }]); + expect((await readFile(calls, "utf8")).split("\n").filter(line => line === "process-start")).toHaveLength(1); + expect(await readFile(join(workspace.cwd, "saved.txt"), "utf8")).toBe("work survives a failed provider session\n"); + } finally { + await bundle.transport.close(); runnerPrpWebSocketInternals.resetForTests(); + server.closeAllConnections(); await new Promise(resolveClose => server.close(() => resolveClose())); + } +} diff --git a/server/src/__tests__/helpers/native-drain-controller.test-worker.ts b/server/src/__tests__/helpers/native-drain-controller.test-worker.ts new file mode 100644 index 00000000000..424d7598a24 --- /dev/null +++ b/server/src/__tests__/helpers/native-drain-controller.test-worker.ts @@ -0,0 +1,50 @@ +import { createServer } from "node:http"; +import { closeRegisteredClients, createDb, heartbeatRunEvents } from "@paperclipai/db"; +import { registerLocalRuntimeApiServer } from "../../runtime-api.js"; +import { setStartupRecoveryPhase } from "../../startup-recovery-state.js"; +import { heartbeatService } from "../../services/heartbeat.js"; +import { executePaperclipNativeSession } from "../../services/native-runtime/native-session-executor.js"; + +process.send?.({ ready: true }); +process.once("message", async (input: { + connectionString: string; + options: Omit[0], "db">; + providerEvidence?: boolean; +}) => { + const api = createServer((_request, response) => response.writeHead(404).end()); + registerLocalRuntimeApiServer(api); + let commands = 0; + let errorName: string | undefined; + let errorMessage: string | undefined; + try { + await new Promise(resolve => api.listen(0, "127.0.0.1", resolve)); + const db = createDb(input.connectionString); + try { + await executePaperclipNativeSession({ ...input.options, db, + runnerExecutionTarget: { ...input.options.runnerExecutionTarget!, + runner: { execute: async () => { commands++; throw new Error("Drained controller launched remote work"); } }, + } as never, + onEvent: async event => { + if (event.payload?.span !== "native.coordinator.claim") return; + if (input.providerEvidence) await db.insert(heartbeatRunEvents).values({ + companyId: input.options.execution.binding.companyId, agentId: input.options.execution.binding.agentId, + runId: input.options.execution.binding.runId, seq: 1_000_000, eventType: "turn.started", + sourceEventId: "controlled-provider-evidence" }); + setStartupRecoveryPhase("draining"); + await new Promise((resolve, reject) => api.close(error => error ? reject(error) : resolve())); + }, + }); + } catch (error) { + errorName = error instanceof Error ? error.name : String(error); + errorMessage = error instanceof Error ? error.message : String(error); + } + const heartbeat = heartbeatService(db); + try { + await heartbeat.drainRunningRunsForShutdown("SIGTERM", new Date(), [input.options.execution.binding.runId]); + } finally { await heartbeat.drainActiveRunExecutions(); } + await closeRegisteredClients(input.connectionString, { scope: "database" }); + process.send?.({ done: true, commands, errorName, errorMessage }, () => process.exit(0)); + } catch (error) { + process.send?.({ error: error instanceof Error ? error.stack : String(error) }, () => process.exit(1)); + } +}); diff --git a/server/src/__tests__/instance-settings-cloud-defaults.test.ts b/server/src/__tests__/instance-settings-cloud-defaults.test.ts index 76ff39b0074..6a9a550032f 100644 --- a/server/src/__tests__/instance-settings-cloud-defaults.test.ts +++ b/server/src/__tests__/instance-settings-cloud-defaults.test.ts @@ -17,6 +17,7 @@ function managedConfig(features: ManagedInstanceConfig["features"] = {}): Manage features, plugins: { autoInstall: [] }, environments: [], + pluginEnvironments: [], }; } diff --git a/server/src/__tests__/instance-settings-service.test.ts b/server/src/__tests__/instance-settings-service.test.ts index 6c2847b6eb9..4ced4dcc4c1 100644 --- a/server/src/__tests__/instance-settings-service.test.ts +++ b/server/src/__tests__/instance-settings-service.test.ts @@ -26,7 +26,7 @@ describe("instance settings service", () => { } expect(applyExperimentalSettingsPatch({}, { enableAgentChat: false }).enableAgentChat).toBe(true); const config: ManagedInstanceConfig = { - v: 1, mode: "cloud", catalogVersion: "test", plugins: { autoInstall: [] }, environments: [], + v: 1, mode: "cloud", catalogVersion: "test", plugins: { autoInstall: [] }, environments: [], pluginEnvironments: [], features: { enableAgentChat: false }, }; const result = applyManagedExperimentalOverlay(normalizeExperimentalSettings({ enableAgentChat: false }), config); @@ -54,7 +54,7 @@ describe("instance settings service", () => { { stored: { remoteWorkspaceStorageEnabled: true }, managed: false, expected: false }, ])("respects Cloud enrollment defaults and managed choices: $expected", ({ stored, managed, expected }) => { const config: ManagedInstanceConfig = { - v: 1, mode: "cloud", catalogVersion: "test", plugins: { autoInstall: [] }, environments: [], + v: 1, mode: "cloud", catalogVersion: "test", plugins: { autoInstall: [] }, environments: [], pluginEnvironments: [], features: managed === undefined ? {} : { remoteWorkspaceStorageEnabled: managed }, }; const normalized = normalizeExperimentalSettings(stored); diff --git a/server/src/__tests__/managed-config.test.ts b/server/src/__tests__/managed-config.test.ts index 226ef9bf401..e01bdbb31d0 100644 --- a/server/src/__tests__/managed-config.test.ts +++ b/server/src/__tests__/managed-config.test.ts @@ -1,6 +1,7 @@ import { describe, expect, it } from "vitest"; import { MANAGED_CONFIG_ENV_KEY, + MANAGED_ENVIRONMENT_DRIVERS, getManagedInstanceConfig, managedFeatureKeySet, parseManagedConfigEnv, @@ -110,6 +111,7 @@ describe("parseManagedConfigEnv", () => { features: { enableApps: false, enableStatusCards: true }, plugins: { autoInstall: ["daytona", "kubernetes"] }, environments: [], + pluginEnvironments: [], }); }); @@ -132,6 +134,7 @@ describe("parseManagedConfigEnv", () => { features: {}, plugins: { autoInstall: [] }, environments: [], + pluginEnvironments: [], }); }); @@ -342,12 +345,17 @@ describe("parseManagedConfigEnv environments section", () => { ).toThrow(/"environments\[0\]" must be an object/); }); - it("rejects more than one entry (single managed sandbox slot)", () => { + it("rejects more than one sandbox entry (single managed sandbox slot)", () => { expect(() => parseManagedConfigEnv( envWith(validDoc({ environments: [entry(), entry({ name: "Other", provider: "kubernetes" })] })), ), - ).toThrow(/at most one entry/); + ).toThrow(/at most one sandbox entry/); + expect(() => + parseManagedConfigEnv( + envWith(validDoc({ environments: [entry(), entry({ name: "Other", driver: "sandbox", provider: "kubernetes" })] })), + ), + ).toThrow(/at most one sandbox entry/); }); it("rejects unknown entry keys", () => { @@ -406,6 +414,94 @@ describe("parseManagedConfigEnv environments section", () => { }); }); +describe("parseManagedConfigEnv plugin environment entries", () => { + const PLUGIN_KEY = "example.remote-host-plugin"; + const remoteHost = (overrides: Record = {}) => ({ + name: "Remote Host", + description: "Runs the agent on a remote host", + driver: "plugin", + pluginKey: PLUGIN_KEY, + driverKey: "tasks", + driverConfig: {}, + ...overrides, + }); + const sandbox = { name: "Managed Sandbox", provider: "daytona", config: { target: "us" } }; + // `plugins.autoInstall` holds catalog keys ("remote-host"); the entry's + // `pluginKey` is the plugin's manifest id, which boot maps through the catalog. + const docWith = (environments: unknown[], autoInstall = ["daytona", "remote-host"]) => + envWith(validDoc({ plugins: { autoInstall }, environments })); + + it("lists the drivers an entry may declare", () => { + expect(MANAGED_ENVIRONMENT_DRIVERS).toEqual(["sandbox", "plugin"]); + }); + + it("parses an old document's sandbox entry unchanged", () => { + const config = parseManagedConfigEnv(docWith([sandbox])); + expect(config?.environments).toEqual([{ name: "Managed Sandbox", provider: "daytona", config: { target: "us" } }]); + expect(config?.pluginEnvironments).toEqual([]); + }); + + it("parses the sandbox entry beside a plugin entry", () => { + const config = parseManagedConfigEnv(docWith([sandbox, remoteHost()])); + expect(config?.environments).toEqual([{ name: "Managed Sandbox", provider: "daytona", config: { target: "us" } }]); + expect(config?.pluginEnvironments).toEqual([{ + driver: "plugin", + name: "Remote Host", + description: "Runs the agent on a remote host", + pluginKey: PLUGIN_KEY, + driverKey: "tasks", + driverConfig: {}, + }]); + expect(Object.isFrozen(config?.pluginEnvironments)).toBe(true); + expect(Object.isFrozen(config?.pluginEnvironments[0])).toBe(true); + }); + + it("parses a plugin entry alone, with optional fields omitted", () => { + const config = parseManagedConfigEnv(docWith([{ name: "Remote", driver: "plugin", pluginKey: PLUGIN_KEY, driverKey: "tasks" }])); + expect(config?.environments).toEqual([]); + expect(config?.pluginEnvironments[0]).toMatchObject({ driverConfig: {} }); + expect(config?.pluginEnvironments[0]?.description).toBeUndefined(); + }); + + it("does not compare the manifest id with autoInstall catalog keys (boot checks the catalog)", () => { + expect(parseManagedConfigEnv(docWith([remoteHost()]))?.pluginEnvironments[0]?.pluginKey).toBe(PLUGIN_KEY); + }); + + it("refuses duplicate names, including across drivers", () => { + expect(() => parseManagedConfigEnv(docWith([sandbox, remoteHost({ name: "Managed Sandbox" })]))) + .toThrow(/"environments\[1\].name" duplicates another entry's name/); + }); + + it("refuses a duplicate pluginKey and driverKey pair", () => { + expect(() => parseManagedConfigEnv(docWith([remoteHost(), remoteHost({ name: "Second" })]))) + .toThrow(/duplicates another plugin entry's pluginKey and driverKey/); + }); + + it("refuses secret-like driverConfig keys at any depth", () => { + expect(() => parseManagedConfigEnv(docWith([remoteHost({ driverConfig: { apiToken: "t" } })]))) + .toThrow(/"environments\[0\].driverConfig" key "apiToken" looks secret-bearing/); + expect(() => parseManagedConfigEnv(docWith([remoteHost({ driverConfig: { auth: [{ password: "p" }] } })]))) + .toThrow(/auth\[0\].password/); + }); + + it("refuses unknown keys, including sandbox keys on a plugin entry and the reverse", () => { + expect(() => parseManagedConfigEnv(docWith([remoteHost({ envVars: {} })]))).toThrow(/unknown key "envVars"/); + expect(() => parseManagedConfigEnv(docWith([remoteHost({ provider: "daytona" })]))).toThrow(/unknown key "provider"/); + expect(() => parseManagedConfigEnv(docWith([{ ...sandbox, pluginKey: PLUGIN_KEY }]))).toThrow(/unknown key "pluginKey"/); + // Not supported by this build on either entry. + expect(() => parseManagedConfigEnv(docWith([remoteHost({ instanceDefault: true })]))).toThrow(/unknown key "instanceDefault"/); + expect(() => parseManagedConfigEnv(docWith([{ ...sandbox, instanceDefault: false }]))).toThrow(/unknown key "instanceDefault"/); + }); + + it("refuses an unknown driver and malformed plugin fields", () => { + expect(() => parseManagedConfigEnv(docWith([remoteHost({ driver: "ssh" })]))).toThrow(/"environments\[0\].driver" must be one of "sandbox", "plugin"/); + expect(() => parseManagedConfigEnv(docWith([{ ...sandbox, driver: null }]))).toThrow(/"environments\[0\].driver" must be one of "sandbox", "plugin" when present \(got null\)/); + expect(() => parseManagedConfigEnv(docWith([remoteHost({ driverKey: "Tasks" })]))).toThrow(/"environments\[0\].driverKey" must start/); + expect(() => parseManagedConfigEnv(docWith([remoteHost({ driverKey: undefined })]))).toThrow(/"environments\[0\].driverKey" must be a non-empty string/); + expect(() => parseManagedConfigEnv(docWith([remoteHost({ driverConfig: [] })]))).toThrow(/"environments\[0\].driverConfig" must be an object/); + }); +}); + describe("getManagedInstanceConfig", () => { it("caches by raw env value and reparses when it changes", () => { const raw = validDoc(); diff --git a/server/src/__tests__/managed-plugin-environments.test.ts b/server/src/__tests__/managed-plugin-environments.test.ts index fc8496b236d..4179d238dbf 100644 --- a/server/src/__tests__/managed-plugin-environments.test.ts +++ b/server/src/__tests__/managed-plugin-environments.test.ts @@ -7,15 +7,42 @@ import { companies, createDb, environments, + instanceSettings, plugins, type Db, } from "@paperclipai/db"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { environmentService } from "../services/environments.js"; +import { MANAGED_CONFIG_ENV_KEY, parseManagedConfigEnv } from "../services/managed-config.js"; +import { applyManagedEnvironments } from "../services/managed-environments.js"; const PLUGIN_KEY = "example.remote-host-plugin"; const MARKER = `${PLUGIN_KEY}:tasks`; +type ReadyListener = (payload: { pluginId: string }) => void; + +function managedConfig(entries: unknown[]) { + return parseManagedConfigEnv({ + [MANAGED_CONFIG_ENV_KEY]: JSON.stringify({ + v: 1, + mode: "cloud", + catalogVersion: "2026.1106.0", + features: {}, + plugins: { autoInstall: ["remote-host"] }, + environments: entries, + }), + }); +} + +const remoteHostEntry = { + name: "Remote Host", + description: "Runs the agent on a remote host", + driver: "plugin", + pluginKey: PLUGIN_KEY, + driverKey: "tasks", + driverConfig: {}, +}; + const support = await getEmbeddedPostgresTestSupport(); (support.supported ? describe : describe.skip)("managed plugin environment rows", () => { let database: Awaited>; @@ -24,7 +51,17 @@ const support = await getEmbeddedPostgresTestSupport(); let running: boolean; const call = vi.fn(async (_id: string, method: string, input: { config: Record }) => method === "environmentValidateConfig" ? { ok: true, normalizedConfig: { ...input.config, normalized: true } } : null); - const workerManager = { isRunning: vi.fn(() => running), call }; + let readyListeners: ReadyListener[] = []; + const workerManager = { + isRunning: vi.fn(() => running), + call, + getWorker: vi.fn(() => ({ + on: (_event: "ready", listener: ReadyListener) => { readyListeners.push(listener); }, + off: (_event: "ready", listener: ReadyListener) => { + readyListeners = readyListeners.filter((candidate) => candidate !== listener); + }, + })), + }; beforeAll(async () => { database = await startEmbeddedPostgresTestDatabase("paperclip-managed-plugin-environment-rows-"); @@ -33,9 +70,11 @@ const support = await getEmbeddedPostgresTestSupport(); afterAll(async () => { await database?.cleanup(); }); beforeEach(async () => { running = true; + readyListeners = []; call.mockClear(); await db.delete(activityLog); await db.delete(agents); + await db.delete(instanceSettings); await db.delete(environments); await db.delete(plugins); await db.delete(companies); @@ -175,4 +214,42 @@ const support = await getEmbeddedPostgresTestSupport(); metadata: { managedPluginEnvironment: MARKER }, })).rejects.toThrow(); }); + + describe("boot (applyManagedEnvironments)", () => { + const boot = (entries: unknown[] = [remoteHostEntry]) => + applyManagedEnvironments(db, managedConfig(entries), { env: {}, workerManager, autoInstalledPluginKeys: [PLUGIN_KEY] }); + const instanceDefault = async () => + (await db.select().from(instanceSettings).where(eq(instanceSettings.singletonKey, "default")))[0]?.defaultEnvironmentId ?? null; + + it("creates the row on the first boot and changes nothing on the second", async () => { + const first = await boot(); + expect(first).toMatchObject({ ensured: 1, added: 1, failed: 0 }); + const [row] = await pluginRows(); + expect(row).toMatchObject({ name: "Remote Host", status: "active", metadata: { managedByPaperclip: true, managedPluginEnvironment: MARKER } }); + + const second = await boot(); + expect(second).toMatchObject({ ensured: 1, added: 0, unchanged: 1 }); + const after = await pluginRows(); + expect(after).toHaveLength(1); + expect(after[0]!.updatedAt.getTime()).toBe(row!.updatedAt.getTime()); + expect(await instanceDefault()).toBeNull(); + }); + + it("archives the row while the plugin is down and reactivates it when the worker recovers", async () => { + await boot(); + running = false; + expect(await boot()).toMatchObject({ ensured: 0, failed: 1 }); + expect((await pluginRows())[0]!.status).toBe("archived"); + + running = true; + for (const listener of [...readyListeners]) listener({ pluginId }); + await vi.waitFor(async () => expect((await pluginRows())[0]?.status).toBe("active")); + }); + + it("never archives a row whose entry left the document", async () => { + await boot(); + expect(await boot([])).toBeNull(); + expect((await pluginRows())[0]!.status).toBe("active"); + }); + }); }); diff --git a/server/src/__tests__/status-card-notifications.test.ts b/server/src/__tests__/status-card-notifications.test.ts index 0566ebdd394..7afd070f973 100644 --- a/server/src/__tests__/status-card-notifications.test.ts +++ b/server/src/__tests__/status-card-notifications.test.ts @@ -25,7 +25,7 @@ it.each([false, true])("keeps execution queue consumers intact and emits only a const write = vi.fn(async () => "written"); expect(await writeExecutionControl(owner as never, DELIVERY_QUEUES.workspaceRecovery, write, scope)).toBe("written"); expect(write).toHaveBeenCalledOnce(); - expect(new Set(settled.map(row => row.queue))).toEqual(new Set([DELIVERY_QUEUES.issueMonitors, DELIVERY_QUEUES.workspaceRecovery, + expect(new Set(settled.map(row => row.queue))).toEqual(new Set([DELIVERY_QUEUES.executionReplacement, DELIVERY_QUEUES.issueMonitors, DELIVERY_QUEUES.workspaceRecovery, ...(scoped ? [DELIVERY_QUEUES.statusCardEvaluation] : [])])); } finally { for (const stop of unsubscribe) stop(); } }); diff --git a/server/src/__tests__/task-interruption-query.test.ts b/server/src/__tests__/task-interruption-query.test.ts index 70cb7cd8ec9..df854e62152 100644 --- a/server/src/__tests__/task-interruption-query.test.ts +++ b/server/src/__tests__/task-interruption-query.test.ts @@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto"; import { eq } from "drizzle-orm"; import { createRunSecretRedactionRegistry } from "../services/run-secret-redaction.js"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; -import { agents, agentWakeupRequests, companies, createDb, applyPendingMigrations, heartbeatRuns, issues, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db"; +import { agents, agentWakeupRequests, companies, createDb, applyPendingMigrations, heartbeatRuns, issues, nativeRunFinalizations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db"; import { taskInterruptionPresentation, taskInterruptionPresentations } from "../services/task-interruption-presentation.js"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; @@ -24,6 +24,28 @@ describe.skipIf(!support.supported)("task interruption latest run query", () => }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); afterAll(async () => { await db?.$client.end(); await database?.cleanup(); }); + it("shows a pending native same-run resume instead of a competing Retry", async () => { + const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Resuming task", issuePrefix: `R${companyId.slice(0, 7)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Resuming agent", adapterType: "paperclip_runner" }); + const [issue] = await db.insert(issues).values({ companyId, title: "Resume after attach failure", + status: "in_progress", assigneeAgentId: agentId }).returning(); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue", + status: "failed", runtimeMode: "native", errorCode: "native_session_interrupted", finishedAt: new Date() }); + const resumeAt = new Date(Date.now() + 30_000); + await db.insert(nativeRunFinalizations).values({ companyId, issueId: issue.id, runId, phase: "retryable_failure", + attempt: 1, nextAttemptAt: resumeAt }); + for (const presentation of [(await taskInterruptionPresentations(db, companyId, [issue])).get(issue.id), + await taskInterruptionPresentation(db, issue)]) { + expect(presentation).toMatchObject({ state: "retry_scheduled", canRetry: false, nextRetryAt: resumeAt.toISOString() }); + } + // Exhausted recovery is terminal: exactly one Retry returns. + await db.update(nativeRunFinalizations).set({ phase: "terminal_failure", nextAttemptAt: null }) + .where(eq(nativeRunFinalizations.runId, runId)); + expect((await taskInterruptionPresentations(db, companyId, [issue])).get(issue.id)) + .toMatchObject({ state: "failed", canRetry: true }); + }); + it("redacts workspace preparation errors on task detail and list reads", async () => { vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "a".repeat(64)); try { diff --git a/server/src/__tests__/tool-access-service.test.ts b/server/src/__tests__/tool-access-service.test.ts index 186b20e3042..69525cf08e0 100644 --- a/server/src/__tests__/tool-access-service.test.ts +++ b/server/src/__tests__/tool-access-service.test.ts @@ -68,6 +68,7 @@ import { GOOGLE_WORKSPACE_DEFAULT_SELECTION, getAvailableConnectionMethod, getConnectableAppDefinition, + createToolConnectionSchema, type CreateToolConnection, type GoogleWorkspaceConnectorProfileId, } from "@paperclipai/shared"; @@ -3363,6 +3364,42 @@ describeEmbeddedPostgres("tool access service", () => { .where(eq(companySecrets.companyId, company.id))).toEqual([]); }); + it.each([ + { name: "channel purpose", input: { connectionPurpose: "channel" as const, transport: "local_stdio" as const } }, + { name: "chat transport", input: { transport: "chat_sdk" as const } }, + { name: "voice transport", input: { transport: "voice" as const } }, + ])("rejects a $name before it writes any row", async ({ input }) => { + const company = await createCompany(db); + const service = createTestToolAccessService(db); + + await expect(service.createConnection(company.id, connectionInput({ + name: "Not a tool", + config: { templateId: "paperclip.echo-calculator-time" }, + ...input, + }))).rejects.toMatchObject({ status: 422 }); + + expect(await db.select().from(toolConnections) + .where(eq(toolConnections.companyId, company.id))).toEqual([]); + expect(await db.select().from(toolApplications) + .where(eq(toolApplications.companyId, company.id))).toEqual([]); + }); + + it("stores config as transportConfig when the request leaves transportConfig out", async () => { + const company = await createCompany(db); + const service = createTestToolAccessService(db); + const body = createToolConnectionSchema.parse({ + name: "Local echo config only", + transport: "local_stdio", + config: { templateId: "paperclip.echo-calculator-time" }, + }); + + const connection = await service.createConnection(company.id, body); + + const [stored] = await db.select().from(toolConnections).where(eq(toolConnections.id, connection.id)); + expect(stored?.config).toMatchObject({ templateId: "paperclip.echo-calculator-time" }); + expect(stored?.transportConfig).toEqual(stored?.config); + }); + it("registers an approved local stdio template and exposes its runtime slot", async () => { const company = await createCompany(db); const service = createTestToolAccessService(db); diff --git a/server/src/app.ts b/server/src/app.ts index 75a3ee1c207..5a765ee0c95 100644 --- a/server/src/app.ts +++ b/server/src/app.ts @@ -1348,6 +1348,9 @@ export async function createApp( logger.error({ err }, "Failed to load ready plugins on startup"); }); app.locals.bundledPluginsStartup = trackIdleWork(bundledPluginsStartup); + // Manifest ids the managed auto-install list resolves to; the managed + // environments bootstrap checks plugin entries against them. + app.locals.managedBundledPluginKeys = managedBundledPluginKeys; void bundledPluginsStartup.then(() => { lifecyclePluginsReady = true; resourceLifecycleDelivery.wake(); agentLifecycleDelivery.wake(); }).catch(() => logger.warn("Agent lifecycle recovery failed; retrying.")); // The shutdown hook runs at most once. It caches the in-flight promise, so a // second caller (for example the `exit` handler) awaits the same completion diff --git a/server/src/index.ts b/server/src/index.ts index e36818bfdee..0ba89a9fd11 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -81,7 +81,7 @@ import { embeddedPostgresOwnerPort } from "./embedded-postgres-owner.js"; import { deliverExecutionStatuses } from "./services/execution-status-delivery.js"; import { deliverReconciledExecutions, settleUnrecoverableExecutions } from "./services/execution-recovery-resolution.js"; import { deliverPendingPauseResumes } from "./services/issue-tree-resume.js"; -import { reconcileSafeNativeReplacements } from "./services/native-runtime/native-safe-replacement.js"; +import { invalidateNativeReplacementDiscovery, reconcileSafeNativeReplacements } from "./services/native-runtime/native-safe-replacement.js"; import { reconcileAbandonedExecutionControl } from "./services/execution-control-reconciliation.js"; import { acknowledgeNativeStartupCleanup } from "./services/native-runtime/native-startup-cleanup.js"; import { executionControlSchedule, registerExecutionControlWork } from "./services/execution-control-work.js"; @@ -1175,8 +1175,8 @@ async function startServerWithDatabaseTeardown( throw err; } - // Ensure sandbox environments declared in the managed-config document - // (`environments` section) before the heartbeat resumes queued runs. The + // Ensure the sandbox and plugin environments declared in the managed-config + // document (`environments` section) before the heartbeat resumes queued runs. The // document already parsed fail-closed above; the ensure step itself is // fail-safe per entry (a degraded boot beats a fleet-wide crash loop), but // a contradictory deployment that also forces PAPERCLIP_EXECUTION_MODE @@ -1186,14 +1186,16 @@ async function startServerWithDatabaseTeardown( // additionally gates each entry on a live plugin worker (and archives the // row of a provider that did not come up). try { - const bundledPluginsStartup = (app as { locals?: { bundledPluginsStartup?: Promise } }) - .locals?.bundledPluginsStartup; + const appLocals = (app as { + locals?: { bundledPluginsStartup?: Promise; managedBundledPluginKeys?: readonly string[] }; + }).locals; const managedEnvironmentsResult = await applyManagedEnvironments(db as any, managedConfig, { - pluginsReady: bundledPluginsStartup, + pluginsReady: appLocals?.bundledPluginsStartup, + autoInstalledPluginKeys: appLocals?.managedBundledPluginKeys, workerManager: pluginWorkerManager, }); if (managedEnvironmentsResult) { - logger.warn(managedEnvironmentsResult, "managed sandbox environments ensured from managed config"); + logger.warn(managedEnvironmentsResult, "managed environments ensured from managed config"); } } catch (err) { logger.error({ err }, "failed to apply managed environments from managed config"); @@ -1342,7 +1344,7 @@ async function startServerWithDatabaseTeardown( { queue: DELIVERY_QUEUES.executionStatus, run: () => deliverExecutionStatuses(db), next: executionSchedule.status }, { queue: DELIVERY_QUEUES.executionDisposition, run: () => settleUnrecoverableExecutions(db), next: executionSchedule.disposition }, ...(heartbeat ? [ - { queue: DELIVERY_QUEUES.executionReplacement, run: () => reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession: run => verifyStoppedNativeSessionForReplacement(db, run) }), next: executionSchedule.replacement }, + { queue: DELIVERY_QUEUES.executionReplacement, run: () => reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession: run => verifyStoppedNativeSessionForReplacement(db, run) }), next: executionSchedule.replacement, onSettled: () => invalidateNativeReplacementDiscovery(db), settleEmptyWhilePaused: true }, { queue: DELIVERY_QUEUES.executionContinuation, run: async () => { await deliverReconciledExecutions(db, heartbeat.wakeup); await deliverPendingPauseResumes(db, heartbeat.wakeup); diff --git a/server/src/routes/environments.ts b/server/src/routes/environments.ts index 9a33f1b66f9..09b2023b7e7 100644 --- a/server/src/routes/environments.ts +++ b/server/src/routes/environments.ts @@ -19,7 +19,10 @@ import { import { conflict, forbidden, unprocessable } from "../errors.js"; import { isCloudManagedInstance } from "../services/cloud-instance.js"; import { getManagedInstanceConfig, SECRET_LIKE_CONFIG_KEY_PATTERN } from "../services/managed-config.js"; -import { MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY } from "../services/managed-plugin-environments.js"; +import { + MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY, + managedPluginEnvironmentMarker, +} from "../services/managed-plugin-environments.js"; import { parseExecutionPolicyBootstrapEnv } from "../services/execution-policy-bootstrap.js"; import { isExecutionForcedToKubernetes } from "../services/execution-allowlist.js"; import { validate } from "../middleware/validate.js"; @@ -182,6 +185,20 @@ function isManagedSandboxProvisioningConfigured(): boolean { } } +/** + * Whether the managed-config document declares the plugin entry this row's + * marker names. Fails closed like `isManagedSandboxProvisioningConfigured`. + */ +function isManagedPluginEnvironmentDeclared(marker: string): boolean { + try { + return (getManagedInstanceConfig()?.pluginEnvironments ?? []).some( + (spec) => managedPluginEnvironmentMarker(spec) === marker, + ); + } catch { + return true; + } +} + /** * Whether this row occupies a provisioner-owned environment slot whose * platform markers are LIVE state — i.e. some platform code path converges @@ -196,6 +213,9 @@ function isManagedSandboxProvisioningConfigured(): boolean { * owned only while a managed-sandbox bootstrap path is configured — * `ensureManagedSandboxEnvironment` then adopts and refreshes whichever * row holds the slot on every boot. + * - The row of a managed-config plugin entry (its + * `managedPluginEnvironment` marker) while the document declares that + * entry: `applyManagedEnvironments` converges the markers on it every boot. * - Any sandbox row bearing the legacy kubernetes marker while the * persisted instance execution policy forces kubernetes: * `findKubernetesEnvironment` selects marked rows (newest first) for @@ -214,6 +234,10 @@ async function isPlatformSlotEnvironment( options: { isForcedKubernetesExecution: () => Promise }, ): Promise { if (environment.driver === "local") return true; + if (environment.driver === "plugin") { + const pluginMarker = environment.metadata?.[MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY]; + return typeof pluginMarker === "string" && isManagedPluginEnvironmentDeclared(pluginMarker); + } if (environment.driver !== "sandbox") return false; if ( environment.metadata?.managedByPaperclip === true && diff --git a/server/src/services/delivery-work-coordinator.ts b/server/src/services/delivery-work-coordinator.ts index 39df770662d..97dc7a8ea87 100644 --- a/server/src/services/delivery-work-coordinator.ts +++ b/server/src/services/delivery-work-coordinator.ts @@ -25,7 +25,7 @@ export function createDeliveryWorkCoordinator(input: { // do not postpone the obligation indefinitely. coalesceMs?: number; run: (signal: AbortSignal) => Promise; - hasPending: () => Promise; + hasPending: (runAllowed?: boolean) => Promise; // Scoped hints belong to the outer transaction, including uncertainty recovery. onSettled?: (hints?: ReadonlySet) => void; // Optional maintenance/retry deadline restored from durable state. @@ -75,7 +75,7 @@ export function createDeliveryWorkCoordinator(input: { let runFailed = false; controller = attempt; running = withDatabaseOperation(`delivery.${queue}`, () => Promise.resolve().then(() => reconcileDatabaseWork(input.owner, queue)) - .then(() => workerStopped || !(input.canRun() || (task.runDuringDrain && input.canReconcile?.())) ? undefined : task.run(attempt.signal).catch(error => { runFailed = true; throw error; })).then(() => workerStopped ? false : task.hasPending()).then(pending => { + .then(() => workerStopped || !(input.canRun() || (task.runDuringDrain && input.canReconcile?.())) ? undefined : task.run(attempt.signal).catch(error => { runFailed = true; throw error; })).then(() => workerStopped ? false : task.hasPending(input.canRun() || Boolean(task.runDuringDrain && input.canReconcile?.()))).then(pending => { pendingWork = pending || databaseWorkPending(input.owner, queue); if (pendingWork) schedule(task.retryMs); else { finishIntent?.(); finishIntent = undefined; } diff --git a/server/src/services/environment-cleanup-notifications.ts b/server/src/services/environment-cleanup-notifications.ts index fbbf3352359..6beca498981 100644 --- a/server/src/services/environment-cleanup-notifications.ts +++ b/server/src/services/environment-cleanup-notifications.ts @@ -31,7 +31,10 @@ export async function notifyEnvironmentCleanup(transaction: object, scope?: Envi await notifyNativeEnvironmentSettlement(transaction, scope); await notifyQueuedRunDispatch(transaction, scope); // A settled run lease can make that run's abandoned native owner releasable. - if (scope?.runId) await notifyDeliveryWork(transaction, DELIVERY_QUEUES.executionFinalization); + if (scope?.runId) { + await notifyDeliveryWork(transaction, DELIVERY_QUEUES.executionFinalization); + await notifyDeliveryWork(transaction, DELIVERY_QUEUES.executionReplacement); + } await notifyDeliveryWork(transaction, DELIVERY_QUEUES.environmentCleanup); await notifyDeliveryWork(transaction, DELIVERY_QUEUES.historicalCleanup); } diff --git a/server/src/services/execution-control-work.ts b/server/src/services/execution-control-work.ts index 7cedc0c2ea3..5a0e24f24e0 100644 --- a/server/src/services/execution-control-work.ts +++ b/server/src/services/execution-control-work.ts @@ -5,7 +5,7 @@ import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator. import { type DeliveryQueue } from "./delivery-work-notifications.js"; import { EXECUTION_RECONCILIATION_INTERVAL_MS } from "./execution-control-deadline.js"; import { obsoleteExecutionConversationHold, unresolvedExecutionDispositions } from "./execution-recovery-resolution.js"; -import { nativeReplacementPending } from "./native-runtime/native-safe-replacement.js"; +import { hasPendingNativeReplacements } from "./native-runtime/native-safe-replacement.js"; import { nextAbandonedNativeOwnerReleaseAt } from "./native-runtime/native-failed-run-owner-release.js"; import { hasPendingPauseResume } from "./issue-tree-resume.js"; @@ -16,6 +16,7 @@ export function registerExecutionControlWork( run: () => Promise; next: () => Promise; onSettled?: () => void; + settleEmptyWhilePaused?: boolean; }>, canRun: () => boolean = () => true, ) { @@ -24,6 +25,7 @@ export function registerExecutionControlWork( let reconciled = 0; let next: number | null = null; let earliestRunAt = 0; + let pausedGeneration = 0, pausedPending = true; const worker = coordinator.register(duty.queue, { retryMs: EXECUTION_RECONCILIATION_INTERVAL_MS, pausePassiveMaintenance: true, @@ -42,7 +44,18 @@ export function registerExecutionControlWork( next = deadline === null ? null : Math.max(deadline, earliestRunAt); reconciled = generation; }, - hasPending: async () => reconciled !== requested, + hasPending: async (runAllowed = true) => { + if (reconciled === requested) return false; + if (runAllowed || !duty.settleEmptyWhilePaused || !canRun() || reconciled === 0) return true; + if (pausedGeneration !== requested) { + const generation = requested; + const deadline = await duty.next(); + if (generation !== requested) return true; + pausedGeneration = generation; pausedPending = deadline !== null; next = deadline; + if (!pausedPending) reconciled = generation; + } + return pausedPending; + }, nextRunAt: () => next, }); return { ...worker, wake: () => { requested += 1; worker.wake(); } }; @@ -74,10 +87,7 @@ export function executionControlSchedule(db: Db) { return at === null ? null : deadline([new Date(at)]); }, async replacement() { - const rows = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns) - .innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)) - .where(nativeReplacementPending()).limit(1); - return rows.length ? retry() : null; + return await hasPendingNativeReplacements(db) ? retry() : null; }, async continuation() { const rows = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and( diff --git a/server/src/services/explicit-native-continuation.test.ts b/server/src/services/explicit-native-continuation.test.ts index 0a9ed4725cc..756410efc6a 100644 --- a/server/src/services/explicit-native-continuation.test.ts +++ b/server/src/services/explicit-native-continuation.test.ts @@ -1586,7 +1586,7 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes await persistHeartbeatRunProcessMetadata(db, source.id, { pid: 999999999, processGroupId: null, startedAt: new Date().toISOString() }); await db.update(heartbeatRuns).set({ processPid: null }).where(eq(heartbeatRuns.id, source.id)); expect(await admit(f, true)).toBeNull(); - expect(await recordNativeLocalProcessStop(db, source)).toBe(true); + expect(await db.transaction(tx => recordNativeLocalProcessStop(tx as unknown as typeof db, source))).toBe(true); await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: source.id, agentId: f.agentId, eventType: PROCESS_START_REQUESTED }); // No PID was stored for the new launch, as when the server dies after spawn. @@ -2440,7 +2440,7 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes await db.update(issues).set({ conversationAgentId: f.agentId, conversationUserId: "board", conversationState: "active", conversationBoundaryCommentId: boundaryId }).where(eq(issues.id, f.issueId)); await db.update(issueRecoveryActions).set({ createdAt: new Date("2026-09-11T09:00:00Z") }).where(eq(issueRecoveryActions.sourceIssueId, f.issueId)); const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId)); - expect(await recordNativeLocalProcessStop(db, source)).toBe(true); + expect(await db.transaction(tx => recordNativeLocalProcessStop(tx as unknown as typeof db, source))).toBe(true); await db.update(heartbeatRuns).set({ processPid: null }).where(eq(heartbeatRuns.id, f.sourceRunId)); if (scenario === "after_reset_workspace_hold") { await db.update(heartbeatRuns).set({ processPid: 999999999, resultJson: { workspaceRestoreFailure: "restore_unsafe_archive" } }).where(eq(heartbeatRuns.id, f.sourceRunId)); diff --git a/server/src/services/heartbeat/deferred-receipt-work.ts b/server/src/services/heartbeat/deferred-receipt-work.ts index b799f10b0a7..915d72b7b07 100644 --- a/server/src/services/heartbeat/deferred-receipt-work.ts +++ b/server/src/services/heartbeat/deferred-receipt-work.ts @@ -5,6 +5,7 @@ import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; import type { DeferredExecutionScope, DeferredReceiptRecoveryOptions } from "../deferred-execution-notifications.js"; import { registerScopedDeadlineWork } from "../scoped-deadline-work.js"; +import { issueBlockersResolvedWakePayloadSql } from "../issue-dependency-wakeups.js"; type Entry = { id: string; companyId: string; dueAt: number; immediate: boolean }; type NativeIntentOptions = DeferredExecutionScope & { @@ -74,6 +75,7 @@ export function registerDeferredReceiptWork( sql`jsonb_typeof(${agentWakeupRequests.payload}#>'{_paperclipWakeContext,wakeCommentIds}') = 'array' and ${agentWakeupRequests.payload}#>'{_paperclipWakeContext,wakeCommentIds}' <> '[]'::jsonb`, sql`${agentWakeupRequests.payload}->>'mutation' = 'interaction'`, + issueBlockersResolvedWakePayloadSql, sql`exists (select 1 from heartbeat_runs r where r.company_id = ${agentWakeupRequests.companyId} and r.runtime_mode = 'legacy' and r.status = 'cancelled' and r.result_json->>'queuedCommentInterruptQueueId' = ${agentWakeupRequests.id}::text)`, diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts index 4da678e4b38..514d0a005b7 100644 --- a/server/src/services/heartbeat/queue.ts +++ b/server/src/services/heartbeat/queue.ts @@ -144,7 +144,11 @@ import { isThrottleCandidateIssueRewake, } from "../issue-rewake-throttle.js"; import { logActivity } from "../activity-log.js"; -import { ISSUE_BLOCKERS_RESOLVED_WAKE_REASON } from "../issue-dependency-wakeups.js"; +import { + ISSUE_BLOCKERS_RESOLVED_WAKE_REASON, + isIssueBlockersResolvedWakePayload, + issueBlockersResolvedWakePayloadSql, +} from "../issue-dependency-wakeups.js"; import { resumeIfPauseAlreadyReleased } from "../issue-tree-resume.js"; import { ISSUE_PAUSED_ERROR_CODE, @@ -1757,7 +1761,8 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende or(and( sql`jsonb_typeof(${agentWakeupRequests.payload} #> '{_paperclipWakeContext,wakeCommentIds}') = 'array'`, sql`${agentWakeupRequests.payload} #> '{_paperclipWakeContext,wakeCommentIds}' <> '[]'::jsonb`, - ), sql`${agentWakeupRequests.payload}->>'mutation' = 'interaction'`), + ), sql`${agentWakeupRequests.payload}->>'mutation' = 'interaction'`, + issueBlockersResolvedWakePayloadSql), sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt' is null`, sql`NOT EXISTS (SELECT 1 FROM heartbeat_runs live_issue_run WHERE live_issue_run.company_id = ${issues.companyId} @@ -1768,7 +1773,8 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende .orderBy(asc(agentWakeupRequests.updatedAt)).limit(50); const releasedNativeSources = new Set(); for (const { wake } of strandedQueues) { - if (!queuedCommentIdsFromWakePayload(wake.payload).length && + const dependencyWake = isIssueBlockersResolvedWakePayload(wake.payload); + if (!dependencyWake && !queuedCommentIdsFromWakePayload(wake.payload).length && !await readQueuedInteractionResponse(db, wake.companyId, String(wake.payload?.issueId), wake.payload)) continue; const [latest] = await db.select({ id: heartbeatRuns.id, diff --git a/server/src/services/issue-dependency-wakeups.ts b/server/src/services/issue-dependency-wakeups.ts index ee09f3e55ff..4d891d583a8 100644 --- a/server/src/services/issue-dependency-wakeups.ts +++ b/server/src/services/issue-dependency-wakeups.ts @@ -1,10 +1,19 @@ -import { and, eq, inArray } from "drizzle-orm"; +import { and, eq, inArray, sql } from "drizzle-orm"; import { createHash } from "node:crypto"; import type { Db } from "@paperclipai/db"; import { agentWakeupRequests } from "@paperclipai/db"; export const ISSUE_BLOCKERS_RESOLVED_WAKE_REASON = "issue_blockers_resolved"; +export const issueBlockersResolvedWakePayloadSql = + sql`${agentWakeupRequests.payload}#>>'{_paperclipWakeContext,wakeReason}' = ${ISSUE_BLOCKERS_RESOLVED_WAKE_REASON}`; + +export function isIssueBlockersResolvedWakePayload(payload: Record | null | undefined): boolean { + const context = payload?._paperclipWakeContext; + return context !== null && typeof context === "object" && !Array.isArray(context) + && (context as Record).wakeReason === ISSUE_BLOCKERS_RESOLVED_WAKE_REASON; +} + /** * Whether moving an issue into `blocked` should immediately wake its assignee * because every recorded blocker is already resolved ("blockers restored"). diff --git a/server/src/services/managed-config.ts b/server/src/services/managed-config.ts index 0bdcb72a304..bea4e28ab6f 100644 --- a/server/src/services/managed-config.ts +++ b/server/src/services/managed-config.ts @@ -9,12 +9,23 @@ * "mode": "cloud", * "catalogVersion": "2026.720.0", * "features": { "": true | false, ... }, - * "plugins": { "autoInstall": ["daytona", "kubernetes"] }, + * "plugins": { "autoInstall": ["daytona", "remote-host"] }, * "environments": [ - * { "name": "Daytona", "provider": "daytona", "config": { "target": "us" } } + * { "name": "Daytona", "provider": "daytona", "config": { "target": "us" } }, + * { + * "name": "Remote Host", + * "driver": "plugin", + * "pluginKey": "example.remote-host-plugin", + * "driverKey": "tasks", + * "driverConfig": {} + * } * ] * } * + * An `environments` entry without `driver` is the single sandbox entry; any + * number of `driver: "plugin"` entries may sit beside it (see + * `MANAGED_ENVIRONMENT_DRIVERS`). + * * Parsing follows the `execution-policy-bootstrap.ts` doctrine: a pure * function over `Record`, strict, and fail * closed — bad JSON, an unsupported `v`, an unknown feature key, a feature @@ -54,8 +65,47 @@ export interface ManagedInstanceConfig { readonly workerEnv?: Readonly>; readonly enabledCompanies?: Readonly>; }; - /** Sandbox environments the control plane provisions at boot (empty when the section is absent). */ + /** + * The sandbox entry of the `environments` section: zero or one, since it + * provisions the single Paperclip-managed sandbox slot (empty when the + * section is absent or declares only plugin entries). + */ environments: readonly ManagedEnvironmentSpec[]; + /** The `driver: "plugin"` entries of the `environments` section (empty when none are declared). */ + pluginEnvironments: readonly ManagedPluginEnvironmentSpec[]; +} + +/** + * The environment drivers a managed-config `environments` entry may declare. + * An entry without `driver` is the sandbox entry, so documents written + * before plugin entries existed parse unchanged. + */ +export const MANAGED_ENVIRONMENT_DRIVERS = ["sandbox", "plugin"] as const; +export type ManagedEnvironmentDriver = (typeof MANAGED_ENVIRONMENT_DRIVERS)[number]; + +/** + * One declared managed plugin environment: an instance-level `plugin`-driver + * environment row served by an auto-installed plugin's environment driver. + * The row is + * identified by its `pluginKey`–`driverKey` pair, which boot stamps on the + * row as `metadata.managedPluginEnvironment`. + */ +export interface ManagedPluginEnvironmentSpec { + driver: "plugin"; + /** Display name of the instance-level environment row (unique per instance). */ + name: string; + description?: string; + /** + * The serving plugin's manifest id (for example + * `example.remote-host-plugin`), not its `plugins.autoInstall` catalog key. + * Boot requires a catalog entry + * whose `key` is in `plugins.autoInstall` to install this id. + */ + pluginKey: string; + /** The plugin's environment driver key. */ + driverKey: string; + /** Driver config, validated by the plugin at ensure time; never carries secrets. */ + driverConfig: Readonly>; } /** @@ -142,6 +192,155 @@ function describeJsonValue(value: unknown): string { return `${JSON.stringify(value)}`; } +const SANDBOX_ENTRY_KEYS = ["name", "description", "driver", "provider", "config"] as const; +const PLUGIN_ENTRY_KEYS = [ + "name", + "description", + "driver", + "pluginKey", + "driverKey", + "driverConfig", +] as const; +/** Same rule as the plugin environment config schema in `environment-config.ts`. */ +const PLUGIN_DRIVER_KEY_PATTERN = /^[a-z0-9][a-z0-9._-]*$/; + +function requireTrimmedString(value: unknown, path: string): string { + if (typeof value !== "string" || value.length === 0 || value.trim() !== value) { + fail(`"${path}" must be a non-empty string without surrounding whitespace (got ${describeJsonValue(value)})`); + } + return value; +} + +function readOptionalDescription(value: unknown, path: string): string | undefined { + if (value === undefined) return undefined; + if (typeof value !== "string" || value.trim().length === 0) { + fail(`"${path}" must be a non-empty string when present (got ${describeJsonValue(value)})`); + } + return value; +} + +function readSecretFreeObject(value: unknown, path: string): Record { + if (value === undefined) return {}; + if (!isPlainObject(value)) { + fail(`"${path}" must be an object (got ${describeJsonValue(value)})`); + } + const secretLikeKey = findSecretLikeConfigKey(value, ""); + if (secretLikeKey) { + fail( + `"${path}" key "${secretLikeKey}" looks secret-bearing; credentials are delivered to managed instances as process environment variables (the provider's documented env fallback), never in the managed-config document`, + ); + } + return { ...value }; +} + +/** + * Parse the `environments` section: at most one sandbox entry (no `driver`, + * or `driver: "sandbox"`) plus any number of `driver: "plugin"` entries. + * Names are unique across the section, and plugin entries are unique by + * their `pluginKey`–`driverKey` pair. + */ +function parseManagedEnvironmentsSection( + raw: unknown, + autoInstall: readonly string[], +): { sandbox: ManagedEnvironmentSpec[]; plugin: ManagedPluginEnvironmentSpec[] } { + const sandbox: ManagedEnvironmentSpec[] = []; + const plugin: ManagedPluginEnvironmentSpec[] = []; + if (raw === undefined) return { sandbox, plugin }; + if (!Array.isArray(raw)) { + fail(`"environments" must be an array of environment objects (got ${describeJsonValue(raw)})`); + } + const names = new Set(); + const pluginDrivers = new Set(); + for (const [index, entry] of raw.entries()) { + const at = `environments[${index}]`; + if (!isPlainObject(entry)) { + fail(`"${at}" must be an object (got ${describeJsonValue(entry)})`); + } + // Only an absent `driver` means the sandbox entry; an explicit `null` is malformed. + const driver = entry.driver === undefined ? "sandbox" : entry.driver; + if (!(MANAGED_ENVIRONMENT_DRIVERS as readonly unknown[]).includes(driver)) { + fail( + `"${at}.driver" must be one of ${MANAGED_ENVIRONMENT_DRIVERS.map((value) => JSON.stringify(value)).join(", ")} when present (got ${describeJsonValue(entry.driver)})`, + ); + } + const allowedKeys: readonly string[] = driver === "plugin" ? PLUGIN_ENTRY_KEYS : SANDBOX_ENTRY_KEYS; + for (const key of Object.keys(entry)) { + if (!allowedKeys.includes(key)) { + fail(`"${at}" has unknown key "${key}" (allowed for a ${driver} entry: ${allowedKeys.join(", ")})`); + } + } + const name = requireTrimmedString(entry.name, `${at}.name`); + if (names.has(name)) { + fail(`"${at}.name" duplicates another entry's name "${name}"; environment names are unique per instance`); + } + names.add(name); + const description = readOptionalDescription(entry.description, `${at}.description`); + + if (driver === "sandbox") { + // The DB enforces at most ONE Paperclip-managed sandbox row per instance + // (partial unique index `environments_managed_sandbox_idx`); every + // sandbox entry provisions that row, so a second one can never be + // satisfied. + if (sandbox.length > 0) { + fail( + `"environments" supports at most one sandbox entry: each sandbox entry provisions the single Paperclip-managed sandbox environment (DB invariant environments_managed_sandbox_idx)`, + ); + } + const provider = requireTrimmedString(entry.provider, `${at}.provider`); + // Coherence: on a managed instance the control plane is the only plugin + // install path, so an environment whose provider plugin is not + // auto-installed could never serve a lease. Catch the skew at parse time. + if (!autoInstall.includes(provider)) { + fail( + `"${at}.provider" is "${provider}", which is not in "plugins.autoInstall"; a managed environment requires its provider plugin to be provisioned`, + ); + } + if (isPlainObject(entry.config) && entry.config.provider !== undefined) { + fail(`"${at}.config" must not set "provider"; it is forced from the entry's provider key`); + } + const config = readSecretFreeObject(entry.config, `${at}.config`); + sandbox.push( + Object.freeze({ + name, + ...(description !== undefined ? { description } : {}), + provider, + config: Object.freeze(config), + }) as ManagedEnvironmentSpec, + ); + continue; + } + + // `pluginKey` is the plugin's manifest id (what the row's config stores), + // not a `plugins.autoInstall` catalog key, so this pure parser cannot map + // it. `applyManagedEnvironments` refuses startup when no auto-installed + // catalog entry installs this manifest id. + const pluginKey = requireTrimmedString(entry.pluginKey, `${at}.pluginKey`); + const driverKey = requireTrimmedString(entry.driverKey, `${at}.driverKey`); + if (!PLUGIN_DRIVER_KEY_PATTERN.test(driverKey)) { + fail( + `"${at}.driverKey" must start with a lowercase letter or digit and contain only lowercase letters, digits, dots, hyphens, or underscores (got ${describeJsonValue(driverKey)})`, + ); + } + const pair = `${pluginKey}:${driverKey}`; + if (pluginDrivers.has(pair)) { + fail(`"${at}" duplicates another plugin entry's pluginKey and driverKey ("${pair}"); each plugin environment driver is declared once`); + } + pluginDrivers.add(pair); + const driverConfig = readSecretFreeObject(entry.driverConfig, `${at}.driverConfig`); + plugin.push( + Object.freeze({ + driver: "plugin", + name, + ...(description !== undefined ? { description } : {}), + pluginKey, + driverKey, + driverConfig: Object.freeze(driverConfig), + }) as ManagedPluginEnvironmentSpec, + ); + } + return { sandbox, plugin }; +} + /** * Parse `PAPERCLIP_MANAGED_CONFIG` from a raw env map. Returns null only when * the variable is absent (self-hosted). Throws with a precise error on a @@ -320,79 +519,10 @@ export function parseManagedConfigEnv(env: ManagedConfigEnv): ManagedInstanceCon // absence is not a dropped security control — it simply declares no managed // environments. When present, the section is validated fail-closed like // everything else. - const environmentSpecs: ManagedEnvironmentSpec[] = []; - if (doc.environments !== undefined) { - if (!Array.isArray(doc.environments)) { - fail(`"environments" must be an array of environment objects (got ${describeJsonValue(doc.environments)})`); - } - // The DB enforces at most ONE Paperclip-managed sandbox row per instance - // (partial unique index `environments_managed_sandbox_idx`); every entry - // here provisions that row, so a longer list can never be satisfied. - if (doc.environments.length > 1) { - fail( - `"environments" supports at most one entry: each entry provisions the single Paperclip-managed sandbox environment (DB invariant environments_managed_sandbox_idx)`, - ); - } - for (const [index, entry] of doc.environments.entries()) { - if (!isPlainObject(entry)) { - fail(`"environments[${index}]" must be an object (got ${describeJsonValue(entry)})`); - } - const allowedEntryKeys = new Set(["name", "description", "provider", "config"]); - for (const key of Object.keys(entry)) { - if (!allowedEntryKeys.has(key)) { - fail(`"environments[${index}]" has unknown key "${key}" (allowed: name, description, provider, config)`); - } - } - if (typeof entry.name !== "string" || entry.name.length === 0 || entry.name.trim() !== entry.name) { - fail( - `"environments[${index}].name" must be a non-empty string without surrounding whitespace (got ${describeJsonValue(entry.name)})`, - ); - } - if ( - entry.description !== undefined - && (typeof entry.description !== "string" || entry.description.trim().length === 0) - ) { - fail(`"environments[${index}].description" must be a non-empty string when present (got ${describeJsonValue(entry.description)})`); - } - if (typeof entry.provider !== "string" || entry.provider.length === 0 || entry.provider.trim() !== entry.provider) { - fail( - `"environments[${index}].provider" must be a non-empty string without surrounding whitespace (got ${describeJsonValue(entry.provider)})`, - ); - } - // Coherence: on a managed instance the control plane is the only plugin - // install path, so an environment whose provider plugin is not - // auto-installed could never serve a lease. Catch the skew at parse time. - if (!autoInstall.includes(entry.provider)) { - fail( - `"environments[${index}].provider" is "${entry.provider}", which is not in "plugins.autoInstall"; a managed environment requires its provider plugin to be provisioned`, - ); - } - const config: Record = {}; - if (entry.config !== undefined) { - if (!isPlainObject(entry.config)) { - fail(`"environments[${index}].config" must be an object (got ${describeJsonValue(entry.config)})`); - } - if (entry.config.provider !== undefined) { - fail(`"environments[${index}].config" must not set "provider"; it is forced from the entry's provider key`); - } - const secretLikeKey = findSecretLikeConfigKey(entry.config, ""); - if (secretLikeKey) { - fail( - `"environments[${index}].config" key "${secretLikeKey}" looks secret-bearing; credentials are delivered to managed instances as process environment variables (the provider's documented env fallback), never in the managed-config document`, - ); - } - Object.assign(config, entry.config); - } - environmentSpecs.push( - Object.freeze({ - name: entry.name, - ...(entry.description !== undefined ? { description: entry.description } : {}), - provider: entry.provider, - config: Object.freeze(config), - }) as ManagedEnvironmentSpec, - ); - } - } + const { sandbox: environmentSpecs, plugin: pluginEnvironmentSpecs } = parseManagedEnvironmentsSection( + doc.environments, + autoInstall, + ); return Object.freeze({ v: SUPPORTED_MANAGED_CONFIG_VERSION, @@ -405,6 +535,7 @@ export function parseManagedConfigEnv(env: ManagedConfigEnv): ManagedInstanceCon ...(enabledCompanies ? { enabledCompanies: Object.freeze(enabledCompanies) } : {}), }), environments: Object.freeze(environmentSpecs), + pluginEnvironments: Object.freeze(pluginEnvironmentSpecs), }) as ManagedInstanceConfig; } diff --git a/server/src/services/managed-environments-plugin-entries.test.ts b/server/src/services/managed-environments-plugin-entries.test.ts new file mode 100644 index 00000000000..e9241b6728f --- /dev/null +++ b/server/src/services/managed-environments-plugin-entries.test.ts @@ -0,0 +1,385 @@ +import { describe, expect, it, vi } from "vitest"; +import { unprocessable } from "../errors.js"; +import { applyManagedEnvironments } from "./managed-environments.js"; +import { + PLUGIN_KEY, + environmentsSeam, + instanceSettingsSeam, + noDb, + parsedConfig, + pluginEnvironmentRow, + readyDriverResolver, + reconciliationResult, + recoveryHandle, + tick, + type InstanceSettingsSeam, + type RecoveryHandle, +} from "./managed-environments.test-support.js"; + +describe("applyManagedEnvironments plugin entries", () => { + const pluginEntry = (overrides: Record = {}) => ({ + name: "Remote Host", + description: "Runs the agent on a remote host", + driver: "plugin", + pluginKey: PLUGIN_KEY, + driverKey: "tasks", + driverConfig: {}, + ...overrides, + }); + const configWith = (environments: unknown[], features: Record = {}) => parsedConfig({ + plugins: { autoInstall: ["daytona", "remote-host"] }, + features, + environments, + }); + const readyPlugin = (status = "ready", declaresDriver = true) => vi.fn(async () => ({ + plugin: { id: "remote-host-plugin", pluginKey: PLUGIN_KEY, status }, + declaresDriver, + })); + const workerManager = (running = true, handle?: RecoveryHandle) => ({ + isRunning: vi.fn(() => running), + call: vi.fn(), + getWorker: vi.fn(() => handle), + }); + + it("ensures a plugin entry with the plugin validator's worker manager", async () => { + const environments = environmentsSeam(); + const result = await applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: {}, + workerManager: workerManager(), + environments, + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + + expect(environments.ensureManagedPluginEnvironment).toHaveBeenCalledWith({ + name: "Remote Host", + description: "Runs the agent on a remote host", + pluginKey: PLUGIN_KEY, + driverKey: "tasks", + driverConfig: {}, + workerManager: { isRunning: expect.any(Function), call: expect.any(Function) }, + }); + expect(result).toMatchObject({ ensured: 1, added: 1, failed: 0 }); + expect(result?.pluginOutcomes).toEqual([{ + environmentId: "plugin-env", + name: "Remote Host", + pluginKey: PLUGIN_KEY, + driverKey: "tasks", + action: "added", + }]); + expect(environments.ensureManagedSandboxEnvironment).not.toHaveBeenCalled(); + }); + + it("runs the sandbox entry first", async () => { + const order: string[] = []; + const environments = environmentsSeam({ + ensureManagedSandboxEnvironment: vi.fn(async () => { + order.push("sandbox"); + return reconciliationResult() as never; + }), + ensureManagedPluginEnvironment: vi.fn(async () => { + order.push("plugin"); + return { environment: pluginEnvironmentRow(), action: "added" as const }; + }), + }); + await applyManagedEnvironments(noDb, configWith([{ name: "Daytona", provider: "daytona" }, pluginEntry()]), { + env: {}, + workerManager: workerManager(), + environments, + instanceSettings: instanceSettingsSeam(), + resolveSandboxProviderDriver: readyDriverResolver(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + expect(order).toEqual(["sandbox", "plugin"]); + }); + + it("refuses startup when no auto-installed catalog entry installs the entry's plugin", async () => { + const environments = environmentsSeam(); + await expect(applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: {}, + workerManager: workerManager(), + environments, + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: ["paperclip.daytona-sandbox-provider"], + })).rejects.toThrow(/uses plugin "example.remote-host-plugin", which no "plugins.autoInstall" catalog entry installs/); + expect(environments.ensureManagedPluginEnvironment).not.toHaveBeenCalled(); + }); + + it("resolves the auto-installed manifest ids from the catalog when the caller passes none", async () => { + // "remote-host" is not in this build's bundled catalog and the test root + // has no distribution catalog, so the catalog lookup refuses. + await expect(applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: { PAPERCLIP_BUNDLED_PLUGIN_ROOT: "/nonexistent-paperclip-catalog" }, + workerManager: workerManager(), + environments: environmentsSeam(), + instanceSettings: instanceSettingsSeam(), + })).rejects.toThrow(/auto-install key "remote-host" is not in the bundled catalog/); + }); + + it("does not claim the sandbox slot, so a forced execution mode can coexist with plugin-only entries", async () => { + await expect(applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: { PAPERCLIP_EXECUTION_MODE: "kubernetes" }, + workerManager: workerManager(), + environments: environmentsSeam(), + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + })).resolves.toMatchObject({ ensured: 1 }); + }); + + it("archives the row of a plugin that is down and reactivates it on worker recovery", async () => { + const { handle, fireReady } = recoveryHandle(); + const environments = environmentsSeam({ + archiveManagedPluginEnvironment: vi.fn().mockResolvedValue(pluginEnvironmentRow({ status: "archived" })), + }); + const manager = workerManager(false, handle); + const result = await applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: {}, + workerManager: manager, + environments, + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + + expect(result).toMatchObject({ ensured: 0, failed: 1 }); + expect(environments.archiveManagedPluginEnvironment).toHaveBeenCalledWith({ pluginKey: PLUGIN_KEY, driverKey: "tasks" }); + expect(environments.ensureManagedPluginEnvironment).not.toHaveBeenCalled(); + + manager.isRunning.mockReturnValue(true); + fireReady(); + fireReady(); + await tick(); + + expect(environments.ensureManagedPluginEnvironment).toHaveBeenCalledTimes(1); + expect(handle.off).toHaveBeenCalledTimes(1); + }); + + it("keeps recovery armed after a worker failure and stops after one success", async () => { + const { handle, fireReady } = recoveryHandle(); + const ensureManagedPluginEnvironment = vi.fn() + .mockRejectedValueOnce(new Error("worker crashed during environmentValidateConfig")) + .mockResolvedValue({ environment: pluginEnvironmentRow(), action: "updated" }); + const environments = environmentsSeam({ ensureManagedPluginEnvironment }); + const manager = workerManager(false, handle); + await applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: {}, + workerManager: manager, + environments, + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + manager.isRunning.mockReturnValue(true); + + fireReady(); + await tick(); + expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(1); + expect(handle.off).not.toHaveBeenCalled(); + + fireReady(); + await tick(); + expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(2); + expect(handle.off).toHaveBeenCalledTimes(1); + + fireReady(); + await tick(); + expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(2); + }); + + it("queues one more attempt when the worker becomes ready during an attempt", async () => { + const { handle, fireReady } = recoveryHandle(); + let failFirst!: (err: Error) => void; + const ensureManagedPluginEnvironment = vi.fn() + .mockImplementationOnce(() => new Promise((_resolve, reject) => { failFirst = reject; })) + .mockResolvedValue({ environment: pluginEnvironmentRow(), action: "updated" }); + const manager = workerManager(false, handle); + await applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: {}, + workerManager: manager, + environments: environmentsSeam({ ensureManagedPluginEnvironment }), + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + manager.isRunning.mockReturnValue(true); + + fireReady(); + fireReady(); + expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(1); + failFirst(new Error("worker crashed")); + await tick(); + await tick(); + expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(2); + expect(handle.off).toHaveBeenCalledTimes(1); + }); + + it("stops recovery when the plugin permanently rejects the config", async () => { + const { handle, fireReady } = recoveryHandle(); + const ensureManagedPluginEnvironment = vi.fn().mockRejectedValue(unprocessable("driverConfig.region is not supported")); + const manager = workerManager(false, handle); + await applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: {}, + workerManager: manager, + environments: environmentsSeam({ ensureManagedPluginEnvironment }), + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + manager.isRunning.mockReturnValue(true); + + fireReady(); + await tick(); + fireReady(); + await tick(); + expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(1); + expect(handle.off).toHaveBeenCalledTimes(1); + }); + + it("archives and arms recovery when the worker dies during the boot ensure", async () => { + const { handle, fireReady } = recoveryHandle(); + const manager = workerManager(true, handle); + const ensureManagedPluginEnvironment = vi.fn() + .mockImplementationOnce(async () => { + manager.isRunning.mockReturnValue(false); + throw new Error("worker crashed during environmentValidateConfig"); + }) + .mockResolvedValue({ environment: pluginEnvironmentRow(), action: "updated" }); + const environments = environmentsSeam({ ensureManagedPluginEnvironment }); + const result = await applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: {}, + workerManager: manager, + environments, + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + await tick(); + + expect(result).toMatchObject({ ensured: 0, failed: 1 }); + expect(environments.archiveManagedPluginEnvironment).toHaveBeenCalledWith({ pluginKey: PLUGIN_KEY, driverKey: "tasks" }); + expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(1); + + manager.isRunning.mockReturnValue(true); + fireReady(); + await tick(); + expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(2); + expect(handle.off).toHaveBeenCalledTimes(1); + }); + + it("keeps the row and retries when the ensure fails while the worker keeps running", async () => { + const { handle, fireReady } = recoveryHandle(); + const ensureManagedPluginEnvironment = vi.fn() + .mockRejectedValueOnce(new Error("environmentValidateConfig timed out")) + .mockRejectedValueOnce(new Error("environmentValidateConfig timed out")) + .mockResolvedValue({ environment: pluginEnvironmentRow(), action: "unchanged" }); + const environments = environmentsSeam({ ensureManagedPluginEnvironment }); + const result = await applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: {}, + workerManager: workerManager(true, handle), + environments, + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + await tick(); + + expect(result).toMatchObject({ ensured: 0, failed: 1 }); + expect(environments.archiveManagedPluginEnvironment).not.toHaveBeenCalled(); + // The immediate retry also failed; the next `ready` succeeds. + expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(2); + fireReady(); + await tick(); + expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(3); + expect(handle.off).toHaveBeenCalledTimes(1); + }); + + it("neither archives nor retries when the boot ensure gets a permanent config rejection", async () => { + const { handle } = recoveryHandle(); + const environments = environmentsSeam({ + ensureManagedPluginEnvironment: vi.fn().mockRejectedValue(unprocessable("driverConfig.region is not supported")), + }); + const manager = workerManager(true, handle); + const result = await applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: {}, + workerManager: manager, + environments, + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + expect(result).toMatchObject({ ensured: 0, failed: 1 }); + expect(environments.archiveManagedPluginEnvironment).not.toHaveBeenCalled(); + expect(manager.getWorker).not.toHaveBeenCalled(); + }); + + it("treats a missing plugin, a plugin without the driver, or a worker manager without call as unavailable", async () => { + for (const [resolveEnvironmentPlugin, manager] of [ + [vi.fn(async () => null), workerManager()], + [readyPlugin("ready", false), workerManager()], + [readyPlugin("error"), workerManager()], + [readyPlugin(), { isRunning: vi.fn(() => true), getWorker: vi.fn(() => undefined) }], + ] as const) { + const environments = environmentsSeam(); + const result = await applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: {}, + workerManager: manager, + environments, + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin, + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + expect(result).toMatchObject({ ensured: 0, failed: 1 }); + expect(environments.archiveManagedPluginEnvironment).toHaveBeenCalledTimes(1); + expect(environments.ensureManagedPluginEnvironment).not.toHaveBeenCalled(); + expect(manager.getWorker).not.toHaveBeenCalled(); + } + }); + + it("never archives a row whose entry was removed from the document", async () => { + const environments = environmentsSeam(); + expect(await applyManagedEnvironments(noDb, configWith([]), { + env: {}, + workerManager: workerManager(), + environments, + instanceSettings: instanceSettingsSeam(), + })).toBeNull(); + expect(environments.archiveManagedPluginEnvironment).not.toHaveBeenCalled(); + }); + + it("never changes the instance default, and the sandbox stamp never replaces a plugin default", async () => { + const settings = instanceSettingsSeam({ + get: vi.fn().mockResolvedValue({ defaultEnvironmentId: "plugin-env" }) as InstanceSettingsSeam["get"], + }); + await applyManagedEnvironments(noDb, configWith( + [{ name: "Daytona", provider: "daytona" }, pluginEntry()], + { enableManagedSandboxOnly: true }, + ), { + env: {}, + workerManager: workerManager(), + instanceSettings: settings, + environments: environmentsSeam({ getById: vi.fn().mockResolvedValue(pluginEnvironmentRow()) }), + resolveSandboxProviderDriver: readyDriverResolver(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + expect(settings.update).not.toHaveBeenCalled(); + }); + + it("is fail-safe: an ensure failure is logged and counted, not thrown", async () => { + const result = await applyManagedEnvironments(noDb, configWith([pluginEntry()]), { + env: {}, + workerManager: workerManager(), + environments: environmentsSeam({ + ensureManagedPluginEnvironment: vi.fn().mockRejectedValue(new Error("validator refused")), + }), + instanceSettings: instanceSettingsSeam(), + resolveEnvironmentPlugin: readyPlugin(), + autoInstalledPluginKeys: [PLUGIN_KEY], + }); + expect(result).toMatchObject({ ensured: 0, failed: 1 }); + }); +}); diff --git a/server/src/services/managed-environments.test-support.ts b/server/src/services/managed-environments.test-support.ts new file mode 100644 index 00000000000..a3019692692 --- /dev/null +++ b/server/src/services/managed-environments.test-support.ts @@ -0,0 +1,139 @@ +/** + * Shared fixtures and service seams for the `applyManagedEnvironments` tests + * (`managed-environments.test.ts` for the sandbox entry and instance default, + * `managed-environments-plugin-entries.test.ts` for plugin entries). + */ +import { vi } from "vitest"; +import type { Db } from "@paperclipai/db"; +import { MANAGED_CONFIG_ENV_KEY, parseManagedConfigEnv } from "./managed-config.js"; +import type { ApplyManagedEnvironmentsOptions } from "./managed-environments.js"; + +export const noDb = null as unknown as Db; + +export function parsedConfig(overrides: Record = {}) { + const config = parseManagedConfigEnv({ + [MANAGED_CONFIG_ENV_KEY]: JSON.stringify({ + v: 1, + mode: "cloud", + catalogVersion: "2026.720.0", + features: {}, + plugins: { autoInstall: ["daytona"] }, + ...overrides, + }), + }); + if (!config) throw new Error("expected a parsed managed config"); + return config; +} + +export function environmentRow(overrides: Record = {}) { + const now = new Date(); + return { + id: "env-1", + name: "Daytona", + description: null, + driver: "sandbox", + status: "active", + config: {}, + envVars: {}, + metadata: {}, + createdAt: now, + updatedAt: now, + ...overrides, + }; +} + +export function reconciliationResult( + overrides: Record = {}, +) { + return { + environment: environmentRow(), + action: "added" as const, + stockStatus: "missing" as const, + updateAvailable: false, + stockHash: "sha256:stock", + ...overrides, + }; +} + +export function readyDriverResolver(status = "ready") { + return vi.fn(async () => ({ + plugin: { id: "plugin-1", pluginKey: "sandbox-providers/daytona", status }, + })); +} + +export function runningWorkerManager(running = true) { + return { isRunning: vi.fn(() => running), getWorker: vi.fn(() => undefined) }; +} + +export type WorkerManagerSeam = NonNullable; +export type RecoveryHandle = NonNullable>; + +/** A worker handle fake that records `ready` listeners so tests can fire them. */ +export function recoveryHandle() { + const readyListeners: Array<(payload: { pluginId: string }) => void> = []; + const handle: RecoveryHandle = { + on: vi.fn((_event, listener) => { + readyListeners.push(listener); + }), + off: vi.fn((_event, listener) => { + const at = readyListeners.indexOf(listener); + if (at !== -1) readyListeners.splice(at, 1); + }), + }; + return { + handle, + fireReady: () => { + for (const listener of [...readyListeners]) listener({ pluginId: "plugin-1" }); + }, + }; +} + +/** Lets the fire-and-forget recovery ensure settle. */ +export function tick() { + return new Promise((resolve) => setImmediate(resolve)); +} + +export type EnvironmentsSeam = NonNullable; +export type InstanceSettingsSeam = NonNullable; + +export function environmentsSeam(overrides: Partial = {}): EnvironmentsSeam { + return { + ensureManagedSandboxEnvironment: + overrides.ensureManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(reconciliationResult()), + archiveManagedSandboxEnvironment: + overrides.archiveManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(null), + getById: overrides.getById ?? vi.fn().mockResolvedValue(null), + update: overrides.update ?? vi.fn().mockResolvedValue(environmentRow()), + findManagedSandboxEnvironment: + overrides.findManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(null), + ensureManagedPluginEnvironment: + overrides.ensureManagedPluginEnvironment + ?? vi.fn().mockResolvedValue({ environment: pluginEnvironmentRow(), action: "added" }), + archiveManagedPluginEnvironment: + overrides.archiveManagedPluginEnvironment ?? vi.fn().mockResolvedValue(null), + }; +} + +export const PLUGIN_KEY = "example.remote-host-plugin"; + +export function pluginEnvironmentRow(overrides: Record = {}) { + return environmentRow({ + id: "plugin-env", + name: "Remote Host", + driver: "plugin", + config: { pluginKey: PLUGIN_KEY, driverKey: "tasks", driverConfig: {} }, + metadata: { managedByPaperclip: true, managedPluginEnvironment: `${PLUGIN_KEY}:tasks` }, + ...overrides, + }) as never; +} + +export function instanceSettingsSeam(overrides: Partial = {}): InstanceSettingsSeam { + return { + get: + overrides.get ?? + (vi.fn().mockResolvedValue({ defaultEnvironmentId: null }) as InstanceSettingsSeam["get"]), + update: + overrides.update ?? + (vi.fn().mockResolvedValue({ defaultEnvironmentId: "env-1" }) as InstanceSettingsSeam["update"]), + }; +} diff --git a/server/src/services/managed-environments.test.ts b/server/src/services/managed-environments.test.ts index 2a38e6c076d..bf8ec41e6bc 100644 --- a/server/src/services/managed-environments.test.ts +++ b/server/src/services/managed-environments.test.ts @@ -1,122 +1,18 @@ import { describe, expect, it, vi } from "vitest"; -import type { Db } from "@paperclipai/db"; -import { MANAGED_CONFIG_ENV_KEY, parseManagedConfigEnv } from "./managed-config.js"; +import { applyManagedEnvironments } from "./managed-environments.js"; import { - applyManagedEnvironments, - type ApplyManagedEnvironmentsOptions, -} from "./managed-environments.js"; - -const noDb = null as unknown as Db; - -function parsedConfig(overrides: Record = {}) { - const config = parseManagedConfigEnv({ - [MANAGED_CONFIG_ENV_KEY]: JSON.stringify({ - v: 1, - mode: "cloud", - catalogVersion: "2026.720.0", - features: {}, - plugins: { autoInstall: ["daytona"] }, - ...overrides, - }), - }); - if (!config) throw new Error("expected a parsed managed config"); - return config; -} - -function environmentRow(overrides: Record = {}) { - const now = new Date(); - return { - id: "env-1", - name: "Daytona", - description: null, - driver: "sandbox", - status: "active", - config: {}, - envVars: {}, - metadata: {}, - createdAt: now, - updatedAt: now, - ...overrides, - }; -} - -function reconciliationResult( - overrides: Record = {}, -) { - return { - environment: environmentRow(), - action: "added" as const, - stockStatus: "missing" as const, - updateAvailable: false, - stockHash: "sha256:stock", - ...overrides, - }; -} - -function readyDriverResolver(status = "ready") { - return vi.fn(async () => ({ - plugin: { id: "plugin-1", pluginKey: "sandbox-providers/daytona", status }, - })); -} - -function runningWorkerManager(running = true) { - return { isRunning: vi.fn(() => running), getWorker: vi.fn(() => undefined) }; -} - -type WorkerManagerSeam = NonNullable; -type RecoveryHandle = NonNullable>; - -/** A worker handle fake that records `ready` listeners so tests can fire them. */ -function recoveryHandle() { - const readyListeners: Array<(payload: { pluginId: string }) => void> = []; - const handle: RecoveryHandle = { - on: vi.fn((_event, listener) => { - readyListeners.push(listener); - }), - off: vi.fn((_event, listener) => { - const at = readyListeners.indexOf(listener); - if (at !== -1) readyListeners.splice(at, 1); - }), - }; - return { - handle, - fireReady: () => { - for (const listener of [...readyListeners]) listener({ pluginId: "plugin-1" }); - }, - }; -} - -/** Lets the fire-and-forget recovery ensure settle. */ -function tick() { - return new Promise((resolve) => setImmediate(resolve)); -} - -type EnvironmentsSeam = NonNullable; -type InstanceSettingsSeam = NonNullable; - -function environmentsSeam(overrides: Partial = {}): EnvironmentsSeam { - return { - ensureManagedSandboxEnvironment: - overrides.ensureManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(reconciliationResult()), - archiveManagedSandboxEnvironment: - overrides.archiveManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(null), - getById: overrides.getById ?? vi.fn().mockResolvedValue(null), - update: overrides.update ?? vi.fn().mockResolvedValue(environmentRow()), - findManagedSandboxEnvironment: - overrides.findManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(null), - }; -} - -function instanceSettingsSeam(overrides: Partial = {}): InstanceSettingsSeam { - return { - get: - overrides.get ?? - (vi.fn().mockResolvedValue({ defaultEnvironmentId: null }) as InstanceSettingsSeam["get"]), - update: - overrides.update ?? - (vi.fn().mockResolvedValue({ defaultEnvironmentId: "env-1" }) as InstanceSettingsSeam["update"]), - }; -} + environmentRow, + environmentsSeam, + instanceSettingsSeam, + noDb, + parsedConfig, + readyDriverResolver, + reconciliationResult, + recoveryHandle, + runningWorkerManager, + tick, + type InstanceSettingsSeam, +} from "./managed-environments.test-support.js"; describe("applyManagedEnvironments", () => { it("no-ops for self-hosted instances and for documents without environments", async () => { diff --git a/server/src/services/managed-environments.ts b/server/src/services/managed-environments.ts index d201728cb7a..36d9c515c3a 100644 --- a/server/src/services/managed-environments.ts +++ b/server/src/services/managed-environments.ts @@ -43,9 +43,17 @@ * sandbox provider falls back to its documented process environment variable * (e.g. `DAYTONA_API_KEY`) when `config` omits the key, so the deployment * delivers secrets as env vars and the managed document stays secret-free. + * + * `driver: "plugin"` entries take the same readiness path after the sandbox + * entry: await `pluginsReady`, require the plugin `ready`, declaring the + * entry's driver, and running a worker; else archive the entry's row and + * re-ensure it on the worker's `ready`. Each entry owns one row (see + * `managed-plugin-environments.ts`). Plugin entries never touch the instance + * default. */ import type { Db } from "@paperclipai/db"; +import { HttpError } from "../errors.js"; import { logger } from "../middleware/logger.js"; import { environmentService } from "./environments.js"; import type { ManagedSandboxEnvironmentReconcileAction } from "./environments.js"; @@ -54,7 +62,10 @@ import type { ManagedInstanceConfig } from "./managed-config.js"; import type { ManagedResourceStockStatus } from "./managed-resource-drift.js"; import { parseExecutionPolicyBootstrapEnv } from "./execution-policy-bootstrap.js"; import { resolvePluginSandboxProviderDriverByKey } from "./plugin-environment-driver.js"; +import { pluginRegistryService } from "./plugin-registry.js"; +import { resolveBundledCatalogRoot, resolveBundledPluginInstalls } from "./bundled-plugins.js"; import type { PluginWorkerManager } from "./plugin-worker-manager.js"; +import type { ManagedPluginEnvironmentReconcileAction } from "./managed-plugin-environments.js"; export interface ApplyManagedEnvironmentsOptions { env?: Record; @@ -75,8 +86,11 @@ export interface ApplyManagedEnvironmentsOptions { * subscription used for post-archive recovery: a worker in crash-restart * backoff re-emits `ready` when the manager respawns it, and the listener * re-ensures the archived environment (see `applyManagedEnvironments`). + * + * Plugin entries also need `call`, to run the plugin's own config + * validation; without it every plugin entry is treated as unavailable. */ - workerManager?: Pick & { + workerManager?: Pick & Partial> & { getWorker(pluginId: string): | { on(event: "ready", listener: (payload: { pluginId: string }) => void): void; @@ -92,6 +106,8 @@ export interface ApplyManagedEnvironmentsOptions { | "getById" | "update" | "findManagedSandboxEnvironment" + | "ensureManagedPluginEnvironment" + | "archiveManagedPluginEnvironment" >; /** Test seam: overrides the instance-settings service built from `db`. */ instanceSettings?: Pick, "get" | "update">; @@ -100,6 +116,32 @@ export interface ApplyManagedEnvironmentsOptions { db: Db; driverKey: string; }) => Promise<{ plugin: { id: string; pluginKey: string; status: string } } | null>; + /** + * Test seam: overrides the lookup of a plugin entry's plugin. Returns null + * when the plugin is not installed; `declaresDriver` is false when its + * manifest does not declare the entry's environment driver. + */ + resolveEnvironmentPlugin?: (input: { + db: Db; + pluginKey: string; + driverKey: string; + }) => Promise<{ plugin: { id: string; pluginKey: string; status: string }; declaresDriver: boolean } | null>; + /** + * Manifest ids (`pluginKey`) that the `plugins.autoInstall` catalog keys + * resolve to: `app.locals.managedBundledPluginKeys`, from + * `resolveBundledPluginInstalls` over the bundled and distribution + * catalogs. A plugin entry whose `pluginKey` is not in this list refuses + * startup. When omitted, the list is resolved here the same way. + */ + autoInstalledPluginKeys?: readonly string[]; +} + +export interface ManagedPluginEnvironmentReconciliationOutcome { + environmentId: string; + name: string; + pluginKey: string; + driverKey: string; + action: ManagedPluginEnvironmentReconcileAction; } export interface ManagedEnvironmentReconciliationOutcome { @@ -123,6 +165,28 @@ export interface ApplyManagedEnvironmentsResult { /** Preserve-only reconciliation never creates replacement backups. */ backedUp: number; outcomes: ManagedEnvironmentReconciliationOutcome[]; + pluginOutcomes: ManagedPluginEnvironmentReconciliationOutcome[]; +} + +/** + * The plugin refused the entry's config (`HttpError` 422 from + * `normalizeEnvironmentConfig` or `validatePluginEnvironmentDriverConfig`). + * Retrying cannot help until the config changes. Every other failure, such + * as a worker that crashed mid-call, is treated as transient. + */ +function isPermanentConfigRejection(err: unknown): boolean { + return err instanceof HttpError && err.status === 422; +} + +async function resolveEnvironmentPluginByKey(input: { db: Db; pluginKey: string; driverKey: string }) { + const plugin = await pluginRegistryService(input.db).getByKey(input.pluginKey); + if (!plugin) return null; + return { + plugin: { id: plugin.id, pluginKey: plugin.pluginKey, status: plugin.status }, + declaresDriver: plugin.manifestJson.environmentDrivers?.some( + (driver) => driver.driverKey === input.driverKey && driver.kind !== "sandbox_provider", + ) ?? false, + }; } /** @@ -176,19 +240,38 @@ export async function applyManagedEnvironments( } } - if (managedConfig.environments.length === 0) return null; + const pluginSpecs = managedConfig.pluginEnvironments; + if (managedConfig.environments.length === 0 && pluginSpecs.length === 0) return null; // The forced-execution-mode bootstrap (`PAPERCLIP_EXECUTION_MODE=kubernetes`) // and this one both own the single Paperclip-managed sandbox row // (`environments_managed_sandbox_idx`). Configuring both is contradictory; - // refuse startup rather than let bootstrap ordering pick a winner. + // refuse startup rather than let bootstrap ordering pick a winner. Plugin + // entries do not touch that slot. const env = opts.env ?? process.env; - if (parseExecutionPolicyBootstrapEnv(env)) { + if (managedConfig.environments.length > 0 && parseExecutionPolicyBootstrapEnv(env)) { throw new Error( `PAPERCLIP_EXECUTION_MODE and the PAPERCLIP_MANAGED_CONFIG "environments" section are mutually exclusive: both manage the single instance sandbox environment`, ); } + // Coherence, fail closed like the parser: a plugin entry names its plugin by + // manifest id, and only `plugins.autoInstall` provisions plugins on a + // managed instance, so some auto-installed catalog entry must install it. + if (pluginSpecs.length > 0) { + const autoInstalledPluginKeys = opts.autoInstalledPluginKeys ?? resolveBundledPluginInstalls( + managedConfig.plugins.autoInstall, + { catalogRoot: resolveBundledCatalogRoot(env), env, enforceCatalogRoot: true }, + ).map((install) => install.pluginKey); + for (const spec of pluginSpecs) { + if (!autoInstalledPluginKeys.includes(spec.pluginKey)) { + throw new Error( + `PAPERCLIP_MANAGED_CONFIG "environments" entry "${spec.name}" uses plugin "${spec.pluginKey}", which no "plugins.autoInstall" catalog entry installs; a managed plugin environment requires its plugin to be provisioned`, + ); + } + } + } + // The heartbeat resumes queued runs right after this bootstrap step, and // lease acquisition fails hard on a provider whose plugin is missing or not // ready. Wait for the bundled-plugin startup pass to finish, then refuse to @@ -266,10 +349,7 @@ export async function applyManagedEnvironments( // worker recovered between the gate check and the subscription. No handle // means no respawn is coming (activation never started a worker), so the // degraded-until-next-boot posture stands. - const scheduleRecoveryReactivation = ( - spec: ManagedInstanceConfig["environments"][number], - pluginId: string, - ): void => { + const scheduleRecoveryReactivation = (pluginId: string, reactivateEntry: () => Promise): void => { const manager = opts.workerManager; const handle = manager?.getWorker(pluginId); if (!handle) return; @@ -278,39 +358,41 @@ export async function applyManagedEnvironments( if (reactivated) return; reactivated = true; handle.off("ready", reactivate); - void environments - .ensureManagedSandboxEnvironment({ - name: spec.name, - description: spec.description, - provider: spec.provider, - config: { ...spec.config }, - stockVersion: managedConfig.catalogVersion, - platformFullyManaged: true, - }) - .then((result) => { - logger.info( - { - environmentId: result.environment.id, - name: spec.name, - provider: spec.provider, - action: result.action, - stockStatus: result.stockStatus, - updateAvailable: result.updateAvailable, - }, - "managed sandbox environment reactivated after provider worker recovery", - ); - }) - .catch((err: unknown) => { - logger.error( - { err, name: spec.name, provider: spec.provider }, - "failed to reactivate managed sandbox environment after provider worker recovery (degraded: environment unavailable until next boot)", - ); - }); + void reactivateEntry(); }; handle.on("ready", reactivate); if (manager?.isRunning(pluginId)) reactivate(); }; + const reactivateSandboxEntry = async (spec: ManagedInstanceConfig["environments"][number]): Promise => { + try { + const result = await environments.ensureManagedSandboxEnvironment({ + name: spec.name, + description: spec.description, + provider: spec.provider, + config: { ...spec.config }, + stockVersion: managedConfig.catalogVersion, + platformFullyManaged: true, + }); + logger.info( + { + environmentId: result.environment.id, + name: spec.name, + provider: spec.provider, + action: result.action, + stockStatus: result.stockStatus, + updateAvailable: result.updateAvailable, + }, + "managed sandbox environment reactivated after provider worker recovery", + ); + } catch (err) { + logger.error( + { err, name: spec.name, provider: spec.provider }, + "failed to reactivate managed sandbox environment after provider worker recovery (degraded: environment unavailable until next boot)", + ); + } + }; + let ensured = 0; let failed = 0; let added = 0; @@ -355,7 +437,7 @@ export async function applyManagedEnvironments( // manager restarts crashed workers, but nothing (re)installs a missing // or non-ready plugin at runtime. if (resolved && resolved.plugin.status === "ready") { - scheduleRecoveryReactivation(spec, resolved.plugin.id); + scheduleRecoveryReactivation(resolved.plugin.id, () => reactivateSandboxEntry(spec)); } continue; } @@ -407,6 +489,179 @@ export async function applyManagedEnvironments( ); } } + const resolvePlugin = opts.resolveEnvironmentPlugin ?? resolveEnvironmentPluginByKey; + const pluginOutcomes: ManagedPluginEnvironmentReconciliationOutcome[] = []; + + const ensurePluginEntry = ( + spec: ManagedInstanceConfig["pluginEnvironments"][number], + workerManager: Pick, + ) => + environments.ensureManagedPluginEnvironment({ + name: spec.name, + description: spec.description, + pluginKey: spec.pluginKey, + driverKey: spec.driverKey, + driverConfig: { ...spec.driverConfig }, + workerManager, + }); + + type PluginSpec = ManagedInstanceConfig["pluginEnvironments"][number]; + type PluginWorkerAccess = Pick; + + const archivePluginEntry = (spec: PluginSpec) => + environments + .archiveManagedPluginEnvironment({ pluginKey: spec.pluginKey, driverKey: spec.driverKey }) + .catch((archiveErr: unknown) => { + logger.error( + { err: archiveErr, name: spec.name, pluginKey: spec.pluginKey, driverKey: spec.driverKey }, + "failed to archive the managed plugin environment of an unavailable plugin", + ); + return null; + }); + + /** + * Re-ensure a plugin entry each time its worker reports `ready`, until one + * ensure succeeds or the plugin permanently rejects the entry's config. A + * worker that crashes during an attempt (for example inside + * `environmentValidateConfig`) leaves the listener armed, so its next + * respawn retries. A `ready` that arrives during an attempt queues one more + * attempt. No handle means no respawn is coming (activation never started a + * worker), so the degraded-until-next-boot posture stands. + */ + const schedulePluginRecovery = (spec: PluginSpec, pluginId: string, workerAccess: PluginWorkerAccess): void => { + const manager = opts.workerManager; + const handle = manager?.getWorker(pluginId); + if (!handle) return; + const logBase = { name: spec.name, pluginKey: spec.pluginKey, driverKey: spec.driverKey }; + let settled = false; + let inFlight = false; + let queued = false; + const settle = (): void => { + settled = true; + handle.off("ready", attempt); + }; + function attempt(): void { + if (settled) return; + if (inFlight) { + queued = true; + return; + } + inFlight = true; + void ensurePluginEntry(spec, workerAccess) + .then((result) => { + settle(); + logger.info( + { ...logBase, environmentId: result.environment.id, action: result.action }, + "managed plugin environment reactivated after plugin worker recovery", + ); + }) + .catch((err: unknown) => { + if (isPermanentConfigRejection(err)) { + settle(); + logger.error( + { err, ...logBase }, + "the plugin rejected the managed plugin environment's config; not retrying (degraded: environment unavailable until the config changes)", + ); + } else { + logger.warn( + { err, ...logBase }, + "failed to reactivate managed plugin environment; retrying when the plugin worker is ready again", + ); + } + }) + .finally(() => { + inFlight = false; + if (queued && !settled) { + queued = false; + attempt(); + } + }); + } + handle.on("ready", attempt); + if (manager?.isRunning(pluginId)) attempt(); + }; + + for (const spec of pluginSpecs) { + const logBase = { name: spec.name, pluginKey: spec.pluginKey, driverKey: spec.driverKey }; + let recoverable: { pluginId: string; workerAccess: PluginWorkerAccess } | null = null; + try { + const resolved = await resolvePlugin({ db, pluginKey: spec.pluginKey, driverKey: spec.driverKey }); + const manager = opts.workerManager; + const call = manager?.call; + // Only a `ready` record that declares the driver can recover without + // another boot: the worker manager restarts crashed workers, but nothing + // (re)installs a missing or non-ready plugin at runtime. + if (resolved && resolved.plugin.status === "ready" && resolved.declaresDriver && manager && call) { + recoverable = { + pluginId: resolved.plugin.id, + workerAccess: { isRunning: manager.isRunning.bind(manager), call: call.bind(manager) }, + }; + } + const workerRunning = resolved != null && (manager?.isRunning(resolved.plugin.id) ?? false); + if (!recoverable || !workerRunning) { + failed += 1; + const archived = await archivePluginEntry(spec); + logger.error( + { + ...logBase, + pluginStatus: resolved?.plugin.status ?? null, + declaresDriver: resolved?.declaresDriver ?? null, + workerRunning, + archivedEnvironmentId: archived?.id ?? null, + }, + "managed plugin environment's plugin is not installed, ready, declaring its driver, and running; skipping ensure and archiving any previously provisioned row (degraded: environment unavailable)", + ); + if (recoverable) schedulePluginRecovery(spec, recoverable.pluginId, recoverable.workerAccess); + continue; + } + const result = await ensurePluginEntry(spec, recoverable.workerAccess); + ensured += 1; + if (result.action === "added") added += 1; + else if (result.action === "unchanged") unchanged += 1; + else updated += 1; + pluginOutcomes.push({ + environmentId: result.environment.id, + name: result.environment.name, + pluginKey: spec.pluginKey, + driverKey: spec.driverKey, + action: result.action, + }); + logger.info( + { ...logBase, environmentId: result.environment.id, action: result.action }, + "managed plugin environment reconciled", + ); + } catch (err) { + failed += 1; + if (recoverable && !isPermanentConfigRejection(err)) { + if (!recoverable.workerAccess.isRunning(recoverable.pluginId)) { + // The worker died after the readiness check (for example inside + // `environmentValidateConfig`): treat it like a plugin that is + // down, so the row is not left active, and retry on its `ready`. + const archived = await archivePluginEntry(spec); + logger.error( + { err, ...logBase, archivedEnvironmentId: archived?.id ?? null }, + "managed plugin environment's worker stopped during ensure; archived any previously provisioned row and retrying when the plugin worker is ready again (degraded: environment unavailable)", + ); + } else { + // The worker is still running (for example the call timed out). + // Keep any previously provisioned row as it is, so agents keep a + // working environment; retry once now and on the worker's next + // `ready`, and otherwise at the next boot. + logger.error( + { err, ...logBase }, + "failed to ensure managed plugin environment while its worker is running; keeping any previously provisioned row and retrying (degraded: declaration not applied)", + ); + } + schedulePluginRecovery(spec, recoverable.pluginId, recoverable.workerAccess); + } else { + logger.error( + { err, ...logBase }, + "failed to ensure managed plugin environment; continuing boot (degraded: environment unavailable)", + ); + } + } + } + return { ensured, failed, @@ -417,5 +672,6 @@ export async function applyManagedEnvironments( removed: 0, backedUp: 0, outcomes, + pluginOutcomes, }; } diff --git a/server/src/services/native-local-process-stop.ts b/server/src/services/native-local-process-stop.ts index 37fc1569498..85d10f95359 100644 --- a/server/src/services/native-local-process-stop.ts +++ b/server/src/services/native-local-process-stop.ts @@ -6,6 +6,7 @@ import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js"; import { nativeSessionProviderEvidence } from "./native-runtime/native-session-resume.js"; import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js"; import os from "node:os"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; export const PROCESS_START_REQUESTED = "native.process_start_requested"; export const PROCESS_IDENTITY_RECORDED = "native.process_identity_recorded"; @@ -56,6 +57,8 @@ export async function recordNativeLocalProcessStop(db: Db, run: Pick lease.provider !== "local")) return false; if ((run.processPid && !absent(run.processPid)) || (run.processGroupId && !absent(-run.processGroupId))) return false; + await notifyDeliveryWork(db, DELIVERY_QUEUES.executionReplacement, + JSON.stringify({ companyId: run.companyId, runId: run.id })); await appendHeartbeatRunEvent(db, { companyId: run.companyId, runId: run.id, diff --git a/server/src/services/native-runtime/native-case-handoff.test.ts b/server/src/services/native-runtime/native-case-handoff.test.ts index 599b1e5584b..c13f5241e7c 100644 --- a/server/src/services/native-runtime/native-case-handoff.test.ts +++ b/server/src/services/native-runtime/native-case-handoff.test.ts @@ -84,7 +84,7 @@ describe("native persisted case handoff runtime", () => { expect(await isCaseHandoffSourceReleased(db, f.input)).toBe(false); const exited = once(child, "exit"); child.kill("SIGTERM"); await exited; const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); - expect(await recordNativeLocalProcessStop(db, run!)).toBe(true); + expect(await db.transaction(tx => recordNativeLocalProcessStop(tx as unknown as typeof db, run!))).toBe(true); // A new worker has no process-local state and reconstructs the release. expect(await isCaseHandoffSourceReleased(db, f.input)).toBe(true); expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]?.resultJson).toHaveProperty("caseHandoffExecutionReleased"); diff --git a/server/src/services/native-runtime/native-runner-restart-preparation.integration.test.ts b/server/src/services/native-runtime/native-runner-restart-preparation.integration.test.ts index 20eff3138be..350eee86282 100644 --- a/server/src/services/native-runtime/native-runner-restart-preparation.integration.test.ts +++ b/server/src/services/native-runtime/native-runner-restart-preparation.integration.test.ts @@ -1,11 +1,21 @@ import { startNativeSessionExecutionLeaseRenewal, type NativeExecutionLeaseRenewal } from "./native-execution-lease.js"; import { randomUUID } from "node:crypto"; -import { resolve } from "node:path"; +import { resolve, join } from "node:path"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; import { eq, sql } from "drizzle-orm"; -import { environmentLeases, environments, heartbeatRuns, nativeRunFinalizations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db"; +import { environmentLeases, environments, heartbeatRuns, nativeRunFinalizations, issueRecoveryActions, issues, completionContracts, nativeRunResults, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db"; import { beforeAll, expect, it, vi } from "vitest"; -import { DurablePrpControlPlane } from "../../vendor/paperclip-runner/index.js"; +import { spawn } from "node:child_process"; +import { createRequire } from "node:module"; +import { DurablePrpControlPlane, createNativeSessionBackend, createRunnerdCodexTransport, defaultCapabilityRunnerdBinary, canonicalNativeRuntimeContextDigest } from "../../vendor/paperclip-runner/index.js"; import { buildNativeExecutionInput } from "./native-execution-input.js"; +import { buildNativeCompletionContract } from "./completion-contracts.js"; +import { reconcileNativeFinalizations, claimNativeSessionResumptions } from "./native-finalization-reconciler.js"; +import { prepareNativeWorkspaceSync } from "./native-workspace-sync.js"; +import { workspaceOperationService } from "../workspace-operations.js"; +import type { EnvironmentLease } from "@paperclipai/shared"; +import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime"; +import { materializeAsset } from "./runtime-context.js"; import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js"; import { executePaperclipNativeSession } from "./native-session-executor.js"; import { registerNativeRunnerCheckpoint } from "./native-runner-checkpoint.js"; @@ -21,6 +31,147 @@ describeEmbeddedPostgres("native runner restart recovery: preparation", () => { let successor: typeof environment.current.successor; beforeAll(() => { ({ runtimeRoot, companyId, agentId, successor } = environment.current); }); + it.each(["recover", "provider event", "cancelled task"] as const)("reconciles the drained remote backend with %s", async mode => { + const fixture = await seedRun(`DRAINED-BACKEND-${mode.replaceAll(" ", "-")}`); + const environmentId = randomUUID(), leaseId = randomUUID(); + await fixture.db.insert(environments).values({ id: environmentId, name: `Drained backend ${environmentId}`, driver: "sandbox" }); + const [lease] = await fixture.db.insert(environmentLeases).values({ id: leaseId, companyId, environmentId, + heartbeatRunId: fixture.runId, issueId: fixture.issueId, status: "active", + provider: "fixture", providerLeaseId: `drained-backend-${leaseId}` }).returning(); + const [issue] = await fixture.db.select().from(issues).where(eq(issues.id, fixture.issueId)); + const [contract] = await fixture.db.select().from(completionContracts).where(eq(completionContracts.issueId, fixture.issueId)); + const runtimeContext = nativeRuntimeContextFixture(); + runtimeContext.instructions.bundle = await materializeAsset([{ path: "AGENTS.md", mode: 0o444, + content: Buffer.from("Complete the controlled restart fixture task.\n") }]); + runtimeContext.aggregateDigest = canonicalNativeRuntimeContextDigest(runtimeContext); + const sandbox = join(runtimeRoot, `workspace-${fixture.runId}`); + await mkdir(sandbox, { recursive: true }); + const savedFile = join(sandbox, "saved-before-drain.txt"); + await writeFile(savedFile, "Preserve admitted work.\n"); + const execution = buildNativeExecutionInput({ companyId, agentId, runId: fixture.runId, + issue: { id: fixture.issueId, identifier: "NRR-DRAINED", title: "Continue after drain", description: null, workMode: "standard" }, + taskPrompt: "Continue the admitted work once.", provider: "codex", + workspace: { id: fixture.runId, cwd: sandbox, repoUrl: null, repoRef: null, branchName: null }, + normalizedSessionId: fixture.native.normalizedSessionId, runtimeContext, + completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1", + contract: buildNativeCompletionContract(issue, { revision: contract.revision }) }, + }); + await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: { + ...fixture.preparedProfile, nativeExecutionInput: execution, + } }).where(eq(heartbeatRuns.id, fixture.runId)); + const remoteSandbox = join(runtimeRoot, `remote-${fixture.runId}`); + const shell: CommandManagedRuntimeRunner = { execute: async input => { + const startedAt = new Date().toISOString(); + const child = spawn(input.command, input.args ?? [], { cwd: input.cwd, + env: { ...process.env, ...input.env }, stdio: ["pipe", "pipe", "pipe"] }); + let stdout = "", stderr = "", timedOut = false; + child.stdout.setEncoding("utf8").on("data", chunk => { stdout += chunk; }); + child.stderr.setEncoding("utf8").on("data", chunk => { stderr += chunk; }); + child.stdin.on("error", () => undefined); + child.stdin.end(input.stdin); + const timeout = setTimeout(() => { timedOut = true; child.kill("SIGKILL"); }, input.timeoutMs ?? 30_000); + try { return await new Promise((resolveResult, reject) => { + child.once("error", reject); + child.once("close", (exitCode, signal) => resolveResult({ stdout, stderr, exitCode, signal, timedOut, + pid: child.pid ?? null, startedAt })); + }); } finally { clearTimeout(timeout); } + } }; + const workspaceSync = await prepareNativeWorkspaceSync({ db: fixture.db, runId: fixture.runId, companyId, + workspaceId: fixture.runId, workspaceLocalDir: sandbox, lease: lease as unknown as EnvironmentLease, + target: { kind: "remote", transport: "sandbox", environmentId, leaseId, remoteCwd: remoteSandbox, runner: shell } }); + expect(workspaceSync).not.toBeNull(); + const child = spawn(process.execPath, ["--import", createRequire(import.meta.url).resolve("tsx"), + resolve(import.meta.dirname, "../../__tests__/helpers/native-drain-controller.test-worker.ts")], + { stdio: ["ignore", "pipe", "pipe", "ipc"], env: process.env }); + const logs: string[] = []; + child.stdout?.on("data", data => logs.push(String(data))); + child.stderr?.on("data", data => logs.push(String(data))); + const exited = new Promise(resolveExit => child.once("exit", resolveExit)); + try { + const result = await new Promise<{ commands: number; errorName: string; errorMessage: string }>((resolveResult, reject) => { + const deadline = setTimeout(() => reject(new Error(`Controller handoff timed out: ${logs.join("").slice(-2000)}`)), 60_000); + child.once("error", error => { clearTimeout(deadline); reject(error); }); + child.once("exit", code => { clearTimeout(deadline); reject(new Error(`Controller exited before handoff (${code}): ${logs.join("").slice(-2000)}`)); }); + child.on("message", (message: { ready?: boolean; done?: boolean; commands: number; errorName: string; errorMessage: string; error?: string }) => { + if (message.ready) child.send({ connectionString: environment.current.temporary.connectionString, providerEvidence: mode === "provider event", + options: { execution, runnerInstanceId: fixture.native.runnerInstanceId, useRunnerd: true, + runnerExecutionTarget: { kind: "remote", transport: "sandbox", environmentId, leaseId, remoteCwd: remoteSandbox } } }); + if (message.done) { clearTimeout(deadline); resolveResult(message); } + if (message.error) { clearTimeout(deadline); reject(new Error(message.error)); } + }); + }); + expect(result).toMatchObject({ commands: 0, errorMessage: "local_runtime_api_listener_unavailable" }); + expect(await exited).toBe(0); + const [handoff] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId)); + if (mode === "provider event") { + expect(handoff).toMatchObject({ phase: "terminal_failure", nextAttemptAt: null, + failureDetail: { recoveryMode: "ambiguous_state", providerSessionEstablished: true } }); + expect(await claimNativeSessionResumptions({ db: fixture.db, runnerInstanceId: "no-provider-replay", + runIds: [fixture.runId], deferIneligible: false })).toHaveLength(0); + return; + } + expect(handoff).toMatchObject({ phase: "retryable_failure", attempt: 1, + failureDetail: { originalFailureCode: "local_runtime_api_listener_unavailable", recoveryMode: "bootstrap_retry", + providerSessionEstablished: false, providerEventsExist: false, checkpointExists: false } }); + if (mode === "cancelled task") { + await fixture.db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, fixture.issueId)); + } + const claims = (await Promise.all(["successor-a", "successor-b"].map(runnerInstanceId => + claimNativeSessionResumptions({ db: fixture.db, runnerInstanceId, runIds: [fixture.runId], + now: new Date(handoff.nextAttemptAt!.getTime() + 1), deferIneligible: false }), + ))).flat(); + if (mode === "cancelled task") { expect(claims).toHaveLength(0); return; } + expect(claims).toHaveLength(1); + const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)); + expect(run).toMatchObject({ status: "running", nativeSessionId: fixture.native.normalizedSessionId }); + const [action] = await fixture.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, fixture.issueId)); + expect(action).toMatchObject({ ownerType: "agent", ownerAgentId: agentId }); + if (process.env.PAPERCLIP_TEST_DRAIN_CONTINUATION === "1") { + const claim = claims[0]; + const root = join(runtimeRoot, `continued-${fixture.runId}`); + const home = join(root, "codex-home"); + await mkdir(home, { recursive: true }); + const report = join(root, "result.json"); + const calls = join(root, "calls.log"); + await writeFile(report, JSON.stringify({ schema: "paperclip.run_result.v1", reportedWorkDisposition: "done", + summary: "Admitted work completed after controller restart.", completionClaim: { + contractRevision: execution.completionContract.contract.revision, objectiveSatisfied: true, + criteria: execution.completionContract.contract.criteria.map(criterion => ({ criterionId: criterion.id, + status: "satisfied", evidenceRefs: [] })), remainingWork: [] }, + evidence: [], verification: [], artifacts: [], attentionRequests: [] })); + const bundle = createRunnerdCodexTransport({ ...transportOptions(fixture, root), + runnerBinary: defaultCapabilityRunnerdBinary(), + codexArgs: ["--state-file", join(root, "provider.json"), "--terminal-result-file", report, + "--call-log", calls, "--record-process-start"], + runtimeContext: execution.runtimeContext, runnerRuntimeContext: execution.runtimeContext, + sourceCodexHome: home, environment: { CODEX_HOME: home, PATH: process.env.PATH }, + lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null }, + }); + const backend = createNativeSessionBackend(execution, { runnerInstanceId: fixture.native.runnerInstanceId, + environment: { CODEX_HOME: home, PATH: process.env.PATH }, codexTransportFactory: () => bundle.transport }); + try { + expect(await executePaperclipNativeSession({ db: fixture.db, execution, backend, useRunnerd: true, + runnerInstanceId: fixture.native.runnerInstanceId, leaseOwner: claim.leaseOwner, + runnerExecutionTarget: { kind: "remote", transport: "sandbox", environmentId, leaseId, remoteCwd: remoteSandbox, + runner: { execute: async () => { throw new Error("Unexpected remote preparation after the controlled backend owns execution"); } } as never }, + turnTimeoutMs: 15_000 })).toMatchObject({ exitCode: 0, timedOut: false }); + const recorder = workspaceOperationService(fixture.db).createRecorder({ companyId, + issueId: fixture.issueId, heartbeatRunId: fixture.runId }); + const saved = await recorder.recordOperation({ phase: "workspace_finalize", cwd: sandbox, + run: async () => { await workspaceSync!.restoreWorkspace(); return { status: "succeeded", exitCode: 0 }; } }); + expect(saved.status).toBe("succeeded"); + await reconcileNativeFinalizations(fixture.db, [fixture.runId]); + expect(await fixture.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, fixture.runId))) + .toMatchObject([{ schemaStatus: "accepted" }]); + const [finished] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId)); + expect(finished, JSON.stringify(finished.failureDetail)).toMatchObject({ phase: "committed", attempt: 2 }); + expect(await readFile(savedFile, "utf8")).toBe("Preserve admitted work.\n"); + expect((await readFile(calls, "utf8")).split("\n").filter(line => line === "process-start")).toHaveLength(1); + } finally { await bundle.transport.close(); } + } + } finally { child.kill("SIGKILL"); await exited; await workspaceSync?.cleanup(); } + }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); + it.each(["preparation", "claim lock"])("gives a fresh remote claim its full lease through blocked %s", async (blockedPhase) => { const fixture = await seedRun(`FRESH-SLOW-${blockedPhase === "claim lock" ? "CLAIM-LOCK" : "PREPARATION"}`); const environmentId = randomUUID(), leaseId = randomUUID(); diff --git a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts index 509213ca403..5241fe2b5b5 100644 --- a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts @@ -2220,10 +2220,15 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( }, ); - it.each([2, 3])("recovers a checkpointed restart interruption within provider budget %s", async (attempt) => { - const fixture = await seedRun(`INTERRUPTED-${attempt}`); + it.each([ + { attempt: 2, driverKind: "codex_app_server" }, + { attempt: 3, driverKind: "codex_app_server" }, + { attempt: 2, driverKind: "acpx_runtime" }, + { attempt: 2, driverKind: "opencode_server" }, + ])("recovers a checkpointed $driverKind restart interruption within provider budget $attempt", async ({ attempt, driverKind }) => { + const fixture = await seedRun(`INTERRUPTED-${driverKind}-${attempt}`); await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: { sessionCheckpoint: { - driverKind: "codex_app_server", + driverKind, sessionId: fixture.native.normalizedSessionId, identity: { companyId, issueId: fixture.issueId, runId: fixture.runId, agentId, sessionId: fixture.native.normalizedSessionId }, providerSessionId: "interrupted-provider", diff --git a/server/src/services/native-runtime/native-safe-replacement.test.ts b/server/src/services/native-runtime/native-safe-replacement.test.ts index 491bae17b2b..088ae1bffc6 100644 --- a/server/src/services/native-runtime/native-safe-replacement.test.ts +++ b/server/src/services/native-runtime/native-safe-replacement.test.ts @@ -16,6 +16,11 @@ import { deliverReconciledExecutions, } from "../execution-recovery-resolution.js"; import { randomUUID } from "node:crypto"; +import { spawn } from "node:child_process"; +import { persistHeartbeatRunProcessMetadata } from "../heartbeat/run-lifecycle.js"; +import { hasNativeLocalProcessStop, releaseSettledNativeRunProcess } from "../native-local-process-stop.js"; +import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; +import { executionControlSchedule } from "../execution-control-work.js"; import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js"; import { tmpdir } from "node:os"; import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; @@ -26,6 +31,7 @@ import { agents, companies, createDb, + subscribeDatabaseWork, heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, @@ -41,7 +47,7 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase, } from "../../__tests__/helpers/embedded-postgres.js"; -import { reconcileSafeNativeReplacements } from "./native-safe-replacement.js"; +import { invalidateNativeReplacementDiscovery, reconcileSafeNativeReplacements } from "./native-safe-replacement.js"; import { reconcileAbandonedExecutionControl } from "../execution-control-reconciliation.js"; const externalDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL; const support = externalDatabaseUrl @@ -219,6 +225,103 @@ const support = externalDatabaseUrl } }, ); + it("wakes a completed replacement scan after actual local stop proof commits", async () => { + const source = await seed(2); + const workspace = await mkdtemp(join(tmpdir(), "paperclip-late-stop-")); + const child = spawn(process.execPath, ["-e", "process.exit(0)"], { detached: true, stdio: "ignore" }); + const pid = child.pid!; + await new Promise((resolve, reject) => { child.once("exit", () => resolve()); child.once("error", reject); }); + await persistHeartbeatRunProcessMetadata(db, source.runId, { pid, processGroupId: pid, + startedAt: new Date().toISOString(), targetKind: "local" }); + await db.update(heartbeatRuns).set({ finishedAt: new Date(), runnerProfileJson: { + nativeExecutionInput: { provider: { kind: "codex" }, workspace: { cwd: workspace } }, + } }).where(eq(heartbeatRuns.id, source.runId)); + await db.update(nativeRunFinalizations).set({ failureCode: "provider_transport_failed" }) + .where(eq(nativeRunFinalizations.runId, source.runId)); + await appendHeartbeatRunEvent(db, { companyId: source.companyId, runId: source.runId, agentId: source.agentId, + eventType: "harness.ready", stream: "system", message: "Provider started before its transport failed." }); + const verifyStoppedSession = async () => await hasNativeLocalProcessStop(db, source.companyId, source.runId) + ? { evidence: {}, retire: () => true } : null; + invalidateNativeReplacementDiscovery(db); + expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 0 }); + expect(await executionControlSchedule(db).replacement()).toBeNull(); + let wakes = 0; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.executionReplacement, event => { + if (event === "settled") { wakes++; invalidateNativeReplacementDiscovery(db); } + }); + try { + await db.transaction(async tx => { + expect(await releaseSettledNativeRunProcess(tx as unknown as typeof db, source.runId)).toBe("released"); + expect(wakes).toBe(0); + }); + expect(wakes).toBeGreaterThan(0); + expect(await executionControlSchedule(db).replacement()).not.toBeNull(); + expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 1 }); + expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 0 }); + expect(await executionControlSchedule(db).replacement()).toBeNull(); + const successors = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, source.runId)); + expect(successors).toHaveLength(1); + } finally { unsubscribe(); await rm(workspace, { recursive: true, force: true }); } + }); + it("wakes completed recovery when the real task becomes eligible", async () => { + const source = await seed(); + const workspace = await mkdtemp(join(tmpdir(), "paperclip-late-eligibility-")); + await db.update(heartbeatRuns).set({ runnerProfileJson: { recoveryEventInventoryVersion: 1, + nativeExecutionInput: { provider: { kind: "codex" }, workspace: { cwd: workspace } }, + } }).where(eq(heartbeatRuns.id, source.runId)); + await issueService(db).update(source.issueId, { status: "backlog" }); + invalidateNativeReplacementDiscovery(db); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 }); + expect(await executionControlSchedule(db).replacement()).toBeNull(); + let wakes = 0; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.executionReplacement, event => { + if (event === "settled") { wakes++; invalidateNativeReplacementDiscovery(db); } + }); + try { + await issueService(db).update(source.issueId, { status: "in_progress" }); + expect(wakes).toBeGreaterThan(0); + expect(await executionControlSchedule(db).replacement()).not.toBeNull(); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 1 }); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 }); + expect(await executionControlSchedule(db).replacement()).toBeNull(); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, source.runId))).toHaveLength(1); + } finally { unsubscribe(); await rm(workspace, { recursive: true, force: true }); } + }); + it("advances past a failed full page and retries it after later work progresses", async () => { + const workspace = await mkdtemp(join(tmpdir(), "paperclip-failed-page-")); + const sources: Awaited>[] = []; + for (let index = 0; index < 26; index++) { + const source = await seed(); sources.push(source); + await db.update(heartbeatRuns).set({ runnerProfileJson: { + nativeExecutionInput: { provider: { kind: "codex" }, workspace: { cwd: workspace } }, + } }).where(eq(heartbeatRuns.id, source.runId)); + await db.update(nativeRunFinalizations).set({ failureCode: "provider_transport_failed" }) + .where(eq(nativeRunFinalizations.runId, source.runId)); + } + sources.sort((a, b) => a.runId.localeCompare(b.runId)); + const later = sources.at(-1)!; + let failing = true; + const verifyStoppedSession = vi.fn(async (run: typeof heartbeatRuns.$inferSelect) => { + if (failing && run.id !== later.runId) throw new Error("Controlled stopped-session verifier failure"); + return { evidence: {}, retire: () => true }; + }); + invalidateNativeReplacementDiscovery(db); + try { + await expect(reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })) + .rejects.toThrow("native_replacement_decision_retry_required"); + expect(verifyStoppedSession).toHaveBeenCalledTimes(25); + expect(await executionControlSchedule(db).replacement()).not.toBeNull(); + expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 1 }); + expect(verifyStoppedSession).toHaveBeenCalledTimes(26); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, later.runId))).toHaveLength(1); + failing = false; + expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 25 }); + expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 0 }); + expect(await executionControlSchedule(db).replacement()).toBeNull(); + const successors = await db.select().from(heartbeatRuns).where(inArray(heartbeatRuns.retryOfRunId, sources.map(source => source.runId))); + expect(successors).toHaveLength(26); + } finally { await rm(workspace, { recursive: true, force: true }); } + }, 60_000); it.each(["unproven", "changed", "verified"] as const)("requires stopped-session proof through commit (%s)", async (mode) => { const source = await seed(2); await db.update(nativeRunFinalizations).set({ failureCode: "native_session_cleanup_quarantined" }).where(eq(nativeRunFinalizations.runId, source.runId)); @@ -235,8 +338,11 @@ const support = externalDatabaseUrl eventType: "tool.execution.started", stream: "system", payload: { name: "paperclip_finish", executionId: "completion", transport: "process" }, }); - await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession }); - await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession }); + for (let sweep = 0; sweep < 2; sweep++) { + const decision = reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession }); + if (mode === "changed") await expect(decision).rejects.toThrow("native_replacement_decision_retry_required"); + else await decision; + } const successors = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, source.runId)); expect(successors).toHaveLength(mode === "verified" ? 1 : 0); const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, source.issueId)); diff --git a/server/src/services/native-runtime/native-safe-replacement.ts b/server/src/services/native-runtime/native-safe-replacement.ts index 89ce98400e5..b2ce3bef1bc 100644 --- a/server/src/services/native-runtime/native-safe-replacement.ts +++ b/server/src/services/native-runtime/native-safe-replacement.ts @@ -5,7 +5,7 @@ import { notifyExecutionRunWork } from "../execution-control-notifications.js"; import { logger } from "../../middleware/logger.js"; import { createHash, randomUUID } from "node:crypto"; import { stat } from "node:fs/promises"; -import { and, eq, inArray, isNull, sql } from "drizzle-orm"; +import { and, asc, eq, gt, inArray, isNull, or, sql } from "drizzle-orm"; import { agentWakeupRequests, environmentLeases, @@ -27,6 +27,8 @@ import { issueRecoveryActionService } from "../issue-recovery-actions.js"; import { buildExecutionContinuation } from "../execution-continuation.js"; import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js"; import { hasRemoteTerminationReceipt } from "../remote-execution-termination.js"; +import { NATIVE_PROVIDER_DURABLE_EVENT_TYPES } from "./native-retired-backup-source.js"; +import { completedWorkspaceRecoveryCondition } from "../workspace-recovery-receipt.js"; export const NATIVE_SAFE_REPLACEMENT_REASON = "native_safe_replacement"; const record = (v: unknown): Record => @@ -45,6 +47,66 @@ const processAlive = (pid: number | null) => { type ReplacementDb = Db | Parameters[0]>[0]; +const listenerUnavailable = "local_runtime_api_listener_unavailable"; +function historicalListenerGuardPending() { + const profile = heartbeatRuns.runnerProfileJson, detail = nativeRunFinalizations.failureDetail; + const providerEvents = sql.join([...NATIVE_PROVIDER_DURABLE_EVENT_TYPES].map(value => sql`${value}`), sql`, `); + return and( + eq(nativeRunFinalizations.failureCode, "native_session_interrupted"), + eq(heartbeatRuns.errorCode, "native_session_interrupted"), eq(heartbeatRuns.error, listenerUnavailable), + sql`${detail}->>'message' = ${listenerUnavailable}`, + sql`${detail}->>'originalFailureCode' = 'native_session_interrupted'`, + sql`${detail}->>'recoveryMode' = 'ambiguous_state'`, + sql`${detail}->'providerSessionEstablished' = 'false'::jsonb`, + sql`${detail}->'providerEventsExist' = 'false'::jsonb`, + sql`${detail}->'checkpointExists' = 'false'::jsonb`, + eq(nativeRunFinalizations.attempt, 1), + isNull(nativeRunFinalizations.leaseOwner), isNull(nativeRunFinalizations.leaseExpiresAt), + isNull(nativeRunFinalizations.assessmentId), isNull(nativeRunFinalizations.decisionId), + isNull(heartbeatRuns.processPid), isNull(heartbeatRuns.processGroupId), + isNull(heartbeatRuns.sessionIdBefore), isNull(heartbeatRuns.sessionIdAfter), isNull(heartbeatRuns.retryOfRunId), + sql`not (coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) ?| array['cancellation', 'nativeCancellation'])`, + sql`coalesce(${profile}->'sessionCheckpoint', 'null'::jsonb) = 'null'::jsonb`, + sql`coalesce(${profile}->'nativeProviderPackAdmission', 'null'::jsonb) = 'null'::jsonb`, + sql`coalesce(${profile}#>>'{nativeProviderPackAdmissionState,state}', 'pending') = 'pending'`, + sql`${profile}#>>'{nativeExecutionInput,binding,runId}' = ${heartbeatRuns.id}::text`, + sql`${profile}#>>'{nativeExecutionInput,binding,companyId}' = ${heartbeatRuns.companyId}::text`, + sql`${profile}#>>'{nativeExecutionInput,binding,agentId}' = ${heartbeatRuns.agentId}::text`, + sql`${profile}#>>'{nativeExecutionInput,binding,issueId}' = ${nativeRunFinalizations.issueId}::text`, + sql`${profile}#>>'{nativeExecutionInput,session,normalizedSessionId}' = ${heartbeatRuns.nativeSessionId}::text`, + sql`not exists (select 1 from ${heartbeatRuns} prior where prior.company_id = ${heartbeatRuns.companyId} + and prior.native_session_id = ${heartbeatRuns.nativeSessionId} and prior.id <> ${heartbeatRuns.id})`, + sql`not exists (select 1 from ${heartbeatRunEvents} event where event.company_id = ${heartbeatRuns.companyId} + and event.run_id = ${heartbeatRuns.id} and (event.source_event_id is not null + or event.event_type in (${providerEvents}) or event.event_type ~ '^(turn|item|tool|semantic_tool|run[.]result)[.]'))`, + sql`not exists (select 1 from ${toolInvocations} invocation where invocation.company_id = ${heartbeatRuns.companyId} + and invocation.run_id = ${heartbeatRuns.id})`, + sql`not exists (select 1 from ${issueRecoveryActions} + where ${issueRecoveryActions.companyId} = ${heartbeatRuns.companyId} + and ${issueRecoveryActions.sourceIssueId} = ${nativeRunFinalizations.issueId} + and ${issueRecoveryActions.cause} = 'native_continuation_requires_reconciliation' + and ${issueRecoveryActions.evidence}->>'runId' = ${heartbeatRuns.id}::text + and ${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked' + and not coalesce((${completedWorkspaceRecoveryCondition(heartbeatRuns.id)}), false))`, + sql`exists (select 1 from ${issues} task where task.company_id = ${heartbeatRuns.companyId} + and task.id = ${nativeRunFinalizations.issueId} and task.assignee_agent_id = ${heartbeatRuns.agentId} + and task.status in ('todo', 'in_progress', 'in_review', 'blocked') + and (task.status <> 'blocked' or exists (select 1 from ${issueRecoveryActions} hold + where hold.company_id = task.company_id and hold.source_issue_id = task.id and hold.kind = 'active_run_watchdog' + and hold.cause = ${nativeRunFinalizations.failureCode} and hold.evidence->>'runId' = ${heartbeatRuns.id}::text + and hold.evidence#>>'{nativeFailureBlock,runId}' = ${heartbeatRuns.id}::text + and hold.evidence#>'{nativeFailureBlock,statusVersion}' = to_jsonb(task.status_version))))`, + ); +} + +async function historicalListenerGuardEvidence(db: ReplacementDb, run: typeof heartbeatRuns.$inferSelect) { + const [evidence] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns) + .innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)).where(and( + eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.id, run.id), historicalListenerGuardPending(), + )).limit(1); + return Boolean(evidence); +} + /** A saved remote source is proved by its current ledger, never a controller path. */ async function remoteWorkspaceReplacementEvidence( db: ReplacementDb, @@ -113,16 +175,95 @@ export function nativeReplacementPending() { return and( eq(heartbeatRuns.status, "failed"), eq(nativeRunFinalizations.phase, "terminal_failure"), - inArray( + or(inArray( nativeRunFinalizations.failureCode, ["native_provider_terminal_failed", "native_session_cleanup_quarantined", "provider_transport_failed"], - ), + ), historicalListenerGuardPending()), isNull(nativeRunFinalizations.resultId), sql`coalesce(${nativeRunFinalizations.failureDetail}->>'successorRunId', '') = ''`, sql`coalesce(${nativeRunFinalizations.failureDetail}->>'replacementDenied', '') = ''`, ); } +const REPLACEMENT_SCAN_LIMIT = 100; +const replacementDiscovery = new WeakMap(); +function replacementDiscoveryState(db: Db) { + let state = replacementDiscovery.get(db); + if (!state) { state = { after: undefined, complete: false, generation: 0, rescan: false }; replacementDiscovery.set(db, state); } + return state; +} + +export function invalidateNativeReplacementDiscovery(db: Db) { + const state = replacementDiscoveryState(db); + state.complete = false; state.rescan = true; state.generation++; +} + +async function replacementHistoryReady(db: Db, run: typeof heartbeatRuns.$inferSelect, issueId: string, retryDatabaseErrors = false) { + try { + const continuation = await buildExecutionContinuation({ db, companyId: run.companyId, issueId, + agentId: run.agentId, context: record(run.contextSnapshot), summary: null, exposeLowTrustRaw: false }); + return continuation.unresolvedInteractionIds.length === 0; + } catch (error) { + let cause: unknown = error; + for (let depth = 0; retryDatabaseErrors && depth < 4 && cause && typeof cause === "object"; depth++) { + const failure = cause as { code?: unknown; cause?: unknown }; + if (typeof failure.code === "string" && /^[A-Z0-9_]+$/.test(failure.code)) throw error; + cause = failure.cause; + } + return null; + } +} + +async function discoverNativeReplacements(db: Db, after: string | undefined, limit = 25) { + const candidates: Array<{ run: typeof heartbeatRuns.$inferSelect; coordinator: typeof nativeRunFinalizations.$inferSelect }> = []; + let scanned = 0; + for (;;) { + const rows = await db.select({ run: heartbeatRuns, coordinator: nativeRunFinalizations }).from(heartbeatRuns) + .innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)) + .where(and(nativeReplacementPending(), after ? gt(heartbeatRuns.id, after) : undefined)) + .orderBy(asc(heartbeatRuns.id)).limit(Math.min(25, REPLACEMENT_SCAN_LIMIT - scanned)); + if (!rows.length) return { candidates, scanned, after: undefined }; + for (const row of rows) { + after = row.run.id; scanned++; + let ready = true; + if (row.coordinator.failureCode === "native_session_interrupted") { + ready = false; + if (row.run.runnerProfileJson?.remoteWorkspace != null) { + const leases = await db.select().from(environmentLeases).where(and( + eq(environmentLeases.companyId, row.run.companyId), eq(environmentLeases.heartbeatRunId, row.run.id), + )); + if (leases.every(lease => lease.releasedAt !== null)) { + const evidence = await remoteWorkspaceReplacementEvidence(db, row.run, row.coordinator.issueId, leases); + ready = Boolean(evidence?.workspacePreserved && evidence.providerStopped) + && await replacementHistoryReady(db, row.run, row.coordinator.issueId, true) === true; + } + } + } + if (ready) candidates.push(row); + if (candidates.length >= limit || scanned >= REPLACEMENT_SCAN_LIMIT) return { candidates, scanned, after }; + } + if (rows.length < 25) return { candidates, scanned, after: undefined }; + } +} + +function settleReplacementDiscovery(state: ReturnType, after: string | undefined, generation: number) { + const rescan = state.rescan || generation !== state.generation; + state.after = after; state.complete = !after && !rescan; + state.rescan = Boolean(after && rescan); +} + +export async function hasPendingNativeReplacements(db: Db) { + const state = replacementDiscoveryState(db); + if (state.complete) return false; + if (state.after) return true; + state.rescan = false; + const generation = state.generation; + const result = await discoverNativeReplacements(db, undefined, 1); + if (result.candidates.length) return true; + settleReplacementDiscovery(state, result.after, generation); + return !state.complete; +} + /** Retryable outbox: terminal_failure plus a new-format failure code is the durable candidate. */ export async function reconcileSafeNativeReplacements( db: Db, @@ -136,18 +277,14 @@ export async function reconcileSafeNativeReplacements( failpoint?: (phase: "successor_inserted" | "lineage_committed") => void; } = {}, ) { - const candidates = await db - .select({ run: heartbeatRuns, coordinator: nativeRunFinalizations }) - .from(heartbeatRuns) - .innerJoin( - nativeRunFinalizations, - eq(nativeRunFinalizations.runId, heartbeatRuns.id), - ) - .where( - nativeReplacementPending(), - ) - .limit(25); - let scheduled = 0; + const discovery = replacementDiscoveryState(db); + if (discovery.complete) discovery.after = undefined; + discovery.complete = false; + if (!discovery.after) discovery.rescan = false; + const generation = discovery.generation; + const selection = await discoverNativeReplacements(db, discovery.after); + const candidates = selection.candidates; + let scheduled = 0, pendingFailures = 0; for (const { run, coordinator } of candidates) { try { if ( @@ -170,11 +307,13 @@ export async function reconcileSafeNativeReplacements( ); // Teardown is still in progress. The next sweep rechecks its durable outcome. if (leases.some((lease) => lease.releasedAt === null)) continue; - const stoppedSession = coordinator.failureCode !== "native_provider_terminal_failed" + const historicalListener = coordinator.failureCode === "native_session_interrupted"; + if (historicalListener && !await historicalListenerGuardEvidence(db, run)) continue; + const stoppedSession = coordinator.failureCode !== "native_provider_terminal_failed" && !historicalListener ? await options.verifyStoppedSession?.(run) ?? null : null; // A transport label alone is not evidence. Keep inspecting these candidates // as process cleanup and the final transcript become durable. - if (coordinator.failureCode !== "native_provider_terminal_failed" && !stoppedSession) continue; + if (coordinator.failureCode !== "native_provider_terminal_failed" && !historicalListener && !stoppedSession) continue; const invocations = await db .select() .from(toolInvocations) @@ -263,23 +402,11 @@ export async function reconcileSafeNativeReplacements( ? [String(p.kind ?? p.type)] : []; }); - let historyComplete = false; - try { - const continuation = await buildExecutionContinuation({ - db, - companyId: run.companyId, - issueId: coordinator.issueId, - agentId: run.agentId, - context: record(run.contextSnapshot), - summary: null, - exposeLowTrustRaw: false, - }); - if (continuation.unresolvedInteractionIds.length > 0) continue; - historyComplete = true; - } catch { - /* The named operator outcome retains incomplete context. */ - } + const history = await replacementHistoryReady(db, run, coordinator.issueId); + if (history === false) continue; + const historyComplete = history === true; const remoteWorkspace = await remoteWorkspaceReplacementEvidence(db, run, coordinator.issueId, leases); + if (historicalListener && (!remoteWorkspace?.workspacePreserved || !remoteWorkspace.providerStopped || invocations.length)) continue; const decision = decideNativeReplacement({ failedSession: true, // A facade transport failure can conceal an authoritative protocol @@ -413,6 +540,13 @@ export async function reconcileSafeNativeReplacements( current.attempt >= 3 ) return false; + let historicalRun: typeof heartbeatRuns.$inferSelect | undefined; + if (historicalListener) { + const [currentRun] = await tx.select().from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), + )).for("update"); + historicalRun = currentRun; + } if (task.status === "blocked") { const failureHolds = await tx.select().from(issueRecoveryActions).where(and( eq(issueRecoveryActions.companyId, run.companyId), @@ -427,6 +561,14 @@ export async function reconcileSafeNativeReplacements( }); if (!ownsBlock) return false; } + const preservationHolds = historicalListener ? await tx.select().from(issueRecoveryActions).where(and( + eq(issueRecoveryActions.companyId, run.companyId), eq(issueRecoveryActions.sourceIssueId, task.id), + eq(issueRecoveryActions.kind, "active_run_watchdog"), + eq(issueRecoveryActions.cause, "native_continuation_requires_reconciliation"), + sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`, + sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`, + )).for("update") : []; + if (historicalListener && (!historicalRun || !await historicalListenerGuardEvidence(tx, historicalRun))) return false; if (remoteWorkspace) { const currentWorkspace = await remoteWorkspaceReplacementEvidence(tx, run, coordinator.issueId, leases, true); if (!currentWorkspace?.workspacePreserved || !currentWorkspace.providerStopped) return false; @@ -548,6 +690,7 @@ export async function reconcileSafeNativeReplacements( eq(issueRecoveryActions.cause, current.failureCode!), sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`, )).for("update"); + holds.push(...preservationHolds); if (holds.some(hold => hold.status === "active" || hold.status === "escalated")) { await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: run.companyId, issueId: task.id }); } @@ -587,11 +730,17 @@ export async function reconcileSafeNativeReplacements( } } catch (error) { if (options.failpoint) throw error; + pendingFailures++; logger.warn( { runId: run.id }, "Native replacement remains pending; another sweep will retry its durable decision", ); } } - return { scanned: candidates.length, scheduled }; + settleReplacementDiscovery(discovery, selection.after, generation); + if (pendingFailures) { + discovery.complete = false; discovery.rescan = true; + throw new Error("native_replacement_decision_retry_required"); + } + return { scanned: selection.scanned, scheduled }; } diff --git a/server/src/services/remote-workspaces.integration.test.ts b/server/src/services/remote-workspaces.integration.test.ts index 912416f1457..9e45c05470b 100644 --- a/server/src/services/remote-workspaces.integration.test.ts +++ b/server/src/services/remote-workspaces.integration.test.ts @@ -13,7 +13,7 @@ import { pathToFileURL } from "node:url"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { and, eq, sql } from "drizzle-orm"; import { activityLog, agents, agentTaskSessions, agentWakeupRequests, applyPendingMigrations, closeRegisteredClients, companies, createDb, environmentLeases, environments, - executionWorkspaceRepositories, executionWorkspaces, executionWorkspaceSnapshots, heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, issues, livenessDb, nativeRunFinalizations, projects, projectWorkspaces, workspaceOperations } from "@paperclipai/db"; + subscribeDatabaseWork, executionWorkspaceRepositories, executionWorkspaces, executionWorkspaceSnapshots, heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, issueThreadInteractions, issues, livenessDb, nativeRunFinalizations, projects, projectWorkspaces, workspaceOperations } from "@paperclipai/db"; import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target"; import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime"; import { buildProjectMentionHref, type Environment, type EnvironmentLease } from "@paperclipai/shared"; @@ -25,12 +25,13 @@ import { createSavedWorkspaceLoader, prepareRemoteWorkspace, readRemoteWorkspace clearRemoteWorkspaceLocationForReleasedLease, createRemoteWorkspaceGitAuth, prepareRemoteWorkspaceRepository, reserveRemoteWorkspace, saveRemoteWorkspace, shouldUseRemoteWorkspaces } from "./remote-workspaces.js"; import type { createGitRemoteAuthProvider } from "./git-credentials.js"; -import { reconcileSafeNativeReplacements } from "./native-runtime/native-safe-replacement.js"; +import { invalidateNativeReplacementDiscovery, reconcileSafeNativeReplacements } from "./native-runtime/native-safe-replacement.js"; import { remoteTerminationReceipt } from "./remote-execution-termination.js"; import { RemoteWorkspaceRestoreError } from "./remote-workspace-transport.js"; import { readRemoteRunnerState } from "./native-runtime/native-session-executor.js"; import { notFound } from "../errors.js"; import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js"; +import { issueThreadInteractionService } from "./issue-thread-interactions.js"; import { createRemoteWorkspaceFiles } from "./remote-workspace-files.js"; import { workspaceFileResourceService } from "./workspace-file-resources.js"; import { nativeWorkspaceExportPendingForLease, preserveRemoteWorkspacePreparationFailure } from "./native-runtime/native-workspace-finalization-state.js"; @@ -47,6 +48,10 @@ import { instanceSettingsService } from "./instance-settings.js"; import { resolveRemoteWorkspaceSource } from "./heartbeat/remote-workspace-source.js"; import { toExecutionWorkspace } from "./execution-workspaces.js"; import { recoveryPostgresSupport, startRecoveryPostgres } from "../__tests__/helpers/recovery-postgres.js"; +import { getExecutionBlocker } from "./execution-blocker.js"; +import { executionControlSchedule } from "./execution-control-work.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import { createSystemRecovery } from "./system-recovery-state.js"; const exec = promisify(execFile); let databaseUrl = process.env.PAPERCLIP_REMOTE_WORKSPACE_TEST_DATABASE_URL; @@ -82,6 +87,7 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem await db.insert(issues).values({ id: ids.issue, companyId: ids.company, title: "Use a remote workspace", status: "in_progress", assigneeAgentId: ids.agent }); await db.insert(heartbeatRuns).values({ id: ids.run, companyId: ids.company, agentId: ids.agent, nativeIssueId: ids.issue, runtimeMode: "native", status: "running" }); + invalidateNativeReplacementDiscovery(db); await db.insert(environmentLeases).values({ id: ids.lease, companyId: ids.company, environmentId: ids.environment, heartbeatRunId: ids.run, issueId: ids.issue, provider: "daytona", providerLeaseId: `allocation-${ids.lease}`, status: "active" }); }); @@ -91,6 +97,10 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem ? (row.metadata.uploads as { objectKey: string }[]).map(item => item.objectKey) : [])].filter((value): value is string => !!value))) { await storage.deleteObject(ids.company, key); } + if (temporary && process.env.PAPERCLIP_TEST_HISTORICAL_LISTENER === "1") await db.transaction(async tx => { + await tx.execute(sql`select set_config('client_min_messages', 'warning', true)`); + await tx.execute(sql`TRUNCATE companies CASCADE`); + }); await db.delete(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.companyId, ids.company)); await db.delete(workspaceOperations).where(eq(workspaceOperations.companyId, ids.company)); await db.delete(activityLog).where(eq(activityLog.companyId, ids.company)); @@ -1712,14 +1722,20 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem } finally { await recovered.cleanup(); } }, 60_000); - it.each(["saved", "dirty", "unsaved", "foreign_snapshot", "delete_requested", "deletion_race", "controller_claim_race", "stale_epoch", "unconfirmed_stop", "unknown_action"] as const)( - "checks durable remote work at the native safe-replacement decision (%s)", async scenario => { + it.each(["saved", "dirty", "unsaved", "foreign_snapshot", "delete_requested", "deletion_race", "controller_claim_race", "stale_epoch", "unconfirmed_stop", "unknown_action", "historical_listener_guard", "historical_listener_provider_event", "historical_listener_checkpoint", "historical_listener_resumed", "historical_listener_other_message", "historical_listener_cancelled", "historical_listener_prior_session", "historical_listener_unconfirmed_stop", "historical_listener_dirty", "historical_listener_stale_block", "historical_listener_unknown_event", "historical_listener_budget", "historical_listener_bad_preservation", "historical_listener_absent", "historical_listener_later_candidate", "historical_listener_late_proof", "historical_listener_lock_timeout", "historical_listener_late_interaction", "historical_listener_interaction_paused", "historical_listener_own_question"] as const)( + "checks durable remote work at the native safe-replacement decision (%s)", async kind => { + const historicalListener = kind.startsWith("historical_listener_"); + const processAcceptance = kind === "historical_listener_guard" && process.env.PAPERCLIP_TEST_HISTORICAL_LISTENER === "1"; + const scenario = kind === "historical_listener_late_proof" ? "dirty" : kind === "historical_listener_unconfirmed_stop" ? "unconfirmed_stop" + : kind === "historical_listener_dirty" ? "dirty" : historicalListener ? "saved" : kind; + const shouldSchedule = scenario === "saved" && (!historicalListener || ["historical_listener_guard", "historical_listener_later_candidate", "historical_listener_lock_timeout"].includes(kind)); const reserved = await reserve({ kind: "empty" }), first = await target("native-replacement-sandbox"); const prepared = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId: ids.run, lease: await lease(), target: first, assertOwned: async () => undefined }); await fs.writeFile(path.join(prepared.remoteRoot, "saved.txt"), "work survives a failed provider session\n"); if (scenario !== "unsaved") await saveRemoteWorkspace({ db, storage, workspace: prepared.workspace, - runId: ids.run, lease: prepared.lease, target: first, assertOwned: async () => undefined }); + runId: ids.run, lease: prepared.lease, target: first, assertOwned: async () => undefined, + boundary: kind === "historical_listener_late_proof" ? "initial" : "final" }); const current = await workspace(reserved.id), authority = readRemoteWorkspaceAuthority(current.metadata)!; if (scenario === "dirty" || scenario === "stale_epoch") await db.update(executionWorkspaces).set({ metadata: { ...current.metadata, remoteWorkspace: { ...authority, ...(scenario === "dirty" ? { dirty: true, writerRunId: ids.run } @@ -1729,29 +1745,137 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem .where(eq(executionWorkspaceSnapshots.id, authority.snapshotId!)); if (scenario === "delete_requested") await db.update(executionWorkspaceSnapshots).set({ status: "delete_requested" }) .where(eq(executionWorkspaceSnapshots.id, authority.snapshotId!)); - const stopped = (await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success", + if (processAcceptance) { + const { recordHistoricalListenerFailure } = await import("../__tests__/helpers/historical-listener-continuation.js"); + await recordHistoricalListenerFailure({ db, connectionString: databaseUrl!, runId: ids.run, cwd: prepared.remoteRoot, + environmentId: ids.environment, leaseId: ids.lease }); + } + const stopped = kind === "historical_listener_late_proof" ? prepared.lease : (await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success", metadata: { ...prepared.lease.metadata, ...(scenario !== "unconfirmed_stop" ? { remoteExecutionTermination: remoteTerminationReceipt(prepared.lease, { providerLeaseId: prepared.lease.providerLeaseId, state: "destroyed", }), } : {}) }, }).where(eq(environmentLeases.id, ids.lease)).returning())[0]! as EnvironmentLease; - if (scenario === "saved") { + if (shouldSchedule) { await db.transaction(tx => clearRemoteWorkspaceLocationForReleasedLease({ db: tx, lease: stopped })); await fs.rm(prepared.remoteRoot, { recursive: true, force: true }); await expect(fs.stat(prepared.remoteRoot)).rejects.toThrow(); } const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, ids.run)); - await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(), runnerProfileJson: { + const sessionId = randomUUID(); + await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(), + ...(historicalListener && !processAcceptance ? { nativeSessionId: sessionId } : {}), + ...(historicalListener ? { error: "local_runtime_api_listener_unavailable", errorCode: "native_session_interrupted" } : {}), runnerProfileJson: { ...source!.runnerProfileJson, recoveryEventInventoryVersion: 1, - nativeExecutionInput: { provider: { kind: "codex" }, workspace: { cwd: prepared.remoteRoot } }, + nativeExecutionInput: processAcceptance ? source!.runnerProfileJson!.nativeExecutionInput : { provider: { kind: "codex" }, workspace: { cwd: prepared.remoteRoot }, + ...(historicalListener ? { binding: { runId: ids.run, companyId: ids.company, agentId: ids.agent, issueId: ids.issue }, + session: { normalizedSessionId: sessionId } } : {}) }, } }).where(eq(heartbeatRuns.id, ids.run)); - await db.insert(nativeRunFinalizations).values({ runId: ids.run, companyId: ids.company, issueId: ids.issue, - phase: "terminal_failure", attempt: 1, failureCode: "native_provider_terminal_failed", - failureDetail: { originalFailureCode: "fixture_checkpoint_unusable" } }); + if (!processAcceptance) await db.insert(nativeRunFinalizations).values({ runId: ids.run, companyId: ids.company, issueId: ids.issue, + phase: "terminal_failure", attempt: 1, failureCode: historicalListener ? "native_session_interrupted" : "native_provider_terminal_failed", + failureDetail: historicalListener ? { originalFailureCode: "native_session_interrupted", message: "local_runtime_api_listener_unavailable", + recoveryMode: "ambiguous_state", providerSessionEstablished: false, providerEventsExist: false, checkpointExists: false } + : { originalFailureCode: "fixture_checkpoint_unusable" } }); + if (historicalListener && !processAcceptance) { + const blocked = await issueService(db).update(ids.issue, { status: "blocked" }); + await db.insert(issueRecoveryActions).values({ companyId: ids.company, sourceIssueId: ids.issue, + kind: "active_run_watchdog", cause: "native_session_interrupted", ownerType: "board", status: "active", + fingerprint: createHash("sha256").update(`${ids.run}:native_session_interrupted`).digest("hex"), + evidence: { runId: ids.run, nativeFailureBlock: { runId: ids.run, statusVersion: blocked!.statusVersion } }, + nextAction: "Reconcile the original startup failure." }); + } + if (kind === "historical_listener_guard" || kind === "historical_listener_bad_preservation") { + const actionId = randomUUID(), preservationId = randomUUID(); + await db.insert(issueRecoveryActions).values({ id: actionId, companyId: ids.company, sourceIssueId: ids.issue, + kind: "active_run_watchdog", cause: "native_continuation_requires_reconciliation", ownerType: "system", status: "resolved", outcome: "blocked", + fingerprint: `native-workspace:${ids.run}`, evidence: { runId: ids.run, + workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [ids.lease] }, + automaticRecovery: { replay: "blocked" }, continuationDelivery: "invalidated", + systemRecovery: { ...createSystemRecovery(ids.run), state: "complete", preservationOperationIds: [preservationId] }, + workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: ids.run, status: "recovered", stage: "saving_receipts", attempt: 1, requestIds: [] }, + }, nextAction: "Files were preserved without replay." }); + await db.insert(workspaceOperations).values([{ id: preservationId, companyId: ids.company, heartbeatRunId: ids.run, issueId: ids.issue, + executionWorkspaceId: reserved.id, phase: "workspace_recovery", status: kind === "historical_listener_bad_preservation" ? "failed" : "succeeded", startedAt: new Date(), + metadata: { recoveryActionId: actionId, snapshotId: authority.snapshotId } }, + { companyId: ids.company, heartbeatRunId: ids.run, issueId: ids.issue, phase: "workspace_finalize", status: "succeeded", startedAt: new Date(), + metadata: { owningService: "workspace_recovery", preservationOperationIds: [preservationId] } }]); + } + if (kind === "historical_listener_absent") await db.update(heartbeatRuns).set({ + runnerProfileJson: sql`${heartbeatRuns.runnerProfileJson} - 'remoteWorkspace'`, + }).where(eq(heartbeatRuns.id, ids.run)); + if (kind === "historical_listener_stale_block") await issueService(db).update(ids.issue, { status: "blocked", actorUserId: "operator" }); + if (kind === "historical_listener_unknown_event") await db.insert(heartbeatRunEvents).values({ companyId: ids.company, runId: ids.run, + agentId: ids.agent, seq: 100, eventType: "future.provider.activity", sourceEventId: "source-provider-event" }); + if (kind === "historical_listener_budget") await db.update(nativeRunFinalizations).set({ attempt: 3 }).where(eq(nativeRunFinalizations.runId, ids.run)); + if (kind === "historical_listener_checkpoint") await db.update(heartbeatRuns).set({ runnerProfileJson: { + ...((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, ids.run)))[0]!.runnerProfileJson), sessionCheckpoint: { providerSessionId: "already-started" }, + } }).where(eq(heartbeatRuns.id, ids.run)); + if (kind === "historical_listener_resumed") await db.update(heartbeatRuns).set({ sessionIdBefore: "prior-session" }).where(eq(heartbeatRuns.id, ids.run)); + if (kind === "historical_listener_prior_session") await db.insert(heartbeatRuns).values({ id: randomUUID(), companyId: ids.company, + agentId: ids.agent, status: "succeeded", nativeSessionId: sessionId }); + if (kind === "historical_listener_other_message") await db.update(nativeRunFinalizations).set({ failureDetail: { + message: "provider reported local_runtime_api_listener_unavailable", originalFailureCode: "native_session_interrupted", recoveryMode: "ambiguous_state", + providerSessionEstablished: false, providerEventsExist: false, checkpointExists: false, + } }).where(eq(nativeRunFinalizations.runId, ids.run)); + if (kind === "historical_listener_cancelled") await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, ids.issue)); + if (kind === "historical_listener_provider_event") await appendHeartbeatRunEvent(db, { companyId: ids.company, runId: ids.run, + agentId: ids.agent, eventType: "turn.started", stream: "system" }); + const interactionWake = ["historical_listener_late_interaction", "historical_listener_interaction_paused", "historical_listener_own_question"].includes(kind); + const interactionId = randomUUID(); + if (interactionWake) { + const priorRunId = randomUUID(); + await db.insert(heartbeatRuns).values({ id: priorRunId, companyId: ids.company, agentId: ids.agent, + nativeIssueId: ids.issue, status: "succeeded", contextSnapshot: { issueId: ids.issue } }); + await db.insert(issueThreadInteractions).values({ id: interactionId, companyId: ids.company, issueId: ids.issue, + kind: "request_confirmation", status: "pending", sourceRunId: kind === "historical_listener_own_question" ? ids.run : priorRunId, + payload: { version: 1, prompt: "An obsolete question from earlier work." } }); + if (kind === "historical_listener_interaction_paused") await issueService(db).update(ids.issue, { status: "backlog" }); + if (kind === "historical_listener_own_question") await appendHeartbeatRunEvent(db, { companyId: ids.company, runId: ids.run, + agentId: ids.agent, eventType: "tool.execution.started", stream: "system", + payload: { name: "request_confirmation", executionId: "source-question", transport: "process" } }); + } if (scenario === "unknown_action") await appendHeartbeatRunEvent(db, { companyId: ids.company, runId: ids.run, agentId: ids.agent, eventType: "tool.execution.started", stream: "system", payload: { name: "send_email", executionId: "unconfirmed-external-action", transport: "process" } }); + if (kind === "historical_listener_later_candidate") { + const sourceNumber = BigInt(`0x${ids.run.replaceAll("-", "")}`); + const parked = Array.from({ length: 100 }, (_, index) => { + const hex = (sourceNumber - BigInt(index + 1)).toString(16).padStart(32, "0"); + return { runId: `${hex.slice(0,8)}-${hex.slice(8,12)}-${hex.slice(12,16)}-${hex.slice(16,20)}-${hex.slice(20)}`, + issueId: randomUUID(), sessionId: randomUUID() }; + }); + await db.insert(issues).values(parked.map(row => ({ id: row.issueId, companyId: ids.company, + title: "Preserve unavailable source", status: "in_progress", assigneeAgentId: ids.agent }))); + await db.insert(heartbeatRuns).values(parked.map(row => ({ id: row.runId, companyId: ids.company, agentId: ids.agent, + nativeIssueId: row.issueId, nativeSessionId: row.sessionId, runtimeMode: "native", status: "failed", + errorCode: "native_session_interrupted", error: "local_runtime_api_listener_unavailable", runnerProfileJson: { + nativeExecutionInput: { binding: { runId: row.runId, companyId: ids.company, agentId: ids.agent, issueId: row.issueId }, + session: { normalizedSessionId: row.sessionId } }, + } }))); + await db.insert(nativeRunFinalizations).values(parked.map(row => ({ runId: row.runId, companyId: ids.company, issueId: row.issueId, + phase: "terminal_failure", attempt: 1, failureCode: "native_session_interrupted", failureDetail: { + message: "local_runtime_api_listener_unavailable", originalFailureCode: "native_session_interrupted", recoveryMode: "ambiguous_state", + providerSessionEstablished: false, providerEventsExist: false, checkpointExists: false, + } }))); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scanned: 100, scheduled: 0 }); + expect(await executionControlSchedule(db).replacement()).not.toBeNull(); + invalidateNativeReplacementDiscovery(db); + } + if (kind === "historical_listener_lock_timeout") { + let release!: () => void, announce!: () => void; + const locked = new Promise(resolve => { announce = resolve; }); + const released = new Promise(resolve => { release = resolve; }); + const holder = db.transaction(async tx => { + await tx.select({ id: issues.id }).from(issues).where(eq(issues.id, ids.issue)).for("update"); + announce(); await released; + }); + await locked; + try { + await expect(reconcileSafeNativeReplacements(db)).rejects.toThrow(); + expect(await executionControlSchedule(db).replacement()).not.toBeNull(); + } finally { release(); await holder; } + } let replacement; if (scenario === "deletion_race" || scenario === "controller_claim_race") { let release!: () => void, announce!: () => void; @@ -1783,12 +1907,102 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem expect(blocked).toBe(true); } finally { release(); await deletion; replacement = await pending; } } else replacement = await reconcileSafeNativeReplacements(db); - expect(replacement).toMatchObject({ scheduled: scenario === "saved" ? 1 : 0 }); + expect(replacement).toMatchObject({ scheduled: shouldSchedule ? 1 : 0 }); const successors = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, ids.run)); - expect(successors).toHaveLength(scenario === "saved" ? 1 : 0); - if (scenario === "saved") expect(successors[0]).toMatchObject({ status: "scheduled_retry", + expect(successors).toHaveLength(shouldSchedule ? 1 : 0); + if (shouldSchedule) expect(successors[0]).toMatchObject({ status: "scheduled_retry", contextSnapshot: { retryOfRunId: ids.run, forceFreshSession: true } }); - else expect(await fs.readFile(path.join(prepared.remoteRoot, "saved.txt"), "utf8")) + if (kind === "historical_listener_later_candidate") { + for (let pass = 0; pass < 3 && await executionControlSchedule(db).replacement() !== null; pass++) { + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 }); + } + expect(await executionControlSchedule(db).replacement()).toBeNull(); + } + if (kind === "historical_listener_guard") expect(await getExecutionBlocker(db, ids.company, ids.issue)).toBeNull(); + if (["historical_listener_provider_event", "historical_listener_checkpoint", "historical_listener_resumed", "historical_listener_other_message", + "historical_listener_cancelled", "historical_listener_prior_session", "historical_listener_unknown_event", "historical_listener_budget", "historical_listener_stale_block", "historical_listener_bad_preservation", "historical_listener_dirty", "historical_listener_unconfirmed_stop", "historical_listener_absent", "historical_listener_late_proof"].includes(kind)) { + expect(await executionControlSchedule(db).replacement()).toBeNull(); + } + if (interactionWake) { + expect(await executionControlSchedule(db).replacement()).toBeNull(); + let wakes = 0; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.executionReplacement, event => { + if (event === "settled") { wakes++; invalidateNativeReplacementDiscovery(db); } + }); + try { + await issueThreadInteractionService(db).skipInteraction({ id: ids.issue, companyId: ids.company, + status: kind === "historical_listener_interaction_paused" ? "backlog" : "blocked" }, interactionId, {}, { userId: "local-board" }); + expect(wakes).toBeGreaterThan(0); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: kind === "historical_listener_late_interaction" ? 1 : 0 }); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 }); + expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, ids.run))) + .toHaveLength(kind === "historical_listener_late_interaction" ? 1 : 0); + expect(await executionControlSchedule(db).replacement()).toBeNull(); + } finally { unsubscribe(); } + } + if (kind === "historical_listener_late_proof") { + let wakes = 0; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.executionReplacement, event => { + if (event === "settled") { wakes++; invalidateNativeReplacementDiscovery(db); } + }); + try { + const saved = await saveRemoteWorkspace({ db, storage, workspace: await workspace(reserved.id), runId: ids.run, + lease: await lease(), target: first, assertOwned: async () => undefined }); + expect(wakes).toBeGreaterThan(0); + expect(await executionControlSchedule(db).replacement()).toBeNull(); + const priorWakes = wakes, active = await lease(), pluginId = randomUUID(); + await db.update(environmentLeases).set({ metadata: { ...active.metadata, driver: "sandbox", pluginId, + provider: "daytona", sandboxProviderPlugin: true } }).where(eq(environmentLeases.id, ids.lease)); + const runtime = environmentRuntimeService(db, { pluginWorkerManager: { + isRunning: () => true, getWorker: () => ({ supportedMethods: ["environmentReleaseLease"] }), + call: async (_id: string, method: string) => { + expect(method).toBe("environmentReleaseLease"); + return { providerLeaseId: active.providerLeaseId, state: "destroyed" }; + }, + } as never }); + await runtime.releaseRunLeases(ids.run, "failed"); + expect(wakes).toBeGreaterThan(priorWakes); + expect(await executionControlSchedule(db).replacement()).not.toBeNull(); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 1 }); + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 }); + expect(await executionControlSchedule(db).replacement()).toBeNull(); + const restored = await createSavedWorkspaceLoader(db, storage)({ companyId: ids.company, workspaceId: reserved.id, snapshotId: saved.snapshotId }); + try { expect(await fs.readFile(path.join(restored.root, "saved.txt"), "utf8")).toBe("work survives a failed provider session\n"); } + finally { await restored.cleanup(); } + } finally { unsubscribe(); } + } + if (kind === "historical_listener_bad_preservation") { + expect(await getExecutionBlocker(db, ids.company, ids.issue)).toMatchObject({ cause: "native_continuation_requires_reconciliation" }); + + } + if (processAcceptance) { + expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 }); + const { completeHistoricalListenerSuccessor } = await import("../__tests__/helpers/historical-listener-continuation.js"); + await completeHistoricalListenerSuccessor({ db, runId: successors[0]!.id, root: path.join(root, "successor-runtime"), + prepareWorkspace: async () => { + const leaseId = randomUUID(), runId = successors[0]!.id; + const [nextLease] = await db.insert(environmentLeases).values({ id: leaseId, companyId: ids.company, + environmentId: ids.environment, heartbeatRunId: runId, issueId: ids.issue, provider: "daytona", + providerLeaseId: `allocation-${leaseId}`, status: "active" }).returning(); + const next = await reserveRemoteWorkspace({ db, storage, companyId: ids.company, issueId: ids.issue, + runId, agentId: ids.agent, environmentId: ids.environment, projectId: null, projectWorkspaceId: null, + existingWorkspaceId: reserved.id, source: { kind: "empty" } }); + const nextTarget = { ...await target("historical-successor"), leaseId, sandboxId: nextLease.providerLeaseId! }; + const nextPrepared = await prepareRemoteWorkspace({ db, storage, workspace: next, runId, + lease: nextLease as EnvironmentLease, target: nextTarget }); + return { cwd: nextPrepared.remoteRoot, finalize: async () => { + const { workspaceOperationService } = await import("./workspace-operations.js"); + const recorder = workspaceOperationService(db).createRecorder({ companyId: ids.company, issueId: ids.issue, heartbeatRunId: runId }); + const saved = await recorder.recordOperation({ phase: "workspace_finalize", cwd: nextPrepared.remoteRoot, run: async () => { + await saveRemoteWorkspace({ db, storage, workspace: nextPrepared.workspace, runId, lease: nextPrepared.lease, + target: nextTarget, assertOwned: async () => undefined }); + return { status: "succeeded", exitCode: 0 }; + } }); + expect(saved.status).toBe("succeeded"); + } }; + } }); + } + if (!shouldSchedule) expect(await fs.readFile(path.join(prepared.remoteRoot, "saved.txt"), "utf8")) .toBe("work survives a failed provider session\n"); if (scenario === "unknown_action") { const [decision] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, ids.run)); diff --git a/server/src/services/remote-workspaces.ts b/server/src/services/remote-workspaces.ts index 63aa0bfe1e1..fbdd45e1e64 100644 --- a/server/src/services/remote-workspaces.ts +++ b/server/src/services/remote-workspaces.ts @@ -1,3 +1,4 @@ +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { workspaceAllocationFiles } from "./remote-workspace-allocation.js"; import { readNativeWorkspaceMigration } from "./native-runtime/native-workspace-migration.js"; import { createHash, randomUUID } from "node:crypto"; @@ -763,7 +764,10 @@ export async function saveRemoteWorkspace(input: { eq(executionWorkspaceSnapshots.executionWorkspaceId, workspace.id), eq(executionWorkspaceSnapshots.status, "uploaded"))).returning({ id: executionWorkspaceSnapshots.id }); if (!committed) throw new Error("remote_workspace_snapshot_publication_changed"); await notifyRemoteWorkspaceSnapshots(tx, `workspace:${workspace.id}`); - if (boundary === "final") await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`); + if (boundary === "final") { + await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`); + await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionReplacement); + } const receipt = object(workspace.metadata?.remoteWorkspaceSnapshotRepositories); const priorRepositories = receipt.snapshotId === current.snapshotId && Array.isArray(receipt.repositories) ? remoteWorkspaceSourceSchema.parse({ kind: "empty", repositories: receipt.repositories }).repositories ?? [] : []; diff --git a/server/src/services/run-dispatch-notifications.ts b/server/src/services/run-dispatch-notifications.ts index 3cd4f00cbe1..36fad0e81a3 100644 --- a/server/src/services/run-dispatch-notifications.ts +++ b/server/src/services/run-dispatch-notifications.ts @@ -12,6 +12,7 @@ export async function notifyRunDispatch(transaction: object, scope?: RunDispatch await notifyIssueRecovery(transaction, scope ? { companyId: scope.companyId } : undefined); const hint = scope ? JSON.stringify({ companyId: scope.companyId }) : "*"; await notifyDeliveryWork(transaction, DELIVERY_QUEUES.nativeProcessSessions, hint); + await notifyDeliveryWork(transaction, DELIVERY_QUEUES.executionReplacement, hint); await notifyQueuedRunDispatch(transaction, scope); } diff --git a/server/src/services/task-interruption-presentation.test.ts b/server/src/services/task-interruption-presentation.test.ts index adbc79479a5..fab317dc824 100644 --- a/server/src/services/task-interruption-presentation.test.ts +++ b/server/src/services/task-interruption-presentation.test.ts @@ -80,6 +80,24 @@ describe("unadmitted chat Retry presentation", () => { }); }); +// The native coordinator resumes the same run itself after an attach failure. +// A manual Retry offered meanwhile would race that automatic resume. +it("shows a pending native same-run resume instead of a competing Retry", () => { + const now = Date.parse("2026-11-06T15:25:50.000Z"); + const failed = { ...run, runtimeMode: "native" as const, errorCode: "native_session_interrupted", resultJson: null }; + const resumeAt = new Date("2026-11-06T15:26:18.687Z"); + expect(projectTaskInterruption(issue, { run: { ...failed, nativeResumeAt: resumeAt }, agent, paused: false, approval: false }, now)) + .toMatchObject({ state: "retry_scheduled", canRetry: false, nextRetryAt: resumeAt.toISOString() }); + // A due resume is being dispatched; it is still not the user's turn. + expect(projectTaskInterruption(issue, { run: { ...failed, nativeResumeAt: new Date(now - 30_000) }, agent, paused: false, approval: false }, now)) + .toMatchObject({ state: "working", canRetry: false }); + // A resume that never ran, or no resume at all, leaves exactly one Retry. + expect(projectTaskInterruption(issue, { run: { ...failed, nativeResumeAt: new Date(now - 10 * 60_000) }, agent, paused: false, approval: false }, now)) + .toMatchObject({ state: "failed", canRetry: true }); + expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false }, now)) + .toMatchObject({ state: "failed", canRetry: true }); +}); + it("explains a persisted Retry wait without exposing the saved payload", () => { const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null }; const wait = { id: "retry", requestedAt: new Date(), sourceRunId: failed.id, diff --git a/server/src/services/task-interruption-presentation.ts b/server/src/services/task-interruption-presentation.ts index 59797d3958e..dec56770f76 100644 --- a/server/src/services/task-interruption-presentation.ts +++ b/server/src/services/task-interruption-presentation.ts @@ -1,6 +1,6 @@ import { and, desc, eq, inArray, isNull, sql, type SQL } from "drizzle-orm"; import { agents, agentWakeupRequests, approvals, chatActions, companies, heartbeatRuns, issueApprovals, issueRecoveryActions, - issues, issueTreeHoldMembers, issueTreeHolds, projects, type Db } from "@paperclipai/db"; + issues, issueTreeHoldMembers, issueTreeHolds, nativeRunFinalizations, projects, type Db } from "@paperclipai/db"; import { unionAll } from "drizzle-orm/pg-core"; import { HEARTBEAT_RUN_STATUSES, readProviderFailure, type TaskInterruptionPresentation } from "@paperclipai/shared"; import { captureTaskInterruptionFailure } from "../sentry.js"; @@ -26,11 +26,17 @@ import { AI_CONNECTION_BUSY_RETRY_REASON, AI_CONNECTION_POOL_WAIT_RETRY_REASON, const resourceWaitRetryReasons = new Set([WORKSPACE_BUSY_RETRY_REASON, AI_CONNECTION_BUSY_RETRY_REASON, AI_CONNECTION_POOL_WAIT_RETRY_REASON, RUNNER_REVISION_WAIT_RETRY_REASON]); +// A due same-run resume dispatches within seconds; past this, offer Retry. +const NATIVE_RESUME_DISPATCH_GRACE_MS = 2 * 60_000; + type ProjectionIssue = Pick; type ProjectionRun = Pick & Partial>; + | "scheduledRetryReason">> & { + /** Due time of the native coordinator's own same-run resume, if one is pending. */ + nativeResumeAt?: Date | null; +}; type ProjectionWait = { id: string; requestedAt: Date; sourceRunId?: string | null; executionWaitReason?: string | null; preparationStatus?: string | null; preparationRetryAt?: number | string | null }; type ProjectionState = { @@ -148,10 +154,16 @@ export function projectTaskInterruption(issue: ProjectionIssue, input: Projectio } if (run.status === "scheduled_retry" && resourceWaitRetryReasons.has(run.scheduledRetryReason ?? "") && !system) return { ...base, state: "working" }; - const retryAt = system?.nextAttemptAt ?? run.scheduledRetryAt?.getTime(); + // The native coordinator resumes this same run by itself. A manual Retry + // would race it, so show the resume until it is due and briefly while its + // dispatch runs. A resume that never ran falls through to one Retry. + const nativeResumeAt = run.nativeResumeAt?.getTime(); + const retryAt = system?.nextAttemptAt ?? nativeResumeAt ?? run.scheduledRetryAt?.getTime(); if (retryAt && retryAt > now) return { ...base, state: "retry_scheduled", - nextRetryAt: new Date(retryAt).toISOString(), canRetry: !preparationPending, + nextRetryAt: new Date(retryAt).toISOString(), canRetry: !preparationPending && !nativeResumeAt, ...(system?.nextAttemptAt && recoveryStep ? { recoveryStep } : {}) }; + if (nativeResumeAt && !system && now - nativeResumeAt < NATIVE_RESUME_DISPATCH_GRACE_MS) + return { ...base, state: "working" }; if (recoveryRunning) return { ...base, state: "working", ...(recoveryStep ? { recoveryStep } : {}) }; if (retryWait?.sourceRunId === run.id && retryWait.executionWaitReason) { const reason = Object.hasOwn(retryWaitMessages, retryWait.executionWaitReason) ? retryWait.executionWaitReason : "execution_recovery"; @@ -203,6 +215,10 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i or ${heartbeatRuns.errorCode} = 'workspace_validation_failed' then ${heartbeatRuns.error} else null end`, errorCode: heartbeatRuns.errorCode, livenessState: heartbeatRuns.livenessState, scheduledRetryAt: heartbeatRuns.scheduledRetryAt, scheduledRetryReason: heartbeatRuns.scheduledRetryReason, + nativeResumeAt: sql`(select ${nativeRunFinalizations.nextAttemptAt} from ${nativeRunFinalizations} + where ${nativeRunFinalizations.companyId} = ${heartbeatRuns.companyId} and ${nativeRunFinalizations.runId} = ${heartbeatRuns.id} + and ${heartbeatRuns.runtimeMode} = 'native' and ${heartbeatRuns.status} = 'failed' + and ${nativeRunFinalizations.resultId} is null and ${nativeRunFinalizations.phase} = 'retryable_failure')`.mapWith(value => value == null ? null : new Date(value)), // The run whose resultJson supplied providerFailure. A scheduled retry // shows its predecessor's reason, so redaction must use that run's registry. providerFailureRunId: sql`case when ${heartbeatRuns.resultJson}->'providerFailure' is null diff --git a/server/src/services/tool-access.ts b/server/src/services/tool-access.ts index 7441bcee493..ae1682df28c 100644 --- a/server/src/services/tool-access.ts +++ b/server/src/services/tool-access.ts @@ -4530,6 +4530,13 @@ export function toolAccessService( let applicationNamespace = input.applicationName ?? input.name; const transport = input.transport; if (!transport) throw badRequest("Tool connection transport is required"); + // This path writes only tool connections. Channel setup has its own + // services. Reject other purposes here, before any row is written, so the + // database purpose/transport check never turns them into a 500. + if (input.connectionPurpose !== undefined && input.connectionPurpose !== "tool") + throw unprocessable("Only tool connections can be created here"); + if (transport === "chat_sdk" || transport === "voice") + throw unprocessable("Chat and voice connections cannot be created here"); const config = normalizeGoogleSheetsConnectionConfig( input.config ?? input.transportConfig ?? {}, ); diff --git a/server/src/services/workspace-recovery.ts b/server/src/services/workspace-recovery.ts index 88f0efc82cc..b4d9a9d6595 100644 --- a/server/src/services/workspace-recovery.ts +++ b/server/src/services/workspace-recovery.ts @@ -390,6 +390,7 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ const activity = await db.transaction(async tx => { await notifyNativeResultCommit(tx, input); await notifyNativeWorkspaceFinalization(tx, input); + await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionReplacement); await notifyNativeStatusReassessment(tx, input); await tx.insert(workspaceOperations).values({ companyId: input.companyId, issueId: input.issueId, heartbeatRunId: input.runId, phase: "workspace_finalize", status: "succeeded", finishedAt: new Date(), diff --git a/tests/e2e/announcement-first-session.spec.ts b/tests/e2e/announcement-first-session.spec.ts new file mode 100644 index 00000000000..265b7924219 --- /dev/null +++ b/tests/e2e/announcement-first-session.spec.ts @@ -0,0 +1,105 @@ +import { test, expect, request as pwRequest, type APIRequestContext, type Page } from "@playwright/test"; + +/** + * E2E (N-13): the in-app announcement well ("Introducing Connectors" upsell) + * + * stays hidden while a new company's onboarding first task is open, so it + * cannot cover the onboarding questions or the account menu in the user's + * first session. Placement (never covering the account menu once shown) is + * covered by announcement-placement.spec.ts. + * + * The public feed is mocked so the spec does not depend on what the live feed + * is publishing; the card mirrors the real 2026-09-14-connectors entry. + */ + +const PORT = Number(process.env.PAPERCLIP_E2E_PORT ?? 3199); +const BASE_URL = `http://127.0.0.1:${PORT}`; +const ANNOUNCEMENT_ID = "e2e-n13-connectors"; +const SETTLE_WAIT_MS = 4_500; // ANNOUNCEMENT_SETTLE_MS (3 s) plus slack. +const ONE_PIXEL_PNG = Buffer.from( + "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=", + "base64", +); + +async function createCompany(board: APIRequestContext, name: string) { + const res = await board.post(`${BASE_URL}/api/companies`, { data: { name: `${name}-${Date.now()}` } }); + if (!res.ok()) throw new Error(`POST /api/companies → ${res.status()}: ${await res.text()}`); + const company = await res.json(); + // A company without agents lands on "Create your first agent"; add one so the normal shell renders. + const agentRes = await board.post(`${BASE_URL}/api/companies/${company.id}/agents`, { + data: { name: "Chief of staff", role: "ceo", adapterType: "process", adapterConfig: { command: "true" } }, + }); + if (!agentRes.ok()) throw new Error(`POST agents → ${agentRes.status()}: ${await agentRes.text()}`); + return { id: company.id as string, prefix: (company.issuePrefix ?? company.prefix) as string }; +} + +async function createIssue(board: APIRequestContext, companyId: string, data: Record) { + const res = await board.post(`${BASE_URL}/api/companies/${companyId}/issues`, { data }); + if (!res.ok()) throw new Error(`POST issues → ${res.status()}: ${await res.text()}`); + return res.json() as Promise<{ id: string; identifier: string; originKind: string | null }>; +} + +async function mockAnnouncementFeed(page: Page) { + await page.route("**/api/announcements/current", (route) => route.fulfill({ json: { + id: ANNOUNCEMENT_ID, + eyebrow: "Introducing Connectors", + title: "Give your agents the access they need", + description: "Connect Gmail, GitHub, Notion, and more. Choose which agents get access, which accounts they use, and which actions they can take.", + image: { path: `assets/${"a".repeat(64)}.png`, alt: "Paperclip agents with their own sets of connected apps." }, + primaryAction: { kind: "route", label: "Explore connectors", path: "/apps" }, + } })); + await page.route(`**/api/announcements/${ANNOUNCEMENT_ID}/image`, (route) => + route.fulfill({ contentType: "image/png", body: ONE_PIXEL_PNG })); + await page.route(`**/api/announcements/${ANNOUNCEMENT_ID}/dismiss`, (route) => route.fulfill({ status: 204 })); + await page.route("**/api/workspace-preferences/me", (route) => route.fulfill({ json: { + userId: "local-board", coordinatorWorkspaceEnabled: false, updatedAt: null, + } })); + await page.addInitScript(() => { + for (const key of Object.keys(window.localStorage)) { + if (key.includes("announcement")) window.localStorage.removeItem(key); + } + }); +} + +test.describe("Announcement well (N-13)", () => { + let board: APIRequestContext; + const companyIds: string[] = []; + + test.beforeAll(async () => { + board = await pwRequest.newContext({ baseURL: BASE_URL }); + }); + + test.afterAll(async () => { + for (const id of companyIds) await board.delete(`${BASE_URL}/api/companies/${id}`).catch(() => {}); + await board.dispose(); + }); + + test.beforeEach(async ({ page }) => { + await page.setViewportSize({ width: 1440, height: 900 }); + await mockAnnouncementFeed(page); + }); + + test("a brand-new company with its onboarding task open does not show the upsell", async ({ page }) => { + const company = await createCompany(board, "E2E-N13-Onboarding"); + companyIds.push(company.id); + const firstTask = await createIssue(board, company.id, { + title: "Plan our launch", status: "todo", onboardingFirstTask: true, + }); + expect(firstTask.originKind).toBe("onboarding_first_task"); + + await page.goto(`/${company.prefix}/issues/${firstTask.identifier}`); + await expect(page.getByRole("button", { name: "Open account menu" })).toBeVisible(); + await page.waitForTimeout(SETTLE_WAIT_MS); + await page.screenshot({ path: test.info().outputPath("onboarding-task.png") }); + const well = page.getByRole("complementary", { name: "Paperclip announcements" }); + await expect(well).toHaveCount(0); + + // Positive control on the same page: once onboarding finishes, the same + // mocked announcement renders without a reload. This proves the absence + // above comes from the onboarding gate, not from a broken feed mock. + const finish = await board.patch(`${BASE_URL}/api/issues/${firstTask.id}`, { data: { status: "done" } }); + if (!finish.ok()) throw new Error(`PATCH issue → ${finish.status()}: ${await finish.text()}`); + await expect(well).toBeVisible({ timeout: 20_000 }); + await expect(well.getByRole("link", { name: "Explore connectors" })).toBeVisible(); + }); +}); diff --git a/ui/src/components/AnnouncementWell.onboarding.test.tsx b/ui/src/components/AnnouncementWell.onboarding.test.tsx new file mode 100644 index 00000000000..f306991daa5 --- /dev/null +++ b/ui/src/components/AnnouncementWell.onboarding.test.tsx @@ -0,0 +1,76 @@ +// @vitest-environment jsdom +import { act } from "react"; +import { createRoot, type Root } from "react-dom/client"; +import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import type { HealthStatus } from "@/api/health"; +import { AnnouncementWell } from "./AnnouncementWell"; +import { announcementPreview } from "@/lib/announcement-preview"; + +// N-13: the connectors announcement must not appear in a new user's first +// session, i.e. while the company's onboarding first task is still open. +const state = vi.hoisted(() => ({ list: vi.fn() })); +vi.mock("@/api/companies-query", () => ({ useAccountIdentity: () => ({ userId: "alice", settled: true }) })); +vi.mock("@/context/CompanyContext", () => ({ useCompany: () => ({ selectedCompanyId: "company", loading: false }) })); +vi.mock("@/context/DialogContext", () => ({ useDialogState: () => ({ onboardingOpen: false }) })); +vi.mock("@/context/ToastContext", () => ({ useOptionalToastActions: () => null, useOptionalToastState: () => [] })); +vi.mock("@/api/issues", () => ({ issuesApi: { list: state.list } })); +vi.mock("@/hooks/useAnnouncement", () => ({ useAnnouncement: (options: { enabled: boolean }) => ({ + announcement: options.enabled ? announcementPreview : null, dismiss: vi.fn(), +}) })); +vi.mock("./AnnouncementCard", () => ({ AnnouncementCard: () =>

Announcement fixture

})); +(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true; + +const firstTask = (status: string) => ({ id: "issue-1", companyId: "company", originKind: "onboarding_first_task", status }); + +describe("announcement well during onboarding", () => { + let container: HTMLDivElement; + let root: Root; + const health = { deploymentMode: "authenticated" } as HealthStatus; + const visible = () => Boolean(container.querySelector('[aria-label="Paperclip announcements"]')); + let client: QueryClient; + const flush = async () => { await act(async () => { await new Promise((resolve) => setTimeout(resolve, 0)); }); }; + const render = async () => { + client = new QueryClient({ defaultOptions: { queries: { retry: false } } }); + await act(async () => root.render()); + await act(async () => { await new Promise((resolve) => setTimeout(resolve, 0)); }); + }; + beforeEach(() => { + state.list.mockReset(); + container = document.createElement("div"); document.body.append(container); root = createRoot(container); + }); + afterEach(async () => { await act(async () => root.unmount()); container.remove(); }); + + it("stays hidden while the onboarding first task is open", async () => { + state.list.mockResolvedValue([firstTask("todo")]); + await render(); + expect(visible()).toBe(false); + }); + + it("shows once the onboarding first task is finished, or when the company never had one", async () => { + state.list.mockResolvedValue([firstTask("done")]); + await render(); + expect(visible()).toBe(true); + await act(async () => root.unmount()); root = createRoot(container); + state.list.mockResolvedValue([]); + await render(); + expect(visible()).toBe(true); + }); + + it("shows without a reload once the open onboarding task finishes and company issues refresh", async () => { + state.list.mockResolvedValue([firstTask("todo")]); + await render(); + expect(visible()).toBe(false); + state.list.mockResolvedValue([firstTask("done")]); + // Issue live updates invalidate the company's issue queries by this prefix. + await act(async () => { await client.invalidateQueries({ queryKey: ["issues", "company"] }); }); + await flush(); + expect(visible()).toBe(true); + }); + + it("stays hidden when onboarding state cannot be read", async () => { + state.list.mockRejectedValue(new Error("offline")); + await render(); + expect(visible()).toBe(false); + }); +}); diff --git a/ui/src/components/AnnouncementWell.test.tsx b/ui/src/components/AnnouncementWell.test.tsx index ebf0a0fabff..0a4930cfebf 100644 --- a/ui/src/components/AnnouncementWell.test.tsx +++ b/ui/src/components/AnnouncementWell.test.tsx @@ -1,6 +1,7 @@ // @vitest-environment jsdom import { act } from "react"; import { createRoot, type Root } from "react-dom/client"; +import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import type { HealthStatus } from "@/api/health"; import { AnnouncementWell } from "./AnnouncementWell"; @@ -18,6 +19,7 @@ vi.mock("@/hooks/useAnnouncement", () => ({ useAnnouncement: (options: { enabled state.hook(options); return { announcement: options.enabled ? announcementPreview : null, dismiss: state.dismiss }; } })); +vi.mock("@/api/issues", () => ({ issuesApi: { list: vi.fn().mockResolvedValue([]) } })); vi.mock("./AnnouncementCard", () => ({ AnnouncementCard: ({ onDismiss }: { onDismiss: () => void }) => })); (globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true; @@ -25,12 +27,16 @@ describe("announcement placement gates", () => { let container: HTMLDivElement; let root: Root; let health: HealthStatus; - const render = async () => { await act(async () => root.render()); }; + let client: QueryClient; + const render = async () => { await act(async () => root.render()); }; const visible = () => Boolean(container.querySelector('[aria-label="Paperclip announcements"]')); beforeEach(() => { Object.assign(state, { userId: "alice", settled: true, companyId: "company", loading: false, onboardingOpen: false, toasts: [] }); vi.clearAllMocks(); health = { deploymentMode: "authenticated" } as HealthStatus; + // Onboarding is finished (no open first task); onboarding gating is covered in AnnouncementWell.onboarding.test.tsx. + client = new QueryClient({ defaultOptions: { queries: { retry: false, staleTime: Infinity } } }); + client.setQueryData(["issues", "company", "onboarding-first-task"], []); container = document.createElement("div"); document.body.append(container); root = createRoot(container); }); afterEach(async () => { await act(async () => root.unmount()); container.remove(); }); diff --git a/ui/src/components/AnnouncementWell.tsx b/ui/src/components/AnnouncementWell.tsx index b71ce16f095..d5d263539a0 100644 --- a/ui/src/components/AnnouncementWell.tsx +++ b/ui/src/components/AnnouncementWell.tsx @@ -1,10 +1,14 @@ import { useEffect, useState } from "react"; +import { useQuery } from "@tanstack/react-query"; +import { ONBOARDING_FIRST_TASK_ORIGIN_KIND } from "@paperclipai/shared"; import { useAccountIdentity } from "@/api/companies-query"; import type { HealthStatus } from "@/api/health"; +import { issuesApi } from "@/api/issues"; import { useCompany } from "@/context/CompanyContext"; import { useDialogState } from "@/context/DialogContext"; import { useOptionalToastActions, useOptionalToastState } from "@/context/ToastContext"; import { useAnnouncement } from "@/hooks/useAnnouncement"; +import { queryKeys } from "@/lib/queryKeys"; import { AnnouncementCard } from "./AnnouncementCard"; // Includes the command palette, sheets and dialogs created outside DialogContext. @@ -22,16 +26,36 @@ function useModalOpen(enabled: boolean) { return open; } +const FINISHED_ONBOARDING_STATUSES = new Set(["done", "cancelled"]); + +/** + * A new user's first session is the onboarding first task. Announcements wait + * until that task is finished so they cannot cover its questions. Unknown + * state (loading or a failed read) keeps the optional card hidden. + */ +function useOnboardingInProgress(companyId: string | null) { + const query = useQuery({ + queryKey: queryKeys.issues.onboardingFirstTask(companyId ?? "__none__"), + queryFn: ({ signal }) => issuesApi.list(companyId!, { originKind: ONBOARDING_FIRST_TASK_ORIGIN_KIND, limit: 1 }, { signal }), + enabled: Boolean(companyId), + staleTime: 60_000, + }); + if (!companyId) return false; + if (!query.isSuccess) return true; + return query.data.some((issue) => issue.originKind === ONBOARDING_FIRST_TASK_ORIGIN_KIND && !FINISHED_ONBOARDING_STATUSES.has(issue.status)); +} + export function AnnouncementWell({ health }: { health?: HealthStatus }) { const { userId: accountId, settled } = useAccountIdentity(); const { selectedCompanyId, loading } = useCompany(); const { onboardingOpen } = useDialogState(); const toastActions = useOptionalToastActions(); const toasts = useOptionalToastState(); + const onboardingInProgress = useOnboardingInProgress(selectedCompanyId); const userId = health?.deploymentMode === "local_trusted" ? "local-board" : (settled ? accountId : null); const { announcement, dismiss } = useAnnouncement({ userId, companyId: selectedCompanyId, - enabled: Boolean(userId && !loading && selectedCompanyId && !onboardingOpen), + enabled: Boolean(userId && !loading && selectedCompanyId && !onboardingOpen && !onboardingInProgress), onSaveFailure: (savedLocally) => toastActions?.pushToast({ title: savedLocally ? "Dismissed in this browser" : "Dismissed for this visit", body: "Couldn’t save across devices. We’ll retry when you reconnect or return.", diff --git a/ui/src/components/IssueMonitorBanner.test.tsx b/ui/src/components/IssueMonitorBanner.test.tsx index 36e749e5570..7e11fcaf134 100644 --- a/ui/src/components/IssueMonitorBanner.test.tsx +++ b/ui/src/components/IssueMonitorBanner.test.tsx @@ -263,13 +263,23 @@ describe("IssueMonitorBanner / IssueMonitorComposerStrip rendering", () => { } finally { flushSync(() => root.unmount()); } }); - it("explains automatic workspace waiting without promising that a reply bypasses the lock", () => { + it("does not raise a pinned banner for a workspace wait the user cannot act on (N-14)", () => { + const issue = { + status: "todo", scheduledRetry: { status: "scheduled_retry", scheduledRetryReason: "workspace_busy", scheduledRetryAt: NOW.toISOString() }, + } as Issue; + const root = createRoot(container); + flushSync(() => root.render()); + expect(container.textContent).toBe(""); + flushSync(() => root.unmount()); + }); + + it("keeps the workspace wait as a quiet composer status without promising that a reply bypasses the lock", () => { const issue = { status: "todo", scheduledRetry: { status: "scheduled_retry", scheduledRetryReason: "workspace_busy", scheduledRetryAt: NOW.toISOString() }, } as Issue; const root = createRoot(container); flushSync(() => root.render(<>)); - expect(container.textContent).toContain("Waiting for workspace"); + expect(container.querySelector('[data-testid="issue-monitor-composer-strip"]')?.textContent).toContain("Waiting for workspace"); expect(container.textContent).toContain("You can keep sending instructions while the agent waits."); expect(container.textContent).not.toContain("wakes the agent now"); expect(container.querySelector("button")).toBeNull(); diff --git a/ui/src/components/IssueMonitorBanner.tsx b/ui/src/components/IssueMonitorBanner.tsx index 52193278153..22ca0474136 100644 --- a/ui/src/components/IssueMonitorBanner.tsx +++ b/ui/src/components/IssueMonitorBanner.tsx @@ -296,6 +296,10 @@ export function IssueMonitorBanner({ const copy = useMonitorSurfaceCopy(issue, reviews.length, hasActiveRun); if (!copy) return null; + // A busy workspace is a transient system wait the user cannot act on. It stays + // a quiet status (composer strip, properties row), never a pinned notice. + if (copy.workspaceWait) return null; + const canCancel = onCancelMonitor && deriveMonitorState(issue).source === "monitor"; return ( diff --git a/ui/src/components/IssueScheduledRetryCard.test.tsx b/ui/src/components/IssueScheduledRetryCard.test.tsx index 8b6fdaea874..8099576fe17 100644 --- a/ui/src/components/IssueScheduledRetryCard.test.tsx +++ b/ui/src/components/IssueScheduledRetryCard.test.tsx @@ -121,11 +121,9 @@ function getRetryNowButton() { } describe("IssueScheduledRetryCard", () => { - it("shows workspace contention as an automatic wait without failure or retry controls", () => { + it("raises no conversation notice for workspace contention the user cannot act on (N-14)", () => { renderWithProviders(); - expect(container.textContent).toContain("Waiting for workspace"); - expect(container.textContent).toContain("Work starts automatically"); - expect(container.textContent).not.toMatch(/failed|Retry|Attempt|Replaces run/); + expect(container.textContent).toBe(""); expect(getRetryNowButton()).toBeNull(); }); it("surfaces model capacity and the automatic retry without a generic failure", () => { diff --git a/ui/src/components/IssueScheduledRetryCard.tsx b/ui/src/components/IssueScheduledRetryCard.tsx index 383fce3be87..dd6325bad5f 100644 --- a/ui/src/components/IssueScheduledRetryCard.tsx +++ b/ui/src/components/IssueScheduledRetryCard.tsx @@ -7,7 +7,6 @@ import { formatRetryReason } from "@/lib/runRetryState"; import type { IssueScheduledRetry } from "@paperclipai/shared"; import { useRetryNowMutation, type RetryNowError } from "../hooks/useRetryNowMutation"; import { Badge } from "@/components/ui/badge"; -import { InlineBanner } from "@/components/InlineBanner"; import { describeError } from "@/api/errors"; const MAX_TURN_CONTINUATION = "max_turns_continuation"; @@ -34,13 +33,9 @@ export function IssueScheduledRetryCard({ if (!scheduledRetry || !issueId) return null; if (scheduledRetry.status !== "scheduled_retry") return null; - if (scheduledRetry.scheduledRetryReason === "workspace_busy") { - return ( - - Another task is using this workspace. Work starts automatically when it is available. - - ); - } + // Workspace contention is a transient wait with nothing for the user to do. + // The composer strip and properties row carry the quiet status instead. + if (scheduledRetry.scheduledRetryReason === "workspace_busy") return null; const continuation = isContinuationReason(scheduledRetry.scheduledRetryReason); const dueAtIso = scheduledRetry.scheduledRetryAt diff --git a/ui/src/components/IssueThreadInteractionCard.tsx b/ui/src/components/IssueThreadInteractionCard.tsx index 68c6f3c2389..6ce2763aa91 100644 --- a/ui/src/components/IssueThreadInteractionCard.tsx +++ b/ui/src/components/IssueThreadInteractionCard.tsx @@ -54,6 +54,7 @@ import { useCoordinatorWorkspaceEnabled } from "@/hooks/useCoordinatorWorkspaceE import { CoordinatorTaskCommitment } from "./coordinator/CoordinatorTaskCommitment"; import { DecisionCardFooter } from "./DecisionCardFooter"; +import { ToolActionAuditHashCopy } from "./ToolActionAuditHashCopy"; const OTHER_ANSWER_ID = "__paperclip_other__"; @@ -1847,12 +1848,12 @@ function RequestToolActionCard({ {isPending ? inspectableArguments ? (
- Arguments and audit hash + Arguments 
             {payload.argumentsSummaryJson}
-

{payload.argumentsHash}

+
) :

Request arguments are unavailable.

: null} {isPending ? ( diff --git a/ui/src/components/OnboardingWizard.test.tsx b/ui/src/components/OnboardingWizard.test.tsx index f4c2dc47e84..38aac59711a 100644 --- a/ui/src/components/OnboardingWizard.test.tsx +++ b/ui/src/components/OnboardingWizard.test.tsx @@ -2138,6 +2138,30 @@ describe("OnboardingWizard restore-gate (stale localStorage across accounts)", ( return { root, queryClient }; } + // N-16: the gateway is an advanced route, not the next thing after the key. + // It sat directly under the API key field, inside the key card; it belongs + // in the step's Advanced disclosure, hidden until that is opened. + it.each([["Claude API", "claude_local", /Claude.*API/], ["Codex API", "codex_local", /OpenAI.*API/]] as const)("keeps Use a custom gateway in the closed Advanced disclosure on the %s key step", async (_path, adapterType, source) => { + const handles = await openStep4({ adapterType, runner: "paperclip", credentialMode: "api" }); + try { + await pickSource(source); + const keyField = document.body.querySelector('input[type="password"]') as HTMLInputElement; + expect(keyField, "the API key field should be showing").toBeTruthy(); + const gateway = () => [...document.body.querySelectorAll("button")].find(entry => entry.textContent?.trim() === "Use a custom gateway"); + expect(gateway(), "the custom gateway control should still exist").toBeTruthy(); + const advanced = gateway()!.closest("details"); + expect(advanced, "Use a custom gateway should live inside the Advanced disclosure").toBeTruthy(); + expect(advanced!.querySelector(":scope > summary")?.textContent?.trim()).toBe("Advanced"); + expect(advanced!.open, "Advanced should start closed").toBe(false); + expect(advanced!.contains(keyField), "the key field stays outside Advanced").toBe(false); + await act(async () => { advanced!.open = true; }); + await flushReact(); + await act(async () => gateway()!.click()); + await flushReact(); + expect(document.querySelector('input[aria-label="Provider URL"]'), "the gateway dialog should still open from Advanced").toBeTruthy(); + } finally { await act(async () => handles.root.unmount()); } + }); + it.each([["successful refresh", false], ["failed refresh and retry", true]] as const)("saves a custom gateway without probing, then explicitly tests and hires with its selected model: %s", async (_scenario, failRefresh) => { const gatewayAccounts: import("@paperclipai/shared").AiConnectionList = { currentUserId: "user-1", canManageConnections: true, connections: [], diff --git a/ui/src/components/OnboardingWizard.tsx b/ui/src/components/OnboardingWizard.tsx index 7b7297f7a31..01d2eaf323b 100644 --- a/ui/src/components/OnboardingWizard.tsx +++ b/ui/src/components/OnboardingWizard.tsx @@ -3157,15 +3157,6 @@ function OnboardingWizardInner({ onChange={setModel} open={modelOpen} onOpenChange={setModelOpen} allowDefault={false} required groupByProvider={false} creatable={connectionModels.models.length === 0} presentation="native" />} - {createdCompanyId && (adapterType === "codex_local" || adapterType === "claude_local") && { - setGatewayConnection({ companyId: createdCompanyId, adapterType, binding }); - setApiKey(""); - setModel(""); - setError(null); - }} />} {(hasGatewaySelection || connectionModels) && + {copyFailed ? ( +

+ {hash} +

+ ) : null} + + ); +} diff --git a/ui/src/components/onboarding/OnboardingGatewaySetup.tsx b/ui/src/components/onboarding/OnboardingGatewaySetup.tsx index e38fabca600..9d41a42caa6 100644 --- a/ui/src/components/onboarding/OnboardingGatewaySetup.tsx +++ b/ui/src/components/onboarding/OnboardingGatewaySetup.tsx @@ -10,6 +10,7 @@ export function OnboardingGatewaySetup({ environmentId, adapterType, disabled, + className, onComplete, }: { companyId: string; @@ -17,11 +18,12 @@ export function OnboardingGatewaySetup({ environmentId?: string; adapterType: "codex_local" | "claude_local"; disabled?: boolean; + className?: string; onComplete: (binding: Exclude) => void; }) { const [open, setOpen] = useState(false); return <> - diff --git a/ui/src/components/task-chat/TaskChatCompactInteractionCard.tsx b/ui/src/components/task-chat/TaskChatCompactInteractionCard.tsx index d4a13a54624..dbcb7487716 100644 --- a/ui/src/components/task-chat/TaskChatCompactInteractionCard.tsx +++ b/ui/src/components/task-chat/TaskChatCompactInteractionCard.tsx @@ -30,6 +30,7 @@ import type { } from "@paperclipai/adapter-utils"; import { InteractionPreparationNotice } from "@/components/InteractionPreparationNotice"; import { IssueThreadInteractionCard } from "@/components/IssueThreadInteractionCard"; +import { ToolActionAuditHashCopy } from "@/components/ToolActionAuditHashCopy"; import { ConnectionIntentInteractionBody } from "@/features/connections/ConnectionIntentInteractionBody"; import { MarkdownBody } from "@/components/MarkdownBody"; import type { MentionOption } from "@/components/MarkdownEditor"; @@ -573,9 +574,7 @@ function ReceiptDisclosure({ 
               {interaction.payload.toolAction.argumentsSummaryJson}
-

- {interaction.payload.toolAction.argumentsHash} -

+ ) : null} {interaction.payload.secretProposal ? ( @@ -934,14 +933,12 @@ function ConfirmationCard({
- Arguments and audit hash + Arguments 
               {interaction.payload.toolAction.argumentsSummaryJson}
-

- {interaction.payload.toolAction.argumentsHash} -

+
) : null} diff --git a/ui/src/components/task-chat/TaskChatInteractionCard.test.tsx b/ui/src/components/task-chat/TaskChatInteractionCard.test.tsx index 2260e4454ae..7b2763446ad 100644 --- a/ui/src/components/task-chat/TaskChatInteractionCard.test.tsx +++ b/ui/src/components/task-chat/TaskChatInteractionCard.test.tsx @@ -3,7 +3,7 @@ import { act, forwardRef, useImperativeHandle, useState, type ForwardedRef } from "react"; import { flushSync } from "react-dom"; import { createRoot, type Root } from "react-dom/client"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { afterEach, beforeEach, describe, expect, it, onTestFinished, vi } from "vitest"; vi.mock("@/hooks/useCoordinatorWorkspaceEnabled", () => ({ useCoordinatorWorkspaceEnabled: () => ({ enabled: false, loaded: true }), @@ -148,7 +148,8 @@ describe("TaskChatInteractionCard", () => { expect(container.querySelector("pre")?.textContent).toBe(interaction.payload.toolAction!.argumentsSummaryJson); expect(details.textContent).toContain("November 8, 2030"); expect(details.textContent).toContain("[redacted]"); - expect(details.textContent).toContain(interaction.payload.toolAction!.argumentsHash); + expect(details.textContent).not.toContain(interaction.payload.toolAction!.argumentsHash); + expect(details.textContent).toContain("Copy audit hash"); await act(async () => details.querySelector("summary")!.click()); expect(details.open).toBe(true); expect(accept).not.toHaveBeenCalled(); @@ -160,6 +161,64 @@ describe("TaskChatInteractionCard", () => { expect(reject).not.toHaveBeenCalled(); }); + it("keeps raw tool ids and the audit hash out of the tool approval card text", async () => { + // Shape of the GitHub create_repository approval from the Nov 5 staging run. + const rawToolName = "mcp.app-gallery-github-42e51e3f-9377-447b-ad45-333beda8a50a-b873f178:create-repository"; + const argumentsHash = "8ca5312877235a4c0f6e9b1d2a7c3e5f9081b2c4d6e8fa1b3c5d7e9f0a2b4c6d"; + const base = structuredClone(pendingToolActionLongProposalInteraction); + const interaction: RequestConfirmationInteraction = { + ...base, + title: "Approve tool action", + summary: "This action needs your approval before it runs.", + payload: { + ...base.payload, + prompt: "Approve create_repository?", + detailsMarkdown: `Tool: \`${rawToolName}\`\nRisk: \`write\``, + target: { type: "custom", key: "tool-action:2f92393c-c044-4367-8b6e-b81e2e732165", revisionId: argumentsHash, label: rawToolName }, + toolAction: { + ...base.payload.toolAction!, + toolName: rawToolName, + toolDisplayName: "create_repository", + appDisplayName: "GitHub", + previewMarkdown: "create_repository · Name: brainrot-storefront-prototype · Private: true", + argumentsSummaryJson: JSON.stringify({ name: "brainrot-storefront-prototype", private: true }), + argumentsHash, + }, + }, + }; + const writeText = vi.fn().mockResolvedValue(undefined); + const originalClipboard = Object.getOwnPropertyDescriptor(navigator, "clipboard"); + Object.defineProperty(navigator, "clipboard", { configurable: true, value: { writeText } }); + onTestFinished(() => { + if (originalClipboard) Object.defineProperty(navigator, "clipboard", originalClipboard); + else delete (navigator as { clipboard?: unknown }).clipboard; + }); + function ReviewSurface() { + const [reviewing, setReviewing] = useState(false); + return setReviewing(true)} + onAcceptInteraction={vi.fn()} onRejectInteraction={vi.fn()} />; + } + flushSync(() => root.render()); + const hashLike = /[0-9a-f]{12,}/i; + expect(container.textContent).not.toMatch(hashLike); + await act(async () => { + Array.from(container.querySelectorAll("button")).find(button => button.textContent === "Review request")!.click(); + }); + const details = container.querySelector("details")!; + await act(async () => details.querySelector("summary")!.click()); + expect(details.open).toBe(true); + expect(container.textContent).toContain("brainrot-storefront-prototype"); + expect(container.textContent).not.toMatch(hashLike); + expect(container.textContent).not.toContain("app-gallery"); + // The audit hash stays reachable for anyone who needs it, through a copy action. + const copy = Array.from(container.querySelectorAll("button")).find(button => button.textContent === "Copy audit hash"); + expect(copy).toBeDefined(); + await act(async () => copy!.click()); + expect(writeText).toHaveBeenCalledWith(argumentsHash); + }); + it("does not render a pending confirmation in the timeline", () => { flushSync(() => { root.render( diff --git a/ui/src/lib/queryKeys.ts b/ui/src/lib/queryKeys.ts index 3c88ecfec28..f13aae268f5 100644 --- a/ui/src/lib/queryKeys.ts +++ b/ui/src/lib/queryKeys.ts @@ -311,6 +311,8 @@ export const queryKeys = { recoveryActions: (id: string, actionId: string | null, runId: string | null) => ["issues", id, "recovery-actions", actionId, runId] as const, list: (companyId: string) => ["issues", companyId] as const, + onboardingFirstTask: (companyId: string) => + ["issues", companyId, "onboarding-first-task"] as const, mentionPool: (companyId: string) => ["issues", companyId, "mention-pool"] as const, search: (