Merge origin/master and preserve monitor wait regressions

Co-Authored-By: Paperclip <noreply@paperclip.ing>

DESIGN.md

@@ -153,7 +153,7 @@ results in place; keep failures actionable inline. Notifications for other work
 remain useful. Repeated delivery of the same run outcome must refresh cached state
 without repeating its toast, including after reconnecting. A terminal outcome
 delivered more than five minutes after the run finished is historical and should
-refresh state silently. Expected cancellation is neutral gray, not an error. The composer's Stop action stops the current response and leaves the composer available for a new message. Pause work is a separate explicit task or subtree action. A paused task replaces the composer with an amber takeover. It says “Task is
+refresh state silently. Expected cancellation is neutral gray, not an error. A busy workspace is a transient system wait with no user action. Show it only as a quiet status in the composer strip and the Workspace property row, never as a pinned banner, conversation notice or toast. The composer's Stop action stops the current response and leaves the composer available for a new message. Pause work is a separate explicit task or subtree action. A paused task replaces the composer with an amber takeover. It says “Task is
 paused.” and “Resume this task to send a message.” with a “Resume task” action.
 Subtrees use “Subtree is paused.” and “Resume subtree.” The takeover cannot be
 dismissed, retains drafts, and hides message inputs until the pause is released.

doc/DEVELOPING.md

@@ -1919,6 +1919,37 @@ that the whole web server is disposable or that all execution and recovery paths
 survive web-server replacement. Run-log/trace storage and producer retention are
 separate changes.
 
+### Managed environments
+
+The `environments` list in `PAPERCLIP_MANAGED_CONFIG` declares instance-level
+environments that boot provisions and tenants cannot edit. It holds at most one
+sandbox entry (no `driver`, or `driver: "sandbox"`) and any number of
+`driver: "plugin"` entries. A plugin entry names a plugin by its manifest id
+(`pluginKey`) and one of its environment drivers:
+
+```json
+{
+  "name": "Remote Host",
+  "description": "Runs the agent on a remote host",
+  "driver": "plugin",
+  "pluginKey": "example.remote-host-plugin",
+  "driverKey": "tasks",
+  "driverConfig": {}
+}
+```
+
+Names and `pluginKey`–`driverKey` pairs must be unique, and `driverConfig` must
+not contain secret-like keys. Any other key refuses startup. `plugins.autoInstall`
+holds catalog keys, not manifest ids, so boot also refuses to start unless an
+auto-installed bundled or distribution catalog entry installs the entry's
+`pluginKey`.
+
+Boot validates the driver config through the plugin, marks the row
+`managedPluginEnvironment: "<pluginKey>:<driverKey>"`, and adopts an existing
+unmarked row for the same plugin and driver. While the plugin is down, the row is
+archived; it comes back when the worker is ready. Removing an entry never archives
+its row. Plugin entries do not change the instance default.
+
 ### Experimental native remote workspace storage
 
 `experimental.remoteWorkspaceStorageEnabled` is a managed feature, on by default

doc/dependency-wake-recovery.md

@@ -13,6 +13,13 @@ unattributed operation in the same shared workspace does not remove its wait.
 Normal readiness, duplicate-wake, interaction, pause, and execution gates still
 apply. A board transition into blocked keeps the existing recovery behavior.
 
+A dependency completion can arrive while the assignee answers a question about
+the blocked task. Its saved wake waits behind that turn. Deferred-receipt recovery
+reconstructs these dependency wakes after the turn and its cleanup finish, even
+when they contain no comment or interaction response. It uses the existing
+release and dispatch checks; task cancellation, agent pause, and pending cleanup
+still prevent execution.
+
 Cancelling a run or pausing its agent suppresses immediate stranded-work recovery.
 A queued user message remains a separate continuation request. The release path
 still preserves independently authorized deferred input; stopping the old run

doc/execution-semantics.md

@@ -2415,8 +2415,11 @@ run, release its environment, or start another provider. Lease expiry alone does
 not authorize that stale preparation to take ownership; normal recovery must
 establish a new claim first.
 
-An exact local Codex restart can restore the conversation after losing its active
-turn. Runnerd records this as `provider_turn_lost_on_restore`, with no invented
+An exact local Codex, OpenCode, or ACPX restart can restore the conversation after
+losing its active turn. ACPX first requires the original provider lifetime to lose
+its fence quorum; without that proof, or with an accepted result, unanswered
+runtime request, or active goal, it closes the session as before. Cursor Cloud
+agents run remotely and are excluded. Runnerd records this as `provider_turn_lost_on_restore`, with no invented
 task result. The admitted restart attempt may send one continuation in that same
 conversation. It preserves the workspace and asks the agent to reconcile unfinished
 commands and external actions before proceeding. It does not resend the task or
@@ -3630,6 +3633,34 @@ re-read dependency readiness, since new dependencies need not change the
 displayed task status. Old blocked rows without a receipt remain held; no
 historical status backfill is performed.
 
+Older native executors recorded the local callback-listener startup guard as
+`native_session_interrupted`. A saved exact guard outcome can enter the same
+bounded safe-replacement path after restart. This requires the first attempt,
+an unused provider-session identity, matching saved execution bindings, no
+provider events, checkpoint, admission receipt, tool call, or cancellation, and
+the existing current snapshot and confirmed stop receipts. A generic error or
+a pending provider-pack receipt alone grants no recovery authority. The same
+transaction can retire that run's completed file-preservation hold only when
+its bound recovery and finalization receipts prove completion. Unrelated holds,
+later task blocks, normal dispatch admission, and the three-attempt budget remain
+effective.
+
+Historical replacement discovery reads at most 100 candidates per pass and
+keeps a cursor only for the remaining scan. A work notification preserves forward
+progress and requests a final rescan, so held earlier rows cannot hide a later
+eligible run. Completed scans of held history become quiet. Final snapshot
+publication, verified lease cleanup, local process-stop proof, completed workspace
+recovery, and dispatch eligibility changes notify the existing replacement worker
+in their transactions. A cleared historical interaction only requests another
+proof check; it grants no new continuation authority. Source provider effects,
+task pause or cancellation, and the original authority checks still prevent replay.
+Temporary decision failures retain the existing worker's retry and advance the
+bounded cursor, with a forced rescan of earlier failures. A failed page cannot
+hide later ready runs. During drain, the initialized replacement worker can read
+readiness once per notification generation to discharge an empty notification
+without starting replacement. Concrete pending work and changes arriving during
+that read remain outstanding. Restart reconstructs the scan from durable records.
+
 For legacy GitHub input without a saved App ID or its own response receipt,
 Task Retry may prove the endpoint's earlier non-null App claim from a processed
 response ledger created before that input. The proof stays within the same

doc/run-log-events.md

@@ -348,13 +348,24 @@ that ledger records `remote_execution_never_claimed` with a
 `bootstrap_incomplete` disposition. The provider attempt remains zero during
 classification; the executor increments it before starting provider work.
 
-For a local Codex turn lost while runnerd was stopped, the canonical `turn.failed`
+For a local Codex, OpenCode, or ACPX turn lost while runnerd was stopped, the canonical `turn.failed`
 event retains `error.code: "provider_turn_lost_on_restore"` and
 `error.recoverable: true`. It records the interrupted turn, not a final task
 outcome. There is no synthesized result for that interruption. An admitted
 same-session continuation emits its own `turn.submitted`, accepted turn ID, and
 terminal events; only the final outcome closes the run. Reconstructed thread
 history retains the original error so reconciliation and live delivery agree.
+An ACPX session records this only after the original provider lifetime loses
+its fence quorum and the same conversation is restored. Otherwise its
+`turn.failed` carries `error.code: "acpx_active_turn_recovery_closed"` with
+`error.recoverable: false`, and the run closes with an honest restart message.
+After an ACPX restore, the controller can still deliver results for semantic
+calls of the lost turn. Runnerd acknowledges each one as `retired` without
+sending it to the restored provider, so the calls settle and live semantic
+dispatch resumes for the continuation. A restored ACPX provider is a new process
+whose MCP request ids restart, so its semantic `callId` values carry a
+per-process prefix (`r<hex>:<id>`). They never reuse an earlier call's id in
+the same run.
 
 ## Native Local Process Stop Evidence
 

packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs

@@ -1,7 +1,7 @@
 use crate::generated_acpx_profiles::{
     acpx_release_profile, ACPX_DRIVER_KIND, QUALIFIED_ACPX_VERSION,
 };
-use std::collections::{BTreeMap, HashMap, HashSet, VecDeque};
+use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet, VecDeque};
 use std::fs::{self, DirBuilder, File};
 use std::io::{Read, Write};
 use std::net::TcpListener;
@@ -70,6 +70,8 @@ pub const ACPX_PROVIDER_STATE_FILE: &str = "acpx-provider-state.json";
 const ACPX_PROVIDER_STATE_SCHEMA: &str = "paperclip.runner.acpx-provider-state.v3";
 const MAX_PROVIDER_STATE_BYTES: u64 = 16 * 1024 * 1024;
 const MAX_PENDING_EVENTS: usize = 8_320;
+// Matches the bounded number of pending ACPX tool calls.
+const MAX_OPEN_SEMANTIC_CALLS: usize = 4_096;
 const MAX_EVENTS_PER_POLL: usize = 128;
 const PROVIDER_LIFETIME_CONFIRMATION_TIMEOUT: Duration = Duration::from_secs(5);
 const PROVIDER_LIFETIME_CONFIRMATION_RETRY: Duration = Duration::from_millis(10);
@@ -597,6 +599,13 @@ struct AcpxDurableState {
     pending_runtime_requests: BTreeMap<String, Value>,
     #[serde(default = "initial_event_sequence")]
     next_event_sequence: u64,
+    /// Semantic tool calls emitted to the controller without a result yet.
+    #[serde(default)]
+    open_semantic_calls: BTreeSet<String>,
+    /// Open calls of a turn lost to a runner restart. The restored provider
+    /// never sees their results; the controller may still deliver them.
+    #[serde(default)]
+    retired_semantic_calls: BTreeSet<String>,
 }
 
 impl AcpxDurableState {
@@ -623,6 +632,8 @@ impl AcpxDurableState {
             pending_events: VecDeque::new(),
             pending_runtime_requests: BTreeMap::new(),
             next_event_sequence: initial_event_sequence(),
+            open_semantic_calls: BTreeSet::new(),
+            retired_semantic_calls: BTreeSet::new(),
         }
     }
 
@@ -789,6 +800,31 @@ impl AcpxDurableState {
             if let Some(id) = event.payload.get("requestId").and_then(Value::as_str) {
                 self.pending_runtime_requests.remove(id);
             }
+        } else if let Some(call_id) = event
+            .payload
+            .pointer("/semantic_tool/callId")
+            .and_then(Value::as_str)
+        {
+            // Remember semantic calls the controller may still answer. A lost
+            // turn's open calls become retired calls on restore, so a later
+            // controller result for them settles instead of failing.
+            match event.event_type.as_str() {
+                "semantic_tool.input"
+                    if self.open_semantic_calls.len() < MAX_OPEN_SEMANTIC_CALLS =>
+                {
+                    self.open_semantic_calls.insert(call_id.to_owned());
+                }
+                "semantic_tool.result" => {
+                    self.open_semantic_calls.remove(call_id);
+                }
+                _ => {}
+            }
+        } else if matches!(
+            event.event_type.as_str(),
+            "turn.completed" | "turn.failed" | "turn.cancelled" | "turn.interrupted"
+        ) {
+            // A provider-observed terminal settles its calls in the live bridge.
+            self.open_semantic_calls.clear();
         }
         self.next_event_sequence = next_sequence;
         self.pending_events.push_back(PolledEvent {
@@ -1116,6 +1152,9 @@ impl AcpxCommandExecutor {
         let previous_turn = state.active_turn_id.clone();
         if unsafe_active {
             self.context.provider_turn_id = None;
+            if self.restore_conversation_after_lost_turn(previous_turn.clone())? {
+                return Ok(());
+            }
             let state = self
                 .state
                 .as_mut()
@@ -1134,6 +1173,13 @@ impl AcpxCommandExecutor {
                     "providerTerminalObserved": false,
                     "code": "acpx_active_turn_recovery_closed",
                     "providerShutdownFailed": true,
+                    // Keep the cause through the controller facade, which
+                    // forwards only the structured error of a failed turn.
+                    "error": {
+                        "code": "acpx_active_turn_recovery_closed",
+                        "recoverable": false,
+                        "message": "The runner restarted during this turn and could not restore the conversation.",
+                    },
                 }),
             })?;
             state.push(NormalizedProviderEvent {
@@ -1171,6 +1217,82 @@ impl AcpxCommandExecutor {
         self.save_state()
     }
 
+    /// Restores the conversation of a turn that was active when the runner
+    /// died, without inventing its outcome. The original provider lifetime
+    /// must have lost its fence quorum first, so no second provider can act on
+    /// the same session. The lost turn is recorded with a recoverable
+    /// `provider_turn_lost_on_restore` cause and the run stays open, so the
+    /// controller can send its single same-conversation continuation. Returns
+    /// false, leaving state untouched, when restoration is not safe.
+    fn restore_conversation_after_lost_turn(
+        &mut self,
+        previous_turn: Option<String>,
+    ) -> Result<bool, DurableRunnerError> {
+        let Some(state) = self.state.as_ref() else {
+            return Ok(false);
+        };
+        // An accepted result, an unanswered runtime request, or an active goal
+        // needs its own reconciliation; a plain continuation cannot settle it.
+        let Some(identity) = state.identity.as_ref() else {
+            return Ok(false);
+        };
+        if previous_turn.is_none()
+            || state.provider_exit_unconfirmed
+            || state.semantic_result.is_some()
+            || !state.pending_runtime_requests.is_empty()
+            || !(state.goal_projection.is_null() || goal_is_dormant(&state.goal_projection))
+        {
+            return Ok(false);
+        }
+        match await_provider_lifetime_fence(identity.provider_lifetime_fence_candidates) {
+            // Release the quorum so the restored provider can inherit it.
+            Ok(listeners) => drop(listeners),
+            Err(_) => return Ok(false),
+        }
+        let Ok(session) = self.start_session(true) else {
+            return Ok(false);
+        };
+        let identity = session.identity().clone();
+        let process_id = session.process_id();
+        let turn_controls = session.turn_control_capabilities();
+        let state = self
+            .state
+            .as_mut()
+            .expect("ACPX state remains available during recovery");
+        state.lifecycle = "session_open".to_owned();
+        state.active_turn_id = None;
+        state.turn_controls = turn_controls;
+        // The controller may still answer the lost turn's semantic calls.
+        // The restored provider never issued them, so settle them on arrival.
+        let open = std::mem::take(&mut state.open_semantic_calls);
+        state.retired_semantic_calls.extend(open);
+        state.push(NormalizedProviderEvent {
+            event_type: "session.resumed".to_owned(),
+            priority: EventPriority::P0,
+            payload: session_event_payload(&state.descriptor, &identity, process_id, turn_controls),
+        })?;
+        state.push(NormalizedProviderEvent {
+            event_type: "turn.failed".to_owned(),
+            priority: EventPriority::P0,
+            payload: json!({
+                "provider": "acpx",
+                "providerTurnId": previous_turn,
+                "status": "failed",
+                "providerTerminalObserved": false,
+                "code": "provider_turn_lost_on_restore",
+                "error": {
+                    "code": "provider_turn_lost_on_restore",
+                    "recoverable": true,
+                    "message": "The runner restored the conversation after process loss, but the previous turn is no longer active.",
+                },
+            }),
+        })?;
+        self.recovery_pending_effects = None;
+        self.session = Some(session);
+        self.save_state()?;
+        Ok(true)
+    }
+
     fn start_session(&self, recovering: bool) -> Result<AcpxProviderSession, DurableRunnerError> {
         let state = self
             .state
@@ -2123,6 +2245,20 @@ impl AcpxCommandExecutor {
         let result: ToolResult = serde_json::from_value(payload.clone()).map_err(|error| {
             DurableRunnerError::invalid(format!("semantic tool result is invalid: {error}"))
         })?;
+        if let Some(state) = self.state.as_ref() {
+            if state.retired_semantic_calls.contains(&result.call_id) {
+                // The call belonged to a turn lost to a runner restart. Its
+                // provider process is gone; acknowledge the controller result
+                // so the call settles, without sending it to the new provider.
+                // The retained receipt makes this idempotent: a delivery left
+                // indeterminate by a crash replays to the same acknowledgement.
+                return Ok(CommandExecution::result(json!({
+                    "status": "delivered",
+                    "callId": result.call_id,
+                    "retired": true,
+                })));
+            }
+        }
         self.session
             .as_mut()
             .ok_or_else(|| DurableRunnerError::invalid("ACPX session is unavailable"))?
@@ -2521,6 +2657,18 @@ fn goal_terminal_disposition<'a>(
 }
 
 impl CommandExecutor for AcpxCommandExecutor {
+    /// A restored session keeps durable receipts for the lost turn's calls.
+    /// Acknowledging one is idempotent and never reaches the provider or the
+    /// business operation, so an indeterminate delivery may replay. Live
+    /// calls have no durable receipt; without retired receipts this stays off.
+    fn can_reconcile_result_delivery(&mut self) -> Result<bool, DurableRunnerError> {
+        self.restore()?;
+        Ok(self
+            .state
+            .as_ref()
+            .is_some_and(|state| !state.retired_semantic_calls.is_empty()))
+    }
+
     fn host_park_ready(&mut self) -> Result<bool, DurableRunnerError> {
         self.ensure_persistence_healthy()?;
         let Some(state) = self.state.as_ref() else {
@@ -5020,6 +5168,14 @@ mod tests {
         assert_eq!(events[0].payload["replayAllowed"], false);
         assert_eq!(events[1].event_type, "turn.failed");
         assert_eq!(events[1].payload["providerShutdownFailed"], true);
+        // An unanswered runtime request (and an unproven original lifetime)
+        // keeps the close path. Its cause survives the controller facade and
+        // is not offered for the restart continuation.
+        assert_eq!(
+            events[1].payload["error"]["code"],
+            "acpx_active_turn_recovery_closed"
+        );
+        assert_eq!(events[1].payload["error"]["recoverable"], false);
         assert_eq!(events[2].event_type, "run.terminal");
         assert_eq!(events[2].payload["schema"], "paperclip.prp.terminal.v1");
         assert_eq!(events[2].payload["turnTerminalState"], "failed");
@@ -5049,6 +5205,74 @@ mod tests {
         fs::remove_dir_all(directory).unwrap();
     }
 
+    #[test]
+    fn lost_turn_restoration_refuses_unreconciled_state_before_touching_the_provider() {
+        let directory = temporary_directory("lost-turn-guards");
+        let provider_descriptor: AcpxProviderDescriptor =
+            serde_json::from_value(descriptor("claude")).unwrap();
+        let operations = Vec::new();
+        let tool_set = AuthorizedToolSet {
+            schema: TOOL_SET_SCHEMA.to_owned(),
+            schema_version: 1,
+            catalog_digest: authorized_tool_catalog_digest(&operations).unwrap(),
+            operations,
+        };
+        let identity = AcpxProviderSessionIdentity {
+            kind: "acpx".to_owned(),
+            normalized_session_id: "session-1".to_owned(),
+            acpx_record_id: "record-1".to_owned(),
+            backend_session_id: "backend-1".to_owned(),
+            agent_session_id: "agent-1".to_owned(),
+            profile_digest: provider_descriptor.command_digest.clone(),
+            workspace_digest: format!("sha256:{}", "a".repeat(64)),
+            requested_model: provider_descriptor.model.clone(),
+            effective_model: provider_descriptor.model.clone(),
+            permission_mode: Some(provider_descriptor.permission_mode),
+            mode: provider_descriptor.mode.clone(),
+            pi_thinking_level: provider_descriptor.pi_thinking_level,
+            claude_effort: provider_descriptor.claude_effort.clone(),
+            grok_effort: provider_descriptor.grok_effort.clone(),
+            grok_disable_web_search: provider_descriptor.grok_disable_web_search.clone(),
+            provider_lifetime_fence_candidates: [60_001, 60_002, 60_003],
+        };
+        let guards: [(&str, fn(&mut AcpxDurableState)); 4] = [
+            ("provider_exit_unconfirmed", |state| {
+                state.provider_exit_unconfirmed = true
+            }),
+            ("accepted_semantic_result", |state| {
+                state.semantic_result = Some(json!({"reportedWorkDisposition": "done"}))
+            }),
+            ("unanswered_runtime_request", |state| {
+                state
+                    .pending_runtime_requests
+                    .insert("input-1".to_owned(), pending_input_request("input-1"));
+            }),
+            ("active_goal", |state| {
+                state.goal_projection = json!({"goal": {"status": "active"}})
+            }),
+        ];
+        for (label, seed) in guards {
+            let mut state =
+                AcpxDurableState::new(provider_descriptor.clone(), tool_set.clone(), "d".into());
+            state.lifecycle = "turn_active".to_owned();
+            state.identity = Some(identity.clone());
+            state.active_turn_id = Some("turn-1".to_owned());
+            seed(&mut state);
+            let mut executor =
+                AcpxCommandExecutor::with_runner_config(&directory, &test_config(&directory, None));
+            executor.state = Some(state.clone());
+            assert!(
+                !executor
+                    .restore_conversation_after_lost_turn(Some("turn-1".to_owned()))
+                    .unwrap(),
+                "{label} must keep the close path"
+            );
+            assert!(executor.session.is_none(), "{label} started a provider");
+            assert_eq!(executor.state, Some(state), "{label} changed saved state");
+        }
+        fs::remove_dir_all(directory).unwrap();
+    }
+
     #[test]
     fn suspension_waits_for_the_original_provider_lifetime_quorum() {
         let (candidates, original_lifetime_fence) = reserve_provider_lifetime_fence();

packages/paperclip-runner/runner/crates/runner-core/src/provider_backend.rs

@@ -421,6 +421,14 @@ fn semantic_result_event(
     }
 }
 
+/// Local Codex app-server and OpenCode server restart their own process and
+/// restore the same conversation after runner loss, so a turn that vanished
+/// meanwhile is a restart interruption the controller may continue once. A
+/// Cursor Cloud agent runs remotely; its lost turn is not a process restart.
+fn restart_lost_turn_is_continuable(provider: &str) -> bool {
+    matches!(provider, "codex" | "opencode")
+}
+
 fn facade_provider_name(provider: &str) -> &'static str {
     match provider {
         "opencode" => "OpenCode",
@@ -2693,6 +2701,7 @@ impl CodexCommandExecutor {
                         // trustworthy success notification to replay. Terminate it
                         // conservatively so the controller cannot wait forever or
                         // mistake an unknown outcome for success.
+                        let restart_continuable = restart_lost_turn_is_continuable(&provider_label);
                         let provider_terminal = NormalizedProviderEvent {
                             event_type: "turn.failed".to_owned(),
                             priority: EventPriority::P0,
@@ -2701,14 +2710,14 @@ impl CodexCommandExecutor {
                                 "providerTurnId": previous_active_turn_id,
                                 "status": "failed",
                                 "providerTerminalObserved": false,
-                                "error": if provider_label == "codex" { json!({
+                                "error": if restart_continuable { json!({
                                     "code": "provider_turn_lost_on_restore",
                                     "recoverable": true,
                                     "message": "The runner restored the conversation after process loss, but the previous turn is no longer active.",
                                 }) } else { Value::Null },
                             }),
                         };
-                        if provider_label == "codex" && !state.active_terminal_tool_rejected {
+                        if restart_continuable && !state.active_terminal_tool_rejected {
                             // No provider result was observed. Preserve the lost
                             // turn as a fact, but leave the run open for the
                             // controller's bounded same-conversation recovery.
@@ -5700,6 +5709,13 @@ impl CommandExecutor for CodexCommandExecutor {
 #[cfg(test)]
 mod tests {
 
+    #[test]
+    fn restart_lost_turns_continue_only_for_locally_restored_providers() {
+        assert!(super::restart_lost_turn_is_continuable("codex"));
+        assert!(super::restart_lost_turn_is_continuable("opencode"));
+        assert!(!super::restart_lost_turn_is_continuable("cursor_cloud"));
+    }
+
     #[test]
     fn mcp_permission_requests_use_durable_turn_and_item_identity() {
         let identity = ProviderEventIdentity {

packages/paperclip-runner/runner/crates/runner-core/tests/acpx_warm_attachment.rs

@@ -166,3 +166,168 @@ fn checkpoints_the_sidecar_and_rebinds_consecutive_warm_runs_before_accepting_wo
     assert_eq!(started.result["status"], "accepted");
     executor.shutdown().unwrap();
 }
+
+fn unused_fence_ports() -> [u16; 3] {
+    // Lifetime fence candidates must be dynamic ports (49152 and above). An
+    // OS-assigned port can be lower, for example on Linux.
+    let mut listeners = Vec::new();
+    for port in (49_152..=u16::MAX).rev() {
+        if let Ok(listener) = std::net::TcpListener::bind(("127.0.0.1", port)) {
+            listeners.push(listener);
+            if listeners.len() == 3 {
+                break;
+            }
+        }
+    }
+    let ports = listeners
+        .iter()
+        .map(|listener| listener.local_addr().unwrap().port())
+        .collect::<Vec<_>>();
+    [ports[0], ports[1], ports[2]]
+}
+
+/// A runner that dies during an active ACPX turn (for example, when the
+/// control plane restarts) must restore the same conversation once the old
+/// provider lifetime is proven gone. It records a recoverable restart-loss
+/// cause instead of a terminal provider failure.
+#[test]
+fn restores_the_conversation_after_losing_an_active_turn_to_a_runner_restart() {
+    let fixture =
+        Fixture(std::env::temp_dir().join(format!("acpx-restart-{}", uuid::Uuid::new_v4())));
+    fs::create_dir_all(&fixture.0).unwrap();
+    fs::set_permissions(&fixture.0, fs::Permissions::from_mode(0o700)).unwrap();
+    let sidecar = fixture.0.join("sidecar");
+    fs::copy(env!("CARGO_BIN_EXE_fake-acpx-sidecar"), &sidecar).unwrap();
+    fs::set_permissions(&sidecar, fs::Permissions::from_mode(0o700)).unwrap();
+    let ports = unused_fence_ports();
+    let args = vec![
+        "--mode".to_owned(),
+        "suspend".to_owned(),
+        "--profile-digest".to_owned(),
+        PROFILE_DIGEST.to_owned(),
+        "--lifetime-fence-ports".to_owned(),
+        format!("{},{},{}", ports[0], ports[1], ports[2]),
+    ];
+    let config = DurableRunnerConfig {
+        connect_url: "ws://127.0.0.1/runner".to_owned(),
+        ca_bundle_path: None,
+        state_dir: fixture.0.clone(),
+        runner_instance_id: "runner-1".to_owned(),
+        environment_lease_id: "lease-1".to_owned(),
+        run_id: "run-1".to_owned(),
+        normalized_session_id: "session-1".to_owned(),
+        turn_id: "turn-1".to_owned(),
+        item_id: "item-1".to_owned(),
+        runner_version: "0.0.0".to_owned(),
+        runner_digest: format!("sha256:{}", "a".repeat(64)),
+        acpx_launch_profile: Some(AcpxLaunchProfile {
+            candidate_admission: None,
+            authority_digest: format!("sha256:{}", "d".repeat(64)),
+            command: sidecar.clone(),
+            args: args.clone(),
+            artifacts: vec![QualifiedLaunchArtifact {
+                path: sidecar.clone(),
+                sha256: format!("sha256:{:x}", Sha256::digest(fs::read(&sidecar).unwrap())),
+            }],
+        }),
+        opencode_launch_profile: None,
+        cursor_cloud_launch_profile: None,
+        max_outbox_bytes: 1024 * 1024,
+        p0_reserve_bytes: 64 * 1024,
+        max_frame_bytes: 1024 * 1024,
+        reconnect_delay: Duration::from_millis(1),
+        reconnect_grace: None,
+        max_runtime: Duration::from_secs(30),
+    };
+    let descriptor = json!({
+        "kind": "acpx", "provider": "acpx", "driver": "acpx_runtime",
+        "providerVersion": "0.13.1", "agent": "codex", "model": "gpt-5.6-sol",
+        "acpxVersion": "0.13.1", "agentServerPackage": "@agentclientprotocol/codex-acp",
+        "agentServerVersion": "1.6.2", "agentRuntimePackage": "@openai/codex",
+        "agentRuntimeVersion": "0.160.0", "commandDigest": PROFILE_DIGEST,
+        "sidecarCommand": sidecar, "sidecarArgs": args, "runtimeDirectory": fixture.0,
+        "normalizedSessionId": "session-1", "runId": "run-1", "cwd": fixture.0,
+        "instructions": "Complete the supplied work.",
+        "permissionMode": "approve-all", "permissionModePinned": true,
+    });
+    let mut original = AcpxCommandExecutor::with_runner_config(&fixture.0, &config);
+    original
+        .execute(&command(1, "run.prepare", json!({"provider": descriptor})))
+        .unwrap();
+    let opened = original
+        .execute(&command(2, "session.open", json!({})))
+        .unwrap();
+    let provider_session = opened.result["providerSessionId"].clone();
+    let started = original
+        .execute(&command(
+            3,
+            "turn.start",
+            json!({"turnId": "provider-turn-1", "text": "Do the work."}),
+        ))
+        .unwrap();
+    assert_eq!(started.result["status"], "accepted");
+    // The runner process dies with the turn still active, after the turn sent
+    // the controller a semantic call that has no result yet.
+    drop(original);
+    let state_path = fixture.0.join("acpx-provider-state.json");
+    let mut saved: Value = serde_json::from_slice(&fs::read(&state_path).unwrap()).unwrap();
+    saved["openSemanticCalls"] = json!(["call-lost"]);
+    fs::write(&state_path, serde_json::to_vec(&saved).unwrap()).unwrap();
+
+    let mut restored = AcpxCommandExecutor::with_runner_config(&fixture.0, &config);
+    let snapshot = restored
+        .execute(&command(4, "session.snapshot", json!({})))
+        .unwrap();
+    assert_ne!(snapshot.result["status"], "closed");
+    assert_eq!(snapshot.result["providerSessionId"], provider_session);
+    let events = restored.poll_events().unwrap();
+    let failed = events
+        .iter()
+        .find(|event| event.event_type == "turn.failed")
+        .expect("the lost turn is terminated");
+    assert_eq!(failed.payload["providerTurnId"], "provider-turn-1");
+    assert_eq!(failed.payload["providerTerminalObserved"], false);
+    assert_eq!(
+        failed.payload["error"]["code"],
+        "provider_turn_lost_on_restore"
+    );
+    assert_eq!(failed.payload["error"]["recoverable"], true);
+    // The run stays open for the controller's single same-conversation continuation.
+    assert!(!events
+        .iter()
+        .any(|event| event.event_type == "run.terminal"));
+    // The controller can still answer the lost call. That result settles the
+    // call instead of failing against the restored provider, which never saw it.
+    let lost_result = json!({
+        "callId": "call-lost", "operationId": "register_deliverable",
+        "result": {"ok": true}, "isError": false,
+    });
+    let settled = restored
+        .execute(&command(5, "semantic_tool.result", lost_result.clone()))
+        .unwrap();
+    assert_eq!(settled.result["retired"], true);
+    // The receipt is durable and idempotent, so a delivery left indeterminate
+    // by a crash may replay to the same acknowledgement.
+    assert!(restored.can_reconcile_result_delivery().unwrap());
+    let replayed = restored
+        .execute(&command(6, "semantic_tool.result", lost_result))
+        .unwrap();
+    assert_eq!(replayed.result["retired"], true);
+    // Calls the lost turn never opened still fail closed.
+    let unknown = json!({
+        "callId": "call-unknown", "operationId": "register_deliverable",
+        "result": {"ok": true}, "isError": false,
+    });
+    assert!(restored
+        .execute(&command(7, "semantic_tool.result", unknown))
+        .is_err());
+    let continued = restored
+        .execute(&command(
+            8,
+            "turn.start",
+            json!({"turnId": "provider-turn-2", "text": "Continue."}),
+        ))
+        .unwrap();
+    assert_eq!(continued.result["status"], "accepted");
+    restored.shutdown().unwrap();
+}

packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs

@@ -7,7 +7,7 @@ use std::os::unix::fs::PermissionsExt;
 
 use paperclip_runner_core::durable::{
     AcpxLaunchProfile, Command, CommandExecutor, CursorCloudLaunchProfile, DurableRunnerConfig,
-    OpenCodeLaunchProfile, QualifiedLaunchArtifact, PROVIDER_STATE_INCOMPATIBLE,
+    OpenCodeLaunchProfile, PolledEvent, QualifiedLaunchArtifact, PROVIDER_STATE_INCOMPATIBLE,
 };
 use paperclip_runner_core::native_provider_backend::NativeProviderCommandExecutor;
 use paperclip_runner_core::provider_bridge::{authorized_tool_catalog_digest, AuthorizedTool};
@@ -579,6 +579,30 @@ fn acpx_runtime_resolution(mode: &str, request_id: &str) -> Value {
     })
 }
 
+/// A turn lost to a runner restart is recorded once as a recoverable loss.
+fn assert_restart_lost_turn(events: &[PolledEvent]) {
+    let lost = events
+        .iter()
+        .filter(|event| event.event_type == "turn.failed")
+        .collect::<Vec<_>>();
+    assert_eq!(lost.len(), 1);
+    assert_eq!(
+        lost[0].payload["error"]["code"],
+        "provider_turn_lost_on_restore"
+    );
+    assert_eq!(lost[0].payload["error"]["recoverable"], true);
+    assert!(!events
+        .iter()
+        .any(|event| event.event_type == "run.terminal"));
+}
+
+/// Restarting the restored, idle session again replays no turn or request.
+fn assert_no_replayed_turn_or_request(events: &[PolledEvent]) {
+    assert!(events
+        .iter()
+        .all(|event| event.event_type == "session.resumed"));
+}
+
 fn shutdown_recovered_acpx_fixture(executor: &mut NativeProviderCommandExecutor, directory: &Path) {
     // The fake sidecar advertises the same three lifetime-fence ports for all
     // fixtures. Parallel recovered shutdowns hold a quorum through state fsync,
@@ -679,20 +703,23 @@ fn acpx_response_delivery_survives_crash_before_journaling_without_replaying_the
             .execute(&command(6, "request.resolve", resolve.payload))
             .is_err());
         assert_eq!(recovered.poll_events().unwrap(), events);
+        // The lost turn's conversation is restored for one continuation. Its
+        // outcome is a recoverable restart loss; the response is not replayed.
         assert_eq!(
             recovered
                 .execute(&command(7, "session.snapshot", json!({})))
                 .unwrap()
                 .result["status"],
-            "closed"
+            "session_open"
         );
+        assert_restart_lost_turn(&events);
         // Polling is not an ACK; only the journal's ACK retires the event.
         recovered.acknowledge_events(events.len()).unwrap();
         assert!(recovered.poll_events().unwrap().is_empty());
         drop(recovered);
         let mut recovered_again =
             NativeProviderCommandExecutor::with_runner_config(&directory, &config);
-        assert!(recovered_again.poll_events().unwrap().is_empty());
+        assert_no_replayed_turn_or_request(&recovered_again.poll_events().unwrap());
         shutdown_recovered_acpx_fixture(&mut recovered_again, &directory);
         fs::remove_dir_all(directory).unwrap();
     }
@@ -864,14 +891,15 @@ fn acpx_failed_settlement_ack_replays_the_same_delivered_event_without_expiry()
             .execute(&command(6, "request.resolve", response))
             .is_err());
         assert_eq!(recovered.poll_events().unwrap(), events);
+        assert_restart_lost_turn(&events);
         recovered.acknowledge_events(events.len()).unwrap();
         assert!(recovered.poll_events().unwrap().is_empty());
         shutdown_recovered_acpx_fixture(&mut recovered, &directory);
         drop(recovered);
         let mut recovered_again =
             NativeProviderCommandExecutor::with_runner_config(&directory, &config);
-        assert!(recovered_again.poll_events().unwrap().is_empty());
-        recovered_again.shutdown().unwrap();
+        assert_no_replayed_turn_or_request(&recovered_again.poll_events().unwrap());
+        shutdown_recovered_acpx_fixture(&mut recovered_again, &directory);
         fs::remove_dir_all(directory).unwrap();
     }
 }
@@ -2278,6 +2306,77 @@ fn rejects_a_mutable_opencode_command_outside_the_runner_launch_profile() {
 }
 
 #[test]
+#[cfg_attr(windows, ignore = "verified launch is not supported on Windows yet")]
+fn restart_lost_opencode_turn_records_a_recoverable_cause() {
+    // A local OpenCode server restarts with runnerd, like Codex app-server:
+    // the lost turn keeps its recoverable cause and the run stays open for
+    // the controller's single same-conversation continuation.
+    let directory = temporary_directory("opencode-restart-lost-turn");
+    let mut config = opencode_config(&directory);
+    let proxy_script = directory.join("qualified-opencode-proxy-script");
+    #[cfg(unix)]
+    fs::set_permissions(&proxy_script, fs::Permissions::from_mode(0o700)).unwrap();
+    fs::write(
+        &proxy_script,
+        format!(
+            "#!/bin/sh\nexec '{}' --state-file '{}' --call-log '{}' --require-completion-contract --durable-turn-ids --linger-after-turn-start\n",
+            env!("CARGO_BIN_EXE_fake-codex-app-server"),
+            directory.join("fake-opencode-state.json").display(),
+            directory.join("fake-opencode-calls.log").display(),
+        ),
+    )
+    .unwrap();
+    #[cfg(unix)]
+    fs::set_permissions(&proxy_script, fs::Permissions::from_mode(0o500)).unwrap();
+    config
+        .opencode_launch_profile
+        .as_mut()
+        .unwrap()
+        .proxy_script = qualified_artifact(proxy_script);
+    let mut first = NativeProviderCommandExecutor::with_runner_config(&directory, &config);
+    first
+        .execute(&command(
+            1,
+            "run.prepare",
+            opencode_prepare_payload(&directory),
+        ))
+        .unwrap();
+    first
+        .execute(&command(2, "session.open", json!({})))
+        .unwrap();
+    first
+        .execute(&command(3, "turn.start", json!({"text": "Edit the file."})))
+        .unwrap();
+    drop(first);
+    let fake_state = directory.join("fake-opencode-state.json");
+    let mut saved: Value = serde_json::from_slice(&fs::read(&fake_state).unwrap()).unwrap();
+    saved["activeTurnId"] = Value::Null;
+    fs::write(&fake_state, serde_json::to_vec(&saved).unwrap()).unwrap();
+
+    let mut restored = NativeProviderCommandExecutor::with_runner_config(&directory, &config);
+    restored
+        .execute(&command(4, "session.snapshot", json!({})))
+        .unwrap();
+    let events = restored.poll_events().unwrap();
+    let lost = events
+        .iter()
+        .find(|event| event.event_type == "turn.failed")
+        .expect("the lost turn is terminated");
+    assert_eq!(
+        lost.payload["error"]["code"],
+        "provider_turn_lost_on_restore"
+    );
+    assert_eq!(lost.payload["error"]["recoverable"], true);
+    assert!(!events
+        .iter()
+        .any(|event| event.event_type == "run.terminal"));
+    assert_eq!(opencode_call_count(&directory, "turn/start"), 1);
+    restored.shutdown().unwrap();
+    fs::remove_dir_all(directory).unwrap();
+}
+
+#[test]
+#[cfg_attr(windows, ignore = "verified launch is not supported on Windows yet")]
 fn rejects_opencode_launch_profile_drift_across_fresh_recovery() {
     let directory = temporary_directory("opencode-profile-recovery");
     let config = opencode_config(&directory);

packages/paperclip-runner/src/cli/acpx-runtime-sidecar.test.ts

@@ -467,6 +467,30 @@ describe("qualified ACPX runtime sidecar", () => {
     }
   });
 
+  it("keeps a restored provider's restarted call ids distinct from earlier calls in the run", async () => {
+    // A restored session starts a new provider process whose MCP request ids
+    // restart. The controller keys calls by run and call id, so a reused id
+    // would be mistaken for an earlier, already answered call.
+    const raw = { reportedWorkDisposition: "done", summary: "Complete", evidence: [], verification: [],
+      completionClaim: { contractRevision: "1", objectiveSatisfied: true, criteria: [], remainingWork: [] } };
+    const tools = new Map<string, any>(), emitted: any[] = [];
+    const handler = loadWaitForTool({ tools, emitted, validate: validatePrpStructuredRunResult, callIdNamespace: "rgeneration1" });
+    const commands = loadActualToolCommands(tools);
+    const bridge = await startRunnerToolBridge({ handler, captureSemanticReceipt: () => () => {} });
+    try {
+      const response = fetch(bridge.url, { method: "POST", headers: { Authorization: `Bearer ${bridge.secret}`, "Content-Type": "application/json" },
+        body: JSON.stringify({ jsonrpc: "2.0", id: 2, method: "tools/call", params: { name: "paperclip_finish", arguments: raw } }) });
+      await vi.waitFor(() => expect(tools.has("rgeneration1:2")).toBe(true));
+      expect(emitted).toEqual([expect.objectContaining({ callId: "rgeneration1:2", operationId: "paperclip_finish" })]);
+      await commands.resolve({ callId: "rgeneration1:2", turnId: "test-turn", result: { accepted: true }, error: null });
+      const body = await (await response).json() as ToolCallResponseBody;
+      expect(body.result.isError).not.toBe(true);
+    } finally {
+      for (const pending of tools.values()) pending.cleanup();
+      await bridge.close();
+    }
+  });
+
   it.each([["monitor", true], ["monitor", false], ["same_agent", true], ["same_agent", false]] as const)("resolves an omitted continuation before the production sidecar validator with saved %s authority=%s", async (kind, eligible) => {
     const raw = { reportedWorkDisposition: "yielded", summary: "The review fix is published.", evidence: [], verification: [],
       completionClaim: { contractRevision: "1", objectiveSatisfied: false, criteria: [], remainingWork: [{ description: "Wait for checks.", blocksCompletion: true }] } };
@@ -1752,6 +1776,7 @@ function loadWaitForTool(input: {
   emitted: unknown[];
   validate?: typeof validatePrpStructuredRunResult;
   emit?: (eventType: string, payload: unknown) => boolean;
+  callIdNamespace?: string | null;
 }): (call: RunnerToolCall) => Promise<unknown> {
   const source = readFileSync(
     fileURLToPath(new URL("./acpx-runtime-sidecar.ts", import.meta.url)),
@@ -1764,7 +1789,7 @@ function loadWaitForTool(input: {
   const factory = new Function(
     "boundedIdentity", "tools", "turnId", "emit", "PRP_COMPLETION_TOOL_NAME",
     "PRP_BLOCK_TOOL_NAME", "validatePrpStructuredRunResult", "boundedSidecarValue", "record", "MAX_PENDING_TOOLS", "stringifyAcpxSidecarFrame",
-    "resolveCompletionContinuation", "runId", "normalizingTools", "randomUUID",
+    "resolveCompletionContinuation", "runId", "normalizingTools", "randomUUID", "callIdNamespace",
     `return (${functionSource});`,
   );
   return factory(
@@ -1786,6 +1811,7 @@ function loadWaitForTool(input: {
     "test-run",
     new Set<string>(),
     randomUUID,
+    input.callIdNamespace ?? null,
   );
 }
 

packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts

@@ -176,6 +176,12 @@ let openParams: AcpxSidecarOpenParams | null = null;
 let claudeCompactionCount = 0;
 let runId: string | null = null;
 let turnId: string | null = null;
+/**
+ * A restored session runs in a new provider process whose MCP request ids
+ * restart. The controller keys semantic calls by run and call id, so prefix
+ * this process's ids to keep them distinct from calls earlier in the run.
+ */
+let callIdNamespace: string | null = null;
 let activeCopilotEvidence: CopilotToolEvidence | undefined;
 let sequence = 0;
 let requestSequence = 0;
@@ -374,6 +380,9 @@ async function dispatch(
     );
     host = openedHost;
     openParams = params;
+    callIdNamespace = params.expectedIdentity === undefined
+      ? null
+      : `r${randomUUID().replaceAll("-", "").slice(0, 12)}`;
     emit("runtime.process", {
       role: "sidecar",
       pid: process.pid,
@@ -821,7 +830,10 @@ async function waitForTool(call: RunnerToolCall): Promise<unknown> {
   if (!activeTurnId || !activeRunId || call.signal.aborted) {
     throw new Error("ACPX tool call is not bound to an active turn");
   }
-  const callId = boundedIdentity(call.callId, "callId");
+  const providerCallId = boundedIdentity(call.callId, "callId");
+  const callId = callIdNamespace === null
+    ? providerCallId
+    : boundedIdentity(`${callIdNamespace}:${providerCallId}`, "callId");
   if (tools.has(callId) || normalizingTools.has(callId)) throw new Error("ACPX tool call is duplicated");
   const operationId = boundedIdentity(call.tool, "operationId");
   if (tools.size + normalizingTools.size >= MAX_PENDING_TOOLS) {

packages/paperclip-runner/src/cli/eval-gateway-runtime.test.ts

@@ -73,6 +73,7 @@ it.each([["codex", undefined], ["acpx", "claude"]])("routes %s/%s with only its
       expect(config).toContain('env_key = "PAPERCLIP_AI_PROVIDER_KEY"');
       expect(config).toContain("requires_openai_auth = false");
       expect(config).toContain("shell_snapshot = false");
+      expect(config).toContain("image_generation = false");
       expect(config).toContain('web_search = "disabled"');
       expect(config).not.toContain(token);
       expect((await stat(join(env.CODEX_HOME!, "config.toml"))).mode & 0o777).toBe(0o600);

packages/paperclip-runner/src/cli/eval-gateway-runtime.ts

@@ -76,7 +76,9 @@ export async function prepareEvalGatewayRuntime(
         'name = "Paperclip eval controller"',
         `base_url = ${JSON.stringify(descriptor.baseUrl + "/v1")}`,
         'wire_api = "responses"', 'requires_openai_auth = false', 'env_key = "PAPERCLIP_AI_PROVIDER_KEY"',
-        '[features]', 'shell_snapshot = false', '',
+        // The controller gateway admits no hosted tools, so qualified evals keep
+        // Codex's hosted image tool off even though product runs enable it.
+        '[features]', 'shell_snapshot = false', 'image_generation = false', '',
       ].join("\n"), { mode: 0o600, flag: "wx" });
     } else {
       Object.assign(environment, {

packages/paperclip-runner/src/contracts/native-session-backend.ts

@@ -95,11 +95,15 @@ export function isNativeRestartInterruption(error: unknown): boolean {
   return failure.code === NATIVE_RESTART_INTERRUPTION_CODE && failure.recoverable === true;
 }
 
+const RESTART_CONTINUATION_DRIVER_KINDS: ReadonlySet<string> = new Set(["codex_app_server", "acpx_runtime", "opencode_server"]);
+
 /** The terminal fingerprint survives a crash between interruption and continuation. */
 export function nativeRestartInterruptedTurnId(snapshot: Pick<PersistedNativeSession,
   "driverKind" | "activeTurnId" | "semanticResult" | "terminalTurns" | "dispositionOnlyRecoveryConsumed" | "pendingRuntimeRequests" | "goal"
 >): string | null {
-  if (snapshot.driverKind !== "codex_app_server" || snapshot.activeTurnId != null ||
+  // Runnerd records this cause for Codex app-server and ACPX sessions after
+  // proving the old provider lifetime ended and restoring the conversation.
+  if (!RESTART_CONTINUATION_DRIVER_KINDS.has(snapshot.driverKind ?? "") || snapshot.activeTurnId != null ||
       snapshot.semanticResult != null || snapshot.dispositionOnlyRecoveryConsumed ||
       snapshot.goal != null || snapshot.pendingRuntimeRequests?.length) return null;
   const terminal = Array.isArray(snapshot.terminalTurns) ? snapshot.terminalTurns.at(-1) : null;

packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.context-control.test.ts

@@ -129,7 +129,6 @@ describe("Codex app-server Codex driver", () => {
         "skills.include_instructions": false,
         include_apps_instructions: false,
         include_collaboration_mode_instructions: true,
-        "features.image_generation": false,
       },
       permissions: "paperclip-runner-workspace-only",
       runtimeWorkspaceRoots: [WORKSPACE],
@@ -160,8 +159,6 @@ describe("Codex app-server Codex driver", () => {
         expect.stringContaining(
           'shell_environment_policy.set={PATH="/bin",LANG="C.UTF-8"}',
         ),
-        "--disable",
-        "image_generation",
       ]),
     );
     expect(

packages/paperclip-runner/src/drivers/codex/codex-security-config.test.ts

@@ -1,6 +1,8 @@
 import { createRequire } from "node:module";
 import { execFileSync } from "node:child_process";
-import { dirname, resolve } from "node:path";
+import { mkdtempSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { dirname, join, resolve } from "node:path";
 import { resolvePinnedCodexCommand } from "./codex-command.js";
 import { evalProviderTransportOptions } from "../../cli/eval-provider-runtime.js";
 import { describe, expect, it } from "vitest";
@@ -358,10 +360,59 @@ describe("Codex security configuration", () => {
       "features.plugins": false,
       "features.multi_agent": false,
       "features.memories": false,
-      "features.image_generation": false,
     });
   });
 
+  it("N-12: lets an API-key Codex agent generate images with its existing provider key instead of asking for another", () => {
+    // Codex API path: the agent's OpenAI key reaches only the trusted
+    // app-server process. Codex's built-in image tool uses that provider auth;
+    // with it disabled the agent falls back to an OPENAI_API_KEY script that
+    // the sandboxed shell can never see, so it asks the user for a second key.
+    const source = {
+      HOME: "/host/home", CODEX_HOME: "/host/codex", PATH: "/safe/bin",
+      OPENAI_API_KEY: "agent-provider-key",
+    };
+    const args = createIsolatedCodexAppServerArgs(source);
+    const serialized = args.join("\n");
+    expect(args).not.toContain("image_generation");
+    for (const mode of ["default", "plan"] as const) {
+      expect(createSecuredCodexThreadParams("/workspace", mode, true, false, source).config)
+        .not.toHaveProperty("features.image_generation");
+    }
+    // The tool saves results under CODEX_HOME/generated_images; the agent must
+    // be able to read (and so copy) them into the workspace, nothing else there.
+    for (const profile of ["paperclip-runner-workspace-only", "paperclip-runner-workspace-read-only"]) {
+      const rules = args.find((arg) => arg.startsWith(`permissions.${profile}.filesystem=`))!;
+      expect(rules).toContain('"/host/codex"="none"');
+      expect(rules).toContain('"/host/codex/generated_images"="read"');
+    }
+    // The existing key is reused by the provider, never exposed to the shell.
+    expect(serialized).not.toContain("OPENAI_API_KEY");
+    expect(serialized).not.toContain("agent-provider-key");
+  });
+
+  it("N-12: grants generated images in preconfigured mode only when CODEX_HOME is outside HOME", () => {
+    const inside = createIsolatedCodexAppServerArgs({ HOME: "/users/executor", CODEX_HOME: "/users/executor/.codex", PAPERCLIP_RUNNER_HOME_MODE: "preconfigured" }).join("\n");
+    expect(inside).not.toContain("generated_images");
+    const outside = createIsolatedCodexAppServerArgs({ HOME: "/users/executor", CODEX_HOME: "/var/lib/codex-home", PAPERCLIP_RUNNER_HOME_MODE: "preconfigured" }).join("\n");
+    expect(outside.match(/"\/var\/lib\/codex-home\/generated_images"="read"/g)).toHaveLength(2);
+    expect(outside).not.toContain('"/var/lib/codex-home"=');
+  });
+
+  it("N-12: the pinned Codex binary keeps built-in image generation enabled under the isolated launch", () => {
+    const codexHome = mkdtempSync(join(tmpdir(), "paperclip-n12-codex-home-"));
+    try {
+      const args = createIsolatedCodexAppServerArgs({ PATH: process.env.PATH, HOME: codexHome, CODEX_HOME: codexHome });
+      const output = execFileSync(resolvePinnedCodexCommand(), [...args.slice(0, -1), "features", "list"], {
+        env: { PATH: process.env.PATH, HOME: codexHome, CODEX_HOME: codexHome },
+        encoding: "utf8", timeout: 30_000, maxBuffer: 256 * 1024,
+      });
+      expect(output.split("\n").find((line) => /^image_generation\s/.test(line))).toMatch(/\strue\s*$/);
+    } finally {
+      rmSync(codexHome, { recursive: true, force: true });
+    }
+  });
+
   it("keeps automatic execution inside the workspace without host credentials and with normal network access", () => {
     const args = createIsolatedCodexAppServerArgs(
       {

packages/paperclip-runner/src/drivers/codex/codex-security-config.ts

@@ -190,7 +190,10 @@ const SKILLLESS_BASE_CONFIG = {
   "features.plugins": false,
   "features.multi_agent": false,
   "features.memories": false,
-  "features.image_generation": false,
+  // Image generation stays on: Codex's built-in tool runs in the trusted
+  // app-server with the agent's existing provider auth. Disabling it makes the
+  // agent fall back to an OPENAI_API_KEY script the sandboxed shell cannot see,
+  // so it asks the user for a second key (N-12).
 } as const;
 
 export function codexCommandEnvironment(
@@ -288,6 +291,16 @@ export function createIsolatedCodexAppServerArgs(
         .map((value) => resolve(value)),
     ),
   ];
+  // Codex saves built-in image results under CODEX_HOME/generated_images and
+  // tells the agent to copy them into the workspace. Expose only that folder.
+  // A preconfigured HOME is already granted, so add the rule only when the
+  // Codex home sits outside it (never narrow an existing HOME write grant).
+  const generatedImagesRoot = typeof source.CODEX_HOME === "string" && source.CODEX_HOME.trim()
+    ? join(resolve(source.CODEX_HOME), "generated_images") : undefined;
+  const preconfiguredHome = usesPreconfiguredHome(source) && source.HOME ? resolve(source.HOME) : undefined;
+  const generatedImageRoots = generatedImagesRoot
+    && !(preconfiguredHome && codexPathInside(preconfiguredHome, generatedImagesRoot))
+    ? [generatedImagesRoot] : [];
   const filesystemRules = [
     `":root"="none"`,
     `":minimal"="read"`,
@@ -296,6 +309,7 @@ export function createIsolatedCodexAppServerArgs(
     ...protectedPaths.map((path) => `${tomlString(resolve(path))}="none"`),
     ...(usesPreconfiguredHome(source) && source.HOME ? [`${tomlString(source.HOME)}="write"`] : []),
     ...readOnlyRoots.map((path) => `${tomlString(resolve(path))}="read"`),
+    ...generatedImageRoots.map((path) => `${tomlString(path)}="read"`),
     ...gitRoots.read.map((path) => `${tomlString(path)}="read"`),
     ...gitRoots.write.map((path) => `${tomlString(path)}="write"`),
     ...(instructionWorkingCopyRoot ? [`${tomlString(resolve(instructionWorkingCopyRoot))}="write"`] : []),
@@ -311,6 +325,7 @@ export function createIsolatedCodexAppServerArgs(
     ...protectedPaths.map((path) => `${tomlString(resolve(path))}="none"`),
     ...(usesPreconfiguredHome(source) && source.HOME ? [`${tomlString(source.HOME)}="read"`] : []),
     ...readOnlyRoots.map((path) => `${tomlString(resolve(path))}="read"`),
+    ...generatedImageRoots.map((path) => `${tomlString(path)}="read"`),
     ...[...gitRoots.read, ...gitRoots.write].map((path) => `${tomlString(path)}="read"`),
     ...(instructionWorkingCopyRoot ? [`${tomlString(resolve(instructionWorkingCopyRoot))}="read"`] : []),
     ...(source.PAPERCLIP_GITHUB_BROKER_TOKEN && source.GH_CONFIG_DIR
@@ -358,8 +373,6 @@ export function createIsolatedCodexAppServerArgs(
     // as the turn plan (`turn/plan/updated` -> `plan.updated`).
     "-c",
     CODEX_UPDATE_PLAN_TOOL_CONFIG,
-    "--disable",
-    "image_generation",
     ...(externalRunnerSandbox
       ? ["--dangerously-bypass-approvals-and-sandbox"]
       : []),

packages/paperclip-runner/src/native-session-runtime.test.ts

@@ -20,6 +20,7 @@ import {
   NativeSessionCloseUnrecoverableError,
   NativeSessionCleanupQuarantinedError,
   NativeSessionProtocolIntegrityError,
+  nativeRestartInterruptedTurnId,
 } from "./contracts/native-session-backend.js";
 import { PROVIDER_STATE_INCOMPATIBLE } from "./contracts/harness-driver.js";
 import type {
@@ -508,6 +509,71 @@ describe("executeNativeSession recovery", () => {
     expect(checkpoints.some(snapshot => snapshot.terminalTurns?.some(turn => turn.turnId === "turn-recovery"))).toBe(true);
   });
 
+  it.each(["acpx_runtime", "opencode_server"] as const)("continues a restart-lost %s turn once in the same conversation", async (driverKind) => {
+    const qualified = resolveQualifiedAcpxProfile("claude", "claude-sonnet-5");
+    const acpxInput: NativeExecutionInputV1 = (driverKind === "acpx_runtime" ? {
+      ...input,
+      session: { ...input.session, driverKind },
+      provider: { kind: "acpx", agent: "claude", model: "claude-sonnet-5", permissionPolicy: "interactive", profile: {
+        driverKind: qualified.driverKind, protocolVersion: qualified.protocolVersion, acpxVersion: qualified.acpxVersion,
+        agent: qualified.agent, agentProfileVersion: qualified.agentProfileVersion,
+        agentServerPackage: qualified.agentServerPackage, agentServerVersion: qualified.agentServerVersion,
+        agentRuntimePackage: qualified.agentRuntimePackage, agentRuntimeVersion: qualified.agentRuntimeVersion,
+        commandDigest: qualified.commandDigest,
+      } },
+    } : {
+      ...input,
+      session: { ...input.session, driverKind },
+      provider: { kind: "opencode", model: "openrouter/deepseek/deepseek-v4-flash-0731" },
+    }) as NativeExecutionInputV1;
+    const failure = { code: "provider_turn_lost_on_restore", recoverable: true };
+    const checkpoint: PersistedNativeSession = {
+      backendKind: "runner", driverKind, sessionId: "record-recovery", identity,
+      providerSessionId: "provider-recovery", cursor: "0", activeTurnId: "turn-recovery",
+      terminalTurns: [], semanticResult: null, pendingRuntimeRequests: [], lineage: [],
+    };
+    const interrupted: PersistedNativeSession = {
+      ...checkpoint, activeTurnId: null,
+      terminalTurns: [{ turnId: "turn-recovery", fingerprint: JSON.stringify({ terminalState: "failed", error: failure, result: null }) }],
+    };
+    const startTurn = vi.fn<NativeSession["startTurn"]>(async () => ({ turnId: "turn-continuation" }));
+    const session: NativeSession = {
+      identity: () => identity,
+      async capabilities() { return { resume: true, typedEvents: true, steering: false, interruption: true, structuredResult: true }; },
+      async *events() {
+        yield runnerEvent(1, "turn.failed", { status: "failed", error: failure });
+        if (!startTurn.mock.calls.length) return;
+        yield { ...runnerEvent(2, "turn.completed", { status: "completed" }), turnId: "turn-continuation" };
+      },
+      startTurn,
+      async result() { return startTurn.mock.calls.length ? { result, terminal, turnId: "turn-continuation" } : null; },
+      async snapshot() { return structuredClone(interrupted); },
+      async close() {},
+    };
+    const backend: NativeSessionBackend = {
+      async descriptor() { return { kind: "runner", name: "restart", version: "1", capabilities: await session.capabilities() }; },
+      async openSession() { throw new Error("must preserve the provider session"); },
+      async recoverSession() { return { recovered: true, session }; },
+    };
+    const port: ControlPlanePort = {
+      async openRun() {},
+      async checkpointSession() {},
+      async appendEvent(event: PrpEvent) { return { cursor: event.sourceSeq, highestContiguousSourceSeq: event.sourceSeq, disposition: "committed" }; },
+      async replayEvents() { return { events: [], highestContiguousSourceSeq: 0 }; },
+      async completeRun() {},
+    };
+    await expect(executeNativeSession({
+      input: acpxInput, backend, controlPlane: port, persistedSession: checkpoint,
+      runnerInstanceId: "runner-recovery", controlPlaneInstanceId: "control-recovery",
+      resumeInterruptedTurn: true,
+    })).resolves.toMatchObject({ result, turnId: "turn-continuation" });
+    expect(startTurn).toHaveBeenCalledOnce();
+    const submitted = startTurn.mock.calls[0]![0] as Parameters<NativeSession["startTurn"]>[0];
+    expect(submitted.continuation).toBe(true);
+    expect(submitted.message.text).toContain("Reconcile any unfinished tool or command");
+    expect(nativeRestartInterruptedTurnId(interrupted)).toBe("turn-recovery");
+  });
+
   it("never continues task work when a restart loses a compaction turn", async () => {
     const failure = { code: "provider_turn_lost_on_restore", recoverable: true };
     const checkpoint: PersistedNativeSession = {

packages/paperclip-runner/src/native-session-runtime.ts

@@ -2448,7 +2448,9 @@ export async function executeNativeSession(
     const resumeInterruptedTurn = async (event: PrpEvent, signal: AbortSignal) => {
       if (restartContinuationStarted || !recovered || !options.resumeInterruptedTurn ||
           // A compaction-only run never continues task work after a restart.
-          input.provider.kind !== "codex" || options.sessionCompaction || options.sessionGoalControl || options.resumeSessionGoalHeartbeat ||
+          // Runnerd records a restart-lost turn only for these providers, after
+          // proving the old process ended and restoring the same conversation.
+          !["codex", "acpx", "opencode"].includes(input.provider.kind) || options.sessionCompaction || options.sessionGoalControl || options.resumeSessionGoalHeartbeat ||
           recoveredSnapshot.goal || persistedSession?.semanticResult || persistedSession?.pendingRuntimeRequests?.length ||
           event.eventType !== "turn.failed" || !isNativeRestartInterruption(event.payload.error) ||
           event.turnId !== persistedSession?.activeTurnId) return false;

packages/shared/src/validators/tool-access.ts

@@ -191,7 +191,8 @@ export const createToolConnectionSchema = z.object({
   status: toolConnectionStatusSchema.optional(),
   connectionKind: toolConnectionKindSchema.default("managed"),
   config: toolTransportConfigSchema.optional(),
-  transportConfig: toolTransportConfigSchema.default({}),
+  // No default: when the caller omits it, the service stores `config` here.
+  transportConfig: toolTransportConfigSchema.optional(),
   credentialRefs: z.array(mcpConnectionCredentialRefSchema).optional(),
   credentialSecretRefs: z.array(toolCredentialSecretRefSchema).default([]),
   enabled: z.boolean().optional(),

server/src/__tests__/deferred-execution-receipt-work.test.ts

@@ -15,7 +15,7 @@ import { issueTreeControlService } from "../services/issue-tree-control.js";
 import { randomUUID } from "node:crypto";
 import { and, eq, sql } from "drizzle-orm";
 import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
-import { activityLog, agents, agentWakeupRequests, authUsers, chatActions, environmentLeases, nativeRunFinalizations, chatEndpoints, toolApplications, toolConnections, companies, companyMemberships, costEvents, createDb, projects, reconcileDatabaseWork, heartbeatRuns, issueComments, issues } from "@paperclipai/db";
+import { activityLog, agents, agentWakeupRequests, authUsers, chatActions, environmentLeases, nativeRunFinalizations, chatEndpoints, toolApplications, toolConnections, companies, companyMemberships, costEvents, createDb, projects, reconcileDatabaseWork, heartbeatRuns, issueComments, issueRelations, issues } from "@paperclipai/db";
 import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
 import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
 import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
@@ -158,6 +158,96 @@ describe("deferred receipt durable dispatch", () => {
       source: "on_demand", status: "completed", reason: "task_retry", payload: { issueId: f.issue.id } });
     await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready; await quiet();
   });
+  async function strandedDependency() {
+    const f = await fixture();
+    await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: f.issue.id,
+      status: "succeeded", errorCode: null, contextSnapshot: { issueId: f.issue.id,
+        wakeReason: "issue_commented", dependencyBlockedInteraction: true } }).where(eq(heartbeatRuns.id, f.source.id));
+    await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issue.id,
+      runId: f.source.id, phase: "committed" });
+    await db.update(issues).set({ status: "blocked", executionRunId: null, checkoutRunId: null }).where(eq(issues.id, f.issue.id));
+    const [blocker] = await db.insert(issues).values({ companyId: f.companyId, title: "Completed review",
+      status: "done", completedAt: new Date() }).returning();
+    await db.insert(issueRelations).values({ companyId: f.companyId, issueId: blocker.id,
+      relatedIssueId: f.issue.id, type: "blocks" });
+    const [intent] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agent.id,
+      source: "automation", reason: "issue_blockers_resolved", status: "coalesced",
+      requestedByActorType: "system", requestedByActorId: "native-status-committer",
+      payload: { issueId: f.issue.id, resolvedBlockerIssueId: blocker.id },
+    }).returning();
+    const [receipt] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agent.id,
+      source: "automation", reason: "issue_execution_deferred", status: "deferred_issue_execution",
+      requestedByActorType: "system", requestedByActorId: `native-status-wake-dispatch:${intent.id}`,
+      payload: { issueId: f.issue.id, resolvedBlockerIssueId: blocker.id, _paperclipWakeContext: {
+        issueId: f.issue.id, wakeReason: "issue_blockers_resolved", statusDecisionSource: "native_status_decision",
+        nativeStatusWakeIntentId: intent.id } },
+    }).returning();
+    return { ...f, receipt, blocker };
+  }
+
+  it.each(["restart", "cleanup"] as const)("recovers a dependency wake after native audit %s into one continuation", async event => {
+    const f = await strandedDependency();
+    const { receipt } = f;
+    const [lease] = event === "cleanup" ? await db.insert(environmentLeases).values({ companyId: f.companyId,
+      heartbeatRunId: f.source.id, issueId: f.issue.id, provider: "daytona", providerLeaseId: "dependency-finalizing",
+      status: "active", leasePolicy: "reuse_by_environment" }).returning() : [];
+    await heartbeat.registerDeferredReceiptWork(coordinator, () => !drain).ready;
+    await heartbeat.registerRunDispatchWork(coordinator, () => !drain).ready;
+    if (lease) {
+      await waitForReal(async () => {
+        await Promise.allSettled([...passes]);
+        expect(coordinator.nextWakeAt()).toBeNull(); expect(idleWorkSnapshot().active).toBe(0);
+      }, { timeout: 10_000 });
+      expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.id)))[0])
+        .toMatchObject({ status: "deferred_issue_execution", runId: null });
+      await environmentService(db).releaseLease(lease.id, "retained", { cleanupStatus: "success" });
+    }
+    await waitForReal(async () => {
+      await Promise.allSettled([...passes]); await heartbeat.drainActiveRunExecutions();
+      const [saved] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.id));
+      const pending = saved.runId ? (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, saved.runId)))[0] : null;
+      expect(saved.status, JSON.stringify({ status: pending?.status, error: pending?.error, errors })).toBe("completed");
+      expect(saved.runId).not.toBeNull();
+    }, { timeout: 15_000 });
+    await finish(f.issue.id);
+    const runs = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId));
+    const dependencyRuns = runs.filter(run => run.contextSnapshot?.wakeReason === "issue_blockers_resolved");
+    expect(dependencyRuns).toHaveLength(1);
+    const [resumed] = dependencyRuns;
+    expect(resumed).toMatchObject({ status: "succeeded", stdoutExcerpt: expect.stringContaining("receipt-proof"),
+      contextSnapshot: { wakeReason: "issue_blockers_resolved" } });
+    expect(resumed.contextSnapshot?.dependencyBlockedInteraction).not.toBe(true);
+    await heartbeat.resumeQueuedRuns();
+    const after = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId));
+    expect(after.filter(run => run.contextSnapshot?.wakeReason === "issue_blockers_resolved")).toHaveLength(1);
+    expect(after.find(run => run.id === f.source.id)?.status).toBe("succeeded");
+    expect((await db.select().from(issues).where(eq(issues.id, f.issue.id)))[0])
+      .toMatchObject({ status: "done", executionRunId: null, checkoutRunId: null });
+    await quiet();
+  });
+  it.each(["paused", "cleanup", "cancelled"] as const)("keeps the %s gate when recovering a dependency wake", async gate => {
+    const f = await strandedDependency();
+    if (gate === "paused") await db.update(agents).set({ status: "paused" }).where(eq(agents.id, f.agent.id));
+    if (gate === "cancelled") await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, f.issue.id));
+    if (gate === "cleanup") await db.insert(environmentLeases).values({ companyId: f.companyId,
+      heartbeatRunId: f.source.id, issueId: f.issue.id, provider: "daytona", providerLeaseId: "dependency-cleanup",
+      status: "active", leasePolicy: "reuse_by_environment" });
+    await heartbeat.resumeQueuedRuns();
+    const [saved] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.receipt.id));
+    expect(saved).toMatchObject({ status: gate === "cancelled" ? "cancelled" : "deferred_issue_execution", runId: null });
+    expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
+  });
+  it("checks unresolved blockers before a recovered dependency wake can execute", async () => {
+    const f = await strandedDependency();
+    await db.update(issues).set({ status: "blocked", completedAt: null }).where(eq(issues.id, f.blocker.id));
+    await heartbeat.resumeQueuedRuns();
+    await heartbeat.drainActiveRunExecutions();
+    const [receipt] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.receipt.id));
+    expect(receipt).toMatchObject({ status: "skipped", runId: expect.any(String) });
+    const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, receipt.runId!));
+    expect(run).toMatchObject({ status: "cancelled", errorCode: "issue_dependencies_blocked", stdoutExcerpt: null });
+    expect((await db.select().from(issues).where(eq(issues.id, f.issue.id)))[0].status).toBe("blocked");
+  });
   it.each([false, true])("reconsiders every saved receipt during repeated company eligibility changes (expired wait: %s)", async expiredWait => {
     const f = await fixture();
     const ids = Array.from({ length: 178 }, (_, index) => `${(index + 1).toString(16).padStart(8, "0")}-0000-4000-8000-000000000000`);

server/src/__tests__/environment-routes.test.ts

@@ -1022,6 +1022,20 @@ describe("environment routes", () => {
 
     describe("managed plugin environment rows", () => {
       const PLUGIN_KEY = "example.remote-host-plugin";
+      const MANAGED_CONFIG_WITH_PLUGIN_ENTRY = JSON.stringify({
+        v: 1,
+        mode: "cloud",
+        catalogVersion: "2026.1106.0",
+        features: {},
+        plugins: { autoInstall: ["remote-host"] },
+        environments: [{
+          name: "Remote Host",
+          driver: "plugin",
+          pluginKey: PLUGIN_KEY,
+          driverKey: "tasks",
+          driverConfig: {},
+        }],
+      });
       const pluginRow = () => ({
         ...createPlatformSandboxEnvironment(),
         id: "env-plugin-1",
@@ -1058,20 +1072,46 @@ describe("environment routes", () => {
         expect(res.body.details).toMatchObject({ code: "environment_platform_managed" });
       });
 
-      it("allows the marker clear while no managed-config entry can own the row", async () => {
-        // Nothing provisions marked plugin rows yet, so a marker is stale and
-        // stays recoverable. The managed-config plugin entry PR refuses this
-        // while the document declares the row's entry.
-        const row = pluginRow();
-        mockEnvironmentService.getById.mockResolvedValue(row);
-        mockEnvironmentService.update.mockResolvedValue({ ...row, metadata: {} });
+      it("refuses a marker clear while the document declares the entry", async () => {
+        process.env.PAPERCLIP_MANAGED_CONFIG = MANAGED_CONFIG_WITH_PLUGIN_ENTRY;
+        try {
+          mockEnvironmentService.getById.mockResolvedValue(pluginRow());
+          const app = createApp(ownerAdminActor);
 
-        const res = await request(createApp(ownerAdminActor))
-          .patch("/api/environments/env-plugin-1")
-          .send({ metadata: { managedByPaperclip: null, managedPluginEnvironment: null } });
+          for (const metadata of [
+            { managedByPaperclip: false },
+            { managedPluginEnvironment: null },
+            { managedByPaperclip: null, managedPluginEnvironment: null },
+          ]) {
+            const res = await request(app).patch("/api/environments/env-plugin-1").send({ metadata });
+            expect(res.status).toBe(403);
+            expect(res.body.details).toMatchObject({ code: "environment_platform_managed" });
+          }
+          expect(mockEnvironmentService.update).not.toHaveBeenCalled();
+        } finally {
+          delete process.env.PAPERCLIP_MANAGED_CONFIG;
+        }
+      });
 
-        expect(res.status).toBe(200);
-        expect(mockEnvironmentService.update).toHaveBeenCalled();
+      it("allows the marker clear once the entry left the document", async () => {
+        process.env.PAPERCLIP_MANAGED_CONFIG = JSON.stringify({
+          ...JSON.parse(MANAGED_CONFIG_WITH_PLUGIN_ENTRY),
+          environments: [],
+        });
+        try {
+          const row = pluginRow();
+          mockEnvironmentService.getById.mockResolvedValue(row);
+          mockEnvironmentService.update.mockResolvedValue({ ...row, metadata: {} });
+
+          const res = await request(createApp(ownerAdminActor))
+            .patch("/api/environments/env-plugin-1")
+            .send({ metadata: { managedByPaperclip: null, managedPluginEnvironment: null } });
+
+          expect(res.status).toBe(200);
+          expect(mockEnvironmentService.update).toHaveBeenCalled();
+        } finally {
+          delete process.env.PAPERCLIP_MANAGED_CONFIG;
+        }
       });
 
       it("refuses a client that stamps the plugin marker on its own row", async () => {

server/src/__tests__/execution-control-work.test.ts

@@ -61,6 +61,41 @@ describe("execution-control lifecycle", () => {
     expect(run).toHaveBeenCalledTimes(runs + 1);
     expect(c.nextWakeAt()).toBeNull();
   });
+  it.each(["empty", "pending", "concurrent"] as const)("settles only proved-empty replacement notifications during drain (%s)", async kind => {
+    vi.useFakeTimers(); const owner = fakeOwner(); let draining = false;
+    const c = coordinator(owner, () => !draining);
+    const run = vi.fn(async () => {}), next = vi.fn(async (): Promise<number | null> => null);
+    await registerExecutionControlWork(c, [{ queue: DELIVERY_QUEUES.executionReplacement, run, next,
+      settleEmptyWhilePaused: true }]).ready;
+    const before = idleWorkSnapshot().active;
+    draining = true;
+    let release: (() => void) | undefined;
+    if (kind === "pending") next.mockResolvedValue(Date.now() + 15_000);
+    if (kind === "concurrent") next.mockImplementationOnce(() => new Promise(resolve => {
+      release = () => resolve(null);
+    }));
+    await owner.transaction(tx => notifyDeliveryWork(tx, DELIVERY_QUEUES.executionReplacement));
+    await vi.advanceTimersByTimeAsync(15_000);
+    if (kind === "concurrent") {
+      expect(release).toBeTypeOf("function");
+      next.mockResolvedValue(Date.now() + 15_000);
+      await owner.transaction(tx => notifyDeliveryWork(tx, DELIVERY_QUEUES.executionReplacement));
+      release!(); await vi.advanceTimersByTimeAsync(1);
+    }
+    expect(run).toHaveBeenCalledTimes(1);
+    expect(idleWorkSnapshot().active).toBe(before + (kind === "empty" ? 0 : 1));
+    const reads = next.mock.calls.length;
+    await vi.advanceTimersByTimeAsync(60_000);
+    expect(next).toHaveBeenCalledTimes(reads);
+    expect(run).toHaveBeenCalledTimes(1);
+    if (kind !== "empty") {
+      draining = false; next.mockResolvedValue(null);
+      await vi.advanceTimersByTimeAsync(15_000);
+      expect(run).toHaveBeenCalledTimes(2);
+    }
+    expect(idleWorkSnapshot().active).toBe(before);
+    expect(c.nextWakeAt()).toBeNull();
+  });
   it("keeps a change arriving during a sweep and does not overlap it", async () => {
     vi.useFakeTimers(); const owner = fakeOwner(); const c = coordinator(owner);
     let release!: () => void;

server/src/__tests__/helpers/historical-listener-continuation.ts

@@ -0,0 +1,134 @@
+import { createServer } from "node:http";
+import { spawn } from "node:child_process";
+import { createRequire } from "node:module";
+import { join, resolve } from "node:path";
+import { mkdir, readFile, writeFile } from "node:fs/promises";
+import { eq } from "drizzle-orm";
+import { completionContracts, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db";
+import { expect } from "vitest";
+import { createRunDispatch } from "../../modules/run-dispatch/index.js";
+import { prepareNativeHeartbeatRun } from "../../services/native-runtime/prepare-native-run.js";
+import { buildNativeExecutionInput } from "../../services/native-runtime/native-execution-input.js";
+import { buildNativeCompletionContract } from "../../services/native-runtime/completion-contracts.js";
+import { nativeRuntimeContextFixture } from "../../services/native-runtime/runtime-context.test-fixture.js";
+import { materializeAsset } from "../../services/native-runtime/runtime-context.js";
+import { executePaperclipNativeSession } from "../../services/native-runtime/native-session-executor.js";
+import { reconcileNativeFinalizations } from "../../services/native-runtime/native-finalization-reconciler.js";
+import { canonicalNativeRuntimeContextDigest, createNativeSessionBackend, createRunnerdCodexTransport, defaultCapabilityRunnerdBinary } from "../../vendor/paperclip-runner/index.js";
+import { registerLocalRuntimeApiServer } from "../../runtime-api.js";
+import { registerRunnerPrpAuthority, runnerPrpWebSocketInternals, setupRunnerPrpWebSocketServer } from "../../realtime/runner-prp-ws.js";
+
+async function prepare(db: Db, runId: string, cwd: string, environmentLeaseId: string) {
+  const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
+  const [issue] = await db.select().from(issues).where(eq(issues.id, run.nativeIssueId ?? String(run.contextSnapshot?.issueId)));
+  await db.update(issues).set({ executionRunId: run.id }).where(eq(issues.id, issue.id));
+  const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId });
+  const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issue.id));
+  const runtimeContext = nativeRuntimeContextFixture();
+  runtimeContext.instructions.bundle = await materializeAsset([{ path: "AGENTS.md", mode: 0o444,
+    content: Buffer.from("Complete the saved task once.\n") }]);
+  runtimeContext.aggregateDigest = canonicalNativeRuntimeContextDigest(runtimeContext);
+  const execution = buildNativeExecutionInput({ companyId: run.companyId, agentId: run.agentId, runId, issue,
+    taskPrompt: "Preserve the saved file and complete the task once.", provider: "codex",
+    workspace: { id: issue.executionWorkspaceId!, cwd, repoUrl: null, repoRef: null, branchName: null },
+    normalizedSessionId: native.normalizedSessionId, runtimeContext,
+    completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1",
+      contract: buildNativeCompletionContract(issue, { revision: contract.revision }) },
+  });
+  const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
+  await db.update(heartbeatRuns).set({ runnerProfileJson: { ...run.runnerProfileJson, ...current.runnerProfileJson, nativeExecutionInput: execution } })
+    .where(eq(heartbeatRuns.id, runId));
+  return { run: current, native, execution };
+}
+
+export async function recordHistoricalListenerFailure(input: {
+  db: Db; connectionString: string; runId: string; cwd: string; environmentId: string; leaseId: string;
+}) {
+  await input.db.update(heartbeatRuns).set({ status: "running", finishedAt: null, error: null, errorCode: null })
+    .where(eq(heartbeatRuns.id, input.runId));
+  const { execution, native } = await prepare(input.db, input.runId, input.cwd, input.leaseId);
+  await input.db.insert(nativeRunFinalizations).values({ runId: input.runId, companyId: execution.binding.companyId,
+    issueId: execution.binding.issueId, phase: "observed" });
+  const child = spawn(process.execPath, ["--import", createRequire(import.meta.url).resolve("tsx"),
+    resolve(import.meta.dirname, "native-drain-controller.test-worker.ts")], { stdio: ["ignore", "pipe", "pipe", "ipc"], env: process.env });
+  const logs: string[] = [];
+  child.stdout?.on("data", data => logs.push(String(data)));
+  child.stderr?.on("data", data => logs.push(String(data)));
+  const exited = new Promise(resolveExit => child.once("exit", resolveExit));
+  try {
+    const result = await new Promise((resolveResult, reject) => {
+      const timeout = setTimeout(() => reject(new Error(logs.join("").slice(-3000))), 60_000);
+      child.once("error", reject);
+      child.on("message", (message: { ready?: boolean; done?: boolean; error?: string }) => {
+        if (message.ready) child.send({ connectionString: input.connectionString, options: { execution,
+          runnerInstanceId: native.runnerInstanceId, useRunnerd: true,
+          runnerExecutionTarget: { kind: "remote", transport: "sandbox", environmentId: input.environmentId,
+            leaseId: input.leaseId, remoteCwd: input.cwd } } });
+        if (message.done || message.error) { clearTimeout(timeout); message.error ? reject(new Error(message.error)) : resolveResult(message); }
+      });
+    });
+    expect(result).toMatchObject({ commands: 0, errorMessage: "local_runtime_api_listener_unavailable" });
+    expect(await exited).toBe(0);
+    const [failed] = await input.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, input.runId));
+    expect(failed).toMatchObject({ phase: "terminal_failure", attempt: 1, failureCode: "native_session_interrupted",
+      failureDetail: { message: "local_runtime_api_listener_unavailable", recoveryMode: "ambiguous_state",
+        providerSessionEstablished: false, providerEventsExist: false, checkpointExists: false } });
+  } finally { child.kill("SIGKILL"); await exited; }
+}
+
+export async function completeHistoricalListenerSuccessor(input: { db: Db; runId: string; root: string;
+  prepareWorkspace: () => Promise<{ cwd: string; finalize: () => Promise<void> }> }) {
+  const [scheduled] = await input.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId));
+  const dispatch = createRunDispatch(input.db);
+  expect(await dispatch.promoteScheduledRetry({ companyId: scheduled.companyId, runId: scheduled.id,
+    now: new Date(scheduled.scheduledRetryAt!.getTime() + 1) })).toMatchObject({ outcome: "promoted" });
+  expect(await dispatch.cancelStaleQueuedRun({ companyId: scheduled.companyId, runId: scheduled.id, expectedStatus: "queued" }))
+    .toMatchObject({ outcome: "not_stale" });
+  await input.db.update(heartbeatRuns).set({ status: "running", runtimeMode: "native",
+    nativeIssueId: String(scheduled.contextSnapshot?.issueId) }).where(eq(heartbeatRuns.id, scheduled.id));
+  await input.db.update(issues).set({ executionRunId: scheduled.id }).where(eq(issues.id, String(scheduled.contextSnapshot?.issueId)));
+  const workspace = await input.prepareWorkspace();
+  const { execution, native } = await prepare(input.db, scheduled.id, workspace.cwd, `controlled-${scheduled.id}`);
+  await input.db.insert(nativeRunFinalizations).values({ runId: scheduled.id, companyId: scheduled.companyId,
+    issueId: execution.binding.issueId, phase: "observed" });
+  const home = join(input.root, "codex-home"), report = join(input.root, "result.json"), calls = join(input.root, "calls.log");
+  await mkdir(home, { recursive: true });
+  await writeFile(report, JSON.stringify({ schema: "paperclip.run_result.v1", reportedWorkDisposition: "done",
+    summary: "Saved task completed after historical controller failure.", completionClaim: {
+      contractRevision: execution.completionContract.contract.revision, objectiveSatisfied: true,
+      criteria: execution.completionContract.contract.criteria.map(criterion => ({ criterionId: criterion.id, status: "satisfied", evidenceRefs: [] })),
+      remainingWork: [] }, evidence: [], verification: [], artifacts: [], attentionRequests: [] }));
+  const server = createServer();
+  registerLocalRuntimeApiServer(server);
+  await new Promise<void>(resolveListen => server.listen(0, "127.0.0.1", resolveListen));
+  const address = server.address();
+  if (!address || typeof address === "string") throw new Error("TCP listener required");
+  setupRunnerPrpWebSocketServer(server, { apiUrl: `http://127.0.0.1:${address.port}` });
+  const bundle = createRunnerdCodexTransport({ runnerBinary: defaultCapabilityRunnerdBinary(),
+    codexCommand: resolve(import.meta.dirname, "../../../../packages/paperclip-runner/runner/target/debug/fake-codex-app-server"),
+    codexArgs: ["--state-file", join(input.root, "provider.json"), "--terminal-result-file", report, "--call-log", calls, "--record-process-start"],
+    stateDirectory: input.root, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null },
+    runtimeContext: execution.runtimeContext, runnerRuntimeContext: execution.runtimeContext,
+    sourceCodexHome: home, environment: { CODEX_HOME: home, PATH: process.env.PATH },
+    prpIdentity: { runnerInstanceId: native.runnerInstanceId, environmentLeaseId: native.environmentLeaseId,
+      runId: scheduled.id, normalizedSessionId: native.normalizedSessionId, turnId: native.turnId, itemId: native.itemId },
+    controlPlaneRegistration: authority => registerRunnerPrpAuthority({ companyId: scheduled.companyId, runId: scheduled.id, authority }),
+  });
+  const backend = createNativeSessionBackend(execution, { runnerInstanceId: native.runnerInstanceId,
+    environment: { CODEX_HOME: home, PATH: process.env.PATH }, codexTransportFactory: () => bundle.transport });
+  try {
+    expect(await executePaperclipNativeSession({ db: input.db, execution, backend, useRunnerd: true,
+      runnerInstanceId: native.runnerInstanceId, turnTimeoutMs: 15_000 })).toMatchObject({ exitCode: 0, timedOut: false });
+    await workspace.finalize();
+    await reconcileNativeFinalizations(input.db, [scheduled.id]);
+    const [finished] = await input.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, scheduled.id));
+    expect(finished, JSON.stringify(finished.failureDetail)).toMatchObject({ phase: "committed", attempt: 2 });
+    expect(await input.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, scheduled.id)))
+      .toMatchObject([{ schemaStatus: "accepted" }]);
+    expect((await readFile(calls, "utf8")).split("\n").filter(line => line === "process-start")).toHaveLength(1);
+    expect(await readFile(join(workspace.cwd, "saved.txt"), "utf8")).toBe("work survives a failed provider session\n");
+  } finally {
+    await bundle.transport.close(); runnerPrpWebSocketInternals.resetForTests();
+    server.closeAllConnections(); await new Promise<void>(resolveClose => server.close(() => resolveClose()));
+  }
+}

server/src/__tests__/helpers/native-drain-controller.test-worker.ts

@@ -0,0 +1,50 @@
+import { createServer } from "node:http";
+import { closeRegisteredClients, createDb, heartbeatRunEvents } from "@paperclipai/db";
+import { registerLocalRuntimeApiServer } from "../../runtime-api.js";
+import { setStartupRecoveryPhase } from "../../startup-recovery-state.js";
+import { heartbeatService } from "../../services/heartbeat.js";
+import { executePaperclipNativeSession } from "../../services/native-runtime/native-session-executor.js";
+
+process.send?.({ ready: true });
+process.once("message", async (input: {
+  connectionString: string;
+  options: Omit<Parameters<typeof executePaperclipNativeSession>[0], "db">;
+  providerEvidence?: boolean;
+}) => {
+  const api = createServer((_request, response) => response.writeHead(404).end());
+  registerLocalRuntimeApiServer(api);
+  let commands = 0;
+  let errorName: string | undefined;
+  let errorMessage: string | undefined;
+  try {
+    await new Promise<void>(resolve => api.listen(0, "127.0.0.1", resolve));
+    const db = createDb(input.connectionString);
+    try {
+      await executePaperclipNativeSession({ ...input.options, db,
+        runnerExecutionTarget: { ...input.options.runnerExecutionTarget!,
+          runner: { execute: async () => { commands++; throw new Error("Drained controller launched remote work"); } },
+        } as never,
+        onEvent: async event => {
+          if (event.payload?.span !== "native.coordinator.claim") return;
+          if (input.providerEvidence) await db.insert(heartbeatRunEvents).values({
+            companyId: input.options.execution.binding.companyId, agentId: input.options.execution.binding.agentId,
+            runId: input.options.execution.binding.runId, seq: 1_000_000, eventType: "turn.started",
+            sourceEventId: "controlled-provider-evidence" });
+          setStartupRecoveryPhase("draining");
+          await new Promise<void>((resolve, reject) => api.close(error => error ? reject(error) : resolve()));
+        },
+      });
+    } catch (error) {
+      errorName = error instanceof Error ? error.name : String(error);
+      errorMessage = error instanceof Error ? error.message : String(error);
+    }
+    const heartbeat = heartbeatService(db);
+    try {
+      await heartbeat.drainRunningRunsForShutdown("SIGTERM", new Date(), [input.options.execution.binding.runId]);
+    } finally { await heartbeat.drainActiveRunExecutions(); }
+    await closeRegisteredClients(input.connectionString, { scope: "database" });
+    process.send?.({ done: true, commands, errorName, errorMessage }, () => process.exit(0));
+  } catch (error) {
+    process.send?.({ error: error instanceof Error ? error.stack : String(error) }, () => process.exit(1));
+  }
+});

server/src/__tests__/instance-settings-cloud-defaults.test.ts

@@ -17,6 +17,7 @@ function managedConfig(features: ManagedInstanceConfig["features"] = {}): Manage
     features,
     plugins: { autoInstall: [] },
     environments: [],
+    pluginEnvironments: [],
   };
 }
 

server/src/__tests__/instance-settings-service.test.ts

@@ -26,7 +26,7 @@ describe("instance settings service", () => {
     }
     expect(applyExperimentalSettingsPatch({}, { enableAgentChat: false }).enableAgentChat).toBe(true);
     const config: ManagedInstanceConfig = {
-      v: 1, mode: "cloud", catalogVersion: "test", plugins: { autoInstall: [] }, environments: [],
+      v: 1, mode: "cloud", catalogVersion: "test", plugins: { autoInstall: [] }, environments: [], pluginEnvironments: [],
       features: { enableAgentChat: false },
     };
     const result = applyManagedExperimentalOverlay(normalizeExperimentalSettings({ enableAgentChat: false }), config);
@@ -54,7 +54,7 @@ describe("instance settings service", () => {
     { stored: { remoteWorkspaceStorageEnabled: true }, managed: false, expected: false },
   ])("respects Cloud enrollment defaults and managed choices: $expected", ({ stored, managed, expected }) => {
     const config: ManagedInstanceConfig = {
-      v: 1, mode: "cloud", catalogVersion: "test", plugins: { autoInstall: [] }, environments: [],
+      v: 1, mode: "cloud", catalogVersion: "test", plugins: { autoInstall: [] }, environments: [], pluginEnvironments: [],
       features: managed === undefined ? {} : { remoteWorkspaceStorageEnabled: managed },
     };
     const normalized = normalizeExperimentalSettings(stored);

server/src/__tests__/managed-config.test.ts

@@ -1,6 +1,7 @@
 import { describe, expect, it } from "vitest";
 import {
   MANAGED_CONFIG_ENV_KEY,
+  MANAGED_ENVIRONMENT_DRIVERS,
   getManagedInstanceConfig,
   managedFeatureKeySet,
   parseManagedConfigEnv,
@@ -110,6 +111,7 @@ describe("parseManagedConfigEnv", () => {
       features: { enableApps: false, enableStatusCards: true },
       plugins: { autoInstall: ["daytona", "kubernetes"] },
       environments: [],
+      pluginEnvironments: [],
     });
   });
 
@@ -132,6 +134,7 @@ describe("parseManagedConfigEnv", () => {
       features: {},
       plugins: { autoInstall: [] },
       environments: [],
+      pluginEnvironments: [],
     });
   });
 
@@ -342,12 +345,17 @@ describe("parseManagedConfigEnv environments section", () => {
     ).toThrow(/"environments\[0\]" must be an object/);
   });
 
-  it("rejects more than one entry (single managed sandbox slot)", () => {
+  it("rejects more than one sandbox entry (single managed sandbox slot)", () => {
     expect(() =>
       parseManagedConfigEnv(
         envWith(validDoc({ environments: [entry(), entry({ name: "Other", provider: "kubernetes" })] })),
       ),
-    ).toThrow(/at most one entry/);
+    ).toThrow(/at most one sandbox entry/);
+    expect(() =>
+      parseManagedConfigEnv(
+        envWith(validDoc({ environments: [entry(), entry({ name: "Other", driver: "sandbox", provider: "kubernetes" })] })),
+      ),
+    ).toThrow(/at most one sandbox entry/);
   });
 
   it("rejects unknown entry keys", () => {
@@ -406,6 +414,94 @@ describe("parseManagedConfigEnv environments section", () => {
   });
 });
 
+describe("parseManagedConfigEnv plugin environment entries", () => {
+  const PLUGIN_KEY = "example.remote-host-plugin";
+  const remoteHost = (overrides: Record<string, unknown> = {}) => ({
+    name: "Remote Host",
+    description: "Runs the agent on a remote host",
+    driver: "plugin",
+    pluginKey: PLUGIN_KEY,
+    driverKey: "tasks",
+    driverConfig: {},
+    ...overrides,
+  });
+  const sandbox = { name: "Managed Sandbox", provider: "daytona", config: { target: "us" } };
+  // `plugins.autoInstall` holds catalog keys ("remote-host"); the entry's
+  // `pluginKey` is the plugin's manifest id, which boot maps through the catalog.
+  const docWith = (environments: unknown[], autoInstall = ["daytona", "remote-host"]) =>
+    envWith(validDoc({ plugins: { autoInstall }, environments }));
+
+  it("lists the drivers an entry may declare", () => {
+    expect(MANAGED_ENVIRONMENT_DRIVERS).toEqual(["sandbox", "plugin"]);
+  });
+
+  it("parses an old document's sandbox entry unchanged", () => {
+    const config = parseManagedConfigEnv(docWith([sandbox]));
+    expect(config?.environments).toEqual([{ name: "Managed Sandbox", provider: "daytona", config: { target: "us" } }]);
+    expect(config?.pluginEnvironments).toEqual([]);
+  });
+
+  it("parses the sandbox entry beside a plugin entry", () => {
+    const config = parseManagedConfigEnv(docWith([sandbox, remoteHost()]));
+    expect(config?.environments).toEqual([{ name: "Managed Sandbox", provider: "daytona", config: { target: "us" } }]);
+    expect(config?.pluginEnvironments).toEqual([{
+      driver: "plugin",
+      name: "Remote Host",
+      description: "Runs the agent on a remote host",
+      pluginKey: PLUGIN_KEY,
+      driverKey: "tasks",
+      driverConfig: {},
+    }]);
+    expect(Object.isFrozen(config?.pluginEnvironments)).toBe(true);
+    expect(Object.isFrozen(config?.pluginEnvironments[0])).toBe(true);
+  });
+
+  it("parses a plugin entry alone, with optional fields omitted", () => {
+    const config = parseManagedConfigEnv(docWith([{ name: "Remote", driver: "plugin", pluginKey: PLUGIN_KEY, driverKey: "tasks" }]));
+    expect(config?.environments).toEqual([]);
+    expect(config?.pluginEnvironments[0]).toMatchObject({ driverConfig: {} });
+    expect(config?.pluginEnvironments[0]?.description).toBeUndefined();
+  });
+
+  it("does not compare the manifest id with autoInstall catalog keys (boot checks the catalog)", () => {
+    expect(parseManagedConfigEnv(docWith([remoteHost()]))?.pluginEnvironments[0]?.pluginKey).toBe(PLUGIN_KEY);
+  });
+
+  it("refuses duplicate names, including across drivers", () => {
+    expect(() => parseManagedConfigEnv(docWith([sandbox, remoteHost({ name: "Managed Sandbox" })])))
+      .toThrow(/"environments\[1\].name" duplicates another entry's name/);
+  });
+
+  it("refuses a duplicate pluginKey and driverKey pair", () => {
+    expect(() => parseManagedConfigEnv(docWith([remoteHost(), remoteHost({ name: "Second" })])))
+      .toThrow(/duplicates another plugin entry's pluginKey and driverKey/);
+  });
+
+  it("refuses secret-like driverConfig keys at any depth", () => {
+    expect(() => parseManagedConfigEnv(docWith([remoteHost({ driverConfig: { apiToken: "t" } })])))
+      .toThrow(/"environments\[0\].driverConfig" key "apiToken" looks secret-bearing/);
+    expect(() => parseManagedConfigEnv(docWith([remoteHost({ driverConfig: { auth: [{ password: "p" }] } })])))
+      .toThrow(/auth\[0\].password/);
+  });
+
+  it("refuses unknown keys, including sandbox keys on a plugin entry and the reverse", () => {
+    expect(() => parseManagedConfigEnv(docWith([remoteHost({ envVars: {} })]))).toThrow(/unknown key "envVars"/);
+    expect(() => parseManagedConfigEnv(docWith([remoteHost({ provider: "daytona" })]))).toThrow(/unknown key "provider"/);
+    expect(() => parseManagedConfigEnv(docWith([{ ...sandbox, pluginKey: PLUGIN_KEY }]))).toThrow(/unknown key "pluginKey"/);
+    // Not supported by this build on either entry.
+    expect(() => parseManagedConfigEnv(docWith([remoteHost({ instanceDefault: true })]))).toThrow(/unknown key "instanceDefault"/);
+    expect(() => parseManagedConfigEnv(docWith([{ ...sandbox, instanceDefault: false }]))).toThrow(/unknown key "instanceDefault"/);
+  });
+
+  it("refuses an unknown driver and malformed plugin fields", () => {
+    expect(() => parseManagedConfigEnv(docWith([remoteHost({ driver: "ssh" })]))).toThrow(/"environments\[0\].driver" must be one of "sandbox", "plugin"/);
+    expect(() => parseManagedConfigEnv(docWith([{ ...sandbox, driver: null }]))).toThrow(/"environments\[0\].driver" must be one of "sandbox", "plugin" when present \(got null\)/);
+    expect(() => parseManagedConfigEnv(docWith([remoteHost({ driverKey: "Tasks" })]))).toThrow(/"environments\[0\].driverKey" must start/);
+    expect(() => parseManagedConfigEnv(docWith([remoteHost({ driverKey: undefined })]))).toThrow(/"environments\[0\].driverKey" must be a non-empty string/);
+    expect(() => parseManagedConfigEnv(docWith([remoteHost({ driverConfig: [] })]))).toThrow(/"environments\[0\].driverConfig" must be an object/);
+  });
+});
+
 describe("getManagedInstanceConfig", () => {
   it("caches by raw env value and reparses when it changes", () => {
     const raw = validDoc();

server/src/__tests__/managed-plugin-environments.test.ts

@@ -7,15 +7,42 @@ import {
   companies,
   createDb,
   environments,
+  instanceSettings,
   plugins,
   type Db,
 } from "@paperclipai/db";
 import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
 import { environmentService } from "../services/environments.js";
+import { MANAGED_CONFIG_ENV_KEY, parseManagedConfigEnv } from "../services/managed-config.js";
+import { applyManagedEnvironments } from "../services/managed-environments.js";
 
 const PLUGIN_KEY = "example.remote-host-plugin";
 const MARKER = `${PLUGIN_KEY}:tasks`;
 
+type ReadyListener = (payload: { pluginId: string }) => void;
+
+function managedConfig(entries: unknown[]) {
+  return parseManagedConfigEnv({
+    [MANAGED_CONFIG_ENV_KEY]: JSON.stringify({
+      v: 1,
+      mode: "cloud",
+      catalogVersion: "2026.1106.0",
+      features: {},
+      plugins: { autoInstall: ["remote-host"] },
+      environments: entries,
+    }),
+  });
+}
+
+const remoteHostEntry = {
+  name: "Remote Host",
+  description: "Runs the agent on a remote host",
+  driver: "plugin",
+  pluginKey: PLUGIN_KEY,
+  driverKey: "tasks",
+  driverConfig: {},
+};
+
 const support = await getEmbeddedPostgresTestSupport();
 (support.supported ? describe : describe.skip)("managed plugin environment rows", () => {
   let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
@@ -24,7 +51,17 @@ const support = await getEmbeddedPostgresTestSupport();
   let running: boolean;
   const call = vi.fn(async (_id: string, method: string, input: { config: Record<string, unknown> }) =>
     method === "environmentValidateConfig" ? { ok: true, normalizedConfig: { ...input.config, normalized: true } } : null);
-  const workerManager = { isRunning: vi.fn(() => running), call };
+  let readyListeners: ReadyListener[] = [];
+  const workerManager = {
+    isRunning: vi.fn(() => running),
+    call,
+    getWorker: vi.fn(() => ({
+      on: (_event: "ready", listener: ReadyListener) => { readyListeners.push(listener); },
+      off: (_event: "ready", listener: ReadyListener) => {
+        readyListeners = readyListeners.filter((candidate) => candidate !== listener);
+      },
+    })),
+  };
 
   beforeAll(async () => {
     database = await startEmbeddedPostgresTestDatabase("paperclip-managed-plugin-environment-rows-");
@@ -33,9 +70,11 @@ const support = await getEmbeddedPostgresTestSupport();
   afterAll(async () => { await database?.cleanup(); });
   beforeEach(async () => {
     running = true;
+    readyListeners = [];
     call.mockClear();
     await db.delete(activityLog);
     await db.delete(agents);
+    await db.delete(instanceSettings);
     await db.delete(environments);
     await db.delete(plugins);
     await db.delete(companies);
@@ -175,4 +214,42 @@ const support = await getEmbeddedPostgresTestSupport();
       metadata: { managedPluginEnvironment: MARKER },
     })).rejects.toThrow();
   });
+
+  describe("boot (applyManagedEnvironments)", () => {
+    const boot = (entries: unknown[] = [remoteHostEntry]) =>
+      applyManagedEnvironments(db, managedConfig(entries), { env: {}, workerManager, autoInstalledPluginKeys: [PLUGIN_KEY] });
+    const instanceDefault = async () =>
+      (await db.select().from(instanceSettings).where(eq(instanceSettings.singletonKey, "default")))[0]?.defaultEnvironmentId ?? null;
+
+    it("creates the row on the first boot and changes nothing on the second", async () => {
+      const first = await boot();
+      expect(first).toMatchObject({ ensured: 1, added: 1, failed: 0 });
+      const [row] = await pluginRows();
+      expect(row).toMatchObject({ name: "Remote Host", status: "active", metadata: { managedByPaperclip: true, managedPluginEnvironment: MARKER } });
+
+      const second = await boot();
+      expect(second).toMatchObject({ ensured: 1, added: 0, unchanged: 1 });
+      const after = await pluginRows();
+      expect(after).toHaveLength(1);
+      expect(after[0]!.updatedAt.getTime()).toBe(row!.updatedAt.getTime());
+      expect(await instanceDefault()).toBeNull();
+    });
+
+    it("archives the row while the plugin is down and reactivates it when the worker recovers", async () => {
+      await boot();
+      running = false;
+      expect(await boot()).toMatchObject({ ensured: 0, failed: 1 });
+      expect((await pluginRows())[0]!.status).toBe("archived");
+
+      running = true;
+      for (const listener of [...readyListeners]) listener({ pluginId });
+      await vi.waitFor(async () => expect((await pluginRows())[0]?.status).toBe("active"));
+    });
+
+    it("never archives a row whose entry left the document", async () => {
+      await boot();
+      expect(await boot([])).toBeNull();
+      expect((await pluginRows())[0]!.status).toBe("active");
+    });
+  });
 });

server/src/__tests__/status-card-notifications.test.ts

@@ -25,7 +25,7 @@ it.each([false, true])("keeps execution queue consumers intact and emits only a
     const write = vi.fn(async () => "written");
     expect(await writeExecutionControl(owner as never, DELIVERY_QUEUES.workspaceRecovery, write, scope)).toBe("written");
     expect(write).toHaveBeenCalledOnce();
-    expect(new Set(settled.map(row => row.queue))).toEqual(new Set([DELIVERY_QUEUES.issueMonitors, DELIVERY_QUEUES.workspaceRecovery,
+    expect(new Set(settled.map(row => row.queue))).toEqual(new Set([DELIVERY_QUEUES.executionReplacement, DELIVERY_QUEUES.issueMonitors, DELIVERY_QUEUES.workspaceRecovery,
       ...(scoped ? [DELIVERY_QUEUES.statusCardEvaluation] : [])]));
   } finally { for (const stop of unsubscribe) stop(); }
 });

server/src/__tests__/task-interruption-query.test.ts

@@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto";
 import { eq } from "drizzle-orm";
 import { createRunSecretRedactionRegistry } from "../services/run-secret-redaction.js";
 import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
-import { agents, agentWakeupRequests, companies, createDb, applyPendingMigrations, heartbeatRuns, issues, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db";
+import { agents, agentWakeupRequests, companies, createDb, applyPendingMigrations, heartbeatRuns, issues, nativeRunFinalizations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db";
 import { taskInterruptionPresentation, taskInterruptionPresentations } from "../services/task-interruption-presentation.js";
 import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
 import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
@@ -24,6 +24,28 @@ describe.skipIf(!support.supported)("task interruption latest run query", () =>
   }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
   afterAll(async () => { await db?.$client.end(); await database?.cleanup(); });
 
+  it("shows a pending native same-run resume instead of a competing Retry", async () => {
+    const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID();
+    await db.insert(companies).values({ id: companyId, name: "Resuming task", issuePrefix: `R${companyId.slice(0, 7)}` });
+    await db.insert(agents).values({ id: agentId, companyId, name: "Resuming agent", adapterType: "paperclip_runner" });
+    const [issue] = await db.insert(issues).values({ companyId, title: "Resume after attach failure",
+      status: "in_progress", assigneeAgentId: agentId }).returning();
+    await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issue.id, scopeKind: "issue",
+      status: "failed", runtimeMode: "native", errorCode: "native_session_interrupted", finishedAt: new Date() });
+    const resumeAt = new Date(Date.now() + 30_000);
+    await db.insert(nativeRunFinalizations).values({ companyId, issueId: issue.id, runId, phase: "retryable_failure",
+      attempt: 1, nextAttemptAt: resumeAt });
+    for (const presentation of [(await taskInterruptionPresentations(db, companyId, [issue])).get(issue.id),
+      await taskInterruptionPresentation(db, issue)]) {
+      expect(presentation).toMatchObject({ state: "retry_scheduled", canRetry: false, nextRetryAt: resumeAt.toISOString() });
+    }
+    // Exhausted recovery is terminal: exactly one Retry returns.
+    await db.update(nativeRunFinalizations).set({ phase: "terminal_failure", nextAttemptAt: null })
+      .where(eq(nativeRunFinalizations.runId, runId));
+    expect((await taskInterruptionPresentations(db, companyId, [issue])).get(issue.id))
+      .toMatchObject({ state: "failed", canRetry: true });
+  });
+
   it("redacts workspace preparation errors on task detail and list reads", async () => {
     vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "a".repeat(64));
     try {

server/src/__tests__/tool-access-service.test.ts

@@ -68,6 +68,7 @@ import {
   GOOGLE_WORKSPACE_DEFAULT_SELECTION,
   getAvailableConnectionMethod,
   getConnectableAppDefinition,
+  createToolConnectionSchema,
   type CreateToolConnection,
   type GoogleWorkspaceConnectorProfileId,
 } from "@paperclipai/shared";
@@ -3363,6 +3364,42 @@ describeEmbeddedPostgres("tool access service", () => {
       .where(eq(companySecrets.companyId, company.id))).toEqual([]);
   });
 
+  it.each([
+    { name: "channel purpose", input: { connectionPurpose: "channel" as const, transport: "local_stdio" as const } },
+    { name: "chat transport", input: { transport: "chat_sdk" as const } },
+    { name: "voice transport", input: { transport: "voice" as const } },
+  ])("rejects a $name before it writes any row", async ({ input }) => {
+    const company = await createCompany(db);
+    const service = createTestToolAccessService(db);
+
+    await expect(service.createConnection(company.id, connectionInput({
+      name: "Not a tool",
+      config: { templateId: "paperclip.echo-calculator-time" },
+      ...input,
+    }))).rejects.toMatchObject({ status: 422 });
+
+    expect(await db.select().from(toolConnections)
+      .where(eq(toolConnections.companyId, company.id))).toEqual([]);
+    expect(await db.select().from(toolApplications)
+      .where(eq(toolApplications.companyId, company.id))).toEqual([]);
+  });
+
+  it("stores config as transportConfig when the request leaves transportConfig out", async () => {
+    const company = await createCompany(db);
+    const service = createTestToolAccessService(db);
+    const body = createToolConnectionSchema.parse({
+      name: "Local echo config only",
+      transport: "local_stdio",
+      config: { templateId: "paperclip.echo-calculator-time" },
+    });
+
+    const connection = await service.createConnection(company.id, body);
+
+    const [stored] = await db.select().from(toolConnections).where(eq(toolConnections.id, connection.id));
+    expect(stored?.config).toMatchObject({ templateId: "paperclip.echo-calculator-time" });
+    expect(stored?.transportConfig).toEqual(stored?.config);
+  });
+
   it("registers an approved local stdio template and exposes its runtime slot", async () => {
     const company = await createCompany(db);
     const service = createTestToolAccessService(db);

server/src/app.ts

@@ -1348,6 +1348,9 @@ export async function createApp(
       logger.error({ err }, "Failed to load ready plugins on startup");
     });
   app.locals.bundledPluginsStartup = trackIdleWork(bundledPluginsStartup);
+  // Manifest ids the managed auto-install list resolves to; the managed
+  // environments bootstrap checks plugin entries against them.
+  app.locals.managedBundledPluginKeys = managedBundledPluginKeys;
   void bundledPluginsStartup.then(() => { lifecyclePluginsReady = true; resourceLifecycleDelivery.wake(); agentLifecycleDelivery.wake(); }).catch(() => logger.warn("Agent lifecycle recovery failed; retrying."));
   // The shutdown hook runs at most once. It caches the in-flight promise, so a
   // second caller (for example the `exit` handler) awaits the same completion

server/src/index.ts

@@ -81,7 +81,7 @@ import { embeddedPostgresOwnerPort } from "./embedded-postgres-owner.js";
 import { deliverExecutionStatuses } from "./services/execution-status-delivery.js";
 import { deliverReconciledExecutions, settleUnrecoverableExecutions } from "./services/execution-recovery-resolution.js";
 import { deliverPendingPauseResumes } from "./services/issue-tree-resume.js";
-import { reconcileSafeNativeReplacements } from "./services/native-runtime/native-safe-replacement.js";
+import { invalidateNativeReplacementDiscovery, reconcileSafeNativeReplacements } from "./services/native-runtime/native-safe-replacement.js";
 import { reconcileAbandonedExecutionControl } from "./services/execution-control-reconciliation.js";
 import { acknowledgeNativeStartupCleanup } from "./services/native-runtime/native-startup-cleanup.js";
 import { executionControlSchedule, registerExecutionControlWork } from "./services/execution-control-work.js";
@@ -1175,8 +1175,8 @@ async function startServerWithDatabaseTeardown(
     throw err;
   }
 
-  // Ensure sandbox environments declared in the managed-config document
-  // (`environments` section) before the heartbeat resumes queued runs. The
+  // Ensure the sandbox and plugin environments declared in the managed-config
+  // document (`environments` section) before the heartbeat resumes queued runs. The
   // document already parsed fail-closed above; the ensure step itself is
   // fail-safe per entry (a degraded boot beats a fleet-wide crash loop), but
   // a contradictory deployment that also forces PAPERCLIP_EXECUTION_MODE
@@ -1186,14 +1186,16 @@ async function startServerWithDatabaseTeardown(
   // additionally gates each entry on a live plugin worker (and archives the
   // row of a provider that did not come up).
   try {
-    const bundledPluginsStartup = (app as { locals?: { bundledPluginsStartup?: Promise<unknown> } })
-      .locals?.bundledPluginsStartup;
+    const appLocals = (app as {
+      locals?: { bundledPluginsStartup?: Promise<unknown>; managedBundledPluginKeys?: readonly string[] };
+    }).locals;
     const managedEnvironmentsResult = await applyManagedEnvironments(db as any, managedConfig, {
-      pluginsReady: bundledPluginsStartup,
+      pluginsReady: appLocals?.bundledPluginsStartup,
+      autoInstalledPluginKeys: appLocals?.managedBundledPluginKeys,
       workerManager: pluginWorkerManager,
     });
     if (managedEnvironmentsResult) {
-      logger.warn(managedEnvironmentsResult, "managed sandbox environments ensured from managed config");
+      logger.warn(managedEnvironmentsResult, "managed environments ensured from managed config");
     }
   } catch (err) {
     logger.error({ err }, "failed to apply managed environments from managed config");
@@ -1342,7 +1344,7 @@ async function startServerWithDatabaseTeardown(
     { queue: DELIVERY_QUEUES.executionStatus, run: () => deliverExecutionStatuses(db), next: executionSchedule.status },
     { queue: DELIVERY_QUEUES.executionDisposition, run: () => settleUnrecoverableExecutions(db), next: executionSchedule.disposition },
     ...(heartbeat ? [
-      { queue: DELIVERY_QUEUES.executionReplacement, run: () => reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession: run => verifyStoppedNativeSessionForReplacement(db, run) }), next: executionSchedule.replacement },
+      { queue: DELIVERY_QUEUES.executionReplacement, run: () => reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession: run => verifyStoppedNativeSessionForReplacement(db, run) }), next: executionSchedule.replacement, onSettled: () => invalidateNativeReplacementDiscovery(db), settleEmptyWhilePaused: true },
       { queue: DELIVERY_QUEUES.executionContinuation, run: async () => {
         await deliverReconciledExecutions(db, heartbeat.wakeup);
         await deliverPendingPauseResumes(db, heartbeat.wakeup);

server/src/routes/environments.ts

@@ -19,7 +19,10 @@ import {
 import { conflict, forbidden, unprocessable } from "../errors.js";
 import { isCloudManagedInstance } from "../services/cloud-instance.js";
 import { getManagedInstanceConfig, SECRET_LIKE_CONFIG_KEY_PATTERN } from "../services/managed-config.js";
-import { MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY } from "../services/managed-plugin-environments.js";
+import {
+  MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY,
+  managedPluginEnvironmentMarker,
+} from "../services/managed-plugin-environments.js";
 import { parseExecutionPolicyBootstrapEnv } from "../services/execution-policy-bootstrap.js";
 import { isExecutionForcedToKubernetes } from "../services/execution-allowlist.js";
 import { validate } from "../middleware/validate.js";
@@ -182,6 +185,20 @@ function isManagedSandboxProvisioningConfigured(): boolean {
   }
 }
 
+/**
+ * Whether the managed-config document declares the plugin entry this row's
+ * marker names. Fails closed like `isManagedSandboxProvisioningConfigured`.
+ */
+function isManagedPluginEnvironmentDeclared(marker: string): boolean {
+  try {
+    return (getManagedInstanceConfig()?.pluginEnvironments ?? []).some(
+      (spec) => managedPluginEnvironmentMarker(spec) === marker,
+    );
+  } catch {
+    return true;
+  }
+}
+
 /**
  * Whether this row occupies a provisioner-owned environment slot whose
  * platform markers are LIVE state — i.e. some platform code path converges
@@ -196,6 +213,9 @@ function isManagedSandboxProvisioningConfigured(): boolean {
  *   owned only while a managed-sandbox bootstrap path is configured —
  *   `ensureManagedSandboxEnvironment` then adopts and refreshes whichever
  *   row holds the slot on every boot.
+ * - The row of a managed-config plugin entry (its
+ *   `managedPluginEnvironment` marker) while the document declares that
+ *   entry: `applyManagedEnvironments` converges the markers on it every boot.
  * - Any sandbox row bearing the legacy kubernetes marker while the
  *   persisted instance execution policy forces kubernetes:
  *   `findKubernetesEnvironment` selects marked rows (newest first) for
@@ -214,6 +234,10 @@ async function isPlatformSlotEnvironment(
   options: { isForcedKubernetesExecution: () => Promise<boolean> },
 ): Promise<boolean> {
   if (environment.driver === "local") return true;
+  if (environment.driver === "plugin") {
+    const pluginMarker = environment.metadata?.[MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY];
+    return typeof pluginMarker === "string" && isManagedPluginEnvironmentDeclared(pluginMarker);
+  }
   if (environment.driver !== "sandbox") return false;
   if (
     environment.metadata?.managedByPaperclip === true &&

server/src/services/delivery-work-coordinator.ts

@@ -25,7 +25,7 @@ export function createDeliveryWorkCoordinator(input: {
       // do not postpone the obligation indefinitely.
       coalesceMs?: number;
       run: (signal: AbortSignal) => Promise<unknown>;
-      hasPending: () => Promise<boolean>;
+      hasPending: (runAllowed?: boolean) => Promise<boolean>;
       // Scoped hints belong to the outer transaction, including uncertainty recovery.
       onSettled?: (hints?: ReadonlySet<string>) => void;
       // Optional maintenance/retry deadline restored from durable state.
@@ -75,7 +75,7 @@ export function createDeliveryWorkCoordinator(input: {
         let runFailed = false;
         controller = attempt;
         running = withDatabaseOperation(`delivery.${queue}`, () => Promise.resolve().then(() => reconcileDatabaseWork(input.owner, queue))
-          .then(() => workerStopped || !(input.canRun() || (task.runDuringDrain && input.canReconcile?.())) ? undefined : task.run(attempt.signal).catch(error => { runFailed = true; throw error; })).then(() => workerStopped ? false : task.hasPending()).then(pending => {
+          .then(() => workerStopped || !(input.canRun() || (task.runDuringDrain && input.canReconcile?.())) ? undefined : task.run(attempt.signal).catch(error => { runFailed = true; throw error; })).then(() => workerStopped ? false : task.hasPending(input.canRun() || Boolean(task.runDuringDrain && input.canReconcile?.()))).then(pending => {
           pendingWork = pending || databaseWorkPending(input.owner, queue);
           if (pendingWork) schedule(task.retryMs);
           else { finishIntent?.(); finishIntent = undefined; }

server/src/services/environment-cleanup-notifications.ts

@@ -31,7 +31,10 @@ export async function notifyEnvironmentCleanup(transaction: object, scope?: Envi
   await notifyNativeEnvironmentSettlement(transaction, scope);
   await notifyQueuedRunDispatch(transaction, scope);
   // A settled run lease can make that run's abandoned native owner releasable.
-  if (scope?.runId) await notifyDeliveryWork(transaction, DELIVERY_QUEUES.executionFinalization);
+  if (scope?.runId) {
+    await notifyDeliveryWork(transaction, DELIVERY_QUEUES.executionFinalization);
+    await notifyDeliveryWork(transaction, DELIVERY_QUEUES.executionReplacement);
+  }
   await notifyDeliveryWork(transaction, DELIVERY_QUEUES.environmentCleanup);
   await notifyDeliveryWork(transaction, DELIVERY_QUEUES.historicalCleanup);
 }

server/src/services/execution-control-work.ts

@@ -5,7 +5,7 @@ import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.
 import { type DeliveryQueue } from "./delivery-work-notifications.js";
 import { EXECUTION_RECONCILIATION_INTERVAL_MS } from "./execution-control-deadline.js";
 import { obsoleteExecutionConversationHold, unresolvedExecutionDispositions } from "./execution-recovery-resolution.js";
-import { nativeReplacementPending } from "./native-runtime/native-safe-replacement.js";
+import { hasPendingNativeReplacements } from "./native-runtime/native-safe-replacement.js";
 import { nextAbandonedNativeOwnerReleaseAt } from "./native-runtime/native-failed-run-owner-release.js";
 import { hasPendingPauseResume } from "./issue-tree-resume.js";
 
@@ -16,6 +16,7 @@ export function registerExecutionControlWork(
     run: () => Promise<unknown>;
     next: () => Promise<number | null>;
     onSettled?: () => void;
+    settleEmptyWhilePaused?: boolean;
   }>,
   canRun: () => boolean = () => true,
 ) {
@@ -24,6 +25,7 @@ export function registerExecutionControlWork(
     let reconciled = 0;
     let next: number | null = null;
     let earliestRunAt = 0;
+    let pausedGeneration = 0, pausedPending = true;
     const worker = coordinator.register(duty.queue, {
       retryMs: EXECUTION_RECONCILIATION_INTERVAL_MS,
       pausePassiveMaintenance: true,
@@ -42,7 +44,18 @@ export function registerExecutionControlWork(
         next = deadline === null ? null : Math.max(deadline, earliestRunAt);
         reconciled = generation;
       },
-      hasPending: async () => reconciled !== requested,
+      hasPending: async (runAllowed = true) => {
+        if (reconciled === requested) return false;
+        if (runAllowed || !duty.settleEmptyWhilePaused || !canRun() || reconciled === 0) return true;
+        if (pausedGeneration !== requested) {
+          const generation = requested;
+          const deadline = await duty.next();
+          if (generation !== requested) return true;
+          pausedGeneration = generation; pausedPending = deadline !== null; next = deadline;
+          if (!pausedPending) reconciled = generation;
+        }
+        return pausedPending;
+      },
       nextRunAt: () => next,
     });
     return { ...worker, wake: () => { requested += 1; worker.wake(); } };
@@ -74,10 +87,7 @@ export function executionControlSchedule(db: Db) {
       return at === null ? null : deadline([new Date(at)]);
     },
     async replacement() {
-      const rows = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
-        .innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id))
-        .where(nativeReplacementPending()).limit(1);
-      return rows.length ? retry() : null;
+      return await hasPendingNativeReplacements(db) ? retry() : null;
     },
     async continuation() {
       const rows = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and(

server/src/services/explicit-native-continuation.test.ts

@@ -1586,7 +1586,7 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes
     await persistHeartbeatRunProcessMetadata(db, source.id, { pid: 999999999, processGroupId: null, startedAt: new Date().toISOString() });
     await db.update(heartbeatRuns).set({ processPid: null }).where(eq(heartbeatRuns.id, source.id));
     expect(await admit(f, true)).toBeNull();
-    expect(await recordNativeLocalProcessStop(db, source)).toBe(true);
+    expect(await db.transaction(tx => recordNativeLocalProcessStop(tx as unknown as typeof db, source))).toBe(true);
     await appendHeartbeatRunEvent(db, { companyId: f.companyId, runId: source.id, agentId: f.agentId,
       eventType: PROCESS_START_REQUESTED });
     // No PID was stored for the new launch, as when the server dies after spawn.
@@ -2440,7 +2440,7 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes
         await db.update(issues).set({ conversationAgentId: f.agentId, conversationUserId: "board", conversationState: "active", conversationBoundaryCommentId: boundaryId }).where(eq(issues.id, f.issueId));
         await db.update(issueRecoveryActions).set({ createdAt: new Date("2026-09-11T09:00:00Z") }).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
         const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.sourceRunId));
-        expect(await recordNativeLocalProcessStop(db, source)).toBe(true);
+        expect(await db.transaction(tx => recordNativeLocalProcessStop(tx as unknown as typeof db, source))).toBe(true);
         await db.update(heartbeatRuns).set({ processPid: null }).where(eq(heartbeatRuns.id, f.sourceRunId));
         if (scenario === "after_reset_workspace_hold") {
           await db.update(heartbeatRuns).set({ processPid: 999999999, resultJson: { workspaceRestoreFailure: "restore_unsafe_archive" } }).where(eq(heartbeatRuns.id, f.sourceRunId));

server/src/services/heartbeat/deferred-receipt-work.ts

@@ -5,6 +5,7 @@ import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator
 import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
 import type { DeferredExecutionScope, DeferredReceiptRecoveryOptions } from "../deferred-execution-notifications.js";
 import { registerScopedDeadlineWork } from "../scoped-deadline-work.js";
+import { issueBlockersResolvedWakePayloadSql } from "../issue-dependency-wakeups.js";
 
 type Entry = { id: string; companyId: string; dueAt: number; immediate: boolean };
 type NativeIntentOptions = DeferredExecutionScope & {
@@ -74,6 +75,7 @@ export function registerDeferredReceiptWork(
               sql`jsonb_typeof(${agentWakeupRequests.payload}#>'{_paperclipWakeContext,wakeCommentIds}') = 'array'
                 and ${agentWakeupRequests.payload}#>'{_paperclipWakeContext,wakeCommentIds}' <> '[]'::jsonb`,
               sql`${agentWakeupRequests.payload}->>'mutation' = 'interaction'`,
+              issueBlockersResolvedWakePayloadSql,
               sql`exists (select 1 from heartbeat_runs r where r.company_id = ${agentWakeupRequests.companyId}
                 and r.runtime_mode = 'legacy' and r.status = 'cancelled'
                 and r.result_json->>'queuedCommentInterruptQueueId' = ${agentWakeupRequests.id}::text)`,

server/src/services/heartbeat/queue.ts

@@ -144,7 +144,11 @@ import {
   isThrottleCandidateIssueRewake,
 } from "../issue-rewake-throttle.js";
 import { logActivity } from "../activity-log.js";
-import { ISSUE_BLOCKERS_RESOLVED_WAKE_REASON } from "../issue-dependency-wakeups.js";
+import {
+  ISSUE_BLOCKERS_RESOLVED_WAKE_REASON,
+  isIssueBlockersResolvedWakePayload,
+  issueBlockersResolvedWakePayloadSql,
+} from "../issue-dependency-wakeups.js";
 import { resumeIfPauseAlreadyReleased } from "../issue-tree-resume.js";
 import {
   ISSUE_PAUSED_ERROR_CODE,
@@ -1757,7 +1761,8 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
         or(and(
           sql`jsonb_typeof(${agentWakeupRequests.payload} #> '{_paperclipWakeContext,wakeCommentIds}') = 'array'`,
           sql`${agentWakeupRequests.payload} #> '{_paperclipWakeContext,wakeCommentIds}' <> '[]'::jsonb`,
-        ), sql`${agentWakeupRequests.payload}->>'mutation' = 'interaction'`),
+        ), sql`${agentWakeupRequests.payload}->>'mutation' = 'interaction'`,
+        issueBlockersResolvedWakePayloadSql),
         sql`${agentWakeupRequests.payload}->'queuedCommentInterrupt' is null`,
         sql`NOT EXISTS (SELECT 1 FROM heartbeat_runs live_issue_run
           WHERE live_issue_run.company_id = ${issues.companyId}
@@ -1768,7 +1773,8 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
       .orderBy(asc(agentWakeupRequests.updatedAt)).limit(50);
     const releasedNativeSources = new Set<string>();
     for (const { wake } of strandedQueues) {
-      if (!queuedCommentIdsFromWakePayload(wake.payload).length &&
+      const dependencyWake = isIssueBlockersResolvedWakePayload(wake.payload);
+      if (!dependencyWake && !queuedCommentIdsFromWakePayload(wake.payload).length &&
           !await readQueuedInteractionResponse(db, wake.companyId, String(wake.payload?.issueId), wake.payload)) continue;
       const [latest] = await db.select({
         id: heartbeatRuns.id,

server/src/services/issue-dependency-wakeups.ts

@@ -1,10 +1,19 @@
-import { and, eq, inArray } from "drizzle-orm";
+import { and, eq, inArray, sql } from "drizzle-orm";
 import { createHash } from "node:crypto";
 import type { Db } from "@paperclipai/db";
 import { agentWakeupRequests } from "@paperclipai/db";
 
 export const ISSUE_BLOCKERS_RESOLVED_WAKE_REASON = "issue_blockers_resolved";
 
+export const issueBlockersResolvedWakePayloadSql =
+  sql`${agentWakeupRequests.payload}#>>'{_paperclipWakeContext,wakeReason}' = ${ISSUE_BLOCKERS_RESOLVED_WAKE_REASON}`;
+
+export function isIssueBlockersResolvedWakePayload(payload: Record<string, unknown> | null | undefined): boolean {
+  const context = payload?._paperclipWakeContext;
+  return context !== null && typeof context === "object" && !Array.isArray(context)
+    && (context as Record<string, unknown>).wakeReason === ISSUE_BLOCKERS_RESOLVED_WAKE_REASON;
+}
+
 /**
  * Whether moving an issue into `blocked` should immediately wake its assignee
  * because every recorded blocker is already resolved ("blockers restored").

server/src/services/managed-config.ts

@@ -9,12 +9,23 @@
  *     "mode": "cloud",
  *     "catalogVersion": "2026.720.0",
  *     "features": { "<feature-key>": true | false, ... },
- *     "plugins":  { "autoInstall": ["daytona", "kubernetes"] },
+ *     "plugins":  { "autoInstall": ["daytona", "remote-host"] },
  *     "environments": [
- *       { "name": "Daytona", "provider": "daytona", "config": { "target": "us" } }
+ *       { "name": "Daytona", "provider": "daytona", "config": { "target": "us" } },
+ *       {
+ *         "name": "Remote Host",
+ *         "driver": "plugin",
+ *         "pluginKey": "example.remote-host-plugin",
+ *         "driverKey": "tasks",
+ *         "driverConfig": {}
+ *       }
  *     ]
  *   }
  *
+ * An `environments` entry without `driver` is the single sandbox entry; any
+ * number of `driver: "plugin"` entries may sit beside it (see
+ * `MANAGED_ENVIRONMENT_DRIVERS`).
+ *
  * Parsing follows the `execution-policy-bootstrap.ts` doctrine: a pure
  * function over `Record<string, string | undefined>`, strict, and fail
  * closed — bad JSON, an unsupported `v`, an unknown feature key, a feature
@@ -54,8 +65,47 @@ export interface ManagedInstanceConfig {
     readonly workerEnv?: Readonly<Record<string, readonly string[]>>;
     readonly enabledCompanies?: Readonly<Record<string, readonly string[]>>;
   };
-  /** Sandbox environments the control plane provisions at boot (empty when the section is absent). */
+  /**
+   * The sandbox entry of the `environments` section: zero or one, since it
+   * provisions the single Paperclip-managed sandbox slot (empty when the
+   * section is absent or declares only plugin entries).
+   */
   environments: readonly ManagedEnvironmentSpec[];
+  /** The `driver: "plugin"` entries of the `environments` section (empty when none are declared). */
+  pluginEnvironments: readonly ManagedPluginEnvironmentSpec[];
+}
+
+/**
+ * The environment drivers a managed-config `environments` entry may declare.
+ * An entry without `driver` is the sandbox entry, so documents written
+ * before plugin entries existed parse unchanged.
+ */
+export const MANAGED_ENVIRONMENT_DRIVERS = ["sandbox", "plugin"] as const;
+export type ManagedEnvironmentDriver = (typeof MANAGED_ENVIRONMENT_DRIVERS)[number];
+
+/**
+ * One declared managed plugin environment: an instance-level `plugin`-driver
+ * environment row served by an auto-installed plugin's environment driver.
+ * The row is
+ * identified by its `pluginKey`–`driverKey` pair, which boot stamps on the
+ * row as `metadata.managedPluginEnvironment`.
+ */
+export interface ManagedPluginEnvironmentSpec {
+  driver: "plugin";
+  /** Display name of the instance-level environment row (unique per instance). */
+  name: string;
+  description?: string;
+  /**
+   * The serving plugin's manifest id (for example
+   * `example.remote-host-plugin`), not its `plugins.autoInstall` catalog key.
+   * Boot requires a catalog entry
+   * whose `key` is in `plugins.autoInstall` to install this id.
+   */
+  pluginKey: string;
+  /** The plugin's environment driver key. */
+  driverKey: string;
+  /** Driver config, validated by the plugin at ensure time; never carries secrets. */
+  driverConfig: Readonly<Record<string, unknown>>;
 }
 
 /**
@@ -142,6 +192,155 @@ function describeJsonValue(value: unknown): string {
   return `${JSON.stringify(value)}`;
 }
 
+const SANDBOX_ENTRY_KEYS = ["name", "description", "driver", "provider", "config"] as const;
+const PLUGIN_ENTRY_KEYS = [
+  "name",
+  "description",
+  "driver",
+  "pluginKey",
+  "driverKey",
+  "driverConfig",
+] as const;
+/** Same rule as the plugin environment config schema in `environment-config.ts`. */
+const PLUGIN_DRIVER_KEY_PATTERN = /^[a-z0-9][a-z0-9._-]*$/;
+
+function requireTrimmedString(value: unknown, path: string): string {
+  if (typeof value !== "string" || value.length === 0 || value.trim() !== value) {
+    fail(`"${path}" must be a non-empty string without surrounding whitespace (got ${describeJsonValue(value)})`);
+  }
+  return value;
+}
+
+function readOptionalDescription(value: unknown, path: string): string | undefined {
+  if (value === undefined) return undefined;
+  if (typeof value !== "string" || value.trim().length === 0) {
+    fail(`"${path}" must be a non-empty string when present (got ${describeJsonValue(value)})`);
+  }
+  return value;
+}
+
+function readSecretFreeObject(value: unknown, path: string): Record<string, unknown> {
+  if (value === undefined) return {};
+  if (!isPlainObject(value)) {
+    fail(`"${path}" must be an object (got ${describeJsonValue(value)})`);
+  }
+  const secretLikeKey = findSecretLikeConfigKey(value, "");
+  if (secretLikeKey) {
+    fail(
+      `"${path}" key "${secretLikeKey}" looks secret-bearing; credentials are delivered to managed instances as process environment variables (the provider's documented env fallback), never in the managed-config document`,
+    );
+  }
+  return { ...value };
+}
+
+/**
+ * Parse the `environments` section: at most one sandbox entry (no `driver`,
+ * or `driver: "sandbox"`) plus any number of `driver: "plugin"` entries.
+ * Names are unique across the section, and plugin entries are unique by
+ * their `pluginKey`–`driverKey` pair.
+ */
+function parseManagedEnvironmentsSection(
+  raw: unknown,
+  autoInstall: readonly string[],
+): { sandbox: ManagedEnvironmentSpec[]; plugin: ManagedPluginEnvironmentSpec[] } {
+  const sandbox: ManagedEnvironmentSpec[] = [];
+  const plugin: ManagedPluginEnvironmentSpec[] = [];
+  if (raw === undefined) return { sandbox, plugin };
+  if (!Array.isArray(raw)) {
+    fail(`"environments" must be an array of environment objects (got ${describeJsonValue(raw)})`);
+  }
+  const names = new Set<string>();
+  const pluginDrivers = new Set<string>();
+  for (const [index, entry] of raw.entries()) {
+    const at = `environments[${index}]`;
+    if (!isPlainObject(entry)) {
+      fail(`"${at}" must be an object (got ${describeJsonValue(entry)})`);
+    }
+    // Only an absent `driver` means the sandbox entry; an explicit `null` is malformed.
+    const driver = entry.driver === undefined ? "sandbox" : entry.driver;
+    if (!(MANAGED_ENVIRONMENT_DRIVERS as readonly unknown[]).includes(driver)) {
+      fail(
+        `"${at}.driver" must be one of ${MANAGED_ENVIRONMENT_DRIVERS.map((value) => JSON.stringify(value)).join(", ")} when present (got ${describeJsonValue(entry.driver)})`,
+      );
+    }
+    const allowedKeys: readonly string[] = driver === "plugin" ? PLUGIN_ENTRY_KEYS : SANDBOX_ENTRY_KEYS;
+    for (const key of Object.keys(entry)) {
+      if (!allowedKeys.includes(key)) {
+        fail(`"${at}" has unknown key "${key}" (allowed for a ${driver} entry: ${allowedKeys.join(", ")})`);
+      }
+    }
+    const name = requireTrimmedString(entry.name, `${at}.name`);
+    if (names.has(name)) {
+      fail(`"${at}.name" duplicates another entry's name "${name}"; environment names are unique per instance`);
+    }
+    names.add(name);
+    const description = readOptionalDescription(entry.description, `${at}.description`);
+
+    if (driver === "sandbox") {
+      // The DB enforces at most ONE Paperclip-managed sandbox row per instance
+      // (partial unique index `environments_managed_sandbox_idx`); every
+      // sandbox entry provisions that row, so a second one can never be
+      // satisfied.
+      if (sandbox.length > 0) {
+        fail(
+          `"environments" supports at most one sandbox entry: each sandbox entry provisions the single Paperclip-managed sandbox environment (DB invariant environments_managed_sandbox_idx)`,
+        );
+      }
+      const provider = requireTrimmedString(entry.provider, `${at}.provider`);
+      // Coherence: on a managed instance the control plane is the only plugin
+      // install path, so an environment whose provider plugin is not
+      // auto-installed could never serve a lease. Catch the skew at parse time.
+      if (!autoInstall.includes(provider)) {
+        fail(
+          `"${at}.provider" is "${provider}", which is not in "plugins.autoInstall"; a managed environment requires its provider plugin to be provisioned`,
+        );
+      }
+      if (isPlainObject(entry.config) && entry.config.provider !== undefined) {
+        fail(`"${at}.config" must not set "provider"; it is forced from the entry's provider key`);
+      }
+      const config = readSecretFreeObject(entry.config, `${at}.config`);
+      sandbox.push(
+        Object.freeze({
+          name,
+          ...(description !== undefined ? { description } : {}),
+          provider,
+          config: Object.freeze(config),
+        }) as ManagedEnvironmentSpec,
+      );
+      continue;
+    }
+
+    // `pluginKey` is the plugin's manifest id (what the row's config stores),
+    // not a `plugins.autoInstall` catalog key, so this pure parser cannot map
+    // it. `applyManagedEnvironments` refuses startup when no auto-installed
+    // catalog entry installs this manifest id.
+    const pluginKey = requireTrimmedString(entry.pluginKey, `${at}.pluginKey`);
+    const driverKey = requireTrimmedString(entry.driverKey, `${at}.driverKey`);
+    if (!PLUGIN_DRIVER_KEY_PATTERN.test(driverKey)) {
+      fail(
+        `"${at}.driverKey" must start with a lowercase letter or digit and contain only lowercase letters, digits, dots, hyphens, or underscores (got ${describeJsonValue(driverKey)})`,
+      );
+    }
+    const pair = `${pluginKey}:${driverKey}`;
+    if (pluginDrivers.has(pair)) {
+      fail(`"${at}" duplicates another plugin entry's pluginKey and driverKey ("${pair}"); each plugin environment driver is declared once`);
+    }
+    pluginDrivers.add(pair);
+    const driverConfig = readSecretFreeObject(entry.driverConfig, `${at}.driverConfig`);
+    plugin.push(
+      Object.freeze({
+        driver: "plugin",
+        name,
+        ...(description !== undefined ? { description } : {}),
+        pluginKey,
+        driverKey,
+        driverConfig: Object.freeze(driverConfig),
+      }) as ManagedPluginEnvironmentSpec,
+    );
+  }
+  return { sandbox, plugin };
+}
+
 /**
  * Parse `PAPERCLIP_MANAGED_CONFIG` from a raw env map. Returns null only when
  * the variable is absent (self-hosted). Throws with a precise error on a
@@ -320,79 +519,10 @@ export function parseManagedConfigEnv(env: ManagedConfigEnv): ManagedInstanceCon
   // absence is not a dropped security control — it simply declares no managed
   // environments. When present, the section is validated fail-closed like
   // everything else.
-  const environmentSpecs: ManagedEnvironmentSpec[] = [];
-  if (doc.environments !== undefined) {
-    if (!Array.isArray(doc.environments)) {
-      fail(`"environments" must be an array of environment objects (got ${describeJsonValue(doc.environments)})`);
-    }
-    // The DB enforces at most ONE Paperclip-managed sandbox row per instance
-    // (partial unique index `environments_managed_sandbox_idx`); every entry
-    // here provisions that row, so a longer list can never be satisfied.
-    if (doc.environments.length > 1) {
-      fail(
-        `"environments" supports at most one entry: each entry provisions the single Paperclip-managed sandbox environment (DB invariant environments_managed_sandbox_idx)`,
-      );
-    }
-    for (const [index, entry] of doc.environments.entries()) {
-      if (!isPlainObject(entry)) {
-        fail(`"environments[${index}]" must be an object (got ${describeJsonValue(entry)})`);
-      }
-      const allowedEntryKeys = new Set(["name", "description", "provider", "config"]);
-      for (const key of Object.keys(entry)) {
-        if (!allowedEntryKeys.has(key)) {
-          fail(`"environments[${index}]" has unknown key "${key}" (allowed: name, description, provider, config)`);
-        }
-      }
-      if (typeof entry.name !== "string" || entry.name.length === 0 || entry.name.trim() !== entry.name) {
-        fail(
-          `"environments[${index}].name" must be a non-empty string without surrounding whitespace (got ${describeJsonValue(entry.name)})`,
-        );
-      }
-      if (
-        entry.description !== undefined
-        && (typeof entry.description !== "string" || entry.description.trim().length === 0)
-      ) {
-        fail(`"environments[${index}].description" must be a non-empty string when present (got ${describeJsonValue(entry.description)})`);
-      }
-      if (typeof entry.provider !== "string" || entry.provider.length === 0 || entry.provider.trim() !== entry.provider) {
-        fail(
-          `"environments[${index}].provider" must be a non-empty string without surrounding whitespace (got ${describeJsonValue(entry.provider)})`,
-        );
-      }
-      // Coherence: on a managed instance the control plane is the only plugin
-      // install path, so an environment whose provider plugin is not
-      // auto-installed could never serve a lease. Catch the skew at parse time.
-      if (!autoInstall.includes(entry.provider)) {
-        fail(
-          `"environments[${index}].provider" is "${entry.provider}", which is not in "plugins.autoInstall"; a managed environment requires its provider plugin to be provisioned`,
-        );
-      }
-      const config: Record<string, unknown> = {};
-      if (entry.config !== undefined) {
-        if (!isPlainObject(entry.config)) {
-          fail(`"environments[${index}].config" must be an object (got ${describeJsonValue(entry.config)})`);
-        }
-        if (entry.config.provider !== undefined) {
-          fail(`"environments[${index}].config" must not set "provider"; it is forced from the entry's provider key`);
-        }
-        const secretLikeKey = findSecretLikeConfigKey(entry.config, "");
-        if (secretLikeKey) {
-          fail(
-            `"environments[${index}].config" key "${secretLikeKey}" looks secret-bearing; credentials are delivered to managed instances as process environment variables (the provider's documented env fallback), never in the managed-config document`,
-          );
-        }
-        Object.assign(config, entry.config);
-      }
-      environmentSpecs.push(
-        Object.freeze({
-          name: entry.name,
-          ...(entry.description !== undefined ? { description: entry.description } : {}),
-          provider: entry.provider,
-          config: Object.freeze(config),
-        }) as ManagedEnvironmentSpec,
-      );
-    }
-  }
+  const { sandbox: environmentSpecs, plugin: pluginEnvironmentSpecs } = parseManagedEnvironmentsSection(
+    doc.environments,
+    autoInstall,
+  );
 
   return Object.freeze({
     v: SUPPORTED_MANAGED_CONFIG_VERSION,
@@ -405,6 +535,7 @@ export function parseManagedConfigEnv(env: ManagedConfigEnv): ManagedInstanceCon
       ...(enabledCompanies ? { enabledCompanies: Object.freeze(enabledCompanies) } : {}),
     }),
     environments: Object.freeze(environmentSpecs),
+    pluginEnvironments: Object.freeze(pluginEnvironmentSpecs),
   }) as ManagedInstanceConfig;
 }
 

server/src/services/managed-environments-plugin-entries.test.ts

@@ -0,0 +1,385 @@
+import { describe, expect, it, vi } from "vitest";
+import { unprocessable } from "../errors.js";
+import { applyManagedEnvironments } from "./managed-environments.js";
+import {
+  PLUGIN_KEY,
+  environmentsSeam,
+  instanceSettingsSeam,
+  noDb,
+  parsedConfig,
+  pluginEnvironmentRow,
+  readyDriverResolver,
+  reconciliationResult,
+  recoveryHandle,
+  tick,
+  type InstanceSettingsSeam,
+  type RecoveryHandle,
+} from "./managed-environments.test-support.js";
+
+describe("applyManagedEnvironments plugin entries", () => {
+  const pluginEntry = (overrides: Record<string, unknown> = {}) => ({
+    name: "Remote Host",
+    description: "Runs the agent on a remote host",
+    driver: "plugin",
+    pluginKey: PLUGIN_KEY,
+    driverKey: "tasks",
+    driverConfig: {},
+    ...overrides,
+  });
+  const configWith = (environments: unknown[], features: Record<string, unknown> = {}) => parsedConfig({
+    plugins: { autoInstall: ["daytona", "remote-host"] },
+    features,
+    environments,
+  });
+  const readyPlugin = (status = "ready", declaresDriver = true) => vi.fn(async () => ({
+    plugin: { id: "remote-host-plugin", pluginKey: PLUGIN_KEY, status },
+    declaresDriver,
+  }));
+  const workerManager = (running = true, handle?: RecoveryHandle) => ({
+    isRunning: vi.fn(() => running),
+    call: vi.fn(),
+    getWorker: vi.fn(() => handle),
+  });
+
+  it("ensures a plugin entry with the plugin validator's worker manager", async () => {
+    const environments = environmentsSeam();
+    const result = await applyManagedEnvironments(noDb, configWith([pluginEntry()]), {
+      env: {},
+      workerManager: workerManager(),
+      environments,
+      instanceSettings: instanceSettingsSeam(),
+      resolveEnvironmentPlugin: readyPlugin(),
+      autoInstalledPluginKeys: [PLUGIN_KEY],
+    });
+
+    expect(environments.ensureManagedPluginEnvironment).toHaveBeenCalledWith({
+      name: "Remote Host",
+      description: "Runs the agent on a remote host",
+      pluginKey: PLUGIN_KEY,
+      driverKey: "tasks",
+      driverConfig: {},
+      workerManager: { isRunning: expect.any(Function), call: expect.any(Function) },
+    });
+    expect(result).toMatchObject({ ensured: 1, added: 1, failed: 0 });
+    expect(result?.pluginOutcomes).toEqual([{
+      environmentId: "plugin-env",
+      name: "Remote Host",
+      pluginKey: PLUGIN_KEY,
+      driverKey: "tasks",
+      action: "added",
+    }]);
+    expect(environments.ensureManagedSandboxEnvironment).not.toHaveBeenCalled();
+  });
+
+  it("runs the sandbox entry first", async () => {
+    const order: string[] = [];
+    const environments = environmentsSeam({
+      ensureManagedSandboxEnvironment: vi.fn(async () => {
+        order.push("sandbox");
+        return reconciliationResult() as never;
+      }),
+      ensureManagedPluginEnvironment: vi.fn(async () => {
+        order.push("plugin");
+        return { environment: pluginEnvironmentRow(), action: "added" as const };
+      }),
+    });
+    await applyManagedEnvironments(noDb, configWith([{ name: "Daytona", provider: "daytona" }, pluginEntry()]), {
+      env: {},
+      workerManager: workerManager(),
+      environments,
+      instanceSettings: instanceSettingsSeam(),
+      resolveSandboxProviderDriver: readyDriverResolver(),
+      resolveEnvironmentPlugin: readyPlugin(),
+      autoInstalledPluginKeys: [PLUGIN_KEY],
+    });
+    expect(order).toEqual(["sandbox", "plugin"]);
+  });
+
+  it("refuses startup when no auto-installed catalog entry installs the entry's plugin", async () => {
+    const environments = environmentsSeam();
+    await expect(applyManagedEnvironments(noDb, configWith([pluginEntry()]), {
+      env: {},
+      workerManager: workerManager(),
+      environments,
+      instanceSettings: instanceSettingsSeam(),
+      resolveEnvironmentPlugin: readyPlugin(),
+      autoInstalledPluginKeys: ["paperclip.daytona-sandbox-provider"],
+    })).rejects.toThrow(/uses plugin "example.remote-host-plugin", which no "plugins.autoInstall" catalog entry installs/);
+    expect(environments.ensureManagedPluginEnvironment).not.toHaveBeenCalled();
+  });
+
+  it("resolves the auto-installed manifest ids from the catalog when the caller passes none", async () => {
+    // "remote-host" is not in this build's bundled catalog and the test root
+    // has no distribution catalog, so the catalog lookup refuses.
+    await expect(applyManagedEnvironments(noDb, configWith([pluginEntry()]), {
+      env: { PAPERCLIP_BUNDLED_PLUGIN_ROOT: "/nonexistent-paperclip-catalog" },
+      workerManager: workerManager(),
+      environments: environmentsSeam(),
+      instanceSettings: instanceSettingsSeam(),
+    })).rejects.toThrow(/auto-install key "remote-host" is not in the bundled catalog/);
+  });
+
+  it("does not claim the sandbox slot, so a forced execution mode can coexist with plugin-only entries", async () => {
+    await expect(applyManagedEnvironments(noDb, configWith([pluginEntry()]), {
+      env: { PAPERCLIP_EXECUTION_MODE: "kubernetes" },
+      workerManager: workerManager(),
+      environments: environmentsSeam(),
+      instanceSettings: instanceSettingsSeam(),
+      resolveEnvironmentPlugin: readyPlugin(),
+      autoInstalledPluginKeys: [PLUGIN_KEY],
+    })).resolves.toMatchObject({ ensured: 1 });
+  });
+
+  it("archives the row of a plugin that is down and reactivates it on worker recovery", async () => {
+    const { handle, fireReady } = recoveryHandle();
+    const environments = environmentsSeam({
+      archiveManagedPluginEnvironment: vi.fn().mockResolvedValue(pluginEnvironmentRow({ status: "archived" })),
+    });
+    const manager = workerManager(false, handle);
+    const result = await applyManagedEnvironments(noDb, configWith([pluginEntry()]), {
+      env: {},
+      workerManager: manager,
+      environments,
+      instanceSettings: instanceSettingsSeam(),
+      resolveEnvironmentPlugin: readyPlugin(),
+      autoInstalledPluginKeys: [PLUGIN_KEY],
+    });
+
+    expect(result).toMatchObject({ ensured: 0, failed: 1 });
+    expect(environments.archiveManagedPluginEnvironment).toHaveBeenCalledWith({ pluginKey: PLUGIN_KEY, driverKey: "tasks" });
+    expect(environments.ensureManagedPluginEnvironment).not.toHaveBeenCalled();
+
+    manager.isRunning.mockReturnValue(true);
+    fireReady();
+    fireReady();
+    await tick();
+
+    expect(environments.ensureManagedPluginEnvironment).toHaveBeenCalledTimes(1);
+    expect(handle.off).toHaveBeenCalledTimes(1);
+  });
+
+  it("keeps recovery armed after a worker failure and stops after one success", async () => {
+    const { handle, fireReady } = recoveryHandle();
+    const ensureManagedPluginEnvironment = vi.fn()
+      .mockRejectedValueOnce(new Error("worker crashed during environmentValidateConfig"))
+      .mockResolvedValue({ environment: pluginEnvironmentRow(), action: "updated" });
+    const environments = environmentsSeam({ ensureManagedPluginEnvironment });
+    const manager = workerManager(false, handle);
+    await applyManagedEnvironments(noDb, configWith([pluginEntry()]), {
+      env: {},
+      workerManager: manager,
+      environments,
+      instanceSettings: instanceSettingsSeam(),
+      resolveEnvironmentPlugin: readyPlugin(),
+      autoInstalledPluginKeys: [PLUGIN_KEY],
+    });
+    manager.isRunning.mockReturnValue(true);
+
+    fireReady();
+    await tick();
+    expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(1);
+    expect(handle.off).not.toHaveBeenCalled();
+
+    fireReady();
+    await tick();
+    expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(2);
+    expect(handle.off).toHaveBeenCalledTimes(1);
+
+    fireReady();
+    await tick();
+    expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(2);
+  });
+
+  it("queues one more attempt when the worker becomes ready during an attempt", async () => {
+    const { handle, fireReady } = recoveryHandle();
+    let failFirst!: (err: Error) => void;
+    const ensureManagedPluginEnvironment = vi.fn()
+      .mockImplementationOnce(() => new Promise((_resolve, reject) => { failFirst = reject; }))
+      .mockResolvedValue({ environment: pluginEnvironmentRow(), action: "updated" });
+    const manager = workerManager(false, handle);
+    await applyManagedEnvironments(noDb, configWith([pluginEntry()]), {
+      env: {},
+      workerManager: manager,
+      environments: environmentsSeam({ ensureManagedPluginEnvironment }),
+      instanceSettings: instanceSettingsSeam(),
+      resolveEnvironmentPlugin: readyPlugin(),
+      autoInstalledPluginKeys: [PLUGIN_KEY],
+    });
+    manager.isRunning.mockReturnValue(true);
+
+    fireReady();
+    fireReady();
+    expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(1);
+    failFirst(new Error("worker crashed"));
+    await tick();
+    await tick();
+    expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(2);
+    expect(handle.off).toHaveBeenCalledTimes(1);
+  });
+
+  it("stops recovery when the plugin permanently rejects the config", async () => {
+    const { handle, fireReady } = recoveryHandle();
+    const ensureManagedPluginEnvironment = vi.fn().mockRejectedValue(unprocessable("driverConfig.region is not supported"));
+    const manager = workerManager(false, handle);
+    await applyManagedEnvironments(noDb, configWith([pluginEntry()]), {
+      env: {},
+      workerManager: manager,
+      environments: environmentsSeam({ ensureManagedPluginEnvironment }),
+      instanceSettings: instanceSettingsSeam(),
+      resolveEnvironmentPlugin: readyPlugin(),
+      autoInstalledPluginKeys: [PLUGIN_KEY],
+    });
+    manager.isRunning.mockReturnValue(true);
+
+    fireReady();
+    await tick();
+    fireReady();
+    await tick();
+    expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(1);
+    expect(handle.off).toHaveBeenCalledTimes(1);
+  });
+
+  it("archives and arms recovery when the worker dies during the boot ensure", async () => {
+    const { handle, fireReady } = recoveryHandle();
+    const manager = workerManager(true, handle);
+    const ensureManagedPluginEnvironment = vi.fn()
+      .mockImplementationOnce(async () => {
+        manager.isRunning.mockReturnValue(false);
+        throw new Error("worker crashed during environmentValidateConfig");
+      })
+      .mockResolvedValue({ environment: pluginEnvironmentRow(), action: "updated" });
+    const environments = environmentsSeam({ ensureManagedPluginEnvironment });
+    const result = await applyManagedEnvironments(noDb, configWith([pluginEntry()]), {
+      env: {},
+      workerManager: manager,
+      environments,
+      instanceSettings: instanceSettingsSeam(),
+      resolveEnvironmentPlugin: readyPlugin(),
+      autoInstalledPluginKeys: [PLUGIN_KEY],
+    });
+    await tick();
+
+    expect(result).toMatchObject({ ensured: 0, failed: 1 });
+    expect(environments.archiveManagedPluginEnvironment).toHaveBeenCalledWith({ pluginKey: PLUGIN_KEY, driverKey: "tasks" });
+    expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(1);
+
+    manager.isRunning.mockReturnValue(true);
+    fireReady();
+    await tick();
+    expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(2);
+    expect(handle.off).toHaveBeenCalledTimes(1);
+  });
+
+  it("keeps the row and retries when the ensure fails while the worker keeps running", async () => {
+    const { handle, fireReady } = recoveryHandle();
+    const ensureManagedPluginEnvironment = vi.fn()
+      .mockRejectedValueOnce(new Error("environmentValidateConfig timed out"))
+      .mockRejectedValueOnce(new Error("environmentValidateConfig timed out"))
+      .mockResolvedValue({ environment: pluginEnvironmentRow(), action: "unchanged" });
+    const environments = environmentsSeam({ ensureManagedPluginEnvironment });
+    const result = await applyManagedEnvironments(noDb, configWith([pluginEntry()]), {
+      env: {},
+      workerManager: workerManager(true, handle),
+      environments,
+      instanceSettings: instanceSettingsSeam(),
+      resolveEnvironmentPlugin: readyPlugin(),
+      autoInstalledPluginKeys: [PLUGIN_KEY],
+    });
+    await tick();
+
+    expect(result).toMatchObject({ ensured: 0, failed: 1 });
+    expect(environments.archiveManagedPluginEnvironment).not.toHaveBeenCalled();
+    // The immediate retry also failed; the next `ready` succeeds.
+    expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(2);
+    fireReady();
+    await tick();
+    expect(ensureManagedPluginEnvironment).toHaveBeenCalledTimes(3);
+    expect(handle.off).toHaveBeenCalledTimes(1);
+  });
+
+  it("neither archives nor retries when the boot ensure gets a permanent config rejection", async () => {
+    const { handle } = recoveryHandle();
+    const environments = environmentsSeam({
+      ensureManagedPluginEnvironment: vi.fn().mockRejectedValue(unprocessable("driverConfig.region is not supported")),
+    });
+    const manager = workerManager(true, handle);
+    const result = await applyManagedEnvironments(noDb, configWith([pluginEntry()]), {
+      env: {},
+      workerManager: manager,
+      environments,
+      instanceSettings: instanceSettingsSeam(),
+      resolveEnvironmentPlugin: readyPlugin(),
+      autoInstalledPluginKeys: [PLUGIN_KEY],
+    });
+    expect(result).toMatchObject({ ensured: 0, failed: 1 });
+    expect(environments.archiveManagedPluginEnvironment).not.toHaveBeenCalled();
+    expect(manager.getWorker).not.toHaveBeenCalled();
+  });
+
+  it("treats a missing plugin, a plugin without the driver, or a worker manager without call as unavailable", async () => {
+    for (const [resolveEnvironmentPlugin, manager] of [
+      [vi.fn(async () => null), workerManager()],
+      [readyPlugin("ready", false), workerManager()],
+      [readyPlugin("error"), workerManager()],
+      [readyPlugin(), { isRunning: vi.fn(() => true), getWorker: vi.fn(() => undefined) }],
+    ] as const) {
+      const environments = environmentsSeam();
+      const result = await applyManagedEnvironments(noDb, configWith([pluginEntry()]), {
+        env: {},
+        workerManager: manager,
+        environments,
+        instanceSettings: instanceSettingsSeam(),
+        resolveEnvironmentPlugin,
+        autoInstalledPluginKeys: [PLUGIN_KEY],
+      });
+      expect(result).toMatchObject({ ensured: 0, failed: 1 });
+      expect(environments.archiveManagedPluginEnvironment).toHaveBeenCalledTimes(1);
+      expect(environments.ensureManagedPluginEnvironment).not.toHaveBeenCalled();
+      expect(manager.getWorker).not.toHaveBeenCalled();
+    }
+  });
+
+  it("never archives a row whose entry was removed from the document", async () => {
+    const environments = environmentsSeam();
+    expect(await applyManagedEnvironments(noDb, configWith([]), {
+      env: {},
+      workerManager: workerManager(),
+      environments,
+      instanceSettings: instanceSettingsSeam(),
+    })).toBeNull();
+    expect(environments.archiveManagedPluginEnvironment).not.toHaveBeenCalled();
+  });
+
+  it("never changes the instance default, and the sandbox stamp never replaces a plugin default", async () => {
+    const settings = instanceSettingsSeam({
+      get: vi.fn().mockResolvedValue({ defaultEnvironmentId: "plugin-env" }) as InstanceSettingsSeam["get"],
+    });
+    await applyManagedEnvironments(noDb, configWith(
+      [{ name: "Daytona", provider: "daytona" }, pluginEntry()],
+      { enableManagedSandboxOnly: true },
+    ), {
+      env: {},
+      workerManager: workerManager(),
+      instanceSettings: settings,
+      environments: environmentsSeam({ getById: vi.fn().mockResolvedValue(pluginEnvironmentRow()) }),
+      resolveSandboxProviderDriver: readyDriverResolver(),
+      resolveEnvironmentPlugin: readyPlugin(),
+      autoInstalledPluginKeys: [PLUGIN_KEY],
+    });
+    expect(settings.update).not.toHaveBeenCalled();
+  });
+
+  it("is fail-safe: an ensure failure is logged and counted, not thrown", async () => {
+    const result = await applyManagedEnvironments(noDb, configWith([pluginEntry()]), {
+      env: {},
+      workerManager: workerManager(),
+      environments: environmentsSeam({
+        ensureManagedPluginEnvironment: vi.fn().mockRejectedValue(new Error("validator refused")),
+      }),
+      instanceSettings: instanceSettingsSeam(),
+      resolveEnvironmentPlugin: readyPlugin(),
+      autoInstalledPluginKeys: [PLUGIN_KEY],
+    });
+    expect(result).toMatchObject({ ensured: 0, failed: 1 });
+  });
+});

server/src/services/managed-environments.test-support.ts

@@ -0,0 +1,139 @@
+/**
+ * Shared fixtures and service seams for the `applyManagedEnvironments` tests
+ * (`managed-environments.test.ts` for the sandbox entry and instance default,
+ * `managed-environments-plugin-entries.test.ts` for plugin entries).
+ */
+import { vi } from "vitest";
+import type { Db } from "@paperclipai/db";
+import { MANAGED_CONFIG_ENV_KEY, parseManagedConfigEnv } from "./managed-config.js";
+import type { ApplyManagedEnvironmentsOptions } from "./managed-environments.js";
+
+export const noDb = null as unknown as Db;
+
+export function parsedConfig(overrides: Record<string, unknown> = {}) {
+  const config = parseManagedConfigEnv({
+    [MANAGED_CONFIG_ENV_KEY]: JSON.stringify({
+      v: 1,
+      mode: "cloud",
+      catalogVersion: "2026.720.0",
+      features: {},
+      plugins: { autoInstall: ["daytona"] },
+      ...overrides,
+    }),
+  });
+  if (!config) throw new Error("expected a parsed managed config");
+  return config;
+}
+
+export function environmentRow(overrides: Record<string, unknown> = {}) {
+  const now = new Date();
+  return {
+    id: "env-1",
+    name: "Daytona",
+    description: null,
+    driver: "sandbox",
+    status: "active",
+    config: {},
+    envVars: {},
+    metadata: {},
+    createdAt: now,
+    updatedAt: now,
+    ...overrides,
+  };
+}
+
+export function reconciliationResult(
+  overrides: Record<string, unknown> = {},
+) {
+  return {
+    environment: environmentRow(),
+    action: "added" as const,
+    stockStatus: "missing" as const,
+    updateAvailable: false,
+    stockHash: "sha256:stock",
+    ...overrides,
+  };
+}
+
+export function readyDriverResolver(status = "ready") {
+  return vi.fn(async () => ({
+    plugin: { id: "plugin-1", pluginKey: "sandbox-providers/daytona", status },
+  }));
+}
+
+export function runningWorkerManager(running = true) {
+  return { isRunning: vi.fn(() => running), getWorker: vi.fn(() => undefined) };
+}
+
+export type WorkerManagerSeam = NonNullable<ApplyManagedEnvironmentsOptions["workerManager"]>;
+export type RecoveryHandle = NonNullable<ReturnType<WorkerManagerSeam["getWorker"]>>;
+
+/** A worker handle fake that records `ready` listeners so tests can fire them. */
+export function recoveryHandle() {
+  const readyListeners: Array<(payload: { pluginId: string }) => void> = [];
+  const handle: RecoveryHandle = {
+    on: vi.fn((_event, listener) => {
+      readyListeners.push(listener);
+    }),
+    off: vi.fn((_event, listener) => {
+      const at = readyListeners.indexOf(listener);
+      if (at !== -1) readyListeners.splice(at, 1);
+    }),
+  };
+  return {
+    handle,
+    fireReady: () => {
+      for (const listener of [...readyListeners]) listener({ pluginId: "plugin-1" });
+    },
+  };
+}
+
+/** Lets the fire-and-forget recovery ensure settle. */
+export function tick() {
+  return new Promise((resolve) => setImmediate(resolve));
+}
+
+export type EnvironmentsSeam = NonNullable<ApplyManagedEnvironmentsOptions["environments"]>;
+export type InstanceSettingsSeam = NonNullable<ApplyManagedEnvironmentsOptions["instanceSettings"]>;
+
+export function environmentsSeam(overrides: Partial<EnvironmentsSeam> = {}): EnvironmentsSeam {
+  return {
+    ensureManagedSandboxEnvironment:
+      overrides.ensureManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(reconciliationResult()),
+    archiveManagedSandboxEnvironment:
+      overrides.archiveManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(null),
+    getById: overrides.getById ?? vi.fn().mockResolvedValue(null),
+    update: overrides.update ?? vi.fn().mockResolvedValue(environmentRow()),
+    findManagedSandboxEnvironment:
+      overrides.findManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(null),
+    ensureManagedPluginEnvironment:
+      overrides.ensureManagedPluginEnvironment
+      ?? vi.fn().mockResolvedValue({ environment: pluginEnvironmentRow(), action: "added" }),
+    archiveManagedPluginEnvironment:
+      overrides.archiveManagedPluginEnvironment ?? vi.fn().mockResolvedValue(null),
+  };
+}
+
+export const PLUGIN_KEY = "example.remote-host-plugin";
+
+export function pluginEnvironmentRow(overrides: Record<string, unknown> = {}) {
+  return environmentRow({
+    id: "plugin-env",
+    name: "Remote Host",
+    driver: "plugin",
+    config: { pluginKey: PLUGIN_KEY, driverKey: "tasks", driverConfig: {} },
+    metadata: { managedByPaperclip: true, managedPluginEnvironment: `${PLUGIN_KEY}:tasks` },
+    ...overrides,
+  }) as never;
+}
+
+export function instanceSettingsSeam(overrides: Partial<InstanceSettingsSeam> = {}): InstanceSettingsSeam {
+  return {
+    get:
+      overrides.get ??
+      (vi.fn().mockResolvedValue({ defaultEnvironmentId: null }) as InstanceSettingsSeam["get"]),
+    update:
+      overrides.update ??
+      (vi.fn().mockResolvedValue({ defaultEnvironmentId: "env-1" }) as InstanceSettingsSeam["update"]),
+  };
+}

server/src/services/managed-environments.test.ts

@@ -1,122 +1,18 @@
 import { describe, expect, it, vi } from "vitest";
-import type { Db } from "@paperclipai/db";
-import { MANAGED_CONFIG_ENV_KEY, parseManagedConfigEnv } from "./managed-config.js";
+import { applyManagedEnvironments } from "./managed-environments.js";
 import {
-  applyManagedEnvironments,
-  type ApplyManagedEnvironmentsOptions,
-} from "./managed-environments.js";
-
-const noDb = null as unknown as Db;
-
-function parsedConfig(overrides: Record<string, unknown> = {}) {
-  const config = parseManagedConfigEnv({
-    [MANAGED_CONFIG_ENV_KEY]: JSON.stringify({
-      v: 1,
-      mode: "cloud",
-      catalogVersion: "2026.720.0",
-      features: {},
-      plugins: { autoInstall: ["daytona"] },
-      ...overrides,
-    }),
-  });
-  if (!config) throw new Error("expected a parsed managed config");
-  return config;
-}
-
-function environmentRow(overrides: Record<string, unknown> = {}) {
-  const now = new Date();
-  return {
-    id: "env-1",
-    name: "Daytona",
-    description: null,
-    driver: "sandbox",
-    status: "active",
-    config: {},
-    envVars: {},
-    metadata: {},
-    createdAt: now,
-    updatedAt: now,
-    ...overrides,
-  };
-}
-
-function reconciliationResult(
-  overrides: Record<string, unknown> = {},
-) {
-  return {
-    environment: environmentRow(),
-    action: "added" as const,
-    stockStatus: "missing" as const,
-    updateAvailable: false,
-    stockHash: "sha256:stock",
-    ...overrides,
-  };
-}
-
-function readyDriverResolver(status = "ready") {
-  return vi.fn(async () => ({
-    plugin: { id: "plugin-1", pluginKey: "sandbox-providers/daytona", status },
-  }));
-}
-
-function runningWorkerManager(running = true) {
-  return { isRunning: vi.fn(() => running), getWorker: vi.fn(() => undefined) };
-}
-
-type WorkerManagerSeam = NonNullable<ApplyManagedEnvironmentsOptions["workerManager"]>;
-type RecoveryHandle = NonNullable<ReturnType<WorkerManagerSeam["getWorker"]>>;
-
-/** A worker handle fake that records `ready` listeners so tests can fire them. */
-function recoveryHandle() {
-  const readyListeners: Array<(payload: { pluginId: string }) => void> = [];
-  const handle: RecoveryHandle = {
-    on: vi.fn((_event, listener) => {
-      readyListeners.push(listener);
-    }),
-    off: vi.fn((_event, listener) => {
-      const at = readyListeners.indexOf(listener);
-      if (at !== -1) readyListeners.splice(at, 1);
-    }),
-  };
-  return {
-    handle,
-    fireReady: () => {
-      for (const listener of [...readyListeners]) listener({ pluginId: "plugin-1" });
-    },
-  };
-}
-
-/** Lets the fire-and-forget recovery ensure settle. */
-function tick() {
-  return new Promise((resolve) => setImmediate(resolve));
-}
-
-type EnvironmentsSeam = NonNullable<ApplyManagedEnvironmentsOptions["environments"]>;
-type InstanceSettingsSeam = NonNullable<ApplyManagedEnvironmentsOptions["instanceSettings"]>;
-
-function environmentsSeam(overrides: Partial<EnvironmentsSeam> = {}): EnvironmentsSeam {
-  return {
-    ensureManagedSandboxEnvironment:
-      overrides.ensureManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(reconciliationResult()),
-    archiveManagedSandboxEnvironment:
-      overrides.archiveManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(null),
-    getById: overrides.getById ?? vi.fn().mockResolvedValue(null),
-    update: overrides.update ?? vi.fn().mockResolvedValue(environmentRow()),
-    findManagedSandboxEnvironment:
-      overrides.findManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(null),
-  };
-}
-
-function instanceSettingsSeam(overrides: Partial<InstanceSettingsSeam> = {}): InstanceSettingsSeam {
-  return {
-    get:
-      overrides.get ??
-      (vi.fn().mockResolvedValue({ defaultEnvironmentId: null }) as InstanceSettingsSeam["get"]),
-    update:
-      overrides.update ??
-      (vi.fn().mockResolvedValue({ defaultEnvironmentId: "env-1" }) as InstanceSettingsSeam["update"]),
-  };
-}
+  environmentRow,
+  environmentsSeam,
+  instanceSettingsSeam,
+  noDb,
+  parsedConfig,
+  readyDriverResolver,
+  reconciliationResult,
+  recoveryHandle,
+  runningWorkerManager,
+  tick,
+  type InstanceSettingsSeam,
+} from "./managed-environments.test-support.js";
 
 describe("applyManagedEnvironments", () => {
   it("no-ops for self-hosted instances and for documents without environments", async () => {

server/src/services/managed-environments.ts

@@ -43,9 +43,17 @@
  * sandbox provider falls back to its documented process environment variable
  * (e.g. `DAYTONA_API_KEY`) when `config` omits the key, so the deployment
  * delivers secrets as env vars and the managed document stays secret-free.
+ *
+ * `driver: "plugin"` entries take the same readiness path after the sandbox
+ * entry: await `pluginsReady`, require the plugin `ready`, declaring the
+ * entry's driver, and running a worker; else archive the entry's row and
+ * re-ensure it on the worker's `ready`. Each entry owns one row (see
+ * `managed-plugin-environments.ts`). Plugin entries never touch the instance
+ * default.
  */
 
 import type { Db } from "@paperclipai/db";
+import { HttpError } from "../errors.js";
 import { logger } from "../middleware/logger.js";
 import { environmentService } from "./environments.js";
 import type { ManagedSandboxEnvironmentReconcileAction } from "./environments.js";
@@ -54,7 +62,10 @@ import type { ManagedInstanceConfig } from "./managed-config.js";
 import type { ManagedResourceStockStatus } from "./managed-resource-drift.js";
 import { parseExecutionPolicyBootstrapEnv } from "./execution-policy-bootstrap.js";
 import { resolvePluginSandboxProviderDriverByKey } from "./plugin-environment-driver.js";
+import { pluginRegistryService } from "./plugin-registry.js";
+import { resolveBundledCatalogRoot, resolveBundledPluginInstalls } from "./bundled-plugins.js";
 import type { PluginWorkerManager } from "./plugin-worker-manager.js";
+import type { ManagedPluginEnvironmentReconcileAction } from "./managed-plugin-environments.js";
 
 export interface ApplyManagedEnvironmentsOptions {
   env?: Record<string, string | undefined>;
@@ -75,8 +86,11 @@ export interface ApplyManagedEnvironmentsOptions {
    * subscription used for post-archive recovery: a worker in crash-restart
    * backoff re-emits `ready` when the manager respawns it, and the listener
    * re-ensures the archived environment (see `applyManagedEnvironments`).
+   *
+   * Plugin entries also need `call`, to run the plugin's own config
+   * validation; without it every plugin entry is treated as unavailable.
    */
-  workerManager?: Pick<PluginWorkerManager, "isRunning"> & {
+  workerManager?: Pick<PluginWorkerManager, "isRunning"> & Partial<Pick<PluginWorkerManager, "call">> & {
     getWorker(pluginId: string):
       | {
           on(event: "ready", listener: (payload: { pluginId: string }) => void): void;
@@ -92,6 +106,8 @@ export interface ApplyManagedEnvironmentsOptions {
     | "getById"
     | "update"
     | "findManagedSandboxEnvironment"
+    | "ensureManagedPluginEnvironment"
+    | "archiveManagedPluginEnvironment"
   >;
   /** Test seam: overrides the instance-settings service built from `db`. */
   instanceSettings?: Pick<ReturnType<typeof instanceSettingsService>, "get" | "update">;
@@ -100,6 +116,32 @@ export interface ApplyManagedEnvironmentsOptions {
     db: Db;
     driverKey: string;
   }) => Promise<{ plugin: { id: string; pluginKey: string; status: string } } | null>;
+  /**
+   * Test seam: overrides the lookup of a plugin entry's plugin. Returns null
+   * when the plugin is not installed; `declaresDriver` is false when its
+   * manifest does not declare the entry's environment driver.
+   */
+  resolveEnvironmentPlugin?: (input: {
+    db: Db;
+    pluginKey: string;
+    driverKey: string;
+  }) => Promise<{ plugin: { id: string; pluginKey: string; status: string }; declaresDriver: boolean } | null>;
+  /**
+   * Manifest ids (`pluginKey`) that the `plugins.autoInstall` catalog keys
+   * resolve to: `app.locals.managedBundledPluginKeys`, from
+   * `resolveBundledPluginInstalls` over the bundled and distribution
+   * catalogs. A plugin entry whose `pluginKey` is not in this list refuses
+   * startup. When omitted, the list is resolved here the same way.
+   */
+  autoInstalledPluginKeys?: readonly string[];
+}
+
+export interface ManagedPluginEnvironmentReconciliationOutcome {
+  environmentId: string;
+  name: string;
+  pluginKey: string;
+  driverKey: string;
+  action: ManagedPluginEnvironmentReconcileAction;
 }
 
 export interface ManagedEnvironmentReconciliationOutcome {
@@ -123,6 +165,28 @@ export interface ApplyManagedEnvironmentsResult {
   /** Preserve-only reconciliation never creates replacement backups. */
   backedUp: number;
   outcomes: ManagedEnvironmentReconciliationOutcome[];
+  pluginOutcomes: ManagedPluginEnvironmentReconciliationOutcome[];
+}
+
+/**
+ * The plugin refused the entry's config (`HttpError` 422 from
+ * `normalizeEnvironmentConfig` or `validatePluginEnvironmentDriverConfig`).
+ * Retrying cannot help until the config changes. Every other failure, such
+ * as a worker that crashed mid-call, is treated as transient.
+ */
+function isPermanentConfigRejection(err: unknown): boolean {
+  return err instanceof HttpError && err.status === 422;
+}
+
+async function resolveEnvironmentPluginByKey(input: { db: Db; pluginKey: string; driverKey: string }) {
+  const plugin = await pluginRegistryService(input.db).getByKey(input.pluginKey);
+  if (!plugin) return null;
+  return {
+    plugin: { id: plugin.id, pluginKey: plugin.pluginKey, status: plugin.status },
+    declaresDriver: plugin.manifestJson.environmentDrivers?.some(
+      (driver) => driver.driverKey === input.driverKey && driver.kind !== "sandbox_provider",
+    ) ?? false,
+  };
 }
 
 /**
@@ -176,19 +240,38 @@ export async function applyManagedEnvironments(
     }
   }
 
-  if (managedConfig.environments.length === 0) return null;
+  const pluginSpecs = managedConfig.pluginEnvironments;
+  if (managedConfig.environments.length === 0 && pluginSpecs.length === 0) return null;
 
   // The forced-execution-mode bootstrap (`PAPERCLIP_EXECUTION_MODE=kubernetes`)
   // and this one both own the single Paperclip-managed sandbox row
   // (`environments_managed_sandbox_idx`). Configuring both is contradictory;
-  // refuse startup rather than let bootstrap ordering pick a winner.
+  // refuse startup rather than let bootstrap ordering pick a winner. Plugin
+  // entries do not touch that slot.
   const env = opts.env ?? process.env;
-  if (parseExecutionPolicyBootstrapEnv(env)) {
+  if (managedConfig.environments.length > 0 && parseExecutionPolicyBootstrapEnv(env)) {
     throw new Error(
       `PAPERCLIP_EXECUTION_MODE and the PAPERCLIP_MANAGED_CONFIG "environments" section are mutually exclusive: both manage the single instance sandbox environment`,
     );
   }
 
+  // Coherence, fail closed like the parser: a plugin entry names its plugin by
+  // manifest id, and only `plugins.autoInstall` provisions plugins on a
+  // managed instance, so some auto-installed catalog entry must install it.
+  if (pluginSpecs.length > 0) {
+    const autoInstalledPluginKeys = opts.autoInstalledPluginKeys ?? resolveBundledPluginInstalls(
+      managedConfig.plugins.autoInstall,
+      { catalogRoot: resolveBundledCatalogRoot(env), env, enforceCatalogRoot: true },
+    ).map((install) => install.pluginKey);
+    for (const spec of pluginSpecs) {
+      if (!autoInstalledPluginKeys.includes(spec.pluginKey)) {
+        throw new Error(
+          `PAPERCLIP_MANAGED_CONFIG "environments" entry "${spec.name}" uses plugin "${spec.pluginKey}", which no "plugins.autoInstall" catalog entry installs; a managed plugin environment requires its plugin to be provisioned`,
+        );
+      }
+    }
+  }
+
   // The heartbeat resumes queued runs right after this bootstrap step, and
   // lease acquisition fails hard on a provider whose plugin is missing or not
   // ready. Wait for the bundled-plugin startup pass to finish, then refuse to
@@ -266,10 +349,7 @@ export async function applyManagedEnvironments(
   // worker recovered between the gate check and the subscription. No handle
   // means no respawn is coming (activation never started a worker), so the
   // degraded-until-next-boot posture stands.
-  const scheduleRecoveryReactivation = (
-    spec: ManagedInstanceConfig["environments"][number],
-    pluginId: string,
-  ): void => {
+  const scheduleRecoveryReactivation = (pluginId: string, reactivateEntry: () => Promise<void>): void => {
     const manager = opts.workerManager;
     const handle = manager?.getWorker(pluginId);
     if (!handle) return;
@@ -278,39 +358,41 @@ export async function applyManagedEnvironments(
       if (reactivated) return;
       reactivated = true;
       handle.off("ready", reactivate);
-      void environments
-        .ensureManagedSandboxEnvironment({
-          name: spec.name,
-          description: spec.description,
-          provider: spec.provider,
-          config: { ...spec.config },
-          stockVersion: managedConfig.catalogVersion,
-          platformFullyManaged: true,
-        })
-        .then((result) => {
-          logger.info(
-            {
-              environmentId: result.environment.id,
-              name: spec.name,
-              provider: spec.provider,
-              action: result.action,
-              stockStatus: result.stockStatus,
-              updateAvailable: result.updateAvailable,
-            },
-            "managed sandbox environment reactivated after provider worker recovery",
-          );
-        })
-        .catch((err: unknown) => {
-          logger.error(
-            { err, name: spec.name, provider: spec.provider },
-            "failed to reactivate managed sandbox environment after provider worker recovery (degraded: environment unavailable until next boot)",
-          );
-        });
+      void reactivateEntry();
     };
     handle.on("ready", reactivate);
     if (manager?.isRunning(pluginId)) reactivate();
   };
 
+  const reactivateSandboxEntry = async (spec: ManagedInstanceConfig["environments"][number]): Promise<void> => {
+    try {
+      const result = await environments.ensureManagedSandboxEnvironment({
+        name: spec.name,
+        description: spec.description,
+        provider: spec.provider,
+        config: { ...spec.config },
+        stockVersion: managedConfig.catalogVersion,
+        platformFullyManaged: true,
+      });
+      logger.info(
+        {
+          environmentId: result.environment.id,
+          name: spec.name,
+          provider: spec.provider,
+          action: result.action,
+          stockStatus: result.stockStatus,
+          updateAvailable: result.updateAvailable,
+        },
+        "managed sandbox environment reactivated after provider worker recovery",
+      );
+    } catch (err) {
+      logger.error(
+        { err, name: spec.name, provider: spec.provider },
+        "failed to reactivate managed sandbox environment after provider worker recovery (degraded: environment unavailable until next boot)",
+      );
+    }
+  };
+
   let ensured = 0;
   let failed = 0;
   let added = 0;
@@ -355,7 +437,7 @@ export async function applyManagedEnvironments(
         // manager restarts crashed workers, but nothing (re)installs a missing
         // or non-ready plugin at runtime.
         if (resolved && resolved.plugin.status === "ready") {
-          scheduleRecoveryReactivation(spec, resolved.plugin.id);
+          scheduleRecoveryReactivation(resolved.plugin.id, () => reactivateSandboxEntry(spec));
         }
         continue;
       }
@@ -407,6 +489,179 @@ export async function applyManagedEnvironments(
       );
     }
   }
+  const resolvePlugin = opts.resolveEnvironmentPlugin ?? resolveEnvironmentPluginByKey;
+  const pluginOutcomes: ManagedPluginEnvironmentReconciliationOutcome[] = [];
+
+  const ensurePluginEntry = (
+    spec: ManagedInstanceConfig["pluginEnvironments"][number],
+    workerManager: Pick<PluginWorkerManager, "isRunning" | "call">,
+  ) =>
+    environments.ensureManagedPluginEnvironment({
+      name: spec.name,
+      description: spec.description,
+      pluginKey: spec.pluginKey,
+      driverKey: spec.driverKey,
+      driverConfig: { ...spec.driverConfig },
+      workerManager,
+    });
+
+  type PluginSpec = ManagedInstanceConfig["pluginEnvironments"][number];
+  type PluginWorkerAccess = Pick<PluginWorkerManager, "isRunning" | "call">;
+
+  const archivePluginEntry = (spec: PluginSpec) =>
+    environments
+      .archiveManagedPluginEnvironment({ pluginKey: spec.pluginKey, driverKey: spec.driverKey })
+      .catch((archiveErr: unknown) => {
+        logger.error(
+          { err: archiveErr, name: spec.name, pluginKey: spec.pluginKey, driverKey: spec.driverKey },
+          "failed to archive the managed plugin environment of an unavailable plugin",
+        );
+        return null;
+      });
+
+  /**
+   * Re-ensure a plugin entry each time its worker reports `ready`, until one
+   * ensure succeeds or the plugin permanently rejects the entry's config. A
+   * worker that crashes during an attempt (for example inside
+   * `environmentValidateConfig`) leaves the listener armed, so its next
+   * respawn retries. A `ready` that arrives during an attempt queues one more
+   * attempt. No handle means no respawn is coming (activation never started a
+   * worker), so the degraded-until-next-boot posture stands.
+   */
+  const schedulePluginRecovery = (spec: PluginSpec, pluginId: string, workerAccess: PluginWorkerAccess): void => {
+    const manager = opts.workerManager;
+    const handle = manager?.getWorker(pluginId);
+    if (!handle) return;
+    const logBase = { name: spec.name, pluginKey: spec.pluginKey, driverKey: spec.driverKey };
+    let settled = false;
+    let inFlight = false;
+    let queued = false;
+    const settle = (): void => {
+      settled = true;
+      handle.off("ready", attempt);
+    };
+    function attempt(): void {
+      if (settled) return;
+      if (inFlight) {
+        queued = true;
+        return;
+      }
+      inFlight = true;
+      void ensurePluginEntry(spec, workerAccess)
+        .then((result) => {
+          settle();
+          logger.info(
+            { ...logBase, environmentId: result.environment.id, action: result.action },
+            "managed plugin environment reactivated after plugin worker recovery",
+          );
+        })
+        .catch((err: unknown) => {
+          if (isPermanentConfigRejection(err)) {
+            settle();
+            logger.error(
+              { err, ...logBase },
+              "the plugin rejected the managed plugin environment's config; not retrying (degraded: environment unavailable until the config changes)",
+            );
+          } else {
+            logger.warn(
+              { err, ...logBase },
+              "failed to reactivate managed plugin environment; retrying when the plugin worker is ready again",
+            );
+          }
+        })
+        .finally(() => {
+          inFlight = false;
+          if (queued && !settled) {
+            queued = false;
+            attempt();
+          }
+        });
+    }
+    handle.on("ready", attempt);
+    if (manager?.isRunning(pluginId)) attempt();
+  };
+
+  for (const spec of pluginSpecs) {
+    const logBase = { name: spec.name, pluginKey: spec.pluginKey, driverKey: spec.driverKey };
+    let recoverable: { pluginId: string; workerAccess: PluginWorkerAccess } | null = null;
+    try {
+      const resolved = await resolvePlugin({ db, pluginKey: spec.pluginKey, driverKey: spec.driverKey });
+      const manager = opts.workerManager;
+      const call = manager?.call;
+      // Only a `ready` record that declares the driver can recover without
+      // another boot: the worker manager restarts crashed workers, but nothing
+      // (re)installs a missing or non-ready plugin at runtime.
+      if (resolved && resolved.plugin.status === "ready" && resolved.declaresDriver && manager && call) {
+        recoverable = {
+          pluginId: resolved.plugin.id,
+          workerAccess: { isRunning: manager.isRunning.bind(manager), call: call.bind(manager) },
+        };
+      }
+      const workerRunning = resolved != null && (manager?.isRunning(resolved.plugin.id) ?? false);
+      if (!recoverable || !workerRunning) {
+        failed += 1;
+        const archived = await archivePluginEntry(spec);
+        logger.error(
+          {
+            ...logBase,
+            pluginStatus: resolved?.plugin.status ?? null,
+            declaresDriver: resolved?.declaresDriver ?? null,
+            workerRunning,
+            archivedEnvironmentId: archived?.id ?? null,
+          },
+          "managed plugin environment's plugin is not installed, ready, declaring its driver, and running; skipping ensure and archiving any previously provisioned row (degraded: environment unavailable)",
+        );
+        if (recoverable) schedulePluginRecovery(spec, recoverable.pluginId, recoverable.workerAccess);
+        continue;
+      }
+      const result = await ensurePluginEntry(spec, recoverable.workerAccess);
+      ensured += 1;
+      if (result.action === "added") added += 1;
+      else if (result.action === "unchanged") unchanged += 1;
+      else updated += 1;
+      pluginOutcomes.push({
+        environmentId: result.environment.id,
+        name: result.environment.name,
+        pluginKey: spec.pluginKey,
+        driverKey: spec.driverKey,
+        action: result.action,
+      });
+      logger.info(
+        { ...logBase, environmentId: result.environment.id, action: result.action },
+        "managed plugin environment reconciled",
+      );
+    } catch (err) {
+      failed += 1;
+      if (recoverable && !isPermanentConfigRejection(err)) {
+        if (!recoverable.workerAccess.isRunning(recoverable.pluginId)) {
+          // The worker died after the readiness check (for example inside
+          // `environmentValidateConfig`): treat it like a plugin that is
+          // down, so the row is not left active, and retry on its `ready`.
+          const archived = await archivePluginEntry(spec);
+          logger.error(
+            { err, ...logBase, archivedEnvironmentId: archived?.id ?? null },
+            "managed plugin environment's worker stopped during ensure; archived any previously provisioned row and retrying when the plugin worker is ready again (degraded: environment unavailable)",
+          );
+        } else {
+          // The worker is still running (for example the call timed out).
+          // Keep any previously provisioned row as it is, so agents keep a
+          // working environment; retry once now and on the worker's next
+          // `ready`, and otherwise at the next boot.
+          logger.error(
+            { err, ...logBase },
+            "failed to ensure managed plugin environment while its worker is running; keeping any previously provisioned row and retrying (degraded: declaration not applied)",
+          );
+        }
+        schedulePluginRecovery(spec, recoverable.pluginId, recoverable.workerAccess);
+      } else {
+        logger.error(
+          { err, ...logBase },
+          "failed to ensure managed plugin environment; continuing boot (degraded: environment unavailable)",
+        );
+      }
+    }
+  }
+
   return {
     ensured,
     failed,
@@ -417,5 +672,6 @@ export async function applyManagedEnvironments(
     removed: 0,
     backedUp: 0,
     outcomes,
+    pluginOutcomes,
   };
 }

server/src/services/native-local-process-stop.ts

@@ -6,6 +6,7 @@ import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
 import { nativeSessionProviderEvidence } from "./native-runtime/native-session-resume.js";
 import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
 import os from "node:os";
+import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
 
 export const PROCESS_START_REQUESTED = "native.process_start_requested";
 export const PROCESS_IDENTITY_RECORDED = "native.process_identity_recorded";
@@ -56,6 +57,8 @@ export async function recordNativeLocalProcessStop(db: Db, run: Pick<typeof hear
   if (leases.some(lease => lease.provider !== "local")) return false;
   if ((run.processPid && !absent(run.processPid)) ||
       (run.processGroupId && !absent(-run.processGroupId))) return false;
+  await notifyDeliveryWork(db, DELIVERY_QUEUES.executionReplacement,
+    JSON.stringify({ companyId: run.companyId, runId: run.id }));
   await appendHeartbeatRunEvent(db, {
     companyId: run.companyId,
     runId: run.id,

server/src/services/native-runtime/native-case-handoff.test.ts

@@ -84,7 +84,7 @@ describe("native persisted case handoff runtime", () => {
     expect(await isCaseHandoffSourceReleased(db, f.input)).toBe(false);
     const exited = once(child, "exit"); child.kill("SIGTERM"); await exited;
     const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
-    expect(await recordNativeLocalProcessStop(db, run!)).toBe(true);
+    expect(await db.transaction(tx => recordNativeLocalProcessStop(tx as unknown as typeof db, run!))).toBe(true);
     // A new worker has no process-local state and reconstructs the release.
     expect(await isCaseHandoffSourceReleased(db, f.input)).toBe(true);
     expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]?.resultJson).toHaveProperty("caseHandoffExecutionReleased");

server/src/services/native-runtime/native-runner-restart-preparation.integration.test.ts

@@ -1,11 +1,21 @@
 import { startNativeSessionExecutionLeaseRenewal, type NativeExecutionLeaseRenewal } from "./native-execution-lease.js";
 import { randomUUID } from "node:crypto";
-import { resolve } from "node:path";
+import { resolve, join } from "node:path";
+import { mkdir, readFile, writeFile } from "node:fs/promises";
 import { eq, sql } from "drizzle-orm";
-import { environmentLeases, environments, heartbeatRuns, nativeRunFinalizations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db";
+import { environmentLeases, environments, heartbeatRuns, nativeRunFinalizations, issueRecoveryActions, issues, completionContracts, nativeRunResults, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db";
 import { beforeAll, expect, it, vi } from "vitest";
-import { DurablePrpControlPlane } from "../../vendor/paperclip-runner/index.js";
+import { spawn } from "node:child_process";
+import { createRequire } from "node:module";
+import { DurablePrpControlPlane, createNativeSessionBackend, createRunnerdCodexTransport, defaultCapabilityRunnerdBinary, canonicalNativeRuntimeContextDigest } from "../../vendor/paperclip-runner/index.js";
 import { buildNativeExecutionInput } from "./native-execution-input.js";
+import { buildNativeCompletionContract } from "./completion-contracts.js";
+import { reconcileNativeFinalizations, claimNativeSessionResumptions } from "./native-finalization-reconciler.js";
+import { prepareNativeWorkspaceSync } from "./native-workspace-sync.js";
+import { workspaceOperationService } from "../workspace-operations.js";
+import type { EnvironmentLease } from "@paperclipai/shared";
+import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
+import { materializeAsset } from "./runtime-context.js";
 import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js";
 import { executePaperclipNativeSession } from "./native-session-executor.js";
 import { registerNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
@@ -21,6 +31,147 @@ describeEmbeddedPostgres("native runner restart recovery: preparation", () => {
   let successor: typeof environment.current.successor;
   beforeAll(() => { ({ runtimeRoot, companyId, agentId, successor } = environment.current); });
 
+  it.each(["recover", "provider event", "cancelled task"] as const)("reconciles the drained remote backend with %s", async mode => {
+    const fixture = await seedRun(`DRAINED-BACKEND-${mode.replaceAll(" ", "-")}`);
+    const environmentId = randomUUID(), leaseId = randomUUID();
+    await fixture.db.insert(environments).values({ id: environmentId, name: `Drained backend ${environmentId}`, driver: "sandbox" });
+    const [lease] = await fixture.db.insert(environmentLeases).values({ id: leaseId, companyId, environmentId,
+      heartbeatRunId: fixture.runId, issueId: fixture.issueId, status: "active",
+      provider: "fixture", providerLeaseId: `drained-backend-${leaseId}` }).returning();
+    const [issue] = await fixture.db.select().from(issues).where(eq(issues.id, fixture.issueId));
+    const [contract] = await fixture.db.select().from(completionContracts).where(eq(completionContracts.issueId, fixture.issueId));
+    const runtimeContext = nativeRuntimeContextFixture();
+    runtimeContext.instructions.bundle = await materializeAsset([{ path: "AGENTS.md", mode: 0o444,
+      content: Buffer.from("Complete the controlled restart fixture task.\n") }]);
+    runtimeContext.aggregateDigest = canonicalNativeRuntimeContextDigest(runtimeContext);
+    const sandbox = join(runtimeRoot, `workspace-${fixture.runId}`);
+    await mkdir(sandbox, { recursive: true });
+    const savedFile = join(sandbox, "saved-before-drain.txt");
+    await writeFile(savedFile, "Preserve admitted work.\n");
+    const execution = buildNativeExecutionInput({ companyId, agentId, runId: fixture.runId,
+      issue: { id: fixture.issueId, identifier: "NRR-DRAINED", title: "Continue after drain", description: null, workMode: "standard" },
+      taskPrompt: "Continue the admitted work once.", provider: "codex",
+      workspace: { id: fixture.runId, cwd: sandbox, repoUrl: null, repoRef: null, branchName: null },
+      normalizedSessionId: fixture.native.normalizedSessionId, runtimeContext,
+      completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1",
+        contract: buildNativeCompletionContract(issue, { revision: contract.revision }) },
+    });
+    await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: {
+      ...fixture.preparedProfile, nativeExecutionInput: execution,
+    } }).where(eq(heartbeatRuns.id, fixture.runId));
+    const remoteSandbox = join(runtimeRoot, `remote-${fixture.runId}`);
+    const shell: CommandManagedRuntimeRunner = { execute: async input => {
+      const startedAt = new Date().toISOString();
+      const child = spawn(input.command, input.args ?? [], { cwd: input.cwd,
+        env: { ...process.env, ...input.env }, stdio: ["pipe", "pipe", "pipe"] });
+      let stdout = "", stderr = "", timedOut = false;
+      child.stdout.setEncoding("utf8").on("data", chunk => { stdout += chunk; });
+      child.stderr.setEncoding("utf8").on("data", chunk => { stderr += chunk; });
+      child.stdin.on("error", () => undefined);
+      child.stdin.end(input.stdin);
+      const timeout = setTimeout(() => { timedOut = true; child.kill("SIGKILL"); }, input.timeoutMs ?? 30_000);
+      try { return await new Promise((resolveResult, reject) => {
+        child.once("error", reject);
+        child.once("close", (exitCode, signal) => resolveResult({ stdout, stderr, exitCode, signal, timedOut,
+          pid: child.pid ?? null, startedAt }));
+      }); } finally { clearTimeout(timeout); }
+    } };
+    const workspaceSync = await prepareNativeWorkspaceSync({ db: fixture.db, runId: fixture.runId, companyId,
+      workspaceId: fixture.runId, workspaceLocalDir: sandbox, lease: lease as unknown as EnvironmentLease,
+      target: { kind: "remote", transport: "sandbox", environmentId, leaseId, remoteCwd: remoteSandbox, runner: shell } });
+    expect(workspaceSync).not.toBeNull();
+    const child = spawn(process.execPath, ["--import", createRequire(import.meta.url).resolve("tsx"),
+      resolve(import.meta.dirname, "../../__tests__/helpers/native-drain-controller.test-worker.ts")],
+      { stdio: ["ignore", "pipe", "pipe", "ipc"], env: process.env });
+    const logs: string[] = [];
+    child.stdout?.on("data", data => logs.push(String(data)));
+    child.stderr?.on("data", data => logs.push(String(data)));
+    const exited = new Promise<number | null>(resolveExit => child.once("exit", resolveExit));
+    try {
+      const result = await new Promise<{ commands: number; errorName: string; errorMessage: string }>((resolveResult, reject) => {
+        const deadline = setTimeout(() => reject(new Error(`Controller handoff timed out: ${logs.join("").slice(-2000)}`)), 60_000);
+        child.once("error", error => { clearTimeout(deadline); reject(error); });
+        child.once("exit", code => { clearTimeout(deadline); reject(new Error(`Controller exited before handoff (${code}): ${logs.join("").slice(-2000)}`)); });
+        child.on("message", (message: { ready?: boolean; done?: boolean; commands: number; errorName: string; errorMessage: string; error?: string }) => {
+          if (message.ready) child.send({ connectionString: environment.current.temporary.connectionString, providerEvidence: mode === "provider event",
+            options: { execution, runnerInstanceId: fixture.native.runnerInstanceId, useRunnerd: true,
+              runnerExecutionTarget: { kind: "remote", transport: "sandbox", environmentId, leaseId, remoteCwd: remoteSandbox } } });
+          if (message.done) { clearTimeout(deadline); resolveResult(message); }
+          if (message.error) { clearTimeout(deadline); reject(new Error(message.error)); }
+        });
+      });
+      expect(result).toMatchObject({ commands: 0, errorMessage: "local_runtime_api_listener_unavailable" });
+      expect(await exited).toBe(0);
+      const [handoff] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
+      if (mode === "provider event") {
+        expect(handoff).toMatchObject({ phase: "terminal_failure", nextAttemptAt: null,
+          failureDetail: { recoveryMode: "ambiguous_state", providerSessionEstablished: true } });
+        expect(await claimNativeSessionResumptions({ db: fixture.db, runnerInstanceId: "no-provider-replay",
+          runIds: [fixture.runId], deferIneligible: false })).toHaveLength(0);
+        return;
+      }
+      expect(handoff).toMatchObject({ phase: "retryable_failure", attempt: 1,
+        failureDetail: { originalFailureCode: "local_runtime_api_listener_unavailable", recoveryMode: "bootstrap_retry",
+          providerSessionEstablished: false, providerEventsExist: false, checkpointExists: false } });
+      if (mode === "cancelled task") {
+        await fixture.db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, fixture.issueId));
+      }
+      const claims = (await Promise.all(["successor-a", "successor-b"].map(runnerInstanceId =>
+        claimNativeSessionResumptions({ db: fixture.db, runnerInstanceId, runIds: [fixture.runId],
+          now: new Date(handoff.nextAttemptAt!.getTime() + 1), deferIneligible: false }),
+      ))).flat();
+      if (mode === "cancelled task") { expect(claims).toHaveLength(0); return; }
+      expect(claims).toHaveLength(1);
+      const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
+      expect(run).toMatchObject({ status: "running", nativeSessionId: fixture.native.normalizedSessionId });
+      const [action] = await fixture.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, fixture.issueId));
+      expect(action).toMatchObject({ ownerType: "agent", ownerAgentId: agentId });
+      if (process.env.PAPERCLIP_TEST_DRAIN_CONTINUATION === "1") {
+        const claim = claims[0];
+        const root = join(runtimeRoot, `continued-${fixture.runId}`);
+        const home = join(root, "codex-home");
+        await mkdir(home, { recursive: true });
+        const report = join(root, "result.json");
+        const calls = join(root, "calls.log");
+        await writeFile(report, JSON.stringify({ schema: "paperclip.run_result.v1", reportedWorkDisposition: "done",
+          summary: "Admitted work completed after controller restart.", completionClaim: {
+            contractRevision: execution.completionContract.contract.revision, objectiveSatisfied: true,
+            criteria: execution.completionContract.contract.criteria.map(criterion => ({ criterionId: criterion.id,
+              status: "satisfied", evidenceRefs: [] })), remainingWork: [] },
+          evidence: [], verification: [], artifacts: [], attentionRequests: [] }));
+        const bundle = createRunnerdCodexTransport({ ...transportOptions(fixture, root),
+          runnerBinary: defaultCapabilityRunnerdBinary(),
+          codexArgs: ["--state-file", join(root, "provider.json"), "--terminal-result-file", report,
+            "--call-log", calls, "--record-process-start"],
+          runtimeContext: execution.runtimeContext, runnerRuntimeContext: execution.runtimeContext,
+          sourceCodexHome: home, environment: { CODEX_HOME: home, PATH: process.env.PATH },
+          lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null },
+        });
+        const backend = createNativeSessionBackend(execution, { runnerInstanceId: fixture.native.runnerInstanceId,
+          environment: { CODEX_HOME: home, PATH: process.env.PATH }, codexTransportFactory: () => bundle.transport });
+        try {
+          expect(await executePaperclipNativeSession({ db: fixture.db, execution, backend, useRunnerd: true,
+            runnerInstanceId: fixture.native.runnerInstanceId, leaseOwner: claim.leaseOwner,
+            runnerExecutionTarget: { kind: "remote", transport: "sandbox", environmentId, leaseId, remoteCwd: remoteSandbox,
+              runner: { execute: async () => { throw new Error("Unexpected remote preparation after the controlled backend owns execution"); } } as never },
+            turnTimeoutMs: 15_000 })).toMatchObject({ exitCode: 0, timedOut: false });
+          const recorder = workspaceOperationService(fixture.db).createRecorder({ companyId,
+            issueId: fixture.issueId, heartbeatRunId: fixture.runId });
+          const saved = await recorder.recordOperation({ phase: "workspace_finalize", cwd: sandbox,
+            run: async () => { await workspaceSync!.restoreWorkspace(); return { status: "succeeded", exitCode: 0 }; } });
+          expect(saved.status).toBe("succeeded");
+          await reconcileNativeFinalizations(fixture.db, [fixture.runId]);
+          expect(await fixture.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, fixture.runId)))
+            .toMatchObject([{ schemaStatus: "accepted" }]);
+          const [finished] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
+          expect(finished, JSON.stringify(finished.failureDetail)).toMatchObject({ phase: "committed", attempt: 2 });
+          expect(await readFile(savedFile, "utf8")).toBe("Preserve admitted work.\n");
+          expect((await readFile(calls, "utf8")).split("\n").filter(line => line === "process-start")).toHaveLength(1);
+        } finally { await bundle.transport.close(); }
+      }
+    } finally { child.kill("SIGKILL"); await exited; await workspaceSync?.cleanup(); }
+  }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
+
   it.each(["preparation", "claim lock"])("gives a fresh remote claim its full lease through blocked %s", async (blockedPhase) => {
     const fixture = await seedRun(`FRESH-SLOW-${blockedPhase === "claim lock" ? "CLAIM-LOCK" : "PREPARATION"}`);
     const environmentId = randomUUID(), leaseId = randomUUID();

server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts

@@ -2220,10 +2220,15 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", (
     },
   );
 
-  it.each([2, 3])("recovers a checkpointed restart interruption within provider budget %s", async (attempt) => {
-    const fixture = await seedRun(`INTERRUPTED-${attempt}`);
+  it.each([
+    { attempt: 2, driverKind: "codex_app_server" },
+    { attempt: 3, driverKind: "codex_app_server" },
+    { attempt: 2, driverKind: "acpx_runtime" },
+    { attempt: 2, driverKind: "opencode_server" },
+  ])("recovers a checkpointed $driverKind restart interruption within provider budget $attempt", async ({ attempt, driverKind }) => {
+    const fixture = await seedRun(`INTERRUPTED-${driverKind}-${attempt}`);
     await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: { sessionCheckpoint: {
-      driverKind: "codex_app_server",
+      driverKind,
       sessionId: fixture.native.normalizedSessionId,
       identity: { companyId, issueId: fixture.issueId, runId: fixture.runId, agentId, sessionId: fixture.native.normalizedSessionId },
       providerSessionId: "interrupted-provider",

server/src/services/native-runtime/native-safe-replacement.test.ts

@@ -16,6 +16,11 @@ import {
   deliverReconciledExecutions,
 } from "../execution-recovery-resolution.js";
 import { randomUUID } from "node:crypto";
+import { spawn } from "node:child_process";
+import { persistHeartbeatRunProcessMetadata } from "../heartbeat/run-lifecycle.js";
+import { hasNativeLocalProcessStop, releaseSettledNativeRunProcess } from "../native-local-process-stop.js";
+import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
+import { executionControlSchedule } from "../execution-control-work.js";
 import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
 import { tmpdir } from "node:os";
 import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
@@ -26,6 +31,7 @@ import {
   agents,
   companies,
   createDb,
+  subscribeDatabaseWork,
   heartbeatRunEvents,
   heartbeatRuns,
   issueRecoveryActions,
@@ -41,7 +47,7 @@ import {
   getEmbeddedPostgresTestSupport,
   startEmbeddedPostgresTestDatabase,
 } from "../../__tests__/helpers/embedded-postgres.js";
-import { reconcileSafeNativeReplacements } from "./native-safe-replacement.js";
+import { invalidateNativeReplacementDiscovery, reconcileSafeNativeReplacements } from "./native-safe-replacement.js";
 import { reconcileAbandonedExecutionControl } from "../execution-control-reconciliation.js";
 const externalDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL;
 const support = externalDatabaseUrl
@@ -219,6 +225,103 @@ const support = externalDatabaseUrl
         }
       },
     );
+    it("wakes a completed replacement scan after actual local stop proof commits", async () => {
+      const source = await seed(2);
+      const workspace = await mkdtemp(join(tmpdir(), "paperclip-late-stop-"));
+      const child = spawn(process.execPath, ["-e", "process.exit(0)"], { detached: true, stdio: "ignore" });
+      const pid = child.pid!;
+      await new Promise<void>((resolve, reject) => { child.once("exit", () => resolve()); child.once("error", reject); });
+      await persistHeartbeatRunProcessMetadata(db, source.runId, { pid, processGroupId: pid,
+        startedAt: new Date().toISOString(), targetKind: "local" });
+      await db.update(heartbeatRuns).set({ finishedAt: new Date(), runnerProfileJson: {
+        nativeExecutionInput: { provider: { kind: "codex" }, workspace: { cwd: workspace } },
+      } }).where(eq(heartbeatRuns.id, source.runId));
+      await db.update(nativeRunFinalizations).set({ failureCode: "provider_transport_failed" })
+        .where(eq(nativeRunFinalizations.runId, source.runId));
+      await appendHeartbeatRunEvent(db, { companyId: source.companyId, runId: source.runId, agentId: source.agentId,
+        eventType: "harness.ready", stream: "system", message: "Provider started before its transport failed." });
+      const verifyStoppedSession = async () => await hasNativeLocalProcessStop(db, source.companyId, source.runId)
+        ? { evidence: {}, retire: () => true } : null;
+      invalidateNativeReplacementDiscovery(db);
+      expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 0 });
+      expect(await executionControlSchedule(db).replacement()).toBeNull();
+      let wakes = 0;
+      const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.executionReplacement, event => {
+        if (event === "settled") { wakes++; invalidateNativeReplacementDiscovery(db); }
+      });
+      try {
+        await db.transaction(async tx => {
+          expect(await releaseSettledNativeRunProcess(tx as unknown as typeof db, source.runId)).toBe("released");
+          expect(wakes).toBe(0);
+        });
+        expect(wakes).toBeGreaterThan(0);
+        expect(await executionControlSchedule(db).replacement()).not.toBeNull();
+        expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 1 });
+        expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 0 });
+        expect(await executionControlSchedule(db).replacement()).toBeNull();
+        const successors = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, source.runId));
+        expect(successors).toHaveLength(1);
+      } finally { unsubscribe(); await rm(workspace, { recursive: true, force: true }); }
+    });
+    it("wakes completed recovery when the real task becomes eligible", async () => {
+      const source = await seed();
+      const workspace = await mkdtemp(join(tmpdir(), "paperclip-late-eligibility-"));
+      await db.update(heartbeatRuns).set({ runnerProfileJson: { recoveryEventInventoryVersion: 1,
+        nativeExecutionInput: { provider: { kind: "codex" }, workspace: { cwd: workspace } },
+      } }).where(eq(heartbeatRuns.id, source.runId));
+      await issueService(db).update(source.issueId, { status: "backlog" });
+      invalidateNativeReplacementDiscovery(db);
+      expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 });
+      expect(await executionControlSchedule(db).replacement()).toBeNull();
+      let wakes = 0;
+      const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.executionReplacement, event => {
+        if (event === "settled") { wakes++; invalidateNativeReplacementDiscovery(db); }
+      });
+      try {
+        await issueService(db).update(source.issueId, { status: "in_progress" });
+        expect(wakes).toBeGreaterThan(0);
+        expect(await executionControlSchedule(db).replacement()).not.toBeNull();
+        expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 1 });
+        expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 });
+        expect(await executionControlSchedule(db).replacement()).toBeNull();
+        expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, source.runId))).toHaveLength(1);
+      } finally { unsubscribe(); await rm(workspace, { recursive: true, force: true }); }
+    });
+    it("advances past a failed full page and retries it after later work progresses", async () => {
+      const workspace = await mkdtemp(join(tmpdir(), "paperclip-failed-page-"));
+      const sources: Awaited<ReturnType<typeof seed>>[] = [];
+      for (let index = 0; index < 26; index++) {
+        const source = await seed(); sources.push(source);
+        await db.update(heartbeatRuns).set({ runnerProfileJson: {
+          nativeExecutionInput: { provider: { kind: "codex" }, workspace: { cwd: workspace } },
+        } }).where(eq(heartbeatRuns.id, source.runId));
+        await db.update(nativeRunFinalizations).set({ failureCode: "provider_transport_failed" })
+          .where(eq(nativeRunFinalizations.runId, source.runId));
+      }
+      sources.sort((a, b) => a.runId.localeCompare(b.runId));
+      const later = sources.at(-1)!;
+      let failing = true;
+      const verifyStoppedSession = vi.fn(async (run: typeof heartbeatRuns.$inferSelect) => {
+        if (failing && run.id !== later.runId) throw new Error("Controlled stopped-session verifier failure");
+        return { evidence: {}, retire: () => true };
+      });
+      invalidateNativeReplacementDiscovery(db);
+      try {
+        await expect(reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession }))
+          .rejects.toThrow("native_replacement_decision_retry_required");
+        expect(verifyStoppedSession).toHaveBeenCalledTimes(25);
+        expect(await executionControlSchedule(db).replacement()).not.toBeNull();
+        expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 1 });
+        expect(verifyStoppedSession).toHaveBeenCalledTimes(26);
+        expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, later.runId))).toHaveLength(1);
+        failing = false;
+        expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 25 });
+        expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 0 });
+        expect(await executionControlSchedule(db).replacement()).toBeNull();
+        const successors = await db.select().from(heartbeatRuns).where(inArray(heartbeatRuns.retryOfRunId, sources.map(source => source.runId)));
+        expect(successors).toHaveLength(26);
+      } finally { await rm(workspace, { recursive: true, force: true }); }
+    }, 60_000);
     it.each(["unproven", "changed", "verified"] as const)("requires stopped-session proof through commit (%s)", async (mode) => {
       const source = await seed(2);
       await db.update(nativeRunFinalizations).set({ failureCode: "native_session_cleanup_quarantined" }).where(eq(nativeRunFinalizations.runId, source.runId));
@@ -235,8 +338,11 @@ const support = externalDatabaseUrl
         eventType: "tool.execution.started", stream: "system",
         payload: { name: "paperclip_finish", executionId: "completion", transport: "process" },
       });
-      await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession });
-      await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession });
+      for (let sweep = 0; sweep < 2; sweep++) {
+        const decision = reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession });
+        if (mode === "changed") await expect(decision).rejects.toThrow("native_replacement_decision_retry_required");
+        else await decision;
+      }
       const successors = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, source.runId));
       expect(successors).toHaveLength(mode === "verified" ? 1 : 0);
       const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, source.issueId));

server/src/services/native-runtime/native-safe-replacement.ts

@@ -5,7 +5,7 @@ import { notifyExecutionRunWork } from "../execution-control-notifications.js";
 import { logger } from "../../middleware/logger.js";
 import { createHash, randomUUID } from "node:crypto";
 import { stat } from "node:fs/promises";
-import { and, eq, inArray, isNull, sql } from "drizzle-orm";
+import { and, asc, eq, gt, inArray, isNull, or, sql } from "drizzle-orm";
 import {
   agentWakeupRequests,
   environmentLeases,
@@ -27,6 +27,8 @@ import { issueRecoveryActionService } from "../issue-recovery-actions.js";
 import { buildExecutionContinuation } from "../execution-continuation.js";
 import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
 import { hasRemoteTerminationReceipt } from "../remote-execution-termination.js";
+import { NATIVE_PROVIDER_DURABLE_EVENT_TYPES } from "./native-retired-backup-source.js";
+import { completedWorkspaceRecoveryCondition } from "../workspace-recovery-receipt.js";
 
 export const NATIVE_SAFE_REPLACEMENT_REASON = "native_safe_replacement";
 const record = (v: unknown): Record<string, unknown> =>
@@ -45,6 +47,66 @@ const processAlive = (pid: number | null) => {
 
 type ReplacementDb = Db | Parameters<Parameters<Db["transaction"]>[0]>[0];
 
+const listenerUnavailable = "local_runtime_api_listener_unavailable";
+function historicalListenerGuardPending() {
+  const profile = heartbeatRuns.runnerProfileJson, detail = nativeRunFinalizations.failureDetail;
+  const providerEvents = sql.join([...NATIVE_PROVIDER_DURABLE_EVENT_TYPES].map(value => sql`${value}`), sql`, `);
+  return and(
+    eq(nativeRunFinalizations.failureCode, "native_session_interrupted"),
+    eq(heartbeatRuns.errorCode, "native_session_interrupted"), eq(heartbeatRuns.error, listenerUnavailable),
+    sql`${detail}->>'message' = ${listenerUnavailable}`,
+    sql`${detail}->>'originalFailureCode' = 'native_session_interrupted'`,
+    sql`${detail}->>'recoveryMode' = 'ambiguous_state'`,
+    sql`${detail}->'providerSessionEstablished' = 'false'::jsonb`,
+    sql`${detail}->'providerEventsExist' = 'false'::jsonb`,
+    sql`${detail}->'checkpointExists' = 'false'::jsonb`,
+    eq(nativeRunFinalizations.attempt, 1),
+    isNull(nativeRunFinalizations.leaseOwner), isNull(nativeRunFinalizations.leaseExpiresAt),
+    isNull(nativeRunFinalizations.assessmentId), isNull(nativeRunFinalizations.decisionId),
+    isNull(heartbeatRuns.processPid), isNull(heartbeatRuns.processGroupId),
+    isNull(heartbeatRuns.sessionIdBefore), isNull(heartbeatRuns.sessionIdAfter), isNull(heartbeatRuns.retryOfRunId),
+    sql`not (coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) ?| array['cancellation', 'nativeCancellation'])`,
+    sql`coalesce(${profile}->'sessionCheckpoint', 'null'::jsonb) = 'null'::jsonb`,
+    sql`coalesce(${profile}->'nativeProviderPackAdmission', 'null'::jsonb) = 'null'::jsonb`,
+    sql`coalesce(${profile}#>>'{nativeProviderPackAdmissionState,state}', 'pending') = 'pending'`,
+    sql`${profile}#>>'{nativeExecutionInput,binding,runId}' = ${heartbeatRuns.id}::text`,
+    sql`${profile}#>>'{nativeExecutionInput,binding,companyId}' = ${heartbeatRuns.companyId}::text`,
+    sql`${profile}#>>'{nativeExecutionInput,binding,agentId}' = ${heartbeatRuns.agentId}::text`,
+    sql`${profile}#>>'{nativeExecutionInput,binding,issueId}' = ${nativeRunFinalizations.issueId}::text`,
+    sql`${profile}#>>'{nativeExecutionInput,session,normalizedSessionId}' = ${heartbeatRuns.nativeSessionId}::text`,
+    sql`not exists (select 1 from ${heartbeatRuns} prior where prior.company_id = ${heartbeatRuns.companyId}
+      and prior.native_session_id = ${heartbeatRuns.nativeSessionId} and prior.id <> ${heartbeatRuns.id})`,
+    sql`not exists (select 1 from ${heartbeatRunEvents} event where event.company_id = ${heartbeatRuns.companyId}
+      and event.run_id = ${heartbeatRuns.id} and (event.source_event_id is not null
+        or event.event_type in (${providerEvents}) or event.event_type ~ '^(turn|item|tool|semantic_tool|run[.]result)[.]'))`,
+    sql`not exists (select 1 from ${toolInvocations} invocation where invocation.company_id = ${heartbeatRuns.companyId}
+      and invocation.run_id = ${heartbeatRuns.id})`,
+    sql`not exists (select 1 from ${issueRecoveryActions}
+      where ${issueRecoveryActions.companyId} = ${heartbeatRuns.companyId}
+        and ${issueRecoveryActions.sourceIssueId} = ${nativeRunFinalizations.issueId}
+        and ${issueRecoveryActions.cause} = 'native_continuation_requires_reconciliation'
+        and ${issueRecoveryActions.evidence}->>'runId' = ${heartbeatRuns.id}::text
+        and ${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'
+        and not coalesce((${completedWorkspaceRecoveryCondition(heartbeatRuns.id)}), false))`,
+    sql`exists (select 1 from ${issues} task where task.company_id = ${heartbeatRuns.companyId}
+      and task.id = ${nativeRunFinalizations.issueId} and task.assignee_agent_id = ${heartbeatRuns.agentId}
+      and task.status in ('todo', 'in_progress', 'in_review', 'blocked')
+      and (task.status <> 'blocked' or exists (select 1 from ${issueRecoveryActions} hold
+        where hold.company_id = task.company_id and hold.source_issue_id = task.id and hold.kind = 'active_run_watchdog'
+          and hold.cause = ${nativeRunFinalizations.failureCode} and hold.evidence->>'runId' = ${heartbeatRuns.id}::text
+          and hold.evidence#>>'{nativeFailureBlock,runId}' = ${heartbeatRuns.id}::text
+          and hold.evidence#>'{nativeFailureBlock,statusVersion}' = to_jsonb(task.status_version))))`,
+  );
+}
+
+async function historicalListenerGuardEvidence(db: ReplacementDb, run: typeof heartbeatRuns.$inferSelect) {
+  const [evidence] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
+    .innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)).where(and(
+      eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.id, run.id), historicalListenerGuardPending(),
+    )).limit(1);
+  return Boolean(evidence);
+}
+
 /** A saved remote source is proved by its current ledger, never a controller path. */
 async function remoteWorkspaceReplacementEvidence(
   db: ReplacementDb,
@@ -113,16 +175,95 @@ export function nativeReplacementPending() {
   return and(
     eq(heartbeatRuns.status, "failed"),
     eq(nativeRunFinalizations.phase, "terminal_failure"),
-    inArray(
+    or(inArray(
       nativeRunFinalizations.failureCode,
       ["native_provider_terminal_failed", "native_session_cleanup_quarantined", "provider_transport_failed"],
-    ),
+    ), historicalListenerGuardPending()),
     isNull(nativeRunFinalizations.resultId),
     sql`coalesce(${nativeRunFinalizations.failureDetail}->>'successorRunId', '') = ''`,
     sql`coalesce(${nativeRunFinalizations.failureDetail}->>'replacementDenied', '') = ''`,
   );
 }
 
+const REPLACEMENT_SCAN_LIMIT = 100;
+const replacementDiscovery = new WeakMap<object, { after: string | undefined; complete: boolean; generation: number; rescan: boolean }>();
+function replacementDiscoveryState(db: Db) {
+  let state = replacementDiscovery.get(db);
+  if (!state) { state = { after: undefined, complete: false, generation: 0, rescan: false }; replacementDiscovery.set(db, state); }
+  return state;
+}
+
+export function invalidateNativeReplacementDiscovery(db: Db) {
+  const state = replacementDiscoveryState(db);
+  state.complete = false; state.rescan = true; state.generation++;
+}
+
+async function replacementHistoryReady(db: Db, run: typeof heartbeatRuns.$inferSelect, issueId: string, retryDatabaseErrors = false) {
+  try {
+    const continuation = await buildExecutionContinuation({ db, companyId: run.companyId, issueId,
+      agentId: run.agentId, context: record(run.contextSnapshot), summary: null, exposeLowTrustRaw: false });
+    return continuation.unresolvedInteractionIds.length === 0;
+  } catch (error) {
+    let cause: unknown = error;
+    for (let depth = 0; retryDatabaseErrors && depth < 4 && cause && typeof cause === "object"; depth++) {
+      const failure = cause as { code?: unknown; cause?: unknown };
+      if (typeof failure.code === "string" && /^[A-Z0-9_]+$/.test(failure.code)) throw error;
+      cause = failure.cause;
+    }
+    return null;
+  }
+}
+
+async function discoverNativeReplacements(db: Db, after: string | undefined, limit = 25) {
+  const candidates: Array<{ run: typeof heartbeatRuns.$inferSelect; coordinator: typeof nativeRunFinalizations.$inferSelect }> = [];
+  let scanned = 0;
+  for (;;) {
+    const rows = await db.select({ run: heartbeatRuns, coordinator: nativeRunFinalizations }).from(heartbeatRuns)
+      .innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id))
+      .where(and(nativeReplacementPending(), after ? gt(heartbeatRuns.id, after) : undefined))
+      .orderBy(asc(heartbeatRuns.id)).limit(Math.min(25, REPLACEMENT_SCAN_LIMIT - scanned));
+    if (!rows.length) return { candidates, scanned, after: undefined };
+    for (const row of rows) {
+      after = row.run.id; scanned++;
+      let ready = true;
+      if (row.coordinator.failureCode === "native_session_interrupted") {
+        ready = false;
+        if (row.run.runnerProfileJson?.remoteWorkspace != null) {
+          const leases = await db.select().from(environmentLeases).where(and(
+            eq(environmentLeases.companyId, row.run.companyId), eq(environmentLeases.heartbeatRunId, row.run.id),
+          ));
+          if (leases.every(lease => lease.releasedAt !== null)) {
+            const evidence = await remoteWorkspaceReplacementEvidence(db, row.run, row.coordinator.issueId, leases);
+            ready = Boolean(evidence?.workspacePreserved && evidence.providerStopped)
+              && await replacementHistoryReady(db, row.run, row.coordinator.issueId, true) === true;
+          }
+        }
+      }
+      if (ready) candidates.push(row);
+      if (candidates.length >= limit || scanned >= REPLACEMENT_SCAN_LIMIT) return { candidates, scanned, after };
+    }
+    if (rows.length < 25) return { candidates, scanned, after: undefined };
+  }
+}
+
+function settleReplacementDiscovery(state: ReturnType<typeof replacementDiscoveryState>, after: string | undefined, generation: number) {
+  const rescan = state.rescan || generation !== state.generation;
+  state.after = after; state.complete = !after && !rescan;
+  state.rescan = Boolean(after && rescan);
+}
+
+export async function hasPendingNativeReplacements(db: Db) {
+  const state = replacementDiscoveryState(db);
+  if (state.complete) return false;
+  if (state.after) return true;
+  state.rescan = false;
+  const generation = state.generation;
+  const result = await discoverNativeReplacements(db, undefined, 1);
+  if (result.candidates.length) return true;
+  settleReplacementDiscovery(state, result.after, generation);
+  return !state.complete;
+}
+
 /** Retryable outbox: terminal_failure plus a new-format failure code is the durable candidate. */
 export async function reconcileSafeNativeReplacements(
   db: Db,
@@ -136,18 +277,14 @@ export async function reconcileSafeNativeReplacements(
     failpoint?: (phase: "successor_inserted" | "lineage_committed") => void;
   } = {},
 ) {
-  const candidates = await db
-    .select({ run: heartbeatRuns, coordinator: nativeRunFinalizations })
-    .from(heartbeatRuns)
-    .innerJoin(
-      nativeRunFinalizations,
-      eq(nativeRunFinalizations.runId, heartbeatRuns.id),
-    )
-    .where(
-      nativeReplacementPending(),
-    )
-    .limit(25);
-  let scheduled = 0;
+  const discovery = replacementDiscoveryState(db);
+  if (discovery.complete) discovery.after = undefined;
+  discovery.complete = false;
+  if (!discovery.after) discovery.rescan = false;
+  const generation = discovery.generation;
+  const selection = await discoverNativeReplacements(db, discovery.after);
+  const candidates = selection.candidates;
+  let scheduled = 0, pendingFailures = 0;
   for (const { run, coordinator } of candidates) {
     try {
       if (
@@ -170,11 +307,13 @@ export async function reconcileSafeNativeReplacements(
         );
       // Teardown is still in progress. The next sweep rechecks its durable outcome.
       if (leases.some((lease) => lease.releasedAt === null)) continue;
-      const stoppedSession = coordinator.failureCode !== "native_provider_terminal_failed"
+      const historicalListener = coordinator.failureCode === "native_session_interrupted";
+      if (historicalListener && !await historicalListenerGuardEvidence(db, run)) continue;
+      const stoppedSession = coordinator.failureCode !== "native_provider_terminal_failed" && !historicalListener
         ? await options.verifyStoppedSession?.(run) ?? null : null;
       // A transport label alone is not evidence. Keep inspecting these candidates
       // as process cleanup and the final transcript become durable.
-      if (coordinator.failureCode !== "native_provider_terminal_failed" && !stoppedSession) continue;
+      if (coordinator.failureCode !== "native_provider_terminal_failed" && !historicalListener && !stoppedSession) continue;
       const invocations = await db
         .select()
         .from(toolInvocations)
@@ -263,23 +402,11 @@ export async function reconcileSafeNativeReplacements(
           ? [String(p.kind ?? p.type)]
           : [];
       });
-      let historyComplete = false;
-      try {
-        const continuation = await buildExecutionContinuation({
-          db,
-          companyId: run.companyId,
-          issueId: coordinator.issueId,
-          agentId: run.agentId,
-          context: record(run.contextSnapshot),
-          summary: null,
-          exposeLowTrustRaw: false,
-        });
-        if (continuation.unresolvedInteractionIds.length > 0) continue;
-        historyComplete = true;
-      } catch {
-        /* The named operator outcome retains incomplete context. */
-      }
+      const history = await replacementHistoryReady(db, run, coordinator.issueId);
+      if (history === false) continue;
+      const historyComplete = history === true;
       const remoteWorkspace = await remoteWorkspaceReplacementEvidence(db, run, coordinator.issueId, leases);
+      if (historicalListener && (!remoteWorkspace?.workspacePreserved || !remoteWorkspace.providerStopped || invocations.length)) continue;
       const decision = decideNativeReplacement({
         failedSession: true,
         // A facade transport failure can conceal an authoritative protocol
@@ -413,6 +540,13 @@ export async function reconcileSafeNativeReplacements(
           current.attempt >= 3
         )
           return false;
+        let historicalRun: typeof heartbeatRuns.$inferSelect | undefined;
+        if (historicalListener) {
+          const [currentRun] = await tx.select().from(heartbeatRuns).where(and(
+            eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId),
+          )).for("update");
+          historicalRun = currentRun;
+        }
         if (task.status === "blocked") {
           const failureHolds = await tx.select().from(issueRecoveryActions).where(and(
             eq(issueRecoveryActions.companyId, run.companyId),
@@ -427,6 +561,14 @@ export async function reconcileSafeNativeReplacements(
           });
           if (!ownsBlock) return false;
         }
+        const preservationHolds = historicalListener ? await tx.select().from(issueRecoveryActions).where(and(
+          eq(issueRecoveryActions.companyId, run.companyId), eq(issueRecoveryActions.sourceIssueId, task.id),
+          eq(issueRecoveryActions.kind, "active_run_watchdog"),
+          eq(issueRecoveryActions.cause, "native_continuation_requires_reconciliation"),
+          sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
+          sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`,
+        )).for("update") : [];
+        if (historicalListener && (!historicalRun || !await historicalListenerGuardEvidence(tx, historicalRun))) return false;
         if (remoteWorkspace) {
           const currentWorkspace = await remoteWorkspaceReplacementEvidence(tx, run, coordinator.issueId, leases, true);
           if (!currentWorkspace?.workspacePreserved || !currentWorkspace.providerStopped) return false;
@@ -548,6 +690,7 @@ export async function reconcileSafeNativeReplacements(
           eq(issueRecoveryActions.cause, current.failureCode!),
           sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
         )).for("update");
+        holds.push(...preservationHolds);
         if (holds.some(hold => hold.status === "active" || hold.status === "escalated")) {
           await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: run.companyId, issueId: task.id });
         }
@@ -587,11 +730,17 @@ export async function reconcileSafeNativeReplacements(
       }
     } catch (error) {
       if (options.failpoint) throw error;
+      pendingFailures++;
       logger.warn(
         { runId: run.id },
         "Native replacement remains pending; another sweep will retry its durable decision",
       );
     }
   }
-  return { scanned: candidates.length, scheduled };
+  settleReplacementDiscovery(discovery, selection.after, generation);
+  if (pendingFailures) {
+    discovery.complete = false; discovery.rescan = true;
+    throw new Error("native_replacement_decision_retry_required");
+  }
+  return { scanned: selection.scanned, scheduled };
 }

server/src/services/remote-workspaces.integration.test.ts

@@ -13,7 +13,7 @@ import { pathToFileURL } from "node:url";
 import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
 import { and, eq, sql } from "drizzle-orm";
 import { activityLog, agents, agentTaskSessions, agentWakeupRequests, applyPendingMigrations, closeRegisteredClients, companies, createDb, environmentLeases, environments,
-  executionWorkspaceRepositories, executionWorkspaces, executionWorkspaceSnapshots, heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, issues, livenessDb, nativeRunFinalizations, projects, projectWorkspaces, workspaceOperations } from "@paperclipai/db";
+  subscribeDatabaseWork, executionWorkspaceRepositories, executionWorkspaces, executionWorkspaceSnapshots, heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, issueThreadInteractions, issues, livenessDb, nativeRunFinalizations, projects, projectWorkspaces, workspaceOperations } from "@paperclipai/db";
 import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
 import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
 import { buildProjectMentionHref, type Environment, type EnvironmentLease } from "@paperclipai/shared";
@@ -25,12 +25,13 @@ import { createSavedWorkspaceLoader, prepareRemoteWorkspace, readRemoteWorkspace
   clearRemoteWorkspaceLocationForReleasedLease, createRemoteWorkspaceGitAuth, prepareRemoteWorkspaceRepository,
   reserveRemoteWorkspace, saveRemoteWorkspace, shouldUseRemoteWorkspaces } from "./remote-workspaces.js";
 import type { createGitRemoteAuthProvider } from "./git-credentials.js";
-import { reconcileSafeNativeReplacements } from "./native-runtime/native-safe-replacement.js";
+import { invalidateNativeReplacementDiscovery, reconcileSafeNativeReplacements } from "./native-runtime/native-safe-replacement.js";
 import { remoteTerminationReceipt } from "./remote-execution-termination.js";
 import { RemoteWorkspaceRestoreError } from "./remote-workspace-transport.js";
 import { readRemoteRunnerState } from "./native-runtime/native-session-executor.js";
 import { notFound } from "../errors.js";
 import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
+import { issueThreadInteractionService } from "./issue-thread-interactions.js";
 import { createRemoteWorkspaceFiles } from "./remote-workspace-files.js";
 import { workspaceFileResourceService } from "./workspace-file-resources.js";
 import { nativeWorkspaceExportPendingForLease, preserveRemoteWorkspacePreparationFailure } from "./native-runtime/native-workspace-finalization-state.js";
@@ -47,6 +48,10 @@ import { instanceSettingsService } from "./instance-settings.js";
 import { resolveRemoteWorkspaceSource } from "./heartbeat/remote-workspace-source.js";
 import { toExecutionWorkspace } from "./execution-workspaces.js";
 import { recoveryPostgresSupport, startRecoveryPostgres } from "../__tests__/helpers/recovery-postgres.js";
+import { getExecutionBlocker } from "./execution-blocker.js";
+import { executionControlSchedule } from "./execution-control-work.js";
+import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
+import { createSystemRecovery } from "./system-recovery-state.js";
 
 const exec = promisify(execFile);
 let databaseUrl = process.env.PAPERCLIP_REMOTE_WORKSPACE_TEST_DATABASE_URL;
@@ -82,6 +87,7 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem
     await db.insert(issues).values({ id: ids.issue, companyId: ids.company, title: "Use a remote workspace", status: "in_progress", assigneeAgentId: ids.agent });
     await db.insert(heartbeatRuns).values({ id: ids.run, companyId: ids.company, agentId: ids.agent, nativeIssueId: ids.issue,
       runtimeMode: "native", status: "running" });
+    invalidateNativeReplacementDiscovery(db);
     await db.insert(environmentLeases).values({ id: ids.lease, companyId: ids.company, environmentId: ids.environment,
       heartbeatRunId: ids.run, issueId: ids.issue, provider: "daytona", providerLeaseId: `allocation-${ids.lease}`, status: "active" });
   });
@@ -91,6 +97,10 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem
       ? (row.metadata.uploads as { objectKey: string }[]).map(item => item.objectKey) : [])].filter((value): value is string => !!value))) {
       await storage.deleteObject(ids.company, key);
     }
+    if (temporary && process.env.PAPERCLIP_TEST_HISTORICAL_LISTENER === "1") await db.transaction(async tx => {
+      await tx.execute(sql`select set_config('client_min_messages', 'warning', true)`);
+      await tx.execute(sql`TRUNCATE companies CASCADE`);
+    });
     await db.delete(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.companyId, ids.company));
     await db.delete(workspaceOperations).where(eq(workspaceOperations.companyId, ids.company));
     await db.delete(activityLog).where(eq(activityLog.companyId, ids.company));
@@ -1712,14 +1722,20 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem
     } finally { await recovered.cleanup(); }
   }, 60_000);
 
-  it.each(["saved", "dirty", "unsaved", "foreign_snapshot", "delete_requested", "deletion_race", "controller_claim_race", "stale_epoch", "unconfirmed_stop", "unknown_action"] as const)(
-    "checks durable remote work at the native safe-replacement decision (%s)", async scenario => {
+  it.each(["saved", "dirty", "unsaved", "foreign_snapshot", "delete_requested", "deletion_race", "controller_claim_race", "stale_epoch", "unconfirmed_stop", "unknown_action", "historical_listener_guard", "historical_listener_provider_event", "historical_listener_checkpoint", "historical_listener_resumed", "historical_listener_other_message", "historical_listener_cancelled", "historical_listener_prior_session", "historical_listener_unconfirmed_stop", "historical_listener_dirty", "historical_listener_stale_block", "historical_listener_unknown_event", "historical_listener_budget", "historical_listener_bad_preservation", "historical_listener_absent", "historical_listener_later_candidate", "historical_listener_late_proof", "historical_listener_lock_timeout", "historical_listener_late_interaction", "historical_listener_interaction_paused", "historical_listener_own_question"] as const)(
+    "checks durable remote work at the native safe-replacement decision (%s)", async kind => {
+      const historicalListener = kind.startsWith("historical_listener_");
+      const processAcceptance = kind === "historical_listener_guard" && process.env.PAPERCLIP_TEST_HISTORICAL_LISTENER === "1";
+      const scenario = kind === "historical_listener_late_proof" ? "dirty" : kind === "historical_listener_unconfirmed_stop" ? "unconfirmed_stop"
+        : kind === "historical_listener_dirty" ? "dirty" : historicalListener ? "saved" : kind;
+      const shouldSchedule = scenario === "saved" && (!historicalListener || ["historical_listener_guard", "historical_listener_later_candidate", "historical_listener_lock_timeout"].includes(kind));
       const reserved = await reserve({ kind: "empty" }), first = await target("native-replacement-sandbox");
       const prepared = await prepareRemoteWorkspace({ db, storage, workspace: reserved, runId: ids.run,
         lease: await lease(), target: first, assertOwned: async () => undefined });
       await fs.writeFile(path.join(prepared.remoteRoot, "saved.txt"), "work survives a failed provider session\n");
       if (scenario !== "unsaved") await saveRemoteWorkspace({ db, storage, workspace: prepared.workspace,
-        runId: ids.run, lease: prepared.lease, target: first, assertOwned: async () => undefined });
+        runId: ids.run, lease: prepared.lease, target: first, assertOwned: async () => undefined,
+        boundary: kind === "historical_listener_late_proof" ? "initial" : "final" });
       const current = await workspace(reserved.id), authority = readRemoteWorkspaceAuthority(current.metadata)!;
       if (scenario === "dirty" || scenario === "stale_epoch") await db.update(executionWorkspaces).set({ metadata: {
         ...current.metadata, remoteWorkspace: { ...authority, ...(scenario === "dirty" ? { dirty: true, writerRunId: ids.run }
@@ -1729,29 +1745,137 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem
         .where(eq(executionWorkspaceSnapshots.id, authority.snapshotId!));
       if (scenario === "delete_requested") await db.update(executionWorkspaceSnapshots).set({ status: "delete_requested" })
         .where(eq(executionWorkspaceSnapshots.id, authority.snapshotId!));
-      const stopped = (await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success",
+      if (processAcceptance) {
+        const { recordHistoricalListenerFailure } = await import("../__tests__/helpers/historical-listener-continuation.js");
+        await recordHistoricalListenerFailure({ db, connectionString: databaseUrl!, runId: ids.run, cwd: prepared.remoteRoot,
+          environmentId: ids.environment, leaseId: ids.lease });
+      }
+      const stopped = kind === "historical_listener_late_proof" ? prepared.lease : (await db.update(environmentLeases).set({ status: "released", releasedAt: new Date(), cleanupStatus: "success",
         metadata: { ...prepared.lease.metadata, ...(scenario !== "unconfirmed_stop" ? {
           remoteExecutionTermination: remoteTerminationReceipt(prepared.lease, {
             providerLeaseId: prepared.lease.providerLeaseId, state: "destroyed",
           }),
         } : {}) },
       }).where(eq(environmentLeases.id, ids.lease)).returning())[0]! as EnvironmentLease;
-      if (scenario === "saved") {
+      if (shouldSchedule) {
         await db.transaction(tx => clearRemoteWorkspaceLocationForReleasedLease({ db: tx, lease: stopped }));
         await fs.rm(prepared.remoteRoot, { recursive: true, force: true });
         await expect(fs.stat(prepared.remoteRoot)).rejects.toThrow();
       }
       const [source] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, ids.run));
-      await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(), runnerProfileJson: {
+      const sessionId = randomUUID();
+      await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(),
+        ...(historicalListener && !processAcceptance ? { nativeSessionId: sessionId } : {}),
+        ...(historicalListener ? { error: "local_runtime_api_listener_unavailable", errorCode: "native_session_interrupted" } : {}), runnerProfileJson: {
         ...source!.runnerProfileJson, recoveryEventInventoryVersion: 1,
-        nativeExecutionInput: { provider: { kind: "codex" }, workspace: { cwd: prepared.remoteRoot } },
+        nativeExecutionInput: processAcceptance ? source!.runnerProfileJson!.nativeExecutionInput : { provider: { kind: "codex" }, workspace: { cwd: prepared.remoteRoot },
+          ...(historicalListener ? { binding: { runId: ids.run, companyId: ids.company, agentId: ids.agent, issueId: ids.issue },
+            session: { normalizedSessionId: sessionId } } : {}) },
       } }).where(eq(heartbeatRuns.id, ids.run));
-      await db.insert(nativeRunFinalizations).values({ runId: ids.run, companyId: ids.company, issueId: ids.issue,
-        phase: "terminal_failure", attempt: 1, failureCode: "native_provider_terminal_failed",
-        failureDetail: { originalFailureCode: "fixture_checkpoint_unusable" } });
+      if (!processAcceptance) await db.insert(nativeRunFinalizations).values({ runId: ids.run, companyId: ids.company, issueId: ids.issue,
+        phase: "terminal_failure", attempt: 1, failureCode: historicalListener ? "native_session_interrupted" : "native_provider_terminal_failed",
+        failureDetail: historicalListener ? { originalFailureCode: "native_session_interrupted", message: "local_runtime_api_listener_unavailable",
+          recoveryMode: "ambiguous_state", providerSessionEstablished: false, providerEventsExist: false, checkpointExists: false }
+          : { originalFailureCode: "fixture_checkpoint_unusable" } });
+      if (historicalListener && !processAcceptance) {
+        const blocked = await issueService(db).update(ids.issue, { status: "blocked" });
+        await db.insert(issueRecoveryActions).values({ companyId: ids.company, sourceIssueId: ids.issue,
+          kind: "active_run_watchdog", cause: "native_session_interrupted", ownerType: "board", status: "active",
+          fingerprint: createHash("sha256").update(`${ids.run}:native_session_interrupted`).digest("hex"),
+          evidence: { runId: ids.run, nativeFailureBlock: { runId: ids.run, statusVersion: blocked!.statusVersion } },
+          nextAction: "Reconcile the original startup failure." });
+      }
+      if (kind === "historical_listener_guard" || kind === "historical_listener_bad_preservation") {
+        const actionId = randomUUID(), preservationId = randomUUID();
+        await db.insert(issueRecoveryActions).values({ id: actionId, companyId: ids.company, sourceIssueId: ids.issue,
+          kind: "active_run_watchdog", cause: "native_continuation_requires_reconciliation", ownerType: "system", status: "resolved", outcome: "blocked",
+          fingerprint: `native-workspace:${ids.run}`, evidence: { runId: ids.run,
+            workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [ids.lease] },
+            automaticRecovery: { replay: "blocked" }, continuationDelivery: "invalidated",
+            systemRecovery: { ...createSystemRecovery(ids.run), state: "complete", preservationOperationIds: [preservationId] },
+            workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: ids.run, status: "recovered", stage: "saving_receipts", attempt: 1, requestIds: [] },
+          }, nextAction: "Files were preserved without replay." });
+        await db.insert(workspaceOperations).values([{ id: preservationId, companyId: ids.company, heartbeatRunId: ids.run, issueId: ids.issue,
+          executionWorkspaceId: reserved.id, phase: "workspace_recovery", status: kind === "historical_listener_bad_preservation" ? "failed" : "succeeded", startedAt: new Date(),
+          metadata: { recoveryActionId: actionId, snapshotId: authority.snapshotId } },
+        { companyId: ids.company, heartbeatRunId: ids.run, issueId: ids.issue, phase: "workspace_finalize", status: "succeeded", startedAt: new Date(),
+          metadata: { owningService: "workspace_recovery", preservationOperationIds: [preservationId] } }]);
+      }
+      if (kind === "historical_listener_absent") await db.update(heartbeatRuns).set({
+        runnerProfileJson: sql`${heartbeatRuns.runnerProfileJson} - 'remoteWorkspace'`,
+      }).where(eq(heartbeatRuns.id, ids.run));
+      if (kind === "historical_listener_stale_block") await issueService(db).update(ids.issue, { status: "blocked", actorUserId: "operator" });
+      if (kind === "historical_listener_unknown_event") await db.insert(heartbeatRunEvents).values({ companyId: ids.company, runId: ids.run,
+        agentId: ids.agent, seq: 100, eventType: "future.provider.activity", sourceEventId: "source-provider-event" });
+      if (kind === "historical_listener_budget") await db.update(nativeRunFinalizations).set({ attempt: 3 }).where(eq(nativeRunFinalizations.runId, ids.run));
+      if (kind === "historical_listener_checkpoint") await db.update(heartbeatRuns).set({ runnerProfileJson: {
+        ...((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, ids.run)))[0]!.runnerProfileJson), sessionCheckpoint: { providerSessionId: "already-started" },
+      } }).where(eq(heartbeatRuns.id, ids.run));
+      if (kind === "historical_listener_resumed") await db.update(heartbeatRuns).set({ sessionIdBefore: "prior-session" }).where(eq(heartbeatRuns.id, ids.run));
+      if (kind === "historical_listener_prior_session") await db.insert(heartbeatRuns).values({ id: randomUUID(), companyId: ids.company,
+        agentId: ids.agent, status: "succeeded", nativeSessionId: sessionId });
+      if (kind === "historical_listener_other_message") await db.update(nativeRunFinalizations).set({ failureDetail: {
+        message: "provider reported local_runtime_api_listener_unavailable", originalFailureCode: "native_session_interrupted", recoveryMode: "ambiguous_state",
+        providerSessionEstablished: false, providerEventsExist: false, checkpointExists: false,
+      } }).where(eq(nativeRunFinalizations.runId, ids.run));
+      if (kind === "historical_listener_cancelled") await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, ids.issue));
+      if (kind === "historical_listener_provider_event") await appendHeartbeatRunEvent(db, { companyId: ids.company, runId: ids.run,
+        agentId: ids.agent, eventType: "turn.started", stream: "system" });
+      const interactionWake = ["historical_listener_late_interaction", "historical_listener_interaction_paused", "historical_listener_own_question"].includes(kind);
+      const interactionId = randomUUID();
+      if (interactionWake) {
+        const priorRunId = randomUUID();
+        await db.insert(heartbeatRuns).values({ id: priorRunId, companyId: ids.company, agentId: ids.agent,
+          nativeIssueId: ids.issue, status: "succeeded", contextSnapshot: { issueId: ids.issue } });
+        await db.insert(issueThreadInteractions).values({ id: interactionId, companyId: ids.company, issueId: ids.issue,
+          kind: "request_confirmation", status: "pending", sourceRunId: kind === "historical_listener_own_question" ? ids.run : priorRunId,
+          payload: { version: 1, prompt: "An obsolete question from earlier work." } });
+        if (kind === "historical_listener_interaction_paused") await issueService(db).update(ids.issue, { status: "backlog" });
+        if (kind === "historical_listener_own_question") await appendHeartbeatRunEvent(db, { companyId: ids.company, runId: ids.run,
+          agentId: ids.agent, eventType: "tool.execution.started", stream: "system",
+          payload: { name: "request_confirmation", executionId: "source-question", transport: "process" } });
+      }
       if (scenario === "unknown_action") await appendHeartbeatRunEvent(db, { companyId: ids.company, runId: ids.run,
         agentId: ids.agent, eventType: "tool.execution.started", stream: "system",
         payload: { name: "send_email", executionId: "unconfirmed-external-action", transport: "process" } });
+      if (kind === "historical_listener_later_candidate") {
+        const sourceNumber = BigInt(`0x${ids.run.replaceAll("-", "")}`);
+        const parked = Array.from({ length: 100 }, (_, index) => {
+          const hex = (sourceNumber - BigInt(index + 1)).toString(16).padStart(32, "0");
+          return { runId: `${hex.slice(0,8)}-${hex.slice(8,12)}-${hex.slice(12,16)}-${hex.slice(16,20)}-${hex.slice(20)}`,
+            issueId: randomUUID(), sessionId: randomUUID() };
+        });
+        await db.insert(issues).values(parked.map(row => ({ id: row.issueId, companyId: ids.company,
+          title: "Preserve unavailable source", status: "in_progress", assigneeAgentId: ids.agent })));
+        await db.insert(heartbeatRuns).values(parked.map(row => ({ id: row.runId, companyId: ids.company, agentId: ids.agent,
+          nativeIssueId: row.issueId, nativeSessionId: row.sessionId, runtimeMode: "native", status: "failed",
+          errorCode: "native_session_interrupted", error: "local_runtime_api_listener_unavailable", runnerProfileJson: {
+            nativeExecutionInput: { binding: { runId: row.runId, companyId: ids.company, agentId: ids.agent, issueId: row.issueId },
+              session: { normalizedSessionId: row.sessionId } },
+          } })));
+        await db.insert(nativeRunFinalizations).values(parked.map(row => ({ runId: row.runId, companyId: ids.company, issueId: row.issueId,
+          phase: "terminal_failure", attempt: 1, failureCode: "native_session_interrupted", failureDetail: {
+            message: "local_runtime_api_listener_unavailable", originalFailureCode: "native_session_interrupted", recoveryMode: "ambiguous_state",
+            providerSessionEstablished: false, providerEventsExist: false, checkpointExists: false,
+          } })));
+        expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scanned: 100, scheduled: 0 });
+        expect(await executionControlSchedule(db).replacement()).not.toBeNull();
+        invalidateNativeReplacementDiscovery(db);
+      }
+      if (kind === "historical_listener_lock_timeout") {
+        let release!: () => void, announce!: () => void;
+        const locked = new Promise<void>(resolve => { announce = resolve; });
+        const released = new Promise<void>(resolve => { release = resolve; });
+        const holder = db.transaction(async tx => {
+          await tx.select({ id: issues.id }).from(issues).where(eq(issues.id, ids.issue)).for("update");
+          announce(); await released;
+        });
+        await locked;
+        try {
+          await expect(reconcileSafeNativeReplacements(db)).rejects.toThrow();
+          expect(await executionControlSchedule(db).replacement()).not.toBeNull();
+        } finally { release(); await holder; }
+      }
       let replacement;
       if (scenario === "deletion_race" || scenario === "controller_claim_race") {
         let release!: () => void, announce!: () => void;
@@ -1783,12 +1907,102 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem
           expect(blocked).toBe(true);
         } finally { release(); await deletion; replacement = await pending; }
       } else replacement = await reconcileSafeNativeReplacements(db);
-      expect(replacement).toMatchObject({ scheduled: scenario === "saved" ? 1 : 0 });
+      expect(replacement).toMatchObject({ scheduled: shouldSchedule ? 1 : 0 });
       const successors = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, ids.run));
-      expect(successors).toHaveLength(scenario === "saved" ? 1 : 0);
-      if (scenario === "saved") expect(successors[0]).toMatchObject({ status: "scheduled_retry",
+      expect(successors).toHaveLength(shouldSchedule ? 1 : 0);
+      if (shouldSchedule) expect(successors[0]).toMatchObject({ status: "scheduled_retry",
         contextSnapshot: { retryOfRunId: ids.run, forceFreshSession: true } });
-      else expect(await fs.readFile(path.join(prepared.remoteRoot, "saved.txt"), "utf8"))
+      if (kind === "historical_listener_later_candidate") {
+        for (let pass = 0; pass < 3 && await executionControlSchedule(db).replacement() !== null; pass++) {
+          expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 });
+        }
+        expect(await executionControlSchedule(db).replacement()).toBeNull();
+      }
+      if (kind === "historical_listener_guard") expect(await getExecutionBlocker(db, ids.company, ids.issue)).toBeNull();
+      if (["historical_listener_provider_event", "historical_listener_checkpoint", "historical_listener_resumed", "historical_listener_other_message",
+        "historical_listener_cancelled", "historical_listener_prior_session", "historical_listener_unknown_event", "historical_listener_budget", "historical_listener_stale_block", "historical_listener_bad_preservation", "historical_listener_dirty", "historical_listener_unconfirmed_stop", "historical_listener_absent", "historical_listener_late_proof"].includes(kind)) {
+        expect(await executionControlSchedule(db).replacement()).toBeNull();
+      }
+      if (interactionWake) {
+        expect(await executionControlSchedule(db).replacement()).toBeNull();
+        let wakes = 0;
+        const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.executionReplacement, event => {
+          if (event === "settled") { wakes++; invalidateNativeReplacementDiscovery(db); }
+        });
+        try {
+          await issueThreadInteractionService(db).skipInteraction({ id: ids.issue, companyId: ids.company,
+            status: kind === "historical_listener_interaction_paused" ? "backlog" : "blocked" }, interactionId, {}, { userId: "local-board" });
+          expect(wakes).toBeGreaterThan(0);
+          expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: kind === "historical_listener_late_interaction" ? 1 : 0 });
+          expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 });
+          expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, ids.run)))
+            .toHaveLength(kind === "historical_listener_late_interaction" ? 1 : 0);
+          expect(await executionControlSchedule(db).replacement()).toBeNull();
+        } finally { unsubscribe(); }
+      }
+      if (kind === "historical_listener_late_proof") {
+        let wakes = 0;
+        const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.executionReplacement, event => {
+          if (event === "settled") { wakes++; invalidateNativeReplacementDiscovery(db); }
+        });
+        try {
+          const saved = await saveRemoteWorkspace({ db, storage, workspace: await workspace(reserved.id), runId: ids.run,
+            lease: await lease(), target: first, assertOwned: async () => undefined });
+          expect(wakes).toBeGreaterThan(0);
+          expect(await executionControlSchedule(db).replacement()).toBeNull();
+          const priorWakes = wakes, active = await lease(), pluginId = randomUUID();
+          await db.update(environmentLeases).set({ metadata: { ...active.metadata, driver: "sandbox", pluginId,
+            provider: "daytona", sandboxProviderPlugin: true } }).where(eq(environmentLeases.id, ids.lease));
+          const runtime = environmentRuntimeService(db, { pluginWorkerManager: {
+            isRunning: () => true, getWorker: () => ({ supportedMethods: ["environmentReleaseLease"] }),
+            call: async (_id: string, method: string) => {
+              expect(method).toBe("environmentReleaseLease");
+              return { providerLeaseId: active.providerLeaseId, state: "destroyed" };
+            },
+          } as never });
+          await runtime.releaseRunLeases(ids.run, "failed");
+          expect(wakes).toBeGreaterThan(priorWakes);
+          expect(await executionControlSchedule(db).replacement()).not.toBeNull();
+          expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 1 });
+          expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 });
+          expect(await executionControlSchedule(db).replacement()).toBeNull();
+          const restored = await createSavedWorkspaceLoader(db, storage)({ companyId: ids.company, workspaceId: reserved.id, snapshotId: saved.snapshotId });
+          try { expect(await fs.readFile(path.join(restored.root, "saved.txt"), "utf8")).toBe("work survives a failed provider session\n"); }
+          finally { await restored.cleanup(); }
+        } finally { unsubscribe(); }
+      }
+      if (kind === "historical_listener_bad_preservation") {
+        expect(await getExecutionBlocker(db, ids.company, ids.issue)).toMatchObject({ cause: "native_continuation_requires_reconciliation" });
+
+      }
+      if (processAcceptance) {
+        expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 });
+        const { completeHistoricalListenerSuccessor } = await import("../__tests__/helpers/historical-listener-continuation.js");
+        await completeHistoricalListenerSuccessor({ db, runId: successors[0]!.id, root: path.join(root, "successor-runtime"),
+          prepareWorkspace: async () => {
+            const leaseId = randomUUID(), runId = successors[0]!.id;
+            const [nextLease] = await db.insert(environmentLeases).values({ id: leaseId, companyId: ids.company,
+              environmentId: ids.environment, heartbeatRunId: runId, issueId: ids.issue, provider: "daytona",
+              providerLeaseId: `allocation-${leaseId}`, status: "active" }).returning();
+            const next = await reserveRemoteWorkspace({ db, storage, companyId: ids.company, issueId: ids.issue,
+              runId, agentId: ids.agent, environmentId: ids.environment, projectId: null, projectWorkspaceId: null,
+              existingWorkspaceId: reserved.id, source: { kind: "empty" } });
+            const nextTarget = { ...await target("historical-successor"), leaseId, sandboxId: nextLease.providerLeaseId! };
+            const nextPrepared = await prepareRemoteWorkspace({ db, storage, workspace: next, runId,
+              lease: nextLease as EnvironmentLease, target: nextTarget });
+            return { cwd: nextPrepared.remoteRoot, finalize: async () => {
+              const { workspaceOperationService } = await import("./workspace-operations.js");
+              const recorder = workspaceOperationService(db).createRecorder({ companyId: ids.company, issueId: ids.issue, heartbeatRunId: runId });
+              const saved = await recorder.recordOperation({ phase: "workspace_finalize", cwd: nextPrepared.remoteRoot, run: async () => {
+                await saveRemoteWorkspace({ db, storage, workspace: nextPrepared.workspace, runId, lease: nextPrepared.lease,
+                  target: nextTarget, assertOwned: async () => undefined });
+                return { status: "succeeded", exitCode: 0 };
+              } });
+              expect(saved.status).toBe("succeeded");
+            } };
+          } });
+      }
+      if (!shouldSchedule) expect(await fs.readFile(path.join(prepared.remoteRoot, "saved.txt"), "utf8"))
         .toBe("work survives a failed provider session\n");
       if (scenario === "unknown_action") {
         const [decision] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, ids.run));

server/src/services/remote-workspaces.ts

@@ -1,3 +1,4 @@
+import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
 import { workspaceAllocationFiles } from "./remote-workspace-allocation.js";
 import { readNativeWorkspaceMigration } from "./native-runtime/native-workspace-migration.js";
 import { createHash, randomUUID } from "node:crypto";
@@ -763,7 +764,10 @@ export async function saveRemoteWorkspace(input: {
         eq(executionWorkspaceSnapshots.executionWorkspaceId, workspace.id), eq(executionWorkspaceSnapshots.status, "uploaded"))).returning({ id: executionWorkspaceSnapshots.id });
       if (!committed) throw new Error("remote_workspace_snapshot_publication_changed");
       await notifyRemoteWorkspaceSnapshots(tx, `workspace:${workspace.id}`);
-      if (boundary === "final") await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`);
+      if (boundary === "final") {
+        await notifyTerminalWorkspaceWork(tx, `workspace:${workspace.id}`);
+        await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionReplacement);
+      }
       const receipt = object(workspace.metadata?.remoteWorkspaceSnapshotRepositories);
       const priorRepositories = receipt.snapshotId === current.snapshotId && Array.isArray(receipt.repositories)
         ? remoteWorkspaceSourceSchema.parse({ kind: "empty", repositories: receipt.repositories }).repositories ?? [] : [];

server/src/services/run-dispatch-notifications.ts

@@ -12,6 +12,7 @@ export async function notifyRunDispatch(transaction: object, scope?: RunDispatch
   await notifyIssueRecovery(transaction, scope ? { companyId: scope.companyId } : undefined);
   const hint = scope ? JSON.stringify({ companyId: scope.companyId }) : "*";
   await notifyDeliveryWork(transaction, DELIVERY_QUEUES.nativeProcessSessions, hint);
+  await notifyDeliveryWork(transaction, DELIVERY_QUEUES.executionReplacement, hint);
   await notifyQueuedRunDispatch(transaction, scope);
 }
 

server/src/services/task-interruption-presentation.test.ts

@@ -80,6 +80,24 @@ describe("unadmitted chat Retry presentation", () => {
   });
 });
 
+// The native coordinator resumes the same run itself after an attach failure.
+// A manual Retry offered meanwhile would race that automatic resume.
+it("shows a pending native same-run resume instead of a competing Retry", () => {
+  const now = Date.parse("2026-11-06T15:25:50.000Z");
+  const failed = { ...run, runtimeMode: "native" as const, errorCode: "native_session_interrupted", resultJson: null };
+  const resumeAt = new Date("2026-11-06T15:26:18.687Z");
+  expect(projectTaskInterruption(issue, { run: { ...failed, nativeResumeAt: resumeAt }, agent, paused: false, approval: false }, now))
+    .toMatchObject({ state: "retry_scheduled", canRetry: false, nextRetryAt: resumeAt.toISOString() });
+  // A due resume is being dispatched; it is still not the user's turn.
+  expect(projectTaskInterruption(issue, { run: { ...failed, nativeResumeAt: new Date(now - 30_000) }, agent, paused: false, approval: false }, now))
+    .toMatchObject({ state: "working", canRetry: false });
+  // A resume that never ran, or no resume at all, leaves exactly one Retry.
+  expect(projectTaskInterruption(issue, { run: { ...failed, nativeResumeAt: new Date(now - 10 * 60_000) }, agent, paused: false, approval: false }, now))
+    .toMatchObject({ state: "failed", canRetry: true });
+  expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false }, now))
+    .toMatchObject({ state: "failed", canRetry: true });
+});
+
 it("explains a persisted Retry wait without exposing the saved payload", () => {
   const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
   const wait = { id: "retry", requestedAt: new Date(), sourceRunId: failed.id,

server/src/services/task-interruption-presentation.ts

@@ -1,6 +1,6 @@
 import { and, desc, eq, inArray, isNull, sql, type SQL } from "drizzle-orm";
 import { agents, agentWakeupRequests, approvals, chatActions, companies, heartbeatRuns, issueApprovals, issueRecoveryActions,
-  issues, issueTreeHoldMembers, issueTreeHolds, projects, type Db } from "@paperclipai/db";
+  issues, issueTreeHoldMembers, issueTreeHolds, nativeRunFinalizations, projects, type Db } from "@paperclipai/db";
 import { unionAll } from "drizzle-orm/pg-core";
 import { HEARTBEAT_RUN_STATUSES, readProviderFailure, type TaskInterruptionPresentation } from "@paperclipai/shared";
 import { captureTaskInterruptionFailure } from "../sentry.js";
@@ -26,11 +26,17 @@ import { AI_CONNECTION_BUSY_RETRY_REASON, AI_CONNECTION_POOL_WAIT_RETRY_REASON,
 const resourceWaitRetryReasons = new Set<string>([WORKSPACE_BUSY_RETRY_REASON, AI_CONNECTION_BUSY_RETRY_REASON,
   AI_CONNECTION_POOL_WAIT_RETRY_REASON, RUNNER_REVISION_WAIT_RETRY_REASON]);
 
+// A due same-run resume dispatches within seconds; past this, offer Retry.
+const NATIVE_RESUME_DISPATCH_GRACE_MS = 2 * 60_000;
+
 type ProjectionIssue = Pick<typeof issues.$inferSelect, "id" | "companyId" | "status" | "assigneeAgentId" | "projectId">;
 type ProjectionRun = Pick<typeof heartbeatRuns.$inferSelect,
   "id" | "status" | "scheduledRetryAt" | "errorCode" | "livenessState" | "resultJson"> &
   Partial<Pick<typeof heartbeatRuns.$inferSelect, "agentId" | "error" | "companyId" | "nativeIssueId" | "runtimeMode" | "startedAt"
-    | "scheduledRetryReason">>;
+    | "scheduledRetryReason">> & {
+  /** Due time of the native coordinator's own same-run resume, if one is pending. */
+  nativeResumeAt?: Date | null;
+};
 type ProjectionWait = { id: string; requestedAt: Date; sourceRunId?: string | null; executionWaitReason?: string | null;
   preparationStatus?: string | null; preparationRetryAt?: number | string | null };
 type ProjectionState = {
@@ -148,10 +154,16 @@ export function projectTaskInterruption(issue: ProjectionIssue, input: Projectio
   }
   if (run.status === "scheduled_retry" && resourceWaitRetryReasons.has(run.scheduledRetryReason ?? "") && !system)
     return { ...base, state: "working" };
-  const retryAt = system?.nextAttemptAt ?? run.scheduledRetryAt?.getTime();
+  // The native coordinator resumes this same run by itself. A manual Retry
+  // would race it, so show the resume until it is due and briefly while its
+  // dispatch runs. A resume that never ran falls through to one Retry.
+  const nativeResumeAt = run.nativeResumeAt?.getTime();
+  const retryAt = system?.nextAttemptAt ?? nativeResumeAt ?? run.scheduledRetryAt?.getTime();
   if (retryAt && retryAt > now) return { ...base, state: "retry_scheduled",
-    nextRetryAt: new Date(retryAt).toISOString(), canRetry: !preparationPending,
+    nextRetryAt: new Date(retryAt).toISOString(), canRetry: !preparationPending && !nativeResumeAt,
     ...(system?.nextAttemptAt && recoveryStep ? { recoveryStep } : {}) };
+  if (nativeResumeAt && !system && now - nativeResumeAt < NATIVE_RESUME_DISPATCH_GRACE_MS)
+    return { ...base, state: "working" };
   if (recoveryRunning) return { ...base, state: "working", ...(recoveryStep ? { recoveryStep } : {}) };
   if (retryWait?.sourceRunId === run.id && retryWait.executionWaitReason) {
     const reason = Object.hasOwn(retryWaitMessages, retryWait.executionWaitReason) ? retryWait.executionWaitReason : "execution_recovery";
@@ -203,6 +215,10 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
         or ${heartbeatRuns.errorCode} = 'workspace_validation_failed' then ${heartbeatRuns.error} else null end`,
       errorCode: heartbeatRuns.errorCode, livenessState: heartbeatRuns.livenessState, scheduledRetryAt: heartbeatRuns.scheduledRetryAt,
       scheduledRetryReason: heartbeatRuns.scheduledRetryReason,
+      nativeResumeAt: sql<Date | null>`(select ${nativeRunFinalizations.nextAttemptAt} from ${nativeRunFinalizations}
+        where ${nativeRunFinalizations.companyId} = ${heartbeatRuns.companyId} and ${nativeRunFinalizations.runId} = ${heartbeatRuns.id}
+          and ${heartbeatRuns.runtimeMode} = 'native' and ${heartbeatRuns.status} = 'failed'
+          and ${nativeRunFinalizations.resultId} is null and ${nativeRunFinalizations.phase} = 'retryable_failure')`.mapWith(value => value == null ? null : new Date(value)),
       // The run whose resultJson supplied providerFailure. A scheduled retry
       // shows its predecessor's reason, so redaction must use that run's registry.
       providerFailureRunId: sql<string>`case when ${heartbeatRuns.resultJson}->'providerFailure' is null

server/src/services/tool-access.ts

@@ -4530,6 +4530,13 @@ export function toolAccessService(
       let applicationNamespace = input.applicationName ?? input.name;
       const transport = input.transport;
       if (!transport) throw badRequest("Tool connection transport is required");
+      // This path writes only tool connections. Channel setup has its own
+      // services. Reject other purposes here, before any row is written, so the
+      // database purpose/transport check never turns them into a 500.
+      if (input.connectionPurpose !== undefined && input.connectionPurpose !== "tool")
+        throw unprocessable("Only tool connections can be created here");
+      if (transport === "chat_sdk" || transport === "voice")
+        throw unprocessable("Chat and voice connections cannot be created here");
       const config = normalizeGoogleSheetsConnectionConfig(
         input.config ?? input.transportConfig ?? {},
       );

server/src/services/workspace-recovery.ts

@@ -390,6 +390,7 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ
       const activity = await db.transaction(async tx => {
         await notifyNativeResultCommit(tx, input);
         await notifyNativeWorkspaceFinalization(tx, input);
+        await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionReplacement);
         await notifyNativeStatusReassessment(tx, input);
         await tx.insert(workspaceOperations).values({ companyId: input.companyId, issueId: input.issueId,
           heartbeatRunId: input.runId, phase: "workspace_finalize", status: "succeeded", finishedAt: new Date(),

tests/e2e/announcement-first-session.spec.ts

@@ -0,0 +1,105 @@
+import { test, expect, request as pwRequest, type APIRequestContext, type Page } from "@playwright/test";
+
+/**
+ * E2E (N-13): the in-app announcement well ("Introducing Connectors" upsell)
+ *
+ * stays hidden while a new company's onboarding first task is open, so it
+ * cannot cover the onboarding questions or the account menu in the user's
+ * first session. Placement (never covering the account menu once shown) is
+ * covered by announcement-placement.spec.ts.
+ *
+ * The public feed is mocked so the spec does not depend on what the live feed
+ * is publishing; the card mirrors the real 2026-09-14-connectors entry.
+ */
+
+const PORT = Number(process.env.PAPERCLIP_E2E_PORT ?? 3199);
+const BASE_URL = `http://127.0.0.1:${PORT}`;
+const ANNOUNCEMENT_ID = "e2e-n13-connectors";
+const SETTLE_WAIT_MS = 4_500; // ANNOUNCEMENT_SETTLE_MS (3 s) plus slack.
+const ONE_PIXEL_PNG = Buffer.from(
+  "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=",
+  "base64",
+);
+
+async function createCompany(board: APIRequestContext, name: string) {
+  const res = await board.post(`${BASE_URL}/api/companies`, { data: { name: `${name}-${Date.now()}` } });
+  if (!res.ok()) throw new Error(`POST /api/companies → ${res.status()}: ${await res.text()}`);
+  const company = await res.json();
+  // A company without agents lands on "Create your first agent"; add one so the normal shell renders.
+  const agentRes = await board.post(`${BASE_URL}/api/companies/${company.id}/agents`, {
+    data: { name: "Chief of staff", role: "ceo", adapterType: "process", adapterConfig: { command: "true" } },
+  });
+  if (!agentRes.ok()) throw new Error(`POST agents → ${agentRes.status()}: ${await agentRes.text()}`);
+  return { id: company.id as string, prefix: (company.issuePrefix ?? company.prefix) as string };
+}
+
+async function createIssue(board: APIRequestContext, companyId: string, data: Record<string, unknown>) {
+  const res = await board.post(`${BASE_URL}/api/companies/${companyId}/issues`, { data });
+  if (!res.ok()) throw new Error(`POST issues → ${res.status()}: ${await res.text()}`);
+  return res.json() as Promise<{ id: string; identifier: string; originKind: string | null }>;
+}
+
+async function mockAnnouncementFeed(page: Page) {
+  await page.route("**/api/announcements/current", (route) => route.fulfill({ json: {
+    id: ANNOUNCEMENT_ID,
+    eyebrow: "Introducing Connectors",
+    title: "Give your agents the access they need",
+    description: "Connect Gmail, GitHub, Notion, and more. Choose which agents get access, which accounts they use, and which actions they can take.",
+    image: { path: `assets/${"a".repeat(64)}.png`, alt: "Paperclip agents with their own sets of connected apps." },
+    primaryAction: { kind: "route", label: "Explore connectors", path: "/apps" },
+  } }));
+  await page.route(`**/api/announcements/${ANNOUNCEMENT_ID}/image`, (route) =>
+    route.fulfill({ contentType: "image/png", body: ONE_PIXEL_PNG }));
+  await page.route(`**/api/announcements/${ANNOUNCEMENT_ID}/dismiss`, (route) => route.fulfill({ status: 204 }));
+  await page.route("**/api/workspace-preferences/me", (route) => route.fulfill({ json: {
+    userId: "local-board", coordinatorWorkspaceEnabled: false, updatedAt: null,
+  } }));
+  await page.addInitScript(() => {
+    for (const key of Object.keys(window.localStorage)) {
+      if (key.includes("announcement")) window.localStorage.removeItem(key);
+    }
+  });
+}
+
+test.describe("Announcement well (N-13)", () => {
+  let board: APIRequestContext;
+  const companyIds: string[] = [];
+
+  test.beforeAll(async () => {
+    board = await pwRequest.newContext({ baseURL: BASE_URL });
+  });
+
+  test.afterAll(async () => {
+    for (const id of companyIds) await board.delete(`${BASE_URL}/api/companies/${id}`).catch(() => {});
+    await board.dispose();
+  });
+
+  test.beforeEach(async ({ page }) => {
+    await page.setViewportSize({ width: 1440, height: 900 });
+    await mockAnnouncementFeed(page);
+  });
+
+  test("a brand-new company with its onboarding task open does not show the upsell", async ({ page }) => {
+    const company = await createCompany(board, "E2E-N13-Onboarding");
+    companyIds.push(company.id);
+    const firstTask = await createIssue(board, company.id, {
+      title: "Plan our launch", status: "todo", onboardingFirstTask: true,
+    });
+    expect(firstTask.originKind).toBe("onboarding_first_task");
+
+    await page.goto(`/${company.prefix}/issues/${firstTask.identifier}`);
+    await expect(page.getByRole("button", { name: "Open account menu" })).toBeVisible();
+    await page.waitForTimeout(SETTLE_WAIT_MS);
+    await page.screenshot({ path: test.info().outputPath("onboarding-task.png") });
+    const well = page.getByRole("complementary", { name: "Paperclip announcements" });
+    await expect(well).toHaveCount(0);
+
+    // Positive control on the same page: once onboarding finishes, the same
+    // mocked announcement renders without a reload. This proves the absence
+    // above comes from the onboarding gate, not from a broken feed mock.
+    const finish = await board.patch(`${BASE_URL}/api/issues/${firstTask.id}`, { data: { status: "done" } });
+    if (!finish.ok()) throw new Error(`PATCH issue → ${finish.status()}: ${await finish.text()}`);
+    await expect(well).toBeVisible({ timeout: 20_000 });
+    await expect(well.getByRole("link", { name: "Explore connectors" })).toBeVisible();
+  });
+});

ui/src/components/AnnouncementWell.onboarding.test.tsx

@@ -0,0 +1,76 @@
+// @vitest-environment jsdom
+import { act } from "react";
+import { createRoot, type Root } from "react-dom/client";
+import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+import type { HealthStatus } from "@/api/health";
+import { AnnouncementWell } from "./AnnouncementWell";
+import { announcementPreview } from "@/lib/announcement-preview";
+
+// N-13: the connectors announcement must not appear in a new user's first
+// session, i.e. while the company's onboarding first task is still open.
+const state = vi.hoisted(() => ({ list: vi.fn() }));
+vi.mock("@/api/companies-query", () => ({ useAccountIdentity: () => ({ userId: "alice", settled: true }) }));
+vi.mock("@/context/CompanyContext", () => ({ useCompany: () => ({ selectedCompanyId: "company", loading: false }) }));
+vi.mock("@/context/DialogContext", () => ({ useDialogState: () => ({ onboardingOpen: false }) }));
+vi.mock("@/context/ToastContext", () => ({ useOptionalToastActions: () => null, useOptionalToastState: () => [] }));
+vi.mock("@/api/issues", () => ({ issuesApi: { list: state.list } }));
+vi.mock("@/hooks/useAnnouncement", () => ({ useAnnouncement: (options: { enabled: boolean }) => ({
+  announcement: options.enabled ? announcementPreview : null, dismiss: vi.fn(),
+}) }));
+vi.mock("./AnnouncementCard", () => ({ AnnouncementCard: () => <p>Announcement fixture</p> }));
+(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true;
+
+const firstTask = (status: string) => ({ id: "issue-1", companyId: "company", originKind: "onboarding_first_task", status });
+
+describe("announcement well during onboarding", () => {
+  let container: HTMLDivElement;
+  let root: Root;
+  const health = { deploymentMode: "authenticated" } as HealthStatus;
+  const visible = () => Boolean(container.querySelector('[aria-label="Paperclip announcements"]'));
+  let client: QueryClient;
+  const flush = async () => { await act(async () => { await new Promise((resolve) => setTimeout(resolve, 0)); }); };
+  const render = async () => {
+    client = new QueryClient({ defaultOptions: { queries: { retry: false } } });
+    await act(async () => root.render(<QueryClientProvider client={client}><AnnouncementWell health={health} /></QueryClientProvider>));
+    await act(async () => { await new Promise((resolve) => setTimeout(resolve, 0)); });
+  };
+  beforeEach(() => {
+    state.list.mockReset();
+    container = document.createElement("div"); document.body.append(container); root = createRoot(container);
+  });
+  afterEach(async () => { await act(async () => root.unmount()); container.remove(); });
+
+  it("stays hidden while the onboarding first task is open", async () => {
+    state.list.mockResolvedValue([firstTask("todo")]);
+    await render();
+    expect(visible()).toBe(false);
+  });
+
+  it("shows once the onboarding first task is finished, or when the company never had one", async () => {
+    state.list.mockResolvedValue([firstTask("done")]);
+    await render();
+    expect(visible()).toBe(true);
+    await act(async () => root.unmount()); root = createRoot(container);
+    state.list.mockResolvedValue([]);
+    await render();
+    expect(visible()).toBe(true);
+  });
+
+  it("shows without a reload once the open onboarding task finishes and company issues refresh", async () => {
+    state.list.mockResolvedValue([firstTask("todo")]);
+    await render();
+    expect(visible()).toBe(false);
+    state.list.mockResolvedValue([firstTask("done")]);
+    // Issue live updates invalidate the company's issue queries by this prefix.
+    await act(async () => { await client.invalidateQueries({ queryKey: ["issues", "company"] }); });
+    await flush();
+    expect(visible()).toBe(true);
+  });
+
+  it("stays hidden when onboarding state cannot be read", async () => {
+    state.list.mockRejectedValue(new Error("offline"));
+    await render();
+    expect(visible()).toBe(false);
+  });
+});

ui/src/components/AnnouncementWell.test.tsx

@@ -1,6 +1,7 @@
 // @vitest-environment jsdom
 import { act } from "react";
 import { createRoot, type Root } from "react-dom/client";
+import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
 import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
 import type { HealthStatus } from "@/api/health";
 import { AnnouncementWell } from "./AnnouncementWell";
@@ -18,6 +19,7 @@ vi.mock("@/hooks/useAnnouncement", () => ({ useAnnouncement: (options: { enabled
   state.hook(options);
   return { announcement: options.enabled ? announcementPreview : null, dismiss: state.dismiss };
 } }));
+vi.mock("@/api/issues", () => ({ issuesApi: { list: vi.fn().mockResolvedValue([]) } }));
 vi.mock("./AnnouncementCard", () => ({ AnnouncementCard: ({ onDismiss }: { onDismiss: () => void }) => <button onClick={onDismiss}>Dismiss fixture</button> }));
 (globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true;
 
@@ -25,12 +27,16 @@ describe("announcement placement gates", () => {
   let container: HTMLDivElement;
   let root: Root;
   let health: HealthStatus;
-  const render = async () => { await act(async () => root.render(<AnnouncementWell health={health} />)); };
+  let client: QueryClient;
+  const render = async () => { await act(async () => root.render(<QueryClientProvider client={client}><AnnouncementWell health={health} /></QueryClientProvider>)); };
   const visible = () => Boolean(container.querySelector('[aria-label="Paperclip announcements"]'));
   beforeEach(() => {
     Object.assign(state, { userId: "alice", settled: true, companyId: "company", loading: false, onboardingOpen: false, toasts: [] });
     vi.clearAllMocks();
     health = { deploymentMode: "authenticated" } as HealthStatus;
+    // Onboarding is finished (no open first task); onboarding gating is covered in AnnouncementWell.onboarding.test.tsx.
+    client = new QueryClient({ defaultOptions: { queries: { retry: false, staleTime: Infinity } } });
+    client.setQueryData(["issues", "company", "onboarding-first-task"], []);
     container = document.createElement("div"); document.body.append(container); root = createRoot(container);
   });
   afterEach(async () => { await act(async () => root.unmount()); container.remove(); });

ui/src/components/AnnouncementWell.tsx

@@ -1,10 +1,14 @@
 import { useEffect, useState } from "react";
+import { useQuery } from "@tanstack/react-query";
+import { ONBOARDING_FIRST_TASK_ORIGIN_KIND } from "@paperclipai/shared";
 import { useAccountIdentity } from "@/api/companies-query";
 import type { HealthStatus } from "@/api/health";
+import { issuesApi } from "@/api/issues";
 import { useCompany } from "@/context/CompanyContext";
 import { useDialogState } from "@/context/DialogContext";
 import { useOptionalToastActions, useOptionalToastState } from "@/context/ToastContext";
 import { useAnnouncement } from "@/hooks/useAnnouncement";
+import { queryKeys } from "@/lib/queryKeys";
 import { AnnouncementCard } from "./AnnouncementCard";
 
 // Includes the command palette, sheets and dialogs created outside DialogContext.
@@ -22,16 +26,36 @@ function useModalOpen(enabled: boolean) {
   return open;
 }
 
+const FINISHED_ONBOARDING_STATUSES = new Set(["done", "cancelled"]);
+
+/**
+ * A new user's first session is the onboarding first task. Announcements wait
+ * until that task is finished so they cannot cover its questions. Unknown
+ * state (loading or a failed read) keeps the optional card hidden.
+ */
+function useOnboardingInProgress(companyId: string | null) {
+  const query = useQuery({
+    queryKey: queryKeys.issues.onboardingFirstTask(companyId ?? "__none__"),
+    queryFn: ({ signal }) => issuesApi.list(companyId!, { originKind: ONBOARDING_FIRST_TASK_ORIGIN_KIND, limit: 1 }, { signal }),
+    enabled: Boolean(companyId),
+    staleTime: 60_000,
+  });
+  if (!companyId) return false;
+  if (!query.isSuccess) return true;
+  return query.data.some((issue) => issue.originKind === ONBOARDING_FIRST_TASK_ORIGIN_KIND && !FINISHED_ONBOARDING_STATUSES.has(issue.status));
+}
+
 export function AnnouncementWell({ health }: { health?: HealthStatus }) {
   const { userId: accountId, settled } = useAccountIdentity();
   const { selectedCompanyId, loading } = useCompany();
   const { onboardingOpen } = useDialogState();
   const toastActions = useOptionalToastActions();
   const toasts = useOptionalToastState();
+  const onboardingInProgress = useOnboardingInProgress(selectedCompanyId);
   const userId = health?.deploymentMode === "local_trusted" ? "local-board" : (settled ? accountId : null);
   const { announcement, dismiss } = useAnnouncement({
     userId, companyId: selectedCompanyId,
-    enabled: Boolean(userId && !loading && selectedCompanyId && !onboardingOpen),
+    enabled: Boolean(userId && !loading && selectedCompanyId && !onboardingOpen && !onboardingInProgress),
     onSaveFailure: (savedLocally) => toastActions?.pushToast({
       title: savedLocally ? "Dismissed in this browser" : "Dismissed for this visit",
       body: "Couldn’t save across devices. We’ll retry when you reconnect or return.",

ui/src/components/IssueMonitorBanner.test.tsx

@@ -263,13 +263,23 @@ describe("IssueMonitorBanner / IssueMonitorComposerStrip rendering", () => {
     } finally { flushSync(() => root.unmount()); }
   });
 
-  it("explains automatic workspace waiting without promising that a reply bypasses the lock", () => {
+  it("does not raise a pinned banner for a workspace wait the user cannot act on (N-14)", () => {
+    const issue = {
+      status: "todo", scheduledRetry: { status: "scheduled_retry", scheduledRetryReason: "workspace_busy", scheduledRetryAt: NOW.toISOString() },
+    } as Issue;
+    const root = createRoot(container);
+    flushSync(() => root.render(<IssueMonitorBanner issue={issue} onCheckNow={vi.fn()} />));
+    expect(container.textContent).toBe("");
+    flushSync(() => root.unmount());
+  });
+
+  it("keeps the workspace wait as a quiet composer status without promising that a reply bypasses the lock", () => {
     const issue = {
       status: "todo", scheduledRetry: { status: "scheduled_retry", scheduledRetryReason: "workspace_busy", scheduledRetryAt: NOW.toISOString() },
     } as Issue;
     const root = createRoot(container);
     flushSync(() => root.render(<><IssueMonitorBanner issue={issue} onCheckNow={vi.fn()} /><IssueMonitorComposerStrip issue={issue} onCheckNow={vi.fn()} /></>));
-    expect(container.textContent).toContain("Waiting for workspace");
+    expect(container.querySelector('[data-testid="issue-monitor-composer-strip"]')?.textContent).toContain("Waiting for workspace");
     expect(container.textContent).toContain("You can keep sending instructions while the agent waits.");
     expect(container.textContent).not.toContain("wakes the agent now");
     expect(container.querySelector("button")).toBeNull();

ui/src/components/IssueMonitorBanner.tsx

@@ -296,6 +296,10 @@ export function IssueMonitorBanner({
   const copy = useMonitorSurfaceCopy(issue, reviews.length, hasActiveRun);
   if (!copy) return null;
 
+  // A busy workspace is a transient system wait the user cannot act on. It stays
+  // a quiet status (composer strip, properties row), never a pinned notice.
+  if (copy.workspaceWait) return null;
+
   const canCancel = onCancelMonitor && deriveMonitorState(issue).source === "monitor";
 
   return (

ui/src/components/IssueScheduledRetryCard.test.tsx

@@ -121,11 +121,9 @@ function getRetryNowButton() {
 }
 
 describe("IssueScheduledRetryCard", () => {
-  it("shows workspace contention as an automatic wait without failure or retry controls", () => {
+  it("raises no conversation notice for workspace contention the user cannot act on (N-14)", () => {
     renderWithProviders(<IssueScheduledRetryCard issueId="issue-1" scheduledRetry={{ ...baseRetry, scheduledRetryReason: "workspace_busy" }} />);
-    expect(container.textContent).toContain("Waiting for workspace");
-    expect(container.textContent).toContain("Work starts automatically");
-    expect(container.textContent).not.toMatch(/failed|Retry|Attempt|Replaces run/);
+    expect(container.textContent).toBe("");
     expect(getRetryNowButton()).toBeNull();
   });
   it("surfaces model capacity and the automatic retry without a generic failure", () => {

ui/src/components/IssueScheduledRetryCard.tsx

@@ -7,7 +7,6 @@ import { formatRetryReason } from "@/lib/runRetryState";
 import type { IssueScheduledRetry } from "@paperclipai/shared";
 import { useRetryNowMutation, type RetryNowError } from "../hooks/useRetryNowMutation";
 import { Badge } from "@/components/ui/badge";
-import { InlineBanner } from "@/components/InlineBanner";
 import { describeError } from "@/api/errors";
 
 const MAX_TURN_CONTINUATION = "max_turns_continuation";
@@ -34,13 +33,9 @@ export function IssueScheduledRetryCard({
   if (!scheduledRetry || !issueId) return null;
   if (scheduledRetry.status !== "scheduled_retry") return null;
 
-  if (scheduledRetry.scheduledRetryReason === "workspace_busy") {
-    return (
-      <InlineBanner tone="info" icon={Clock} title="Waiting for workspace" className="mb-3">
-        Another task is using this workspace. Work starts automatically when it is available.
-      </InlineBanner>
-    );
-  }
+  // Workspace contention is a transient wait with nothing for the user to do.
+  // The composer strip and properties row carry the quiet status instead.
+  if (scheduledRetry.scheduledRetryReason === "workspace_busy") return null;
 
   const continuation = isContinuationReason(scheduledRetry.scheduledRetryReason);
   const dueAtIso = scheduledRetry.scheduledRetryAt

ui/src/components/IssueThreadInteractionCard.tsx

@@ -54,6 +54,7 @@ import { useCoordinatorWorkspaceEnabled } from "@/hooks/useCoordinatorWorkspaceE
 import { CoordinatorTaskCommitment } from "./coordinator/CoordinatorTaskCommitment";
 
 import { DecisionCardFooter } from "./DecisionCardFooter";
+import { ToolActionAuditHashCopy } from "./ToolActionAuditHashCopy";
 
 const OTHER_ANSWER_ID = "__paperclip_other__";
 
@@ -1847,12 +1848,12 @@ function RequestToolActionCard({
       {isPending ? inspectableArguments ? (
         <details className="mt-3">
           <summary className="cursor-pointer text-xs font-medium text-muted-foreground">
-            Arguments and audit hash
+            Arguments
           </summary>
           <pre className="mt-2 max-h-40 overflow-auto whitespace-pre-wrap break-words font-mono text-xs text-foreground">
             {payload.argumentsSummaryJson}
           </pre>
-          <p className="mt-1 break-all font-mono text-xs text-muted-foreground">{payload.argumentsHash}</p>
+          <ToolActionAuditHashCopy hash={payload.argumentsHash} />
         </details>
       ) : <p className="mt-3 text-xs text-muted-foreground">Request arguments are unavailable.</p> : null}
       {isPending ? (

ui/src/components/OnboardingWizard.test.tsx

@@ -2138,6 +2138,30 @@ describe("OnboardingWizard restore-gate (stale localStorage across accounts)", (
       return { root, queryClient };
     }
 
+    // N-16: the gateway is an advanced route, not the next thing after the key.
+    // It sat directly under the API key field, inside the key card; it belongs
+    // in the step's Advanced disclosure, hidden until that is opened.
+    it.each([["Claude API", "claude_local", /Claude.*API/], ["Codex API", "codex_local", /OpenAI.*API/]] as const)("keeps Use a custom gateway in the closed Advanced disclosure on the %s key step", async (_path, adapterType, source) => {
+      const handles = await openStep4({ adapterType, runner: "paperclip", credentialMode: "api" });
+      try {
+        await pickSource(source);
+        const keyField = document.body.querySelector('input[type="password"]') as HTMLInputElement;
+        expect(keyField, "the API key field should be showing").toBeTruthy();
+        const gateway = () => [...document.body.querySelectorAll("button")].find(entry => entry.textContent?.trim() === "Use a custom gateway");
+        expect(gateway(), "the custom gateway control should still exist").toBeTruthy();
+        const advanced = gateway()!.closest("details");
+        expect(advanced, "Use a custom gateway should live inside the Advanced disclosure").toBeTruthy();
+        expect(advanced!.querySelector(":scope > summary")?.textContent?.trim()).toBe("Advanced");
+        expect(advanced!.open, "Advanced should start closed").toBe(false);
+        expect(advanced!.contains(keyField), "the key field stays outside Advanced").toBe(false);
+        await act(async () => { advanced!.open = true; });
+        await flushReact();
+        await act(async () => gateway()!.click());
+        await flushReact();
+        expect(document.querySelector('input[aria-label="Provider URL"]'), "the gateway dialog should still open from Advanced").toBeTruthy();
+      } finally { await act(async () => handles.root.unmount()); }
+    });
+
     it.each([["successful refresh", false], ["failed refresh and retry", true]] as const)("saves a custom gateway without probing, then explicitly tests and hires with its selected model: %s", async (_scenario, failRefresh) => {
       const gatewayAccounts: import("@paperclipai/shared").AiConnectionList = {
         currentUserId: "user-1", canManageConnections: true, connections: [],

ui/src/components/OnboardingWizard.tsx

@@ -3157,15 +3157,6 @@ function OnboardingWizardInner({
                           onChange={setModel} open={modelOpen} onOpenChange={setModelOpen}
                           allowDefault={false} required groupByProvider={false}
                           creatable={connectionModels.models.length === 0} presentation="native" />}
-                        {createdCompanyId && (adapterType === "codex_local" || adapterType === "claude_local") && <OnboardingGatewaySetup
-                          companyId={createdCompanyId} agentId={setupAgentId ?? undefined} environmentId={resolvedLoginEnvironmentId ?? undefined}
-                          adapterType={adapterType} disabled={loading || adapterEnvLoading}
-                          onComplete={binding => {
-                            setGatewayConnection({ companyId: createdCompanyId, adapterType, binding });
-                            setApiKey("");
-                            setModel("");
-                            setError(null);
-                          }} />}
                         {(hasGatewaySelection || connectionModels) && <Button type="button" variant="ghost" disabled={loading || adapterEnvLoading} onClick={() => {
                           setGatewayConnection(undefined);
                           setSelectedSavedKey({ companyId: createdCompanyId!, envKey: apiKeyEnvKeyFor(adapterType), id: "" });
@@ -3280,6 +3271,22 @@ function OnboardingWizardInner({
 
                   {sourceSelected && ["codex_local", "claude_local", "opencode_local", "pi_local"].includes(adapterType) && <details className="space-y-3">
                     <summary className="cursor-pointer text-sm text-muted-foreground">Advanced</summary>
+                    {/* The gateway is an alternative route, not the next
+                        thing after the key, so it waits here rather than
+                        sitting under the API key field (N-16). Same gates as
+                        the key card it used to live in. */}
+                    {connectCardLive && connectCardMounted && !connectProgress && credentialMode === "api" && createdCompanyId && (adapterType === "codex_local" || adapterType === "claude_local") && <OnboardingGatewaySetup
+                      companyId={createdCompanyId} agentId={setupAgentId ?? undefined} environmentId={resolvedLoginEnvironmentId ?? undefined}
+                      adapterType={adapterType} disabled={loading || adapterEnvLoading}
+                      // Ghost padding pulled back so the label lines up with
+                      // the Advanced summary above it.
+                      className="-ml-3"
+                      onComplete={binding => {
+                        setGatewayConnection({ companyId: createdCompanyId, adapterType, binding });
+                        setApiKey("");
+                        setModel("");
+                        setError(null);
+                      }} />}
                     <AgentRunnerSelect value={runner} disabled={loading || adapterEnvLoading}
                       pending={runnerDiscovery.isFetching} error={runnerDiscovery.error ? describeError(runnerDiscovery.error).body : undefined} onRetry={() => { void runnerDiscovery.refetch(); }}
                       defaultRunner={runnerDiscovery.data?.find(item => item.type === adapterType)?.defaultRunner}

ui/src/components/ToolActionAuditHashCopy.test.tsx

@@ -0,0 +1,46 @@
+// @vitest-environment jsdom
+
+import { act } from "react";
+import { createRoot, type Root } from "react-dom/client";
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+
+const copyTextToClipboard = vi.fn<(text: string) => Promise<void>>();
+vi.mock("@/lib/clipboard", () => ({ copyTextToClipboard: (text: string) => copyTextToClipboard(text) }));
+
+import { ToolActionAuditHashCopy } from "./ToolActionAuditHashCopy";
+
+const hash = "8ca5312877235a4c0f6e9b1d2a7c3e5f9081b2c4d6e8fa1b3c5d7e9f0a2b4c6d";
+
+describe("ToolActionAuditHashCopy", () => {
+  let container: HTMLDivElement;
+  let root: Root;
+
+  beforeEach(() => {
+    copyTextToClipboard.mockReset();
+    container = document.createElement("div");
+    document.body.appendChild(container);
+    root = createRoot(container);
+  });
+
+  afterEach(() => {
+    act(() => root.unmount());
+    container.remove();
+  });
+
+  it("copies the hash without showing it", async () => {
+    copyTextToClipboard.mockResolvedValue(undefined);
+    await act(async () => root.render(<ToolActionAuditHashCopy hash={hash} />));
+    expect(container.textContent).toBe("Copy audit hash");
+    await act(async () => container.querySelector("button")!.click());
+    expect(copyTextToClipboard).toHaveBeenCalledWith(hash);
+    expect(container.textContent).toBe("Copied");
+  });
+
+  it("shows the hash as selectable text when the clipboard is blocked", async () => {
+    copyTextToClipboard.mockRejectedValue(new Error("blocked"));
+    await act(async () => root.render(<ToolActionAuditHashCopy hash={hash} />));
+    await act(async () => container.querySelector("button")!.click());
+    expect(container.textContent).toContain("Couldn’t copy");
+    expect(container.textContent).toContain(hash);
+  });
+});

ui/src/components/ToolActionAuditHashCopy.tsx

@@ -0,0 +1,42 @@
+import { useState } from "react";
+import { Check, Copy, X } from "lucide-react";
+import { useCopyAction } from "@/lib/use-copy-action";
+import { Button } from "./ui/button";
+
+/**
+ * The reviewed-arguments hash identifies exactly what an approval covers, which
+ * matters for audits but means nothing to the person approving. Keep it off the
+ * card and reachable through a copy action instead of printing the raw digest.
+ * When the clipboard is blocked, show the hash as selectable text so the audit
+ * value never becomes unreachable.
+ */
+export function ToolActionAuditHashCopy({ hash }: { hash: string }) {
+  const { status, copy } = useCopyAction();
+  const [copyFailed, setCopyFailed] = useState(false);
+  const label = status === "copied" ? "Copied" : status === "failed" ? "Couldn’t copy" : "Copy audit hash";
+  const Icon = status === "copied" ? Check : status === "failed" ? X : Copy;
+  return (
+    <div className="mt-1 space-y-1">
+      <Button
+        type="button"
+        size="xs"
+        variant="ghost"
+        className="text-muted-foreground"
+        title="Copy the hash that identifies these exact arguments"
+        onClick={() => {
+          void copy(hash).then(result => {
+            if (result === "failed") setCopyFailed(true);
+          });
+        }}
+      >
+        <Icon aria-hidden="true" />
+        {label}
+      </Button>
+      {copyFailed ? (
+        <p className="select-all break-all font-mono text-xs text-muted-foreground" aria-label="Audit hash">
+          {hash}
+        </p>
+      ) : null}
+    </div>
+  );
+}

ui/src/components/onboarding/OnboardingGatewaySetup.tsx

@@ -10,6 +10,7 @@ export function OnboardingGatewaySetup({
   environmentId,
   adapterType,
   disabled,
+  className,
   onComplete,
 }: {
   companyId: string;
@@ -17,11 +18,12 @@ export function OnboardingGatewaySetup({
   environmentId?: string;
   adapterType: "codex_local" | "claude_local";
   disabled?: boolean;
+  className?: string;
   onComplete: (binding: Exclude<AiConnectionBinding, { mode: "responsible_user" }>) => void;
 }) {
   const [open, setOpen] = useState(false);
   return <>
-    <Button type="button" variant="ghost" disabled={disabled} onClick={() => setOpen(true)}>
+    <Button type="button" variant="ghost" className={className} disabled={disabled} onClick={() => setOpen(true)}>
       Use a custom gateway
     </Button>
     <Dialog open={open} onOpenChange={setOpen}>

ui/src/components/task-chat/TaskChatCompactInteractionCard.tsx

@@ -30,6 +30,7 @@ import type {
 } from "@paperclipai/adapter-utils";
 import { InteractionPreparationNotice } from "@/components/InteractionPreparationNotice";
 import { IssueThreadInteractionCard } from "@/components/IssueThreadInteractionCard";
+import { ToolActionAuditHashCopy } from "@/components/ToolActionAuditHashCopy";
 import { ConnectionIntentInteractionBody } from "@/features/connections/ConnectionIntentInteractionBody";
 import { MarkdownBody } from "@/components/MarkdownBody";
 import type { MentionOption } from "@/components/MarkdownEditor";
@@ -573,9 +574,7 @@ function ReceiptDisclosure({
             <pre className="mt-2 max-h-40 overflow-auto whitespace-pre-wrap break-words font-mono text-xs">
               {interaction.payload.toolAction.argumentsSummaryJson}
             </pre>
-            <p className="mt-1 break-all font-mono text-xs text-muted-foreground">
-              {interaction.payload.toolAction.argumentsHash}
-            </p>
+            <ToolActionAuditHashCopy hash={interaction.payload.toolAction.argumentsHash} />
           </div>
         ) : null}
         {interaction.payload.secretProposal ? (
@@ -934,14 +933,12 @@ function ConfirmationCard({
           </MarkdownBody>
           <details>
             <summary className="cursor-pointer text-xs font-medium text-muted-foreground">
-              Arguments and audit hash
+              Arguments
             </summary>
             <pre className="mt-2 max-h-40 overflow-auto whitespace-pre-wrap break-words font-mono text-xs text-foreground">
               {interaction.payload.toolAction.argumentsSummaryJson}
             </pre>
-            <p className="mt-1 break-all font-mono text-xs text-muted-foreground">
-              {interaction.payload.toolAction.argumentsHash}
-            </p>
+            <ToolActionAuditHashCopy hash={interaction.payload.toolAction.argumentsHash} />
           </details>
         </div>
       ) : null}

ui/src/components/task-chat/TaskChatInteractionCard.test.tsx

@@ -3,7 +3,7 @@
 import { act, forwardRef, useImperativeHandle, useState, type ForwardedRef } from "react";
 import { flushSync } from "react-dom";
 import { createRoot, type Root } from "react-dom/client";
-import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
+import { afterEach, beforeEach, describe, expect, it, onTestFinished, vi } from "vitest";
 
 vi.mock("@/hooks/useCoordinatorWorkspaceEnabled", () => ({
   useCoordinatorWorkspaceEnabled: () => ({ enabled: false, loaded: true }),
@@ -148,7 +148,8 @@ describe("TaskChatInteractionCard", () => {
     expect(container.querySelector("pre")?.textContent).toBe(interaction.payload.toolAction!.argumentsSummaryJson);
     expect(details.textContent).toContain("November 8, 2030");
     expect(details.textContent).toContain("[redacted]");
-    expect(details.textContent).toContain(interaction.payload.toolAction!.argumentsHash);
+    expect(details.textContent).not.toContain(interaction.payload.toolAction!.argumentsHash);
+    expect(details.textContent).toContain("Copy audit hash");
     await act(async () => details.querySelector("summary")!.click());
     expect(details.open).toBe(true);
     expect(accept).not.toHaveBeenCalled();
@@ -160,6 +161,64 @@ describe("TaskChatInteractionCard", () => {
     expect(reject).not.toHaveBeenCalled();
   });
 
+  it("keeps raw tool ids and the audit hash out of the tool approval card text", async () => {
+    // Shape of the GitHub create_repository approval from the Nov 5 staging run.
+    const rawToolName = "mcp.app-gallery-github-42e51e3f-9377-447b-ad45-333beda8a50a-b873f178:create-repository";
+    const argumentsHash = "8ca5312877235a4c0f6e9b1d2a7c3e5f9081b2c4d6e8fa1b3c5d7e9f0a2b4c6d";
+    const base = structuredClone(pendingToolActionLongProposalInteraction);
+    const interaction: RequestConfirmationInteraction = {
+      ...base,
+      title: "Approve tool action",
+      summary: "This action needs your approval before it runs.",
+      payload: {
+        ...base.payload,
+        prompt: "Approve create_repository?",
+        detailsMarkdown: `Tool: \`${rawToolName}\`\nRisk: \`write\``,
+        target: { type: "custom", key: "tool-action:2f92393c-c044-4367-8b6e-b81e2e732165", revisionId: argumentsHash, label: rawToolName },
+        toolAction: {
+          ...base.payload.toolAction!,
+          toolName: rawToolName,
+          toolDisplayName: "create_repository",
+          appDisplayName: "GitHub",
+          previewMarkdown: "create_repository · Name: brainrot-storefront-prototype · Private: true",
+          argumentsSummaryJson: JSON.stringify({ name: "brainrot-storefront-prototype", private: true }),
+          argumentsHash,
+        },
+      },
+    };
+    const writeText = vi.fn().mockResolvedValue(undefined);
+    const originalClipboard = Object.getOwnPropertyDescriptor(navigator, "clipboard");
+    Object.defineProperty(navigator, "clipboard", { configurable: true, value: { writeText } });
+    onTestFinished(() => {
+      if (originalClipboard) Object.defineProperty(navigator, "clipboard", originalClipboard);
+      else delete (navigator as { clipboard?: unknown }).clipboard;
+    });
+    function ReviewSurface() {
+      const [reviewing, setReviewing] = useState(false);
+      return <TaskChatInteractionCard item={interactionItem(interaction)}
+        presentation={reviewing ? "takeover" : "timeline"}
+        onReviewRequest={() => setReviewing(true)}
+        onAcceptInteraction={vi.fn()} onRejectInteraction={vi.fn()} />;
+    }
+    flushSync(() => root.render(<TooltipProvider><ThemeProvider><ReviewSurface /></ThemeProvider></TooltipProvider>));
+    const hashLike = /[0-9a-f]{12,}/i;
+    expect(container.textContent).not.toMatch(hashLike);
+    await act(async () => {
+      Array.from(container.querySelectorAll("button")).find(button => button.textContent === "Review request")!.click();
+    });
+    const details = container.querySelector("details")!;
+    await act(async () => details.querySelector("summary")!.click());
+    expect(details.open).toBe(true);
+    expect(container.textContent).toContain("brainrot-storefront-prototype");
+    expect(container.textContent).not.toMatch(hashLike);
+    expect(container.textContent).not.toContain("app-gallery");
+    // The audit hash stays reachable for anyone who needs it, through a copy action.
+    const copy = Array.from(container.querySelectorAll("button")).find(button => button.textContent === "Copy audit hash");
+    expect(copy).toBeDefined();
+    await act(async () => copy!.click());
+    expect(writeText).toHaveBeenCalledWith(argumentsHash);
+  });
+
   it("does not render a pending confirmation in the timeline", () => {
     flushSync(() => {
       root.render(

ui/src/lib/queryKeys.ts

@@ -311,6 +311,8 @@ export const queryKeys = {
     recoveryActions: (id: string, actionId: string | null, runId: string | null) =>
       ["issues", id, "recovery-actions", actionId, runId] as const,
     list: (companyId: string) => ["issues", companyId] as const,
+    onboardingFirstTask: (companyId: string) =>
+      ["issues", companyId, "onboarding-first-task"] as const,
     mentionPool: (companyId: string) =>
       ["issues", companyId, "mention-pool"] as const,
     search: (