fix(daytona): keep a runner sandbox alive through the controller reconnect grace (#19754)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote agent run executes in a Daytona sandbox and keeps working
while the Paperclip server restarts
> - The server reattaches that runner within a 7-minute reconnect grace
> - Daytona stops an idle sandbox after its `autoStopInterval`, and it
counts only SDK and lifecycle calls as activity
> - With the server down, nothing touches the sandbox, so a 5-minute
auto-stop kills the runner before the server returns
> - This pull request makes the host tell the provider its reconnect
grace, so Daytona never auto-stops a runner sandbox inside that grace
> - The benefit is that a controller outage of up to 7 minutes no longer
turns a healthy remote run into a blocked task

## Linked Issues or Issue Description

Refs #19199 (the same floor for the warm runner idle window). Refs
#19741 (follow-up delivery through restarts; no overlap).

**What happened?**
A restart-chaos wave on Daytona killed the server for 4m51s while a
per-turn remote run was active. The environment used `autoStopInterval:
5`. The last sandbox activity was at 06:09:16. The server came back at
06:14:19 and then crashed again. At 06:15:21 recovery called the
sandbox. The plugin found it not started and started it again. The
runner identity marker was still on disk, but its process was gone:
`native.remote_runner_recovery_failed {stage: process_verification,
reason: process_not_running}`, then
`runner_remote_process_identity_unavailable`. The task went from done to
blocked. An earlier wave showed the same failure after a 4m45s outage.

**Expected behavior**
A sandbox that holds a live remote runner survives any controller outage
up to the reconnect grace. The server reattaches the runner and the run
continues.

**Steps to reproduce**
1. Create a Daytona environment with `autoStopInterval: 5` and a
`paperclip_runner` agent.
2. Start a remote run and kill the server for more than 5 minutes (less
than 7).
3. Restart the server. Recovery reports `process_not_running` and the
task becomes blocked.

## What Changed

- `server`: add `resolveRunnerReconnectGraceMsForRun`. It returns
`REMOTE_RUNNER_RECONNECT_GRACE_MS` for `paperclip_runner` on a sandbox
environment.
- `server`: forward `runnerReconnectGraceMs` to the provider worker on
lease acquisition and on the recovery reattach resume. It uses the same
provider-only path as `warmRunnerIdleTimeoutMs`. It is not lease
identity, lease metadata, or saved environment config.
- Daytona plugin: the effective auto-stop is at least the largest host
floor (warm idle window or reconnect grace) plus 5 minutes. With the
current grace this is 12 minutes. The plugin applies it at creation and
raises an older sandbox's shorter interval on resume (bounded, best
effort).
- Daytona plugin: keep the forwarded grace out of the normalized config.
Update the `autoStopInterval` manifest description.
- `doc/execution-semantics.md`: document the reconnect-grace floor next
to the warm floor.

## Verification

- Red/green Daytona plugin tests: `cd
packages/plugins/sandbox-providers/daytona && npx vitest run --config
vitest.config.ts`. The new cases failed before the fix (for example
`expected 120000 to be >= 720000`, and no `setAutostopInterval` call on
resume). All 605 tests pass after the fix.
- Server tests: `cd server && npx vitest run
src/__tests__/environment-run-orchestrator.test.ts
src/__tests__/environment-runtime.test.ts`. 247 pass. They prove
acquisition forwards the grace only for `paperclip_runner` on a sandbox,
that the reattach resume forwards it, and that it does not reach lease
metadata or saved config.
- `pnpm typecheck` for the plugin and `tsc --noEmit` for the server are
clean.
- `pnpm verify` asks for broad CI coverage because a doc changed.
Required CI supplies it.
- Evidence from the kept wave database: lease metadata shows
`autoStopInterval: 5`, `reuseLease: false`, `runnerLifecycleMode:
per_turn`. The run had no events between 06:09:16 and 06:14:19.

## Risks

- Cost: a runner sandbox that leaks (no cleanup) now auto-stops after 12
minutes instead of a shorter configured value. Cleanup still stops or
destroys released sandboxes. Explicit `0` and longer values are
unchanged. Auto-archive and auto-delete are unchanged.
- I did not inspect the sandbox state in Daytona directly. Cleanup
destroyed it before this investigation. The proof is the timing, the
lease config, and the `process_not_running` result after a restart that
kept the disk.
- No live Daytona replay ran.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edit, code execution) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

packages/plugins/sandbox-providers/daytona/src/manifest.ts

@@ -160,7 +160,7 @@ const manifest: PaperclipPluginManifestV1 = {
           autoStopInterval: {
             type: "number",
             description:
-              "Daytona auto-stop interval in minutes. `0` disables auto-stop. Defaults to 15 when unset. A reusable sandbox that keeps a warm runner never auto-stops before the warm idle window plus 5 minutes.",
+              "Daytona auto-stop interval in minutes. `0` disables auto-stop. Defaults to 15 when unset. A reusable sandbox that keeps a warm runner never auto-stops before the warm idle window plus 5 minutes, and a sandbox running a Paperclip runner never auto-stops before the host's runner reconnect grace plus 5 minutes.",
             default: 15,
           },
           autoArchiveInterval: {

packages/plugins/sandbox-providers/daytona/src/plugin.test.ts

@@ -1045,6 +1045,57 @@ describe("Daytona sandbox provider plugin", () => {
       }
     });
 
+    // RC-54: a per-turn, non-reusable sandbox running a remote runner must not
+    // auto-stop during a controller outage the host still treats as
+    // reattachable. The host forwards its reconnect grace for runner leases.
+    it.each([
+      [{ reuseLease: false, runnerLifecycleMode: "per_turn", autoStopInterval: 5 }],
+      [{ reuseLease: false, autoStopInterval: 2 }],
+      [{ reuseLease: true, runnerLifecycleMode: "per_turn", autoStopInterval: 5 }],
+      [{ reuseLease: false }],
+    ])("keeps a runner sandbox alive through the host reconnect grace plus a margin (%o)", async (config) => {
+      const autoStopMs = (await acquire({ ...config, runnerReconnectGraceMs: 420_000 })) * 60_000;
+      expect(autoStopMs).toBeGreaterThanOrEqual(420_000 + marginMs);
+    });
+
+    it("combines the reconnect grace floor with the warm floor and explicit settings", async () => {
+      expect(await acquire({ reuseLease: true, warmRunnerIdleTimeoutMs: 900_000, runnerReconnectGraceMs: 420_000 }))
+        .toBe(Math.ceil(900_000 / 60_000) + WARM_IDLE_AUTO_STOP_MARGIN_MINUTES);
+      expect(await acquire({ reuseLease: false, runnerReconnectGraceMs: 420_000, autoStopInterval: 60 })).toBe(60);
+      expect(await acquire({ reuseLease: false, runnerReconnectGraceMs: 420_000, autoStopInterval: 0 })).toBe(0);
+      // Without a host-forwarded grace (not a runner lease), the configured value stands.
+      expect(await acquire({ reuseLease: false, autoStopInterval: 5 })).toBe(5);
+    });
+
+    it("raises the auto-stop of an existing runner sandbox on recovery resume", async () => {
+      // The stopped sandbox from the outage restarts on resume and keeps its
+      // original 5-minute auto-stop unless the provider raises it.
+      process.env.DAYTONA_API_KEY = "host-key";
+      const sandbox = createMockSandbox({ id: "sandbox-runner", state: "stopped" });
+      (sandbox as { autoStopInterval: number }).autoStopInterval = 5;
+      sandbox.process.executeCommand.mockImplementation(async (command: string) => ({ exitCode: 0,
+        result: command.startsWith("cat ") ? JSON.stringify({ token: "sentinel-token" }) : "bash" }));
+      mockGet.mockResolvedValue(sandbox);
+      await plugin.definition.onEnvironmentResumeLease!({
+        driverKey: "daytona", companyId: "company-1", environmentId: "env-1", providerLeaseId: sandbox.id,
+        config: { reuseLease: false, runnerLifecycleMode: "per_turn", autoStopInterval: 5, runnerReconnectGraceMs: 420_000 },
+        leaseMetadata: { workspaceSentinel: {
+          path: "/home/daytona/paperclip-workspace/.paperclip-runtime/reusable-sandbox-lease.json",
+          token: "sentinel-token", result: "written" } },
+      });
+      expect(sandbox.setAutostopInterval).toHaveBeenCalledTimes(1);
+      const [minutes] = sandbox.setAutostopInterval.mock.calls[0] as [number];
+      expect(minutes * 60_000).toBeGreaterThanOrEqual(420_000 + marginMs);
+    });
+
+    it("keeps the host reconnect grace out of the saved provider config", async () => {
+      const result = await plugin.definition.onEnvironmentValidateConfig!({ driverKey: "daytona",
+        config: { apiKey: "key", image: "node:20", runnerReconnectGraceMs: 420_000, warmRunnerIdleTimeoutMs: 900_000, reuseLease: true } });
+      expect(result.ok).toBe(true);
+      expect(result.normalizedConfig).not.toHaveProperty("runnerReconnectGraceMs");
+      expect(result.normalizedConfig).not.toHaveProperty("warmIdleTimeoutMs");
+    });
+
     it("does not change auto-stop on resume of a per-turn reusable sandbox", async () => {
       const sandbox = await resume({ reuseLease: true, runnerLifecycleMode: "per_turn" });
       expect(sandbox.setAutostopInterval).not.toHaveBeenCalled();

packages/plugins/sandbox-providers/daytona/src/plugin.ts

@@ -200,6 +200,13 @@ interface DaytonaDriverConfig {
    * normalized provider config; it only raises the effective auto-stop.
    */
   warmIdleTimeoutMs: number | null;
+  /**
+   * Host-owned remote runner reconnect grace (ms): how long the host still
+   * reattaches a live remote runner after a controller outage. Null when the
+   * lease hosts no remote runner. Never persisted in the normalized provider
+   * config; it only raises the effective auto-stop.
+   */
+  runnerReconnectGraceMs: number | null;
 }
 
 type WorkspaceSentinelResult = {
@@ -256,17 +263,37 @@ const DEFAULT_AUTO_STOP_INTERVAL_MINUTES = 15;
 export const DEFAULT_WARM_RUNNER_IDLE_TIMEOUT_MS = 300_000;
 export const WARM_IDLE_AUTO_STOP_MARGIN_MINUTES = 5;
 
+// A remote runner keeps working while its controller is offline, but Daytona
+// counts only SDK and lifecycle events as activity. With no controller there are
+// none, so an auto-stop shorter than the host's reconnect grace stops the
+// sandbox (and kills the runner) while the host would still reattach it. The
+// host forwards its grace for runner leases; the provider backstop then fires
+// only after the grace plus this margin. Idle sandboxes still stop.
+
+/**
+ * Host-owned floors (ms) the provider auto-stop must outlive: the warm runner
+ * idle window and the remote runner reconnect grace.
+ */
+function hostAutoStopFloorsMs(
+  config: Pick<DaytonaDriverConfig, "warmIdleTimeoutMs" | "runnerReconnectGraceMs">,
+): number[] {
+  return [config.warmIdleTimeoutMs, config.runnerReconnectGraceMs]
+    .filter((value): value is number => value != null && value > 0);
+}
+
 /**
  * Effective Daytona auto-stop (minutes) for a sandbox. An explicit 0 (disabled)
- * is preserved. A warm reusable lease never auto-stops before the host's warm
- * idle window plus {@link WARM_IDLE_AUTO_STOP_MARGIN_MINUTES}.
+ * is preserved. A sandbox never auto-stops before a host-owned window (the warm
+ * idle window of a warm reusable lease, or the remote runner reconnect grace)
+ * plus {@link WARM_IDLE_AUTO_STOP_MARGIN_MINUTES}.
  */
 export function effectiveAutoStopIntervalMinutes(
-  config: Pick<DaytonaDriverConfig, "autoStopInterval" | "warmIdleTimeoutMs">,
+  config: Pick<DaytonaDriverConfig, "autoStopInterval" | "warmIdleTimeoutMs" | "runnerReconnectGraceMs">,
 ): number | null {
   const configured = config.autoStopInterval;
-  if (configured === 0 || config.warmIdleTimeoutMs == null) return configured;
-  const floor = Math.ceil(config.warmIdleTimeoutMs / 60_000) + WARM_IDLE_AUTO_STOP_MARGIN_MINUTES;
+  const floorsMs = hostAutoStopFloorsMs(config);
+  if (configured === 0 || floorsMs.length === 0) return configured;
+  const floor = Math.ceil(Math.max(...floorsMs) / 60_000) + WARM_IDLE_AUTO_STOP_MARGIN_MINUTES;
   return Math.max(configured ?? DEFAULT_AUTO_STOP_INTERVAL_MINUTES, floor);
 }
 
@@ -347,6 +374,11 @@ function parseOptionalNumber(value: unknown): number | null {
   return Number.isFinite(parsed) ? parsed : null;
 }
 
+function parsePositiveInteger(value: unknown): number | null {
+  const parsed = parseOptionalInteger(value);
+  return parsed != null && parsed > 0 ? parsed : null;
+}
+
 function parseDriverConfig(raw: Record<string, unknown>): DaytonaDriverConfig {
   const timeoutMs = Number(raw.timeoutMs ?? DEFAULT_DAYTONA_OPERATION_TIMEOUT_MS);
   const livenessTimeoutMs = Number(raw.livenessTimeoutMs ?? DEFAULT_LIVENESS_TIMEOUT_MS);
@@ -370,6 +402,7 @@ function parseDriverConfig(raw: Record<string, unknown>): DaytonaDriverConfig {
     reuseLease: raw.reuseLease === true,
     archiveOnRelease: raw.archiveOnRelease === true,
     warmIdleTimeoutMs: parseWarmIdleTimeoutMs(raw),
+    runnerReconnectGraceMs: parsePositiveInteger(raw.runnerReconnectGraceMs),
   };
 }
 
@@ -2744,8 +2777,10 @@ const plugin = definePlugin(trackDaytonaHooks({
       return { ok: false, errors };
     }
 
-    // The warm idle window is host-owned; keep it out of the saved provider config.
-    const { warmIdleTimeoutMs: _warmIdleTimeoutMs, ...normalizedConfig } = config;
+    // The warm idle window and reconnect grace are host-owned; keep them out of
+    // the saved provider config.
+    const { warmIdleTimeoutMs: _warmIdleTimeoutMs, runnerReconnectGraceMs: _runnerReconnectGraceMs,
+      ...normalizedConfig } = config;
     return {
       ok: true,
       normalizedConfig,
@@ -3028,13 +3063,14 @@ const plugin = definePlugin(trackDaytonaHooks({
         }
         const resumedFromState = sandbox.state ?? null;
         await ensureSandboxStarted(sandbox, toTimeoutSeconds(config.timeoutMs), scope);
-        // A sandbox created before the warm floor (or under a shorter warm
-        // window) keeps its old auto-stop. Raise it on resume so the provider
-        // backstop cannot race the host's warm idle window. Best effort: the
-        // host's bounded resume retry still covers a refused resume.
+        // A sandbox created before a host floor (or under a shorter warm window
+        // or reconnect grace) keeps its old auto-stop. Raise it on resume so the
+        // provider backstop cannot race the host's warm idle window or a later
+        // controller outage. Best effort: the host's bounded resume retry still
+        // covers a refused resume.
         const autoStopInterval = effectiveAutoStopIntervalMinutes(config);
         const currentAutoStop = sandbox.autoStopInterval;
-        if (autoStopInterval != null && autoStopInterval > 0 && config.warmIdleTimeoutMs != null
+        if (autoStopInterval != null && autoStopInterval > 0 && hostAutoStopFloorsMs(config).length > 0
           && typeof currentAutoStop === "number" && currentAutoStop > 0 && currentAutoStop < autoStopInterval) {
           await withLivenessTimeout("sandbox.setAutostopInterval",
             Math.max(1, Math.min(config.livenessTimeoutMs > 0 ? config.livenessTimeoutMs : AUTO_STOP_UPDATE_TIMEOUT_MS, AUTO_STOP_UPDATE_TIMEOUT_MS)),