fix(daytona): keep a runner sandbox alive through the controller reconnect grace (#19754)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote agent run executes in a Daytona sandbox and keeps working
while the Paperclip server restarts
> - The server reattaches that runner within a 7-minute reconnect grace
> - Daytona stops an idle sandbox after its `autoStopInterval`, and it
counts only SDK and lifecycle calls as activity
> - With the server down, nothing touches the sandbox, so a 5-minute
auto-stop kills the runner before the server returns
> - This pull request makes the host tell the provider its reconnect
grace, so Daytona never auto-stops a runner sandbox inside that grace
> - The benefit is that a controller outage of up to 7 minutes no longer
turns a healthy remote run into a blocked task

## Linked Issues or Issue Description

Refs #19199 (the same floor for the warm runner idle window). Refs
#19741 (follow-up delivery through restarts; no overlap).

**What happened?**
A restart-chaos wave on Daytona killed the server for 4m51s while a
per-turn remote run was active. The environment used `autoStopInterval:
5`. The last sandbox activity was at 06:09:16. The server came back at
06:14:19 and then crashed again. At 06:15:21 recovery called the
sandbox. The plugin found it not started and started it again. The
runner identity marker was still on disk, but its process was gone:
`native.remote_runner_recovery_failed {stage: process_verification,
reason: process_not_running}`, then
`runner_remote_process_identity_unavailable`. The task went from done to
blocked. An earlier wave showed the same failure after a 4m45s outage.

**Expected behavior**
A sandbox that holds a live remote runner survives any controller outage
up to the reconnect grace. The server reattaches the runner and the run
continues.

**Steps to reproduce**
1. Create a Daytona environment with `autoStopInterval: 5` and a
`paperclip_runner` agent.
2. Start a remote run and kill the server for more than 5 minutes (less
than 7).
3. Restart the server. Recovery reports `process_not_running` and the
task becomes blocked.

## What Changed

- `server`: add `resolveRunnerReconnectGraceMsForRun`. It returns
`REMOTE_RUNNER_RECONNECT_GRACE_MS` for `paperclip_runner` on a sandbox
environment.
- `server`: forward `runnerReconnectGraceMs` to the provider worker on
lease acquisition and on the recovery reattach resume. It uses the same
provider-only path as `warmRunnerIdleTimeoutMs`. It is not lease
identity, lease metadata, or saved environment config.
- Daytona plugin: the effective auto-stop is at least the largest host
floor (warm idle window or reconnect grace) plus 5 minutes. With the
current grace this is 12 minutes. The plugin applies it at creation and
raises an older sandbox's shorter interval on resume (bounded, best
effort).
- Daytona plugin: keep the forwarded grace out of the normalized config.
Update the `autoStopInterval` manifest description.
- `doc/execution-semantics.md`: document the reconnect-grace floor next
to the warm floor.

## Verification

- Red/green Daytona plugin tests: `cd
packages/plugins/sandbox-providers/daytona && npx vitest run --config
vitest.config.ts`. The new cases failed before the fix (for example
`expected 120000 to be >= 720000`, and no `setAutostopInterval` call on
resume). All 605 tests pass after the fix.
- Server tests: `cd server && npx vitest run
src/__tests__/environment-run-orchestrator.test.ts
src/__tests__/environment-runtime.test.ts`. 247 pass. They prove
acquisition forwards the grace only for `paperclip_runner` on a sandbox,
that the reattach resume forwards it, and that it does not reach lease
metadata or saved config.
- `pnpm typecheck` for the plugin and `tsc --noEmit` for the server are
clean.
- `pnpm verify` asks for broad CI coverage because a doc changed.
Required CI supplies it.
- Evidence from the kept wave database: lease metadata shows
`autoStopInterval: 5`, `reuseLease: false`, `runnerLifecycleMode:
per_turn`. The run had no events between 06:09:16 and 06:14:19.

## Risks

- Cost: a runner sandbox that leaks (no cleanup) now auto-stops after 12
minutes instead of a shorter configured value. Cleanup still stops or
destroys released sandboxes. Explicit `0` and longer values are
unchanged. Auto-archive and auto-delete are unchanged.
- I did not inspect the sandbox state in Daytona directly. Cleanup
destroyed it before this investigation. The proof is the timing, the
lease config, and the `process_not_running` result after a restart that
kept the disk.
- No live Daytona replay ran.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edit, code execution) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

server/src/__tests__/environment-run-orchestrator.test.ts

@@ -69,6 +69,7 @@ import {
   environmentRunOrchestrator,
   EnvironmentRunError,
 } from "../services/environment-run-orchestrator.ts";
+import { REMOTE_RUNNER_RECONNECT_GRACE_MS } from "../services/native-runtime/native-runner-recovery-contract.js";
 import type { Environment, EnvironmentLease, ExecutionWorkspace } from "@paperclipai/shared";
 import type { RealizedExecutionWorkspace } from "../services/workspace-runtime.ts";
 import type { EnvironmentRuntimeService } from "../services/environment-runtime.ts";
@@ -868,6 +869,26 @@ describe("native runner lifecycle changes before lease acquisition", () => {
     });
     expect(acquireRunLease.mock.calls[0][0].environment.config.reuseLease).toBe(expectedReuse);
   });
+
+  // RC-54: a per-turn remote runner sandbox must outlive a controller outage
+  // the host still recovers, so acquisition forwards the reconnect grace.
+  it.each([
+    ["paperclip_runner", "sandbox", REMOTE_RUNNER_RECONNECT_GRACE_MS],
+    ["claude_local", "sandbox", undefined],
+    ["paperclip_runner", "local", undefined],
+  ] as const)("forwards the remote runner reconnect grace for %s on %s", async (adapterType, driver, expected) => {
+    const environment = { ...makeEnvironment(driver), config: driver === "sandbox"
+      ? { provider: "daytona", reuseLease: false, runnerLifecycleMode: "per_turn" } : {} };
+    mockGetEnvironment.mockResolvedValue(environment);
+    const acquireRunLease = vi.fn(async (input) => ({ environment: input.environment, lease: makeLease(), leaseContext: { executionWorkspaceId: null, executionWorkspaceMode: null } }));
+    const orchestrator = environmentRunOrchestrator({} as never, { environmentRuntime: makeMockRuntime({ acquireRunLease }) });
+    await orchestrator.acquireForRun({
+      companyId: "company-1", selectedEnvironmentId: "env-1", localEnvironmentId: "local", adapterType,
+      adapterConfig: {}, issueId: "task-1", heartbeatRunId: "run-1", agentId: "agent-1",
+      persistedExecutionWorkspace: null, executionWorkspaceSettings: null,
+    });
+    expect(acquireRunLease.mock.calls[0][0].runnerReconnectGraceMs).toBe(expected);
+  });
 });
 
 
@@ -917,8 +938,11 @@ describe("live remote runner lease reattachment", () => {
     const result = await environmentRunOrchestrator({} as never, { environmentRuntime: runtime }).acquireForRun(input);
     expect(mockGetLease).toHaveBeenCalledWith("lease-1");
     expect(runtime.acquireRunLease).not.toHaveBeenCalled();
+    // RC-54: the reattach resume carries the host reconnect grace, so the
+    // provider raises an older sandbox's shorter idle auto-stop.
     expect(runtime.resumeRunLease).toHaveBeenCalledWith({
       environment: result.environment, lease: savedLease, requestedCwd: "/remote/workspace",
+      runnerReconnectGraceMs: REMOTE_RUNNER_RECONNECT_GRACE_MS,
     });
     expect(result.lease).toEqual(savedLease);
     expect(result.leaseContext.executionWorkspaceId).toBe("ew-1");
@@ -933,7 +957,7 @@ describe("live remote runner lease reattachment", () => {
     });
     expect(runtime.resumeRunLease).toHaveBeenCalledWith({
       environment: result.environment, lease: savedLease, requestedCwd: "/remote/workspace",
-      warmRunnerIdleTimeoutMs: 1_800_000,
+      warmRunnerIdleTimeoutMs: 1_800_000, runnerReconnectGraceMs: REMOTE_RUNNER_RECONNECT_GRACE_MS,
     });
     expect(runtime.acquireRunLease).not.toHaveBeenCalled();
   });

server/src/__tests__/environment-runtime.test.ts

@@ -416,6 +416,29 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
     expect(acquired.lease.metadata).not.toHaveProperty("warmRunnerIdleTimeoutMs");
   });
 
+  it("forwards the run's remote runner reconnect grace to the provider worker without changing lease identity", async () => {
+    const pluginId = randomUUID();
+    const { companyId, environment: baseEnvironment, runId } = await seedEnvironment();
+    const config = { provider: "daytona", image: "node:20", timeoutMs: 1234, autoStopInterval: 5 };
+    await insertSandboxProviderPlugin(pluginId, { environmentDrivers: [{ driverKey: "daytona", kind: "sandbox_provider", displayName: "Daytona", configSchema: { type: "object" } }] });
+    await environmentService(db).update(baseEnvironment.id, { driver: "sandbox", config });
+    const environment: Environment = { ...baseEnvironment, driver: "sandbox", config };
+    const worker = createFreshPluginLeaseWorker(pluginId);
+    vi.mocked(worker.call).mockImplementation(async (_id, method) => {
+      if (method !== "environmentAcquireLease") throw new Error(`Unexpected ${method}`);
+      return { providerLeaseId: "runner-lease", metadata: { provider: "daytona", image: "node:20", remoteCwd: "/workspace" } };
+    });
+    const service = environmentRuntimeService(db, { pluginWorkerManager: worker });
+    const acquired = await service.acquireRunLease({ companyId, environment, configuredEnvironment: environment,
+      issueId: null, heartbeatRunId: runId, persistedExecutionWorkspace: null,
+      adapterType: "paperclip_runner", runnerReconnectGraceMs: 420_000 });
+    expect(worker.call).toHaveBeenCalledWith(pluginId, "environmentAcquireLease", expect.objectContaining({
+      config: expect.objectContaining({ autoStopInterval: 5, runnerReconnectGraceMs: 420_000 }),
+    }), expect.any(Number));
+    expect(acquired.lease.metadata).not.toHaveProperty("runnerReconnectGraceMs");
+    expect((await environmentService(db).getById(environment.id))?.config).not.toHaveProperty("runnerReconnectGraceMs");
+  });
+
   it.each([["paperclip_runner", false, false], ["paperclip_runner", true, false], ["paperclip_runner", true, undefined], ["claude_local", false, false], [null, false, false]] as const)("pins qualified snapshots only for native execution (%s, warm=%s, savedReuse=%s)", async (adapterType, warm, savedReuse) => {
     const pluginId = randomUUID();
     const { companyId, environment: baseEnvironment, runId } = await seedEnvironment();

server/src/__tests__/heartbeat-native-remote-failure-retention.test.ts

@@ -16,6 +16,7 @@ import { instanceSettingsService } from "../services/instance-settings.js";
 import { buildNativeExecutionInput } from "../services/native-runtime/native-execution-input.js";
 import { nativeRuntimeContextFixture } from "../services/native-runtime/runtime-context.test-fixture.js";
 import { registerNativeRunnerCheckpoint } from "../services/native-runtime/native-runner-checkpoint.js";
+import { REMOTE_RUNNER_RECONNECT_GRACE_MS } from "../services/native-runtime/native-runner-recovery-contract.js";
 import { describeEmbeddedPostgres, nativeRestartFixture } from "./helpers/native-restart.js";
 
 describeEmbeddedPostgres("heartbeat native remote preparation failure retention", () => {
@@ -169,6 +170,8 @@ describeEmbeddedPostgres("heartbeat native remote preparation failure retention"
           executionWorkspaceId: workspaceId, heartbeatRunId: f.runId, providerLeaseId,
           status: "active", leasePolicy: "ephemeral" }),
         requestedCwd: remoteCwd,
+        // The reattach resume tells the provider the reconnect grace (RC-54).
+        runnerReconnectGraceMs: REMOTE_RUNNER_RECONNECT_GRACE_MS,
       });
       expect(acquireAllocation).not.toHaveBeenCalled();
       expect(resumed).toHaveBeenCalledOnce();