From d14a3b88b188cf5eee0e44c826facead8221062b Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 15 Nov 2026 10:09:32 -0600 Subject: [PATCH] fix(native): release an idle warm session whose Runner is proven gone, and retry a failed idle close (#21782) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm agent session keeps its Runner alive in a sandbox between turns. The server closes idle sessions, and closes them again before it exits. > - When an idle close failed, the idle-timer path kept the session registered and never tried again, unless the sandbox lease was already inactive. > - The controller-restart path tried once more, but it accepted no proof that the session was already gone. It checked a destroyed sandbox only for a forced environment delete. > - In a release-swap test, one idle close failed with `runner_remote_process_identity_unavailable`. The session stayed registered until shutdown, the shutdown close failed again, and the sandbox was held until the host stopped it. > - This pull request treats a failed close as complete when the Runner is proven gone or the sandbox is stopped or destroyed, retries a failed idle close a bounded number of times, and logs which Runner check failed. > - The benefit is that a dead warm session is released, and the next failure names its cause. ## Linked Issues or Issue Description Refs #21627 **What happened?** In a zero-downtime release-swap test, a warm session's idle close failed 4 s after the server left new run starts. The failure was `runner_remote_process_identity_unavailable`. The idle-timer path kept the session registered with no retry. At SIGTERM, `closeIdleWarmNativeSessionsForRestart` tried again and failed the same way. The log showed "idle warm native sessions did not close" and the restart-chaos harness counted a warm-shutdown failure. The log did not say which Runner check failed, so the cause of the unreadable identity is not known. **Expected behavior** A close whose Runner is proven gone, or whose sandbox is stopped or destroyed, releases the session. A failed idle close is tried again a bounded number of times. The failure text names the Runner check stage and reason. **Steps to reproduce** 1. Keep a warm remote session idle. 2. Make its close fail with a `RemoteRunnerRecoveryError` (for example, `process_verification/process_not_running`). 3. Call `closeIdleWarmNativeSessionsForRestart()`. On master the result is `failed: 1`. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: - New `warmCloseFailureProvesOwnerGone`. It uses existing proofs only: the close error's Runner diagnostic is `process_verification/process_not_running`, or a lease row for the sandbox has a termination receipt (`hasRemoteTerminationReceipt`) with state `destroyed`, or the warm run's own lease row has a receipt with state `stopped`. `pending_cleanup` and a lease without a receipt prove nothing. A lease read error counts as no proof. - `closeIdleWarmNativeSessions` (controller restart, environment delete, idle sleep) uses this proof for every failed close, not only for a forced environment delete. A released Runner detach failure is unchanged. - The idle-timer path uses the same proof. Its earlier status-only rule from #21504 (any lease status other than `active` or `retained`) is removed, because `pending_cleanup` is written before the provider stop. Without proof, it re-arms the timer and tries again, every 5 s, at most 3 times (`WARM_IDLE_CLOSE_RETRY_MS`, `WARM_IDLE_CLOSE_MAX_RETRIES`). The log line records the attempt number. - `describeWarmCloseFailure` appends the Runner check as `[stage/reason/lifecycle]` when the error carries one. - `server/src/services/native-runtime/native-session-executor.test.ts`: three tests. - "an idle warm session whose close fails with runner_remote_process_identity_unavailable and whose Runner is proven gone is released at controller restart" - "names the Runner check in a warm close failure and keeps an unproven owner registered at controller restart" - "retries a failed idle warm close a bounded number of times" - The #21504 idle-path test is now "forgets a warm owner whose idle close failed only once its sandbox has a stop or destroy receipt". A stop or destroy receipt releases the owner. `pending_cleanup`, `released` without a receipt, and `retained` keep it. - "reads the warm owner's own lease rows at controller restart" (a stop receipt for this run releases the owner; `pending_cleanup`, no receipt, or another run's receipt do not) ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts`: 1519 passed, 7 skipped. - Red on master (with only the two exported constants added so the file compiles): all three new tests fail. The first fails with `"failed": 1`. - `npx tsc --noEmit -p server`: no errors. ## Risks - A session is released without a checkpoint only when its Runner is proven gone or its sandbox is stopped or destroyed. The last saved checkpoint stays the recovery source, which is the same as the forced environment delete path today. - Environment delete and idle sleep now also accept the proof. Before, they accepted only a destroyed receipt during a forced delete. `pending_cleanup` is written before the provider stop, so it never counts as proof here. - The idle retry adds at most 3 more close attempts per failed session, 5 s apart. - A failed idle close whose sandbox stop is still pending now keeps its owner until a receipt lands or a successor retires it. Before, the status alone released it. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code with tool use (shell, file edit) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../native-session-executor.test.ts | 136 +++++++++++++++++- .../native-runtime/native-session-executor.ts | 70 +++++---- 2 files changed, 170 insertions(+), 36 deletions(-) diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 8e463045596..c0011b7afc0 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -434,6 +434,8 @@ import { nativeSessionWorkspaceScope, reserveWarmNativeInstructionDirectory, closeIdleWarmNativeSessionsForRestart, + WARM_IDLE_CLOSE_MAX_RETRIES, + WARM_IDLE_CLOSE_RETRY_MS, readNativeSessionIdleWork, createGovernedWaitEventObservation, createRemoteRunnerProcessLauncher, @@ -11614,10 +11616,12 @@ describe("native warm session supervision", () => { }); it.each([ - { lease: "pending_cleanup", forgotten: true }, - { lease: "released", forgotten: true }, + { lease: "stopped-receipt", forgotten: true }, + { lease: "destroyed-receipt", forgotten: true }, + { lease: "pending_cleanup", forgotten: false }, + { lease: "released", forgotten: false }, { lease: "retained", forgotten: false }, - ])("forgets a warm owner whose idle close failed only once the host gave up its sandbox: $lease", async ({ lease, forgotten }) => { + ])("forgets a warm owner whose idle close failed only once its sandbox has a stop or destroy receipt: $lease", async ({ lease, forgotten }) => { const identity = `warm-idle-close-sandbox-${lease}`; const warmExecution = { ...execution, binding: { ...execution.binding, runId: identity, executionWorkspaceId: `${identity}-workspace` }, session: { ...execution.session, normalizedSessionId: identity, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 50 } }, @@ -11632,7 +11636,7 @@ describe("native warm session supervision", () => { }); const copy = (runId: string) => ({ runId, root: `/tmp/${identity}`, preparationKey: "same", hasChanges: async () => false, collectStopped: async () => undefined, retirementFailed: async () => undefined }); - let leaseRows: Array<{ status: string }> = []; + let leaseRows: Array> = []; const base = leaseDb(warmExecution); const db = { ...base, select: (fields?: unknown) => ({ from: (table: unknown) => table === environmentLeases ? { where: async () => leaseRows } : base.select(fields as never).from(table as never) }) } as unknown as Db; @@ -11641,7 +11645,14 @@ describe("native warm session supervision", () => { runnerExecutionTarget: { kind: "remote", transport: "sandbox", environmentId: identity, remoteCwd: `/tmp/${identity}`, sandboxLeaseAcquisition: { providerLeaseId: `${identity}-sandbox` } } as never, instructionWorkingCopy: copy(identity) }); - leaseRows = [{ status: lease }]; + const receipt = lease.endsWith("-receipt") ? lease.slice(0, -"-receipt".length) : null; + leaseRows = [{ id: `${identity}-lease`, companyId: warmExecution.binding.companyId, environmentId: identity, + heartbeatRunId: identity, provider: "daytona", providerLeaseId: `${identity}-sandbox`, + status: receipt ? "released" : lease, releasedAt: lease === "retained" ? null : new Date(), + cleanupStatus: receipt || lease === "released" ? "success" : "failed", leasePolicy: "reuse_by_environment", + metadata: receipt ? { remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", + companyId: warmExecution.binding.companyId, runId: identity, leaseId: `${identity}-lease`, provider: "daytona", + providerLeaseId: `${identity}-sandbox`, state: receipt, confirmedAt: new Date().toISOString() } } : {} }]; await vi.waitFor(() => expect(session.close).toHaveBeenCalledTimes(1)); const followUpRunId = `${identity}-follow-up`; const adopt = vi.fn(async () => null); @@ -12197,6 +12208,79 @@ describe("native warm session supervision", () => { expect(collectStopped).toHaveBeenCalledOnce(); }); + it("an idle warm session whose close fails with runner_remote_process_identity_unavailable and whose Runner is proven gone is released at controller restart", async () => { + const name = "warm-close-runner-gone"; + const current = { ...execution, + binding: { ...execution.binding, runId: name, executionWorkspaceId: name }, + session: { ...execution.session, normalizedSessionId: name, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 } }, + } as NativeExecutionInputV1; + const close = vi.fn<() => Promise>(async () => { + throw new RemoteRunnerRecoveryError({ stage: "process_verification", reason: "process_not_running" }); + }); + state.execute.mockReset().mockImplementationOnce(async options => { + await options.onSession?.({ close }); + return { result: { summary: "completed" }, terminal: { runTerminalState: "succeeded" }, + turnId: name, normalizedSessionId: name, providerSessionId: name, driverKind: "test", driverVersion: "1", + nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null }; + }); + await expect(executePaperclipNativeSession({ db: leaseDb(current), execution: current, runnerInstanceId: name })) + .resolves.toMatchObject({ exitCode: 0 }); + expect(close).not.toHaveBeenCalled(); + expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 1, busy: 0, failed: 0, failures: [] }); + expect(close).toHaveBeenCalledOnce(); + expect(await closeIdleWarmNativeSessionsForRestart()).toEqual({ closed: 0, busy: 0, failed: 0, failures: [] }); + }); + + it("names the Runner check in a warm close failure and keeps an unproven owner registered at controller restart", async () => { + const name = "warm-close-marker-missing"; + const current = { ...execution, + binding: { ...execution.binding, runId: name, executionWorkspaceId: name }, + session: { ...execution.session, normalizedSessionId: name, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 } }, + } as NativeExecutionInputV1; + const close = vi.fn<() => Promise>(async () => { + throw new RemoteRunnerRecoveryError({ stage: "process_verification", reason: "marker_missing" }); + }); + state.execute.mockReset().mockImplementationOnce(async options => { + await options.onSession?.({ close }); + return { result: { summary: "completed" }, terminal: { runTerminalState: "succeeded" }, + turnId: name, normalizedSessionId: name, providerSessionId: name, driverKind: "test", driverVersion: "1", + nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null }; + }); + await executePaperclipNativeSession({ db: leaseDb(current), execution: current, runnerInstanceId: name }); + expect(await closeIdleWarmNativeSessionsForRestart()).toMatchObject({ closed: 0, failed: 1, failures: [{ runId: name, + reason: "close failed: runner_remote_process_identity_unavailable: runner_remote_process_identity_unavailable [process_verification/marker_missing]" }] }); + close.mockResolvedValue(undefined); + expect(await closeIdleWarmNativeSessionsForRestart()).toMatchObject({ closed: 1, failed: 0 }); + }); + + it("retries a failed idle warm close a bounded number of times", async () => { + const name = "warm-idle-close-retry"; + const current = { ...execution, + binding: { ...execution.binding, runId: name, executionWorkspaceId: name }, + session: { ...execution.session, normalizedSessionId: name, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 20 } }, + } as NativeExecutionInputV1; + const close = vi.fn<() => Promise>(async () => { throw new Error("native_runner_checkpoint_owner_lost"); }); + state.execute.mockReset().mockImplementationOnce(async options => { + await options.onSession?.({ close }); + return { result: { summary: "completed" }, terminal: { runTerminalState: "succeeded" }, + turnId: name, normalizedSessionId: name, providerSessionId: name, driverKind: "test", driverVersion: "1", + nativeEventCount: 1, highestContiguousSourceSeq: 1, usage: null }; + }); + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] }); + try { + await executePaperclipNativeSession({ db: leaseDb(current), execution: current, runnerInstanceId: name }); + await vi.advanceTimersByTimeAsync(20); + expect(close).toHaveBeenCalledTimes(1); + await vi.advanceTimersByTimeAsync(WARM_IDLE_CLOSE_RETRY_MS); + expect(close).toHaveBeenCalledTimes(2); + await vi.advanceTimersByTimeAsync(WARM_IDLE_CLOSE_RETRY_MS * 10); + expect(close).toHaveBeenCalledTimes(1 + WARM_IDLE_CLOSE_MAX_RETRIES); + expect(readNativeSessionIdleWork()).toBe("present"); + } finally { vi.useRealTimers(); } + close.mockResolvedValue(undefined); + expect(await closeIdleWarmNativeSessionsForRestart()).toMatchObject({ closed: 1, failed: 0 }); + }); + it.each(["failed", "cancelled"] as const)("retires a %s session even if its warm checkpoint rejects", async (runTerminalState) => { const name = `warm-checkpoint-rejects-${runTerminalState}`; const current = { ...execution, @@ -13245,6 +13329,48 @@ describe("native warm session supervision", () => { }, ); + it.each([ + { lease: "released-stop-receipt", closed: 1 }, + { lease: "pending_cleanup", closed: 0 }, + { lease: "released-without-receipt", closed: 0 }, + { lease: "stop-receipt-for-another-run", closed: 0 }, + ] as const)("reads the warm owner's own lease rows at controller restart: $lease", async ({ lease: kind, closed }) => { + const { base } = fixture(`restart-lease-${kind}`); + const environmentId = `restart-lease-env-${kind}`, providerLeaseId = `restart-lease-sandbox-${kind}`; + const close = vi.fn(async (): Promise => { throw new Error("owned close not yet confirmed"); }); + state.execute.mockReset().mockImplementationOnce(async options => { + await options.onSession?.({ close }); return result; + }); + const leaseRunId = kind === "stop-receipt-for-another-run" ? "earlier-run" : base.binding.runId; + const terminal = kind !== "pending_cleanup"; + const row = { id: `lease-${kind}`, companyId: base.binding.companyId, environmentId, heartbeatRunId: leaseRunId, + provider: "daytona", providerLeaseId, status: terminal ? "released" : "pending_cleanup", releasedAt: new Date(), + cleanupStatus: terminal ? "success" : "failed", leasePolicy: "reuse_by_environment", + metadata: kind === "released-stop-receipt" || kind === "stop-receipt-for-another-run" + ? { remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId: base.binding.companyId, + runId: leaseRunId, leaseId: `lease-${kind}`, provider: "daytona", providerLeaseId, state: "stopped", + confirmedAt: new Date().toISOString() } } + : { failureReason: "sandbox_stop_pending" } }; + let leaseRows: unknown[] = []; + const ownerDb = leaseDb(base), select = ownerDb.select.bind(ownerDb); + ownerDb.select = ((fields?: Record) => ({ from: (table: unknown) => { + if (table !== environmentLeases) return select(fields as never).from(table as never); + const query = { then: (...args: Parameters["then"]>) => Promise.resolve(leaseRows).then(...args), + where: () => query, limit: () => query }; + return query; + } })) as unknown as Db["select"]; + try { + await executePaperclipNativeSession({ db: ownerDb, execution: base, runnerInstanceId: "runner", + runnerExecutionTarget: { kind: "remote", transport: "sandbox", environmentId, remoteCwd: `/tmp/${environmentId}`, + sandboxLeaseAcquisition: { outcome: "created", providerLeaseId } } }); + leaseRows = [row]; + expect(await closeIdleWarmNativeSessionsForRestart()).toMatchObject({ closed, failed: 1 - closed }); + } finally { + close.mockResolvedValue(undefined); + await closeIdleWarmNativeSessionsForRestart(); + } + }); + it("retains failed retirement for retry and never launches over an uncontained owner", async () => { const { base, next, run } = fixture("identity-close-failure"); const close = vi.fn().mockRejectedValueOnce(new Error("containment unavailable")).mockResolvedValue(undefined); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index cffe9cd56c0..26badf5ba56 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -743,12 +743,15 @@ type WarmNativeSession = { controllerDetachedForRestart?: boolean; finishRelease?: () => Promise; closeOnReleaseReason?: string; + idleCloseRetries?: number; idleTimer: ReturnType | null; lastActivityAt: string; }; const WARM_CLOSE_FAILURE_DETAIL_MAX = 2_000; const WARM_NATIVE_SESSION_CLOSE_CONCURRENCY = 4; +export const WARM_IDLE_CLOSE_RETRY_MS = 5_000; +export const WARM_IDLE_CLOSE_MAX_RETRIES = 3; export function describeWarmCloseFailure(error: unknown): string { const describe = (value: unknown): string => { @@ -757,8 +760,10 @@ export function describeWarmCloseFailure(error: unknown): string { const code = "code" in value && typeof value.code === "string" ? `${value.code}: ` : ""; const settlement = value instanceof NativeSessionCloseUnrecoverableError && value.settlement ? ` ${JSON.stringify(value.settlement)}` : ""; + const remote = readRemoteRunnerRecoveryError(value); + const recovery = remote ? ` [${remote.stage}/${remote.reason}${remote.runnerLifecycle ? `/${remote.runnerLifecycle}` : ""}]` : ""; const cause = value.cause === undefined ? "" : ` (cause: ${describe(value.cause)})`; - return `${code}${value.message}${settlement}${cause}`; + return `${code}${value.message}${recovery}${settlement}${cause}`; }; const text = describe(error).replaceAll(/[\r\n]+/g, " "); return text.length > WARM_CLOSE_FAILURE_DETAIL_MAX ? `${text.slice(0, WARM_CLOSE_FAILURE_DETAIL_MAX)}…` : text; @@ -1107,17 +1112,7 @@ export async function closeWarmNativeSessionsForEnvironment(input: { return closeIdleWarmNativeSessions(input); } -async function warmSandboxGivenUpByHost(entry: WarmNativeSession): Promise { - if (!entry.sandboxLeaseDb || !entry.providerLeaseId || !entry.environmentId) return false; - const leases = await entry.sandboxLeaseDb.select({ status: environmentLeases.status }).from(environmentLeases).where(and( - eq(environmentLeases.companyId, entry.companyId), - eq(environmentLeases.environmentId, entry.environmentId), - eq(environmentLeases.providerLeaseId, entry.providerLeaseId), - )); - return leases.length > 0 && leases.every(lease => lease.status !== "active" && lease.status !== "retained"); -} - -async function warmSandboxDestroyed(db: Db, entry: WarmNativeSession): Promise { +async function warmSandboxTerminated(db: Db, entry: WarmNativeSession): Promise { if (!entry.providerLeaseId || !entry.environmentId) return false; const leases = await db.select().from(environmentLeases).where(and( eq(environmentLeases.companyId, entry.companyId), @@ -1125,8 +1120,23 @@ async function warmSandboxDestroyed(db: Db, entry: WarmNativeSession): Promise resolveRemoteTerminationLease(db, lease))); - return resolved.some(lease => hasRemoteTerminationReceipt(lease) - && (lease.metadata?.remoteExecutionTermination as { state?: unknown } | undefined)?.state === "destroyed"); + return resolved.some(lease => { + if (!hasRemoteTerminationReceipt(lease)) return false; + const state = (lease.metadata?.remoteExecutionTermination as { state?: unknown } | undefined)?.state; + return state === "destroyed" || (state === "stopped" && lease.heartbeatRunId === entry.runId); + }); +} + +/** A failed close is complete when its Runner is proven gone, its sandbox is destroyed, or the run's own sandbox lease has a stop receipt. */ +async function warmCloseFailureProvesOwnerGone(entry: WarmNativeSession, error: unknown, db = entry.sandboxLeaseDb): Promise { + const diagnostic = readRemoteRunnerRecoveryError(error); + if (diagnostic?.stage === "process_verification" && diagnostic.reason === "process_not_running") return true; + try { + return db ? await warmSandboxTerminated(db, entry) : false; + } catch (proofError) { + logger.warn({ err: proofError, runId: entry.runId }, "could not read the warm session's sandbox lease"); + return false; + } } /** Detach released owners and checkpoint unreleased owners before a controller @@ -1195,16 +1205,10 @@ async function closeIdleWarmNativeSessions(input: { busy += 1; return; } - const destroyed = !detach && input.forceStopUnsettled && input.db - ? await warmSandboxDestroyed(input.db, entry).catch(proofError => { - logger.warn({ err: proofError, runId: entry.runId }, "could not read the warm session's sandbox lease"); - return false; - }) - : false; - if (destroyed) { + if (!detach && await warmCloseFailureProvesOwnerGone(entry, error, input.db ?? entry.sandboxLeaseDb)) { logger.warn({ runId: entry.runId, issueId: entry.issueId, environmentId: entry.environmentId, providerLeaseId: entry.providerLeaseId ?? null, reason: input.reason, closeFailure: describeWarmCloseFailure(error) }, - "warm native session's sandbox was already destroyed; released its owner"); + "warm native session's Runner or sandbox is gone; released its owner"); forgetWarmNativeSession(sessionId, entry); closed += 1; return; @@ -8860,17 +8864,21 @@ function armWarmNativeSessionIdleTimer( return; } logger.warn({ runId: current.runId, issueId: current.issueId, environmentId: current.environmentId, - providerLeaseId: current.providerLeaseId ?? null, closeFailure: describeWarmCloseFailure(error) }, + providerLeaseId: current.providerLeaseId ?? null, closeFailure: describeWarmCloseFailure(error), + closeAttempt: (current.idleCloseRetries ?? 0) + 1 }, released ? "idle warm native session detach failed" : "idle warm native session close failed"); if (released) return; - return warmSandboxGivenUpByHost(current).then(givenUp => { - if (!givenUp) return; - logger.warn({ runId: current.runId, issueId: current.issueId, environmentId: current.environmentId, - providerLeaseId: current.providerLeaseId ?? null }, - "warm native session's sandbox was stopped by the host; released its owner"); - forgetWarmNativeSession(sessionId, current); - }, proofError => { - logger.warn({ err: proofError, runId: current.runId }, "could not read the warm session's sandbox lease"); + return warmCloseFailureProvesOwnerGone(current, error).then(gone => { + if (gone) { + logger.warn({ runId: current.runId, issueId: current.issueId, environmentId: current.environmentId, + providerLeaseId: current.providerLeaseId ?? null }, + "warm native session's Runner or sandbox is gone; released its owner"); + forgetWarmNativeSession(sessionId, current); + return; + } + if ((current.idleCloseRetries ?? 0) >= WARM_IDLE_CLOSE_MAX_RETRIES) return; + current.idleCloseRetries = (current.idleCloseRetries ?? 0) + 1; + armWarmNativeSessionIdleTimer(sessionId, current, WARM_IDLE_CLOSE_RETRY_MS); }); }) .finally(() => {