From e680fa68140b151aad69f3c7f55f1c825274a2c8 Mon Sep 17 00:00:00 2001
From: Dotta <34892728+cryppadotta@users.noreply.github.com>
Date: Fri, 6 Nov 2026 08:27:59 -0600
Subject: [PATCH] fix(native): retry startup interrupted by a closed runtime
 API listener (#19462)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs prepare a remote workspace, then start a runner and a
provider session
> - A graceful server shutdown closes the HTTP listener before it drains
runs
> - A run that is still preparing then asks for the local runtime API
origin, and the lookup fails with
`local_runtime_api_listener_unavailable`
> - Recovery did not know this code. It treated the failure as an
ambiguous provider state and blocked the task behind a board-owned
action, although no runner or provider process existed
> - This pull request classifies the closed listener as a definite
pre-session failure
> - The benefit is that a deploy no longer strands a preparing task. The
same run gets a bounded bootstrap retry that the next server process
claims

## Linked Issues or Issue Description

**What happened**
A native Codex task on a remote sandbox spent about ten minutes in
workspace preparation. A deploy started a graceful shutdown during that
time. When preparation finished, the executor called
`resolveLocalRuntimeApiUrl()` to build the tool authority. The listener
was already closed, so it threw
`local_runtime_api_listener_unavailable`. No runner backend or provider
session had started. The failure classifier returned the generic
`native_session_interrupted` code. `nativeProviderRecoveryEvidence` then
returned `ambiguous_state`. The run became a terminal failure, the task
became blocked, and the recovery action told the board to resolve an
ambiguous session and to not open a replacement session. The same
signature occurred on three other runs during an earlier deploy.

**Expected behavior**
A failure that occurs before any runner or provider process exists is
safe to retry on the same run. The task must stay in progress with a
scheduled, bounded bootstrap retry. If a provider event or checkpoint
exists, recovery must stay operator-owned.

**Steps to reproduce**
Start a native run. Close the server's HTTP listener before the executor
creates the runner backend. Observe a `terminal_failure` with
`recoveryMode: "ambiguous_state"` and a blocked task. The new
integration test reproduces this with the real
`resolveLocalRuntimeApiUrl()` and a real Postgres database.

**Environment**
Server native runtime, remote sandbox target, `codex_app_server` driver.
Related recovery work: #19249, #19304, #19305.

## What Changed

- `nativeSessionFailureSourceCode` returns a new
`local_runtime_api_listener_unavailable` source code for this error.
- `nativeProviderRecoveryEvidence` treats that code as definitely
pre-session. The existing guard stays: the run must have no checkpoint
and no durable provider event.
- The chat failed-run retry accepts the code in its list of ordinary
pre-session failures, so an exhausted startup retry can still be retried
from chat.
- New integration tests in `native-startup-retry.integration.test.ts`
cover the retry path and the operator-owned path when a provider event
exists.

## Verification

- Red before the fix: the new test failed with `phase:
"terminal_failure"`, `recoveryMode: "ambiguous_state"`, `recoverable:
false` and a board-owned recovery. This matches the failed run on the
deployed instance.
- Green after the fix: `cd server && npx vitest run
src/services/native-runtime/native-startup-retry.integration.test.ts`
passes 26 of 26 tests. The retry test asserts `retryable_failure`,
`bootstrap_retry`, backoff timing, an agent-owned `resume_native_run`
wake, and a claim through `claimNativeSessionResumptions`.
- Safety case: when a durable `turn.started` event exists, the same
error stays `terminal_failure` with `ambiguous_state` and a blocked
task.
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"listener|classif|pre-session"` passes.
- Not run: a live deploy that interrupts a preparing sandbox run. The
test uses the real listener lookup and the real recovery tables, but not
a real sandbox.

## Risks

- Low risk. The change only affects runs that fail with this exact error
and have no checkpoint and no provider event.
- The retry uses the existing bootstrap budget of three attempts with
backoff. Repeated failures still stop and hand the task to the board.
- The GitHub callback bridge that started in the sandbox before the
failure is a per-attempt helper. The existing cleanup path releases it.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use, code execution and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
---
 .../chat-channels/failed-run-retry.ts         |  1 +
 .../native-runtime/native-session-executor.ts |  8 ++
 .../native-startup-retry.integration.test.ts  | 77 +++++++++++++++++++
 3 files changed, 86 insertions(+)

diff --git a/server/src/services/chat-channels/failed-run-retry.ts b/server/src/services/chat-channels/failed-run-retry.ts
index d90c7a6c989..533f013345c 100644
--- a/server/src/services/chat-channels/failed-run-retry.ts
+++ b/server/src/services/chat-channels/failed-run-retry.ts
@@ -566,6 +566,7 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
       "provider_transport_failed",
       "native_runner_process_exited",
       "provider_session_open_rejected",
+      "local_runtime_api_listener_unavailable",
     ];
     if (
       !coordinator ||
diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts
index 6013a88ba67..ac41ebd8d80 100644
--- a/server/src/services/native-runtime/native-session-executor.ts
+++ b/server/src/services/native-runtime/native-session-executor.ts
@@ -7310,6 +7310,7 @@ export function nativeSessionFailureSourceCode(
   | "provider_session_open_rejected"
   | "runner_ingress_access_rejected"
   | "runner_ingress_acquisition_failed"
+  | "local_runtime_api_listener_unavailable"
   | "native_session_interrupted" {
   if (error instanceof NativePermissionDeclinedError) return "native_permission_declined";
   if (error instanceof NativeProviderStateIncompatibleError) return NATIVE_PROVIDER_STATE_INCOMPATIBLE;
@@ -7410,6 +7411,12 @@ export function nativeSessionFailureSourceCode(
     }
     return "provider_session_open_rejected";
   }
+  // This process has no HTTP listener (graceful shutdown closes it first), so
+  // the control plane cannot hand a runner its callback origin. The lookup runs
+  // before any runner or provider process is created.
+  if (/\blocal_runtime_api_listener_unavailable\b/.test(message)) {
+    return "local_runtime_api_listener_unavailable";
+  }
   return "native_session_interrupted";
 }
 
@@ -7505,6 +7512,7 @@ export async function nativeProviderRecoveryEvidence(input: {
     "provider_request_protocol_error",
     "native_runner_process_exited",
     "provider_session_open_rejected",
+    "local_runtime_api_listener_unavailable",
   ]).has(input.sourceFailureCode);
   if (!checkpointExists && !providerEventsExist && (definitelyPreSession
     || unclaimedBootstrap && input.bootstrap?.transientIngressFailure)) {
diff --git a/server/src/services/native-runtime/native-startup-retry.integration.test.ts b/server/src/services/native-runtime/native-startup-retry.integration.test.ts
index 117280af19d..e847c5d27f8 100644
--- a/server/src/services/native-runtime/native-startup-retry.integration.test.ts
+++ b/server/src/services/native-runtime/native-startup-retry.integration.test.ts
@@ -40,6 +40,7 @@ import { environmentRunnerIngressErrorData, JsonRpcCallError, PluginEnvironmentR
 import { accountRunCost } from "../run-cost-accounting.js";
 import { settleUnrecoverableExecutions } from "../execution-recovery-resolution.js";
 import { agentProviderLimitCooldown } from "./native-startup-retry.js";
+import { resolveLocalRuntimeApiUrl, runtimeApiInternals } from "../../runtime-api.js";
 
 type SessionOpenStatus = "failed" | "rejected" | "indeterminate";
 
@@ -479,6 +480,82 @@ describe("native provider startup retry", () => {
     expect(state.issue.status).toBe("in_progress");
   });
 
+  it("retries startup that a draining server interrupted before any provider session", async () => {
+    const seeded = await seedRun(await createAgent());
+    // A graceful shutdown closes the HTTP listener first; startup that reaches
+    // the runtime API lookup afterwards fails before a runner or provider exists.
+    runtimeApiInternals.resetForTests();
+    provider.execute.mockReset().mockImplementation(async () => {
+      resolveLocalRuntimeApiUrl();
+      throw new Error("expected the closed listener to stop startup");
+    });
+    const failedAt = new Date();
+    await expect(executePaperclipNativeSession({
+      db,
+      execution: seeded.execution,
+      runnerInstanceId: seeded.runnerInstanceId,
+      backend: {
+        descriptor: async () => { throw new Error("Unexpected provider descriptor call"); },
+        openSession: async () => { throw new Error("Unexpected provider session call"); },
+      },
+    })).rejects.toThrow("local_runtime_api_listener_unavailable");
+
+    const state = await read(seeded);
+    expect(state.coordinator).toMatchObject({
+      phase: "retryable_failure",
+      recoveryState: "resuming_session",
+      attempt: 1,
+      leaseOwner: null,
+      failureDetail: expect.objectContaining({
+        originalFailureCode: "local_runtime_api_listener_unavailable",
+        recoveryMode: "bootstrap_retry",
+        providerSessionEstablished: false,
+        providerEventsExist: false,
+        checkpointExists: false,
+      }),
+    });
+    const [low, high] = backoffWindow(failedAt, 1);
+    expect(state.coordinator.nextAttemptAt!.getTime()).toBeGreaterThanOrEqual(low);
+    expect(state.coordinator.nextAttemptAt!.getTime()).toBeLessThanOrEqual(high + 5_000);
+    expect(state.issue.status).toBe("in_progress");
+    expect(state.action).toMatchObject({
+      ownerType: "agent",
+      ownerAgentId: seeded.agentId,
+      wakePolicy: { kind: "resume_native_run", runId: seeded.runId },
+    });
+    // The next server process claims the same run; no replacement run is created.
+    await claimRetry(seeded);
+  });
+
+  it("keeps a closed-listener failure operator-owned once provider work is durable", async () => {
+    const seeded = await seedRun(await createAgent());
+    runtimeApiInternals.resetForTests();
+    provider.execute.mockReset().mockImplementation(async () => {
+      await db.insert(heartbeatRunEvents).values({
+        companyId, runId: seeded.runId, agentId: seeded.agentId, seq: 900,
+        eventType: "turn.started", stream: "system",
+      });
+      resolveLocalRuntimeApiUrl();
+    });
+    await expect(executePaperclipNativeSession({
+      db,
+      execution: seeded.execution,
+      runnerInstanceId: seeded.runnerInstanceId,
+      backend: {
+        descriptor: async () => { throw new Error("Unexpected provider descriptor call"); },
+        openSession: async () => { throw new Error("Unexpected provider session call"); },
+      },
+    })).rejects.toThrow("local_runtime_api_listener_unavailable");
+
+    const state = await read(seeded);
+    expect(state.coordinator).toMatchObject({
+      phase: "terminal_failure",
+      nextAttemptAt: null,
+      failureDetail: expect.objectContaining({ recoveryMode: "ambiguous_state", providerSessionEstablished: true }),
+    });
+    expect(state.issue.status).toBe("blocked");
+  });
+
   it.each(["grok_model_mismatch", "requested_model_unsupported"])(
     "stops at a session.open %s rejection with a choose-model action instead of a doomed retry",
     async (classification) => {