fix(native): relaunch a remote Runner that exited for a fresh bootstrap (#21374)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs often use a remote Runner in a sandbox. After a restart,
a deploy or a same-run retry, the server reattaches to that Runner or
starts a new Runner over its saved state
> - A Runner that no controller reaches before its connection lease
expires stops its provider, saves its state as `recoverable_failure`
with `lease_expired_requires_bootstrap`, and exits. The Runner protocol
says that only a new process with a fresh bootstrap ticket can
reconnect, and that the new process continues from the same cursor
> - Reattachment started a new Runner over saved state only for a
settled `suspended` exit (or an exit before `session.open`). So it
refused this exit, and the run ended with
`runner_remote_process_identity_unavailable`
(`native_continuation_requires_reconciliation`)
> - This pull request lets reattachment launch a Runner over the exact
saved state of a Runner that exited to ask for a fresh bootstrap
> - The benefit is that the run continues by itself after a long
controller gap, with no Retry and no manual sandbox repair

## Linked Issues or Issue Description

Refs #21329, #21362, #20853

**What happened?**
On hosted staging (build `2b69cdfcd6`, read-only trace), Codex runs on
remote Daytona Runners ended with
`native_continuation_requires_reconciliation` and the error
`runner_remote_process_identity_unavailable`, about 5 per hour, after
#21329 was deployed. All five sample runs (c6fabe9a, 31d54b02, 995b63df,
dd595934, 50c6e6e6) ended the same way. The last
`native.remote_runner_recovery_failed` event recorded
`process_verification` / `process_not_running`, `runnerLifecycle:
recoverable_failure`, `runnerRecoverableFailure:
lease_expired_requires_bootstrap`, and the Runner diagnostic "connection
lease expired; a fresh bootstrap is required". Since 10:30 UTC, 8 of the
15 runs that ended with this error had this exact exit.

Timeline of run c6fabe9a:
- 10:46 to 10:54: `runner.session.startup` took 476 s in a loaded
sandbox.
- 10:54:02: the controller sent `turn.start`. At 10:54:47 the command
timed out: "PRP command turn.start timed out (runner connected; pending
ahead 0; last runner event seq 2 91s ago)". The run resumed in the same
run and the turn then ran (tools completed from 11:14).
- 11:39: a graceful server shutdown suspended the controller. At 11:43
the next controller started the reattachment, but it did not reconnect
to the Runner until the server restarted again at 11:58. In that gap the
Runner's lease expired and the Runner exited.
- 11:58:53: the reattachment proved the process gone and refused the
`recoverable_failure` state. The run failed at 11:59.

Why `turn.start` timed out while the Runner was connected: the Runner
answers `turn.start` only after Codex answers its `turn/start` JSON-RPC
request. While it waits, the Runner buffers Codex notifications and
commits no events, so the controller sees a quiet connected Runner with
nothing pending ahead. The controller's bound for the `turn.start`
command result is a fixed 30 s. #21329 changed only the later wait for
`turn/started`. In a sandbox that needed 8 minutes to start, Codex
answered later than 30 s. That timeout is recoverable, and the run did
recover from it. This pull request does not change it (see Risks). The
failure that ended the runs is the refused reattachment.

**Expected behavior**
When a remote Runner exits because its lease expired (or its reconnect
grace or deadline ended) and its saved state is intact, reattachment
starts a new Runner from that state and the run continues.

**Steps to reproduce**
1. Start a native Codex run on a remote sandbox Runner.
2. Keep every controller from reaching the Runner for longer than its
connection lease (on staging: a deploy that stalled the reattachment for
about 15 minutes).
3. The Runner saves `recoverable_failure` /
`lease_expired_requires_bootstrap` and exits.
4. The next reattachment fails with
`runner_remote_process_identity_unavailable`, and the run ends.

**Agent adapter(s) involved**
`paperclip_runner` with Codex on a Daytona remote sandbox. The rule is
not provider-specific.

## What Changed

- `remote-runner-recovery-diagnostics.ts`: new
`exitedRunnerRequestedFreshBootstrap` (diagnostic) and
`runnerStateRequestsFreshBootstrap` (saved state). They accept lifecycle
`recoverable_failure` with `lease_expired_requires_bootstrap`,
`transport_reconnect_grace_exceeded` or
`transport_reconnect_deadline_exceeded`. These are the three codes the
Rust Runner records after `shutdown_preserving_cleanup`. The state check
also requires no pending terminal delivery and no warm transition.
- `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): when
the exact Runner process is proven gone and its diagnostic names one of
these exits, the server reads the saved state again. If the state has
the same six identity fields and still names that exit, the reattachment
launches a Runner from it instead of failing. The outbox does not need
to be empty and a turn may be active: the new Runner replays from the
same cursor. The attempt bound is the one the reattachment already
admits (3, or 4 for an unanswered-Runner replacement claim). Other exits
are unchanged.
- `native-session-executor.ts` (host preparation in
`createRunnerdBackend`): a resumed sandbox for a reattachment that did
not adopt a process re-reads the Runner state. When it still has the
same six identity fields and names the fresh-bootstrap exit, preparation
keeps it in place, as for an adopted live Runner. It no longer fails
with `runner_harness_state_mismatch: resumed_not_suspended` (found by
Greptile).
- `doc/execution-semantics.md`: one paragraph describes the rule and the
staging evidence.

## Verification

- Red on `origin/master`: the 5 new "launches a new Runner over the
saved state after ..." cases failed (the reattachment threw
`runner_remote_process_identity_unavailable`). The 7 new fail-closed
cases passed.
- Green: `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "remote
controller restart adoption"` passes 118 tests.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
1712 passed, 21 skipped, 1 failed. The failure ("verifies a live warm
owner before refreshing run authority", warm-session supervision, not
touched here) passes when run alone. It is a timing failure under
full-file load.
- Launch path: "keeps the saved mid-turn state of a remote Runner that
exited for lease_expired_requires_bootstrap when the reattachment
prepares its sandbox" runs `createRunnerdBackend` and
`prepareExternalRunnerState` with a mid-turn Codex checkpoint. Red
before the host-preparation change (`resumed_not_suspended`), green
after. An unknown recoverable failure stays fail-closed.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts`
on the final revision: 1494 passed, 7 skipped.
- `cd server && node scripts/typecheck-layers.mjs` (includes test
programs): passes.
- Fail-closed cases covered: an unknown recoverable failure, a warm
transition, a pending terminal delivery, a state that changed after the
exit, a different Runner instance, attempt 4 on an ordinary claim, and
attempt 5 on a replacement claim.
- Staging read (read-only): the 5 sample runs and 69 runs with this
error in the last 9 hours were classified from their
`native.remote_runner_recovery_failed` events.

## Risks

- Medium. Reattachment now starts a new Runner over a saved state that
can be mid-turn. The Runner itself stops its provider before it saves
this state. The new Runner restores the session. It continues the turn,
or it reports the lost turn as recoverable only after it proves that the
old provider process ended. A recoverable lost turn on a remote
reattachment ends the run as `native_provider_terminal_failed` until
#21362 merges; the replacement lane then continues the task. That is
still better than the current reconciliation hold.
- The attempt bound does not grow. A Runner that exits this way again on
a later reattachment is bounded by the same provider attempt budget.
- This does not change the 30 s `turn.start` command bound or the deploy
gaps that let the lease expire. Those are separate follow-ups. 57 older
runs with this error had a different exit (`ready`, not settled). This
pull request keeps that case fail-closed; #20853 covers the
settled-`ready` part.
- No schema, API, UI or Runner image change.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude
Code with tool use (shell, file edit, code execution, read-only browser
reads of staging run data).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 06:53:38 -06:00
1 parent e3faf579a1
commit eb54831d28
4 files changed
+147 -4

No files matched your search

+19
View File
@@ -6016,6 +6016,25 @@ cannot. Process verification and the final state read share one recovery
deadline. A timeout, command failure, transport failure or invalid saved JSON on
that final read keeps its closed recovery stage and reason in the diagnostic.
A Runner that no controller reaches before its connection lease expires, or
before its reconnect grace or deadline ends, stops its provider, saves its state
with lifecycle `recoverable_failure` and `lease_expired_requires_bootstrap`,
`transport_reconnect_grace_exceeded` or `transport_reconnect_deadline_exceeded`,
and exits. Only a new process with a fresh bootstrap ticket can reconnect, and
that process continues from the same cursor. A reattachment that finds this exit
re-reads the state after the process is proven gone. When it has the same six
identity fields, the same lifecycle and code, and no pending terminal delivery
or warm transition, the reattachment launches a Runner from that state, even
mid-turn and with events still in the outbox. Sandbox preparation reads that
state once more and keeps it in place, as it does for an adopted live Runner;
it does not require a `suspended` state or an idle provider checkpoint for this
exit. The new Runner restores the
conversation: it continues the turn, or reports the lost turn as recoverable
after it proves that the old provider process ended. This path admits the same
attempts as a reattachment, including the extra attempt of an unanswered-Runner
replacement. On staging, a deploy that kept a reattachment from reconnecting for
more than the lease ended these runs with `runner_remote_process_identity_unavailable`.
Reattachment keeps the complete durable PRP binding, including its turn and item
attachment IDs. A healthy warm handoff can create new attachment IDs within the
retained Runner. Replacing that run's controller must preserve those IDs; deriving
@@ -1146,6 +1146,34 @@ describe("remote controller restart adoption", () => {
target: exitedRunner(state) as never, identity, runId: "run", normalizedSessionId: "session", providerAttempt }).catch(caught => caught);
expect(readRemoteRunnerRecoveryError(error)).toMatchObject({ stage: "process_verification", reason: "process_not_running" });
});
describe("a reattachment whose Runner exited on its own and asked for a fresh bootstrap", () => {
const leaseExpired = { lifecycle: "recoverable_failure", recoverableFailure: "lease_expired_requires_bootstrap",
outbox: [{ id: 412 }], pendingTerminalDelivery: null, pendingProviderCleanup: null, warmTransition: null };
const unanswered = { ...claim, replaceUnansweredRunner: true as const };
it.each([
["a lease expiry on attempt 2", claim, leaseExpired, 2],
["a lease expiry on the last attempt", claim, leaseExpired, 3],
["a lease expiry on the admitted replacement attempt", unanswered, leaseExpired, 4],
["an elapsed reconnect deadline", claim, { ...leaseExpired, recoverableFailure: "transport_reconnect_deadline_exceeded" }, 2],
["an elapsed reconnect grace", claim, { ...leaseExpired, recoverableFailure: "transport_reconnect_grace_exceeded" }, 2],
] as const)("launches a new Runner over the saved state after %s", async (_case, recoveryClaim, state, providerAttempt) => {
await expect(verifyRemoteRunnerOrSettledExit({ claim: recoveryClaim, target: exitedRunner(state) as never, identity,
runId: "run", normalizedSessionId: "session", providerAttempt })).resolves.toBeUndefined();
});
it.each([
["an unknown recoverable failure", claim, leaseExpired, { ...leaseExpired, recoverableFailure: "provider_failed" }, 2],
["a warm transition", claim, leaseExpired, { ...leaseExpired, warmTransition: { transitionId: "warm" } }, 2],
["a pending terminal delivery", claim, leaseExpired, { ...leaseExpired, pendingTerminalDelivery: { eventId: "event" } }, 2],
["a state that changed after the exit", claim, leaseExpired, { ...leaseExpired, lifecycle: "turn_active" }, 2],
["a different Runner after the exit", claim, leaseExpired, { ...leaseExpired, runnerInstanceId: "other" }, 2],
["an attempt beyond the budget", claim, leaseExpired, leaseExpired, 4],
["an attempt beyond the replacement budget", unanswered, leaseExpired, leaseExpired, 5],
] as const)("keeps the reattachment fail-closed for %s", async (_case, recoveryClaim, state, afterExit, providerAttempt) => {
const error = await verifyRemoteRunnerOrSettledExit({ claim: recoveryClaim, target: exitedRunner(state, afterExit) as never,
identity, runId: "run", normalizedSessionId: "session", providerAttempt }).catch(caught => caught);
expect(readRemoteRunnerRecoveryError(error)).toMatchObject({ stage: "process_verification", reason: "process_not_running" });
});
});
describe("a same-run reattachment whose adopted Runner never answered its recovery read", () => {
const unanswered = { ...claim, replaceUnansweredRunner: true as const };
const midTurn = { lifecycle: "turn_active", outbox: [{ id: 338 }], pendingTerminalDelivery: null, pendingProviderCleanup: null };
@@ -17767,6 +17795,63 @@ describe("runnerd provider runtime wiring", () => {
|| request.args.includes("paperclip-runner-claim-unstarted-session"))).toBe(false);
});
it.each([
["keeps", "lease_expired_requires_bootstrap"],
["refuses", "provider_failed"],
] as const)("%s the saved mid-turn state of a remote Runner that exited for %s when the reattachment prepares its sandbox", async (outcome, recoverableFailure) => {
const current = scopedNativeExecution(`fresh-bootstrap-relaunch-${recoverableFailure}`);
const identity = { runId: current.binding.runId, normalizedSessionId: current.session.normalizedSessionId!,
runnerInstanceId: "lease-expired-runner", environmentLeaseId: "lease-expired-lease",
turnId: `turn_attachment_${randomUUID()}`, itemId: `item_attachment_${randomUUID()}` };
const root = await initializeScopedRunnerRoot(current, identity.runnerInstanceId);
await mkdir(join(root, "control-plane"), { recursive: true });
await writeFile(join(root, "control-plane", "control-plane-state.json"), JSON.stringify(durableControlPlaneState(identity)));
const runnerState = { ...identity, lifecycle: "recoverable_failure", recoverableFailure, outbox: [{ id: 412 }],
pendingTerminalDelivery: null, pendingProviderCleanup: null, warmTransition: null };
const providerState = { schema: "paperclip.runner.codex-provider-state.v1", lifecycle: "session_open",
threadId: "lease-expired-thread", providerSessionId: null, activeProviderTurnId: "provider-turn-1",
config: { provider: "codex", driver: "codex_app_server" } };
const execute = vi.fn(async (request: { args: string[] }) => {
const script = request.args[1] ?? "";
if (request.args[2] === "paperclip-runner-recovery-check") return { exitCode: 45, timedOut: false, stderr: "", stdout: "" };
const stdout = request.args[0] === "--build-metadata" ? JSON.stringify({
schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd",
packageName: "@paperclipai/paperclip-runner", binaryContractVersion: 2,
durableSessionCapabilities: ["unlimited_runtime", "connection_lease_renewal"], prpTransportModes: ["listen_ws"],
}) : request.args[0] === "--version" ? "codex-cli 0.160.0"
: script.includes("base64") && script.includes("runner-state.json") ? Buffer.from(JSON.stringify(runnerState)).toString("base64")
: script.includes("base64") ? Buffer.from(JSON.stringify(providerState)).toString("base64")
: script.includes("cat --") ? `nonce\n123\n2026-09-19T10:00:00.000Z\n${identity.runnerInstanceId}\nlinux:abcd-1234:100\n` : "";
return { exitCode: 0, timedOut: false, stderr: "", stdout };
});
const syncIn = vi.fn(async () => undefined);
state.createBackend.mockClear(); state.createTransport.mockClear();
const created = createRunnerdBackend({ db: leaseDb(current), execution: current, runnerInstanceId: identity.runnerInstanceId,
runnerIngressAuthorized: true,
controllerPersistence: { bootId: "successor", runnerArtifact: { version: "admitted-version", digest: `sha256:${"a".repeat(64)}` } },
runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace",
environmentId: "environment", leaseId: identity.environmentLeaseId, providerKey: "daytona", runner: { execute, syncIn },
effectiveCapabilities: { runnerWebSocketIngress: true },
sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "sandbox" } } as never,
restartRecovery: { kind: "reattach_remote_runner", runId: current.binding.runId, leaseOwner: "controller",
controllerGeneration: 2, providerAttempt: 2, restartKind: "hard", recoveryRequestId: null,
remote: { providerLeaseId: "sandbox", remoteCwd: "/workspace" } },
});
if (outcome === "refuses") {
await expect(created).rejects.toThrow("runner_remote_process_identity_unavailable");
return;
}
await created;
state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
const options = state.createTransport.mock.calls.at(-1)![0] as RunnerTransportOptions & { prepareExternalRunnerState: () => Promise<void> };
expect(options).toMatchObject({ prpIdentity: identity });
expect((options as { adoptExistingRunner?: unknown }).adoptExistingRunner).toBeUndefined();
await expect(options.prepareExternalRunnerState()).resolves.toBeUndefined();
expect(syncIn).not.toHaveBeenCalled();
expect(execute.mock.calls.some(([request]) => request.args.includes("paperclip-runner-claim-unstarted-session")
|| request.args.some(argument => argument.includes(".continuity-break-") || argument.includes("rm -")))).toBe(false);
});
it.each(["claude", "pi"] as const)("relays the assigned gateway when a deploy lands mid-run and the same ACPX %s run reattaches its surviving runner", async agent => {
const base = scopedNativeExecution(`relay-mid-run-${agent}`);
const current = { ...base, provider: { kind: "acpx", agent, model: null },
@@ -291,7 +291,8 @@ import { reportRunFailure } from "../run-failure-report.js";
import {
REMOTE_RECOVERY_MARKER_READ_SCRIPT, REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT,
REMOTE_RUNNER_EXIT_EVIDENCE_SCRIPT, RemoteRunnerRecoveryError, attachRemoteRunnerRecoveryDiagnostic, boundRemoteRunnerExitEvidence,
describeExitedRemoteRunner, describeExitedRemoteRunnerReason, readRemoteRunnerRecoveryError, remoteRecoveryProbeReason,
describeExitedRemoteRunner, describeExitedRemoteRunnerReason, exitedRunnerRequestedFreshBootstrap, readRemoteRunnerRecoveryError,
remoteRecoveryProbeReason, runnerStateRequestsFreshBootstrap,
} from "./remote-runner-recovery-diagnostics.js";
import { isPluginRpcTimeout, readRemoteRunnerRecoveryProbe, REMOTE_RUNNER_PLUGIN_STALL_BUDGET_MS, REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS, remoteRunnerPluginStall, type RemoteRunnerPluginStall } from "./remote-runner-recovery-probe.js";
import { describeOwnFailedAttachment, inspectRemoteWarmSessionRecovery, isSettledRemoteWarmControlState, summarizePostTerminalEvents, type RemoteWarmSessionRecovery, unsettledControllerCommands } from "./remote-warm-session-recovery.js";
@@ -15372,14 +15373,18 @@ export async function verifyRemoteRunnerOrSettledExit(
} catch (error) {
const diagnostic = readRemoteRunnerRecoveryError(error);
const exitedBeforeSessionOpen = input.claim.exitedBeforeSessionOpen === true;
if (input.providerAttempt > NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT || diagnostic?.stage !== "process_verification" ||
const freshBootstrap = diagnostic !== null && exitedRunnerRequestedFreshBootstrap(diagnostic);
const attemptLimit = freshBootstrap && input.claim.replaceUnansweredRunner === true
? NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT + 1 : NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT;
if (input.providerAttempt > attemptLimit || diagnostic?.stage !== "process_verification" ||
diagnostic.reason !== "process_not_running" ||
(!exitedBeforeSessionOpen && (diagnostic.runnerLifecycle !== "suspended" || diagnostic.runnerSettled !== true)) ||
(!exitedBeforeSessionOpen && !freshBootstrap &&
(diagnostic.runnerLifecycle !== "suspended" || diagnostic.runnerSettled !== true)) ||
target?.kind !== "remote" || target.transport !== "sandbox" || !target.runner) throw error;
const state = await readRemoteRunnerState({ runner: target.runner,
stateDirectory: remoteRunnerStateDirectory(target.remoteCwd, input.normalizedSessionId), recoveryDeadline, stall });
if (REMOTE_RUNNER_IDENTITY_FIELDS.some(field => state[field] !== input.identity[field])) throw error;
if (exitedBeforeSessionOpen ? !runnerStateAllowsFreshStart(state)
if (freshBootstrap ? !runnerStateRequestsFreshBootstrap(state) : exitedBeforeSessionOpen ? !runnerStateAllowsFreshStart(state)
: state.lifecycle !== "suspended" || !describeExitedRemoteRunner(state).runnerSettled) throw error;
return undefined;
}
@@ -17177,6 +17182,13 @@ async function createRunnerdBackendWithinSessionClaim(
};
};
const retainsFreshBootstrapRunnerState = async (): Promise<boolean> => {
if (!remoteCommandRunner || !remoteStateDirectory || !durableIdentity) return false;
const runnerState = await readRemoteRunnerState({ runner: remoteCommandRunner, stateDirectory: remoteStateDirectory });
return runnerStateRequestsFreshBootstrap(runnerState)
&& REMOTE_RUNNER_IDENTITY_FIELDS.every(field => runnerState[field] === durableIdentity[field]);
};
const claimUntouchedRemoteSession = async (): Promise<boolean> => {
if (!remoteCommandRunner || !remoteRuntimeRoot || !remoteSessionRoot) return false;
const controllerState = runnerControlPlane?.store.state;
@@ -17470,6 +17482,10 @@ async function createRunnerdBackendWithinSessionClaim(
throw new RemoteRunnerRecoveryError({ stage: "process_verification", reason: "process_not_running" });
}
});
} else if (sandboxLeaseAcquisition?.outcome === "resumed" && !adoptedProcess && !stagedRecovery &&
input.restartRecovery?.kind === "reattach_remote_runner" &&
await measureNativeRunnerSpan(input.trace, "sandbox.lease.resume", retainsFreshBootstrapRunnerState)) {
await input.onLog?.("stdout", "[paperclip-runner] Starting a new runner from the saved state of the runner whose connection lease ended.\n");
} else if (sandboxLeaseAcquisition?.outcome === "resumed") {
const reuseStartedAtMs = Date.now();
const state = await measureNativeRunnerSpan(
@@ -90,6 +90,29 @@ export function describeExitedRemoteRunner(state: Record<string, unknown>): Pick
};
}
/**
* The Runner exits with these codes after it stops its provider and saves its
* state, because only a new process with a fresh bootstrap ticket can
* reconnect. The replacement continues from the same cursor.
*/
const FRESH_BOOTSTRAP_RECOVERABLE_FAILURES = new Set([
"lease_expired_requires_bootstrap", "transport_reconnect_deadline_exceeded", "transport_reconnect_grace_exceeded",
]);
export function exitedRunnerRequestedFreshBootstrap(
diagnostic: Pick<RemoteRunnerRecoveryDiagnostic, "runnerLifecycle" | "runnerRecoverableFailure">,
): boolean {
return diagnostic.runnerLifecycle === "recoverable_failure"
&& FRESH_BOOTSTRAP_RECOVERABLE_FAILURES.has(diagnostic.runnerRecoverableFailure ?? "");
}
/** The saved state of an exited Runner that asked for a fresh bootstrap, re-read after its exit. */
export function runnerStateRequestsFreshBootstrap(state: Record<string, unknown>): boolean {
return exitedRunnerRequestedFreshBootstrap({ ...describeExitedRemoteRunner(state),
...describeExitedRemoteRunnerReason(state, text => text) })
&& state.pendingTerminalDelivery == null && state.warmTransition == null;
}
const transportDiagnostics = new WeakMap<object, RemoteRunnerRecoveryDiagnostic>();
export function attachRemoteRunnerRecoveryDiagnostic(error: unknown, diagnostic: RemoteRunnerRecoveryDiagnostic): unknown {