fix(native): keep a slow but live Runner on retry instead of replacing it (#21462)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a recovery read to a remote Runner goes unanswered, the next
attempt of the same run carries `replaceUnansweredRunner`
> - That attempt verified the Runner was alive and then refused to adopt
it anyway. On a resumed sandbox lease the reuse check then failed with
`resumed_not_suspended` and left the Runner running. Otherwise a second
Runner was launched, and its launch fence killed the busy Runner, which
lost the provider turn.
> - After a late reconnect a Runner is often slow, not dead: it is still
replaying the events it saved with no controller attached
> - This pull request makes the attempt check progress first. It adopts
a live Runner that is making progress and replaces only a Runner that
made no progress.
> - The benefit is that a slow but working Runner keeps its turn, and
the retry no longer causes the failure it was meant to fix.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13. The retry after a
first-attempt timeout failed with these codes (counts from the traced
sample of 136 failures):

- `runner_ingress_unavailable: runnerd process failed:
runner_harness_state_mismatch: resumed_not_suspended` (part of 40
`ingress` failures)
- `runner_remote_process_identity_unavailable` (25): the Runner was gone
by the time the retry checked, because the launch fence or a previous
retry had killed it
- "restored the conversation after process loss, but…" /
`provider_turn_lost_on_restore` (10): the replacement Runner restored
from the old state and the provider turn was gone

Runs that crossed a swap with a reconnect gap under 150 s: 0 of 9
failed. Over 420 s: 14 of 15 failed. The process ID of the Runner
changed inside the same sandbox in the failed runs, which shows the
replacement.

Code path: `nativeRecoveryCommandUnanswered` grants an extra attempt
with `replaceUnansweredRunner: true`. `verifyRemoteRunnerOrSettledExit`
verifies the Runner is alive and then returns `undefined` because of
that flag, so `adoptedProcess` is `undefined`.

**Expected behavior**

A live Runner that is making progress is adopted. A Runner that made no
progress is stopped with proof and then restored from its own journal.

**Steps to reproduce**

1. Reattach a remote run whose previous attempt left a recovery read
unanswered (`replaceUnansweredRunner: true`), with the Runner alive and
its saved event sequence past the controller's last acknowledgement.
2. On master the attempt launches a second Runner, or fails with
`resumed_not_suspended` on a resumed sandbox lease.
3. The new tests in `native-session-executor.test.ts` and
`native-runner-restart-recovery.integration.test.ts` do this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

- `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): with
`replaceUnansweredRunner`, the attempt adopts the live Runner, launching
nothing and signalling nothing, when:
- the Runner's saved event sequence is past the controller's last
acknowledgement, or
- its saved state (event and command sequences, generation, lifecycle)
changes within a bounded wait of about 10 s.
- The baseline is the first successful state read; every successful read
is also compared against the controller's acknowledgement. A Runner
whose state cannot be read at all during the wait is adopted, not
replaced: missing evidence is not proof of a wedge. A Runner whose
identity changes during the wait fails closed.
- A settled exit during the wait keeps the attempt allowance that
admitted the replacement, so it also restores on the extra fourth
attempt.
- Only a Runner that made no progress is replaced. The relaunch's
existing launch fence stops it and its process lineage with proof, and
the new Runner restores its own journal. A resumed sandbox lease no
longer refuses that relaunch with `resumed_not_suspended`.
- A Runner that exits during the wait follows the existing exited-Runner
rules: a settled suspension restores, and a mid-turn exit stays
fail-closed.
- The wait runs only on this recovery path, never on a normal run start.
- `doc/execution-semantics.md`: describes adopt versus replace.
- The claim flag `replaceUnansweredRunner` is set in claim logic that
another session owns. This PR does not change it; it changes only what
the executor does with it.

## Verification

Red on master, green on this branch:

- `native-session-executor.test.ts`
- `adopts a live Runner that emitted events past the controller's last
acknowledgement`
- `adopts a live Runner whose saved state advances within the bounded
wait`
- `hands a live Runner that made no progress to the relaunch, whose
launch fence stops it, without signalling it here`
- `restores from the journal of a Runner that suspended and exited
during the wait`
- `stays fail-closed for a Runner that exited mid-turn during the wait`
- `native-runner-restart-recovery.integration.test.ts`: `reattaches a
live %s Runner whose recovery read went unanswered on a resumed sandbox
lease` (progressing, wedged)
- Review fixes, `native-runner-restart-recovery.integration.test.ts`
(the first four were red before the fix):
- `adopts a live Runner whose first state read failed and whose later
reads advance`
- `adopts a live Runner whose first readable state is already past the
controller's acknowledgement`
- `does not replace a live Runner whose state could not be read at all
during the wait`
- `restores from the journal of a Runner that suspended and exited
during the wait on the admitted fourth attempt`
- `fails closed when the Runner's identity changes during the wait`
(pins existing behavior)
- `native-remote-retry-admission.test.ts`: unchanged, still passes.

Local runs on this branch: `tsc --noEmit` is clean for `server` on this
branch (master c6b497d831, which contains #21442).
`native-runner-restart-recovery.integration.test.ts`, the
adopt-or-replace block: 12 of 12 pass. The full
`native-session-executor.test.ts` and
`native-remote-retry-admission.test.ts` runs did not finish on this
shared machine (load 20 to 96); CI ran them on this head and they
passed.

Conflict resolved: master added a relaunch branch for a Runner that
exited for a fresh bootstrap (#21374) at the same place. Both branches
are kept; the cheaper check for an unanswered Runner that made no
progress comes first.

Daytona results that exist. They ran on a combined branch with this PR,
#21450, #21442 (merged) and #21362 (merged), so they cannot be credited
to this PR alone:

- G5 baseline (master d1d71a7755; 480 s delay injected on the first
identity write per run): one Claude run reproduced the staging chain:
authentication 419 ms, identity +423.5 s, attempt 1 `native session
provider recovery timed out after 900000ms`, then the retry failed in 6
s with `runner_harness_state_mismatch: resumed_not_suspended`.
- G5c (same cell, combined branch 588c6ba267): 6 of 6 tasks finished
hands-off. The chain did not recur. All runs that crossed swap 2 were
adopted on the same Runner process ID.
- Caveat: the adopt-or-replace retry path that this PR changes was never
exercised in a cell. In G5c no first attempt failed, because all
identity writes started together 4 to 5 s after the claim. G5c also ran
before the phase timing fix in #21450.

## Risks

- Not proven on Daytona: the adopt-or-replace path itself (see the
caveat above).
- Without #21442 (merged) an adopted Runner can still time out on its
first command while it replays a backlog. This PR stops the replacement;
#21442 (merged) makes the adopted Runner answer.
- A Runner whose state cannot be read during the whole wait is adopted.
If it is in fact wedged, the adopted attempt times out again and the
attempt limit bounds it.
- A wedged Runner that still emits a few events counts as progressing
and is adopted. The existing attempt limit bounds this.
- The progress check adds a wait of up to about 10 s on this recovery
path only. A normal run start has no added wait.
- The claim construction (`native-restart-recovery.ts`) is owned by
another session and is unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 18:56:11 -06:00
1 parent 47a2c82d35
commit 2178c3df88
4 files changed
+300 -5

No files matched your search

@@ -1644,6 +1644,150 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
.toBe(unanswered ? true : undefined);
});
describe("a same-run reattachment claimed to replace a Runner that left its recovery read unanswered", () => {
async function replacementClaim(label: string) {
const fixture = await seedRun(`remote-unanswered-progress-${label}`);
await fixture.db.update(heartbeatRuns).set({ processPid: null, runnerProfileJson: {
nativeWorkspaceSync: remotePreparedWorkspace(),
sessionCheckpoint: { identity: { companyId, agentId, issueId: fixture.issueId, runId: fixture.runId,
sessionId: fixture.native.normalizedSessionId }, providerSessionId: "provider-session" },
} }).where(eq(heartbeatRuns.id, fixture.runId));
await fixture.db.update(nativeRunFinalizations).set({ attempt: 1, failureDetail: {
message: "native_session_recovery_failed: Error: runner_ingress_unavailable: PRP command runner.drain timed out (runner connected; pending ahead 0; last runner event seq 338 16s ago)",
recoveryMode: "exact_checkpoint_resume", checkpointExists: true, adoptedRunnerUnanswered: true,
} }).where(eq(nativeRunFinalizations.runId, fixture.runId));
const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
restartKind: "graceful", runIds: [fixture.runId] });
if (claim?.kind !== "reattach_remote_runner" || claim.replaceUnansweredRunner !== true) {
throw new Error(`expected a replacement claim, got ${JSON.stringify(claim)}`);
}
const identity = { runId: fixture.runId, normalizedSessionId: fixture.native.normalizedSessionId,
runnerInstanceId: "unanswered-runner", environmentLeaseId: "lease", turnId: "turn", itemId: "item" };
return { claim, identity };
}
function sandbox(providerLeaseId: string, identity: Record<string, unknown>, runnerState: (read: number) => Record<string, unknown> | null,
alive: (check: number) => boolean) {
let reads = 0, checks = 0;
const execute = vi.fn(async (request: { args: string[] }) => {
const script = request.args[1] ?? "";
if (request.args[2] === "paperclip-runner-recovery-check") {
return { stdout: "", stderr: "", exitCode: alive(checks++) ? 0 : 45, timedOut: false };
}
if (script.includes("base64")) {
const state = runnerState(reads++);
if (state === null) return { stdout: "", stderr: "state read failed", exitCode: 1, timedOut: false };
return { stdout: Buffer.from(JSON.stringify({ ...identity, outbox: [], pendingTerminalDelivery: null, pendingProviderCleanup: null,
...state })).toString("base64"), stderr: "", exitCode: 0, timedOut: false };
}
if (script.includes("cat --")) {
return { stdout: `nonce\n4242\n2026-11-13T09:00:00.000Z\n${String(identity.runnerInstanceId)}\nlinux:abcd-1234:100\n`,
stderr: "", exitCode: 0, timedOut: false };
}
return { stdout: "", stderr: "", exitCode: 0, timedOut: false };
});
const target = { kind: "remote" as const, transport: "sandbox" as const, remoteCwd: "/workspace", providerKey: "daytona",
leaseId: "lease", sandboxLeaseAcquisition: { outcome: "resumed" as const, providerLeaseId }, runner: { execute } };
const signalled = () => execute.mock.calls.some(([request]) => /kill -/.test(request.args.join(" ")));
return { target, signalled };
}
async function reattach(label: string, runnerState: (read: number) => Record<string, unknown> | null,
alive: (check: number) => boolean = () => true, controllerAckedSourceSeq: number | null = 338, providerAttempt?: number) {
const { claim, identity } = await replacementClaim(label);
if (claim.remote.kind === "plugin_task") throw new Error("expected a sandbox claim");
const { target, signalled } = sandbox(claim.remote.providerLeaseId, identity, runnerState, alive);
const onReplace = vi.fn();
const outcome = await verifyRemoteRunnerOrSettledExit({ claim, target: target as never, identity, runId: identity.runId,
normalizedSessionId: identity.normalizedSessionId, providerAttempt: providerAttempt ?? claim.providerAttempt,
unansweredRunnerProgress: { controllerAckedSourceSeq, onReplace, windowMs: 200, pollMs: 20 } }).catch(error => error as Error);
return { outcome, onReplace, signalled };
}
it("adopts a live Runner that emitted events past the controller's last acknowledgement", async () => {
const { outcome, onReplace, signalled } = await reattach("emitted",
() => ({ lifecycle: "turn_active", nextSourceSeq: 2345, ackedSourceSeq: 338, lastControllerCommandSeq: 40 }));
expect(outcome).toMatchObject({ pid: 4242 });
expect(onReplace).not.toHaveBeenCalled();
expect(signalled()).toBe(false);
});
it("adopts a live Runner whose saved state advances within the bounded wait", async () => {
const { outcome, onReplace, signalled } = await reattach("advancing",
read => ({ lifecycle: "turn_active", nextSourceSeq: 339 + read, ackedSourceSeq: 338, lastControllerCommandSeq: 40 }),
() => true, null);
expect(outcome).toMatchObject({ pid: 4242 });
expect(onReplace).not.toHaveBeenCalled();
expect(signalled()).toBe(false);
});
it("hands a live Runner that made no progress to the relaunch, whose launch fence stops it, without signalling it here", async () => {
const { outcome, onReplace, signalled } = await reattach("wedged",
() => ({ lifecycle: "turn_active", nextSourceSeq: 339, ackedSourceSeq: 338, lastControllerCommandSeq: 40 }));
expect(outcome).toBeUndefined();
expect(onReplace).toHaveBeenCalledOnce();
expect(signalled()).toBe(false);
});
it("restores from the journal of a Runner that suspended and exited during the wait", async () => {
const { outcome, onReplace } = await reattach("suspended-exit",
read => read === 0 ? { lifecycle: "turn_active", nextSourceSeq: 339, ackedSourceSeq: 338, lastControllerCommandSeq: 40 }
: { lifecycle: "suspended", nextSourceSeq: 339, ackedSourceSeq: 339, lastControllerCommandSeq: 40 },
check => check === 0);
expect(outcome).toBeUndefined();
expect(onReplace).not.toHaveBeenCalled();
});
it("adopts a live Runner whose first state read failed and whose later reads advance", async () => {
const { outcome, onReplace, signalled } = await reattach("first-read-failed",
read => read === 1 ? null : { lifecycle: "turn_active", nextSourceSeq: 339 + read, ackedSourceSeq: 338, lastControllerCommandSeq: 40 },
() => true, null);
expect(outcome).toMatchObject({ pid: 4242 });
expect(onReplace).not.toHaveBeenCalled();
expect(signalled()).toBe(false);
});
it("adopts a live Runner whose first readable state is already past the controller's acknowledgement", async () => {
const { outcome, onReplace } = await reattach("first-read-failed-emitted",
read => read === 0 ? { lifecycle: "turn_active", nextSourceSeq: 339, ackedSourceSeq: 338, lastControllerCommandSeq: 40 }
: read === 1 ? null : { lifecycle: "turn_active", nextSourceSeq: 2345, ackedSourceSeq: 338, lastControllerCommandSeq: 40 });
expect(outcome).toMatchObject({ pid: 4242 });
expect(onReplace).not.toHaveBeenCalled();
});
it("does not replace a live Runner whose state could not be read at all during the wait", async () => {
const { outcome, onReplace, signalled } = await reattach("unreadable",
read => read === 0 ? { lifecycle: "turn_active", nextSourceSeq: 339, ackedSourceSeq: 338, lastControllerCommandSeq: 40 } : null);
expect(outcome).toMatchObject({ pid: 4242 });
expect(onReplace).not.toHaveBeenCalled();
expect(signalled()).toBe(false);
});
it("fails closed when the Runner's identity changes during the wait", async () => {
const { outcome, onReplace } = await reattach("identity-changed",
read => read === 0 ? { lifecycle: "turn_active", nextSourceSeq: 339, ackedSourceSeq: 338, lastControllerCommandSeq: 40 }
: { lifecycle: "turn_active", nextSourceSeq: 339, ackedSourceSeq: 338, lastControllerCommandSeq: 40, runnerInstanceId: "another-runner" });
expect(outcome).toBeInstanceOf(Error);
expect((outcome as Error).message).toContain("runner_remote_recovery_identity_mismatch");
expect(onReplace).not.toHaveBeenCalled();
});
it("restores from the journal of a Runner that suspended and exited during the wait on the admitted fourth attempt", async () => {
const { outcome, onReplace } = await reattach("suspended-exit-fourth",
read => read === 0 ? { lifecycle: "turn_active", nextSourceSeq: 339, ackedSourceSeq: 338, lastControllerCommandSeq: 40 }
: { lifecycle: "suspended", nextSourceSeq: 339, ackedSourceSeq: 339, lastControllerCommandSeq: 40 },
check => check === 0, 338, 4);
expect(outcome).toBeUndefined();
expect(onReplace).not.toHaveBeenCalled();
});
it("stays fail-closed for a Runner that exited mid-turn during the wait", async () => {
const { outcome, onReplace } = await reattach("midturn-exit",
() => ({ lifecycle: "turn_active", nextSourceSeq: 339, ackedSourceSeq: 338, lastControllerCommandSeq: 40 }),
check => check === 0);
expect(outcome).toBeInstanceOf(Error);
expect(onReplace).not.toHaveBeenCalled();
});
});
it("keeps a restart recovery's attempt credit while the agent directory is busy, then resumes at attempt 3", async () => {
const fixture = await seedRun("remote-busy-directory");
await fixture.db.update(heartbeatRuns).set({
@@ -18306,6 +18306,50 @@ describe("runnerd provider runtime wiring", () => {
expect(existsSync(join(root, "control-plane", "control-plane-state.json"))).toBe(true);
});
it.each(["progressing", "wedged"] as const)("reattaches a live %s Runner whose recovery read went unanswered on a resumed sandbox lease", async mode => {
const current = scopedNativeExecution(`unanswered-live-runner-${mode}`);
const identity = { runId: current.binding.runId, normalizedSessionId: current.session.normalizedSessionId!,
runnerInstanceId: "unanswered-codex", environmentLeaseId: "unanswered-lease",
turnId: `turn_attachment_${randomUUID()}`, itemId: `item_attachment_${randomUUID()}` };
const root = await initializeScopedRunnerRoot(current, identity.runnerInstanceId);
await mkdir(join(root, "control-plane"), { recursive: true });
await writeFile(join(root, "control-plane", "control-plane-state.json"),
JSON.stringify({ ...durableControlPlaneState(identity), ackedSourceSeq: 2204 }));
const execute = vi.fn(async (request: { args: string[] }) => {
const script = request.args[1] ?? "";
return { exitCode: 0, timedOut: false, stderr: "",
stdout: request.args[0] === "--build-metadata" ? JSON.stringify({
schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd",
packageName: "@paperclipai/paperclip-runner", binaryContractVersion: 2,
durableSessionCapabilities: ["unlimited_runtime", "connection_lease_renewal"], prpTransportModes: ["listen_ws"],
}) : request.args[0] === "--version" ? "codex-cli 0.160.0" : script.includes("base64")
? Buffer.from(JSON.stringify({ ...identity, lifecycle: "turn_active", outbox: [{ id: 2205 }], pendingTerminalDelivery: null,
nextSourceSeq: mode === "progressing" ? 2345 : 2205, ackedSourceSeq: 2204, lastControllerCommandSeq: 61 })).toString("base64")
: script.includes("cat --") ? `nonce\n1806\n2026-11-13T08:30:36.187Z\n${identity.runnerInstanceId}\nlinux:abcd-1234:100\n` : "" };
});
state.createBackend.mockClear(); state.createTransport.mockClear();
const logs: string[] = [];
await createRunnerdBackend({ db: leaseDb(current), execution: current, runnerInstanceId: identity.runnerInstanceId,
runnerIngressAuthorized: true, onLog: async (_stream, chunk) => { logs.push(chunk); },
controllerPersistence: { bootId: "successor", runnerArtifact: { version: "admitted-version", digest: `sha256:${"a".repeat(64)}` } },
runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace", environmentId: "environment",
leaseId: identity.environmentLeaseId, providerKey: "daytona", runner: { execute },
effectiveCapabilities: { runnerWebSocketIngress: true },
sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "sandbox" } } as never,
restartRecovery: { kind: "reattach_remote_runner", runId: current.binding.runId, leaseOwner: "controller",
controllerGeneration: 6, providerAttempt: 2, restartKind: "hard", recoveryRequestId: null,
remote: { providerLeaseId: "sandbox", remoteCwd: "/workspace" }, replaceUnansweredRunner: true },
});
state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
const options = state.createTransport.mock.calls.at(-1)![0] as RunnerTransportOptions & { prepareExternalRunnerState: () => Promise<void> };
if (mode === "progressing") expect(options.adoptExistingRunner).toMatchObject({ pid: 1806 });
else expect(options.adoptExistingRunner).toBeUndefined();
await expect(options.prepareExternalRunnerState()).resolves.toBeUndefined();
expect(execute.mock.calls.some(([request]) => /kill -/.test(request.args.join(" ")))).toBe(false);
expect(execute.mock.calls.some(([request]) => request.args.includes("paperclip-runner-launch"))).toBe(false);
expect(logs.some(line => line.includes("made no progress"))).toBe(mode === "wedged");
}, 30_000);
it.each(["failed", "timeout", "succeeded"] as const)("retains the local controller checkpoint until remote continuity archiving succeeds: %s", async mode => {
const current = scopedNativeExecution("failed-remote-continuity-archive");
state.createBackend.mockClear(); state.createTransport.mockClear();
@@ -7124,6 +7124,18 @@ type RunnerdDurableIdentity = Record<string, unknown> & {
environmentLeaseId: string;
};
function readRunnerdControllerAckedSourceSeq(root: string): number | null {
const statePath = resolve(root, "control-plane", "control-plane-state.json");
if (!isSafeNativeStateDirectory(root) || !existsSync(statePath)) return null;
try {
const acked = record(JSON.parse(readBoundedNativeFile(statePath, NATIVE_CONTROL_PLANE_STATE_MAX_BYTES,
"runner_durable_identity_too_large").toString("utf8"))).ackedSourceSeq;
return typeof acked === "number" && Number.isSafeInteger(acked) && acked >= 0 ? acked : null;
} catch {
return null;
}
}
function readRunnerdDurableIdentity(
root: string,
allowExternal = false,
@@ -15388,10 +15400,68 @@ async function readRemoteRunnerExitEvidence(runner: CommandManagedRuntimeRunner,
}
}
export interface UnansweredRunnerProgressProbe {
controllerAckedSourceSeq: number | null;
onReplace: () => void;
windowMs?: number;
pollMs?: number;
}
const UNANSWERED_RUNNER_PROGRESS_WINDOW_MS = 10_000;
const UNANSWERED_RUNNER_PROGRESS_POLL_MS = 2_000;
async function observeUnansweredRunnerProgress(input: {
runner: CommandManagedRuntimeRunner;
stateDirectory: string;
identity: Record<string, unknown>;
probe: UnansweredRunnerProgressProbe;
isAlive: () => Promise<boolean>;
recoveryDeadline: number;
stall: RemoteRunnerPluginStall;
}): Promise<"progressing" | "wedged" | "exited"> {
const read = async () => {
const state = await readRemoteRunnerState({ runner: input.runner, stateDirectory: input.stateDirectory,
recoveryDeadline: input.recoveryDeadline, stall: input.stall }).catch(() => null);
if (state && REMOTE_RUNNER_IDENTITY_FIELDS.some(field => state[field] !== input.identity[field])) {
throw new Error("runner_remote_recovery_identity_mismatch");
}
return state;
};
const progressMarker = (state: Record<string, unknown> | null) => state ? JSON.stringify([state.generation ?? null,
state.reconnectCount ?? null, state.nextSourceSeq ?? null, state.ackedSourceSeq ?? null,
state.lastControllerCommandSeq ?? null, state.lifecycle ?? null]) : null;
const settle = async () => await input.isAlive() ? "progressing" as const : "exited" as const;
const emittedPastAck = (state: Record<string, unknown>) => {
const emittedSourceSeq = Number(state.nextSourceSeq) - 1;
return input.probe.controllerAckedSourceSeq !== null && Number.isSafeInteger(emittedSourceSeq)
&& emittedSourceSeq > input.probe.controllerAckedSourceSeq;
};
let baseline: string | null = null;
const progressed = (state: Record<string, unknown> | null) => {
if (!state) return false;
if (emittedPastAck(state)) return true;
const marker = progressMarker(state);
if (baseline === null) {
baseline = marker;
return false;
}
return marker !== baseline;
};
if (progressed(await read())) return settle();
const end = Math.min(Date.now() + (input.probe.windowMs ?? UNANSWERED_RUNNER_PROGRESS_WINDOW_MS), input.recoveryDeadline);
while (Date.now() < end) {
await new Promise(done => setTimeout(done, Math.min(input.probe.pollMs ?? UNANSWERED_RUNNER_PROGRESS_POLL_MS, end - Date.now())));
if (progressed(await read())) return settle();
}
if (baseline === null) return settle();
return await input.isAlive() ? "wedged" : "exited";
}
export async function verifyRemoteRunnerOrSettledExit(
input: Parameters<typeof verifyRemoteRunnerReattachment>[0] & { providerAttempt: number;
claim: { exitedBeforeSessionOpen?: true; replaceUnansweredRunner?: true } },
) {
claim: { exitedBeforeSessionOpen?: true; replaceUnansweredRunner?: true };
unansweredRunnerProgress?: UnansweredRunnerProgressProbe; admittedAttemptLimit?: number },
): Promise<Awaited<ReturnType<typeof verifyRemoteRunnerReattachment>> | undefined> {
const recoveryDeadline = Math.min(input.recoveryDeadline ?? Number.POSITIVE_INFINITY,
Date.now() + REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS);
const stall = input.stall ?? remoteRunnerPluginStall(input.recoveryDeadline);
@@ -15406,8 +15476,8 @@ export async function verifyRemoteRunnerOrSettledExit(
const diagnostic = readRemoteRunnerRecoveryError(error);
const exitedBeforeSessionOpen = input.claim.exitedBeforeSessionOpen === true;
const freshBootstrap = diagnostic !== null && exitedRunnerRequestedFreshBootstrap(diagnostic);
const attemptLimit = freshBootstrap && input.claim.replaceUnansweredRunner === true
? NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT + 1 : NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT;
const attemptLimit = Math.max(input.admittedAttemptLimit ?? 0, freshBootstrap && input.claim.replaceUnansweredRunner === true
? NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT + 1 : NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT);
if (input.providerAttempt > attemptLimit || diagnostic?.stage !== "process_verification" ||
diagnostic.reason !== "process_not_running" ||
(!exitedBeforeSessionOpen && !freshBootstrap &&
@@ -15420,7 +15490,20 @@ export async function verifyRemoteRunnerOrSettledExit(
: state.lifecycle !== "suspended" || !describeExitedRemoteRunner(state).runnerSettled) throw error;
return undefined;
}
return replaceUnanswered ? undefined : verified;
if (!replaceUnanswered) return verified;
if (!input.unansweredRunnerProgress) return undefined;
const progress = await observeUnansweredRunnerProgress({ runner: target.runner!,
stateDirectory: remoteRunnerStateDirectory(target.remoteCwd, input.normalizedSessionId), identity: input.identity,
probe: input.unansweredRunnerProgress, isAlive: () => verified.isAlive(), stall,
recoveryDeadline: Math.min(input.recoveryDeadline ?? Number.POSITIVE_INFINITY, Date.now() + REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS) });
if (progress === "progressing") return verified;
if (progress === "exited") {
const { replaceUnansweredRunner: _replace, ...claim } = input.claim;
return verifyRemoteRunnerOrSettledExit({ ...input, claim, unansweredRunnerProgress: undefined,
admittedAttemptLimit: NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT + 1 });
}
input.unansweredRunnerProgress.onReplace();
return undefined;
}
function runnerStateAllowsFreshStart(state: Record<string, unknown>): boolean {
@@ -16778,6 +16861,7 @@ async function createRunnerdBackendWithinSessionClaim(
// exited. The transport then relaunches it from durable state, which needs
// the admitted provider pack's launch inputs that adoption does not.
let relaunchesSettledRemoteRunner = false;
let replacesUnansweredLiveRunner = false;
const explicitRemoteCodex = input.runnerRemoteCodexPath?.trim() || null;
const remoteCodexNpmSpec = input.runnerRemoteCodexNpmSpec?.trim() || null;
if (explicitRemoteCodex && remoteCodexNpmSpec) {
@@ -17515,6 +17599,9 @@ async function createRunnerdBackendWithinSessionClaim(
throw new RemoteRunnerRecoveryError({ stage: "process_verification", reason: "process_not_running" });
}
});
} else if (sandboxLeaseAcquisition?.outcome === "resumed" && replacesUnansweredLiveRunner) {
await input.onLog?.("stdout", "[paperclip-runner] The runner left its recovery read unanswered and made no progress; "
+ "the relaunch stops it and its process lineage, then restores its saved journal.\n");
} else if (sandboxLeaseAcquisition?.outcome === "resumed" && !adoptedProcess && !stagedRecovery &&
input.restartRecovery?.kind === "reattach_remote_runner" &&
await measureNativeRunnerSpan(input.trace, "sandbox.lease.resume", retainsFreshBootstrapRunnerState)) {
@@ -18170,6 +18257,10 @@ async function createRunnerdBackendWithinSessionClaim(
runId: input.execution.binding.runId,
normalizedSessionId: nativeSessionKey(input.execution),
providerAttempt: input.restartRecovery.providerAttempt,
unansweredRunnerProgress: {
controllerAckedSourceSeq: readRunnerdControllerAckedSourceSeq(root),
onReplace: () => { replacesUnansweredLiveRunner = true; },
},
})
: target.kind === "local" && localRecoveryProcess
? {