fix(native): keep a slow but live Runner on retry instead of replacing it (#21462)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a recovery read to a remote Runner goes unanswered, the next
attempt of the same run carries `replaceUnansweredRunner`
> - That attempt verified the Runner was alive and then refused to adopt
it anyway. On a resumed sandbox lease the reuse check then failed with
`resumed_not_suspended` and left the Runner running. Otherwise a second
Runner was launched, and its launch fence killed the busy Runner, which
lost the provider turn.
> - After a late reconnect a Runner is often slow, not dead: it is still
replaying the events it saved with no controller attached
> - This pull request makes the attempt check progress first. It adopts
a live Runner that is making progress and replaces only a Runner that
made no progress.
> - The benefit is that a slow but working Runner keeps its turn, and
the retry no longer causes the failure it was meant to fix.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13. The retry after a
first-attempt timeout failed with these codes (counts from the traced
sample of 136 failures):

- `runner_ingress_unavailable: runnerd process failed:
runner_harness_state_mismatch: resumed_not_suspended` (part of 40
`ingress` failures)
- `runner_remote_process_identity_unavailable` (25): the Runner was gone
by the time the retry checked, because the launch fence or a previous
retry had killed it
- "restored the conversation after process loss, but…" /
`provider_turn_lost_on_restore` (10): the replacement Runner restored
from the old state and the provider turn was gone

Runs that crossed a swap with a reconnect gap under 150 s: 0 of 9
failed. Over 420 s: 14 of 15 failed. The process ID of the Runner
changed inside the same sandbox in the failed runs, which shows the
replacement.

Code path: `nativeRecoveryCommandUnanswered` grants an extra attempt
with `replaceUnansweredRunner: true`. `verifyRemoteRunnerOrSettledExit`
verifies the Runner is alive and then returns `undefined` because of
that flag, so `adoptedProcess` is `undefined`.

**Expected behavior**

A live Runner that is making progress is adopted. A Runner that made no
progress is stopped with proof and then restored from its own journal.

**Steps to reproduce**

1. Reattach a remote run whose previous attempt left a recovery read
unanswered (`replaceUnansweredRunner: true`), with the Runner alive and
its saved event sequence past the controller's last acknowledgement.
2. On master the attempt launches a second Runner, or fails with
`resumed_not_suspended` on a resumed sandbox lease.
3. The new tests in `native-session-executor.test.ts` and
`native-runner-restart-recovery.integration.test.ts` do this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

- `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): with
`replaceUnansweredRunner`, the attempt adopts the live Runner, launching
nothing and signalling nothing, when:
- the Runner's saved event sequence is past the controller's last
acknowledgement, or
- its saved state (event and command sequences, generation, lifecycle)
changes within a bounded wait of about 10 s.
- The baseline is the first successful state read; every successful read
is also compared against the controller's acknowledgement. A Runner
whose state cannot be read at all during the wait is adopted, not
replaced: missing evidence is not proof of a wedge. A Runner whose
identity changes during the wait fails closed.
- A settled exit during the wait keeps the attempt allowance that
admitted the replacement, so it also restores on the extra fourth
attempt.
- Only a Runner that made no progress is replaced. The relaunch's
existing launch fence stops it and its process lineage with proof, and
the new Runner restores its own journal. A resumed sandbox lease no
longer refuses that relaunch with `resumed_not_suspended`.
- A Runner that exits during the wait follows the existing exited-Runner
rules: a settled suspension restores, and a mid-turn exit stays
fail-closed.
- The wait runs only on this recovery path, never on a normal run start.
- `doc/execution-semantics.md`: describes adopt versus replace.
- The claim flag `replaceUnansweredRunner` is set in claim logic that
another session owns. This PR does not change it; it changes only what
the executor does with it.

## Verification

Red on master, green on this branch:

- `native-session-executor.test.ts`
- `adopts a live Runner that emitted events past the controller's last
acknowledgement`
- `adopts a live Runner whose saved state advances within the bounded
wait`
- `hands a live Runner that made no progress to the relaunch, whose
launch fence stops it, without signalling it here`
- `restores from the journal of a Runner that suspended and exited
during the wait`
- `stays fail-closed for a Runner that exited mid-turn during the wait`
- `native-runner-restart-recovery.integration.test.ts`: `reattaches a
live %s Runner whose recovery read went unanswered on a resumed sandbox
lease` (progressing, wedged)
- Review fixes, `native-runner-restart-recovery.integration.test.ts`
(the first four were red before the fix):
- `adopts a live Runner whose first state read failed and whose later
reads advance`
- `adopts a live Runner whose first readable state is already past the
controller's acknowledgement`
- `does not replace a live Runner whose state could not be read at all
during the wait`
- `restores from the journal of a Runner that suspended and exited
during the wait on the admitted fourth attempt`
- `fails closed when the Runner's identity changes during the wait`
(pins existing behavior)
- `native-remote-retry-admission.test.ts`: unchanged, still passes.

Local runs on this branch: `tsc --noEmit` is clean for `server` on this
branch (master c6b497d831, which contains #21442).
`native-runner-restart-recovery.integration.test.ts`, the
adopt-or-replace block: 12 of 12 pass. The full
`native-session-executor.test.ts` and
`native-remote-retry-admission.test.ts` runs did not finish on this
shared machine (load 20 to 96); CI ran them on this head and they
passed.

Conflict resolved: master added a relaunch branch for a Runner that
exited for a fresh bootstrap (#21374) at the same place. Both branches
are kept; the cheaper check for an unanswered Runner that made no
progress comes first.

Daytona results that exist. They ran on a combined branch with this PR,
#21450, #21442 (merged) and #21362 (merged), so they cannot be credited
to this PR alone:

- G5 baseline (master d1d71a7755; 480 s delay injected on the first
identity write per run): one Claude run reproduced the staging chain:
authentication 419 ms, identity +423.5 s, attempt 1 `native session
provider recovery timed out after 900000ms`, then the retry failed in 6
s with `runner_harness_state_mismatch: resumed_not_suspended`.
- G5c (same cell, combined branch 588c6ba267): 6 of 6 tasks finished
hands-off. The chain did not recur. All runs that crossed swap 2 were
adopted on the same Runner process ID.
- Caveat: the adopt-or-replace retry path that this PR changes was never
exercised in a cell. In G5c no first attempt failed, because all
identity writes started together 4 to 5 s after the claim. G5c also ran
before the phase timing fix in #21450.

## Risks

- Not proven on Daytona: the adopt-or-replace path itself (see the
caveat above).
- Without #21442 (merged) an adopted Runner can still time out on its
first command while it replays a backlog. This PR stops the replacement;
#21442 (merged) makes the adopted Runner answer.
- A Runner whose state cannot be read during the whole wait is adopted.
If it is in fact wedged, the adopted attempt times out again and the
attempt limit bounds it.
- A wedged Runner that still emits a few events counts as progressing
and is adopted. The existing attempt limit bounds this.
- The progress check adds a wait of up to about 10 s on this recovery
path only. A normal run start has no added wait.
- The claim construction (`native-restart-recovery.ts`) is owned by
another session and is unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 18:56:11 -06:00
1 parent 47a2c82d35
commit 2178c3df88
4 files changed
+300 -5

No files matched your search

+16
View File
@@ -1860,6 +1860,22 @@ mid-turn can leave provider or tool processes that the fence cannot find, so it
keeps the fail-closed path below, and the task continues in a fresh session from
the saved work.
A slow answer is not a dead Runner. After a release swap the new server can reach
a live Runner late, while the Runner is still replaying the events it saved with
no controller attached; its first command then times out although the Runner is
working. So a `replaceUnansweredRunner` attempt first checks whether the live
Runner is making progress. It adopts the Runner, launching no second one and
signalling nothing, when the Runner's saved event sequence is past the
controller's last acknowledgement, or when its saved state (event and command
sequences, generation, lifecycle) changes within a bounded wait of about ten
seconds. Only a Runner that made no progress in that wait is replaced as above:
the relaunch's launch fence stops it and its process lineage with proof, and the
new Runner restores its own journal. This replacement also proceeds on a resumed
sandbox lease, where the in-place reuse check would otherwise refuse the live
Runner's unsuspended state (`runner_harness_state_mismatch: resumed_not_suspended`)
and leave it running. A Runner that exits during the wait follows the exited-Runner
rules: a settled suspension restores, a mid-turn exit stays fail-closed. A state read that fails is no evidence either way: the first successful read sets the baseline, and a Runner whose state could not be read at all during the wait is adopted, not replaced. A Runner whose identity changes during the wait fails closed. A settled exit during the wait restores on every attempt that admitted the replacement, including the extra fourth attempt.
On every `reattach_remote_runner` attempt, a turn the Runner reports as
`provider_turn_lost_on_restore` with `recoverable: true` continues once in the same
conversation, as on a local dead-runner resume. The Runner reports that cause only
@@ -1644,6 +1644,150 @@ process.on("SIGTERM", () => child.kill("SIGTERM"));
.toBe(unanswered ? true : undefined);
});
describe("a same-run reattachment claimed to replace a Runner that left its recovery read unanswered", () => {
async function replacementClaim(label: string) {
const fixture = await seedRun(`remote-unanswered-progress-${label}`);
await fixture.db.update(heartbeatRuns).set({ processPid: null, runnerProfileJson: {
nativeWorkspaceSync: remotePreparedWorkspace(),
sessionCheckpoint: { identity: { companyId, agentId, issueId: fixture.issueId, runId: fixture.runId,
sessionId: fixture.native.normalizedSessionId }, providerSessionId: "provider-session" },
} }).where(eq(heartbeatRuns.id, fixture.runId));
await fixture.db.update(nativeRunFinalizations).set({ attempt: 1, failureDetail: {
message: "native_session_recovery_failed: Error: runner_ingress_unavailable: PRP command runner.drain timed out (runner connected; pending ahead 0; last runner event seq 338 16s ago)",
recoveryMode: "exact_checkpoint_resume", checkpointExists: true, adoptedRunnerUnanswered: true,
} }).where(eq(nativeRunFinalizations.runId, fixture.runId));
const [claim] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor,
restartKind: "graceful", runIds: [fixture.runId] });
if (claim?.kind !== "reattach_remote_runner" || claim.replaceUnansweredRunner !== true) {
throw new Error(`expected a replacement claim, got ${JSON.stringify(claim)}`);
}
const identity = { runId: fixture.runId, normalizedSessionId: fixture.native.normalizedSessionId,
runnerInstanceId: "unanswered-runner", environmentLeaseId: "lease", turnId: "turn", itemId: "item" };
return { claim, identity };
}
function sandbox(providerLeaseId: string, identity: Record<string, unknown>, runnerState: (read: number) => Record<string, unknown> | null,
alive: (check: number) => boolean) {
let reads = 0, checks = 0;
const execute = vi.fn(async (request: { args: string[] }) => {
const script = request.args[1] ?? "";
if (request.args[2] === "paperclip-runner-recovery-check") {
return { stdout: "", stderr: "", exitCode: alive(checks++) ? 0 : 45, timedOut: false };
}
if (script.includes("base64")) {
const state = runnerState(reads++);
if (state === null) return { stdout: "", stderr: "state read failed", exitCode: 1, timedOut: false };
return { stdout: Buffer.from(JSON.stringify({ ...identity, outbox: [], pendingTerminalDelivery: null, pendingProviderCleanup: null,
...state })).toString("base64"), stderr: "", exitCode: 0, timedOut: false };
}
if (script.includes("cat --")) {
return { stdout: `nonce\n4242\n2026-11-13T09:00:00.000Z\n${String(identity.runnerInstanceId)}\nlinux:abcd-1234:100\n`,
stderr: "", exitCode: 0, timedOut: false };
}
return { stdout: "", stderr: "", exitCode: 0, timedOut: false };
});
const target = { kind: "remote" as const, transport: "sandbox" as const, remoteCwd: "/workspace", providerKey: "daytona",
leaseId: "lease", sandboxLeaseAcquisition: { outcome: "resumed" as const, providerLeaseId }, runner: { execute } };
const signalled = () => execute.mock.calls.some(([request]) => /kill -/.test(request.args.join(" ")));
return { target, signalled };
}
async function reattach(label: string, runnerState: (read: number) => Record<string, unknown> | null,
alive: (check: number) => boolean = () => true, controllerAckedSourceSeq: number | null = 338, providerAttempt?: number) {
const { claim, identity } = await replacementClaim(label);
if (claim.remote.kind === "plugin_task") throw new Error("expected a sandbox claim");
const { target, signalled } = sandbox(claim.remote.providerLeaseId, identity, runnerState, alive);
const onReplace = vi.fn();
const outcome = await verifyRemoteRunnerOrSettledExit({ claim, target: target as never, identity, runId: identity.runId,
normalizedSessionId: identity.normalizedSessionId, providerAttempt: providerAttempt ?? claim.providerAttempt,
unansweredRunnerProgress: { controllerAckedSourceSeq, onReplace, windowMs: 200, pollMs: 20 } }).catch(error => error as Error);
return { outcome, onReplace, signalled };
}
it("adopts a live Runner that emitted events past the controller's last acknowledgement", async () => {
const { outcome, onReplace, signalled } = await reattach("emitted",
() => ({ lifecycle: "turn_active", nextSourceSeq: 2345, ackedSourceSeq: 338, lastControllerCommandSeq: 40 }));
expect(outcome).toMatchObject({ pid: 4242 });
expect(onReplace).not.toHaveBeenCalled();
expect(signalled()).toBe(false);
});
it("adopts a live Runner whose saved state advances within the bounded wait", async () => {
const { outcome, onReplace, signalled } = await reattach("advancing",
read => ({ lifecycle: "turn_active", nextSourceSeq: 339 + read, ackedSourceSeq: 338, lastControllerCommandSeq: 40 }),
() => true, null);
expect(outcome).toMatchObject({ pid: 4242 });
expect(onReplace).not.toHaveBeenCalled();
expect(signalled()).toBe(false);
});
it("hands a live Runner that made no progress to the relaunch, whose launch fence stops it, without signalling it here", async () => {
const { outcome, onReplace, signalled } = await reattach("wedged",
() => ({ lifecycle: "turn_active", nextSourceSeq: 339, ackedSourceSeq: 338, lastControllerCommandSeq: 40 }));
expect(outcome).toBeUndefined();
expect(onReplace).toHaveBeenCalledOnce();
expect(signalled()).toBe(false);
});
it("restores from the journal of a Runner that suspended and exited during the wait", async () => {
const { outcome, onReplace } = await reattach("suspended-exit",
read => read === 0 ? { lifecycle: "turn_active", nextSourceSeq: 339, ackedSourceSeq: 338, lastControllerCommandSeq: 40 }
: { lifecycle: "suspended", nextSourceSeq: 339, ackedSourceSeq: 339, lastControllerCommandSeq: 40 },
check => check === 0);
expect(outcome).toBeUndefined();
expect(onReplace).not.toHaveBeenCalled();
});
it("adopts a live Runner whose first state read failed and whose later reads advance", async () => {
const { outcome, onReplace, signalled } = await reattach("first-read-failed",
read => read === 1 ? null : { lifecycle: "turn_active", nextSourceSeq: 339 + read, ackedSourceSeq: 338, lastControllerCommandSeq: 40 },
() => true, null);
expect(outcome).toMatchObject({ pid: 4242 });
expect(onReplace).not.toHaveBeenCalled();
expect(signalled()).toBe(false);
});
it("adopts a live Runner whose first readable state is already past the controller's acknowledgement", async () => {
const { outcome, onReplace } = await reattach("first-read-failed-emitted",
read => read === 0 ? { lifecycle: "turn_active", nextSourceSeq: 339, ackedSourceSeq: 338, lastControllerCommandSeq: 40 }
: read === 1 ? null : { lifecycle: "turn_active", nextSourceSeq: 2345, ackedSourceSeq: 338, lastControllerCommandSeq: 40 });
expect(outcome).toMatchObject({ pid: 4242 });
expect(onReplace).not.toHaveBeenCalled();
});
it("does not replace a live Runner whose state could not be read at all during the wait", async () => {
const { outcome, onReplace, signalled } = await reattach("unreadable",
read => read === 0 ? { lifecycle: "turn_active", nextSourceSeq: 339, ackedSourceSeq: 338, lastControllerCommandSeq: 40 } : null);
expect(outcome).toMatchObject({ pid: 4242 });
expect(onReplace).not.toHaveBeenCalled();
expect(signalled()).toBe(false);
});
it("fails closed when the Runner's identity changes during the wait", async () => {
const { outcome, onReplace } = await reattach("identity-changed",
read => read === 0 ? { lifecycle: "turn_active", nextSourceSeq: 339, ackedSourceSeq: 338, lastControllerCommandSeq: 40 }
: { lifecycle: "turn_active", nextSourceSeq: 339, ackedSourceSeq: 338, lastControllerCommandSeq: 40, runnerInstanceId: "another-runner" });
expect(outcome).toBeInstanceOf(Error);
expect((outcome as Error).message).toContain("runner_remote_recovery_identity_mismatch");
expect(onReplace).not.toHaveBeenCalled();
});
it("restores from the journal of a Runner that suspended and exited during the wait on the admitted fourth attempt", async () => {
const { outcome, onReplace } = await reattach("suspended-exit-fourth",
read => read === 0 ? { lifecycle: "turn_active", nextSourceSeq: 339, ackedSourceSeq: 338, lastControllerCommandSeq: 40 }
: { lifecycle: "suspended", nextSourceSeq: 339, ackedSourceSeq: 339, lastControllerCommandSeq: 40 },
check => check === 0, 338, 4);
expect(outcome).toBeUndefined();
expect(onReplace).not.toHaveBeenCalled();
});
it("stays fail-closed for a Runner that exited mid-turn during the wait", async () => {
const { outcome, onReplace } = await reattach("midturn-exit",
() => ({ lifecycle: "turn_active", nextSourceSeq: 339, ackedSourceSeq: 338, lastControllerCommandSeq: 40 }),
check => check === 0);
expect(outcome).toBeInstanceOf(Error);
expect(onReplace).not.toHaveBeenCalled();
});
});
it("keeps a restart recovery's attempt credit while the agent directory is busy, then resumes at attempt 3", async () => {
const fixture = await seedRun("remote-busy-directory");
await fixture.db.update(heartbeatRuns).set({
@@ -18306,6 +18306,50 @@ describe("runnerd provider runtime wiring", () => {
expect(existsSync(join(root, "control-plane", "control-plane-state.json"))).toBe(true);
});
it.each(["progressing", "wedged"] as const)("reattaches a live %s Runner whose recovery read went unanswered on a resumed sandbox lease", async mode => {
const current = scopedNativeExecution(`unanswered-live-runner-${mode}`);
const identity = { runId: current.binding.runId, normalizedSessionId: current.session.normalizedSessionId!,
runnerInstanceId: "unanswered-codex", environmentLeaseId: "unanswered-lease",
turnId: `turn_attachment_${randomUUID()}`, itemId: `item_attachment_${randomUUID()}` };
const root = await initializeScopedRunnerRoot(current, identity.runnerInstanceId);
await mkdir(join(root, "control-plane"), { recursive: true });
await writeFile(join(root, "control-plane", "control-plane-state.json"),
JSON.stringify({ ...durableControlPlaneState(identity), ackedSourceSeq: 2204 }));
const execute = vi.fn(async (request: { args: string[] }) => {
const script = request.args[1] ?? "";
return { exitCode: 0, timedOut: false, stderr: "",
stdout: request.args[0] === "--build-metadata" ? JSON.stringify({
schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd",
packageName: "@paperclipai/paperclip-runner", binaryContractVersion: 2,
durableSessionCapabilities: ["unlimited_runtime", "connection_lease_renewal"], prpTransportModes: ["listen_ws"],
}) : request.args[0] === "--version" ? "codex-cli 0.160.0" : script.includes("base64")
? Buffer.from(JSON.stringify({ ...identity, lifecycle: "turn_active", outbox: [{ id: 2205 }], pendingTerminalDelivery: null,
nextSourceSeq: mode === "progressing" ? 2345 : 2205, ackedSourceSeq: 2204, lastControllerCommandSeq: 61 })).toString("base64")
: script.includes("cat --") ? `nonce\n1806\n2026-11-13T08:30:36.187Z\n${identity.runnerInstanceId}\nlinux:abcd-1234:100\n` : "" };
});
state.createBackend.mockClear(); state.createTransport.mockClear();
const logs: string[] = [];
await createRunnerdBackend({ db: leaseDb(current), execution: current, runnerInstanceId: identity.runnerInstanceId,
runnerIngressAuthorized: true, onLog: async (_stream, chunk) => { logs.push(chunk); },
controllerPersistence: { bootId: "successor", runnerArtifact: { version: "admitted-version", digest: `sha256:${"a".repeat(64)}` } },
runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace", environmentId: "environment",
leaseId: identity.environmentLeaseId, providerKey: "daytona", runner: { execute },
effectiveCapabilities: { runnerWebSocketIngress: true },
sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "sandbox" } } as never,
restartRecovery: { kind: "reattach_remote_runner", runId: current.binding.runId, leaseOwner: "controller",
controllerGeneration: 6, providerAttempt: 2, restartKind: "hard", recoveryRequestId: null,
remote: { providerLeaseId: "sandbox", remoteCwd: "/workspace" }, replaceUnansweredRunner: true },
});
state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
const options = state.createTransport.mock.calls.at(-1)![0] as RunnerTransportOptions & { prepareExternalRunnerState: () => Promise<void> };
if (mode === "progressing") expect(options.adoptExistingRunner).toMatchObject({ pid: 1806 });
else expect(options.adoptExistingRunner).toBeUndefined();
await expect(options.prepareExternalRunnerState()).resolves.toBeUndefined();
expect(execute.mock.calls.some(([request]) => /kill -/.test(request.args.join(" ")))).toBe(false);
expect(execute.mock.calls.some(([request]) => request.args.includes("paperclip-runner-launch"))).toBe(false);
expect(logs.some(line => line.includes("made no progress"))).toBe(mode === "wedged");
}, 30_000);
it.each(["failed", "timeout", "succeeded"] as const)("retains the local controller checkpoint until remote continuity archiving succeeds: %s", async mode => {
const current = scopedNativeExecution("failed-remote-continuity-archive");
state.createBackend.mockClear(); state.createTransport.mockClear();
@@ -7124,6 +7124,18 @@ type RunnerdDurableIdentity = Record<string, unknown> & {
environmentLeaseId: string;
};
function readRunnerdControllerAckedSourceSeq(root: string): number | null {
const statePath = resolve(root, "control-plane", "control-plane-state.json");
if (!isSafeNativeStateDirectory(root) || !existsSync(statePath)) return null;
try {
const acked = record(JSON.parse(readBoundedNativeFile(statePath, NATIVE_CONTROL_PLANE_STATE_MAX_BYTES,
"runner_durable_identity_too_large").toString("utf8"))).ackedSourceSeq;
return typeof acked === "number" && Number.isSafeInteger(acked) && acked >= 0 ? acked : null;
} catch {
return null;
}
}
function readRunnerdDurableIdentity(
root: string,
allowExternal = false,
@@ -15388,10 +15400,68 @@ async function readRemoteRunnerExitEvidence(runner: CommandManagedRuntimeRunner,
}
}
export interface UnansweredRunnerProgressProbe {
controllerAckedSourceSeq: number | null;
onReplace: () => void;
windowMs?: number;
pollMs?: number;
}
const UNANSWERED_RUNNER_PROGRESS_WINDOW_MS = 10_000;
const UNANSWERED_RUNNER_PROGRESS_POLL_MS = 2_000;
async function observeUnansweredRunnerProgress(input: {
runner: CommandManagedRuntimeRunner;
stateDirectory: string;
identity: Record<string, unknown>;
probe: UnansweredRunnerProgressProbe;
isAlive: () => Promise<boolean>;
recoveryDeadline: number;
stall: RemoteRunnerPluginStall;
}): Promise<"progressing" | "wedged" | "exited"> {
const read = async () => {
const state = await readRemoteRunnerState({ runner: input.runner, stateDirectory: input.stateDirectory,
recoveryDeadline: input.recoveryDeadline, stall: input.stall }).catch(() => null);
if (state && REMOTE_RUNNER_IDENTITY_FIELDS.some(field => state[field] !== input.identity[field])) {
throw new Error("runner_remote_recovery_identity_mismatch");
}
return state;
};
const progressMarker = (state: Record<string, unknown> | null) => state ? JSON.stringify([state.generation ?? null,
state.reconnectCount ?? null, state.nextSourceSeq ?? null, state.ackedSourceSeq ?? null,
state.lastControllerCommandSeq ?? null, state.lifecycle ?? null]) : null;
const settle = async () => await input.isAlive() ? "progressing" as const : "exited" as const;
const emittedPastAck = (state: Record<string, unknown>) => {
const emittedSourceSeq = Number(state.nextSourceSeq) - 1;
return input.probe.controllerAckedSourceSeq !== null && Number.isSafeInteger(emittedSourceSeq)
&& emittedSourceSeq > input.probe.controllerAckedSourceSeq;
};
let baseline: string | null = null;
const progressed = (state: Record<string, unknown> | null) => {
if (!state) return false;
if (emittedPastAck(state)) return true;
const marker = progressMarker(state);
if (baseline === null) {
baseline = marker;
return false;
}
return marker !== baseline;
};
if (progressed(await read())) return settle();
const end = Math.min(Date.now() + (input.probe.windowMs ?? UNANSWERED_RUNNER_PROGRESS_WINDOW_MS), input.recoveryDeadline);
while (Date.now() < end) {
await new Promise(done => setTimeout(done, Math.min(input.probe.pollMs ?? UNANSWERED_RUNNER_PROGRESS_POLL_MS, end - Date.now())));
if (progressed(await read())) return settle();
}
if (baseline === null) return settle();
return await input.isAlive() ? "wedged" : "exited";
}
export async function verifyRemoteRunnerOrSettledExit(
input: Parameters<typeof verifyRemoteRunnerReattachment>[0] & { providerAttempt: number;
claim: { exitedBeforeSessionOpen?: true; replaceUnansweredRunner?: true } },
) {
claim: { exitedBeforeSessionOpen?: true; replaceUnansweredRunner?: true };
unansweredRunnerProgress?: UnansweredRunnerProgressProbe; admittedAttemptLimit?: number },
): Promise<Awaited<ReturnType<typeof verifyRemoteRunnerReattachment>> | undefined> {
const recoveryDeadline = Math.min(input.recoveryDeadline ?? Number.POSITIVE_INFINITY,
Date.now() + REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS);
const stall = input.stall ?? remoteRunnerPluginStall(input.recoveryDeadline);
@@ -15406,8 +15476,8 @@ export async function verifyRemoteRunnerOrSettledExit(
const diagnostic = readRemoteRunnerRecoveryError(error);
const exitedBeforeSessionOpen = input.claim.exitedBeforeSessionOpen === true;
const freshBootstrap = diagnostic !== null && exitedRunnerRequestedFreshBootstrap(diagnostic);
const attemptLimit = freshBootstrap && input.claim.replaceUnansweredRunner === true
? NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT + 1 : NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT;
const attemptLimit = Math.max(input.admittedAttemptLimit ?? 0, freshBootstrap && input.claim.replaceUnansweredRunner === true
? NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT + 1 : NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT);
if (input.providerAttempt > attemptLimit || diagnostic?.stage !== "process_verification" ||
diagnostic.reason !== "process_not_running" ||
(!exitedBeforeSessionOpen && !freshBootstrap &&
@@ -15420,7 +15490,20 @@ export async function verifyRemoteRunnerOrSettledExit(
: state.lifecycle !== "suspended" || !describeExitedRemoteRunner(state).runnerSettled) throw error;
return undefined;
}
return replaceUnanswered ? undefined : verified;
if (!replaceUnanswered) return verified;
if (!input.unansweredRunnerProgress) return undefined;
const progress = await observeUnansweredRunnerProgress({ runner: target.runner!,
stateDirectory: remoteRunnerStateDirectory(target.remoteCwd, input.normalizedSessionId), identity: input.identity,
probe: input.unansweredRunnerProgress, isAlive: () => verified.isAlive(), stall,
recoveryDeadline: Math.min(input.recoveryDeadline ?? Number.POSITIVE_INFINITY, Date.now() + REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS) });
if (progress === "progressing") return verified;
if (progress === "exited") {
const { replaceUnansweredRunner: _replace, ...claim } = input.claim;
return verifyRemoteRunnerOrSettledExit({ ...input, claim, unansweredRunnerProgress: undefined,
admittedAttemptLimit: NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT + 1 });
}
input.unansweredRunnerProgress.onReplace();
return undefined;
}
function runnerStateAllowsFreshStart(state: Record<string, unknown>): boolean {
@@ -16778,6 +16861,7 @@ async function createRunnerdBackendWithinSessionClaim(
// exited. The transport then relaunches it from durable state, which needs
// the admitted provider pack's launch inputs that adoption does not.
let relaunchesSettledRemoteRunner = false;
let replacesUnansweredLiveRunner = false;
const explicitRemoteCodex = input.runnerRemoteCodexPath?.trim() || null;
const remoteCodexNpmSpec = input.runnerRemoteCodexNpmSpec?.trim() || null;
if (explicitRemoteCodex && remoteCodexNpmSpec) {
@@ -17515,6 +17599,9 @@ async function createRunnerdBackendWithinSessionClaim(
throw new RemoteRunnerRecoveryError({ stage: "process_verification", reason: "process_not_running" });
}
});
} else if (sandboxLeaseAcquisition?.outcome === "resumed" && replacesUnansweredLiveRunner) {
await input.onLog?.("stdout", "[paperclip-runner] The runner left its recovery read unanswered and made no progress; "
+ "the relaunch stops it and its process lineage, then restores its saved journal.\n");
} else if (sandboxLeaseAcquisition?.outcome === "resumed" && !adoptedProcess && !stagedRecovery &&
input.restartRecovery?.kind === "reattach_remote_runner" &&
await measureNativeRunnerSpan(input.trace, "sandbox.lease.resume", retainsFreshBootstrapRunnerState)) {
@@ -18170,6 +18257,10 @@ async function createRunnerdBackendWithinSessionClaim(
runId: input.execution.binding.runId,
normalizedSessionId: nativeSessionKey(input.execution),
providerAttempt: input.restartRecovery.providerAttempt,
unansweredRunnerProgress: {
controllerAckedSourceSeq: readRunnerdControllerAckedSourceSeq(root),
onReplace: () => { replacesUnansweredLiveRunner = true; },
},
})
: target.kind === "local" && localRecoveryProcess
? {