mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-18 14:35:55 +01:00
fix(native): keep a slow but live Runner on retry instead of replacing it (#21462)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a recovery read to a remote Runner goes unanswered, the next attempt of the same run carries `replaceUnansweredRunner` > - That attempt verified the Runner was alive and then refused to adopt it anyway. On a resumed sandbox lease the reuse check then failed with `resumed_not_suspended` and left the Runner running. Otherwise a second Runner was launched, and its launch fence killed the busy Runner, which lost the provider turn. > - After a late reconnect a Runner is often slow, not dead: it is still replaying the events it saved with no controller attached > - This pull request makes the attempt check progress first. It adopts a live Runner that is making progress and replaces only a Runner that made no progress. > - The benefit is that a slow but working Runner keeps its turn, and the retry no longer causes the failure it was meant to fix. ## Linked Issues or Issue Description **What happened?** Hosted staging, release swaps on 2026-11-13. The retry after a first-attempt timeout failed with these codes (counts from the traced sample of 136 failures): - `runner_ingress_unavailable: runnerd process failed: runner_harness_state_mismatch: resumed_not_suspended` (part of 40 `ingress` failures) - `runner_remote_process_identity_unavailable` (25): the Runner was gone by the time the retry checked, because the launch fence or a previous retry had killed it - "restored the conversation after process loss, but…" / `provider_turn_lost_on_restore` (10): the replacement Runner restored from the old state and the provider turn was gone Runs that crossed a swap with a reconnect gap under 150 s: 0 of 9 failed. Over 420 s: 14 of 15 failed. The process ID of the Runner changed inside the same sandbox in the failed runs, which shows the replacement. Code path: `nativeRecoveryCommandUnanswered` grants an extra attempt with `replaceUnansweredRunner: true`. `verifyRemoteRunnerOrSettledExit` verifies the Runner is alive and then returns `undefined` because of that flag, so `adoptedProcess` is `undefined`. **Expected behavior** A live Runner that is making progress is adopted. A Runner that made no progress is stopped with proof and then restored from its own journal. **Steps to reproduce** 1. Reattach a remote run whose previous attempt left a recovery read unanswered (`replaceUnansweredRunner: true`), with the Runner alive and its saved event sequence past the controller's last acknowledgement. 2. On master the attempt launches a second Runner, or fails with `resumed_not_suspended` on a resumed sandbox lease. 3. The new tests in `native-session-executor.test.ts` and `native-runner-restart-recovery.integration.test.ts` do this. **Paperclip version or commit** `master` at426e8a35dc. ## What Changed - `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): with `replaceUnansweredRunner`, the attempt adopts the live Runner, launching nothing and signalling nothing, when: - the Runner's saved event sequence is past the controller's last acknowledgement, or - its saved state (event and command sequences, generation, lifecycle) changes within a bounded wait of about 10 s. - The baseline is the first successful state read; every successful read is also compared against the controller's acknowledgement. A Runner whose state cannot be read at all during the wait is adopted, not replaced: missing evidence is not proof of a wedge. A Runner whose identity changes during the wait fails closed. - A settled exit during the wait keeps the attempt allowance that admitted the replacement, so it also restores on the extra fourth attempt. - Only a Runner that made no progress is replaced. The relaunch's existing launch fence stops it and its process lineage with proof, and the new Runner restores its own journal. A resumed sandbox lease no longer refuses that relaunch with `resumed_not_suspended`. - A Runner that exits during the wait follows the existing exited-Runner rules: a settled suspension restores, and a mid-turn exit stays fail-closed. - The wait runs only on this recovery path, never on a normal run start. - `doc/execution-semantics.md`: describes adopt versus replace. - The claim flag `replaceUnansweredRunner` is set in claim logic that another session owns. This PR does not change it; it changes only what the executor does with it. ## Verification Red on master, green on this branch: - `native-session-executor.test.ts` - `adopts a live Runner that emitted events past the controller's last acknowledgement` - `adopts a live Runner whose saved state advances within the bounded wait` - `hands a live Runner that made no progress to the relaunch, whose launch fence stops it, without signalling it here` - `restores from the journal of a Runner that suspended and exited during the wait` - `stays fail-closed for a Runner that exited mid-turn during the wait` - `native-runner-restart-recovery.integration.test.ts`: `reattaches a live %s Runner whose recovery read went unanswered on a resumed sandbox lease` (progressing, wedged) - Review fixes, `native-runner-restart-recovery.integration.test.ts` (the first four were red before the fix): - `adopts a live Runner whose first state read failed and whose later reads advance` - `adopts a live Runner whose first readable state is already past the controller's acknowledgement` - `does not replace a live Runner whose state could not be read at all during the wait` - `restores from the journal of a Runner that suspended and exited during the wait on the admitted fourth attempt` - `fails closed when the Runner's identity changes during the wait` (pins existing behavior) - `native-remote-retry-admission.test.ts`: unchanged, still passes. Local runs on this branch: `tsc --noEmit` is clean for `server` on this branch (masterc6b497d831, which contains #21442). `native-runner-restart-recovery.integration.test.ts`, the adopt-or-replace block: 12 of 12 pass. The full `native-session-executor.test.ts` and `native-remote-retry-admission.test.ts` runs did not finish on this shared machine (load 20 to 96); CI ran them on this head and they passed. Conflict resolved: master added a relaunch branch for a Runner that exited for a fresh bootstrap (#21374) at the same place. Both branches are kept; the cheaper check for an unanswered Runner that made no progress comes first. Daytona results that exist. They ran on a combined branch with this PR, #21450, #21442 (merged) and #21362 (merged), so they cannot be credited to this PR alone: - G5 baseline (master d1d71a7755; 480 s delay injected on the first identity write per run): one Claude run reproduced the staging chain: authentication 419 ms, identity +423.5 s, attempt 1 `native session provider recovery timed out after 900000ms`, then the retry failed in 6 s with `runner_harness_state_mismatch: resumed_not_suspended`. - G5c (same cell, combined branch 588c6ba267): 6 of 6 tasks finished hands-off. The chain did not recur. All runs that crossed swap 2 were adopted on the same Runner process ID. - Caveat: the adopt-or-replace retry path that this PR changes was never exercised in a cell. In G5c no first attempt failed, because all identity writes started together 4 to 5 s after the claim. G5c also ran before the phase timing fix in #21450. ## Risks - Not proven on Daytona: the adopt-or-replace path itself (see the caveat above). - Without #21442 (merged) an adopted Runner can still time out on its first command while it replays a backlog. This PR stops the replacement; #21442 (merged) makes the adopted Runner answer. - A Runner whose state cannot be read during the whole wait is adopted. If it is in fact wedged, the adopted attempt times out again and the attempt limit bounds it. - A wedged Runner that still emits a few events counts as progressing and is adopted. The existing attempt limit bounds this. - The progress check adds a wait of up to about 10 s on this recovery path only. A normal run start has no added wait. - The claim construction (`native-restart-recovery.ts`) is owned by another session and is unchanged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
47a2c82d35
commit
2178c3df88
4 files changed
+300
-5
No files matched your search
@@ -7124,6 +7124,18 @@ type RunnerdDurableIdentity = Record<string, unknown> & {
|
||||
environmentLeaseId: string;
|
||||
};
|
||||
|
||||
function readRunnerdControllerAckedSourceSeq(root: string): number | null {
|
||||
const statePath = resolve(root, "control-plane", "control-plane-state.json");
|
||||
if (!isSafeNativeStateDirectory(root) || !existsSync(statePath)) return null;
|
||||
try {
|
||||
const acked = record(JSON.parse(readBoundedNativeFile(statePath, NATIVE_CONTROL_PLANE_STATE_MAX_BYTES,
|
||||
"runner_durable_identity_too_large").toString("utf8"))).ackedSourceSeq;
|
||||
return typeof acked === "number" && Number.isSafeInteger(acked) && acked >= 0 ? acked : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function readRunnerdDurableIdentity(
|
||||
root: string,
|
||||
allowExternal = false,
|
||||
@@ -15388,10 +15400,68 @@ async function readRemoteRunnerExitEvidence(runner: CommandManagedRuntimeRunner,
|
||||
}
|
||||
}
|
||||
|
||||
export interface UnansweredRunnerProgressProbe {
|
||||
controllerAckedSourceSeq: number | null;
|
||||
onReplace: () => void;
|
||||
windowMs?: number;
|
||||
pollMs?: number;
|
||||
}
|
||||
|
||||
const UNANSWERED_RUNNER_PROGRESS_WINDOW_MS = 10_000;
|
||||
const UNANSWERED_RUNNER_PROGRESS_POLL_MS = 2_000;
|
||||
|
||||
async function observeUnansweredRunnerProgress(input: {
|
||||
runner: CommandManagedRuntimeRunner;
|
||||
stateDirectory: string;
|
||||
identity: Record<string, unknown>;
|
||||
probe: UnansweredRunnerProgressProbe;
|
||||
isAlive: () => Promise<boolean>;
|
||||
recoveryDeadline: number;
|
||||
stall: RemoteRunnerPluginStall;
|
||||
}): Promise<"progressing" | "wedged" | "exited"> {
|
||||
const read = async () => {
|
||||
const state = await readRemoteRunnerState({ runner: input.runner, stateDirectory: input.stateDirectory,
|
||||
recoveryDeadline: input.recoveryDeadline, stall: input.stall }).catch(() => null);
|
||||
if (state && REMOTE_RUNNER_IDENTITY_FIELDS.some(field => state[field] !== input.identity[field])) {
|
||||
throw new Error("runner_remote_recovery_identity_mismatch");
|
||||
}
|
||||
return state;
|
||||
};
|
||||
const progressMarker = (state: Record<string, unknown> | null) => state ? JSON.stringify([state.generation ?? null,
|
||||
state.reconnectCount ?? null, state.nextSourceSeq ?? null, state.ackedSourceSeq ?? null,
|
||||
state.lastControllerCommandSeq ?? null, state.lifecycle ?? null]) : null;
|
||||
const settle = async () => await input.isAlive() ? "progressing" as const : "exited" as const;
|
||||
const emittedPastAck = (state: Record<string, unknown>) => {
|
||||
const emittedSourceSeq = Number(state.nextSourceSeq) - 1;
|
||||
return input.probe.controllerAckedSourceSeq !== null && Number.isSafeInteger(emittedSourceSeq)
|
||||
&& emittedSourceSeq > input.probe.controllerAckedSourceSeq;
|
||||
};
|
||||
let baseline: string | null = null;
|
||||
const progressed = (state: Record<string, unknown> | null) => {
|
||||
if (!state) return false;
|
||||
if (emittedPastAck(state)) return true;
|
||||
const marker = progressMarker(state);
|
||||
if (baseline === null) {
|
||||
baseline = marker;
|
||||
return false;
|
||||
}
|
||||
return marker !== baseline;
|
||||
};
|
||||
if (progressed(await read())) return settle();
|
||||
const end = Math.min(Date.now() + (input.probe.windowMs ?? UNANSWERED_RUNNER_PROGRESS_WINDOW_MS), input.recoveryDeadline);
|
||||
while (Date.now() < end) {
|
||||
await new Promise(done => setTimeout(done, Math.min(input.probe.pollMs ?? UNANSWERED_RUNNER_PROGRESS_POLL_MS, end - Date.now())));
|
||||
if (progressed(await read())) return settle();
|
||||
}
|
||||
if (baseline === null) return settle();
|
||||
return await input.isAlive() ? "wedged" : "exited";
|
||||
}
|
||||
|
||||
export async function verifyRemoteRunnerOrSettledExit(
|
||||
input: Parameters<typeof verifyRemoteRunnerReattachment>[0] & { providerAttempt: number;
|
||||
claim: { exitedBeforeSessionOpen?: true; replaceUnansweredRunner?: true } },
|
||||
) {
|
||||
claim: { exitedBeforeSessionOpen?: true; replaceUnansweredRunner?: true };
|
||||
unansweredRunnerProgress?: UnansweredRunnerProgressProbe; admittedAttemptLimit?: number },
|
||||
): Promise<Awaited<ReturnType<typeof verifyRemoteRunnerReattachment>> | undefined> {
|
||||
const recoveryDeadline = Math.min(input.recoveryDeadline ?? Number.POSITIVE_INFINITY,
|
||||
Date.now() + REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS);
|
||||
const stall = input.stall ?? remoteRunnerPluginStall(input.recoveryDeadline);
|
||||
@@ -15406,8 +15476,8 @@ export async function verifyRemoteRunnerOrSettledExit(
|
||||
const diagnostic = readRemoteRunnerRecoveryError(error);
|
||||
const exitedBeforeSessionOpen = input.claim.exitedBeforeSessionOpen === true;
|
||||
const freshBootstrap = diagnostic !== null && exitedRunnerRequestedFreshBootstrap(diagnostic);
|
||||
const attemptLimit = freshBootstrap && input.claim.replaceUnansweredRunner === true
|
||||
? NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT + 1 : NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT;
|
||||
const attemptLimit = Math.max(input.admittedAttemptLimit ?? 0, freshBootstrap && input.claim.replaceUnansweredRunner === true
|
||||
? NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT + 1 : NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT);
|
||||
if (input.providerAttempt > attemptLimit || diagnostic?.stage !== "process_verification" ||
|
||||
diagnostic.reason !== "process_not_running" ||
|
||||
(!exitedBeforeSessionOpen && !freshBootstrap &&
|
||||
@@ -15420,7 +15490,20 @@ export async function verifyRemoteRunnerOrSettledExit(
|
||||
: state.lifecycle !== "suspended" || !describeExitedRemoteRunner(state).runnerSettled) throw error;
|
||||
return undefined;
|
||||
}
|
||||
return replaceUnanswered ? undefined : verified;
|
||||
if (!replaceUnanswered) return verified;
|
||||
if (!input.unansweredRunnerProgress) return undefined;
|
||||
const progress = await observeUnansweredRunnerProgress({ runner: target.runner!,
|
||||
stateDirectory: remoteRunnerStateDirectory(target.remoteCwd, input.normalizedSessionId), identity: input.identity,
|
||||
probe: input.unansweredRunnerProgress, isAlive: () => verified.isAlive(), stall,
|
||||
recoveryDeadline: Math.min(input.recoveryDeadline ?? Number.POSITIVE_INFINITY, Date.now() + REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS) });
|
||||
if (progress === "progressing") return verified;
|
||||
if (progress === "exited") {
|
||||
const { replaceUnansweredRunner: _replace, ...claim } = input.claim;
|
||||
return verifyRemoteRunnerOrSettledExit({ ...input, claim, unansweredRunnerProgress: undefined,
|
||||
admittedAttemptLimit: NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT + 1 });
|
||||
}
|
||||
input.unansweredRunnerProgress.onReplace();
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function runnerStateAllowsFreshStart(state: Record<string, unknown>): boolean {
|
||||
@@ -16778,6 +16861,7 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
// exited. The transport then relaunches it from durable state, which needs
|
||||
// the admitted provider pack's launch inputs that adoption does not.
|
||||
let relaunchesSettledRemoteRunner = false;
|
||||
let replacesUnansweredLiveRunner = false;
|
||||
const explicitRemoteCodex = input.runnerRemoteCodexPath?.trim() || null;
|
||||
const remoteCodexNpmSpec = input.runnerRemoteCodexNpmSpec?.trim() || null;
|
||||
if (explicitRemoteCodex && remoteCodexNpmSpec) {
|
||||
@@ -17515,6 +17599,9 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
throw new RemoteRunnerRecoveryError({ stage: "process_verification", reason: "process_not_running" });
|
||||
}
|
||||
});
|
||||
} else if (sandboxLeaseAcquisition?.outcome === "resumed" && replacesUnansweredLiveRunner) {
|
||||
await input.onLog?.("stdout", "[paperclip-runner] The runner left its recovery read unanswered and made no progress; "
|
||||
+ "the relaunch stops it and its process lineage, then restores its saved journal.\n");
|
||||
} else if (sandboxLeaseAcquisition?.outcome === "resumed" && !adoptedProcess && !stagedRecovery &&
|
||||
input.restartRecovery?.kind === "reattach_remote_runner" &&
|
||||
await measureNativeRunnerSpan(input.trace, "sandbox.lease.resume", retainsFreshBootstrapRunnerState)) {
|
||||
@@ -18170,6 +18257,10 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
runId: input.execution.binding.runId,
|
||||
normalizedSessionId: nativeSessionKey(input.execution),
|
||||
providerAttempt: input.restartRecovery.providerAttempt,
|
||||
unansweredRunnerProgress: {
|
||||
controllerAckedSourceSeq: readRunnerdControllerAckedSourceSeq(root),
|
||||
onReplace: () => { replacesUnansweredLiveRunner = true; },
|
||||
},
|
||||
})
|
||||
: target.kind === "local" && localRecoveryProcess
|
||||
? {
|
||||
|
||||
Reference in new issue
Block a user