mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-18 14:35:55 +01:00
fix(native): keep a slow but live Runner on retry instead of replacing it (#21462)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a recovery read to a remote Runner goes unanswered, the next attempt of the same run carries `replaceUnansweredRunner` > - That attempt verified the Runner was alive and then refused to adopt it anyway. On a resumed sandbox lease the reuse check then failed with `resumed_not_suspended` and left the Runner running. Otherwise a second Runner was launched, and its launch fence killed the busy Runner, which lost the provider turn. > - After a late reconnect a Runner is often slow, not dead: it is still replaying the events it saved with no controller attached > - This pull request makes the attempt check progress first. It adopts a live Runner that is making progress and replaces only a Runner that made no progress. > - The benefit is that a slow but working Runner keeps its turn, and the retry no longer causes the failure it was meant to fix. ## Linked Issues or Issue Description **What happened?** Hosted staging, release swaps on 2026-11-13. The retry after a first-attempt timeout failed with these codes (counts from the traced sample of 136 failures): - `runner_ingress_unavailable: runnerd process failed: runner_harness_state_mismatch: resumed_not_suspended` (part of 40 `ingress` failures) - `runner_remote_process_identity_unavailable` (25): the Runner was gone by the time the retry checked, because the launch fence or a previous retry had killed it - "restored the conversation after process loss, but…" / `provider_turn_lost_on_restore` (10): the replacement Runner restored from the old state and the provider turn was gone Runs that crossed a swap with a reconnect gap under 150 s: 0 of 9 failed. Over 420 s: 14 of 15 failed. The process ID of the Runner changed inside the same sandbox in the failed runs, which shows the replacement. Code path: `nativeRecoveryCommandUnanswered` grants an extra attempt with `replaceUnansweredRunner: true`. `verifyRemoteRunnerOrSettledExit` verifies the Runner is alive and then returns `undefined` because of that flag, so `adoptedProcess` is `undefined`. **Expected behavior** A live Runner that is making progress is adopted. A Runner that made no progress is stopped with proof and then restored from its own journal. **Steps to reproduce** 1. Reattach a remote run whose previous attempt left a recovery read unanswered (`replaceUnansweredRunner: true`), with the Runner alive and its saved event sequence past the controller's last acknowledgement. 2. On master the attempt launches a second Runner, or fails with `resumed_not_suspended` on a resumed sandbox lease. 3. The new tests in `native-session-executor.test.ts` and `native-runner-restart-recovery.integration.test.ts` do this. **Paperclip version or commit** `master` at426e8a35dc. ## What Changed - `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): with `replaceUnansweredRunner`, the attempt adopts the live Runner, launching nothing and signalling nothing, when: - the Runner's saved event sequence is past the controller's last acknowledgement, or - its saved state (event and command sequences, generation, lifecycle) changes within a bounded wait of about 10 s. - The baseline is the first successful state read; every successful read is also compared against the controller's acknowledgement. A Runner whose state cannot be read at all during the wait is adopted, not replaced: missing evidence is not proof of a wedge. A Runner whose identity changes during the wait fails closed. - A settled exit during the wait keeps the attempt allowance that admitted the replacement, so it also restores on the extra fourth attempt. - Only a Runner that made no progress is replaced. The relaunch's existing launch fence stops it and its process lineage with proof, and the new Runner restores its own journal. A resumed sandbox lease no longer refuses that relaunch with `resumed_not_suspended`. - A Runner that exits during the wait follows the existing exited-Runner rules: a settled suspension restores, and a mid-turn exit stays fail-closed. - The wait runs only on this recovery path, never on a normal run start. - `doc/execution-semantics.md`: describes adopt versus replace. - The claim flag `replaceUnansweredRunner` is set in claim logic that another session owns. This PR does not change it; it changes only what the executor does with it. ## Verification Red on master, green on this branch: - `native-session-executor.test.ts` - `adopts a live Runner that emitted events past the controller's last acknowledgement` - `adopts a live Runner whose saved state advances within the bounded wait` - `hands a live Runner that made no progress to the relaunch, whose launch fence stops it, without signalling it here` - `restores from the journal of a Runner that suspended and exited during the wait` - `stays fail-closed for a Runner that exited mid-turn during the wait` - `native-runner-restart-recovery.integration.test.ts`: `reattaches a live %s Runner whose recovery read went unanswered on a resumed sandbox lease` (progressing, wedged) - Review fixes, `native-runner-restart-recovery.integration.test.ts` (the first four were red before the fix): - `adopts a live Runner whose first state read failed and whose later reads advance` - `adopts a live Runner whose first readable state is already past the controller's acknowledgement` - `does not replace a live Runner whose state could not be read at all during the wait` - `restores from the journal of a Runner that suspended and exited during the wait on the admitted fourth attempt` - `fails closed when the Runner's identity changes during the wait` (pins existing behavior) - `native-remote-retry-admission.test.ts`: unchanged, still passes. Local runs on this branch: `tsc --noEmit` is clean for `server` on this branch (masterc6b497d831, which contains #21442). `native-runner-restart-recovery.integration.test.ts`, the adopt-or-replace block: 12 of 12 pass. The full `native-session-executor.test.ts` and `native-remote-retry-admission.test.ts` runs did not finish on this shared machine (load 20 to 96); CI ran them on this head and they passed. Conflict resolved: master added a relaunch branch for a Runner that exited for a fresh bootstrap (#21374) at the same place. Both branches are kept; the cheaper check for an unanswered Runner that made no progress comes first. Daytona results that exist. They ran on a combined branch with this PR, #21450, #21442 (merged) and #21362 (merged), so they cannot be credited to this PR alone: - G5 baseline (master d1d71a7755; 480 s delay injected on the first identity write per run): one Claude run reproduced the staging chain: authentication 419 ms, identity +423.5 s, attempt 1 `native session provider recovery timed out after 900000ms`, then the retry failed in 6 s with `runner_harness_state_mismatch: resumed_not_suspended`. - G5c (same cell, combined branch 588c6ba267): 6 of 6 tasks finished hands-off. The chain did not recur. All runs that crossed swap 2 were adopted on the same Runner process ID. - Caveat: the adopt-or-replace retry path that this PR changes was never exercised in a cell. In G5c no first attempt failed, because all identity writes started together 4 to 5 s after the claim. G5c also ran before the phase timing fix in #21450. ## Risks - Not proven on Daytona: the adopt-or-replace path itself (see the caveat above). - Without #21442 (merged) an adopted Runner can still time out on its first command while it replays a backlog. This PR stops the replacement; #21442 (merged) makes the adopted Runner answer. - A Runner whose state cannot be read during the whole wait is adopted. If it is in fact wedged, the adopted attempt times out again and the attempt limit bounds it. - A wedged Runner that still emits a few events counts as progressing and is adopted. The existing attempt limit bounds this. - The progress check adds a wait of up to about 10 s on this recovery path only. A normal run start has no added wait. - The claim construction (`native-restart-recovery.ts`) is owned by another session and is unchanged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
47a2c82d35
commit
2178c3df88
4 files changed
+300
-5
No files matched your search
@@ -18306,6 +18306,50 @@ describe("runnerd provider runtime wiring", () => {
|
||||
expect(existsSync(join(root, "control-plane", "control-plane-state.json"))).toBe(true);
|
||||
});
|
||||
|
||||
it.each(["progressing", "wedged"] as const)("reattaches a live %s Runner whose recovery read went unanswered on a resumed sandbox lease", async mode => {
|
||||
const current = scopedNativeExecution(`unanswered-live-runner-${mode}`);
|
||||
const identity = { runId: current.binding.runId, normalizedSessionId: current.session.normalizedSessionId!,
|
||||
runnerInstanceId: "unanswered-codex", environmentLeaseId: "unanswered-lease",
|
||||
turnId: `turn_attachment_${randomUUID()}`, itemId: `item_attachment_${randomUUID()}` };
|
||||
const root = await initializeScopedRunnerRoot(current, identity.runnerInstanceId);
|
||||
await mkdir(join(root, "control-plane"), { recursive: true });
|
||||
await writeFile(join(root, "control-plane", "control-plane-state.json"),
|
||||
JSON.stringify({ ...durableControlPlaneState(identity), ackedSourceSeq: 2204 }));
|
||||
const execute = vi.fn(async (request: { args: string[] }) => {
|
||||
const script = request.args[1] ?? "";
|
||||
return { exitCode: 0, timedOut: false, stderr: "",
|
||||
stdout: request.args[0] === "--build-metadata" ? JSON.stringify({
|
||||
schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd",
|
||||
packageName: "@paperclipai/paperclip-runner", binaryContractVersion: 2,
|
||||
durableSessionCapabilities: ["unlimited_runtime", "connection_lease_renewal"], prpTransportModes: ["listen_ws"],
|
||||
}) : request.args[0] === "--version" ? "codex-cli 0.160.0" : script.includes("base64")
|
||||
? Buffer.from(JSON.stringify({ ...identity, lifecycle: "turn_active", outbox: [{ id: 2205 }], pendingTerminalDelivery: null,
|
||||
nextSourceSeq: mode === "progressing" ? 2345 : 2205, ackedSourceSeq: 2204, lastControllerCommandSeq: 61 })).toString("base64")
|
||||
: script.includes("cat --") ? `nonce\n1806\n2026-11-13T08:30:36.187Z\n${identity.runnerInstanceId}\nlinux:abcd-1234:100\n` : "" };
|
||||
});
|
||||
state.createBackend.mockClear(); state.createTransport.mockClear();
|
||||
const logs: string[] = [];
|
||||
await createRunnerdBackend({ db: leaseDb(current), execution: current, runnerInstanceId: identity.runnerInstanceId,
|
||||
runnerIngressAuthorized: true, onLog: async (_stream, chunk) => { logs.push(chunk); },
|
||||
controllerPersistence: { bootId: "successor", runnerArtifact: { version: "admitted-version", digest: `sha256:${"a".repeat(64)}` } },
|
||||
runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace", environmentId: "environment",
|
||||
leaseId: identity.environmentLeaseId, providerKey: "daytona", runner: { execute },
|
||||
effectiveCapabilities: { runnerWebSocketIngress: true },
|
||||
sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "sandbox" } } as never,
|
||||
restartRecovery: { kind: "reattach_remote_runner", runId: current.binding.runId, leaseOwner: "controller",
|
||||
controllerGeneration: 6, providerAttempt: 2, restartKind: "hard", recoveryRequestId: null,
|
||||
remote: { providerLeaseId: "sandbox", remoteCwd: "/workspace" }, replaceUnansweredRunner: true },
|
||||
});
|
||||
state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
|
||||
const options = state.createTransport.mock.calls.at(-1)![0] as RunnerTransportOptions & { prepareExternalRunnerState: () => Promise<void> };
|
||||
if (mode === "progressing") expect(options.adoptExistingRunner).toMatchObject({ pid: 1806 });
|
||||
else expect(options.adoptExistingRunner).toBeUndefined();
|
||||
await expect(options.prepareExternalRunnerState()).resolves.toBeUndefined();
|
||||
expect(execute.mock.calls.some(([request]) => /kill -/.test(request.args.join(" ")))).toBe(false);
|
||||
expect(execute.mock.calls.some(([request]) => request.args.includes("paperclip-runner-launch"))).toBe(false);
|
||||
expect(logs.some(line => line.includes("made no progress"))).toBe(mode === "wedged");
|
||||
}, 30_000);
|
||||
|
||||
it.each(["failed", "timeout", "succeeded"] as const)("retains the local controller checkpoint until remote continuity archiving succeeds: %s", async mode => {
|
||||
const current = scopedNativeExecution("failed-remote-continuity-archive");
|
||||
state.createBackend.mockClear(); state.createTransport.mockClear();
|
||||
|
||||
Reference in new issue
Block a user