mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(runner): keep remote recovery and launch through a short plugin RPC stall (#20945)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote run starts `paperclip-runnerd` in a sandbox. The controller
reads and controls it through the environment plugin
(`environmentExecute`).
> - #20940 made a stalled supervision probe "unknown" while the Runner
stays connected. Other callers still turn the same plugin stall into a
terminal failure.
> - During restart reattachment, the saved-state read, the
identity-marker read and the process check share a 30-second budget. One
plugin RPC that stalls for 40 seconds uses all of it, so a short stall
fails a reattachment to a healthy Runner.
> - At launch, a stalled launch call or identity read fails the run with
`runner_process_termination_unconfirmed` or
`runner_remote_process_identity_unavailable`, although the runner may be
healthy.
> - This pull request treats a plugin RPC timeout as "unknown" and
retries the read-only call within a bounded stall budget. A stalled
launch call is not repeated: the controller reads the launch nonce's
identity marker instead.
> - The benefit is that a short plugin stall during recovery or launch
no longer fails a healthy run. Every positive answer from the sandbox
keeps its meaning, and the bound still fails closed.
## Linked Issues or Issue Description
Refs #20940, #20909
**What happened?**
A plugin `environmentExecute` RPC stall of more than 40 seconds makes a
remote recovery read fail. Plugin RPCs time out at the requested timeout
plus a 30-second overhead, so a 10-second probe can take 40 seconds. The
reattachment read budget is 30 seconds. One stalled read therefore
failed reattachment with `probe_transport_failed` or `probe_timeout`. At
launch, a stalled launch call rejected the run with exit unconfirmed. A
stalled identity read (20-second wait, 2-second reads that can take 32
seconds) failed the run with
`runner_remote_process_identity_unavailable`.
**Expected behavior**
A plugin RPC timeout is a stall of the plugin, not an answer from the
sandbox. Read-only recovery reads retry within a bound. A launch call
that stalls is not repeated. The launch nonce's marker decides whether
the one runner was launched. At the bound, the run fails closed and
Retry works.
**Steps to reproduce**
1. Start a remote (Daytona) run.
2. Restart the controller so that the run reattaches, or start a new
run.
3. While the controller reads the sandbox, stop the plugin worker for 45
seconds.
4. The reattachment or launch fails.
**Paperclip version or commit**
master at 48d23d1530.
## What Changed
- `remote-runner-recovery-probe.ts`: adds `isPluginRpcTimeout` (an
error, or its cause chain, with `PLUGIN_RPC_ERROR_CODES.TIMEOUT`). It
also adds a per-recovery `RemoteRunnerPluginStall` with a 3-minute
budget (`REMOTE_RUNNER_PLUGIN_STALL_BUDGET_MS`). The caller's recovery
deadline caps it. After the first plugin RPC timeout in a recovery, its
read-only probes use the stall deadline instead of the 30-second
deadline. Signals are never retried. Other transport failures keep the
30-second budget.
- `verifyRemoteRunnerReattachment` and `verifyRemoteRunnerOrSettledExit`
share one stall budget across the saved-state read, the identity-marker
read, the process check and the post-exit settled-state read
(`readRemoteRunnerState`).
- `waitForRemoteRunnerProcessIdentity`: a plugin RPC timeout extends the
20-second identity wait to the stall budget.
- Launch: a launch call that fails with a plugin RPC timeout is not
repeated. The controller goes on to the identity-marker wait for the
same nonce. A found marker adopts the one launched runner. A missing
marker uses the existing nonce cleanup, and the result stays unconfirmed
when the cleanup fails.
- `doc/execution-semantics.md` describes these rules.
## Verification
- Red on master: 7 new tests in
`server/src/services/native-runtime/native-session-executor.test.ts`
failed. Each one stalls a plugin RPC for 40 seconds with a
`JsonRpcCallError` `TIMEOUT`:
- reattachment through a stall in the saved-state, identity-marker and
process-check reads (the reattachment failed with the stall error);
- the bound test (it settled before the stall budget);
- the post-exit settled-state read (it failed with the stall error);
- launch-call and identity-read stalls (they failed with the stall error
or `runner_remote_process_identity_unavailable`).
- Green: all 7 pass, and a control test proves that a non-stall
transport error keeps the 30-second bound.
`native-session-executor.test.ts` and
`remote-runner-recovery-probe.test.ts` pass (1400 tests). `tsc --noEmit`
passes for `server`.
- The tests also assert that one launch call happens, no cleanup or
signal runs on the success path, and the bound fails closed without exit
proof.
## Risks
- During a real plugin stall, recovery and launch can now wait up to 3
minutes instead of about 30 seconds before they fail closed. The longer
wait applies only after a plugin RPC timeout. A sandbox that answers, or
a non-timeout transport error, keeps the old timing.
- No wait is added before `turn.start` on the normal path.
- State saved by an older version is not changed.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, code search). Extended reasoning was
on.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
5e2243f081
commit
757a20d2f1
4 files changed
+199
-18
No files matched your search
@@ -84,6 +84,7 @@ import { nativeSha256 } from "./canonical.js";
|
||||
import { resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js";
|
||||
import { REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT, RemoteRunnerRecoveryError, readRemoteRunnerRecoveryError } from "./remote-runner-recovery-diagnostics.js";
|
||||
import { REMOTE_RUNNER_RECONNECT_GRACE_MS } from "./native-runner-recovery-contract.js";
|
||||
import { JsonRpcCallError, PLUGIN_RPC_ERROR_CODES } from "@paperclipai/plugin-sdk";
|
||||
import { agentDirectoryWorkingCopyService } from "../agent-directory-working-copies.js";
|
||||
import { probeAgentDirectory } from "../agent-directory-probe.js";
|
||||
import { AGENT_FILES_CONTRACT } from "../agent-file-store.js";
|
||||
@@ -862,6 +863,96 @@ describe("remote controller restart adoption", () => {
|
||||
expect(probes).toBe(2);
|
||||
expect(execute.mock.calls.every(([request]) => !request.args.join(" ").match(/nohup|mkdir|rm -|mv -|kill -"/))).toBe(true);
|
||||
});
|
||||
const pluginStall = () => new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.TIMEOUT,
|
||||
message: 'RPC call "environmentExecute" timed out after 40000ms' });
|
||||
it.each(["runner_state", "identity_marker", "process_verification"] as const)(
|
||||
"reattaches the same runner through a plugin RPC stall during the %s read", async stage => {
|
||||
vi.useFakeTimers();
|
||||
onTestFinished(() => { vi.useRealTimers(); });
|
||||
const { target, execute } = fixture();
|
||||
const original = execute.getMockImplementation()!;
|
||||
const operation = stage === "identity_marker" ? "paperclip-runner-recovery-identity" : "paperclip-runner-recovery-check";
|
||||
let stalls = 0;
|
||||
execute.mockImplementation(async request => {
|
||||
if ((stage === "runner_state" ? request.args[1].includes("base64") : request.args[2] === operation) && stalls < 2) {
|
||||
stalls++;
|
||||
await new Promise(resolveWait => setTimeout(resolveWait, 40_000));
|
||||
throw pluginStall();
|
||||
}
|
||||
return original(request);
|
||||
});
|
||||
const outcome = verifyRemoteRunnerReattachment({
|
||||
claim, target: target as never, identity, runId: "run", normalizedSessionId: "session",
|
||||
}).catch(error => error as Error);
|
||||
await vi.advanceTimersByTimeAsync(90_000);
|
||||
const adopted = await outcome;
|
||||
expect(adopted).not.toBeInstanceOf(Error);
|
||||
expect((adopted as { pid: number }).pid).toBe(123);
|
||||
expect(stalls).toBe(2);
|
||||
expect(execute.mock.calls.every(([request]) => !request.args.join(" ").match(/nohup|mkdir|rm -|mv -|kill -"/))).toBe(true);
|
||||
});
|
||||
it("fails reattachment closed when the plugin stall outlasts its bound", async () => {
|
||||
vi.useFakeTimers();
|
||||
onTestFinished(() => { vi.useRealTimers(); });
|
||||
const { target, execute } = fixture();
|
||||
const original = execute.getMockImplementation()!;
|
||||
execute.mockImplementation(async request => {
|
||||
if (request.args[2] !== "paperclip-runner-recovery-check") return original(request);
|
||||
await new Promise(resolveWait => setTimeout(resolveWait, 40_000));
|
||||
throw pluginStall();
|
||||
});
|
||||
let settled = false;
|
||||
const outcome = verifyRemoteRunnerReattachment({
|
||||
claim, target: target as never, identity, runId: "run", normalizedSessionId: "session",
|
||||
}).catch(error => { settled = true; return error as Error; });
|
||||
await vi.advanceTimersByTimeAsync(150_000);
|
||||
expect(settled).toBe(false);
|
||||
await vi.advanceTimersByTimeAsync(80_000);
|
||||
expect(readRemoteRunnerRecoveryError(await outcome)).toEqual({ stage: "process_verification", reason: "probe_transport_failed" });
|
||||
});
|
||||
it("keeps the 30-second bound for a transport failure that is not a plugin stall", async () => {
|
||||
vi.useFakeTimers();
|
||||
onTestFinished(() => { vi.useRealTimers(); });
|
||||
const { target, execute } = fixture();
|
||||
const original = execute.getMockImplementation()!;
|
||||
execute.mockImplementation(async request => {
|
||||
if (request.args[2] !== "paperclip-runner-recovery-check") return original(request);
|
||||
throw new Error("Sandbox with ID test-deleted-sandbox not found");
|
||||
});
|
||||
const startedAt = Date.now();
|
||||
const outcome = verifyRemoteRunnerReattachment({
|
||||
claim, target: target as never, identity, runId: "run", normalizedSessionId: "session",
|
||||
}).catch(error => error as Error);
|
||||
await vi.advanceTimersByTimeAsync(30_000);
|
||||
expect(readRemoteRunnerRecoveryError(await outcome)).toEqual({ stage: "process_verification", reason: "probe_transport_failed" });
|
||||
expect(Date.now() - startedAt).toBeLessThanOrEqual(30_000);
|
||||
});
|
||||
it("reads the settled state of an exited runner through a plugin RPC stall", async () => {
|
||||
vi.useFakeTimers();
|
||||
onTestFinished(() => { vi.useRealTimers(); });
|
||||
const settled = { lifecycle: "suspended", outbox: [], pendingTerminalDelivery: null, pendingProviderCleanup: null };
|
||||
const { target, execute } = fixture(settled);
|
||||
const original = execute.getMockImplementation()!;
|
||||
let exited = false;
|
||||
let stalls = 0;
|
||||
execute.mockImplementation(async request => {
|
||||
if (request.args[2] === "paperclip-runner-recovery-check") {
|
||||
exited = true;
|
||||
return { stdout: "", stderr: "", exitCode: 45, timedOut: false };
|
||||
}
|
||||
if (exited && request.args[1].includes("base64") && stalls === 0) {
|
||||
stalls++;
|
||||
await new Promise(resolveWait => setTimeout(resolveWait, 40_000));
|
||||
throw pluginStall();
|
||||
}
|
||||
return original(request);
|
||||
});
|
||||
const outcome = verifyRemoteRunnerOrSettledExit({ claim, target: target as never, identity,
|
||||
runId: "run", normalizedSessionId: "session", providerAttempt: 2 }).catch(error => error as Error);
|
||||
await vi.advanceTimersByTimeAsync(60_000);
|
||||
expect(await outcome).toBeUndefined();
|
||||
expect(stalls).toBe(1);
|
||||
});
|
||||
it("rejects a marker that cannot prove its process generation", async () => {
|
||||
const { target, execute } = fixture();
|
||||
const original = execute.getMockImplementation()!;
|
||||
@@ -1672,6 +1763,50 @@ fs.readFileSync = function(path, ...args) {
|
||||
} finally { vi.useRealTimers(); }
|
||||
});
|
||||
|
||||
it.each(["launch", "identity"] as const)("supervises the one launched runner through a plugin RPC stall on the %s call", async stalled => {
|
||||
vi.useFakeTimers();
|
||||
let nonce = "";
|
||||
let markerWritten = false;
|
||||
let stalls = 0;
|
||||
const ok = { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" };
|
||||
const stall = async () => {
|
||||
stalls++;
|
||||
await new Promise(resolve => setTimeout(resolve, 40_000));
|
||||
throw new JsonRpcCallError({ code: PLUGIN_RPC_ERROR_CODES.TIMEOUT, message: 'RPC call "environmentExecute" timed out after 40000ms' });
|
||||
};
|
||||
const execute = vi.fn(async (input: { args?: string[] }) => {
|
||||
const label = input.args?.[2];
|
||||
if (label === "paperclip-runner-launch") {
|
||||
nonce = input.args![4]!;
|
||||
markerWritten = true;
|
||||
if (stalled === "launch") return stall();
|
||||
return ok;
|
||||
}
|
||||
if (label === "paperclip-runner-process-identity") {
|
||||
if (stalled === "identity" && stalls === 0) return stall();
|
||||
return markerWritten ? { ...ok, stdout: `${nonce}\n4321\n2026-09-06T00:00:00.000Z\nrunner-remote\n` } : { ...ok, exitCode: 1 };
|
||||
}
|
||||
if (label === "paperclip-runner-monitor") return { ...ok, exitCode: 45 };
|
||||
return { ...ok, exitCode: 1 };
|
||||
});
|
||||
const onSpawn = vi.fn(async () => undefined);
|
||||
const handle = createRemoteRunnerProcessLauncher({
|
||||
target: { kind: "remote", transport: "sandbox", environmentId: "environment", leaseId: "lease", remoteCwd: "/workspace" },
|
||||
runner: { execute } as never, remoteBinary: "/runtime/runnerd", processIdentityPath: "/runtime/identity",
|
||||
stateDirectory: "/runtime", diagnosticsDirectory: "/runtime/diagnostics", runnerInstanceId: "runner-remote", onSpawn,
|
||||
})({ command: "/controller/runnerd", args: [], cwd: "/controller", environment: {} });
|
||||
const outcome = handle.completion.catch(error => error as Error);
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(60_000);
|
||||
expect(await outcome).toMatchObject({ code: null });
|
||||
expect(stalls).toBe(1);
|
||||
expect(onSpawn).toHaveBeenCalledExactlyOnceWith({ pid: 4321, processGroupId: null, startedAt: "2026-09-06T00:00:00.000Z" });
|
||||
expect(handle.isExitConfirmed?.()).toBe(true);
|
||||
expect(execute.mock.calls.filter(([x]) => x.args?.[2] === "paperclip-runner-launch")).toHaveLength(1);
|
||||
expect(execute.mock.calls.some(([x]) => x.args?.[2] === "paperclip-runner-identity-failure-cleanup")).toBe(false);
|
||||
} finally { vi.useRealTimers(); }
|
||||
});
|
||||
|
||||
it("bounds supervision without exit proof when a connected runner's probes never answer", async () => {
|
||||
vi.useFakeTimers();
|
||||
let nonce = "";
|
||||
|
||||
Reference in new issue
Block a user