mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 19:02:37 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent turns run in a remote Runner (`paperclip-runnerd`) that the server launches in a sandbox. > - A deploy can stop the server after the launch command reached the sandbox but before the server read back the Runner's process identity (a lost acknowledgement). > - The database then has no PID for the run, so the successor server launches the Runner again on the same state folder. > - Two Runners must never serve the same run's state. #20852 proposed a large launch-intent design for this window. Since then, #21079 added a fence that stops a leftover Runner on the same state folder before each launch. > - This pull request adds a Linux regression test that drives the real launcher through the lost-acknowledgement window. > - The benefit is that a future change to the launch path cannot bring back a second Runner without failing this test. ## Linked Issues or Issue Description Refs #20852 (superseded: current master already prevents the duplicate Runner through #21079). Refs #21079. **What happened?** #20852 claimed that a relaunch after a lost launch acknowledgement could start a second Runner for the same run. Nothing proved whether current master still has this problem. **Expected behavior** After a lost acknowledgement, the successor's relaunch stops the orphaned Runner. Exactly one Runner serves the run's state folder, and the process marker names it. **Steps to reproduce** 1. Launch a remote Runner, and lose the launch RPC's acknowledgement before the identity read (the old server stops). 2. Relaunch the same run on the same sandbox from a new launcher (the successor). 3. Count the live `paperclip-runnerd` processes that serve the run's `--state-dir`. ## What Changed - New `server/src/services/native-runtime/remote-runner-lost-launch-ack.test.ts`. It is a test only and changes no product code. - It runs only where `/proc` exists (Linux CI), like the existing #21079 fence tests. - It drives the exported `createRemoteRunnerProcessLauncher` with a local command runner. The old launcher's runner runs the launch command, then throws `remote_launch_ack_lost`. After that, every command fails, because the old server is gone. - A small bash stand-in for `paperclip-runnerd` keeps the Runner's argv[0] and `--state-dir`. Its `reap-lineage` reports a verified, empty lineage. - The test checks these results: the orphan was alive and had no PID on the handle; the successor's Runner has a new PID; the orphan stops; exactly one Runner serves the state folder; the marker names the successor's PID. ## Verification - Linux reproduction (Docker `node:24-bookworm-slim`). I used the exact outer launch command and `REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT`, which I extracted from this head: - master scripts: `runners after successor relaunch [61], orphan alive false`. Result: one Runner (PASS). - the same scripts without the #21079 stale-state fence (counterfactual): `runners after successor relaunch [24, 56], orphan alive true`. Result: duplicate Runner (FAIL). - macOS: `npx vitest run src/services/native-runtime/remote-runner-lost-launch-ack.test.ts` reports the file as skipped (1 skipped), as intended, because there is no `/proc`. - The test runs for real only on Linux CI. CI on this head is the first real run of the vitest file. ## Risks - Low risk. The change is a test only. - The test starts real local processes. Its teardown kills only the PIDs it found on its own temporary state folder, and it never signals PID 0. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, code search, Docker, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
120 lines
6.5 KiB
TypeScript
120 lines
6.5 KiB
TypeScript
import { spawn } from "node:child_process";
|
|
import { existsSync, readdirSync, readFileSync } from "node:fs";
|
|
import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
|
import { tmpdir } from "node:os";
|
|
import { basename, join } from "node:path";
|
|
import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
|
|
import { describe, expect, it, vi } from "vitest";
|
|
import { createRemoteRunnerProcessLauncher } from "./native-session-executor.js";
|
|
|
|
type Execute = CommandManagedRuntimeRunner["execute"];
|
|
|
|
/** Runs each sandbox command on this machine, the way a sandbox command RPC would. */
|
|
const localExecute: Execute = request => new Promise(resolve => {
|
|
const child = spawn(request.command, request.args ?? [], { cwd: request.cwd,
|
|
env: { ...process.env, ...request.env }, stdio: ["ignore", "pipe", "pipe"] });
|
|
let stdout = "", stderr = "";
|
|
child.stdout.on("data", chunk => { stdout += String(chunk); });
|
|
child.stderr.on("data", chunk => { stderr += String(chunk); });
|
|
child.once("close", (code, signal) => resolve({ exitCode: code, signal, timedOut: false, pid: child.pid ?? null,
|
|
startedAt: null, stdout, stderr }));
|
|
});
|
|
|
|
const alive = (pid: number) => {
|
|
try { return !readFileSync(`/proc/${pid}/stat`, "utf8").split(") ")[1]!.startsWith("Z"); } catch { return false; }
|
|
};
|
|
|
|
/** Every live `paperclip-runnerd` that serves this state folder, read from /proc as the launch fence does. */
|
|
function runnersServing(stateDirectory: string): number[] {
|
|
const pids: number[] = [];
|
|
for (const entry of readdirSync("/proc")) {
|
|
if (String(Number(entry)) !== entry) continue;
|
|
let argv: string[];
|
|
try { argv = readFileSync(`/proc/${entry}/cmdline`, "utf8").split("\0"); } catch { continue; }
|
|
if (basename(argv[0] ?? "") !== "paperclip-runnerd" || argv[1] === "reap-lineage") continue;
|
|
if (argv.some((value, index) => value === "--state-dir" && argv[index + 1] === stateDirectory) && alive(Number(entry))) {
|
|
pids.push(Number(entry));
|
|
}
|
|
}
|
|
return pids;
|
|
}
|
|
|
|
// The server can stop after the launch command reached the sandbox but before
|
|
// it read back the Runner's process identity (a lost acknowledgement). The
|
|
// database then holds no PID for the run, and the successor launches again.
|
|
// This guards that a relaunch never leaves two Runners on the run's state.
|
|
describe.skipIf(!existsSync("/proc/self/cmdline"))("remote Runner launch whose acknowledgement was lost", () => {
|
|
it("leaves exactly one Runner on the run's state folder after the successor relaunches", async () => {
|
|
const root = await mkdtemp(join(tmpdir(), "runner-lost-launch-ack-"));
|
|
const stateDirectory = join(root, "sandbox", "state");
|
|
const binary = join(root, "bin", "paperclip-runnerd");
|
|
await mkdir(join(root, "bin"), { recursive: true });
|
|
await mkdir(stateDirectory, { recursive: true });
|
|
// Stands in for runnerd: a long-lived process whose argv[0] is the Runner
|
|
// binary. `reap-lineage` reports a verified, empty lineage.
|
|
await writeFile(binary, [
|
|
"#!/bin/bash",
|
|
'if test "$1" = reap-lineage; then',
|
|
` printf '%s\\n' '${JSON.stringify({ schema: "paperclip.runner.process-lineage-reap.v1", verified: true, reason: null, journals: 1, signalled: 0, remaining: [] })}'`,
|
|
" exit 0",
|
|
"fi",
|
|
`exec -a "$0" ${JSON.stringify(process.execPath)} -e 'setInterval(() => {}, 1000)' -- "$@"`,
|
|
"",
|
|
].join("\n"), { mode: 0o755 });
|
|
const launcherInput = {
|
|
target: { kind: "remote" as const, transport: "sandbox" as const, environmentId: "environment", leaseId: "lease",
|
|
remoteCwd: join(root, "sandbox") },
|
|
remoteBinary: binary,
|
|
processIdentityPath: join(stateDirectory, "runner-process.identity"),
|
|
stateDirectory,
|
|
diagnosticsDirectory: join(root, "sandbox", "diagnostics"),
|
|
runnerInstanceId: "runner-lost-ack",
|
|
};
|
|
const spec = { command: binary, args: ["--runner-id", "runner-lost-ack", "--state-dir", stateDirectory],
|
|
cwd: root, environment: {} };
|
|
|
|
// The old server: its launch command runs in the sandbox, then the
|
|
// connection is lost before the acknowledgement, and the server is gone.
|
|
let serverGone = false;
|
|
const oldExecute = vi.fn<Execute>(async request => {
|
|
if (serverGone) throw new Error("old_controller_gone");
|
|
if (request.args?.[2] !== "paperclip-runner-launch") return localExecute(request);
|
|
await localExecute(request);
|
|
serverGone = true;
|
|
throw new Error("remote_launch_ack_lost");
|
|
});
|
|
const oldHandle = createRemoteRunnerProcessLauncher({ ...launcherInput, runner: { execute: oldExecute } })(spec);
|
|
const successorSpawned = vi.fn<(meta: { pid: number }) => Promise<void>>(async () => undefined);
|
|
let successorHandle: ReturnType<ReturnType<typeof createRemoteRunnerProcessLauncher>> | undefined;
|
|
let orphanPid = 0;
|
|
try {
|
|
await expect(oldHandle.completion).rejects.toThrow("remote_launch_ack_lost");
|
|
// The Runner the old server started is alive, and nothing recorded its PID.
|
|
await vi.waitFor(() => expect(runnersServing(stateDirectory)).toHaveLength(1), { timeout: 10_000 });
|
|
orphanPid = runnersServing(stateDirectory)[0]!;
|
|
expect(oldHandle.child.pid).toBeUndefined();
|
|
|
|
// The successor relaunches the same run on the same sandbox.
|
|
successorHandle = createRemoteRunnerProcessLauncher({ ...launcherInput, runner: { execute: localExecute },
|
|
onSpawn: successorSpawned })(spec);
|
|
void successorHandle.completion.catch(() => undefined);
|
|
await vi.waitFor(() => expect(successorSpawned).toHaveBeenCalledTimes(1), { timeout: 30_000 });
|
|
const successorPid = successorSpawned.mock.calls[0]![0].pid;
|
|
|
|
expect(successorPid).not.toBe(orphanPid);
|
|
await vi.waitFor(() => expect(alive(orphanPid)).toBe(false), { timeout: 10_000 });
|
|
expect(runnersServing(stateDirectory)).toEqual([successorPid]);
|
|
expect((await readFile(launcherInput.processIdentityPath, "utf8")).split("\n")[1]).toBe(String(successorPid));
|
|
} finally {
|
|
// Fresh /proc reads only: a saved PID may have been reused by now.
|
|
for (const pid of runnersServing(stateDirectory)) {
|
|
try { process.kill(pid, "SIGKILL"); } catch { /* already gone */ }
|
|
}
|
|
// The successor's supervision settles once it observes its Runner gone.
|
|
if (successorHandle) await Promise.race([successorHandle.completion.catch(() => undefined),
|
|
new Promise(resolve => setTimeout(resolve, 15_000))]);
|
|
await rm(root, { recursive: true, force: true });
|
|
}
|
|
}, 60_000);
|
|
});
|