mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-19 16:29:42 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on Daytona start a detached `paperclip-runnerd` in the sandbox, and the controller adopts it after a restart > - On staging, recovery fails with `runner_remote_process_identity_unavailable` when the Runner is gone, but nobody can see why it exited > - The launch script sends runnerd's stdout and stderr to `/dev/null`, so only a panic or a graceful error reaches `runnerd.stderr.log`, and the recovery event does not read that file > - This pull request keeps runnerd's stderr in a bounded file in its private diagnostics directory, and adds the exit reason to the `native.remote_runner_recovery_failed` event > - The benefit is that the reason a Runner exited shows in the run log on staging without SSH, which the class R investigation needs ## Linked Issues or Issue Description **What happened?** Staging runs fail with `runner_remote_process_identity_unavailable` after the remote Runner exits (`native.remote_runner_recovery_failed` with `stage: process_verification`, `reason: process_not_running`). The event does not say why the Runner exited. **What did you expect?** The run log names the Runner's exit reason: its durable `recoverableFailure` and diagnostics, and what it wrote to stderr. **Steps to reproduce** Restart-chaos re-wake and stall cells reproduce the refusal. Each failed run log shows only `runnerLifecycle` and `runnerSettled`. Refs #20893 ## What Changed - `REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT` sends runnerd's stderr to `diagnostics/runnerd.process-stderr.log` (mode 0600, in the 0700 directory) instead of `/dev/null`. - At launch, a file above 1 MiB is rotated once to `.1`. - While runnerd runs, a detached trimmer is a disk backstop. Past 8 MiB, it keeps the last 1 MiB. It checks every 15 s and ends when runnerd ends. It is double-forked with its own argv, so it is not a child of runnerd and does not carry runnerd's arguments. runnerd's own exit reason goes to `runnerd.stderr.log`, which is never trimmed. - A symlink or non-file at that path keeps the old `/dev/null` behavior. - Stdout is unchanged (`/dev/null`). Runnerd's PID is still the launch shell's PID, so the identity marker and fingerprint are unchanged. - `remote-runner-recovery-diagnostics.ts` has a read-only `REMOTE_RUNNER_EXIT_EVIDENCE_SCRIPT`. It prints a 2 KiB tail of `runnerd.stderr.log` and of `runnerd.process-stderr.log`. It reads the session diagnostics directory first, then the shared runtime-root directory used by external Runners (Dot, Muse, grok_bot). It never reads through a symlink. Each file keeps its own share of the 4 KiB budget. - When recovery's process verification finds `process_not_running`, the diagnostic adds these fields: - `runnerRecoverableFailure`: a code-shaped string from `runner-state.json`; - `runnerDiagnostics`: the last 4 entries, redacted, 500 characters each; - `runnerExitEvidence`: the redacted file tails, 4 KiB at most. `native.remote_runner_recovery_failed` carries them. - The live process monitor's exit report now puts the durable state summary first, then the file tails. Before, it used one or the other. - `run-failure-diagnostics.ts` is unchanged. It still copies only the closed labels, so no free text reaches failure reports. ## Verification - `pnpm exec vitest run server/src/services/native-runtime/remote-runner-launch-stderr.test.ts`: the launch writer keeps stderr in the file, writes the identity marker with the child's PID, appends across relaunches, rotates an oversized file, trims a growing file while the child runs, and never writes through a symlink. - `pnpm exec vitest run server/src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts`: the evidence script output is bounded and refuses symlinks. Redaction and bounds hold, and the new fields survive the event payload. - `pnpm exec vitest run server/src/services/native-runtime/native-session-executor.test.ts` passes (1374 tests). One expectation was updated for the combined exit report. - `npx tsc --noEmit -p server/tsconfig.json` is clean. - Older runner images: the script is server-side. It adds only `wc`, `tr`, `tail`, `cat`, `mv`, `sleep` and `kill`. The script already needed `node`, `date`, `chmod`, `mv` and `printf`, so any image that launched runnerd before has these tools. The runner content ID is unchanged, so the current Daytona image runs it unmodified. The restart-chaos waves run on that image. ## Risks - Low. A sandbox disk that is already full could make the stderr redirect fail. `exec ... 2>>file` then fails, and the launch reports `runner_remote_process_identity_unavailable` the same way a launch that cannot write its identity marker does. The marker is written first, in the same directory tree. - Stderr content stays in the sandbox. Only a redacted 4 KiB tail goes into the instance's own run log (run-log data path; not Telemetry and not Observability). - No wait is added before `turn.start`. The evidence read runs only on the recovery failure path, with its own 10 s bound. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
75 lines
4.4 KiB
TypeScript
75 lines
4.4 KiB
TypeScript
import { spawn } from "node:child_process";
|
|
import { lstat, mkdir, mkdtemp, readFile, rm, stat, symlink, writeFile } from "node:fs/promises";
|
|
import { tmpdir } from "node:os";
|
|
import { join } from "node:path";
|
|
import { describe, expect, it } from "vitest";
|
|
import {
|
|
REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT, REMOTE_RUNNER_PROCESS_STDERR_FILE, REMOTE_RUNNER_PROCESS_STDERR_MAX_BYTES,
|
|
REMOTE_RUNNER_PROCESS_STDERR_TRIM_BYTES,
|
|
} from "./native-session-executor.js";
|
|
|
|
/** Runs the launch writer as the remote launcher does, with a Node child standing in for runnerd. */
|
|
async function launch(root: string, childSource: string, script = REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT) {
|
|
const child = spawn("/bin/sh", ["-c", script, "paperclip-runner-child", join(root, "state", "identity"), "nonce", "runner-id",
|
|
join(root, "diagnostics"), process.execPath, "-e", childSource, "--", "--runner-id", "runner-id"],
|
|
{ stdio: ["ignore", "pipe", "pipe"] });
|
|
let stderr = "";
|
|
child.stderr.on("data", chunk => { stderr += String(chunk); });
|
|
const code = await new Promise<number | null>(resolve => child.once("exit", resolve));
|
|
return { code, stderr, pid: child.pid! };
|
|
}
|
|
|
|
describe("remote runnerd stderr evidence", () => {
|
|
it("keeps runnerd's stderr in its private diagnostics directory and still writes the identity marker", async () => {
|
|
const root = await mkdtemp(join(tmpdir(), "runner-launch-stderr-"));
|
|
try {
|
|
await mkdir(join(root, "state"));
|
|
const result = await launch(root, 'process.stderr.write("paperclip-runnerd: lease expired\\n"); process.exit(3)');
|
|
expect(result.code).toBe(3);
|
|
// Nothing reaches the launcher's own stderr; the reason stays in the sandbox file.
|
|
expect(result.stderr).toBe("");
|
|
const file = join(root, "diagnostics", REMOTE_RUNNER_PROCESS_STDERR_FILE);
|
|
expect(await readFile(file, "utf8")).toBe("paperclip-runnerd: lease expired\n");
|
|
expect((await stat(file)).mode & 0o077).toBe(0);
|
|
expect((await readFile(join(root, "state", "identity"), "utf8")).split("\n")[1]).toBe(String(result.pid));
|
|
// A second launch appends: the earlier exit reason survives the relaunch.
|
|
await launch(root, 'process.stderr.write("second\\n")');
|
|
expect(await readFile(file, "utf8")).toBe("paperclip-runnerd: lease expired\nsecond\n");
|
|
} finally { await rm(root, { recursive: true, force: true }); }
|
|
});
|
|
|
|
it("rotates an oversized file at launch and trims it while runnerd runs", async () => {
|
|
const root = await mkdtemp(join(tmpdir(), "runner-launch-rotate-"));
|
|
try {
|
|
await mkdir(join(root, "state"));
|
|
await mkdir(join(root, "diagnostics"), { mode: 0o700 });
|
|
const file = join(root, "diagnostics", REMOTE_RUNNER_PROCESS_STDERR_FILE);
|
|
await writeFile(file, "o".repeat(REMOTE_RUNNER_PROCESS_STDERR_MAX_BYTES + 1));
|
|
await launch(root, 'process.stderr.write("fresh\\n")');
|
|
expect((await stat(`${file}.1`)).size).toBe(REMOTE_RUNNER_PROCESS_STDERR_MAX_BYTES + 1);
|
|
// The trimmer checks every 15 s in production; check it quickly here.
|
|
const fast = REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT.replace("sleep 15", "sleep 0.2");
|
|
expect(fast).not.toBe(REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT);
|
|
await launch(root, `process.stderr.write("z".repeat(${REMOTE_RUNNER_PROCESS_STDERR_TRIM_BYTES + 4096}) + "\\nend\\n"); setTimeout(() => {}, 1500)`, fast);
|
|
const size = (await stat(file)).size;
|
|
// Past the 8 MiB backstop the trimmer keeps the last 1 MiB, the most recent lines included.
|
|
expect(size).toBeLessThanOrEqual(REMOTE_RUNNER_PROCESS_STDERR_MAX_BYTES + 4096);
|
|
expect((await readFile(file, "utf8")).endsWith("end\n")).toBe(true);
|
|
} finally { await rm(root, { recursive: true, force: true }); }
|
|
});
|
|
|
|
it("never writes through a symlink at the stderr path", async () => {
|
|
const root = await mkdtemp(join(tmpdir(), "runner-launch-symlink-"));
|
|
try {
|
|
await mkdir(join(root, "state"));
|
|
await mkdir(join(root, "diagnostics"), { mode: 0o700 });
|
|
await writeFile(join(root, "target"), "");
|
|
await symlink(join(root, "target"), join(root, "diagnostics", REMOTE_RUNNER_PROCESS_STDERR_FILE));
|
|
const result = await launch(root, 'process.stderr.write("must not land\\n")');
|
|
expect(result.code).toBe(0);
|
|
expect(await readFile(join(root, "target"), "utf8")).toBe("");
|
|
expect((await lstat(join(root, "diagnostics", REMOTE_RUNNER_PROCESS_STDERR_FILE))).isSymbolicLink()).toBe(true);
|
|
} finally { await rm(root, { recursive: true, force: true }); }
|
|
});
|
|
});
|