Files
PaperClipAI/server/src/services/native-runtime/remote-runner-launch-stderr.test.ts
T
DottaandClaude 80b2369f26 fix(native): keep remote runnerd exit evidence and report it on failed recovery (#20907)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on Daytona start a detached `paperclip-runnerd` in the
sandbox, and the controller adopts it after a restart
> - On staging, recovery fails with
`runner_remote_process_identity_unavailable` when the Runner is gone,
but nobody can see why it exited
> - The launch script sends runnerd's stdout and stderr to `/dev/null`,
so only a panic or a graceful error reaches `runnerd.stderr.log`, and
the recovery event does not read that file
> - This pull request keeps runnerd's stderr in a bounded file in its
private diagnostics directory, and adds the exit reason to the
`native.remote_runner_recovery_failed` event
> - The benefit is that the reason a Runner exited shows in the run log
on staging without SSH, which the class R investigation needs

## Linked Issues or Issue Description

**What happened?**
Staging runs fail with `runner_remote_process_identity_unavailable`
after the remote Runner exits (`native.remote_runner_recovery_failed`
with `stage: process_verification`, `reason: process_not_running`). The
event does not say why the Runner exited.

**What did you expect?**
The run log names the Runner's exit reason: its durable
`recoverableFailure` and diagnostics, and what it wrote to stderr.

**Steps to reproduce**
Restart-chaos re-wake and stall cells reproduce the refusal. Each failed
run log shows only `runnerLifecycle` and `runnerSettled`.

Refs #20893

## What Changed

- `REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT` sends runnerd's stderr to
`diagnostics/runnerd.process-stderr.log` (mode 0600, in the 0700
directory) instead of `/dev/null`.
  - At launch, a file above 1 MiB is rotated once to `.1`.
- While runnerd runs, a detached trimmer is a disk backstop. Past 8 MiB,
it keeps the last 1 MiB. It checks every 15 s and ends when runnerd
ends. It is double-forked with its own argv, so it is not a child of
runnerd and does not carry runnerd's arguments. runnerd's own exit
reason goes to `runnerd.stderr.log`, which is never trimmed.
- A symlink or non-file at that path keeps the old `/dev/null` behavior.
- Stdout is unchanged (`/dev/null`). Runnerd's PID is still the launch
shell's PID, so the identity marker and fingerprint are unchanged.
- `remote-runner-recovery-diagnostics.ts` has a read-only
`REMOTE_RUNNER_EXIT_EVIDENCE_SCRIPT`. It prints a 2 KiB tail of
`runnerd.stderr.log` and of `runnerd.process-stderr.log`. It reads the
session diagnostics directory first, then the shared runtime-root
directory used by external Runners (Dot, Muse, grok_bot). It never reads
through a symlink. Each file keeps its own share of the 4 KiB budget.
- When recovery's process verification finds `process_not_running`, the
diagnostic adds these fields:
- `runnerRecoverableFailure`: a code-shaped string from
`runner-state.json`;
- `runnerDiagnostics`: the last 4 entries, redacted, 500 characters
each;
  - `runnerExitEvidence`: the redacted file tails, 4 KiB at most.
  
  `native.remote_runner_recovery_failed` carries them.
- The live process monitor's exit report now puts the durable state
summary first, then the file tails. Before, it used one or the other.
- `run-failure-diagnostics.ts` is unchanged. It still copies only the
closed labels, so no free text reaches failure reports.

## Verification

- `pnpm exec vitest run
server/src/services/native-runtime/remote-runner-launch-stderr.test.ts`:
the launch writer keeps stderr in the file, writes the identity marker
with the child's PID, appends across relaunches, rotates an oversized
file, trims a growing file while the child runs, and never writes
through a symlink.
- `pnpm exec vitest run
server/src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts`:
the evidence script output is bounded and refuses symlinks. Redaction
and bounds hold, and the new fields survive the event payload.
- `pnpm exec vitest run
server/src/services/native-runtime/native-session-executor.test.ts`
passes (1374 tests). One expectation was updated for the combined exit
report.
- `npx tsc --noEmit -p server/tsconfig.json` is clean.
- Older runner images: the script is server-side. It adds only `wc`,
`tr`, `tail`, `cat`, `mv`, `sleep` and `kill`. The script already needed
`node`, `date`, `chmod`, `mv` and `printf`, so any image that launched
runnerd before has these tools. The runner content ID is unchanged, so
the current Daytona image runs it unmodified. The restart-chaos waves
run on that image.

## Risks

- Low. A sandbox disk that is already full could make the stderr
redirect fail. `exec ... 2>>file` then fails, and the launch reports
`runner_remote_process_identity_unavailable` the same way a launch that
cannot write its identity marker does. The marker is written first, in
the same directory tree.
- Stderr content stays in the sandbox. Only a redacted 4 KiB tail goes
into the instance's own run log (run-log data path; not Telemetry and
not Observability).
- No wait is added before `turn.start`. The evidence read runs only on
the recovery failure path, with its own 10 s bound.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-11 15:31:04 -06:00

75 lines
4.4 KiB
TypeScript

import { spawn } from "node:child_process";
import { lstat, mkdir, mkdtemp, readFile, rm, stat, symlink, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import {
REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT, REMOTE_RUNNER_PROCESS_STDERR_FILE, REMOTE_RUNNER_PROCESS_STDERR_MAX_BYTES,
REMOTE_RUNNER_PROCESS_STDERR_TRIM_BYTES,
} from "./native-session-executor.js";
/** Runs the launch writer as the remote launcher does, with a Node child standing in for runnerd. */
async function launch(root: string, childSource: string, script = REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT) {
const child = spawn("/bin/sh", ["-c", script, "paperclip-runner-child", join(root, "state", "identity"), "nonce", "runner-id",
join(root, "diagnostics"), process.execPath, "-e", childSource, "--", "--runner-id", "runner-id"],
{ stdio: ["ignore", "pipe", "pipe"] });
let stderr = "";
child.stderr.on("data", chunk => { stderr += String(chunk); });
const code = await new Promise<number | null>(resolve => child.once("exit", resolve));
return { code, stderr, pid: child.pid! };
}
describe("remote runnerd stderr evidence", () => {
it("keeps runnerd's stderr in its private diagnostics directory and still writes the identity marker", async () => {
const root = await mkdtemp(join(tmpdir(), "runner-launch-stderr-"));
try {
await mkdir(join(root, "state"));
const result = await launch(root, 'process.stderr.write("paperclip-runnerd: lease expired\\n"); process.exit(3)');
expect(result.code).toBe(3);
// Nothing reaches the launcher's own stderr; the reason stays in the sandbox file.
expect(result.stderr).toBe("");
const file = join(root, "diagnostics", REMOTE_RUNNER_PROCESS_STDERR_FILE);
expect(await readFile(file, "utf8")).toBe("paperclip-runnerd: lease expired\n");
expect((await stat(file)).mode & 0o077).toBe(0);
expect((await readFile(join(root, "state", "identity"), "utf8")).split("\n")[1]).toBe(String(result.pid));
// A second launch appends: the earlier exit reason survives the relaunch.
await launch(root, 'process.stderr.write("second\\n")');
expect(await readFile(file, "utf8")).toBe("paperclip-runnerd: lease expired\nsecond\n");
} finally { await rm(root, { recursive: true, force: true }); }
});
it("rotates an oversized file at launch and trims it while runnerd runs", async () => {
const root = await mkdtemp(join(tmpdir(), "runner-launch-rotate-"));
try {
await mkdir(join(root, "state"));
await mkdir(join(root, "diagnostics"), { mode: 0o700 });
const file = join(root, "diagnostics", REMOTE_RUNNER_PROCESS_STDERR_FILE);
await writeFile(file, "o".repeat(REMOTE_RUNNER_PROCESS_STDERR_MAX_BYTES + 1));
await launch(root, 'process.stderr.write("fresh\\n")');
expect((await stat(`${file}.1`)).size).toBe(REMOTE_RUNNER_PROCESS_STDERR_MAX_BYTES + 1);
// The trimmer checks every 15 s in production; check it quickly here.
const fast = REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT.replace("sleep 15", "sleep 0.2");
expect(fast).not.toBe(REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT);
await launch(root, `process.stderr.write("z".repeat(${REMOTE_RUNNER_PROCESS_STDERR_TRIM_BYTES + 4096}) + "\\nend\\n"); setTimeout(() => {}, 1500)`, fast);
const size = (await stat(file)).size;
// Past the 8 MiB backstop the trimmer keeps the last 1 MiB, the most recent lines included.
expect(size).toBeLessThanOrEqual(REMOTE_RUNNER_PROCESS_STDERR_MAX_BYTES + 4096);
expect((await readFile(file, "utf8")).endsWith("end\n")).toBe(true);
} finally { await rm(root, { recursive: true, force: true }); }
});
it("never writes through a symlink at the stderr path", async () => {
const root = await mkdtemp(join(tmpdir(), "runner-launch-symlink-"));
try {
await mkdir(join(root, "state"));
await mkdir(join(root, "diagnostics"), { mode: 0o700 });
await writeFile(join(root, "target"), "");
await symlink(join(root, "target"), join(root, "diagnostics", REMOTE_RUNNER_PROCESS_STDERR_FILE));
const result = await launch(root, 'process.stderr.write("must not land\\n")');
expect(result.code).toBe(0);
expect(await readFile(join(root, "target"), "utf8")).toBe("");
expect((await lstat(join(root, "diagnostics", REMOTE_RUNNER_PROCESS_STDERR_FILE))).isSymbolicLink()).toBe(true);
} finally { await rm(root, { recursive: true, force: true }); }
});
});