mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(native): keep remote runnerd exit evidence and report it on failed recovery (#20907)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on Daytona start a detached `paperclip-runnerd` in the sandbox, and the controller adopts it after a restart > - On staging, recovery fails with `runner_remote_process_identity_unavailable` when the Runner is gone, but nobody can see why it exited > - The launch script sends runnerd's stdout and stderr to `/dev/null`, so only a panic or a graceful error reaches `runnerd.stderr.log`, and the recovery event does not read that file > - This pull request keeps runnerd's stderr in a bounded file in its private diagnostics directory, and adds the exit reason to the `native.remote_runner_recovery_failed` event > - The benefit is that the reason a Runner exited shows in the run log on staging without SSH, which the class R investigation needs ## Linked Issues or Issue Description **What happened?** Staging runs fail with `runner_remote_process_identity_unavailable` after the remote Runner exits (`native.remote_runner_recovery_failed` with `stage: process_verification`, `reason: process_not_running`). The event does not say why the Runner exited. **What did you expect?** The run log names the Runner's exit reason: its durable `recoverableFailure` and diagnostics, and what it wrote to stderr. **Steps to reproduce** Restart-chaos re-wake and stall cells reproduce the refusal. Each failed run log shows only `runnerLifecycle` and `runnerSettled`. Refs #20893 ## What Changed - `REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT` sends runnerd's stderr to `diagnostics/runnerd.process-stderr.log` (mode 0600, in the 0700 directory) instead of `/dev/null`. - At launch, a file above 1 MiB is rotated once to `.1`. - While runnerd runs, a detached trimmer is a disk backstop. Past 8 MiB, it keeps the last 1 MiB. It checks every 15 s and ends when runnerd ends. It is double-forked with its own argv, so it is not a child of runnerd and does not carry runnerd's arguments. runnerd's own exit reason goes to `runnerd.stderr.log`, which is never trimmed. - A symlink or non-file at that path keeps the old `/dev/null` behavior. - Stdout is unchanged (`/dev/null`). Runnerd's PID is still the launch shell's PID, so the identity marker and fingerprint are unchanged. - `remote-runner-recovery-diagnostics.ts` has a read-only `REMOTE_RUNNER_EXIT_EVIDENCE_SCRIPT`. It prints a 2 KiB tail of `runnerd.stderr.log` and of `runnerd.process-stderr.log`. It reads the session diagnostics directory first, then the shared runtime-root directory used by external Runners (Dot, Muse, grok_bot). It never reads through a symlink. Each file keeps its own share of the 4 KiB budget. - When recovery's process verification finds `process_not_running`, the diagnostic adds these fields: - `runnerRecoverableFailure`: a code-shaped string from `runner-state.json`; - `runnerDiagnostics`: the last 4 entries, redacted, 500 characters each; - `runnerExitEvidence`: the redacted file tails, 4 KiB at most. `native.remote_runner_recovery_failed` carries them. - The live process monitor's exit report now puts the durable state summary first, then the file tails. Before, it used one or the other. - `run-failure-diagnostics.ts` is unchanged. It still copies only the closed labels, so no free text reaches failure reports. ## Verification - `pnpm exec vitest run server/src/services/native-runtime/remote-runner-launch-stderr.test.ts`: the launch writer keeps stderr in the file, writes the identity marker with the child's PID, appends across relaunches, rotates an oversized file, trims a growing file while the child runs, and never writes through a symlink. - `pnpm exec vitest run server/src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts`: the evidence script output is bounded and refuses symlinks. Redaction and bounds hold, and the new fields survive the event payload. - `pnpm exec vitest run server/src/services/native-runtime/native-session-executor.test.ts` passes (1374 tests). One expectation was updated for the combined exit report. - `npx tsc --noEmit -p server/tsconfig.json` is clean. - Older runner images: the script is server-side. It adds only `wc`, `tr`, `tail`, `cat`, `mv`, `sleep` and `kill`. The script already needed `node`, `date`, `chmod`, `mv` and `printf`, so any image that launched runnerd before has these tools. The runner content ID is unchanged, so the current Daytona image runs it unmodified. The restart-chaos waves run on that image. ## Risks - Low. A sandbox disk that is already full could make the stderr redirect fail. `exec ... 2>>file` then fails, and the launch reports `runner_remote_process_identity_unavailable` the same way a launch that cannot write its identity marker does. The marker is written first, in the same directory tree. - Stderr content stays in the sandbox. Only a redacted 4 KiB tail goes into the instance's own run log (run-log data path; not Telemetry and not Observability). - No wait is added before `turn.start`. The evidence read runs only on the recovery failure path, with its own 10 s bound. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
1 parent
b2779054d4
commit
80b2369f26
5 files changed
+265
-11
No files matched your search
@@ -278,8 +278,8 @@ import { issueRecoveryActionService } from "../issue-recovery-actions.js";
|
||||
import { reportRunFailure } from "../run-failure-report.js";
|
||||
import {
|
||||
REMOTE_RECOVERY_MARKER_READ_SCRIPT, REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT,
|
||||
RemoteRunnerRecoveryError, attachRemoteRunnerRecoveryDiagnostic, describeExitedRemoteRunner, readRemoteRunnerRecoveryError,
|
||||
remoteRecoveryProbeReason,
|
||||
REMOTE_RUNNER_EXIT_EVIDENCE_SCRIPT, RemoteRunnerRecoveryError, attachRemoteRunnerRecoveryDiagnostic, boundRemoteRunnerExitEvidence,
|
||||
describeExitedRemoteRunner, describeExitedRemoteRunnerReason, readRemoteRunnerRecoveryError, remoteRecoveryProbeReason,
|
||||
} from "./remote-runner-recovery-diagnostics.js";
|
||||
import { readRemoteRunnerRecoveryProbe, REMOTE_RUNNER_RECOVERY_PROBE_BUDGET_MS } from "./remote-runner-recovery-probe.js";
|
||||
import { describeOwnFailedAttachment, inspectRemoteWarmSessionRecovery, isSettledRemoteWarmControlState, summarizePostTerminalEvents, type RemoteWarmSessionRecovery, unsettledControllerCommands } from "./remote-warm-session-recovery.js";
|
||||
@@ -14556,10 +14556,44 @@ export const REMOTE_RUNNER_AUTHORITY_ARCHIVE_SCRIPT =
|
||||
'elif test -f "$1" && test ! -L "$1" && test -f "$3" && test ! -L "$3" && command -v cmp >/dev/null 2>&1 && cmp -s "$1" "$3"; then rm -f -- "$1"; ' +
|
||||
'else exit 1; fi; base64 < "$3"';
|
||||
|
||||
/**
|
||||
* A detached remote runnerd has no controller-side stderr. Its stderr goes to
|
||||
* this file in its private diagnostics directory instead of /dev/null, so the
|
||||
* reason a Runner exited stays in the sandbox for recovery to read. The file
|
||||
* never leaves the sandbox on its own; recovery reads a short redacted tail.
|
||||
*/
|
||||
export const REMOTE_RUNNER_PROCESS_STDERR_FILE = "runnerd.process-stderr.log";
|
||||
/** Above this size the file is rotated once at launch, so each runnerd starts with a fresh file. */
|
||||
export const REMOTE_RUNNER_PROCESS_STDERR_MAX_BYTES = 1_048_576;
|
||||
/** The disk backstop for one runnerd's lifetime: above this size the trimmer keeps the last 1 MiB. */
|
||||
export const REMOTE_RUNNER_PROCESS_STDERR_TRIM_BYTES = 8_388_608;
|
||||
const REMOTE_RUNNER_PROCESS_STDERR_KEEP_BYTES = 1_048_576;
|
||||
|
||||
/**
|
||||
* The disk backstop for a running runnerd's stderr file ($1 pid, $2 file). It
|
||||
* runs detached from runnerd (double fork) with its own argv and ends with
|
||||
* runnerd. Rewriting the kept tail can drop a line runnerd appends between the
|
||||
* copy and the rewrite, so it acts only past 8 MiB, which ordinary stderr
|
||||
* volume never reaches. runnerd's own exit reason is not on this stream: it
|
||||
* writes that to `runnerd.stderr.log` itself, which no trimmer touches.
|
||||
*/
|
||||
const REMOTE_RUNNER_PROCESS_STDERR_TRIM_SCRIPT =
|
||||
`pid=$1; log=$2; while kill -0 "$pid" 2>/dev/null; do sleep 15; size=$(wc -c 2>/dev/null < "$log" | tr -d " "); ` +
|
||||
`if test "\${size:-0}" -gt ${REMOTE_RUNNER_PROCESS_STDERR_TRIM_BYTES} && test -f "$log" && test ! -L "$log"; then ` +
|
||||
`tail -c ${REMOTE_RUNNER_PROCESS_STDERR_KEEP_BYTES} -- "$log" > "$log.trim" && cat -- "$log.trim" > "$log"; rm -f -- "$log.trim"; fi; done`;
|
||||
|
||||
export const REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT =
|
||||
'set -eu; identity_path=$1; identity_nonce=$2; runner_instance_id=$3; diagnostics_directory=$4; shift 4; umask 077; test ! -L "$diagnostics_directory"; if test -e "$diagnostics_directory"; then test -d "$diagnostics_directory"; else mkdir -p -- "$diagnostics_directory"; fi; chmod 0700 "$diagnostics_directory"; started_at=$(date -u +"%Y-%m-%dT%H:%M:%S.%3NZ"); pid=$$; process_fingerprint=""; if ' +
|
||||
REMOTE_RUNNER_PROCESS_FINGERPRINT_SCRIPT +
|
||||
'; then :; else process_fingerprint=""; fi; identity_tmp="${identity_path}.tmp.$$"; printf "%s\\n%s\\n%s\\n%s\\n%s\\n" "$identity_nonce" "$$" "$started_at" "$runner_instance_id" "$process_fingerprint" > "$identity_tmp"; chmod 0600 "$identity_tmp"; mv -f -- "$identity_tmp" "$identity_path"; exec "$@"';
|
||||
'; then :; else process_fingerprint=""; fi; ' +
|
||||
// Bounded stderr evidence: rotate an oversized file once, then trim it in place while runnerd runs.
|
||||
// A symlink or non-file at the path keeps the old /dev/null behavior.
|
||||
`stderr_log="$diagnostics_directory/${REMOTE_RUNNER_PROCESS_STDERR_FILE}"; ` +
|
||||
'if test -L "$stderr_log" || { test -e "$stderr_log" && test ! -f "$stderr_log"; }; then stderr_log=/dev/null; else ' +
|
||||
'stderr_size=0; if test -f "$stderr_log"; then stderr_size=$(wc -c < "$stderr_log" | tr -d " "); fi; ' +
|
||||
`if test "\${stderr_size:-0}" -gt ${REMOTE_RUNNER_PROCESS_STDERR_MAX_BYTES}; then mv -f -- "$stderr_log" "$stderr_log.1"; fi; ` +
|
||||
`( ( exec sh -c '${REMOTE_RUNNER_PROCESS_STDERR_TRIM_SCRIPT}' paperclip-runner-stderr-trim "$pid" "$stderr_log" </dev/null >/dev/null 2>&1 ) & ); fi; ` +
|
||||
'identity_tmp="${identity_path}.tmp.$$"; printf "%s\\n%s\\n%s\\n%s\\n%s\\n" "$identity_nonce" "$$" "$started_at" "$runner_instance_id" "$process_fingerprint" > "$identity_tmp"; chmod 0600 "$identity_tmp"; mv -f -- "$identity_tmp" "$identity_path"; exec "$@" 2>>"$stderr_log"';
|
||||
|
||||
const REMOTE_RUNNER_FAILED_IDENTITY_CLEANUP_SCRIPT =
|
||||
'set -eu; identity_path=$1; expected_nonce=$2; expected_runner_id=$3; marker_wait=0; while { test ! -f "$identity_path" || test -L "$identity_path"; } && test "$marker_wait" -lt 50; do marker_wait=$((marker_wait + 1)); sleep 0.1; done; test -f "$identity_path" && test ! -L "$identity_path" || exit 3; { IFS= read -r nonce; IFS= read -r pid; IFS= read -r started_at; IFS= read -r runner_id; } < "$identity_path"; test "$nonce" = "$expected_nonce" && test "$runner_id" = "$expected_runner_id" && test -n "$started_at" || exit 4; case "$pid" in ""|*[!0-9]*) exit 4 ;; esac; test "$pid" -gt 0 || exit 4; if kill -0 "$pid" 2>/dev/null; then if test -r "/proc/$pid/cmdline"; then command_line=$(tr "\\000" "\\n" < "/proc/$pid/cmdline"); printf "%s\\n" "$command_line" | grep -Fqx -- "--runner-id" || exit 4; printf "%s\\n" "$command_line" | grep -Fqx -- "$expected_runner_id" || exit 4; fi; signal_target=$pid; if command -v ps >/dev/null 2>&1; then session_id=$(ps -o sid= -p "$pid" 2>/dev/null | tr -d " ") || true; if test "$session_id" = "$pid"; then signal_target="-$pid"; fi; fi; kill -TERM -- "$signal_target" 2>/dev/null || kill -TERM "$pid" 2>/dev/null || true; term_wait=0; while kill -0 "$pid" 2>/dev/null && test "$term_wait" -lt 50; do term_wait=$((term_wait + 1)); sleep 0.1; done; if kill -0 "$pid" 2>/dev/null; then kill -KILL -- "$signal_target" 2>/dev/null || kill -KILL "$pid" 2>/dev/null || true; kill_wait=0; while kill -0 "$pid" 2>/dev/null && test "$kill_wait" -lt 50; do kill_wait=$((kill_wait + 1)); sleep 0.1; done; fi; kill -0 "$pid" 2>/dev/null && exit 5; fi; test -f "$identity_path" && test ! -L "$identity_path" || exit 4; { IFS= read -r final_nonce; IFS= read -r final_pid; IFS= read -r final_started_at; IFS= read -r final_runner_id; } < "$identity_path"; test "$final_nonce" = "$nonce" && test "$final_pid" = "$pid" && test "$final_started_at" = "$started_at" && test "$final_runner_id" = "$runner_id" || exit 4; rm -f -- "$identity_path"';
|
||||
@@ -14704,7 +14738,15 @@ export async function verifyRemoteRunnerReattachment(input: {
|
||||
const verified = await check(undefined, probeDeadline);
|
||||
if (!verified.alive) {
|
||||
throw new RemoteRunnerRecoveryError({ stage: "process_verification", reason: verified.reason,
|
||||
...(verified.reason === "process_not_running" ? describeExitedRemoteRunner(runnerState) : {}) });
|
||||
...(verified.reason === "process_not_running" ? {
|
||||
...describeExitedRemoteRunner(runnerState),
|
||||
// Why the Runner exited, from its own durable state and sandbox-local files, so the run log names it.
|
||||
...describeExitedRemoteRunnerReason(runnerState, redactSensitiveText),
|
||||
// A session Runner logs beside its state; external (Dot, Muse, grok_bot) Runners log in the shared runtime root.
|
||||
...await readRemoteRunnerExitEvidence(runner, [posix.join(posix.dirname(stateDirectory), "diagnostics"),
|
||||
posix.join(target.remoteCwd, ".paperclip-runtime", "paperclip-runner", "diagnostics")],
|
||||
Math.min(input.recoveryDeadline ?? Number.POSITIVE_INFINITY, Date.now() + 10_000)),
|
||||
} : {}) });
|
||||
}
|
||||
// These checks authorize an attempt to reconnect, not the task itself. The
|
||||
// transport still requires authentication against the exact durable PRP key.
|
||||
@@ -14716,6 +14758,23 @@ export async function verifyRemoteRunnerReattachment(input: {
|
||||
};
|
||||
}
|
||||
|
||||
/** Best effort: a missing, slow or unreadable file leaves the recovery outcome unchanged. */
|
||||
async function readRemoteRunnerExitEvidence(runner: CommandManagedRuntimeRunner, diagnosticsDirectories: string[], deadline: number):
|
||||
Promise<{ runnerExitEvidence?: string }> {
|
||||
try {
|
||||
const result = await readRemoteRunnerRecoveryProbe({
|
||||
runner, stage: "process_verification", deadline, readOnly: true,
|
||||
request: { command: "sh", args: ["-c", REMOTE_RUNNER_EXIT_EVIDENCE_SCRIPT, "paperclip-runner-exit-evidence", ...diagnosticsDirectories],
|
||||
bypassSession: true },
|
||||
});
|
||||
if (result.exitCode !== 0 || result.timedOut) return {};
|
||||
const evidence = boundRemoteRunnerExitEvidence(result.stdout, redactSensitiveText);
|
||||
return evidence ? { runnerExitEvidence: evidence } : {};
|
||||
} catch {
|
||||
return {};
|
||||
}
|
||||
}
|
||||
|
||||
export async function verifyRemoteRunnerOrSettledExit(
|
||||
input: Parameters<typeof verifyRemoteRunnerReattachment>[0] & { providerAttempt: number },
|
||||
) {
|
||||
@@ -15102,7 +15161,7 @@ export function createRemoteRunnerProcessLauncher(input: {
|
||||
command: "sh",
|
||||
args: [
|
||||
"-c",
|
||||
'set -eu; directory=$1; file="$directory/runnerd.stderr.log"; test -d "$directory" && test ! -L "$directory" && test -f "$file" && test ! -L "$file"; tail -c 65536 -- "$file"',
|
||||
REMOTE_RUNNER_EXIT_EVIDENCE_SCRIPT,
|
||||
"paperclip-runner-diagnostics",
|
||||
input.diagnosticsDirectory,
|
||||
],
|
||||
@@ -15139,7 +15198,8 @@ export function createRemoteRunnerProcessLauncher(input: {
|
||||
code: null,
|
||||
signal: null,
|
||||
stdout: "",
|
||||
stderr: diagnosticTail || stateSummary,
|
||||
// The durable state names a graceful exit; the files name a panic or what runnerd wrote to stderr.
|
||||
stderr: diagnosticTail ? `${stateSummary}\n${diagnosticTail}` : stateSummary,
|
||||
};
|
||||
}
|
||||
})().catch((error: unknown) => {
|
||||
|
||||
Reference in new issue
Block a user