mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 20:29:18 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm agent session keeps its Runner alive in a sandbox between
turns. The server closes an idle session after 5 minutes. A monitor loop
polls the Runner process with `REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT`
until the process exits.
> - The check reads `/proc/<pid>/cmdline` and exits 48
(`process_runner_id_missing`) when the command line does not hold
`--runner-id`.
> - A Linux process that is exiting releases its memory before it
becomes a zombie. For that short time its command line reads as empty,
but the process still answers `kill -0` and its stat is not yet `Z`.
> - So a check that lands in that window reports exit 48, and the close
fails although the Runner is exiting as asked.
> - This pull request makes the check wait up to 1 second for a process
of the saved generation whose command line is empty. If the process
becomes a zombie or goes away, the check exits 45
(`process_not_running`).
> - The benefit is that an idle warm close no longer fails, and is no
longer logged as a failure, when it catches the Runner while it exits.
## Linked Issues or Issue Description
Refs #21853
**What happened?**
Restart-chaos rerun 11 A1 (master 22d3f9a03c, wave
w-20261117T084017-s7106) logged `idle warm native session close failed`
with `runner_remote_process_identity_unavailable
[process_verification/process_runner_id_missing]` for run `6871fda4`
(t011, Codex, Daytona), close attempt 1, at 09:09:33Z.
**Evidence**
- The run succeeded at 09:04:34Z. The idle close ran 5 minutes later.
The server logged `transport_close_requested` at 09:09:33Z, then the
close failure in the same second.
- The Runner marker holds pid 714, with a saved process generation
(`linux:<boot>:<ticks>`). So the generation check passed, and the
process at pid 714 was still the Runner.
- The Runner's log-rotation watcher (`while kill -0 "$pid"; do sleep 15;
…`) started its last `sleep 15` at 09:09:21Z, so pid 714 was alive then.
The sandbox capture at 09:09:35Z shows no pid 714. So the Runner exited
between 09:09:21Z and 09:09:35Z, during the close.
- Exit 48 needs a live pid whose generation matches and whose command
line lacks `--runner-id`. The only way for the same process to lose its
command line is the exit window, after the kernel releases its memory
and before it becomes a zombie.
- The next evidence line shows the sandbox Runner `suspended`,
`settled`, identity matching, and no Runner process.
`warmCloseFailureProvesOwnerGone` accepted this proof (#21853) and
released the owner on attempt 1 (`warm native session's Runner or
sandbox is gone; released its owner`). No retry was needed.
`warm-shutdown.json` reports pass, 0 leaked sandboxes and the
environment deleted.
**Expected behavior**
A check that catches the Runner while it exits reports
`process_not_running`, so the monitor confirms the exit and the idle
close completes cleanly.
**Why the close failed spuriously**
The monitor loop in `native-session-executor.ts` accepts only exit 0
(alive, poll again) or exit 45 (exit confirmed). Any other exit code
throws `RemoteRunnerRecoveryError`, which fails the close.
## What Changed
-
`server/src/services/native-runtime/remote-runner-recovery-diagnostics.ts`:
in `REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT`, when the command line lacks
the runner id, is empty, and the saved process generation matched, the
check polls `/proc/<pid>/stat` and `kill -0` every 50 ms for up to 1
second. It exits 45 when the process becomes `Z` or `X`, or is gone.
Otherwise it exits 48, as before.
-
`server/src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts`:
two regression tests. A process of the saved generation with an empty
command line and state `R` becomes a zombie, or is reaped, while the
check runs: the check exits 45. The existing cases are unchanged: a live
process with an empty command line (state `S`) still exits 48, and
without a saved generation a zombie still exits 48. The helper timeout
goes from 2 to 5 seconds, because the unchanged empty-command-line case
now waits 1 second.
## Verification
- `cd server && npx vitest run
src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts
src/services/native-runtime/sandbox-runner-port-holders.test.ts`: 18
passed.
- Red on master: with the master script, both new tests fail with
`expected 48 to be 45`.
- `pnpm verify` selected the full server Vitest suite, because many
server modules import this file. I stopped that local run so that it did
not load the machine while a chaos wave ran. Required CI supplies the
broad coverage.- No chaos wave was run, and no Daytona sandbox was
touched. Rerun 11 holds the slot.
## Risks
- Low risk. The wait runs only when the command line is empty and the
saved generation matches, so it applies only to the Runner's own process
while it exits. A live process that keeps an empty command line still
exits 48, after 1 extra second.
- A check without a saved generation does not change, because there a
pid can be reused.
- The script changes no persisted state. Markers, state files and
checkpoints saved by the previous release are read the same way.
- The wait is in the recovery and close checks only. It adds no work
before `turn.start`.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits) for the log analysis, code and
tests.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
267 lines
13 KiB
TypeScript
267 lines
13 KiB
TypeScript
export const REMOTE_RUNNER_RECOVERY_ERROR_CODE = "runner_remote_process_identity_unavailable";
|
|
|
|
const stages = ["runner_state", "identity_marker", "process_verification"] as const;
|
|
const reasons = [
|
|
"marker_missing", "marker_unsafe", "marker_read_failed", "marker_invalid",
|
|
"fingerprint_missing", "process_not_running", "process_identity_changed", "process_runner_id_missing",
|
|
"fingerprint_unavailable", "process_generation_changed", "probe_timeout",
|
|
"probe_transport_failed", "probe_failed",
|
|
"state_read_failed", "state_invalid",
|
|
] as const;
|
|
|
|
const runnerLifecycles = [
|
|
"connecting", "ready", "backpressure", "reconnecting", "session_opening", "session_open",
|
|
"turn_starting", "turn_active", "running", "suspended", "recoverable_failure", "reconciliation_required",
|
|
"provider_exited", "unrecoverable", "fenced", "failed", "completed", "closed",
|
|
"other", "unavailable",
|
|
] as const;
|
|
|
|
export interface RemoteRunnerRecoveryDiagnostic {
|
|
stage: typeof stages[number];
|
|
reason: typeof reasons[number];
|
|
runnerLifecycle?: typeof runnerLifecycles[number];
|
|
runnerSettled?: boolean;
|
|
/** Present only as `false`: the controller journal proves no turn command was ever issued. */
|
|
providerTurnIssued?: false;
|
|
/** process_not_running: the exited Runner's own recoverable-failure code from its durable state. */
|
|
runnerRecoverableFailure?: string;
|
|
/** process_not_running: the last durable state diagnostics, redacted and bounded. */
|
|
runnerDiagnostics?: string[];
|
|
/** process_not_running: a redacted tail of the Runner's sandbox-local stderr and exit diagnostic files. */
|
|
runnerExitEvidence?: string;
|
|
}
|
|
|
|
/** Bounds for the exit evidence a recovery event carries. */
|
|
export const REMOTE_RUNNER_EXIT_EVIDENCE_MAX_CHARS = 4_096;
|
|
const RUNNER_DIAGNOSTIC_MAX_CHARS = 500;
|
|
const RUNNER_DIAGNOSTIC_COUNT = 4;
|
|
const RECOVERABLE_FAILURE_PATTERN = /^[A-Za-z0-9_.:-]{1,120}$/;
|
|
|
|
/**
|
|
* Prints a bounded tail of the exited Runner's sandbox-local exit evidence: the
|
|
* bounded exit diagnostic runnerd writes itself (`runnerd.stderr.log`) and its
|
|
* redirected stderr (`runnerd.process-stderr.log`). Each argument is a candidate
|
|
* diagnostics directory, most specific first; the first that holds either file
|
|
* is the one read. Read-only; missing or unsafe files print nothing.
|
|
*/
|
|
export const REMOTE_RUNNER_EXIT_EVIDENCE_SCRIPT =
|
|
'for directory in "$@"; do test -d "$directory" && test ! -L "$directory" || continue; found=""; ' +
|
|
'for name in runnerd.stderr.log runnerd.process-stderr.log; do file="$directory/$name"; ' +
|
|
'if test -f "$file" && test ! -L "$file"; then found=1; printf "== %s\\n" "$name"; tail -c 2048 -- "$file"; printf "\\n"; fi; done; ' +
|
|
'test -z "$found" || exit 0; done; exit 0';
|
|
|
|
/** The durable reason an exited Runner recorded, redacted and bounded for a run event. */
|
|
export function describeExitedRemoteRunnerReason(state: Record<string, unknown> | null, redact: (text: string) => string):
|
|
Pick<RemoteRunnerRecoveryDiagnostic, "runnerRecoverableFailure" | "runnerDiagnostics"> {
|
|
if (!state) return {};
|
|
const failure = typeof state.recoverableFailure === "string" ? state.recoverableFailure
|
|
: typeof state.recoverable_failure === "string" ? state.recoverable_failure : null;
|
|
const diagnostics = Array.isArray(state.diagnostics)
|
|
? state.diagnostics.filter((value): value is string => typeof value === "string").slice(-RUNNER_DIAGNOSTIC_COUNT)
|
|
.map(value => redact(value).slice(-RUNNER_DIAGNOSTIC_MAX_CHARS))
|
|
: [];
|
|
return {
|
|
...(failure && RECOVERABLE_FAILURE_PATTERN.test(failure) ? { runnerRecoverableFailure: failure } : {}),
|
|
...(diagnostics.length ? { runnerDiagnostics: diagnostics } : {}),
|
|
};
|
|
}
|
|
|
|
/** The tail of the exit evidence script's output, redacted and bounded; null when there is none. */
|
|
export function boundRemoteRunnerExitEvidence(output: string, redact: (text: string) => string): string | null {
|
|
// Each file gets its own share of the budget, so a long stderr tail cannot crowd out the exit diagnostic.
|
|
const sections = redact(output).split(/^(?=== runnerd\.)/m).map(section => section.trim()).filter(Boolean);
|
|
if (!sections.length) return null;
|
|
const share = Math.floor(REMOTE_RUNNER_EXIT_EVIDENCE_MAX_CHARS / sections.length) - 1;
|
|
return sections.map(section => {
|
|
if (section.length <= share) return section;
|
|
const header = section.slice(0, section.indexOf("\n") + 1);
|
|
return header + section.slice(-(share - header.length));
|
|
}).join("\n");
|
|
}
|
|
|
|
export function describeExitedRemoteRunner(state: Record<string, unknown>): Pick<RemoteRunnerRecoveryDiagnostic,
|
|
"runnerLifecycle" | "runnerSettled"> {
|
|
const lifecycle = state.lifecycle;
|
|
return {
|
|
runnerLifecycle: typeof lifecycle !== "string" ? "unavailable"
|
|
: runnerLifecycles.find(candidate => candidate === lifecycle && candidate !== "other" && candidate !== "unavailable") ?? "other",
|
|
runnerSettled: Array.isArray(state.outbox) && state.outbox.length === 0 &&
|
|
state.pendingTerminalDelivery == null && state.pendingProviderCleanup == null && state.warmTransition == null,
|
|
};
|
|
}
|
|
|
|
/**
|
|
* The Runner exits with these codes after it stops its provider and saves its
|
|
* state, because only a new process with a fresh bootstrap ticket can
|
|
* reconnect. The replacement continues from the same cursor.
|
|
*/
|
|
const FRESH_BOOTSTRAP_RECOVERABLE_FAILURES = new Set([
|
|
"lease_expired_requires_bootstrap", "transport_reconnect_deadline_exceeded", "transport_reconnect_grace_exceeded",
|
|
]);
|
|
|
|
export function exitedRunnerRequestedFreshBootstrap(
|
|
diagnostic: Pick<RemoteRunnerRecoveryDiagnostic, "runnerLifecycle" | "runnerRecoverableFailure">,
|
|
): boolean {
|
|
return diagnostic.runnerLifecycle === "recoverable_failure"
|
|
&& FRESH_BOOTSTRAP_RECOVERABLE_FAILURES.has(diagnostic.runnerRecoverableFailure ?? "");
|
|
}
|
|
|
|
/** The saved state of an exited Runner that asked for a fresh bootstrap, re-read after its exit. */
|
|
export function runnerStateRequestsFreshBootstrap(state: Record<string, unknown>): boolean {
|
|
return exitedRunnerRequestedFreshBootstrap({ ...describeExitedRemoteRunner(state),
|
|
...describeExitedRemoteRunnerReason(state, text => text) })
|
|
&& state.pendingTerminalDelivery == null && state.warmTransition == null;
|
|
}
|
|
|
|
const transportDiagnostics = new WeakMap<object, RemoteRunnerRecoveryDiagnostic>();
|
|
|
|
export function attachRemoteRunnerRecoveryDiagnostic(error: unknown, diagnostic: RemoteRunnerRecoveryDiagnostic): unknown {
|
|
if (error && typeof error === "object") {
|
|
transportDiagnostics.set(error, diagnostic);
|
|
return error;
|
|
}
|
|
return new RemoteRunnerRecoveryError(diagnostic);
|
|
}
|
|
|
|
export class RemoteRunnerRecoveryError extends Error {
|
|
readonly code = REMOTE_RUNNER_RECOVERY_ERROR_CODE;
|
|
|
|
constructor(readonly remoteRunnerRecovery: RemoteRunnerRecoveryDiagnostic) {
|
|
super(REMOTE_RUNNER_RECOVERY_ERROR_CODE);
|
|
}
|
|
}
|
|
|
|
export function readRemoteRunnerRecoveryDiagnostic(value: unknown): RemoteRunnerRecoveryDiagnostic | null {
|
|
if (!value || typeof value !== "object") return null;
|
|
try {
|
|
const { stage, reason, runnerLifecycle, runnerSettled, providerTurnIssued, runnerRecoverableFailure, runnerDiagnostics,
|
|
runnerExitEvidence } = value as Record<string, unknown>;
|
|
if (!stages.some(candidate => candidate === stage) || !reasons.some(candidate => candidate === reason)) return null;
|
|
const diagnostics = Array.isArray(runnerDiagnostics)
|
|
? runnerDiagnostics.filter((entry): entry is string => typeof entry === "string").slice(-RUNNER_DIAGNOSTIC_COUNT)
|
|
.map(entry => entry.slice(-RUNNER_DIAGNOSTIC_MAX_CHARS))
|
|
: [];
|
|
return {
|
|
stage, reason,
|
|
...(runnerLifecycles.some(candidate => candidate === runnerLifecycle) ? { runnerLifecycle } : {}),
|
|
...(typeof runnerSettled === "boolean" ? { runnerSettled } : {}),
|
|
...(providerTurnIssued === false ? { providerTurnIssued } : {}),
|
|
...(typeof runnerRecoverableFailure === "string" && RECOVERABLE_FAILURE_PATTERN.test(runnerRecoverableFailure)
|
|
? { runnerRecoverableFailure } : {}),
|
|
...(diagnostics.length ? { runnerDiagnostics: diagnostics } : {}),
|
|
...(typeof runnerExitEvidence === "string" && runnerExitEvidence
|
|
? { runnerExitEvidence: runnerExitEvidence.slice(-REMOTE_RUNNER_EXIT_EVIDENCE_MAX_CHARS) } : {}),
|
|
} as RemoteRunnerRecoveryDiagnostic;
|
|
} catch {
|
|
return null;
|
|
}
|
|
}
|
|
|
|
export function readRemoteRunnerRecoveryError(error: unknown): RemoteRunnerRecoveryDiagnostic | null {
|
|
let candidate = error;
|
|
for (let depth = 0; depth < 4 && candidate && typeof candidate === "object"; depth++) {
|
|
const transportDiagnostic = transportDiagnostics.get(candidate);
|
|
if (transportDiagnostic) return transportDiagnostic;
|
|
try {
|
|
const record = candidate as Record<string, unknown>;
|
|
if (record.code === REMOTE_RUNNER_RECOVERY_ERROR_CODE) return readRemoteRunnerRecoveryDiagnostic(record.remoteRunnerRecovery);
|
|
candidate = record.cause;
|
|
} catch {
|
|
return null;
|
|
}
|
|
}
|
|
return null;
|
|
}
|
|
|
|
export const REMOTE_RECOVERY_MARKER_READ_SCRIPT =
|
|
'test ! -L "$1" || exit 32; test -f "$1" || exit 31; cat -- "$1" || exit 33';
|
|
|
|
export const REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT = [
|
|
"set -eu",
|
|
"identity_path=$1; expected_nonce=$2; expected_runner_id=$3; expected_pid=$4",
|
|
'test ! -L "$identity_path" || exit 42',
|
|
'test -f "$identity_path" || exit 41',
|
|
'{ IFS= read -r nonce; IFS= read -r pid; IFS= read -r started_at; IFS= read -r runner_id; } < "$identity_path" || exit 43',
|
|
'test "$nonce" = "$expected_nonce" && test "$runner_id" = "$expected_runner_id" && test "$pid" = "$expected_pid" && test -n "$started_at" || exit 44',
|
|
String.raw`node -e '
|
|
const fs = require("node:fs");
|
|
const [pid, runnerId, expectedGeneration] = process.argv.slice(1);
|
|
if (!/^[1-9][0-9]*$/.test(pid) || !Number.isSafeInteger(Number(pid))) process.exit(44);
|
|
function requireLiveProcess() {
|
|
try { process.kill(Number(pid), 0); }
|
|
catch (error) { process.exit(error.code === "ESRCH" ? 45 : 46); }
|
|
}
|
|
function verifyGeneration() {
|
|
try {
|
|
const stat = fs.readFileSync("/proc/" + pid + "/stat", "utf8");
|
|
const end = stat.lastIndexOf(") ");
|
|
const fields = stat.slice(end + 2).trim().split(/\s+/);
|
|
const ticks = fields[19];
|
|
const boot = fs.readFileSync("/proc/sys/kernel/random/boot_id", "utf8").trim();
|
|
if (end < 0 || !stat.startsWith(pid + " (") || !/^[0-9]+$/.test(ticks || "") || !/^[-a-fA-F0-9]+$/.test(boot)) process.exit(46);
|
|
if ("linux:" + boot + ":" + ticks !== expectedGeneration) process.exit(47);
|
|
if (fields[0] === "Z") process.exit(45);
|
|
} catch (error) {
|
|
if (error.code === "ENOENT") requireLiveProcess();
|
|
process.exit(46);
|
|
}
|
|
}
|
|
function awaitExitingProcess() {
|
|
const pause = new Int32Array(new SharedArrayBuffer(4));
|
|
const until = Date.now() + 1000;
|
|
while (Date.now() < until) {
|
|
let state;
|
|
try {
|
|
const stat = fs.readFileSync("/proc/" + pid + "/stat", "utf8");
|
|
state = stat.slice(stat.lastIndexOf(") ") + 2).trim().split(/\s+/)[0];
|
|
} catch (error) {
|
|
if (error.code === "ENOENT") process.exit(45);
|
|
return;
|
|
}
|
|
if (state === "Z" || state === "X" || state === "x") process.exit(45);
|
|
try { process.kill(Number(pid), 0); }
|
|
catch (error) {
|
|
if (error.code === "ESRCH") process.exit(45);
|
|
return;
|
|
}
|
|
Atomics.wait(pause, 0, 0, 50);
|
|
}
|
|
}
|
|
requireLiveProcess();
|
|
const commandPath = "/proc/" + pid + "/cmdline";
|
|
if (process.platform !== "linux" && !expectedGeneration && !fs.existsSync(commandPath)) process.exit(0);
|
|
let args;
|
|
try { args = fs.readFileSync(commandPath, "utf8").split("\0"); }
|
|
catch (error) {
|
|
requireLiveProcess();
|
|
if (error.code === "ENOENT" && expectedGeneration) verifyGeneration();
|
|
process.exit(46);
|
|
}
|
|
if (expectedGeneration) verifyGeneration();
|
|
if (!args.includes("--runner-id") || !args.includes(runnerId)) {
|
|
requireLiveProcess();
|
|
if (expectedGeneration && args.join("") === "") awaitExitingProcess();
|
|
process.exit(48);
|
|
}
|
|
' "$pid" "$expected_runner_id"` + ' "${5:-}"',
|
|
].join("; ");
|
|
|
|
export function remoteRecoveryProbeReason(stage: RemoteRunnerRecoveryDiagnostic["stage"], exitCode: number | null, timedOut: boolean): RemoteRunnerRecoveryDiagnostic["reason"] {
|
|
if (timedOut) return "probe_timeout";
|
|
if (stage === "identity_marker") {
|
|
if (exitCode === 31) return "marker_missing";
|
|
if (exitCode === 32) return "marker_unsafe";
|
|
if (exitCode === 33) return "marker_read_failed";
|
|
} else if (stage === "process_verification") {
|
|
if (exitCode === 41) return "marker_missing";
|
|
if (exitCode === 42) return "marker_unsafe";
|
|
if (exitCode === 43) return "marker_read_failed";
|
|
if (exitCode === 44) return "process_identity_changed";
|
|
if (exitCode === 45) return "process_not_running";
|
|
if (exitCode === 46) return "fingerprint_unavailable";
|
|
if (exitCode === 47) return "process_generation_changed";
|
|
if (exitCode === 48) return "process_runner_id_missing";
|
|
}
|
|
return "probe_failed";
|
|
}
|