mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native Runner path runs an agent turn through a runnerd process that the control plane owns > - A restart of the control plane must keep the task going without a click, if that is safe > - A graceful shutdown closes the HTTP listener first and then drains the heartbeat runs > - A native run that was between provider dispatch and runnerd backend setup in that gap failed with `local_runtime_api_listener_unavailable`, so the drain did not see it as running and did not suspend it for the next boot > - This pull request makes that failure a restart detach: the run stays running with its claim, the drain suspends it, and the next controller adopts it > - The benefit is that a SIGTERM in this window no longer spends a failure and a same-run retry, and the task continues by itself ## Linked Issues or Issue Description Refs #19530 (the same restart window, before provider dispatch). **What happened?** In a restart chaos replay (ACPX Claude, local runner), a restart-recovered run got SIGTERM while `createRunnerdBackendWithinSessionClaim` built the runnerd backend. `resolveLocalRuntimeApiUrl()` threw `local_runtime_api_listener_unavailable` because shutdown had already closed the listener. The run settled `failed`. The drain then logged `restartSuspendedRunIds: []`. The next boot used the one bootstrap retry. A second SIGTERM hit the same window, and the task stopped at a manual Retry. **Expected behavior** A run that this process cannot continue because it is shutting down is handed to the next controller. It is not settled as failed. **Steps to reproduce** 1. Start a native Runner task and restart the server (K1) so that startup recovery resumes the run. 2. Send SIGTERM while the resumed run sets up its runnerd backend. 3. See the run fail with `local_runtime_api_listener_unavailable` and the drain report no suspended runs. ## What Changed - `server/src/runtime-api.ts`: `resolveLocalRuntimeApiUrl()` throws the new `LocalRuntimeApiListenerClosedError` when this process had a listener and closed it (the `listening` flag is false or the `close` event fired). The error message stays `local_runtime_api_listener_unavailable`, so the existing classifiers do not change. A listener that never started still throws the plain error. - `server/src/services/native-runtime/native-session-executor.ts`: for a local Runner, the startup failure handler treats `LocalRuntimeApiListenerClosedError` like a restart detach. It stops lease renewal, keeps the claim and throws `NativeControllerDetachedForRestartError`. Heartbeat already keeps such a run `running`. A remote Runner keeps the same-run bootstrap retry from #19462, because startup recovery has no process evidence to adopt a claimed remote startup (`native-runner-restart-preparation.integration.test.ts` still covers that path). - `doc/execution-semantics.md`: describe the new rule. - Tests: a heartbeat regression that closes the real runtime API listener before backend setup, then drains and restarts; a unit test for the closed versus never-started listener; two `vi.mock` factories keep the real module exports. ## Verification - Red before the fix: `pnpm exec vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts -t listener` failed with `restartSuspendedRunIds: []` and the run `failed` (`local_runtime_api_listener_unavailable`, `bootstrap_retry`). - Green after the fix: the same test passes. The drain suspends the run, the run stays `running` with `awaiting_runner_reattach`, and the next controller claims it and reaches the backend once. - `pnpm exec vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts src/http/server.test.ts src/__tests__/server-startup-feedback-export.test.ts src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-recovered-owner-followup.test.ts`: 1333 tests pass. One timing test in `native-session-executor.test.ts` failed once under host load average ~30 and passed alone. - `pnpm exec tsc --noEmit -p server/tsconfig.json`: pass. - Live, restart chaos harness, local ACPX Claude (`claude-sonnet-5`), `--rc06-masked`, this branch ata35ec8b845(before the remote-scope commit5dc63f24fb, which does not change the local path). The original replay schedule no longer reached the window, because the task now ends before the second fault. So I moved the faults earlier: K1 at 10 s, then K2 as soon as the recovered server is ready. - K1 + K2, 2 runs: both runs PASS with **R1 pass**. Each drain logged `restartSuspendedRunIds: [<run>]` (before: `[]`). The next boot classified the run `resume_dead_runner` and the same run completed. No `local_runtime_api_listener_unavailable` failure. - K1 + K2 + K2, 2 runs: each drain suspended the run. The fourth boot blocked the run with `native_restart_recovery_blocked`, because three restart claims used the three-attempt recovery budget. Retry then completed the task (R3 pass). That budget count is a separate open item. This pull request does not change it. ## Risks - Low. The new path applies only after this process closed its own listener, which happens only during shutdown. The run then waits for the drain or the next boot, as for the other restart detaches. - If shutdown does not reach the drain (for example SIGKILL after the listener closes), the run keeps its live claim until the claim expires. Then startup recovery handles it as a hard restart. This is the same as any run that is running at a hard kill. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, GitHub CLI). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
248 lines
8.3 KiB
TypeScript
248 lines
8.3 KiB
TypeScript
import os from "node:os";
|
|
import type { Server } from "node:http";
|
|
import { isIP } from "node:net";
|
|
|
|
const runtimeApiListeners = new Set<Server>();
|
|
const registeredRuntimeApiListeners = new WeakSet<Server>();
|
|
let runtimeApiListenerClosed = false;
|
|
|
|
/**
|
|
* The process had a runtime API listener and has since closed it, which only
|
|
* happens while it shuts down. Work that needs a callback origin must hand off
|
|
* to the next controller instead of failing as a provider or startup error.
|
|
*/
|
|
export class LocalRuntimeApiListenerClosedError extends Error {
|
|
constructor() {
|
|
// Keep the established failure code for classifiers and persisted rows.
|
|
super("local_runtime_api_listener_unavailable");
|
|
this.name = "LocalRuntimeApiListenerClosedError";
|
|
}
|
|
}
|
|
|
|
/** Track listener ownership without treating configured or advertised URLs as authority. */
|
|
export function registerLocalRuntimeApiServer(server: Server): void {
|
|
if (!registeredRuntimeApiListeners.has(server)) {
|
|
registeredRuntimeApiListeners.add(server);
|
|
server.on("listening", () => {
|
|
runtimeApiListeners.add(server);
|
|
runtimeApiListenerClosed = false;
|
|
});
|
|
server.on("close", () => {
|
|
if (runtimeApiListeners.delete(server)) runtimeApiListenerClosed = true;
|
|
});
|
|
}
|
|
if (server.listening) {
|
|
runtimeApiListeners.add(server);
|
|
runtimeApiListenerClosed = false;
|
|
}
|
|
}
|
|
|
|
function normalizeHost(value: string | null | undefined): string {
|
|
return (value ?? "").trim();
|
|
}
|
|
|
|
function isLoopbackHost(host: string): boolean {
|
|
const normalized = normalizeHost(host).toLowerCase();
|
|
return normalized === "127.0.0.1" || normalized === "localhost" || normalized === "::1";
|
|
}
|
|
|
|
function isWildcardHost(host: string): boolean {
|
|
const normalized = normalizeHost(host).toLowerCase();
|
|
return normalized === "0.0.0.0" || normalized === "::";
|
|
}
|
|
|
|
function isLinkLocalHost(host: string): boolean {
|
|
const normalized = normalizeHost(host).toLowerCase();
|
|
if (normalized.startsWith("169.254.")) return true;
|
|
// IPv6 link-local block is fe80::/10 (fe80:: through febf::)
|
|
if (/^fe[89ab][0-9a-f]:/.test(normalized)) return true;
|
|
return false;
|
|
}
|
|
|
|
function formatOrigin(protocol: string, host: string, port: number): string {
|
|
const normalizedHost = host.includes(":") && !host.startsWith("[") && !host.endsWith("]")
|
|
? `[${host}]`
|
|
: host;
|
|
return `${protocol}//${normalizedHost}:${port}`;
|
|
}
|
|
|
|
function pushCandidate(
|
|
candidates: string[],
|
|
seen: Set<string>,
|
|
rawUrl: string | null | undefined,
|
|
): void {
|
|
const trimmed = rawUrl?.trim();
|
|
if (!trimmed) return;
|
|
try {
|
|
const normalized = new URL(trimmed).origin;
|
|
if (seen.has(normalized)) return;
|
|
seen.add(normalized);
|
|
candidates.push(normalized);
|
|
} catch {
|
|
// Ignore malformed candidates.
|
|
}
|
|
}
|
|
|
|
// A base URL written as `host:port` (no `//`) is a VALID URL with an opaque
|
|
// scheme, and the origin of an opaque-scheme URL serializes to the *string*
|
|
// "null". That string is truthy, so it escapes every `if (origin)` guard and is
|
|
// then handed to `new URL(origin)`, which throws `TypeError: Invalid URL`.
|
|
// Treat it as absent config, exactly like input that fails to parse at all.
|
|
function parsePublicOrigin(rawUrl: string | null | undefined): string | null {
|
|
const trimmed = rawUrl?.trim();
|
|
if (!trimmed) return null;
|
|
let origin: string;
|
|
try {
|
|
origin = new URL(trimmed).origin;
|
|
} catch {
|
|
return null;
|
|
}
|
|
return origin === "null" ? null : origin;
|
|
}
|
|
|
|
export function chooseLocalRuntimeApiUrl(input: {
|
|
bindHost: string;
|
|
port: number;
|
|
}): string {
|
|
const bindHost = normalizeHost(input.bindHost).replace(/^\[([^\]]+)\]$/, "$1");
|
|
if (isIP(bindHost) === 0 || !Number.isInteger(input.port) || input.port < 1 || input.port > 65535) {
|
|
throw new Error("local_runtime_api_listener_address_invalid");
|
|
}
|
|
const connectHost = bindHost === "::" ? "::1"
|
|
: bindHost === "0.0.0.0" ? "127.0.0.1"
|
|
: bindHost;
|
|
return formatOrigin("http:", connectHost, input.port);
|
|
}
|
|
|
|
export function resolveLocalRuntimeApiUrl(): string {
|
|
const listeners = [...runtimeApiListeners].filter(server => server.listening);
|
|
// `server.close()` clears `listening` at once; its close event waits for
|
|
// admitted requests. Either state means this process stopped serving.
|
|
if (listeners.length === 0 && (runtimeApiListenerClosed || runtimeApiListeners.size > 0)) {
|
|
throw new LocalRuntimeApiListenerClosedError();
|
|
}
|
|
if (listeners.length !== 1) {
|
|
throw new Error(listeners.length ? "local_runtime_api_listener_ambiguous" : "local_runtime_api_listener_unavailable");
|
|
}
|
|
const address = listeners[0]!.address();
|
|
if (!address || typeof address === "string") throw new Error("local_runtime_api_listener_unavailable");
|
|
return chooseLocalRuntimeApiUrl({ bindHost: address.address, port: address.port });
|
|
}
|
|
|
|
export const runtimeApiInternals = {
|
|
resetForTests(): void {
|
|
runtimeApiListeners.clear();
|
|
runtimeApiListenerClosed = false;
|
|
},
|
|
};
|
|
|
|
export function choosePrimaryRuntimeApiUrl(input: {
|
|
authPublicBaseUrl?: string | null;
|
|
allowedHostnames: string[];
|
|
bindHost: string;
|
|
port: number;
|
|
}): string {
|
|
const explicitOrigin = parsePublicOrigin(input.authPublicBaseUrl);
|
|
if (explicitOrigin) return explicitOrigin;
|
|
|
|
const bindHost = normalizeHost(input.bindHost);
|
|
if (bindHost && !isWildcardHost(bindHost) && isLoopbackHost(bindHost)) {
|
|
return formatOrigin("http:", bindHost, input.port);
|
|
}
|
|
|
|
const allowedHostname = input.allowedHostnames
|
|
.map((value) => value.trim())
|
|
.find(Boolean);
|
|
if (allowedHostname) {
|
|
return formatOrigin("http:", allowedHostname, input.port);
|
|
}
|
|
|
|
if (bindHost && !isWildcardHost(bindHost)) {
|
|
return formatOrigin("http:", bindHost, input.port);
|
|
}
|
|
|
|
return formatOrigin("http:", "localhost", input.port);
|
|
}
|
|
|
|
export function collectReachableInterfaceHosts(input: {
|
|
networkInterfacesMap?: NodeJS.Dict<os.NetworkInterfaceInfo[]>;
|
|
} = {}): string[] {
|
|
const interfaces = input.networkInterfacesMap ?? os.networkInterfaces();
|
|
const rankedHosts: Array<{ host: string; rank: number; index: number }> = [];
|
|
const seen = new Set<string>();
|
|
let index = 0;
|
|
|
|
for (const entries of Object.values(interfaces)) {
|
|
for (const entry of entries ?? []) {
|
|
if (entry.internal) continue;
|
|
const host = normalizeHost(entry.address);
|
|
if (!host || isLoopbackHost(host) || isWildcardHost(host) || isLinkLocalHost(host)) continue;
|
|
if (seen.has(host)) continue;
|
|
seen.add(host);
|
|
rankedHosts.push({
|
|
host,
|
|
rank: entry.family === "IPv4" ? 0 : 1,
|
|
index: index++,
|
|
});
|
|
}
|
|
}
|
|
|
|
return rankedHosts
|
|
.sort((left, right) => left.rank - right.rank || left.index - right.index)
|
|
.map((entry) => entry.host);
|
|
}
|
|
|
|
export function buildRuntimeApiCandidateUrls(input: {
|
|
preferredApiUrl?: string | null;
|
|
authPublicBaseUrl?: string | null;
|
|
allowedHostnames: string[];
|
|
bindHost: string;
|
|
port: number;
|
|
networkInterfacesMap?: NodeJS.Dict<os.NetworkInterfaceInfo[]>;
|
|
}): string[] {
|
|
const candidates: string[] = [];
|
|
const seen = new Set<string>();
|
|
const explicitOrigin = parsePublicOrigin(input.authPublicBaseUrl);
|
|
const protocol = explicitOrigin ? new URL(explicitOrigin).protocol : "http:";
|
|
|
|
pushCandidate(candidates, seen, input.preferredApiUrl);
|
|
pushCandidate(candidates, seen, explicitOrigin);
|
|
|
|
for (const rawHost of input.allowedHostnames) {
|
|
const host = normalizeHost(rawHost);
|
|
if (!host) continue;
|
|
pushCandidate(candidates, seen, formatOrigin(protocol, host, input.port));
|
|
}
|
|
|
|
const bindHost = normalizeHost(input.bindHost);
|
|
if (bindHost && !isWildcardHost(bindHost)) {
|
|
pushCandidate(candidates, seen, formatOrigin(protocol, bindHost, input.port));
|
|
}
|
|
|
|
if (explicitOrigin) {
|
|
const hostname = new URL(explicitOrigin).hostname;
|
|
if (isLoopbackHost(hostname)) {
|
|
pushCandidate(candidates, seen, formatOrigin(protocol, "host.docker.internal", input.port));
|
|
}
|
|
}
|
|
|
|
for (const host of collectReachableInterfaceHosts({ networkInterfacesMap: input.networkInterfacesMap })) {
|
|
pushCandidate(candidates, seen, formatOrigin(protocol, host, input.port));
|
|
}
|
|
|
|
if (candidates.length === 0) {
|
|
pushCandidate(
|
|
candidates,
|
|
seen,
|
|
choosePrimaryRuntimeApiUrl({
|
|
authPublicBaseUrl: input.authPublicBaseUrl,
|
|
allowedHostnames: input.allowedHostnames,
|
|
bindHost: input.bindHost,
|
|
port: input.port,
|
|
}),
|
|
);
|
|
}
|
|
|
|
return candidates;
|
|
}
|