Files
PaperClipAI/server/src/shutdown.ts
T
DottaandClaude Opus 5.5 cf09024292 fix(heartbeat): count an early native hand-off only after its suspend commits (#21572)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A server restart hands each live Runner run to the next server, so
the agent keeps working without a retry
> - At shutdown start, the outgoing server hands off native runs early,
in parallel with the rest of shutdown
> - The early hand-off put a run in the "handed off" set before its
suspend finished, and the drain counted every run in that set as
suspended without waiting
> - The early hand-off shares the 20 s pre-drain budget, and one Runner
detach can take 20-30 s, so the process could exit with the suspend
unfinished
> - Then the finalization lease stays live, the next server gets no
hand-over signal, and the run waits for lease expiry or the reaper
> - This pull request makes the drain wait for each in-flight early
hand-off, and fall back to its own hand-off when the early one failed or
did not finish
> - The benefit is that a run counts as handed over only after its
hand-over is committed, so the next server can always reach it at once

## Linked Issues or Issue Description

**What happened?**
A passing zero-downtime swap test showed a latent race in the shutdown
hand-off. `handOffNativeRunsForShutdown` added a run to
`nativeRunsHandedOffEarly` before `suspendNativeRunForShutdown`
finished. `drainRunningRunsForShutdown` then reported every run in that
set as `restartSuspended` without awaiting it. Because the early
hand-off shares the 20 s pre-drain budget and a Runner detach can wait
20-30 s, the process can exit before the lease release commits. Also, a
timed-out `prepareHotRestartShutdown` could still write the hot-restart
snapshot and detach sessions after the fallback drain started. In the
same test, swap 1's early hand-off took 2.08 s, and no log showed which
phase was slow.

**Expected behavior**
A run counts as handed over only after its suspend commits. If the early
suspend fails or does not finish, the drain hands the run off itself. A
timed-out hot-restart preparation does no further work after the drain
starts. The hand-off logs its phase timings.

**Steps to reproduce**
1. Start a native Runner run on a remote sandbox.
2. Send SIGTERM while the Runner detach is slow (or fails).
3. The drain reports the run as restart-suspended while its suspend is
still pending, and the process can exit before the lease release.

**Paperclip version or commit**
`master` at bbed32450c.

**Deployment mode**
Hosted, zero-downtime release swap.

## What Changed

- `nativeRunsHandedOffEarly` now holds each run's in-flight suspend
promise, not only its id.
- The shutdown drain awaits that promise (bounded at 30 s). If it fails,
returns false or does not finish, the drain runs its own suspend, as it
does for a run without an early hand-off.
- `runBoundedShutdownStep` and the pre-drain budget pass an
`AbortSignal` to each step and abort it on timeout.
- `prepareHotRestartShutdown` takes that signal and stops before the
snapshot write, the run events, and the session detach when the step
timed out.
- `handOffNativeRunsForShutdown` logs one structured line, "native run
shutdown hand-off phase timings": intent read, controller identity,
select, max per-run detach, max per-run transaction, and total.

## Verification

- Red, then green (each failed against `master` source and passes with
the fix), all through the public shutdown functions in
`server/src/services/heartbeat/recovery.test.ts`:
- `does not count an early hand-off as suspended while its detach is
still pending at the shutdown drain`
- `hands off a run at the shutdown drain when its pending early hand-off
fails`
- `writes no hot-restart snapshot and detaches nothing after a timed-out
preparation hands over to the drain`
- New test in `server/src/shutdown.test.ts`: `aborts the signal of a
step that outlives its bound, and only that step`.
- `pnpm vitest run src/shutdown.test.ts
src/services/heartbeat/recovery.test.ts
src/__tests__/heartbeat-native-early-hand-off.test.ts` passes (145
tests).
- `tsc --noEmit` for `server` passes. `pnpm verify` passes for the
selected tests.

## Risks

- When an early hand-off does not finish within 30 s, the drain starts a
second suspend. The second detach joins the first, because the session
memoizes its restart detach. The run can then get two "suspended" run
events. That is preferred to reporting a hand-over that did not happen.
- The drain can now wait up to 30 s for a slow early hand-off before the
process exits. Before, it exited without the hand-over.
- The abort signal only stops work between steps of the preparation. A
snapshot write that already started still completes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:02:25 -06:00

377 lines
13 KiB
TypeScript

type HotRestartShutdownPreparation = {
skipDrain: boolean;
};
type ShutdownLogger = {
info(obj: object, msg: string): void;
error(obj: object, msg: string): void;
};
export const PRE_DRAIN_SHUTDOWN_BUDGET_MS = 20_000;
export type ShutdownStepResult<T> =
| { outcome: "completed"; value: T; durationMs: number }
| { outcome: "timed_out"; durationMs: number }
| { outcome: "failed"; error: unknown; durationMs: number };
export async function afterStalledShutdownStep<T>(result: ShutdownStepResult<T>, onStalled: () => Promise<void>) {
if (result.outcome !== "completed") await onStalled();
return result;
}
export async function runBoundedShutdownStep<T>(input: {
step: string;
signal: "SIGINT" | "SIGTERM";
timeoutMs: number;
run: (abort: AbortSignal) => Promise<T>;
log: ShutdownLogger;
now?: () => number;
}): Promise<ShutdownStepResult<T>> {
const now = input.now ?? Date.now;
const startedAt = now();
const timeoutMs = Math.max(0, input.timeoutMs);
const abort = new AbortController();
let timer: NodeJS.Timeout | null = null;
try {
const result = await Promise.race([
Promise.resolve()
.then(() => input.run(abort.signal))
.then(
(value) => ({ outcome: "completed" as const, value }),
(error: unknown) => ({ outcome: "failed" as const, error }),
),
new Promise<{ outcome: "timed_out" }>((resolve) => {
timer = setTimeout(() => resolve({ outcome: "timed_out" }), timeoutMs);
timer.unref?.();
}),
]);
const durationMs = now() - startedAt;
const fields = { signal: input.signal, step: input.step, durationMs, timeoutMs };
if (result.outcome === "completed") {
input.log.info(fields, "shutdown step completed");
return { outcome: "completed", value: result.value, durationMs };
}
if (result.outcome === "failed") {
input.log.error({ ...fields, err: result.error }, "shutdown step failed; continuing shutdown");
return { outcome: "failed", error: result.error, durationMs };
}
abort.abort(new Error(`shutdown step ${input.step} timed out`));
input.log.info(fields, "shutdown step timed out; continuing shutdown without it");
return { outcome: "timed_out", durationMs };
} finally {
if (timer) clearTimeout(timer);
}
}
export function createPreDrainShutdownBudget(input: {
signal: "SIGINT" | "SIGTERM";
budgetMs?: number;
log: ShutdownLogger;
now?: () => number;
}) {
const now = input.now ?? Date.now;
const deadline = now() + (input.budgetMs ?? PRE_DRAIN_SHUTDOWN_BUDGET_MS);
return {
run<T>(step: string, run: (abort: AbortSignal) => Promise<T>) {
return runBoundedShutdownStep({
step,
signal: input.signal,
timeoutMs: deadline - now(),
run,
log: input.log,
now,
});
},
};
}
export async function drainRunExecutionFinalizersForShutdown(input: {
signal: "SIGINT" | "SIGTERM";
drain: (() => Promise<void>) | null;
timeoutMs?: number;
log: ShutdownLogger;
}): Promise<"drained" | "timed_out" | "unavailable"> {
if (!input.drain) return "unavailable";
const timeoutMs = input.timeoutMs ?? 5_000;
let timer: NodeJS.Timeout | null = null;
try {
const result = await Promise.race([
input.drain().then(() => "drained" as const),
new Promise<"timed_out">((resolve) => {
timer = setTimeout(() => resolve("timed_out"), timeoutMs);
timer.unref?.();
}),
]);
if (result === "timed_out") {
input.log.info(
{ signal: input.signal, timeoutMs },
"bounded heartbeat execution finalizer drain timed out",
);
}
return result;
} finally {
if (timer) clearTimeout(timer);
}
}
type ShutdownHttpListener = {
listening: boolean;
close(callback?: (err?: Error) => void): unknown;
closeIdleConnections?: () => void;
closeAllConnections?: () => void;
};
/**
* Stops the HTTP listener from accepting new requests and waits, for at most
* `timeoutMs`, for the open connections to finish. Idle keep-alive sockets
* close at once; whatever is still open when the grace period ends is closed
* forcibly, so the teardown never hangs on a long-lived client. Call this
* before the database pool ends, so no request can reach a route after
* `sql.end()` and fail with a connection-ended error.
*/
export async function closeHttpListenerForShutdown(input: {
server: ShutdownHttpListener;
signal: "SIGINT" | "SIGTERM";
timeoutMs?: number;
drainRequests?: (signal: AbortSignal) => Promise<void>;
closeRunAdmission?: () => void;
log: ShutdownLogger;
}): Promise<"closed" | "timed_out" | "not_listening"> {
input.closeRunAdmission?.();
if (!input.server.listening) return "not_listening";
const timeoutMs = input.timeoutMs ?? 5_000;
let timer: NodeJS.Timeout | null = null;
const controller = new AbortController();
try {
const closed = new Promise<"closed">((resolve) => {
input.server.close((err) => {
if (err && (err as NodeJS.ErrnoException).code !== "ERR_SERVER_NOT_RUNNING") {
input.log.error({ err, signal: input.signal }, "HTTP listener close failed");
}
resolve("closed");
});
input.server.closeIdleConnections?.();
});
return await Promise.race([
input.drainRequests ? input.drainRequests(controller.signal).then(() => {
input.server.closeIdleConnections?.();
return "closed" as const;
}) : closed,
new Promise<"timed_out">((resolve) => {
timer = setTimeout(() => {
input.log.info(
{ signal: input.signal, timeoutMs },
"HTTP listener drain timed out; closing the remaining connections",
);
input.server.closeAllConnections?.();
resolve("timed_out");
}, timeoutMs);
timer.unref?.();
}),
]);
} finally {
if (timer) clearTimeout(timer);
controller.abort();
}
}
/**
* Runs the final, ordered teardown of the server. It awaits the application
* service cleanup first, so a live setup-token login session stops and releases
* its sandbox lease before the database and the provider stop. The caller runs
* `process.exit(0)` only after this helper resolves, so an orderly shutdown
* never leaves a sandbox lease or confidential login state alive past the
* process exit.
*
* A step that rejects does not stop the teardown. The helper logs the error and
* continues to the next step. A failed setup-token lease release stays a
* durable record for the startup reaper; the helper surfaces it in the log
* instead of blocking the exit path.
*/
export async function finalizeServerShutdown(input: {
signal: "SIGINT" | "SIGTERM";
shutdownAppServices: (() => Promise<void>) | undefined;
/**
* Stops the HTTP listener and drains its connections (see
* `closeHttpListenerForShutdown`). Runs first, while every application
* service is still available to the requests being drained, so no request
* runs against a half-dismantled service or an ended pool.
*/
closeHttpListener?: (() => Promise<unknown>) | null;
/**
* Waits for every run-failure Sentry report still in flight. Runs after the
* application services and before the database pool ends, so a report that
* started just before shutdown still gets its database read and reaches
* Sentry before `shutdownSentry` flushes and closes the client.
*/
drainPendingRunFailureReports?: (() => Promise<void>) | null;
/**
* Ends the server's PostgreSQL client pools. Runs after the application
* services (which still need the database) and before the embedded
* provider stops, so the backends close in order and none outlive the
* process.
*/
closeDatabase?: (() => Promise<void>) | null;
stopEmbeddedPostgres: (() => Promise<void>) | null;
shutdownInstrumentation: () => Promise<void>;
shutdownSentry: () => Promise<void>;
log: ShutdownLogger;
}): Promise<void> {
const { signal } = input;
// Stop accepting requests and drain the open ones before any service goes
// away, so a request that is still in flight sees a fully working server.
if (input.closeHttpListener) {
try {
await input.closeHttpListener();
} catch (err) {
input.log.error({ err, signal }, "HTTP listener shutdown failed");
}
}
// Await the application service cleanup, so a live setup-token login session
// releases its sandbox lease before the database and the provider stop. A
// rejected cleanup stays durable for the reaper; it does not block the exit.
try {
await input.shutdownAppServices?.();
} catch (err) {
input.log.error({ err, signal }, "Application service shutdown failed");
}
// Wait for every in-flight run-failure Sentry report before the database
// pool ends. `reportRunFailure` is fire-and-forget: without this wait, a
// report that started just before shutdown can lose its database read to
// the pool end below, or lose its Sentry call to the flush further down.
if (input.drainPendingRunFailureReports) {
try {
await input.drainPendingRunFailureReports();
} catch (err) {
input.log.error({ err, signal }, "run-failure report drain failed");
}
}
// End the client pools once nothing needs them any more. Without this the
// process exit leaves the pooled backends to PostgreSQL's own TCP keepalive
// reaping, and a restart loop can pile up enough of them to hit
// `max_connections` before the next boot gets a connection.
if (input.closeDatabase) {
try {
await input.closeDatabase();
} catch (err) {
input.log.error({ err, signal }, "Database client shutdown failed");
}
}
if (input.stopEmbeddedPostgres) {
input.log.info({ signal }, "Stopping embedded PostgreSQL");
try {
await input.stopEmbeddedPostgres();
} catch (err) {
input.log.error({ err }, "Failed to stop embedded PostgreSQL cleanly");
}
}
// Flush buffered OTel spans before the process goes away; without this await
// the exporter's final batch is dropped on exit.
await input.shutdownInstrumentation();
// Flush buffered Sentry events before the process goes away; without this
// await the last events are dropped on exit.
await input.shutdownSentry();
}
const COORDINATED_SHUTDOWN_SIGNALS = ["SIGINT", "SIGTERM"] as const;
type ShutdownSignalTarget = {
rawListeners(eventName: string): Function[];
removeListener(eventName: string, listener: (...args: any[]) => void): unknown;
};
/**
* Some dependencies eagerly install process signal handlers as an import side
* effect. Paperclip must remain the sole owner of SIGINT/SIGTERM ordering: its
* handler first snapshots live heartbeat runs and only then stops embedded
* infrastructure. Remove only listeners added by the supplied import, while
* preserving every listener that was already registered.
*/
export async function loadWithoutCoordinatedShutdownSignalHooks<T>(
load: () => Promise<T>,
signalTarget: ShutdownSignalTarget = process,
) {
const listenersBeforeLoad = new Map(
COORDINATED_SHUTDOWN_SIGNALS.map((signal) => [
signal,
signalTarget.rawListeners(signal),
]),
);
let loaded: T;
try {
loaded = await load();
} finally {
for (const signal of COORDINATED_SHUTDOWN_SIGNALS) {
const remainingBeforeLoad = [...(listenersBeforeLoad.get(signal) ?? [])];
for (const listener of signalTarget.rawListeners(signal)) {
const existingIndex = remainingBeforeLoad.indexOf(listener);
if (existingIndex >= 0) {
remainingBeforeLoad.splice(existingIndex, 1);
continue;
}
signalTarget.removeListener(signal, listener as (...args: any[]) => void);
}
}
}
return loaded;
}
export async function coordinateHeartbeatSchedulerShutdown<
TPreparation extends HotRestartShutdownPreparation,
>(input: {
signal: "SIGINT" | "SIGTERM";
prepareHotRestartShutdown: ((signal: "SIGINT" | "SIGTERM") => Promise<TPreparation>) | null;
waitForHeartbeatSchedulerIdle: () => Promise<void>;
}): Promise<{
hotRestart: TPreparation | null;
preparationError: unknown;
waitedForSchedulerIdle: boolean;
}> {
let hotRestart: TPreparation | null = null;
let preparationError: unknown = null;
// The signal handler stops the scheduler before entering this coordinator.
// Quiesce any callback that was already in flight before querying running
// rows for the shutdown snapshot, otherwise a late queue claim can create a
// run that is absent from both the snapshot and the selective drain set.
await input.waitForHeartbeatSchedulerIdle();
if (input.prepareHotRestartShutdown) {
try {
hotRestart = await input.prepareHotRestartShutdown(input.signal);
} catch (err) {
preparationError = err;
}
}
return {
hotRestart,
preparationError,
waitedForSchedulerIdle: true,
};
}
export async function finishServiceShutdown(
stop: () => Promise<void>,
cleanup: () => Promise<void>,
): Promise<void> {
const failures: unknown[] = [];
let retryStop = false;
try { await stop(); } catch { retryStop = true; }
try { await cleanup(); } catch (error) { failures.push(error); }
if (retryStop) {
try { await stop(); } catch (error) { failures.unshift(error); }
}
if (failures.length === 1) throw failures[0];
if (failures.length > 1) throw new AggregateError(failures, "Service shutdown failed");
}