mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A server restart hands each live Runner run to the next server, so
the agent keeps working without a retry
> - At shutdown start, the outgoing server hands off native runs early,
in parallel with the rest of shutdown
> - The early hand-off put a run in the "handed off" set before its
suspend finished, and the drain counted every run in that set as
suspended without waiting
> - The early hand-off shares the 20 s pre-drain budget, and one Runner
detach can take 20-30 s, so the process could exit with the suspend
unfinished
> - Then the finalization lease stays live, the next server gets no
hand-over signal, and the run waits for lease expiry or the reaper
> - This pull request makes the drain wait for each in-flight early
hand-off, and fall back to its own hand-off when the early one failed or
did not finish
> - The benefit is that a run counts as handed over only after its
hand-over is committed, so the next server can always reach it at once
## Linked Issues or Issue Description
**What happened?**
A passing zero-downtime swap test showed a latent race in the shutdown
hand-off. `handOffNativeRunsForShutdown` added a run to
`nativeRunsHandedOffEarly` before `suspendNativeRunForShutdown`
finished. `drainRunningRunsForShutdown` then reported every run in that
set as `restartSuspended` without awaiting it. Because the early
hand-off shares the 20 s pre-drain budget and a Runner detach can wait
20-30 s, the process can exit before the lease release commits. Also, a
timed-out `prepareHotRestartShutdown` could still write the hot-restart
snapshot and detach sessions after the fallback drain started. In the
same test, swap 1's early hand-off took 2.08 s, and no log showed which
phase was slow.
**Expected behavior**
A run counts as handed over only after its suspend commits. If the early
suspend fails or does not finish, the drain hands the run off itself. A
timed-out hot-restart preparation does no further work after the drain
starts. The hand-off logs its phase timings.
**Steps to reproduce**
1. Start a native Runner run on a remote sandbox.
2. Send SIGTERM while the Runner detach is slow (or fails).
3. The drain reports the run as restart-suspended while its suspend is
still pending, and the process can exit before the lease release.
**Paperclip version or commit**
`master` at bbed32450c.
**Deployment mode**
Hosted, zero-downtime release swap.
## What Changed
- `nativeRunsHandedOffEarly` now holds each run's in-flight suspend
promise, not only its id.
- The shutdown drain awaits that promise (bounded at 30 s). If it fails,
returns false or does not finish, the drain runs its own suspend, as it
does for a run without an early hand-off.
- `runBoundedShutdownStep` and the pre-drain budget pass an
`AbortSignal` to each step and abort it on timeout.
- `prepareHotRestartShutdown` takes that signal and stops before the
snapshot write, the run events, and the session detach when the step
timed out.
- `handOffNativeRunsForShutdown` logs one structured line, "native run
shutdown hand-off phase timings": intent read, controller identity,
select, max per-run detach, max per-run transaction, and total.
## Verification
- Red, then green (each failed against `master` source and passes with
the fix), all through the public shutdown functions in
`server/src/services/heartbeat/recovery.test.ts`:
- `does not count an early hand-off as suspended while its detach is
still pending at the shutdown drain`
- `hands off a run at the shutdown drain when its pending early hand-off
fails`
- `writes no hot-restart snapshot and detaches nothing after a timed-out
preparation hands over to the drain`
- New test in `server/src/shutdown.test.ts`: `aborts the signal of a
step that outlives its bound, and only that step`.
- `pnpm vitest run src/shutdown.test.ts
src/services/heartbeat/recovery.test.ts
src/__tests__/heartbeat-native-early-hand-off.test.ts` passes (145
tests).
- `tsc --noEmit` for `server` passes. `pnpm verify` passes for the
selected tests.
## Risks
- When an early hand-off does not finish within 30 s, the drain starts a
second suspend. The second detach joins the first, because the session
memoizes its restart detach. The run can then get two "suspended" run
events. That is preferred to reporting a hand-over that did not happen.
- The drain can now wait up to 30 s for a slow early hand-off before the
process exits. Before, it exited without the hand-over.
- The abort signal only stops work between steps of the preparation. A
snapshot write that already started still completes.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
377 lines
13 KiB
TypeScript
377 lines
13 KiB
TypeScript
type HotRestartShutdownPreparation = {
|
|
skipDrain: boolean;
|
|
};
|
|
|
|
type ShutdownLogger = {
|
|
info(obj: object, msg: string): void;
|
|
error(obj: object, msg: string): void;
|
|
};
|
|
|
|
export const PRE_DRAIN_SHUTDOWN_BUDGET_MS = 20_000;
|
|
|
|
export type ShutdownStepResult<T> =
|
|
| { outcome: "completed"; value: T; durationMs: number }
|
|
| { outcome: "timed_out"; durationMs: number }
|
|
| { outcome: "failed"; error: unknown; durationMs: number };
|
|
|
|
export async function afterStalledShutdownStep<T>(result: ShutdownStepResult<T>, onStalled: () => Promise<void>) {
|
|
if (result.outcome !== "completed") await onStalled();
|
|
return result;
|
|
}
|
|
|
|
export async function runBoundedShutdownStep<T>(input: {
|
|
step: string;
|
|
signal: "SIGINT" | "SIGTERM";
|
|
timeoutMs: number;
|
|
run: (abort: AbortSignal) => Promise<T>;
|
|
log: ShutdownLogger;
|
|
now?: () => number;
|
|
}): Promise<ShutdownStepResult<T>> {
|
|
const now = input.now ?? Date.now;
|
|
const startedAt = now();
|
|
const timeoutMs = Math.max(0, input.timeoutMs);
|
|
const abort = new AbortController();
|
|
let timer: NodeJS.Timeout | null = null;
|
|
try {
|
|
const result = await Promise.race([
|
|
Promise.resolve()
|
|
.then(() => input.run(abort.signal))
|
|
.then(
|
|
(value) => ({ outcome: "completed" as const, value }),
|
|
(error: unknown) => ({ outcome: "failed" as const, error }),
|
|
),
|
|
new Promise<{ outcome: "timed_out" }>((resolve) => {
|
|
timer = setTimeout(() => resolve({ outcome: "timed_out" }), timeoutMs);
|
|
timer.unref?.();
|
|
}),
|
|
]);
|
|
const durationMs = now() - startedAt;
|
|
const fields = { signal: input.signal, step: input.step, durationMs, timeoutMs };
|
|
if (result.outcome === "completed") {
|
|
input.log.info(fields, "shutdown step completed");
|
|
return { outcome: "completed", value: result.value, durationMs };
|
|
}
|
|
if (result.outcome === "failed") {
|
|
input.log.error({ ...fields, err: result.error }, "shutdown step failed; continuing shutdown");
|
|
return { outcome: "failed", error: result.error, durationMs };
|
|
}
|
|
abort.abort(new Error(`shutdown step ${input.step} timed out`));
|
|
input.log.info(fields, "shutdown step timed out; continuing shutdown without it");
|
|
return { outcome: "timed_out", durationMs };
|
|
} finally {
|
|
if (timer) clearTimeout(timer);
|
|
}
|
|
}
|
|
|
|
export function createPreDrainShutdownBudget(input: {
|
|
signal: "SIGINT" | "SIGTERM";
|
|
budgetMs?: number;
|
|
log: ShutdownLogger;
|
|
now?: () => number;
|
|
}) {
|
|
const now = input.now ?? Date.now;
|
|
const deadline = now() + (input.budgetMs ?? PRE_DRAIN_SHUTDOWN_BUDGET_MS);
|
|
return {
|
|
run<T>(step: string, run: (abort: AbortSignal) => Promise<T>) {
|
|
return runBoundedShutdownStep({
|
|
step,
|
|
signal: input.signal,
|
|
timeoutMs: deadline - now(),
|
|
run,
|
|
log: input.log,
|
|
now,
|
|
});
|
|
},
|
|
};
|
|
}
|
|
|
|
export async function drainRunExecutionFinalizersForShutdown(input: {
|
|
signal: "SIGINT" | "SIGTERM";
|
|
drain: (() => Promise<void>) | null;
|
|
timeoutMs?: number;
|
|
log: ShutdownLogger;
|
|
}): Promise<"drained" | "timed_out" | "unavailable"> {
|
|
if (!input.drain) return "unavailable";
|
|
const timeoutMs = input.timeoutMs ?? 5_000;
|
|
let timer: NodeJS.Timeout | null = null;
|
|
try {
|
|
const result = await Promise.race([
|
|
input.drain().then(() => "drained" as const),
|
|
new Promise<"timed_out">((resolve) => {
|
|
timer = setTimeout(() => resolve("timed_out"), timeoutMs);
|
|
timer.unref?.();
|
|
}),
|
|
]);
|
|
if (result === "timed_out") {
|
|
input.log.info(
|
|
{ signal: input.signal, timeoutMs },
|
|
"bounded heartbeat execution finalizer drain timed out",
|
|
);
|
|
}
|
|
return result;
|
|
} finally {
|
|
if (timer) clearTimeout(timer);
|
|
}
|
|
}
|
|
|
|
type ShutdownHttpListener = {
|
|
listening: boolean;
|
|
close(callback?: (err?: Error) => void): unknown;
|
|
closeIdleConnections?: () => void;
|
|
closeAllConnections?: () => void;
|
|
};
|
|
|
|
/**
|
|
* Stops the HTTP listener from accepting new requests and waits, for at most
|
|
* `timeoutMs`, for the open connections to finish. Idle keep-alive sockets
|
|
* close at once; whatever is still open when the grace period ends is closed
|
|
* forcibly, so the teardown never hangs on a long-lived client. Call this
|
|
* before the database pool ends, so no request can reach a route after
|
|
* `sql.end()` and fail with a connection-ended error.
|
|
*/
|
|
export async function closeHttpListenerForShutdown(input: {
|
|
server: ShutdownHttpListener;
|
|
signal: "SIGINT" | "SIGTERM";
|
|
timeoutMs?: number;
|
|
drainRequests?: (signal: AbortSignal) => Promise<void>;
|
|
closeRunAdmission?: () => void;
|
|
log: ShutdownLogger;
|
|
}): Promise<"closed" | "timed_out" | "not_listening"> {
|
|
input.closeRunAdmission?.();
|
|
if (!input.server.listening) return "not_listening";
|
|
const timeoutMs = input.timeoutMs ?? 5_000;
|
|
let timer: NodeJS.Timeout | null = null;
|
|
const controller = new AbortController();
|
|
try {
|
|
const closed = new Promise<"closed">((resolve) => {
|
|
input.server.close((err) => {
|
|
if (err && (err as NodeJS.ErrnoException).code !== "ERR_SERVER_NOT_RUNNING") {
|
|
input.log.error({ err, signal: input.signal }, "HTTP listener close failed");
|
|
}
|
|
resolve("closed");
|
|
});
|
|
input.server.closeIdleConnections?.();
|
|
});
|
|
return await Promise.race([
|
|
input.drainRequests ? input.drainRequests(controller.signal).then(() => {
|
|
input.server.closeIdleConnections?.();
|
|
return "closed" as const;
|
|
}) : closed,
|
|
new Promise<"timed_out">((resolve) => {
|
|
timer = setTimeout(() => {
|
|
input.log.info(
|
|
{ signal: input.signal, timeoutMs },
|
|
"HTTP listener drain timed out; closing the remaining connections",
|
|
);
|
|
input.server.closeAllConnections?.();
|
|
resolve("timed_out");
|
|
}, timeoutMs);
|
|
timer.unref?.();
|
|
}),
|
|
]);
|
|
} finally {
|
|
if (timer) clearTimeout(timer);
|
|
controller.abort();
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Runs the final, ordered teardown of the server. It awaits the application
|
|
* service cleanup first, so a live setup-token login session stops and releases
|
|
* its sandbox lease before the database and the provider stop. The caller runs
|
|
* `process.exit(0)` only after this helper resolves, so an orderly shutdown
|
|
* never leaves a sandbox lease or confidential login state alive past the
|
|
* process exit.
|
|
*
|
|
* A step that rejects does not stop the teardown. The helper logs the error and
|
|
* continues to the next step. A failed setup-token lease release stays a
|
|
* durable record for the startup reaper; the helper surfaces it in the log
|
|
* instead of blocking the exit path.
|
|
*/
|
|
export async function finalizeServerShutdown(input: {
|
|
signal: "SIGINT" | "SIGTERM";
|
|
shutdownAppServices: (() => Promise<void>) | undefined;
|
|
/**
|
|
* Stops the HTTP listener and drains its connections (see
|
|
* `closeHttpListenerForShutdown`). Runs first, while every application
|
|
* service is still available to the requests being drained, so no request
|
|
* runs against a half-dismantled service or an ended pool.
|
|
*/
|
|
closeHttpListener?: (() => Promise<unknown>) | null;
|
|
/**
|
|
* Waits for every run-failure Sentry report still in flight. Runs after the
|
|
* application services and before the database pool ends, so a report that
|
|
* started just before shutdown still gets its database read and reaches
|
|
* Sentry before `shutdownSentry` flushes and closes the client.
|
|
*/
|
|
drainPendingRunFailureReports?: (() => Promise<void>) | null;
|
|
/**
|
|
* Ends the server's PostgreSQL client pools. Runs after the application
|
|
* services (which still need the database) and before the embedded
|
|
* provider stops, so the backends close in order and none outlive the
|
|
* process.
|
|
*/
|
|
closeDatabase?: (() => Promise<void>) | null;
|
|
stopEmbeddedPostgres: (() => Promise<void>) | null;
|
|
shutdownInstrumentation: () => Promise<void>;
|
|
shutdownSentry: () => Promise<void>;
|
|
log: ShutdownLogger;
|
|
}): Promise<void> {
|
|
const { signal } = input;
|
|
|
|
// Stop accepting requests and drain the open ones before any service goes
|
|
// away, so a request that is still in flight sees a fully working server.
|
|
if (input.closeHttpListener) {
|
|
try {
|
|
await input.closeHttpListener();
|
|
} catch (err) {
|
|
input.log.error({ err, signal }, "HTTP listener shutdown failed");
|
|
}
|
|
}
|
|
|
|
// Await the application service cleanup, so a live setup-token login session
|
|
// releases its sandbox lease before the database and the provider stop. A
|
|
// rejected cleanup stays durable for the reaper; it does not block the exit.
|
|
try {
|
|
await input.shutdownAppServices?.();
|
|
} catch (err) {
|
|
input.log.error({ err, signal }, "Application service shutdown failed");
|
|
}
|
|
|
|
// Wait for every in-flight run-failure Sentry report before the database
|
|
// pool ends. `reportRunFailure` is fire-and-forget: without this wait, a
|
|
// report that started just before shutdown can lose its database read to
|
|
// the pool end below, or lose its Sentry call to the flush further down.
|
|
if (input.drainPendingRunFailureReports) {
|
|
try {
|
|
await input.drainPendingRunFailureReports();
|
|
} catch (err) {
|
|
input.log.error({ err, signal }, "run-failure report drain failed");
|
|
}
|
|
}
|
|
|
|
// End the client pools once nothing needs them any more. Without this the
|
|
// process exit leaves the pooled backends to PostgreSQL's own TCP keepalive
|
|
// reaping, and a restart loop can pile up enough of them to hit
|
|
// `max_connections` before the next boot gets a connection.
|
|
if (input.closeDatabase) {
|
|
try {
|
|
await input.closeDatabase();
|
|
} catch (err) {
|
|
input.log.error({ err, signal }, "Database client shutdown failed");
|
|
}
|
|
}
|
|
|
|
if (input.stopEmbeddedPostgres) {
|
|
input.log.info({ signal }, "Stopping embedded PostgreSQL");
|
|
try {
|
|
await input.stopEmbeddedPostgres();
|
|
} catch (err) {
|
|
input.log.error({ err }, "Failed to stop embedded PostgreSQL cleanly");
|
|
}
|
|
}
|
|
|
|
// Flush buffered OTel spans before the process goes away; without this await
|
|
// the exporter's final batch is dropped on exit.
|
|
await input.shutdownInstrumentation();
|
|
|
|
// Flush buffered Sentry events before the process goes away; without this
|
|
// await the last events are dropped on exit.
|
|
await input.shutdownSentry();
|
|
}
|
|
|
|
const COORDINATED_SHUTDOWN_SIGNALS = ["SIGINT", "SIGTERM"] as const;
|
|
|
|
type ShutdownSignalTarget = {
|
|
rawListeners(eventName: string): Function[];
|
|
removeListener(eventName: string, listener: (...args: any[]) => void): unknown;
|
|
};
|
|
|
|
/**
|
|
* Some dependencies eagerly install process signal handlers as an import side
|
|
* effect. Paperclip must remain the sole owner of SIGINT/SIGTERM ordering: its
|
|
* handler first snapshots live heartbeat runs and only then stops embedded
|
|
* infrastructure. Remove only listeners added by the supplied import, while
|
|
* preserving every listener that was already registered.
|
|
*/
|
|
export async function loadWithoutCoordinatedShutdownSignalHooks<T>(
|
|
load: () => Promise<T>,
|
|
signalTarget: ShutdownSignalTarget = process,
|
|
) {
|
|
const listenersBeforeLoad = new Map(
|
|
COORDINATED_SHUTDOWN_SIGNALS.map((signal) => [
|
|
signal,
|
|
signalTarget.rawListeners(signal),
|
|
]),
|
|
);
|
|
|
|
let loaded: T;
|
|
try {
|
|
loaded = await load();
|
|
} finally {
|
|
for (const signal of COORDINATED_SHUTDOWN_SIGNALS) {
|
|
const remainingBeforeLoad = [...(listenersBeforeLoad.get(signal) ?? [])];
|
|
for (const listener of signalTarget.rawListeners(signal)) {
|
|
const existingIndex = remainingBeforeLoad.indexOf(listener);
|
|
if (existingIndex >= 0) {
|
|
remainingBeforeLoad.splice(existingIndex, 1);
|
|
continue;
|
|
}
|
|
signalTarget.removeListener(signal, listener as (...args: any[]) => void);
|
|
}
|
|
}
|
|
}
|
|
|
|
return loaded;
|
|
}
|
|
|
|
export async function coordinateHeartbeatSchedulerShutdown<
|
|
TPreparation extends HotRestartShutdownPreparation,
|
|
>(input: {
|
|
signal: "SIGINT" | "SIGTERM";
|
|
prepareHotRestartShutdown: ((signal: "SIGINT" | "SIGTERM") => Promise<TPreparation>) | null;
|
|
waitForHeartbeatSchedulerIdle: () => Promise<void>;
|
|
}): Promise<{
|
|
hotRestart: TPreparation | null;
|
|
preparationError: unknown;
|
|
waitedForSchedulerIdle: boolean;
|
|
}> {
|
|
let hotRestart: TPreparation | null = null;
|
|
let preparationError: unknown = null;
|
|
|
|
// The signal handler stops the scheduler before entering this coordinator.
|
|
// Quiesce any callback that was already in flight before querying running
|
|
// rows for the shutdown snapshot, otherwise a late queue claim can create a
|
|
// run that is absent from both the snapshot and the selective drain set.
|
|
await input.waitForHeartbeatSchedulerIdle();
|
|
|
|
if (input.prepareHotRestartShutdown) {
|
|
try {
|
|
hotRestart = await input.prepareHotRestartShutdown(input.signal);
|
|
} catch (err) {
|
|
preparationError = err;
|
|
}
|
|
}
|
|
|
|
return {
|
|
hotRestart,
|
|
preparationError,
|
|
waitedForSchedulerIdle: true,
|
|
};
|
|
}
|
|
|
|
export async function finishServiceShutdown(
|
|
stop: () => Promise<void>,
|
|
cleanup: () => Promise<void>,
|
|
): Promise<void> {
|
|
const failures: unknown[] = [];
|
|
let retryStop = false;
|
|
try { await stop(); } catch { retryStop = true; }
|
|
try { await cleanup(); } catch (error) { failures.push(error); }
|
|
if (retryStop) {
|
|
try { await stop(); } catch (error) { failures.unshift(error); }
|
|
}
|
|
if (failures.length === 1) throw failures[0];
|
|
if (failures.length > 1) throw new AggregateError(failures, "Service shutdown failed");
|
|
}
|