fix(heartbeat): count an early native hand-off only after its suspend commits (#21572)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A server restart hands each live Runner run to the next server, so
the agent keeps working without a retry
> - At shutdown start, the outgoing server hands off native runs early,
in parallel with the rest of shutdown
> - The early hand-off put a run in the "handed off" set before its
suspend finished, and the drain counted every run in that set as
suspended without waiting
> - The early hand-off shares the 20 s pre-drain budget, and one Runner
detach can take 20-30 s, so the process could exit with the suspend
unfinished
> - Then the finalization lease stays live, the next server gets no
hand-over signal, and the run waits for lease expiry or the reaper
> - This pull request makes the drain wait for each in-flight early
hand-off, and fall back to its own hand-off when the early one failed or
did not finish
> - The benefit is that a run counts as handed over only after its
hand-over is committed, so the next server can always reach it at once

## Linked Issues or Issue Description

**What happened?**
A passing zero-downtime swap test showed a latent race in the shutdown
hand-off. `handOffNativeRunsForShutdown` added a run to
`nativeRunsHandedOffEarly` before `suspendNativeRunForShutdown`
finished. `drainRunningRunsForShutdown` then reported every run in that
set as `restartSuspended` without awaiting it. Because the early
hand-off shares the 20 s pre-drain budget and a Runner detach can wait
20-30 s, the process can exit before the lease release commits. Also, a
timed-out `prepareHotRestartShutdown` could still write the hot-restart
snapshot and detach sessions after the fallback drain started. In the
same test, swap 1's early hand-off took 2.08 s, and no log showed which
phase was slow.

**Expected behavior**
A run counts as handed over only after its suspend commits. If the early
suspend fails or does not finish, the drain hands the run off itself. A
timed-out hot-restart preparation does no further work after the drain
starts. The hand-off logs its phase timings.

**Steps to reproduce**
1. Start a native Runner run on a remote sandbox.
2. Send SIGTERM while the Runner detach is slow (or fails).
3. The drain reports the run as restart-suspended while its suspend is
still pending, and the process can exit before the lease release.

**Paperclip version or commit**
`master` at bbed32450c.

**Deployment mode**
Hosted, zero-downtime release swap.

## What Changed

- `nativeRunsHandedOffEarly` now holds each run's in-flight suspend
promise, not only its id.
- The shutdown drain awaits that promise (bounded at 30 s). If it fails,
returns false or does not finish, the drain runs its own suspend, as it
does for a run without an early hand-off.
- `runBoundedShutdownStep` and the pre-drain budget pass an
`AbortSignal` to each step and abort it on timeout.
- `prepareHotRestartShutdown` takes that signal and stops before the
snapshot write, the run events, and the session detach when the step
timed out.
- `handOffNativeRunsForShutdown` logs one structured line, "native run
shutdown hand-off phase timings": intent read, controller identity,
select, max per-run detach, max per-run transaction, and total.

## Verification

- Red, then green (each failed against `master` source and passes with
the fix), all through the public shutdown functions in
`server/src/services/heartbeat/recovery.test.ts`:
- `does not count an early hand-off as suspended while its detach is
still pending at the shutdown drain`
- `hands off a run at the shutdown drain when its pending early hand-off
fails`
- `writes no hot-restart snapshot and detaches nothing after a timed-out
preparation hands over to the drain`
- New test in `server/src/shutdown.test.ts`: `aborts the signal of a
step that outlives its bound, and only that step`.
- `pnpm vitest run src/shutdown.test.ts
src/services/heartbeat/recovery.test.ts
src/__tests__/heartbeat-native-early-hand-off.test.ts` passes (145
tests).
- `tsc --noEmit` for `server` passes. `pnpm verify` passes for the
selected tests.

## Risks

- When an early hand-off does not finish within 30 s, the drain starts a
second suspend. The second detach joins the first, because the session
memoizes its restart detach. The run can then get two "suspended" run
events. That is preferred to reporting a hand-over that did not happen.
- The drain can now wait up to 30 s for a slow early hand-off before the
process exits. Before, it exited without the hand-over.
- The abort signal only stops work between steps of the preparation. A
snapshot write that already started still completes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-14 03:02:25 -06:00
1 parent 633dec7e43
commit cf09024292
5 files changed
+200 -18

No files matched your search

+3 -3
View File
@@ -1303,7 +1303,7 @@ async function startServerWithDatabaseTeardown(
) => Promise<unknown>) | null = null;
let drainHeartbeatExecutionFinalizers: (() => Promise<void>) | null = null;
let handOffNativeRunsForShutdown: ((signal: "SIGINT" | "SIGTERM") => Promise<string[]>) | null = null;
let prepareHotRestartShutdown: ((signal: "SIGINT" | "SIGTERM") => Promise<{
let prepareHotRestartShutdown: ((signal: "SIGINT" | "SIGTERM", now?: Date, abort?: AbortSignal) => Promise<{
skipDrain: boolean;
drainRunIds?: string[];
}>) | null = null;
@@ -2281,8 +2281,8 @@ async function startServerWithDatabaseTeardown(
signal,
prepareHotRestartShutdown: prepareHotRestart
? async (preparationSignal) => {
const preparation = await preDrain.run("hot_restart_prepare", () =>
prepareHotRestart(preparationSignal));
const preparation = await preDrain.run("hot_restart_prepare", abort =>
prepareHotRestart(preparationSignal, undefined, abort));
if (preparation.outcome === "completed") return preparation.value;
if (preparation.outcome === "failed") throw preparation.error;
throw new Error("hot-restart shutdown preparation timed out");
@@ -31,6 +31,7 @@ import { persistHeartbeatRunProcessMetadata } from "./run-lifecycle.js";
import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
import { hasNativeLocalProcessStop, PROCESS_START_REQUESTED, releaseSettledNativeRunProcess } from "../native-local-process-stop.js";
import { legacyControllerBootId } from "../legacy-controller-lease.js";
import { runBoundedShutdownStep } from "../../shutdown.js";
type Run = typeof heartbeatRuns.$inferSelect;
@@ -44,10 +45,12 @@ const native = vi.hoisted(() => ({
}));
const hotRestart = vi.hoisted(() => ({
read: vi.fn<() => Promise<HotRestartIntent | null>>(async () => null),
write: vi.fn(async () => undefined),
}));
vi.mock("../hot-restart.js", async (importOriginal) => ({
...await importOriginal<typeof import("../hot-restart.js")>(),
readHotRestartIntent: hotRestart.read,
writeHotRestartShutdownSnapshot: hotRestart.write,
}));
vi.mock("../native-runtime/index.js", async (importOriginal) => ({
...await importOriginal<typeof import("../native-runtime/index.js")>(),
@@ -1500,6 +1503,122 @@ describePostgres("heartbeat recovery module ownership", () => {
}
});
async function seedOwnedNativeRun() {
const run = await seedRun();
const identity = await currentNativeControllerIdentity();
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, run.agentId));
const [nativeRun] = await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: run.issueId, nativeSessionId: randomUUID() })
.where(eq(heartbeatRuns.id, run.id)).returning();
await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!,
phase: "observed", controllerBootId: identity.bootId, leaseOwner: "departing-server", leaseExpiresAt: new Date(Date.now() + 60_000) });
return nativeRun;
}
async function removeNativeRun(runId: string) {
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
}
const finalization = async (runId: string) =>
(await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)))[0];
const drainHasStarted = () => new Promise(resolve => setTimeout(resolve, 250));
it("does not count an early hand-off as suspended while its detach is still pending at the shutdown drain", async () => {
const run = await seedOwnedNativeRun(), recovery = createHeartbeatRecovery(db, callbacks(db));
let finishDetach!: () => void;
native.detach.mockImplementationOnce(() => new Promise<number>(resolve => { finishDetach = () => resolve(0); }));
try {
const early = recovery.handOffNativeRunsForShutdown("SIGTERM");
await vi.waitFor(() => expect(native.detach).toHaveBeenCalledTimes(1));
let drained = false;
const drain = recovery.drainRunningRunsForShutdown("SIGTERM", new Date(), [run.id]).finally(() => { drained = true; });
await drainHasStarted();
expect(drained).toBe(false);
expect((await finalization(run.id)).recoveryState).toBeNull();
finishDetach();
expect(await early).toEqual([run.id]);
expect((await drain).restartSuspendedRunIds).toEqual([run.id]);
expect(native.detach).toHaveBeenCalledTimes(1);
expect(await finalization(run.id)).toMatchObject({ recoveryState: "awaiting_runner_reattach" });
} finally {
finishDetach?.();
await removeNativeRun(run.id);
}
});
it("hands off a run at the shutdown drain when its pending early hand-off fails", async () => {
const run = await seedOwnedNativeRun(), recovery = createHeartbeatRecovery(db, callbacks(db));
let failDetach!: (error: Error) => void;
native.detach.mockImplementationOnce(() => new Promise<number>((_resolve, reject) => { failDetach = reject; }));
try {
const early = recovery.handOffNativeRunsForShutdown("SIGTERM");
await vi.waitFor(() => expect(native.detach).toHaveBeenCalledTimes(1));
const drain = recovery.drainRunningRunsForShutdown("SIGTERM", new Date(), [run.id]);
await drainHasStarted();
failDetach(new Error("runner detach failed"));
expect(await early).toEqual([]);
expect((await drain).restartSuspendedRunIds).toEqual([run.id]);
expect(native.detach).toHaveBeenCalledTimes(2);
expect(await finalization(run.id)).toMatchObject({ recoveryState: "awaiting_runner_reattach" });
} finally {
await removeNativeRun(run.id);
}
});
it("writes no hot-restart snapshot and detaches nothing after a timed-out preparation hands over to the drain", async () => {
const run = await seedOwnedNativeRun(), recovery = createHeartbeatRecovery(db, callbacks(db));
hotRestart.read.mockResolvedValue({ version: 1, requestedAt: new Date().toISOString(), previousServerPid: process.pid,
previousServerVersion: null, drainRequired: false, requestedByRunId: null, preflightActiveRunIds: [] });
let finishIdleClose!: () => void;
native.closeIdle.mockImplementationOnce(() => new Promise(resolve => { finishIdleClose = () => resolve({ closed: 0, failed: 0 }); }));
let preparation: ReturnType<typeof recovery.prepareHotRestartShutdown> | undefined;
try {
const step = await runBoundedShutdownStep({ step: "hot_restart_prepare", signal: "SIGTERM", timeoutMs: 50,
log: { info: vi.fn(), error: vi.fn() },
run: abort => (preparation = recovery.prepareHotRestartShutdown("SIGTERM", new Date(), abort)) });
expect(step.outcome).toBe("timed_out");
expect((await recovery.drainRunningRunsForShutdown("SIGTERM", new Date(), [run.id])).restartSuspendedRunIds).toEqual([run.id]);
expect(native.detach).toHaveBeenCalledTimes(1);
finishIdleClose();
await expect(preparation).rejects.toThrow("timed out");
expect(hotRestart.write).not.toHaveBeenCalled();
expect(native.detach).toHaveBeenCalledTimes(1);
} finally {
finishIdleClose?.();
await preparation?.catch(() => undefined);
await removeNativeRun(run.id);
}
});
it("records no more hot-restart run events after the preparation times out during its first event", async () => {
const runs = [await seedOwnedNativeRun(), await seedOwnedNativeRun(), await seedOwnedNativeRun()];
const deps = callbacks(db), recovery = createHeartbeatRecovery(db, deps);
hotRestart.read.mockResolvedValue({ version: 1, requestedAt: new Date().toISOString(), previousServerPid: process.pid,
previousServerVersion: null, drainRequired: false, requestedByRunId: null, preflightActiveRunIds: [] });
let finishFirstEvent!: () => void;
const firstEvent = new Promise<undefined>(resolve => { finishFirstEvent = () => resolve(undefined); });
deps.appendRunEvent.mockImplementationOnce(() => firstEvent);
const hotRestartEvents = () => deps.appendRunEvent.mock.calls
.filter(([, event]) => event.message?.startsWith("Hot restart requested") === true).length;
let preparation: ReturnType<typeof recovery.prepareHotRestartShutdown> | undefined;
try {
const step = await runBoundedShutdownStep({ step: "hot_restart_prepare", signal: "SIGTERM", timeoutMs: 2_000,
log: { info: vi.fn(), error: vi.fn() },
run: abort => (preparation = recovery.prepareHotRestartShutdown("SIGTERM", new Date(), abort)) });
expect(step.outcome).toBe("timed_out");
expect(hotRestartEvents()).toBe(1);
const runIds = runs.map(run => run.id);
expect((await recovery.drainRunningRunsForShutdown("SIGTERM", new Date(), runIds)).restartSuspendedRunIds.sort())
.toEqual([...runIds].sort());
finishFirstEvent();
await expect(preparation).rejects.toThrow("timed out");
expect(hotRestartEvents()).toBe(1);
expect(native.detach).toHaveBeenCalledTimes(runs.length);
} finally {
finishFirstEvent?.();
await preparation?.catch(() => undefined);
for (const run of runs) await removeNativeRun(run.id);
}
});
it("does not dispatch recovery for a run still preparing in this heartbeat process", async () => {
const run = await seedRun();
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, run.agentId));
+58 -12
View File
@@ -213,6 +213,8 @@ export const DETACHED_PROCESS_ERROR_CODE = "process_detached";
const NATIVE_OWNERSHIP_UNVERIFIED_MESSAGE =
"Native execution ownership could not be verified; automatic recovery is blocked";
const EARLY_NATIVE_HAND_OFF_DRAIN_WAIT_MS = 30_000;
// The reaper sweeps at most this many pending_cleanup leases per tick.
const PENDING_CLEANUP_SWEEP_PAGE_SIZE = 20;
// An environment delete tries each of that environment's cleanups once, from
@@ -490,12 +492,14 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
async function prepareHotRestartShutdown(
signal: "SIGINT" | "SIGTERM",
now = new Date(),
abort?: AbortSignal,
) {
enterShutdown();
const idleSessions = await closeIdleWarmNativeSessionsForRestart();
if (idleSessions.failed > 0) {
logger.warn({ idleSessions }, "idle native session cleanup failed before controller shutdown");
}
abort?.throwIfAborted();
let intent: Awaited<ReturnType<typeof readHotRestartIntent>>;
try {
intent = await readHotRestartIntent();
@@ -544,6 +548,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
.from(heartbeatRuns)
.innerJoin(agents, eq(heartbeatRuns.agentId, agents.id))
.where(eq(heartbeatRuns.status, "running"));
abort?.throwIfAborted();
const snapshotRuns = activeRuns.map(toHotRestartIntentRun);
const intentWithVersion = {
...intent,
@@ -589,8 +594,8 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
activeRuns: snapshotRuns,
capturedAt: now,
});
for (const { run } of activeRuns) {
abort?.throwIfAborted();
await appendRunEvent(run, {
eventType: "lifecycle",
stream: "system",
@@ -614,6 +619,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
isNativeSessionId(run.nativeSessionId),
)
.map(({ run }) => run.id);
abort?.throwIfAborted();
for (const runId of nativeRunIds) markHandedOffByThisController(runId);
const detachedNativeSessions =
await detachNativeSessionsForRestart(nativeRunIds);
@@ -1158,10 +1164,27 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
}
}
const nativeRunsHandedOffEarly = new Set<string>();
const nativeRunsHandedOffEarly = new Map<string, Promise<boolean>>();
async function settleEarlyNativeHandOff(runId: string, handOff: Promise<boolean>) {
let timer: ReturnType<typeof setTimeout> | undefined;
try {
const outcome = await Promise.race([handOff, new Promise<"unfinished">(resolve => {
timer = setTimeout(() => resolve("unfinished"), EARLY_NATIVE_HAND_OFF_DRAIN_WAIT_MS);
timer.unref?.();
})]);
if (outcome === "unfinished") {
logger.warn({ runId, waitedMs: EARLY_NATIVE_HAND_OFF_DRAIN_WAIT_MS },
"early native run hand-off is unfinished at the shutdown drain; the drain hands it off");
}
return outcome === true;
} finally {
clearTimeout(timer);
}
}
async function suspendNativeRunForShutdown(run: typeof heartbeatRuns.$inferSelect, signal: "SIGINT" | "SIGTERM",
now: Date, controllerBootId: string) {
now: Date, controllerBootId: string, timing?: { detachMs: number; transactionMs: number }) {
const [owner] = await db.select({ bootId: nativeRunFinalizations.controllerBootId })
.from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
if (owner?.bootId && owner.bootId !== controllerBootId) return false;
@@ -1170,7 +1193,9 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
// finalizer interrupt/suspend Claude while the next server is adopting
// the same turn.
markHandedOffByThisController(run.id);
const detachStartedAt = performance.now();
await detachNativeSessionsForRestart([run.id]);
if (timing) timing.detachMs = Math.round(performance.now() - detachStartedAt);
const recoveryHistoryEntry = JSON.stringify({
at: now.toISOString(),
restartKind: "graceful",
@@ -1183,6 +1208,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
// execution lease must still be released: the detached executor will
// not finalize, and the next controller's reconciler skips leased rows.
// A finalizer lease belongs to finalization already in progress here.
const transactionStartedAt = performance.now();
const released = await db.transaction(async tx => {
await notifyLiveRunRecovery(tx, { companyId: run.companyId, runId: run.id });
await notifyReattachHandOver(tx, run.id);
@@ -1217,6 +1243,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
)
.returning({ resultId: nativeRunFinalizations.resultId, leaseOwner: nativeRunFinalizations.leaseOwner });
});
if (timing) timing.transactionMs = Math.round(performance.now() - transactionStartedAt);
const resultAccepted = Boolean(released[0]?.resultId);
const finalizerHeld = resultAccepted && Boolean(released[0]?.leaseOwner?.startsWith("native-finalizer:"));
await appendRunEvent(run, {
@@ -1243,25 +1270,40 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
}
async function handOffNativeRunsForShutdown(signal: "SIGINT" | "SIGTERM", now = new Date()) {
const startedAt = performance.now();
const elapsed = (from: number) => Math.round(performance.now() - from);
const intent = await readHotRestartIntent().catch(() => null);
const intentReadMs = elapsed(startedAt);
if (intent && !intent.drainRequired && shouldHonorHotRestartIntentForProcess(intent)) return [];
const identityStartedAt = performance.now();
const nativeController = await currentNativeControllerIdentity();
const controllerIdentityMs = elapsed(identityStartedAt);
const selectStartedAt = performance.now();
const liveRuns = await db.select({ run: heartbeatRuns }).from(heartbeatRuns)
.innerJoin(agents, eq(heartbeatRuns.agentId, agents.id))
.where(and(eq(heartbeatRuns.status, "running"), eq(heartbeatRuns.runtimeMode, "native"),
eq(agents.adapterType, "paperclip_runner")));
const selectMs = elapsed(selectStartedAt);
const handedOff: string[] = [];
let maxDetachMs = 0, maxTransactionMs = 0, failed = 0;
await Promise.allSettled(liveRuns.map(async ({ run }) => {
if (isNativeRunnerOwnershipHeld(run) || nativeRunsHandedOffEarly.has(run.id)) return;
nativeRunsHandedOffEarly.add(run.id);
try {
if (await suspendNativeRunForShutdown(run, signal, now, nativeController.bootId)) handedOff.push(run.id);
else nativeRunsHandedOffEarly.delete(run.id);
} catch (error) {
nativeRunsHandedOffEarly.delete(run.id);
const timing = { detachMs: 0, transactionMs: 0 };
const handOff = suspendNativeRunForShutdown(run, signal, now, nativeController.bootId, timing).catch(error => {
failed += 1;
logger.warn({ err: error, runId: run.id }, "early native run hand-off failed; the shutdown drain retries it");
}
return false;
});
nativeRunsHandedOffEarly.set(run.id, handOff);
const suspended = await handOff;
maxDetachMs = Math.max(maxDetachMs, timing.detachMs);
maxTransactionMs = Math.max(maxTransactionMs, timing.transactionMs);
if (suspended) handedOff.push(run.id);
else if (nativeRunsHandedOffEarly.get(run.id) === handOff) nativeRunsHandedOffEarly.delete(run.id);
}));
logger.info({ signal, liveRuns: liveRuns.length, handedOff: handedOff.length, failed, intentReadMs,
controllerIdentityMs, selectMs, maxDetachMs, maxTransactionMs, totalMs: elapsed(startedAt) },
"native run shutdown hand-off phase timings");
return handedOff;
}
@@ -1311,8 +1353,12 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
run.runtimeMode === "native" &&
agent.adapterType === "paperclip_runner"
) {
if (nativeRunsHandedOffEarly.has(run.id)
|| await suspendNativeRunForShutdown(run, signal, now, nativeController.bootId)) restartSuspendedRunIds.push(run.id);
const earlyHandOff = nativeRunsHandedOffEarly.get(run.id);
if (earlyHandOff && await settleEarlyNativeHandOff(run.id, earlyHandOff)) {
restartSuspendedRunIds.push(run.id);
return;
}
if (await suspendNativeRunForShutdown(run, signal, now, nativeController.bootId)) restartSuspendedRunIds.push(run.id);
return;
}
if (agent.adapterType === "paperclip_runner" && run.runtimeMode === "legacy" && run.scopeKind === "issue" &&
+15
View File
@@ -410,6 +410,21 @@ describe("runBoundedShutdownStep", () => {
}
});
it("aborts the signal of a step that outlives its bound, and only that step", async () => {
vi.useFakeTimers();
try {
const signals: AbortSignal[] = [];
const budget = createPreDrainShutdownBudget({ signal: "SIGTERM", budgetMs: 250, log: stubLogger() });
await budget.run("delivery_work_stop", async abort => { signals.push(abort); });
const pending = budget.run("hot_restart_prepare", abort => { signals.push(abort); return new Promise<void>(() => undefined); });
await vi.advanceTimersByTimeAsync(250);
await expect(pending).resolves.toMatchObject({ outcome: "timed_out" });
expect(signals.map(signal => signal.aborted)).toEqual([false, true]);
} finally {
vi.useRealTimers();
}
});
it("logs a step that rejects and does not reject the shutdown", async () => {
const log = stubLogger();
const error = new Error("skill sync stop failed");
+5 -3
View File
@@ -23,18 +23,19 @@ export async function runBoundedShutdownStep<T>(input: {
step: string;
signal: "SIGINT" | "SIGTERM";
timeoutMs: number;
run: () => Promise<T>;
run: (abort: AbortSignal) => Promise<T>;
log: ShutdownLogger;
now?: () => number;
}): Promise<ShutdownStepResult<T>> {
const now = input.now ?? Date.now;
const startedAt = now();
const timeoutMs = Math.max(0, input.timeoutMs);
const abort = new AbortController();
let timer: NodeJS.Timeout | null = null;
try {
const result = await Promise.race([
Promise.resolve()
.then(input.run)
.then(() => input.run(abort.signal))
.then(
(value) => ({ outcome: "completed" as const, value }),
(error: unknown) => ({ outcome: "failed" as const, error }),
@@ -54,6 +55,7 @@ export async function runBoundedShutdownStep<T>(input: {
input.log.error({ ...fields, err: result.error }, "shutdown step failed; continuing shutdown");
return { outcome: "failed", error: result.error, durationMs };
}
abort.abort(new Error(`shutdown step ${input.step} timed out`));
input.log.info(fields, "shutdown step timed out; continuing shutdown without it");
return { outcome: "timed_out", durationMs };
} finally {
@@ -70,7 +72,7 @@ export function createPreDrainShutdownBudget(input: {
const now = input.now ?? Date.now;
const deadline = now() + (input.budgetMs ?? PRE_DRAIN_SHUTDOWN_BUDGET_MS);
return {
run<T>(step: string, run: () => Promise<T>) {
run<T>(step: string, run: (abort: AbortSignal) => Promise<T>) {
return runBoundedShutdownStep({
step,
signal: input.signal,