From cf09024292136a07c47a65f8d31c928240e6e0d5 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 14 Nov 2026 03:02:25 -0600 Subject: [PATCH] fix(heartbeat): count an early native hand-off only after its suspend commits (#21572) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A server restart hands each live Runner run to the next server, so the agent keeps working without a retry > - At shutdown start, the outgoing server hands off native runs early, in parallel with the rest of shutdown > - The early hand-off put a run in the "handed off" set before its suspend finished, and the drain counted every run in that set as suspended without waiting > - The early hand-off shares the 20 s pre-drain budget, and one Runner detach can take 20-30 s, so the process could exit with the suspend unfinished > - Then the finalization lease stays live, the next server gets no hand-over signal, and the run waits for lease expiry or the reaper > - This pull request makes the drain wait for each in-flight early hand-off, and fall back to its own hand-off when the early one failed or did not finish > - The benefit is that a run counts as handed over only after its hand-over is committed, so the next server can always reach it at once ## Linked Issues or Issue Description **What happened?** A passing zero-downtime swap test showed a latent race in the shutdown hand-off. `handOffNativeRunsForShutdown` added a run to `nativeRunsHandedOffEarly` before `suspendNativeRunForShutdown` finished. `drainRunningRunsForShutdown` then reported every run in that set as `restartSuspended` without awaiting it. Because the early hand-off shares the 20 s pre-drain budget and a Runner detach can wait 20-30 s, the process can exit before the lease release commits. Also, a timed-out `prepareHotRestartShutdown` could still write the hot-restart snapshot and detach sessions after the fallback drain started. In the same test, swap 1's early hand-off took 2.08 s, and no log showed which phase was slow. **Expected behavior** A run counts as handed over only after its suspend commits. If the early suspend fails or does not finish, the drain hands the run off itself. A timed-out hot-restart preparation does no further work after the drain starts. The hand-off logs its phase timings. **Steps to reproduce** 1. Start a native Runner run on a remote sandbox. 2. Send SIGTERM while the Runner detach is slow (or fails). 3. The drain reports the run as restart-suspended while its suspend is still pending, and the process can exit before the lease release. **Paperclip version or commit** `master` at bbed32450c. **Deployment mode** Hosted, zero-downtime release swap. ## What Changed - `nativeRunsHandedOffEarly` now holds each run's in-flight suspend promise, not only its id. - The shutdown drain awaits that promise (bounded at 30 s). If it fails, returns false or does not finish, the drain runs its own suspend, as it does for a run without an early hand-off. - `runBoundedShutdownStep` and the pre-drain budget pass an `AbortSignal` to each step and abort it on timeout. - `prepareHotRestartShutdown` takes that signal and stops before the snapshot write, the run events, and the session detach when the step timed out. - `handOffNativeRunsForShutdown` logs one structured line, "native run shutdown hand-off phase timings": intent read, controller identity, select, max per-run detach, max per-run transaction, and total. ## Verification - Red, then green (each failed against `master` source and passes with the fix), all through the public shutdown functions in `server/src/services/heartbeat/recovery.test.ts`: - `does not count an early hand-off as suspended while its detach is still pending at the shutdown drain` - `hands off a run at the shutdown drain when its pending early hand-off fails` - `writes no hot-restart snapshot and detaches nothing after a timed-out preparation hands over to the drain` - New test in `server/src/shutdown.test.ts`: `aborts the signal of a step that outlives its bound, and only that step`. - `pnpm vitest run src/shutdown.test.ts src/services/heartbeat/recovery.test.ts src/__tests__/heartbeat-native-early-hand-off.test.ts` passes (145 tests). - `tsc --noEmit` for `server` passes. `pnpm verify` passes for the selected tests. ## Risks - When an early hand-off does not finish within 30 s, the drain starts a second suspend. The second detach joins the first, because the session memoizes its restart detach. The run can then get two "suspended" run events. That is preferred to reporting a hand-over that did not happen. - The drain can now wait up to 30 s for a slow early hand-off before the process exits. Before, it exited without the hand-over. - The abort signal only stops work between steps of the preparation. A snapshot write that already started still completes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- server/src/index.ts | 6 +- .../src/services/heartbeat/recovery.test.ts | 119 ++++++++++++++++++ server/src/services/heartbeat/recovery.ts | 70 +++++++++-- server/src/shutdown.test.ts | 15 +++ server/src/shutdown.ts | 8 +- 5 files changed, 200 insertions(+), 18 deletions(-) diff --git a/server/src/index.ts b/server/src/index.ts index ad8fde47768..340b576632c 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1303,7 +1303,7 @@ async function startServerWithDatabaseTeardown( ) => Promise) | null = null; let drainHeartbeatExecutionFinalizers: (() => Promise) | null = null; let handOffNativeRunsForShutdown: ((signal: "SIGINT" | "SIGTERM") => Promise) | null = null; - let prepareHotRestartShutdown: ((signal: "SIGINT" | "SIGTERM") => Promise<{ + let prepareHotRestartShutdown: ((signal: "SIGINT" | "SIGTERM", now?: Date, abort?: AbortSignal) => Promise<{ skipDrain: boolean; drainRunIds?: string[]; }>) | null = null; @@ -2281,8 +2281,8 @@ async function startServerWithDatabaseTeardown( signal, prepareHotRestartShutdown: prepareHotRestart ? async (preparationSignal) => { - const preparation = await preDrain.run("hot_restart_prepare", () => - prepareHotRestart(preparationSignal)); + const preparation = await preDrain.run("hot_restart_prepare", abort => + prepareHotRestart(preparationSignal, undefined, abort)); if (preparation.outcome === "completed") return preparation.value; if (preparation.outcome === "failed") throw preparation.error; throw new Error("hot-restart shutdown preparation timed out"); diff --git a/server/src/services/heartbeat/recovery.test.ts b/server/src/services/heartbeat/recovery.test.ts index f1336e9c1ad..dc7683b7dc9 100644 --- a/server/src/services/heartbeat/recovery.test.ts +++ b/server/src/services/heartbeat/recovery.test.ts @@ -31,6 +31,7 @@ import { persistHeartbeatRunProcessMetadata } from "./run-lifecycle.js"; import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js"; import { hasNativeLocalProcessStop, PROCESS_START_REQUESTED, releaseSettledNativeRunProcess } from "../native-local-process-stop.js"; import { legacyControllerBootId } from "../legacy-controller-lease.js"; +import { runBoundedShutdownStep } from "../../shutdown.js"; type Run = typeof heartbeatRuns.$inferSelect; @@ -44,10 +45,12 @@ const native = vi.hoisted(() => ({ })); const hotRestart = vi.hoisted(() => ({ read: vi.fn<() => Promise>(async () => null), + write: vi.fn(async () => undefined), })); vi.mock("../hot-restart.js", async (importOriginal) => ({ ...await importOriginal(), readHotRestartIntent: hotRestart.read, + writeHotRestartShutdownSnapshot: hotRestart.write, })); vi.mock("../native-runtime/index.js", async (importOriginal) => ({ ...await importOriginal(), @@ -1500,6 +1503,122 @@ describePostgres("heartbeat recovery module ownership", () => { } }); + async function seedOwnedNativeRun() { + const run = await seedRun(); + const identity = await currentNativeControllerIdentity(); + await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, run.agentId)); + const [nativeRun] = await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: run.issueId, nativeSessionId: randomUUID() }) + .where(eq(heartbeatRuns.id, run.id)).returning(); + await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!, + phase: "observed", controllerBootId: identity.bootId, leaseOwner: "departing-server", leaseExpiresAt: new Date(Date.now() + 60_000) }); + return nativeRun; + } + async function removeNativeRun(runId: string) { + await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)); + await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + } + const finalization = async (runId: string) => + (await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)))[0]; + const drainHasStarted = () => new Promise(resolve => setTimeout(resolve, 250)); + + it("does not count an early hand-off as suspended while its detach is still pending at the shutdown drain", async () => { + const run = await seedOwnedNativeRun(), recovery = createHeartbeatRecovery(db, callbacks(db)); + let finishDetach!: () => void; + native.detach.mockImplementationOnce(() => new Promise(resolve => { finishDetach = () => resolve(0); })); + try { + const early = recovery.handOffNativeRunsForShutdown("SIGTERM"); + await vi.waitFor(() => expect(native.detach).toHaveBeenCalledTimes(1)); + let drained = false; + const drain = recovery.drainRunningRunsForShutdown("SIGTERM", new Date(), [run.id]).finally(() => { drained = true; }); + await drainHasStarted(); + expect(drained).toBe(false); + expect((await finalization(run.id)).recoveryState).toBeNull(); + finishDetach(); + expect(await early).toEqual([run.id]); + expect((await drain).restartSuspendedRunIds).toEqual([run.id]); + expect(native.detach).toHaveBeenCalledTimes(1); + expect(await finalization(run.id)).toMatchObject({ recoveryState: "awaiting_runner_reattach" }); + } finally { + finishDetach?.(); + await removeNativeRun(run.id); + } + }); + + it("hands off a run at the shutdown drain when its pending early hand-off fails", async () => { + const run = await seedOwnedNativeRun(), recovery = createHeartbeatRecovery(db, callbacks(db)); + let failDetach!: (error: Error) => void; + native.detach.mockImplementationOnce(() => new Promise((_resolve, reject) => { failDetach = reject; })); + try { + const early = recovery.handOffNativeRunsForShutdown("SIGTERM"); + await vi.waitFor(() => expect(native.detach).toHaveBeenCalledTimes(1)); + const drain = recovery.drainRunningRunsForShutdown("SIGTERM", new Date(), [run.id]); + await drainHasStarted(); + failDetach(new Error("runner detach failed")); + expect(await early).toEqual([]); + expect((await drain).restartSuspendedRunIds).toEqual([run.id]); + expect(native.detach).toHaveBeenCalledTimes(2); + expect(await finalization(run.id)).toMatchObject({ recoveryState: "awaiting_runner_reattach" }); + } finally { + await removeNativeRun(run.id); + } + }); + + it("writes no hot-restart snapshot and detaches nothing after a timed-out preparation hands over to the drain", async () => { + const run = await seedOwnedNativeRun(), recovery = createHeartbeatRecovery(db, callbacks(db)); + hotRestart.read.mockResolvedValue({ version: 1, requestedAt: new Date().toISOString(), previousServerPid: process.pid, + previousServerVersion: null, drainRequired: false, requestedByRunId: null, preflightActiveRunIds: [] }); + let finishIdleClose!: () => void; + native.closeIdle.mockImplementationOnce(() => new Promise(resolve => { finishIdleClose = () => resolve({ closed: 0, failed: 0 }); })); + let preparation: ReturnType | undefined; + try { + const step = await runBoundedShutdownStep({ step: "hot_restart_prepare", signal: "SIGTERM", timeoutMs: 50, + log: { info: vi.fn(), error: vi.fn() }, + run: abort => (preparation = recovery.prepareHotRestartShutdown("SIGTERM", new Date(), abort)) }); + expect(step.outcome).toBe("timed_out"); + expect((await recovery.drainRunningRunsForShutdown("SIGTERM", new Date(), [run.id])).restartSuspendedRunIds).toEqual([run.id]); + expect(native.detach).toHaveBeenCalledTimes(1); + finishIdleClose(); + await expect(preparation).rejects.toThrow("timed out"); + expect(hotRestart.write).not.toHaveBeenCalled(); + expect(native.detach).toHaveBeenCalledTimes(1); + } finally { + finishIdleClose?.(); + await preparation?.catch(() => undefined); + await removeNativeRun(run.id); + } + }); + + it("records no more hot-restart run events after the preparation times out during its first event", async () => { + const runs = [await seedOwnedNativeRun(), await seedOwnedNativeRun(), await seedOwnedNativeRun()]; + const deps = callbacks(db), recovery = createHeartbeatRecovery(db, deps); + hotRestart.read.mockResolvedValue({ version: 1, requestedAt: new Date().toISOString(), previousServerPid: process.pid, + previousServerVersion: null, drainRequired: false, requestedByRunId: null, preflightActiveRunIds: [] }); + let finishFirstEvent!: () => void; + const firstEvent = new Promise(resolve => { finishFirstEvent = () => resolve(undefined); }); + deps.appendRunEvent.mockImplementationOnce(() => firstEvent); + const hotRestartEvents = () => deps.appendRunEvent.mock.calls + .filter(([, event]) => event.message?.startsWith("Hot restart requested") === true).length; + let preparation: ReturnType | undefined; + try { + const step = await runBoundedShutdownStep({ step: "hot_restart_prepare", signal: "SIGTERM", timeoutMs: 2_000, + log: { info: vi.fn(), error: vi.fn() }, + run: abort => (preparation = recovery.prepareHotRestartShutdown("SIGTERM", new Date(), abort)) }); + expect(step.outcome).toBe("timed_out"); + expect(hotRestartEvents()).toBe(1); + const runIds = runs.map(run => run.id); + expect((await recovery.drainRunningRunsForShutdown("SIGTERM", new Date(), runIds)).restartSuspendedRunIds.sort()) + .toEqual([...runIds].sort()); + finishFirstEvent(); + await expect(preparation).rejects.toThrow("timed out"); + expect(hotRestartEvents()).toBe(1); + expect(native.detach).toHaveBeenCalledTimes(runs.length); + } finally { + finishFirstEvent?.(); + await preparation?.catch(() => undefined); + for (const run of runs) await removeNativeRun(run.id); + } + }); + it("does not dispatch recovery for a run still preparing in this heartbeat process", async () => { const run = await seedRun(); await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, run.agentId)); diff --git a/server/src/services/heartbeat/recovery.ts b/server/src/services/heartbeat/recovery.ts index b2a866808bd..210ab21ff02 100644 --- a/server/src/services/heartbeat/recovery.ts +++ b/server/src/services/heartbeat/recovery.ts @@ -213,6 +213,8 @@ export const DETACHED_PROCESS_ERROR_CODE = "process_detached"; const NATIVE_OWNERSHIP_UNVERIFIED_MESSAGE = "Native execution ownership could not be verified; automatic recovery is blocked"; +const EARLY_NATIVE_HAND_OFF_DRAIN_WAIT_MS = 30_000; + // The reaper sweeps at most this many pending_cleanup leases per tick. const PENDING_CLEANUP_SWEEP_PAGE_SIZE = 20; // An environment delete tries each of that environment's cleanups once, from @@ -490,12 +492,14 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD async function prepareHotRestartShutdown( signal: "SIGINT" | "SIGTERM", now = new Date(), + abort?: AbortSignal, ) { enterShutdown(); const idleSessions = await closeIdleWarmNativeSessionsForRestart(); if (idleSessions.failed > 0) { logger.warn({ idleSessions }, "idle native session cleanup failed before controller shutdown"); } + abort?.throwIfAborted(); let intent: Awaited>; try { intent = await readHotRestartIntent(); @@ -544,6 +548,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD .from(heartbeatRuns) .innerJoin(agents, eq(heartbeatRuns.agentId, agents.id)) .where(eq(heartbeatRuns.status, "running")); + abort?.throwIfAborted(); const snapshotRuns = activeRuns.map(toHotRestartIntentRun); const intentWithVersion = { ...intent, @@ -589,8 +594,8 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD activeRuns: snapshotRuns, capturedAt: now, }); - for (const { run } of activeRuns) { + abort?.throwIfAborted(); await appendRunEvent(run, { eventType: "lifecycle", stream: "system", @@ -614,6 +619,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD isNativeSessionId(run.nativeSessionId), ) .map(({ run }) => run.id); + abort?.throwIfAborted(); for (const runId of nativeRunIds) markHandedOffByThisController(runId); const detachedNativeSessions = await detachNativeSessionsForRestart(nativeRunIds); @@ -1158,10 +1164,27 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD } } - const nativeRunsHandedOffEarly = new Set(); + const nativeRunsHandedOffEarly = new Map>(); + + async function settleEarlyNativeHandOff(runId: string, handOff: Promise) { + let timer: ReturnType | undefined; + try { + const outcome = await Promise.race([handOff, new Promise<"unfinished">(resolve => { + timer = setTimeout(() => resolve("unfinished"), EARLY_NATIVE_HAND_OFF_DRAIN_WAIT_MS); + timer.unref?.(); + })]); + if (outcome === "unfinished") { + logger.warn({ runId, waitedMs: EARLY_NATIVE_HAND_OFF_DRAIN_WAIT_MS }, + "early native run hand-off is unfinished at the shutdown drain; the drain hands it off"); + } + return outcome === true; + } finally { + clearTimeout(timer); + } + } async function suspendNativeRunForShutdown(run: typeof heartbeatRuns.$inferSelect, signal: "SIGINT" | "SIGTERM", - now: Date, controllerBootId: string) { + now: Date, controllerBootId: string, timing?: { detachMs: number; transactionMs: number }) { const [owner] = await db.select({ bootId: nativeRunFinalizations.controllerBootId }) .from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id)); if (owner?.bootId && owner.bootId !== controllerBootId) return false; @@ -1170,7 +1193,9 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD // finalizer interrupt/suspend Claude while the next server is adopting // the same turn. markHandedOffByThisController(run.id); + const detachStartedAt = performance.now(); await detachNativeSessionsForRestart([run.id]); + if (timing) timing.detachMs = Math.round(performance.now() - detachStartedAt); const recoveryHistoryEntry = JSON.stringify({ at: now.toISOString(), restartKind: "graceful", @@ -1183,6 +1208,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD // execution lease must still be released: the detached executor will // not finalize, and the next controller's reconciler skips leased rows. // A finalizer lease belongs to finalization already in progress here. + const transactionStartedAt = performance.now(); const released = await db.transaction(async tx => { await notifyLiveRunRecovery(tx, { companyId: run.companyId, runId: run.id }); await notifyReattachHandOver(tx, run.id); @@ -1217,6 +1243,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD ) .returning({ resultId: nativeRunFinalizations.resultId, leaseOwner: nativeRunFinalizations.leaseOwner }); }); + if (timing) timing.transactionMs = Math.round(performance.now() - transactionStartedAt); const resultAccepted = Boolean(released[0]?.resultId); const finalizerHeld = resultAccepted && Boolean(released[0]?.leaseOwner?.startsWith("native-finalizer:")); await appendRunEvent(run, { @@ -1243,25 +1270,40 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD } async function handOffNativeRunsForShutdown(signal: "SIGINT" | "SIGTERM", now = new Date()) { + const startedAt = performance.now(); + const elapsed = (from: number) => Math.round(performance.now() - from); const intent = await readHotRestartIntent().catch(() => null); + const intentReadMs = elapsed(startedAt); if (intent && !intent.drainRequired && shouldHonorHotRestartIntentForProcess(intent)) return []; + const identityStartedAt = performance.now(); const nativeController = await currentNativeControllerIdentity(); + const controllerIdentityMs = elapsed(identityStartedAt); + const selectStartedAt = performance.now(); const liveRuns = await db.select({ run: heartbeatRuns }).from(heartbeatRuns) .innerJoin(agents, eq(heartbeatRuns.agentId, agents.id)) .where(and(eq(heartbeatRuns.status, "running"), eq(heartbeatRuns.runtimeMode, "native"), eq(agents.adapterType, "paperclip_runner"))); + const selectMs = elapsed(selectStartedAt); const handedOff: string[] = []; + let maxDetachMs = 0, maxTransactionMs = 0, failed = 0; await Promise.allSettled(liveRuns.map(async ({ run }) => { if (isNativeRunnerOwnershipHeld(run) || nativeRunsHandedOffEarly.has(run.id)) return; - nativeRunsHandedOffEarly.add(run.id); - try { - if (await suspendNativeRunForShutdown(run, signal, now, nativeController.bootId)) handedOff.push(run.id); - else nativeRunsHandedOffEarly.delete(run.id); - } catch (error) { - nativeRunsHandedOffEarly.delete(run.id); + const timing = { detachMs: 0, transactionMs: 0 }; + const handOff = suspendNativeRunForShutdown(run, signal, now, nativeController.bootId, timing).catch(error => { + failed += 1; logger.warn({ err: error, runId: run.id }, "early native run hand-off failed; the shutdown drain retries it"); - } + return false; + }); + nativeRunsHandedOffEarly.set(run.id, handOff); + const suspended = await handOff; + maxDetachMs = Math.max(maxDetachMs, timing.detachMs); + maxTransactionMs = Math.max(maxTransactionMs, timing.transactionMs); + if (suspended) handedOff.push(run.id); + else if (nativeRunsHandedOffEarly.get(run.id) === handOff) nativeRunsHandedOffEarly.delete(run.id); })); + logger.info({ signal, liveRuns: liveRuns.length, handedOff: handedOff.length, failed, intentReadMs, + controllerIdentityMs, selectMs, maxDetachMs, maxTransactionMs, totalMs: elapsed(startedAt) }, + "native run shutdown hand-off phase timings"); return handedOff; } @@ -1311,8 +1353,12 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD run.runtimeMode === "native" && agent.adapterType === "paperclip_runner" ) { - if (nativeRunsHandedOffEarly.has(run.id) - || await suspendNativeRunForShutdown(run, signal, now, nativeController.bootId)) restartSuspendedRunIds.push(run.id); + const earlyHandOff = nativeRunsHandedOffEarly.get(run.id); + if (earlyHandOff && await settleEarlyNativeHandOff(run.id, earlyHandOff)) { + restartSuspendedRunIds.push(run.id); + return; + } + if (await suspendNativeRunForShutdown(run, signal, now, nativeController.bootId)) restartSuspendedRunIds.push(run.id); return; } if (agent.adapterType === "paperclip_runner" && run.runtimeMode === "legacy" && run.scopeKind === "issue" && diff --git a/server/src/shutdown.test.ts b/server/src/shutdown.test.ts index 728896f0535..43e2147c77a 100644 --- a/server/src/shutdown.test.ts +++ b/server/src/shutdown.test.ts @@ -410,6 +410,21 @@ describe("runBoundedShutdownStep", () => { } }); + it("aborts the signal of a step that outlives its bound, and only that step", async () => { + vi.useFakeTimers(); + try { + const signals: AbortSignal[] = []; + const budget = createPreDrainShutdownBudget({ signal: "SIGTERM", budgetMs: 250, log: stubLogger() }); + await budget.run("delivery_work_stop", async abort => { signals.push(abort); }); + const pending = budget.run("hot_restart_prepare", abort => { signals.push(abort); return new Promise(() => undefined); }); + await vi.advanceTimersByTimeAsync(250); + await expect(pending).resolves.toMatchObject({ outcome: "timed_out" }); + expect(signals.map(signal => signal.aborted)).toEqual([false, true]); + } finally { + vi.useRealTimers(); + } + }); + it("logs a step that rejects and does not reject the shutdown", async () => { const log = stubLogger(); const error = new Error("skill sync stop failed"); diff --git a/server/src/shutdown.ts b/server/src/shutdown.ts index 894a05f2e35..63f7f1e2905 100644 --- a/server/src/shutdown.ts +++ b/server/src/shutdown.ts @@ -23,18 +23,19 @@ export async function runBoundedShutdownStep(input: { step: string; signal: "SIGINT" | "SIGTERM"; timeoutMs: number; - run: () => Promise; + run: (abort: AbortSignal) => Promise; log: ShutdownLogger; now?: () => number; }): Promise> { const now = input.now ?? Date.now; const startedAt = now(); const timeoutMs = Math.max(0, input.timeoutMs); + const abort = new AbortController(); let timer: NodeJS.Timeout | null = null; try { const result = await Promise.race([ Promise.resolve() - .then(input.run) + .then(() => input.run(abort.signal)) .then( (value) => ({ outcome: "completed" as const, value }), (error: unknown) => ({ outcome: "failed" as const, error }), @@ -54,6 +55,7 @@ export async function runBoundedShutdownStep(input: { input.log.error({ ...fields, err: result.error }, "shutdown step failed; continuing shutdown"); return { outcome: "failed", error: result.error, durationMs }; } + abort.abort(new Error(`shutdown step ${input.step} timed out`)); input.log.info(fields, "shutdown step timed out; continuing shutdown without it"); return { outcome: "timed_out", durationMs }; } finally { @@ -70,7 +72,7 @@ export function createPreDrainShutdownBudget(input: { const now = input.now ?? Date.now; const deadline = now() + (input.budgetMs ?? PRE_DRAIN_SHUTDOWN_BUDGET_MS); return { - run(step: string, run: () => Promise) { + run(step: string, run: (abort: AbortSignal) => Promise) { return runBoundedShutdownStep({ step, signal: input.signal,