mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-18 23:08:26 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A server restart hands each live Runner run to the next server, so
the agent keeps working without a retry
> - At shutdown start, the outgoing server hands off native runs early,
in parallel with the rest of shutdown
> - The early hand-off put a run in the "handed off" set before its
suspend finished, and the drain counted every run in that set as
suspended without waiting
> - The early hand-off shares the 20 s pre-drain budget, and one Runner
detach can take 20-30 s, so the process could exit with the suspend
unfinished
> - Then the finalization lease stays live, the next server gets no
hand-over signal, and the run waits for lease expiry or the reaper
> - This pull request makes the drain wait for each in-flight early
hand-off, and fall back to its own hand-off when the early one failed or
did not finish
> - The benefit is that a run counts as handed over only after its
hand-over is committed, so the next server can always reach it at once
## Linked Issues or Issue Description
**What happened?**
A passing zero-downtime swap test showed a latent race in the shutdown
hand-off. `handOffNativeRunsForShutdown` added a run to
`nativeRunsHandedOffEarly` before `suspendNativeRunForShutdown`
finished. `drainRunningRunsForShutdown` then reported every run in that
set as `restartSuspended` without awaiting it. Because the early
hand-off shares the 20 s pre-drain budget and a Runner detach can wait
20-30 s, the process can exit before the lease release commits. Also, a
timed-out `prepareHotRestartShutdown` could still write the hot-restart
snapshot and detach sessions after the fallback drain started. In the
same test, swap 1's early hand-off took 2.08 s, and no log showed which
phase was slow.
**Expected behavior**
A run counts as handed over only after its suspend commits. If the early
suspend fails or does not finish, the drain hands the run off itself. A
timed-out hot-restart preparation does no further work after the drain
starts. The hand-off logs its phase timings.
**Steps to reproduce**
1. Start a native Runner run on a remote sandbox.
2. Send SIGTERM while the Runner detach is slow (or fails).
3. The drain reports the run as restart-suspended while its suspend is
still pending, and the process can exit before the lease release.
**Paperclip version or commit**
`master` at bbed32450c.
**Deployment mode**
Hosted, zero-downtime release swap.
## What Changed
- `nativeRunsHandedOffEarly` now holds each run's in-flight suspend
promise, not only its id.
- The shutdown drain awaits that promise (bounded at 30 s). If it fails,
returns false or does not finish, the drain runs its own suspend, as it
does for a run without an early hand-off.
- `runBoundedShutdownStep` and the pre-drain budget pass an
`AbortSignal` to each step and abort it on timeout.
- `prepareHotRestartShutdown` takes that signal and stops before the
snapshot write, the run events, and the session detach when the step
timed out.
- `handOffNativeRunsForShutdown` logs one structured line, "native run
shutdown hand-off phase timings": intent read, controller identity,
select, max per-run detach, max per-run transaction, and total.
## Verification
- Red, then green (each failed against `master` source and passes with
the fix), all through the public shutdown functions in
`server/src/services/heartbeat/recovery.test.ts`:
- `does not count an early hand-off as suspended while its detach is
still pending at the shutdown drain`
- `hands off a run at the shutdown drain when its pending early hand-off
fails`
- `writes no hot-restart snapshot and detaches nothing after a timed-out
preparation hands over to the drain`
- New test in `server/src/shutdown.test.ts`: `aborts the signal of a
step that outlives its bound, and only that step`.
- `pnpm vitest run src/shutdown.test.ts
src/services/heartbeat/recovery.test.ts
src/__tests__/heartbeat-native-early-hand-off.test.ts` passes (145
tests).
- `tsc --noEmit` for `server` passes. `pnpm verify` passes for the
selected tests.
## Risks
- When an early hand-off does not finish within 30 s, the drain starts a
second suspend. The second detach joins the first, because the session
memoizes its restart detach. The run can then get two "suspended" run
events. That is preferred to reporting a hand-over that did not happen.
- The drain can now wait up to 30 s for a slow early hand-off before the
process exits. Before, it exited without the hand-over.
- The abort signal only stops work between steps of the preparation. A
snapshot write that already started still completes.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
719 lines
25 KiB
TypeScript
719 lines
25 KiB
TypeScript
import { EventEmitter } from "node:events";
|
|
import { describe, expect, it, vi } from "vitest";
|
|
import {
|
|
closeHttpListenerForShutdown,
|
|
coordinateHeartbeatSchedulerShutdown,
|
|
createPreDrainShutdownBudget,
|
|
drainRunExecutionFinalizersForShutdown,
|
|
finalizeServerShutdown,
|
|
finishServiceShutdown,
|
|
loadWithoutCoordinatedShutdownSignalHooks,
|
|
runBoundedShutdownStep,
|
|
} from "./shutdown.js";
|
|
|
|
function deferred<T = void>() {
|
|
let resolve!: (value: T) => void;
|
|
let reject!: (reason?: unknown) => void;
|
|
const promise = new Promise<T>((res, rej) => {
|
|
resolve = res;
|
|
reject = rej;
|
|
});
|
|
return { promise, resolve, reject };
|
|
}
|
|
|
|
function stubLogger() {
|
|
return { info: vi.fn(), error: vi.fn() };
|
|
}
|
|
|
|
describe("finalizeServerShutdown", () => {
|
|
it("awaits the setup-token cleanup before the database stop and the process exit", async () => {
|
|
const order: string[] = [];
|
|
// The held promise models the setup-token session cancellation and its
|
|
// sandbox lease release. The teardown must not continue while it is pending.
|
|
const release = deferred();
|
|
const shutdownAppServices = vi.fn(async () => {
|
|
order.push("appServices:start");
|
|
await release.promise;
|
|
order.push("appServices:settled");
|
|
});
|
|
const closeHttpListener = vi.fn(async () => {
|
|
order.push("listener:close");
|
|
});
|
|
const closeDatabase = vi.fn(async () => {
|
|
order.push("database:close");
|
|
});
|
|
const stopEmbeddedPostgres = vi.fn(async () => {
|
|
order.push("postgres:stop");
|
|
});
|
|
const shutdownInstrumentation = vi.fn(async () => {
|
|
order.push("instrumentation:flush");
|
|
});
|
|
const shutdownSentry = vi.fn(async () => {
|
|
order.push("sentry:flush");
|
|
});
|
|
|
|
let exited = false;
|
|
const finalize = finalizeServerShutdown({
|
|
signal: "SIGTERM",
|
|
shutdownAppServices,
|
|
closeHttpListener,
|
|
closeDatabase,
|
|
stopEmbeddedPostgres,
|
|
shutdownInstrumentation,
|
|
shutdownSentry,
|
|
log: stubLogger(),
|
|
}).then(() => {
|
|
// This models the caller's `process.exit(0)` continuation.
|
|
exited = true;
|
|
order.push("exit");
|
|
});
|
|
|
|
// The cleanup is in flight. The database stop, the instrumentation flush,
|
|
// and the process exit continuation must all wait for it to settle.
|
|
await vi.waitFor(() => expect(shutdownAppServices).toHaveBeenCalledOnce());
|
|
// The listener already closed: requests are drained while every service
|
|
// is still available, and nothing after this point can be reached.
|
|
expect(closeHttpListener).toHaveBeenCalledOnce();
|
|
expect(closeDatabase).not.toHaveBeenCalled();
|
|
expect(stopEmbeddedPostgres).not.toHaveBeenCalled();
|
|
expect(shutdownInstrumentation).not.toHaveBeenCalled();
|
|
expect(exited).toBe(false);
|
|
|
|
release.resolve();
|
|
await finalize;
|
|
|
|
expect(exited).toBe(true);
|
|
expect(order).toEqual([
|
|
"listener:close",
|
|
"appServices:start",
|
|
"appServices:settled",
|
|
"database:close",
|
|
"postgres:stop",
|
|
"instrumentation:flush",
|
|
"sentry:flush",
|
|
"exit",
|
|
]);
|
|
});
|
|
|
|
it("keeps the teardown durable and still exits when the setup-token release fails", async () => {
|
|
const order: string[] = [];
|
|
// The held promise rejects, which models a lease release that failed. The
|
|
// reaper owns the durable retry, so the teardown must log the failure and
|
|
// continue rather than swallow it or block the exit.
|
|
const release = deferred();
|
|
const releaseError = new Error("lease release failed");
|
|
const shutdownAppServices = vi.fn(async () => {
|
|
await release.promise;
|
|
});
|
|
const stopEmbeddedPostgres = vi.fn(async () => {
|
|
order.push("postgres:stop");
|
|
});
|
|
const shutdownInstrumentation = vi.fn(async () => {
|
|
order.push("instrumentation:flush");
|
|
});
|
|
const shutdownSentry = vi.fn(async () => undefined);
|
|
const log = stubLogger();
|
|
|
|
let exited = false;
|
|
const finalize = finalizeServerShutdown({
|
|
signal: "SIGTERM",
|
|
shutdownAppServices,
|
|
stopEmbeddedPostgres,
|
|
shutdownInstrumentation,
|
|
shutdownSentry,
|
|
log,
|
|
}).then(() => {
|
|
exited = true;
|
|
});
|
|
|
|
await vi.waitFor(() => expect(shutdownAppServices).toHaveBeenCalledOnce());
|
|
expect(stopEmbeddedPostgres).not.toHaveBeenCalled();
|
|
expect(exited).toBe(false);
|
|
|
|
release.reject(releaseError);
|
|
await finalize;
|
|
|
|
// The teardown surfaced the failure in the log, then finished the ordered
|
|
// teardown and reached the exit continuation.
|
|
expect(log.error).toHaveBeenCalledWith(
|
|
expect.objectContaining({ err: releaseError, signal: "SIGTERM" }),
|
|
expect.any(String),
|
|
);
|
|
expect(order).toEqual(["postgres:stop", "instrumentation:flush"]);
|
|
expect(exited).toBe(true);
|
|
});
|
|
|
|
it("logs a failed database close and still stops the provider and exits", async () => {
|
|
const order: string[] = [];
|
|
const closeError = new Error("pool end timed out");
|
|
const closeDatabase = vi.fn(async () => {
|
|
order.push("database:close");
|
|
throw closeError;
|
|
});
|
|
const stopEmbeddedPostgres = vi.fn(async () => {
|
|
order.push("postgres:stop");
|
|
});
|
|
const log = stubLogger();
|
|
|
|
await finalizeServerShutdown({
|
|
signal: "SIGTERM",
|
|
shutdownAppServices: vi.fn(async () => undefined),
|
|
closeDatabase,
|
|
stopEmbeddedPostgres,
|
|
shutdownInstrumentation: vi.fn(async () => undefined),
|
|
shutdownSentry: vi.fn(async () => undefined),
|
|
log,
|
|
});
|
|
|
|
expect(order).toEqual(["database:close", "postgres:stop"]);
|
|
expect(log.error).toHaveBeenCalledWith(
|
|
expect.objectContaining({ err: closeError, signal: "SIGTERM" }),
|
|
"Database client shutdown failed",
|
|
);
|
|
});
|
|
|
|
it("waits for pending run-failure reports before the database stop and the Sentry flush", async () => {
|
|
const order: string[] = [];
|
|
const release = deferred();
|
|
const drainPendingRunFailureReports = vi.fn(async () => {
|
|
order.push("runFailureReports:start");
|
|
await release.promise;
|
|
order.push("runFailureReports:settled");
|
|
});
|
|
const closeDatabase = vi.fn(async () => {
|
|
order.push("database:close");
|
|
});
|
|
const shutdownSentry = vi.fn(async () => {
|
|
order.push("sentry:flush");
|
|
});
|
|
|
|
const finalize = finalizeServerShutdown({
|
|
signal: "SIGTERM",
|
|
shutdownAppServices: vi.fn(async () => undefined),
|
|
drainPendingRunFailureReports,
|
|
closeDatabase,
|
|
stopEmbeddedPostgres: null,
|
|
shutdownInstrumentation: vi.fn(async () => undefined),
|
|
shutdownSentry,
|
|
log: stubLogger(),
|
|
});
|
|
|
|
await vi.waitFor(() => expect(drainPendingRunFailureReports).toHaveBeenCalledOnce());
|
|
expect(closeDatabase).not.toHaveBeenCalled();
|
|
expect(shutdownSentry).not.toHaveBeenCalled();
|
|
|
|
release.resolve();
|
|
await finalize;
|
|
|
|
expect(order).toEqual([
|
|
"runFailureReports:start",
|
|
"runFailureReports:settled",
|
|
"database:close",
|
|
"sentry:flush",
|
|
]);
|
|
});
|
|
|
|
it("logs a failed run-failure report drain and still stops the database and exits", async () => {
|
|
const drainError = new Error("drain failed");
|
|
const drainPendingRunFailureReports = vi.fn(async () => {
|
|
throw drainError;
|
|
});
|
|
const closeDatabase = vi.fn(async () => undefined);
|
|
const log = stubLogger();
|
|
|
|
await finalizeServerShutdown({
|
|
signal: "SIGTERM",
|
|
shutdownAppServices: vi.fn(async () => undefined),
|
|
drainPendingRunFailureReports,
|
|
closeDatabase,
|
|
stopEmbeddedPostgres: null,
|
|
shutdownInstrumentation: vi.fn(async () => undefined),
|
|
shutdownSentry: vi.fn(async () => undefined),
|
|
log,
|
|
});
|
|
|
|
expect(closeDatabase).toHaveBeenCalledOnce();
|
|
expect(log.error).toHaveBeenCalledWith(
|
|
expect.objectContaining({ err: drainError, signal: "SIGTERM" }),
|
|
"run-failure report drain failed",
|
|
);
|
|
});
|
|
|
|
it("skips the database stop when no embedded PostgreSQL runs in this process", async () => {
|
|
const shutdownAppServices = vi.fn(async () => undefined);
|
|
const shutdownInstrumentation = vi.fn(async () => undefined);
|
|
const shutdownSentry = vi.fn(async () => undefined);
|
|
const log = stubLogger();
|
|
|
|
await finalizeServerShutdown({
|
|
signal: "SIGINT",
|
|
shutdownAppServices,
|
|
stopEmbeddedPostgres: null,
|
|
shutdownInstrumentation,
|
|
shutdownSentry,
|
|
log,
|
|
});
|
|
|
|
expect(shutdownAppServices).toHaveBeenCalledOnce();
|
|
expect(shutdownInstrumentation).toHaveBeenCalledOnce();
|
|
expect(log.info).not.toHaveBeenCalled();
|
|
});
|
|
});
|
|
|
|
describe("closeHttpListenerForShutdown", () => {
|
|
function fakeServer(input: { listening: boolean; closeDelayMs?: number | null }) {
|
|
const closeIdleConnections = vi.fn();
|
|
const closeAllConnections = vi.fn();
|
|
const close = vi.fn((callback?: (err?: Error) => void) => {
|
|
if (input.closeDelayMs === null) return;
|
|
setTimeout(() => callback?.(), input.closeDelayMs ?? 0);
|
|
});
|
|
return { listening: input.listening, close, closeIdleConnections, closeAllConnections };
|
|
}
|
|
|
|
it("stops accepting, closes idle keep-alive sockets, and resolves once the listener closed", async () => {
|
|
const server = fakeServer({ listening: true, closeDelayMs: 0 });
|
|
await expect(
|
|
closeHttpListenerForShutdown({ server, signal: "SIGTERM", timeoutMs: 1_000, log: stubLogger() }),
|
|
).resolves.toBe("closed");
|
|
expect(server.close).toHaveBeenCalledOnce();
|
|
expect(server.closeIdleConnections).toHaveBeenCalledOnce();
|
|
expect(server.closeAllConnections).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("closes run admission before it stops accepting requests", async () => {
|
|
const order: string[] = [];
|
|
const server = fakeServer({ listening: true, closeDelayMs: 0 });
|
|
server.close.mockImplementation((callback?: (err?: Error) => void) => { order.push("listener"); setTimeout(() => callback?.(), 0); });
|
|
await closeHttpListenerForShutdown({ server, signal: "SIGTERM", timeoutMs: 1_000, log: stubLogger(),
|
|
closeRunAdmission: () => order.push("run_admission") });
|
|
expect(order).toEqual(["run_admission", "listener"]);
|
|
const stopped = vi.fn();
|
|
await expect(closeHttpListenerForShutdown({ server: fakeServer({ listening: false }), signal: "SIGTERM", log: stubLogger(),
|
|
closeRunAdmission: stopped })).resolves.toBe("not_listening");
|
|
expect(stopped).toHaveBeenCalledOnce();
|
|
});
|
|
|
|
it("closes the remaining connections when the drain outlives the grace period", async () => {
|
|
vi.useFakeTimers();
|
|
try {
|
|
const server = fakeServer({ listening: true, closeDelayMs: null });
|
|
const log = stubLogger();
|
|
const pending = closeHttpListenerForShutdown({ server, signal: "SIGINT", timeoutMs: 250, log });
|
|
await vi.advanceTimersByTimeAsync(250);
|
|
await expect(pending).resolves.toBe("timed_out");
|
|
expect(server.closeAllConnections).toHaveBeenCalledOnce();
|
|
expect(log.info).toHaveBeenCalledWith(
|
|
expect.objectContaining({ timeoutMs: 250 }),
|
|
expect.stringContaining("timed out"),
|
|
);
|
|
} finally {
|
|
vi.useRealTimers();
|
|
}
|
|
});
|
|
|
|
it("waits for HTTP work without waiting for upgraded sockets to close", async () => {
|
|
const server = fakeServer({ listening: true, closeDelayMs: null });
|
|
const requests = deferred();
|
|
let finished = false;
|
|
const drain = closeHttpListenerForShutdown({ server, signal: "SIGTERM", log: stubLogger(),
|
|
drainRequests: () => requests.promise }).then(result => { finished = true; return result; });
|
|
await Promise.resolve();
|
|
expect(finished).toBe(false);
|
|
requests.resolve();
|
|
await expect(drain).resolves.toBe("closed");
|
|
expect(server.close).toHaveBeenCalledOnce();
|
|
expect(server.closeAllConnections).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("does nothing when the listener was never bound", async () => {
|
|
const server = fakeServer({ listening: false });
|
|
await expect(
|
|
closeHttpListenerForShutdown({ server, signal: "SIGTERM", log: stubLogger() }),
|
|
).resolves.toBe("not_listening");
|
|
expect(server.close).not.toHaveBeenCalled();
|
|
});
|
|
});
|
|
|
|
describe("drainRunExecutionFinalizersForShutdown", () => {
|
|
it("awaits bounded execution finalizers", async () => {
|
|
const release = deferred();
|
|
const drain = vi.fn(() => release.promise);
|
|
const pending = drainRunExecutionFinalizersForShutdown({
|
|
signal: "SIGTERM",
|
|
drain,
|
|
timeoutMs: 1_000,
|
|
log: stubLogger(),
|
|
});
|
|
await vi.waitFor(() => expect(drain).toHaveBeenCalledOnce());
|
|
release.resolve();
|
|
await expect(pending).resolves.toBe("drained");
|
|
});
|
|
|
|
it("returns after the bounded timeout when an adopted run remains active", async () => {
|
|
vi.useFakeTimers();
|
|
try {
|
|
const log = stubLogger();
|
|
const pending = drainRunExecutionFinalizersForShutdown({
|
|
signal: "SIGINT",
|
|
drain: () => new Promise<void>(() => undefined),
|
|
timeoutMs: 250,
|
|
log,
|
|
});
|
|
await vi.advanceTimersByTimeAsync(250);
|
|
await expect(pending).resolves.toBe("timed_out");
|
|
expect(log.info).toHaveBeenCalledWith(
|
|
expect.objectContaining({ timeoutMs: 250 }),
|
|
expect.stringContaining("timed out"),
|
|
);
|
|
} finally {
|
|
vi.useRealTimers();
|
|
}
|
|
});
|
|
});
|
|
|
|
describe("runBoundedShutdownStep", () => {
|
|
it("returns the step value and logs its duration", async () => {
|
|
const log = stubLogger();
|
|
await expect(runBoundedShutdownStep({
|
|
step: "delivery_work_stop",
|
|
signal: "SIGTERM",
|
|
timeoutMs: 1_000,
|
|
run: async () => "stopped",
|
|
log,
|
|
})).resolves.toMatchObject({ outcome: "completed", value: "stopped" });
|
|
expect(log.info).toHaveBeenCalledWith(
|
|
expect.objectContaining({ step: "delivery_work_stop", signal: "SIGTERM", durationMs: expect.any(Number) }),
|
|
"shutdown step completed",
|
|
);
|
|
});
|
|
|
|
it("stops waiting for a step that outlives its bound", async () => {
|
|
vi.useFakeTimers();
|
|
try {
|
|
const log = stubLogger();
|
|
const pending = runBoundedShutdownStep({
|
|
step: "scheduler_idle",
|
|
signal: "SIGTERM",
|
|
timeoutMs: 250,
|
|
run: () => new Promise<void>(() => undefined),
|
|
log,
|
|
});
|
|
await vi.advanceTimersByTimeAsync(250);
|
|
await expect(pending).resolves.toMatchObject({ outcome: "timed_out" });
|
|
expect(log.info).toHaveBeenCalledWith(
|
|
expect.objectContaining({ step: "scheduler_idle", timeoutMs: 250 }),
|
|
expect.stringContaining("timed out"),
|
|
);
|
|
} finally {
|
|
vi.useRealTimers();
|
|
}
|
|
});
|
|
|
|
it("aborts the signal of a step that outlives its bound, and only that step", async () => {
|
|
vi.useFakeTimers();
|
|
try {
|
|
const signals: AbortSignal[] = [];
|
|
const budget = createPreDrainShutdownBudget({ signal: "SIGTERM", budgetMs: 250, log: stubLogger() });
|
|
await budget.run("delivery_work_stop", async abort => { signals.push(abort); });
|
|
const pending = budget.run("hot_restart_prepare", abort => { signals.push(abort); return new Promise<void>(() => undefined); });
|
|
await vi.advanceTimersByTimeAsync(250);
|
|
await expect(pending).resolves.toMatchObject({ outcome: "timed_out" });
|
|
expect(signals.map(signal => signal.aborted)).toEqual([false, true]);
|
|
} finally {
|
|
vi.useRealTimers();
|
|
}
|
|
});
|
|
|
|
it("logs a step that rejects and does not reject the shutdown", async () => {
|
|
const log = stubLogger();
|
|
const error = new Error("skill sync stop failed");
|
|
await expect(runBoundedShutdownStep({
|
|
step: "skill_source_sync_stop",
|
|
signal: "SIGINT",
|
|
timeoutMs: 1_000,
|
|
run: async () => { throw error; },
|
|
log,
|
|
})).resolves.toMatchObject({ outcome: "failed", error });
|
|
expect(log.error).toHaveBeenCalledWith(
|
|
expect.objectContaining({ step: "skill_source_sync_stop", err: error }),
|
|
expect.stringContaining("failed"),
|
|
);
|
|
});
|
|
|
|
it("contains a step that throws synchronously", async () => {
|
|
const result = await runBoundedShutdownStep({
|
|
step: "systemd_notify",
|
|
signal: "SIGTERM",
|
|
timeoutMs: 1_000,
|
|
run: () => { throw new Error("spawn failed"); },
|
|
log: stubLogger(),
|
|
});
|
|
expect(result.outcome).toBe("failed");
|
|
});
|
|
});
|
|
|
|
describe("createPreDrainShutdownBudget", () => {
|
|
it("gives each step only the time that is left in the shared budget", async () => {
|
|
vi.useFakeTimers();
|
|
try {
|
|
const log = stubLogger();
|
|
const budget = createPreDrainShutdownBudget({ signal: "SIGTERM", budgetMs: 1_000, log });
|
|
const first = budget.run("skill_source_sync_stop", () =>
|
|
new Promise<void>((resolve) => setTimeout(resolve, 700)));
|
|
await vi.advanceTimersByTimeAsync(700);
|
|
await expect(first).resolves.toMatchObject({ outcome: "completed" });
|
|
|
|
const second = budget.run("scheduler_idle", () => new Promise<void>(() => undefined));
|
|
await vi.advanceTimersByTimeAsync(300);
|
|
await expect(second).resolves.toMatchObject({ outcome: "timed_out" });
|
|
expect(log.info).toHaveBeenCalledWith(
|
|
expect.objectContaining({ step: "scheduler_idle", timeoutMs: 300 }),
|
|
expect.stringContaining("timed out"),
|
|
);
|
|
} finally {
|
|
vi.useRealTimers();
|
|
}
|
|
});
|
|
|
|
it("reaches the run drain within the budget when every pre-drain step hangs", async () => {
|
|
vi.useFakeTimers();
|
|
try {
|
|
const budget = createPreDrainShutdownBudget({ signal: "SIGTERM", budgetMs: 1_000, log: stubLogger() });
|
|
const hang = () => new Promise<void>(() => undefined);
|
|
const drain = vi.fn(async () => ({ interrupted: 1 }));
|
|
const shutdown = (async () => {
|
|
await budget.run("skill_source_sync_stop", hang);
|
|
await budget.run("delivery_work_stop", hang);
|
|
await coordinateHeartbeatSchedulerShutdown({
|
|
signal: "SIGTERM",
|
|
prepareHotRestartShutdown: async () => {
|
|
const preparation = await budget.run("hot_restart_prepare", () =>
|
|
new Promise<{ skipDrain: boolean }>(() => undefined));
|
|
if (preparation.outcome === "completed") return preparation.value;
|
|
throw new Error("hot-restart shutdown preparation timed out");
|
|
},
|
|
waitForHeartbeatSchedulerIdle: async () => {
|
|
await budget.run("scheduler_idle", hang);
|
|
},
|
|
});
|
|
return drain();
|
|
})();
|
|
|
|
const startedAt = Date.now();
|
|
await vi.advanceTimersByTimeAsync(1_000);
|
|
await vi.runAllTimersAsync();
|
|
await expect(shutdown).resolves.toEqual({ interrupted: 1 });
|
|
expect(drain).toHaveBeenCalledOnce();
|
|
expect(Date.now() - startedAt).toBeLessThan(1_010);
|
|
} finally {
|
|
vi.useRealTimers();
|
|
}
|
|
});
|
|
});
|
|
|
|
describe("loadWithoutCoordinatedShutdownSignalHooks", () => {
|
|
it("removes the eager signal handlers from the real embedded-postgres import", async () => {
|
|
const before = {
|
|
SIGINT: process.rawListeners("SIGINT"),
|
|
SIGTERM: process.rawListeners("SIGTERM"),
|
|
};
|
|
const moduleName = "embedded-postgres";
|
|
|
|
await loadWithoutCoordinatedShutdownSignalHooks(() => import(moduleName));
|
|
|
|
expect(process.rawListeners("SIGINT")).toEqual(before.SIGINT);
|
|
expect(process.rawListeners("SIGTERM")).toEqual(before.SIGTERM);
|
|
});
|
|
|
|
it("keeps the database available for a marker-backed SIGTERM snapshot", async () => {
|
|
const signalTarget = new EventEmitter();
|
|
const preexistingSignalListener = vi.fn();
|
|
signalTarget.on("SIGTERM", preexistingSignalListener);
|
|
|
|
let databaseAvailable = true;
|
|
const embeddedPostgresExitHook = vi.fn(() => {
|
|
databaseAvailable = false;
|
|
});
|
|
await loadWithoutCoordinatedShutdownSignalHooks(
|
|
async () => {
|
|
signalTarget.on("SIGINT", embeddedPostgresExitHook);
|
|
signalTarget.on("SIGTERM", embeddedPostgresExitHook);
|
|
return { default: class EmbeddedPostgres {} };
|
|
},
|
|
signalTarget,
|
|
);
|
|
|
|
let shutdown: Promise<unknown> | null = null;
|
|
let snapshotCaptured = false;
|
|
signalTarget.once("SIGTERM", () => {
|
|
shutdown = coordinateHeartbeatSchedulerShutdown({
|
|
signal: "SIGTERM",
|
|
prepareHotRestartShutdown: async () => {
|
|
// This models the real failure path: a valid intent exists, and the
|
|
// snapshot must query embedded PostgreSQL after SIGTERM is delivered.
|
|
expect(databaseAvailable).toBe(true);
|
|
snapshotCaptured = true;
|
|
return { mode: "hot_restart" as const, skipDrain: true };
|
|
},
|
|
waitForHeartbeatSchedulerIdle: vi.fn(async () => undefined),
|
|
});
|
|
});
|
|
|
|
signalTarget.emit("SIGTERM");
|
|
await shutdown;
|
|
|
|
expect(preexistingSignalListener).toHaveBeenCalledOnce();
|
|
expect(embeddedPostgresExitHook).not.toHaveBeenCalled();
|
|
expect(snapshotCaptured).toBe(true);
|
|
});
|
|
});
|
|
|
|
describe("coordinateHeartbeatSchedulerShutdown", () => {
|
|
it("quiesces active scheduler work before capturing a hot-restart snapshot", async () => {
|
|
let snapshotCaptured = false;
|
|
let releaseScheduler!: () => void;
|
|
const schedulerIdle = new Promise<void>((resolve) => {
|
|
releaseScheduler = resolve;
|
|
});
|
|
const waitForHeartbeatSchedulerIdle = vi.fn(() => schedulerIdle);
|
|
|
|
const shutdown = coordinateHeartbeatSchedulerShutdown({
|
|
signal: "SIGTERM",
|
|
prepareHotRestartShutdown: vi.fn(async () => {
|
|
snapshotCaptured = true;
|
|
return { mode: "prepared" as const, skipDrain: true };
|
|
}),
|
|
waitForHeartbeatSchedulerIdle,
|
|
});
|
|
|
|
await vi.waitFor(() => expect(waitForHeartbeatSchedulerIdle).toHaveBeenCalledOnce());
|
|
expect(snapshotCaptured).toBe(false);
|
|
releaseScheduler();
|
|
|
|
const result = await shutdown;
|
|
expect(snapshotCaptured).toBe(true);
|
|
expect(result).toEqual({
|
|
hotRestart: { mode: "prepared", skipDrain: true },
|
|
preparationError: null,
|
|
waitedForSchedulerIdle: true,
|
|
});
|
|
});
|
|
|
|
it("quiesces scheduler work before selecting server-stdio runs to drain", async () => {
|
|
const waitForHeartbeatSchedulerIdle = vi.fn(async () => undefined);
|
|
|
|
const result = await coordinateHeartbeatSchedulerShutdown({
|
|
signal: "SIGTERM",
|
|
prepareHotRestartShutdown: vi.fn(async () => ({
|
|
mode: "acp_drain_required" as const,
|
|
skipDrain: false,
|
|
drainRunIds: ["acp-run"],
|
|
})),
|
|
waitForHeartbeatSchedulerIdle,
|
|
});
|
|
|
|
expect(waitForHeartbeatSchedulerIdle).toHaveBeenCalledOnce();
|
|
expect(result).toEqual({
|
|
hotRestart: {
|
|
mode: "acp_drain_required",
|
|
skipDrain: false,
|
|
drainRunIds: ["acp-run"],
|
|
},
|
|
preparationError: null,
|
|
waitedForSchedulerIdle: true,
|
|
});
|
|
});
|
|
|
|
it("preserves the scheduler idle wait for normal graceful shutdown", async () => {
|
|
let releaseScheduler!: () => void;
|
|
const schedulerIdle = new Promise<void>((resolve) => {
|
|
releaseScheduler = resolve;
|
|
});
|
|
const waitForHeartbeatSchedulerIdle = vi.fn(() => schedulerIdle);
|
|
let settled = false;
|
|
|
|
const shutdown = coordinateHeartbeatSchedulerShutdown({
|
|
signal: "SIGTERM",
|
|
prepareHotRestartShutdown: vi.fn(async () => ({
|
|
mode: "not_requested" as const,
|
|
skipDrain: false,
|
|
})),
|
|
waitForHeartbeatSchedulerIdle,
|
|
}).finally(() => {
|
|
settled = true;
|
|
});
|
|
|
|
await vi.waitFor(() => expect(waitForHeartbeatSchedulerIdle).toHaveBeenCalledOnce());
|
|
expect(settled).toBe(false);
|
|
|
|
releaseScheduler();
|
|
|
|
await expect(shutdown).resolves.toEqual({
|
|
hotRestart: { mode: "not_requested", skipDrain: false },
|
|
preparationError: null,
|
|
waitedForSchedulerIdle: true,
|
|
});
|
|
});
|
|
|
|
it("waits for scheduler idle when hot-restart preparation is unavailable", async () => {
|
|
const waitForHeartbeatSchedulerIdle = vi.fn(async () => undefined);
|
|
|
|
const result = await coordinateHeartbeatSchedulerShutdown({
|
|
signal: "SIGTERM",
|
|
prepareHotRestartShutdown: null,
|
|
waitForHeartbeatSchedulerIdle,
|
|
});
|
|
|
|
expect(waitForHeartbeatSchedulerIdle).toHaveBeenCalledOnce();
|
|
expect(result).toEqual({
|
|
hotRestart: null,
|
|
preparationError: null,
|
|
waitedForSchedulerIdle: true,
|
|
});
|
|
});
|
|
|
|
it("falls back to the scheduler idle wait when hot-restart preparation fails", async () => {
|
|
const preparationError = new Error("snapshot failed");
|
|
const waitForHeartbeatSchedulerIdle = vi.fn(async () => undefined);
|
|
|
|
const result = await coordinateHeartbeatSchedulerShutdown({
|
|
signal: "SIGTERM",
|
|
prepareHotRestartShutdown: vi.fn(async () => {
|
|
throw preparationError;
|
|
}),
|
|
waitForHeartbeatSchedulerIdle,
|
|
});
|
|
|
|
expect(waitForHeartbeatSchedulerIdle).toHaveBeenCalledOnce();
|
|
expect(result).toEqual({
|
|
hotRestart: null,
|
|
preparationError,
|
|
waitedForSchedulerIdle: true,
|
|
});
|
|
});
|
|
});
|
|
|
|
describe("service shutdown failure isolation", () => {
|
|
it("awaits remaining cleanup before reporting a scheduler failure", async () => {
|
|
const failure = new Error("plugin result could not be saved");
|
|
const gate = deferred();
|
|
const cleanup = vi.fn(() => gate.promise);
|
|
let finished = false;
|
|
const stopping = finishServiceShutdown(async () => { throw failure; }, cleanup)
|
|
.then(() => { finished = true; return undefined; }, error => { finished = true; return error; });
|
|
try {
|
|
await vi.waitFor(() => expect(cleanup).toHaveBeenCalledOnce());
|
|
expect(finished).toBe(false);
|
|
} finally { gate.resolve(); }
|
|
expect(await stopping).toBe(failure);
|
|
});
|
|
it("reports both failures after attempting cleanup", async () => {
|
|
const stopFailure = new Error("scheduler failure"), cleanupFailure = new Error("worker failure");
|
|
const cleanup = vi.fn(async () => { throw cleanupFailure; });
|
|
await expect(finishServiceShutdown(async () => { throw stopFailure; }, cleanup))
|
|
.rejects.toMatchObject({ errors: [stopFailure, cleanupFailure] });
|
|
expect(cleanup).toHaveBeenCalledOnce();
|
|
});
|
|
});
|