mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
fix(heartbeat): count an early native hand-off only after its suspend commits (#21572)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A server restart hands each live Runner run to the next server, so
the agent keeps working without a retry
> - At shutdown start, the outgoing server hands off native runs early,
in parallel with the rest of shutdown
> - The early hand-off put a run in the "handed off" set before its
suspend finished, and the drain counted every run in that set as
suspended without waiting
> - The early hand-off shares the 20 s pre-drain budget, and one Runner
detach can take 20-30 s, so the process could exit with the suspend
unfinished
> - Then the finalization lease stays live, the next server gets no
hand-over signal, and the run waits for lease expiry or the reaper
> - This pull request makes the drain wait for each in-flight early
hand-off, and fall back to its own hand-off when the early one failed or
did not finish
> - The benefit is that a run counts as handed over only after its
hand-over is committed, so the next server can always reach it at once
## Linked Issues or Issue Description
**What happened?**
A passing zero-downtime swap test showed a latent race in the shutdown
hand-off. `handOffNativeRunsForShutdown` added a run to
`nativeRunsHandedOffEarly` before `suspendNativeRunForShutdown`
finished. `drainRunningRunsForShutdown` then reported every run in that
set as `restartSuspended` without awaiting it. Because the early
hand-off shares the 20 s pre-drain budget and a Runner detach can wait
20-30 s, the process can exit before the lease release commits. Also, a
timed-out `prepareHotRestartShutdown` could still write the hot-restart
snapshot and detach sessions after the fallback drain started. In the
same test, swap 1's early hand-off took 2.08 s, and no log showed which
phase was slow.
**Expected behavior**
A run counts as handed over only after its suspend commits. If the early
suspend fails or does not finish, the drain hands the run off itself. A
timed-out hot-restart preparation does no further work after the drain
starts. The hand-off logs its phase timings.
**Steps to reproduce**
1. Start a native Runner run on a remote sandbox.
2. Send SIGTERM while the Runner detach is slow (or fails).
3. The drain reports the run as restart-suspended while its suspend is
still pending, and the process can exit before the lease release.
**Paperclip version or commit**
`master` at bbed32450c.
**Deployment mode**
Hosted, zero-downtime release swap.
## What Changed
- `nativeRunsHandedOffEarly` now holds each run's in-flight suspend
promise, not only its id.
- The shutdown drain awaits that promise (bounded at 30 s). If it fails,
returns false or does not finish, the drain runs its own suspend, as it
does for a run without an early hand-off.
- `runBoundedShutdownStep` and the pre-drain budget pass an
`AbortSignal` to each step and abort it on timeout.
- `prepareHotRestartShutdown` takes that signal and stops before the
snapshot write, the run events, and the session detach when the step
timed out.
- `handOffNativeRunsForShutdown` logs one structured line, "native run
shutdown hand-off phase timings": intent read, controller identity,
select, max per-run detach, max per-run transaction, and total.
## Verification
- Red, then green (each failed against `master` source and passes with
the fix), all through the public shutdown functions in
`server/src/services/heartbeat/recovery.test.ts`:
- `does not count an early hand-off as suspended while its detach is
still pending at the shutdown drain`
- `hands off a run at the shutdown drain when its pending early hand-off
fails`
- `writes no hot-restart snapshot and detaches nothing after a timed-out
preparation hands over to the drain`
- New test in `server/src/shutdown.test.ts`: `aborts the signal of a
step that outlives its bound, and only that step`.
- `pnpm vitest run src/shutdown.test.ts
src/services/heartbeat/recovery.test.ts
src/__tests__/heartbeat-native-early-hand-off.test.ts` passes (145
tests).
- `tsc --noEmit` for `server` passes. `pnpm verify` passes for the
selected tests.
## Risks
- When an early hand-off does not finish within 30 s, the drain starts a
second suspend. The second detach joins the first, because the session
memoizes its restart detach. The run can then get two "suspended" run
events. That is preferred to reporting a hand-over that did not happen.
- The drain can now wait up to 30 s for a slow early hand-off before the
process exits. Before, it exited without the hand-over.
- The abort signal only stops work between steps of the preparation. A
snapshot write that already started still completes.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
633dec7e43
commit
cf09024292
5 files changed
+200
-18
No files matched your search
+3
-3
@@ -1303,7 +1303,7 @@ async function startServerWithDatabaseTeardown(
|
||||
) => Promise<unknown>) | null = null;
|
||||
let drainHeartbeatExecutionFinalizers: (() => Promise<void>) | null = null;
|
||||
let handOffNativeRunsForShutdown: ((signal: "SIGINT" | "SIGTERM") => Promise<string[]>) | null = null;
|
||||
let prepareHotRestartShutdown: ((signal: "SIGINT" | "SIGTERM") => Promise<{
|
||||
let prepareHotRestartShutdown: ((signal: "SIGINT" | "SIGTERM", now?: Date, abort?: AbortSignal) => Promise<{
|
||||
skipDrain: boolean;
|
||||
drainRunIds?: string[];
|
||||
}>) | null = null;
|
||||
@@ -2281,8 +2281,8 @@ async function startServerWithDatabaseTeardown(
|
||||
signal,
|
||||
prepareHotRestartShutdown: prepareHotRestart
|
||||
? async (preparationSignal) => {
|
||||
const preparation = await preDrain.run("hot_restart_prepare", () =>
|
||||
prepareHotRestart(preparationSignal));
|
||||
const preparation = await preDrain.run("hot_restart_prepare", abort =>
|
||||
prepareHotRestart(preparationSignal, undefined, abort));
|
||||
if (preparation.outcome === "completed") return preparation.value;
|
||||
if (preparation.outcome === "failed") throw preparation.error;
|
||||
throw new Error("hot-restart shutdown preparation timed out");
|
||||
|
||||
@@ -31,6 +31,7 @@ import { persistHeartbeatRunProcessMetadata } from "./run-lifecycle.js";
|
||||
import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
|
||||
import { hasNativeLocalProcessStop, PROCESS_START_REQUESTED, releaseSettledNativeRunProcess } from "../native-local-process-stop.js";
|
||||
import { legacyControllerBootId } from "../legacy-controller-lease.js";
|
||||
import { runBoundedShutdownStep } from "../../shutdown.js";
|
||||
|
||||
type Run = typeof heartbeatRuns.$inferSelect;
|
||||
|
||||
@@ -44,10 +45,12 @@ const native = vi.hoisted(() => ({
|
||||
}));
|
||||
const hotRestart = vi.hoisted(() => ({
|
||||
read: vi.fn<() => Promise<HotRestartIntent | null>>(async () => null),
|
||||
write: vi.fn(async () => undefined),
|
||||
}));
|
||||
vi.mock("../hot-restart.js", async (importOriginal) => ({
|
||||
...await importOriginal<typeof import("../hot-restart.js")>(),
|
||||
readHotRestartIntent: hotRestart.read,
|
||||
writeHotRestartShutdownSnapshot: hotRestart.write,
|
||||
}));
|
||||
vi.mock("../native-runtime/index.js", async (importOriginal) => ({
|
||||
...await importOriginal<typeof import("../native-runtime/index.js")>(),
|
||||
@@ -1500,6 +1503,122 @@ describePostgres("heartbeat recovery module ownership", () => {
|
||||
}
|
||||
});
|
||||
|
||||
async function seedOwnedNativeRun() {
|
||||
const run = await seedRun();
|
||||
const identity = await currentNativeControllerIdentity();
|
||||
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, run.agentId));
|
||||
const [nativeRun] = await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: run.issueId, nativeSessionId: randomUUID() })
|
||||
.where(eq(heartbeatRuns.id, run.id)).returning();
|
||||
await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!,
|
||||
phase: "observed", controllerBootId: identity.bootId, leaseOwner: "departing-server", leaseExpiresAt: new Date(Date.now() + 60_000) });
|
||||
return nativeRun;
|
||||
}
|
||||
async function removeNativeRun(runId: string) {
|
||||
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
|
||||
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
}
|
||||
const finalization = async (runId: string) =>
|
||||
(await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)))[0];
|
||||
const drainHasStarted = () => new Promise(resolve => setTimeout(resolve, 250));
|
||||
|
||||
it("does not count an early hand-off as suspended while its detach is still pending at the shutdown drain", async () => {
|
||||
const run = await seedOwnedNativeRun(), recovery = createHeartbeatRecovery(db, callbacks(db));
|
||||
let finishDetach!: () => void;
|
||||
native.detach.mockImplementationOnce(() => new Promise<number>(resolve => { finishDetach = () => resolve(0); }));
|
||||
try {
|
||||
const early = recovery.handOffNativeRunsForShutdown("SIGTERM");
|
||||
await vi.waitFor(() => expect(native.detach).toHaveBeenCalledTimes(1));
|
||||
let drained = false;
|
||||
const drain = recovery.drainRunningRunsForShutdown("SIGTERM", new Date(), [run.id]).finally(() => { drained = true; });
|
||||
await drainHasStarted();
|
||||
expect(drained).toBe(false);
|
||||
expect((await finalization(run.id)).recoveryState).toBeNull();
|
||||
finishDetach();
|
||||
expect(await early).toEqual([run.id]);
|
||||
expect((await drain).restartSuspendedRunIds).toEqual([run.id]);
|
||||
expect(native.detach).toHaveBeenCalledTimes(1);
|
||||
expect(await finalization(run.id)).toMatchObject({ recoveryState: "awaiting_runner_reattach" });
|
||||
} finally {
|
||||
finishDetach?.();
|
||||
await removeNativeRun(run.id);
|
||||
}
|
||||
});
|
||||
|
||||
it("hands off a run at the shutdown drain when its pending early hand-off fails", async () => {
|
||||
const run = await seedOwnedNativeRun(), recovery = createHeartbeatRecovery(db, callbacks(db));
|
||||
let failDetach!: (error: Error) => void;
|
||||
native.detach.mockImplementationOnce(() => new Promise<number>((_resolve, reject) => { failDetach = reject; }));
|
||||
try {
|
||||
const early = recovery.handOffNativeRunsForShutdown("SIGTERM");
|
||||
await vi.waitFor(() => expect(native.detach).toHaveBeenCalledTimes(1));
|
||||
const drain = recovery.drainRunningRunsForShutdown("SIGTERM", new Date(), [run.id]);
|
||||
await drainHasStarted();
|
||||
failDetach(new Error("runner detach failed"));
|
||||
expect(await early).toEqual([]);
|
||||
expect((await drain).restartSuspendedRunIds).toEqual([run.id]);
|
||||
expect(native.detach).toHaveBeenCalledTimes(2);
|
||||
expect(await finalization(run.id)).toMatchObject({ recoveryState: "awaiting_runner_reattach" });
|
||||
} finally {
|
||||
await removeNativeRun(run.id);
|
||||
}
|
||||
});
|
||||
|
||||
it("writes no hot-restart snapshot and detaches nothing after a timed-out preparation hands over to the drain", async () => {
|
||||
const run = await seedOwnedNativeRun(), recovery = createHeartbeatRecovery(db, callbacks(db));
|
||||
hotRestart.read.mockResolvedValue({ version: 1, requestedAt: new Date().toISOString(), previousServerPid: process.pid,
|
||||
previousServerVersion: null, drainRequired: false, requestedByRunId: null, preflightActiveRunIds: [] });
|
||||
let finishIdleClose!: () => void;
|
||||
native.closeIdle.mockImplementationOnce(() => new Promise(resolve => { finishIdleClose = () => resolve({ closed: 0, failed: 0 }); }));
|
||||
let preparation: ReturnType<typeof recovery.prepareHotRestartShutdown> | undefined;
|
||||
try {
|
||||
const step = await runBoundedShutdownStep({ step: "hot_restart_prepare", signal: "SIGTERM", timeoutMs: 50,
|
||||
log: { info: vi.fn(), error: vi.fn() },
|
||||
run: abort => (preparation = recovery.prepareHotRestartShutdown("SIGTERM", new Date(), abort)) });
|
||||
expect(step.outcome).toBe("timed_out");
|
||||
expect((await recovery.drainRunningRunsForShutdown("SIGTERM", new Date(), [run.id])).restartSuspendedRunIds).toEqual([run.id]);
|
||||
expect(native.detach).toHaveBeenCalledTimes(1);
|
||||
finishIdleClose();
|
||||
await expect(preparation).rejects.toThrow("timed out");
|
||||
expect(hotRestart.write).not.toHaveBeenCalled();
|
||||
expect(native.detach).toHaveBeenCalledTimes(1);
|
||||
} finally {
|
||||
finishIdleClose?.();
|
||||
await preparation?.catch(() => undefined);
|
||||
await removeNativeRun(run.id);
|
||||
}
|
||||
});
|
||||
|
||||
it("records no more hot-restart run events after the preparation times out during its first event", async () => {
|
||||
const runs = [await seedOwnedNativeRun(), await seedOwnedNativeRun(), await seedOwnedNativeRun()];
|
||||
const deps = callbacks(db), recovery = createHeartbeatRecovery(db, deps);
|
||||
hotRestart.read.mockResolvedValue({ version: 1, requestedAt: new Date().toISOString(), previousServerPid: process.pid,
|
||||
previousServerVersion: null, drainRequired: false, requestedByRunId: null, preflightActiveRunIds: [] });
|
||||
let finishFirstEvent!: () => void;
|
||||
const firstEvent = new Promise<undefined>(resolve => { finishFirstEvent = () => resolve(undefined); });
|
||||
deps.appendRunEvent.mockImplementationOnce(() => firstEvent);
|
||||
const hotRestartEvents = () => deps.appendRunEvent.mock.calls
|
||||
.filter(([, event]) => event.message?.startsWith("Hot restart requested") === true).length;
|
||||
let preparation: ReturnType<typeof recovery.prepareHotRestartShutdown> | undefined;
|
||||
try {
|
||||
const step = await runBoundedShutdownStep({ step: "hot_restart_prepare", signal: "SIGTERM", timeoutMs: 2_000,
|
||||
log: { info: vi.fn(), error: vi.fn() },
|
||||
run: abort => (preparation = recovery.prepareHotRestartShutdown("SIGTERM", new Date(), abort)) });
|
||||
expect(step.outcome).toBe("timed_out");
|
||||
expect(hotRestartEvents()).toBe(1);
|
||||
const runIds = runs.map(run => run.id);
|
||||
expect((await recovery.drainRunningRunsForShutdown("SIGTERM", new Date(), runIds)).restartSuspendedRunIds.sort())
|
||||
.toEqual([...runIds].sort());
|
||||
finishFirstEvent();
|
||||
await expect(preparation).rejects.toThrow("timed out");
|
||||
expect(hotRestartEvents()).toBe(1);
|
||||
expect(native.detach).toHaveBeenCalledTimes(runs.length);
|
||||
} finally {
|
||||
finishFirstEvent?.();
|
||||
await preparation?.catch(() => undefined);
|
||||
for (const run of runs) await removeNativeRun(run.id);
|
||||
}
|
||||
});
|
||||
|
||||
it("does not dispatch recovery for a run still preparing in this heartbeat process", async () => {
|
||||
const run = await seedRun();
|
||||
await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, run.agentId));
|
||||
|
||||
@@ -213,6 +213,8 @@ export const DETACHED_PROCESS_ERROR_CODE = "process_detached";
|
||||
const NATIVE_OWNERSHIP_UNVERIFIED_MESSAGE =
|
||||
"Native execution ownership could not be verified; automatic recovery is blocked";
|
||||
|
||||
const EARLY_NATIVE_HAND_OFF_DRAIN_WAIT_MS = 30_000;
|
||||
|
||||
// The reaper sweeps at most this many pending_cleanup leases per tick.
|
||||
const PENDING_CLEANUP_SWEEP_PAGE_SIZE = 20;
|
||||
// An environment delete tries each of that environment's cleanups once, from
|
||||
@@ -490,12 +492,14 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
async function prepareHotRestartShutdown(
|
||||
signal: "SIGINT" | "SIGTERM",
|
||||
now = new Date(),
|
||||
abort?: AbortSignal,
|
||||
) {
|
||||
enterShutdown();
|
||||
const idleSessions = await closeIdleWarmNativeSessionsForRestart();
|
||||
if (idleSessions.failed > 0) {
|
||||
logger.warn({ idleSessions }, "idle native session cleanup failed before controller shutdown");
|
||||
}
|
||||
abort?.throwIfAborted();
|
||||
let intent: Awaited<ReturnType<typeof readHotRestartIntent>>;
|
||||
try {
|
||||
intent = await readHotRestartIntent();
|
||||
@@ -544,6 +548,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
.from(heartbeatRuns)
|
||||
.innerJoin(agents, eq(heartbeatRuns.agentId, agents.id))
|
||||
.where(eq(heartbeatRuns.status, "running"));
|
||||
abort?.throwIfAborted();
|
||||
const snapshotRuns = activeRuns.map(toHotRestartIntentRun);
|
||||
const intentWithVersion = {
|
||||
...intent,
|
||||
@@ -589,8 +594,8 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
activeRuns: snapshotRuns,
|
||||
capturedAt: now,
|
||||
});
|
||||
|
||||
for (const { run } of activeRuns) {
|
||||
abort?.throwIfAborted();
|
||||
await appendRunEvent(run, {
|
||||
eventType: "lifecycle",
|
||||
stream: "system",
|
||||
@@ -614,6 +619,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
isNativeSessionId(run.nativeSessionId),
|
||||
)
|
||||
.map(({ run }) => run.id);
|
||||
abort?.throwIfAborted();
|
||||
for (const runId of nativeRunIds) markHandedOffByThisController(runId);
|
||||
const detachedNativeSessions =
|
||||
await detachNativeSessionsForRestart(nativeRunIds);
|
||||
@@ -1158,10 +1164,27 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
}
|
||||
}
|
||||
|
||||
const nativeRunsHandedOffEarly = new Set<string>();
|
||||
const nativeRunsHandedOffEarly = new Map<string, Promise<boolean>>();
|
||||
|
||||
async function settleEarlyNativeHandOff(runId: string, handOff: Promise<boolean>) {
|
||||
let timer: ReturnType<typeof setTimeout> | undefined;
|
||||
try {
|
||||
const outcome = await Promise.race([handOff, new Promise<"unfinished">(resolve => {
|
||||
timer = setTimeout(() => resolve("unfinished"), EARLY_NATIVE_HAND_OFF_DRAIN_WAIT_MS);
|
||||
timer.unref?.();
|
||||
})]);
|
||||
if (outcome === "unfinished") {
|
||||
logger.warn({ runId, waitedMs: EARLY_NATIVE_HAND_OFF_DRAIN_WAIT_MS },
|
||||
"early native run hand-off is unfinished at the shutdown drain; the drain hands it off");
|
||||
}
|
||||
return outcome === true;
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
|
||||
async function suspendNativeRunForShutdown(run: typeof heartbeatRuns.$inferSelect, signal: "SIGINT" | "SIGTERM",
|
||||
now: Date, controllerBootId: string) {
|
||||
now: Date, controllerBootId: string, timing?: { detachMs: number; transactionMs: number }) {
|
||||
const [owner] = await db.select({ bootId: nativeRunFinalizations.controllerBootId })
|
||||
.from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id));
|
||||
if (owner?.bootId && owner.bootId !== controllerBootId) return false;
|
||||
@@ -1170,7 +1193,9 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
// finalizer interrupt/suspend Claude while the next server is adopting
|
||||
// the same turn.
|
||||
markHandedOffByThisController(run.id);
|
||||
const detachStartedAt = performance.now();
|
||||
await detachNativeSessionsForRestart([run.id]);
|
||||
if (timing) timing.detachMs = Math.round(performance.now() - detachStartedAt);
|
||||
const recoveryHistoryEntry = JSON.stringify({
|
||||
at: now.toISOString(),
|
||||
restartKind: "graceful",
|
||||
@@ -1183,6 +1208,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
// execution lease must still be released: the detached executor will
|
||||
// not finalize, and the next controller's reconciler skips leased rows.
|
||||
// A finalizer lease belongs to finalization already in progress here.
|
||||
const transactionStartedAt = performance.now();
|
||||
const released = await db.transaction(async tx => {
|
||||
await notifyLiveRunRecovery(tx, { companyId: run.companyId, runId: run.id });
|
||||
await notifyReattachHandOver(tx, run.id);
|
||||
@@ -1217,6 +1243,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
)
|
||||
.returning({ resultId: nativeRunFinalizations.resultId, leaseOwner: nativeRunFinalizations.leaseOwner });
|
||||
});
|
||||
if (timing) timing.transactionMs = Math.round(performance.now() - transactionStartedAt);
|
||||
const resultAccepted = Boolean(released[0]?.resultId);
|
||||
const finalizerHeld = resultAccepted && Boolean(released[0]?.leaseOwner?.startsWith("native-finalizer:"));
|
||||
await appendRunEvent(run, {
|
||||
@@ -1243,25 +1270,40 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
}
|
||||
|
||||
async function handOffNativeRunsForShutdown(signal: "SIGINT" | "SIGTERM", now = new Date()) {
|
||||
const startedAt = performance.now();
|
||||
const elapsed = (from: number) => Math.round(performance.now() - from);
|
||||
const intent = await readHotRestartIntent().catch(() => null);
|
||||
const intentReadMs = elapsed(startedAt);
|
||||
if (intent && !intent.drainRequired && shouldHonorHotRestartIntentForProcess(intent)) return [];
|
||||
const identityStartedAt = performance.now();
|
||||
const nativeController = await currentNativeControllerIdentity();
|
||||
const controllerIdentityMs = elapsed(identityStartedAt);
|
||||
const selectStartedAt = performance.now();
|
||||
const liveRuns = await db.select({ run: heartbeatRuns }).from(heartbeatRuns)
|
||||
.innerJoin(agents, eq(heartbeatRuns.agentId, agents.id))
|
||||
.where(and(eq(heartbeatRuns.status, "running"), eq(heartbeatRuns.runtimeMode, "native"),
|
||||
eq(agents.adapterType, "paperclip_runner")));
|
||||
const selectMs = elapsed(selectStartedAt);
|
||||
const handedOff: string[] = [];
|
||||
let maxDetachMs = 0, maxTransactionMs = 0, failed = 0;
|
||||
await Promise.allSettled(liveRuns.map(async ({ run }) => {
|
||||
if (isNativeRunnerOwnershipHeld(run) || nativeRunsHandedOffEarly.has(run.id)) return;
|
||||
nativeRunsHandedOffEarly.add(run.id);
|
||||
try {
|
||||
if (await suspendNativeRunForShutdown(run, signal, now, nativeController.bootId)) handedOff.push(run.id);
|
||||
else nativeRunsHandedOffEarly.delete(run.id);
|
||||
} catch (error) {
|
||||
nativeRunsHandedOffEarly.delete(run.id);
|
||||
const timing = { detachMs: 0, transactionMs: 0 };
|
||||
const handOff = suspendNativeRunForShutdown(run, signal, now, nativeController.bootId, timing).catch(error => {
|
||||
failed += 1;
|
||||
logger.warn({ err: error, runId: run.id }, "early native run hand-off failed; the shutdown drain retries it");
|
||||
}
|
||||
return false;
|
||||
});
|
||||
nativeRunsHandedOffEarly.set(run.id, handOff);
|
||||
const suspended = await handOff;
|
||||
maxDetachMs = Math.max(maxDetachMs, timing.detachMs);
|
||||
maxTransactionMs = Math.max(maxTransactionMs, timing.transactionMs);
|
||||
if (suspended) handedOff.push(run.id);
|
||||
else if (nativeRunsHandedOffEarly.get(run.id) === handOff) nativeRunsHandedOffEarly.delete(run.id);
|
||||
}));
|
||||
logger.info({ signal, liveRuns: liveRuns.length, handedOff: handedOff.length, failed, intentReadMs,
|
||||
controllerIdentityMs, selectMs, maxDetachMs, maxTransactionMs, totalMs: elapsed(startedAt) },
|
||||
"native run shutdown hand-off phase timings");
|
||||
return handedOff;
|
||||
}
|
||||
|
||||
@@ -1311,8 +1353,12 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
run.runtimeMode === "native" &&
|
||||
agent.adapterType === "paperclip_runner"
|
||||
) {
|
||||
if (nativeRunsHandedOffEarly.has(run.id)
|
||||
|| await suspendNativeRunForShutdown(run, signal, now, nativeController.bootId)) restartSuspendedRunIds.push(run.id);
|
||||
const earlyHandOff = nativeRunsHandedOffEarly.get(run.id);
|
||||
if (earlyHandOff && await settleEarlyNativeHandOff(run.id, earlyHandOff)) {
|
||||
restartSuspendedRunIds.push(run.id);
|
||||
return;
|
||||
}
|
||||
if (await suspendNativeRunForShutdown(run, signal, now, nativeController.bootId)) restartSuspendedRunIds.push(run.id);
|
||||
return;
|
||||
}
|
||||
if (agent.adapterType === "paperclip_runner" && run.runtimeMode === "legacy" && run.scopeKind === "issue" &&
|
||||
|
||||
@@ -410,6 +410,21 @@ describe("runBoundedShutdownStep", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("aborts the signal of a step that outlives its bound, and only that step", async () => {
|
||||
vi.useFakeTimers();
|
||||
try {
|
||||
const signals: AbortSignal[] = [];
|
||||
const budget = createPreDrainShutdownBudget({ signal: "SIGTERM", budgetMs: 250, log: stubLogger() });
|
||||
await budget.run("delivery_work_stop", async abort => { signals.push(abort); });
|
||||
const pending = budget.run("hot_restart_prepare", abort => { signals.push(abort); return new Promise<void>(() => undefined); });
|
||||
await vi.advanceTimersByTimeAsync(250);
|
||||
await expect(pending).resolves.toMatchObject({ outcome: "timed_out" });
|
||||
expect(signals.map(signal => signal.aborted)).toEqual([false, true]);
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
|
||||
it("logs a step that rejects and does not reject the shutdown", async () => {
|
||||
const log = stubLogger();
|
||||
const error = new Error("skill sync stop failed");
|
||||
|
||||
@@ -23,18 +23,19 @@ export async function runBoundedShutdownStep<T>(input: {
|
||||
step: string;
|
||||
signal: "SIGINT" | "SIGTERM";
|
||||
timeoutMs: number;
|
||||
run: () => Promise<T>;
|
||||
run: (abort: AbortSignal) => Promise<T>;
|
||||
log: ShutdownLogger;
|
||||
now?: () => number;
|
||||
}): Promise<ShutdownStepResult<T>> {
|
||||
const now = input.now ?? Date.now;
|
||||
const startedAt = now();
|
||||
const timeoutMs = Math.max(0, input.timeoutMs);
|
||||
const abort = new AbortController();
|
||||
let timer: NodeJS.Timeout | null = null;
|
||||
try {
|
||||
const result = await Promise.race([
|
||||
Promise.resolve()
|
||||
.then(input.run)
|
||||
.then(() => input.run(abort.signal))
|
||||
.then(
|
||||
(value) => ({ outcome: "completed" as const, value }),
|
||||
(error: unknown) => ({ outcome: "failed" as const, error }),
|
||||
@@ -54,6 +55,7 @@ export async function runBoundedShutdownStep<T>(input: {
|
||||
input.log.error({ ...fields, err: result.error }, "shutdown step failed; continuing shutdown");
|
||||
return { outcome: "failed", error: result.error, durationMs };
|
||||
}
|
||||
abort.abort(new Error(`shutdown step ${input.step} timed out`));
|
||||
input.log.info(fields, "shutdown step timed out; continuing shutdown without it");
|
||||
return { outcome: "timed_out", durationMs };
|
||||
} finally {
|
||||
@@ -70,7 +72,7 @@ export function createPreDrainShutdownBudget(input: {
|
||||
const now = input.now ?? Date.now;
|
||||
const deadline = now() + (input.budgetMs ?? PRE_DRAIN_SHUTDOWN_BUDGET_MS);
|
||||
return {
|
||||
run<T>(step: string, run: () => Promise<T>) {
|
||||
run<T>(step: string, run: (abort: AbortSignal) => Promise<T>) {
|
||||
return runBoundedShutdownStep({
|
||||
step,
|
||||
signal: input.signal,
|
||||
|
||||
Reference in new issue
Block a user