mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A deploy swaps servers with an overlap: the outgoing server still serves while the incoming server starts new runs > - A warm agent session stays open in the server process that ran the last turn, after that run finished > - The incoming server took over a warm session while the outgoing server was alive and still held it, because the finished run's finalization lease ends at commit > - The takeover then failed, the task went blocked with a Retry, and in one case a retry loop ran 27 times on the same sandbox > - This pull request makes the incoming server wait while the server that holds the warm session is alive, makes the outgoing server close idle warm sessions as soon as it leaves new runs to the replacement, and stops the retry loop when the promised fresh sandbox does not come > - The benefit is that a follow-up during a deploy finishes without a person, and no Runner ever gets two controllers ## Linked Issues or Issue Description **What happened?** A zero-downtime release-swap test (Gate B, wave B1, masteraaec67b891) found two failures with one cause. Task 1: run A finished on the outgoing server at 15:13:56. Its warm session stayed idle in that server. The incoming server claimed the follow-up at 15:13:57. `inspectRemoteWarmSessionRecovery` asked `evaluateNativeControllerTakeover`, which allowed the takeover with `remote_controller_lease_released_or_expired`. That lease covers the finished run, not the warm session. The outgoing server was alive until 15:18:18. `runner.prp.authenticate` failed after 62.3 s, the run failed with `remote_warm_session_unproven`, the failure was saved as ambiguous and owned by the board, and the task stayed blocked with a Retry. Task 2: the same shape on a later swap started 27 automatic runs in 13 minutes. Fourteen of them said "The next attempt starts on a fresh sandbox", but each one resumed the same sandbox and got the same refusal (`journal_unsettled`). **Expected behavior** A follow-up claimed during an overlap waits for the warm session owner to let go, then continues with no Retry banner and no second controller. If the outgoing server dies with the session open, the takeover continues after its ownership ends. A message that promises a fresh sandbox is true, or the retries stop after a bounded number with a true state. **Steps to reproduce** 1. Run the restart-chaos release-swap preset with a warm (opencode or codex) staging mix. 2. Let a run finish on the outgoing server after the traffic switch, with a follow-up comment already queued. 3. See the follow-up fail with `remote_warm_session_unproven` and the task go blocked. **Paperclip version or commit** masteraaec67b891and45531b719e. **Deployment mode** Hosted deploy overlap (release swap) with Daytona sandboxes. ## What Changed - `remote-warm-session-recovery.ts`: before live adoption, the run proves that the server process that ran the finished turn is gone. It uses the shared owner proof (`proveOwnerGone`): a PID check in the same process space, or the `controller_instances` lease in another process space. While that server is alive, the run defers with the existing `native_controller_recovery_waiting_for_owner` wait and checks again after 5 s. The wait ends when the server exits or is fenced, or when the finished turn's warm window (idle timeout plus two minutes) ends. - `native-session-executor.ts` and `run-start-handover.ts`: a server that leaves new run starts to a replacement closes its idle warm sessions after 1 s, through the existing idle-timeout close. A session that becomes idle after that closes after 1 s too. This is the same close that shutdown already does, done earlier. - `predecessor-retirement-fallback.ts`: when a retry meets an unretired owner on the same sandbox that its source already asked to replace, it does not ask again. The run fails without another automatic retry, and the task shows a Retry. - `predecessor-retirement.ts`: the error no longer says "The next attempt starts on a fresh sandbox", because the replacement is only a request. - `doc/execution-semantics.md`: documents the owner wait, the hand-off at the swap, and the retry stop. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-controller-recovery.integration.test.ts` - New: "waits while the outgoing server that ran the finished turn still holds its warm session, then takes it over once that server exits" (red on master: master returned a takeover while the owner was alive). - New: "waits for a live outgoing server in another process space until its controller lease ends" (red on master). - New: "stops waiting for a live outgoing server once the finished turn's warm window has closed" (bound check). - `cd server && npx vitest run src/__tests__/heartbeat-native-predispatch-retry.test.ts` - New: "stops retrying with a truthful failure when the next attempt met the same unretired sandbox instead of a fresh one" (red on master: master scheduled another retry on the same sandbox). - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "replacement server starts"` - New: "closes a warm session promptly when a replacement server starts this server's runs (before/after the turn ends)" (both red on master). - Focused suites: native-session-executor, remote-warm-session-recovery, native-controller-recovery, native-recovered-owner-followup, native-runner-restart-recovery, heartbeat-native-predispatch-retry, environment-runtime-reusable-resume-replacement, task-interruption-presentation, sandbox-runner-port-holders. - `npx tsc --noEmit -p server/tsconfig.json` passes. `pnpm verify` exits 2 (a doc is a global CI input), so required CI supplies broad coverage. - Not run here: the restart-chaos release-swap wave. The maintainer runs it after merge. ## Risks - A follow-up can now wait up to the outgoing server's remaining overlap when that server cannot close its idle session. Before, it failed and needed a Retry. The wait is bounded by the owner's exit or the warm window. - The outgoing server closes idle warm sessions earlier during an overlap. The next turn then resumes from a suspended Runner instead of a live one, as it already does after a shutdown. - A retry chain that meets the same unretired sandbox twice now stops one attempt earlier than the shared cap. The task shows a Retry instead of looping. - Rows without a recorded process space keep the existing one-release local proof, so they behave as before outside the same host. - Previous-release state: this change adds no persisted field. Finalization rows saved by the previous release already record the controller boot, PID, process space and start time, so they carry forward. A follow-up already deferred by the previous release resumes through the same wait code. - Runtime cost: the owner proof adds one `ps` start-time read (same process space) or one `controller_instances` row read (other process space) per inspection. It runs once per attempt, at most every 5 s while a live owner holds the session. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
404 lines
19 KiB
TypeScript
404 lines
19 KiB
TypeScript
import { sql, type SQL } from "drizzle-orm";
|
|
import { controllerLeaseDb, type Db } from "@paperclipai/db";
|
|
import { notifyDatabaseWorkOnCommit } from "@paperclipai/db/work-signals";
|
|
import { logger } from "../middleware/logger.js";
|
|
import { currentControllerBootId, currentProcessSpaceHash } from "./controller-process-identity.js";
|
|
import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
|
|
import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
|
|
import { subscribeTaskDrainChanges, subscribeTaskDrainExpiry } from "./task-admission.js";
|
|
|
|
/**
|
|
* Hand-over of new run starts during a deploy overlap.
|
|
*
|
|
* A replacement server reports that it starts runs by setting `dispatching_at`
|
|
* and a short `dispatch_lease_expires_at` on its own `controller_instances` row,
|
|
* once its queued-run dispatch is running. An older server that is still
|
|
* serving then leaves NEW run starts (queued-run claims) to it; runs it already
|
|
* started carry on. The lease is renewed only while an older server may still
|
|
* be serving, so a lone server writes once and the database can still sleep.
|
|
* If the replacement stops, drains or dies, its lease is cleared or lapses and
|
|
* the older server starts runs again within seconds: the hand-over never leaves
|
|
* the company without a server that starts runs.
|
|
*
|
|
* Readers outside this module (for example a deployer) can read "a replacement
|
|
* is starting runs" as a row with a higher epoch than the old boot where
|
|
* `dispatching_at is not null and dispatch_lease_expires_at > now()
|
|
* and stopped_at is null and fenced_at is null`.
|
|
*
|
|
* Database use: every statement of this module (marker writes, reads, and the
|
|
* commit-bound notifications) runs in a small transaction on the dedicated
|
|
* one-connection controller-lease pool with a 1 s statement timeout. It never
|
|
* touches the shared pool and never registers tracked database work, so the
|
|
* worker coordinator has nothing of it to reconcile there. That pool is shared
|
|
* only with legacy controller-lease renewals: a statement here waits at most one
|
|
* legacy batch (10 s statement timeout), inside the 15 s between a renewal and
|
|
* the lease's expiry, and delays a legacy renewal (60 s lease, renewed every
|
|
* 10 s) by at most 1 s. At most one read, one write and one hint are in flight.
|
|
* An unanswered read never keeps this server yielding: a dispatch that cannot
|
|
* get an answer within a second admits, and a yield always ends when the
|
|
* replacement's last seen lease runs out on this server's clock.
|
|
*/
|
|
export const RUN_START_HANDOVER_RENEW_MS = 5_000;
|
|
export const RUN_START_HANDOVER_TTL_MS = 20_000;
|
|
/** A replacement renews the hand-over for at most this long after it began starting runs. */
|
|
export const RUN_START_HANDOVER_MAX_MS = 30 * 60_000;
|
|
/** A yielding server checks this often whether the replacement still starts runs. */
|
|
export const RUN_START_YIELD_RECHECK_MS = 3_000;
|
|
|
|
const DEFAULT_TIMING = {
|
|
renewMs: RUN_START_HANDOVER_RENEW_MS,
|
|
ttlMs: RUN_START_HANDOVER_TTL_MS,
|
|
maxMs: RUN_START_HANDOVER_MAX_MS,
|
|
/** An older boot counts as possibly serving while its lease was live this recently. */
|
|
olderBootWindowMs: RUN_START_HANDOVER_MAX_MS,
|
|
recheckMs: RUN_START_YIELD_RECHECK_MS,
|
|
/** A dispatch pass first waits for a read when the last one is older than this: the bound on a stale answer. */
|
|
staleAwaitMs: 2_000,
|
|
/** A dispatch waits at most this long for that read; the read itself is cancelled by the database. */
|
|
readWaitMs: 1_000,
|
|
readStatementTimeoutMs: 1_000,
|
|
/** Short, so a marker write can delay a legacy lease renewal on the shared lease connection by at most this. */
|
|
writeStatementTimeoutMs: 1_000,
|
|
/** Skipped starts are announced to the replacement at most this often. */
|
|
skipHintMinMs: 1_000,
|
|
/** A clear that failed while this server cannot start runs is retried this often until the lease would lapse. */
|
|
clearRetryMs: 1_000,
|
|
};
|
|
|
|
const interval = (ms: number) => sql`${`${ms} milliseconds`}::interval`;
|
|
|
|
export type RunStartHandover = ReturnType<typeof createRunStartHandover>;
|
|
|
|
export function createRunStartHandover(input: {
|
|
db: Db;
|
|
/** The server's shared worker coordinator: renewals and rechecks are its deadlines. */
|
|
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>;
|
|
/** This server may start runs now (not stopping, draining, or in standby). */
|
|
canDispatch: () => boolean;
|
|
/** Called when a yielding server starts runs again. */
|
|
onResume?: () => void;
|
|
bootId?: string;
|
|
processSpace?: string;
|
|
pid?: number;
|
|
timing?: Partial<typeof DEFAULT_TIMING>;
|
|
}) {
|
|
const { db } = input;
|
|
const t = { ...DEFAULT_TIMING, ...input.timing };
|
|
const bootId = input.bootId ?? currentControllerBootId();
|
|
let stopped = false;
|
|
const listeners = new Set<() => void>();
|
|
const changed = () => { for (const listener of listeners) { try { listener(); } catch { /* never blocks */ } } };
|
|
|
|
// Replacement side. Every marker write runs in this one chain, so a release
|
|
// always lands after a renewal that was already in flight.
|
|
let intendedAt: number | null = null;
|
|
let released = false;
|
|
let olderMayServe = true;
|
|
let leaseHeld = false;
|
|
/** A marker was written and no clear has succeeded since: a release must (re)try the clear. */
|
|
let markerMayBeLive = false;
|
|
let lastWriteAt = Number.NEGATIVE_INFINITY;
|
|
let lastClearAttemptAt = Number.NEGATIVE_INFINITY;
|
|
let writes: Promise<void> = Promise.resolve();
|
|
const handingOver = () => intendedAt !== null && !released && olderMayServe && Date.now() - intendedAt < t.maxMs;
|
|
let reportedHandingOver = false;
|
|
const reportHandingOver = () => {
|
|
const now = handingOver();
|
|
if (now !== reportedHandingOver) { reportedHandingOver = now; changed(); }
|
|
};
|
|
|
|
// Older-server side: the in-memory flag every run start reads.
|
|
let yieldingTo: string | null = null;
|
|
/** Local-clock end of the replacement's last seen lease: a yield is never kept past it, read or no read. */
|
|
let yieldUntil = Number.NEGATIVE_INFINITY;
|
|
let lastReadAt = Number.NEGATIVE_INFINITY;
|
|
let reading: Promise<void> | null = null;
|
|
let readRequested = false;
|
|
let lastSkipHintAt = Number.NEGATIVE_INFINITY;
|
|
let hinting: Promise<void> | null = null;
|
|
|
|
/**
|
|
* One statement under a statement timeout, so a slow database cancels it instead of holding a pool connection.
|
|
* With `announce`, the marker-change notification is part of the same transaction on the same connection: it is
|
|
* delivered on commit, and nothing in the write path waits on another pool.
|
|
*/
|
|
function bounded<T>(statement: SQL, timeoutMs: number, announce = false): Promise<T[]> {
|
|
// Every statement of this module runs here: never on the shared pool, whose stalls must not move run starts.
|
|
return controllerLeaseDb(db).transaction(async tx => {
|
|
await tx.execute(sql`select set_config('statement_timeout', ${String(timeoutMs)}, true)`);
|
|
// Commit-bound and untracked: a failed write leaves no uncertain transaction for the coordinator to
|
|
// reconcile on the shared pool before the next renewal.
|
|
if (announce) await notifyDatabaseWorkOnCommit(tx, DELIVERY_QUEUES.runStartHandover);
|
|
return await tx.execute(statement) as unknown as T[];
|
|
});
|
|
}
|
|
|
|
const serialize = (work: () => Promise<void>) => {
|
|
const next = writes.then(work);
|
|
writes = next.catch(error => { logger.warn({ err: error }, "run start hand-over write failed; its lease lapses unless a later write succeeds"); });
|
|
return writes;
|
|
};
|
|
|
|
async function writeLease(announce: boolean) {
|
|
const [row] = await bounded<{ olderMayServe: boolean }>(sql`
|
|
insert into controller_instances (boot_id, process_space, pid, lease_expires_at, dispatching_at, dispatch_lease_expires_at)
|
|
values (${bootId}, ${input.processSpace ?? currentProcessSpaceHash()}, ${input.pid ?? process.pid}, now(), now(),
|
|
now() + ${interval(t.ttlMs)})
|
|
on conflict (boot_id) do update set
|
|
dispatching_at = coalesce(controller_instances.dispatching_at, now()),
|
|
dispatch_lease_expires_at = now() + ${interval(t.ttlMs)}
|
|
where controller_instances.fenced_at is null and controller_instances.stopped_at is null
|
|
returning exists (select 1 from controller_instances older
|
|
where older.epoch < controller_instances.epoch and older.boot_id <> ${bootId}
|
|
and older.stopped_at is null and older.fenced_at is null
|
|
and older.lease_expires_at > now() - ${interval(t.olderBootWindowMs)}) as "olderMayServe"`, t.writeStatementTimeoutMs,
|
|
announce);
|
|
return row ?? null;
|
|
}
|
|
|
|
async function clearLease() {
|
|
await bounded(sql`update controller_instances set dispatch_lease_expires_at = null
|
|
where boot_id = ${bootId} and dispatch_lease_expires_at is not null`, t.writeStatementTimeoutMs, true);
|
|
// Only a clear that succeeded ends the marker; a failed one is retried by the next release or renewal.
|
|
leaseHeld = false;
|
|
markerMayBeLive = false;
|
|
}
|
|
|
|
/** Set, renew, or clear the marker to match whether this server can start runs now. */
|
|
function maintainLease(): Promise<void> {
|
|
return serialize(async () => {
|
|
if (intendedAt === null || released || !handingOver() && leaseHeld) return;
|
|
if (!input.canDispatch()) {
|
|
// Draining, standby or admission closed: the older server takes run starts back now.
|
|
if (markerMayBeLive) {
|
|
lastClearAttemptAt = Date.now();
|
|
await clearLease();
|
|
}
|
|
return;
|
|
}
|
|
if (Date.now() - lastWriteAt < t.renewMs / 2) return;
|
|
// Counted from the attempt, so a failing database is retried on the renewal cadence, not in a loop.
|
|
lastWriteAt = Date.now();
|
|
markerMayBeLive = true;
|
|
let row: Awaited<ReturnType<typeof writeLease>>;
|
|
// A first or regained marker tells the older server in the same transaction.
|
|
try { row = await writeLease(!leaseHeld); }
|
|
catch (error) {
|
|
// Unconfirmed: the next successful write counts as regained and tells the older server again.
|
|
leaseHeld = false;
|
|
throw error;
|
|
}
|
|
const regained = row !== null && !leaseHeld;
|
|
leaseHeld = row !== null;
|
|
// No older server left to hand over from: the lease lapses without further writes.
|
|
olderMayServe = row?.olderMayServe ?? false;
|
|
if (regained) {
|
|
if (olderMayServe) logger.info("this server now starts runs; an older server still serving leaves new run starts to it");
|
|
}
|
|
}).finally(reportHandingOver);
|
|
}
|
|
|
|
/** Read whether a newer boot starts runs now. A failed or cancelled read resumes run starts here. */
|
|
async function read(): Promise<void> {
|
|
let next: string | null = null;
|
|
let until = Number.NEGATIVE_INFINITY;
|
|
const askedAt = Date.now();
|
|
try {
|
|
const [row] = await bounded<{ bootId: string; remainingMs: number }>(sql`
|
|
select newer.boot_id as "bootId",
|
|
extract(epoch from newer.dispatch_lease_expires_at - now()) * 1000 as "remainingMs"
|
|
from controller_instances me
|
|
join controller_instances newer on newer.epoch > me.epoch and newer.boot_id <> me.boot_id
|
|
where me.boot_id = ${bootId} and newer.dispatching_at is not null
|
|
and newer.dispatch_lease_expires_at > now()
|
|
and newer.stopped_at is null and newer.fenced_at is null
|
|
order by newer.epoch desc limit 1`, t.readStatementTimeoutMs);
|
|
next = row?.bootId ?? null;
|
|
// Counted from when the read was asked, so time spent waiting for the answer only shortens the yield.
|
|
if (row) until = askedAt + Number(row.remainingMs);
|
|
} catch (error) {
|
|
logger.warn({ err: error }, "run start hand-over read failed; this server keeps starting runs");
|
|
}
|
|
lastReadAt = Date.now();
|
|
if (stopped) return;
|
|
const previous = yieldingTo;
|
|
yieldingTo = next && until > Date.now() ? next : null;
|
|
yieldUntil = until;
|
|
next = yieldingTo;
|
|
if (next && !previous) {
|
|
logger.info({ replacementBootId: next }, "a newer server is starting runs; this server leaves new run starts to it");
|
|
skipHint(true);
|
|
changed();
|
|
worker.wake();
|
|
} else if (!next && previous) {
|
|
logger.info({ replacementBootId: previous }, "the newer server stopped starting runs; this server starts runs again");
|
|
changed();
|
|
input.onResume?.();
|
|
}
|
|
// Schedule the next recheck from this answer.
|
|
if (yieldingTo) worker.wake();
|
|
}
|
|
|
|
/** The replacement's lease ran out on this server's clock: start runs again even if no read has answered. */
|
|
function expireYield() {
|
|
if (!yieldingTo || Date.now() < yieldUntil) return;
|
|
logger.warn({ replacementBootId: yieldingTo }, "the newer server's run start lease ran out; this server starts runs again");
|
|
yieldingTo = null;
|
|
changed();
|
|
queueMicrotask(() => input.onResume?.());
|
|
}
|
|
|
|
/** Single flight: a slow read stays tracked until it settles, so reads never pile up on the pool. */
|
|
function refresh(): Promise<void> {
|
|
reading ??= read().finally(() => { reading = null; });
|
|
return reading;
|
|
}
|
|
|
|
/** Runs this server skips stay queued; wake the replacement's dispatch so they do not wait for a sweep. */
|
|
function skipHint(force = false) {
|
|
const now = Date.now();
|
|
// Best effort and single flight: a busy pool delays one hint, never a queue of them (the replacement's own
|
|
// queued-run redrive still covers a lost hint).
|
|
if (hinting || (!force && now - lastSkipHintAt < t.skipHintMinMs)) return;
|
|
lastSkipHintAt = now;
|
|
hinting = controllerLeaseDb(db).transaction(async tx => {
|
|
await tx.execute(sql`select set_config('statement_timeout', ${String(t.writeStatementTimeoutMs)}, true)`);
|
|
await notifyDatabaseWorkOnCommit(tx, DELIVERY_QUEUES.queuedRunDispatch);
|
|
})
|
|
.catch(error => logger.warn({ err: error }, "queued run dispatch hint for the newer server failed"))
|
|
.finally(() => { hinting = null; });
|
|
}
|
|
|
|
function nextDeadline(): number | null {
|
|
const deadlines: number[] = [];
|
|
// A clear that has not succeeded yet, while this server cannot start runs: retry until the lease would lapse.
|
|
if (handingOver() && !input.canDispatch() && markerMayBeLive && Date.now() < lastWriteAt + t.ttlMs) {
|
|
deadlines.push(lastClearAttemptAt + t.clearRetryMs);
|
|
}
|
|
// Renew while handing over and able to start runs; a drain change wakes the worker otherwise.
|
|
if (handingOver() && input.canDispatch()) deadlines.push(lastWriteAt + t.renewMs);
|
|
if (yieldingTo) deadlines.push(reading ? yieldUntil : Math.min(lastReadAt + t.recheckMs, yieldUntil));
|
|
return deadlines.length ? Math.max(Date.now(), Math.min(...deadlines)) : null;
|
|
}
|
|
|
|
const worker = input.coordinator.register(DELIVERY_QUEUES.runStartHandover, {
|
|
retryMs: t.renewMs,
|
|
// Clearing the marker while this server drains is what lets the older one resume.
|
|
runDuringDrain: true,
|
|
// Another server's marker changed.
|
|
onSettled: () => { readRequested = true; },
|
|
run: async () => {
|
|
if (stopped) return;
|
|
expireYield();
|
|
if (handingOver()) await maintainLease();
|
|
if (readRequested || (yieldingTo && !reading && Date.now() - lastReadAt >= t.recheckMs)) {
|
|
readRequested = false;
|
|
// Not awaited: a read stuck behind the pool must not hold the worker, whose lease-expiry deadline still fires.
|
|
void refresh();
|
|
}
|
|
},
|
|
hasPending: async () => false,
|
|
nextRunAt: nextDeadline,
|
|
});
|
|
const stopDrainChanges = subscribeTaskDrainChanges(worker.wake);
|
|
const stopDrainExpiry = subscribeTaskDrainExpiry(worker.wake);
|
|
|
|
return {
|
|
ready: worker.ready,
|
|
/** Replacement side: this boot starts runs now. Older boots leave new starts to it. A drain defers it. */
|
|
async begin(): Promise<void> {
|
|
if (stopped || released || intendedAt !== null) return;
|
|
intendedAt = Date.now();
|
|
await maintainLease();
|
|
worker.wake();
|
|
},
|
|
/** Replacement side, final (shutdown): stop claiming run starts. Older boots resume at once. */
|
|
async release(): Promise<void> {
|
|
released = true;
|
|
reportHandingOver();
|
|
// Runs after any write already in flight, so nothing restores the marker afterwards.
|
|
await serialize(async () => { if (markerMayBeLive) await clearLease(); });
|
|
},
|
|
/** The hot-path flag: a newer server starts runs, so this one starts none. Never queries inline. */
|
|
isYielding(): boolean {
|
|
expireYield();
|
|
if (yieldingTo) skipHint();
|
|
return yieldingTo !== null;
|
|
},
|
|
/** Before a dispatch pass: when the last read is older than staleAwaitMs, wait (briefly) for a fresh one. */
|
|
refreshIfStale(): Promise<void> | null {
|
|
if (stopped || Date.now() - lastReadAt < t.staleAwaitMs) return null;
|
|
let timer: ReturnType<typeof setTimeout> | undefined;
|
|
const read = refresh();
|
|
// The read keeps running (and stays the single flight) if this caller stops waiting. Until it
|
|
// answers, an unconfirmed yield is not trusted: this server admits rather than risk a pause.
|
|
const waited = new Promise<void>(resolve => {
|
|
timer = setTimeout(() => {
|
|
if (reading === read && yieldingTo) {
|
|
logger.warn({ replacementBootId: yieldingTo }, "run start hand-over read is slow; this server starts runs until it answers");
|
|
yieldingTo = null;
|
|
changed();
|
|
}
|
|
resolve();
|
|
}, t.readWaitMs);
|
|
});
|
|
return Promise.race([read, waited]).finally(() => clearTimeout(timer));
|
|
},
|
|
refresh,
|
|
/** This server holds a hand-over an older server may be reading. */
|
|
isHandingOver: () => handingOver(),
|
|
isLeavingRunStarts: () => yieldingTo !== null,
|
|
/** A hand-over involves this server (either side). Pure read, no refresh. */
|
|
isActive: () => handingOver() || yieldingTo !== null,
|
|
subscribe(listener: () => void) {
|
|
listeners.add(listener);
|
|
return () => { listeners.delete(listener); };
|
|
},
|
|
async stop(): Promise<void> {
|
|
await this.release();
|
|
stopped = true;
|
|
stopDrainChanges();
|
|
stopDrainExpiry();
|
|
yieldingTo = null;
|
|
await worker.stop();
|
|
// A pending read or best-effort hint is not awaited: on the shared pool it could wait behind slow work,
|
|
// and it settles (or fails) when the pool answers or closes. Neither can change the marker.
|
|
},
|
|
};
|
|
}
|
|
|
|
// The process's hand-over, when this deployment can hand over (cloud overlap).
|
|
let current: RunStartHandover | null = null;
|
|
const processListeners = new Set<() => void>();
|
|
let unsubscribeCurrent: () => void = () => {};
|
|
|
|
export function registerRunStartHandover(handover: RunStartHandover | null): void {
|
|
unsubscribeCurrent();
|
|
current = handover;
|
|
unsubscribeCurrent = handover?.subscribe(() => { for (const listener of processListeners) listener(); }) ?? (() => {});
|
|
}
|
|
|
|
/** True while a newer server starts this server's new runs. No query on this path. */
|
|
export function isYieldingRunStarts(): boolean {
|
|
return current?.isYielding() ?? false;
|
|
}
|
|
|
|
/** Null (nothing to await) unless this server has not read the hand-over recently. */
|
|
export function refreshRunStartYieldIfStale(): Promise<void> | null {
|
|
return current?.refreshIfStale() ?? null;
|
|
}
|
|
|
|
export function isLeavingRunStartsToReplacement(): boolean {
|
|
return current?.isLeavingRunStarts() ?? false;
|
|
}
|
|
|
|
/** Either side of a hand-over is in progress here; a pure read for listeners. */
|
|
export function isRunStartHandoverActive(): boolean {
|
|
return current?.isActive() ?? false;
|
|
}
|
|
|
|
export function subscribeRunStartHandoverChanges(listener: () => void): () => void {
|
|
processListeners.add(listener);
|
|
return () => { processListeners.delete(listener); };
|
|
}
|