mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Cloud replaces the server process on each release. The old server keeps serving until the new one passes health checks and takes the traffic. > - During that overlap the old server keeps starting new runs right up to SIGTERM, although the new server can already start them. > - SIGTERM then cuts off exactly those runs while they still prepare. In one audited swap the old server started 30 runs after the new server booted, and 23 of them ended `process_lost`. > - Admission must never pause, so the fix must hand over who starts new runs, not stop admission. > - This pull request makes the new server record when it starts runs. The old server then leaves new run starts to it, and starts them again if the new server stops or dies. > - The benefit is that a deploy no longer starts runs on a server that is about to stop, and the company always has one server that starts runs. ## Linked Issues or Issue Description Refs #20852 (preserves preparing and launching turns that a deploy interrupts; this PR reduces how many such turns the old server starts). Related: #21046, #21107 (shared controller-instance owner proof), #20986 (queued-run dispatch signal). **What existing behavior does this improve?** The start of new runs while two servers overlap during a Cloud deploy. **Subsystem affected** Server run admission (`server/src/services/heartbeat/queue.ts`), the controller instance table, and startup and shutdown in `server/src/index.ts`. **Current behavior** The old server starts queued runs until it receives SIGTERM. Runs that it starts in the last minutes of the overlap are interrupted while they prepare. Nothing tells the old server that its replacement already starts runs: `controller_instances.epoch` is written at process start, and that happens minutes before the new server can start runs. **Proposed behavior** The new server records `dispatching_at` and a short `dispatch_lease_expires_at` on its `controller_instances` row right after its startup queued-run pass. The old server reads this through an in-memory flag at run admission and leaves new run starts to the new server. Runs it already started continue and are handed over at SIGTERM as today. If the new server stops, drains or dies, the old server starts runs again within seconds. **Reason and benefit** Fewer runs are cut off by each swap. There is never a moment without a server that starts runs. **Breaking changes** None. The migration adds two nullable columns. Releases without this code ignore the columns, and their rows never carry the marker, so a mixed-version overlap behaves as today. ## What Changed - `controller_instances`: new nullable columns `dispatching_at` and `dispatch_lease_expires_at` (migration `0471_controller_instances_dispatch_marker`, two `ADD COLUMN` statements, no lock beyond the catalog update). - New `server/src/services/run-start-handover.ts`. It is one worker on the shared delivery coordinator (`DELIVERY_QUEUES.runStartHandover`). Its renewal and recheck are coordinator deadlines, and a drain change or expiry wakes it. - Replacement side: `begin()` records the intent and writes the marker (it creates the boot's row if the row is missing). If the server drains at that moment, the worker writes the marker when the drain ends. The worker then publishes a `run-start-handover` hint and renews the 20 s lease every 5 s. It renews only while an older boot that is not stopped or fenced may still serve, and for at most 30 min. All marker writes run in one serialized chain. `release()` at shutdown lands after any renewal already in flight. A clear that fails is retried, and shutdown waits for the release. While the server cannot dispatch (drain, quiesce, admission closed), the worker clears the lease. - Old-server side: `isYielding()` is an in-memory read. The hint refreshes it, and so does one read that a dispatch pass waits for (at most 1 s) when the last read is more than 2 s old. A read that gets no answer within 1 s never keeps the server yielding. The server admits, and the yield also ends when the replacement's last seen lease runs out on the old server's own clock. Reads are a single flight on the dedicated connection. While it yields, the server rechecks every 3 s, calls `onResume` when the marker is gone, and sends a coalesced queued-run-dispatch hint for skipped starts. - It yields only to a newer epoch with a live marker. It never yields to an older boot, to itself, or to a row without the marker, and it does not yield without a row of its own. - **Database invariant:** every statement of the hand-over runs in a small transaction on the dedicated one-connection controller-lease pool (`controllerLeaseDb`) with a 1 s statement timeout. That covers reads, marker writes, the `run-start-handover` notification (commit-bound, in the marker write's own transaction) and the skipped-start hint. None of it touches the shared pool or registers tracked database work. The pool is shared only with legacy controller-lease renewals, and never with the liveness pool that native lease renewals use. A hand-over statement waits at most one legacy batch (10 s), inside the 15 s between a marker renewal and the marker's expiry. It delays a legacy renewal (60 s lease, renewed every 10 s) by at most 1 s. I did not piggyback on the controller-instance lease renewal, because that renewal runs only while the server holds work and only every 30 s. - **Import audit of `run-start-handover.ts`:** - `drizzle-orm` `sql`: builds statements, no I/O. - `controllerLeaseDb`: the dedicated connection, used by every statement here. - `notifyDatabaseWorkOnCommit` (new in `work-signals.ts`): runs `pg_notify` only on the transaction it is given and registers no intent, so the coordinator's `reconcileDatabaseWork` has nothing of ours to check. - `logger`: no I/O to the database. - `controller-process-identity`: in-memory boot id and host hash. - `delivery-work-coordinator`: type only. The worker is registered for lifecycle and deadlines; its `run` awaits only the serialized marker chain, and reads are not awaited. - `DELIVERY_QUEUES`: constants. - `task-admission` drain subscriptions: in-memory events. - `heartbeat.ts`: run admission is also closed while the server yields. A `runStartHandover` service option (default: the process's registered hand-over) lets one process host two assembled services in tests. `queue.ts`: `startNextQueuedRunForAgent` and `resumeQueuedRunsAfterDrain` first call `refreshRunAdmission()`. That call returns `null` (nothing to await) unless the hand-over read is stale. - `index.ts`: creates the hand-over only when the instance lease is on, the deployment notification channel exists, and the scheduler is enabled. It calls `begin()` after the RunDispatch startup pass and wakes that queue. It releases the hand-over when run admission closes at shutdown, and stops it before the lease stops. - `deployment-notifications.ts`: keeps the LISTEN connection open while a hand-over involves this server, so skipped starts and lease changes reach the other server at once. - `doc/execution-semantics.md`: new section "Run starts during a deploy overlap". It covers the marker, who yields, failure handling, what counts as starting a new run, and what a post-deploy audit should show. **Marker contract for other components (for example the deployer):** a replacement starts runs when its `controller_instances` row has an `epoch` greater than the old boot's, and `dispatching_at is not null and dispatch_lease_expires_at > now() and stopped_at is null and fenced_at is null`. `dispatching_at` stays set after the lease lapses, as the time the boot began starting runs. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-run-start-handover.test.ts`: assembled-app test with two `heartbeatService`s, each with its own pool, listener, coordinator and boot. A wake that the old server accepts stays queued there. The replacement's real queued-run dispatch starts it, and claims are observed per pool. After a replacement release, the old server starts the run itself. Red: with the yield removed from `isRunAdmissionClosed`, the old server claims the run. - `cd server && npx vitest run src/__tests__/run-start-handover.test.ts`: 19 tests pass against embedded PostgreSQL. Each test uses two servers, each with its own pool, boot id, LISTEN connection and delivery coordinator. The tests cover: - the old server is ready and the new server starts; the new server sets the marker; the old server yields; the queued run is claimed only by the new server, and a skipped start wakes the new server's dispatch - the old server learns from its own dispatch path without a notification: a read older than 2 s is refreshed before the start - the new server dies after it sets the marker: the old server resumes and runs its queued-run pass - the new server shuts down: the old server resumes at once - the new server drains: it releases the marker and takes it back afterwards - a replacement that starts up during a drain takes over when the drain ends - a shutdown release during an in-flight renewal stays released - a clear that fails at shutdown (statement timeout behind a row lock) is retried by a later release - an older server that outlives the lease TTL keeps yielding while the replacement renews - a yield that the old server cannot confirm within 1 s (its only connection is busy) does not stop starts (red/green) - a yield ends when the replacement's lease runs out, even while every read waits for a connection (red/green) - renewals blocked past the TTL hand starts back once, then the replacement takes them again (bounded flapping) - a clear that fails when the replacement starts draining is retried well inside the lease (red/green) - with both servers' shared pools blocked past the TTL, the marker is written, renewed and released, and the old server yields and resumes, all before the block ends (red when hand-over statements use the shared pool) - an older-release row without the marker changes nothing - a server never yields to an older boot or to itself, and does not yield without a row of its own - a lone server writes once, never renews, and admits from memory - without a hand-over (self-hosted) there is nothing to await - Red: on `origin/master` the test file cannot run, because the module and the columns do not exist. The old server has no signal and keeps starting the handed-over run. - `cd server && npx vitest run src/__tests__/deployment-notifications.test.ts src/services/heartbeat/dispatch-work.test.ts src/__tests__/delivery-work-coordinator.test.ts src/__tests__/controller-instance-lease.test.ts src/shutdown.test.ts`: all pass. - `cd packages/db && pnpm check:migrations`: safety and compatibility checks pass. `cd server && npx tsc --noEmit -p .`: clean. - `pnpm verify` asks for broad verification because `doc/execution-semantics.md` is a global CI input. Required CI supplies that coverage. ## Risks - Run starts move from the old server to the new server during the overlap. If the new server cannot dispatch, the old server resumes on the lease clear (about 3 s) or the lease lapse (up to about 23 s after a crash). If a hand-over read fails, the server keeps admitting. - If the old server was not listening for notifications, it can start runs for up to about 2 s after the marker (the age of its last read). On a dispatching server, the pre-dispatch read adds one small bounded query at most every 2 s. - An older boot that has been idle for more than 30 min counts as gone. The new server then does not renew, and the old server admits again as today. - The migration adds nullable columns only. The deployer can apply it while the previous release serves. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
492 lines
24 KiB
TypeScript
492 lines
24 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { and, eq, sql } from "drizzle-orm";
|
|
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import { agents, companies, controllerInstances, controllerLeaseDb, createDb, heartbeatRuns } from "@paperclipai/db";
|
|
import { DATABASE_WORK_CHANNEL, enableDatabaseWorkNotifications, receiveDatabaseWorkNotification,
|
|
subscribeDatabaseWork } from "@paperclipai/db/work-signals";
|
|
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
|
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
|
|
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
|
import { readTaskDrain, startTaskDrain, stopTaskDrain } from "../services/task-admission.js";
|
|
import { createRunStartHandover, isRunStartHandoverActive, isYieldingRunStarts, refreshRunStartYieldIfStale,
|
|
registerRunStartHandover, type RunStartHandover } from "../services/run-start-handover.js";
|
|
|
|
const support = await getEmbeddedPostgresTestSupport();
|
|
const suite = support.supported ? describe : describe.skip;
|
|
|
|
// Short timings keep the production order of events while the test waits seconds, not minutes.
|
|
const timing = { renewMs: 200, ttlMs: 1_500, recheckMs: 150, staleAwaitMs: 100, skipHintMinMs: 50 };
|
|
|
|
type Server = {
|
|
name: string;
|
|
db: ReturnType<typeof createDb>;
|
|
handover: RunStartHandover;
|
|
resumed: ReturnType<typeof vi.fn>;
|
|
dispatchHints: number;
|
|
stop: () => Promise<void>;
|
|
};
|
|
|
|
suite("run start hand-over between an old and a new server during a deploy overlap", () => {
|
|
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
|
let admin!: ReturnType<typeof createDb>;
|
|
let companyId = "";
|
|
let agentId = "";
|
|
const servers: Server[] = [];
|
|
const claims: Array<{ server: string; runId: string }> = [];
|
|
|
|
beforeAll(async () => {
|
|
temporary = await startEmbeddedPostgresTestDatabase("paperclip-run-start-handover-");
|
|
admin = createDb(temporary.connectionString);
|
|
companyId = randomUUID();
|
|
agentId = randomUUID();
|
|
await admin.insert(companies).values({ id: companyId, name: "Paperclip", issuePrefix: "RSH", requireBoardApprovalForNewAgents: false });
|
|
await admin.insert(agents).values({ id: agentId, companyId, name: "Builder", adapterType: "codex_local", status: "idle" });
|
|
}, 30_000);
|
|
|
|
afterEach(async () => {
|
|
registerRunStartHandover(null);
|
|
for (const server of servers.splice(0)) await server.stop();
|
|
claims.length = 0;
|
|
await admin.delete(heartbeatRuns);
|
|
await admin.delete(controllerInstances);
|
|
});
|
|
|
|
afterAll(async () => { await temporary?.cleanup(); });
|
|
|
|
/** A server process: its own pool, its own boot id, and the cross-server notification listener. */
|
|
async function bootServer(name: string, options: {
|
|
canDispatch?: () => boolean; listen?: boolean; maxConnections?: number;
|
|
timing?: Partial<typeof timing> & { writeStatementTimeoutMs?: number; readWaitMs?: number };
|
|
} = {}): Promise<Server> {
|
|
const canDispatch = options.canDispatch ?? (() => true);
|
|
const db = createDb(temporary!.connectionString, options.maxConnections ? { maxConnections: options.maxConnections } : undefined);
|
|
enableDatabaseWorkNotifications(db);
|
|
const listener = options.listen === false ? null : createDb(temporary!.connectionString, { maxConnections: 1 }).$client;
|
|
await listener?.listen(DATABASE_WORK_CHANNEL, payload => receiveDatabaseWorkNotification(db, payload));
|
|
const resumed = vi.fn();
|
|
const errors: unknown[] = [];
|
|
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
|
|
const handover = createRunStartHandover({ db, coordinator, canDispatch, onResume: resumed, bootId: `boot-${name}-${randomUUID()}`,
|
|
processSpace: name, pid: 1, timing: { ...timing, ...options.timing } });
|
|
await handover.ready;
|
|
const server: Server = { name, db, handover, resumed, dispatchHints: 0, stop: async () => {} };
|
|
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.queuedRunDispatch, event => {
|
|
if (event === "settled") server.dispatchHints += 1;
|
|
});
|
|
server.stop = async () => {
|
|
unsubscribe();
|
|
await handover.stop();
|
|
await coordinator.stop();
|
|
expect(errors).toEqual([]);
|
|
await listener?.end({ timeout: 1 });
|
|
await db.$client.end({ timeout: 1 });
|
|
};
|
|
servers.push(server);
|
|
return server;
|
|
}
|
|
|
|
async function queueRun(): Promise<string> {
|
|
const id = randomUUID();
|
|
await admin.insert(heartbeatRuns).values({ id, companyId, agentId, status: "queued", invocationSource: "on_demand" });
|
|
return id;
|
|
}
|
|
|
|
/** The queue's start path: refresh a stale answer, read the in-memory flag, then the atomic claim. */
|
|
async function dispatch(server: Server, runId: string): Promise<boolean> {
|
|
const refreshing = server.handover.refreshIfStale();
|
|
if (refreshing) await refreshing;
|
|
if (server.handover.isYielding()) return false;
|
|
const [row] = await server.db.update(heartbeatRuns).set({ status: "running", startedAt: new Date() })
|
|
.where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.status, "queued"))).returning({ id: heartbeatRuns.id });
|
|
if (row) claims.push({ server: server.name, runId });
|
|
return Boolean(row);
|
|
}
|
|
|
|
async function marker(server: Server) {
|
|
const [row] = await admin.execute(sql`select dispatching_at, dispatch_lease_expires_at > now() as live
|
|
from controller_instances where process_space = ${server.name}`) as unknown as Array<{ dispatching_at: Date | null; live: boolean | null }>;
|
|
return row;
|
|
}
|
|
|
|
it("the old server stops starting runs once the new one starts them, and the new one starts the queued run", async () => {
|
|
// The old server booted alone long ago: it recorded its own start and holds no hand-over.
|
|
const old = await bootServer("old");
|
|
await old.handover.begin();
|
|
expect(old.handover.isHandingOver()).toBe(false);
|
|
const before = await queueRun();
|
|
expect(await dispatch(old, before)).toBe(true);
|
|
|
|
// The new server boots; while it is still starting up the old one keeps starting runs.
|
|
const next = await bootServer("new");
|
|
const duringStartup = await queueRun();
|
|
expect(await dispatch(old, duringStartup)).toBe(true);
|
|
|
|
// Its queued-run dispatch is running: it records that and the old server learns from the notification.
|
|
await next.handover.begin();
|
|
expect(next.handover.isHandingOver()).toBe(true);
|
|
expect(await marker(next)).toMatchObject({ live: true });
|
|
await vi.waitFor(() => expect(old.handover.isActive()).toBe(true), { timeout: 10_000 });
|
|
|
|
const handedOver = await queueRun();
|
|
const hintsBefore = next.dispatchHints;
|
|
expect(await dispatch(old, handedOver)).toBe(false);
|
|
// The skipped start wakes the new server's queued-run dispatch instead of waiting for a sweep.
|
|
await vi.waitFor(() => expect(next.dispatchHints).toBeGreaterThan(hintsBefore), { timeout: 10_000 });
|
|
expect(await dispatch(next, handedOver)).toBe(true);
|
|
expect(claims.filter(claim => claim.runId === handedOver)).toEqual([{ server: "new", runId: handedOver }]);
|
|
|
|
// The old server's SIGTERM ends the hand-over; the new server stops renewing without further writes.
|
|
await admin.update(controllerInstances).set({ stoppedAt: sql`now()` }).where(eq(controllerInstances.processSpace, "old"));
|
|
await vi.waitFor(() => expect(next.handover.isHandingOver()).toBe(false), { timeout: 10_000 });
|
|
});
|
|
|
|
it("the old server learns of the hand-over on its next start even without a notification", async () => {
|
|
// The old server cannot hear this replacement: only its own dispatch path can tell it.
|
|
const old = await bootServer("old", { listen: false });
|
|
await old.handover.begin();
|
|
// Its last read saw no replacement; then the replacement takes over while the old server is idle.
|
|
await old.handover.refresh();
|
|
expect(old.handover.isActive()).toBe(false);
|
|
const next = await bootServer("new");
|
|
await next.handover.begin();
|
|
await new Promise(resolve => setTimeout(resolve, timing.staleAwaitMs + 50));
|
|
// The next start waits for one fresh read instead of trusting the stale one.
|
|
const run = await queueRun();
|
|
expect(await dispatch(old, run)).toBe(false);
|
|
expect(old.handover.isActive()).toBe(true);
|
|
expect(await dispatch(next, run)).toBe(true);
|
|
});
|
|
|
|
it("a yield the old server cannot confirm in time does not stop it from starting runs", async () => {
|
|
// A small pool that is busy: the old server's read waits for a connection.
|
|
const old = await bootServer("old", { maxConnections: 1, timing: { readWaitMs: 200 } });
|
|
await old.handover.begin();
|
|
const next = await bootServer("new");
|
|
await next.handover.begin();
|
|
await vi.waitFor(() => expect(old.handover.isActive()).toBe(true), { timeout: 10_000 });
|
|
|
|
// The replacement dies, and the old server's only connection is busy past the wait.
|
|
// A drizzle query runs only once awaited: start it now and let it take the connection.
|
|
const busy = controllerLeaseDb(old.db).execute(sql`select pg_sleep(1.5)`).then(() => undefined);
|
|
await new Promise(resolve => setTimeout(resolve, 50));
|
|
await next.handover.stop();
|
|
expect(await marker(next)).toMatchObject({ live: null });
|
|
await new Promise(resolve => setTimeout(resolve, timing.staleAwaitMs + 50));
|
|
const run = await queueRun();
|
|
const started = Date.now();
|
|
expect(await dispatch(old, run)).toBe(true);
|
|
expect(Date.now() - started).toBeLessThan(5_000);
|
|
await busy;
|
|
});
|
|
|
|
it("a yield ends when the replacement's lease runs out, even while every read waits for a connection", async () => {
|
|
const old = await bootServer("old", { maxConnections: 1 });
|
|
await old.handover.begin();
|
|
// A replacement that recorded its start and then died: its lease is never renewed.
|
|
await admin.execute(sql`insert into controller_instances (boot_id, process_space, pid, lease_expires_at, dispatching_at, dispatch_lease_expires_at)
|
|
values (${`boot-crashed-${randomUUID()}`}, 'crashed', 9, now(), now(), now() + ${`${timing.ttlMs} milliseconds`}::interval)`);
|
|
await old.handover.refresh();
|
|
expect(old.handover.isActive()).toBe(true);
|
|
|
|
// The old server's only connection stays busy well past the lease: no read can answer.
|
|
const busy = controllerLeaseDb(old.db).execute(sql`select pg_sleep(${(timing.ttlMs + 3_000) / 1000})`).then(() => undefined);
|
|
await new Promise(resolve => setTimeout(resolve, 50));
|
|
const resumedAt = await vi.waitFor(() => {
|
|
expect(old.resumed).toHaveBeenCalledTimes(1);
|
|
return Date.now();
|
|
}, { timeout: timing.ttlMs + 2_000 });
|
|
let busyDone = false;
|
|
void busy.then(() => { busyDone = true; });
|
|
expect(busyDone).toBe(false);
|
|
expect(old.handover.isYielding()).toBe(false);
|
|
expect(resumedAt).toBeGreaterThan(0);
|
|
await busy;
|
|
});
|
|
|
|
it("renewals blocked past the TTL hand starts back once, then the replacement takes them again", async () => {
|
|
const old = await bootServer("old");
|
|
await old.handover.begin();
|
|
const next = await bootServer("new", { timing: { writeStatementTimeoutMs: 300 } });
|
|
await next.handover.begin();
|
|
await vi.waitFor(() => expect(old.handover.isActive()).toBe(true), { timeout: 10_000 });
|
|
|
|
// Every renewal is cancelled while the replacement's row is held, until its lease lapses.
|
|
let unlock!: () => void;
|
|
const locked = new Promise<void>(resolve => { unlock = resolve; });
|
|
let held!: () => void;
|
|
const holding = new Promise<void>(resolve => { held = resolve; });
|
|
const lock = admin.transaction(async tx => {
|
|
await tx.execute(sql`select 1 from controller_instances where process_space = 'new' for update`);
|
|
held();
|
|
await locked;
|
|
});
|
|
await holding;
|
|
await vi.waitFor(() => expect(old.resumed).toHaveBeenCalledTimes(1), { timeout: 10_000 });
|
|
expect(old.handover.isActive()).toBe(false);
|
|
expect(await dispatch(old, await queueRun())).toBe(true);
|
|
|
|
unlock();
|
|
await lock;
|
|
// The first renewal that succeeds again announces the marker, and the old server leaves starts to it.
|
|
await vi.waitFor(() => expect(old.handover.isActive()).toBe(true), { timeout: 10_000 });
|
|
expect(old.resumed).toHaveBeenCalledTimes(1);
|
|
const run = await queueRun();
|
|
expect(await dispatch(old, run)).toBe(false);
|
|
expect(await dispatch(next, run)).toBe(true);
|
|
});
|
|
|
|
it("with both servers' shared pools blocked past the TTL, the marker is written, renewed and released on time", async () => {
|
|
const old = await bootServer("old", { maxConnections: 1 });
|
|
await old.handover.begin();
|
|
const next = await bootServer("new", { maxConnections: 1 });
|
|
// Every shared-pool connection on both servers is busy for longer than the whole hand-over below.
|
|
const blockMs = timing.ttlMs * 4;
|
|
const busy = Promise.all([old.db, next.db].map(server =>
|
|
server.execute(sql`select pg_sleep(${blockMs / 1000})`).then(() => undefined)));
|
|
await new Promise(resolve => setTimeout(resolve, 50));
|
|
const blockedUntil = Date.now() + blockMs - 100;
|
|
|
|
await next.handover.begin();
|
|
expect(await marker(next)).toMatchObject({ live: true });
|
|
await vi.waitFor(() => expect(old.handover.isActive()).toBe(true), { timeout: 1_000 });
|
|
await new Promise(resolve => setTimeout(resolve, timing.ttlMs * 1.5));
|
|
expect(await marker(next)).toMatchObject({ live: true });
|
|
expect(old.handover.isActive()).toBe(true);
|
|
expect(old.resumed).not.toHaveBeenCalled();
|
|
|
|
const started = Date.now();
|
|
await next.handover.release();
|
|
expect(Date.now() - started).toBeLessThan(1_000);
|
|
expect(await marker(next)).toMatchObject({ live: null });
|
|
await vi.waitFor(() => expect(old.resumed).toHaveBeenCalledTimes(1), { timeout: 1_000 });
|
|
// All of the above happened while both shared pools were still blocked.
|
|
expect(Date.now()).toBeLessThan(blockedUntil);
|
|
await busy;
|
|
});
|
|
|
|
it("an older server that outlives the lease TTL keeps leaving starts to the renewing replacement", async () => {
|
|
const old = await bootServer("old");
|
|
await old.handover.begin();
|
|
const next = await bootServer("new");
|
|
await next.handover.begin();
|
|
await vi.waitFor(() => expect(old.handover.isActive()).toBe(true), { timeout: 10_000 });
|
|
await new Promise(resolve => setTimeout(resolve, timing.ttlMs * 2));
|
|
expect(next.handover.isHandingOver()).toBe(true);
|
|
expect(await marker(next)).toMatchObject({ live: true });
|
|
expect(old.resumed).not.toHaveBeenCalled();
|
|
const run = await queueRun();
|
|
expect(await dispatch(old, run)).toBe(false);
|
|
expect(await dispatch(next, run)).toBe(true);
|
|
});
|
|
|
|
it("a lease clear that fails at shutdown is retried by a later release", async () => {
|
|
const old = await bootServer("old");
|
|
await old.handover.begin();
|
|
const next = await bootServer("new", { timing: { writeStatementTimeoutMs: 300 } });
|
|
await next.handover.begin();
|
|
await vi.waitFor(() => expect(old.handover.isActive()).toBe(true), { timeout: 10_000 });
|
|
|
|
// The clear cannot take the row while it is held, and its statement timeout cancels it.
|
|
let unlock!: () => void;
|
|
const locked = new Promise<void>(resolve => { unlock = resolve; });
|
|
let held!: () => void;
|
|
const holding = new Promise<void>(resolve => { held = resolve; });
|
|
const lock = admin.transaction(async tx => {
|
|
await tx.execute(sql`select 1 from controller_instances where process_space = 'new' for update`);
|
|
held();
|
|
await locked;
|
|
});
|
|
await holding;
|
|
await next.handover.release();
|
|
unlock();
|
|
await lock;
|
|
expect(await marker(next)).toMatchObject({ live: true });
|
|
expect(old.resumed).not.toHaveBeenCalled();
|
|
|
|
await next.handover.release();
|
|
expect(await marker(next)).toMatchObject({ live: null });
|
|
await vi.waitFor(() => expect(old.resumed).toHaveBeenCalledTimes(1), { timeout: 10_000 });
|
|
});
|
|
|
|
it("the old server starts runs again within seconds when the new server dies after taking over", async () => {
|
|
const old = await bootServer("old");
|
|
await old.handover.begin();
|
|
// A replacement that recorded its start and then died: its hand-over lease is never renewed.
|
|
await admin.execute(sql`insert into controller_instances (boot_id, process_space, pid, lease_expires_at, dispatching_at, dispatch_lease_expires_at)
|
|
values (${`boot-crashed-${randomUUID()}`}, 'crashed', 9, now(), now(), now() + ${`${timing.ttlMs} milliseconds`}::interval)`);
|
|
await old.handover.refresh();
|
|
expect(old.handover.isActive()).toBe(true);
|
|
const run = await queueRun();
|
|
expect(await dispatch(old, run)).toBe(false);
|
|
|
|
// Once the lease lapses the old server resumes and re-runs its queued-run pass.
|
|
await vi.waitFor(() => expect(old.resumed).toHaveBeenCalledTimes(1), { timeout: 10_000 });
|
|
expect(old.handover.isActive()).toBe(false);
|
|
expect(await dispatch(old, run)).toBe(true);
|
|
});
|
|
|
|
it("the old server resumes at once when the new server shuts down (rollback or failed deploy)", async () => {
|
|
const old = await bootServer("old");
|
|
await old.handover.begin();
|
|
const next = await bootServer("new");
|
|
await next.handover.begin();
|
|
await vi.waitFor(() => expect(old.handover.isActive()).toBe(true), { timeout: 10_000 });
|
|
|
|
await next.handover.release();
|
|
expect(await marker(next)).toMatchObject({ live: null });
|
|
await vi.waitFor(() => expect(old.resumed).toHaveBeenCalledTimes(1), { timeout: 10_000 });
|
|
const run = await queueRun();
|
|
expect(await dispatch(old, run)).toBe(true);
|
|
});
|
|
|
|
it("the new server releases the hand-over while it drains and takes it back when the drain ends", async () => {
|
|
const old = await bootServer("old");
|
|
await old.handover.begin();
|
|
const next = await bootServer("new", { canDispatch: () => readTaskDrain(new Date()) === null });
|
|
await next.handover.begin();
|
|
await vi.waitFor(() => expect(old.handover.isActive()).toBe(true), { timeout: 10_000 });
|
|
|
|
try {
|
|
startTaskDrain();
|
|
await vi.waitFor(() => expect(old.resumed).toHaveBeenCalledTimes(1), { timeout: 10_000 });
|
|
expect(await marker(next)).toMatchObject({ live: null });
|
|
} finally { stopTaskDrain(); }
|
|
await vi.waitFor(() => expect(old.handover.isActive()).toBe(true), { timeout: 10_000 });
|
|
});
|
|
|
|
it("a clear that fails when the replacement starts draining is retried well inside the lease", async () => {
|
|
const old = await bootServer("old");
|
|
await old.handover.begin();
|
|
// A long lease, so the old server can only resume early through a retried clear.
|
|
const next = await bootServer("new", { canDispatch: () => readTaskDrain(new Date()) === null,
|
|
timing: { ttlMs: 8_000, writeStatementTimeoutMs: 300 } });
|
|
await next.handover.begin();
|
|
await vi.waitFor(() => expect(old.handover.isActive()).toBe(true), { timeout: 10_000 });
|
|
|
|
let unlock!: () => void;
|
|
const locked = new Promise<void>(resolve => { unlock = resolve; });
|
|
let held!: () => void;
|
|
const holding = new Promise<void>(resolve => { held = resolve; });
|
|
const lock = admin.transaction(async tx => {
|
|
await tx.execute(sql`select 1 from controller_instances where process_space = 'new' for update`);
|
|
held();
|
|
await locked;
|
|
});
|
|
await holding;
|
|
try {
|
|
startTaskDrain();
|
|
// The first clear is cancelled by its statement timeout while the row is held.
|
|
await new Promise(resolve => setTimeout(resolve, 700));
|
|
expect(await marker(next)).toMatchObject({ live: true });
|
|
unlock();
|
|
await lock;
|
|
const unlockedAt = Date.now();
|
|
await vi.waitFor(() => expect(old.resumed).toHaveBeenCalledTimes(1), { timeout: 10_000 });
|
|
expect(Date.now() - unlockedAt).toBeLessThan(4_000);
|
|
expect(await marker(next)).toMatchObject({ live: null });
|
|
} finally {
|
|
unlock();
|
|
stopTaskDrain();
|
|
}
|
|
});
|
|
|
|
it("a replacement that starts up during a drain takes run starts over once the drain ends", async () => {
|
|
const old = await bootServer("old");
|
|
await old.handover.begin();
|
|
const next = await bootServer("new", { canDispatch: () => readTaskDrain(new Date()) === null });
|
|
try {
|
|
startTaskDrain();
|
|
await next.handover.begin();
|
|
expect(await marker(next)).toBeUndefined();
|
|
await old.handover.refresh();
|
|
expect(old.handover.isActive()).toBe(false);
|
|
} finally { stopTaskDrain(); }
|
|
await vi.waitFor(() => expect(old.handover.isActive()).toBe(true), { timeout: 10_000 });
|
|
expect(await marker(next)).toMatchObject({ live: true });
|
|
});
|
|
|
|
it("a shutdown release during an in-flight renewal is not undone by that renewal", async () => {
|
|
const old = await bootServer("old");
|
|
await old.handover.begin();
|
|
const next = await bootServer("new");
|
|
await next.handover.begin();
|
|
await vi.waitFor(() => expect(old.handover.isActive()).toBe(true), { timeout: 10_000 });
|
|
|
|
// Hold the replacement's row so its next renewal is in flight when shutdown releases the marker.
|
|
let unlock!: () => void;
|
|
const locked = new Promise<void>(resolve => { unlock = resolve; });
|
|
let held!: () => void;
|
|
const holding = new Promise<void>(resolve => { held = resolve; });
|
|
const lock = admin.transaction(async tx => {
|
|
await tx.execute(sql`select 1 from controller_instances where process_space = 'new' for update`);
|
|
held();
|
|
await locked;
|
|
});
|
|
await holding;
|
|
await new Promise(resolve => setTimeout(resolve, timing.renewMs * 2));
|
|
const release = next.handover.release();
|
|
unlock();
|
|
await lock;
|
|
await release;
|
|
expect(await marker(next)).toMatchObject({ live: null });
|
|
await vi.waitFor(() => expect(old.resumed).toHaveBeenCalledTimes(1), { timeout: 10_000 });
|
|
await new Promise(resolve => setTimeout(resolve, timing.renewMs * 3));
|
|
expect(await marker(next)).toMatchObject({ live: null });
|
|
expect(old.handover.isActive()).toBe(false);
|
|
});
|
|
|
|
it("an instance from an older release, without the marker, changes nothing", async () => {
|
|
const old = await bootServer("old");
|
|
await old.handover.begin();
|
|
// A newer boot from a release without this code: a lease row, no dispatching_at.
|
|
await admin.insert(controllerInstances).values({ bootId: `boot-legacy-${randomUUID()}`, processSpace: "legacy", pid: 3,
|
|
leaseExpiresAt: sql`now() + interval '90 seconds'` });
|
|
await old.handover.refresh();
|
|
expect(old.handover.isActive()).toBe(false);
|
|
expect(await dispatch(old, await queueRun())).toBe(true);
|
|
});
|
|
|
|
it("a server never yields to an older instance or to itself", async () => {
|
|
const older = await bootServer("older");
|
|
await older.handover.begin();
|
|
const current = await bootServer("current");
|
|
await current.handover.begin();
|
|
// The older boot still advertises a live hand-over lease (for example, from its own overlap).
|
|
await admin.update(controllerInstances).set({ dispatchLeaseExpiresAt: sql`now() + interval '1 minute'` });
|
|
await current.handover.refresh();
|
|
expect(current.handover.isActive()).toBe(current.handover.isHandingOver());
|
|
expect(current.handover.isYielding()).toBe(false);
|
|
expect(await dispatch(current, await queueRun())).toBe(true);
|
|
});
|
|
|
|
it("a server that has no row of its own never yields", async () => {
|
|
const next = await bootServer("new");
|
|
await next.handover.begin();
|
|
const unknown = await bootServer("unknown");
|
|
await unknown.handover.refresh();
|
|
expect(unknown.handover.isYielding()).toBe(false);
|
|
});
|
|
|
|
it("a lone server records its start once and does not renew or read on the start path", async () => {
|
|
const only = await bootServer("only");
|
|
await only.handover.begin();
|
|
expect(only.handover.isHandingOver()).toBe(false);
|
|
const first = await admin.select({ lease: controllerInstances.dispatchLeaseExpiresAt }).from(controllerInstances);
|
|
await new Promise(resolve => setTimeout(resolve, timing.renewMs * 3));
|
|
const later = await admin.select({ lease: controllerInstances.dispatchLeaseExpiresAt }).from(controllerInstances);
|
|
expect(later).toEqual(first);
|
|
|
|
// Once read, a start is admitted from memory: nothing to await.
|
|
await only.handover.refresh();
|
|
expect(only.handover.refreshIfStale()).toBeNull();
|
|
expect(await dispatch(only, await queueRun())).toBe(true);
|
|
});
|
|
|
|
it("a process without a hand-over (self-hosted, lease off) admits from memory with nothing to await", () => {
|
|
registerRunStartHandover(null);
|
|
expect(isYieldingRunStarts()).toBe(false);
|
|
expect(refreshRunStartYieldIfStale()).toBeNull();
|
|
expect(isRunStartHandoverActive()).toBe(false);
|
|
});
|
|
});
|