mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A release swap runs two servers for a short overlap. The old server drains, and the new server takes over the runs. > - Each server proves that it is alive with a controller-instance lease. Recovery uses that lease to decide whether a run's controller is gone. > - In local swap cells, a healthy old server exited during the overlap, because its lease renewal waited for a connection on the busy main pool. Every run it drove was interrupted. > - When a controller crashed, its runs waited for another server boot, or for a timer that restarted on every look. Tasks stood still for 21 to 28 minutes. > - This pull request renews the lease on its own connection, and lets the live server end a crashed controller's runs within a bounded time. > - The benefit is that a swap does not kill a healthy server, and a crashed server's runs end in about 2 minutes, so their tasks continue by themselves. ## Linked Issues or Issue Description Refs #21135, #21222 **What happened?** 1. During a two-server release swap (local cells, load 14 to 34), the outgoing server exited with "controller instance lease lost while holding work". Five lease renewals in a row timed out after 10 s. The 10 s timeout included the wait for a connection on the shared main pool, which the traffic switch and the replacement's startup pass kept busy. 2. When the old server was killed in the middle of the overlap, its runs stayed in progress for 21 to 28 minutes. The live server did not re-check a crashed controller's runs: only a new boot did. The unverified-controller wait started when a server first looked at the run, not when the controller left, and the successor's own wait record counted as the dead owner's sign of life. **Expected behavior** A healthy server keeps its lease during a swap. When a controller is gone (its lease lapsed and it is fenced or stopped), the live server ends that controller's runs within a bounded time, without another server boot, so restart recovery continues the tasks. **Steps to reproduce** 1. Run two servers against one database with a release swap and a 200 s overlap. 2. Kill the old server 40 s into the overlap (SIGKILL). 3. See that its runs stay in progress for more than 20 minutes, until another server starts. ## What Changed - Lease renewal on its own connection (`packages/db/src/client.ts`, `packages/db/src/index.ts`, `server/src/services/controller-instance-lease.ts`): - New `controllerInstanceLeaseDb(db)`: one private 1-connection pool (`application_name` `paperclip-controller-instance-lease`), made the same way as the existing `controllerLeaseDb` and closed by `closeLivenessDb`. - The renewal is one statement on that connection, as before. - The connection sends `statement_timeout` = 8 s (below the 10 s client timeout) as a startup parameter (new `DatabaseClientOptions.statementTimeoutMs`, used only by this pool). The server cancels a slow renewal, so it never holds the single connection that the next renewal needs, and there is no extra round trip per renewal. (An earlier revision set it per transaction; that added three round trips to every renewal and broke the idle-server receipt test.) - If a connection pooler refuses the startup parameter ("unsupported startup parameter"), the lease logs a warning and renews on a replacement connection without it. - A crashed controller's runs (`server/src/services/recovery/ownership-registry.ts`, `server/src/index.ts`, `server/src/services/native-runtime/native-restart-recovery.ts`): - The ownership reconciler on the live server hands runs whose controller is proven gone to restart recovery. It does not wait for another server boot. - The unverified-controller wait is bounded from the moment the controller's lease lapsed, not from the first look. - When the controller boot is `stopped` or `fenced`, the run ends at once. - A successor's wait record no longer refreshes the dead owner's sign of life. Safety of ending a run when its controller boot is fenced or stopped: - Only `stopped` (written by the boot's own graceful stop) and `fenced` (a newer epoch fenced it after its lease expired plus 30 s grace) count. `live` and `grace` never count, so a controller that is alive and renewing is never treated as gone. `expired` alone does not count either. - A boot that cannot renew stops itself at its self-fence window (about its lease expiry), and a boot that finds itself fenced exits at once. So when another server can fence it, the old process has already stopped driving runs. The lease change above removes the main cause of healthy boots failing to renew. - If the old process is still alive (partition or stall) and its Runner continues: the old run is fenced. Its agent token is refused for writes (`agent_run_superseded`), its remote workspace epoch is abandoned or the continuation gets a new workspace, and the native coordinator records `replacementDenied`. There are never two writers in one workspace. ## Verification Red-to-green tests (each failed before its fix and passes after it): - `server/src/__tests__/controller-instance-lease.test.ts` "own connection": red (10004 ms timeout while the main pool is held) to green. Full file green (26). - `server/src/__tests__/controller-instance-lease.test.ts` "has the server cancel a renewal blocked on a row lock, and renews on the same connection next time": red without the connection-level timeout (the renewal runs to the 10 s client timeout and stays active), green with it (cancelled at about 8 s, no active lease statement left, the next renewal succeeds in under 2 s on the same backend). - `packages/db/src/client-options.test.ts`: the statement timeout is sent as a startup parameter. - `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`: "lapsed longer ago" / "end of the bound" (wait counted from the lease lapse), the live-server re-check of a crashed controller, the fenced/stopped boot ending the run at once, and the successor's wait record. Full file green. Changed existing expectation (it asserted the old wait for a stopped or fenced controller): - `server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts` "keeps foreign local Runner and provider ownership after its controller is stopped/fenced": the disposition is now `blocked` (`controller_process_space_unverified`) instead of `awaiting_evidence`, the run is `failed` with `native_restart_recovery_blocked`, and the finalization is `terminal_failure` with `executorStopEvidence: "unverified"`. The safety checks stay: no local PID probe, no claim, no stamp, no local stop proof, the run keeps its process identity and session checkpoint, and no result is written. Local commands, on this branch based on current master: - `cd server && npx vitest run src/__tests__/controller-instance-lease.test.ts src/__tests__/heartbeat-native-restart-suspension.test.ts`: 2 files, 140 tests passed. - `npx vitest run src/__tests__/controller-instance-lease.test.ts src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`: 225 passed, 14 skipped (real-process cases that need CI). - `pnpm exec vitest run src/__tests__/maintenance-server-routes.test.ts -t "releases an ordinary held task Retry"`: passed (idle receipt: one lease statement after the work ends). Local swap cells (two servers, embedded Postgres, real agents, old server SIGKILLed 40 s into a 200 s overlap): - Before: runs of the killed server stayed in progress 21 to 28 minutes. - N1 (2 tasks): 2/2 finished hands-off. Kill to run end 5 min 31 s (the successor's wait write still moved the timer; fixed by the last commit). - R1 (6 tasks, 3 adapters, subtasks, follow-ups, 2 swaps): 4/4 hands-off, 0 banners, 0 duplicates, 0 invariant violations. Kill to run end 3 min 39 s. - N2 (same seed as N1, all fixes): 2/2 hands-off. Kill to run end 2 min 01 s; fresh runs started at 2 min 32 s and 3 min 02 s. The floor is the lease TTL (90 s) plus the fence grace (30 s). - Hour-long release-swap wave (w-20261112T233630-s7201) on the branch that holds this fix and the related recovery fixes: 11/11 tasks finished hands-off (master: 9/11). The lease and crashed-controller fixes were visibly working in it. ## Risks - A pooled database URL whose pooler refuses `statement_timeout` as a startup parameter: the first renewal fails over to a connection without it (logged), so the lease keeps working without the server-side cancel. - One extra database connection per server, held while the server runs. It is idle except for one renewal every few seconds. - A run whose controller boot is fenced or stopped now ends at once instead of after the bound. See the safety notes above: only a boot that is not alive can be in these states. - No migration. No schema change. Nothing changes before `turn.start` on the normal path. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no documented behavior or command changes) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
314 lines
17 KiB
TypeScript
314 lines
17 KiB
TypeScript
import { and, eq, isNull, sql } from "drizzle-orm";
|
|
import { controllerInstanceLeaseDb, controllerInstanceLeaseDbWithoutStatementTimeout, controllerInstances, type Db } from "@paperclipai/db";
|
|
import { logger } from "../middleware/logger.js";
|
|
import {
|
|
currentControllerBootId, currentProcessSpaceHash, onControllerStamp,
|
|
} from "./controller-process-identity.js";
|
|
import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
|
|
import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
|
|
import { setControllerLeaseQuiesce, subscribeIdleWorkSettled } from "./task-admission.js";
|
|
|
|
export const CONTROLLER_LEASE_TTL_MS = 90_000;
|
|
export const CONTROLLER_LEASE_RENEW_MS = 30_000;
|
|
/** Renewal attempts start failing from their own start; this boot stops after this long while holding work. */
|
|
export const CONTROLLER_LEASE_SELF_FENCE_MS = 60_000;
|
|
/** A prover fences a boot only this long after the boot's last sign of life. */
|
|
export const CONTROLLER_LEASE_FENCE_GRACE_MS = 30_000;
|
|
/** A stamp counts as held work for a full lease, so a slow ownership write commits before the lease can lapse. */
|
|
const STAMP_COMMIT_GRACE_MS = CONTROLLER_LEASE_TTL_MS;
|
|
/** A settle of other work re-checks holdings at most this often. */
|
|
const SETTLE_RECHECK_MIN_MS = 5_000;
|
|
/** A renewal that has not answered within this bound counts as failed. */
|
|
const RENEW_TIMEOUT_MS = 10_000;
|
|
/** Server-side cancel for a renewal statement, below the client timeout. */
|
|
export const RENEW_STATEMENT_TIMEOUT_MS = 8_000;
|
|
const RETRY_MS = 5_000;
|
|
|
|
export type ControllerInstanceState = "live" | "grace" | "expired" | "fenced" | "stopped";
|
|
/** "holding" saw an in-progress record held by this boot; "holding_stamp" held only for a recent stamp. */
|
|
type RenewResult = "holding" | "holding_stamp" | "idle" | "fenced";
|
|
/** A stamp whose record no renewal has seen keeps the lease held until the server's tracked work next settles
|
|
* (its write then committed or rolled back), and at most this long. */
|
|
const UNOBSERVED_STAMP_CAP_MS = 10 * 60_000;
|
|
|
|
type LeaseState = {
|
|
onLost: (reason: "fenced" | "lease_lost") => void;
|
|
now: () => number;
|
|
renew: (stampedRecently: boolean) => Promise<RenewResult>;
|
|
lastStampAt: number;
|
|
/** Start of the first renewal attempt in the current failing streak. */
|
|
failingSince: number | null;
|
|
/** Renewal is due: this boot stamped work, or its last renewal found work it still holds. */
|
|
active: boolean;
|
|
epoch: number | null;
|
|
stopped: boolean;
|
|
/** Wall-clock time of the last stamp (the stamp window follows the server's clock). */
|
|
lastStampWallAt: number;
|
|
lastTickAt: number;
|
|
/** Counts stamps, so a stamp taken while a renewal is in flight is never lost. */
|
|
stampSeq: number;
|
|
/** Wall time of the oldest stamp no renewal has yet seen a held record for. */
|
|
unobservedStampSince: number | null;
|
|
unsubscribe: () => void;
|
|
wake: () => void;
|
|
};
|
|
|
|
let state: LeaseState | null = null;
|
|
let registeredDb: Db | null = null;
|
|
|
|
export function controllerInstanceLeaseEnabled(env: NodeJS.ProcessEnv = process.env): boolean {
|
|
return env.PAPERCLIP_CONTROLLER_INSTANCE_LEASE !== "off";
|
|
}
|
|
|
|
function exitOnLostLease(reason: "fenced" | "lease_lost") {
|
|
logger.fatal({ reason }, "controller instance lease lost while holding work; stopping so a newer server can recover it");
|
|
process.exit(1);
|
|
}
|
|
|
|
/** In-progress records stamped with this boot id: the process-owned kinds of the ownership registry. */
|
|
function holdsStampedWorkSql(bootId: string) {
|
|
return sql`(
|
|
exists (select 1 from environment_leases where
|
|
(status = 'active' and metadata->'leaseController'->>'bootId' = ${bootId})
|
|
or (status = 'pending_cleanup' and metadata->>'pendingCleanupInFlight' = 'true'
|
|
and metadata->'pendingCleanupController'->>'bootId' = ${bootId}))
|
|
or exists (select 1 from sandbox_creation_operations where state = 'pending' and metadata->'creationController'->>'bootId' = ${bootId})
|
|
or exists (select 1 from workspace_operations where status in ('running', 'pending') and metadata->'finalizeOwner'->>'bootId' = ${bootId})
|
|
or exists (select 1 from workspace_operations where status = 'running' and metadata->'commandOwner'->>'bootId' = ${bootId})
|
|
or exists (select 1 from workspace_operations, jsonb_each(case when jsonb_typeof(metadata#>'{saveWork,owners}') = 'object'
|
|
then metadata#>'{saveWork,owners}' else '{}'::jsonb end) as save_owner(invocation_id, stamp)
|
|
where status in ('running', 'pending') and phase = 'remote_workspace_snapshot' and save_owner.stamp->>'bootId' = ${bootId})
|
|
or exists (select 1 from agents where lifecycle_operation->'leaseHolder'->>'bootId' = ${bootId})
|
|
or exists (select 1 from native_run_finalizations where lease_owner is not null
|
|
and lease_holder->>'token' = lease_owner and lease_holder->>'bootId' = ${bootId})
|
|
or exists (select 1 from native_run_finalizations where controller_boot_id = ${bootId}
|
|
and phase not in ('committed', 'terminal_failure', 'applied'))
|
|
or exists (select 1 from heartbeat_runs where status = 'running' and controller_boot_id::text = ${bootId}))`;
|
|
}
|
|
|
|
/**
|
|
* One statement: create this boot's row on first use, and extend its lease only while the boot
|
|
* holds work (a stamp within the TTL, or an in-progress record stamped with its boot id).
|
|
* Returns whether it still holds work, or that a newer boot fenced it.
|
|
*/
|
|
function renewInDatabase(db: Db): (stampedRecently: boolean) => Promise<RenewResult> {
|
|
return async stampedRecently => {
|
|
const bootId = currentControllerBootId();
|
|
const records = holdsStampedWorkSql(bootId);
|
|
const holds = sql`(${stampedRecently} or ${records})`;
|
|
const expires = sql`now() + ${`${CONTROLLER_LEASE_TTL_MS} milliseconds`}::interval`;
|
|
// Its own connection: a busy main pool (request bursts at a traffic switch,
|
|
// a replacement's startup pass) must not delay the proof that this boot is
|
|
// alive. One statement per renewal, so an idle boot's last check stays a
|
|
// single round trip. The connection's own statement_timeout (below the
|
|
// client timeout) makes the server cancel a slow renewal, so it never holds
|
|
// the connection the next renewal needs.
|
|
const renew = (lease: Db) => lease.execute(sql`
|
|
insert into controller_instances (boot_id, process_space, pid, lease_expires_at)
|
|
values (${bootId}, ${currentProcessSpaceHash()}, ${process.pid}, ${expires})
|
|
on conflict (boot_id) do update set
|
|
lease_expires_at = case when ${holds} then ${expires} else controller_instances.lease_expires_at end,
|
|
renewed_at = case when ${holds} then now() else controller_instances.renewed_at end
|
|
where controller_instances.fenced_at is null and controller_instances.stopped_at is null
|
|
returning epoch, ${holds} as holds, ${records} as records`) as unknown as Promise<Array<{ epoch: string | number; holds: boolean; records: boolean }>>;
|
|
let rows: Awaited<ReturnType<typeof renew>>;
|
|
try {
|
|
rows = await renew(controllerInstanceLeaseDb(db, { statementTimeoutMs: RENEW_STATEMENT_TIMEOUT_MS }));
|
|
} catch (error) {
|
|
// A connection pooler may refuse the startup parameter: renew without it.
|
|
if (!/unsupported startup parameter/i.test(error instanceof Error ? error.message : String(error))) throw error;
|
|
logger.warn({ err: error }, "controller instance lease connection refused statement_timeout; renewing without it");
|
|
rows = await renew(controllerInstanceLeaseDbWithoutStatementTimeout(db));
|
|
}
|
|
const row = rows[0];
|
|
if (!row) return "fenced";
|
|
if (state) state.epoch = Number(row.epoch);
|
|
return row.records ? "holding" : row.holds ? "holding_stamp" : "idle";
|
|
};
|
|
}
|
|
|
|
function withTimeout<T>(work: Promise<T>, ms: number): Promise<T> {
|
|
let timer: ReturnType<typeof setTimeout> | undefined;
|
|
return Promise.race([work, new Promise<never>((_, reject) => {
|
|
timer = setTimeout(() => reject(new Error("controller instance lease renewal timed out")), ms);
|
|
})]).finally(() => clearTimeout(timer));
|
|
}
|
|
|
|
/**
|
|
* Registers this boot's lease with the shared delivery coordinator. The lease is renewed only
|
|
* while the boot holds work, so an idle boot issues no statements and its database can sleep.
|
|
*/
|
|
export function registerControllerInstanceLeaseWork(db: Db, coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
|
|
options: {
|
|
onLost?: (reason: "fenced" | "lease_lost") => void;
|
|
now?: () => number;
|
|
renew?: (stampedRecently: boolean) => Promise<RenewResult>;
|
|
} = {}) {
|
|
registeredDb = db;
|
|
const current: LeaseState = {
|
|
onLost: options.onLost ?? exitOnLostLease, now: options.now ?? (() => performance.now()),
|
|
renew: options.renew ?? renewInDatabase(db), lastStampAt: Number.NEGATIVE_INFINITY, failingSince: null,
|
|
active: false, epoch: null, stopped: false, unsubscribe: () => {}, wake: () => {},
|
|
lastStampWallAt: Number.NEGATIVE_INFINITY, lastTickAt: Number.NEGATIVE_INFINITY, stampSeq: 0,
|
|
unobservedStampSince: null,
|
|
};
|
|
state = current;
|
|
const worker = coordinator.register(DELIVERY_QUEUES.controllerInstanceLease, {
|
|
retryMs: RETRY_MS,
|
|
runDuringDrain: true,
|
|
run: () => tickControllerInstanceLease(),
|
|
hasPending: async () => false,
|
|
nextRunAt: () => nextLeaseDeadline(current),
|
|
});
|
|
current.wake = worker.wake;
|
|
const unsubscribeStamp = onControllerStamp(() => {
|
|
const wasActive = current.active;
|
|
current.lastStampAt = current.now();
|
|
current.lastStampWallAt = Date.now();
|
|
current.stampSeq += 1;
|
|
current.unobservedStampSince ??= Date.now();
|
|
current.active = true;
|
|
// The first stamp after an idle lapse renews at once, without delaying the stamping write.
|
|
if (!wasActive && !current.stopped) worker.wake();
|
|
});
|
|
// When the server's work settles, check holdings now rather than one renewal later,
|
|
// so an idle server issues no statement after its last work.
|
|
// The rate limit also absorbs the settle of this worker's own run.
|
|
const unsubscribeSettle = subscribeIdleWorkSettled(() => {
|
|
// All tracked work finished: any ownership write it made has committed or rolled back,
|
|
// so the holdings query is authoritative from here on.
|
|
current.unobservedStampSince = null;
|
|
if (current.active && !current.stopped && Date.now() - current.lastTickAt >= SETTLE_RECHECK_MIN_MS) worker.wake();
|
|
});
|
|
current.unsubscribe = () => { unsubscribeStamp(); unsubscribeSettle(); };
|
|
return {
|
|
ready: worker.ready,
|
|
stop: async () => {
|
|
current.unsubscribe();
|
|
await worker.stop();
|
|
},
|
|
};
|
|
}
|
|
|
|
function nextLeaseDeadline(current: LeaseState): number | null {
|
|
if (current.stopped) return null;
|
|
if (current.failingSince !== null) return Date.now() + RETRY_MS;
|
|
return current.active ? Date.now() + CONTROLLER_LEASE_RENEW_MS : null;
|
|
}
|
|
|
|
/** Renew while holding work; quiesce on failure; stop only if the failure outlasts the self-fence window with work held. */
|
|
export async function tickControllerInstanceLease(): Promise<void> {
|
|
const current = state;
|
|
if (!current || current.stopped || (!current.active && current.failingSince === null)) return;
|
|
const attemptStartedAt = current.now();
|
|
const stampSeqAtStart = current.stampSeq;
|
|
current.lastTickAt = Date.now();
|
|
// Held until a renewal sees the stamped record committed (or rolled back past the cap).
|
|
const stampedRecently = Date.now() - current.lastStampWallAt < STAMP_COMMIT_GRACE_MS
|
|
|| (current.unobservedStampSince !== null && Date.now() - current.unobservedStampSince < UNOBSERVED_STAMP_CAP_MS);
|
|
let result: RenewResult | "failed";
|
|
try { result = await withTimeout(current.renew(stampedRecently), RENEW_TIMEOUT_MS); } catch (error) {
|
|
logger.warn({ err: error }, "controller instance lease renewal failed");
|
|
result = "failed";
|
|
}
|
|
if (result === "holding" || result === "holding_stamp" || result === "idle") {
|
|
current.failingSince = null;
|
|
// Idle: the lease lapses with no further statements until the next stamp wakes it.
|
|
// A stamp taken during this attempt was not seen by its snapshot: check again now.
|
|
const stampedDuringAttempt = current.stampSeq !== stampSeqAtStart;
|
|
// A held record does not show that every stamped write has committed: another stamp's
|
|
// write may still be pending. Only the work-settled signal (or the cap) clears it.
|
|
current.active = result !== "idle" || stampedDuringAttempt;
|
|
if (result === "idle" && stampedDuringAttempt) current.wake();
|
|
setControllerLeaseQuiesce(false);
|
|
return;
|
|
}
|
|
// Fenced or unreachable: dispatch nothing new while the outcome is open.
|
|
setControllerLeaseQuiesce(true);
|
|
if (result === "fenced") { current.stopped = true; current.onLost("fenced"); return; }
|
|
current.failingSince ??= attemptStartedAt;
|
|
if (current.now() - current.failingSince >= CONTROLLER_LEASE_SELF_FENCE_MS && (current.active || stampedRecently)) {
|
|
current.stopped = true;
|
|
current.onLost("lease_lost");
|
|
}
|
|
}
|
|
|
|
/** Graceful shutdown: this boot no longer owns anything, so its stamps are gone at once. */
|
|
export async function stopControllerInstanceLease(db: Db | null = registeredDb): Promise<void> {
|
|
const current = state;
|
|
if (current) { current.stopped = true; current.unsubscribe(); }
|
|
state = null;
|
|
if (!db || current?.epoch == null) return;
|
|
await db.update(controllerInstances).set({ stoppedAt: sql`now()` })
|
|
.where(eq(controllerInstances.bootId, currentControllerBootId()))
|
|
.catch(error => logger.warn({ err: error }, "controller instance stop record failed"));
|
|
}
|
|
|
|
/**
|
|
* The lease state of another boot, for a record it last wrote at `touchedAt`. The boot's last
|
|
* sign of life is the later of its lease expiry and that write plus the TTL (a boot that resumed
|
|
* after an idle lapse renews on its first stamp). Within the grace after it, the owner is still
|
|
* treated as alive so the record stays watched; past it, the boot is fenced (only by a newer
|
|
* epoch) and its records are gone. A boot that held work exits before this point.
|
|
*/
|
|
export async function proveControllerInstance(db: Db, bootId: string, touchedAt: Date | null,
|
|
options: { readOnly?: boolean } = {}): Promise<ControllerInstanceState | null> {
|
|
const [row] = await db.select({
|
|
fenced: sql<boolean>`${controllerInstances.fencedAt} is not null`,
|
|
stopped: sql<boolean>`${controllerInstances.stoppedAt} is not null`,
|
|
live: sql<boolean>`${controllerInstances.leaseExpiresAt} > now()`,
|
|
pastGrace: sql<boolean>`greatest(${controllerInstances.leaseExpiresAt},
|
|
${touchedAt?.toISOString() ?? null}::timestamptz + ${`${CONTROLLER_LEASE_TTL_MS} milliseconds`}::interval)
|
|
< now() - ${`${CONTROLLER_LEASE_FENCE_GRACE_MS} milliseconds`}::interval`,
|
|
}).from(controllerInstances).where(eq(controllerInstances.bootId, bootId));
|
|
if (!row) return null;
|
|
if (row.stopped) return "stopped";
|
|
if (row.fenced) return "fenced";
|
|
if (row.live) return "live";
|
|
if (!touchedAt) return "expired";
|
|
if (!row.pastGrace) return "grace";
|
|
// A read-only pass reports; only a mutating pass fences (or creates this boot's row).
|
|
if (options.readOnly) return "expired";
|
|
// Only a boot that holds a lease row of its own may fence, and only an older epoch.
|
|
const epoch = await ownEpoch();
|
|
if (epoch === null) return "expired";
|
|
const [fenced] = await db.update(controllerInstances).set({ fencedAt: sql`now()`, fencedByEpoch: epoch })
|
|
.where(and(eq(controllerInstances.bootId, bootId), isNull(controllerInstances.fencedAt), isNull(controllerInstances.stoppedAt),
|
|
sql`greatest(${controllerInstances.leaseExpiresAt}, ${touchedAt.toISOString()}::timestamptz
|
|
+ ${`${CONTROLLER_LEASE_TTL_MS} milliseconds`}::interval)
|
|
< now() - ${`${CONTROLLER_LEASE_FENCE_GRACE_MS} milliseconds`}::interval`,
|
|
sql`${controllerInstances.epoch} < ${epoch}`))
|
|
.returning({ bootId: controllerInstances.bootId });
|
|
if (fenced) return "fenced";
|
|
const [after] = await db.select({ fenced: sql<boolean>`${controllerInstances.fencedAt} is not null` })
|
|
.from(controllerInstances).where(eq(controllerInstances.bootId, bootId));
|
|
return after?.fenced ? "fenced" : "expired";
|
|
}
|
|
|
|
/** This boot's epoch, creating its row on first need (one statement). Null when the lease is off or unreachable. */
|
|
async function ownEpoch(): Promise<number | null> {
|
|
const current = state;
|
|
if (!current || current.stopped) return null;
|
|
if (current.epoch === null) {
|
|
const result = await withTimeout(current.renew(false), RENEW_TIMEOUT_MS).catch(() => null);
|
|
if (result === "fenced") return null;
|
|
}
|
|
return current.epoch;
|
|
}
|
|
|
|
/** Lets the owner proof read other boots' leases, whether or not this boot holds one. */
|
|
export function registerControllerInstanceDb(db: Db): void {
|
|
registeredDb = db;
|
|
}
|
|
|
|
export function registeredControllerInstanceDb(): Db | null {
|
|
return registeredDb;
|
|
}
|
|
|
|
export function resetControllerInstanceLeaseForTests(): void {
|
|
state?.unsubscribe();
|
|
state = null;
|
|
registeredDb = null;
|
|
setControllerLeaseQuiesce(false);
|
|
}
|