Files
PaperClipAI/server/src/services/controller-instance-lease.ts
T
DottaandClaude Opus 5.5 500bc694b5 fix(recovery): keep a draining server's lease alive and end a crashed server's runs in about 2 minutes (#21238)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A release swap runs two servers for a short overlap. The old server
drains, and the new server takes over the runs.
> - Each server proves that it is alive with a controller-instance
lease. Recovery uses that lease to decide whether a run's controller is
gone.
> - In local swap cells, a healthy old server exited during the overlap,
because its lease renewal waited for a connection on the busy main pool.
Every run it drove was interrupted.
> - When a controller crashed, its runs waited for another server boot,
or for a timer that restarted on every look. Tasks stood still for 21 to
28 minutes.
> - This pull request renews the lease on its own connection, and lets
the live server end a crashed controller's runs within a bounded time.
> - The benefit is that a swap does not kill a healthy server, and a
crashed server's runs end in about 2 minutes, so their tasks continue by
themselves.

## Linked Issues or Issue Description

Refs #21135, #21222

**What happened?**
1. During a two-server release swap (local cells, load 14 to 34), the
outgoing server exited with "controller instance lease lost while
holding work". Five lease renewals in a row timed out after 10 s. The 10
s timeout included the wait for a connection on the shared main pool,
which the traffic switch and the replacement's startup pass kept busy.
2. When the old server was killed in the middle of the overlap, its runs
stayed in progress for 21 to 28 minutes. The live server did not
re-check a crashed controller's runs: only a new boot did. The
unverified-controller wait started when a server first looked at the
run, not when the controller left, and the successor's own wait record
counted as the dead owner's sign of life.

**Expected behavior**
A healthy server keeps its lease during a swap. When a controller is
gone (its lease lapsed and it is fenced or stopped), the live server
ends that controller's runs within a bounded time, without another
server boot, so restart recovery continues the tasks.

**Steps to reproduce**
1. Run two servers against one database with a release swap and a 200 s
overlap.
2. Kill the old server 40 s into the overlap (SIGKILL).
3. See that its runs stay in progress for more than 20 minutes, until
another server starts.

## What Changed

- Lease renewal on its own connection (`packages/db/src/client.ts`,
`packages/db/src/index.ts`,
`server/src/services/controller-instance-lease.ts`):
- New `controllerInstanceLeaseDb(db)`: one private 1-connection pool
(`application_name` `paperclip-controller-instance-lease`), made the
same way as the existing `controllerLeaseDb` and closed by
`closeLivenessDb`.
  - The renewal is one statement on that connection, as before.
- The connection sends `statement_timeout` = 8 s (below the 10 s client
timeout) as a startup parameter (new
`DatabaseClientOptions.statementTimeoutMs`, used only by this pool). The
server cancels a slow renewal, so it never holds the single connection
that the next renewal needs, and there is no extra round trip per
renewal. (An earlier revision set it per transaction; that added three
round trips to every renewal and broke the idle-server receipt test.)
- If a connection pooler refuses the startup parameter ("unsupported
startup parameter"), the lease logs a warning and renews on a
replacement connection without it.
- A crashed controller's runs
(`server/src/services/recovery/ownership-registry.ts`,
`server/src/index.ts`,
`server/src/services/native-runtime/native-restart-recovery.ts`):
- The ownership reconciler on the live server hands runs whose
controller is proven gone to restart recovery. It does not wait for
another server boot.
- The unverified-controller wait is bounded from the moment the
controller's lease lapsed, not from the first look.
- When the controller boot is `stopped` or `fenced`, the run ends at
once.
- A successor's wait record no longer refreshes the dead owner's sign of
life.

Safety of ending a run when its controller boot is fenced or stopped:
- Only `stopped` (written by the boot's own graceful stop) and `fenced`
(a newer epoch fenced it after its lease expired plus 30 s grace) count.
`live` and `grace` never count, so a controller that is alive and
renewing is never treated as gone. `expired` alone does not count
either.
- A boot that cannot renew stops itself at its self-fence window (about
its lease expiry), and a boot that finds itself fenced exits at once. So
when another server can fence it, the old process has already stopped
driving runs. The lease change above removes the main cause of healthy
boots failing to renew.
- If the old process is still alive (partition or stall) and its Runner
continues: the old run is fenced. Its agent token is refused for writes
(`agent_run_superseded`), its remote workspace epoch is abandoned or the
continuation gets a new workspace, and the native coordinator records
`replacementDenied`. There are never two writers in one workspace.

## Verification

Red-to-green tests (each failed before its fix and passes after it):
- `server/src/__tests__/controller-instance-lease.test.ts` "own
connection": red (10004 ms timeout while the main pool is held) to
green. Full file green (26).
- `server/src/__tests__/controller-instance-lease.test.ts` "has the
server cancel a renewal blocked on a row lock, and renews on the same
connection next time": red without the connection-level timeout (the
renewal runs to the 10 s client timeout and stays active), green with it
(cancelled at about 8 s, no active lease statement left, the next
renewal succeeds in under 2 s on the same backend).
- `packages/db/src/client-options.test.ts`: the statement timeout is
sent as a startup parameter.
- `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`:
"lapsed longer ago" / "end of the bound" (wait counted from the lease
lapse), the live-server re-check of a crashed controller, the
fenced/stopped boot ending the run at once, and the successor's wait
record. Full file green.

Changed existing expectation (it asserted the old wait for a stopped or
fenced controller):
-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`
"keeps foreign local Runner and provider ownership after its controller
is stopped/fenced": the disposition is now `blocked`
(`controller_process_space_unverified`) instead of `awaiting_evidence`,
the run is `failed` with `native_restart_recovery_blocked`, and the
finalization is `terminal_failure` with `executorStopEvidence:
"unverified"`. The safety checks stay: no local PID probe, no claim, no
stamp, no local stop proof, the run keeps its process identity and
session checkpoint, and no result is written.

Local commands, on this branch based on current master:
- `cd server && npx vitest run
src/__tests__/controller-instance-lease.test.ts
src/__tests__/heartbeat-native-restart-suspension.test.ts`: 2 files, 140
tests passed.
- `npx vitest run src/__tests__/controller-instance-lease.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
225 passed, 14 skipped (real-process cases that need CI).
- `pnpm exec vitest run src/__tests__/maintenance-server-routes.test.ts
-t "releases an ordinary held task Retry"`: passed (idle receipt: one
lease statement after the work ends).

Local swap cells (two servers, embedded Postgres, real agents, old
server SIGKILLed 40 s into a 200 s overlap):
- Before: runs of the killed server stayed in progress 21 to 28 minutes.
- N1 (2 tasks): 2/2 finished hands-off. Kill to run end 5 min 31 s (the
successor's wait write still moved the timer; fixed by the last commit).
- R1 (6 tasks, 3 adapters, subtasks, follow-ups, 2 swaps): 4/4
hands-off, 0 banners, 0 duplicates, 0 invariant violations. Kill to run
end 3 min 39 s.
- N2 (same seed as N1, all fixes): 2/2 hands-off. Kill to run end 2 min
01 s; fresh runs started at 2 min 32 s and 3 min 02 s. The floor is the
lease TTL (90 s) plus the fence grace (30 s).
- Hour-long release-swap wave (w-20261112T233630-s7201) on the branch
that holds this fix and the related recovery fixes: 11/11 tasks finished
hands-off (master: 9/11). The lease and crashed-controller fixes were
visibly working in it.

## Risks

- A pooled database URL whose pooler refuses `statement_timeout` as a
startup parameter: the first renewal fails over to a connection without
it (logged), so the lease keeps working without the server-side cancel.
- One extra database connection per server, held while the server runs.
It is idle except for one renewal every few seconds.
- A run whose controller boot is fenced or stopped now ends at once
instead of after the bound. See the safety notes above: only a boot that
is not alive can be in these states.
- No migration. No schema change. Nothing changes before `turn.start` on
the normal path.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:11:27 -06:00

314 lines
17 KiB
TypeScript

import { and, eq, isNull, sql } from "drizzle-orm";
import { controllerInstanceLeaseDb, controllerInstanceLeaseDbWithoutStatementTimeout, controllerInstances, type Db } from "@paperclipai/db";
import { logger } from "../middleware/logger.js";
import {
currentControllerBootId, currentProcessSpaceHash, onControllerStamp,
} from "./controller-process-identity.js";
import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
import { setControllerLeaseQuiesce, subscribeIdleWorkSettled } from "./task-admission.js";
export const CONTROLLER_LEASE_TTL_MS = 90_000;
export const CONTROLLER_LEASE_RENEW_MS = 30_000;
/** Renewal attempts start failing from their own start; this boot stops after this long while holding work. */
export const CONTROLLER_LEASE_SELF_FENCE_MS = 60_000;
/** A prover fences a boot only this long after the boot's last sign of life. */
export const CONTROLLER_LEASE_FENCE_GRACE_MS = 30_000;
/** A stamp counts as held work for a full lease, so a slow ownership write commits before the lease can lapse. */
const STAMP_COMMIT_GRACE_MS = CONTROLLER_LEASE_TTL_MS;
/** A settle of other work re-checks holdings at most this often. */
const SETTLE_RECHECK_MIN_MS = 5_000;
/** A renewal that has not answered within this bound counts as failed. */
const RENEW_TIMEOUT_MS = 10_000;
/** Server-side cancel for a renewal statement, below the client timeout. */
export const RENEW_STATEMENT_TIMEOUT_MS = 8_000;
const RETRY_MS = 5_000;
export type ControllerInstanceState = "live" | "grace" | "expired" | "fenced" | "stopped";
/** "holding" saw an in-progress record held by this boot; "holding_stamp" held only for a recent stamp. */
type RenewResult = "holding" | "holding_stamp" | "idle" | "fenced";
/** A stamp whose record no renewal has seen keeps the lease held until the server's tracked work next settles
* (its write then committed or rolled back), and at most this long. */
const UNOBSERVED_STAMP_CAP_MS = 10 * 60_000;
type LeaseState = {
onLost: (reason: "fenced" | "lease_lost") => void;
now: () => number;
renew: (stampedRecently: boolean) => Promise<RenewResult>;
lastStampAt: number;
/** Start of the first renewal attempt in the current failing streak. */
failingSince: number | null;
/** Renewal is due: this boot stamped work, or its last renewal found work it still holds. */
active: boolean;
epoch: number | null;
stopped: boolean;
/** Wall-clock time of the last stamp (the stamp window follows the server's clock). */
lastStampWallAt: number;
lastTickAt: number;
/** Counts stamps, so a stamp taken while a renewal is in flight is never lost. */
stampSeq: number;
/** Wall time of the oldest stamp no renewal has yet seen a held record for. */
unobservedStampSince: number | null;
unsubscribe: () => void;
wake: () => void;
};
let state: LeaseState | null = null;
let registeredDb: Db | null = null;
export function controllerInstanceLeaseEnabled(env: NodeJS.ProcessEnv = process.env): boolean {
return env.PAPERCLIP_CONTROLLER_INSTANCE_LEASE !== "off";
}
function exitOnLostLease(reason: "fenced" | "lease_lost") {
logger.fatal({ reason }, "controller instance lease lost while holding work; stopping so a newer server can recover it");
process.exit(1);
}
/** In-progress records stamped with this boot id: the process-owned kinds of the ownership registry. */
function holdsStampedWorkSql(bootId: string) {
return sql`(
exists (select 1 from environment_leases where
(status = 'active' and metadata->'leaseController'->>'bootId' = ${bootId})
or (status = 'pending_cleanup' and metadata->>'pendingCleanupInFlight' = 'true'
and metadata->'pendingCleanupController'->>'bootId' = ${bootId}))
or exists (select 1 from sandbox_creation_operations where state = 'pending' and metadata->'creationController'->>'bootId' = ${bootId})
or exists (select 1 from workspace_operations where status in ('running', 'pending') and metadata->'finalizeOwner'->>'bootId' = ${bootId})
or exists (select 1 from workspace_operations where status = 'running' and metadata->'commandOwner'->>'bootId' = ${bootId})
or exists (select 1 from workspace_operations, jsonb_each(case when jsonb_typeof(metadata#>'{saveWork,owners}') = 'object'
then metadata#>'{saveWork,owners}' else '{}'::jsonb end) as save_owner(invocation_id, stamp)
where status in ('running', 'pending') and phase = 'remote_workspace_snapshot' and save_owner.stamp->>'bootId' = ${bootId})
or exists (select 1 from agents where lifecycle_operation->'leaseHolder'->>'bootId' = ${bootId})
or exists (select 1 from native_run_finalizations where lease_owner is not null
and lease_holder->>'token' = lease_owner and lease_holder->>'bootId' = ${bootId})
or exists (select 1 from native_run_finalizations where controller_boot_id = ${bootId}
and phase not in ('committed', 'terminal_failure', 'applied'))
or exists (select 1 from heartbeat_runs where status = 'running' and controller_boot_id::text = ${bootId}))`;
}
/**
* One statement: create this boot's row on first use, and extend its lease only while the boot
* holds work (a stamp within the TTL, or an in-progress record stamped with its boot id).
* Returns whether it still holds work, or that a newer boot fenced it.
*/
function renewInDatabase(db: Db): (stampedRecently: boolean) => Promise<RenewResult> {
return async stampedRecently => {
const bootId = currentControllerBootId();
const records = holdsStampedWorkSql(bootId);
const holds = sql`(${stampedRecently} or ${records})`;
const expires = sql`now() + ${`${CONTROLLER_LEASE_TTL_MS} milliseconds`}::interval`;
// Its own connection: a busy main pool (request bursts at a traffic switch,
// a replacement's startup pass) must not delay the proof that this boot is
// alive. One statement per renewal, so an idle boot's last check stays a
// single round trip. The connection's own statement_timeout (below the
// client timeout) makes the server cancel a slow renewal, so it never holds
// the connection the next renewal needs.
const renew = (lease: Db) => lease.execute(sql`
insert into controller_instances (boot_id, process_space, pid, lease_expires_at)
values (${bootId}, ${currentProcessSpaceHash()}, ${process.pid}, ${expires})
on conflict (boot_id) do update set
lease_expires_at = case when ${holds} then ${expires} else controller_instances.lease_expires_at end,
renewed_at = case when ${holds} then now() else controller_instances.renewed_at end
where controller_instances.fenced_at is null and controller_instances.stopped_at is null
returning epoch, ${holds} as holds, ${records} as records`) as unknown as Promise<Array<{ epoch: string | number; holds: boolean; records: boolean }>>;
let rows: Awaited<ReturnType<typeof renew>>;
try {
rows = await renew(controllerInstanceLeaseDb(db, { statementTimeoutMs: RENEW_STATEMENT_TIMEOUT_MS }));
} catch (error) {
// A connection pooler may refuse the startup parameter: renew without it.
if (!/unsupported startup parameter/i.test(error instanceof Error ? error.message : String(error))) throw error;
logger.warn({ err: error }, "controller instance lease connection refused statement_timeout; renewing without it");
rows = await renew(controllerInstanceLeaseDbWithoutStatementTimeout(db));
}
const row = rows[0];
if (!row) return "fenced";
if (state) state.epoch = Number(row.epoch);
return row.records ? "holding" : row.holds ? "holding_stamp" : "idle";
};
}
function withTimeout<T>(work: Promise<T>, ms: number): Promise<T> {
let timer: ReturnType<typeof setTimeout> | undefined;
return Promise.race([work, new Promise<never>((_, reject) => {
timer = setTimeout(() => reject(new Error("controller instance lease renewal timed out")), ms);
})]).finally(() => clearTimeout(timer));
}
/**
* Registers this boot's lease with the shared delivery coordinator. The lease is renewed only
* while the boot holds work, so an idle boot issues no statements and its database can sleep.
*/
export function registerControllerInstanceLeaseWork(db: Db, coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
options: {
onLost?: (reason: "fenced" | "lease_lost") => void;
now?: () => number;
renew?: (stampedRecently: boolean) => Promise<RenewResult>;
} = {}) {
registeredDb = db;
const current: LeaseState = {
onLost: options.onLost ?? exitOnLostLease, now: options.now ?? (() => performance.now()),
renew: options.renew ?? renewInDatabase(db), lastStampAt: Number.NEGATIVE_INFINITY, failingSince: null,
active: false, epoch: null, stopped: false, unsubscribe: () => {}, wake: () => {},
lastStampWallAt: Number.NEGATIVE_INFINITY, lastTickAt: Number.NEGATIVE_INFINITY, stampSeq: 0,
unobservedStampSince: null,
};
state = current;
const worker = coordinator.register(DELIVERY_QUEUES.controllerInstanceLease, {
retryMs: RETRY_MS,
runDuringDrain: true,
run: () => tickControllerInstanceLease(),
hasPending: async () => false,
nextRunAt: () => nextLeaseDeadline(current),
});
current.wake = worker.wake;
const unsubscribeStamp = onControllerStamp(() => {
const wasActive = current.active;
current.lastStampAt = current.now();
current.lastStampWallAt = Date.now();
current.stampSeq += 1;
current.unobservedStampSince ??= Date.now();
current.active = true;
// The first stamp after an idle lapse renews at once, without delaying the stamping write.
if (!wasActive && !current.stopped) worker.wake();
});
// When the server's work settles, check holdings now rather than one renewal later,
// so an idle server issues no statement after its last work.
// The rate limit also absorbs the settle of this worker's own run.
const unsubscribeSettle = subscribeIdleWorkSettled(() => {
// All tracked work finished: any ownership write it made has committed or rolled back,
// so the holdings query is authoritative from here on.
current.unobservedStampSince = null;
if (current.active && !current.stopped && Date.now() - current.lastTickAt >= SETTLE_RECHECK_MIN_MS) worker.wake();
});
current.unsubscribe = () => { unsubscribeStamp(); unsubscribeSettle(); };
return {
ready: worker.ready,
stop: async () => {
current.unsubscribe();
await worker.stop();
},
};
}
function nextLeaseDeadline(current: LeaseState): number | null {
if (current.stopped) return null;
if (current.failingSince !== null) return Date.now() + RETRY_MS;
return current.active ? Date.now() + CONTROLLER_LEASE_RENEW_MS : null;
}
/** Renew while holding work; quiesce on failure; stop only if the failure outlasts the self-fence window with work held. */
export async function tickControllerInstanceLease(): Promise<void> {
const current = state;
if (!current || current.stopped || (!current.active && current.failingSince === null)) return;
const attemptStartedAt = current.now();
const stampSeqAtStart = current.stampSeq;
current.lastTickAt = Date.now();
// Held until a renewal sees the stamped record committed (or rolled back past the cap).
const stampedRecently = Date.now() - current.lastStampWallAt < STAMP_COMMIT_GRACE_MS
|| (current.unobservedStampSince !== null && Date.now() - current.unobservedStampSince < UNOBSERVED_STAMP_CAP_MS);
let result: RenewResult | "failed";
try { result = await withTimeout(current.renew(stampedRecently), RENEW_TIMEOUT_MS); } catch (error) {
logger.warn({ err: error }, "controller instance lease renewal failed");
result = "failed";
}
if (result === "holding" || result === "holding_stamp" || result === "idle") {
current.failingSince = null;
// Idle: the lease lapses with no further statements until the next stamp wakes it.
// A stamp taken during this attempt was not seen by its snapshot: check again now.
const stampedDuringAttempt = current.stampSeq !== stampSeqAtStart;
// A held record does not show that every stamped write has committed: another stamp's
// write may still be pending. Only the work-settled signal (or the cap) clears it.
current.active = result !== "idle" || stampedDuringAttempt;
if (result === "idle" && stampedDuringAttempt) current.wake();
setControllerLeaseQuiesce(false);
return;
}
// Fenced or unreachable: dispatch nothing new while the outcome is open.
setControllerLeaseQuiesce(true);
if (result === "fenced") { current.stopped = true; current.onLost("fenced"); return; }
current.failingSince ??= attemptStartedAt;
if (current.now() - current.failingSince >= CONTROLLER_LEASE_SELF_FENCE_MS && (current.active || stampedRecently)) {
current.stopped = true;
current.onLost("lease_lost");
}
}
/** Graceful shutdown: this boot no longer owns anything, so its stamps are gone at once. */
export async function stopControllerInstanceLease(db: Db | null = registeredDb): Promise<void> {
const current = state;
if (current) { current.stopped = true; current.unsubscribe(); }
state = null;
if (!db || current?.epoch == null) return;
await db.update(controllerInstances).set({ stoppedAt: sql`now()` })
.where(eq(controllerInstances.bootId, currentControllerBootId()))
.catch(error => logger.warn({ err: error }, "controller instance stop record failed"));
}
/**
* The lease state of another boot, for a record it last wrote at `touchedAt`. The boot's last
* sign of life is the later of its lease expiry and that write plus the TTL (a boot that resumed
* after an idle lapse renews on its first stamp). Within the grace after it, the owner is still
* treated as alive so the record stays watched; past it, the boot is fenced (only by a newer
* epoch) and its records are gone. A boot that held work exits before this point.
*/
export async function proveControllerInstance(db: Db, bootId: string, touchedAt: Date | null,
options: { readOnly?: boolean } = {}): Promise<ControllerInstanceState | null> {
const [row] = await db.select({
fenced: sql<boolean>`${controllerInstances.fencedAt} is not null`,
stopped: sql<boolean>`${controllerInstances.stoppedAt} is not null`,
live: sql<boolean>`${controllerInstances.leaseExpiresAt} > now()`,
pastGrace: sql<boolean>`greatest(${controllerInstances.leaseExpiresAt},
${touchedAt?.toISOString() ?? null}::timestamptz + ${`${CONTROLLER_LEASE_TTL_MS} milliseconds`}::interval)
< now() - ${`${CONTROLLER_LEASE_FENCE_GRACE_MS} milliseconds`}::interval`,
}).from(controllerInstances).where(eq(controllerInstances.bootId, bootId));
if (!row) return null;
if (row.stopped) return "stopped";
if (row.fenced) return "fenced";
if (row.live) return "live";
if (!touchedAt) return "expired";
if (!row.pastGrace) return "grace";
// A read-only pass reports; only a mutating pass fences (or creates this boot's row).
if (options.readOnly) return "expired";
// Only a boot that holds a lease row of its own may fence, and only an older epoch.
const epoch = await ownEpoch();
if (epoch === null) return "expired";
const [fenced] = await db.update(controllerInstances).set({ fencedAt: sql`now()`, fencedByEpoch: epoch })
.where(and(eq(controllerInstances.bootId, bootId), isNull(controllerInstances.fencedAt), isNull(controllerInstances.stoppedAt),
sql`greatest(${controllerInstances.leaseExpiresAt}, ${touchedAt.toISOString()}::timestamptz
+ ${`${CONTROLLER_LEASE_TTL_MS} milliseconds`}::interval)
< now() - ${`${CONTROLLER_LEASE_FENCE_GRACE_MS} milliseconds`}::interval`,
sql`${controllerInstances.epoch} < ${epoch}`))
.returning({ bootId: controllerInstances.bootId });
if (fenced) return "fenced";
const [after] = await db.select({ fenced: sql<boolean>`${controllerInstances.fencedAt} is not null` })
.from(controllerInstances).where(eq(controllerInstances.bootId, bootId));
return after?.fenced ? "fenced" : "expired";
}
/** This boot's epoch, creating its row on first need (one statement). Null when the lease is off or unreachable. */
async function ownEpoch(): Promise<number | null> {
const current = state;
if (!current || current.stopped) return null;
if (current.epoch === null) {
const result = await withTimeout(current.renew(false), RENEW_TIMEOUT_MS).catch(() => null);
if (result === "fenced") return null;
}
return current.epoch;
}
/** Lets the owner proof read other boots' leases, whether or not this boot holds one. */
export function registerControllerInstanceDb(db: Db): void {
registeredDb = db;
}
export function registeredControllerInstanceDb(): Db | null {
return registeredDb;
}
export function resetControllerInstanceLeaseForTests(): void {
state?.unsubscribe();
state = null;
registeredDb = null;
setControllerLeaseQuiesce(false);
}