Files
PaperClipAI/server/src/services/legacy-controller-lease.ts
T
DottaandClaude Opus 5.5 6917d0568d fix(heartbeat): start each claimed run at once so its controller lease cannot lapse unrenewed (#21242)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server starts queued agent runs. A legacy-runtime run gets a 60
s controller lease when it is claimed. Only the run's own execution
renews that lease.
> - The queued-run dispatch claimed all the runs of one agent in a loop.
It started their executions only after the last claim finished.
> - When the database is slow (for example, while a flood of recovery
runs fills the queue), one claim can take tens of seconds. The earlier
claims' leases then expire before their executions start. The run fails
with `controller_lease_lost` and `renewalCount: 0`.
> - Also, the lease watch waited for its first 10 s tick before it
renewed, even when the claim-time lease was almost spent.
> - This pull request starts each run as soon as it is claimed. It also
makes the lease watch renew immediately when the lease is older than one
renewal interval.
> - The benefit is that a slow claim batch no longer makes runs fail
before they do any work.

## Linked Issues or Issue Description

Refs #21127, #21042, #21222

**What happened?**
On staging, legacy-runtime runs failed with `controller_lease_lost`.
Every loss record showed `renewalCount: 0`, `renewalPendingMs: null`,
and `sinceRenewalMs` equal to `deadlineLateMs` (1–13 ms). So the lease
watch started on a lease that had already expired, and no renewal was
ever sent. In one run, the claim was at 23:20:02 and the loss was at
23:23:11. Losses came in groups on the same agent at the same moment.
For example, runs claimed at 23:19:51, 23:20:02 and 23:21:59 were all
lost at 23:23:11.6.

**Why did it happen?**
`startNextQueuedRunForAgent` claimed every run of a batch before it
called `executeRun` for any of them. Each claim commits a 60 s lease,
but nothing renews it until the run's execution starts its lease watch.
A queue flood (fixed separately in #21222) made each claim slow, so a
batch took minutes. This bug predates today's changes. The queue flood
exposed it. The run-start hand-over (#21127) did not cause it: no
renewal was queued on the controller-lease connection.

**Expected behavior**
A claimed run starts its execution, and renews its lease, without
waiting for other claims. A lease watch that starts late renews at once.
A lease that lapsed and was revoked, reaped or taken over is still lost
and is never revived.

## What Changed

- `server/src/services/heartbeat/queue.ts`: `startNextQueuedRunForAgent`
starts each run's execution right after its own claim, not after the
whole batch. The concurrency cap, the agent start lock and the
run-admission check before each claim are unchanged. If a later claim
throws, the runs claimed before it are already executing, not stranded.
After merging master, the loop walks master's queued-run dispatcher
(#21234: person-initiated runs first, paced recovery delivery): each
successful claim is reported to the dispatcher (`dispatcher.claimed`)
and then started at once, so master's order and pacing are unchanged.
Master's long-queued-run bookkeeping (#21349) is kept too: a claim that
returns nothing is still recorded as declined (`noteDeclinedAgedClaim`),
and a successful claim still clears that record before the run starts.
- `server/src/services/legacy-controller-lease.ts`:
`watchLegacyControllerLease` renews immediately when the claim-time
lease has less than 50 s left (older than one renewal interval). It uses
the same conditional renewal statement. That statement extends only a
lease that this boot still holds on a running row. The deadline is
unchanged.
- Tests: red/green tests for both changes. A test also proves that a
late watch on a revoked, reaped or lapsed lease still loses it and does
not change the row.

## Verification

- `cd server && npx vitest run
src/services/legacy-controller-lease.test.ts
src/services/heartbeat/queue.test.ts --maxWorkers=2`: all tests pass.
- Red on master (source reverted, new tests kept): "starts each claimed
run before claiming the next" fails (`executeRun` not called while the
second claim is in progress). "renews at once when the watch starts on a
lease older than one renewal interval" fails (aborted at the claim-time
deadline). The three "starts late ... still loses it" cases fail on
master because master loses them by deadline and does not check
ownership.
- After deploy: `controller_lease_lost` with `renewalCount: 0` should
drop to about zero on staging, also with a full queue. A late watch now
shows `renewal_rejected` or a non-null `renewalPendingMs` instead of a
silent expired start.

## Risks

- Low. The executions of a batch now start while later claims of the
same batch are still in progress. Execution does not claim runs and is
not awaited by the loop, so the cap and the lock are not affected.
- A late lease watch sends one extra renewal at start. A normal claim
starts its watch within milliseconds with about 60 s left, so it sends
no extra statement and adds no wait to a turn start.
- No schema change. Mixed versions are safe: the renewal statement is
unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:08:38 -06:00

467 lines
23 KiB
TypeScript

import { notifyLiveRunRecovery } from "./live-run-recovery-notifications.js";
import { and, eq, gt, isNull, lte, sql, type SQL } from "drizzle-orm";
import { controllerLeaseDb, heartbeatRuns, type Db } from "@paperclipai/db";
import type { LegacyControllerLeaseLossCause, LegacyControllerLeaseLossTiming } from "./run-cancellation.js";
import { randomUUID } from "node:crypto";
import {
currentControllerBootId,
currentControllerProcessStamp,
readControllerProcess,
} from "./controller-process-identity.js";
import { proveOwnerGone, type LocalOwnerStamp, type OwnerProof } from "./owner-proof.js";
import type { InProgressRecord, SettleOutcome } from "./recovery/ownership-registry.js";
// A boot UUID has meaning across containers; a numeric PID does not. Legacy
// leases stamp the single process boot id shared with native claims.
export const legacyControllerBootId = currentControllerBootId();
export const LEGACY_CONTROLLER_LEASE_MS = 60_000;
export const LEGACY_CONTROLLER_RENEW_MS = 10_000;
type Run = typeof heartbeatRuns.$inferSelect;
/** A lease lost before adapter dispatch: no provider work began, so the
* continuation is retryable rather than an opaque `setup_failed`. */
export const CONTROLLER_LEASE_LOST_FAILURE_CODE = "controller_lease_lost";
/** Abort reason for a lost lease, so completion can attribute the stop. */
export class LegacyControllerLeaseLostError extends Error {
constructor(
readonly leaseLossCause: LegacyControllerLeaseLossCause,
readonly leaseLossTiming?: LegacyControllerLeaseLossTiming,
) {
super("Legacy controller lease lost");
this.name = "LegacyControllerLeaseLostError";
}
}
/** Commit these fields in the same UPDATE that claims a queued run. */
export function legacyControllerClaim(runtimeMode: string) {
if (runtimeMode === "native") return {};
return {
legacyProcessSettlement: null,
controllerBootId: legacyControllerBootId,
// This process, so a hard-killed controller is proven gone on the shared
// ownership path (no I/O: the claim is on the dispatch path).
controllerProcess: currentControllerProcessStamp(),
controllerLeaseExpiresAt: sql`clock_timestamp() + interval '60 seconds'`,
executionStage: "preparing",
};
}
type RenewalTarget = Pick<Run, "id" | "companyId"> & { stage?: "dispatching" };
type RenewalOutcome = "renewed" | "rejected" | "busy";
async function renewLegacyControllerLeasesInTransaction(
tx: Db,
targets: RenewalTarget[],
): Promise<Map<string, RenewalOutcome>> {
const requested = sql.join(targets.map(target => sql`(${target.id}::uuid, ${target.companyId}::uuid)`), sql`, `);
const dispatching = targets.filter(target => target.stage === "dispatching").map(target => sql`${target.id}::uuid`);
const stage = dispatching.length
? sql`case when r.id in (${sql.join(dispatching, sql`, `)}) then 'dispatching' else r.execution_stage end`
: sql`r.execution_stage`;
const rows = await tx.execute<{ id: string; renewed: boolean; examined: boolean }>(sql`
with requested(id, company_id) as (values ${requested}),
examined as (
select r.id from ${heartbeatRuns} r join requested q on r.id = q.id and r.company_id = q.company_id
for update of r skip locked
),
renewed as (
update ${heartbeatRuns} r set controller_lease_expires_at = clock_timestamp() + interval '60 seconds',
execution_stage = ${stage}
from examined e
where r.id = e.id and r.runtime_mode = 'legacy' and r.status = 'running'
and r.controller_boot_id = ${legacyControllerBootId}
and r.controller_lease_expires_at > clock_timestamp()
returning r.id
)
select q.id::text as id, exists (select 1 from renewed n where n.id = q.id) as renewed,
exists (select 1 from examined e where e.id = q.id) as examined
from requested q`);
const outcomes = new Map<string, RenewalOutcome>();
for (const row of rows) outcomes.set(row.id, row.renewed ? "renewed" : row.examined ? "rejected" : "busy");
return outcomes;
}
export async function renewLegacyControllerLease(
db: Db,
run: Pick<Run, "id" | "companyId" | "controllerBootId">,
stage?: "dispatching",
): Promise<boolean> {
const outcomes = await db.transaction(tx => renewLegacyControllerLeasesInTransaction(tx as unknown as Db,
[{ id: run.id, companyId: run.companyId, stage }]));
return outcomes.get(run.id) === "renewed";
}
/** One attempt may wait at most this long; a timed-out attempt is retried
* until the lease deadline instead of holding the renewal for the whole lease. */
export const LEGACY_CONTROLLER_RENEWAL_STATEMENT_TIMEOUT_MS = 10_000;
/** Pause before retrying a renewal that failed with an error, so a dropped
* connection is retried without spinning against an unavailable database. */
export const LEGACY_CONTROLLER_RENEWAL_RETRY_DELAY_MS = 1_000;
export const LEGACY_CONTROLLER_RENEWAL_BUSY_RETRY_DELAY_MS = 250;
function isStatementTimeout(error: unknown): boolean {
for (let cause = error, depth = 0; cause && depth < 4; cause = (cause as { cause?: unknown }).cause, depth++) {
if ((cause as { code?: unknown }).code === "57014") return true;
}
return false;
}
type RenewalResult = { outcome: RenewalOutcome; issuedAt: number };
type RenewalRequest = {
target: RenewalTarget;
statementTimeoutMs: number;
resolve(result: RenewalResult): void;
reject(error: unknown): void;
};
type RenewalBatch = { pending: RenewalRequest[]; flushing: boolean };
const renewalBatches = new WeakMap<object, RenewalBatch>();
function renewThroughBatch(db: Db, target: RenewalTarget, statementTimeoutMs: number): Promise<RenewalResult> {
let batch = renewalBatches.get(db);
if (!batch) renewalBatches.set(db, batch = { pending: [], flushing: false });
const outcome = new Promise<RenewalResult>((resolve, reject) => {
batch.pending.push({ target, statementTimeoutMs, resolve, reject });
});
if (!batch.flushing) void flushRenewals(db, batch);
return outcome;
}
async function flushRenewals(db: Db, batch: RenewalBatch) {
batch.flushing = true;
try {
await Promise.resolve();
while (batch.pending.length) {
const requests = batch.pending.splice(0);
const targets = new Map<string, RenewalTarget>();
for (const { target } of requests) {
const existing = targets.get(target.id);
targets.set(target.id, existing ? { ...existing, stage: existing.stage ?? target.stage } : target);
}
try {
const statementTimeoutMs = Math.min(...requests.map(request => request.statementTimeoutMs));
let issuedAt = 0;
const outcomes = await controllerLeaseDb(db).transaction(async (tx) => {
issuedAt = Date.now();
await tx.execute(sql`select set_config('statement_timeout', ${`${statementTimeoutMs}ms`}, true)`);
return renewLegacyControllerLeasesInTransaction(tx as unknown as Db, [...targets.values()]);
});
for (const request of requests) {
const sameCompany = targets.get(request.target.id)!.companyId === request.target.companyId;
request.resolve({ outcome: sameCompany ? outcomes.get(request.target.id) ?? "rejected" : "rejected", issuedAt });
}
} catch (error) {
for (const request of requests) request.reject(error);
}
}
} finally {
batch.flushing = false;
}
}
export async function liveLegacyControllerExpiresAt(db: Db, run: Run): Promise<Date | null> {
if (run.runtimeMode === "native" || !run.controllerBootId) return null;
const [owner] = await db.select({ expiresAt: heartbeatRuns.controllerLeaseExpiresAt }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId),
eq(heartbeatRuns.status, "running"),
gt(heartbeatRuns.controllerLeaseExpiresAt, sql`clock_timestamp()`),
));
return owner?.expiresAt ?? null;
}
export async function hasLiveLegacyController(db: Db, run: Run): Promise<boolean> {
return (await liveLegacyControllerExpiresAt(db, run)) !== null;
}
export type RevokedLegacyControllerState = Pick<Run, "executionStage" | "processPid" | "processGroupId"> & { portablePreparationLeaseId?: string };
function legacyPreparationStage() {
return and(eq(heartbeatRuns.runtimeMode, "legacy"), eq(heartbeatRuns.executionStage, "preparing"),
isNull(heartbeatRuns.processPid), isNull(heartbeatRuns.processGroupId), isNull(heartbeatRuns.processStartedAt));
}
export async function handoffLegacyControllerPreparation(db: Db, run: Run, preparationLease: SQL<string | null>, onHandoff: () => void) {
return db.transaction(async tx => {
await notifyLiveRunRecovery(tx, { companyId: run.companyId, runId: run.id });
const [released] = await tx.update(heartbeatRuns).set({ controllerLeaseExpiresAt: sql`clock_timestamp()` })
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId),
eq(heartbeatRuns.status, "running"), eq(heartbeatRuns.controllerBootId, legacyControllerBootId),
legacyPreparationStage(), sql`${preparationLease} is not null`)).returning({ id: heartbeatRuns.id });
if (!released) return false;
onHandoff();
return true;
});
}
/** Atomically revoke an expired controller. Renewal and revocation serialize on
* the run row. Expiry permits cleanup, never dispatch of a replacement agent.
* Returns the stage and process identifiers committed with the revocation, or
* null when another writer owns the row. They cannot advance afterwards,
* because renewal requires the replaced boot ID. */
export async function revokeExpiredLegacyControllerWithState(
db: Db,
run: Run,
reclaimPreparation?: SQL<string | null>,
): Promise<RevokedLegacyControllerState | null> {
if (run.runtimeMode === "native" || !run.controllerBootId) {
return { executionStage: run.executionStage, processPid: run.processPid, processGroupId: run.processGroupId };
}
const controllerBootId = run.controllerBootId;
const [revoked] = await db.transaction(async tx => {
await notifyLiveRunRecovery(tx, { companyId: run.companyId, runId: run.id });
const expired = and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId),
eq(heartbeatRuns.status, "running"), eq(heartbeatRuns.controllerBootId, controllerBootId),
lte(heartbeatRuns.controllerLeaseExpiresAt, sql`clock_timestamp()`));
const state = { executionStage: heartbeatRuns.executionStage, processPid: heartbeatRuns.processPid,
processGroupId: heartbeatRuns.processGroupId };
if (reclaimPreparation) {
const [reclaimed] = await tx.update(heartbeatRuns).set(legacyControllerClaim("legacy")).where(and(expired, legacyPreparationStage(),
sql`${reclaimPreparation} is not null`)).returning({ ...state, portablePreparationLeaseId: reclaimPreparation });
if (reclaimed?.portablePreparationLeaseId) return [{ ...reclaimed, portablePreparationLeaseId: reclaimed.portablePreparationLeaseId }];
}
return tx.update(heartbeatRuns).set({ controllerBootId: randomUUID(),
controllerLeaseExpiresAt: sql`clock_timestamp() + interval '60 seconds'`,
}).where(expired).returning(state);
});
return revoked ?? null;
}
/**
* The recorded controller process of a legacy lease (plan Phase 2), or null.
* `controller_process` names the controller only while its boot id equals
* `controller_boot_id`: revocation swaps in a random boot id, so a revoked
* lease never keeps it. Rows saved by the previous release have none; their
* lease stays a boot-id fence (expiry, revocation and the age guard).
*/
export function legacyControllerOwnerStamp(
run: Pick<Run, "controllerBootId" | "controllerProcess">,
): LocalOwnerStamp | null {
const controller = readControllerProcess(run.controllerProcess);
if (!controller || !run.controllerBootId || controller.bootId !== run.controllerBootId) return null;
return { kind: "local", scope: "legacy_controller_lease", generation: null, host: null,
processSpace: controller.hostHash, bootId: controller.bootId, pid: controller.pid,
processStartedAt: controller.processStartedAt, processGroupId: null };
}
/**
* The controller process that ran a run which is already terminal, also when
* the orphan reaper revoked its lease: revocation swaps in a random boot id
* but keeps the recorded process. Only for judging holds on finished runs,
* never a live lease (legacyControllerOwnerStamp owns those).
*/
export function finishedLegacyControllerOwnerStamp(
run: Pick<Run, "status" | "controllerBootId" | "controllerProcess">,
): LocalOwnerStamp | null {
const current = legacyControllerOwnerStamp(run);
if (current || run.status === "running" || run.status === "queued") return current;
const controller = readControllerProcess(run.controllerProcess);
return controller ? legacyControllerOwnerStamp({ controllerBootId: controller.bootId, controllerProcess: run.controllerProcess })
: null;
}
/** The shared prover's verdict on the recorded legacy controller, or null when
* the row names none (a native run, a revoked lease, or an older row). */
export async function proveLegacyController(
run: Pick<Run, "runtimeMode" | "controllerBootId" | "controllerProcess">,
): Promise<OwnerProof | null> {
if (run.runtimeMode === "native") return null;
const stamp = legacyControllerOwnerStamp(run);
return stamp === null ? null : proveOwnerGone(stamp);
}
/** True only when the shared prover proves the recorded legacy controller gone. */
export async function legacyControllerProvenGone(
run: Pick<Run, "runtimeMode" | "controllerBootId" | "controllerProcess">,
): Promise<boolean> {
return (await proveLegacyController(run))?.verdict === "gone";
}
/**
* How long a run whose lease lapsed stays with its owner while the owner's
* server process is proven alive. The owner settles the run itself: its local
* deadline stops the adapter and records `controller_lease_lost`. A run still
* running after this grace has an owner that no longer drives it, and the
* orphan reaper recovers it.
*/
export const LEGACY_LIVE_OWNER_SETTLE_GRACE_MS = 5 * 60_000;
/**
* The boot id of a run's legacy controller whose lease has not expired yet but
* whose recorded process is proven gone, or null. A drained server terminalizes
* its runs and exits with their leases unexpired; nothing of that controller is
* left to unwind a Stop.
*/
export async function provenGoneLegacyControllerBootId(db: Db, companyId: string, runId: string): Promise<string | null> {
const [run] = await db.select({ runtimeMode: heartbeatRuns.runtimeMode, controllerBootId: heartbeatRuns.controllerBootId,
controllerProcess: heartbeatRuns.controllerProcess }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.runtimeMode, "legacy"),
gt(heartbeatRuns.controllerLeaseExpiresAt, sql`clock_timestamp()`)));
if (!run?.controllerBootId) return null;
return await legacyControllerProvenGone(run) ? run.controllerBootId : null;
}
/**
* Settle of the `legacy_controller_lease` ownership kind. The shared
* reconciler calls it only after `proveOwnerGone` proved the recorded
* controller gone. It expires exactly that controller's outstanding lease
* (compare-and-swap on run, boot id and recorded pid) after waking live-run
* recovery, which then revokes it and settles the run as a lost process under
* its own checks (a live child process still keeps it running). It settles
* ownership only.
*/
export async function settleLegacyControllerLease(db: Db, record: InProgressRecord,
context: { now: Date; proof: OwnerProof }): Promise<SettleOutcome> {
if (record.owner.kind !== "process" || !record.companyId) return { settled: false, reason: "owner_not_a_process" };
const stamp = record.owner.stamp;
if (!stamp.bootId) return { settled: false, reason: "owner_not_a_process" };
const companyId = record.companyId;
const bootId = stamp.bootId;
return db.transaction(async transaction => {
const tx = transaction as unknown as Db;
const [row] = await tx.select({ agentId: heartbeatRuns.agentId, controllerProcess: heartbeatRuns.controllerProcess,
controllerBootId: heartbeatRuns.controllerBootId })
.from(heartbeatRuns).where(and(eq(heartbeatRuns.id, record.recordId), eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.runtimeMode, "legacy"), eq(heartbeatRuns.status, "running"),
eq(heartbeatRuns.controllerBootId, bootId), gt(heartbeatRuns.controllerLeaseExpiresAt, sql`clock_timestamp()`)))
.for("update");
if (!row) return { settled: false, reason: "lease_no_longer_held" };
const current = legacyControllerOwnerStamp(row);
// The proof judged this exact process; the lease has moved on since.
if (!current || current.pid !== stamp.pid || current.processSpace !== stamp.processSpace
|| (current.processStartedAt ?? null) !== (stamp.processStartedAt ?? null)) {
return { settled: false, reason: "owner_changed_since_proof" };
}
// Work intent before the write it announces (sleep-aware background work).
await notifyLiveRunRecovery(tx, { companyId, runId: record.recordId, agentId: row.agentId });
const updated = await tx.update(heartbeatRuns).set({ controllerLeaseExpiresAt: sql`clock_timestamp()` })
.where(and(eq(heartbeatRuns.id, record.recordId), eq(heartbeatRuns.companyId, companyId),
eq(heartbeatRuns.status, "running"), eq(heartbeatRuns.controllerBootId, bootId),
gt(heartbeatRuns.controllerLeaseExpiresAt, sql`clock_timestamp()`)))
.returning({ id: heartbeatRuns.id });
return updated.length ? { settled: true, reason: context.proof.reason } : { settled: false, reason: "lease_no_longer_held" };
});
}
export async function revokeExpiredLegacyController(db: Db, run: Run): Promise<boolean> {
return (await revokeExpiredLegacyControllerWithState(db, run)) !== null;
}
/** Abort the adapter if the controller cannot renew. Bound each check by the
* lease duration even when the database connection never settles. */
export function watchLegacyControllerLease(
db: Db,
run: Run,
controller: AbortController,
{ renewalStatementTimeoutMs = LEGACY_CONTROLLER_RENEWAL_STATEMENT_TIMEOUT_MS } = {},
) {
if (run.runtimeMode === "native" || !run.controllerBootId) {
return { stop() {}, async assertOwned(_stage?: "dispatching") {} };
}
let stopped = false;
let pending = false;
// Timing evidence for a loss: a renewal still in flight at the deadline
// points at the database; a late deadline callback points at the event loop.
let lastRenewedAt = Date.now();
let renewalCount = 0;
// A non-timeout error since the last successful renewal, so a deadline that
// passes during the retries is attributed to the error, not to a stall.
let renewalFailed = false;
const renewalsInFlight = new Set<{ startedAt: number }>();
let deadlineDueAt = 0;
let deadline: ReturnType<typeof setTimeout> | undefined;
const lost = (cause: LegacyControllerLeaseLossCause, atDeadline = false) => {
if (stopped || controller.signal.aborted) return;
const now = Date.now();
const oldestRenewal = Math.min(...[...renewalsInFlight].map((renewal) => renewal.startedAt));
controller.abort(new LegacyControllerLeaseLostError(cause, {
sinceRenewalMs: now - lastRenewedAt,
renewalPendingMs: renewalsInFlight.size > 0 ? now - oldestRenewal : null,
deadlineLateMs: atDeadline ? Math.max(0, now - deadlineDueAt) : null,
renewalCount,
}));
};
const deadlinePassed = () => lost(renewalFailed ? "renewal_error" : "renewal_deadline", true);
const armDeadline = (delayMs: number) => {
clearTimeout(deadline);
deadlineDueAt = Date.now() + delayMs;
deadline = setTimeout(deadlinePassed, delayMs);
deadline.unref();
};
const initialRemainingMs = (run.controllerLeaseExpiresAt?.getTime() ?? 0) - Date.now();
armDeadline(Math.max(0, initialRemainingMs));
const assertOwned = async (stage?: "dispatching") => {
if (stopped) return;
controller.signal.throwIfAborted();
const startedAt = Date.now();
const renewal = { startedAt };
renewalsInFlight.add(renewal);
let onAbort!: () => void;
const aborted = new Promise<never>((_, reject) => {
onAbort = () => reject(controller.signal.reason);
controller.signal.addEventListener("abort", onAbort, { once: true });
});
// The successful attempt's UPDATE writes a fresh lease, so the local
// deadline is measured from that attempt rather than the first one.
// Failed attempts are retried until the deadline aborts the controller:
// the UPDATE is idempotent and still requires this controller's boot ID.
let renewedFrom = startedAt;
const renew = async () => {
for (;;) {
try {
const { outcome, issuedAt } = await renewThroughBatch(db, { id: run.id, companyId: run.companyId, stage },
renewalStatementTimeoutMs);
renewedFrom = issuedAt;
if (outcome !== "busy") return outcome === "renewed";
await new Promise((resolve) => setTimeout(resolve, LEGACY_CONTROLLER_RENEWAL_BUSY_RETRY_DELAY_MS));
if (stopped || controller.signal.aborted) return false;
} catch (error) {
if (stopped || controller.signal.aborted) throw error;
// A statement timeout is a stall; it stays a `renewal_deadline` loss.
if (!isStatementTimeout(error)) {
renewalFailed = true;
await new Promise((resolve) => setTimeout(resolve, LEGACY_CONTROLLER_RENEWAL_RETRY_DELAY_MS));
if (stopped || controller.signal.aborted) throw error;
}
}
}
};
let renewed: boolean;
try {
renewed = await Promise.race([renew(), aborted]);
} finally {
renewalsInFlight.delete(renewal);
controller.signal.removeEventListener("abort", onAbort);
}
if (stopped) return;
if (!renewed) {
lost("renewal_rejected");
controller.signal.throwIfAborted();
}
controller.signal.throwIfAborted();
if (!stopped) {
lastRenewedAt = Date.now();
renewalCount += 1;
renewalFailed = false;
armDeadline(Math.max(0, LEGACY_CONTROLLER_LEASE_MS - (Date.now() - renewedFrom)));
}
};
const tick = () => {
if (pending || stopped) return;
pending = true;
void assertOwned().catch(() => lost("renewal_error")).finally(() => { pending = false; });
};
const timer = setInterval(tick, LEGACY_CONTROLLER_RENEW_MS);
timer.unref();
// The watch can start well after the claim that wrote the lease (a slow
// batch of claims, a loaded database). A lease older than one renewal
// interval is renewed now rather than on the next tick. The renewal is the
// same conditional statement as every other one: it only extends a lease
// this boot still holds on a running row, so a lease that lapsed and was
// revoked, reaped or taken over is still lost.
if (initialRemainingMs < LEGACY_CONTROLLER_LEASE_MS - LEGACY_CONTROLLER_RENEW_MS) tick();
return { assertOwned, stop() { stopped = true; clearInterval(timer); clearTimeout(deadline); } };
}