fix(heartbeat): renew legacy controller leases on a private pool (#17032)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A legacy (non-native) agent run keeps a controller lease on its
`heartbeat_runs` row. The server renews the lease every 10 seconds. If
no renewal succeeds in 60 seconds, the watchdog stops the run with
`renewal_deadline`.
> - #16359 added timings to lease-loss reports. The first staging event
with these timings shows `deadlineLateMs=0`, `renewalPendingMs=50000`,
and 108 renewals in about 22.5 minutes, where about 135 are expected.
Thus the event loop was not blocked. One renewal UPDATE was in flight
for 50 seconds, and renewals were slow for a long time before the loss.
> - The agent had exited with code 0. The run was in post-execution
`sync_back`. Other runs on the same host completed. No code path held a
lock on the run row across slow work. A later `instruction_save` UPDATE
on the same row completed while the earlier renewal was still pending,
so a row-lock wait does not explain the stall.
> - postgres.js 3.4.9 sends a new query to a busy connection when no
connection is idle (`handler` → `go(busy.shift(), query)` in
`src/index.js`). The query waits behind the statement that is in flight
on that connection. The renewal had no timeout, and the `pending` flag
stopped all later renewals. Thus one slow unrelated statement on the
shared pool can stop a healthy run.
> - This pull request moves lease renewal to a small private pool,
bounds each attempt with a statement timeout, and retries a timed-out
attempt until the existing lease deadline.
> - The benefit is that slow application queries on the shared pool can
no longer cancel a healthy legacy run.

## Linked Issues or Issue Description

Refs #16359 (lease-loss timings that identified this cause)
Refs #16358 (pre-dispatch lease loss becomes retryable; different path)
Refs #16917 (open: keeps a clean adapter result after a lease lapsed; it
does not prevent the loss)
Refs #13389 (open: host sleep / late timers; different cause)

**What happened?**
A staging legacy `claude_local` run was cancelled with `Legacy
controller lease lost (renewal_deadline)` after the agent exited with
code 0. The timings show that the deadline timer ran on time, and one
renewal UPDATE was pending for 50 seconds.

**Expected behavior**
A live controller keeps its lease. Slow unrelated database work does not
cancel the run.

**Steps to reproduce**
1. Create a database client with one pooled connection.
2. Start `select pg_sleep(5)` on that client.
3. Call `watchLegacyControllerLease(...).assertOwned()` with the same
client.
4. Before this change, the renewal waits about 5 seconds behind
`pg_sleep`. With a long enough slow statement, the lease deadline stops
the run.

## What Changed

- `packages/db`: Add `livenessDb(db)`. It gives each `createDb` client a
private pool of at most 2 connections
(`application_name=paperclip-controller-lease`). The pool is created on
first use. Test doubles and transactions get the original client back.
- `packages/db`: Add `closeLivenessDb(db, timeoutSeconds)`. The server
calls it in `closeDatabaseClients`, so shutdown and failed startup end
the private pool too.
- `legacy-controller-lease.ts`: The watchdog renews in a transaction on
the private pool. Each attempt sets a transaction-local
`statement_timeout` of 10 seconds. A timed-out attempt (`57014`) is
retried. After a successful retry, the local deadline is measured from
that attempt's start, because that attempt's UPDATE wrote the fresh 60
second lease. The existing 60 second deadline, the abort race, and the
`renewal_rejected` / `renewal_error` attribution do not change.
`renewLegacyControllerLease` itself does not change, so the
runtime-selection caller inside its own transaction is not affected.
- Tests: a saturated shared pool no longer delays renewal; a renewal
that times out on a locked run row retries and keeps the lease; the
private pool is cached per client, has its own `application_name`, and
closes; after a 45 second timed-out attempt and a successful retry, the
next deadline is a full 60 seconds after the retry. The existing test
doubles now model the transaction.
- `docs/deploy/database.md`: Count the up to 2 extra connections per
server process when you size `max_connections`.

## Verification

- `cd server && npx vitest run
src/services/legacy-controller-lease.test.ts`: 20 passed, on embedded
Postgres.
- Red/green: with the old `legacy-controller-lease.ts` and the new
tests, the saturated-pool test fails with `expected 5018 to be less than
2000`. With the fix, it passes.
- Red/green for the retry deadline: with the deadline measured from the
first attempt, the retry-deadline test fails (`expected true to be
false`) because the run is aborted 15 seconds after the retry. With the
fix, it passes.
- Mutation check: when the timeout retry is removed, the locked-row test
fails with `canceling statement due to statement timeout`.
- `npx vitest run src/__tests__/heartbeat-process-recovery.test.ts -t
"lease|dispatch|controller|interaction"`: 43 passed.
- `npx vitest run src/services/heartbeat/run-completion.test.ts
src/__tests__/heartbeat-orphaned-active-lease-sweep.test.ts`: 53 passed.
- `npx vitest run src/__tests__/server-startup-feedback-export.test.ts`:
passed.
- `cd packages/db && npx vitest run src/work-signals.test.ts
src/client-options.test.ts src/client-teardown-registry.test.ts`: 24
passed. `npx tsc --noEmit` for `packages/db` passed.
- Not run locally: the full server `tsc` and the full server suite. The
sandbox has a 4 GB memory limit, and `tsc` was killed (exit 137). CI is
the gate for the server typecheck, the full tests, and the build.

## Risks

- Each server process can open up to 2 more Postgres connections.
Deployments that are at their `max_connections` limit must allow for
them. The docs now say this.
- A renewal attempt is now 4 statements (BEGIN, `set_config`, UPDATE,
COMMIT) instead of 1. Each statement is a short primary-key operation on
a dedicated connection.
- If many runs wait on locked run rows at the same time, renewals for
other runs can queue on the 2 private connections. The 10 second
statement timeout limits each wait, which is less than the 60 second
lease.
- Behavior for a real lease loss does not change. A rejected renewal and
a non-timeout error still stop the run at once, and the 60 second
deadline still applies.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code in agent mode with tool use (shell, file edit, and test execution).
A read-only subagent helped with code search.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

docs/deploy/database.md

@@ -77,7 +77,7 @@ The server opens one postgres.js pool for its own queries (and a second one when
 | `DATABASE_APPLICATION_NAME` | `paperclip` | Value of `application_name` in `pg_stat_activity`, so you can find Paperclip's backends: `SELECT * FROM pg_stat_activity WHERE application_name = 'paperclip';` |
 | `DATABASE_PREPARED_STATEMENTS` | `true` (driver) | Set `false` behind a transaction-mode pooler (see above). |
 
-The server ends its pools during shutdown (SIGINT/SIGTERM) and when startup fails after the pool was opened, so a restarting server does not leave idle backends behind. Size `max_connections` on the PostgreSQL side for at least `DATABASE_POOL_MAX` per server process plus your other clients.
+The server ends its pools during shutdown (SIGINT/SIGTERM) and when startup fails after the pool was opened, so a restarting server does not leave idle backends behind. Size `max_connections` on the PostgreSQL side for at least `DATABASE_POOL_MAX` per server process plus your other clients. Each server process also opens up to 2 connections (`application_name=paperclip-controller-lease`) for agent-run lease renewals, so they never queue behind slow application queries; count those too.
 
 ## Switching Between Modes
 

packages/db/src/client.ts

@@ -267,6 +267,32 @@ export async function withDedicatedDbConnection<T>(db: Db, action: (dedicated: D
   finally { await dedicated.$client.end({ timeout: 1 }); }
 }
 
+// postgres.js pipelines a query onto a busy connection when none is idle, so
+// a short liveness write can wait behind an unrelated slow statement with no
+// client-side bound. Liveness writes (controller-lease renewal) get a small
+// private pool, created on first use and shared for the owner's lifetime.
+const livenessDbFactories = new WeakMap<object, () => Db>();
+const livenessDbs = new WeakMap<object, Db>();
+const LIVENESS_POOL_MAX_CONNECTIONS = 2;
+
+/** The private liveness pool for a `createDb` client; other clients (test
+ * doubles, transactions) are returned unchanged. */
+export function livenessDb(db: Db): Db {
+  const existing = livenessDbs.get(db);
+  if (existing) return existing;
+  const factory = livenessDbFactories.get(db);
+  if (!factory) return db;
+  const created = factory();
+  livenessDbs.set(db, created);
+  return created;
+}
+
+/** Ends the liveness pool if one was created. The ended pool stays cached, so
+ * a late renewal fails like a query on the ended primary pool. */
+export async function closeLivenessDb(db: Db, timeoutSeconds: number): Promise<void> {
+  await livenessDbs.get(db)?.$client.end({ timeout: timeoutSeconds });
+}
+
 export function createDb(url: string, options?: DatabaseClientOptions) {
   const resolved = resolveDatabaseClientOptions(options ?? databaseClientOptionsFromEnv());
   return createDbClient(url, resolved);
@@ -284,6 +310,9 @@ function createDbClient(url: string, resolved: DatabaseClientOptions, owner?: ob
   dedicatedDbFactories.set(db, () => createDbClient(url, {
     ...resolved, maxConnections: 1, applicationName: "paperclip-workspace-finalization-lock",
   }, db));
+  livenessDbFactories.set(db, () => createDbClient(url, {
+    ...resolved, maxConnections: LIVENESS_POOL_MAX_CONNECTIONS, applicationName: "paperclip-controller-lease",
+  }, db));
   return db;
 }
 

packages/db/src/index.ts

@@ -1,6 +1,8 @@
 export {
   createDb,
   withDedicatedDbConnection,
+  livenessDb,
+  closeLivenessDb,
   closeRegisteredClients,
   getPostgresDataDirectory,
   ensurePostgresDatabase,

server/src/__tests__/server-startup-feedback-export.test.ts

@@ -258,6 +258,7 @@ vi.mock("@paperclipai/db", () => ({
   reconcilePendingMigrationHistory: vi.fn(async () => ({ repairedMigrations: [] })),
   formatDatabaseBackupResult: vi.fn(() => "ok"),
   runDatabaseBackup: vi.fn(),
+  closeLivenessDb: vi.fn(async () => {}),
   authUsers: {},
   companies: {},
   companyMemberships: {},

server/src/index.ts

@@ -66,6 +66,7 @@ import {
   companies,
   companyMemberships,
   instanceUserRoles,
+  closeLivenessDb,
 } from "@paperclipai/db";
 import detectPort from "detect-port";
 import { createApp } from "./app.js";
@@ -670,7 +671,7 @@ async function startServerWithDatabaseTeardown(
   const closeDatabaseClients = async () => {
     setSentryRunLinkResolver(null);
     const clients = pluginMigrationDb === db ? [db] : [db, pluginMigrationDb];
-    await Promise.all(clients.map((client) => endDatabaseClient(client, 5)));
+    await Promise.all([...clients.map((client) => endDatabaseClient(client, 5)), closeLivenessDb(db, 5)]);
   };
   startupDatabase.close = closeDatabaseClients;
   setSentryRunLinkResolver(createSentryRunLinkResolver(db));

server/src/services/legacy-controller-lease.test.ts

@@ -1,13 +1,21 @@
 import { randomUUID } from "node:crypto";
 import { eq, sql } from "drizzle-orm";
 import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
-import { agents, companies, createDb, heartbeatRuns } from "@paperclipai/db";
+import { agents, closeLivenessDb, companies, createDb, heartbeatRuns, livenessDb } from "@paperclipai/db";
 import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
 import { heartbeatService } from "./heartbeat.js";
 import { hasLiveLegacyController, LEGACY_CONTROLLER_LEASE_MS, LEGACY_CONTROLLER_RENEW_MS, legacyControllerBootId, legacyControllerClaim, LegacyControllerLeaseLostError,
   renewLegacyControllerLease, revokeExpiredLegacyController, revokeExpiredLegacyControllerWithState,
   watchLegacyControllerLease } from "./legacy-controller-lease.js";
 
+type WatchArgs = Parameters<typeof watchLegacyControllerLease>;
+// A renewal test double: each attempt runs in a transaction on the liveness pool.
+function fakeRenewalDb(returning: () => Promise<unknown>) {
+  const fake = { update: () => ({ set: () => ({ where: () => ({ returning }) }) }),
+    execute: async () => [], transaction: (action: (tx: unknown) => Promise<unknown>) => action(fake) };
+  return fake as unknown as WatchArgs[0];
+}
+
 const support = await getEmbeddedPostgresTestSupport();
 (support.supported ? describe : describe.skip)("durable legacy controller ownership", () => {
   let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
@@ -115,7 +123,7 @@ const support = await getEmbeddedPostgresTestSupport();
   });
   it("rejects dispatch at the lease deadline even if the database query never settles", async () => {
     const run = await seed();
-    const hungDb = { update: () => ({ set: () => ({ where: () => ({ returning: () => new Promise(() => {}) }) }) }) } as unknown as typeof db;
+    const hungDb = fakeRenewalDb(() => new Promise(() => {}));
     vi.useFakeTimers();
     const controller = new AbortController();
     const watch = watchLegacyControllerLease(hungDb, { ...run, controllerLeaseExpiresAt: new Date(Date.now() + 100) }, controller);
@@ -128,6 +136,60 @@ const support = await getEmbeddedPostgresTestSupport();
     } finally { watch.stop(); vi.useRealTimers(); }
   });
 
+  it("renews without queueing behind a slow statement on a saturated shared pool", async () => {
+    const run = await seed();
+    // postgres.js pipelines onto a busy connection when none is idle.
+    const shared = createDb(database.connectionString, { maxConnections: 1 });
+    const slow = shared.execute(sql`select pg_sleep(5)`).then(() => undefined); // drizzle sends on then()
+    const controller = new AbortController();
+    const watch = watchLegacyControllerLease(shared, run, controller);
+    try {
+      const startedAt = Date.now();
+      await watch.assertOwned("dispatching");
+      expect(Date.now() - startedAt).toBeLessThan(2_000);
+      expect(controller.signal.aborted).toBe(false);
+      const [saved] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id));
+      expect(saved.executionStage).toBe("dispatching");
+    } finally { watch.stop(); await slow; }
+  }, 15_000);
+
+  it("renews on one private pool per client that shutdown can end", async () => {
+    const run = await seed();
+    const owner = createDb(database.connectionString);
+    const pool = livenessDb(owner);
+    expect(pool).not.toBe(owner);
+    expect(livenessDb(owner)).toBe(pool);
+    const watch = watchLegacyControllerLease(owner, run, new AbortController());
+    try {
+      await watch.assertOwned();
+      const [{ name }] = await pool.execute<{ name: string }>(sql`select current_setting('application_name') as name`);
+      expect(name).toBe("paperclip-controller-lease");
+    } finally { watch.stop(); }
+    await closeLivenessDb(owner, 1);
+    await expect(livenessDb(owner).execute(sql`select 1`)).rejects.toThrow();
+  });
+
+  it("retries a renewal that times out waiting on the run row instead of losing the lease", async () => {
+    const run = await seed();
+    let lockTaken!: () => void, releaseLock!: () => void;
+    const taken = new Promise<void>((resolve) => { lockTaken = resolve; });
+    const released = new Promise<void>((resolve) => { releaseLock = resolve; });
+    const holder = db.transaction(async (tx) => {
+      await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)).for("update");
+      lockTaken();
+      await released;
+    });
+    await taken;
+    const controller = new AbortController();
+    const watch = watchLegacyControllerLease(db, run, controller, { renewalStatementTimeoutMs: 200 });
+    try {
+      const renewal = watch.assertOwned();
+      setTimeout(releaseLock, 1_000);
+      await expect(renewal).resolves.toBeUndefined();
+      expect(controller.signal.aborted).toBe(false);
+    } finally { releaseLock(); watch.stop(); await holder; }
+  }, 15_000);
+
   it("leaves native controller ownership to the native coordinator", () => {
     expect(legacyControllerClaim("native")).toEqual({});
   });
@@ -135,9 +197,7 @@ const support = await getEmbeddedPostgresTestSupport();
 
 describe("legacy controller lease loss attribution", () => {
   it("attributes a failing renewal query to renewal_error", async () => {
-    type WatchArgs = Parameters<typeof watchLegacyControllerLease>;
-    const failingDb = { update: () => ({ set: () => ({ where: () => ({
-      returning: () => Promise.reject(new Error("connection reset")) }) }) }) } as unknown as WatchArgs[0];
+    const failingDb = fakeRenewalDb(() => Promise.reject(new Error("connection reset")));
     vi.useFakeTimers();
     const controller = new AbortController();
     const run = { id: randomUUID(), companyId: randomUUID(), runtimeMode: "legacy", controllerBootId: legacyControllerBootId,
@@ -151,16 +211,15 @@ describe("legacy controller lease loss attribution", () => {
     } finally { watch.stop(); vi.useRealTimers(); }
   });
 
-  type WatchArgs = Parameters<typeof watchLegacyControllerLease>;
   const watchedRun = (leaseMs: number) => ({ id: randomUUID(), companyId: randomUUID(), runtimeMode: "legacy",
     controllerBootId: legacyControllerBootId, controllerLeaseExpiresAt: new Date(Date.now() + leaseMs) }) as WatchArgs[1];
 
   it("records a renewal still in flight when a stalled query reaches the deadline", async () => {
     let calls = 0;
-    const stallingDb = { update: () => ({ set: () => ({ where: () => ({ returning: () => {
+    const stallingDb = fakeRenewalDb(() => {
       calls += 1;
       return calls <= 2 ? Promise.resolve([{ id: "run" }]) : new Promise(() => {});
-    } }) }) }) } as unknown as WatchArgs[0];
+    });
     vi.useFakeTimers();
     const controller = new AbortController();
     const watch = watchLegacyControllerLease(stallingDb, watchedRun(LEGACY_CONTROLLER_LEASE_MS), controller);
@@ -174,8 +233,33 @@ describe("legacy controller lease loss attribution", () => {
     } finally { watch.stop(); vi.useRealTimers(); }
   });
 
+  it("measures the next deadline from the retry that renewed the lease", async () => {
+    const timeoutAfterMs = 45_000;
+    let calls = 0;
+    const retryingDb = fakeRenewalDb(() => {
+      calls += 1;
+      if (calls === 1) {
+        return new Promise((_, reject) => setTimeout(() => reject(Object.assign(new Error("statement timeout"), { code: "57014" })), timeoutAfterMs));
+      }
+      return calls === 2 ? Promise.resolve([{ id: "run" }]) : new Promise(() => {});
+    });
+    vi.useFakeTimers();
+    const controller = new AbortController();
+    const watch = watchLegacyControllerLease(retryingDb, watchedRun(LEGACY_CONTROLLER_LEASE_MS), controller);
+    try {
+      await vi.advanceTimersByTimeAsync(LEGACY_CONTROLLER_RENEW_MS + timeoutAfterMs);
+      expect(calls).toBe(2);
+      // The retry wrote a fresh 60s lease; the first attempt's start must not shorten it.
+      await vi.advanceTimersByTimeAsync(LEGACY_CONTROLLER_LEASE_MS - 1);
+      expect(controller.signal.aborted).toBe(false);
+      await vi.advanceTimersByTimeAsync(2);
+      expect(controller.signal.reason.leaseLossCause).toBe("renewal_deadline");
+      expect(controller.signal.reason.leaseLossTiming).toMatchObject({ sinceRenewalMs: LEGACY_CONTROLLER_LEASE_MS, renewalCount: 1 });
+    } finally { watch.stop(); vi.useRealTimers(); }
+  });
+
   it("records how late the deadline ran when the event loop was blocked", async () => {
-    const idleDb = { update: () => ({ set: () => ({ where: () => ({ returning: () => new Promise(() => {}) }) }) }) } as unknown as WatchArgs[0];
+    const idleDb = fakeRenewalDb(() => new Promise(() => {}));
     const controller = new AbortController();
     const watch = watchLegacyControllerLease(idleDb, watchedRun(10), controller);
     try {

server/src/services/legacy-controller-lease.ts

@@ -1,6 +1,6 @@
 import { randomUUID } from "node:crypto";
 import { and, eq, gt, lte, sql } from "drizzle-orm";
-import { heartbeatRuns, type Db } from "@paperclipai/db";
+import { heartbeatRuns, livenessDb, type Db } from "@paperclipai/db";
 import type { LegacyControllerLeaseLossCause, LegacyControllerLeaseLossTiming } from "./run-cancellation.js";
 
 // A boot UUID has meaning across containers; a numeric PID does not.
@@ -52,6 +52,31 @@ export async function renewLegacyControllerLease(
   return Boolean(renewed);
 }
 
+/** One attempt may wait at most this long; a timed-out attempt is retried
+ * until the lease deadline instead of holding the renewal for the whole lease. */
+export const LEGACY_CONTROLLER_RENEWAL_STATEMENT_TIMEOUT_MS = 10_000;
+
+function isStatementTimeout(error: unknown): boolean {
+  for (let cause = error, depth = 0; cause && depth < 4; cause = (cause as { cause?: unknown }).cause, depth++) {
+    if ((cause as { code?: unknown }).code === "57014") return true;
+  }
+  return false;
+}
+
+/** Renew on the private liveness pool so the write cannot queue behind an
+ * unrelated slow statement on a busy shared connection. */
+async function renewOnLivenessPool(
+  db: Db,
+  run: Pick<Run, "id" | "companyId" | "controllerBootId">,
+  stage: "dispatching" | undefined,
+  statementTimeoutMs: number,
+): Promise<boolean> {
+  return livenessDb(db).transaction(async (tx) => {
+    await tx.execute(sql`select set_config('statement_timeout', ${`${statementTimeoutMs}ms`}, true)`);
+    return renewLegacyControllerLease(tx as unknown as Db, run, stage);
+  });
+}
+
 export async function hasLiveLegacyController(db: Db, run: Run): Promise<boolean> {
   if (run.runtimeMode === "native" || !run.controllerBootId) return false;
   const [owner] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
@@ -98,7 +123,12 @@ export async function revokeExpiredLegacyController(db: Db, run: Run): Promise<b
 
 /** Abort the adapter if the controller cannot renew. Bound each check by the
  * lease duration even when the database connection never settles. */
-export function watchLegacyControllerLease(db: Db, run: Run, controller: AbortController) {
+export function watchLegacyControllerLease(
+  db: Db,
+  run: Run,
+  controller: AbortController,
+  { renewalStatementTimeoutMs = LEGACY_CONTROLLER_RENEWAL_STATEMENT_TIMEOUT_MS } = {},
+) {
   if (run.runtimeMode === "native" || !run.controllerBootId) {
     return { stop() {}, async assertOwned(_stage?: "dispatching") {} };
   }
@@ -141,9 +171,22 @@ export function watchLegacyControllerLease(db: Db, run: Run, controller: AbortCo
       onAbort = () => reject(controller.signal.reason);
       controller.signal.addEventListener("abort", onAbort, { once: true });
     });
+    // The successful attempt's UPDATE writes a fresh lease, so the local
+    // deadline is measured from that attempt rather than the first one.
+    let attemptStartedAt = startedAt;
+    const renew = async () => {
+      for (;;) {
+        attemptStartedAt = Date.now();
+        try {
+          return await renewOnLivenessPool(db, run, stage, renewalStatementTimeoutMs);
+        } catch (error) {
+          if (!isStatementTimeout(error) || stopped || controller.signal.aborted) throw error;
+        }
+      }
+    };
     let renewed: boolean;
     try {
-      renewed = await Promise.race([renewLegacyControllerLease(db, run, stage), aborted]);
+      renewed = await Promise.race([renew(), aborted]);
     } finally {
       renewalsInFlight.delete(renewal);
       controller.signal.removeEventListener("abort", onAbort);
@@ -157,7 +200,7 @@ export function watchLegacyControllerLease(db: Db, run: Run, controller: AbortCo
     if (!stopped) {
       lastRenewedAt = Date.now();
       renewalCount += 1;
-      armDeadline(Math.max(0, LEGACY_CONTROLLER_LEASE_MS - (Date.now() - startedAt)));
+      armDeadline(Math.max(0, LEGACY_CONTROLLER_LEASE_MS - (Date.now() - attemptStartedAt)));
     }
   };
   const timer = setInterval(() => {