Files
PaperClipAI/server/src/services/native-runtime/native-execution-lease-batch.test.ts
T
DottaandClaude Opus 5.5 754b1f36d3 perf(native): renew all of a process's execution leases in one transaction, without a recovery wake (#21444)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every running native agent. Each run
holds a native execution lease that the process must renew before it
expires (60 seconds for a remote run, 20 minutes for a local run)
> - Each run renewed its lease on its own timer, in its own transaction,
on a private pool of two connections. At 1,000 remote runs that is about
50 transactions each second through two connections
> - When renewals queue behind each other, a renewal can miss its 60
second deadline, and the run is lost although the server is healthy
> - Each renewal also woke six live-run recovery queues for its run. The
recovery owner then re-checked a healthy run about every 20 seconds,
although a renewal only moves the deadline later
> - This pull request renews all leases of one process with one shared
timer. All leases that are due go into one transaction of fixed size,
the same model that the legacy controller lease already uses. A renewal
no longer wakes recovery, as #20924 did for the legacy lease
> - The benefit is that the renewal cost no longer grows with the number
of runs, and all safety properties of a single renewal stay the same

## Linked Issues or Issue Description

Refs #20924 (the same change for the legacy controller lease), #21242.

**What existing behavior does this improve?**
Native execution lease renewal in
`server/src/services/native-runtime/native-execution-lease.ts`. Each run
had one `setInterval` and one transaction per renewal: statement
timeout, row lock, recovery notification, update, commit.

**Subsystem affected**
Server: native runtime, controller lease renewal on the liveness
connection pool.

**Current behavior**
The renewal cost grows with the number of runs. A local load model with
1,000 remote runs sends 35.7 lease transactions and 212.8 lease
statements each second, and the event-loop utilization is 0.34. All of
them use a pool of two connections.

**Proposed behavior**
One timer for each database client renews all due leases in one
transaction for each 200 runs. With 1,000 remote runs the same model
sends 0.29 lease transactions and 1.7 lease statements each second, and
the event-loop utilization is 0.07.

**Reason and benefit**
The plan for 1,000 concurrent agents needs renewals that do not scale
with the run count (item 1d). A renewal that misses its deadline loses
the run.

**Breaking changes**
None. The `NativeExecutionLeaseRenewal` contract (`stop()`,
`remainingMs()`) and `renewNativeSessionExecutionLease` do not change.

## What Changed

- `native-execution-lease.ts`: a renewal (batched or single) no longer
calls `notifyLiveRunRecovery`. A renewal only moves the lease deadline
later. Recovery already holds a deadline at the earlier expiry. At that
deadline it re-reads the row and moves to the new expiry. A lapsed lease
is still found at its expiry. This change does not touch the signals of
the writers that release, revoke or take over a lease, or that change
the run status. This also removes six notifications for each renewal,
which was 1,200 notifications for a batch of 200 runs during a deploy
overlap.
- `native-execution-lease.ts`: one shared schedule for each database
client replaces the timer of each run. When the first lease is due, all
leases that are due within a quarter of their own interval renew with
it. Thus the leases of a process converge on one batch for each
interval.
- A batch is one transaction on the liveness pool for each 200 runs:
`statement_timeout`, one `SELECT … FOR UPDATE SKIP LOCKED` of the
coordinator rows, and one `UPDATE … FROM (VALUES …)`.
- The batch keeps the guarantees of a single renewal:
- It locks each coordinator row before it reads the expiry, as before.
Thus an owner whose lease expired while another writer held the row is
never revived.
- It extends a row only for the exact owner: run, company, task, lease
owner, attempt, controller boot id, pid, process start, and a live
lease.
- The confirmed deadline is sampled after the locks and before the
update, as before.
- A run whose row was locked and not extended fails with
`native_session_lease_lost`. The other runs in the batch succeed.
- These rows do not renew in the batch. They renew alone through the
unchanged single-run path:
- A row that another writer holds (`SKIP LOCKED`). Thus one busy row
cannot hold the batch until the statement timeout.
- A row that the lock did not find (another owner, or no row). The
single path gives the same answer as before.
- A local lease that lapsed (host sleep). Only the single path may take
it back, under the run row lock.
- A batch that is cancelled (statement timeout `57014` or deadlock
`40P01`) extends nothing. Its runs retry after 1 second, and each run
keeps its own deadline timer. Before this change, a statement timeout
waited for the next 20 second tick.
- Any other batch failure, also an error thrown while the statement is
built, may come from one bad row. Each run of that batch then renews
alone at once through the single path, so a bad row can cost only its
own run. A run that fails alone while others in its batch renew keeps
renewing alone (out of later batches) until a single renewal succeeds.
During a database outage the single renewals also fail; those runs
return to the batch at their next interval, not after a 1 second retry,
so an outage causes at most one queued single transaction for each run
and interval.
- Runs that become due while a batch is in flight wait for that batch. A
batch that runs for more than 15 seconds no longer holds back the next
batch.
- A tick that passed while a renewal was in flight is skipped, as a
fixed interval skips it. Thus a slow database gets no extra load.
- The shared timer replaces the interval timer of each run. It exists
only while this process holds a lease, and it fires at the next renewal
deadline. With no held lease there is no timer and no SQL.
- New export `renewNativeExecutionLeasesNow(db)` renews all leases of
the process at once. Tests use it.

## Verification

- New PostgreSQL tests in `native-execution-lease-batch.test.ts`:
- 3 runs and 60 runs each renew in 1 transaction with the same number of
statements.
  - 450 runs renew in 3 transactions.
- In one batch, the runs whose lease was taken over, whose controller
boot changed, or whose lease expired fail with
`native_session_lease_lost`. Their rows do not change. The healthy runs
renew. A newer attempt under the same owner is not lost, as before.
- Interleaving: another transaction holds a coordinator row from before
its lease expires until after it expired. The healthy run in the same
batch renews at once. After the lock is released, the expired owner is
lost and its row stays expired.
- A takeover that commits before the batch: the batch extends nothing
and the run is lost.
- A local and a remote lease renew in one transaction, each for its own
lease length.
- A poison binding (an attempt outside the integer range) makes the
shared UPDATE fail with `22003`. The other 4 runs still renew in the
same tick, only the poison run is lost, and the next tick renews the 4
runs in 1 transaction again.
- An invalid date throws while the statement is built. The other runs
renew, the broken run is lost alone, and no lease stays busy (the next
call renews again in 1 transaction).
- New PostgreSQL test in `live-recovery-work.test.ts` with the real
registered recovery owners: another server's native run renews its
lease. No recovery pass runs after the renewal. At the old expiry
recovery re-reads the row, keeps the run running and moves its deadline
to the new expiry. When renewals stop, recovery acts on the run within
10 seconds after the end of the last lease.
- Red first: with a shim on `origin/master` that renews each lease
alone, the transaction test failed (`expected 3 to be 1`) and the
450-run test failed.
- The unit tests in `native-execution-lease.test.ts` now drive the
batch. New tests show that a non-cancelled batch failure renews the run
alone at once, and that an outage (batch and single renewal both fail)
does not retry every second and loses no run.
- `native-execution-lease-reacquire.test.ts`: the host-sleep tests now
go through the batch, which hands the lapsed local lease to the single
path.
- Load model (`startNativeSessionExecutionLeaseRenewal` for N remote
runs that start at random times over 20 seconds, 70 second window, local
PostgreSQL, machine load average about 50):

| Runs | Lease transactions/s before → after | Lease statements/s before
→ after | Event-loop utilization before → after | Smallest margin to the
60 s deadline | Lost |
| --- | --- | --- | --- | --- | --- |
| 50 | 1.81 → 0.11 | 10.9 → 0.69 | 0.015 → 0.095 (noise) | 40.0 s → 40.0
s | 0 → 0 |
| 1,000 | 35.67 → 0.29 | 212.8 → 1.71 | 0.341 → 0.065 | 39.0 s → 40.0 s
| 0 → 0 |

The window includes the 20 second ramp, so the steady rate before is
higher (one transaction each 20 seconds for each run, 50/s at 1,000
runs). After the change, one batch carries up to 200 runs.

## Risks

- **Behavior change: batch-wide error handling.** One shared transaction
now carries up to 200 runs, so one database error reaches all of them. A
cancelled batch only skips the extension and retries after 1 second. Any
other batch error sends each run to the single path at once, so one bad
row (a bad value, a cast error, an error while the statement is built)
costs only its own run. In a database outage, up to one single
transaction for each run waits on the two-connection pool at each
interval. Each run's own deadline is unchanged.
- **Behavior change: no recovery signal on renewal.** See below.
- During a deploy overlap the old server renews its runs one by one and
the new server renews in batches. Both use the same row lock and the
same conditions, so they do not conflict. Rows written by the old
version have the same columns.
- `SKIP LOCKED` sends a busy row to the single path, which waits for the
lock as before.
- **No recovery signal on renewal:** recovery of a healthy run of
another server now runs once for each lease period at the old expiry,
not after each renewal. A dead run is found at the same time as before:
at the expiry of its last lease.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:16 -06:00

293 lines
15 KiB
TypeScript

import { randomUUID } from "node:crypto";
import { eq, inArray, sql } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import {
agents, closeLivenessDb, companies, createDb, heartbeatRuns, issues, livenessDb, nativeRunFinalizations,
} from "@paperclipai/db";
import {
renewNativeExecutionLeasesNow, startNativeSessionExecutionLeaseRenewal, type NativeExecutionLeaseRenewal,
} from "./native-execution-lease.js";
import { REMOTE_CONTROLLER_LEASE_MS } from "./native-runner-recovery-contract.js";
import { meterDatabaseStatements } from "../../__tests__/helpers/database-statement-meter.js";
import {
getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase,
} from "../../__tests__/helpers/embedded-postgres.js";
const support = await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = support.supported ? describe : describe.skip;
if (!support.supported) console.warn(`Skipping batched native lease renewal tests: ${support.reason ?? "unsupported environment"}`);
const controller = { bootId: `boot-${randomUUID()}`, pid: 5151, processStartedAt: new Date("2026-11-13T08:00:00.000Z") };
/**
* One server process renews the native execution leases of all its runs in
* one shared transaction per tick, on the private two-connection liveness pool.
*/
describeEmbeddedPostgres("batched native execution lease renewal", () => {
let db!: ReturnType<typeof createDb>;
let owner!: ReturnType<typeof createDb>;
let meter!: ReturnType<typeof meterDatabaseStatements>;
let connectionString!: string;
let stopDatabase: (() => Promise<void>) | undefined;
let companyId!: string;
let agentId!: string;
beforeAll(async () => {
const started = await startEmbeddedPostgresTestDatabase("paperclip-native-lease-batch-");
stopDatabase = started.cleanup;
connectionString = started.connectionString;
db = createDb(connectionString);
owner = createDb(connectionString);
meter = meterDatabaseStatements(livenessDb(owner), connectionString);
companyId = randomUUID();
agentId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Lease batch", issuePrefix: "LBT",
requireBoardApprovalForNewAgents: false });
await db.insert(agents).values({ id: agentId, companyId, name: "Runner", role: "engineer", status: "active",
adapterType: "paperclip_runner", adapterConfig: { provider: "codex" }, runtimeConfig: {}, permissions: {} });
}, 120_000);
afterAll(async () => {
if (owner) await closeLivenessDb(owner, 1).catch(() => undefined);
await owner?.$client.end({ timeout: 1 });
await db?.$client.end({ timeout: 1 });
await stopDatabase?.();
});
type Seeded = { runId: string; issueId: string; leaseOwner: string };
async function seedRuns(count: number, leaseMs = 30_000): Promise<Seeded[]> {
const runs = Array.from({ length: count }, () => ({ runId: randomUUID(), issueId: randomUUID(),
leaseOwner: `runner:${randomUUID()}` }));
await db.insert(issues).values(runs.map(run => ({ id: run.issueId, companyId, identifier: `LBT-${run.runId.slice(0, 8)}`,
title: "batch", status: "in_progress", priority: "medium", workMode: "standard", assigneeAgentId: agentId })));
await db.insert(heartbeatRuns).values(runs.map(run => ({ id: run.runId, companyId, agentId, status: "running",
runtimeMode: "native", nativeIssueId: run.issueId, invocationSource: "assignment", triggerDetail: "system",
contextSnapshot: { issueId: run.issueId } })));
await db.insert(nativeRunFinalizations).values(runs.map(run => ({ runId: run.runId, companyId, issueId: run.issueId,
phase: "observed", attempt: 1, leaseOwner: run.leaseOwner,
leaseExpiresAt: sql`clock_timestamp() + (${leaseMs} * interval '1 millisecond')`,
controllerBootId: controller.bootId, controllerPid: controller.pid,
controllerProcessStartedAt: controller.processStartedAt, controllerGeneration: 1 })));
return runs;
}
function start(run: Seeded, overrides: { remote?: boolean; controllerGeneration?: number } = {}) {
const onLost = vi.fn(async () => undefined);
const guard = startNativeSessionExecutionLeaseRenewal({ db: owner, companyId, ...run, attempt: 1, controller,
remote: overrides.remote ?? true, controllerGeneration: overrides.controllerGeneration,
confirmedDeadline: performance.now() + 25_000, onLost });
return { run, guard, onLost };
}
async function expiries(runIds: string[]) {
const rows = await db.select({ runId: nativeRunFinalizations.runId, expiresAt: nativeRunFinalizations.leaseExpiresAt,
leaseOwner: nativeRunFinalizations.leaseOwner })
.from(nativeRunFinalizations).where(inArray(nativeRunFinalizations.runId, runIds));
return new Map(rows.map(row => [row.runId, row]));
}
async function stopAll(guards: NativeExecutionLeaseRenewal[]) {
await Promise.all(guards.map(guard => guard.stop().catch(() => undefined)));
}
async function renewCost(count: number) {
const runs = await seedRuns(count);
const started = runs.map(run => start(run));
try {
// Warm the pool so connection setup is not counted.
await renewNativeExecutionLeasesNow(owner);
const before = meter.read();
await renewNativeExecutionLeasesNow(owner);
const after = meter.read();
for (const { guard, onLost } of started) {
expect(onLost).not.toHaveBeenCalled();
expect(guard.remainingMs()).toBeGreaterThan(REMOTE_CONTROLLER_LEASE_MS - 5_000);
}
const saved = await expiries(runs.map(run => run.runId));
for (const run of runs) expect(saved.get(run.runId)!.expiresAt!.getTime()).toBeGreaterThan(Date.now() + 50_000);
return { transactions: after.transactions - before.transactions, statements: after.statements - before.statements };
} finally {
await stopAll(started.map(item => item.guard));
}
}
it("renews every lease of the process in one transaction whose size does not grow with the run count", async () => {
const few = await renewCost(3);
const many = await renewCost(60);
expect(few.transactions).toBe(1);
expect(many.transactions).toBe(1);
expect(many.statements).toBe(few.statements);
}, 60_000);
it("splits a very large tick into bounded transactions", async () => {
const cost = await renewCost(450);
expect(cost.transactions).toBe(3);
}, 120_000);
it("fails only the runs whose lease is no longer theirs and renews the rest", async () => {
const runs = await seedRuns(6);
const [healthy, other, takenOver, newAttempt, otherBoot, expired] = runs;
await db.update(nativeRunFinalizations).set({ leaseOwner: `native-finalizer:${randomUUID()}` })
.where(eq(nativeRunFinalizations.runId, takenOver!.runId));
await db.update(nativeRunFinalizations).set({ attempt: 2 }).where(eq(nativeRunFinalizations.runId, newAttempt!.runId));
await db.update(nativeRunFinalizations).set({ controllerBootId: `boot-${randomUUID()}` })
.where(eq(nativeRunFinalizations.runId, otherBoot!.runId));
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: sql`clock_timestamp() - interval '1 second'` })
.where(eq(nativeRunFinalizations.runId, expired!.runId));
const before = await expiries(runs.map(run => run.runId));
const started = runs.map(run => start(run));
try {
await renewNativeExecutionLeasesNow(owner);
// A newer attempt under the same owner is the owner's own executor
// moving on: not extended and not lost, as for a single renewal.
const [ok, ok2, takenOverRenewal, advanced, otherBootRenewal, expiredRenewal] = started;
const lost = [takenOverRenewal!, otherBootRenewal!, expiredRenewal!];
expect(advanced!.onLost).not.toHaveBeenCalled();
for (const item of [ok!, ok2!]) {
expect(item.onLost).not.toHaveBeenCalled();
await expect(item.guard.stop()).resolves.toBeUndefined();
}
for (const item of lost) {
await vi.waitFor(() => expect(item.onLost).toHaveBeenCalledTimes(1));
await expect(item.guard.stop()).rejects.toThrow("native_session_lease_lost");
}
const after = await expiries(runs.map(run => run.runId));
for (const run of [healthy!, other!]) {
expect(after.get(run.runId)!.expiresAt!.getTime()).toBeGreaterThan(before.get(run.runId)!.expiresAt!.getTime());
}
// Nothing the batch could not extend was written, and the expired owner stays expired.
for (const run of [takenOver!, newAttempt!, otherBoot!, expired!]) expect(after.get(run.runId)).toEqual(before.get(run.runId));
} finally {
await stopAll(started.map(item => item.guard));
}
}, 60_000);
it("never revives an owner that expired while another writer held its row, and does not wait for that row", async () => {
const [contended, healthy] = await seedRuns(2, 1_500);
// Another actor (for example restart recovery) holds the coordinator row
// from before the lease expires until after it expired, then commits.
let release!: () => void;
let locked!: () => void;
const held = new Promise<void>(resolve => { release = resolve; });
const acquired = new Promise<void>(resolve => { locked = resolve; });
const writer = db.transaction(async tx => {
await tx.execute(sql`select 1 from ${nativeRunFinalizations} where ${nativeRunFinalizations.runId} = ${contended!.runId} for update`);
locked();
await held;
});
await acquired;
const contendedRenewal = start(contended!);
const healthyRenewal = start(healthy!);
try {
const renewing = renewNativeExecutionLeasesNow(owner);
// The busy row does not hold up the rest of the batch.
await vi.waitFor(async () => expect((await expiries([healthy!.runId])).get(healthy!.runId)!.expiresAt!.getTime())
.toBeGreaterThan(Date.now() + 50_000), { timeout: 5_000 });
await new Promise(resolve => setTimeout(resolve, 2_000));
expect((await expiries([contended!.runId])).get(contended!.runId)!.expiresAt!.getTime()).toBeLessThan(Date.now());
release();
await writer;
await renewing;
await vi.waitFor(() => expect(contendedRenewal.onLost).toHaveBeenCalledTimes(1));
await expect(contendedRenewal.guard.stop()).rejects.toThrow("native_session_lease_lost");
expect((await expiries([contended!.runId])).get(contended!.runId)!.expiresAt!.getTime()).toBeLessThan(Date.now());
expect(healthyRenewal.onLost).not.toHaveBeenCalled();
} finally {
release();
await writer.catch(() => undefined);
await stopAll([contendedRenewal.guard, healthyRenewal.guard]);
}
}, 60_000);
it("never revives an owner whose row a takeover rewrote while the batch waited", async () => {
const [run] = await seedRuns(1, 60_000);
const renewal = start(run!);
// A takeover commits between the batch's statements on another connection;
// the batch locks the row and reads its committed owner, so it extends nothing.
await db.update(nativeRunFinalizations).set({ leaseOwner: `native-cleanup:${randomUUID()}`,
leaseExpiresAt: sql`clock_timestamp() + interval '5 minutes'` }).where(eq(nativeRunFinalizations.runId, run!.runId));
const before = await expiries([run!.runId]);
try {
await renewNativeExecutionLeasesNow(owner);
await vi.waitFor(() => expect(renewal.onLost).toHaveBeenCalledTimes(1));
expect(await expiries([run!.runId])).toEqual(before);
} finally {
await renewal.guard.stop().catch(() => undefined);
}
}, 60_000);
it("confines a binding that makes the shared statement fail to its own run", async () => {
const runs = await seedRuns(5);
const before = await expiries(runs.map(run => run.runId));
const healthy = runs.slice(1).map(run => start(run));
// An attempt outside the integer range: PostgreSQL rejects the whole
// shared UPDATE (22003) for every run in it.
const poison = { run: runs[0]!, onLost: vi.fn(async () => undefined), guard: startNativeSessionExecutionLeaseRenewal({
db: owner, companyId, ...runs[0]!, attempt: 2 ** 40, controller, remote: true,
confirmedDeadline: performance.now() + 25_000, onLost: async () => { await poison.onLost(); } }) };
try {
await renewNativeExecutionLeasesNow(owner);
const after = await expiries(runs.map(run => run.runId));
for (const { run, onLost, guard } of healthy) {
expect(onLost).not.toHaveBeenCalled();
expect(after.get(run.runId)!.expiresAt!.getTime()).toBeGreaterThan(before.get(run.runId)!.expiresAt!.getTime());
expect(guard.remainingMs()).toBeGreaterThan(REMOTE_CONTROLLER_LEASE_MS - 5_000);
}
expect(after.get(poison.run.runId)).toEqual(before.get(poison.run.runId));
// The healthy runs go back to one shared transaction; nothing is stuck busy.
const cost = meter.read();
await renewNativeExecutionLeasesNow(owner);
const healthyCost = meter.read();
expect(healthyCost.transactions - cost.transactions).toBe(1);
for (const { onLost } of healthy) expect(onLost).not.toHaveBeenCalled();
// Only the bad binding's own run is lost.
await vi.waitFor(() => expect(poison.onLost).toHaveBeenCalledTimes(1));
} finally {
await stopAll([...healthy.map(item => item.guard), poison.guard]);
}
}, 60_000);
it("handles an error thrown while the shared statement is built and leaves no lease busy", async () => {
const runs = await seedRuns(3);
const healthy = runs.slice(1).map(run => start(run));
const onLost = vi.fn(async () => undefined);
// An invalid date throws in toISOString() while the VALUES list is built.
const broken = startNativeSessionExecutionLeaseRenewal({ db: owner, companyId, ...runs[0]!, attempt: 1,
controller: { ...controller, processStartedAt: new Date(Number.NaN) }, remote: true,
confirmedDeadline: performance.now() + 25_000, onLost });
try {
await renewNativeExecutionLeasesNow(owner);
const after = await expiries(runs.map(run => run.runId));
for (const { run, onLost: healthyLost } of healthy) {
expect(healthyLost).not.toHaveBeenCalled();
expect(after.get(run.runId)!.expiresAt!.getTime()).toBeGreaterThan(Date.now() + 50_000);
}
// Every lease settled: a second call renews again instead of finding them busy.
const before = meter.read();
await renewNativeExecutionLeasesNow(owner);
expect(meter.read().transactions - before.transactions).toBe(1);
for (const { guard } of healthy) expect(guard.remainingMs()).toBeGreaterThan(REMOTE_CONTROLLER_LEASE_MS - 5_000);
await vi.waitFor(() => expect(onLost).toHaveBeenCalledTimes(1));
} finally {
await stopAll([...healthy.map(item => item.guard), broken]);
}
}, 60_000);
it("renews local and remote leases together, each for its own lease length", async () => {
const [remote, local] = await seedRuns(2);
const started = [start(remote!), start(local!, { remote: false, controllerGeneration: 1 })];
try {
const before = meter.read();
await renewNativeExecutionLeasesNow(owner);
expect(meter.read().transactions - before.transactions).toBe(1);
const saved = await expiries([remote!.runId, local!.runId]);
expect(saved.get(remote!.runId)!.expiresAt!.getTime()).toBeLessThan(Date.now() + 2 * 60_000);
expect(saved.get(local!.runId)!.expiresAt!.getTime()).toBeGreaterThan(Date.now() + 15 * 60_000);
expect(started[1]!.guard.remainingMs()).toBeGreaterThan(15 * 60_000);
} finally {
await stopAll(started.map(item => item.guard));
}
}, 60_000);
});