mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 20:12:20 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process controls every running native agent. Each run holds a native execution lease that the process must renew before it expires (60 seconds for a remote run, 20 minutes for a local run) > - Each run renewed its lease on its own timer, in its own transaction, on a private pool of two connections. At 1,000 remote runs that is about 50 transactions each second through two connections > - When renewals queue behind each other, a renewal can miss its 60 second deadline, and the run is lost although the server is healthy > - Each renewal also woke six live-run recovery queues for its run. The recovery owner then re-checked a healthy run about every 20 seconds, although a renewal only moves the deadline later > - This pull request renews all leases of one process with one shared timer. All leases that are due go into one transaction of fixed size, the same model that the legacy controller lease already uses. A renewal no longer wakes recovery, as #20924 did for the legacy lease > - The benefit is that the renewal cost no longer grows with the number of runs, and all safety properties of a single renewal stay the same ## Linked Issues or Issue Description Refs #20924 (the same change for the legacy controller lease), #21242. **What existing behavior does this improve?** Native execution lease renewal in `server/src/services/native-runtime/native-execution-lease.ts`. Each run had one `setInterval` and one transaction per renewal: statement timeout, row lock, recovery notification, update, commit. **Subsystem affected** Server: native runtime, controller lease renewal on the liveness connection pool. **Current behavior** The renewal cost grows with the number of runs. A local load model with 1,000 remote runs sends 35.7 lease transactions and 212.8 lease statements each second, and the event-loop utilization is 0.34. All of them use a pool of two connections. **Proposed behavior** One timer for each database client renews all due leases in one transaction for each 200 runs. With 1,000 remote runs the same model sends 0.29 lease transactions and 1.7 lease statements each second, and the event-loop utilization is 0.07. **Reason and benefit** The plan for 1,000 concurrent agents needs renewals that do not scale with the run count (item 1d). A renewal that misses its deadline loses the run. **Breaking changes** None. The `NativeExecutionLeaseRenewal` contract (`stop()`, `remainingMs()`) and `renewNativeSessionExecutionLease` do not change. ## What Changed - `native-execution-lease.ts`: a renewal (batched or single) no longer calls `notifyLiveRunRecovery`. A renewal only moves the lease deadline later. Recovery already holds a deadline at the earlier expiry. At that deadline it re-reads the row and moves to the new expiry. A lapsed lease is still found at its expiry. This change does not touch the signals of the writers that release, revoke or take over a lease, or that change the run status. This also removes six notifications for each renewal, which was 1,200 notifications for a batch of 200 runs during a deploy overlap. - `native-execution-lease.ts`: one shared schedule for each database client replaces the timer of each run. When the first lease is due, all leases that are due within a quarter of their own interval renew with it. Thus the leases of a process converge on one batch for each interval. - A batch is one transaction on the liveness pool for each 200 runs: `statement_timeout`, one `SELECT … FOR UPDATE SKIP LOCKED` of the coordinator rows, and one `UPDATE … FROM (VALUES …)`. - The batch keeps the guarantees of a single renewal: - It locks each coordinator row before it reads the expiry, as before. Thus an owner whose lease expired while another writer held the row is never revived. - It extends a row only for the exact owner: run, company, task, lease owner, attempt, controller boot id, pid, process start, and a live lease. - The confirmed deadline is sampled after the locks and before the update, as before. - A run whose row was locked and not extended fails with `native_session_lease_lost`. The other runs in the batch succeed. - These rows do not renew in the batch. They renew alone through the unchanged single-run path: - A row that another writer holds (`SKIP LOCKED`). Thus one busy row cannot hold the batch until the statement timeout. - A row that the lock did not find (another owner, or no row). The single path gives the same answer as before. - A local lease that lapsed (host sleep). Only the single path may take it back, under the run row lock. - A batch that is cancelled (statement timeout `57014` or deadlock `40P01`) extends nothing. Its runs retry after 1 second, and each run keeps its own deadline timer. Before this change, a statement timeout waited for the next 20 second tick. - Any other batch failure, also an error thrown while the statement is built, may come from one bad row. Each run of that batch then renews alone at once through the single path, so a bad row can cost only its own run. A run that fails alone while others in its batch renew keeps renewing alone (out of later batches) until a single renewal succeeds. During a database outage the single renewals also fail; those runs return to the batch at their next interval, not after a 1 second retry, so an outage causes at most one queued single transaction for each run and interval. - Runs that become due while a batch is in flight wait for that batch. A batch that runs for more than 15 seconds no longer holds back the next batch. - A tick that passed while a renewal was in flight is skipped, as a fixed interval skips it. Thus a slow database gets no extra load. - The shared timer replaces the interval timer of each run. It exists only while this process holds a lease, and it fires at the next renewal deadline. With no held lease there is no timer and no SQL. - New export `renewNativeExecutionLeasesNow(db)` renews all leases of the process at once. Tests use it. ## Verification - New PostgreSQL tests in `native-execution-lease-batch.test.ts`: - 3 runs and 60 runs each renew in 1 transaction with the same number of statements. - 450 runs renew in 3 transactions. - In one batch, the runs whose lease was taken over, whose controller boot changed, or whose lease expired fail with `native_session_lease_lost`. Their rows do not change. The healthy runs renew. A newer attempt under the same owner is not lost, as before. - Interleaving: another transaction holds a coordinator row from before its lease expires until after it expired. The healthy run in the same batch renews at once. After the lock is released, the expired owner is lost and its row stays expired. - A takeover that commits before the batch: the batch extends nothing and the run is lost. - A local and a remote lease renew in one transaction, each for its own lease length. - A poison binding (an attempt outside the integer range) makes the shared UPDATE fail with `22003`. The other 4 runs still renew in the same tick, only the poison run is lost, and the next tick renews the 4 runs in 1 transaction again. - An invalid date throws while the statement is built. The other runs renew, the broken run is lost alone, and no lease stays busy (the next call renews again in 1 transaction). - New PostgreSQL test in `live-recovery-work.test.ts` with the real registered recovery owners: another server's native run renews its lease. No recovery pass runs after the renewal. At the old expiry recovery re-reads the row, keeps the run running and moves its deadline to the new expiry. When renewals stop, recovery acts on the run within 10 seconds after the end of the last lease. - Red first: with a shim on `origin/master` that renews each lease alone, the transaction test failed (`expected 3 to be 1`) and the 450-run test failed. - The unit tests in `native-execution-lease.test.ts` now drive the batch. New tests show that a non-cancelled batch failure renews the run alone at once, and that an outage (batch and single renewal both fail) does not retry every second and loses no run. - `native-execution-lease-reacquire.test.ts`: the host-sleep tests now go through the batch, which hands the lapsed local lease to the single path. - Load model (`startNativeSessionExecutionLeaseRenewal` for N remote runs that start at random times over 20 seconds, 70 second window, local PostgreSQL, machine load average about 50): | Runs | Lease transactions/s before → after | Lease statements/s before → after | Event-loop utilization before → after | Smallest margin to the 60 s deadline | Lost | | --- | --- | --- | --- | --- | --- | | 50 | 1.81 → 0.11 | 10.9 → 0.69 | 0.015 → 0.095 (noise) | 40.0 s → 40.0 s | 0 → 0 | | 1,000 | 35.67 → 0.29 | 212.8 → 1.71 | 0.341 → 0.065 | 39.0 s → 40.0 s | 0 → 0 | The window includes the 20 second ramp, so the steady rate before is higher (one transaction each 20 seconds for each run, 50/s at 1,000 runs). After the change, one batch carries up to 200 runs. ## Risks - **Behavior change: batch-wide error handling.** One shared transaction now carries up to 200 runs, so one database error reaches all of them. A cancelled batch only skips the extension and retries after 1 second. Any other batch error sends each run to the single path at once, so one bad row (a bad value, a cast error, an error while the statement is built) costs only its own run. In a database outage, up to one single transaction for each run waits on the two-connection pool at each interval. Each run's own deadline is unchanged. - **Behavior change: no recovery signal on renewal.** See below. - During a deploy overlap the old server renews its runs one by one and the new server renews in batches. Both use the same row lock and the same conditions, so they do not conflict. Rows written by the old version have the same columns. - `SKIP LOCKED` sends a busy row to the single path, which waits for the lock as before. - **No recovery signal on renewal:** recovery of a healthy run of another server now runs once for each lease period at the old expiry, not after each renewal. A dead run is found at the same time as before: at the expiry of its last lease. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
293 lines
15 KiB
TypeScript
293 lines
15 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { eq, inArray, sql } from "drizzle-orm";
|
|
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import {
|
|
agents, closeLivenessDb, companies, createDb, heartbeatRuns, issues, livenessDb, nativeRunFinalizations,
|
|
} from "@paperclipai/db";
|
|
import {
|
|
renewNativeExecutionLeasesNow, startNativeSessionExecutionLeaseRenewal, type NativeExecutionLeaseRenewal,
|
|
} from "./native-execution-lease.js";
|
|
import { REMOTE_CONTROLLER_LEASE_MS } from "./native-runner-recovery-contract.js";
|
|
import { meterDatabaseStatements } from "../../__tests__/helpers/database-statement-meter.js";
|
|
import {
|
|
getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase,
|
|
} from "../../__tests__/helpers/embedded-postgres.js";
|
|
|
|
const support = await getEmbeddedPostgresTestSupport();
|
|
const describeEmbeddedPostgres = support.supported ? describe : describe.skip;
|
|
if (!support.supported) console.warn(`Skipping batched native lease renewal tests: ${support.reason ?? "unsupported environment"}`);
|
|
|
|
const controller = { bootId: `boot-${randomUUID()}`, pid: 5151, processStartedAt: new Date("2026-11-13T08:00:00.000Z") };
|
|
|
|
/**
|
|
* One server process renews the native execution leases of all its runs in
|
|
* one shared transaction per tick, on the private two-connection liveness pool.
|
|
*/
|
|
describeEmbeddedPostgres("batched native execution lease renewal", () => {
|
|
let db!: ReturnType<typeof createDb>;
|
|
let owner!: ReturnType<typeof createDb>;
|
|
let meter!: ReturnType<typeof meterDatabaseStatements>;
|
|
let connectionString!: string;
|
|
let stopDatabase: (() => Promise<void>) | undefined;
|
|
let companyId!: string;
|
|
let agentId!: string;
|
|
|
|
beforeAll(async () => {
|
|
const started = await startEmbeddedPostgresTestDatabase("paperclip-native-lease-batch-");
|
|
stopDatabase = started.cleanup;
|
|
connectionString = started.connectionString;
|
|
db = createDb(connectionString);
|
|
owner = createDb(connectionString);
|
|
meter = meterDatabaseStatements(livenessDb(owner), connectionString);
|
|
companyId = randomUUID();
|
|
agentId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Lease batch", issuePrefix: "LBT",
|
|
requireBoardApprovalForNewAgents: false });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Runner", role: "engineer", status: "active",
|
|
adapterType: "paperclip_runner", adapterConfig: { provider: "codex" }, runtimeConfig: {}, permissions: {} });
|
|
}, 120_000);
|
|
|
|
afterAll(async () => {
|
|
if (owner) await closeLivenessDb(owner, 1).catch(() => undefined);
|
|
await owner?.$client.end({ timeout: 1 });
|
|
await db?.$client.end({ timeout: 1 });
|
|
await stopDatabase?.();
|
|
});
|
|
|
|
type Seeded = { runId: string; issueId: string; leaseOwner: string };
|
|
|
|
async function seedRuns(count: number, leaseMs = 30_000): Promise<Seeded[]> {
|
|
const runs = Array.from({ length: count }, () => ({ runId: randomUUID(), issueId: randomUUID(),
|
|
leaseOwner: `runner:${randomUUID()}` }));
|
|
await db.insert(issues).values(runs.map(run => ({ id: run.issueId, companyId, identifier: `LBT-${run.runId.slice(0, 8)}`,
|
|
title: "batch", status: "in_progress", priority: "medium", workMode: "standard", assigneeAgentId: agentId })));
|
|
await db.insert(heartbeatRuns).values(runs.map(run => ({ id: run.runId, companyId, agentId, status: "running",
|
|
runtimeMode: "native", nativeIssueId: run.issueId, invocationSource: "assignment", triggerDetail: "system",
|
|
contextSnapshot: { issueId: run.issueId } })));
|
|
await db.insert(nativeRunFinalizations).values(runs.map(run => ({ runId: run.runId, companyId, issueId: run.issueId,
|
|
phase: "observed", attempt: 1, leaseOwner: run.leaseOwner,
|
|
leaseExpiresAt: sql`clock_timestamp() + (${leaseMs} * interval '1 millisecond')`,
|
|
controllerBootId: controller.bootId, controllerPid: controller.pid,
|
|
controllerProcessStartedAt: controller.processStartedAt, controllerGeneration: 1 })));
|
|
return runs;
|
|
}
|
|
|
|
function start(run: Seeded, overrides: { remote?: boolean; controllerGeneration?: number } = {}) {
|
|
const onLost = vi.fn(async () => undefined);
|
|
const guard = startNativeSessionExecutionLeaseRenewal({ db: owner, companyId, ...run, attempt: 1, controller,
|
|
remote: overrides.remote ?? true, controllerGeneration: overrides.controllerGeneration,
|
|
confirmedDeadline: performance.now() + 25_000, onLost });
|
|
return { run, guard, onLost };
|
|
}
|
|
|
|
async function expiries(runIds: string[]) {
|
|
const rows = await db.select({ runId: nativeRunFinalizations.runId, expiresAt: nativeRunFinalizations.leaseExpiresAt,
|
|
leaseOwner: nativeRunFinalizations.leaseOwner })
|
|
.from(nativeRunFinalizations).where(inArray(nativeRunFinalizations.runId, runIds));
|
|
return new Map(rows.map(row => [row.runId, row]));
|
|
}
|
|
|
|
async function stopAll(guards: NativeExecutionLeaseRenewal[]) {
|
|
await Promise.all(guards.map(guard => guard.stop().catch(() => undefined)));
|
|
}
|
|
|
|
async function renewCost(count: number) {
|
|
const runs = await seedRuns(count);
|
|
const started = runs.map(run => start(run));
|
|
try {
|
|
// Warm the pool so connection setup is not counted.
|
|
await renewNativeExecutionLeasesNow(owner);
|
|
const before = meter.read();
|
|
await renewNativeExecutionLeasesNow(owner);
|
|
const after = meter.read();
|
|
for (const { guard, onLost } of started) {
|
|
expect(onLost).not.toHaveBeenCalled();
|
|
expect(guard.remainingMs()).toBeGreaterThan(REMOTE_CONTROLLER_LEASE_MS - 5_000);
|
|
}
|
|
const saved = await expiries(runs.map(run => run.runId));
|
|
for (const run of runs) expect(saved.get(run.runId)!.expiresAt!.getTime()).toBeGreaterThan(Date.now() + 50_000);
|
|
return { transactions: after.transactions - before.transactions, statements: after.statements - before.statements };
|
|
} finally {
|
|
await stopAll(started.map(item => item.guard));
|
|
}
|
|
}
|
|
|
|
it("renews every lease of the process in one transaction whose size does not grow with the run count", async () => {
|
|
const few = await renewCost(3);
|
|
const many = await renewCost(60);
|
|
expect(few.transactions).toBe(1);
|
|
expect(many.transactions).toBe(1);
|
|
expect(many.statements).toBe(few.statements);
|
|
}, 60_000);
|
|
|
|
it("splits a very large tick into bounded transactions", async () => {
|
|
const cost = await renewCost(450);
|
|
expect(cost.transactions).toBe(3);
|
|
}, 120_000);
|
|
|
|
it("fails only the runs whose lease is no longer theirs and renews the rest", async () => {
|
|
const runs = await seedRuns(6);
|
|
const [healthy, other, takenOver, newAttempt, otherBoot, expired] = runs;
|
|
await db.update(nativeRunFinalizations).set({ leaseOwner: `native-finalizer:${randomUUID()}` })
|
|
.where(eq(nativeRunFinalizations.runId, takenOver!.runId));
|
|
await db.update(nativeRunFinalizations).set({ attempt: 2 }).where(eq(nativeRunFinalizations.runId, newAttempt!.runId));
|
|
await db.update(nativeRunFinalizations).set({ controllerBootId: `boot-${randomUUID()}` })
|
|
.where(eq(nativeRunFinalizations.runId, otherBoot!.runId));
|
|
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: sql`clock_timestamp() - interval '1 second'` })
|
|
.where(eq(nativeRunFinalizations.runId, expired!.runId));
|
|
const before = await expiries(runs.map(run => run.runId));
|
|
const started = runs.map(run => start(run));
|
|
try {
|
|
await renewNativeExecutionLeasesNow(owner);
|
|
// A newer attempt under the same owner is the owner's own executor
|
|
// moving on: not extended and not lost, as for a single renewal.
|
|
const [ok, ok2, takenOverRenewal, advanced, otherBootRenewal, expiredRenewal] = started;
|
|
const lost = [takenOverRenewal!, otherBootRenewal!, expiredRenewal!];
|
|
expect(advanced!.onLost).not.toHaveBeenCalled();
|
|
for (const item of [ok!, ok2!]) {
|
|
expect(item.onLost).not.toHaveBeenCalled();
|
|
await expect(item.guard.stop()).resolves.toBeUndefined();
|
|
}
|
|
for (const item of lost) {
|
|
await vi.waitFor(() => expect(item.onLost).toHaveBeenCalledTimes(1));
|
|
await expect(item.guard.stop()).rejects.toThrow("native_session_lease_lost");
|
|
}
|
|
const after = await expiries(runs.map(run => run.runId));
|
|
for (const run of [healthy!, other!]) {
|
|
expect(after.get(run.runId)!.expiresAt!.getTime()).toBeGreaterThan(before.get(run.runId)!.expiresAt!.getTime());
|
|
}
|
|
// Nothing the batch could not extend was written, and the expired owner stays expired.
|
|
for (const run of [takenOver!, newAttempt!, otherBoot!, expired!]) expect(after.get(run.runId)).toEqual(before.get(run.runId));
|
|
} finally {
|
|
await stopAll(started.map(item => item.guard));
|
|
}
|
|
}, 60_000);
|
|
|
|
it("never revives an owner that expired while another writer held its row, and does not wait for that row", async () => {
|
|
const [contended, healthy] = await seedRuns(2, 1_500);
|
|
// Another actor (for example restart recovery) holds the coordinator row
|
|
// from before the lease expires until after it expired, then commits.
|
|
let release!: () => void;
|
|
let locked!: () => void;
|
|
const held = new Promise<void>(resolve => { release = resolve; });
|
|
const acquired = new Promise<void>(resolve => { locked = resolve; });
|
|
const writer = db.transaction(async tx => {
|
|
await tx.execute(sql`select 1 from ${nativeRunFinalizations} where ${nativeRunFinalizations.runId} = ${contended!.runId} for update`);
|
|
locked();
|
|
await held;
|
|
});
|
|
await acquired;
|
|
const contendedRenewal = start(contended!);
|
|
const healthyRenewal = start(healthy!);
|
|
try {
|
|
const renewing = renewNativeExecutionLeasesNow(owner);
|
|
// The busy row does not hold up the rest of the batch.
|
|
await vi.waitFor(async () => expect((await expiries([healthy!.runId])).get(healthy!.runId)!.expiresAt!.getTime())
|
|
.toBeGreaterThan(Date.now() + 50_000), { timeout: 5_000 });
|
|
await new Promise(resolve => setTimeout(resolve, 2_000));
|
|
expect((await expiries([contended!.runId])).get(contended!.runId)!.expiresAt!.getTime()).toBeLessThan(Date.now());
|
|
release();
|
|
await writer;
|
|
await renewing;
|
|
await vi.waitFor(() => expect(contendedRenewal.onLost).toHaveBeenCalledTimes(1));
|
|
await expect(contendedRenewal.guard.stop()).rejects.toThrow("native_session_lease_lost");
|
|
expect((await expiries([contended!.runId])).get(contended!.runId)!.expiresAt!.getTime()).toBeLessThan(Date.now());
|
|
expect(healthyRenewal.onLost).not.toHaveBeenCalled();
|
|
} finally {
|
|
release();
|
|
await writer.catch(() => undefined);
|
|
await stopAll([contendedRenewal.guard, healthyRenewal.guard]);
|
|
}
|
|
}, 60_000);
|
|
|
|
it("never revives an owner whose row a takeover rewrote while the batch waited", async () => {
|
|
const [run] = await seedRuns(1, 60_000);
|
|
const renewal = start(run!);
|
|
// A takeover commits between the batch's statements on another connection;
|
|
// the batch locks the row and reads its committed owner, so it extends nothing.
|
|
await db.update(nativeRunFinalizations).set({ leaseOwner: `native-cleanup:${randomUUID()}`,
|
|
leaseExpiresAt: sql`clock_timestamp() + interval '5 minutes'` }).where(eq(nativeRunFinalizations.runId, run!.runId));
|
|
const before = await expiries([run!.runId]);
|
|
try {
|
|
await renewNativeExecutionLeasesNow(owner);
|
|
await vi.waitFor(() => expect(renewal.onLost).toHaveBeenCalledTimes(1));
|
|
expect(await expiries([run!.runId])).toEqual(before);
|
|
} finally {
|
|
await renewal.guard.stop().catch(() => undefined);
|
|
}
|
|
}, 60_000);
|
|
|
|
it("confines a binding that makes the shared statement fail to its own run", async () => {
|
|
const runs = await seedRuns(5);
|
|
const before = await expiries(runs.map(run => run.runId));
|
|
const healthy = runs.slice(1).map(run => start(run));
|
|
// An attempt outside the integer range: PostgreSQL rejects the whole
|
|
// shared UPDATE (22003) for every run in it.
|
|
const poison = { run: runs[0]!, onLost: vi.fn(async () => undefined), guard: startNativeSessionExecutionLeaseRenewal({
|
|
db: owner, companyId, ...runs[0]!, attempt: 2 ** 40, controller, remote: true,
|
|
confirmedDeadline: performance.now() + 25_000, onLost: async () => { await poison.onLost(); } }) };
|
|
try {
|
|
await renewNativeExecutionLeasesNow(owner);
|
|
const after = await expiries(runs.map(run => run.runId));
|
|
for (const { run, onLost, guard } of healthy) {
|
|
expect(onLost).not.toHaveBeenCalled();
|
|
expect(after.get(run.runId)!.expiresAt!.getTime()).toBeGreaterThan(before.get(run.runId)!.expiresAt!.getTime());
|
|
expect(guard.remainingMs()).toBeGreaterThan(REMOTE_CONTROLLER_LEASE_MS - 5_000);
|
|
}
|
|
expect(after.get(poison.run.runId)).toEqual(before.get(poison.run.runId));
|
|
// The healthy runs go back to one shared transaction; nothing is stuck busy.
|
|
const cost = meter.read();
|
|
await renewNativeExecutionLeasesNow(owner);
|
|
const healthyCost = meter.read();
|
|
expect(healthyCost.transactions - cost.transactions).toBe(1);
|
|
for (const { onLost } of healthy) expect(onLost).not.toHaveBeenCalled();
|
|
// Only the bad binding's own run is lost.
|
|
await vi.waitFor(() => expect(poison.onLost).toHaveBeenCalledTimes(1));
|
|
} finally {
|
|
await stopAll([...healthy.map(item => item.guard), poison.guard]);
|
|
}
|
|
}, 60_000);
|
|
|
|
it("handles an error thrown while the shared statement is built and leaves no lease busy", async () => {
|
|
const runs = await seedRuns(3);
|
|
const healthy = runs.slice(1).map(run => start(run));
|
|
const onLost = vi.fn(async () => undefined);
|
|
// An invalid date throws in toISOString() while the VALUES list is built.
|
|
const broken = startNativeSessionExecutionLeaseRenewal({ db: owner, companyId, ...runs[0]!, attempt: 1,
|
|
controller: { ...controller, processStartedAt: new Date(Number.NaN) }, remote: true,
|
|
confirmedDeadline: performance.now() + 25_000, onLost });
|
|
try {
|
|
await renewNativeExecutionLeasesNow(owner);
|
|
const after = await expiries(runs.map(run => run.runId));
|
|
for (const { run, onLost: healthyLost } of healthy) {
|
|
expect(healthyLost).not.toHaveBeenCalled();
|
|
expect(after.get(run.runId)!.expiresAt!.getTime()).toBeGreaterThan(Date.now() + 50_000);
|
|
}
|
|
// Every lease settled: a second call renews again instead of finding them busy.
|
|
const before = meter.read();
|
|
await renewNativeExecutionLeasesNow(owner);
|
|
expect(meter.read().transactions - before.transactions).toBe(1);
|
|
for (const { guard } of healthy) expect(guard.remainingMs()).toBeGreaterThan(REMOTE_CONTROLLER_LEASE_MS - 5_000);
|
|
await vi.waitFor(() => expect(onLost).toHaveBeenCalledTimes(1));
|
|
} finally {
|
|
await stopAll([...healthy.map(item => item.guard), broken]);
|
|
}
|
|
}, 60_000);
|
|
|
|
it("renews local and remote leases together, each for its own lease length", async () => {
|
|
const [remote, local] = await seedRuns(2);
|
|
const started = [start(remote!), start(local!, { remote: false, controllerGeneration: 1 })];
|
|
try {
|
|
const before = meter.read();
|
|
await renewNativeExecutionLeasesNow(owner);
|
|
expect(meter.read().transactions - before.transactions).toBe(1);
|
|
const saved = await expiries([remote!.runId, local!.runId]);
|
|
expect(saved.get(remote!.runId)!.expiresAt!.getTime()).toBeLessThan(Date.now() + 2 * 60_000);
|
|
expect(saved.get(local!.runId)!.expiresAt!.getTime()).toBeGreaterThan(Date.now() + 15 * 60_000);
|
|
expect(started[1]!.guard.remainingMs()).toBeGreaterThan(15 * 60_000);
|
|
} finally {
|
|
await stopAll(started.map(item => item.guard));
|
|
}
|
|
}, 60_000);
|
|
});
|