mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The scaling plan (`doc/plans/2026-11-13-thousand-concurrent-agents.md`) has a finish line that a real-agent ramp harness checks (#21695) > - Four finish-line criteria need numbers from inside each server process: main-thread CPU, run event lateness, agent API time before the handler, and missed lease renewals > - The server had no way to give them, so every harness run was at best "incomplete" > - This pull request adds one instance-admin endpoint that returns those numbers per process over a short sliding window, off by default, and connects the harness to it > - The benefit is that a ramp run can measure every server criterion it can see, with near-zero cost when on and no cost when off ## Linked Issues or Issue Description Refs #21695 **Subsystem affected** Server (instance routes, run event append, native lease renewal), database client (statement counter), and the scale-ramp harness in `scripts/scale-ramp/`. **Problem or motivation** The scale-ramp harness reads `PAPERCLIP_SCALE_METRICS_URL`, but the server has no endpoint for it. The criteria `main_thread_cpu`, `run_event_lateness_p99`, `agent_api_pre_handler_p99` and `lease_renewals_missed_zero` are therefore always "not measured", and no run can pass. **Proposed solution** Add `GET /api/instance/server-metrics`. It returns a JSON snapshot of one server process over the last 50 to 60 s. Only an instance admin can read it. The env flag `PAPERCLIP_SERVER_METRICS` (read once at start, default off) turns on collection and the route. Off, the path answers 404 and no collector starts. **Alternatives considered** - The postgres.js `debug` option for the statement counter: rejected, because it captures a stack (`new Error()`) for every query. - Process CPU from `process.cpuUsage()` as `mainThreadCpu`: rejected, because it includes GC and libuv threads. It is returned as `processCpu` instead, and `mainThreadCpu` reads the main thread from `/proc` (Linux only, null elsewhere). - Default on: the measured cost is negligible (numbers below), but the event-loop monitor wakes the process every 20 ms when idle, and this is a new instance-wide surface. It stays off; the runbook says how to turn it on for a ramp run. ## What Changed - `server/src/services/server-metrics.ts`: a per-process collector. It uses `performance.eventLoopUtilization()`, `monitorEventLoopDelay` (20 ms), `process.cpuUsage()`, the main thread's `/proc/self/task/<pid>/stat` on Linux, and six 10 s buckets of native `perf_hooks` histograms. One unref'd timer rotates the buckets. Nothing on the hot path awaits, allocates per call or does I/O. - `server/src/routes/instance-server-metrics.ts`: `GET /api/instance/server-metrics`, guarded by the existing `assertInstanceAdmin` (the check that `/instance/database-backups` and other instance routes use). It returns numbers, a random per-process `processId`, `uptimeSec`, `buildRevision` and `windowMs`. It returns no SQL, names, IDs or hostnames. - `server/src/app.ts`: when the flag is on, two middlewares time agent API requests from arrival to the API router, the route is mounted, and shutdown stops the collector. When the flag is off, nothing is mounted. - The snapshot also carries `dedicatedConnections` from `dedicatedDbConnectionStats()` (#21693): `limit`, `inUse`, `waiting` at request time and `peakInUse`, `queued`, `overLimit`, `nestedOverLimit` since start. The harness ignores it today; it shows dedicated-connection queueing in a restart burst. - `packages/db/src/statement-count.ts` and `lazy-transaction-queries.ts`: a Drizzle logger that increments one counter per statement replaces Drizzle's no-op logger. No edit to `client.ts`. - `paperclip-control-plane-port.ts`: one call records runner `emittedAt` to committed row for each committed (not duplicate) runner event. - `native-execution-lease.ts`: one call counts a lease whose renewal deadline passed. - Harness: missed lease renewals are summed per `processId` (several processes, or a restart, no longer cancel out). A metrics URL on the target origin gets the board token in the `Authorization` header; `PAPERCLIP_SCALE_METRICS_TOKEN` still overrides it, and another origin gets no token. - Contract: `scripts/scale-ramp/server-metrics.fixture.json` is the serializer's own output. A server test asserts the serializer produces exactly this file, and a harness test parses it and checks every field. - `rules/company-scope-exceptions.mdc`: adds the snapshot as an instance-wide exception with its reason. The runbook in the plan doc documents the path, the flag, the credential and each field. | Field | Provided | Source | | --- | --- | --- | | `mainThreadCpu` | yes (Linux; null elsewhere) | `/proc/self/task/<pid>/stat` utime+stime | | `eventLoopUtilization` | yes | `performance.eventLoopUtilization()` | | `eventLoopDelayP99Ms` | yes | `monitorEventLoopDelay`, highest 10 s p99 in the window | | `statementsPerSecond` | yes | Drizzle logger counter (not BEGIN/COMMIT or raw driver calls) | | `poolWaitP99Ms` | absent | postgres.js has no cheap hook for queue time | | `runEventLatenessP99Ms` | yes (null with no events) | runner event append | | `agentApiPreHandlerP99Ms` | yes (null with no agent requests) | arrival and router-entry middlewares | | `leaseRenewalsMissed` | yes, counter since process start | native lease deadline timer | ## Verification - `cd server && npx vitest run src/__tests__/instance-server-metrics-routes.test.ts src/__tests__/server-metrics.test.ts`: through `createApp` on the embedded test database. 200 with the exact field set for an instance admin; 403 for a company owner who is not an instance admin, for an agent key, and unauthenticated; 404 and no collector when the flag is off; values move after agent requests, statements, a late run event and a missed renewal; the body has no IDs, keys or SQL. - `cd server && npx vitest run src/services/native-runtime/native-execution-lease.test.ts src/services/native-runtime/paperclip-control-plane-port.test.ts src/__tests__/api-server-timing.test.ts`: the lease and run-event hooks (red/green checked for the lease hook by removing the call). - `cd packages/db && npx vitest run src/lazy-transaction-queries.test.ts src/client-options.test.ts`: the counter counts statements inside transactions and savepoints. - `node --test scripts/scale-ramp/scale-ramp.test.ts` and `tsc -p scripts/scale-ramp/tsconfig.json`: 38 pass, including the fixture contract and the board-token header. - `node server/scripts/typecheck-layers.mjs` and `tsc --noEmit` in `packages/db`: pass. Overhead, measured on a shared, loaded laptop (load average 8 to 20), median with min and max: | Measure | Off | On | | --- | --- | --- | | Logger call per statement (5M calls x 7) | 0.28 ns (0.24 to 0.97) | 3.36 ns (3.32 to 6.16) | | CPU per `select 1` statement, embedded Postgres (3,000 x 10) | 28.7 us (20.3 to 42.4) | 32.3 us (20.7 to 45.5) | | CPU per HTTP request, client and server in one process (2,000 x 6) | 67.0 us (40.4 to 107.4) | 64.2 us (40.9 to 83.3) | | Idle process CPU, 5 s x 3 | 0.01% of a core (0.00 to 2.21) | 0.08% (0.05 to 0.09) | | Record one run event (Date.parse + histogram) | - | 148 ns (135 to 188) | | One snapshot | - | 55 us | The statement and request differences are inside the run-to-run spread. The direct cost is about 3 ns per statement, against about 30 us of CPU per statement. ## Risks - Low risk. Off by default; off, only the statement counter runs, and it costs one increment more than Drizzle's no-op logger. - `mainThreadCpu` assumes Linux USER_HZ of 100 and is null on other platforms. The harness then uses `eventLoopUtilization`. - Run event lateness compares the runner's clock with the server's clock, so clock skew adds to it. Negative values count as zero. - With several processes, each request reaches one process. The harness keeps the highest gauge and sums counter growth per process, but a process it never samples is not seen. - The statement counter does not count BEGIN/COMMIT or raw postgres.js calls. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
269 lines
14 KiB
TypeScript
269 lines
14 KiB
TypeScript
import { afterEach, describe, expect, it, vi } from "vitest";
|
|
import type { SQL } from "drizzle-orm";
|
|
import { PgDialect } from "drizzle-orm/pg-core";
|
|
import type { Db } from "@paperclipai/db";
|
|
import { startNativeSessionExecutionLeaseRenewal } from "./native-execution-lease.js";
|
|
import { startServerMetrics, stopServerMetrics } from "../server-metrics.js";
|
|
|
|
const dialect = new PgDialect();
|
|
|
|
function fixture() {
|
|
// `returning` decides each renewal: the batched UPDATE and the single-run
|
|
// UPDATE both resolve through it.
|
|
const returning = vi.fn(async () => [{ runId: "run" }]);
|
|
const coordinator = vi.fn(async (): Promise<Array<{ attempt: number; leaseOwner: string }>> => []);
|
|
const renewed = vi.fn((_values: unknown) => undefined);
|
|
const update = vi.fn(() => ({ set: (values: unknown) => { renewed(values); return { where: (condition: unknown) => { renewed(condition); return { returning }; } }; } }));
|
|
const select = () => ({ from: () => ({ where: () => ({ limit: coordinator }) }) });
|
|
// The batch locks the coordinator row first; `lock` is what that lock reads.
|
|
const lock = { present: true, live: true };
|
|
const execute = vi.fn(async (query: SQL) => {
|
|
const text = dialect.sqlToQuery(query).sql; if (text.includes("skip locked")) return lock.present ? [{ run_id: "run", live: lock.live }] : [];
|
|
if (text.includes("set lease_expires_at")) return (await returning()).length ? [{ run_id: "run" }] : [];
|
|
return [];
|
|
});
|
|
const transaction = vi.fn(async (run: (tx: unknown) => Promise<unknown>) => run({ update, select, execute }));
|
|
const db = { update, transaction };
|
|
return { update, returning, transaction, coordinator, lock, input: { db: db as unknown as Db, runId: "run", companyId: "company", issueId: "issue",
|
|
leaseOwner: "owner", attempt: 1, controller: { bootId: "boot", pid: 1, processStartedAt: new Date() }, remote: true,
|
|
onLost: vi.fn(async () => undefined) } };
|
|
}
|
|
afterEach(() => vi.useRealTimers());
|
|
describe("native execution renewal ownership", () => {
|
|
it("reports only the remaining confirmed ownership budget", async () => {
|
|
vi.useFakeTimers();
|
|
const { input, returning } = fixture();
|
|
let complete!: (value: { runId: string }[]) => void;
|
|
returning.mockImplementationOnce(() => new Promise(resolve => { complete = resolve; }));
|
|
const guard = startNativeSessionExecutionLeaseRenewal({ ...input, confirmedDeadline: performance.now() + 45_000 });
|
|
expect(guard.remainingMs()).toBe(45_000);
|
|
await vi.advanceTimersByTimeAsync(40_000);
|
|
expect(guard.remainingMs()).toBe(5_000);
|
|
complete([{ runId: "run" }]);
|
|
await vi.advanceTimersByTimeAsync(0);
|
|
expect(guard.remainingMs()).toBe(40_000);
|
|
await guard.stop();
|
|
});
|
|
it("keeps a successfully renewed lease through repeated slow connection acquisition", async () => {
|
|
vi.useFakeTimers();
|
|
const { input, returning, transaction } = fixture();
|
|
const connected = transaction.getMockImplementation()!;
|
|
transaction.mockImplementation(async run => {
|
|
await new Promise(resolve => setTimeout(resolve, 30_000));
|
|
return connected(run);
|
|
});
|
|
const guard = startNativeSessionExecutionLeaseRenewal(input);
|
|
try {
|
|
// Renewals start at 20s/60s and extend the database lease at 50s/90s.
|
|
// Connection waits must not shorten the successfully extended ownership.
|
|
await vi.advanceTimersByTimeAsync(95_000);
|
|
expect(returning).toHaveBeenCalledTimes(2);
|
|
expect(input.onLost).not.toHaveBeenCalled();
|
|
} finally { await guard.stop().catch(() => undefined); }
|
|
expect(vi.getTimerCount()).toBe(0);
|
|
});
|
|
it("shares one timer across preparation and execution and stops idle SQL after both release", async () => {
|
|
vi.useFakeTimers();
|
|
const { input, returning } = fixture();
|
|
const preparation = startNativeSessionExecutionLeaseRenewal(input);
|
|
const execution = startNativeSessionExecutionLeaseRenewal(input);
|
|
expect(returning).not.toHaveBeenCalled();
|
|
await vi.advanceTimersByTimeAsync(40_000);
|
|
expect(returning).toHaveBeenCalledTimes(2);
|
|
await preparation.stop();
|
|
await preparation.stop();
|
|
await vi.advanceTimersByTimeAsync(20_000);
|
|
expect(returning).toHaveBeenCalledTimes(3);
|
|
await execution.stop();
|
|
await vi.advanceTimersByTimeAsync(120_000);
|
|
expect(returning).toHaveBeenCalledTimes(3);
|
|
expect(vi.getTimerCount()).toBe(0);
|
|
});
|
|
it("retries cancelled SQL within the original deadline without losing healthy ownership", async () => {
|
|
vi.useFakeTimers();
|
|
const { input, returning } = fixture();
|
|
returning.mockRejectedValueOnce(Object.assign(new Error("statement timeout"), { code: "57014" }));
|
|
const guard = startNativeSessionExecutionLeaseRenewal(input);
|
|
await vi.advanceTimersByTimeAsync(60_000);
|
|
expect(returning).toHaveBeenCalledTimes(3);
|
|
expect(input.onLost).not.toHaveBeenCalled();
|
|
await guard.stop();
|
|
expect(vi.getTimerCount()).toBe(0);
|
|
});
|
|
it("renews alone at once after a shared transaction fails for a reason other than a cancelled statement", async () => {
|
|
vi.useFakeTimers();
|
|
const { input, returning, update } = fixture();
|
|
returning.mockRejectedValueOnce(Object.assign(new Error("integer out of range"), { code: "22003" }));
|
|
const guard = startNativeSessionExecutionLeaseRenewal({ ...input, confirmedDeadline: performance.now() + 60_000 });
|
|
await vi.advanceTimersByTimeAsync(20_000);
|
|
// The batch UPDATE failed, then the single path renewed this run alone.
|
|
expect(returning).toHaveBeenCalledTimes(2);
|
|
expect(update).toHaveBeenCalledTimes(1);
|
|
expect(guard.remainingMs()).toBe(60_000);
|
|
expect(input.onLost).not.toHaveBeenCalled();
|
|
await guard.stop();
|
|
});
|
|
it("does not retry every second when the shared and the single renewal both fail (database outage)", async () => {
|
|
vi.useFakeTimers();
|
|
const { input, returning, coordinator } = fixture();
|
|
const outage = Object.assign(new Error("Connection terminated unexpectedly"), { code: "08006" });
|
|
returning.mockRejectedValueOnce(outage).mockRejectedValueOnce(outage);
|
|
coordinator.mockRejectedValueOnce(outage);
|
|
const guard = startNativeSessionExecutionLeaseRenewal({ ...input, confirmedDeadline: performance.now() + 60_000 });
|
|
await vi.advanceTimersByTimeAsync(20_000);
|
|
expect(returning).toHaveBeenCalledTimes(2);
|
|
await vi.advanceTimersByTimeAsync(19_000);
|
|
expect(returning).toHaveBeenCalledTimes(2);
|
|
// The next interval goes back to the shared batch.
|
|
await vi.advanceTimersByTimeAsync(1_000);
|
|
expect(returning).toHaveBeenCalledTimes(3);
|
|
expect(guard.remainingMs()).toBe(60_000);
|
|
await guard.stop();
|
|
});
|
|
it("detaches at the deadline while a database response is missing and cannot regain authority from a late response", async () => {
|
|
vi.useFakeTimers();
|
|
const { input, returning } = fixture();
|
|
let complete!: (value: { runId: string }[]) => void;
|
|
returning.mockImplementationOnce(() => new Promise(resolve => { complete = resolve; }));
|
|
const guard = startNativeSessionExecutionLeaseRenewal(input);
|
|
await vi.advanceTimersByTimeAsync(60_000);
|
|
expect(returning).toHaveBeenCalledTimes(1);
|
|
expect(input.onLost).toHaveBeenCalledTimes(1);
|
|
complete([{ runId: "run" }]);
|
|
await expect(guard.stop()).rejects.toThrow("native_session_lease_renewal_deadline");
|
|
await vi.advanceTimersByTimeAsync(120_000);
|
|
expect(returning).toHaveBeenCalledTimes(1);
|
|
expect(vi.getTimerCount()).toBe(0);
|
|
});
|
|
it("counts a renewal deadline that passes as a missed lease renewal for the server metrics", async () => {
|
|
const metrics = startServerMetrics();
|
|
try {
|
|
vi.useFakeTimers();
|
|
const { input, returning } = fixture();
|
|
let complete!: (value: { runId: string }[]) => void;
|
|
returning.mockImplementationOnce(() => new Promise(resolve => { complete = resolve; }));
|
|
const guard = startNativeSessionExecutionLeaseRenewal(input);
|
|
await vi.advanceTimersByTimeAsync(30_000);
|
|
expect(metrics.snapshot().leaseRenewalsMissed).toBe(0);
|
|
await vi.advanceTimersByTimeAsync(30_000);
|
|
expect(metrics.snapshot().leaseRenewalsMissed).toBe(1);
|
|
complete([{ runId: "run" }]);
|
|
await expect(guard.stop()).rejects.toThrow("native_session_lease_renewal_deadline");
|
|
} finally {
|
|
stopServerMetrics(metrics);
|
|
}
|
|
});
|
|
it("hands preparation renewal to the same owner's executor at the next attempt", async () => {
|
|
vi.useFakeTimers();
|
|
const { input, returning } = fixture();
|
|
const preparation = startNativeSessionExecutionLeaseRenewal(input);
|
|
expect(() => startNativeSessionExecutionLeaseRenewal({ ...input, attempt: 0 })).toThrow("native_session_renewal_binding_changed");
|
|
const executorLost = vi.fn(async () => undefined);
|
|
const execution = startNativeSessionExecutionLeaseRenewal({ ...input, attempt: 2, onLost: executorLost });
|
|
await preparation.stop();
|
|
await vi.advanceTimersByTimeAsync(20_000);
|
|
expect(returning).toHaveBeenCalledTimes(1);
|
|
returning.mockResolvedValueOnce([]);
|
|
await vi.advanceTimersByTimeAsync(20_000);
|
|
expect(executorLost).toHaveBeenCalledTimes(1);
|
|
expect(input.onLost).not.toHaveBeenCalled();
|
|
await expect(execution.stop()).rejects.toThrow("native_session_lease_lost");
|
|
});
|
|
it("gives the executor that takes over at the next attempt its own later deadline", async () => {
|
|
vi.useFakeTimers();
|
|
const { input, returning } = fixture();
|
|
let complete!: (value: { runId: string }[]) => void;
|
|
const preparation = startNativeSessionExecutionLeaseRenewal({ ...input, confirmedDeadline: performance.now() + 25_000 });
|
|
returning.mockImplementationOnce(() => new Promise(resolve => { complete = resolve; }));
|
|
await vi.advanceTimersByTimeAsync(20_000);
|
|
const executorLost = vi.fn(async () => undefined);
|
|
const execution = startNativeSessionExecutionLeaseRenewal({ ...input, attempt: 2, onLost: executorLost,
|
|
confirmedDeadline: performance.now() + 60_000 });
|
|
expect(execution.remainingMs()).toBe(60_000);
|
|
const released = preparation.stop().catch(() => undefined);
|
|
await vi.advanceTimersByTimeAsync(30_000);
|
|
expect(executorLost).not.toHaveBeenCalled();
|
|
complete([{ runId: "run" }]);
|
|
await vi.advanceTimersByTimeAsync(0);
|
|
expect(execution.remainingMs()).toBeGreaterThanOrEqual(30_000);
|
|
await released;
|
|
await execution.stop();
|
|
});
|
|
it("reacquires a lapsed local lease with the generation of the attempt that took over", async () => {
|
|
vi.useFakeTimers();
|
|
const { input, update, returning, lock } = fixture();
|
|
// The local lease lapsed (host sleep): only the single-run path may take it back.
|
|
lock.live = false;
|
|
const conditions: unknown[] = [];
|
|
update.mockImplementation(() => ({ set: () => ({ where: (condition: unknown) => { conditions.push(condition); return { returning }; } }) }));
|
|
const local = { ...input, remote: false, controllerGeneration: 3 };
|
|
const preparation = startNativeSessionExecutionLeaseRenewal(local);
|
|
const execution = startNativeSessionExecutionLeaseRenewal({ ...local, attempt: 2, controllerGeneration: 4 });
|
|
await preparation.stop();
|
|
await vi.advanceTimersByTimeAsync(5 * 60_000);
|
|
const values = (root: unknown) => {
|
|
const found = new Set<unknown>(), seen = new WeakSet<object>();
|
|
const walk = (node: unknown) => {
|
|
if (node === null || typeof node !== "object") { found.add(node); return; }
|
|
if (seen.has(node)) return; seen.add(node);
|
|
for (const child of Object.values(node)) walk(child);
|
|
};
|
|
walk(root);
|
|
return found;
|
|
};
|
|
expect(conditions.length).toBeGreaterThan(0);
|
|
expect(values(conditions.at(-1)).has(4)).toBe(true);
|
|
expect(values(conditions.at(-1)).has(3)).toBe(false);
|
|
await execution.stop();
|
|
});
|
|
it("does not count a renewal that failed because the owner's executor advanced the attempt", async () => {
|
|
vi.useFakeTimers();
|
|
const { input, returning } = fixture();
|
|
let complete!: (value: { runId: string }[]) => void;
|
|
returning.mockImplementationOnce(() => new Promise(resolve => { complete = resolve; }));
|
|
const preparation = startNativeSessionExecutionLeaseRenewal(input);
|
|
await vi.advanceTimersByTimeAsync(20_000);
|
|
expect(returning).toHaveBeenCalledTimes(1);
|
|
const execution = startNativeSessionExecutionLeaseRenewal({ ...input, attempt: 2 });
|
|
complete([]);
|
|
await vi.advanceTimersByTimeAsync(20_000);
|
|
expect(returning).toHaveBeenCalledTimes(2);
|
|
expect(input.onLost).not.toHaveBeenCalled();
|
|
await preparation.stop();
|
|
await execution.stop();
|
|
expect(vi.getTimerCount()).toBe(0);
|
|
});
|
|
it("keeps renewing when the coordinator shows its own owner already claimed the next attempt", async () => {
|
|
vi.useFakeTimers();
|
|
const { input, returning, coordinator } = fixture();
|
|
coordinator.mockResolvedValue([{ attempt: 2, leaseOwner: "owner" }]);
|
|
returning.mockResolvedValueOnce([]);
|
|
const preparation = startNativeSessionExecutionLeaseRenewal(input);
|
|
await vi.advanceTimersByTimeAsync(20_000);
|
|
expect(coordinator).toHaveBeenCalledTimes(1);
|
|
expect(input.onLost).not.toHaveBeenCalled();
|
|
const execution = startNativeSessionExecutionLeaseRenewal({ ...input, attempt: 2 });
|
|
await preparation.stop();
|
|
await vi.advanceTimersByTimeAsync(20_000);
|
|
expect(returning).toHaveBeenCalledTimes(2);
|
|
expect(input.onLost).not.toHaveBeenCalled();
|
|
await execution.stop();
|
|
});
|
|
it("does not queue SQL behind a slow renewal and fences a lost owner", async () => {
|
|
vi.useFakeTimers();
|
|
const { input, returning } = fixture();
|
|
let complete!: (value: { runId: string }[]) => void;
|
|
returning.mockImplementationOnce(() => new Promise(resolve => { complete = resolve; }));
|
|
const guard = startNativeSessionExecutionLeaseRenewal(input);
|
|
await vi.advanceTimersByTimeAsync(50_000);
|
|
expect(returning).toHaveBeenCalledTimes(1);
|
|
complete([]);
|
|
await vi.advanceTimersByTimeAsync(100_000);
|
|
expect(returning).toHaveBeenCalledTimes(1);
|
|
expect(input.onLost).toHaveBeenCalledTimes(1);
|
|
expect(() => startNativeSessionExecutionLeaseRenewal(input)).toThrow("native_session_lease_lost");
|
|
await expect(guard.stop()).rejects.toThrow("native_session_lease_lost");
|
|
expect(vi.getTimerCount()).toBe(0);
|
|
});
|
|
});
|