mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
test(server): advance the maintenance fake clock only after database connections finish startup (#22523)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The maintenance server tests boot the real server in a child process with a fake clock. The test moves the clock forward (up to 30 s per step) to reach deadlines fast. > - The fake clock also replaces the timers of the PostgreSQL driver. The driver starts a 30 s connect timer when it opens a connection, and stops it at the first ReadyForQuery. > - If the test moves the clock 30 s while a new database connection is still in startup, the connect timer fires at once. The query fails with `CONNECT_TIMEOUT`, the queue goes into retry, and the server does not settle. > - This race stopped the Release for856f14e507in `verify / Serialized tests (2/5)`. > - This pull request makes the child wait until every database connection has finished startup before it moves the clock. > - The benefit is that clock steps never fire a real-network timeout, so these tests no longer depend on runner speed. ## Linked Issues or Issue Description **What happened?** The Cloud readiness run 44954319430 (Release for856f14e507) failed in `verify / Serialized tests (2/5)`. The failed test is `maintenance-server-routes.test.ts > assembled maintenance server production wiring > reacts to the first ordinary status-card source write and accepts authorized controlled summarizer output`. The error is "Maintenance did not settle". The child log shows `write CONNECT_TIMEOUT 127.0.0.1:<proxy>` from `connectTimedOut` in postgres.js, called from `MockTimers.tick` at `server-child.ts:86`. That line is the `advance` command. The same log shows a false 28.6 s "event loop stall" from the same 30 s clock step. **Expected behavior** A fake clock step moves only the server's own deadlines. It must not fire the driver's connect timeout for a connection that is still in startup on the real network. **Steps to reproduce** 1. Run `pnpm --filter @paperclipai/server exec vitest run src/__tests__/maintenance-clock-database-startup.test.ts`. 2. The first test starts a PostgreSQL connection against a server that holds its startup reply, moves fake timers by 30 s, and then lets the reply through. The query fails with `CONNECT_TIMEOUT`. This is the CI failure in a deterministic form. 3. The second test waits with the new startup tracker before the same 30 s step. The query succeeds. **Paperclip version or commit** master at856f14e507. The race exists since the fake-clock `advance` command came in; the parent-side guard (`proxy.starting`) did not cover a connection that opens after the guard check. Classification: timing flake in the test harness (fake clock against a real-network driver timer). It is not a product regression. The test passed in every earlier Cloud readiness run on master. ## What Changed - New helper `server/src/__tests__/helpers/maintenance/database-startup.ts`. It uses the `net.client.socket` diagnostics channel to track each client socket from `connect()` until the first ReadyForQuery from the database (or until it closes or is not a database socket). It reads only backend frames with the existing protocol observer. - `server-child.ts` waits for that tracker before each `advance`, and then moves the clock in the same synchronous step. The driver starts its connect timer in the same synchronous call that publishes `net.client.socket`, so no connection can start its timer between the check and the step. - New test `server/src/__tests__/maintenance-clock-database-startup.test.ts` with the red case (unguarded step gives `CONNECT_TIMEOUT`) and the green case (guarded step succeeds). - No product code changes. No timeout changes. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/maintenance-clock-database-startup.test.ts`: 2 of 2 pass. - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/maintenance-server-routes.test.ts -t "first ordinary status-card source write"`: passes locally. - Full `maintenance-server-routes.test.ts`, `maintenance-postgres-proxy.test.ts` and `maintenance-harness-resources.test.ts`: pass locally. - `node scripts/typecheck-test-projects.mjs server/src/__tests__/helpers/maintenance`: passes. ## Risks - Low risk. Test-harness change only. If a database connection never finishes startup, `advance` now fails after 30 s with a clear message instead of a later "did not settle". ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, GitHub CLI) for investigation, the fix and local verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
782bd8d8eb
commit
7a4a5bba8a
3 files changed
+178
-2
No files matched your search
@@ -0,0 +1,53 @@
|
|||||||
|
import { channel } from "node:diagnostics_channel";
|
||||||
|
import type { Socket } from "node:net";
|
||||||
|
import { createPostgresProtocolObserver } from "./postgres-proxy.js";
|
||||||
|
|
||||||
|
const realTimeout = globalThis.setTimeout;
|
||||||
|
|
||||||
|
export function databasePort(databaseUrl: string) {
|
||||||
|
return Number(new URL(databaseUrl).port) || 5432;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function trackDatabaseConnectionStartup(port: number) {
|
||||||
|
const starting = new Set<Socket>();
|
||||||
|
let errors: Error[] = [];
|
||||||
|
const subscriber = (message: unknown) => {
|
||||||
|
const socket = (message as { socket: Socket }).socket;
|
||||||
|
starting.add(socket);
|
||||||
|
socket.once("close", () => starting.delete(socket));
|
||||||
|
socket.once("connect", () => {
|
||||||
|
if (socket.remotePort !== port) { starting.delete(socket); return; }
|
||||||
|
const observer = createPostgresProtocolObserver(() => {});
|
||||||
|
const onData = (chunk: Buffer) => {
|
||||||
|
try { observer.backend(chunk); }
|
||||||
|
catch (error) { errors.push(error as Error); starting.delete(socket); socket.off("data", onData); return; }
|
||||||
|
if (observer.ready) { starting.delete(socket); socket.off("data", onData); }
|
||||||
|
};
|
||||||
|
socket.on("data", onData);
|
||||||
|
});
|
||||||
|
};
|
||||||
|
channel("net.client.socket").subscribe(subscriber);
|
||||||
|
function failIfObservationFailed() {
|
||||||
|
if (!errors.length) return;
|
||||||
|
const failed = errors;
|
||||||
|
errors = [];
|
||||||
|
throw new AggregateError(failed, "Database connection startup could not be observed");
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
get starting() { return starting.size; },
|
||||||
|
async waitUntilStarted(timeoutMs = 30_000) {
|
||||||
|
const deadline = performance.now() + timeoutMs;
|
||||||
|
while (starting.size > 0) {
|
||||||
|
failIfObservationFailed();
|
||||||
|
if (performance.now() >= deadline) throw new Error(`Database connection startup did not finish within ${timeoutMs}ms`);
|
||||||
|
await new Promise(resolve => realTimeout(resolve, 5));
|
||||||
|
}
|
||||||
|
failIfObservationFailed();
|
||||||
|
},
|
||||||
|
async advanceAfterStartup(tick: (ms: number) => void, ms: number, timeoutMs = 30_000) {
|
||||||
|
do await this.waitUntilStarted(timeoutMs); while (starting.size > 0);
|
||||||
|
tick(ms);
|
||||||
|
},
|
||||||
|
dispose() { channel("net.client.socket").unsubscribe(subscriber); },
|
||||||
|
};
|
||||||
|
}
|
||||||
@@ -2,6 +2,7 @@ import { channel } from "node:diagnostics_channel";
|
|||||||
import { mock } from "node:test";
|
import { mock } from "node:test";
|
||||||
import type { Db } from "@paperclipai/db";
|
import type { Db } from "@paperclipai/db";
|
||||||
import type { Socket } from "node:net";
|
import type { Socket } from "node:net";
|
||||||
|
import { databasePort, trackDatabaseConnectionStartup } from "./database-startup.js";
|
||||||
import { observeMaintenanceIncomingHttp } from "./incoming-http-observation.js";
|
import { observeMaintenanceIncomingHttp } from "./incoming-http-observation.js";
|
||||||
|
|
||||||
const realTimeout = globalThis.setTimeout, realInterval = globalThis.setInterval, clearRealInterval = globalThis.clearInterval, realNow = Date.now;
|
const realTimeout = globalThis.setTimeout, realInterval = globalThis.setInterval, clearRealInterval = globalThis.clearInterval, realNow = Date.now;
|
||||||
@@ -26,11 +27,12 @@ if (process.env.PAPERCLIP_MAINTENANCE_GITHUB_ORIGIN) {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const databaseStartup = trackDatabaseConnectionStartup(databasePort(process.env.DATABASE_URL!));
|
||||||
let outgoingHttpRequests = 0;
|
let outgoingHttpRequests = 0;
|
||||||
const sockets: Array<{ socket: Socket; database: boolean }> = [];
|
const sockets: Array<{ socket: Socket; database: boolean }> = [];
|
||||||
channel("net.client.socket").subscribe(message => {
|
channel("net.client.socket").subscribe(message => {
|
||||||
const socket = (message as { socket: Socket }).socket;
|
const socket = (message as { socket: Socket }).socket;
|
||||||
socket.once("connect", () => sockets.push({ socket, database: socket.remotePort === Number(new URL(process.env.DATABASE_URL!).port) }));
|
socket.once("connect", () => sockets.push({ socket, database: socket.remotePort === databasePort(process.env.DATABASE_URL!) }));
|
||||||
});
|
});
|
||||||
function network() {
|
function network() {
|
||||||
return { nonDatabaseConnections: sockets.filter(entry => !entry.database).length,
|
return { nonDatabaseConnections: sockets.filter(entry => !entry.database).length,
|
||||||
@@ -83,7 +85,7 @@ try {
|
|||||||
if (message.command === "realtime" && fakeClock) runClock();
|
if (message.command === "realtime" && fakeClock) runClock();
|
||||||
if (message.command === "advance") {
|
if (message.command === "advance") {
|
||||||
if (!fakeClock || typeof message.ms !== "number" || !Number.isFinite(message.ms) || message.ms < 0 || message.ms > 30_000) throw new Error("Invalid maintenance clock advance");
|
if (!fakeClock || typeof message.ms !== "number" || !Number.isFinite(message.ms) || message.ms < 0 || message.ms > 30_000) throw new Error("Invalid maintenance clock advance");
|
||||||
mock.timers.tick(message.ms);
|
await databaseStartup.advanceAfterStartup(ms => mock.timers.tick(ms), message.ms);
|
||||||
}
|
}
|
||||||
if (message.command === "drain") startTaskDrain({ purpose: "idle" });
|
if (message.command === "drain") startTaskDrain({ purpose: "idle" });
|
||||||
if (message.command === "resume") stopTaskDrain();
|
if (message.command === "resume") stopTaskDrain();
|
||||||
|
|||||||
@@ -0,0 +1,121 @@
|
|||||||
|
import { createRequire } from "node:module";
|
||||||
|
import { once } from "node:events";
|
||||||
|
import { createConnection, createServer, type Server, type Socket } from "node:net";
|
||||||
|
import { pathToFileURL } from "node:url";
|
||||||
|
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||||
|
import { databasePort, trackDatabaseConnectionStartup } from "./helpers/maintenance/database-startup.js";
|
||||||
|
|
||||||
|
const databaseRequire = createRequire(createRequire(import.meta.url).resolve("@paperclipai/db"));
|
||||||
|
type PostgresClient = {
|
||||||
|
(strings: TemplateStringsArray): { simple(): { execute(): Promise<unknown[]> } };
|
||||||
|
end(options: { timeout: number }): Promise<void>;
|
||||||
|
};
|
||||||
|
const postgres = (await import(pathToFileURL(databaseRequire.resolve("postgres")).href)).default as (options: Record<string, unknown>) => PostgresClient;
|
||||||
|
const realTimeout = globalThis.setTimeout;
|
||||||
|
const realImmediate = globalThis.setImmediate;
|
||||||
|
|
||||||
|
function frame(type: string, body: Buffer) {
|
||||||
|
const header = Buffer.alloc(5);
|
||||||
|
header[0] = type.charCodeAt(0); header.writeInt32BE(body.length + 4, 1);
|
||||||
|
return Buffer.concat([header, body]);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function startHeldPostgres(startupReply = Buffer.concat([frame("R", Buffer.alloc(4)), frame("Z", Buffer.from("I"))])) {
|
||||||
|
const held: Socket[] = [], sockets = new Set<Socket>();
|
||||||
|
const server: Server = createServer(socket => {
|
||||||
|
sockets.add(socket);
|
||||||
|
socket.once("close", () => sockets.delete(socket));
|
||||||
|
socket.once("data", () => {
|
||||||
|
held.push(socket);
|
||||||
|
socket.on("data", () => socket.write(Buffer.concat([frame("C", Buffer.from("SELECT 0\0")), frame("Z", Buffer.from("I"))])));
|
||||||
|
});
|
||||||
|
socket.on("error", () => {});
|
||||||
|
});
|
||||||
|
await new Promise<void>((resolve, reject) => { server.once("error", reject); server.listen(0, "127.0.0.1", resolve); });
|
||||||
|
const port = (server.address() as { port: number }).port;
|
||||||
|
return {
|
||||||
|
port,
|
||||||
|
held,
|
||||||
|
release() {
|
||||||
|
for (const socket of held.splice(0)) {
|
||||||
|
if (!socket.destroyed) socket.write(startupReply);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
async stop() {
|
||||||
|
for (const socket of sockets) socket.destroy();
|
||||||
|
await new Promise<void>((resolve, reject) => server.close(error => error ? reject(error) : resolve()));
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async function startConnection(port: number, tracker: ReturnType<typeof trackDatabaseConnectionStartup>, upstream: { held: Socket[] }, cleanup: Array<() => unknown>) {
|
||||||
|
const sql = postgres({ host: "127.0.0.1", port, username: "maintenance", database: "maintenance", password: "unused",
|
||||||
|
ssl: false, fetch_types: false, connect_timeout: 30, max: 1, onnotice: () => {} });
|
||||||
|
const query = sql`select 1`.simple().execute();
|
||||||
|
query.catch(() => {});
|
||||||
|
cleanup.push(() => sql.end({ timeout: 0 }));
|
||||||
|
const deadline = performance.now() + 5_000;
|
||||||
|
while (tracker.starting === 0 || upstream.held.length === 0) {
|
||||||
|
if (performance.now() > deadline) throw new Error("PostgreSQL client did not start its connection");
|
||||||
|
vi.advanceTimersByTime(0);
|
||||||
|
await new Promise(resolve => realImmediate(resolve));
|
||||||
|
}
|
||||||
|
return { query };
|
||||||
|
}
|
||||||
|
|
||||||
|
describe("maintenance fake clock and database connection startup", () => {
|
||||||
|
const cleanup: Array<() => unknown> = [];
|
||||||
|
afterEach(async () => {
|
||||||
|
vi.useRealTimers();
|
||||||
|
for (const step of cleanup.splice(0).reverse()) await step();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("fires the driver connect timeout when the clock advances during connection startup", async () => {
|
||||||
|
const upstream = await startHeldPostgres(); cleanup.push(() => upstream.stop());
|
||||||
|
const tracker = trackDatabaseConnectionStartup(upstream.port); cleanup.push(() => tracker.dispose());
|
||||||
|
vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] });
|
||||||
|
const { query } = await startConnection(upstream.port, tracker, upstream, cleanup);
|
||||||
|
vi.advanceTimersByTime(30_000);
|
||||||
|
upstream.release();
|
||||||
|
await expect(query).rejects.toMatchObject({ code: "CONNECT_TIMEOUT" });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("advances the clock only after every database connection has finished startup", async () => {
|
||||||
|
const upstream = await startHeldPostgres(); cleanup.push(() => upstream.stop());
|
||||||
|
const tracker = trackDatabaseConnectionStartup(upstream.port); cleanup.push(() => tracker.dispose());
|
||||||
|
vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] });
|
||||||
|
const { query } = await startConnection(upstream.port, tracker, upstream, cleanup);
|
||||||
|
expect(tracker.starting).toBe(1);
|
||||||
|
realTimeout(() => upstream.release(), 50);
|
||||||
|
const ticks: number[] = [];
|
||||||
|
await tracker.advanceAfterStartup(ms => { ticks.push(ms); expect(tracker.starting).toBe(0); vi.advanceTimersByTime(ms); }, 30_000, 5_000);
|
||||||
|
expect(ticks).toEqual([30_000]);
|
||||||
|
await expect(query).resolves.toHaveLength(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("reports an unreadable startup reply once and keeps later advances usable", async () => {
|
||||||
|
const upstream = await startHeldPostgres(Buffer.from([0x52, 0, 0, 0, 0])); cleanup.push(() => upstream.stop());
|
||||||
|
const tracker = trackDatabaseConnectionStartup(upstream.port); cleanup.push(() => tracker.dispose());
|
||||||
|
const client = createConnection({ host: "127.0.0.1", port: upstream.port });
|
||||||
|
let clientError: Error | undefined;
|
||||||
|
client.on("error", error => { clientError = error; }); cleanup.push(() => client.destroy());
|
||||||
|
await once(client, "connect");
|
||||||
|
client.write(Buffer.from([0, 0, 0, 8, 0, 3, 0, 0]));
|
||||||
|
const deadline = performance.now() + 5_000;
|
||||||
|
while (upstream.held.length === 0) {
|
||||||
|
if (clientError) throw clientError;
|
||||||
|
if (performance.now() > deadline) throw new Error("PostgreSQL stub did not receive the startup message");
|
||||||
|
await new Promise(resolve => realTimeout(resolve, 5));
|
||||||
|
}
|
||||||
|
expect(tracker.starting).toBe(1);
|
||||||
|
upstream.release();
|
||||||
|
await expect(tracker.waitUntilStarted(5_000)).rejects.toThrow("Database connection startup could not be observed");
|
||||||
|
expect(tracker.starting).toBe(0);
|
||||||
|
await expect(tracker.waitUntilStarted(5_000)).resolves.toBeUndefined();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("uses the PostgreSQL default port when the database URL omits it", () => {
|
||||||
|
expect(databasePort("postgres://maintenance@127.0.0.1/maintenance_test")).toBe(5432);
|
||||||
|
expect(databasePort("postgres://maintenance@127.0.0.1:6543/maintenance_test")).toBe(6543);
|
||||||
|
});
|
||||||
|
});
|
||||||
Reference in new issue
Block a user