Files
PaperClipAI/server/src/__tests__/idle-admission.test.ts
T
DottaandPaperclip 006d5691a6 feat(server): support volume-free web server handoff (#18518)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its web server also owns controller admission and user commands.
> - A rolling deployment starts a replacement before the old server
exits.
> - Shared database state alone does not preserve local agent files,
browser chunks, or in-process commands.
> - This pull request adds an opt-in ephemeral server contract and a
local overlap test.
> - The benefit is a web server that can relinquish traffic and local
files without losing acknowledged work.

## Linked Issues or Issue Description

Refs #18403, #18327, #18335, #18341.

**Current behavior**

The control-plane web server still has local agent-file authority and
local command targets. A new container can miss saved bytes or send a
command to the wrong process. An existing browser tab also needs chunks
from its original build.

**Proposed behavior**

Eligible servers share only PostgreSQL, object storage and stable
secrets. A replacement waits for readiness. The old server drains
admitted requests, keeps owner commands available through handoff, and
releases controller leases. Explicit maintenance adopts existing files
before the volume is removed.

**Reason and benefit**

This is the application prerequisite for rolling app-only deployments.
Schema migrations and permanent multi-replica operation are outside this
change.

## What Changed

- Store agent-file manifests in PostgreSQL and immutable bytes in S3.
Treat local roots as disposable projections. Preserve existing
permission checks and edit conflicts.
- Add an explicit migration, eligibility and object-retention command to
the production runtime bundle.
- Add `/api/ready`, admitted-request drain, browser restart signaling
and retained immutable UI chunks.
- Carry transaction-aware work hints across controllers during handoff
and active execution. Keep listeners closed when idle.
- Deliver durable Stop and runtime responses to the execution owner.
Recheck request lifecycle and access before response dispatch.
- Reject local-only configuration in ephemeral mode. Keep the feature
disabled for existing installations by default.
- Add an isolated PostgreSQL/S3/proxy/browser deployment harness with
independent server homes and retained failure evidence.

## Verification

- All CI checks pass on `ce17b95019d3cf1cabd357d19b6d37f96782cc5c`:
workspace typecheck, build, Runner checks, server/general tests and
browser shards. Greptile reviewed the runtime changes at 5/5 with all
findings resolved; the final commit updates maps and the plan.
- Local targeted checks passed: coordinator 27; Stop/recovery 33;
PostgreSQL command delivery 6; route/workspace regressions 8; bootstrap
Stop 1; restart recovery 75. PostgreSQL command tests simulate the
provider edge.
- Local browser proof passed 91 reads and 19 saves; reconnect took
1,013ms. The tab opened Audit from S3 after deletion of A's whole
home/build. This uses compiled workspace code and local-trusted auth,
not released containers or live agents.
- Local workspace typecheck/build passed; current server and runtime
bundle passed. The earlier broad local suite was stopped after three
hours and is not a passing receipt. Its retained failures and isolated
reruns are documented in the plan.
- Stacked #18526 adds stricter graceful-exit proof, an unready
replacement control, live Runner qualification and a Stop handoff
correction. Hosted enablement remains gated on those results and Cloud
#1096. No hosted stack has been enabled.

## Risks

- Apply migration 0422 before enabling the feature. Existing data must
pass each owner's migration and eligibility checks. Do not discard a
volume based on a single migration page.
- Durable manifests remain authoritative after the opt-in is disabled.
Rolling back to a binary that ignores them is unsupported.
- Cold directory restoration and complete shutdown timing still need
qualification before enabling a hosted stack.
- Retained UI assets must cover the supported browser and rollback
window. Cleanup is explicit; no new idle maintenance polling is
introduced.
- This change does not claim production zero downtime or authorize a
fleet-wide enablement.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and
context-window limit are not exposed in this session. Used reasoning,
code editing, shell execution and browser automation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-01 01:58:54 -06:00

170 lines
7.5 KiB
TypeScript

import express from "express";
import request from "supertest";
import { request as httpRequest } from "node:http";
import { createServer as createViteServer } from "vite";
import { afterEach, describe, expect, it } from "vitest";
import { drainHttpRequestWork, idleAdmissionMiddleware, trackIdleRequestHandlers } from "../middleware/idle-admission.js";
import { idleWorkSnapshot, startTaskDrain, stopTaskDrain, readTaskDrain, beginIdleTrackedWork } from "../services/task-admission.js";
function deferred() {
let resolve!: () => void;
const promise = new Promise<void>(done => { resolve = done; });
return { promise, resolve };
}
function app() { const app = express(); app.use(idleAdmissionMiddleware); return app; }
afterEach(() => stopTaskDrain());
describe("idle admission", () => {
it("preserves legacy drains, but blocks all new work during an owned idle hold", async () => {
const server = app();
server.all("/{*path}", (_req, res) => res.sendStatus(204));
trackIdleRequestHandlers(server);
startTaskDrain({ ttlMs: 60_000 });
await request(server).post("/api/issues").expect(204);
startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
for (const path of ["/api/issues", "/api/chat-webhooks/test", "/mcp", "/api/auth/callback"]) {
await request(server).get(path).expect(503).expect("Retry-After", "1");
await request(server).post(path).expect(503);
}
await request(server).get("/api/health").expect(204);
await request(server).get("/api/instance/task-drain/").expect(204);
await request(server).post("/api/instance/task-drain").expect(204);
stopTaskDrain();
await request(server).post("/api/issues").expect(204);
expect(idleWorkSnapshot().active).toBe(0);
});
it("counts a nested async handler after it sends a response", async () => {
const server = app();
const router = express.Router();
const done = deferred();
router.post("/work", async (_req, res) => { res.sendStatus(202); await done.promise; });
server.use("/api", router);
trackIdleRequestHandlers(server);
await request(server).post("/api/work").expect(202);
expect(idleWorkSnapshot().active).toBe(1);
let drained = false;
const drain = drainHttpRequestWork(server, new AbortController().signal).then(() => { drained = true; });
await new Promise(resolve => setImmediate(resolve));
expect(drained).toBe(false);
startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
await request(server).post("/api/work").expect(503);
done.resolve();
await drain;
expect(drained).toBe(true);
expect(idleWorkSnapshot().active).toBe(0);
});
it("tracks ordinary middleware with non-router stack metadata", async () => {
const server = app(), done = deferred();
const middleware = Object.assign(async (_req: express.Request, res: express.Response) => {
res.sendStatus(202); await done.promise;
}, { stack: { name: "middleware metadata" } });
server.use(middleware);
expect(() => trackIdleRequestHandlers(server)).not.toThrow();
await request(server).get("/work").expect(202);
expect(idleWorkSnapshot().active).toBe(1);
done.resolve();
await new Promise(resolve => setImmediate(resolve));
expect(idleWorkSnapshot().active).toBe(0);
});
it("supports real Vite Connect route metadata without losing async work", async () => {
const vite = await createViteServer({ configFile: false, appType: "custom",
server: { middlewareMode: true, watch: null }, optimizeDeps: { noDiscovery: true, include: [] } });
const server = app(), done = deferred();
try {
server.use(vite.middlewares);
server.post("/work", async (_req, res) => { res.sendStatus(202); await done.promise; });
expect(() => trackIdleRequestHandlers(server)).not.toThrow();
await request(server).post("/work").expect(202);
expect(idleWorkSnapshot().active).toBe(1);
done.resolve();
await new Promise(resolve => setImmediate(resolve));
expect(idleWorkSnapshot().active).toBe(0);
} finally { await vite.close(); }
});
it("tracks real Vite middleware without confusing Connect mount paths with Express routes", async () => {
const { createServer } = await import("vite");
const vite = await createServer({
configFile: false,
appType: "custom",
server: { middlewareMode: true, hmr: false, watch: null },
optimizeDeps: { noDiscovery: true, include: [] },
});
const server = app(), done = deferred();
vite.middlewares.use("/vite-work", async (_req, res) => {
res.statusCode = 202;
res.end();
await done.promise;
});
server.use(vite.middlewares);
try {
expect(() => trackIdleRequestHandlers(server)).not.toThrow();
await request(server).get("/vite-work").expect(202);
expect(idleWorkSnapshot().active).toBe(1);
startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
await request(server).get("/vite-work").expect(503);
done.resolve();
await new Promise(resolve => setImmediate(resolve));
expect(idleWorkSnapshot().active).toBe(0);
} finally {
done.resolve();
await vite.close();
}
});
it("does not mistake a client disconnect for completed accepted work", async () => {
const server = app();
const entered = deferred(), done = deferred(), closed = deferred();
server.post("/work", async (_req, res) => { res.once("close", closed.resolve); entered.resolve(); await done.promise; /* client has left */ });
trackIdleRequestHandlers(server);
const listener = server.listen(0, "127.0.0.1");
await new Promise<void>(resolve => listener.once("listening", resolve));
const address = listener.address() as { port: number };
const client = httpRequest({ host: "127.0.0.1", port: address.port, path: "/work", method: "POST" });
client.on("error", () => undefined);
client.end();
try {
await entered.promise;
client.destroy();
await closed.promise;
startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
expect(idleWorkSnapshot().active).toBe(1);
done.resolve();
await new Promise(resolve => setImmediate(resolve));
expect(idleWorkSnapshot().active).toBe(0);
} finally {
done.resolve(); client.destroy();
await new Promise<void>(resolve => listener.close(() => resolve()));
}
});
it("settles errors through async Express error middleware", async () => {
const server = app(), done = deferred();
server.get("/work", async () => { throw new Error("test"); });
server.use(async (_err: Error, _req: express.Request, res: express.Response, _next: express.NextFunction) => {
res.sendStatus(500); await done.promise;
});
trackIdleRequestHandlers(server);
await request(server).get("/work").expect(500);
expect(idleWorkSnapshot().active).toBe(1);
done.resolve(); await new Promise(resolve => setImmediate(resolve));
expect(idleWorkSnapshot().active).toBe(0);
});
it("uses unique owners even for simultaneous starts and expires the hold", () => {
const first = startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
const second = startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
expect(second.ownerId).not.toBe(first.ownerId);
expect(readTaskDrain(second.expiresAt!)).toBeNull();
});
it("records even work that starts and finishes between observations", () => {
const before = idleWorkSnapshot();
const done = beginIdleTrackedWork(); done(); done();
expect(idleWorkSnapshot()).toEqual({ active: before.active, generation: before.generation + 2 });
});
});